I'm thinking it's like this- Harry spent so much time at work, as an auror, and part-time teacher at Hogwarts (according to his Wiki page) that he was never there for his kids. He missed trips to Plalform 9 & 3/4, birthdays, the occasional christmas. They began to resent him, and eventually decided to take what he loves most- his wives- from him as revenge.