Charles Spencer, the brother of the late Princess Diana, spoke out this week to talk about the trauma he and his sister shared as children when their parents split up in 1969.

“Diana and I had two older sisters who were away at school, so she and I were very much in it together and I did talk to her about it,” Spencer told The Sunday Times. The two older sisters that he was referring to were Lady Sarah McCorquodale, 65, and Jane Fellowes, Baroness Fellowes, 63.

While Spencer had fond memories of the parenting of his father, John Spencer, the 8th Earl Spencer, he found his mother Frances Shand Kydd to be more problematic.

“Our father was a quiet and constant source of love, but our mother wasn’t cut out for maternity. Not her fault, she couldn’t do it,” he said. “While she was packing her stuff to leave, she promised Diana [then aged five] she’d come back to see her. Diana used to wait on the doorstep for her, but she never came.”

Kydd passed away in 2004, seven years after Diana was killed in a car crash at the age of 36.

Spencer went on to say that in order to do “very profound work on [his] unhappy childhood,” he has been “in and out of therapy for 20 years.” Though he has found this work to be “agonizing and horrible,” he noted that “the result has been cathartic.”

“Coming out the other side has been good,” he said. “The one thing I’ve learned through all the stuff I’ve tackled is that very few people set out to be destructive.”

Until he met his third wife Karen Villeneuve, who he married in 2011, Spencer said that he had “a predisposition for rescuing people” because of his rough upbringing.

“We got engaged quickly and I’m sure people thought, ‘Here he goes again,’ but I was confident,” Spencer said.

Villeneuve said that she feels similarly to her husband.

“I’ve been a first wife and a third and — trust me on this — you really want to be the third, because men are so motivated at this point,” she explained. “They are really, really paying attention. The most appealing thing about Charles for me was how willing he was to work on himself, and continue to do so.”

“I love that we are a supportive partnership,” Villeneuve added. “We’re both ambitious and there’s been a lot of give and take.