As a Hollywood A-list star, Matthew McConaughey may seem like he has it all, but the actor himself believes that being affluent presents its own set of problems that most people may not realize.

Fox News reported that McConaughey and his wife Camila Alves are the parents of three children: Levi, 11, Vida, 10 and Livingston, 7. When asked if his success has changed the way he parents his kids, McConaughey explained that being an affluent father means that he is put in the position of making the decision not to give his kids everything they want.

“Sometimes loving your kids means giving them just what they want. Other times it means tough love. Affluent people can give their kids everything they want, but they’re not usually going to get what they need,” McConaughey said. “Loving a child is a lot harder if you really give a damn. ‘No’ takes a lot more energy. It’s a lot easier to say ‘yes.’

McConaughey and Alves have made helping children a priority in their lives ever since they started the Just Keep Livin’ Foundation back in 2008. The organization aims to give kids a place to practice physical activities and focuses on teaching them gratitude through service projects. For McConaughey, it was never a question that his foundation would focus on children.

“I always knew I wanted it to be kids, because I wanted the entire foundation to provide prevention before we needed a cure,” he told Town And Country Magazine. “High school’s the last place you can catch somebody and get them on the right track, before they no longer have to listen to a teacher or parent. If they’re screwing up in high school and do some of the same stuff after they graduate, they may go to jail—or worse.”

When asked if he could relate to youths on the cusp of making the wrong decisions that would impact their lives forever, McConaughey said he could not.

“No. I was very fortunate. I grew up upper-middle-class, and I had a roof over my head, food on the table, parents that loved and supported me, a car that was paid for, and a job, and I made straight A’s,” he said.

McConaughey added that despite this, his upbringing wasn’t the most stable given the fact that his parents divorced three times and remarried twice.

“My parents were probably just saying, ‘I’ve had enough of you for a while. I’ll see you in a year or so,'” he said. “It wasn’t a perfect upbringing, but I was in a position where I was never going to be a victim.”

Alves revealed that her parents did the exact same thing, making her the perfect partner for McConaughey.

“I would call them two people extremely in love who cannot find a way to communicate,” she said. “It’s all fine now. But when I was growing up, in Brazil, my mom would always extol to me and my brother the importance of giving back.”

Though McConaughey may seem selfless when it comes to parenting and charity work, he insists that he is still a selfish hedonist.

“Your hypothesis is backward. I’ve always been a hedonist, and part of having a foundation is putting a capital H on hedonism,” McConaughey explained. “Being able to give back in ways is a selfish endeavor. It feels good to me to see a smile or hear a young person say thank you. That’s called a selfless act? I call it a selfish act.”