Speculation is mounting that Prince Harry is having regrets about stepping down as a senior member of the British royal family and moving to Los Angeles, California with his wife Meghan Markle. It has just been reported that Harry is struggling with the fact that he is both friendless and jobless in his new home.

“He has a lot of friends in the military community in the UK and of course he misses them. This is a very strange time for us all, but I think Harry is missing having a structure to his life right now,” a source close to Harry told Vanity Fair. “He doesn’t have friends in L.A. like Meghan and he doesn’t have a job. So at the moment he’s a bit rudderless, but it won’t always be like this, and he knows that.”

Harry and Meghan shocked the world earlier this year when they announced that they were stepping down as senior members of the British royal family. After making this announcement, they initially moved to Vancouver, Canada before relocating to Meghan’s hometown of L.A. While Meghan is used to life in Hollywood after spending years trying to make it there as an actress, reports had already been coming in that Harry was struggling.

“He has been telling friends that he still can’t believe this has happened,” another source close to Harry told the U.K. Telegraph. “He can’t believe his life has been turned upside down.”

The insider went on to say that Harry is particularly missing life in the Army, as he had to give up his military appointments when he stepped down as a royal.

“Harry has told friends he is really missing the Army as well his military appointments,” the source said. “He misses the camaraderie of being in the forces. He was in a happy place when he was serving in the Army, then he met Meghan and since then life has been great. But I don’t think he foresaw things turning out quite as they did.”

The insider concluded by claiming that Harry “doesn’t blame Meghan” for all of the changes in his life, adding that “there is just a sense that he might have been better protected if he was still in the Army.”