A former royal protection officer has just spoken out to give Prince Harry and Meghan Markle a warning about life in Los Angeles, California that may cause them to think twice about their decision to move their permanently.

Though Harry and Meghan appeared to be eager to relocate to her hometown of L.A., ex-royal protection officer Simon Morgan just warned that they will undoubtedly have some struggles living there when it comes to privacy.

“I think a lot of it will depend on how they actually engage with people, both in terms of the ordinary person in the streets and the media, and how much access they allow with that,” Morgan told Us Weekly. “The more you withdraw, the more people want to know what you’re up to. There just becomes that fascination.”

He went on to say that the public will be fascinated by Harry and Meghan, regardless of what they do.

“Harry, you know, he is still a senior member of the British Royal family. Both in terms of … bloodline and position, or maybe not entitled at the moment. So therefore there is still a fascination around that, certainly with the [former] duchess, you know,” Morgan explained. “If she does go back to work in that particular sector, then there will always be an interest in what she’s doing, because it would be part and parcel of the very nature of what she’s doing.”

He added that the protection team for the couple will undoubtedly have their work cut out for them.

“The work is always … on the protection team,” he said, adding that he was stunned to see them leave the U.K. because the country is a “secure environment” in which they had “a regular routine and a footprint.”

“But then, you know, to make the move to L.A. kind of puts you very much in the spotlight,” Morgan said, describing Los Angeles as a “constant” and “24/7” environment.

“So I think, yeah — from a protection perspective, you know, it’s not an ideal choice,” he continued. “But there again, you know, people want to lead their lives and you as the protection team just have to work around that.”

Harry and Meghan have been living at the $18 million mansion of the Hollywood mogul Tyler Perry with their 1 year-old son Archie over the past few weeks while they reportedly have been looking for their own home in the area.