“This will be unwelcome reputational damage to the institution,” a source close to Charles said to the Times of London.

Even though Charles “loves his brother and has the ability to have sympathy for the slings and arrows that his brother endures,” it was decided “long ago concluded that it is probably an unsolvable problem.”

“This will probably further strengthen in the prince’s mind that a way back for the duke is demonstrably not possible, because the specter of this [accusation] raises its head with hideous regularity.”

As we previously reported, Prince Andrew released a statement about stepping down from the Royal family.

RELATED: Queen Elizabeth Faces Backlash For Wishing Her Son Prince Andrew A Happy Birthday

“It has become clear to me over the last few days that the circumstances relating to my former association with Jeffrey Epstein has become a major disruption to my family’s work and the valuable work going on in the many organisations and charities that I am proud to support,” he said in the statement at the time.

“Therefore, I have asked Her Majesty if I may step back from public duties for the foreseeable future, and she has given her permission.”

“I continue to unequivocally regret my ill-judged association with Jeffrey Epstein. His suicide has left many unanswered questions, particularly for his victims, and I deeply sympathise with everyone who has been affected and wants some form of closure. I can only hope that, in time, they will be able to rebuild their lives. Of course, I am willing to help any appropriate law enforcement agency with their investigations, if required.”