As with many organisations in Highcliffe the Golf Club has a close association with Highcliffe Castle. The person responsible for the first Highcliffe Mansion was a former Prime Minister, John Stuart, The Third Earl of Bute. He commissioned Robert Adam to build him a house that would "command the fairest outlook in England" and he called it High Cliff . Later, Lord Bute's son, Sir Charles Stuart, found the expense and annoyance of landslips so intolerable that he sold the mansion.When Sir Charles Stuart's son returned from his first appointment as Ambassador in Paris in 1824 he adapted the old High Cliff House, which had been virtually demolished, into what is now known as Highcliffe Castle.
Having been impressed by the beauty of french chateaux, he bought one Les andelys in Normandy, which had been built in the 16th Century for the Bourbon family. He then had it taken down, transferred in barges across the channel, hauled up the cliff and re-erected (to adapt the existing building).
The carved panelling and the great stained glass windows were very beautiful and the tall elegantly sculpted spires made it look like a fairy castle.