The oldest historical record of a building at the location of the current castle dates to 1391. In that year, the De Haar family received the castle and the surrounding lands as a fiefdom from Hendrik van Woerden. The castle remained in the ownership of the De Haar family until 1440, when the last male heir died childless. The castle then passed to the Van Zuylen family. In 1482, the castle was burned down and the walls were destroyed, except for the parts that did not have a military function. These parts probably were incorporated into the castle when it was rebuilt during the early 16th century. The castle is mentioned in an inventory of the possessions of Steven van Zuylen from 1506, and again in a list of fiefdoms in the province Utrecht from 1536. The oldest image of the castle dates to 1554 and shows that the castle had been largely rebuilt by then. After 1641, when Johan van Zuylen van de Haar died childless, the castle seems to have gradually fallen into ruins. The castle escaped total destruction by the French during the Rampjaar 1672.

In 1801 the last Catholic van Zuylen in the Netherlands, the bachelor Anton-Martinus van Zuylen van Nijevelt (1708–1801), bequeathed the property to his cousin Jean-Jacques van Zuylen van Nyevelt (1752–1846) of the Catholic branch in the Southern Netherlands. [Wikipedia]