The castle itself was first mentioned in the second half of the 14th century and was acquired by the diocese about a hundred years later. The building and grounds became the bishop's main summer residence in the 18th century. However, after the French Revolution, the entire complex was sold as national property and converted into an annex of the major seminary. In 1906, the definitive separation of church and state led to the closure of the seminary. Like its counterpart a few kilometers away (the former minor seminary – see Hôpital Plaza), the buildings were used as a military hospital during the First World War. In the 1960s, the buildings were used for   summer camps. Since the early 1980s, it has served as a residence for the Pain de Vie community, and finally for the Charles de Jesus Brotherhood from around 2007 to February 2014. After standing empty for several years, the building was purchased in 2016 by a resident of the municipality, who intends to give it a socio-cultural purpose.