In urbex circles, people sometimes refer to this prison as 'Frauenknast' or 'Women's Prison'. This misconception arose from the urbex name 'Hinter Gittern: Frauenknast', which was inspired by a German television series with the same name. In reality, the prison housed both male and female inmates. In fact the population was even predominantly male.

This originally medieval moated castle was rebuilt several times. The oldest sources date back to the year 860. During that period, the castle was successively inhabited by the counts and archbishops who governed this region. Around 1200, the counts had the castle rebuilt into a typical late Romanesque moated castle, covering an area of ​​almost 12,000 m². The four-winged core castle forms a rectangle of 62 x 42 m (2604 m²) and was surrounded by a moat. The 35-meter-high tower in the north wing and the tall forebuilding in the northeast date back to that period. The tower housed a tower watchman who sounded the horn when danger threatened or an enemy became visible.

From 1375 onwards, the castle was restored after a fire and gradually expanded into a closed, four-winged complex. In 1522, the "Hohes Haus" (Holy House) was built, featuring a large hall and the typical late Gothic window frames.

Around 1555, a new extension was added, completed with the Renaissance bay window on the courtyard, where an inscription refers to the year 1555. This is where the actual urbex name 'Prison 1555' originates.

Between 1640 and 1860, the castle came into the hands of another noble family, who used it as a second residence.

From 1861 to 1991, the building served as a prison and penitentiary. After the founding of the state, it was expanded in 1873 by adding a third floor to the south and west wings and an extension on the north side. 282 individual cells were created. At that time, a prisoner climbed the tower every day to wind the clock.