Medieval diplomas mention Szász (meaning Saxon in Hungarian) village from the 13th century, which became the most significant marketplace of the region and the center of the Mecsek ridge fiefdoms of the diocese of Pécs in the 16th century. Bishop Valentine Alsáni of Pécs built the Saint John the Baptist’s chapel next to the marketplace along his manor house. This multi-level building soon received a stone walled garden. After the death of King Louis the Great in 1382, bishop Valentine – already a cardinal at the time – converted the manor into a castle fortified with towers. The garden became a castle yard where a robust gate tower operated as an entrance into the building. The old staircase tower was converted into a chapel, and a new staircase tower was built into the middle of the façade facing the castle yard. A little later, the kitchen was built in between the double castle walls. After the death of Bishop Valentine, John Albeni and later his brother Henry were appointed to the bishop’s seat. They continued the development of the building too, a new multi-level palace building has been erected in the castle yard, and the walls have been strengthened as well. In 1439 a diploma already mentions the site as a proper castle. During the 15 century, new walls have been erected around the castle and cylinder shaped towers rose on the corners. Bishop Sigismund Ernuszt of Pécs built a new palace wing above the old kitchen in the eastern section of the castle yard in the 1490s with ornate renaissance windows. The old gate tower was converted into a keep while a new entrance was cut on the inner wall of the castle. In 1540 Simon Athinai occupied the castle for Queen Isabelle, but in 1543 the defenders gave up the castle to the Ottoman army, who created a bureaucratic centre, a nahije in the captured edifice. The outer wall, strengthened by eight towers, was built in the 16th century. The destruction of the castle occurred in the late 17th century, when Christian armies defeated the Turks. In the 18th century the ruinous building was used as basis for the new church and episcopal manor. Some parts of the building still stand, and many of its walls are the original medieval ones in their core, in our time the castle is being renovated and rebuilt to once again show some of its once existed glory.