Dudley Castle was a Norman castle and was built in the Midlands by William Fitz-Ansculf just after the Conquest in 1071. The land had been held by Edwin, Earl of Mercia in the time of Edward the Confessor and he was allowed to retain his lands after the Conquest until he rebelled against William the Conqueror and was killed. Dudey was held by supported of Matilda against King Stephen until 1138 when Stephen attacked and destroyed the castle and the surrounding area. In 1165, Gervase, a son of the Castle's owner founded a priory at Dudley.