Dudley Castle was originally a Norman castle and was built in the Midlands by William Fitz-Ansculf just after the Norman Conquest in 1071. The land had been held by Edwin, Earl of Mercia, in the time of Edward the Confessor and he was allowed to retain the lands after the Conquest. Edwin was later killed after rebelling against William the Conqueror. During the civil war between Matilda and King Stephen the castle was held by a supporter of Matilda until 1138 when Stephen attacked and destroyed the castle and the surrounding area. In 1165 Gervase, a son of the Castle's owner, founded a priory at Dudley.