Tamworth Castle is a typical Norman motte and bailey castle. Its sandstone walls and superb herringbone wall are thought to date from the 1180's. They replaced a wooden tower on the present artificial mound (motte) with an enclosure (lower bailey) which would have been constructed shortly after the Norman Conquest. The stone Castle is a polygonal Shell-Keep with a square tower set into its walls. The lower bailey would have been bordered by a timber palisade with a ditch outside. The footpath over the Herringbone Wall, all that survives of the "Curtain Wall" of the bailey, leads up to the Castle Keep and down to the excavations of the late 13th century Gatehouse. Only the lower portion of the double tower of the gatehouse remains; the moat was dry, and the drawbridge raised and lowered over a stone causeway.

Numerous additions and alterations have been made to the Castle by succeeding generations of owners. The oldest surviving section within the Shell-Keep, apart from the Tower itself, is the north wing with its 13th century arched doorway. The Banqueting Hall added in the early 15th century, and the Warder's Lodge at the entrance to the Courtyard (upper bailey) is Tudor. With the construction of the South Wing in the early 17th century, the 12th century Keep now housed an "H" plan country gentlemen's residence. The Castle was much neglected in the 18th century, but between 1783 and 1811 extensive alterations were made which included the removal of the bay windows, tall chimneys and characteristic gable-roofed attic storey and the re-facing of the exterior of the entire South Wing.

There has been some confusion in the past over the identity of the castle' s first Lord. There is evidence that before the reign of King Stephen ( 1135 - 1154 ), it was held by both Robert de Despencer and Robert Marmion. Robert de Despencer evidently left no surviving legitimate son when he died, either a daughter or a niece married into the Marmion family. But as the name Despencer means Steward, (a feudal official), it is now believed that, they may have been one and the same person. Marmion had performed the office of Champion to William the Conqueror in Normandy and the gift of Tamworth Castle and the Manor of Scrivelsby in Lincolnshire required him to render service as Royal Champion to the King of England. He was to come "to the Coronation of the Lord King, completely armed with royal arms of the livery of the Lord King, and sitting upon the principal royal war-horse, and oppose himself against any person who should gainsay the Royal Champion".

The Marmions held the Castle until 1291 when Philip, the last and eighth Baron died. When Lady Jane Marmion died without an heir, the Castle was granted by Edward I to her niece's husband, Sir Alexander de Freville. He was the last holder of the Castle to perform the office of Royal Champion; he appeared in this role at the Coronation of Edward III in 1327. The youngest daughter of the last of the Marmions had been granted her father's estate at Scrivelsby, and her successors claimed successfully that the office of Royal Champion was attached to that Manor and not to Tamworth Castle.

In 1423 the male line of Freville failed and the Castle passed to Sir Thomas Ferrers of Groby, who had married into the Freville family. From the Ferrers, the Castle passed by marriage to the Shirleys of Chartley in 1688; again by marriage to the Comptons, Earls of Northampton in 1715, and finally to the Townshends of Raynham in 1751. The Castle remained in the Townshend family until 1897 apart from a brief period in the early 19th century when it came into the possession of a London auctioneer, Mr. John Robins. In 1897 the Castle was purchased by Tamworth Corporation for the sum for £3000, and was formally opened to the public on the 22nd May 1899.