Colchester has been an important military garrison since the Roman era. The Colchester Garrison is currently home to the 16th Air Assault Brigade. The Army's only military corrective training centre, known colloquially within the forces and locally as "The Glasshouse" after the original military prison in Aldershot,[36] is in Berechurch Hall Road, on the outskirts of Colchester.[37] The centre holds servicemen and women from all three services who are sentenced to serve periods of detention.

From 1998 to 2008, the garrison area of the town underwent massive redevelopment. A lot of the Ministry of Defence land was sold for private housing development and parts of the garrison were moved. Many parts of the garrison now stand empty awaiting the second phase of the development.

Since 2006, Colchester has been one of 12 places in the UK where Royal Salutes are fired to mark Royal anniversaries and visits by foreign heads of state. From 2009, these salutes have taken place in Castle Park.
 

Colchester houses several museums. The Castle Museum, found within Colchester Castle, features an extensive exhibit on Roman Colchester. Nearby are Hollytrees Museum, a social history museum with children's exhibits in the former home of Charles Gray, and the town's Natural History Museum, located in the former All Saints' Church. Tymperley's Clock Museum, located in the town centre in a 15th-century timber-framed house, once home to William Gilbert, now houses the Bernard Mason clock collection.

The town's link with football had begun with the amateur club Colchester Town, formed in 1867 and dissolved in 1937. They were succeeded by professional club Colchester United, who compete in the Football League One and play home games at Colchester Community Stadium. Founded in 1937, the club entered the Football League in 1950. Colchester United Ladies play in the FA Women's Premier League Southern Division. Other sports teams based in the town include Colchester Rugby Football Club, Colchester Gladiators American Football Club, Colchester Weight Lifting Club and Colchester & East Essex Cricket Club. Essex County Cricket Club play some of their home games at Castle Park Cricket Ground, home of Colchester & East Essex.

Sports facilities in Colchester include the sports centre, Colchester Leisure World, Colchester Garrison Athletics Stadium (a co-operative facility used by both the army and civilian population), and a skatepark.
 
Colchester Zoo is a large zoo based on the outskirts of the town.
 

Medieval Colchester's main landmark is Colchester Castle, which is an 11th-century Norman keep, and built on top of the vaults of the old Roman temple. There are notable medieval ruins in Colchester, including the surviving gateway of the Benedictine abbey of St. John the Baptist (known locally as "St. John's Abbey"), and the ruins of the Augustinian priory of St. Botolph (known locally as "St. Botolph's Priory"). Many of Colchester's parish churches date from this period.

Colchester developed rapidly during the later 14th century as a centre of the woollen cloth industry, and became famous in many parts of Europe for its russets (fabrics of a grey-brown colour). This allowed the population to recover exceptionally rapidly from the effects of the Black Death, particularly by immigration into the town.

Between 1550 and 1600, a large number of weavers and clothmakers from Flanders emigrated to Colchester and the surrounding areas. They were famed for the production of "Bays and Says" cloths which were woven from wool and are normally associated with Baize and Serge although surviving examples show that they were rather different from their modern equivalents. An area in Colchester town centre is still known as the Dutch Quarter and many buildings there date from the Tudor period. During this period Colchester was one of the most prosperous wool towns in England, and was also famed for its oysters. The old Roman wall runs along Northgate Street in the Dutch Quarter.

In 1648, during the Second English Civil War, a Royalist army led by Lord Goring entered the town. A pursuing Parliamentary army led by Sir Thomas Fairfax and Henry Ireton surrounded the town for eleven and a half weeks, a period known as the Siege of Colchester. It started on the 13 June. The Royalists surrendered in the late summer (on the 27th August Lord Goring signed the surrender document in the Kings Head Inn) and Sir Charles Lucas and Sir George Lisle were executed in the grounds of Colchester Castle. A small obelisk marks the spot where they fell.

Daniel Defoe mentions in A tour through England and Wales that the town lost over 5000 people to the plague in 1665, "more in proportion than any of its neighbours, or than the city of London". By the time he wrote this in 1722, however, he estimated its population to be around 40,000 (including "out-villages").

In 1884, the town was struck by the Colchester earthquake, estimated to have been 4.7 on the Richter Scale causing extensive regional damage.

In the early 20th century Colchester lobbied to be the seat for a new Church of England diocese for Essex, to be split off from the existing Diocese of Rochester. The bid was unsuccessful, with county town Chelmsford forming the seat of the new diocese. The University of Essex was established on the outskirts of the town at Wivenhoe Park in 1961. The £22.7M eight-mile A120 Colchester Eastern Bypass opened in June 1982.