In the wake of the 2016 referendum, the idea that 'imperial nostalgia' motivated the Leave vote became a staple of academic commentary. Yet such claims suffer from four important flaws. They are usually polemical in character; they suggest, at least implicitly, that only Leave voters are subject to imperial patterns of thought; they fail to differentiate between Commonwealth and imperial loyalties; and they conflate 'nostalgia' with 'amnesia'. This article deploys a longer historical perspective to offer a new reading of the relationship between Brexit and Empire, focusing on the ways in which empire is remembered and articulated. It shows how imperial modes of thought shaped the views of pro-Europeans, as well as their opponents, and explores the changing uses of the Commonwealth. It pays particular attention to the views of Black and Asian voters - a cohort that disrupts many conventional assumptions about Brexit - and shows how empire was excised from histories of 'Global Britain', in a manner that minimises the significance of decolonisation. As such, it presents the legacies of empire, not as a disorder to which only half the population is subject, but as a common cultural inheritance through which all sides of the European debate think and argue.