The prison, as the centrepiece of our justice system, is seen as fixed, unassailable, necessary. Yet, globally, COVID-19 has opened safety valves in previously impregnable, impermeable institutions, from maximum-security prisons to immigration detention. Some Australian prisons - bursting at record-high levels with our most dispossessed, wounded and reviled - have created space in ways that bring cautious hope. Might the crisis license/allow a reimagining of how we might do justice differently? The pandemic has given us pause to consider what 'justice' means, what we expect of our penal system, and to rethink our justice machinery. As never-imprisoned and formerly imprisoned co-authors, we reflect critically on COVID-19 inspired commentary, scholarly arguments for decarceration, and the lived experience of criminalisation and imprisonment. We weigh the social and economic costs of a system of punishment that dehumanises, reproduces violence, and causes countless harms to countless lives. We find it unable to rehabilitate, unable to keep us safe, unable to justify its vast expenditure. We find it unsustainable. We reflect on the UN call for a global ceasefire amid the pandemic to imagine a justice system that represents both a 'ceasefire', in the short-term, and an architecture of sustainable justice practices into the future.