In this paper, we argue that the creation of Responsible AI over the past four decades has predominantly relied on two approaches: contextual and technical. While both are indispensable, we contend that a third equally vital approach, focusing on human-AI interaction design, has been relatively neglected, despite the ongoing efforts of pioneers in the field. Through the presentation of four case studies of real-world AI systems, we illustrate, however, how small design choices impact an AI platform's responsibleness independent of the technical or contextual level. We advocate, therefore, for a larger role for design in the creation of Responsible AI, and we call for increased research efforts in this area to advance our understanding of human-AI interaction design in Responsible AI. This includes both smaller case studies, such as those presented in this paper, that replicate or swiftly test earlier findings in different contexts, as well as larger, more comprehensive studies that lay the foundations for a framework.