Many classical Islamic sources argue that it is not permissible for non-Muslims to reside in the Hijaz, especially Mecca and Medina. Such arguments are usually based on a famous Prophetic saying, "Two religions should not join/remain in the peninsula/land of the Arabs", and on the reported action taken by the second caliph. Umar b. al-Khattab to remove non-Muslims from settlements in western Arabia. In this article, it is argued that the contradictory nature of the evidence for this expulsion casts serious doubt on whether such a widespread action actually took place, certainly not in the decades immediately following Muhammad's death. It concludes that the widely attested classical prohibition on non-Muslims residing in the Hijaz rather had much more to do with the gradually evolving need to draw up firmer communal boundaries, which could help distinguish Muslims from others, and the role played by sacred spaces in doing so.