Despite the increasing popularity of modular buildings, there are relatively few published case studies considering the overall structural responses to wind and earthquake load. As a result, there is limited structural performance data for the overall behaviour of modular buildings, which in turn impedes the development of appropriate design standards. At the same time, different inter-module connections exhibit different structural behaviours, and the overall response of a building assembled from interconnected modules may be influenced by the inter-module connection. However, the structural behaviours of inter-module connections in modular building structures subjected to lateral dynamic loads are not well known in the existing literature. This paper, therefore, presents a numerical investigation into the overall structural behaviour of a selected case study building. In particular, the moment-rotation and force-displacement behaviours of the inter-module connection were simplified, and incorporated in the overall structural model. The numerical technique was verified by comparison with a previous study, and the verified technique was then extended to the selected six storey case study building. The results showed that the overall response was affected by the inter-module connection stiffness, especially the translational stiffness in the along load direction.