Music listening is heavily influenced by contexts, and contextual factors can shape users' interaction with music information retrieval (MIR) systems. To better design context-sensitive user experiences in MIR systems, in this paper, we present a review of prior studies on how contexts are associated with user behavior in MIR systems. Contextual factors considered include interaction design, age, personality, time of day, activity, motivation, nationality, etc. Based on the review, we introduce a framework to consider these contextual factors in a consistent and organized way. The framework is adapted from Ingwersen and Jarvelin's 2006 nested contexts framework, and has four layers: 1) MIR/system contexts that focus on MIR systems themselves, including both hardware and software; 2) situational contexts that describe varied and transient daily situations where users interact with MIR systems; 3) personal contexts that focus on the more stable personal characteristics; and 4) social and cultural contexts that describe the characteristics of users' environments. We also present an example to illustrate how to systematically analyze user contexts by using the framework. Finally, we discuss several areas for possible future studies.