As a cost-effective paradigm, Sparse Crowdsensing aims to recruit workers to perform a part of sensing tasks and infer the rest, which has broad applications, including environmental monitoring, urban sensing, etc. In most cases, workers will participate in real time, and thus their sensing data are coming dynamically. Taking full advantage of the online coming data to complete the full sensing map is an important problem for Sparse Crowdsensing. However, for data completion, the importance of data collected from different spatio-temporal areas is usually different and time-varying. For example, the newly obtained data in the center is often more important than the old ones from edges. Moreover, the area importance may also influence the following worker selection, i.e., selecting suitable workers to actively sense important areas (instead of passively waiting for given data) for improving completion accuracy. To this end, in this paper, we propose a framework for online Sparse Crowdsensing, called OS-MCS, which consists of three parts: matrix completion, importance estimation, and worker selection. We start from the dynamically coming data and propose an online matrix completion algorithm with spatio-temporal constraints. Based on that, we estimate the spatio-temporal area importance by conducting a reinforcement learning-based up-to-date model. Finally, we utilize the prophet secretary problem to select suitable workers to sense important areas for accurate completion in an online manner. Extensive experiments on real-world data sets show the effectiveness of our proposals.