Today's peer-to-peer (P2P) Video-on-Demand (VoD) systems are known to be highly scalable in a steady state. For the dynamic scenario, much effort has been spent on accommodating sharply increasing requests (known as flash crowd) with effective solutions being developed. The high popularity upon a flash crowd however does not necessarily last long, and indeed often drops very fast after the peak. Compared to growth, a decay is seemingly less challenging or even beneficial given the less user demands. While this is true in a conventional client/server system, we find that it is not the case for peer-to-peer. A quick decay can easily de-stabilize an established overlay, and the resultant smaller overlay is generally less effective for content sharing. The replication of data segments, which is critical during flash crowd, will not promptly respond to a fast and globalized population decay, either. Many of the replicas can become redundant and, even worse, their spaces cannot be utilized for an extended period. In this paper, we seek to understand the impact of such decays and the key influential factors. To this end, we develop a mathematical model to trace the evolution of peer upload and replication during population churns, specifically during decays. Our model captures peer behaviors with common data replication and scheduling strategies in state-of-the-art peer-to-peer VoD systems. It quantitatively reveals the root causes toward escalating server load during a population decay. The model also facilitates the design of a flexible server provision to serve highly time-varying demands.