We study the variability of passive scalar diffusion via the statistics of stochastic particle dispersion in a chaotic flow. We find that at intermediate times when the statistics of individual trajectories start to exhibit scaling-law behaviors, scalar variance over the entire domain exhibits multimodal structure. We demarcate the domain based on Lagrangian coherent structures and find that the conditional statistics exhibit strong unimodal behavior, indicating coherence of effective diffusion among each Lagrangian partition of the flow.