Steep slope segments of rivers with large sediment inflow or gravelly rivers transport gravelly sediments and fill up the river channel. Originally, the vegetation cover was low in these gravelly areas. However, severe vegetation encroachment can currently be seen on these riparian areas. Therefore, we hypothesized that gravelly sediments inhibit vegetation colonization and that a reduction in movable gravelly sediments caused the change. Accordingly, the objective of our study was to assess the effect of sediment deposition and erosion on vegetation colonization. To achieve this objective, the recovery process of herbs and trees after sediment deposition or erosion was investigated using aerial photos from six rivers. A field investigation was conducted before and after a large flood at depositional and erosional locations. The aerial photo survey elucidated that the recovery of herbs or woody vegetation after flushing is substantially delayed at sites where gravel is deposited in comparison to eroded sites. The recovery of vegetation was relatively fast in sandy rivers compared to gravelly rivers. Tree recovery slowly began mainly from new recruitment at deposited sites of gravelly sediments, whereas in eroded sites, new shoots sprouted in the next spring from the collapsed live trees, achieving a rapid recovery of tree density in the succeeding years. The nutrient and moisture content of the sediment was significantly higher in the eroded sites in comparison to gravel deposited sites. The above results provided sufficient information on the effect of gravel deposition for vegetation colonization in riparian zone. The gravel layers are deposited after washing and being segregated from fine sediments during floods. Thus, they are low in moisture and nutrients compared to eroded sites in which the underlying sediments are exposed by the flood. Fine sediments released by an upstream dam enhance the nutrient and moisture contents to increase vegetation colonization. Therefore, the reduction of gravelly sediments due to gravel mining, river regulation and modifications of river basins can have a substantial effect on vegetation colonization.