Concrete gravity dams should withstand huge hydraulic pressure and seismic load with their weight and the shear resistance of their foundations. However, rock foundations for dams are not generally homogeneous, but are composed of several kinds of rocks with different deformation and strength characteristics. It is necessary to develop an accurate evaluation method for the shear strength characteristics of such rock masses in order to rationalize the design of concrete gravity dams. First, composite rock mass models containing hard rock and soft rock masses were prepared, and the laboratory shear tests were done, varying the location and area ratio of the soft rock mass. In addition, nonlinear FEM analysis was conducted in order to reproduce the shear failure process of a rock mass model and to consider the shear strength characteristics of composite rock masses. In conclusion, we can generally adopt the area weighted mean strength as the design shear strength of composite rock masses containing hard rocks and soft rocks for concrete gravity dams. However, when soft rocks are located at the downstream side of the base of a dam, or when there is a big difference in the deformability and shear strength of the hard rock and soft rock, we should perform nonlinear FEM analysis to consider the detailed stress distribution and shear failure process.