By laboratory experiments, we investigate instability behaviors in a differential rotation system derived from the Czochralski silicon crystal growth. The experimental apparatus consists of a rotating disc (radius a, rotation rate omega) on the surface of homogeneous salt water contained in a rotating cylindrical tank (radius R>a, rotation rate Omega<omega). The differential rotation induces the vertical Stewartson layer at the disc radius, and various flow patterns appear as the rotation rate difference omega-Omega is increased. At relatively small omega-Omega, the well-known barotropic, or circular shear, instability generates multiple vertically coherent vortices along the Stewartson layer. The number of vortices decreases with increasing omega-Omega. When there are two vortices, irregular vertical motion is observed near the axis beyond certain critical values of omega-Omega. The streamlines near the axis are elliptical and the flow behavior has properties similar to those of the elliptic instability. As omega-Omega is further increased, the irregular motion ceases and the rotation center of the flow below the disc deviates from the machine axis. The rotation center itself orbits around the machine axis. This flow behavior resembles the single-vortex mode of the circular shear instability. We focus on these two instability behaviors: The vertical irregular motion and the off-axis rotation, and examine the instability onset conditions in the light of the elliptic instability and the circular shear instability, respectively. (C) 2004 American Institute of Physics.