How is it that complex numbers, involving the imaginary i = square-root -1 play such an important role in physics, which always measures real quantities? An answer can be given in the framework of the vector algebra of three-dimensional space, in which an associative, invertible product of vectors is defined. In this mathematical structure, also known as the Pauli algebra, i arises naturally and carries geometrical significance. In particular, i enters as the unit volume element, and imaginary vectors are pseudovectors which represent planes, such as planes of rotation or reflection. The i from the vector algebra is related to common applications of imaginary numbers in physics, including rotations in a plane, electromagnetic waves, and phase factors and operator relations in quantum mechanics. Moreover, the same algebra of real three-dimensional vectors which yields complex numbers also forms the basis for a complex four-dimensional space with the Minkowski metric and provides a natural formalism for compact, covariant treatments of relativistic phenomena.