Karger, Motwani, and Sudan [J. ACM, 45 (1998), pp. 246-265] introduced the notion of a vector coloring of a graph. In particular, they showed that every k-colorable graph is also vector k-colorable, and that for constant k, graphs that are vector k-colorable can be colored by roughly Delta(1-2/k) colors. Here Delta is the maximum degree in the graph and is assumed to be of the order of n(delta) for some 0 < delta < 1. Their results play a major role in the best approximation algorithms used for coloring and for maximum independent sets. We show that for every positive integer k there are graphs that are vector k-colorable but do not have independent sets significantly larger than n/Delta(1-2/k) (and hence cannot be colored with significantly fewer than Delta(1-2/k) colors). For k = O(log n/log log n) we show vector k-colorable graphs that do not have independent sets of size (log n) c, for some constant c. This shows that the vector chromatic number does not approximate the chromatic number within factors better than n/polylogn. As part of our proof, we analyze "property testing" algorithms that distinguish between graphs that have an independent set of size n/k, and graphs that are "far" from having such an independent set. Our bounds on the sample size improve previous bounds of Goldreich, Goldwasser, and Ron [J. ACM, 45 (1998), pp. 653-750] for this problem.