We consider Wilson's SU(N) lattice gauge theory (without fermions) at negative values of beta=2N/g(2) and for N=2 or 3. We show that in the limit beta-->-infinity, the path integral is dominated by configurations where links variables are set to a nontrivial element of the center on selected nonintersecting lines. For N=2, these configurations can be characterized by a unique gauge invariant set of variables, while for N=3 a multiplicity growing with the volume as the number of configurations of an Ising model is observed. In general, there is a discontinuity in the average plaquette when g(2) changes its sign which prevents us from having a convergent series in g(2) for this quantity. For N=2, a change of variables relates the gauge invariant observables at positive and negative values of beta. For N=3, we derive an identity relating the observables at beta with those at beta rotated by +/-2pi/3 in the complex plane and show numerical evidence for a Ising like first order phase transition near beta=-22. We discuss the possibility of having lines of first order phase transitions ending at a second order phase transition in an extended bare parameter space.