We prove four theorems about groups with a dihedral (or cyclic) image containing a difference set. For the first two, suppose G, a group of order 2p (q) over tilde with p an odd prime, contains a nontrivial (v, k, lambda) difference set D with order n = k - lambda prime to p and self-conjugate modulo p. If G has an image of order p, then 0 less than or equal to 2a + epsilon root n less than or equal to 2 (q) over tilde for a unique choice of epsilon = +/-1, and for a = (k-epsilonrootn)/2p. If G has an image of order 2p, then rootn less than or equal to (q) over tilde and lambda greater than or equal to rootn(rootn-1)/((q) over tilde -1). There are further constraints on n, a and epsilon. We give examples in which these theorems imply no difference set can exist in a group of a specified order, including filling in some entries in Smith's extension to nonabelian groups of Lander's tables. A similar theorem covers the case when p|n. Finally, we show that if G contains a nontrivial (v, k, lambda) difference set D and has a dihedral image D-2m with either (n, m) = 1 or m = p(t) for p an odd prime dividing n, then one of the C-2 intersection numbers of D is divisible by m. Again, this gives some non-existence results.