Let G be a finite p-group and let Aut(c)(G) be the set of all central automorphisms of G. For any group G, the center of the inner automorphism group, Z(Inn(G)), is always contained in Aut(c) (G). In this paper, we study finite p-groups G for which Aut(c)(G) is of minimal possible, that is Aut(c)(G) = Z(Inn(G)). We characterize the groups in some special cases, including p-groups G with C-G (Z(Phi(G))) not equal Phi(G), p-groups with an abelian maximal subgroup, metacyclic p-groups with p > 2, p-groups of order p(m) and exponent p(m-2) and Camina p-groups.