Optical potentials with non-local (gradient) terms are used to describe the spectra of kaonic atoms. The strength of the non-local terms is determined from a many-body calculation of the kaon self energy in nuclear matter. We find that the non-local terms are quantitatively important and the results depend strongly on the way the gradient terms are arranged. Phenomenologically successful description is obtained for p-wave like optical potentials. It is suggested that the microscopic form of the non-local interaction terms is obtained systematically by means of a semi-classical expansion of the nucleus structure.