We establish the equality of classical and tropical curve counts for elliptic curves on toric surfaces with fixed j-invariant, refining results of Mikhalkin and Nishinou-Siebert. As an application, we determine a formula for such counts on P-2 and all Hirzebruch surfaces. This formula relates the count of elliptic curves with the number of rational curves on the surface satisfying a small number of tangency conditions with the toric boundary. Furthermore, the combinatorial tropical multiplicities of Kerber and Markwig for counts in P-2 are derived and explained algebro-geometrically, using Berkovich geometry and logarithmic Gromov-Witten theory. As a consequence, a new proof of Pandharipande's formula for counts of elliptic curves in P-2 with fixed j-invariant is obtained.