This paper analyzes open dielectric waveguides using the vector finite-element method and boundary integral equations derived from the second Green's theorem. This finite-element formulation, together with the boundary operator, is solved using a penalty function method. Comparison with previously published results shows good agreement for the analysis of the rectangular dielectric waveguide. (C) 2008 Optical Society of America