We show that the isometry group of a polyhedral Hilbert geometry coincides with its group of collineations (projectivities) if and only if the polyhedron is not an n-simplex with n >= 2. Moreover, we determine the isometry group of the Hilbert geometry on the n-simplex for all n >= 2, and find that it has the collineation group as an index-two subgroup. The results confirm several conjectures of P. de la Harpe for the class of polyhedral Hilbert geometries.