In the loop representation of quantum gravity in 3+1 dimensions, there is a space of physical states consisting of invariants of links in S3. The con-ect inner product on this space of states is not known, or in other words, the *-algebra structure of the algebra of observables has not been determined. In order to address this issue, we work instead with quantum gravity on D3, and work with a space K spanned by tangles. A certain algebra T, the 'tangle algebra', acts as operators on K. The 'empty link' psi0, corresponding to the class of the empty set, is shown to be a cyclic vector for T. We construct inner products on quotients of K from link invariants, show that these quotients are representations of T, and calculate the *-algebra structures of T in these representations. In particular, taking the link invariant to be the Jones polynomial (or more precisely, Kauffman bracket), we obtain the inner product for states of quantum gravity arising from SU(2) Chem-Simons theory.