BC type z-measures and determinantal point processes

The (BC type) z-measures are a family of four parameter z, z', a, b probability measures on the path space of the nonnegative Gelfand Tsetlin graph with Jacobi-edge multiplicities. We can interpret the z-measures as random point processes on the punctured positive real line X = R->0 \ {1}. Our main result is that these random processes are determinantal and moreover we compute their correlation kernels explicitly in terms of hypergeometric functions. For very special values of the parameters z, z', the processes P-z,P-z',P-a,P-b on X are essentially scaling limits of Racah orthogonal polynomial ensembles and their correlation kernels can be computed simply from some limits of the Racah polynomials. Thus, in the language of random matrices, we study certain analytic continuations of processes that are limits of Racah ensembles, and such that they retain the determinantal structure. Another interpretation of our results, and the main motivation of this paper, is the representation theory of big groups. In representation-theoretic terms, this paper solves a natural problem of harmonic analysis for several infinite dimensional symmetric spaces. (C) 2018 Elsevier Inc. All rights reserved.