Robustness measures for molecular detections using high-resolution transmission spectroscopy of exoplanets

Ground-based high-resolution transmission spectroscopy has emerged as a promising technique for detecting chemicals in transiting exoplanetary atmospheres. Despite chemical inferences in several exoplanets and previous robustness studies, a robust and consistent detrending method to remove telluric and stellar features from transmission spectra has yet to be agreed upon. In this work, we investigate the robustness of metrics used to optimize principle component analysis (PCA)-based detrending for high-resolution transmission spectra of exoplanets in the near-infrared. As a case study, we consider observations of the hot Jupiter HD 189733 b obtained using the CARMENES spectrograph on the 3.5 m CAHA telescope. We confirm that optimizing the detrending parameters to maximize the signal-to-noise ratio (S/N) of a cross-correlation signal in the presence of noise has the potential to bias the detection significance at the planetary velocity of optimization. However, we find that optimization using the difference between a signal-injected cross-correlation function and the direct cross-correlation function (CCF) is more robust against over-optimization of noise and spurious signals. We additionally examine the robustness of weighting the contribution of each order to the final CCF, and of S/N calculations. Using a prescribed robust methodology, we confirm H2O in the atmosphere of HD 189733 b (S/N = 6.1). We then investigate two further case studies, of exoplanets HD 209458 b and WASP-76 b, confirming OH in the atmosphere of WASP-76 b (S/N = 4.7), and demonstrating how non-robust methods may induce false positive or inflated detections. Our findings pave the way towards a robust framework for homogeneous characterization of exoplanetary atmospheres using high-resolution transmission spectroscopy in the near-infrared.