Ground thermal conductivity, As, is one of the important design factors for the ground source heat pump system. This study aimed to investigate the effect of environmental thermal disturbance on the As estimation during the thermal response test (TRT), which is interpreted by the infinite line source model. For this purpose, a detailed numerical model considering the heat exchange between the circulating fluid inside the above-ground pipe of TRT rig and the outdoor environment in two representative seasons was developed. The numerical model was compared with the model with the adiabatic boundary assigned to the outer surface of above-ground pipe to evaluate the fluctuations of As on account of the atmospheric disturbance. The parametric and sensitivity analyses were carried out to quantify the impacts of TRT seasons and above-ground pipe parameters including the surface absorptivity, thickness, and thermal conductivity of insulation material, and the length of the above-ground pipe on the As and corresponding design borehole length. An approach of prolonging the TRT duration was proposed to impair the disturbance when the pipe is not insulated. Results show that As fluctuates strongly and is largely misestimated considering the atmospheric disturbance, especially when TRT duration is limited. The As varies widely as the length of above-ground pipe, insulation surface absorptivity, and insulation thermal conductivity increase, insulation thickness decreases, and TRT starts in summer. The pipe length is the most significant factor to the uncertainty of As. The test time is recommended to last at least 3 days in summer and 3.6 days in winter when the pipe is not wrapped by a layer of insulation material. This study quantitatively assesses the impacts of environmental thermal disturbance during the TRT and gives guidance to carry out TRT properly.