In teleoperation, it is crucial to monitor human operators' cognitive workload. This is because the movement of the end-effector of the robot in a distance is ultimately determined by the decisions, controls, and commands of the human operators. There are various factors that affect cognitive load, and among them, time pressure or schedule pressure is the common stress that human workers usually experience in construction to meet the project completion deadline. They have to make up for it due to factors that delay the schedule during the construction project (e.g., weather, clients' change orders). It is hard to expect that there will not be time pressure even when teleoperating construction robots since it is known that teleoperation takes longer to accomplish the same amount of work as onboarding. Such time pressure typically affects human workers' cognitive load and work performance in various contexts. There is still a lack of knowledge about operators' perspective when it comes to teleoperating a robot for a challenging task at a construction site under time constraints. Therefore, it is necessary to examine the cognitive load of the operator and overall human-robot interaction performance with time pressure. This study takes account of different levels of time pressure at a construction site rather than a binary situation, that is, with or without time pressure. The experimental results showed that cognitive load increased and performance decreased when excessive time pressure was given, whereas cognitive load decreased and performance increased when reasonable time pressure was applied.