An integrated heating, ventilation and air-conditioning (HVAC) system is one of the most important components to determine the energy consumption of the entire building. For commercial buildings, particularly office buildings and schools, the heating and cooling loads are largely dependent on the occupant behavioral patterns such as occupant density and their activities. Therefore, if HVAC system can respond to dynamic occupancy profiles, there is a large potential for reducing energy consumption. However, currently, most of existing HVAC systems are being operated without the ability to adjust supply air rate in response to the dynamic profiles of occupants. Due to this inefficiency, much of the HVAC energy use is wasted, particularly when the conditioned spaces are unoccupied or under-occupied (fewer occupants than the intended design). The solution to this inefficiency is to control HVAC system based on dynamic occupant profiles. Motivated by this, the research provided a real-time vision-based occupant pattern recognition system for occupancy counting as well as activity level classification. The research was divided into two parts. The first part was to use an open source library based on deep learning for real-time occupancy counting and background subtraction method for activity level classification with a static RGB camera. The second part utilized a DOE reference office building model with dynamic set-point control and conventional HVAC control to identify the potential energy savings and thermal comfort. The research results revealed that the vision-based system can detect occupants and classify activity level in real time with accuracy around 90% when there are not many occlusions. Additionally, the dynamic set-point control strategies indeed can bring about energy savings and thermal comfort improvements.