Home energy audits are a commonly used method to assess the performance and energy efficiency of residential buildings. In many cases these audits are offered to residential homeowners as a way to gauge the need for and possible implementation of energy efficiency retrofits. However, these energy audits are also time-intensive, and can have low benefit-cost ratios if the recommendations developed are not acted on in the form of energy efficiency investments on the part of the homeowner. Additionally many homeowners can be hesitant to conduct an energy audit, limiting the amount of homes that can be targeted for improvements. Given the increasing amount of data and information available today, particularly thanks to smart grid infrastructure, smart meters, IoT devices, reanalysis-based weather data, utility data, etc, as well as improved connectivity and communication, this offers an opportunity to utilize data-driven methods to assess the performance of a residential building. Using data-driven techniques, this research works towards an assessment tool that can be used to assess the efficiency of residential buildings without the need for a physical energy audit. Energy use data and weather data are collected for 74 homes in multiple climate zones to develop inverse models. These inverse models are then used, in conjunction with a community-level model comparisons, to detect abnormalities. These point outliers or use-pattern abnormalities may indicate an inefficiency, or a relatively lower-performing home in comparison to neighboring homes. Types of inefficiencies are identified and several case study homes are analyzed. The results of this work will help to the construction community to target homes in need of energy efficiency upgrades, ultimately motivating improved sustainability of residential buildings.