In fact, nonlinearity is generic and may arise in a damaged engineering structure. For example, the subsequently open and close of cracks in concrete structures under dynamic loadings make the vibration behavior nonlinear. However, nonlinear dynamical systems theory is far less established than linear system. Usually, the damage is represented by the decrease in structural stiffness and most of the currently available vibration-based system identification and damage detection approaches are based on modal parameters, namely the natural frequencies, mode shapes and damping ratios, and/or their derivations, which are only suitable for linear systems and the basis of modal analysis is no longer valid in the presence of nonlinearity. It has been widely recognized that one of the major challenges in damage detection, early warning and damage prognosis is to obtain reasonably accurate identification of minor nonlinearity such as hysteresis performance of structural members under dynamic excitations which is the direct indicator of damage initiation and development. In this study, a data-based model free nonlinearity identification approach in the form of hysteresis using structural dynamic response and complete and incomplete excitation measurement time series was proposed and validated experimentally with a 4-story frame structure equipped with smart devices of magneto-rheological (MR) dampers to simulate nonlinear performance. Firstly, as an optimization method, the least-squares technique was employed to identify the system matrices of an equivalent linear system of the nonlinear structure model basing on the excitation force by hammer and the corresponding vibration measurements; and secondly, the nonlinear restoring force of the structure was identified and compared with the test measurements finally. Results show that the proposed data-based approach is capable of identifying the nonlinear behavior of engineering structures and present an applicable way for the evaluation of structure damage initiation and development under dynamic loads.