AMR(Autonomous Mobile Robot) is being used to improve working environment through collaboration such as transporting goods between workers. For collaboration such as transporting goods, AMR tracks the workers and carries out goods transport. Object tracking is possible based on a deep learning model trained using big data, built as an object to be tracked. When the worker changes frequently, such as in a work environment, there is a problem in that big data construction and deep learning model learning are required whenever an object to be tracked is changed. There is a need for a method for tracking objects that change frequently while providing small amounts of data. This paper proposes a deep learning-based framework for tracking changeable object. An object to be tracked, such as a worker, is defined as a ToI (Target-of-Interest) object. The proposed framework utilizes a two-stage deep learning model to track a changeable ToI object. In the deep learning model of the first stage, an object of the same type as the ToI object is tracked. In the deep learning model of the second stage, the ToI object is found among the objects being tracked. The position of the ToI object is transformed into the coordinate system of the AMR so that the AMR can track the ToI object. In the experiment, the results of tracking the ToI object by using the proposed method were verified. When tracking ToI objects with a single-stage deep learning model with a small amount of data, the accuracy of tracking the ToI objects decreased according to the amount of data. In the case of the proposed method, the tracking of the ToI object was not affected by the amount of data. © ICROS 2022.