Pavement maintenance causes an instant change in the pavement's material or structural properties and affects the subsequent development of pavement distresses and performance. The occurrence of maintenance action significantly limits the applicability of predictive models that rely heavily on the continuity of time and pavement exposure conditions. Using rutting of asphalt pavement as an example, this study treated the partitioned rutting development as short-term time series. The proposed average cosine similarity of pavement rutting development effectively integrated the data collected within a characteristic length in the longitudinal direction of the pavement. Integration of the raw data effectively mitigated the data inconsistency caused by measuring errors and simplified the model construction. We employed convolutional neural network (CNN) and long short-term memory (LSTM) as two typical deep-learning (DL) models to capture the characteristics of rutting development from limited data and make corresponding predictions. The effects of model hyperparameters and input type on the model performance (e.g., accuracy and stability) were investigated to identify the optimal setting for various modeling data. The comparisons with two statistical models-exponential smoothing (ES) model and autoregressive integrated moving average (ARIMA) model-indicated the potential of applied DL models in accurately predicting rutting development in field pavement. Finally, three strategies of improving model performance were explored and discussed for future applications, i.e., increasing the input length, input dimensionality, and model complexity. This study stands in the position of transportation engineers to solve a practical problem with computer-aided tools, which is the prediction of asphalt pavement performance with deep-learning (DL) models. The data utilized in this study were collected from the field, resulting in inherently challenging characteristics such as being short-term and noisy. This study investigated corresponding strategies to allow DL models to provide accurate and stable predictions. Besides, this study makes DL models more accessible to practicing engineers. A significant amount of model information obtained in this study can be extended to various types of performance indices. Finally, DL models constructed in this study were compared with classic and currently applied models using the same data. The results reveal that DL models can serve as reliable and promising tools for predicting asphalt pavement performance in the natural deterioration stage of the pavement.