Insufficient knowledge about a new bug or a new developer, in the context of recommendations done in software bug repositories (SBR) mining, impacts the recommender-system performance and gives rise to a cold start problem (CSP). Many recent cold start solutions based on machine learning in general, and specifically on reinforcement and deep learning, have been published, but the insights from these works are not presented comprehensively and remain scattered, as a result, it is difficult for budding researchers to conclude further enhancements. Also, there is a lack of a survey covering both ML and RL-based solutions for CSP under one hood. So, to bridge these gaps, this article presents a critical review using the PRISMA model. Both ML and RL-based solutions for Cold start problems have been presented in this model through a well-defined taxonomy along with its detailed bibliometric analysis. This article provides 78 significant primary studies published from 2012 to 2022. Findings from this review indicate that different solution strategies based on MABs as well as CMABs, need to be designed for handling cold start settings in the bug and developer context. Moreover, there is a great scope for performance improvement in the state-of-the-art solutions by either improving the accuracy, feature engineering integration, different process metrics exploration, or hyper-parameter tuning. This review will give directions to novice researchers, academicians, and practitioners to work ahead on the issues identified in this contemporary challenging problem.