With the rapid development of Internet and multimedia technologies, the volume and dimension of data increase significantly which lead to the need for effective management of large scale and high dimensional data in many applications. Approximate Nearest Neighbor search is one of the most basic problems. Given a query vector, how to retrieve its nearest neighbor vectors in a large scale dataset quickly and accurately, the study of this problem faces many bottlenecks. Among them, the establishment of an efficient index structure to reconstruct the original dataset, making the output indexing table balanced and effectively fitting the dataset's underlying distribution is one of the most important tasks. In addition, due to the curse of dimension, estimating the distance between high-dimensional vectors effectively is also one of the bottlenecks in the nearest neighbor search. Aiming these difficult problems, we propose a neighbor search method based on product quantization in clusters. First, recombine the hierarchical relationship between vector quantization and product quantization to generate a more compact and balanced index structure which can better fit the original large scale and high dimensional dataset, and the empty bucket rate of the index table is significantly reduced. By adjusting the centroid scale of each layer, the intra-cluster quantization tree structure can be flexibly applied to different neighbor search tasks on different large scale datasets. Second, we design a new approach for generating neighbor clusters based on greedy queues. The probability of containing neighboring vectors of the query in its nearest neighbor cluster is low, so a certain number of neighbor clusters are needed to provide enough candidate vectors to apply to the re-ranking part. We propose a greedy queue, which can generate sufficient neighbor clusters for query vector quickly compared with greedy algorithms. Third, our approach also includes an improved re-ranking method called plane quantilization approaching method which is inspired by line quantilization. The calculation of the distance between the candidate set and the query vector is an important part of the re-ranking phase and directly affects the query accuracy. However accurate Euclidean distance calculation between vectors in large-scale and high-dimensional datasets is time consuming and sometimes impossible to achieve under limited physical resources. Therefore, it is necessary to adopt approximate distance calculation approaches. Our method can more efficiently calculate the distance between high dimensional vectors and cause smaller approximation error compared to the state-of-art quantilization approaches, which significantly improved the accuracy of the nearest neighbor retrieval. In the experiments, we use several large scale and high dimensional dataset described by the operator Sift and Gist to measure our method respectively. Compared with the product quantization tree technique, the accuracy of the first recall is increased by 57.7% and the reduction rate of empty bin rate is more than 50%. Compared with local optimized product quantization technology, the recall rate can also be as high as 0.97, but the required query time is reduced by 8 times. The experimental results show that our method based on product quantization in clusters can significantly improve the performance of nearest neighbor search, and is an effective solution for nearest neighbor search on large scale and high dimensional datasets. © 2020, Science Press. All right reserved.
