Nearest Neighbor Search with Neighborhood Graph and Tree for High-dimensional Data
Neighborhood Graph and Tree for Indexing High-dimensional Data
NGT provides commands and a library for performing high-speed approximate nearest neighbor searches against a large volume of data (several million to several 10 million items of data) in high dimensional vector data space (several ten to several thousand dimensions).
$ brew install ngt
$ unzip NGT-x.x.x.zip $ cd NGT-x.x.x $ mkdir build $ cd build $ cmake .. $ make $ make install $ ldconfig /usr/local/lib
$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" $ brew install cmake $ brew install [email protected] $ export CXX=/usr/local/bin/g++-9 $ export CC=/usr/local/bin/gcc-9 $ unzip NGT-x.x.x.zip $ cd NGT-x.x.x $ mkdir build $ cd build $ cmake .. $ make $ make install
The index can be placed in shared memory with memory mapped files. Using shared memory can reduce the amount of memory needed when multiple processes are using the same index. In addition, it can not only handle an index with a large number of objects that cannot be loaded into memory, but also reduce time to open it. Since changes become necessary at build time, please add the following parameter when executing "cmake" in order to use shared memory.
$ cmake -DNGT_SHARED_MEMORY_ALLOCATOR=ON ..
Note: Since there is no lock function, the index should be used only for reference when multiple processes are using the same index.
When you insert more than about 5 million objects, please add the following parameter to improve the search time.
$ cmake -DNGT_LARGE_DATASET=ON ..
The followings are the results of ann benchmarks on an AWS c5.4xlarge instance for NGT v1.7.5.
Copyright (C) 2015-2020 Yahoo Japan Corporation
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License. You may obtain a copy of the License at
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Yahoo Japan Corporation has acquired several patents on the technologies utilized in this software. However, the patent rights shall not be exercised under Apache License Version 2.0, only when the patented techniques are used with this software.
This project requires contributors to accept the terms in the Contributor License Agreement (CLA).
Please note that contributors to the NGT repository on GitHub (https://github.com/yahoojapan/NGT) shall be deemed to have accepted the CLA without individual written agreements.