Mala Mahadevan digs into an algorithm:
DiskANN is meant to help with searching a billion-vector dataset from a single machine using SSDs.
Microsoft Research’s original work demonstrated a billion-point index on a workstation with 64 GB of RAM and an SSD, while targeting high recall and low query latency. Before we get to ‘why ssd’ and details of storage, we need to understand the basics of vector search and underlying terminologies.
Click through for a depiction of the paper and quite a few examples of how the mechanisms work.