a composable vector quantization framework
VQ-bench is an open benchmark and harness for vector quantization algorithms. It implements several primitives in Rust and builds quantizers by composing these primitives. It supports several high-dimensional embedding datasets and measures how different quantizers trade off size (e.g., bits-per-dimension, encoding memory) against
quality (e.g., reconstruction error, recall). VQ-bench reports these metrics side-by-side and lets you explore the trade-offs directly. The goal is for it to eventually provide an open-source, reproducible comparison of existing quantization algorithms and a testbed for designing new quantization algorithms.
Each method evaluated in VQ-bench, with a link to its source. Colors match the plot series. Scalar and MinMax are uniform min–max baselines (per-coordinate and per-vector, respectively) with no dedicated reference. All datasets are from VIBE (Jääsääri et al. 2025) and use dot-product similarity. Calibration queries is the number of available sample queries for training.
Two experiments
are available in the selector.
benchmark displays the results from running each quantizer across all datasets.
whitening-ablation is a proof-of-concept experiment that takes advantage of the compositional interface of VQ-bench. It investigates whether whitening (making the query distribution isotropic via SVD) improves quantization performance. It applies only to datasets with calibration queries, and leads to improvements on COCO but regressions on LAION.
We also report size and cost metrics (bits per dimension, peak / encode memory, encode / scoring time). Softmax metrics are reported at temperatures τ ∈ {0.5, 1, 2}; recall and SOS at k ∈ {1, 10, 50}.Quantizers
Quantizer Paper Year Datasets
Dataset Embedding model / dim Vectors Calibration queries ArXiv nomic / 7681,344,643 0 CCNews nomic / 768495,328 0 Yahoo MiniLM / 384677,305 0 COCO nomic / 768282,360 114,054 LAION CLIP / 5121,000,448 999,448 Experiments
Metrics
Metric What it measures Better KV-cache values · signal compression reconstruction MSE Average squared distance between the true vector and the reconstructed vector. lower reconstruction bias Squared norm of the average residual vector (true vector − approximated vector). lower Model weights compression score MSE Average squared error of the inner-product estimates between queries and candidates. lower score bias Bias of the inner-product estimates. → 0 Vector search · recommendations recall@k Fraction of the true top-k neighbors recovered contained in the approximated top-k. higher SOSk (sum of top scores) Ratio of summed top-k approximated scores and summed true top-k scores. higher KV-cache keys · softmax model layers attention KL (τ) KL-divergence between the softmax (i.e., attention) distributions induced by true and approximate candidate scores, at temperature τ. lower attention TV (τ) TV-distance between the softmax (i.e., attention) distributions, at temperature τ. lower
VQ-bench is maintained by Amir Ingber and Edo Liberty (Pinecone) and Ashwin Padaki (University of Pennsylvania). The code will become open source in late July 2026.