a composable vector quantization framework
VQ-bench is an open-source benchmark and framework for vector quantization. It implements several algorithmic primitives and builds quantizers by composing these primitives. It evaluates the performance of existing quantizers on several size and quality metrics, and we publish the results on this page.
Each method evaluated in VQ-bench, with a link to its source.
| Quantizer | Paper | Year |
|---|
MinMax (per-vector) and Scalar (per-dimension) are uniform min–max quantization baselines with no dedicated reference.
| Dataset | Vectors | Dimension | Calibration queries |
|---|---|---|---|
| ArXiv | 1,344,643 | 768 | 0 |
| CCNews | 495,328 | 768 | 0 |
| Yahoo | 677,305 | 384 | 0 |
| COCO | 282,360 | 768 | 114,054 |
| LAION | 1,000,448 | 512 | 999,448 |
All datasets are from VIBE (Jääsääri et al. 2025) and use inner-product similarity. Calibration queries is the number of available sample queries for training.
| Metric | What it measures | Better |
|---|---|---|
| KV-cache values / signal compression | ||
| reconstruction MSE | average squared distance between the true vector and the reconstructed vector | lower |
| reconstruction bias | squared norm of the average residual vector (true vector − approximated vector) | lower |
| Model weights compression | ||
| score MSE | average squared error of the inner-product estimates between queries and candidates | lower |
| score bias | bias of the inner-product estimates | → 0 |
| Vector search / recommendations | ||
| recall@k | fraction of the true top-k neighbors contained in the approximated top-k | higher |
| SOS@k | ratio of summed top-k approximated scores and summed true top-k scores | higher |
| exp-SOS@k (τ) | SOS@k over exponentiated scores exp(score / τ), at temperature τ | higher |
| KV-cache keys | ||
| attention KL (τ) | KL-divergence between the softmax distributions induced by true and approximate candidate scores, at temperature τ | lower |
| attention TV (τ) | TV-distance between the softmax distributions, at temperature τ | lower |
We also report size and efficiency metrics (bits per dimension, peak / encode memory, encode / scoring time). All time and memory statistics were measured on an Apple M2 Pro with 16GB RAM using 6 threads.
VQ-bench is actively maintained by Amir Ingber, Edo Liberty (Pinecone), and Ashwin Padaki (UPenn).