Google TurboQuant: 3-bit Quantized KV Cache with Zero Precision Loss, Inference Speed Up to 8x Faster

According to 1M AI News monitoring, Google Research has released a quantization compression algorithm called TurboQuant, which can compress the KV cache of large language models to 3 bits, reducing memory usage by at least 6 times without training or fine-tuning and without losing model accuracy. In 4-bit mode, the speed of computing attention on NVIDIA H100 GPUs is up to 8 times faster than the 32-bit unquantized baseline.

The research team validated TurboQuant on long-context benchmarks such as LongBench, Needle In A Haystack, and ZeroSCROLLS using Gemma and Mistral models, achieving optimal performance in all tests. The algorithm consists of two sub-algorithms: PolarQuant, which eliminates traditional quantization memory overhead through polar coordinate transformation, and QJL, which corrects residual errors with only 1 bit.

Led by Google Research’s Amir Zandieh and Vice President and Google Fellow Vahab Mirrokni, in collaboration with KAIST in South Korea and New York University, the study will be published at ICLR 2026. Google states that one of the main applications of this technology is to address the KV cache bottleneck in models like Gemini.