AXELRAM: Quantize Once, Never Dequantize

Large language model (LLM) inference is bottlenecked by key-value (KV) cache memory: an 8B-parameter model at context length $T{=}4096$ stores 512 MB in FP16. KV cache quantization to 2–4 bits reduces storage, but the quantized keys must be dequantized before computing attention scores—$T$ times per query.

We ask: can the SRAM that stores the quantized cache also compute attention scores, eliminating dequantization entirely?

AXELRAM answers this by integrating orthogonal-transform-based quantization with table-lookup attention in a single SRAM macro (Fig. 1). The design-time fixed codebook (30 bytes for $b{=}3$) is shared across all $d$ dimensions and stored in ROM.

Contributions.

1. 1.

   We propose AXELRAM, a smart SRAM macro with an asymmetric write/read path that reduces multiplications by $102.4\times$ (Section III).

2. 2.

   We discover sign pattern sensitivity in rotation-based KV cache quantization through 10-seed evaluation across 3 models, finding catastrophic spikes ($\Delta>50$) unreported in prior work (Section IV).

3. 3.

   We propose a gradient-free sign pattern selection that eliminates spikes with zero additional hardware cost (Section V).

For unit-norm $\hat{\mathbf{x}}\in\mathbb{R}^{d}$ and orthogonal $R$, each coordinate of $\mathbf{y}=R\hat{\mathbf{x}}$ follows approximately $\mathcal{N}(0,1/d)$ for $d\geq 64$ (concentration of measure [1]). The Lloyd-Max quantizer for this distribution yields centroids and boundaries as functions of $(d,b)$ alone—a design-time fixed codebook. For $b{=}3$, $d{=}128$: 8 centroids + 7 boundaries = 15 FP16 values = 30 bytes, verified to satisfy the Lloyd-Max optimality conditions to machine precision ($<10^{-11}$).

The inner product is preserved under orthogonal transforms: $\mathbf{q}^{\top}\mathbf{k}=(R\mathbf{q})^{\top}(R\mathbf{k})$. Instead of inverse-transforming each stored key ($T$ times), we transform the query once and compute in the rotated transform domain:

This is the asymmetric path that enables table-lookup attention.

Contrast with TurboQuant. TurboQuant [1] computes $\langle\mathbf{q},\;R^{-1}\cdot\mathrm{codebook}[\mathrm{idx}]\cdot\|\mathbf{k}\|\rangle$, applying inverse rotation to each key. Equation (1) moves the rotation from keys ($T$ times) to the query (once), enabling the pre-computation table. This rearrangement is not disclosed or suggested in [1].

Rotation-based quantization. TurboQuant [1] established the rotation + Lloyd-Max framework for KV cache with near-optimal distortion rate. QuaRot [4] and QuIP# [5] apply Hadamard rotation for weight quantization. KVLinC [11] uses Hadamard with linear correction. PolarQuant [12] uses polar decomposition requiring per-block scale parameters. All report single-seed evaluations.

Rotation seed sensitivity. SpinQuant [2] (ICLR 2025) first demonstrated rotation seed sensitivity in W4A4 quantization, finding 6-point accuracy variance across 100 Hadamard rotations. They propose Cayley-parameterized gradient optimization requiring backpropagation through the full model. ParoQuant [3] samples 100 seeds but reports only averages without variance analysis. Our work extends this analysis to KV cache quantization, where catastrophic spikes ($\Delta>50$) occur—qualitatively more severe than SpinQuant’s findings—and proposes a gradient-free mitigation with zero hardware cost.

ADC-based attention. LOOKAT [8] (January 2026) proposes ADC-based attention with K-Means codebooks. Product Quantization [13] established ADC for nearest-neighbor search. AXELRAM differs in: (1) fixed codebook (not learned), (2) shared across all dimensions (capacity independent of $d$), and (3) targeting SRAM-integrated hardware.

CIM accelerators. MADDNESS [9] uses data-dependent LUT-based SRAM. Google’s PIM patent [10] proposes processing-in-memory for attention without rotation-based quantization. AXELRAM is distinguished by its data-independent ROM codebook and asymmetric path.

Calibration-free vs. lightweight calibration. The basic AXELRAM configuration (random sign pattern) is calibration-free and works reliably on models with homogeneous layer norms (e.g., LLaMA-3.1-8B: norm ratio $<2\times$). For models with heterogeneous norms (e.g., Qwen2.5-3B: norm ratio $>5\times$), lightweight sign pattern calibration (8 samples, one-time, 36 seconds) is recommended. The fixed codebook itself remains calibration-free and model-independent in both cases.

The design philosophy of AXELRAM is captured by the title of this paper: each incoming key vector is quantized once through the write path and subsequently used for attention score computation entirely through table lookup and accumulation, with no reconstruction of the original vector at any point in the read path. This property is an invariant of the hardware architecture itself—the read path physically contains no inverse transform circuit—and holds regardless of how the rotation parameters were determined. When sign pattern calibration is employed for models with heterogeneous layer norms, the calibration selects which values are written to the sign vector ROM before the system begins processing tokens. Once deployment begins, the runtime behavior is identical to the uncalibrated case: every token’s key vector is quantized once, and every attention score is computed without dequantization. The calibration step thus fits naturally into the standard model deployment pipeline, alongside other one-time setup tasks such as weight loading and memory allocation.

Practical guideline. We recommend checking the layer-wise key norm ratio ($\max/\min$ across layers) as a predictor of sign pattern sensitivity:

• •

  Ratio $<2\times$: calibration-free (default signs) is safe.

• •

  Ratio $>5\times$: sign pattern optimization is recommended.

On the 2-bit limitation of Qwen3-8B. After sign pattern optimization, Qwen3-8B at 2-bit still exhibits $\Delta=+9.26$, while 3-bit ($+0.35$) and 4-bit ($+0.01$) are excellent. This is not a failure of the sign pattern selection: even the best of 10 random seeds yields $\Delta=+9.10$, comparable to the optimized result. The root cause is that 2-bit quantization provides only 4 representable levels, which is insufficient to capture the distribution tails of Qwen3-8B’s key vectors. We note that this limitation is not unique to AXELRAM—any 2-bit scalar quantization method would face the same precision floor on this model. In our experiments, 3-bit quantization ($5.1\times$ compression) operated well on all three models tested, while 4-bit provided near-lossless quality on every model. The appropriate bit-width depends on the model architecture and the acceptable quality–compression tradeoff; we recommend per-model validation before deployment at any bit-width.

Hardware status. No hardware has been fabricated for AXELRAM. All results in this paper are derived from bit-accurate software simulation of the proposed architecture. The multiplication counts (e.g., $102.4\times$ reduction) are mathematical identities that hold exactly in any implementation. The PPL evaluations simulate the quantize–dequantize cycle in PyTorch and measure the impact on language modeling quality. The architecture is presented as a proposal; physical implementation and silicon validation remain as future work.

Reproducibility. All evaluation code, codebook solver, sign pattern optimizer, and result JSON files are released under MIT license at https://github.com/Axelidea/AXELRAM. The codebook solver is an independent implementation using quantile-based initialization and scipy.stats; no code is shared with or derived from any prior implementation.

AXELRAM is a smart SRAM macro that eliminates dequantization from quantized-KV attention, achieving $102.4\times$ multiplication reduction via the asymmetric path design. Through 10-seed evaluation across 3 models, we discover sign pattern sensitivity—catastrophic PPL spikes ($\Delta>50$) on models with heterogeneous layer norms—extending SpinQuant’s rotation variance finding to the KV cache domain. A gradient-free sign pattern selection eliminates these spikes with zero additional hardware cost (the optimized signs occupy the same ROM as the default signs). No hardware has been fabricated; we present the architecture as a proposal validated by bit-accurate simulation.

Part of the computational work was performed using the TSUBAME4.0 supercomputer at Institute of Science Tokyo.

Finally, we observe that Penguins—awkward in their waddle on land, yet remarkably graceful once they enter water—remind us that choosing the right domain makes all the difference. We thank the Penguins for the inspiration.