Read the preprint (PDF)

Reinforcement learning is how language models learn to reason better, but updating every weight is expensive: dense AdamW keeps a gradient and two optimizer moments for every trained weight. Randomized Gradient Compression (RGC) removes both. It trains all 1.4 billion attention and MLP projection weights from a compressed gradient and stores no optimizer state at all.

On Qwen3-1.7B-Base trained with GRPO on GSM8K, normalized RGC at rank 64 reached 83.24% test accuracy, compared with 83.70% for dense AdamW, using 8.53 GiB instead of 24.66 GiB of policy-training memory. That’s 65% less training memory, or 40% less over the full loop, including response generation.

Accuracy and peak GPU memory

RGC, normalized SGD LoRA Dense AdamW

Policy training

Policy training accuracy and peak memoryTest accuracy on GSM8K versus peak allocated GPU memory. AdamW: 83.70 percent, 24.66 GiB; LoRA rank 64: 83.62 percent, 6.75 GiB; LoRA rank 128: 83.78 percent, 7.78 GiB; LoRA rank 256: 83.62 percent, 9.96 GiB; RGC rank 64: 83.24 percent, 8.53 GiB; RGC rank 128: 82.03 percent, 8.73 GiB; RGC rank 256: 82.34 percent, 9.13 GiB. Labels identify rank.0510152025808182838485Peak allocated memory (GiB)Test accuracy (%)AdamW: 83.70%, 24.66 GiBAdamWLoRA 64: 83.62%, 6.75 GiB64LoRA 128: 83.78%, 7.78 GiB128LoRA 256: 83.62%, 9.96 GiB256RGC 64: 83.24%, 8.53 GiB64RGC 128: 82.03%, 8.73 GiB128RGC 256: 82.34%, 9.13 GiB256

Full loop

Full loop accuracy and peak memoryTest accuracy on GSM8K versus peak allocated GPU memory. AdamW: 83.70 percent, 25.37 GiB; LoRA rank 64: 83.62 percent, 13.16 GiB; LoRA rank 128: 83.78 percent, 13.93 GiB; LoRA rank 256: 83.62 percent, 15.12 GiB; RGC rank 64: 83.24 percent, 15.10 GiB; RGC rank 128: 82.03 percent, 15.11 GiB; RGC rank 256: 82.34 percent, 15.69 GiB. Labels identify rank.0510152025808182838485Peak allocated memory (GiB)Test accuracy (%)AdamW: 83.70%, 25.37 GiBAdamWLoRA 64: 83.62%, 13.16 GiB64LoRA 128: 83.78%, 13.93 GiB128LoRA 256: 83.62%, 15.12 GiB256RGC 64: 83.24%, 15.10 GiB64RGC 128: 82.03%, 15.11 GiB128RGC 256: 82.34%, 15.69 GiB256
Figure 1 Test accuracy and peak allocated GPU memory on Qwen3-1.7B-Base and GSM8K after 200 GRPO steps. Point labels give rank. The full loop includes response generation.

How it works

For a linear layer with input X and output derivative D, the weight gradient is D.T X. RGC never forms it. Each selected layer holds a random projection P with r columns and accumulates a narrow matrix instead:

C += D.T (X P)

The layer saves X P, not X, for backpropagation. After the 64 responses of a training step, RGC rebuilds the update H = C P.T 128 rows at a time and applies it to the original weights, so no dense gradient or dense update is ever held in memory. A new random projection is drawn after every update.1

We tried two ways of applying the update, neither of which stores momentum:

plain SGD:       W <- W - eta * H
normalized SGD:  W <- W - tau * norm(W) * H / norm(H)

The normalized rule, in the spirit of LARS, moves every weight matrix by the same fraction of its own norm. It beat plain SGD at all three ranks, by 11 to 27 test questions.

Both rules start from a calibrated step size: on the untrained model, we measured the size of a first AdamW step at learning rate 1e-6 and set RGC’s base step to match it, then tried that step and three times it.

Results

We trained each configuration for 200 GRPO steps with one training seed and selected the step size on development accuracy before any test evaluation. Training peak covers the policy update; whole-loop peak includes generating responses. Both columns report peak allocated GPU memory.

Method Test acc. Training peak Whole-loop peak
Untrained base 64.90% - -
Dense AdamW 83.70% 24.66 GiB 25.37 GiB
LoRA, rank 64 83.62% 6.75 GiB 13.16 GiB
LoRA, rank 128 83.78% 7.78 GiB 13.93 GiB
LoRA, rank 256 83.62% 9.96 GiB 15.12 GiB
RGC normalized, 64 83.24% 8.53 GiB 15.10 GiB
RGC normalized, 128 82.03% 8.73 GiB 15.11 GiB
RGC normalized, 256 82.34% 9.13 GiB 15.69 GiB

Every configuration improved substantially on the untrained base’s 64.90%, and every normalized RGC run finished above 82%. Across ranks 64, 128 and 256, normalized RGC finished within 0.4-1.7 test-accuracy points of same-rank LoRA and within 1.7 points of dense AdamW, using 63-65% less policy-training memory and 38-40% less whole-loop memory than dense AdamW.2

Memory you can predict

Above the BF16 model, RGC’s training state has two parts. Its compressed state is small: about 0.4 GiB at rank 128 for this model, and per rank about a fifth the size of LoRA’s adapter state. The larger part is that RGC keeps the weights it trains in FP32, a 2.6 GiB premium that LoRA avoids by freezing a BF16 base. LoRA instead pays for its adapters and their Adam state, which grows with rank.

Setting the two equal predicts where they cross: LoRA should use less training memory at ranks 64 and 128, and RGC less at rank 256. That is what we measured, with every predicted gap within about 0.1 GiB of the measured one. At rank 256, RGC’s training peak is 0.84 GiB (8.4%) below LoRA’s. Over the full loop, generation sets the peak, and RGC stays above LoRA at every rank, by 0.6 to 1.9 GiB.

The memory crossover, predicted before training

Predicted Measured, normalized RGC
Predicted and measured training memory relative to LoRARGC minus same-rank LoRA memory, in GiB. Rank 64: predicted plus 1.784, measured plus 1.783. Rank 128: predicted plus 0.943, measured plus 0.948. Rank 256: predicted minus 0.738, measured minus 0.838. Negative values mean RGC uses less training memory.-10+1+264128256Rank 64: predicted 1.784, measured 1.783 GiB+1.783 GiBRank 128: predicted 0.943, measured 0.948 GiB+0.948 GiBRank 256: predicted -0.738, measured -0.838 GiB-0.838 GiBRGC uses more memoryRGC uses less memoryRankRGC − LoRA training memory (GiB)
Figure 2 RGC minus same-rank LoRA training memory. The predicted gaps were +1.784, +0.943 and −0.738 GiB; measured gaps were +1.783, +0.948 and −0.838 GiB. Negative values mean RGC uses less memory.

What’s next

We’re building RGC into Actuator and plan to release it soon, so you can use it in your own RL training runs.

The full method, calibration, grid, paired comparisons and every candidate’s development trajectory are in the paper (PDF).

  1. P is a random orthonormal frame, obtained by QR factorization of a Gaussian matrix with the standard sign correction and scaled by sqrt(input width / rank). GoLore samples its low-rank gradient projections from the same distribution, unscaled. In exact arithmetic, RGC’s scale makes H an unbiased gradient estimate and attains the smallest worst-case error among unbiased reconstructions of the form G P P.T. The paper gives the derivation. ↩

  2. None of normalized RGC’s paired 95% intervals against same-rank LoRA or dense AdamW excludes zero. These paired-question intervals hold the selected checkpoints fixed; they do not measure variation across training seeds. ↩