Read the RGC preprint (PDF) for the full method, experiments and full-loop recipe. Developed with the help of Marvin, our industry-leading agentic co-scientist. Last updated 25 October 2026.
Reinforcement learning improves how language models reason, but updating every weight is expensive: dense AdamW keeps FP32 weights, a gradient and two optimizer moments. Randomized Gradient Compression (RGC) trains the original attention and MLP projection weights from a compressed gradient, keeps those weights in BF16, and stores no optimizer state at all. For Qwen3-8B at rank 64, its persistent training state is 2.06 bytes per parameter, the model’s BF16 weights plus 3%, against 13.9 for dense AdamW.
Trained with GRPO on GSM8K, RGC uses less memory than LoRA at every matched rank, in training and over the full RL loop. Chunked generation and loss bring the full-loop peaks to 18.7 GiB for Qwen3-8B and 29.1 GiB for Qwen3-14B, only 6% above the 14B model’s own weights. Dense AdamW needs about 106 and 200 GiB respectively for its weights, gradients and optimizer state alone.
Memory use and test accuracy
Memory bars share a linear 0–30 GiB scale. RGC uses BF16 weights and normalized SGD.
How it works
For a linear layer with input X and output derivative D, the weight gradient is D.T X. RGC never forms it. Each selected layer holds a random projection P with r columns and accumulates a narrow matrix instead:
C += D.T (X P)
The layer saves X P, not X, for backpropagation. After the 64 responses of a training step, RGC rebuilds the update H = C P.T 128 rows at a time and applies it to the original weights, so no dense gradient or dense update is ever held in memory. A new random projection is drawn after every update.1
LoRA leaves each pretrained matrix fixed and trains a rank-r term added to it. RGC changes the matrices themselves, and redrawing the projection lets successive updates accumulate beyond rank r.
Normalized SGD moves each weight matrix by the same fraction of its own norm, with no momentum:
W <- W - tau * norm(W) * H / norm(H)
The weights stay in BF16 without an FP32 copy. A typical update is one to three millionths, about 0.4-1.2% of the gap between neighbouring BF16 values, so ordinary rounding would erase most updates. With stochastic rounding, a weight that should move 1% of the gap moves the whole gap 1% of the time, making it correct on average. In our runs, an update changed 1.8-4.4% of the weights on average.
We calibrated the step size to match a first AdamW step at learning rate 1e-6 on the untrained model, then tried that step and three times it on the 1.7B model.
Results
Runs train for 200 GRPO steps, with 1.7B step sizes selected on development accuracy before test evaluation. Training peak covers the policy update; full-loop peak includes response generation. Both report peak allocated GPU memory.
| Method | Test acc. | Training peak | Full-loop peak |
|---|---|---|---|
| Untrained base | 64.90% | - | - |
| Dense AdamW | 83.70% | 24.66 GiB | 25.37 GiB |
| LoRA, rank 16 | 82.11% | 5.95 GiB | 12.44 GiB |
| LoRA, rank 64 | 83.62% | 6.75 GiB | 13.16 GiB |
| LoRA, rank 128 | 83.78% | 7.78 GiB | 13.93 GiB |
| LoRA, rank 256 | 83.62% | 9.96 GiB | 15.12 GiB |
| RGC, rank 8 | 82.94% | 5.71 GiB | 12.27 GiB |
| RGC, rank 16 | 83.32% | 5.74 GiB | 12.42 GiB |
| RGC, rank 64 | 83.70% | 5.89 GiB | 12.57 GiB |
| RGC, rank 128 | 80.59% | 6.08 GiB | 12.77 GiB |
| RGC, rank 256 | 82.11% | 6.48 GiB | 13.04 GiB |
RGC’s training peak stays between 5.71 and 6.48 GiB from rank 8 to 256. At matched rank it uses 3.5-35% less training memory than LoRA. At rank 64 it matches dense AdamW’s 83.70% accuracy; at rank 16 it scores 1.2 points above LoRA at the same rank.2
The full loop
Generation usually sets an RL step’s memory peak: decoding all 64 responses at once holds a key/value cache for each, while the loss holds full-vocabulary logits for a whole response. We decode 16 responses at a time at 8B and four at 14B, compute the output head and loss over 128-token chunks, and release RGC’s compressed buffers during generation. The recipe also works with LoRA.
On Qwen3-8B, this cuts RGC’s full-loop peak by a third, from 27.95 to 18.67 GiB, with essentially unchanged training-loop time and 93.10% test accuracy.
Full-weight RL, close to model memory
Qwen3-8B-Base
Qwen3-14B-Base
Weight rows show BF16 model storage; other rows show measured full-loop peaks with the recipe. Bars share a linear 0–32 GiB scale.
Qwen3-8B-Base
| Method | Test acc. | Training peak | Full-loop peak |
|---|---|---|---|
| Untrained base | 85.82% | - | - |
| RGC, rank 64 | 92.34% | 18.45 GiB | 27.95 GiB |
| RGC, rank 64, second run | 92.80% | 18.44 GiB | 27.78 GiB |
| RGC, rank 128 | 93.18% | 18.94 GiB | 28.27 GiB |
| RGC, rank 64, full-loop recipe | 93.10% | 16.43 GiB | 18.67 GiB |
| LoRA, rank 64 | 93.78% | 20.57 GiB | 29.24 GiB |
| LoRA, rank 16, full-loop recipe | 92.04% | 16.67 GiB | 18.70 GiB |
| Dense AdamW | Out of memory during its first optimizer step | - | - |
With the full-loop recipe, RGC at rank 64 and LoRA at rank 16 both peak at 18.7 GiB. RGC scores 1.1 points higher and completes its 200-step training loop in 336 minutes versus LoRA’s 451. Dense AdamW needs about 106 GiB for its weights, gradients and optimizer state alone, and ran out of memory during its first optimizer step.
Qwen3-14B-Base
RGC trains every attention and MLP projection with a 29.1 GiB full-loop peak, just 6% above the model’s own 27.51 GiB of weights, and lifts accuracy from 90.2% to 94.8%. Dense AdamW would need about 200 GiB for its weights, gradients and optimizer state alone, before activations.
Memory you can predict
Above the BF16 model, RGC keeps only its compressed state, about a fifth the size of LoRA’s adapters and Adam state at the same rank. That puts RGC below LoRA at every rank, with savings that grow as rank increases.
We predicted RGC’s training-memory gap to LoRA before each run, and all nine predictions landed within 0.12 GiB. At 1.7B, predicted savings at ranks 16, 64, 128 and 256 were 0.21, 0.84, 1.68 and 3.36 GiB; measured savings were 0.21, 0.86, 1.69 and 3.48 GiB. At 8B, RGC at ranks 64 and 128 saved 2.12 and 1.63 GiB against LoRA at rank 64, versus predictions of 2.11 and 1.62 GiB.
Memory savings, predicted before training
The advantage also grows with sequence length. LoRA keeps each layer’s full input for its adapter gradients; RGC keeps only the rank-r projection. At 16,384 tokens, RGC at rank 64 uses 1.6 GiB less policy-training memory than LoRA at rank 16 on the 8B model.
What’s next
We’re building RGC into Actuator and plan to release it soon, so you can use it in your own RL training runs.
The full method, calibration, paired comparisons, full-loop recipe and every run’s development trajectory are in the paper (PDF).
-
Pis a random orthonormal frame, obtained by QR factorization of a Gaussian matrix with the standard sign correction and scaled bysqrt(input width / rank). GoLore samples its low-rank gradient projections from the same distribution, unscaled. In exact arithmetic, RGC’s scale makesHan unbiased gradient estimate and attains the smallest worst-case error among unbiased reconstructions of the formG P P.T. The paper gives the derivation. ↩ -
Accuracy is measured on 1,319 GSM8K test questions. Paired-question intervals and run details are in the paper. The reported training-loop times exclude setup, evaluation and checkpointing. ↩