The LoRA paper's headline is that it trains 10,000 times fewer parameters, and the number is honest — GPT-3 175B with rank-4 adapters on the query and value projections has 18.9M trainable parameters against 175.2B, a ratio of 9,281. What almost nobody checks is what that ratio does to the three quantities a training run is actually billed for. Peak memory falls 7.9 times, not 9,281. Training FLOPs fall 1.5 times. Merged inference latency falls not at all. The gap between the headline and the bill is the whole subject, and it is arithmetic rather than opinion.

<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 760 470" width="100%" style="max-width:760px" font-family="Helvetica,Arial,sans-serif">
<rect width="760" height="470" fill="#0d1117"/>
<text x="28" y="34" fill="#e6edf3" font-size="19" font-weight="700">The 803x is real. It divides three things nobody was paying for.</text>
<text x="28" y="54" fill="#8b98a5" font-size="13">Llama-2 7B, LoRA r=16 on q and v, mixed-precision AdamW &#8212; one configuration, six ratios.</text>
<text x="28" y="82" fill="#e6edf3" font-size="13" font-weight="700">what LoRA divides, and by how much</text>
<line x1="176.0" y1="96" x2="176.0" y2="326" stroke="#21262d" stroke-width="1"/>
<text x="176.0" y="338" fill="#6e7681" font-size="10" text-anchor="start">1x</text>
<line x1="242.0" y1="96" x2="242.0" y2="326" stroke="#21262d" stroke-width="1"/>
<text x="242.0" y="338" fill="#6e7681" font-size="10" text-anchor="middle">10x</text>
<line x1="308.0" y1="96" x2="308.0" y2="326" stroke="#21262d" stroke-width="1"/>
<text x="308.0" y="338" fill="#6e7681" font-size="10" text-anchor="middle">100x</text>
<line x1="374.0" y1="96" x2="374.0" y2="326" stroke="#21262d" stroke-width="1"/>
<text x="374.0" y="338" fill="#6e7681" font-size="10" text-anchor="middle">1,000x</text>
<text x="166" y="118" fill="#8b98a5" font-size="12" text-anchor="end">trainable parameters</text>
<rect x="176" y="104" width="191.7" height="19" fill="#58a6ff" rx="2"/>
<text x="359.7" y="118" fill="#0d1117" font-size="12" font-weight="700" text-anchor="end">803x</text>
<text x="166" y="158" fill="#8b98a5" font-size="12" text-anchor="end">AdamW optimizer state</text>
<rect x="176" y="144" width="191.7" height="19" fill="#58a6ff" rx="2"/>
<text x="359.7" y="158" fill="#0d1117" font-size="12" font-weight="700" text-anchor="end">803x</text>
<text x="166" y="198" fill="#8b98a5" font-size="12" text-anchor="end">gradient buffer</text>
<rect x="176" y="184" width="191.7" height="19" fill="#58a6ff" rx="2"/>
<text x="359.7" y="198" fill="#0d1117" font-size="12" font-weight="700" text-anchor="end">803x</text>
<text x="166" y="238" fill="#8b98a5" font-size="12" text-anchor="end">peak training memory</text>
<rect x="176" y="224" width="59.4" height="19" fill="#d29922" rx="2"/>
<text x="242.4" y="238" fill="#d29922" font-size="12" font-weight="700">7.9x</text>
<text x="166" y="278" fill="#8b98a5" font-size="12" text-anchor="end">training FLOPs per step</text>
<rect x="176" y="264" width="11.6" height="19" fill="#f85149" rx="2"/>
<text x="194.6" y="278" fill="#f85149" font-size="12" font-weight="700">1.5x</text>
<text x="166" y="318" fill="#8b98a5" font-size="12" text-anchor="end">inference latency, merged</text>
<rect x="176" y="304" width="1.5" height="19" fill="#f85149" rx="2"/>
<text x="183.0" y="318" fill="#f85149" font-size="12" font-weight="700">1.0x</text>
<line x1="172" y1="100" x2="172" y2="216" stroke="#58a6ff" stroke-width="2"/>
<line x1="172" y1="220" x2="172" y2="332" stroke="#f85149" stroke-width="2"/>
<text x="28" y="352" fill="#58a6ff" font-size="11" font-weight="700">the top three are accounting.</text>
<text x="28" y="367" fill="#8b98a5" font-size="11">they count parameters that were never the binding cost.</text>
<text x="28" y="388" fill="#f85149" font-size="11" font-weight="700">the bottom three are the bill.</text>
<text x="28" y="403" fill="#8b98a5" font-size="11">memory falls 7.9x, compute 1.5x, latency not at all.</text>
<line x1="432" y1="76" x2="432" y2="418" stroke="#21262d" stroke-width="1"/>
<text x="462" y="82" fill="#e6edf3" font-size="13" font-weight="700">where the 100.4 GB actually sits</text>
<rect x="462" y="198.5" width="74" height="157.5" fill="#f85149"/>
<text x="499" y="274" fill="#0d1117" font-size="11" font-weight="700" text-anchor="middle">optimizer</text>
<text x="499" y="288" fill="#0d1117" font-size="11" text-anchor="middle">75.3 GB</text>
<rect x="462" y="172.2" width="74" height="26.2" fill="#d29922"/>
<text x="499" y="189" fill="#0d1117" font-size="11" font-weight="700" text-anchor="middle">gradients</text>
<rect x="462" y="146.0" width="74" height="26.2" fill="#58a6ff"/>
<text x="499" y="163" fill="#0d1117" font-size="11" font-weight="700" text-anchor="middle">weights</text>
<text x="499" y="138.0" fill="#e6edf3" font-size="12" font-weight="700" text-anchor="middle">100.4 GB</text>
<text x="499" y="373" fill="#8b98a5" font-size="12" text-anchor="middle">full</text>
<rect x="566" y="354.8" width="74" height="1.2" fill="#3fb950"/>
<rect x="566" y="328.6" width="74" height="26.2" fill="#58a6ff"/>
<text x="603" y="346" fill="#0d1117" font-size="11" font-weight="700" text-anchor="middle">frozen</text>
<text x="603" y="320.6" fill="#e6edf3" font-size="12" font-weight="700" text-anchor="middle">12.7 GB</text>
<text x="603" y="373" fill="#8b98a5" font-size="12" text-anchor="middle">LoRA</text>
<rect x="670" y="354.8" width="74" height="1.2" fill="#3fb950"/>
<rect x="670" y="348.2" width="74" height="6.6" fill="#8957e5"/>
<text x="707" y="340.2" fill="#e6edf3" font-size="12" font-weight="700" text-anchor="middle">3.2 GB</text>
<text x="707" y="373" fill="#8b98a5" font-size="12" text-anchor="middle">QLoRA</text>
<line x1="452" y1="356" x2="746" y2="356" stroke="#30363d" stroke-width="1"/>
<line x1="636" y1="328" x2="636" y2="300" stroke="#3fb950" stroke-width="1"/>
<line x1="636" y1="300" x2="684" y2="300" stroke="#3fb950" stroke-width="1"/>
<text x="746" y="278" fill="#3fb950" font-size="11" font-weight="700" text-anchor="end">adapters: 0.11 GB</text>
<text x="746" y="292" fill="#8b98a5" font-size="11" text-anchor="end">99% of the rest is the frozen model</text>
<text x="462" y="398" fill="#8957e5" font-size="11" font-weight="700">QLoRA reaches 31x, and the extra factor of 3.9</text>
<text x="462" y="413" fill="#8b98a5" font-size="11">comes from a 4-bit number format, not from the rank.</text>
<line x1="28" y1="436" x2="732" y2="436" stroke="#21262d" stroke-width="1"/>
<text x="28" y="456" fill="#8b98a5" font-size="12">Nothing on the left is a rank effect. Rank sets what an adapter can learn; these ratios are set by which tensors carry optimizer state.</text>
</svg>

The parameter ratio is real, and it is a ratio of the wrong quantity. Under the standard mixed-precision AdamW recipe a trainable parameter costs 16 bytes: two for the bf16 weight used in the forward pass, two for the bf16 gradient, and twelve for the fp32 master copy plus the first and second moments. A frozen parameter costs two. So the memory of a run is

and that limit is the entire story. Driving the trainable fraction from one part in eight hundred to one part in ten thousand moves the memory ratio from 7.92 to 7.99. The ratio was never going to exceed 8, because 14 of the 16 bytes belong to the optimizer and the other 2 belong to weights that have to be resident whether they move or not. Llama-2 7B is 100.4 GB to fine-tune fully and 12.7 GB under LoRA at rank 16 on q and v — and 12.55 GB of that 12.7 GB is the frozen model itself. The adapters, their gradients and their Adam moments together are 0.11 GB, under one percent of what remains. Every additional order of magnitude of parameter efficiency is competing for that last one percent.

Read the last two columns against the fifth. The parameter ratio moves by a factor of 11.6 down that table and the memory ratio moves by 1.3 percent. They are not measuring the same thing and they never were.

LoRA removes half of the backward pass, not the backward pass. A forward pass through a linear layer costs one unit of compute, call it C. The backward pass costs 2C, and the two halves are different jobs: one C propagates the gradient of the loss to the layer's input, and one C contracts that gradient with the layer's input to form the gradient of the weights. Freezing a matrix deletes the second job and leaves the first entirely intact, because the adapters in earlier layers still need their gradients and the only route to them is back through every frozen matrix above. Full fine-tuning is therefore 1C + 2C = 3C per step and LoRA is 1C + 1C = 2C, a ratio of 0.667 and a saving of 33 percent — set against a trainable-parameter saving of 99.88 percent in the same configuration. Turn on gradient checkpointing, which nearly every long-context recipe does, and the recompute adds one more C to both sides: 4C against 3C, a saving of 25 percent. This is why a LoRA run does not finish in a tenth of the time. It finishes in about two-thirds of the time, and the step-time win that people report as much larger than that is almost always an optimizer-state effect showing up as a larger batch that fits, not as faster arithmetic.

Once the optimizer term is gone, the frozen weights are the memory, so the next factor has to come from a number format. This is the observation QLoRA is built on, and it follows directly from the identity above: with the 14 bytes of optimizer and gradient removed, the 2N term is all that is left, so the only remaining lever is the number of bytes per frozen parameter. Dettmers and colleagues replaced bf16 with a 4-bit NormalFloat quantisation and the 2N became 0.5N. Llama-2 7B goes 100.4 GB to 12.7 GB to 3.3 GB — 31 times a full fine-tune, and the last factor of 3.9 is a data type, not a rank. The 70B case is the one that changed practice: a full fine-tune holds 771 GB of AdamW state alone, ten A100-80GB cards before a single activation tensor, and the 4-bit LoRA holds the entire model plus its adapters in 32.6 GB, which is one card. Almost every claim in circulation about LoRA making 70B fine-tuning possible on a single GPU is a claim about quantisation wearing LoRA's name.

A rank sweep at fixed learning rate is also a learning-rate sweep. LoRA does not train B and A, it trains the product s·BA where s is a scaling constant, and the original formulation sets s = alpha/r. With B initialised to zero the first gradient step moves that product by roughly the learning rate times s squared, so the rank sits inside the effective step size, squared. Rank goes up by 16 and the effective step falls by 256. Running the experiment rather than asserting it — least-squares regression onto a rank-2 target, so that every rank tested is over-parameterised and no part of the result can be an expressivity effect, identical learning rate and alpha throughout, median of three seeds — the number of steps needed to cut the loss by 90 percent goes 2,180 at r=4, 2,960 at r=8, 4,240 at r=16, 7,100 at r=32 and 12,300 at r=64. More rank is strictly more expressive, and under alpha/r more rank converges 5.6 times slower. The sweep runs backwards. Kalajdzievski's rank-stabilised scaling replaces alpha/r with alpha over the square root of r, and on the identical problem the same sweep goes 1,780 to 1,180 steps — 1.5 times faster with 16 times the rank, which is the direction expressivity predicts. The practical consequence is blunt: a rank sweep at a fixed learning rate under the default scaling measures the learning rate, and the widely repeated finding that rank barely matters is partly an artifact of that confound.

Where the adapters go decides what a given rank buys. Rank is one dimension of the decision and target-module selection is the other, and the second is cheaper. On Llama-2 7B, adapters on q and v at rank 8 cost 4.19M parameters; on all four attention projections, 8.39M; on every linear matrix in the block including the gate, up and down projections, 19.99M. That all-linear configuration at rank 8 costs less than q and v alone at rank 64 — 19.99M against 33.55M — and it reaches the feed-forward block, which in a gated transformer layer holds about 67 percent of the parameters against attention's 33 percent. Rank 64 on q and v is a very expressive adapter attached to a third of the model. Rank 8 on everything is a less expressive adapter attached to all of it, for fewer trainable parameters, and on most instruction-tuning tasks it wins. Hugging Face PEFT's all-linear target mode exists because this comparison keeps coming out the same way.

Merging is exact and free for one adapter, and it is catastrophic for a thousand. The merge is not an approximation: W plus s·BA is a plain matrix of the same shape, so a merged adapter adds exactly zero inference parameters and exactly zero latency, and every benchmark showing LoRA inference is as fast as the base model is measuring a merged one. The trouble is that merging is destructive of sharing. Ten task-specific adapters merged into ten copies of a 7B model in bf16 is 125.5 GB of weights; one shared base plus ten unmerged adapters is 12.7 GB. At a thousand adapters it is 12,551 GB against 28.2 GB, a factor of 445. So a multi-tenant server keeps adapters unmerged and pays a small batched low-rank matmul per token, which is what S-LoRA, Punica and the multi-LoRA paths in vLLM are for, while a single-tenant deployment merges and pays nothing. LoRA's inference cost is a serving-topology decision. It is not a property of the method, and the two published answers to "does LoRA slow down inference" are both correct about different deployments.

LoRA's real failure mode is that it cannot move what it cannot reach. Everything above is accounting; this is the part that decides whether the run should have been LoRA at all. The frozen set in a standard configuration includes the embedding matrix and the output head — 262M parameters on 7B, four percent of the model and all of the token geometry. A task that needs new tokens, a new script, or a language the tokenizer fragments badly is asking the model to move exactly the tensors LoRA left alone, and no rank fixes that. The same argument applies to continued pretraining on a new domain, where the update is genuinely high-rank and spread across every layer: the low-rank constraint is a real constraint and the literature that finds LoRA matching full fine-tuning is almost entirely instruction-tuning and preference-optimisation literature, where the update is small, late and close to something the base model already does. Biderman and colleagues' careful comparison found LoRA lagging full fine-tuning substantially on code and mathematics continued pretraining and matching it on instruction tuning, and that split is the honest boundary of the method — with the compensating property, in their results, that LoRA forgets the base distribution less, because it moved less.

{ "rows": [
  { "type": "nodes", "items": [
    { "title": "\"use LoRA, it is 10,000x cheaper\"", "sub": "a ratio of trainable parameters, and of nothing else", "tone": "orange" }
  ] },
  { "type": "arrow" },
  { "type": "group", "title": "First: which resource is actually binding?", "note": "the parameter ratio predicts one of these three and it is not the one you think", "items": [
    { "title": "Memory", "sub": "16N to 2N + 16T, so the ratio asymptotes at 8x however small the rank", "tone": "green" },
    { "title": "Compute", "sub": "3C to 2C, a 33% saving; 25% once gradient checkpointing is on", "tone": "orange" },
    { "title": "Inference latency", "sub": "1.0x merged, and a batched low-rank matmul unmerged", "tone": "red" }
  ] },
  { "type": "arrow" },
  { "type": "group", "title": "If memory is binding, the lever is bytes per frozen weight", "note": "2N is all that is left once the optimizer term is gone", "items": [
    { "title": "bf16 LoRA", "sub": "7B in 12.7 GB, of which 12.55 GB is the frozen model", "tone": "green" },
    { "title": "NF4 QLoRA", "sub": "7B in 3.3 GB, 70B in 32.6 GB - one card, from the data type", "tone": "green" }
  ] },
  { "type": "arrow" },
  { "type": "group", "title": "Then configure, in this order", "note": "placement first, scaling second, rank last", "items": [
    { "title": "Target modules", "sub": "all-linear at r=8 costs 19.99M, less than q,v at r=64 at 33.55M", "tone": "green" },
    { "title": "Scaling", "sub": "alpha/sqrt(r) or retune lr per rank - alpha/r hides a 256x step change", "tone": "orange" },
    { "title": "Rank", "sub": "the cheapest dimension to be wrong about, and the one most often swept", "tone": "orange" }
  ] },
  { "type": "arrow" },
  { "type": "group", "title": "Reconsider LoRA entirely when the update is not small", "note": "the frozen set includes 262M parameters of embedding and output head on 7B", "items": [
    { "title": "New tokens, scripts or languages", "sub": "the tensors that must move are the frozen ones", "tone": "red" },
    { "title": "Continued pretraining on a new domain", "sub": "genuinely high-rank, spread over every layer", "tone": "red" },
    { "title": "Serving hundreds of tasks", "sub": "LoRA is the right answer here for a serving reason, not a training one", "tone": "green" }
  ] }
] }

Read LoRA through a memory-hierarchy lens rather than a parameter-count lens. The parameter count is a true statement about an object that was never expensive; the bill is written in optimizer bytes, in backward-pass FLOPs and in how many copies of the base model a serving fleet has to hold, and each of those has its own ratio and its own ceiling. Memory stops at 8 times because the frozen weights still have to be resident. Compute stops at 1.5 times because the gradient still has to reach the earliest adapter. Latency does not move at all once the matrices are merged. The one genuinely large factor anywhere in the method — QLoRA's 31 times — comes from writing the frozen weights in four bits, which is a decision about number formats that LoRA merely made affordable. Rank is the last thing to tune and the first thing everybody tunes, and the question that should come before any of it is whether the change being asked for is small enough to live in a low-rank correction to the matrices that were left unfrozen.