LoRA

A parameter-efficient technique for adapting large pretrained models.

What is LoRA?

LoRA keeps the original model weights frozen and introduces small trainable low-rank matrices into selected layers. Only these added parameters are updated during fine-tuning.

This greatly reduces memory and storage requirements compared with updating the complete model. Multiple LoRA adapters can also be created for different tasks while sharing the same base model.

Read the short notes and pictures.Do everything inside a blue frame.

What is LoRA?

LoRA stands for low-rank adaptation. It is a way to fine-tune a large model for a new job without retraining all of it. The original model is left untouched, and a small set of new numbers is trained beside it. The result is a tiny file, called an adapter, that changes how the big model behaves.

The original weights are frozen, and two small matrices are added.
The original weights are frozen, and two small matrices are added.

How LoRA works

  1. Freeze the model

    The billions of original weights are locked. They do not change.

  2. Add two small matrices

    Next to each chosen layer, LoRA adds two thin matrices, usually called A and B.

  3. Train only those

    Training adjusts A and B only: often well under one percent of the model’s size.

  4. Combine at run time

    The model’s output is its original weights plus the small correction from A and B.

One base model, with small adapters swapped in for different jobs.
One base model, with small adapters swapped in for different jobs.

The words you will meet

Fine-tuning

Training an existing model further on your own examples.

Adapter

The small file LoRA produces. It can be loaded or removed at any time.

Rank

How thin the two matrices are. A lower rank means a smaller, cheaper adapter.

Where it is used

Teaching a model your company’s tone or format

Custom styles for image generators

Fine-tuning on a single graphics card

Serving many customers from one base model

Keep these

  • LoRA fine-tunes a model by training a small add-on, not the whole model.
  • The original weights stay frozen.
  • The result is a small adapter file that can be swapped in and out.
  • It makes fine-tuning far cheaper and faster.

Begin the first module

Become AI native. It is the real deal today, and if it is not for you, you have lost nothing and learnt a new skill.