Fine-Tuning Guide: LoRA vs Full Fine-Tune for LLMs

When base models don't quite fit your needs, fine-tuning is the answer. But which approach should you use? Let's break down LoRA (Low-Rank Adaptation) vs full fine-tuning.

The Fine-Tuning Spectrum

Less Resource-Intensive                    More Resource-Intensive
         │                                          │
         ▼                                          ▼
┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐
│   Prompt    │  │    LoRA     │  │   QLoRA     │  │    Full     │
│ Engineering │  │             │  │  (4-bit)    │  │ Fine-Tune   │
└─────────────┘  └─────────────┘  └─────────────┘  └─────────────┘
     Free           Low Cost       Medium Cost       High Cost
   No Training     Minutes-Hours    Hours-Days       Days-Weeks

What is LoRA?

LoRA (Low-Rank Adaptation) freezes the original model weights and injects trainable low-rank matrices into each layer. Instead of updating all 7B+ parameters, you only train a small fraction.

How LoRA Works

Original Model Weights (Frozen)
         │
         ▼
    ┌─────────┐
    │  W₀     │  (frozen, say 4096 x 4096)
    └─────────┘
         │
         ├──────────────────────┐
         │                      │
         ▼                      ▼
    ┌─────────┐            ┌─────────┐
    │    A    │            │    B    │
    │ (4096×r)│            │ (r×4096)│
    └─────────┘            └─────────┘
         │                      │
         └──────────┬───────────┘
                    │
                    ▼
              W = W₀ + BA

Where r (rank) is typically 8-64, making trainable parameters < 1% of total.

When to Use LoRA

✅ Good For:

Domain adaptation: Teaching a model legal, medical, or technical jargon
Style transfer: Making outputs match your brand voice
Code generation: Specialized for your tech stack
Quick iteration: Testing multiple approaches rapidly
Limited compute: Consumer GPUs (RTX 3090, 4090)

❌ Not Ideal For:

Major behavior changes: Fundamental shifts in model capabilities
Multi-modal tasks: Adding new modalities
Knowledge injection: Teaching large amounts of new factual info

When to Use Full Fine-Tuning

✅ Good For:

Custom base models: Building specialized models from scratch
Significant capability changes: Major behavioral modifications
Enterprise deployments: When quality matters more than cost
Knowledge-intensive tasks: Legal research, medical diagnosis

❌ Not Ideal For:

Small datasets: Risk of overfitting
Limited budget: Requires significant compute
Fast iteration: Slow turnaround time

Hyperparameter Recipes

LoRA Config (Good Starting Point)

lora_config = LoraConfig(
    r=16,                       # Rank (8-64 typically)
    lora_alpha=32,              # Scaling factor (usually 2x rank)
    lora_dropout=0.1,           # Dropout for regularization
    bias="none",                # Don't train biases
    task_type="CAUSAL_LM",      # For text generation
    target_modules=[            # Which layers to adapt
        "q_proj",
        "k_proj", 
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj"
    ]
)

Training Arguments

training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.1,
    logging_steps=10,
    save_steps=100,
    fp16=True,
)

QLoRA: The Best of Both Worlds

QLoRA combines LoRA with 4-bit quantization, making it possible to fine-tune large models on consumer hardware.

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=bnb_config,
    device_map="auto",
)

Hardware Requirements

Method	VRAM Required	Time (7B model)	Cost
LoRA	16-24GB	Hours	$
QLoRA	8-16GB	Hours	$
Full Fine-Tune	80GB+	Days	$$$$

Data Quality > Data Quantity

For fine-tuning, quality matters more than quantity:

1,000 high-quality examples often beats 100,000 mediocre ones
Ensure diversity in your training data
Clean and deduplicate rigorously
Format consistently (same prompt template)

Pre-Made Fine-Tuning Recipes

Our marketplace offers ready-to-use fine-tuning configurations:

Code generation LoRAs for specific languages/frameworks
Domain-specific adapters (legal, medical, financial)
Style transfer configs for different writing voices
Hyperparameter presets tested on popular models

Save weeks of experimentation with battle-tested recipes from the community.