LoRA Target Modules Explained: q_proj, v_proj, k_proj & More

Confused by LoRA target modules? Learn what q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj actually do. When to use them. How to choose. With code and real analogies.

The Problem That Brought You Here

You decide to fine-tune an LLM with LoRA. You install Hugging Face PEFT, open a tutorial, and see this:

target_modules=["q_proj", "v_proj"]

Easy enough. Then another tutorial says:

target_modules=[
    "q_proj",
    "k_proj",
    "v_proj",
    "o_proj"
]

Then somebody on Reddit or GitHub recommends:

target_modules=[
    "q_proj",
    "k_proj",
    "v_proj",
    "o_proj",
    "gate_proj",
    "up_proj",
    "down_proj"
]

And now you’re sitting there thinking:

“Bhai, konsa sahi hai?”

An even more basic question usually comes first:

“What the hell do all these _proj things actually do?”

Most LoRA tutorials show you the config but never explain the decision hiding behind it. This guide fixes that.

By the end, you’ll know:

• what q_proj, k_proj, v_proj, and o_proj actually do (no jargon)

• what gate_proj, up_proj, and down_proj are for

• why everyone and their mother uses q_proj and v_proj

• why QLoRA goes nuclear with all-linear

• how to stop blindly copying configs from the internet

The 30-Second Cheat Sheet (For the Impatient)

Quick Starting Point (If You Just Want Code)

That table is useful—but understanding why those options differ is much more valuable.

First: What Does target_modules Even Mean in LoRA?

LoRA does not retrain every parameter of your LLM. That would be expensive and stupid.

Instead, the pretrained model stays mostly frozen while LoRA inserts small trainable low-rank matrices into selected weight matrices.

Think of it like this:

You have a giant library with 100,000 books. Instead of rewriting every single book to add your new knowledge, you just add sticky notes to a few carefully chosen books.

target_modules tells LoRA which books get the sticky notes.

from peft import LoraConfig

config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"]  # Only these get sticky notes
)

This tells PEFT: “Find modules matching q_proj and v_proj and inject LoRA adapters there.”

It does not mean the entire Transformer is being adapted. That makes target_modules one of the biggest knobs controlling LoRA capacity.

Where Do These Projection Modules Actually Live?

Before we dive into each one, let’s see where they sit in the Transformer.

A simplified Llama-style Transformer block looks like this:

                    ┌─────────────────────────────────┐
                    │         Input Token             │
                    └───────────────┬─────────────────┘
                                    │
                                    ▼
                    ┌─────────────────────────────────┐
                    │        SELF-ATTENTION           │
                    │                                 │
                    │  ┌──────┐  ┌──────┐  ┌──────┐  │
                    │  │q_proj│  │k_proj│  │v_proj│  │
                    │  └──┬───┘  └──┬───┘  └──┬───┘  │
                    │     │         │         │       │
                    │     └─────────┼─────────┘       │
                    │               │                 │
                    │          Attention              │
                    │               │                 │
                    │               ▼                 │
                    │          ┌──────────┐           │
                    │          │  o_proj  │           │
                    │          └──────────┘           │
                    └───────────────┬─────────────────┘
                                    │
                                    ▼
                    ┌─────────────────────────────────┐
                    │        MLP / FFN                │
                    │                                 │
                    │  ┌──────────┐  ┌──────────┐    │
                    │  │ gate_proj│  │ up_proj  │    │
                    │  └────┬─────┘  └────┬─────┘    │
                    │       │             │          │
                    │       └──────┬──────┘          │
                    │              │                 │
                    │         Activation             │
                    │              │                 │
                    │              ▼                 │
                    │         ┌──────────┐           │
                    │         │down_proj │           │
                    │         └──────────┘           │
                    └───────────────┬─────────────────┘
                                    │
                                    ▼
                    ┌─────────────────────────────────┐
                    │        Next Block              │
                    └─────────────────────────────────┘

The first four modules live in ATTENTION.
The last three live in the MLP (Feed-Forward Network).

Now let’s understand each one.

q_proj — The “What Am I Looking For?” Guy

q_proj produces the Query vectors. It’s the part of the model that asks:

“What information am I looking for?”

Example:

Suppose the model sees: “The capital of France is…”

The current token needs to figure out: “What previous information might be useful here?” The Query projection creates that search-like representation.

Mathematically:

Q = X × W_Q

Where X is the incoming representation and W_Q is the Query matrix.

Real-world analogy:

Imagine Google Search. You type something in. q_proj is like the search box — it’s how you express what you’re looking for. Same knowledge base. Different way of asking.

When you target q_proj with LoRA: You’re teaching the model how to ask better questions.

k_proj — The “What Do I Have?” Guy

If Query is “What am I looking for?” then Key is:

“What information do I contain that might be useful to someone else?”

Mathematically:

K = X × W_K

Attention compares Queries with Keys to figure out who should talk to whom.

Real-world analogy:

Imagine a conference. You need someone who understands PyTorch. One person’s badge says “Frontend Developer.” Another says “PyTorch Performance Engineer.” Those badges act as Keys. Your Query is compared against those Keys to find the right person.

When you target k_proj with LoRA: You’re teaching the model how to advertise its own relevance to others.

v_proj — The “What Info Actually Moves?” Guy

Now suppose attention has found the right tokens. What information actually flows through?

That’s Values.

V = X × W_V

The simplified equation:

Attention = softmax((Q × K^T) / √d) × V

Notice how Values only appear at the end? That’s because they’re the payload — the actual information being transferred.

Real-world analogy:

• Query = “Who in this room knows about CUDA?”

• Key = Badges showing what everyone knows

• Value = The actual knowledge that gets shared when you find the right person

When you target v_proj with LoRA: You’re teaching the model what information should actually flow through attention.

o_proj — The “Let Me Combine Everything” Guy

Transformers use multiple attention heads. Each head focuses on something different:

• One head might look at grammar

• Another might track relationships

• Another might connect questions to answers

These need to be combined back into one coherent representation. That’s o_proj.

Think: “How should all the attention heads’ findings be merged together?”

Real-world analogy:

You have a team of analysts. Each analyst produces a report. o_proj is the person who reads all the reports and writes one final summary that actually makes sense.

When you target o_proj with LoRA: You’re teaching the model how to combine information from different attention heads.

Attention Is Only Half the Story !!

A common beginner misconception: “Transformers are just attention.”

They’re not.

After attention, each block has a large feed-forward network (MLP) that does most of the heavy lifting.

In Llama/Mistral architectures, you’ll see three more projections:

• gate_proj

• up_proj

• down_proj

These are also major linear transformations—and natural LoRA targets.

up_proj — The “Let Me Think Bigger” Guy

The model’s hidden representation has a certain size (d_model). The MLP expands it into a much larger space.

Why? Because having more space gives the network room to construct richer features.

Conceptually:

Hidden state → up_proj → Giant workspace

Real-world analogy:

You start with a short question. Then you open a giant whiteboard and spread out: calculations, possibilities, relationships, ideas. That giant workspace is up_proj.

When you target up_proj with LoRA: You’re teaching the model how to expand features for deeper thinking.

gate_proj — The “What Actually Matters?” Guy

Many modern models use a gated MLP (like SwiGLU). The gate decides:

“Which intermediate features should actually matter for this token?”

Real-world analogy:

Imagine a kitchen with 100 ingredients. up_proj puts all 100 on the counter. gate_proj decides which 5 actually go into the dish.

When you target gate_proj with LoRA: You’re teaching the model how to filter what’s important.

down_proj — The “Bring It Back” Guy

After the MLP has done its thing, the expanded representation needs to be compressed back to the model’s hidden dimension.

That’s down_proj.

Conceptually:

Hidden state → up_proj → Giant workspace → down_proj → Hidden state

Real-world analogy:

If up_proj opens the giant whiteboard, down_proj writes the key conclusions back into your notebook.

When you target down_proj with LoRA: You’re teaching the model how to summarize expanded thinking back into the main stream.

Why Does Everyone Use Only q_proj and v_proj?

Because it works well enough for most tasks while being extremely lightweight.

The original LoRA paper showed that full fine-tuning isn’t required. You can get great results by just adapting a few strategic matrices.

So ["q_proj", "v_proj"] became the default.

But remember: It’s a baseline, not a universal optimum.

Why QLoRA Uses Everything

QLoRA does things differently. It uses all-linear because:

1. It can. Quantization frees up enough VRAM to adapt more modules.

2. It often works better. Adapting more places gives the model more capacity to learn.

3. It’s convenient. "all-linear" saves you from typing all seven names.

# QLoRA-style config
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules="all-linear",  # Everything!
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM
)

But: all-linear doesn’t mean the same thing everywhere. Different architectures expose different modules. Always verify:

print(peft_model.targeted_module_names)

How Many Parameters Are We Talking About?

This is the part most tutorials skip.

Suppose a matrix has shape 4096 × 4096. With rank r = 16:

Trainable params for one matrix = r × (d_in + d_out)
                                = 16 × (4096 + 4096)
                                = 16 × 8192
                                = 131,072

Now:

Lesson: Rank controls capacity per module. Target modules control how many modules you insert that capacity into.

Both decisions matter.

A Better Way to Choose: Run Three Experiments

Instead of arguing which is theoretically best, run controlled experiments.

Experiment A — Minimal

target_modules=["q_proj", "v_proj"]

Experiment B — Full Attention

target_modules=[
    "q_proj",
    "k_proj",
    "v_proj",
    "o_proj"
]

Experiment C — Broad

target_modules="all-linear"

Keep everything else constant:

• same base model

• same dataset

• same rank

• same alpha

• same seed

Then compare:

That turns target selection from guesswork into engineering.

The Golden Rule: Inspect Your Damn Model

Different architectures use different names.

If you copy ["q_proj", "v_proj"] from a Llama tutorial and use it on GPT-2, it will fail.

Inspect Like a Pro :

# Print all module names
for name, module in model.named_modules():
    print(name)

Even better — inspect only linear modules:

import torch

for name, module in model.named_modules():
    if isinstance(module, torch.nn.Linear):
        print(name)

Verify After Configuration

# After creating the PEFT model
peft_model.print_trainable_parameters()
print(peft_model.targeted_module_names)

This simple check can save you hours of wasted training.

Common Mistakes That Waste Your Time

1. Copying Configs From a Different Architecture

A Llama config won’t work on GPT-2. Inspect first.

2. Looking Only at Rank

r = 32, targets = ["q_proj", "v_proj"]  # ~262K params
r = 16, targets = all 7 modules          # ~917K params

These are not comparable by rank alone. Total adaptation budget = rank × number of targets.

3. Assuming all-linear Does the Same Thing Everywhere

It doesn’t. Different architectures → different modules.

4. Evaluating Only Training Loss

Lower training loss can just mean “I memorized the training set.” Use a real validation set.

Practical PEFT Configurations :

Attention-Only LoRA

from peft import LoraConfig, TaskType

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj"
    ],
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM
)

QLoRA-Style Broad Targeting

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules="all-linear",
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM
)

The second configuration is more convenient when you want to cover most linear projections without manually naming them.

The Decision Table :

The Bottom Line :

Stop thinking of LoRA target modules as magic incantations. They’re just places where you’re allowing the model to learn.

Start simple:

target_modules=["q_proj", "v_proj"]

Expand only if needed.

Never blindly copy a config from the internet.

And always remember:

The rank controls how much capacity each adapter has. Target modules control how many places you use that capacity.

Both matter. Choose wisely.

FAQ (Because People Keep Asking)

What are LoRA target modules?

They’re the model layers into which LoRA adapters are inserted. In Hugging Face PEFT, you specify them using target_modules inside LoraConfig.

Why are q_proj and v_proj commonly used?

They provide a compact way to adapt important parts of self-attention without adding LoRA adapters everywhere. Lightweight baseline.

What does q_proj do?

It creates Query vectors. Mental model: “What information should this token look for?”

What does k_proj do?

It creates Key vectors. Mental model: “What information does this token advertise as relevant?”

What does v_proj do?

It creates Value vectors. Mental model: “What information from this token should actually flow through attention?”

What does o_proj do?

It projects the combined multi-head attention output back into the model’s hidden representation.

What are gate_proj, up_proj, and down_proj?

They’re part of the MLP/feed-forward network. up_proj expands, gate_proj filters, down_proj compresses.

Should I target all seven modules in Llama?

Not automatically. It’s a common broad configuration, especially for QLoRA, but you should compare it against a smaller Q/V or Q/K/V/O configuration.

What does target_modules=”all-linear” do?

It asks PEFT to target applicable linear layers automatically. For Llama/Mistral, this covers the major attention and MLP projections. Always verify.

Is all-linear better than q_proj and v_proj?

Not universally. It provides more adaptation capacity but also increases trainable parameters, VRAM, and compute. Let your data decide.

How do I find LoRA modules for an unknown model?

for name, module in model.named_modules():
    if isinstance(module, torch.nn.Linear):
        print(name)

LoRA configs on the internet are suggestions. Your model, your data, and your validation loss are the only things that actually matter. So stop copying, start experimenting, and let your metrics be the judge. 

If you’re still confused about what r, alpha, and target modules actually do, start with our beginner-friendly breakdown before diving into production configs. Check out these links:- 

1. Demystifying LoRA: What Rank (r), Alpha, and Target Modules Actually Do – Demystifying LoRA: What Rank (r), Alpha, and Target Modules – neuralninjas.in
2. LoRA in LLMs and Agentic RAG: The Complete Production Guide – LoRA in LLMs and Agentic RAG: The Complete Production Guide – neuralninjas.in
3. Fine-Tune Factory 🏭 – Fine-Tune Factory — LLM Fine-Tuning, Live