Fine-Tuning a Sub-2B LLM for Text-to-SQL on a Single Consumer GPU

This post is the story of a pipeline I built as a final-project-style exercise. The brief, in its boring form:

Can a sub-2-billion-parameter open-weight model, fine-tuned with LoRA on a single desktop GPU, reach usable accuracy on text-to-SQL — and then actually serve its own predictions, not just emit numbers in a notebook?

The headline result: 73.8 % combined execution accuracy on a 940-prompt held-out Spider-derived test set, using a Qwen3-1.7B base

a rank-16 wide LoRA, merged to GGUF, served by llama-server on a single AMD RX 9060 XT (16 GB, ROCm 7.13).

What follows covers the data, the training runs, the ablations that told me which knob mattered, and the engineering around running the whole thing end-to-end on commodity hardware.

Why sub-2B, and why on a single GPU

Larger models would have been easier. A 7B-class base would have smoothed over a lot of the fragility I had to chase down. But:

The 16 GB on my RX 9060 XT is the budget, not the goal.
I wanted the merged GGUF to be reasonable to ship — a 1–2 B model in Q4 fits in roughly 1.2 GB at inference.
The interesting story isn't "GPT-4 got it." It's which trade-offs collapse when you force the model below 2 B.

Two bases that fit:

Model	Params	Why I picked it
Llama-3.2-1B	1.24 B	Smallest Llama-family model — sanity baseline
Qwen3-1.7B	1.7 B	Reported strong structured-output behavior at this size

I did not evaluate larger models. The point of the project was the hardware constraint, not a leaderboard chase.

Data: Spider-derived, no chain-of-thought

I used the AI4DS/sql_generator_no_cot dataset — a Spider-derived split with question/SQL pairs and no chain-of-thought scaffolding in the target. That's deliberate: I wanted to measure whether the base model could produce SQL directly when fine-tuned, not whether it could narrate its way to a query.

9 399 question/SQL pairs total
90 / 10 split → 8 459 train / 940 held-out test
Each training row gets an explicit stop token so the model learns to stop
Test set is never seen during training or model selection

The test set is evaluated execution-based, not by string match: I parse the predicted SQL, run it against an in-memory SQLite engine with a canonical schema, and compare the result rows. This catches SELECT name vs SELECT NAME and alias-swap mistakes that string-exact metrics reward and execution-reality would punish. (I report both numbers, because string-exact is what the leaderboards publish — the gap between the two is the most interesting part of the result.)

Training: two LoRA shapes, one shared ladder

I ran five configurations on a single ladder:

Configuration	Combined accuracy	String-exact	Execution-equiv
Llama-3.2-1B base (zero-shot)	0.0 %	0.0 %	0.0 %
Qwen3-1.7B base (zero-shot)	1.8 %	0.2 %	3.3 %
Llama-3.2-1B + attn-only LoRA	63.7 %	12.3 %	50.7 %
Llama-3.2-1B + wide LoRA	69.5 %	14.2 %	54.5 %
Qwen3-1.7B + wide LoRA	73.8 %	16.5 %	57.3 %

Three effects are cleanly separable:

LoRA vs nothing. Both bases go from < 2 % to > 63 % with rank 8 on q,k,v,o_proj only. Fine-tuning is doing the real work; the base models genuinely do not speak SQL.
attn-only vs wide. Adding gate_proj, up_proj, down_proj and bumping rank 8 → 16 jumps both bases by ~6 points on combined accuracy. On a 1B-class model the MLP matters.
Llama-3.2-1B vs Qwen3-1.7B. Qwen wins by 4 points with the same wide-LoRA recipe. Same training data, same pipeline, same hardware. The base model isn't background scenery.

All training was done with LLaMA-Factory on top of ROCm 7.13. The training configs are kept for reproducibility — there's nothing magic in them.

Merging + GGUF quantization: where the project would have died

A merged GGUF was the make-or-break engineering step. Without it, the project would have been a notebook result, not a system. Three pieces had to be glued together:

HF → merged HF with LLaMA-Factory's llamafactory-cli merge
merged HF → GGUF with the upstream convert_hf_to_gguf.py
ROCm llama-server running the resulting GGUF in 4-bit

The full pipeline (merge → GGUF → quantize → serve) was wired up end-to-end. Re-running it cleanly reproduces the headline numbers within ±0.5 % on the same hardware.

The non-obvious failure modes, in case someone else hits them:

ROCm needs the matching prebuilt llama.cpp binary, not pip install. Pull the right ROCm artifact and the GGUF converter explicitly.
Qwen3 GGUF conversion currently expects the instruct tokenizer config, not base. Account for that explicitly.
VOCAB-merging mismatches show up as a silent, garbage-text symptom — don't trust a clean exit code without diffing a few generations by hand.

Serving + manual testing: actually using the model

Once the GGUF was served on :8080, I wanted to use it on real schemas. Three test fixtures covered the spectrum:

A small school schema — three tables, ~30 rows, five graded prompts.
A medium ecommerce schema — a few tables, ~17 rows, four graded prompts.
A larger university schema — ten tables, ~850 rows, twelve graded prompts.

The interactive REPL is the part I like most. It loads a fixture into SQLite, gives the model the schema in the prompt, and lets you type questions in natural English:

fixture> how many students are enrolled in the deep-learning unit?
generated SQL:
  SELECT COUNT(*) FROM enrolment e JOIN course c ON e.course_id = c.id
  WHERE c.code = 'COMP9999'
result: 47

After each turn the REPL prints the model's generated SQL, the rows it returned, and — if you're running a graded fixture — whether the rows match the expected answer. The gradedness is what made the 940-prompt eval trustworthy.

What the ablations actually cost (in time)

Reproducing all five configurations on a single RX 9060 XT 16 GB:

Step	Approx wall time
Download base models	~10 min
Download + prep dataset	~2 min
Llama wide-LoRA train	~80 min
Qwen wide-LoRA train	~110 min
Llama attn-only train	~25 min
Zero-shot eval (940 × 2)	~80 min
Merge → GGUF → quant	~10 min
Total	~5 h

This is not a "leave it overnight" job. You can run a single configuration during a coffee break and have evaluation JSONs to look at before lunch.

What I'd do differently (with infinite time)

Schema-conditioning via system prompt. Feeding the full DDL at inference time, not just in training, would probably close half the remaining 26 % gap.
Constrained decoding. A grammar-based decoder (Outlines, Guidance) guarantees well-formed SQL tokens and pushes the string-exact number up dramatically. The merged GGUF path makes this awkward — I'd need to keep the HF path in the loop for safety-critical serving.
More base models. Phi-3.5-mini and Gemma-2-2B were on my shortlist but didn't fit the time budget.

The takeaway is unsurprising but worth saying out loud: at the sub-2B scale, base-model choice and LoRA coverage matter more than hyperparameter sweeping. The difference between Llama-3.2-1B and Qwen3-1.7B at the same LoRA recipe is larger than anything a learning-rate sweep on a single model will find.

Closing notes

The interesting numbers in this post were reached by combining a small open-weight base, a careful LoRA shape choice, and a working merged-GGUF + ROCm serving path on commodity hardware. The pipeline itself isn't shared publicly — this write-up is the artifact.

If you want to chase the same result, the recipe in one line is: fine-tune a ~1–2 B base on a Spider-derived text-to-SQL dataset (no chain-of-thought), evaluate execution-based against an in-memory SQLite engine, and don't trust a single number that hasn't been checked against the rows it returned.