CoachAI
LLM Fine-Tuning - QLoRA - Evals & Guardrails
An AI coaching evaluation platform that fine-tunes Llama 3.1 8B with QLoRA and compares base and adapted models for fitness, nutrition, recovery, and performance coaching, with guardrails and three-method evaluation.
Problem
Would fine-tuning a small open model actually beat the base model for domain coaching? I assumed yes. The point of the project was to find out properly rather than assume.
Approach
QLoRA (4-bit NF4) on Llama 3.1 8B over 545 coaching Q&A examples, producing two adapters (r=16 and r=64) served side by side on a Modal A10G GPU. Evaluated three ways: manual spot-checks, LLM-as-judge scoring with Mistral 7B across weighted criteria, and live human preference logged to Supabase. A keyword blocklist plus an LLM intent classifier gate every query before inference.
Trade-off
The evaluation disagreed with me. The base model won on five of six weighted criteria: 545 examples were not enough, and not clean enough, to beat a well-trained base. I could have tuned the eval until the result flattered the work. I took the result instead.
Architecture
Fine-tuned Llama 3.1 8B using QLoRA (4-bit NF4) on 545 coaching Q&A examples, producing two LoRA adapters (r=16 and r=64) hosted on Modal A10G GPU. Evaluation uses three methods: manual spot-checks, LLM-as-judge (Mistral 7B), and human preference feedback logged to Supabase. Guardrails apply a keyword blocklist plus an LLM intent classifier to block off-topic queries. A React/Vite frontend loads both adapters side-by-side for response comparison, latency measurement, and feedback collection.
Results
The honest finding is more useful than the flattering one: this became a project about knowing when fine-tuning is and is not worth it. The real failure cases the evaluation surfaced went back into expanding the guardrail classifier, which is where they actually improved the product.
Highlights
- QLoRA 4-bit NF4 fine-tuning (r=16 and r=64)
- LLM-as-judge evaluation (Mistral 7B)
- Keyword + LLM intent guardrails
Technologies
- React
- Vite
- FastAPI
- Modal
- Transformers
- PEFT
- QLoRA
- Llama 3.1
- Supabase
Screenshots
