Dataset Design for Finetuning

Data quality beats model size

Small, clean datasets usually outperform large noisy datasets. The highest leverage work is curation.

A practical schema for instruction data

Each row should include:

1. instruction: what the user asks.
2. context: optional grounding.
3. response: ideal assistant output.
4. tags: difficulty, domain, safety labels.

{
  "instruction": "Summarize this outage report in 5 bullets.",
  "context": "<incident log>",
  "response": "- Root cause...",
  "tags": ["ops", "summary", "format:bullets", "difficulty:medium"]
}

Coverage checklist

Make sure your data spans:

1. Easy, medium, and hard examples.
2. Ambiguous prompts requiring clarification.
3. Safety-sensitive requests and refusal behavior.
4. Edge-case formatting and long-context scenarios.

Common failure modes

• Mode collapse: model repeats one style for all prompts.
• Overfitting to template phrasing.
• Missing negative examples (what not to do).
• Label inconsistency across annotators.

Quick annotation rules

1. Keep responses concise unless depth is explicitly requested.
2. Prefer deterministic formats for operational tasks.
3. Ban hallucinated sources.
4. Add one "gold" example per high-risk workflow and re-test each release.