Best-practice lists for model weights tend to be bland and safe. This one is not. These are opinionated positions earned from watching projects succeed and fail, and each comes with the reasoning so you can decide whether it applies to you.
The through-line is restraint. Most teams do too much: too big a model, too aggressive a quantization, too eager a fine-tune. The practices below push you toward doing less, measuring more, and earning each escalation. That discipline is what separates teams that ship from teams that tinker.
If you only adopt one thing here, make it the first practice. It prevents more wasted effort than all the others combined.
Default to the Smallest Model That Works
Start small and force yourself to justify going bigger. This inverts the usual instinct, and that is the point.
A smaller model is cheaper, faster, easier to run, and easier to fine-tune. The reasoning is that parameter count is a capacity ceiling, not a quality floor, and most real tasks sit well below the ceiling of a modern 7B-to-13B model. You should only move up when you have concrete test cases the smaller model fails.
- Begin with a small, well-regarded model.
- Build your evaluation set first so you can detect failure.
- Scale up only against measured shortfalls, never on a hunch.
This single habit prevents the most expensive mistake in the field, which the Common Mistakes article covers as its number one entry.
Treat Evaluation as a First-Class Artifact
Your test set is more valuable than your model. Build it before you do anything else and maintain it like code.
The reasoning is simple: without measurement, every decision about size, precision, and fine-tuning becomes a guess. A good evaluation set of 20 to 50 representative cases turns those guesses into experiments.
What makes a good eval set
- Real examples from your actual use case, not toy prompts.
- Hard cases and edge cases, because those are where quantization and fine-tuning quietly break.
- A clear notion of what "correct" means for each case, even if scored by hand.
Run this set every time you change the model, the precision, or the weights. The How-To guide builds baselining into its workflow for exactly this reason.
Quantize Deliberately, Not Maximally
Quantization is a tool, not a default setting to crank. The best practice is to quantize to the largest precision that fits and verify the cost.
The reasoning is that lower precision saves memory but spends quality, and the quality loss is uneven. It often hides on hard cases while looking fine on easy ones. So:
- Try the model at native precision first if it fits.
- Drop to 8-bit before 4-bit, and only go lower under real memory pressure.
- Measure the drop on your hard cases, not just average benchmarks.
A model that fits but fails your edge cases is worse than a smaller model that fits comfortably at higher precision.
Make Fine-Tuning the Last Resort
Reach for prompting and retrieval before you ever touch the weights. This is the most contested practice here, and the most important.
The reasoning is that fine-tuning adds permanent cost: data preparation, training, evaluation, versioning, and maintenance across model updates. Prompting and retrieval get you a large fraction of the benefit with none of that overhead. Fine-tune only when you have proven the base model cannot do the task even with excellent context.
When you do fine-tune, prefer parameter-efficient methods like LoRA. They freeze the original weights, run on modest hardware, and produce small swappable adapters that are easy to version and revert.
Use Conservative Training Settings
If you fine-tune, err toward slow and stable. Aggressive settings are the fastest way to ruin a model.
- Start with a low learning rate and increase only if learning stalls.
- Use fewer training steps than you think you need, then add more if results justify it.
- Watch for catastrophic forgetting, where general ability erodes as narrow skill grows.
The reasoning is that small datasets and high learning rates push weights too far, overfitting to your examples and losing the broad competence that made the base model useful.
Treat Weights as Supply-Chain Artifacts
Every weight file you load is code from the internet. Handle it accordingly.
- Prefer safetensors, which cannot execute code on load.
- Checksum downloads against the published hash.
- Record the exact base model, precision, and source for everything you ship.
The reasoning is both security and reproducibility. Legacy formats can run code on load, and undocumented weights cannot be reproduced or audited later. The Tools roundup covers the libraries that make this verification routine.
Version Everything You Ship
Treat models like releases. Name them, record what produced them, and keep the ability to roll back.
When a base model updates or a fine-tune regresses, versioning is the difference between a quick revert and a frantic investigation. Keep the checksum, the dataset, and the settings alongside each shipped model. The Checklist turns this into a standing routine.
Why These Practices Cluster Around Restraint
Read the practices back to back and a pattern emerges: nearly every one tells you to do less than your instinct suggests. Default to the smallest model. Quantize less aggressively. Fine-tune as a last resort. Train conservatively. This is not timidity; it is an accurate response to how these systems actually behave.
The reason restraint wins is that the costs of overreach are asymmetric. Reaching for a larger model, a lower precision, or an eager fine-tune adds cost, complexity, and failure modes immediately, while the benefit is often zero because the smaller, simpler choice already met the bar. You rarely regret starting conservative and scaling up on evidence, but you frequently regret starting aggressive and unwinding it. The Examples article shows both sides of this asymmetry playing out in real scenarios.
The one place restraint does not apply is measurement and verification. There you should do more, not less: a bigger evaluation set, stricter checksums, more thorough documentation. Those investments are cheap and pay back every time something changes downstream.
Frequently Asked Questions
Why default to small models when large ones are available?
Because parameter count is a capacity ceiling, not a quality guarantee, and most tasks do not need a large model. Small models cost less, run faster, and are easier to adapt. Starting small and scaling only on measured failure saves money and complexity without sacrificing results on most real work.
How big should my evaluation set be?
Twenty to fifty representative cases is a practical starting point for most tasks, weighted toward hard and edge cases. The exact number matters less than the quality and realism of the examples. The goal is enough coverage that a change in model, precision, or weights produces a visible, trustworthy signal.
Is fine-tuning ever the right first move?
Rarely. Prompting and retrieval solve most problems with far less overhead, so fine-tuning should follow only after those approaches demonstrably fall short. The exception is when you need a specific style, format, or behavior that prompting cannot reliably produce, and even then, parameter-efficient methods come first.
What precision should I serve a model at?
The largest precision your hardware allows. Native 16-bit gives full quality if it fits; 8-bit is a strong compromise; 4-bit is for real memory pressure. Always confirm the quality cost on your own hard cases rather than assuming the benchmark average applies to your task.
How do I keep fine-tuning from breaking the model?
Use a conservative learning rate, fewer steps, and parameter-efficient methods that freeze the original weights. Watch a held-out evaluation set for signs of catastrophic forgetting, where general ability declines. If quality on broad cases drops, reduce the learning rate or the number of steps and try again.
Key Takeaways
- Default to the smallest model that works and scale up only against measured failures.
- Build and maintain an evaluation set before anything else; it is more valuable than the model.
- Quantize to the largest precision that fits and verify the cost on hard cases.
- Make fine-tuning a last resort, prefer LoRA, and use conservative training settings.
- Treat weights as supply-chain artifacts: verify, document, and version everything you ship.