Custom Continued Pretraining
Most enterprises do not need a foundation model trained from scratch — they need an open base like Llama 3, Qwen 2.5, or Mistral that deeply understands their jargon, products, and document distributions. Continued pretraining fills that gap by extending training on a curated domain corpus before any instruction tuning happens. The customer problem is consistent: generic models hallucinate internal acronyms, misinterpret contracts or clinical notes, and require massive prompt scaffolding to stay on-domain. A continued-pretrained model reduces that friction at the weight level, which pays dividends across every downstream task from retrieval to classification to chat.
Delivery starts with corpus engineering, not GPUs. You audit the customer's data estate — wikis, tickets, code, PDFs, transcripts — then clean, deduplicate, filter for PII and licensing risk, and mix domain tokens with general replay data to avoid catastrophic forgetting. Training itself runs on managed multi-node capacity with checkpointing, loss-spike monitoring, and intermediate eval probes on both domain perplexity and general benchmarks like MMLU. You version every run with data manifests, tokenizer decisions, hyperparameters, and eval scores so the model is reproducible and auditable months later.
What makes this a factory rather than consulting is standardization of gates and artifacts. No model promotes without beating the base on an agreed domain eval suite while staying within a tolerance band on general capability and safety probes. Outputs land in a signed model registry with SBOM-style provenance, license notes, and a model card covering intended uses, limitations, and evaluation methodology. The customer owns the resulting weights and can self-host or hand them to your managed inference offering.
Commercially, package this as a fixed-scope data assessment followed by a per-run build fee plus registry and maintenance retainer. Acceptance criteria should be numeric and pre-registered: for example, +X points on internal QA accuracy, no more than Y points regression on general benchmarks, and passing toxicity and PII-leakage probes. This protects both sides from vibe-based acceptance and creates natural upsell into fine-tuning, distillation, and drift monitoring as the domain evolves.
Distillation Service
Frontier models are powerful but expensive to serve, slow at high concurrency, and often impossible to deploy on-prem or on-device. Distillation solves the economics by transferring capability from a large teacher into a small open-weights student that is cheaper, faster, and customer-owned. The buyer is typically a product team with a working prototype on a closed API and a painful inference bill, or a regulated buyer who loves the quality but cannot send data to a third party. Distillation gives them a path to keep 90%+ of the task quality at a fraction of the cost.
Execution requires care around rights and methodology. You only distill from teachers you have the right to distill from — customer-owned models, permissively licensed outputs, or synthetic data generated under compliant terms — and you document that chain explicitly. Practically, you collect or generate task-specific prompts, sample teacher completions with diversity controls, filter for correctness with judges and human spot-checks, then train the student via supervised fine-tuning and optionally preference optimization on judge-ranked pairs. You evaluate on held-out task sets plus stress sets for robustness, and you report quality alongside latency, throughput, and cost per thousand requests.
The tradeoff conversation is central to delivery. Smaller students excel on narrow, well-specified tasks but lose generality, long-context reasoning, and edge-case handling compared to the teacher. Your job is to map the task envelope honestly: define where the student is authorized to serve autonomously, where it should escalate to a larger model via routing, and where human review remains. That routing design often becomes a follow-on managed inference engagement, which is good for retention.
Package distillation as a cost-reduction program with a before/after $/correct report — dollars per correct task completion with confidence intervals — rather than raw accuracy alone. Pricing can include a success component tied to verified inference savings, plus a serving contract for the distilled artifact. Acceptance includes quality parity thresholds, latency SLOs, and a license memo confirming the student weights are cleanly owned and deployable in the customer's environment.
Model Merging & Surgery
Customers often accumulate multiple fine-tunes — one for support tone, one for code, one for a second language — and face a combinatorial explosion of endpoints to maintain. Model merging offers a cheaper alternative to joint retraining by combining existing checkpoints into a single multi-capable model through techniques like SLERP, TIES, or DARE. The value proposition is speed and cost: no new large training run, just systematic search over merge configurations validated against evals. It is explicitly experimental, which is why it works well as an opt-in factory lane with clear disclaimers.
Delivery is a search problem with guardrails. You constrain merges to compatible architectures and tokenizers, then run a structured sweep over layer-wise interpolation weights and sparsification parameters. Each candidate is scored on a composite suite covering every donor capability plus general safety and fluency probes to catch interference where one skill degrades another. Only Pareto-improving candidates advance, and you keep full lineage of parent checkpoints and merge recipes so results are reproducible and rollback is trivial.
Expect uneven results and say so upfront. Merges often preserve complementary skills well but struggle when donors conflict on style, language, or factual priors. Part of the service is diagnosing those conflicts and recommending when to stop merging and instead route between specialists or distill into a unified student. Vocabulary expansion, tokenizer swaps, and cross-architecture fusion fall under surgery and carry higher risk — quote them separately with explicit no-guarantee research terms.
Commercially, sell merging as a low-cost experiment sprint: fixed fee for a bounded search with a report showing per-task deltas versus maintaining separate models. If a merged candidate wins, it promotes through the same registry and eval-gated deployment as any factory output. If nothing beats the baseline, the customer still gains a map of task interference that informs the next fine-tune — and you gain credibility for honest gating rather than shipping a regression.