Open-Weights Agent Stacks with Tool Use
Agents fail in production for operational reasons: unreliable tool calling, unbounded loops, unversioned prompts, and no evaluation of multi-step trajectories. An open-weights agent stack solves this by standardizing orchestration, tool schemas, and observability on self-hosted models, so customer data never transits a third-party API. The buyer is usually a product team with a demo that works on closed models and a security review that blocks deployment — you give them the same capability inside their boundary.
Delivery centers on the agent loop as an engineered system. You define tool contracts with strict input/output validation, budgeted step limits, and deterministic fallback paths when the model emits malformed calls. Prompts, tool definitions, and routing policies are versioned artifacts behind the same eval-gated CI as any LLM change. Tracing captures the full span tree — model calls, retrievals, tool executions — so failures are debuggable as structured traces rather than chat transcripts. Small fine-tunes for tool-calling reliability on the customer's schema often outperform prompt patching alone.
Evaluation must cover trajectories, not just final answers. Offline suites score task completion, tool-call validity, faithfulness to retrieved context, and safety constraints across multi-step scenarios; online evals sample production trajectories for drift in escalation rates and loop lengths. Cost controls matter because agentic loops multiply tokens — per-task budgets, early-exit heuristics, and escalation to larger models only on hard steps keep unit economics viable. Human-in-the-loop hooks for high-stakes actions are designed in from the start, not bolted on after an incident.
Package agent stacks as use-case sprints: one workflow automated end-to-end with evals, runbooks, and a rollback path, then expansion to adjacent workflows. Pricing combines build fees with per-task serving and eval maintenance retainers. Acceptance includes trajectory-level eval scores beating the prototype baseline, tool-error rates below threshold, and a live red-team session demonstrating bounded behavior on adversarial inputs. The orchestrator itself stays fungible — durable assets are the prompts, evaluators, datasets, and tool contracts the customer owns.
RAG-as-a-Service on Customer Infra
Retrieval-augmented generation remains the highest-ROI enterprise pattern: ground open models in customer documents to cut hallucinations without retraining. But production RAG is a data pipeline, not a demo — ingestion, chunking, embeddings, indexing, access control, and retrieval evals all determine whether answers are trustworthy. RAG-as-a-service delivers that pipeline inside the customer's infrastructure with vector databases, embedding jobs, and evaluation harnesses as managed primitives rather than bespoke builds per workload.
Technical delivery starts with document engineering. You build connectors for SharePoint, Confluence, ticketing systems, and file shares, then implement parsing, deduplication, chunking strategies tuned per document type, and metadata extraction for filtering. Embeddings run as versioned batch and streaming jobs with index versioning so retrieval behavior is reproducible. At query time, hybrid retrieval combining dense vectors with keyword search plus reranking consistently beats pure vector search on enterprise corpora, and access-control filtering ensures users only retrieve documents they are permitted to see — a non-negotiable for regulated buyers.
What elevates this from plumbing to a service is retrieval evaluation. You maintain golden QA sets with known supporting passages, scoring context recall, faithfulness of answers to retrieved context, and abstention quality when nothing relevant exists. Chunking or embedding changes pass through eval gates before promotion, and production traces showing low retrieval scores feed curation loops that fix source documents or add missing content. Object storage for corpora plus audit logs of what was retrieved for each answer complete the compliance story.
Commercially, scope by corpus and use case: pilot on one high-value knowledge base with clear success metrics like answer faithfulness and support deflection, then expand connectors and languages. Pricing mixes ingestion volume, index hosting, and quarterly eval maintenance. Acceptance criteria should be behavioral — for example, faithfulness above threshold on held-out questions with correct access-control enforcement verified by permission-matrix tests — plus a demonstration that index rebuilds are versioned and rollback-capable.