AI agents + RAG: reference architectures that survive production

AI agents + RAG: reference architectures that survive production

Enterprises don't need hobby projects; they need AI agents that answer, act, and audit reliably. Retrieval augmented generation consulting exists because the difference between a demo and a durable system is architecture, tooling, and discipline. Below are pragmatic reference architectures, the tools that actually ship, and the failure modes we see repeatedly in full-cycle product engineering.

Reference architecture 1: Orchestrated agent atop RAG microservices

Use a dedicated orchestrator (Temporal, LangGraph, or custom state machine) to coordinate retrieval, tool use, reasoning, and guardrails. Treat RAG as a microservice with strict contracts and observability. Key components:

• Ingestion: streaming connectors (Kafka, Debezium) normalize docs; apply PII scrubbing and canonical schemas.
• Indexing: hybrid store (PGVector or Pinecone + BM25) with per-domain namespaces; maintain a change log for backfills.
• Retrieval: lexical + dense retrieval; add a cross-encoder reranker (Cohere Rerank or bge-rerank) for precision.
• Chunking: semantic splitting with overlap; create "document profiles" (length, entropy, update cadence) to tune chunk size.
• Reasoning: tool-aware LLM with function calling; plan/execute/verify loops are capped by cost and latency budgets.
• Guardrails: schema validators, policy filters, and red-teaming probes in CI.
• Observability: trace each turn; log prompts, retrieved spans, and final citations for audit.

Reference architecture 2: Event-driven RAG for streaming decisions

For support, trading, or ops, an event bus drives low-latency retrieval with time-aware ranking. Maintain an online store for hot embeddings and an offline store for cold history. Apply time decay to favor recent facts, and snapshot the context included in decisions to ensure replayability during audits.

Tooling choices and trade-offs

Frameworks: LangChain accelerates prototypes; Semantic Kernel fits .NET estates; LlamaIndex shines for retrieval plumbing. Pick one, then freeze versions. Vector DBs: Pinecone for managed scale, Weaviate for hybrid search and filters, PGVector when you want fewer moving parts. Embeddings: OpenAI text-3-large or Voyage for multilingual; E5-large or bge for self-hosted cost control. Rerankers deliver more value per dollar than ever-larger embeddings-adopt them early. For agents, prefer constrained tools with JSON schemas over free-form instructions; it reduces hallucinated API calls dramatically.

Pitfalls to avoid

• Chunking mismatch: giant chunks bury answers; tiny chunks shatter context. Measure retrieval hit rate vs. answer exactness, not intuition.
• Retriever drift: schema or taxonomy changes silently degrade recall. Schedule synthetic queries that cover every domain and alert on drops.
• Stale indices: without CDC-based reindexing and tombstones, you cite deprecated facts. Tie index versions to source commits.
• Over-embedding: embedding everything every hour burns budget. Track document entropy and update only changed spans.
• Metric myopia: BLEU/ROUGE won't save you. Use groundedness, citation accuracy, and task success time.
• Context bloat: shoving 200K tokens increases latency and contradictions. Tighten retriever precision before enlarging context windows.
• Security gaps: retrieved snippets can exfiltrate secrets. Classify, mask, and enforce ABAC at retrieval time.

Evaluation and governance

Adopt a two-tier eval: offline synthetic suites to block regressions, online interleaving to measure user impact. For RAG, track the "GRC triad": grounding (evidence present), relevance (evidence matches question), and consistency (answer aligns with evidence). Add refusal quality for unsupported queries. Canary new retrievers to 5% traffic with shadow logging; rollback on grounding dips beyond threshold. Keep a red-team corpus of jailbreaks and policy edge cases in CI.

Case snapshots

• B2B knowledge base: After adding hybrid retrieval + rerank and per-product namespaces, first-contact resolution rose 28%, while average handle time fell 17%.
• E-commerce support: Agent tools with strict JSON schemas cut API mishits by 63%; caching embeddings on SKU updates saved 42% in monthly spend.
• Compliance summarization: Time-decayed retrieval prevented outdated policy citations; human review workload dropped 31% with auditable citations.

Team models that work

Cross-functional pods (platform, data, app, security) deliver faster than siloed teams. Gun.io engineers shine when you need rapid augmentation of senior talent to unblock orchestration, retrieval, or ops. Likewise, slashdev.io provides excellent remote engineers and software agency expertise for business owners and startups to realize their ideas without sacrificing enterprise rigor. If you need burst capacity or domain specialists, Retrieval augmented generation consulting partners can de-risk design while your core team focuses on full-cycle product engineering.

Implementation checklist

• Define tasks and SLOs (latency, cost/query, groundedness floor); choose a rollback condition.
• Map data sources, access rules, and retention; design CDC for indexes.
• Select a retriever triad: BM25 + dense + rerank; calibrate per domain.
• Establish eval harnesses with synthetic and real queries; automate drift alerts.
• Build an orchestrator with tool schemas, retries, and budget-aware planning.
• Ship an audit trail: prompts, retrieved spans, citations, and tool outputs.
• Run a canary, compare business KPIs, then scale progressively.

Budgeting and ROI

Control spend with caching (prompt and embedding), dynamic truncation, and query brokering (cheap model for recall, premium for answer). Set latency SLOs aligned to business value; for async tasks, batch and stream partials. Most savings come from fewer, better chunks plus reranking-not from bigger LLMs. Treat RAG like search: relevance wins, and everything else is decoration.