GeorgeZhang
ZH
Home Work Lab CV Focus Writing Contact 中文
Back to list

2025-03 - Present

Production-grade Agent Platform Architecture

Owned a cross-industry AI Agent customer-service SaaS platform from architecture evolution through production operations and expanded the team AI development harness across 40 repositories.
  • Agent Runtime
  • Tool-use
  • LLMOps
  • Evaluation
  • AI Harness

Overview

YC provides enterprise SaaS services across multiple industries, with AI Agent customer service as the core scenario. The Agent platform role covered architecture, runtime governance, knowledge and tool systems, quality governance, and team engineering standards.

The platform supports 2k+ tenants, 230k+ users, a production peak of 60k requests/day, and about 7s average end-to-end response time.

Platform Scope

Core capabilities include:

  1. Agent Runtime: session-level runtime, complex task handling, termination conditions, concurrency control, async execution, and callback delivery.
  2. Tool / MCP / Skills: HTTP Tool, MCP adapter, dynamic Skill load/unload, tool gating, and side-effect planner.
  3. RAG / Memory: documents, QA, websites, attachments, products, session memory, and context augmentation.
  4. Observability / Cost: correlation_id, events, inspection, token usage, callback payloads, and load-test metrics.
  5. Quality Governance: production issue cases, content gaps, human review, knowledge/Skill/Memory updates, replay evaluation, and release gates.

Data Feedback and Knowledge Evolution

Built an unknown-question clustering and topic discovery platform for customer-service Agents, turning unresolved, missed, or low-confidence questions into analyzable topic clusters for knowledge base updates, RAG optimization, and Skill iteration.

The module entered platform support in 2025. At that stage, the platform scale is estimated at about 50% of the current footprint: about 1,000+ tenants, 115,000+ users, and a peak of about 30,000 requests/day. On a weekly request basis, unknown questions moved from about 30% in the early stage to about 10%, then stabilized around 5%, corresponding to about 63,000, 21,000, and 10,500 unknown questions processed per week.

The technical flow included multilingual text cleaning, embeddings, UMAP/HDBSCAN, BERTopic, LLM topic representation, clustering quality metrics, experiment management, visualization reports, and result persistence. About 50% of clusters fed knowledge base, RAG configuration, or Skill iteration; high-frequency topic handling covered about 70% of identified themes; and content-gap closure contributed to about 30% improvement in customer-service Agent auto-resolution.

Architecture Evolution

Agent 1.0 centered on a controlled runtime. Guideline, Journey, and ARQ constrained LLM behavior inside an explainable, traceable, and reviewable business execution framework.

Agent 2.0 evolved into a model-driven runtime. The LLM handled tool calling and multi-step decisions, while the runtime owned context, action space, dynamic Skill loading, RAG/Memory, tool boundaries, concurrency, and cost governance.

The architecture continuously balances control, task quality, context efficiency, tool risk, and extension cost, turning production constraints into Runtime capabilities.

Team Engineering Governance

YC has a product-engineering team of about 20 people. The AI Agent track included a 3-person squad of 2 developers and 1 QA. The AI development harness was expanded across 40 repositories so AI-assisted engineering became part of a controlled delivery process.

The governance model has three layers:

  1. Outer Loop: architecture rules, language profiles, layering boundaries, naming, error handling, logging, and observability expectations.
  2. Inner Loop: Spec / EPIC workflow for non-trivial changes, including goals, non-goals, acceptance criteria, phases, tasks, and status.
  3. Supervisor Gate: change summary, changed files, self-check result, AI Harness trace, and acceptance checklist as delivery gates.

Unified specifications, process checks, and delivery gates reduce architecture drift, runaway refactors, inconsistent quality, and acceptance gaps while improving review efficiency, delivery consistency, and rollback readiness.

Stack

The main stack includes Python, TypeScript, Rust, Java, FastAPI, Milvus, Alibaba Cloud, Google Cloud, and Gemini.

The stack supports one platform goal: Agent Runtime, tool systems, RAG/Memory, evaluation gates, observability, and team engineering governance.

Delivery Value

Core delivery value includes:

  • Bringing Agent capabilities into real customer-service workflows.
  • Platformizing tool integration, approval, runtime authorization, and audit boundaries.
  • Bringing prompts, tool schemas, model versions, and RAG strategies into release governance.
  • Bringing unknown-question clustering, content-gap detection, and topic discovery into the knowledge-evolution loop.
  • Turning Agent Runtime, Skills, RAG/Memory, and multi-channel delivery into reusable platform assets.
  • Owning reliability, cost, quality, delivery process, and business outcomes.