Config Enhancer Semantic Guardrails

Date: 2026-07-23 Scope: services/agent Status: implemented locally; not deployed

Incident

Online session 84664120166982656 used chatbot 69ff0f7c7eef888daa8369d8 and config hash 8b59ff1880c515850498c5e5abb4e0f5.

The cached enhanced config exposed three independent problems:

1. The generated instructions string contained a serialized copy of sibling fields such as timers, constraints, and skills.
2. The source sequence “after 20 minutes, then after another 10 minutes” became two independent idle timers with delays 1200 and 600. Idle timers share the latest-user-activity baseline, so the closing message would run before the first follow-up.
3. Skill enhancement rewrote runtime instructions as well as routing descriptions. Several configured conditions were shortened or lost, and a concrete repeated-charge complaint was routed to a generic loan-consultation skill.

The final answer also treated its self-reported evidence value as sufficient support. A general instruction was used to infer a specific merchant relationship and charge cause that were not established by the available instructions, skill, Knowledge Base material, or tool result.

Production Scope

A six-hour online log sample contained:

Measure	Count
Prompt contexts	812
Chatbots	20
Config hashes	23
Sessions	319
Embedded sibling-JSON occurrences	115

All 115 occurrences belonged to chatbot 69ff0f7c7eef888daa8369d8 and config hash 8b59ff1880c515850498c5e5abb4e0f5. Every observed prompt for that config was affected; the other 22 sampled config hashes had no matching occurrence. This was therefore deterministic for one persisted enhanced config, not a platform-wide prompt-logging behavior.

Enhanced Config Comparison

The cached config was inspected through:

https://omni-flux.ycloud.com/api/v1/config/69ff0f7c7eef888daa8369d8

Area	Raw configuration	Enhanced configuration	Assessment
Main instructions	2,968 characters	4,809 characters	The clean 2,868-character prefix equals the raw text after removing the 100-character timeout section; the remaining tail is unrelated sibling JSON
Timeout sequence	Follow up after 20 minutes; close after another 10 minutes	Independent delays 1200 and 600	Semantic inversion; runtime would close first
Consultation-fee policy	Broad rule that all products include a consultation fee	Preserved unchanged	The risky policy originated in user configuration, not enhancement
Constraints	Present in the SOP	Copied to top-level constraints and retained in the SOP	Matches the current copy-only contract
Skills	Seven raw skills with detailed execution rules	Seven skills with rewritten descriptions and instructions	Membership and tools remained, but several execution details were shortened or changed

Skill Semantic Review

Skill	Enhancement result	Risk
Own bank account / IBAN	Core rule retained; post-thanks continuation and several explicit prohibitions removed	Medium
Amount / borrowing intent	Amount range retained; boundary with generic consultation remained weak and explicit forbidden-output details were reduced	Medium
Website / application entry	Conditional button/card capability became unconditional	High
Loan rejection	Core behavior retained	Low
Proxy application	Core behavior retained	Low
Page/upload failure	“Website or contact information” condition was narrowed to “link”	Medium
Generic loan consultation	Repayment option and useful routing distinctions were removed; selected for a concrete repeated-charge complaint	High

The agreed platform boundary is not to rewrite these customer-owned business rules. Skills are outside enhancement and pass through exactly as configured.

Case Root Cause

The observed answer was produced by a combined failure chain:

1. The user supplied transaction notifications and later reported that the charge had happened three times.
2. The skill router selected the generic loan-consultation skill with confidence 0.85, despite the latest message describing a concrete payment complaint.
3. The customer-authored SOP contained a broad consultation-fee rule.
4. No trusted source established the specific merchant relationship or charge cause.
5. The model marked evidence=instruction, but the value was self-reported and did not perform claim-level validation.
6. The first final envelope was invalid. During envelope retry, the model added a new fee explanation instead of limiting the retry to JSON repair.

The embedded sibling JSON was a separate config-integrity defect. It made the prompt confusing and oversized, but it was not the direct source of the fee claim.

Agreed Scope

• Keep the existing full-config JSON response contract for ConfigEnhancer.
• Assign retained SOP text to instructions and keep every sibling output in its declared top-level field.
• Model idle and fixed timer baselines explicitly. Chained relative waits sharing a mode and baseline are cumulative; independent actions keep their source delays.
• Exclude skills from enhancement input and output; preserve raw skills exactly.
• Strengthen answer grounding so an evidence label cannot authorize a claim and a broad policy cannot establish a specific entity, event, transaction, or causal relationship.
• Preserve supported answer meaning during final-envelope repair; a JSON retry must not introduce new business facts.
• Remove the redundant original_config cache field. Build the performance evaluation's originalAgentConfig view from basic_settings.instruction and top-level skills when requested.
• Bump CONFIG_CACHE_VERSION to config_enhance_v2, causing old L1/L2 entries to miss and be enhanced again.

Non-Goals

• No customer-specific SOP or skill rewrite.
• No incident-specific production matching branch.
• No extraction-plan/source-span architecture.
• No change to timer runtime scheduling.

Prompt Design

Production wording uses three general semantic invariants:

1. Field ownership: each generated value belongs to its declared schema field.
2. Time semantics: timer mode determines the baseline, and chained relative waits on the same baseline are cumulative.
3. Claim grounding: trusted support entails the complete business claim.

The production prompts contain no chatbot, merchant, fee, language, or incident-specific examples. Positive statements define the target result; negative wording remains limited to existing security and transport boundaries where omission would weaken the contract.

Implementation

Config parser

The parser prompt must state:

• instructions contains the retained SOP text.
• Sibling fields appear exactly once at the top level.
• The response ends after the single top-level JSON object.
• idle counts from the latest user activity and resets on new activity.
• fixed counts from initial session timer registration and retains its schedule across user activity.
• A chained “another N” delay is cumulative only when actions share the same mode and baseline.

Skills are omitted from parser input and response schema. Runtime assembly keeps the raw skill array without LLM rewriting or parser post-processing.

Main-agent grounding

Before emitting a business fact, the model must check that a trusted source entails the complete claim: subject, relationship, condition, scope, and, when present, cause. User text and the evidence enum do not add authority.

When final JSON parsing requires a retry, the retry instruction requires the model to retain already-supported meaning and fix the envelope without adding facts.

Cache invalidation

CONFIG_CACHE_VERSION = "config_enhance_v2" invalidates incompatible L1 and L2 results through the existing version-miss path. Re-enhancement remains lazy and coalesced per (tenant_id, chatbot_id, config_hash) inside each process.

Redundant config removal

original_config.instructions duplicated basic_settings.instruction, while original_config.skills duplicated top-level runtime skills after skill enhancement was removed. Agent storage and config/observability APIs therefore omit original_config. The performance API derives originalAgentConfig from those existing fields so evaluation context remains complete without duplicate Mongo storage.

Existing Mongo documents remain readable because cache deserialization selects declared fields. Agent database initialization automatically applies a batched, restartable migration that removes the legacy field with $unset; it does not write empty or null values. This storage-only cleanup does not bump the cache version or trigger re-enhancement.

Database migrations use one document in the versions collection: {"version": N}. A missing document means version 0. Code runs ordered, idempotent migrations above the stored version and advances the value only after each migration succeeds. MongoDB owns the document _id; migration code does not assign or depend on it. Every Mongo-backed Agent instance checks the version before service initialization. Migration failure is logged without advancing the version or blocking service startup; the next restart retries from the last completed version.

Acceptance Criteria

• Config parser prompt assigns every output to its declared field.
• Config parser prompt defines both timer modes and same-baseline cumulative delays.
• Skills are absent from parser input/output and remain identical to raw config.
• Prompt grounding states that evidence is a reporting label, not permission.
• Final-envelope retry cannot add new business claims.
• Runtime config and observability contain no persisted original_config; performance derives its source-config view from existing fields.
• Agent startup automatically runs pending migrations in bounded batches.
• A failed migration does not advance the version or block startup, and the next restart retries it.
• A current database version does not scan the business collection.
• Old cache versions miss and successful enhancement persists config_enhance_v2.
• Focused prompt, config service, and workflow-agent tests pass.

Implemented Changes

File	Change
services/agent/api/services/config/enhancer.py	Added instructions field boundaries, mode-aware cumulative timer semantics, and raw skill passthrough
services/agent/bu_agent_sdk/prompts/templates.py	Added claim-entailment grounding, made evidence non-authorizing, and prohibited new facts during final-envelope retry
services/agent/api/services/config/service.py	Bumped cache version to config_enhance_v2
Agent config model, repository, and query services	Removed redundant original-config persistence and response fields
Agent/omni-base performance contract	Derives source instruction and skills without persisted duplication
Agent database infrastructure	Added startup-driven, ordered, versioned, restartable batched Mongo migrations
docs/agent/configs/config-enhance.md	Updated the authoritative config-enhance contract
Focused test files	Added prompt, schema, passthrough, grounding, retry, and cache-version regression coverage

Verification

Executed:

cd services/agent
PYTHONPATH=. uv run pytest \
  tests/prompts/test_config_enhancer.py \
  tests/prompts/test_prompt_builder.py \
  tests/runtime/test_workflow_agent.py \
  tests/config/test_config_service.py \
  tests/api/internal/test_observability_api.py \
  tests/api/internal/test_performance_api.py \
  tests/api/v1/test_session_api.py \
  tests/session/test_session_management.py \
  tests/services/database/test_migrations.py \
  tests/test_container_database_startup.py -q

Result: 398 passed in 3.91s.

omni-base performance contract:

cd services/omni-base
uv run pytest \
  tests/unit/test_agent_runtime_performance_client.py \
  tests/unit/test_ai_agent_performance_signals.py -q

Result: 51 passed in 0.45s.

Repository quick verification also passed with the project Python:

PATH="$PWD/.venv/bin:$PATH" bash scripts/verify.sh

The unqualified system python3 is 3.9.6 and cannot compile the project's Python 3.11+ match syntax.

Local startup verification:

• First startup applied migration v1, removed five local legacy fields, wrote database version 1, and returned HTTP 200 from /api/v1/health.
• Second startup read database version 1 without applying the migration and returned HTTP 200 from /api/v1/health.
• Both locally started Agent processes shut down cleanly after verification.

Real-provider config enhancement and production deployment were not performed as part of this local bugfix.

Evidence

• Session log: services/agent/automation_test/case/online/0723/session-84664120166982656.log
• Sampled production logs showed the embedded sibling JSON on every observed prompt for this config hash and not on the other sampled config hashes.
• Runtime timer registration starts all configured timers independently; reset_idle() restarts only idle timers and leaves fixed timers on their original plan in services/agent/api/services/timer/manager.py.