How it Works
Plain-language overview first — expand Technical reference for APIs, agents, and LangSmith details.
How this system works
Think of it as an experiment analyst assistant: it reads your test data, checks quality, estimates what worked, and suggests what to run next. Dell analysts stay in control — nothing launches without your approval.
Load your experiment data
The system pulls past test results, variants (arms), and metrics for the experiment you selected — like opening a folder of A/B test history.
Quality check
An automatic checklist asks: Is the data complete? Do arms match traffic? Are sample sizes big enough? You get GO, CAUTION, or STOP.
What worked?
A causal step estimates lift and uncertainty — which variant seemed to help, and how confident we are (mostly math, not guesswork).
What to try next
Candidates are scored and ranked. The top row is the suggested next test. You still approve before anything would launch in production.
Quick glossary
- Experiment ID
- Name of one test in the registry (e.g. exp_sanity_001_calibrated).
- Validation
- Safety check on data quality before trusting numbers.
- Caution
- Proceed with care — warnings exist but analysis still ran.
- Recommendation
- Ranked suggestion for the next variant to try.
- Approve / Proceed anyway
- Your sign-off in the app; stored in this browser for the demo.
Simple picture of the flow
HTTP Surface
VITE_API_URL · no standalone /recommend or /analyze
POST /orchestrate Pipeline
AdaptiveExperimentationOrchestrator.run · schema v1.0
Ranking inputs: one candidate per metrics row from retrieval before optional generation merge. Validation stop → HTTP 500, pipeline halted.
ValidationAgent Graph
LangSmith span names
First step of /orchestrate
ValidationAgent graph
Programmatic or ReAct loop
Stub or LLM when enabled
RecommendationSkill rank step (name is legacy)
Only when using CoordinatorAgent / CLI — not FastAPI by default
Smoke path only (skips causal)
Agents
Canonical v1 owner (POST /orchestrate): retrieval → validation → causal → optional generation → recommendation. Halts on validation stop.
LangGraph validation graph (Skill 2). Benchmark + world_spec checks; optional Azure capstone-mini diagnostics.
Programmatic causal by default; LangGraph create_react_agent when ENABLE_CAUSAL_AGENT_LOOP=true.
Structured experiment proposals when ENABLE_GENERATION_AGENT=true (Azure capstone-standard).
LangSmith umbrella (coordinator_run). Delegates to orchestrator; minimal_demo skips causal for smokes only.
Skills Catalog
Loads benchmark parquets (BENCHMARK_DATA_DIR) via load_benchmark_context; falls back to a minimal stub if the bundle is missing for an experiment id.
LangGraph gate: structural → metrics → benchmark → world_spec → decide → llm_diagnostics. go / caution / stop is code-only; optional Azure narrative when ENABLE_VALIDATION_LLM.
Default: programmatic causal estimates + schema validation. Optional LangGraph ReAct (ENABLE_CAUSAL_AGENT_LOOP) with Azure stat deployment (capstone-code).
When ENABLE_GENERATION_AGENT=true: LLM proposals (Azure generation deployment). Otherwise stub candidates; orchestrator also builds ranking inputs from metrics rows.
RecommendationSkill: deterministic score + rank (heuristic). LangSmith span name is legacy; not a separate LangGraph agent in the API repo.
Environment (Lovable + Railway)
Railway FastAPI base URL (no trailing slash)
Parquet bundle on API host
SQLite path for /orchestrate demo history
Foundry / Azure OpenAI resource
Chat completions key
Azure narrative in llm_diagnostics
LangGraph ReAct causal path
GenerationAgent vs stub candidates
Deployment names (e.g. capstone-mini)
DATA: benchmark parquets under BENCHMARK_DATA_DIR · RUN_HISTORY_DB SQLite is ephemeral on Railway without a persistent volume.
LLM: capstone-mini (validation) · capstone-standard (generation) · capstone-code (causal ReAct) — deployment names in Azure Foundry.
Causal Agent Tools
Always-call deterministic baseline for the causal agent. Required first step of the ReAct loop.
Optional progressive-disclosure panels surfacing relevant analysis references.
Optional sandbox for analysis code (when CAUSAL_REQUIRE_SANDBOX or agent loop).
Required final tool. Emits JSON validated against the EvaluationArtifact schema.
loop order: run_programmatic_baseline → optional disclosure/sandbox → submit_evaluation