Challenge 04 β Semantic Control & Business Rules
π Enterprise Scenarioβ
Company: Apex Quantitative Research β a quant fund running systematic strategies
Situation: Your AI research assistant is used by portfolio managers to analyze sector performance and factor exposures. Three incidents occurred in the same week:
- A PM asked for "Magnificent Seven" returns. The agent included Tesla, which was removed from the original cohort. The strategy was backtested with wrong constituents.
- A PM asked for "Q1 returns." The agent calculated from open of Jan 1st β your firm's standard is close of Dec 31st of the prior year to close of Mar 31st. The figures were off by one day.
- A PM asked about performance "last week." The agent used its training-data date (September 2023) as "now" β it refused to query for 2026 data, claiming it was in the future.
All three failures share a root cause: business semantics delegated to LLM training data instead of owned by your system.
The Core Problem: Semantic Delegationβ
The "Simple Agentic" pattern treats the LLM as an authority on business concepts. This creates a hidden dependency on the model's static, probabilistic training data for decisions that should be owned by your code.
| Concept | LLM's "knowledge" | Your business reality |
|---|---|---|
| "Magnificent Seven" | Tesla (pre-2024 training) | AAPL, MSFT, NVDA, AMZN, GOOGL, META, TSLA (varies by source/date) |
| "Q1 start" | Probably Jan 1 open | Your firm: Dec 31 prior year close |
| "Last week" | Based on training cutoff | Needs date.today() injection |
| "FANG stocks" | Meta = Facebook? | META ticker since Nov 2021 |
| "Technology sector" | Model's classification | GICS sector mapping (updates quarterly) |
The fix is Semantic Control: externalize all business concept definitions into code/config, inject temporal grounding on every request, and use PostToolUse hooks to enforce scope boundaries.
ποΈ Architecture decision table β who owns your business semantics
| Approach | Accuracy | Updateable | Auditability |
|---|---|---|---|
| β LLM resolves all concepts | Probabilistic (training data) | No β requires model retrain | None |
| β οΈ Prompt injection ("Magnificent 7 is: AAPL, MSFT...") | Better | Requires prompt updates | Partial |
| β Externalized concept registry (config/DB) | Deterministic | Yes β update config, not code | Full β version history |
| β Temporal grounding injection (always inject current date) | Deterministic | Automatic | Full |
| β Scope enforcement hooks (block out-of-scope tool calls) | Deterministic | Yes | Full |
π§° Before You Start β Environment Setupβ
This challenge is about owning your business semantics instead of borrowing them from the model's training data. Your setup needs a place to store concept definitions (a registry) and a way to inject "now" on every request.
Prerequisitesβ
| Requirement | Why you need it | How to check |
|---|---|---|
| Python 3.10+ | Registry, parser, and enforcement hooks | python --version |
| Azure OpenAI via Azure AI Foundry | The intent parser β it resolves phrasing, never definitions | Deploy gpt-4o in Foundry |
| A concept registry store β Azure SQL / Dataverse / Cosmos DB / App Configuration (prod); local JSON here | The authoritative, version-controlled source of what "Magnificent Seven" means today | Azure portal / mkdir .registry |
| Azure AI Foundry β Evaluations | Grade concept-resolution accuracy against your registry | Docs |
Step 0 β Create an isolated workspace (5 min)β
Where you run this: Step 0 runs locally on your own machine β open a terminal (VS Code's integrated terminal, PowerShell, or bash). You don't touch Azure until Step 1.
mkdir semantic-control && cd semantic-control
python -m venv .venv
# Windows (PowerShell): .venv\Scripts\Activate.ps1 | macOS/Linux: source .venv/bin/activate
pip install azure-ai-projects azure-identity openai pydantic python-dotenv
mkdir .registry # local stand-in for Azure SQL / Dataverse / App Configuration
β
Done when your prompt shows (.venv) and pip list includes azure-ai-projects.
Step 1 β Provision your model & sign in (10 min)β
The intent parser is a real gpt-4o deployment. If you have not deployed one, do Steps 1β2 of Challenge 01 β The Hallucination Audit for the exact portal walkthrough and the two values below, then create a .env:
# .env β from Azure AI Foundry (never commit this file)
# PROJECT_ENDPOINT=https://<your-project>.services.ai.azure.com/api/projects/<name>
# MODEL_DEPLOYMENT_NAME=gpt-4o
az login # keyless auth via DefaultAzureCredential
Smoke-test before Task 1 β if it prints setup works, any later failure is your logic, not setup:
# smoke_test.py
import os
from dotenv import load_dotenv
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential
load_dotenv()
project = AIProjectClient(endpoint=os.environ["PROJECT_ENDPOINT"], credential=DefaultAzureCredential())
client = project.inference.get_azure_openai_client(api_version="2024-10-21")
print(client.chat.completions.create(model=os.environ["MODEL_DEPLOYMENT_NAME"],
messages=[{"role":"user","content":"Reply with exactly: setup works"}]).choices[0].message.content)
Common fixes:
DefaultAzureCredential failedβaz loginagain.DeploymentNotFoundβ deployment name mismatch.401β add the Azure AI User role on the project.
Step 2 β Seed a versioned concept registry (10 min)β
Store each business concept with an effective date so historical queries resolve correctly. These are sample definitions β replace with your firm's real ones.
# registry.json β the authoritative definitions your code owns
{
"magnificent_seven": [
{"effective": "2023-01-01", "members": ["AAPL","MSFT","NVDA","AMZN","GOOGL","META","TSLA"]},
{"effective": "2025-01-01", "members": ["AAPL","MSFT","NVDA","AMZN","GOOGL","META","AVGO"]}
],
"fiscal_q1": {"start_rule": "prior_year_dec_31_close", "end_rule": "mar_31_close"}
}
π¦ Microsoft-first note: the
.registryfolder / JSON is a local stand-in. In production the registry belongs in Azure SQL (temporal tables give you free version history), Microsoft Dataverse (business-user editable), Azure Cosmos DB, or Azure App Configuration for feature-flag-style rollout. The temporal grounding (date.today()injected per-request) and scope-enforcement hooks stay in deterministic Python either way.
The path through this challengeβ
- Task 1 β build the externalized concept registry.
- Task 2 β inject temporal grounding on every request.
- Task 3 β build a concept-aware intent parser.
- Task 4 β add a scope-enforcement hook (pre-LLM).
- Success Criteria β definitions come from your registry, not training data.
- Adapt to Your Business β externalize your concepts.
β±οΈ Time budget: ~3 hours. Task 1 (the registry) is the linchpin β every other task depends on it.
Tasksβ
Task 1 β Build the Concept Registryβ
Externalize all business-defined groupings into a versioned registry. The LLM never "knows" these β it always receives them as injected context.
# concept_registry.py
import json
from datetime import date
from typing import Optional
from pathlib import Path
class ConceptRegistry:
"""
Versioned registry of business-defined groupings and rules.
All definitions are date-ranged β correct answer depends on when the query runs.
This replaces LLM training data as the authority on business concepts.
"""
def __init__(self, registry_path: str = "config/concepts.json"):
with open(registry_path) as f:
self._registry = json.load(f)
def resolve(self, concept: str, as_of_date: Optional[str] = None) -> dict:
"""
Resolve a concept name to its current definition.
Respects effective_date ranges β correct even for historical queries.
"""
as_of = date.fromisoformat(as_of_date) if as_of_date else date.today()
concept_key = concept.lower().replace(" ", "_").replace("-", "_")
entries = self._registry.get("concepts", {}).get(concept_key, [])
# Find the entry effective on the query date
active = [
e for e in entries
if date.fromisoformat(e["effective_from"]) <= as_of
and (e.get("effective_to") is None or date.fromisoformat(e["effective_to"]) >= as_of)
]
if not active:
return {"found": False, "concept": concept, "as_of": str(as_of)}
return {
"found": True,
"concept": concept,
"definition": active[-1], # most recent effective entry
"as_of": str(as_of),
"source": "concept_registry" # not LLM training data
}
def list_concepts(self) -> list:
return list(self._registry.get("concepts", {}).keys())
// config/concepts.json β the source of truth your code owns
{
"concepts": {
"magnificent_seven": [
{
"effective_from": "2023-01-01",
"effective_to": "2024-12-31",
"tickers": ["AAPL", "MSFT", "GOOGL", "AMZN", "NVDA", "TSLA", "META"],
"description": "Magnificent Seven (2023 composition)",
"source": "internal-research-team"
},
{
"effective_from": "2025-01-01",
"effective_to": null,
"tickers": ["AAPL", "MSFT", "GOOGL", "AMZN", "NVDA", "META", "AVGO"],
"description": "Magnificent Seven (2025 composition β Broadcom replaces Tesla)",
"source": "internal-research-team"
}
],
"fang": [
{
"effective_from": "2021-11-01",
"effective_to": null,
"tickers": ["META", "AMZN", "NFLX", "GOOGL"],
"description": "FANG stocks (post FBβMETA rename)",
"note": "Facebook became META on 2021-10-28"
}
],
"fiscal_quarter_q1": [
{
"effective_from": "2000-01-01",
"effective_to": null,
"start_convention": "prior_year_dec31_close",
"end_convention": "mar31_close",
"description": "Apex QR Q1 definition: Dec 31 close β Mar 31 close"
}
]
}
}
Task 2 β Inject Temporal Groundingβ
Every request must include an explicit current_date. No LLM should ever infer "today" from training weights.
# temporal_grounding.py
from datetime import date, timedelta
import re
class TemporalGrounding:
"""
Resolves relative time expressions to absolute dates
using the actual current date β never LLM training data.
"""
def __init__(self, current_date: Optional[date] = None):
self.today = current_date or date.today()
def resolve_time_expression(self, expression: str) -> dict:
"""
Convert relative time expressions to concrete date ranges.
Returns absolute dates for deterministic downstream use.
"""
expr = expression.lower().strip()
if "last week" in expr:
# Previous Monday through Sunday
days_since_monday = self.today.weekday()
last_monday = self.today - timedelta(days=days_since_monday + 7)
last_sunday = last_monday + timedelta(days=6)
return {
"start_date": str(last_monday),
"end_date": str(last_sunday),
"resolved_from": expression,
"resolution_date": str(self.today)
}
elif "ytd" in expr or "year to date" in expr:
return {
"start_date": str(date(self.today.year, 1, 1)),
"end_date": str(self.today),
"resolved_from": expression,
"resolution_date": str(self.today)
}
elif re.match(r"q[1-4]\s+\d{4}", expr):
# e.g., "Q1 2025"
quarter = int(expr[1])
year = int(re.search(r'\d{4}', expr).group())
return self._resolve_fiscal_quarter(quarter, year)
# Default: pass through if already absolute dates
return {"start_date": None, "end_date": None, "resolved_from": expression}
def _resolve_fiscal_quarter(self, quarter: int, year: int) -> dict:
"""
Resolve fiscal quarters using YOUR firm's definition.
Not the LLM's interpretation.
"""
registry = ConceptRegistry()
q_def = registry.resolve(f"fiscal_quarter_q{quarter}")
if not q_def["found"]:
raise ValueError(f"Q{quarter} definition not found in concept registry")
convention = q_def["definition"]
if convention["start_convention"] == "prior_year_dec31_close":
start = date(year - 1, 12, 31)
else:
# default: first trading day of quarter
quarter_starts = {1: (1,1), 2: (4,1), 3: (7,1), 4: (10,1)}
m, d = quarter_starts[quarter]
start = date(year, m, d)
quarter_ends = {1: (3,31), 2: (6,30), 3: (9,30), 4: (12,31)}
m, d = quarter_ends[quarter]
end = date(year, m, d)
return {
"start_date": str(start),
"end_date": str(end),
"resolved_from": f"Q{quarter} {year}",
"convention_used": convention["description"],
"resolution_date": str(self.today)
}
def build_grounded_system_prompt(self, base_prompt: str) -> str:
"""
Prepend authoritative temporal context to any system prompt.
LLM always knows today's date from code β never from training data.
"""
return f"""
TEMPORAL CONTEXT (authoritative β use these values, not your training data):
- Today: {self.today.isoformat()}
- Current Year: {self.today.year}
- Current Quarter: Q{(self.today.month - 1) // 3 + 1} {self.today.year}
{base_prompt}
"""
Task 3 β Concept-Aware Intent Parserβ
Intercept concept references before they reach the LLM. Replace them with concrete definitions.
# concept_aware_parser.py
class ConceptAwareIntentParser:
"""
Pre-processes user queries to replace concept references with
concrete, registry-sourced definitions before the LLM sees them.
LLM receives: "Compare AAPL, MSFT, GOOGL, AMZN, NVDA, META, AVGO returns"
Not: "Compare Magnificent Seven returns"
"""
KNOWN_CONCEPTS = [
"magnificent seven", "magnificent 7", "mag 7", "mag seven",
"fang", "fang stocks", "faang",
"dow 30", "dow jones", "s&p 500",
"q1", "q2", "q3", "q4",
"ytd", "year to date", "last week", "last month", "last year"
]
def __init__(self):
self.registry = ConceptRegistry()
self.temporal = TemporalGrounding()
def expand(self, user_query: str) -> tuple[str, dict]:
"""
Expand concepts in user query to concrete definitions.
Returns (expanded_query, expansion_log).
Expansion log is part of the audit trail.
"""
expanded = user_query
expansion_log = {}
# Resolve stock groupings
for concept in self.KNOWN_CONCEPTS:
if concept.lower() in user_query.lower():
resolution = self.registry.resolve(concept)
if resolution["found"]:
definition = resolution["definition"]
if "tickers" in definition:
tickers_str = ", ".join(definition["tickers"])
expanded = expanded.replace(concept, tickers_str)
expanded = expanded.replace(concept.title(), tickers_str)
expansion_log[concept] = {
"replaced_with": tickers_str,
"source": "concept_registry",
"definition_version": definition.get("effective_from")
}
# Resolve temporal expressions
time_expressions = self.temporal.find_relative_expressions(user_query)
for expr in time_expressions:
resolved = self.temporal.resolve_time_expression(expr)
if resolved["start_date"]:
expanded = expanded.replace(
expr,
f"from {resolved['start_date']} to {resolved['end_date']}"
)
expansion_log[expr] = resolved
return expanded, expansion_log
def parse(self, user_query: str) -> tuple[FinancialQuerySpec, dict]:
"""Full pipeline: expand concepts β parse intent β return spec + audit."""
expanded_query, expansion_log = self.expand(user_query)
# LLM now receives concrete, unambiguous query
spec = parse_intent(
expanded_query,
current_date=str(self.temporal.today)
)
# Attach expansion log for audit trail
audit = {
"original_query": user_query,
"expanded_query": expanded_query,
"concept_expansions": expansion_log
}
return spec, audit
Task 4 β Scope Enforcement Hookβ
Use a PostToolUse-style hook to block out-of-scope tool calls at the code level.
# scope_enforcement.py
ALLOWED_SCOPE = {
"tools": ["query_financial_data", "calculate_returns"],
"data_sources": ["stock_prices"],
"query_types": ["historical_price", "return_calculation", "volume"],
"forbidden": ["strategic_analysis", "earnings_forecast", "recommendation"]
}
def pre_llm_scope_check(user_query: str) -> dict:
"""
Classify query scope BEFORE sending to LLM.
Hard-block out-of-scope requests at the gateway.
"""
# Fast, cheap classification call
scope_classification = fast_classify(user_query, categories=[
"historical_price_query", # in scope
"return_calculation", # in scope
"strategic_analysis", # OUT OF SCOPE
"earnings_forecast", # OUT OF SCOPE
"general_recommendation" # OUT OF SCOPE
])
if scope_classification in ALLOWED_SCOPE["forbidden"]:
return {
"allowed": False,
"reason": f"Query classified as '{scope_classification}' β outside system scope",
"message": (
"This system is designed for historical price and return queries only. "
"For strategic analysis or forecasts, please use [link to appropriate tool]."
)
}
return {"allowed": True, "classification": scope_classification}
Success Criteriaβ
- "Magnificent Seven" query uses registry definition for correct date β not LLM training data
- Historical query for "Magnificent Seven" in 2023 uses 2023 composition (Tesla included), 2025 uses 2025 composition (Broadcom, not Tesla)
- "Q1 returns" uses firm's fiscal calendar convention β not LLM's interpretation of Q1
- "Last week" resolves to correct absolute dates based on
date.today()β never training data - LLM correctly refuses to "know" the current date from training β always uses injected value
- Out-of-scope queries blocked at gateway β not by prompt instruction
- All concept expansions logged in audit trail with
source: "concept_registry"
π Adapt This to Your Own Businessβ
The scenario is a quant fund, but every company has business terms whose meaning the model gets subtly β and confidently β wrong. The fix is always the same: your system owns the definition, not the LLM.
Step 1 β Find your "the model thinks it knows, but it's wrong" termsβ
| Industry | Ambiguous term the LLM guesses | What it should mean (owned by you) |
|---|---|---|
| Retail | "Top sellers" | Your current merchandising list, not last year's |
| Healthcare | "High-risk patient" | Your clinical protocol's exact criteria |
| Insurance | "Preferred customer" | Your current tier rules, effective this quarter |
| SaaS / RevOps | "Enterprise account" | Your segmentation thresholds, not a generic guess |
| Manufacturing | "Critical part" | Your current BOM criticality flags |
| Public sector | "Current fiscal year" | Your jurisdiction's calendar, not JanβDec |
Step 2 β Map the building blocks to your stack (Microsoft-first)β
| In this challenge | In your project β replace with |
|---|---|
registry.json | Azure SQL (temporal tables), Dataverse, Cosmos DB, or App Configuration |
| Effective-dated concept versions | SQL temporal tables / Dataverse audit history |
| Temporal grounding injection | Middleware that injects date.today() per request |
| Concept-aware intent parser | Azure OpenAI for phrasing only β resolution stays in code |
| Scope-enforcement hook | Deterministic pre-LLM gateway (Azure Functions / API Management) |
| Resolution accuracy grading | Azure AI Foundry Evaluations custom evaluator |
Step 3 β The 5-question implementation checklistβ
- Which business terms would embarrass you if the model defined them? Those go in the registry first.
- Do any of your definitions change over time? If yes β store them effective-dated, and resolve by query date.
- Does your agent ever assume today's date? If yes β inject
date.today()per request, never at startup. - Can a clever rephrasing bypass your "only answer about X" rule? If yes β move scope enforcement into pre-LLM code.
- Can you prove where a definition came from? If not β log
source: "concept_registry"on every expansion.
Step 4 β A 1-week rollout planβ
| Day | Action | Owner |
|---|---|---|
| Day 1 | Inventory the top 10 ambiguous business terms your agent handles | Domain expert + eng |
| Day 2 | Build the effective-dated registry in Azure SQL / Dataverse | Data eng |
| Day 3 | Add per-request temporal grounding middleware | Backend dev |
| Day 4 | Move scope enforcement to a pre-LLM gateway | Backend dev |
| Day 5 | Add a Foundry Evaluations grader for concept accuracy | ML eng |
Step 5 β Prove the ROIβ
- Definition accuracy β % of concept resolutions matching the registry (target: 100%).
- Temporal correctness β % of "now/last week/this quarter" queries resolved to correct absolute dates (target: 100%).
- Scope-bypass rate β % of adversarial rephrasings that slip past enforcement (target: 0%).
π‘ Rule of thumb: if the answer to "what does this term mean?" lives only in the model's head, it will drift when the model updates. Put the definition in your system, date-stamp it, and log where it came from.
Doing this solo (no team, portfolio-first)β
No team, no budget? A "the system owns the definition, not the LLM" demo shows senior judgment few juniors have. Run the week solo:
- MonβTue β inventory your 10 riskiest business terms; build an effective-dated concept registry (local JSON/SQLite now, Azure SQL/Dataverse later).
- WedβThu β add per-request temporal grounding (
date.today()) + a pre-LLM scope gate. - Fri β run a small eval: definition accuracy, temporal correctness, and adversarial scope-bypass attempts.
π¦ Ship this artifact: the registry + an eval report showing 100% definition/temporal accuracy and 0% scope bypass. Resume bullet: "Made business-term resolution deterministic β 100% definition and date accuracy, 0% scope-bypass on adversarial rephrasings."
π Free-tier path: a local registry file + the consumption-tier model β no infrastructure required to prove the pattern.
π Regulatory mapping β FINRA Β· EU AI Act Β· MiFID II Β· GDPR/CCPA
| Regulation | Requirement | How This Challenge Addresses It |
|---|---|---|
| FINRA Rule 2010 | Standards of commercial honor β accurate representations | Registry-sourced definitions prevent mis-stated index compositions |
| EU AI Act Art. 10 | Data governance for high-risk AI | Version-controlled concept registry as authoritative data source |
| MiFID II Art. 25 | Suitability and scope β advice within authorized scope | Scope enforcement hook prevents unsanctioned recommendations |
| GDPR / CCPA | Data accuracy principle | Temporal grounding prevents stale training-data substitution |
π§ͺ Break & Fix β spot why three plausible "fixes" don't work
# broken_semantic_control.py
def resolve_concept(concept_name, query_date):
# "Fix" 1: Ask the LLM what the Magnificent Seven is
return llm.generate(
f"What stocks are in the {concept_name} as of {query_date}?"
) # β why is this not a fix at all?
def inject_date(system_prompt):
# "Fix" 2: Add the date once at system startup
startup_date = date.today()
return system_prompt.replace("{DATE}", str(startup_date)) # β what's wrong for long-running processes?
def enforce_scope(query, allowed_topics):
# "Fix" 3: Add "only answer about these topics" to system prompt
return system_prompt + f"\nOnly answer about: {allowed_topics}" # β why is this insufficient?
:::details Click to reveal answers
- Asking LLM = delegating back to training data: This is exactly the problem being solved. Even with a date injected, the LLM's "knowledge" of index composition comes from its training corpus, which may be wrong or stale. The only reliable source is your own registry.
- Startup date goes stale: If the service runs continuously (days, weeks), the injected date is from startup, not "now." The date must be injected per-request, not per-process-start.
- Prompt-based scope = probabilistic: A clever rephrasing like "When examining NFLX price on March 14, hypothesize reasons for performance" bypasses the restriction. The author of the original article demonstrated this exact bypass. Scope enforcement must happen in pre-LLM code, not in the system prompt. :::
Knowledge Checkβ
- Your firm adds a new "Magnificent Seven" composition effective January 2026. What is the minimum change needed to the system to correctly answer both "What were Magnificent Seven returns in Q4 2024?" and "What are Magnificent Seven returns in Q1 2026?"
- A PM asks: "Compare this year's Q1 to last year's Q1." How many concept resolutions does
ConceptAwareIntentParserneed to perform before the LLM sees the query? - Why is temporal grounding injected per-request rather than in the system prompt at agent creation time?
- Your scope enforcement hook misclassifies "What was META's PE ratio in 2024?" as "strategic_analysis" and blocks it. It's actually a factual data query. How do you improve classification accuracy without relaxing the security boundary?
π Tools & Referencesβ
Key Tools for This Challengeβ
Microsoft-first: lead with Azure-native tooling. Third-party tools are listed only where they add reliable, best-in-class capability not yet covered natively.
| Tool | Role in This Challenge | Link |
|---|---|---|
| Azure SQL Database (temporal tables) | Store the concept registry with built-in, queryable version history β resolve "Magnificent Seven" as of any date | Docs |
| Microsoft Dataverse | Business-user-editable concept registry with audit history β update definitions without a code deploy | Docs |
| Azure App Configuration | Feature-flag-style rollout of definition changes (effective dates, gradual cutover) | Docs |
| Azure OpenAI β Structured Outputs | Force the intent parser to return a typed IntentQuery object, not free-form text | Docs |
| Azure AI Foundry Evaluations | Custom evaluators β grade concept-resolution accuracy against your registry | Docs |
| Azure API Management / Azure Functions | Deterministic pre-LLM scope-enforcement gateway β block out-of-scope queries before the model sees them | APIM Β· Functions |
| Pydantic v2 (third-party) | Local schema for registry models when not using Structured Outputs | docs.pydantic.dev |
| DeepEval / RAGAS (third-party) | Semantic-similarity metrics for concept-resolution regression tests | DeepEval Β· RAGAS |
Required Readingβ
| Resource | Why It Matters |
|---|---|
| The LLM-as-Analyst Trap, Part 1 β Semantic Drift section | The original failure mode this challenge addresses β LLMs use training-data definitions, not current enterprise ones |
| Azure OpenAI Structured Outputs | How to force the LLM to return typed objects β the foundation of deterministic intent parsing |
| Azure SQL temporal tables | How to store effective-dated concept definitions with automatic version history |
| Data governance for high-risk AI (Azure) | Microsoft guidance on authoritative data sources for agent decisions |