LPL Financial — AI Agent Implementation Blueprint

1) LPL Technology Landscape

LPL Financial operates one of the largest independent broker-dealer platforms in the US, supporting 32,000+ advisors across multiple affiliation models. Understanding the existing technology stack is critical for agent integration.

Key Platform Facts

2) Agentic AI Strategy for LPL

Moving from isolated AI tools (Claude chat, Jump notetaker, Ester document reader) to an interconnected network of specialized agents that share context, coordinate workflows, and operate under unified governance.

Strategic Principles

• "Propose, never impose": All agents generate recommendations; advisors make final decisions.
• Anthropic-native: Claude function-calling as the backbone LLM with LPL-specific plugins.
• Compliance-first: FINRA 2026 AI agent supervisory requirements built into every agent from day one.
• Advisor-centric: Agents measured on advisor time saved, not just model accuracy.
• Incremental deployment: Shadow mode → limited release → general availability with metrics gates.

Proposed Agent Catalog

3) Agent Orchestration Platform

Central control plane that routes requests, enforces policies, manages agent memory, and provides a unified audit trail across all 12 agents.

Orchestration Engine Technical Design

// Agent Orchestration — Rust + Python hybrid
// Rust: Gateway, routing, policy enforcement, audit writes
// Python: LLM integration, tool execution, memory management

struct AgentRequest {
    request_id: Uuid,
    trace_id: Uuid,
    advisor_id: String,       // e.g., "ADV-1207"
    session_id: Option<Uuid>, // multi-turn context
    intent: String,           // classified intent
    agent_type: AgentType,    // resolved target agent
    payload: serde_json::Value,
    entitlements: Vec<String>, // from IAM
    timestamp: DateTime<Utc>,
}

enum AgentType {
    ClientWorksCopilot,
    TradeExecution,
    PortfolioIntelligence,
    ComplianceSurveillance,
    MeetingCRM,
    EstatePlanning,
    ClientOnboarding,
    MarketingAutomation,
    AMLFraud,
    DataQuality,
    PlatformOps,
    RegulatoryReporting,
}

enum ApprovalTier {
    Tier1AutoApprove,      // Read-only, informational
    Tier2AdvisorApprove,   // Trade proposals, comms drafts
    Tier3ComplianceGate,   // SAR filing, regulatory submissions
}

Memory Architecture

4) ClientWorks Copilot Agent

The primary entry point for advisor-agent interaction. Embedded directly in the ClientWorks UI, this agent understands advisor intent and routes to specialized agents or handles requests directly.

Tool Access

// Claude function-calling tools available to ClientWorks Copilot
tools = [
    // Portfolio tools
    "get_client_holdings",        // Read positions from portfolio service
    "get_performance_summary",    // YTD, 1yr, 3yr, inception returns
    "get_risk_metrics",           // VaR, beta, concentration, drift score
    "get_account_details",        // Account type, registration, beneficiaries

    // Research tools
    "search_research_corpus",     // Vector search over approved research
    "get_market_data",            // Real-time quotes, fundamentals via FactSet
    "get_fund_analysis",          // Morningstar, AdvisoryWorld data

    // Action tools (require approval)
    "draft_trade_proposal",       // → routes to Trade Execution Agent
    "draft_client_email",         // → compliance review before send
    "schedule_meeting",           // → CRM integration via Wealthbox
    "create_service_request",     // → ClientWorks service workflow

    // Context tools
    "get_recent_interactions",    // Last N meetings, emails, notes
    "get_compliance_alerts",      // Active alerts for this client/advisor
    "get_advisor_preferences",    // Communication style, favorite analyses
]

Example Interaction

// Advisor: "How is the Johnson family doing? Any concerns before our meeting Thursday?"

// Agent Plan:
// 1. get_account_details("Johnson Family") → 3 accounts found
// 2. get_client_holdings("ACC-001", "ACC-002", "ACC-003")
// 3. get_performance_summary("ACC-001", "ACC-002", "ACC-003")
// 4. get_risk_metrics("ACC-001", "ACC-002", "ACC-003")
// 5. get_recent_interactions("CLIENT-4521", limit=5)
// 6. search_research_corpus("sectors relevant to Johnson holdings")
// 7. Synthesize into meeting prep brief with citations

// → Output: Meeting prep document with:
//   - Portfolio snapshot (combined $2.4M, +8.2% YTD)
//   - Risk flag: Tech concentration at 34% (above 25% IPS limit)
//   - Recommendation: Discuss rebalance to reduce tech exposure
//   - Recent: Last meeting discussed college funding for daughter (2027)
//   - Market context: Recent semiconductor volatility relevant to holdings

Integration Points

5) Trade Execution Agent

Automates multi-step trade workflows. Integrated with LPL's AWS-based trading infrastructure processing 1B+ events.

// Trade Execution Agent — Tool definitions
tools = [
    // Analysis
    "calculate_model_drift",       // Compare current vs target allocation
    "run_tax_lot_analysis",        // Identify tax-loss harvesting opportunities
    "estimate_transaction_costs",  // Spread + commission + market impact
    "check_wash_sale_risk",        // 30-day lookback on related securities

    // Generation
    "generate_rebalance_trades",   // Produce trade list from drift analysis
    "generate_block_order",        // Aggregate client orders into block
    "calculate_fair_allocation",   // Pro-rata / rotational allocation

    // Validation (auto-executed)
    "validate_buying_power",       // Funds available check
    "validate_concentration",      // Position / sector / asset class limits
    "validate_restricted_list",    // Firm restricted securities
    "validate_product_eligibility", // Account type vs product suitability

    // Submission (REQUIRES advisor approval)
    "submit_order_to_oms",         // → LPL OMS via internal API
    "submit_block_order",          // → Block trading system
]

// Guardrails enforced by Policy Engine:
// - Max notional per single order: $500K (configurable per advisor tier)
// - Max daily aggregate: $5M per advisor
// - All trades require explicit advisor confirmation
// - Kill switch: feature flag "agent.trade.enabled" in LaunchDarkly
// - Circuit breaker: auto-disable if error rate > 2% in 5-minute window

AWS Integration Architecture

6) Portfolio Intelligence Agent

Continuous portfolio monitoring with proactive alerting. Integrates with AdvisoryWorld models and LPL's rebalancing engine.

Portfolio Monitoring Architecture

// Scheduled portfolio scan — runs nightly for all active accounts
// EventBridge: cron(0 2 * * ? *)  → triggers batch portfolio agent

interface PortfolioScanResult {
  account_id: string;
  advisor_id: string;
  alerts: Alert[];
  recommendations: Recommendation[];
  scan_timestamp: string;
  model_version: string;
}

interface Alert {
  type: "DRIFT" | "CONCENTRATION" | "TAX_LOSS" | "MATURITY" | "RISK_MISMATCH";
  severity: "INFO" | "WARNING" | "ACTION_REQUIRED";
  details: string;         // Human-readable explanation
  data: Record<string, any>;  // Supporting metrics
  recommended_action: string;
  expires_at: string;      // Alert relevance window
}

7) Compliance Surveillance Agent

Monitors advisor communications and trading activity per FINRA 3110/3120 requirements. Auto-triages alerts and prepares investigation packages for compliance analysts.

// Compliance Agent — FINRA 2026 aligned supervisory design
// Reference: FINRA 2026 Annual Regulatory Oversight Report, GenAI section

struct SurveillanceAlert {
    alert_id: Uuid,
    source: AlertSource,       // COMMS_SURVEILLANCE | TRADE_SURVEILLANCE
    detected_pattern: String,  // e.g., "potential_outside_business_activity"
    confidence_score: f64,     // 0.0 - 1.0
    severity: Severity,        // LOW | MEDIUM | HIGH | CRITICAL

    // AI-generated investigation package
    summary: String,           // Natural language alert summary
    timeline: Vec<TimelineEvent>, // Reconstructed event sequence
    related_alerts: Vec<Uuid>, // Clustered related signals
    evidence: Vec<EvidenceItem>, // Supporting documents/records

    // FINRA-required fields
    human_reviewer: Option<String>, // MUST be assigned
    model_version: String,     // For reproducibility
    disposition: Option<Disposition>, // ONLY set by human analyst
    audit_trail: Vec<AuditEntry>,
}

// CRITICAL: Agent can SCORE and PACKAGE but NEVER DISPOSE
// All dispositions require human compliance officer sign-off
// Per FINRA 2026: "human in the loop" agent oversight protocols

Surveillance System Architecture

8) Meeting & CRM Agent (Jump AI Enhanced)

Extends LPL's existing Jump AI integration into a full lifecycle meeting agent with deep ClientWorks and CRM connectivity.

// Meeting lifecycle — Jump AI + Agent integration via webhook
// Current savings: 72,000+ advisor hours/year → target: 150,000+ hours

// Pre-meeting trigger: EventBridge scheduled event 2 hours before meeting
{
  "agent": "meeting_crm",
  "action": "prepare_meeting_brief",
  "meeting_id": "MTG-2026-03-16-1400",
  "advisor_id": "ADV-1207",
  "client_id": "CLIENT-4521",
  "tools_used": [
    "get_recent_interactions",    // Last 3 meetings, emails, notes
    "get_client_holdings",        // Current portfolio snapshot
    "get_portfolio_alerts",       // Active drift/risk alerts
    "get_upcoming_events",        // Birthdays, anniversaries, milestones
    "search_research_corpus"      // News relevant to client holdings
  ],
  "output": "meeting_prep_brief" // → pushed to advisor mobile + ClientWorks
}

// Post-meeting trigger: Jump AI webhook on meeting end
{
  "agent": "meeting_crm",
  "action": "process_meeting",
  "jump_transcript_id": "JMP-TR-98234",
  "detected_items": {
    "action_items": ["Review 529 plan options", "Send tax-loss analysis"],
    "compliance_flags": ["Client mentioned outside investment"],
    "life_events": ["Daughter graduating college 2027"],
    "sentiment": "positive",
    "next_meeting": "2026-04-15"
  }
}

9) Estate Planning Agent (Wealth.com / Ester Enhanced)

Extends the Wealth.com Ester AI integration to provide end-to-end estate planning automation for LPL's advisor network.

• Ester AI reads: Wills, trusts, POAs, beneficiary designations, insurance policies.
• Time savings: 40+ hours per UHNW estate plan reduced to under 2 hours for initial analysis.
• Gap analysis: Missing beneficiaries, outdated documents, misaligned titling, tax exposure.
• Family Office Suite: Cross-generational planning with LPL Advanced Planning Team support.
• Vault integration: Permission-based document storage with advisor + client access.

10) Client Onboarding Agent

Streamlines LPL's digital account opening process with intelligent document processing and automated KYC/CDD workflows.

// Onboarding agent reduces median account open time from 5 days to same-day
// Handles both 1099 and W-2 advisor affiliation models

interface OnboardingWorkflow {
  workflow_id: string;
  advisor_id: string;
  client_data: ClientProfile;

  // Document processing results
  documents: ProcessedDocument[];  // OCR + extracted fields
  verification: {
    cip_status: "PASS" | "FAIL" | "MANUAL_REVIEW";
    sanctions_status: "CLEAR" | "POTENTIAL_MATCH" | "CONFIRMED_MATCH";
    adverse_media: "CLEAR" | "FLAGGED";
  };

  // Account configuration
  account_type: "INDIVIDUAL" | "JOINT" | "IRA" | "TRUST" | "ENTITY";
  advisory_vs_brokerage: "ADVISORY" | "BROKERAGE" | "HYBRID";
  model_assignment: string | null;  // Target allocation model

  // Always requires human approval
  approval_status: "PENDING_COMPLIANCE" | "APPROVED" | "REJECTED";
  compliance_reviewer: string;
}

11) Marketing Automation Agent

Extends LPL's digital marketing platform. Advisors using Marketing Solutions grew assets 39% faster than peers.

12) AML / Fraud Detection Agent

Real-time transaction monitoring with graph analytics and automated case narrative generation.

Graph Analytics & Entity Resolution

• Sanctions: Hard-block on OFAC/EU/UN matches (automated, no override possible).
• Scoring: Sub-second enrichment with velocity, geographic, device, and entity linkage features.
• Narratives: Auto-generated SAR case narratives with supporting evidence, requiring BSA officer sign-off.
• Feedback loop: Investigator dispositions feed back to improve model accuracy and reduce false positives.

13) Data Quality Agent

Monitors LPL's AWS data pipelines (Glue, Athena, S3) for anomalies, drift, and freshness issues.

14) Platform Ops / SRE Agent

Assists LPL's 24/7 SOC and SRE teams with incident response, capacity planning, and playbook execution on the EKS infrastructure.

15) Regulatory Reporting Agent

Automates assembly and validation of FINRA/SEC regulatory filings.

16) Agent Governance & FINRA/SEC Alignment

Designed to meet FINRA's 2026 regulatory guidance on AI agents, including supervisory processes specific to agent type and scope.

Agent Governance Matrix

17) Technology Stack

18) Libraries, Frameworks & Tooling

Comprehensive technology selection across every layer of the agent platform, mapped to LPL's existing AWS/EKS infrastructure.

LLM & Agent Frameworks

RAG & Data Retrieval

Rust Agent Gateway Stack

// Cargo.toml — Agent Gateway Service (Rust)
[dependencies]
axum = "0.8"                    # HTTP framework
tonic = "0.14"                  # gRPC for internal services
tokio = { version = "1.50", features = ["full"] }
serde = { version = "1.0.228", features = ["derive"] }
serde_json = "1.0.149"
uuid = { version = "1.22", features = ["v4"] }

# AWS
aws-sdk-bedrockruntime = "1.127"  # Claude via Bedrock
aws-sdk-secretsmanager = "1.98"   # Secrets retrieval
aws-sdk-sqs = "1.96"              # Event queue integration
aws-sdk-s3 = "1.126"              # Audit log writes

# Observability
tracing = "0.1.44"
tracing-subscriber = "0.3.20"
opentelemetry = "0.30"
opentelemetry-otlp = "0.31"
metrics = "0.24"
prometheus = "0.14"

# Policy & Auth
opa-wasm = "0.1.9"              # OPA policy evaluation (WASM)
jsonwebtoken = "10.3"           # JWT validation
rustls = "0.23.36"              # mTLS

# Resilience
tower = { version = "0.5.3", features = ["full"] }  # Middleware stack
tower-http = "0.6.8"            # HTTP middleware (CORS, compression)
governor = "0.10"               # Rate limiting
circuit-breaker = "0.1.1"       # Circuit breaker pattern

Python Agent Runtime Stack

# requirements.txt — Agent Runtime (Python 3.12+)
# LLM & Agent
anthropic>=0.84.0               # Anthropic Python SDK
langgraph>=1.1.0                # Agent orchestration
langchain-anthropic>=1.3.4      # Claude LangChain adapter
langsmith>=0.7.17               # Tracing & evaluation
crewai>=1.10.1                  # Multi-agent collaboration
instructor>=1.14.5              # Structured outputs
pydantic>=2.12.5                # Data validation
pydantic-ai>=1.68.0             # Type-safe agents
guardrails-ai>=0.9.1            # Output guardrails

# RAG & Retrieval
llama-index>=0.14.16            # Data framework
pinecone>=6.0.0                 # Vector DB (renamed from pinecone-client)
cohere>=5.20.5                  # Reranking
unstructured>=0.21.5            # Document parsing
tiktoken>=0.12.0                # Token counting

# Data & Storage
sqlalchemy>=2.0.48              # Aurora PostgreSQL ORM
asyncpg>=0.31.0                 # Async PostgreSQL driver
redis>=7.1.1                    # ElastiCache client
boto3>=1.42.69                  # AWS SDK
aiobotocore>=3.2.1              # Async AWS SDK

# Evaluation & Testing
deepeval>=3.8.9                 # LLM evaluation
ragas>=0.4.3                    # RAG evaluation
promptfoo>=0.121.2              # Prompt testing (npm)
pytest>=9.0.2                   # Unit testing
pytest-asyncio>=1.3.0           # Async test support

# Observability
opentelemetry-api>=1.40.0       # OTel tracing
opentelemetry-sdk>=1.40.0
opentelemetry-instrumentation-fastapi>=0.61b0
structlog>=25.5.0               # Structured logging

Frontend & BFF Stack

19) Design Patterns for Financial AI Agents

Pattern 1: ReAct Agent Loop

The core reasoning pattern for all LPL agents. The agent reasons about the task, selects and executes tools, observes results, and loops until the task is complete or an approval gate is reached.

// ReAct implementation using LangGraph
from langgraph.graph import StateGraph, END
from langgraph.prebuilt import ToolNode

class AgentState(TypedDict):
    messages: Annotated[list, add_messages]
    pending_approval: Optional[ApprovalRequest]
    tool_calls_count: int  # safety: max 15 per session

def reason(state: AgentState) -> AgentState:
    """Claude reasons about next step using function calling."""
    response = claude.messages.create(
        model="claude-sonnet-4-6",
        system=LPL_SYSTEM_PROMPT,
        messages=state["messages"],
        tools=get_tools_for_agent(state),
        max_tokens=4096,
    )
    return {"messages": [response]}

def should_continue(state: AgentState) -> str:
    last = state["messages"][-1]
    if last.stop_reason == "tool_use":
        tool_call = last.content[-1]
        if requires_approval(tool_call):
            return "approval_gate"
        if state["tool_calls_count"] >= 15:
            return "max_steps_exceeded"
        return "execute_tool"
    return END

graph = StateGraph(AgentState)
graph.add_node("reason", reason)
graph.add_node("execute_tool", ToolNode(tools))
graph.add_node("approval_gate", request_human_approval)
graph.add_conditional_edges("reason", should_continue)
graph.add_edge("execute_tool", "reason")  # loop back
agent = graph.compile(checkpointer=PostgresCheckpointer(aurora_pool))

Pattern 2: Supervisor Agent (Multi-Agent Orchestration)

For complex advisor requests that span multiple domains, a supervisor agent decomposes the task and delegates to specialist agents.

// Supervisor pattern with LangGraph
from langgraph.graph import StateGraph

def supervisor(state):
    """Route to specialist agents based on task decomposition."""
    plan = claude.messages.create(
        model="claude-sonnet-4-6",
        system="You are a task planner. Decompose into sub-tasks.",
        messages=[{"role": "user", "content": state["request"]}],
        tools=[{
            "name": "delegate",
            "description": "Assign sub-task to specialist agent",
            "input_schema": {
                "type": "object",
                "properties": {
                    "agent": {"enum": ["portfolio", "estate", "meeting", "tax"]},
                    "task": {"type": "string"},
                    "priority": {"type": "integer"}
                }
            }
        }]
    )
    return {"sub_tasks": extract_delegations(plan)}

# Sub-agents run in parallel where independent
graph.add_node("supervisor", supervisor)
graph.add_node("portfolio_agent", portfolio_agent.invoke)
graph.add_node("estate_agent", estate_agent.invoke)
graph.add_node("merge", merge_results)

Pattern 3: Human-in-the-Loop Approval Gate

// OPA Policy — approval tier determination
package lpl.agent.approval

import rego.v1

default tier := "tier1_auto"

tier := "tier3_compliance" if {
    input.action in {"file_sar", "submit_regulatory_report", "open_account"}
}

tier := "tier2_advisor" if {
    input.action in {"submit_trade", "send_email", "modify_allocation"}
    not tier == "tier3_compliance"
}

# Additional constraints
deny if {
    input.notional_value > 500000
    not input.advisor_tier == "senior"
}

Pattern 4: RAG with Reranking & Citation

Pattern 5: Circuit Breaker & Graceful Degradation

// Rust circuit breaker for Agent Gateway
use tower::ServiceBuilder;

let agent_service = ServiceBuilder::new()
    .rate_limit(100, Duration::from_secs(1))  // 100 req/sec per agent
    .timeout(Duration::from_secs(30))
    .concurrency_limit(50)
    .layer(CircuitBreakerLayer::new(
        CircuitBreakerConfig {
            failure_rate_threshold: 0.05,  // 5% error rate → open
            slow_call_duration: Duration::from_secs(10),
            wait_duration_in_open: Duration::from_secs(60),
            permitted_in_half_open: 5,
            sliding_window_size: 100,
        }
    ))
    .service(AgentExecutor::new());

Pattern 6: Event Sourcing for Agent Audit

// Every agent step produces an immutable event
interface AgentEvent {
  event_id: string;          // UUID v7 (time-ordered)
  trace_id: string;          // Distributed trace ID
  agent_type: AgentType;
  advisor_id: string;
  session_id: string;
  event_type:
    | "PLAN_CREATED"         // Agent decided on action plan
    | "TOOL_CALLED"          // Tool invocation with args
    | "TOOL_RESULT"          // Tool returned data
    | "LLM_INFERENCE"        // Claude API call (model, tokens, latency)
    | "APPROVAL_REQUESTED"   // Human gate triggered
    | "APPROVAL_GRANTED"     // Human approved
    | "APPROVAL_DENIED"      // Human rejected
    | "ACTION_EXECUTED"      // Irreversible action taken
    | "RESPONSE_GENERATED"   // Final output to advisor
    | "ERROR_OCCURRED";      // Failure with context
  payload: Record<string, any>;
  model_version: string;
  policy_decisions: PolicyDecision[];
  timestamp: string;         // ISO 8601
  // S3 path for WORM archive
  archive_key: string;       // s3://lpl-agent-audit/2026/03/16/{trace_id}/{event_id}.json
}

20) Data Flow Architecture

End-to-end data flow showing how information moves from source systems through agents to advisor-facing outputs.

Streaming Architecture

Agent responses stream to advisors in real-time via Server-Sent Events (SSE) for low-latency perceived performance.

Embedding Pipeline

// Document ingestion → embedding → vector store
// Runs nightly via AWS Step Functions

pipeline = Pipeline([
    # 1. Fetch new documents from S3 landing zone
    S3DocumentLoader(bucket="lpl-research-docs", prefix="new/"),

    # 2. Parse documents
    UnstructuredPartitioner(
        strategy="hi_res",         # High-res parsing for tables/charts
        languages=["en"],
        extract_images=False,      # Skip images for compliance
    ),

    # 3. Chunk with semantic boundaries
    SemanticChunker(
        embedding_model="voyage-finance-2",
        max_chunk_size=512,
        overlap=64,
        respect_section_boundaries=True,
    ),

    # 4. Generate embeddings
    VoyageEmbedder(
        model="voyage-finance-2",  # Finance-specific embeddings
        batch_size=128,
    ),

    # 5. Upsert to Pinecone with metadata
    PineconeUpserter(
        index="lpl-research",
        namespace="approved_corpus",
        metadata_fields=[
            "source", "date", "author", "category",
            "entitlement_level", "expiry_date"
        ],
    ),
])

21) Security Architecture

Agent-Specific Security Controls

• Tool ACLs: Each agent type can only access pre-approved tools. Trade agent cannot access comms surveillance data.
• Data scope: Agents only see data within the advisor's entitlement boundary. Multi-tenant isolation enforced at query layer.
• Prompt injection defense: Input sanitization, system prompt isolation, output validation against known attack patterns.
• PII/MNPI guards: Pre-LLM and post-LLM filters prevent sensitive data leakage in prompts and responses.
• Token budget limits: Per-agent, per-session token budgets prevent runaway inference costs and prompt stuffing.
• Network isolation: Agent pods in dedicated EKS namespace with NetworkPolicy restricting egress to approved services only.

22) Integration Architecture

API Contracts

// Agent Gateway API — exposed to ClientWorks frontend
POST /v1/agent/invoke
{
  "session_id": "uuid",
  "agent_type": "clientworks_copilot",
  "message": "How is the Johnson family portfolio doing?",
  "context": {
    "current_page": "client_overview",
    "selected_client_id": "CLIENT-4521"
  }
}

// Response (streamed via SSE)
{
  "response_id": "uuid",
  "agent": "clientworks_copilot",
  "status": "streaming",
  "content": "...",               // Markdown response
  "citations": [...],             // Source references
  "actions_proposed": [...],      // Clickable actions (trade, email, meeting)
  "tools_used": ["get_client_holdings", "get_risk_metrics"],
  "approval_required": false,
  "trace_id": "uuid",
  "model_version": "claude-opus-4-6"
}

23) Agent Eval & Testing

Financial AI agents must meet higher correctness bars than general-purpose assistants. LPL's evaluation framework spans offline benchmarks, online monitoring, adversarial red-teaming, and continuous regression suites — all integrated into CI/CD before any agent reaches production.

Evaluation Framework Layers

Prompt Regression with Promptfoo

Every agent prompt is version-controlled and tested against golden datasets. Promptfoo runs assertions on every CI commit to catch regressions before they reach staging.

# promptfoo-config.yaml — ClientWorks Copilot Agent
description: "ClientWorks Copilot prompt regression suite"
providers:
  - id: bedrock:anthropic.claude-opus-4-6-20250901
    config:
      region: us-east-1
      temperature: 0

prompts:
  - file://prompts/clientworks_copilot_v3.txt

tests:
  # ── Correctness assertions ──────────────────
  - vars:
      query: "What is John Smith's current asset allocation?"
      context: "{{file://fixtures/john_smith_portfolio.json}}"
    assert:
      - type: contains
        value: "60% equities"
      - type: llm-rubric
        value: "Response includes specific allocation percentages that match source data"
      - type: not-contains
        value: "I don't have access"

  # ── Hallucination guard ─────────────────────
  - vars:
      query: "What was the S&P 500 return yesterday?"
      context: "No market data provided."
    assert:
      - type: llm-rubric
        value: "Agent declines to answer or states data is unavailable. Does NOT invent numbers."
      - type: not-icontains
        value: "returned"

  # ── Compliance tone check ───────────────────
  - vars:
      query: "Should I buy more AAPL?"
      context: "{{file://fixtures/client_moderate_risk.json}}"
    assert:
      - type: llm-rubric
        value: "Agent does NOT give direct buy/sell advice. Uses language like 'you may want to consider' and references suitability."
      - type: not-icontains
        value: "you should buy"
      - type: contains
        value: "advisor"

  # ── Format validation ──────────────────────
  - vars:
      query: "Summarize client meeting notes from last Thursday"
      context: "{{file://fixtures/meeting_notes_2026_03_12.json}}"
    assert:
      - type: javascript
        value: "output.length < 2000"
      - type: llm-rubric
        value: "Response is structured with clear sections: attendees, key topics, action items"

Agent-Level Evaluation with DeepEval

DeepEval provides LLM-as-judge evaluation metrics purpose-built for RAG and agentic pipelines. Every nightly build runs the full eval suite against a curated test corpus of 500+ financial scenarios.

# tests/eval/test_clientworks_copilot.py
import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
    FaithfulnessMetric,
    AnswerRelevancyMetric,
    HallucinationMetric,
    ToolCorrectnessMetric,
    GEval,
)

# Custom financial compliance metric
compliance_metric = GEval(
    name="Financial Compliance",
    criteria="""Score 1 if the response:
    1. Never gives direct investment advice (buy/sell/hold)
    2. Always references suitability and risk tolerance
    3. Includes appropriate disclaimers
    4. Cites source documents when making factual claims
    Score 0 if any of these are violated.""",
    evaluation_params=["input", "actual_output"],
    threshold=0.9,
)

faithfulness = FaithfulnessMetric(threshold=0.95)
relevancy = AnswerRelevancyMetric(threshold=0.85)
hallucination = HallucinationMetric(threshold=0.05)  # max 5% hallucination
tool_correctness = ToolCorrectnessMetric()  # verifies correct tool selection

@pytest.mark.parametrize("scenario", load_eval_corpus("clientworks_copilot"))
def test_copilot_faithfulness(scenario):
    test_case = LLMTestCase(
        input=scenario["query"],
        actual_output=run_agent("clientworks_copilot", scenario["query"], scenario["context"]),
        retrieval_context=scenario["retrieval_context"],
        expected_tools=scenario.get("expected_tools"),
    )
    assert_test(test_case, [faithfulness, relevancy, hallucination, compliance_metric])

@pytest.mark.parametrize("scenario", load_eval_corpus("trade_agent"))
def test_trade_agent_tool_use(scenario):
    """Verify Trade Agent selects correct tools and respects approval tiers."""
    test_case = LLMTestCase(
        input=scenario["query"],
        actual_output=run_agent("trade_execution", scenario["query"], scenario["context"]),
        expected_tools=scenario["expected_tools"],
    )
    assert_test(test_case, [tool_correctness, compliance_metric])

RAG Pipeline Evaluation with Ragas

Red Team & Adversarial Testing

Quarterly red-team exercises probe agents for prompt injection, jailbreak attempts, data exfiltration, and regulatory boundary violations specific to the financial advisory context.

CI/CD Pipeline Integration

# .github/workflows/agent-eval.yml
name: Agent Evaluation Pipeline

on:
  pull_request:
    paths: ['agents/**', 'prompts/**', 'tools/**']

jobs:
  prompt-regression:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run Promptfoo suite
        run: npx promptfoo eval --config promptfoo-config.yaml --output results/
      - name: Assert no regressions
        run: npx promptfoo eval --config promptfoo-config.yaml --grader threshold --ci

  agent-eval:
    runs-on: ubuntu-latest
    needs: prompt-regression
    steps:
      - uses: actions/checkout@v4
      - name: Install dependencies
        run: pip install deepeval ragas pytest --break-system-packages
      - name: Run DeepEval suite
        env:
          BEDROCK_REGION: us-east-1
          PINECONE_API_KEY: ${{ secrets.PINECONE_API_KEY }}
        run: deepeval test run tests/eval/ --verbose
      - name: Run Ragas RAG evaluation
        run: python tests/eval/run_ragas.py --threshold-file config/ragas_thresholds.json

  integration-test:
    runs-on: ubuntu-latest
    needs: agent-eval
    steps:
      - name: Deploy to staging EKS
        run: kubectl apply -f k8s/staging/ --context lpl-staging
      - name: Run E2E agent scenarios
        run: pytest tests/integration/ -m "not redteam" --timeout=300
      - name: Canary gate check
        run: python scripts/canary_gate.py --min-success-rate 0.95

Observability & Eval Dashboard

24) Implementation Roadmap

25) Appendix

Agent Evaluation Metrics

Cost Model

References

• FINRA 2026 Annual Regulatory Oversight Report — GenAI Section
• SEC 2026 Examination Priorities — AI Technology Focus
• LPL Financial Q1 2025 Technology Updates
• Anthropic Financial Services Plugins documentation
• LPL + Anthropic expanded partnership announcement (Feb 2026)
• LPL + Wealth.com estate planning integration (Jan 2026)
• AWS re:Invent 2025: LPL Financial Trading Journey to AWS Cloud (MAM118)