AI Data Quality Agent
An autonomous LLM-powered agent that continuously monitors data lake tables, classifies quality issues by severity, and proposes or applies SQL fixes — reducing manual triage time by over 60%.
Problem Solved
Data quality enforcement in large Lakehouse environments is a manual, reactive process. Engineers write Great Expectations rules for known failure modes, but novel issues — schema drift, upstream API format changes, silent null explosions — slip through because no one thought to write a rule for them first.
This agent reverses the model: instead of maintaining a library of hand-crafted rules, it uses an LLM to continuously inspect table statistics, infer what "normal" looks like, and generate targeted SQL or PySpark fixes when anomalies are detected — creating an audit trail that feeds back into the rule library over time.
Key Features
Autonomous Issue Classification
The agent profiles every registered table daily and uses Claude to classify issues into five severity tiers — from cosmetic (trailing whitespace) to critical (referential integrity breaks) — routing each to the right response workflow.
Self-Healing Pipelines
For low-risk issues (null coalescing, type casting, dedup), the agent applies a generated fix directly to a staging table and promotes it after a configurable approval window with no human in the loop.
Natural-Language Explanations
Every detected issue includes a plain-English summary written for data consumers, so business analysts understand the impact without reading SQL error logs.
MLflow Experiment Tracking
All agent runs, prompts, generated SQL, and outcomes are logged as MLflow experiments, giving the team a full audit trail and a dataset for fine-tuning future fix quality.
Architecture
Architecture Overview
A Databricks Workflow triggers the agent on a 6-hour cadence. The orchestrator (LangChain ReAct agent) is given a tool palette: read table stats, run SQL probes, call the Claude API, write findings to a Delta audit table, and optionally submit a fix job via the Databricks REST API.
Claude acts as the reasoning engine — it receives a structured context block containing table schema, recent statistics, and sample rows, then decides whether to classify the issue, request more data, or emit a fix.
- Orchestration: Databricks Workflows + LangChain ReAct
- Reasoning: Anthropic Claude API (claude-sonnet-4-6)
- Storage: Delta Lake audit table + MLflow experiment store
- Fix delivery: Databricks REST API for notebook/job submission
- Observability: Databricks monitoring + Slack webhook alerts
Key Decisions
ReAct agent over a simple chain
Quality issues require multi-step investigation — the agent often needs to query a second table to confirm a foreign-key break, or pull a sample of rows to verify a suspected encoding issue. A linear chain can't handle this branching; ReAct's tool-call loop can.
Claude as reasoning engine, not a fine-tuned model
Fine-tuning on SQL fix patterns would require continuous labelled data curation as schemas evolve. Using Claude with a well-structured prompt is cheaper to maintain, easier to audit, and improves automatically as the base model improves.
Staging-then-promote for auto-fixes
Applying fixes directly to production tables is too risky, even for low-severity issues. Writing to a staging partition with an approval window (defaulting to 24 h) gives engineers a rollback window while still automating the bulk of routine fixes.
Screenshots
Interested in something similar?
We scope, design, and ship data and AI tools that solve the problems your team actually has.
Let's Talk