NTB DQ Framework

Section 1Overview

One framework that answers, for every table in the lakehouse: can this data be trusted?

Quality dimensions

Rule types

Delta audit tables

162

Automated tests

The problem it solves. Without a shared framework, data quality checking is scattered: each pipeline hand-rolls its own validation (or has none), bad rows land silently in warehouse tables, and the first person to notice is a business user looking at a wrong number in a dashboard. There is no central record of what was checked, when, or what failed.

What the framework changes:

Rules are defined once, centrally. A rule is a small config — table, column, rule type, severity, owner — stored in a Delta registry. Identical rules are deduplicated automatically, and rules can be deactivated without deleting history.
Every run leaves an audit trail. Run logs, per-rule results, and the actual failing rows are persisted to Delta tables that dashboards can query directly.
Bad data can be blocked, not just reported. Critical-severity failures come back as a DataFrame of the exact offending rows, so the pipeline can filter them out before writing (Section 4).
Every rule declares an owner — accountability for data quality is written into the rule itself.

Section 2How it works

A pipeline calls one method — run_checks(table) — and the engine does the rest.

dq_rule_registry

rules registered once with add_rule() — table, column, rule type, severity, owner

engine.run_checks(target_table)

loads every active rule for that table and runs them in one pass

16 rule types · 6 dimensions

each check counts rows evaluated, passed, and failed — and captures the failing rows

dq_run_log · dq_result_table · dq_failed_records

full audit trail persisted to Delta — queryable by any dashboard

RunSummary → critical_failed_rows_df

returned to the pipeline: pass/fail counts plus the exact rows that failed critical rules

Non-blocking by design. A failing check never crashes the pipeline. run_checks() always returns a complete RunSummary — the pipeline stays in control and decides what to do with the failures.

✓

Rules describe good data. Every rule defines what valid rows look like — an allowed list, a regex, a range, a business condition — and the framework reports whatever doesn't match. One consistent mental model across all 16 rule types.

Section 3Six dimensions, sixteen rule types

The industry-standard data quality dimensions, each covered by concrete, configurable rule types.

Completeness

3 rule types

Is anything missing?

null_checkrows where a column is null or empty

missing_partitionexpected date partitions that never arrived

volume_checkrow count vs. a baseline threshold

Validity

4 rule types

Are values well-formed and in range?

regex_checkvalues not matching a format pattern (e.g. 13-digit ID card)

allowed_valuesvalues outside an allowed list

range_checkvalues outside min/max bounds

type_checkvalues that can't be cast to the expected type

Accuracy

3 rule types

Does it match a trusted reference?

reference_matchrow-level mismatches against a reference table

tolerance_matchsame, but allowing a numeric tolerance

reconciliationaggregate totals (sum/count) vs. the source system

Consistency

3 rule types

Does it agree across tables and systems?

referential_integrityorphan rows missing from a reference table

cross_system_matchcolumn values that differ between systems

business_ruleany SQL condition that should hold, e.g. end_date >= start_date

Timeliness

2 rule types

Is the data fresh?

freshness_checkhours of delay since the latest timestamp

sla_checkfails when the delay breaches an SLA threshold

Uniqueness

1 rule type

Are there duplicates?

duplicate_checkduplicate groups on a set of key columns

Section 4The quarantine workflow

Two severity levels separate watch this from never let this through — and every load takes one of three paths.

How a load flows through the checkpoint

incoming load — pipeline DataFrame

engine.run_checks(target_table)

256 active rules · every load, every day since 21 May 2026

Pass

target table

99.997% of all row-checks — clean data flows through untouched

Warning

target table + DQ log

11,832 rows logged to date — the row still lands, a policy agreed with data owners, and the trend is watched on dashboards

Critical

Quarantine zone

3,735 rows stopped to date — diverted to dq_failed_records, never reaching the target table

Live figures from lakehouse.data_quality as of 13 Aug 2026.

Who reviews the quarantine? The data engineer and the data owner review quarantined rows together — the engineer diagnoses the pipeline side, the owner rules on the business side.

✓

Calibrated, not trigger-happy. Critical status is earned: only 22 of 256 active rules are critical today. A rule that proves too strict is deactivated in the registry — a soft-delete that keeps full history — and quarantined rows are never deleted, so data can be reloaded once the rule is corrected.

Warning — observe & investigate

Result and pass/fail counts recorded on every run
A sample of failing rows (up to 100 by default) saved for diagnosis
The row still lands in the target table — a policy agreed with data owners; the trend is watched on dashboards

Critical — block & quarantine

All failing rows saved to dq_failed_records — no sampling cap
Failing rows returned as RunSummary.critical_failed_rows_df
The pipeline diverts them to the quarantine zone — they never reach the target table

Filtering bad rows before an upsert

summary = engine.run_checks("my_catalog.my_schema.customers")

if summary.critical_failed_rows_df is not None:
    clean_df = df.join(summary.critical_failed_rows_df, how="left_anti")  # drop bad rows
    clean_df.write.mode("overwrite").saveAsTable("my_catalog.my_schema.customers")

This is the framework's sharpest capability: quality checking stops being a report someone reads later and becomes a gate inside the pipeline — bad records are quarantined in the audit table while clean data flows through.

Section 5Delta audit tables

On first initialization the framework creates four Delta tables — the permanent record of what was checked and what failed.

Table	What it holds
`dq_rule_registry`	Every rule definition, with owner, severity, and an activation flag — rules are deactivated, never lost
`dq_run_log`	One row per run: when it ran, how many rules passed and failed, total rows evaluated
`dq_result_table`	Per-rule results for every run — pass rate, failed count, error details
`dq_failed_records`	The failing rows themselves — all rows for critical rules, a sample for warnings

✓

Zero setup. Tables are created automatically the first time a DQEngine is initialized in a catalog — no DDL scripts, no provisioning step.

Section 6Monitoring views

Built-in aggregations turn the audit tables into dashboard-ready answers — no hand-written SQL required.

Method	Question it answers
`daily_summary()`	How did each table do today — runs, pass rates, failures by table and date?
`dimension_summary()`	Which quality dimension is failing — is it a completeness problem or a freshness problem?
`rule_trend(days=30)`	Is a specific rule getting better or worse over time?
`top_offenders(top_n=10)`	Which rules have the lowest pass rates — where should the team focus first?

Section 7Guardrails & scale

Engineering that keeps the framework safe to open up to many teams and many pipelines at once.

SQL guard — injection protection built in

Rules accept user-supplied SQL expressions, so every expression and identifier is validated before it reaches Spark SQL. DDL/DML statements (DROP, DELETE, MERGE, …), SQL comments, UNION SELECT, and stored-procedure patterns are all blocked at add_rule() time with a clear DQValidationError. Normal boolean expressions like total_amount > 0 pass through untouched.

Row scoping with `filter_expression` new in 1.1.0

Any rule can be scoped to a subset of rows with a SQL filter — for example COALESCE(is_deleted, false) = false checks only live rows and treats soft-deleted ones as passing. It works on every rule type and goes through the same SQL guard.

Safe parallel runs

Many pipelines can run checks concurrently against the same DQ tables: primary keys are UUIDs generated in Python (no identity-column metadata conflicts), tables are partitioned by target_table, and writes use WriteSerializable isolation. This was hardened iteratively — versions 0.2.1 through 0.3.1 each removed a real concurrency failure seen in production.

Local time, local audit

All DQ timestamps are persisted in Asia/Bangkok time, so run logs and dashboards line up with the business day.

Section 8Developer experience

A pipeline adds full quality checking in about ten lines.

from ntb_dq_framework import DQEngine

engine = DQEngine(spark, catalog="my_catalog")   # creates DQ tables if missing

engine.add_rule(
    rule_name="null_email",
    dimension="Completeness",
    target_table="my_catalog.my_schema.customers",
    target_column="email",
    rule_type="null_check",
    severity="critical",
    owner="data-team",
)

summary = engine.run_checks("my_catalog.my_schema.customers")
print(summary.total_rows_passed, summary.total_rows_failed)

Ships with its own AI assistant

The package bundles a Claude Code agent: after pip install, one command — ntb-dq install-agent — drops it into the project. The agent then activates automatically whenever someone mentions DQEngine or a quality dimension, and helps engineers design rules, write correct add_rule() calls, and debug results.

Tested like a library, not a script

162 automated tests across ~2,700 lines of framework code — conventional unit tests plus property-based tests (Hypothesis) that probe invariants like row-count preservation and critical-severity handling with generated edge cases. Several past bugs have exploration-and-fix test pairs preserved in the suite.

Section 9Release history

Eight releases on PyPI — each one driven by real production use.

1.1.0

Row scoping

filter_expression lets any rule check only the rows that matter (e.g. skip soft-deleted records).

Current

0.4.0

Bundled AI agent

Claude Code agent ships inside the package; ntb-dq install-agent CLI installs it per project or per user.

0.3.3

Bangkok timestamps

All DQ timestamps persisted in Asia/Bangkok so audit logs match the business day.

0.3.1

True parallel safety

UUID primary keys replaced identity columns, eliminating metadata conflicts when pipelines run checks concurrently.

0.2.1

The critical gate

critical_failed_rows_df introduced — pipelines can now filter bad rows out before writing. Tables partitioned for parallel runs.

0.2.0

Non-blocking runs

Failed checks no longer raise — the pipeline receives a full RunSummary and stays in control.

0.1.5

Initial release

Six-dimension rule-based checks with Delta table persistence.

Section 1Overview

Section 2How it works

Section 3Six dimensions, sixteen rule types

Completeness

Validity

Accuracy

Consistency

Timeliness

Uniqueness

Section 4The quarantine workflow

How a load flows through the checkpoint

Filtering bad rows before an upsert

Section 5Delta audit tables

Section 6Monitoring views

Section 7Guardrails & scale

SQL guard — injection protection built in

Row scoping with filter_expression new in 1.1.0

Safe parallel runs

Local time, local audit

Section 8Developer experience

Ships with its own AI assistant

Tested like a library, not a script

Section 9Release history

Row scoping with `filter_expression` new in 1.1.0