Error Injection
Clean synthetic data cannot test a data-quality pipeline. Every rule passes, every reconciliation balances, and nothing is proven. What a quality suite needs is a fixture whose defects are known, varied and reproducible — so a test can assert not just "the pipeline flagged something" but "the pipeline flagged exactly these rows".
errors injects seeded defects as the very last pipeline stage — after calculated features and after derived tables. That ordering is a hard requirement rather than a preference: a representation violation turns a numeric column into text, so all arithmetic must already have happened. It also enables the interesting case — corrupt a derived table and the detail no longer rolls up to it.
The configuration block
Keyed by table name (the base table under table_name), with a columns section and a table section:
errors={
"payroll": {
"columns": {
"gross_pay": {"format_violation": {"rate": 0.08,
"styles": ["words", "currency"]},
"missing": 0.02},
"base_pay": {"out_of_range": {"rate": 0.04, "factor": 1000}},
},
"table": {
"duplicate_rows": 0.05,
"mapping_break": {"columns": ["employee_id", "cost_centre"], "rate": 0.05},
},
},
}
Every operator takes a rate between 0 and 1, either bare ("missing": 0.02) or in its expanded form ({"rate": 0.02}).
Column operators
| Operator | Effect |
|---|---|
missing |
Cells set to null. |
format_violation |
The same value, wrongly represented. Optional styles; when omitted, the set matching the column's dtype is used. |
out_of_range |
Numeric values scaled by factor (default 1000) — absurd, but still numeric. Requires a numeric column. |
format_violation styles:
| Numeric styles | Example |
|---|---|
words |
1200 → "one thousand, two hundred" |
thousands_sep |
"1,200.00", or a decimal comma "53,68" below 1000 |
currency |
"£1,200.00" |
trailing_space |
"1200.0 " |
| String styles | Effect |
|---|---|
truncate |
Value cut short |
case_flip |
Casing inverted |
whitespace |
Stray leading/trailing whitespace |
Table operators
| Operator | Effect |
|---|---|
duplicate_rows |
Random rows appended verbatim — duplicate identifiers included. |
mapping_break |
{"columns": [key, mapped], "rate": ...} — the mapped column takes a value belonging to a different key, violating a supposedly strict mapping. |
Worked example
A monthly payroll register generated twice from the same configuration — once clean as the reference, once corrupted — plus a derived cost-centre summary corrupted after derivation so the detail no longer reconciles.
"""
Example: error injection — generating data that is realistically WRONG.
Clean synthetic data cannot test a data-quality pipeline: every rule passes, so
nothing is proven. `errors` injects seeded, reproducible defects as the VERY
LAST stage — after calculated features and derived tables — which is what makes
it safe: identities and aggregates are computed on clean data, and the
corruption exists only in the delivered output.
Configuration is keyed by table name (the base table under `table_name`), with
a `columns` section and a `table` section:
column operators
missing cells set to null
format_violation the same value, wrongly REPRESENTED
numeric styles: words, thousands_sep, currency,
trailing_space
string styles: truncate, case_flip, whitespace
out_of_range numeric values scaled by `factor` — absurd, still numeric
table operators
duplicate_rows random rows appended verbatim (duplicate identifiers)
mapping_break a supposedly strict key -> value mapping violated
Note the deliberate design: a column carrying corrupted REPRESENTATIONS is
delivered as a string column — a Dataset column has one type, and "one thousand,
two hundred" is text. `missing` alone never changes a column's kind.
The table here is a monthly payroll register plus a derived cost-centre summary
that is corrupted AFTER derivation, so the detail no longer rolls up to it —
the unresolved mismatch a reconciliation engine has to catch.
Run with: python faker_error_injection.py
"""
import pandas as pd
from ydata.metadata import Metadata
from ydata.metadata.builder import MetadataConfigurationBuilder
from ydata.synthesizers import FakerSynthesizer
employees = pd.DataFrame({
"employee_id": [f"EMP-{i:04d}" for i in range(1, 13)],
"cost_centre": (["CC-ENG"] * 5) + (["CC-SALES"] * 4) + (["CC-OPS"] * 3),
"contract": (["permanent"] * 8) + (["contractor"] * 4),
})
columns = {
"base_pay": {"datatype": "numerical", "vartype": "float",
"min": 1_200.0, "max": 9_500.0},
"overtime_hours": {"datatype": "numerical", "vartype": "int",
"min": 0, "max": 40},
"reviewer": {"datatype": "string", "vartype": "string", "characteristic": "name"},
}
metadata = Metadata(configuration_builder=MetadataConfigurationBuilder(columns))
BASE_CONFIG = dict(
table_name="payroll",
scaffold={
"dimensions": [
{"column": "employee_id", "from_table": "employees",
"from_column": "employee_id", "include": ["cost_centre", "contract"]},
{"column": "pay_month",
"date_range": {"start": "2026-01-01", "end": "2026-03-01", "freq": "MS"}},
]
},
reference_data={"employees": employees},
distributions={"base_pay": {"distribution": "lognormal", "mean": 8.1, "sigma": 0.35,
"min": 1_200.0, "max": 9_500.0}},
calculated_features=[
{"calculated_features": "overtime_pay",
"function": lambda pay, hours: (pay / 160 * 1.5 * hours).round(2),
"calculated_from": ["base_pay", "overtime_hours"]},
{"calculated_features": "gross_pay",
"function": lambda pay, ot: (pay + ot).round(2),
"calculated_from": ["base_pay", "overtime_pay"]},
],
derived_tables=[
{"name": "cost_centre_summary", "type": "aggregate",
"group_by": ["pay_month", "cost_centre"],
"aggregations": {
"total_gross": {"column": "gross_pay", "agg": "sum"},
"headcount": {"column": "employee_id", "agg": "nunique"},
}},
],
)
# ---------------------------------------------------------------------------
# 1. The clean reference run — same configuration, no `errors` block
# ---------------------------------------------------------------------------
clean_synth = FakerSynthesizer(locale="en")
clean_synth.fit(metadata, **BASE_CONFIG)
clean = clean_synth.sample(random_state=7)
clean_payroll = clean["payroll"].to_pandas()
clean_summary = clean["cost_centre_summary"].to_pandas()
# clean data reconciles perfectly — that is the baseline a DQ suite starts from
recomputed = (clean_payroll.groupby(["pay_month", "cost_centre"])["gross_pay"]
.sum().round(2))
reported = clean_summary.set_index(["pay_month", "cost_centre"])["total_gross"].round(2)
assert (reported == recomputed.reindex(reported.index)).all()
# ---------------------------------------------------------------------------
# 2. The same run, corrupted
# ---------------------------------------------------------------------------
dirty_synth = FakerSynthesizer(locale="en")
dirty_synth.fit(
metadata,
**BASE_CONFIG,
errors={
"payroll": {
"columns": {
# numbers arriving as prose and as currency strings
"gross_pay": {"format_violation": {"rate": 0.08,
"styles": ["words", "currency"]}},
# a plain hole; the column stays numeric
"overtime_hours": {"missing": 0.06},
# absurd but still numeric — an outlier detector's job
"base_pay": {"out_of_range": {"rate": 0.04, "factor": 1000}},
# string styles on a string column
"reviewer": {"format_violation": {"rate": 0.10,
"styles": ["case_flip", "whitespace"]}},
},
"table": {
"duplicate_rows": 0.05,
# employee_id -> cost_centre is supposed to be strict
"mapping_break": {"columns": ["employee_id", "cost_centre"],
"rate": 0.05},
},
},
# corrupting a DERIVED table breaks reconciliation with the detail
"cost_centre_summary": {
"columns": {"total_gross": {"out_of_range": {"rate": 0.15, "factor": 100}}},
},
},
)
dirty = dirty_synth.sample(random_state=7)
payroll = dirty["payroll"].to_pandas()
summary = dirty["cost_centre_summary"].to_pandas()
# ---------------------------------------------------------------------------
# 3. Every defect class a quality pipeline should now flag
# ---------------------------------------------------------------------------
NUMERIC = r"-?\d+(\.\d+)?"
text_cells = (payroll["gross_pay"].notna()
& ~payroll["gross_pay"].astype(str).str.fullmatch(NUMERIC))
print("Representation violations in gross_pay:")
print(payroll.loc[text_cells, ["employee_id", "pay_month", "gross_pay"]]
.head(4).to_string(index=False))
assert text_cells.any()
assert pd.api.types.is_string_dtype(payroll["gross_pay"]), \
"a corrupted representation makes the column text"
missing = payroll["overtime_hours"].isna().sum()
print(f"\nMissing overtime_hours: {missing}")
assert missing > 0
assert pd.api.types.is_numeric_dtype(payroll["overtime_hours"]), \
"`missing` alone keeps the column numeric"
outliers = pd.to_numeric(payroll["base_pay"], errors="coerce") > 9_500.0
print(f"base_pay values outside the declared domain: {outliers.sum()}")
assert outliers.any()
truth = dict(zip(employees["employee_id"], employees["cost_centre"]))
broken = [e for e, c in zip(payroll["employee_id"], payroll["cost_centre"])
if truth[e] != c]
print(f"Broken employee -> cost_centre mappings: {len(broken)}")
assert broken
duplicates = int(payroll.duplicated().sum())
print(f"Duplicated payroll rows: {duplicates}")
assert duplicates > 0
assert len(payroll) > len(clean_payroll), "duplicates are appended, not substituted"
# the derived summary no longer agrees with the detail
parseable = pd.to_numeric(payroll["gross_pay"], errors="coerce")
recomputed = parseable.groupby([payroll["pay_month"], payroll["cost_centre"]]).sum()
reported = summary.set_index(["pay_month", "cost_centre"])["total_gross"]
mismatches = (reported.round(2) != recomputed.reindex(reported.index).round(2)).sum()
print(f"Cost-centre groups that no longer reconcile: {mismatches} of {len(reported)}")
assert mismatches > 0
# ---------------------------------------------------------------------------
# 4. Reproducible: the same seed corrupts the same cells, every run
# ---------------------------------------------------------------------------
again = dirty_synth.sample(random_state=7)["payroll"].to_pandas()
assert payroll.equals(again), "error injection must be reproducible under random_state"
print(f"\n{len(clean_payroll)} clean rows -> {len(payroll)} delivered rows")
print("All injected defect classes present, and reproducible under random_state=7.")
Notes
A corrupted representation makes the column text
A Dataset column has one type, and "one thousand, two hundred" is text. So a column touched by format_violation is delivered as a string column — the untouched numbers rendered as text ("12.5") beside the corrupted cells. That is deliberate: it is exactly what a downstream parser has to survive. missing alone never changes a column's kind — numerics stay numeric via nullable dtypes.
Operators compose in a fixed order
out_of_range runs before format_violation, so both may target the same column: a cell can be scaled and then have its representation corrupted, never a corrupted string multiplied.
Generate the clean twin
Run the same configuration with and without the errors block, under the same random_state, and you get a matched pair: the ground truth and the corrupted delivery. Asserting what a pipeline should have caught then becomes a diff.
All draws are reproducible under sample(random_state=...) — the same run corrupts the same cells every time.
Related Materials
