Skip to content

Faker Synthesizer

Overview

YData SDK includes the Faker Synthesizer, a method for generating synthetic data from a schema or metadata only — no real dataset is required. You define column types and characteristics (such as id, name, email, phone, address, zipcode), optional regex patterns, categorical values, and date ranges. The synthesizer then generates rows that match this schema, making it ideal when you need plausible synthetic data without access to any real source data.

You can build the schema with MetadataConfigurationBuilder or a dictionary configuration, and optionally derive metadata from an existing dataset and then generate from it. Locale support allows you to control the language and format of generated values.

Key Features

  • No Source Data Required: Generate data purely from a schema or metadata definition.
  • Metadata- and Schema-Driven: Define columns by datatype, vartype, and characteristics (id, name, email, phone, address, zipcode, date).
  • Flexible Constraints: Use regex patterns for string columns and categorical distributions for controlled variety.
  • Date Control: Set min, max, and format for date columns.
  • Dual Configuration: Use either MetadataConfigurationBuilder for programmatic setup or a dictionary config.
  • Locale Support: Choose a locale (e.g. "en") for language and regional formatting.
  • Fit from Data or Scratch: Fit from existing data’s metadata (e.g. infer schema then generate) or from a schema built from scratch.
  • Structural Scaffolds: Pin the table's structure with a declarative dimensional grid (dates × entities × slots, with density rules) — coverage and row counts guaranteed by construction.
  • Custom Distributions: Choose the statistical shape of numerical columns (normal, lognormal, poisson, ...) instead of uniform-within-domain.
  • Conditional Generation: Sample a column differently per segment defined by one or more other columns — first-match rules with distribution or value-probability outcomes.
  • Calculated Features (Business Rules): Columns computed exactly from other columns after sampling — arithmetic identities, sign rules, derived flags.
  • Time Series: Rows that depend on earlier rows of the same entity — seasonality, trend and persistence on a column; reactions to past values, balances that follow from their flows, regimes and lifecycles on the table.
  • Multi-table Databases: Several tables from one schema (fit(tables=...)): primary and foreign keys, children per parent, parent filters, 1:1 extensions, cross-table calculated features, and labelled errors including orphan foreign keys.

Use Cases

  • Prototyping and Demos: Quickly create sample tables for UI demos or product mockups.
  • Test Fixtures and Dev Databases: Populate test and development environments with realistic-looking data.
  • Schema-Only or Masking Workflows: Generate data that respects a target schema without exposing real data.
  • No Real Data Available: Produce synthetic data when data collection is impossible or not yet done.

Best Practices

  • Define Clear Datatypes and Characteristics: Use the right characteristic (e.g. email, phone) so generated values match expectations.
  • Use Categories and Regex for Control: Categorical distributions and regex keep variety predictable and valid.
  • Set Date Ranges Where Needed: Use min/max and format for date columns to avoid invalid or inconsistent values.
  • Prefer the Builder for Complex Schemas: MetadataConfigurationBuilder keeps large schemas readable and maintainable.

Worked Examples

Each configuration block below has a runnable, self-contained example that isolates one concept and asserts the property it is supposed to guarantee. Start with the one that matches the problem you have:

Example The question it answers
Structural Scaffolds Which rows exist? Grids, calendars, density, and reference attributes carried in by include.
Distributions & Conditions What shape do the values have, and how does one column depend on another?
Calculated Features Which columns must be computed rather than sampled, so the arithmetic holds on every row?
Sequence & Per-Partition Features How do I express "the previous row of the same entity" — running totals, deltas, lags?
Time Series How do rows depend on earlier rows — seasonality, memory, reactions to past values, balances that follow from their flows, regimes and lifespans?
Derived Tables & Aggregations How do I get summaries that reconcile with the detail by construction?
Per-Group Totals How do I hold a figure that belongs to a group — a daily cap, a monthly amount — rather than to a row?
Error Injection How do I generate data that is realistically wrong, for testing quality pipelines?
Multi-table Databases How do I generate several related tables, with every foreign key resolving and a plausible number of children per parent?

Every example lives under examples/synthesizers/fakersynthesizer/ and runs as a plain script.

Advanced Usage

The Faker Synthesizer supports two configuration styles:

  • Builder: Use MetadataConfigurationBuilder to add columns with datatype, vartype, characteristic, and optional regex, categories, or date min/max. Then build a Metadata and pass it to fit().
  • Dictionary: Pass a dictionary mapping column names to config (datatype, vartype, characteristic, regex, categories, min, max, format, etc.) into MetadataConfigurationBuilder(config) and then into Metadata.

You can fit from scratch (metadata built only from schema) or from existing data by computing Metadata(data) and fitting the synthesizer on that metadata to generate data that matches the same structure.

Structural Scaffolds

Some tables have a structure that must hold exactly — one row per product per day, every day of the month present, quieter weekends. Column-wise generation cannot guarantee that shape; a scaffold can. A scaffold is a declarative dimensional grid: the cross product of its dimensions (a date_range, a static values list, or a from_table column sourced from a reference DataFrame), optionally filtered by density rules, becomes the table's seed rows. Scaffold columns are emitted verbatim; every other metadata column is generated per seed row.

synth = FakerSynthesizer()
synth.fit(
    metadata,
    scaffold={
        "dimensions": [
            {"column": "product_id", "from_table": "products", "from_column": "product_id"},
            {"column": "booking_date",
             "date_range": {"start": "2026-03-01", "end": "2026-03-31", "freq": "D"}},
        ],
        "density": {
            "by": {"table": "products", "column": "category"},
            "rates": {"fuel": 1.0, "beverages": 0.85, "snacks": 0.7},
            "multipliers": [{"when": "weekend", "factor": 1.25, "except": ["fuel"]}],
        },
    },
    reference_data={"products": products_df},   # backs `from_table` dimensions
)
data = synth.sample(random_state=42)            # rows defined by the grid; reproducible

When a scaffold is configured, sample_size is ignored — the grid defines the row count — and random_state makes the grid and the sampled columns reproducible. The same scaffold configuration model is used by the LLM synthesizer, so a structural definition moves between the two engines unchanged.

A scaffold can also be declared in a YAML configuration file alongside the columns, using the reserved top-level scaffold key, and passed to fit(tables=...):

columns:
  quantity:
    vartype: float
    datatype: numerical
    min: 1.0
    max: 60.0

scaffold:
  dimensions:
    - column: product_id
      from_table: products
      from_column: product_id
    - column: booking_date
      date_range: {start: "2026-03-01", end: "2026-03-31", freq: D}

synth.fit(tables="faker_config.yaml", reference_data={"products": products_df})

The file may also carry distributions, conditions, sequence, totals, derived_tables and errors. Every Faker configuration file loads into one type, FakerSchema: a file without a top-level tables: key is a single table, and one with it is a database (see Multi-table Databases). load_faker_schema(path) returns it for inspection or reuse:

from ydata.synthesizers.faker import load_faker_schema

schema = load_faker_schema("faker_config.yaml")   # one table, named "data"
schema["data"].scaffold, schema["data"].conditions
synth.fit(tables=schema, reference_data={"products": products_df})

A plain single table (no keys or relationships) is generated exactly as with fit(metadata, ...), and its calculated features may name columns bare ("amount") or qualified ("data.amount").

Custom Distributions

By default, numerical columns are sampled uniformly within their declared domain. The distributions argument picks the shape instead — supported: uniform, normal, lognormal, exponential, poisson, triangular, beta. Optional min/max clip the values (falling back to the column's domain), and integer columns are rounded:

synth.fit(
    metadata,
    distributions={
        "amount": {"distribution": "lognormal", "mean": 3.0, "sigma": 0.5},
        "n_items": {"distribution": "poisson", "lam": 3.0},
        "score": {"distribution": "normal", "loc": 50, "scale": 10, "min": 0, "max": 100},
    },
)

Being pure data, distributions can also live in the YAML configuration under the reserved distributions key.

Conditional Generation

A column's behavior often depends on other columns: fuel settlements are large, shop settlements small; credit notes are negative; a premium segment buys different products. The conditions argument samples a column per segment, where segments are defined by when predicates over one or more columns (equality or is-in, AND semantics). Rules are evaluated in order — first match wins — and unmatched rows fall back to default or to the column's base sampling:

synth.fit(
    metadata,
    conditions=[
        {   # numerical column: different distribution per segment
            "column": "amount",
            "rules": [
                {"when": {"category": "fuel"},
                 "distribution": {"distribution": "lognormal", "mean": 8.6, "sigma": 0.55}},
                {"when": {"category": ["shop", "commission"], "channel": "online"},
                 "distribution": {"distribution": "lognormal", "mean": 6.9, "sigma": 0.7}},
            ],
            "default": {"distribution": {"distribution": "uniform", "low": 1, "high": 50}},
        },
        {   # categorical column: different value probabilities per segment
            "column": "doc_type",
            "rules": [
                {"when": {"category": "fuel"},
                 "values": {"Invoice": 0.9, "Credit Note": 0.1}},
            ],
            "default": {"values": {"Invoice": 0.7, "Debit Note": 0.2, "Credit Note": 0.1}},
        },
    ],
)

when columns may be other metadata columns, scaffold dimensions (per-product behavior on a grid), or other conditioned columns — dependencies are ordered automatically and cycles rejected at fit() time. Conditions cannot target columns marked unique (per-segment outcomes cannot guarantee uniqueness across segments — rejected at fit()); distribution overrides on unique columns keep the constraint enforced, falling back to duplicates with a warning only when the distribution's support is too small. Conditions compose with the rest of the pipeline in a fixed order: scaffold → column sampling (with distributions) → conditioned columns → calculated features. Like distributions, conditions are pure data and can live in the YAML configuration under the reserved conditions key.

Sequences and Derived Tables

Two blocks turn single-table generation into a small, reconcilable database:

sequence declares row ordering — the output is stably sorted by partition_by + order_by, and calculated features marked "per_partition": True are applied within each partition, in order. That makes temporal relations exact per entity (running balances via cumsum, lags via shift inside the callable). Scaffold dimensions may also carry extra source columns into the rows via include (e.g. an account's opening_balance), joined deterministically:

synth.fit(
    metadata,
    scaffold={"dimensions": [
        {"column": "account_id", "from_table": "accounts",
         "from_column": "account_id", "include": ["opening_balance"]},
        {"column": "date", "date_range": {"start": "2026-03-01", "end": "2026-03-31"}},
    ]},
    reference_data={"accounts": accounts_df},
    sequence={"order_by": "date", "partition_by": ["account_id"]},
    calculated_features=[
        {"calculated_features": "balance",
         "function": lambda opening, amount: opening + amount.cumsum(),
         "calculated_from": ["opening_balance", "amount"],
         "per_partition": True},
    ],
)

derived_tables computes additional tables FROM the generated table — declarative aggregate roll-ups (sum, mean, min, max, count, nunique, first, last; the order-dependent first/last require a sequence) or a custom DataFrame→DataFrame callable. Aggregates reconcile to the detail by construction: sum the detail and you get the aggregate, exactly — generate the transaction grain, derive the settlement/statement grains. With derived tables configured, sample() returns a dict of table name → Dataset (the base table under table_name). Conditions also accept calendar predicates on date columns ({"weekday_in": [5, 6]}, {"month_in": [12]}, {"day_in_month": [25]}) for paydays and seasonality. See the full flow — transactions, exact running balances, and derived monthly statements — in examples/synthesizers/fakersynthesizer/ledger_generation.py.

Per-Group Totals

Some figures belong to a group rather than to a row: a site takes 2,400 litres of fuel a day, a supplier issues GBP 500 of credit a month, a store's takings cannot exceed the float. A distribution shapes one value at a time and cannot express that, and derived_tables only measures the total once it exists. totals imposes it — each group is scaled by a single factor, so the mix and the spread the model drew survive and only the magnitude moves:

synth.fit(
    metadata,
    totals=[
        # a ceiling: groups already under it are untouched
        {"columns": ["net_value", "vat_value", "gross_value"],
         "measure": "gross_value",
         "group_by": ["billing_date"],
         "max": 5000.00, "round": 2},

        # an exact figure: every group is scaled to it, up or down
        {"columns": "credit_amount", "group_by": ["supplier", "month"],
         "target": 500.00, "round": 2},
    ],
)

columns scale together, which is how co-dependent monetary columns keep their identities (gross == net + vat survives the rescale) — totals run after calculated features, so a feature is not recomputed afterwards; list the whole family. measure is the column the figure is read on (the last of columns by default) and must be one of them. Give exactly one of max or target. round quantizes after scaling, and with target the rounding residual is added to the group's largest-magnitude row, so the total is exact rather than nearly right — three rows scaled to 100 give 33.33, 33.33, 33.34. An integer column is widened to float before scaling, since assignment would otherwise truncate and the group would miss its figure; round: 0 keeps whole numbers. A group whose measure sums to zero cannot be scaled into a non-zero figure and is left alone.

Totals sit between calculated features and derived tables, so aggregates reconcile to the adjusted detail, and before error injection, which stays last. Being pure data, they can also live in the YAML configuration under the reserved totals key.

Error Injection

Perfect data doesn't test anything. The errors argument injects deterministic, seeded corruption as the very last pipeline stage — after calculated features and derived tables, so identities and aggregates are computed on clean data and the errors land only in the delivered output. It is keyed by table name, and derived tables can be targeted too — corrupting an aggregate manufactures exactly the detail-vs-summary mismatch a reconciliation or data-quality engine must alert on:

synth.fit(
    metadata,
    errors={
        "daily_sales": {   # the detail table
            "columns": {
                "revenue": {
                    # representation corruption: 1200 -> "one thousand, two hundred"
                    "format_violation": {"rate": 0.03, "styles": ["words", "currency"]},
                    "missing": 0.02,
                },
            },
            "table": {
                "duplicate_rows": 0.01,
                "mapping_break": {"columns": ["sku", "department"], "rate": 0.02},
            },
        },
        "department_summary": {   # the derived aggregate -> reconciliation alerts fire
            "columns": {"total_revenue": {"out_of_range": {"rate": 0.05, "factor": 100}}},
        },
    },
)

Column operators: missing, format_violation (styles words, thousands_sep, currency, trailing_space for numerics; truncate, case_flip, whitespace for strings — a corrupted column is deliberately delivered as a string column, untouched numbers rendered as text next to the corrupted cells, which is itself part of what downstream parsers must survive; missing alone keeps numeric columns numeric) and out_of_range (absurd but still numeric). Table operators: duplicate_rows, mapping_break (a supposedly-strict mapping violated in a fraction of rows) and copy_column (one column takes another column's value in the same row). Everything is reproducible under random_state — the same run always corrupts the same cells, so tests can assert exactly which rows should be flagged. Like the other blocks, errors is pure data and can live in the YAML configuration.

Calculated Features (Business Rules)

Columns bound by exact rules — vat = 20% of net, totals, sign conventions — should be computed, not sampled. The Faker Synthesizer accepts the same single-table calculated-features format as the RegularSynthesizer (bare column names); features are excluded from sampling and applied afterwards, in list order, so later features may consume earlier outputs:

synth.fit(
    metadata,
    calculated_features=[
        {"calculated_features": "gross_amount",
         "function": lambda qty, price: (qty * price).round(2),
         "calculated_from": ["quantity", "unit_price"]},
        {"calculated_features": "vat_value",           # chained on gross_amount
         "function": lambda gross: (gross * 0.20).round(2),
         "calculated_from": ["gross_amount"]},
    ],
)

Calculated features may consume scaffold columns but never overwrite them. Cross-table reference_keys are supported in schema mode only (see below). Being Python callables, calculated features cannot be declared in YAML — pass them in code. See the full guide: Calculated Features (Business Rules).

Multi-table Databases

Passing tables= instead of metadata switches the synthesizer to schema mode: one schema describes several tables, their primary keys and the foreign keys between them, and sample() returns a MultiDataset in which every foreign key resolves. Each table keeps its single-table configuration (columns, distributions, conditions, errors, ...) and adds its relationships:

tables:
  customers:
    primary_key: customer_id
    key_format: "CUS-{n:06d}"
    columns:
      customer_id: {datatype: string, vartype: string}
      segment:     {datatype: categorical, vartype: string, categories: {consumer: 80, business: 20}}
  orders:
    primary_key: order_id
    foreign_keys:
      - column: customer_id
        referenced_table: customers
        cardinality: {distribution: poisson, lam: 2.5, min: 1, max: 12}   # orders per customer
    columns:
      order_id:    {datatype: numerical, vartype: int}
      customer_id: {datatype: string, vartype: string}
      status:      {datatype: categorical, vartype: string, categories: {open: 30, shipped: 70}}
  shipments:
    primary_key: shipment_id
    foreign_keys:
      - column: order_id
        referenced_table: orders
        cardinality: {min: 1, max: 1}
        where: {status: shipped}                                          # only shipped orders
    columns:
      shipment_id: {datatype: numerical, vartype: int}
      order_id:    {datatype: numerical, vartype: int}
      carrier:     {datatype: categorical, vartype: string, categories: {UPS: 60, DHL: 40}}

synth = FakerSynthesizer()
synth.fit(tables="shop.yaml")                              # or the equivalent dict
db = synth.sample(sample_size={"customers": 1_000}, random_state=42)

Only root tables are sized; the others follow from their relationships. Cross-table calculated features (reference_keys), context_columns, 1:1 extensions, supplied tables, referential_integrity errors, error_labels_ and checking the result with ReferentialIntegrityValidator are covered in Multi-table Databases.

Related Materials