Skip to content

Structural Scaffolds

Column-wise generation decides what each value looks like. It cannot decide which rows exist. Ask for 1,000 rows of a store-inventory table and you get 1,000 plausible rows — with some stores missing on some days, some days missing entirely, and no way to say "beans are counted twice a day, everywhere, all week".

A scaffold settles that question before a single value is sampled. It is a declarative dimensional grid: the cross product of its dimensions becomes the table's seed rows, filtered by optional density rules. Scaffold columns are emitted verbatim — never sampled, never overwritten — and every other metadata column is generated per seed row.

The configuration block

scaffold={
    "dimensions": [...],   # one entry per axis of the grid — at least one
    "density": {...},      # optional: which grid rows survive
}

Dimensions

Each dimension declares a column and exactly one source:

Key Meaning
from_table (+ from_column) Values taken from a named DataFrame passed via reference_data. from_column defaults to that table's primary key.
date_range An inclusive calendar axis: {"start": ..., "end": ..., "freq": "D"} — any pandas frequency.
values A static list.
include Optional, from_table only. Extra columns of the source table carried into the seed rows by a deterministic join.

include is what keeps reference attributes honest. Carrying a store's region into every row means the region is joined, never invented, so store_id → region holds exactly — and those columns become available to conditions and calculated_features downstream.

Density

Density decides which grid rows survive, as a keep-probability per category of a source-table column:

Key Meaning
by {"table": ..., "column": ...} — the source table and the column whose values key the rates.
rates Keep-probability per category value, each between 0 and 1.
default_rate Applied to categories absent from rates. Defaults to 1.0.
multipliers Conditional adjustments: {"when": "weekend" or "weekday", "factor": ..., "except": [...]}. except lists category values the multiplier skips.

Worked example

A coffee-shop chain's daily stock counts: store × product × day × shift, with pastries counted every day, beans only on some, and busier weekends.

"""
Example: structural scaffolds — pinning WHICH rows exist.

Column-wise generation decides what each value looks like; it cannot guarantee
that every store is present on every day, or that quiet categories have quiet
days. A `scaffold` does: it is a declarative dimensional grid whose cross
product becomes the table's seed rows, before a single value is sampled.

This example builds the daily stock-count table of a coffee-shop chain:

  - a `from_table` dimension over the store master, with `include` carrying
    the store's region and format into every row (joined, never invented)
  - a `date_range` dimension over one week
  - a static `values` dimension for the two daily count shifts
  - `density` rules: pastries are counted every day, beans only on some, and
    weekends are busier — except for pastries, which are exempt

Only `units_on_hand` and `units_wasted` are sampled; every other column is
structural and is emitted verbatim.

Run with: python faker_scaffold.py
"""
import pandas as pd

from ydata.metadata import Metadata
from ydata.metadata.builder import MetadataConfigurationBuilder
from ydata.synthesizers import FakerSynthesizer

WEEK_START, WEEK_END = "2026-03-02", "2026-03-08"   # Monday .. Sunday
SHIFTS = ["morning", "evening"]

# ---------------------------------------------------------------------------
# Store master (in practice: read from your reference system)
# ---------------------------------------------------------------------------
stores = pd.DataFrame({
    "store_id": ["ST-01", "ST-02", "ST-03", "ST-04"],
    "region": ["north", "north", "south", "south"],
    "store_format": ["flagship", "kiosk", "flagship", "kiosk"],
})

products = pd.DataFrame({
    "product_id": ["P-BEANS", "P-PASTRY", "P-MILK"],
    "category": ["beans", "pastries", "dairy"],
})

# only the MEASURED columns are declared as metadata; the structural columns
# come from the scaffold
columns = {
    "units_on_hand": {"datatype": "numerical", "vartype": "int", "min": 0, "max": 400},
    "units_wasted": {"datatype": "numerical", "vartype": "int", "min": 0, "max": 25},
}
metadata = Metadata(configuration_builder=MetadataConfigurationBuilder(columns))

synth = FakerSynthesizer(locale="en")
synth.fit(
    metadata,
    scaffold={
        "dimensions": [
            # values from a reference table, with extra columns carried along
            {"column": "store_id", "from_table": "stores", "from_column": "store_id",
             "include": ["region", "store_format"]},
            {"column": "product_id", "from_table": "products",
             "from_column": "product_id"},
            # an inclusive calendar axis
            {"column": "count_date",
             "date_range": {"start": WEEK_START, "end": WEEK_END, "freq": "D"}},
            # a static axis
            {"column": "shift", "values": SHIFTS},
        ],
        # WHICH grid rows survive: a keep-probability per product category,
        # boosted at weekends (pastries exempt — they are counted daily anyway)
        "density": {
            "by": {"table": "products", "column": "category"},
            "rates": {"pastries": 1.0, "beans": 0.6, "dairy": 0.8},
            "default_rate": 1.0,
            "multipliers": [{"when": "weekend", "factor": 1.3, "except": ["pastries"]}],
        },
    },
    reference_data={"stores": stores, "products": products},
)

# sample_size is IGNORED when a scaffold is configured — the grid sets the
# row count. random_state makes both the grid and the sampled values reproducible.
counts = synth.sample(random_state=42).to_pandas()

# ---------------------------------------------------------------------------
# What the scaffold guarantees
# ---------------------------------------------------------------------------
full_grid = len(stores) * len(products) * 7 * len(SHIFTS)
print(f"Full grid would be {full_grid} rows; density kept {len(counts)}.")
assert len(counts) < full_grid, "density should drop some rows"

# structure: the grid columns are emitted verbatim, never sampled
assert set(counts["store_id"]) == set(stores["store_id"])
assert set(counts["shift"]) == set(SHIFTS)
dates = pd.to_datetime(counts["count_date"])
assert dates.min() == pd.Timestamp(WEEK_START) and dates.max() == pd.Timestamp(WEEK_END)

# `include` columns are JOINED from the master, so the mapping is exact
truth = dict(zip(stores["store_id"], stores["region"]))
assert all(truth[s] == r for s, r in zip(counts["store_id"], counts["region"]))

# density: pastries are counted on every store x day x shift, beans are not
per_category = counts.merge(products, on="product_id")
pastry_rows = (per_category["category"] == "pastries").sum()
assert pastry_rows == len(stores) * 7 * len(SHIFTS), "rate 1.0 keeps every pastry row"
beans_rows = (per_category["category"] == "beans").sum()
print(f"pastries kept {pastry_rows} rows (rate 1.0); beans kept {beans_rows} (rate 0.6)")
assert beans_rows < pastry_rows

# the weekend multiplier lifts the keep-rate for the non-exempt categories
is_weekend = pd.to_datetime(per_category["count_date"]).dt.weekday >= 5
beans = per_category[per_category["category"] == "beans"]
beans_weekend = (pd.to_datetime(beans["count_date"]).dt.weekday >= 5).mean()
print(f"share of kept bean counts falling on a weekend: {beans_weekend:.2f} "
      f"(2/7 = {2/7:.2f} without the multiplier)")

# reproducibility
again = synth.sample(random_state=42).to_pandas()
assert counts.equals(again), "same random_state must give the same table"

print(counts.head(8).to_string(index=False))
print("\nAll structure, coverage, join and density checks passed.")

Notes

sample_size is ignored

When a scaffold is configured, the grid defines the row count — sample(sample_size=...) has no effect. Pass random_state to make both the surviving grid and the sampled values reproducible.

Scaffold columns need no metadata entry

Dimension columns and include columns are structural: declare them in the scaffold, not in the MetadataConfigurationBuilder configuration. Only the columns you want generated belong in the metadata.

The same scaffold works on the LLM synthesizer

Scaffolds use a shared configuration model, so a structural definition moves between the Faker and LLM engines unchanged — deterministic structure in both, with only the semantic columns filled differently.

A scaffold is pure data, so it can also live in a YAML configuration file under the reserved top-level scaffold key and be loaded with load_faker_config. See Faker Synthesizer for that form.

Related Materials