Time Series

FakerSynthesizer also supports the generation of time series: rows whose values depend on earlier rows of the same entity, such as a monthly balance per account or daily deposits per branch.

A time series is defined by the following elements:

Entity: what each series belongs to (an account, a branch, a branch × product), set by partition_by in the sequence.
Time: the order of the rows within each entity, set by order_by in the sequence.
Column behaviour: seasonality, trend and persistence on a numerical column.
Table behaviour: reacts_to, balances, regime and lifecycle, for dependencies across columns and rows.

On a column

deposit_rate:
  datatype: numerical
  vartype: float
  min: 0
  max: 1
  seasonality: {on: month_start, Dec: 1.5, Feb: 1.3, Mar: 1.2, Apr: 1.3}   # bonuses, tax refunds
  trend: 0.03            # +3% a year on the mean, compounding
  persistence: 0.6       # memory from one row to the next

Key	Meaning
`seasonality`	Multipliers on the column's mean by calendar position; positions not listed are `1.0`. `by`: `month` (default, keys `Jan`…`Dec` or `1`…`12`), `quarter` (`Q1`…`Q4`), `weekday` (`Mon`…`Sun`), `day_of_month` (`1`…`31`). `on`: the date column (default: the first `order_by` column). `normalize` (default `true`) rescales the multipliers to average 1 over the cycle, so the declared distribution stays the average one; `false` applies them literally. `mode: additive` shifts the mean instead (columns that can be negative, e.g. rate changes).
`trend`	Growth of the mean per year from the first date, compounding: `0.03` is +3%/year. `{per_year: 0.001, mode: additive}` adds instead.
`persistence`	AR(1) memory of the draw itself, between 0 and 0.99. Implemented with a Gaussian copula: each row is still drawn from exactly its declared distribution — only the dependence between consecutive rows changes.

In practice: with distributions: {deposit_rate: {distribution: beta, mean: 0.02, concentration: 60}}, the multipliers above average 1.108 over the year, so, normalised, December's mean is 2.7% and the unlisted months' 1.8%; the yearly average stays 2%. Every December row of every entity draws from that December distribution, so monthly deposit totals show the same peak every year. With normalize: false December is 3% and the unlisted months 2%.

Seasonality, trend and reactions move the mean and keep the shape: a beta stays a beta with the same concentration, a lognormal keeps its sigma. They act on the distribution the row would have had anyway — from distributions, from the matching conditions rule, or uniform over min/max — so segment effects and time effects combine.

On the table

`reacts_to` — a column responds to earlier values

reacts_to:
  withdrawal_rate:
    - {column: competitor_rate_gap, lag: 1, window: 2, effect: 20}

The signal is the sum (stat: mean for the average) of column over window rows, ending lag rows before the current one, within the entity. The mean becomes mean × exp(effect × signal): after two months in which competitors paid one point more (a gap of 0.01 each month, 0.02 in total), withdrawals run exp(0.4) ≈ 1.49× higher. Several entries multiply. column can be any numerical column of the table, or one generated through time (then lag ≥ 1).

`balances` — stocks that follow from their flows

balances:
  balance:
    start: opening_balance        # value before the entity's first row (column or number)
    growth: interest_rate         # credited interest, multiplicative per row
    inflow: deposit_rate          # rate x previous balance
    outflow: withdrawal_rate      # rate x previous balance, at most what is available
    flows_as: {inflow: deposits, outflow: withdrawals}   # emit the amounts too

Per row, with previous = start on the entity's first row:

inflow    = previous × inflow_rate
available = previous × (1 + growth) + inflow
outflow   = min(previous × outflow_rate, available)
balance   = available − outflow

The identity balance = previous × (1 + growth) + inflow − outflow holds exactly on every row, and the balance never goes negative, because an outflow takes at most what is available — nothing is clipped afterwards. Aggregates of the balance and its flows reconcile. growth, inflow and outflow are columns or numbers; at least one is required. Inputs that would make a non-negative balance meaningless are rejected, with the offending values: a negative or missing start, growth below −1, negative rates.

`regime` — a categorical state with memory

regime:
  column: regime                                   # default name
  avg_duration: {stable: 18, rate_competition: 3}  # rows

A Markov chain per entity: each state lasts its average duration, and the long-run share of each state is proportional to it (here stable 86%, rate competition 14%). transition: {stable: {stable: 0.95, rate_competition: 0.05}, ...} gives the matrix explicitly instead; initial sets the first row's shares. The regime is generated before everything else, so conditions can use it:

conditions:
  - column: competitor_rate_gap
    rules: [{when: {regime: rate_competition},
             distribution: {distribution: beta, mean: 0.01, concentration: 400}}]

`lifecycle` — contiguous spans

lifecycle: {start: random, avg_length: 36, min_length: 3}

Each entity covers one contiguous span of its rows — products launch and retire at a branch, accounts open and close — and rows outside it are dropped. start: first begins at the entity's first row; random anywhere that leaves room for min_length. Lengths average avg_length (spans reaching the end of the grid are cut there). Balances start from start, and reactions only see rows inside the span.

Worked example: a monthly deposits warehouse

Savings products held at bank branches: a date dimension with the rate environment, a branch and a product dimension, the products each branch offers, and a monthly fact table whose balances follow from interest, deposits and withdrawals.

# Time-series FakerSynthesizer configuration: a monthly deposits warehouse.
#
# Savings products held at bank branches. Rows depend on earlier rows of the same
# entity (a branch x product): balances follow from interest, deposits and
# withdrawals; rates are seasonal and persistent; withdrawals react to the
# competitor rate gap, which a rate-competition regime drives.
#
#   dim_month ──────────────┐
#   dim_branch ──< branch_products >── dim_product
#                       └──< fact_balances (branch x product x month) >── dim_month
#                                └── kpi_product_month (derived)

tables:
  dim_month:
    primary_key: month_key                 # 1..60, one per month
    scaffold:
      dimensions:
        - {column: month_start, date_range: {start: "2020-01-01", end: "2024-12-01", freq: MS}}
    sequence: {order_by: [month_start]}
    regime: {avg_duration: {stable: 18, rate_competition: 3}}
    columns:
      month_key: {datatype: numerical, vartype: int}
      interest_rate: {datatype: numerical, vartype: float, min: 0, max: 0.01, persistence: 0.9}
      competitor_rate_gap: {datatype: numerical, vartype: float, min: 0, max: 0.05, persistence: 0.5}
    distributions:
      interest_rate: {distribution: beta, mean: 0.002, concentration: 2000}
      competitor_rate_gap: {distribution: beta, mean: 0.002, concentration: 400}
    conditions:
      - column: competitor_rate_gap
        rules: [{when: {regime: rate_competition},
                 distribution: {distribution: beta, mean: 0.01, concentration: 400}}]

  dim_branch:
    primary_key: branch_id
    key_format: "BR{n:04d}"
    columns:
      branch_id: {datatype: string, vartype: string}
      location: {datatype: categorical, vartype: string, categories: {urban: 60, rural: 40}}

  dim_product:
    primary_key: product_id
    key_format: "SAV{n:02d}"
    columns:
      product_id: {datatype: string, vartype: string}
      popularity: {datatype: numerical, vartype: float, min: 0.01, max: 100}
    distributions:
      popularity: {distribution: lognormal, mean: 0, sigma: 1}

  branch_products:                         # which products each branch offers
    primary_key: [branch_id, product_id]
    foreign_keys:
      - {column: branch_id, referenced_table: dim_branch, cardinality: {min: 2, max: 6}}
      - {column: product_id, referenced_table: dim_product, weight_column: popularity}
    columns:
      branch_id: {datatype: string, vartype: string}
      product_id: {datatype: string, vartype: string}
      opening_balance: {datatype: numerical, vartype: float, min: 1e4, max: 5e8}
    distributions:
      opening_balance: {distribution: lognormal, mean: 15, sigma: 1}

  fact_balances:
    primary_key: [branch_id, product_id, month_key]
    scaffold:
      dimensions:
        - {column: _branch_product, from_table: branch_products,
           include: [branch_id, product_id, opening_balance]}
        - {column: month_key, from_table: dim_month,
           include: [month_start, interest_rate, competitor_rate_gap]}
    foreign_keys:
      - {column: branch_id, referenced_table: dim_branch, context_columns: [location]}
      - {column: product_id, referenced_table: dim_product}
      - {column: month_key, referenced_table: dim_month}
    sequence: {partition_by: [branch_id, product_id], order_by: [month_key]}
    columns:
      branch_id: {datatype: string, vartype: string}
      product_id: {datatype: string, vartype: string}
      month_key: {datatype: numerical, vartype: int}
      deposit_rate:
        {datatype: numerical, vartype: float, min: 0, max: 0.5, persistence: 0.6,
         seasonality: {on: month_start, Dec: 1.5, Feb: 1.3, Mar: 1.2, Apr: 1.3}}
      withdrawal_rate:
        {datatype: numerical, vartype: float, min: 0, max: 0.5, persistence: 0.6,
         seasonality: {on: month_start, Apr: 1.4, Nov: 1.1, Dec: 1.3}}   # tax payments, holidays
    distributions:
      deposit_rate: {distribution: beta, mean: 0.02, concentration: 60}
      withdrawal_rate: {distribution: beta, mean: 0.018, concentration: 80}
    conditions:                            # rural depositors move less
      - column: withdrawal_rate
        rules: [{when: {branch_id.location: rural},
                 distribution: {distribution: beta, mean: 0.012, concentration: 80}}]
    reacts_to:
      withdrawal_rate: [{column: competitor_rate_gap, lag: 1, window: 2, effect: 20}]
    balances:
      balance: {start: opening_balance, growth: interest_rate, inflow: deposit_rate,
                outflow: withdrawal_rate, flows_as: {inflow: deposits, outflow: withdrawals}}
    lifecycle: {start: random, avg_length: 36}
    derived_tables:
      - name: kpi_product_month
        type: aggregate
        group_by: [product_id, month_key]
        aggregations:
          balance: {column: balance, agg: sum}
          withdrawals: {column: withdrawals, agg: sum}

"""
Example: a time-series data warehouse with the FakerSynthesizer.

The schema (faker_timeseries.yaml) is a monthly deposits warehouse: date, branch
and product dimensions, a branch x product bridge, and a fact table whose balances
follow from interest, deposits and withdrawals — with seasonality, persistence, a
rate-competition regime, reactions to the competitor rate gap and product
lifecycles — plus a KPI table derived from the facts.

Run with: python faker_timeseries.py
"""
from pathlib import Path

from ydata.metadata import MultiMetadata
from ydata.report import ReferentialIntegrityValidator
from ydata.synthesizers import FakerSynthesizer

HERE = Path(__file__).parent
OUTPUT = Path("deposits_warehouse")

synth = FakerSynthesizer()
synth.fit(tables=HERE / "faker_timeseries.yaml")

# only root tables are sized; every other table follows from its relationships
db = synth.sample(sample_size={"dim_branch": 300, "dim_product": 12}, random_state=42)

# referential integrity of the generated warehouse (ydata-report)
integrity = ReferentialIntegrityValidator(synthetic_data=db, metadata=MultiMetadata(db)).validate()
summary = integrity["summary"]
print(f"PK violations: {summary['total_pk_violations']}, "
      f"FK violations: {summary['total_fk_violations']}")

# save every table as a local file — Parquet keeps the dtypes and scales to
# millions of rows; use .to_csv(...) instead for CSV
OUTPUT.mkdir(exist_ok=True)
for name in db.keys():
    table = db[name].to_pandas()
    table.to_csv(OUTPUT / f"{name}.csv", index=False)
    print(f"{name:18s} {len(table):>7,} rows -> {OUTPUT / f'{name}.csv'}")

In code, without a schema, the same goes to fit(..., sequence=..., timeseries={"columns": {...}, "reacts_to": ..., "balances": ..., "regime": ..., "lifecycle": ...}).

Notes

Which columns can vary through time

Time behaviour applies to sampled numerical columns, because seasonality, trend and reactions work by moving a column's mean. Keys, scaffold columns and unique columns define the structure of the table, so they stay fixed.

Condition outcomes for a time-varying column are distributions, so each segment has a mean that time effects can move. Additive mode shifts the mean rather than scaling it, so it uses distributions that can be shifted: normal, beta, uniform and triangular.

To include missing values, add them with error injection rather than missings. Errors are applied after the balances are computed, so balances and lags stay consistent and the missing cells appear only in the delivered data.

Related Materials

Time Series

On a column

On the table

reacts_to — a column responds to earlier values

balances — stocks that follow from their flows

regime — a categorical state with memory

lifecycle — contiguous spans