Sequence and Per-Partition Features
Row order is meaningless in a plain tabular generator. That makes a whole class of columns impossible to express: a running balance, a meter reading, a delta against the previous row, a rolling average, any lag. All of them mean "the previous row of the same entity", and without an order there is no previous row.
sequence supplies one. The frame is stably sorted before calculated features run, the output keeps that order, and a calculated feature marked per_partition is applied within each partition, in order — so cumsum, shift, diff and rolling inside your callable mean exactly what they say.
The configuration block
| Key | Meaning |
|---|---|
order_by |
The ordering column, or a list of columns. Required. |
partition_by |
Optional. The entity column(s) whose rows form independent sequences. Omit it and the whole table is one sequence. |
The frame is sorted by partition_by + order_by, stably, so ties keep their generated order. Columns may be metadata columns or scaffold dimensions.
Then, on any calculated feature:
{
"calculated_features": "meter_reading",
"function": lambda initial, kwh: (initial + kwh.cumsum()).round(3),
"calculated_from": ["initial_index", "energy_kwh"],
"per_partition": True, # <- applied within each device, in order
}
The callable receives one Series per partition rather than one Series for the whole table. Partition boundaries are hard: diff() yields NaN on each entity's first row rather than a value borrowed from the previous entity, and a cumsum never leaks across entities.
Leave per_partition off for row-wise rules — they look at no neighbours, so partitioning would only cost time.
Pre-defined windowed rules
The common windows are pre-defined business rules: CUMULATIVE_SUM, RUNNING_BALANCE, DECLINING_BALANCE, PREVIOUS, CHANGE and PCT_CHANGE. They need only the sequence — no per_partition — and are computed across all partitions at once rather than with one Python call per partition, which matters with thousands of accounts:
{"calculated_features": "meter_reading", # initial_index + cumulative energy_kwh
"function": BusinessRule.CUMULATIVE_SUM,
"calculated_from": ["energy_kwh", "initial_index"]}
per_partition with your own function remains the way to express anything else — a rolling mean, an exponential smoothing, a recurrence.
Worked example
Hourly readings from a fleet of energy meters: an exact per-device running total and a delta against the previous reading (both pre-defined rules), a rolling mean (your own per-partition function), and a plain row-wise rule for contrast.
"""
Example: `sequence` and per-partition calculated features — order as a
first-class citizen.
Row order is meaningless in a plain tabular generator, so anything that depends
on "the previous row of the same entity" — a running total, a delta, a rolling
average, a lag — cannot be expressed. `sequence` fixes that:
sequence={"order_by": [...], "partition_by": [...]}
The frame is stably sorted by `partition_by + order_by` before calculated
features run, and the output keeps that order. A calculated feature marked
`"per_partition": True` is then applied WITHIN each partition, in order — so
`cumsum`, `shift`, `diff` and `rolling` inside your callable mean exactly what
they say.
The common windows are pre-defined business rules (`BusinessRule.CUMULATIVE_SUM`,
`RUNNING_BALANCE`, `CHANGE`, ...). They need only the `sequence` — no
`per_partition` — and run across all partitions at once instead of one call
per partition.
This example generates hourly readings from a fleet of energy meters:
- `energy_kwh` is the consumption sampled for each hour
- `meter_reading` is the running total per device — `BusinessRule.CUMULATIVE_SUM`
- `delta_vs_previous` is `BusinessRule.CHANGE`, undefined on each device's first row
- `rolling_6h` is a per-device rolling mean — your own `per_partition` function
- `is_peak_hour` is a plain (non-partitioned) row-wise rule, for contrast
Run with: python faker_sequence_partitions.py
"""
import numpy as np
import pandas as pd
from ydata.calculated import BusinessRule
from ydata.metadata import Metadata
from ydata.metadata.builder import MetadataConfigurationBuilder
from ydata.synthesizers import FakerSynthesizer
HOURS = list(range(24))
# meter master: the initial index reading is carried into the rows via `include`
meters = pd.DataFrame({
"device_id": ["MTR-A1", "MTR-A2", "MTR-B1", "MTR-B2", "MTR-B3"],
"site": ["plant-a", "plant-a", "plant-b", "plant-b", "plant-b"],
"initial_index": [102_450.0, 88_310.5, 5_120.0, 44_902.25, 17_003.75],
})
columns = {
"energy_kwh": {"datatype": "numerical", "vartype": "float",
"min": 0.1, "max": 300.0},
}
metadata = Metadata(configuration_builder=MetadataConfigurationBuilder(columns))
synth = FakerSynthesizer(locale="en")
synth.fit(
metadata,
scaffold={ # one reading per device per hour, over three days
"dimensions": [
{"column": "device_id", "from_table": "meters",
"from_column": "device_id", "include": ["site", "initial_index"]},
{"column": "reading_date",
"date_range": {"start": "2026-03-02", "end": "2026-03-04", "freq": "D"}},
{"column": "hour", "values": HOURS},
]
},
reference_data={"meters": meters},
# -----------------------------------------------------------------------
# Declare the ordering. `order_by` may be a single column or a list;
# `partition_by` is optional — omit it and the whole table is one sequence.
# -----------------------------------------------------------------------
sequence={"order_by": ["reading_date", "hour"], "partition_by": ["device_id"]},
conditions=[
{ # consumption peaks during the working day
"column": "energy_kwh",
"rules": [
{"when": {"hour": [8, 9, 10, 11, 12, 13, 14, 15, 16, 17]},
"distribution": {"distribution": "normal", "loc": 180.0,
"scale": 35.0, "min": 20.0}},
],
"default": {"distribution": {"distribution": "normal", "loc": 45.0,
"scale": 15.0, "min": 0.1}},
},
],
calculated_features=[
# pre-defined windowed rules: per device, in sequence order.
# meter_reading = initial_index + cumulative energy_kwh
{"calculated_features": "meter_reading",
"function": BusinessRule.CUMULATIVE_SUM,
"calculated_from": ["energy_kwh", "initial_index"]},
{"calculated_features": "delta_vs_previous",
"function": BusinessRule.CHANGE,
"calculated_from": ["energy_kwh"]},
# your own function, applied within each device, in sequence order
{"calculated_features": "rolling_6h",
"function": lambda kwh: kwh.rolling(6, min_periods=1).mean().round(3),
"calculated_from": ["energy_kwh"],
"per_partition": True},
# a plain row-wise rule needs no partition — it looks at no neighbours
{"calculated_features": "is_peak_hour",
"function": lambda hour: hour.between(8, 17),
"calculated_from": ["hour"]},
],
)
readings = synth.sample(random_state=42).to_pandas()
# ---------------------------------------------------------------------------
# What ordering guarantees
# ---------------------------------------------------------------------------
# 1. the output is sorted by partition, then by the ordering columns
expected_order = readings.sort_values(["device_id", "reading_date", "hour"],
kind="stable").reset_index(drop=True)
assert readings.equals(expected_order), "output preserves the declared sequence"
# 2. the running total is exact per device — and never leaks across devices
for device, rows in readings.groupby("device_id", sort=False):
expected = rows["initial_index"].iloc[0] + rows["energy_kwh"].cumsum()
assert np.allclose(rows["meter_reading"], expected), f"{device}: reading drifted"
assert rows["meter_reading"].is_monotonic_increasing
# 3. `diff` is undefined on the FIRST row of each partition — exactly one NaN
# per device, never a value borrowed from the previous device
first_rows = readings.groupby("device_id", sort=False).head(1)
assert first_rows["delta_vs_previous"].isna().all()
assert readings["delta_vs_previous"].isna().sum() == readings["device_id"].nunique()
# 4. the peak-hour rule and the conditioned distribution agree
peak_mean = readings[readings["is_peak_hour"]]["energy_kwh"].mean()
offpeak_mean = readings[~readings["is_peak_hour"]]["energy_kwh"].mean()
print(f"Mean kWh — peak hours {peak_mean:.1f} vs off-peak {offpeak_mean:.1f}")
assert peak_mean > offpeak_mean * 2
print()
print(readings.head(10).to_string(index=False))
print(f"\n{len(readings)} readings across {readings['device_id'].nunique()} meters")
print(readings.groupby("device_id")[["initial_index", "meter_reading"]]
.agg({"initial_index": "first", "meter_reading": "last"}).to_string())
print("\nAll ordering, running-total and partition-boundary checks passed.")
Notes
per_partition requires a sequence
Without a declared ordering there is no meaningful "within each partition, in order", so the combination is rejected at fit() time.
Recurrences, not just running totals
Anything expressible over an ordered Series works inside the callable: shift for lags, rolling/ewm for smoothing, cumsum/cumprod for accumulation, scipy.signal.lfilter for AR-style recurrences. The synthesizer supplies the order; the semantics stay in your code.
When sampling should see the past
A per-partition feature computes columns from generated values; it does not feed back into sampling. For rows whose distribution depends on earlier rows — seasonality, persistence, reactions to past values, balances that follow from their flows, regimes, lifecycles — use Time Series.
sequence is also what makes the order-aware aggregations first and last available to derived tables.
Related Materials
