Sequence and Per-Partition Features
Row order is meaningless in a plain tabular generator. That makes a whole class of columns impossible to express: a running balance, a meter reading, a delta against the previous row, a rolling average, any lag. All of them mean "the previous row of the same entity", and without an order there is no previous row.
sequence supplies one. The frame is stably sorted before calculated features run, the output keeps that order, and a calculated feature marked per_partition is applied within each partition, in order — so cumsum, shift, diff and rolling inside your callable mean exactly what they say.
The configuration block
| Key | Meaning |
|---|---|
order_by |
The ordering column, or a list of columns. Required. |
partition_by |
Optional. The entity column(s) whose rows form independent sequences. Omit it and the whole table is one sequence. |
The frame is sorted by partition_by + order_by, stably, so ties keep their generated order. Columns may be metadata columns or scaffold dimensions.
Then, on any calculated feature:
{
"calculated_features": "meter_reading",
"function": lambda initial, kwh: (initial + kwh.cumsum()).round(3),
"calculated_from": ["initial_index", "energy_kwh"],
"per_partition": True, # <- applied within each device, in order
}
The callable receives one Series per partition rather than one Series for the whole table. Partition boundaries are hard: diff() yields NaN on each entity's first row rather than a value borrowed from the previous entity, and a cumsum never leaks across entities.
Leave per_partition off for row-wise rules — they look at no neighbours, so partitioning would only cost time.
Worked example
Hourly readings from a fleet of energy meters: an exact per-device running total, a delta against the previous reading, a rolling mean, and a plain row-wise rule for contrast.
"""
Example: `sequence` and per-partition calculated features — order as a
first-class citizen.
Row order is meaningless in a plain tabular generator, so anything that depends
on "the previous row of the same entity" — a running total, a delta, a rolling
average, a lag — cannot be expressed. `sequence` fixes that:
sequence={"order_by": [...], "partition_by": [...]}
The frame is stably sorted by `partition_by + order_by` before calculated
features run, and the output keeps that order. A calculated feature marked
`"per_partition": True` is then applied WITHIN each partition, in order — so
`cumsum`, `shift`, `diff` and `rolling` inside your callable mean exactly what
they say.
This example generates hourly readings from a fleet of energy meters:
- `energy_kwh` is the consumption sampled for each hour
- `meter_reading` is the running total per device — the exact cumulative sum
- `delta_vs_previous` uses `diff()`, undefined on each device's first row
- `rolling_6h` is a per-device rolling mean
- `is_peak_hour` is a plain (non-partitioned) row-wise rule, for contrast
Run with: python faker_sequence_partitions.py
"""
import pandas as pd
from ydata.metadata import Metadata
from ydata.metadata.builder import MetadataConfigurationBuilder
from ydata.synthesizers import FakerSynthesizer
HOURS = list(range(24))
# meter master: the initial index reading is carried into the rows via `include`
meters = pd.DataFrame({
"device_id": ["MTR-A1", "MTR-A2", "MTR-B1", "MTR-B2", "MTR-B3"],
"site": ["plant-a", "plant-a", "plant-b", "plant-b", "plant-b"],
"initial_index": [102_450.0, 88_310.5, 5_120.0, 44_902.25, 17_003.75],
})
columns = {
"energy_kwh": {"datatype": "numerical", "vartype": "float",
"min": 0.1, "max": 300.0},
}
metadata = Metadata(configuration_builder=MetadataConfigurationBuilder(columns))
synth = FakerSynthesizer(locale="en")
synth.fit(
metadata,
scaffold={ # one reading per device per hour, over three days
"dimensions": [
{"column": "device_id", "from_table": "meters",
"from_column": "device_id", "include": ["site", "initial_index"]},
{"column": "reading_date",
"date_range": {"start": "2026-03-02", "end": "2026-03-04", "freq": "D"}},
{"column": "hour", "values": HOURS},
]
},
reference_data={"meters": meters},
# -----------------------------------------------------------------------
# Declare the ordering. `order_by` may be a single column or a list;
# `partition_by` is optional — omit it and the whole table is one sequence.
# -----------------------------------------------------------------------
sequence={"order_by": ["reading_date", "hour"], "partition_by": ["device_id"]},
conditions=[
{ # consumption peaks during the working day
"column": "energy_kwh",
"rules": [
{"when": {"hour": [8, 9, 10, 11, 12, 13, 14, 15, 16, 17]},
"distribution": {"distribution": "normal", "loc": 180.0,
"scale": 35.0, "min": 20.0}},
],
"default": {"distribution": {"distribution": "normal", "loc": 45.0,
"scale": 15.0, "min": 0.1}},
},
],
calculated_features=[
# per_partition: applied within each device, in sequence order
{"calculated_features": "meter_reading",
"function": lambda initial, kwh: (initial + kwh.cumsum()).round(3),
"calculated_from": ["initial_index", "energy_kwh"],
"per_partition": True},
{"calculated_features": "delta_vs_previous",
"function": lambda kwh: kwh.diff().round(3),
"calculated_from": ["energy_kwh"],
"per_partition": True},
{"calculated_features": "rolling_6h",
"function": lambda kwh: kwh.rolling(6, min_periods=1).mean().round(3),
"calculated_from": ["energy_kwh"],
"per_partition": True},
# a plain row-wise rule needs no partition — it looks at no neighbours
{"calculated_features": "is_peak_hour",
"function": lambda hour: hour.between(8, 17),
"calculated_from": ["hour"]},
],
)
readings = synth.sample(random_state=42).to_pandas()
# ---------------------------------------------------------------------------
# What ordering guarantees
# ---------------------------------------------------------------------------
# 1. the output is sorted by partition, then by the ordering columns
expected_order = readings.sort_values(["device_id", "reading_date", "hour"],
kind="stable").reset_index(drop=True)
assert readings.equals(expected_order), "output preserves the declared sequence"
# 2. the running total is exact per device — and never leaks across devices
for device, rows in readings.groupby("device_id", sort=False):
expected = (rows["initial_index"].iloc[0] + rows["energy_kwh"].cumsum()).round(3)
assert (rows["meter_reading"] == expected).all(), f"{device}: reading drifted"
assert rows["meter_reading"].is_monotonic_increasing
# 3. `diff` is undefined on the FIRST row of each partition — exactly one NaN
# per device, never a value borrowed from the previous device
first_rows = readings.groupby("device_id", sort=False).head(1)
assert first_rows["delta_vs_previous"].isna().all()
assert readings["delta_vs_previous"].isna().sum() == readings["device_id"].nunique()
# 4. the peak-hour rule and the conditioned distribution agree
peak_mean = readings[readings["is_peak_hour"]]["energy_kwh"].mean()
offpeak_mean = readings[~readings["is_peak_hour"]]["energy_kwh"].mean()
print(f"Mean kWh — peak hours {peak_mean:.1f} vs off-peak {offpeak_mean:.1f}")
assert peak_mean > offpeak_mean * 2
print()
print(readings.head(10).to_string(index=False))
print(f"\n{len(readings)} readings across {readings['device_id'].nunique()} meters")
print(readings.groupby("device_id")[["initial_index", "meter_reading"]]
.agg({"initial_index": "first", "meter_reading": "last"}).to_string())
print("\nAll ordering, running-total and partition-boundary checks passed.")
Notes
per_partition requires a sequence
Without a declared ordering there is no meaningful "within each partition, in order", so the combination is rejected at fit() time.
Recurrences, not just running totals
Anything expressible over an ordered Series works inside the callable: shift for lags, rolling/ewm for smoothing, cumsum/cumprod for accumulation, scipy.signal.lfilter for AR-style recurrences. The synthesizer supplies the order; the semantics stay in your code.
Sampling does not see the past
A per-partition feature computes columns from generated values. It cannot feed back into sampling — a distribution parameter cannot depend on a previously generated row. Reach for a simulation framework when you need that.
sequence is also what makes the order-aware aggregations first and last available to derived tables.
Related Materials
