Per-Group Totals
Some figures do not belong to a row. A forecourt takes 2,400 litres of fuel a day; a supplier issues GBP 500 of credit a month; a till cannot hold more than its float. The number constrains a group of rows, and no per-column distribution can express it — a distribution shapes one value at a time and has no idea what the other rows in the day drew.
The usual workaround is to generate, measure, and rescale in application code. totals makes that a configuration block: the group is scaled by a single factor, so the mix and the spread the model drew survive and only the magnitude moves.
The configuration block
totals=[
# a ceiling: groups already under it are untouched
{"columns": ["net_value", "vat_value", "gross_value"],
"measure": "gross_value",
"group_by": ["billing_date"],
"max": 5000.00,
"round": 2},
# an exact figure: every group is scaled to it, up or down
{"columns": "credit_amount",
"group_by": ["supplier", "month"],
"target": 500.00,
"round": 2},
]
| Key | Meaning |
|---|---|
columns |
The column, or columns, to scale. Scaled together by one factor. |
measure |
The column the figure is read on. Must be one of columns; defaults to the last. |
group_by |
Grouping columns. Omit for a single group covering the whole table. |
max |
A ceiling. A group already under it is left alone. |
target |
An exact figure. Every group is scaled to it, up or down. |
round |
Decimal places to quantize to after scaling. |
Give exactly one of max or target.
Why the columns scale together
Totals run after calculated features, so a feature is not recomputed afterwards. Scaling gross_value alone would leave gross == net + vat false on every touched row. Listing the whole family instead applies one factor to all of them, and the identity survives the rescale:
The same rule applies to a quantity and the value derived from it. If a column is not listed, it keeps its original value — which is occasionally what you want, and usually a bug.
Exactness
round quantizes the scaled values, and rounding does not distribute evenly: three rows scaled to a target of 100 round to 33.33 each and sum to 99.99. With target, the residual is added to the group's largest-magnitude row, so the group's total is the figure that was asked for — 33.33, 33.33, 33.34. With max there is nothing to correct, since a ceiling is a bound rather than an equality.
Worked example
Daily wholesale deliveries under a ceiling, and a supplier's monthly credit notes set to an exact figure — with assertions that the totals hold, that the proportions the model drew are unchanged, and that gross == net + vat still reconciles row by row.
"""
Example: per-group totals — holding a figure that belongs to a GROUP.
A distribution shapes one value at a time. It cannot say "this site takes 2,400
litres a day" or "this supplier issues GBP 500 of credit a month", because those
numbers constrain a group of rows, not a row. `derived_tables` only measures the
total once it exists; `totals` imposes it.
Each group is scaled by a SINGLE factor, so the mix and the spread the model drew
survive and only the magnitude moves:
columns the column(s) to scale — scaled together, by one factor
measure the column the figure is read on (defaults to the last of `columns`)
group_by the grouping; omit for one group covering the whole table
max a ceiling — a group already under it is left alone
target an exact figure — every group is scaled to it, up or down
round decimal places, applied after scaling
Listing co-dependent columns TOGETHER is what keeps their identities true:
totals run after calculated features, so a feature is not recomputed afterwards,
and scaling `gross` alone would leave `gross == net + vat` false on every row it
touched.
The table here is a wholesaler's delivery lines: daily deliveries held under a
depot's capacity, and monthly credit notes set to the exact figure a scenario
asks for.
Run with: python faker_totals.py
"""
import pandas as pd
from ydata.metadata import Metadata
from ydata.metadata.builder import MetadataConfigurationBuilder
from ydata.synthesizers import FakerSynthesizer
sites = pd.DataFrame({
"site_code": ["S-01", "S-02", "S-03", "S-04"],
"region": ["north", "north", "south", "south"],
})
columns = {
"units": {"datatype": "numerical", "vartype": "int", "min": 1, "max": 90},
"unit_price": {"datatype": "numerical", "vartype": "float",
"min": 0.80, "max": 14.50},
}
metadata = Metadata(configuration_builder=MetadataConfigurationBuilder(columns))
BASE_CONFIG = dict(
scaffold={
"dimensions": [
{"column": "site_code", "from_table": "sites",
"from_column": "site_code", "include": ["region"]},
{"column": "delivery_date",
"date_range": {"start": "2026-04-01", "end": "2026-04-10", "freq": "D"}},
]
},
reference_data={"sites": sites},
calculated_features=[
{"calculated_features": ["net_value", "vat_value", "gross_value"],
"function": lambda units, price: (
(net := (units * price).round(2)),
(vat := (net * 0.20).round(2)),
(net + vat).round(2),
),
"calculated_from": ["units", "unit_price"]},
],
)
DAILY_CAP = 1_500.00
# ---------------------------------------------------------------------------
# 1. Without totals — the daily figure is whatever the draws happen to sum to
# ---------------------------------------------------------------------------
plain_synth = FakerSynthesizer(locale="en")
plain_synth.fit(metadata, **BASE_CONFIG)
plain = plain_synth.sample(random_state=13).to_pandas()
plain_daily = plain.groupby("delivery_date")["gross_value"].sum().round(2)
print("Daily gross without totals:")
print(plain_daily.to_string())
print(f"days above the {DAILY_CAP:,.2f} depot capacity: "
f"{int((plain_daily > DAILY_CAP).sum())} of {len(plain_daily)}")
# ---------------------------------------------------------------------------
# 2. A ceiling on the day, and an exact monthly figure per region
# ---------------------------------------------------------------------------
synth = FakerSynthesizer(locale="en")
synth.fit(
metadata,
**BASE_CONFIG,
totals=[
# the whole monetary family scales together, so gross == net + vat holds
{"columns": ["net_value", "vat_value", "gross_value"],
"measure": "gross_value",
"group_by": ["delivery_date"],
"max": DAILY_CAP,
"round": 2},
],
)
df = synth.sample(random_state=13).to_pandas()
daily = df.groupby("delivery_date")["gross_value"].sum().round(2)
print("\nDaily gross with the ceiling applied:")
print(daily.to_string())
assert (daily <= DAILY_CAP).all(), "a ceiling is a hard bound, rounding included"
# a day that was already under the ceiling is untouched
for day, before in plain_daily.items():
if before <= DAILY_CAP:
assert daily[day] == before, "a group under the ceiling must not move"
# the shape the model drew survives: only the magnitude changed. The comparison
# is made to the precision `round` asked for — quantizing to the cent perturbs
# each line's share slightly, which is the point of rounding
busiest = plain_daily.idxmax()
before = plain.loc[plain.delivery_date == busiest, "gross_value"].to_numpy()
after = df.loc[df.delivery_date == busiest, "gross_value"].to_numpy()
drift = abs(after / after.sum() - before / before.sum()).max()
print(f"\nlargest change in any line's share of {busiest}: {drift:.2e}")
assert drift < 1e-3, "proportions within the group must be preserved"
# co-dependent columns kept their identity through the rescale
assert (abs(df["net_value"] + df["vat_value"] - df["gross_value"]) <= 0.011).all(), \
"gross == net + vat must survive scaling"
# ---------------------------------------------------------------------------
# 3. An exact figure, and why the rounding residual matters
# ---------------------------------------------------------------------------
CREDIT_PER_REGION = 500.00
credit_synth = FakerSynthesizer(locale="en")
credit_synth.fit(
metadata,
**BASE_CONFIG,
totals=[{"columns": "gross_value", "group_by": ["region"],
"target": CREDIT_PER_REGION, "round": 2}],
)
credits = credit_synth.sample(random_state=13).to_pandas()
print(f"\nGross per region, targeted at {CREDIT_PER_REGION:,.2f}:")
for region, group in credits.groupby("region"):
total = round(float(group["gross_value"].sum()), 2)
print(f" {region}: {total:,.2f} across {len(group)} lines")
assert total == CREDIT_PER_REGION, \
"the residual left by rounding is added to the largest row, so the " \
"group total is the figure asked for — not a cent under it"
print("\nCeiling respected, proportions preserved, identities intact, "
"targets exact to the cent.")
Notes
Where it sits in the pipeline
scaffold → column sampling → conditioned columns → sequence sort → calculated features → totals → derived tables → errors.
After calculated features, so the figure can be read on a derived monetary column. Before derived tables, so aggregates reconcile to the adjusted detail. Before error injection, which stays last.
An integer column is widened to float
Scaling 10, 20, 30 to a target of 100 gives 16.67, 33.33, 50.00 — values an integer column cannot hold, and assignment would truncate them silently, leaving the group short of its figure. Integer columns are therefore widened before scaling. Ask for round: 0 when you want whole numbers.
A group that sums to zero is left alone
There is no factor that scales zero into a non-zero figure. Such a group passes through untouched rather than failing the run.
Pure data, so it lives in YAML too
totals is a reserved top-level key in a configuration file, alongside scaffold, distributions, conditions, sequence, derived_tables and errors.
Related Materials
