Calculated Features

Some columns are not observations, they are consequences. A line total is quantity × unit price. VAT is a statutory rate applied to a base. An approval flag is a comparison. Sampling those columns independently produces a table that is realistic column by column and wrong row by row.

calculated_features computes them instead, with your own Python function. The columns are excluded from generation and applied after everything else has been sampled.

The configuration block

calculated_features=[
    {
        "calculated_features": "vat_value",                      # output column(s)
        "function": lambda net: (net * 0.23).round(2),           # your callable
        "calculated_from": ["net_value"],                        # input column(s)
    },
]

Key	Meaning
calculated_features	The output column name, or a list of names for a multi-output feature.
function	A callable receiving one pandas.Series per calculated_from entry, positionally — parameter names are yours to choose. Returns a Series or 1-D array; for multiple outputs, a tuple/list of Series, a 2-D array, or a DataFrame with one column per output.
calculated_from	Input columns, in the order the function expects them.
per_partition	Optional. Applies the feature within each partition of a declared sequence — see Sequence and Per-Partition Features.

Features are applied in list order, so a later feature may consume an earlier one's output. That is how a chain such as line_net → discount_value → taxable_base → vat_value → line_total is expressed: one small, readable rule per step, rather than one opaque function.

Worked example

An invoice-line table demonstrating a two-input feature, a five-step chain, a rate lookup driven by another column, a multi-output feature, and a non-numeric flag.

"""
Example: calculated features — the columns that must NOT be sampled.

Some columns are not observations, they are consequences: a line total is
quantity x unit price, VAT is a rate applied to a base, a margin flag is a
comparison. Sampling them independently produces a table that is realistic
column by column and wrong row by row — invoices whose lines do not add up.

`calculated_features` computes those columns with your own Python function,
after everything else has been sampled. They are excluded from generation and
applied IN LIST ORDER, so a later feature can consume an earlier one's output.

This example builds an invoice-line table and demonstrates:

  - a single-output feature from two inputs (`line_net`)
  - a chain: `discount_value` -> `taxable_base` -> `vat_value` -> `line_total`
  - a lookup driven by another column (`vat_rate` per tax region)
  - a MULTI-OUTPUT feature returning two columns in one pass
  - a non-numeric output (a boolean quality flag)

Run with: python faker_calculated_features.py
"""
import numpy as np
import pandas as pd

from ydata.metadata import Metadata
from ydata.metadata.builder import MetadataConfigurationBuilder
from ydata.synthesizers import FakerSynthesizer

# statutory VAT rates — a business fact, not something to be sampled
VAT_RATES = {"PT": 0.23, "ES": 0.21, "DE": 0.19, "LU": 0.17}

columns = {
    "invoice_id": {"datatype": "string", "vartype": "string",
                   "regex": "INV-2026-[0-9]{5}"},
    "tax_region": {"datatype": "categorical", "vartype": "string",
                   "categories": {"PT": 40, "ES": 30, "DE": 20, "LU": 10}},
    "quantity": {"datatype": "numerical", "vartype": "int", "min": 1, "max": 40},
    "unit_price": {"datatype": "numerical", "vartype": "float",
                   "min": 2.5, "max": 900.0},
    "discount_pct": {"datatype": "numerical", "vartype": "float",
                     "min": 0.0, "max": 0.30},
}
metadata = Metadata(configuration_builder=MetadataConfigurationBuilder(columns))

synth = FakerSynthesizer(locale="en")
synth.fit(
    metadata,
    distributions={
        "unit_price": {"distribution": "lognormal", "mean": 3.4, "sigma": 0.9},
        "discount_pct": {"distribution": "beta", "a": 1.5, "b": 8.0, "min": 0.0, "max": 0.30},
    },
    calculated_features=[
        # 1. two inputs, one output. The function receives one pandas Series
        #    per `calculated_from` entry, POSITIONALLY — argument names are
        #    yours to choose.
        {"calculated_features": "line_net",
         "function": lambda qty, price: (qty * price).round(2),
         "calculated_from": ["quantity", "unit_price"]},

        # 2. chained on the output of (1)
        {"calculated_features": "discount_value",
         "function": lambda net, pct: (net * pct).round(2),
         "calculated_from": ["line_net", "discount_pct"]},

        {"calculated_features": "taxable_base",
         "function": lambda net, disc: (net - disc).round(2),
         "calculated_from": ["line_net", "discount_value"]},

        # 3. a lookup: the rate is decided by another column, never invented
        {"calculated_features": "vat_rate",
         "function": lambda region: region.map(VAT_RATES),
         "calculated_from": ["tax_region"]},

        {"calculated_features": "vat_value",
         "function": lambda base, rate: (base * rate).round(2),
         "calculated_from": ["taxable_base", "vat_rate"]},

        {"calculated_features": "line_total",
         "function": lambda base, vat: (base + vat).round(2),
         "calculated_from": ["taxable_base", "vat_value"]},

        # 4. MULTI-OUTPUT: one function, two columns, returned as a tuple of
        #    Series in the same order as the declared outputs.
        {"calculated_features": ["net_per_unit", "total_per_unit"],
         "function": lambda base, total, qty: ((base / qty).round(4),
                                               (total / qty).round(4)),
         "calculated_from": ["taxable_base", "line_total", "quantity"]},

        # 5. outputs need not be numeric — a rule can produce a flag
        {"calculated_features": "requires_approval",
         "function": lambda total, pct: (total > 5_000) | (pct > 0.25),
         "calculated_from": ["line_total", "discount_pct"]},
    ],
)

lines = synth.sample(2_000, random_state=42).to_pandas()

# ---------------------------------------------------------------------------
# Every identity holds on every row — that is the whole point
# ---------------------------------------------------------------------------
assert np.allclose(lines["line_net"], (lines["quantity"] * lines["unit_price"]).round(2))
assert np.allclose(lines["taxable_base"], lines["line_net"] - lines["discount_value"])
assert np.allclose(lines["vat_value"], (lines["taxable_base"] * lines["vat_rate"]).round(2))
assert np.allclose(lines["line_total"], lines["taxable_base"] + lines["vat_value"])
assert np.allclose(lines["total_per_unit"], (lines["line_total"] / lines["quantity"]).round(4))

# the lookup is exact: one rate per region, no drift
observed = lines.groupby("tax_region")["vat_rate"].unique()
assert all(len(v) == 1 for v in observed), "each region must carry exactly one rate"
assert {r: v[0] for r, v in observed.items()} == {
    r: VAT_RATES[r] for r in observed.index}

# the boolean rule agrees with its own definition
expected_flag = (lines["line_total"] > 5_000) | (lines["discount_pct"] > 0.25)
assert (lines["requires_approval"] == expected_flag).all()

print(lines.head(6).to_string(index=False))
print(f"\n{len(lines)} invoice lines | "
      f"{lines['requires_approval'].sum()} flagged for approval")
print("VAT rate per region:")
print(lines.groupby("tax_region")["vat_rate"].first().to_string())
print("\nAll arithmetic, lookup and flag identities hold on every row.")

Notes

Callables cannot live in YAML

Every other Faker configuration block is pure data and can be declared in a configuration file. Calculated features are Python functions — pass them to fit(calculated_features=...) in code.

Scaffold columns are read-only

A calculated feature may consume scaffold dimensions and include columns, but never overwrite them. Cross-table reference_keys are likewise not supported by this single-table synthesizer.

When is a column a calculated feature?

Whenever it has a single correct answer given the other columns. If a validation script could check the column, a calculated feature should produce it. Columns carrying new information — names, free text, amounts, categories — belong to generation.

Related Materials

• Sequence and Per-Partition Features
• Derived Tables and Aggregations
• Calculated Features (Business Rules)