Calculated Features

Some columns are not observations, they are consequences. A line total is quantity × unit price. VAT is a statutory rate applied to a base. An approval flag is a comparison. Sampling those columns independently produces a table that is realistic column by column and wrong row by row.

calculated_features computes them instead — with a pre-defined business rule when one fits, or with your own Python function. The columns are excluded from generation and applied after everything else has been sampled.

The configuration block

calculated_features=[
    {
        "calculated_features": "vat_value",                      # output column(s)
        "function": lambda net: (net * 0.23).round(2),           # your callable
        "calculated_from": ["net_value"],                        # input column(s)
    },
]

Key	Meaning
calculated_features	The output column name, or a list of names for a multi-output feature.
function	A pre-defined BusinessRule (see below), or a callable receiving one pandas.Series per calculated_from entry, positionally — parameter names are yours to choose. Returns a Series or 1-D array; for multiple outputs, a tuple/list of Series, a 2-D array, or a DataFrame with one column per output.
calculated_from	Input columns, in the order the function expects them.
per_partition	Optional. Applies the feature within each partition of a declared sequence — see Sequence and Per-Partition Features.

Features are applied in list order, so a later feature may consume an earlier one's output. That is how a chain such as line_net → discount_value → taxable_base → vat_value → line_total is expressed: one small, readable rule per step, rather than one opaque function.

Pre-defined business rules

The common arithmetic is already written. Pass a BusinessRule as function, with the input columns in the order of its signature:

from ydata.calculated import BusinessRule

{"calculated_features": "net_amount",                 # quantity × unit_price − discount
 "function": BusinessRule.LINE_AMOUNT,
 "calculated_from": ["quantity", "unit_price", "discount"]}

Constants are fixed by calling the rule, e.g. BusinessRule.GROSS_UP(rate=0.23). The windowed rules (RUNNING_BALANCE, CUMULATIVE_SUM, …) need only a sequence — see Sequence and Per-Partition Features. The full catalog is in Calculated Features (Business Rules).

Worked example

An invoice-line table demonstrating pre-defined business rules, a five-step chain, a rate lookup driven by another column, a multi-output feature, and a non-numeric flag.

"""
Example: calculated features — the columns that must NOT be sampled.

Some columns are not observations, they are consequences: a line total is
quantity x unit price, VAT is a rate applied to a base, a margin flag is a
comparison. Sampling them independently produces a table that is realistic
column by column and wrong row by row — invoices whose lines do not add up.

`calculated_features` computes those columns after everything else has been
sampled — with a pre-defined business rule (`BusinessRule`) when one fits, or
with your own Python function when it does not. They are excluded from generation and
applied IN LIST ORDER, so a later feature can consume an earlier one's output.

This example builds an invoice-line table and demonstrates:

  - pre-defined business rules for the arithmetic (`BusinessRule.PRODUCT`,
    `DIFFERENCE`, `SUM`)
  - a chain: `line_net` -> `discount_value` -> `taxable_base` -> `vat_value`
    -> `line_total`
  - a lookup driven by another column (`vat_rate` per tax region) — your own
    function, as no pre-defined rule covers a lookup
  - a MULTI-OUTPUT feature returning two columns in one pass
  - a non-numeric output (a boolean quality flag)

Run with: python faker_calculated_features.py
"""
import numpy as np
import pandas as pd

from ydata.calculated import BusinessRule
from ydata.metadata import Metadata
from ydata.metadata.builder import MetadataConfigurationBuilder
from ydata.synthesizers import FakerSynthesizer

# statutory VAT rates — a business fact, not something to be sampled
VAT_RATES = {"PT": 0.23, "ES": 0.21, "DE": 0.19, "LU": 0.17}

columns = {
    "invoice_id": {"datatype": "string", "vartype": "string",
                   "regex": "INV-2026-[0-9]{5}"},
    "tax_region": {"datatype": "categorical", "vartype": "string",
                   "categories": {"PT": 40, "ES": 30, "DE": 20, "LU": 10}},
    "quantity": {"datatype": "numerical", "vartype": "int", "min": 1, "max": 40},
    "unit_price": {"datatype": "numerical", "vartype": "float",
                   "min": 2.5, "max": 900.0},
    "discount_pct": {"datatype": "numerical", "vartype": "float",
                     "min": 0.0, "max": 0.30},
}
metadata = Metadata(configuration_builder=MetadataConfigurationBuilder(columns))

synth = FakerSynthesizer(locale="en")
synth.fit(
    metadata,
    distributions={
        "unit_price": {"distribution": "lognormal", "mean": 3.4, "sigma": 0.9},
        "discount_pct": {"distribution": "beta", "a": 1.5, "b": 8.0, "min": 0.0, "max": 0.30},
    },
    calculated_features=[
        # 1. a pre-defined business rule: line_net = quantity × unit_price.
        #    The rule receives one pandas Series per `calculated_from` entry,
        #    POSITIONALLY, in the order of its signature.
        {"calculated_features": "line_net",
         "function": BusinessRule.PRODUCT,
         "calculated_from": ["quantity", "unit_price"]},

        # 2. chained on the output of (1)
        {"calculated_features": "discount_value",
         "function": BusinessRule.PRODUCT,
         "calculated_from": ["line_net", "discount_pct"]},

        {"calculated_features": "taxable_base",
         "function": BusinessRule.DIFFERENCE,
         "calculated_from": ["line_net", "discount_value"]},

        # 3. a lookup: the rate is decided by another column, never invented.
        #    No business rule covers this, so it is your own function.
        {"calculated_features": "vat_rate",
         "function": lambda region: region.map(VAT_RATES),
         "calculated_from": ["tax_region"]},

        {"calculated_features": "vat_value",
         "function": BusinessRule.PRODUCT,
         "calculated_from": ["taxable_base", "vat_rate"]},

        {"calculated_features": "line_total",
         "function": BusinessRule.SUM,
         "calculated_from": ["taxable_base", "vat_value"]},

        # 4. MULTI-OUTPUT: one function, two columns, returned as a tuple of
        #    Series in the same order as the declared outputs.
        {"calculated_features": ["net_per_unit", "total_per_unit"],
         "function": lambda base, total, qty: ((base / qty).round(4),
                                               (total / qty).round(4)),
         "calculated_from": ["taxable_base", "line_total", "quantity"]},

        # 5. outputs need not be numeric — a rule can produce a flag
        {"calculated_features": "requires_approval",
         "function": lambda total, pct: (total > 5_000) | (pct > 0.25),
         "calculated_from": ["line_total", "discount_pct"]},
    ],
)

lines = synth.sample(2_000, random_state=42).to_pandas()

# ---------------------------------------------------------------------------
# Every identity holds on every row — that is the whole point
# ---------------------------------------------------------------------------
assert np.allclose(lines["line_net"], lines["quantity"] * lines["unit_price"])
assert np.allclose(lines["taxable_base"], lines["line_net"] - lines["discount_value"])
assert np.allclose(lines["vat_value"], lines["taxable_base"] * lines["vat_rate"])
assert np.allclose(lines["line_total"], lines["taxable_base"] + lines["vat_value"])
assert np.allclose(lines["total_per_unit"], (lines["line_total"] / lines["quantity"]).round(4))

# the lookup is exact: one rate per region, no drift
observed = lines.groupby("tax_region")["vat_rate"].unique()
assert all(len(v) == 1 for v in observed), "each region must carry exactly one rate"
assert {r: v[0] for r, v in observed.items()} == {
    r: VAT_RATES[r] for r in observed.index}

# the boolean rule agrees with its own definition
expected_flag = (lines["line_total"] > 5_000) | (lines["discount_pct"] > 0.25)
assert (lines["requires_approval"] == expected_flag).all()

print(lines.head(6).to_string(index=False))
print(f"\n{len(lines)} invoice lines | "
      f"{lines['requires_approval'].sum()} flagged for approval")
print("VAT rate per region:")
print(lines.groupby("tax_region")["vat_rate"].first().to_string())
print("\nAll arithmetic, lookup and flag identities hold on every row.")

Notes

Calculated features are passed in code

Every other Faker configuration block is pure data and can be declared in a configuration file. Calculated features are passed to fit(calculated_features=...) in code. Pre-defined business rules can already be referenced by name ("function": "line_amount"); declaring them in the YAML configuration is planned.

Scaffold columns are read-only

A calculated feature may consume scaffold dimensions and include columns, but never overwrite them. Cross-table reference_keys are likewise not supported by this single-table synthesizer.

When is a column a calculated feature?

Whenever it has a single correct answer given the other columns. If a validation script could check the column, a calculated feature should produce it. Columns carrying new information — names, free text, amounts, categories — belong to generation.

Related Materials

• Sequence and Per-Partition Features
• Derived Tables and Aggregations
• Calculated Features (Business Rules)