PII identification

To overcome the concerns around data privacy and enable secure data sharing, YData SDK incorporates an automated Personal Identifiable Information (PII) identification engine to help detect and handle potential PII.

What can be considered Personal Identifiable Information (PII)?

PII is information that, when used alone or with other relevant data, can uniquely identify an individual. PII may contain direct indentifiers (e.g., ID, VAT, Credit Card Number) and/or quasi-identifiers (e.g., age, gender, race, occupation). Correctly classifying these is crucial to reduce the risk of re-identification. Learn more about how YData mitigates the risk of re-identification using synthetic data.

YData SDK offers a standardized classification of PII that automatically highlights and tags potential PII. The automatic detection of PII can be enabled when reading the data from a connector and can be leveraged to generate privacy-preserving synthetic data.

Characteristics

PII is represented as characteristics of a column (ydata.characteristics.ColumnCharacteristic):

Characteristic	Meaning	Detected automatically from
`email`	Email address	values, column name
`person`	Person name	values, column name
`location`	Place: city, region, country	values, column name
`url`	URL	values, column name
`credit_card`	Card number	values, column name
`iban`	IBAN	values, column name
`phone`	Phone number	column name
`address`	Street or full address	column name
`zipcode`	Postal code	column name
`country`	Country	column name
`vat`	VAT / tax identification number	column name
`id`	Identifier	metadata (unique values)
`name`, `uuid`, `pii`	Name, UUID, generic PII	declared by the user only

Detecting characteristics

Detection runs when the metadata is computed with infer_characteristics=True:

from ydata.dataset import Dataset
from ydata.metadata import Metadata

metadata = Metadata(Dataset(df), infer_characteristics=True)

for col, details in metadata.columns.items():
    if details.characteristics:
        print(col, sorted(c.value for c in details.characteristics))
# customer_ref ['id']
# full_name ['id', 'person']
# email ['email', 'id']
# phone ['id', 'phone']
# city ['id', 'location']
# website ['id', 'url']

It combines three sources of evidence:

Values. Each distinct value of the string columns is analysed with pattern and checksum recognizers for emails, URLs, card numbers (issuer prefix + Luhn), IBANs and phone numbers, and a spaCy model for person and location names. For each value, the entity with the highest score counts, if the score reaches the threshold for its characteristic. A column is tagged when the share of matching values reaches characteristics_threshold (80% by default). Placeholder values such as unknown, n/a or - are ignored. Columns with more than 11,000 distinct values are analysed on a sample of 11,000 values.
Column names. Names are matched against keywords in English, Portuguese, Spanish, French, German, Italian and Dutch. Specific names tag the column directly (last_name, apellidos, codigo_postal, telefone, city, morada, vat_id). Generic names (name, email, website, card_number) only lower the bar: at least 50% of the values must confirm the characteristic. Names describing something else (company_name, email_domain, card_type) are ignored.
Metadata. Columns with unique values, no missing values and a categorical or string data type are tagged as id.

How each characteristic was found

metadata.summary["characteristics"] holds, for each column, the share of values that matched (value) and the source of the evidence (values, column_name, metadata or manual):

metadata.summary["characteristics"]["phone"]
# [{'characteristic': ColumnCharacteristic.PHONE, 'value': 1, 'upper_bound': 1, 'lower_bound': 1, 'source': 'column_name'},
#  {'characteristic': ColumnCharacteristic.ID, 'value': 1, 'upper_bound': 1, 'lower_bound': 1, 'source': 'metadata'}]

When a column is analysed on a sample, lower_bound and upper_bound give the 99% confidence interval of the share.

Parameters

Parameter	Default	Effect
`infer_characteristics`	`False`	Run the detection.
`characteristics`	`None`	Known characteristics, `{column: characteristic}` or `{column: [characteristics]}`. Always applied and merged with the detected ones.
`characteristics_language`	`"en"`	Language of the values, selects the spaCy model for names and places: `en`, `pt`, `es`, `fr`, `de`, `it`, `nl`, or `xx` (multilingual). Models other than English are downloaded on first use (~500 MB); install them beforehand where there is no internet access (`python -m spacy download pt_core_news_lg`).
`characteristics_threshold`	`0.8`	Share of a column's values that must match. Lower it for columns with mixed content (e.g. a contact column with emails and phone numbers).
`characteristics_score_thresholds`	`0.8`, URL `0.6`	Minimum detection score of a single value, per characteristic, e.g. `{"url": 0.5}`.

Lowering the score thresholds

Pattern-only matches get a fixed detection score: 0.5 for URLs without a scheme (www.site.com, site.com) and 0.4 for phone numbers. Accepting those scores detects more columns but also matches other values: at 0.5, company names such as Castro Co.Ltd and usernames such as cindy.harmon are read as URLs; at 0.4, most 10-digit identifiers (e.g. account numbers) are read as phone numbers.

To run the detection on an existing Metadata without recomputing it, use metadata.compute_characteristics(dataset).

What to expect

Share of values identified, per type of data, with the default settings and the English model. Columns are tagged from their values when the share is above 80%; otherwise only through their column name.

Data	Values identified	Tagged from values only	Tagged with a descriptive column name
Full names (English), any case	95–99%	yes	yes
Full names (pt, es, de, fr, it, nl)	88–95%	yes	yes
First names only	91%	yes	yes
Last names only	77%	no	yes
Names in non-Latin scripts (Devanagari, Chinese, Japanese)	< 6%	no	no
Cities	13–19% (mostly read as person names)	no	yes
Countries (English) / US states	86% / 100%	yes	yes
Street or full addresses	2–27%	no	yes
URLs with a scheme (`https://...`)	94–98%	yes	yes
`www.` or bare domains	0% at the default threshold	no	yes
Emails	100%	yes	yes
Card numbers (mixed brands)	93%	yes	yes
IBANs	100%	yes	yes
Phone numbers	0% at the default threshold	no	yes

With characteristics_language="pt", Portuguese cities (97%) and country names in Portuguese (91%) are tagged from their values, while upper-case names and URLs with a path are detected less often. Use the language parameter only when the data is in that language.

These figures were measured on generated values; real data is usually less regular.

Reviewing the result

Detected characteristics are suggestions. Review them before generating synthetic data, and add or remove what is needed:

metadata.add_characteristics({"segment": "pii"})
metadata.remove_characteristics({"website": "url"})

Columns with characteristics are anonymized automatically by the synthesizers. metadata.get_anonymizers() returns the anonymizer fit will apply to each column, and the anonymize argument of fit overrides them: {column: <anonymizer>} to use another anonymizer, {column: None} to keep the original values.

metadata.get_anonymizers()
# {'customer_ref': <function mask_anonymizer>, 'full_name': AnonymizerType.NAME, 'email': AnonymizerType.EMAIL,
#  'phone': AnonymizerType.PHONE, 'city': <function mask_anonymizer>, 'website': AnonymizerType.URL}

id and location columns are masked: digits and letters are replaced at random and other characters kept, so values keep their format without being real (CUS-1234567 → XQR-8302114, New York → Qwe Rtyu). See the anonymization examples for the anonymizer configuration.

Example

The PII detection example covers detection, inspection, manual characteristics and the detection parameters.