Dataset

The Dataset object in ydata-sdk is a schema-aware data structure built on pandas, providing a consistent interface for data manipulation, profiling, and synthetic data generation workflows. It wraps a pandas DataFrame and adds type management, missing-value tracking, and convenience methods used across the entire YData ecosystem.

The Dataset object integrates natively with pandas and numpy, allowing effortless conversion to and from both formats.

Key features

• Schema Awareness: Stores and maintains column metadata, variable types, and structure for enhanced data integrity.
• Seamless Integration: Easily converts to pandas DataFrames or numpy arrays for flexible data manipulation.
• In-Memory Processing: Designed for datasets that fit in memory, powered by pandas for reliable single-machine performance.
• Supports Data Preprocessing – Provides built-in utilities for type casting, sampling, filtering, and data transformation.

ydata.dataset.dataset.Dataset

Dataset class provides the interface to handle data within YData's package.

Parameters:

Name	Type	Description	Default
df	Union[DataFrame, ndarray]	The data to be manipulated.	required
schema	Optional[Dict]	Mapping of column names to variable types.	None
sample	float	Fraction of the data to be sampled as the Dataset	0.2
index	Optional[str]	Name of the column to be used as index, if any. This is an optional input, specially recommended for TimeSeries data.	None

Properties

columns (list[str]): list of column names that are part of the Dataset schema nrows (int): number of rows from the Dataset ncols (int): number of columns shape (tuple): tuple of (nrows, ncols) memory_usage (int): total memory usage of the dataset in bytes nmissings (int): total number of missings in Dataset dtypes (Dict[str, str]): mapping of data type per column, either provided or inferred index (str): Returns the name of the index column

Magic Methods

columns property

A list with the Dataset column names.

index property

The name of the logical index column, if any.

loc property

Label location based indexer for selection. Delegates to the underlying pandas DataFrame loc indexer.

df.loc["b"] df.loc["b":"d"]

memory_usage property

A property that returns the total memory usage of the Dataset in bytes. Returns: memory_usage (int): Total memory usage of the Dataset in bytes.

ncols property

Number of columns in the Dataset.

nmissings property

Total number of missing values in the Dataset.

nrows property

Number of rows in the Dataset.

schema property writable

A dictionary mapping column names to their VariableType.

Returns:

Name	Type	Description
schema	dict	{column_name: VariableType}

apply(function, axis=1, raw=False, args=None)

Apply a function along an axis of the Dataset.

Parameters:

Name	Type	Description	Default
function	callable	Function to apply to each row.	required
axis	Union[int, str]	1/'columns' apply function to each row. 0/'index' apply function to each column is not supported.	1
raw	bool	If True, the function receives numpy arrays instead of Series.	False
args	Optional[Tuple]	Positional arguments to pass to function.	None

Returns:

Name	Type	Description
df	Dataset	A dataset object output of function.

astype(column, vartype, format=None)

Convert a column in the dataset to a specified data type.

Parameters:

Name	Type	Description	Default
column	str	The name of the column to convert.	required
vartype	VariableType | str	The target data type.	required
format	str	Format string for date parsing.	None

copy()

Copy a Dataset instance.

Returns:

Name	Type	Description
dataset	Dataset	A new Dataset instance with the same schema and index.

drop_columns(columns, inplace=False)

Drop specified columns from a Dataset.

Parameters:

Name	Type	Description	Default
columns	str or list	Column labels to drop.	required
inplace	bool	If False, return a copy. Otherwise, drop inplace.	False

head(n=5)

Return the first n rows of the dataset.

Parameters:

Name	Type	Description	Default
n	int	Number of rows to return.	5

Returns:

Name	Type	Description
dataset	DataFrame	First n rows.

infer_dtypes(schema=None)

Infer and assign data types to dataset columns.

Parameters:

Name	Type	Description	Default
schema	Optional[dict]	A dictionary of manually assigned types. If None, types are inferred automatically.	None

missings()

Calculates the number of missing values per column in a Dataset. Returns: missings (pandas.Series): Missing value count per column.

query(query)

Filter the dataset using a pandas query expression.

Parameters:

Name	Type	Description	Default
query	str	The query expression to filter the dataset.	required

Returns:

Name	Type	Description
dataset	Dataset	The filtered dataset.

reorder_columns(columns)

Reorder the columns of the dataset.

Parameters:

Name	Type	Description	Default
columns	List[str]	Ordered list of column names.	required

Returns:

Name	Type	Description
dataset	Dataset	Dataset with columns in the specified order.

sample(size, strategy='random', **strategy_params)

Generate a sampled subset of the dataset.

Parameters:

Name	Type	Description	Default
size	Union[float, int]	Size (number of rows or fraction) of the sample.	required
strategy	str	'random' or 'stratified'.	'random'

Returns:

Name	Type	Description
dataset	Dataset	The sampled subset.

select_columns(columns, copy=True)

Returns a Dataset containing only a subset with the specified columns.

Parameters:

Name	Type	Description	Default
columns	str or list	Column labels to select.	required
copy	bool	If True, return a copy.	True

select_dtypes(include=None, exclude=None)

Return a subset of the dataset containing only specified data types.

Parameters:

Name	Type	Description	Default
include	str | list | None	Variable types to include.	None
exclude	str | list | None	Variable types to exclude.	None

Returns:

Name	Type	Description
dataset	Dataset	Filtered dataset.

shape()

Return the (nrows, ncols) shape of the Dataset.

Returns:

Name	Type	Description
shape	tuple	A tuple of (nrows, ncols).

sort_values(by, ignore_index=True, inplace=False)

Sort the dataset by one or more columns.

Parameters:

Name	Type	Description	Default
by	List[str]	Column(s) to sort by.	required
ignore_index	bool	Whether to reset the index after sorting.	True
inplace	bool	Whether to sort in-place.	False

Returns:

Name	Type	Description
dataset	Dataset	Sorted dataset (if inplace is False).

sorted_index(by)

Get the sorted index positions of the dataset based on specified columns.

Parameters:

Name	Type	Description	Default
by	List[str]	Column(s) to sort by.	required

Returns:

Name	Type	Description
index	Series	Sorted index positions.

tail(n=5)

Return the last n rows of the dataset.

Parameters:

Name	Type	Description	Default
n	int	Number of rows to return.	5

Returns:

Name	Type	Description
dataset	DataFrame	Last n rows.

to_numpy()

Converts the Dataset to a numpy ndarray.

Returns:

Name	Type	Description
dataset	ndarray	The underlying data as a numpy array.

to_pandas()

Converts the Dataset to a pandas DataFrame.

Returns:

Name	Type	Description
dataset	DataFrame	The underlying pandas DataFrame.

uniques(col)

Compute the exact number of unique values in a column.

Parameters:

Name	Type	Description	Default
col	str	The column name.	required

Returns:

Name	Type	Description
nuniques	int	The number of unique values.

update_types(dtypes)

Batch update data types for multiple columns.

Parameters:

Name	Type	Description	Default
dtypes	list	List of dicts, each with 'column' and 'vartype' keys.	required

value_counts(col)

Return the count of occurrences for each unique value in a column.

Parameters:

Name	Type	Description	Default
col	str	The name of the column to count values for.	required

Returns:

Name	Type	Description
value_counts	Series	Value counts per unique value.