Profiling#

class dataforge_ml.profiling.orchestrator.StructuralProfiler(config=None, observer=None)[source]#

Bases: _ObservabilityMixin

Phase 1 orchestrator — runs all sub-processors and assembles StructuralProfileResult.

Parameters:

config (PipelineConfig, optional) – Master pipeline configuration. Defaults to PipelineConfig() which applies all sub-processor defaults exactly as they were before Scope 15.
observer (Callable[[PipelineEvent], None], optional) – Live Progress Observer invoked with each PipelineEvent this orchestrator emits. None (the default) disables Progress; Trace still flows to the dataforge_ml logger regardless (ADR-0054).

profile(data)[source]#

Profile a Polars DataFrame and return a full StructuralProfileResult.

Runs all Phase 1 sub-processors in sequence: modality stats, missingness, type detection, per-column distribution profiling, target profiling, and (optionally) correlation analysis. Sub-processor thresholds are drawn from the nested sub-configs on config.profiling.

Parameters:

data (Any) – Must be a polars.DataFrame.

Returns:

Fully populated profile result for the dataset.

Return type:

StructuralProfileResult

Raises:

TypeError – When data is not a polars.DataFrame.
OverrideCoercionError – When a column carrying TypeFlag.UserOverride completely fails coercion to its overridden SemanticType (zero usable values remain despite the original column having non-null data).

class dataforge_ml.profiling._config.ProfileConfig(modality=Modality.Tabular, target_columns=<factory>, compute_correlation=False, correlation_target_column=None, memory_threshold_mb=500.0, chunk_size=100000, row_drop_threshold=0.5, missingness=<factory>, numeric=<factory>, type_detection=<factory>, categorical=<factory>, correlation=<factory>, datetime_=<factory>, compute_nonlinearity=False, nonlinearity=<factory>, numeric_sentinels=<property object>, string_sentinels=<property object>, datetime_epoch_units=<property object>, datetime_formats=<property object>)[source]#

Bases: object

Controls the structural profiler’s behaviour.

Parameters:

modality (Modality) – Data modality. Currently only Tabular is implemented.
target_columns (list[str]) – Names of label/target columns, if any.
compute_correlation (bool) – Whether to compute the feature-feature correlation matrix.
correlation_target_column (Optional[str]) – Column used for feature-target correlation metrics.
memory_threshold_mb (float) – Memory (MB) above which chunked processing activates.
chunk_size (int) – Rows per chunk when chunked processing is active.
row_drop_threshold (float) – Fraction of columns that must be missing in a row for that row to be counted as a drop candidate in RowMissingnessDistribution.
missingness (MissingnessProfileConfig) – Threshold configuration for the missingness sub-processor.
numeric (NumericProfileConfig) – Threshold configuration for the numeric distribution sub-processor.
type_detection (TypeDetectionConfig) – Threshold configuration for the type-detection sub-processor.
categorical (CategoricalProfileConfig) – Threshold configuration for the categorical sub-processor.
correlation (CorrelationProfileConfig) – Threshold configuration for the correlation sub-processor.
datetime (DatetimeProfileConfig) – Threshold configuration for the datetime sub-processor.
compute_nonlinearity (bool) – Whether to run NonlinearityProfiler and populate the four nonlinearity signal fields on NumericStats. When True and compute_correlation is also True, Pearson and Spearman matrices are reused from the correlation step rather than recomputed. Default False.
nonlinearity (NonlinearityProfileConfig) – Threshold configuration for the nonlinearity sub-processor.
datetime_ (DatetimeProfileConfig)
numeric_sentinels (dataclasses.InitVar[Optional[dict[str, list[float]]]])
string_sentinels (dataclasses.InitVar[Optional[dict[str, list[str]]]])
datetime_epoch_units (dataclasses.InitVar[Optional[dict[str, Union[str, dataforge_ml.profiling._config.EpochUnit]]]])
datetime_formats (dataclasses.InitVar[Optional[dict[str, str]]])

categorical: CategoricalProfileConfig#

chunk_size: int = 100000#

compute_correlation: bool = False#

compute_nonlinearity: bool = False#

correlation: CorrelationProfileConfig#

correlation_target_column: str | None = None#

datetime_: DatetimeProfileConfig#

property datetime_epoch_units: MappingProxyType#

Get the per-column datetime epoch units.

Returns:: Read-only mapping of column names to epoch units.
Return type:: MappingProxyType[str, EpochUnit]

property datetime_formats: MappingProxyType#

Get the per-column declared datetime format strings.

Keys are column names; values are strftime-style format strings (e.g. {"Year": "%Y"}) applied by DatetimeProfiler with strict=False when coercing that column to Datetime. A declaration applies to any column profiled as Datetime, whether overridden or auto-detected. Format strings are not validated against strftime grammar at declaration time — a bad format surfaces at profiling time. Defaults to an empty dict — columns with no declaration fall back to Polars format inference.

Returns:: Read-only mapping of column names to declared datetime formats.
Return type:: MappingProxyType[str, str]

classmethod from_dict(data)[source]#

Construct a ProfileConfig from a plain dictionary.

Parameters:: data (dict) – Mapping produced by to_dict(). Missing keys fall back to field defaults.
Returns:: Reconstructed config instance with all sub-configs deserialised.
Return type:: ProfileConfig

classmethod from_json(json_str)[source]#

Construct a ProfileConfig from a JSON string.

Parameters:: json_str (str) – JSON produced by to_json().
Returns:: Reconstructed config instance.
Return type:: ProfileConfig

memory_threshold_mb: float = 500.0#

missingness: MissingnessProfileConfig#

modality: Modality = 'tabular'#

nonlinearity: NonlinearityProfileConfig#

numeric: NumericProfileConfig#

property numeric_sentinels: MappingProxyType#

Get the per-column numeric sentinel declarations.

Keys are column names; values are lists of float-compatible sentinel values that should be treated as effective nulls (e.g. {"age": [-999.0, 9999.0]}). Applies to any column whose dtype passes _numeric_sentinel_eligible (all integer and float Polars dtypes). Defaults to an empty dict — columns with no declaration are completely unaffected.

Returns:: Read-only mapping of column names to numeric sentinel values.
Return type:: MappingProxyType[str, list[float]]

row_drop_threshold: float = 0.5#

set_datetime_epoch_unit(column, unit)[source]#

Set epoch unit for one or more columns.

Parameters:

column (str or list of str) – Column name or list of column names to apply the epoch unit to.
unit (str or EpochUnit) – Epoch unit (e.g. “s”, “ms”).

Raises:

ValueError – If the unit is not a valid EpochUnit.

Return type:

None

set_datetime_format(column, format)[source]#

Declare a datetime format string for one or more columns.

The format is applied by DatetimeProfiler with strict=False when coercing the column to Datetime, and is not validated against strftime grammar or the data at declaration time — a bad format surfaces at profiling time, consistent with set_column_type and set_datetime_epoch_unit.

Parameters:

column (str or list of str) – Column name or list of column names to apply the format to.
format (str) – A non-empty strftime-style format string (e.g. "%Y").

Raises:

ValueError – If any column name is empty, or if format is not a non-empty string.

Return type:

None

set_numeric_sentinel(column, values)[source]#

Set numeric sentinel values for one or more columns.

Parameters:

column (str or list of str) – Column name or list of column names to apply the sentinels to.
values (list of float) – List of sentinel values. Must be numeric and non-empty.

Raises:

ValueError – If the values list is empty, or if any element is not numeric.

Return type:

None

set_string_sentinel(column, values)[source]#

Set string sentinel values for one or more columns.

Parameters:

column (str or list of str) – Column name or list of column names to apply the sentinels to.
values (list of str) – List of string sentinel values. Must be strings and non-empty.

Raises:

ValueError – If the values list is empty, or if any element is not a string.

Return type:

None

property string_sentinels: MappingProxyType#

Get the per-column user-declared string sentinel declarations.

Keys are column names; values are lists of string values that should be treated as effective nulls for that column (e.g. {"status": ["N/A", "missing"]}). Uses replace semantics: when a declaration exists for a column, only the declared values are matched (case-insensitive); the hardcoded defaults ("NA", "NAN", "NULL", "NONE", "?") are not applied for that column. Empty/whitespace-only strings are always effective null regardless of any declaration. Defaults to an empty dict — columns with no declaration continue to use the hardcoded defaults unchanged.

Returns:: Read-only mapping of column names to string sentinel values.
Return type:: MappingProxyType[str, list[str]]

target_columns: list[str]#

to_dict()[source]#

Serialise the config to a plain dictionary.

Returns:: All field values, with nested sub-configs serialised recursively.
Return type:: dict

to_json()[source]#

Serialise the config to a JSON string.

Returns:: JSON representation of to_dict().
Return type:: str

type_detection: TypeDetectionConfig#

class dataforge_ml.profiling._missingness_config.MissingnessProfileConfig(severity_minor=0.01, severity_moderate=0.05, severity_high=0.2, mar_correlation_threshold=0.6, col_drop_threshold=0.5)[source]#

Bases: object

Threshold configuration for the missingness sub-processor.

All fields default to the library’s original hard-coded constants so that constructing MissingnessProfileConfig() produces identical behaviour to the pre-config implementation.

Parameters:

severity_minor (float) – Effective null ratio upper bound (exclusive) for MissingSeverity.Minor. Columns with a ratio below this value are classified as Minor.
severity_moderate (float) – Effective null ratio upper bound (exclusive) for MissingSeverity.Moderate.
severity_high (float) – Effective null ratio upper bound (exclusive) for MissingSeverity.High. Columns at or above this value are classified as MissingSeverity.Severe.
mar_correlation_threshold (float) – Minimum absolute Pearson correlation between binary missingness indicators for a column pair to receive the MissingnessFlag.MARSuspect flag.
col_drop_threshold (float) – Effective null ratio above which a non-fully-null column receives the MissingnessFlag.DropCandidate flag.

col_drop_threshold: float = 0.5#

classmethod from_dict(data)[source]#

Construct a MissingnessProfileConfig from a plain dictionary.

Parameters:: data (dict) – Mapping produced by to_dict(). Missing keys fall back to field defaults.
Returns:: Reconstructed config instance.
Return type:: MissingnessProfileConfig

mar_correlation_threshold: float = 0.6#

severity_high: float = 0.2#

severity_minor: float = 0.01#

severity_moderate: float = 0.05#

to_dict()[source]#

Serialise the config to a plain dictionary.

Returns:: All field values keyed by field name.
Return type:: dict

class dataforge_ml.profiling._numeric_config.NumericProfileConfig(skew_normal=0.5, skew_moderate=1.0, skew_high=2.0, kurt_platykurtic_upper=-1.0, kurt_leptokurtic_lower=3.0, near_constant_threshold=0.9, scale_orders_of_magnitude=3, bimodal_dip_p_value_threshold=0.05, bimodal_min_separation_threshold=2.0, bimodal_min_component_weight=0.05, tail_asymmetry_right_share_threshold=0.6666666666666666, tail_asymmetry_left_share_threshold=0.3333333333333333, outlier_sigma_threshold=3.0, high_outlier_density_threshold=0.05)[source]#

Bases: object

Threshold configuration for the numeric distribution sub-processor.

All fields default to the library’s original hard-coded constants so that constructing NumericProfileConfig() produces identical behaviour to the pre-config implementation.

Parameters:

skew_normal (float) – Absolute skewness upper bound (inclusive) for SkewSeverity.Normal.
skew_moderate (float) – Absolute skewness upper bound (inclusive) for SkewSeverity.Moderate. Columns with |skew| > skew_normal and |skew| <= skew_moderate are classified as Moderate.
skew_high (float) – Absolute skewness upper bound (inclusive) for SkewSeverity.High. Columns with |skew| > skew_moderate and |skew| <= skew_high are classified as High. Columns above this bound are Severe.
kurt_platykurtic_upper (float) – Excess kurtosis upper bound (exclusive) for KurtosisTag.Platykurtic. Columns with kurtosis < kurt_platykurtic_upper are Platykurtic.
kurt_leptokurtic_lower (float) – Excess kurtosis lower bound (exclusive) for KurtosisTag.Leptokurtic. Columns with kurtosis > kurt_leptokurtic_lower are Leptokurtic. All others are Mesokurtic.
near_constant_threshold (float) – Mode frequency above which a column receives NumericFlag.NearConstant. Expressed as a fraction of total rows (e.g. 0.90 = 90%).
scale_orders_of_magnitude (int) – Number of orders of magnitude the absolute value range must span for a column to receive NumericFlag.ScaleAnomaly (i.e. ratio >= 10^n).
bimodal_dip_p_value_threshold (float) – P-value threshold for Hartigan’s Dip Test to classify as bimodal. Default 0.05.
bimodal_min_separation_threshold (float) – Ashman’s D separation threshold above which bimodality is confirmed. Default 2.0.
bimodal_min_component_weight (float) – Minimum mixing weight for the smaller component to confirm bimodality. Default 0.05.
tail_asymmetry_right_share_threshold (float) – Threshold for tail asymmetry share above which the distribution is RightHeavy. Default 2/3.
tail_asymmetry_left_share_threshold (float) – Threshold for tail asymmetry share below which the distribution is LeftHeavy. Default 1/3.
outlier_sigma_threshold (float) – Sigma threshold for outlier detection. Default 3.0.
high_outlier_density_threshold (float) – Threshold for outlier density above which HighOutlierDensity flag is raised. Default 0.05.

bimodal_dip_p_value_threshold: float = 0.05#

bimodal_min_component_weight: float = 0.05#

bimodal_min_separation_threshold: float = 2.0#

classmethod from_dict(data)[source]#

Construct a NumericProfileConfig from a plain dictionary.

Parameters:: data (dict) – Mapping produced by to_dict(). Missing keys fall back to field defaults.
Returns:: Reconstructed config instance.
Return type:: NumericProfileConfig

high_outlier_density_threshold: float = 0.05#

kurt_leptokurtic_lower: float = 3.0#

kurt_platykurtic_upper: float = -1.0#

near_constant_threshold: float = 0.9#

outlier_sigma_threshold: float = 3.0#

scale_orders_of_magnitude: int = 3#

skew_high: float = 2.0#

skew_moderate: float = 1.0#

skew_normal: float = 0.5#

tail_asymmetry_left_share_threshold: float = 0.3333333333333333#

tail_asymmetry_right_share_threshold: float = 0.6666666666666666#

to_dict()[source]#

Serialise the config to a plain dictionary.

Returns:: All field values keyed by field name.
Return type:: dict

class dataforge_ml.profiling._numeric_config.NonlinearityProfileConfig(spearman_pearson_discrepancy_threshold=0.1, mutual_information_threshold=0.05, r2_gap_threshold=0.1, heteroscedasticity_p_value_threshold=0.05, r2_rf_unpredictable_threshold=0.05, min_rows=20, bootstrap_sample_size=500, random_state=42)[source]#

Bases: object

Threshold configuration for the NonlinearityProfiler sub-processor.

Defaults are calibrated for typical tabular datasets (hundreds to thousands of rows):

A Spearman/Pearson discrepancy ≥ 0.10 reliably separates monotonic non-linear relationships from linear ones across common real-world data.
p < 0.05 is the standard significance level for the Breusch-Pagan test.
An R² gap ≥ 0.10 represents a meaningful improvement of a non-linear model over a linear baseline (calibrated against held-out evaluation).
R²_RF < 0.05 (cross-validated on a bootstrap sample) indicates the column is not meaningfully predictable from its numeric co-variates.

Parameters:

spearman_pearson_discrepancy_threshold (float) – Minimum max-over-predictors |Spearman_r − Pearson_r| that triggers the monotonic-nonlinear signal. Values in [0, 1]; default 0.10.
mutual_information_threshold (float) – Minimum mean mutual information (nats) across predictors that contributes to a ComplexNonlinear classification. Default 0.05.
r2_gap_threshold (float) – Minimum R²_RF − R²_linear (both cross-validated on a bootstrap sample) to classify a column as ComplexNonlinear rather than MonotonicNonlinear. Default 0.10.
heteroscedasticity_p_value_threshold (float) – Breusch-Pagan p-value below which heteroscedasticity is detected, contributing to a MonotonicNonlinear classification. Default 0.05.
r2_rf_unpredictable_threshold (float) – Cross-validated R²_RF below which a column is tagged Unpredictable. Default 0.05.
min_rows (int) – Minimum number of complete (non-null) rows required across the target column and all its predictors to run the profiler for that column. Default 20.
bootstrap_sample_size (int) – Maximum number of rows sampled for the R² gap and Breusch-Pagan computations. Larger datasets are down-sampled uniformly at random. Default 500.
random_state (int) – Random seed used for row sampling and RandomForestRegressor. Default 42.

bootstrap_sample_size: int = 500#

classmethod from_dict(data)[source]#

Construct a NonlinearityProfileConfig from a plain dictionary.

Parameters:: data (dict) – Mapping produced by to_dict(). Missing keys fall back to field defaults.
Returns:: Reconstructed config instance.
Return type:: NonlinearityProfileConfig

heteroscedasticity_p_value_threshold: float = 0.05#

min_rows: int = 20#

mutual_information_threshold: float = 0.05#

r2_gap_threshold: float = 0.1#

r2_rf_unpredictable_threshold: float = 0.05#

random_state: int = 42#

spearman_pearson_discrepancy_threshold: float = 0.1#

to_dict()[source]#

Serialise the config to a plain dictionary.

Returns:: All field values keyed by field name.
Return type:: dict

class dataforge_ml.profiling._type_detection_config.TypeDetectionConfig(numeric_coerce_threshold=0.95, datetime_coerce_threshold=0.8, encoded_category_max_unique=15, encoded_category_max_ratio=0.05, identifier_unique_ratio=0.99, identifier_max_median_length=40, free_text_avg_words=3, free_text_median_chars=20, free_text_p90_chars=35, free_text_min_unique_ratio=0.4, free_text_high_unique_with_spaces=0.7)[source]#

Bases: object

Threshold configuration for the type-detection sub-processor.

All fields default to the library’s original hard-coded constants so that constructing TypeDetectionConfig() produces identical behaviour to the pre-config implementation.

Parameters:

numeric_coerce_threshold (float) – Minimum fraction of non-null values that must cast successfully to Float64 for a Utf8 column to be reclassified as SemanticType.Numeric.
datetime_coerce_threshold (float) – Minimum fraction of non-null values that must parse successfully as a datetime for a Utf8 column to be reclassified as SemanticType.Datetime.
encoded_category_max_unique (int) – Maximum absolute number of unique values for a non-tight-sequence integer column to be labelled TypeFlag.EncodedCategory and classified as SemanticType.Categorical.
encoded_category_max_ratio (float) – Maximum ratio of unique values to non-null rows for an integer column to be labelled TypeFlag.EncodedCategory.
identifier_unique_ratio (float) – Minimum ratio of unique values to total rows above which a column is flagged as TypeFlag.IdentifierColumn and classified as SemanticType.Identifier.
identifier_max_median_length (int) – Maximum median character length allowed for a Utf8 column to be classified as SemanticType.Identifier. Columns with longer median length are treated as free text instead.
free_text_avg_words (int) – Median word count above which a Utf8 column is flagged as TypeFlag.FreeTextCandidate.
free_text_median_chars (int) – Median character length above which a multi-word Utf8 column is flagged as TypeFlag.FreeTextCandidate.
free_text_p90_chars (int) – 90th-percentile character length above which a high-cardinality Utf8 column is flagged as TypeFlag.FreeTextCandidate.
free_text_min_unique_ratio (float) – Minimum unique ratio required (alongside free_text_p90_chars) to flag a Utf8 column as TypeFlag.FreeTextCandidate.
free_text_high_unique_with_spaces (float) – Unique ratio above which a multi-token Utf8 column is flagged as TypeFlag.FreeTextCandidate regardless of character length.

datetime_coerce_threshold: float = 0.8#

encoded_category_max_ratio: float = 0.05#

encoded_category_max_unique: int = 15#

free_text_avg_words: int = 3#

free_text_high_unique_with_spaces: float = 0.7#

free_text_median_chars: int = 20#

free_text_min_unique_ratio: float = 0.4#

free_text_p90_chars: int = 35#

classmethod from_dict(data)[source]#

Construct a TypeDetectionConfig from a plain dictionary.

Parameters:: data (dict) – Mapping produced by to_dict(). Missing keys fall back to field defaults.
Returns:: Reconstructed config instance.
Return type:: TypeDetectionConfig

identifier_max_median_length: int = 40#

identifier_unique_ratio: float = 0.99#

numeric_coerce_threshold: float = 0.95#

to_dict()[source]#

Serialise the config to a plain dictionary.

Returns:: All field values keyed by field name.
Return type:: dict

class dataforge_ml.profiling._categorical_config.CategoricalProfileConfig(rare_threshold_pct=0.01, stratification_rare_threshold_pct=0.05, mixed_type_min_minor_pct=0.05, near_constant_threshold=0.9)[source]#

Bases: object

Threshold configuration for the categorical column sub-processor.

All fields default to the library’s original hard-coded constants so that constructing CategoricalProfileConfig() produces identical behaviour to the pre-config implementation.

Note: near_constant_threshold is independent of the equivalent field in NumericProfileConfig — they share a default but are separately tunable.

Parameters:

rare_threshold_pct (float) – Fraction of total rows below which a category is counted as rare for diagnostic purposes (RareCategoryStats.rare_category_count).
stratification_rare_threshold_pct (float) – Fraction of total rows below which a category is added to RareCategoryStats.rare_label_values, used by the stratified splitter to protect minority classes.
mixed_type_min_minor_pct (float) – Minimum Wilson-interval lower bound for the minority type fraction required to set CategoricalFlag.MixedType.
near_constant_threshold (float) – Mode frequency above which a column receives CategoricalFlag.NearConstant. Expressed as a fraction of total rows (e.g. 0.90 = 90%).

classmethod from_dict(data)[source]#

Construct a CategoricalProfileConfig from a plain dictionary.

Parameters:: data (dict) – Mapping produced by to_dict(). Missing keys fall back to field defaults.
Returns:: Reconstructed config instance.
Return type:: CategoricalProfileConfig

mixed_type_min_minor_pct: float = 0.05#

near_constant_threshold: float = 0.9#

rare_threshold_pct: float = 0.01#

stratification_rare_threshold_pct: float = 0.05#

to_dict()[source]#

Serialise the config to a plain dictionary.

Returns:: All field values keyed by field name.
Return type:: dict

class dataforge_ml.profiling._correlation_config.CorrelationProfileConfig(near_redundant_pearson_threshold=0.95, near_redundant_cramer_v_threshold=0.8, near_redundant_eta_squared_threshold=0.5, mi_min_rows=10)[source]#

Bases: object

Threshold configuration for the correlation sub-processor.

All fields default to the library’s original hard-coded constants so that constructing CorrelationProfileConfig() produces identical behaviour to the pre-config implementation.

Parameters:

near_redundant_pearson_threshold (float) – Maximum absolute Pearson or Spearman |r| below which a numeric column pair is not flagged near_redundant. Pairs whose max(|pearson_r|, |spearman_r|) exceeds this value are flagged.
near_redundant_cramer_v_threshold (float) – Cramér’s V above which a categorical column pair is flagged near_redundant.
near_redundant_eta_squared_threshold (float) – Eta-squared (η²) above which a numeric-categorical pair is flagged near_redundant.
mi_min_rows (int) – Minimum number of complete-case rows required for a k-NN mutual information estimate to be computed. Columns with fewer valid rows are silently skipped.

classmethod from_dict(data)[source]#

Construct a CorrelationProfileConfig from a plain dictionary.

Parameters:: data (dict) – Mapping produced by to_dict(). Missing keys fall back to field defaults.
Returns:: Reconstructed config instance.
Return type:: CorrelationProfileConfig

mi_min_rows: int = 10#

near_redundant_cramer_v_threshold: float = 0.8#

near_redundant_eta_squared_threshold: float = 0.5#

near_redundant_pearson_threshold: float = 0.95#

to_dict()[source]#

Serialise the config to a plain dictionary.

Returns:: All field values keyed by field name.
Return type:: dict

class dataforge_ml.profiling._datetime_config.DatetimeProfileConfig(mnar_null_ratio_threshold=0.05, high_gap_cv_threshold=1.0, recent_window_fraction=0.1)[source]#

Bases: object

Threshold configuration for the datetime sub-processor.

All fields default to the library’s original hard-coded constants so that constructing DatetimeProfileConfig() produces identical behaviour to the pre-config implementation.

Parameters:

mnar_null_ratio_threshold (float) – Null ratio above which a datetime column receives the DatetimeFlag.MnarSuspected flag. A column whose parse-level null ratio (including values that failed datetime coercion) exceeds this value is considered potentially Missing Not At Random.
high_gap_cv_threshold (float) – Coefficient of variation (std / mean) of consecutive time gaps above which a column receives the DatetimeFlag.HighGapVariance flag, indicating irregular temporal spacing.
recent_window_fraction (float) – Fraction of the total date range considered the “recent” window when checking for DatetimeFlag.RecentDateMissing. A value of 0.10 means the last 10 % of the observed date range is examined for data sparsity.

classmethod from_dict(data)[source]#

Construct a DatetimeProfileConfig from a plain dictionary.

Parameters:: data (dict) – Mapping produced by to_dict(). Missing keys fall back to field defaults.
Returns:: Reconstructed config instance.
Return type:: DatetimeProfileConfig

high_gap_cv_threshold: float = 1.0#

mnar_null_ratio_threshold: float = 0.05#

recent_window_fraction: float = 0.1#

to_dict()[source]#

Serialise the config to a plain dictionary.

Returns:: All field values keyed by field name.
Return type:: dict

class dataforge_ml.profiling._config.StructuralProfileResult(columns=<factory>, dataset=<factory>, targets=<factory>, numeric_sentinels=<factory>, string_sentinels=<factory>)[source]#

Bases: object

Top-level result returned by StructuralProfiler.profile().

Contains per-column profiles, dataset-level statistics, any target variable analyses requested via ProfileConfig.target_columns, and the declared sentinel mappings copied from ProfileConfig so Phase 2 can consume them without holding a config reference.

Parameters:

columns (dict[str, ColumnProfile])
dataset (DatasetStats)
targets (dict[str, TargetProfileResult])
numeric_sentinels (dict[str, list[float]])
string_sentinels (dict[str, list[str]])

columns: dict[str, ColumnProfile]#

dataset: DatasetStats#

classmethod from_dict(data)[source]#

Reconstruct a StructuralProfileResult from a plain dictionary.

The inverse of to_dict(), rebuilding every nested column, dataset, and target profile so from_dict(result.to_dict()) is structurally equal to result.

Parameters:: data (dict) – Mapping produced by to_dict().
Returns:: Reconstructed profiling result.
Return type:: StructuralProfileResult

numeric_sentinels: dict[str, list[float]]#

string_sentinels: dict[str, list[str]]#

targets: dict[str, TargetProfileResult]#

to_dict()[source]#

Serialise the full profiling result to a plain dictionary.

Returns:: Nested dictionary with columns, dataset, and targets keys, each recursively serialised.
Return type:: dict

to_json(indent=2)[source]#

Serialise the full profiling result to a JSON string.

Parameters:: indent (int) – Number of spaces used for JSON indentation.
Returns:: JSON representation of to_dict().
Return type:: str

to_markdown()[source]#

Produce the lossless Markdown Profile Report.

Every field present in to_dict() — including histogram bins, full correlation matrices, memory breakdown, and all per-column fields — is rendered as Markdown. For an 82-column dataset this produces roughly 1 MB of text; that size is an accepted consequence of the Rendering Contract (ADR-0086), and the bounded view of a profile is to_dict() plus the caller’s own selection.

Returns:: Markdown string containing a summary table followed by per-column detail sections and dataset-level statistics.
Return type:: str

class dataforge_ml.profiling._config.ColumnProfile(name='', semantic_type=None, numeric_kind=None, type_flags=<factory>, original_dtype='', inferred_dtype='', missingness=None, is_target=False, stats=None)[source]#

Bases: object

Per-column result produced by the structural profiler.

Carries the type classification, missingness summary, and computed statistics for a single column after Phase 1 profiling completes.

Parameters:

name (str)
semantic_type (SemanticType | None)
numeric_kind (NumericKind | None)
type_flags (list[TypeFlag])
original_dtype (str)
inferred_dtype (str)
missingness (ColumnMissingnessProfile | None)
is_target (bool)
stats (NumericStats | CategoricalStats | DatetimeStats | BooleanStats | TextStats | None)

classmethod from_dict(data)[source]#

Reconstruct a ColumnProfile from a plain dictionary.

stats is dispatched to the correct stats subtype (NumericStats, CategoricalStats, DatetimeStats, BooleanStats, or TextStats) using semantic_type as the discriminator, matching the dispatch the profiler orchestrator itself uses.

Parameters:: data (dict) – Mapping produced by to_dict().
Returns:: Reconstructed column profile instance.
Return type:: ColumnProfile

inferred_dtype: str = ''#

is_target: bool = False#

missingness: ColumnMissingnessProfile | None = None#

name: str = ''#

numeric_kind: NumericKind | None = None#

original_dtype: str = ''#

semantic_type: SemanticType | None = None#

stats: NumericStats | CategoricalStats | DatetimeStats | BooleanStats | TextStats | None = None#

to_dict()[source]#

Serialise the column profile to a plain dictionary.

Returns:: All field values with nested objects serialised to their string or dict representations.
Return type:: dict

type_flags: list[TypeFlag]#

class dataforge_ml.profiling._config.DatasetStats(modality=Modality.Tabular, row_count=0, column_count=0, memory_bytes=0, memory_breakdown=None, duplicate_count=0, duplicate_ratio=0.0, overall_sparsity=0.0, was_chunked=False, missingness_matrix=None, row_distribution=<factory>, feature_correlation=None, target_correlations=<factory>)[source]#

Bases: object

Dataset-level statistics produced by the structural profiler.

Aggregates row and memory counts, duplicate and sparsity ratios, the missingness matrix, row-level missingness distribution, and correlation results for the full profiled DataFrame.

Parameters:

modality (Modality)
row_count (int)
column_count (int)
memory_bytes (int)
memory_breakdown (MemoryBreakdown | None)
duplicate_count (int)
duplicate_ratio (float)
overall_sparsity (float)
was_chunked (bool)
missingness_matrix (dict[str, dict[str, float]] | None)
row_distribution (RowMissingnessDistribution)
feature_correlation (CorrelationProfileResult | None)
target_correlations (dict[str, CorrelationProfileResult])

column_count: int = 0#

duplicate_count: int = 0#

duplicate_ratio: float = 0.0#

feature_correlation: CorrelationProfileResult | None = None#

classmethod from_dict(data)[source]#

Reconstruct the dataset stats from a plain dictionary.

Parameters:: data (dict) – Mapping produced by to_dict().
Returns:: Reconstructed instance.
Return type:: DatasetStats

memory_breakdown: MemoryBreakdown | None = None#

memory_bytes: int = 0#

missingness_matrix: dict[str, dict[str, float]] | None = None#

modality: Modality = 'tabular'#

overall_sparsity: float = 0.0#

row_count: int = 0#

row_distribution: RowMissingnessDistribution#

target_correlations: dict[str, CorrelationProfileResult]#

to_dict()[source]#

Serialise the dataset stats to a plain dictionary.

Returns:: All field values with nested objects serialised to their dict representations.
Return type:: dict

was_chunked: bool = False#

class dataforge_ml.profiling._config.NumericKind(*values)[source]#

Bases: StrEnum

BoundedDiscrete = 'bounded_discrete'#

Continuous = 'continuous'#

class dataforge_ml.profiling._base.OverrideCoercionError[source]#

Bases: Exception

Raised when a user-overridden column completely fails coercion.

This exception is triggered when zero usable values remain after coercion, despite the original column having at least one non-null value.