Imputation#

The imputation door is user-orchestrated (ADR-0060, ADR-0071): decide() builds the pure plan, a caller-written fit_unit loop trains it one unit at a time, and FittedImputer.compose rolls the trained units into the whole-dataframe imputer.

The plan has two authors (ADR-0083): decide() derives it from a profile, and author() takes it from a person who knows their own data. The plan either produces is the same object, and everything downstream treats them alike.

author() is also how your own estimator gets onto a plan: author(..., estimators={"mice": my_model}) stamps the MICE block ModelChoice.Custom and trains it with my_model itself — the object you passed, not a clone. Two consequences: mutating it after authoring moves every plan holding it, and it is never serialized, so a saved plan reloads as Custom with the slot empty and raises at fit time until you supply the estimator again.

The two authors also compose: author(map, base=plan) re-authors a plan you already have, so a user who ran decide() and disagrees with two of forty columns changes those two and keeps the thirty-eight the router got right. A column the map does not name keeps its decision verbatim; sentinels, config_snapshot and custom estimators carry over; the decided hyperparameter base is gap-filled for new units and never overwritten for carried ones; and tuning deltas ride along for units that survive the edit but are removed outright for those it dissolved. Re-authoring a deserialized plan works, but the units saved beside it were fitted under the old strategies — the result is a plan to fit fresh, never one to pair with those units.

The guarantee boundary#

A hand-authored plan is the same object as a decided one, so almost everything the library promises holds for it unchanged. What narrows is narrow and specific: the promises that depend on the library having chosen the estimator, and the promises that depend on the door having seen a profile.

These hold for every plan, whichever author wrote it:

Guarantee	Why it still holds
Frame coverage — every column of `columns` gets a decision	By construction: `default` stamps every column the map does not name
Observed-Value Preservation (ADR-0078)	Enforced inside each unit’s own `transform`, below the plan
Composition — `FittedImputer.compose`’s exact-coverage gate (ADR-0071)	Units are a pure projection of `column_decisions`
Execution — `fit_unit`, single-track raise, unit independence	Nothing on the fit path asks who wrote the plan
Persistence of the plan and of each fitted unit (ADR-0072)	Unchanged, except a `Custom` estimator, which is dropped and re-supplied
Sentinel normalisation at fit and at transform	Via `author`’s plan-level `numeric_sentinels` / `string_sentinels`
Dial validation — `with_hyperparameters` against a complete base	The door writes the same decided base `decide()` does, from the same table
`core_budget` arithmetic	For library estimators; see the row below for `Custom`

These hold only where the library chose the estimator, or only where a profile was read:

Guarantee	How it narrows for a hand-authored plan
`core_budget` arithmetic for a `Custom` unit	Priced at `1` — the truthful count of cores the library spends. The budget cannot see parallelism you configured inside your own estimator
Determinism across `n_jobs` (ADR-0069’s RandomForest pinning)	Holds only where the library built the estimator. A foreign estimator’s determinism is its own
Routing rationale (`signals`)	Empty, legally: a hand-authored decision explains no routing, because there was none
Size-guard warnings before fit	None — the door has no frame to measure. A plan that cannot train raises at fit
Reload-and-fit without further input	Not for a `Custom` plan: `build_from_choice` raises until you supply the estimator again
Tuning of the estimator’s own dials	Not for a foreign estimator — the library cannot enumerate dials it does not define

One asymmetry follows from the door having no frame. A manual author gets a hard UnitNotTrainableError at fit where a forced automatic one gets a FitSignals warning: parity would mean handing a decide-time door a frame, and decide-time doors do not touch data.

A self-parallelising estimator#

core_budget prices a Custom unit at 1 in every branch, because the library never sets a foreign estimator’s parameters. If the estimator you supplied parallelises itself — n_jobs, nthread, num_threads, or an internal pool exposed as no parameter at all — the budget is structurally blind to it, and there is deliberately no probe: one would fire on correct plans while missing the spellings it does not know.

The documented pattern is to fit that unit outside your pool, where it has the machine to itself, and to use total_cores as the reservation hatch for the rest:

budget = core_budget(plan, max_workers=n_workers, total_cores=n_cores - reserved)

# The self-parallelising unit runs alone, with the whole machine.
(mice,) = plan.units_for(ImputationStrategy.MICE)
custom = fit_unit(plan, mice, df)

with ThreadPoolExecutor(max_workers=n_workers) as pool:
    rest = [
        pool.submit(fit_unit, plan, unit, df, n_jobs_inner=budget[unit.unit_id])
        for unit in plan.units
        if unit.strategy is not ImputationStrategy.MICE
    ]
    ...

Decision layer#

dataforge_ml.imputation._decision_assembler.decide(profile, n_rows, config=None)[source]#

Build the pure, immutable imputation plan for a profiled dataset.

The single decide-time entry point of the Decision/Execution split (ADR-0060). Routes every profiled column to its imputation strategy, resolves the concrete estimator family for model-based columns, surfaces domain-snap bounds and the indicator/mnar/drop flags, and materialises the execution units — all as a pure function of (profile, shape, config) that trains nothing and never touches a DataFrame.

Every fact about the data is declared here and only here: the bimodal centres and feature_cols are measurements lowered from the profile, and the grouping variable and the constant fill are lowered from config, all onto ColumnImputationDecision (ADR-0083). No fitter reads config for them, so a saved plan fits without the config that produced it — and editing config between decide() and fit_unit() no longer has any effect.

decide resolves its own active column set for the Imputation phase from config (Hard Exclusions plus Imputation-phase Soft Exclusions) — no upstream filtering is relied on. A hard-excluded column is omitted from the plan entirely (recoverable via the config snapshot); a soft-excluded column stays visible as a Passthrough decision carrying a signal that records the exclusion. Excluded columns leave the numeric block wholesale: they are neither targets nor predictors, and every block-derived value — feature counts, MICE/KNN block membership, multi-MAR counting, regression and correlated-feature lists, block hyperparameter signals, and the plan’s decided-for shape — is computed over active columns only.

The plan is decided for the shape execution will face, so n_rows is the train row count, not the profile’s full-dataset count (ADR-0066). It feeds both the size guards and the row-dependent hyperparameters (ADR-0062), so a plan decided against the wrong count can request a model the train split cannot train. A caller genuinely planning against the whole dataset passes profile.dataset.row_count explicitly.

Parameters:

profile (StructuralProfileResult) – Full-dataset Phase 1 profile. Supplies the routing signals, the imputable-column population, and the declared sentinel maps carried onto the plan (ADR-0068).
n_rows (int) – Row count the plan is decided for — the train split’s. Only the row dimension is accepted: splitting is row-wise, so n_features and the column set stay derived from profile.
config (PipelineConfig, optional) – Pipeline configuration supplying the imputation thresholds, MNAR declarations, per-column overrides, and the exclusion lists resolved into the Imputation-phase active column set. Defaults to PipelineConfig() when omitted.

Returns:

The immutable plan: per-column decisions, materialised units, the shape it was decided for, the config snapshot, the declared sentinel maps, and the profile provenance.

Return type:

ImputationDecision

Raises:

ValueError – If n_rows is negative, or if a hard- or Imputation-soft-excluded column is also named in mnar_columns, per_column_strategy, per_column_constant_fill, or add_indicator_columns — contradictory configuration surfaces at decide-time, before any routing, with every offending column reported in one raise.

dataforge_ml.imputation._authoring.author(columns_map, *, columns=None, default=None, estimators=None, numeric_sentinels=None, string_sentinels=None, base=None)[source]#

Build an imputation plan by hand, from column names and declared strategies.

The manual end of the automated↔manual spectrum (ADR-0083), sibling to decide(): both take a description and return an ImputationDecision, one from a profile and one from a person. The returned plan carries a complete decided hyperparameter base written from the same dial table decide reads, so with_hyperparameters(), fit_unit(), compose(), transform, core_budget() and persistence all behave exactly as they do for a decided plan.

No data is touched: columns is a sequence of names, so a plan can be authored in CI against a schema before any frame exists. What the door derives rather than accepts: semantic_type (Numeric, or the base’s own value when re-authoring), indicator_flag and mnar (from strategy == MNAR), drop (from strategy == Dropped), and an {col}_missing Indicator entry per MNAR column. signals is empty on an authored column — there was no routing to explain — and config_snapshot is empty, which truthfully reads back as library defaults. Both come from the base instead when one is given.

A MICE column is stamped with BayesianRidge — decide’s own neutral branch — so the plainest plan an author can write actually trains; override it with with_model_choice(), or hand the block your own tuned estimator through estimators.

Estimators enter here; dials do not. An estimator has no other way onto a plan — with_model_choice takes a label, and there is no with_estimator — whereas dials keep their one grammar, with_hyperparameters(), identical whichever author made the plan (ADR-0083).

Re-authoring. base= hands the door a plan instead of a column universe, so a user who ran decide and disagrees with two of forty columns changes those two and keeps the thirty-eight the router got right. This amends ADR-0082 rather than reopening it: a structural edit still invalidates what is derived downstream of the unit list, but the door re-derives that state — with_strategy stays deleted because it did not. See base for the field-by-field carry-over rules.

What the door cannot check. It sees names, never data, so a strategy/dtype mismatch and every size guard stay fit-time failures. That leaves one accepted asymmetry: a hand-authored unit too small or too typed to train raises a hard UnitNotTrainableError, where the same unit forced onto the automatic path is merely reported through FitSignals as a warning. Parity would mean handing a decide-time door a frame for the sake of one diagnostic. Which library guarantees narrow for a hand-authored plan and which do not is tabulated under “The guarantee boundary” in the imputation API guide; briefly, everything below the plan is unchanged, while routing rationale, pre-fit size warnings, and — for a user-supplied estimator — the n_jobs determinism pinning, the core_budget count and dial tuning do not apply.

Parameters:

columns_map (Mapping[str, ImputationStrategy or AuthoredColumn]) – The columns the author has an opinion about, keyed by column name. A bare strategy declares what happens; an AuthoredColumn declares that plus the facts its fitter reads. Every key must appear in columns.
columns (Sequence[str], optional) – Every column of the frames this plan will be fitted and applied to, in order. Required unless base is given, and keyword-only: transform rejects a frame column the plan never described, so a partial map without the full universe would build an imputer that refuses its own author’s frame.
default (ImputationStrategy, optional, default ImputationStrategy.Passthrough) – Strategy stamped on every column not named in columns_map. Note that it flips which case is silent: under default=Median a forgotten column is quietly imputed rather than left alone. Not accepted with base, where an unnamed column keeps the base’s decision instead.
estimators (Mapping[str, Any], optional) – Sklearn-compatible estimator instances keyed by unit id — in practice {"mice": my_model}, MICE being the only strategy with an estimator slot. Every column of the named unit is stamped Custom and the instance is carried on the plan’s custom_estimators map, unchanged and uncloned. Keyed by unit rather than by column because a per-column spelling would let two estimators be named for one joint block. A pre-fitted estimator is accepted; its state is inert, since IterativeImputer clones and refits from scratch. Two consequences worth knowing: the object is shared, not copied, so mutating it after authoring silently moves every plan holding it; and it is never serialized, so a plan saved and reloaded keeps the Custom label with an empty slot and raises at fit time until the estimator is supplied again. With base, the base’s estimators are carried and this argument overwrites the entry it names.
numeric_sentinels (Mapping[str, Sequence[float]], optional) – Declared numeric sentinel values per column (e.g. -999), normalised to nulls before any fit or transform (ADR-0068). Plan-level rather than per-column, so a sentinel-bearing column the author has no opinion about is still normalised. With base, the base’s map is carried and an entry given here overwrites that column’s.
string_sentinels (Mapping[str, Sequence[str]], optional) – Declared string sentinel values per column. See numeric_sentinels.
base (ImputationDecision, optional) – A plan to re-author. Its columns are the universe, so columns and default are refused alongside it — two sources for one fact, and nothing left to default. Carry-over runs field by field: a column absent from columns_map keeps its decision verbatim (semantic type, model choice, routing signals and all), while a named one is rebuilt from the map and takes only its semantic type from the base; config_snapshot and both sentinel maps are carried verbatim, since a re-authoring user may no longer hold the profile or the config; custom_estimators is carried; the decided hyperparameter base is gap-filled, never overwritten, because decide’s dials are profile-computed and blanket recalculation would be a silent quality regression rather than a loud crash; and tuning deltas are carried for surviving units. Everything belonging to a unit the edit dissolved — a delta, an estimator, a decided base — is removed, not left inert: a dead entry survives to_dict and a save/load and would spring back to life when a column re-enters that unit later. Knowingly kept: two provenances of settings in one plan, and a stale n_neighbors when a block’s membership changes, since a structural edit hands dial responsibility to the author. Nothing records the mix — mixed provenance gets no field. Re-authoring a deserialized plan is allowed and unguarded, but the units saved beside it were fitted under the old strategies: the result is a plan to fit fresh, never one to pair with those units.

Returns:

The immutable plan: one decision per column of the universe (columns or, when re-authoring, the base’s own), the derived MNAR indicator entries, the materialised units, the complete decided hyperparameter base, any supplied estimators, and the sentinel maps.

Return type:

ImputationDecision

Raises:

ValueError – If neither columns nor base is given, or if base is combined with columns or default; if a columns_map key is absent from the column universe; if an entry declares a semantic type other than Numeric; if Indicator is declared; if Constant carries no fill; if a bimodal strategy carries fewer than two centres; if ClusterConditional carries neither a grouping variable nor a non-empty feature_cols; if MICE is declared for exactly one column, which leaves the block a target and no predictor; or if an estimators key names no unit of the plan, names a unit whose strategy has no estimator slot, or carries None.
TypeError – If a columns_map value is neither an ImputationStrategy nor an AuthoredColumn.

class dataforge_ml.imputation._authoring.AuthoredColumn(strategy, center1=None, center2=None, feature_cols=None, grouping_variable=None, constant_fill=None, domain_snap_bounds=None)[source]#

Bases: object

One column’s hand-written imputation declaration.

The richer half of author()’s input: a bare ImputationStrategy says what happens to a column, and this says it alongside the facts about the data that the strategy’s fitter reads — the bimodal centres, the centroid features, the grouping variable, the constant fill, the domain-snap bounds (ADR-0083).

Deliberately not a ColumnImputationDecision: there is no column to restate (the map key is the column), no semantic_type (the door stamps Numeric), no drop, mnar or indicator_flag (all derived from strategy) and no signals (a hand-authored decision explains no routing). Dials are not here either — they keep their one grammar, with_hyperparameters(), whichever author made the plan.

Parameters:

strategy (ImputationStrategy) – The strategy this column takes. Every strategy that answers “what happens to this column?” is allowed; Indicator is refused, being the label of a column that arrives as a consequence of MNAR rather than as a choice.
center1 (float, optional) – First of the two mode centres the GMMSampling and ClusterConditional strategies split on. Required by both.
center2 (float, optional) – Second mode centre. See center1.
feature_cols (tuple[str, ...], optional) – Columns the ClusterConditional centroid branch measures its per-cluster centroids over. Either this or grouping_variable must be supplied for that strategy.
grouping_variable (str, optional) – Column whose groups the ClusterConditional group-wise branch aggregates within. Selects that branch over the centroid one.
constant_fill (float, optional) – The fill value a Constant column takes. Required by that strategy.
domain_snap_bounds (tuple[float, float], optional) – (min, max) a model-based strategy’s predictions are rounded and clipped into, for a column with a bounded discrete domain.

center1: float | None = None#

center2: float | None = None#

constant_fill: float | None = None#

domain_snap_bounds: tuple[float, float] | None = None#

feature_cols: tuple[str, ...] | None = None#

grouping_variable: str | None = None#

strategy: ImputationStrategy#

class dataforge_ml.imputation._config.ImputationDecision(column_decisions, config_snapshot, decided_hyperparameters=<factory>, override_hyperparameters=<factory>, custom_estimators=<factory>, numeric_sentinels=<factory>, string_sentinels=<factory>)[source]#

Bases: object

The pure, immutable imputation plan produced by decide().

The keystone of the Decision/Execution split (ADR-0060): a complete, inspectable, editable description of what will happen to every column, derived purely from (profile, shape, config) and holding no value the execution layer learns from training data. The execution layer consumes it, persistence serialises it, and the user can inspect and edit it before anything trains.

The plan is immutable: with_model_choice() and with_hyperparameters() return a new ImputationDecision rather than mutating in place, and units is re-derived from column_decisions at every construction, so a stale unit list is structurally impossible and every plan that exists is valid by construction.

Those edits are dial edits: they change how a unit trains, never which units exist, so they cannot invalidate the decided hyperparameter base. Changing which strategy a column takes is a structural edit and belongs to decide() alone, declared through per_column_strategy (ADR-0082).

Parameters:

column_decisions (dict[str, ColumnImputationDecision]) – Per-column plan entries keyed by column name, in decision order. Held as an internal copy so the constructed plan is independent of the caller’s mapping.
config_snapshot (dict) – Serialised PipelineConfig (config.to_dict()) the plan was decided under.
decided_hyperparameters (dict[str, tuple[tuple[str, Any], ...]]) – The decide-time hyperparameter base per unit id, written by the authoring function and never by an edit (ADR-0073). Complete for each unit’s strategy: it carries every dial the strategy has in _STRATEGY_DIALS, an invariant from_dict() re-establishes on load, plus any profile facts the unit’s fitter reads.
override_hyperparameters (dict[str, tuple[tuple[str, Any], ...]]) – The sparse per-unit override delta, written only by with_hyperparameters(). _derive_units stamps the per-key merge decided ⊕ delta onto each unit, so the two-map split is invisible below the plan surface (ADR-0073).
custom_estimators (dict[str, Any]) – User-supplied estimator instances keyed by unit id, filled by author()’s estimators= channel and paired with ModelChoice.Custom on the unit’s columns (ADR-0083). The caller’s own object, never a clone, here and on every derived copy: cloning would break identity, make get_params a construction-time requirement stricter than sklearn’s own, and silently strip fitted state. Executing a plan cannot mutate it — IterativeImputer clones per column — so the only live hazard is deliberate post-authoring mutation, which is closed by this paragraph rather than by a guard. Unit-keyed and not per-column: MICE is the only strategy with an estimator slot, so a per-column spelling would let two estimators be named for one block. Not serialised — to_dict() drops it and the plan reloads as Custom with the slot empty. It stays in ==, so a Custom plan compares unequal to its own round trip; that is the truth, since the restored plan cannot fit. Nothing learned is lost: the estimator is a line of the user’s own code.
numeric_sentinels (dict[str, list[float]]) – Declared numeric sentinel values per column, carried from the source profile so the execution layer can normalise effective nulls without it (ADR-0068).
string_sentinels (dict[str, list[str]]) – Declared string sentinel values per column, carried from the source profile for the same reason as numeric_sentinels (ADR-0068).
units (tuple[ImputationUnit, ...]) – Materialised execution units, re-derived from column_decisions at construction. Not an init argument.
dropped_columns (tuple[str, ...]) – Convenience projection of the columns routed to Dropped. Not an init argument.

Notes

column_decisions is the single source of truth; units and dropped_columns are always projections of it and are never set directly.

A plan carrying custom_estimators may hold a pre-fitted estimator: the library tolerates it and never reads its state (sklearn refits from scratch), and no guard refuses it, because any check would recognise only sklearn’s trailing-underscore spelling and would read as a guarantee it is not. ADR-0060’s value-free claim therefore narrows from a structural guarantee to a statement about the library: the library never writes learned state onto a plan.

column_decisions: dict[str, ColumnImputationDecision]#

config_snapshot: dict#

custom_estimators: dict[str, Any]#

decided_hyperparameters: dict[str, tuple[tuple[str, Any], ...]]#

dropped_columns: tuple = ()#

classmethod from_dict(data)[source]#

Reconstruct an ImputationDecision from a plain dictionary.

The inverse of to_dict(). column_decisions is the single source of truth: units and dropped_columns are re-derived, never read from data, so from_dict(plan.to_dict()) is structurally equal to plan even if the serialised unit list was tampered with.

The decided base is gap-filled against _STRATEGY_DIALS on the way in — a dial the payload is missing takes the table’s neutral value, and a dial the payload carries is never overwritten. “Complete for the unit’s strategy” is thereby true of the type rather than of the two authoring functions, which is what lets a fitter subscript a dial outright. Cost taken knowingly: an artifact saved before a dial existed silently gains it rather than dying on a bare KeyError.

Parameters:: data (dict) – Mapping produced by to_dict().
Returns:: Reconstructed plan instance.
Return type:: ImputationDecision

numeric_sentinels: dict[str, list[float]]#

override_hyperparameters: dict[str, tuple[tuple[str, Any], ...]]#

string_sentinels: dict[str, list[str]]#

to_dict()[source]#

Serialise the plan to a plain, JSON-friendly dictionary.

units and dropped_columns are emitted for readability but are re-derived — not consumed — on from_dict(), so a hand-edited unit list can never desynchronise a reloaded plan.

custom_estimators is omitted: a user-supplied estimator is a live object, not data, and the payload stays JSON-native (ADR-0083). The reloaded plan keeps ModelChoice.Custom with an empty slot, and fitting it raises rather than falling back to a library estimator. Rejected alternatives: refusing to serialize such a plan at all, and pickling the estimator into the envelope.

Returns:: The plan’s fields with nested objects serialised to dicts.
Return type:: dict

to_markdown()[source]#

Render the whole plan as a Markdown document.

A document per rule 5 of the Rendering Contract (ADR-0086): it owns the # and ## heading levels and delegates to the ColumnImputationDecision and ImputationUnit fragments beneath them. Every field of the plan is covered, so a plan from decide() and one built by hand through author() render identically in shape and are comparable by eye.

custom_estimators holds live user objects rather than data, so it is reported by unit id and estimator class name only.

Returns:: Markdown document with a summary, the per-column decisions, the execution units, the declared sentinels, and the config snapshot the plan was decided under.
Return type:: str

units: tuple = ()#

units_for(strategy)[source]#

Return the plan’s units that execute strategy.

The selection surface for a caller driving their own fit loop: it replaces scanning units against a hand-typed unit_id literal, so the id stays an internal name the caller never spells.

Always returns a tuple — empty when the plan routed nothing to strategy, which is an ordinary outcome rather than an error. A strategy that is structural, or that no column reached, therefore needs no guard at the call site: the loop simply runs zero times. MICE and KNN yield at most one unit each, being joint blocks.

Parameters:: strategy (ImputationStrategy) – The strategy to select on.
Returns:: The matching units, in plan order. Empty if there are none.
Return type:: tuple[ImputationUnit, …]

with_hyperparameters(unit_id, hyperparameters)[source]#

Return a new plan overriding named hyperparameters on one unit.

A per-key merge onto the unit’s always-complete decided base (ADR-0073): the named keys are overridden and every other decided dial is kept, so the plan the user inspects is exactly the plan that fits. The override is stored as a sparse delta; units re-derives the merge at construction. Successive edits to the same unit accumulate per key. Passing None resets the unit to its decided values by clearing its delta. Per-unit only: a joint unit shares one estimator, so per-column overrides on it would be meaningless.

Every key must be one of the unit’s strategy’s dials (_STRATEGY_DIALS) — the same table both authors build the decided base from, so what is dialable is one fact rather than a per-author allowlist. A strategy with no row has no dials, and every key is rejected for it. The decided base may carry more than the dials (profile facts a fitter reads, such as a bimodal unit’s centres); those are not dialable. Values are not type-checked: sklearn rejects an ill-typed value at fit time.

Parameters:

unit_id (str) – ID of the unit to override (e.g. "mice", "knn", or "median:age"). Must already be present in the plan.
hyperparameters (dict[str, Any] or None) – The keys to override and their replacement values, or None to reset the unit to its decided base.

Returns:

A new plan with the override applied.

Return type:

ImputationDecision

Raises:

KeyError – If unit_id is not part of the plan’s derived units.
ValueError – If hyperparameters names a key that is not one of the unit’s strategy’s dials, identifying the unit and the offending key.

with_model_choice(column, model_choice)[source]#

Return a new plan overriding column’s estimator family.

The plan is immutable; this builds a fresh ImputationDecision with the one column’s model_choice replaced and every unit re-derived so the change is visible on the affected unit.

Parameters:

column (str) – Column whose estimator family to override. Must already be present in the plan.
model_choice (str or ModelChoice or None) – The replacement estimator family, or None to clear it.

Returns:

A new plan with the edit applied.

Return type:

ImputationDecision

Raises:

KeyError – If column is not part of the plan.

class dataforge_ml.imputation._config.ColumnImputationDecision(column, semantic_type, strategy, signals=(), model_choice=None, domain_snap_bounds=None, center1=None, center2=None, feature_cols=None, grouping_variable=None, constant_fill=None, indicator_flag=False, mnar=False, drop=False)[source]#

Bases: object

Pure, value-free per-column imputation plan entry.

The decided half of the Decision/Execution split (ADR-0060): a complete, inspectable description of what will happen to one column and why, derived purely from (profile, shape, config). It structurally cannot hold anything the execution layer learns from training data — no fill value, no fitted model, no convergence/n_iter — so “what was decided” is un-blurrably separate from “what was learned”. ColumnImputationRecord composes this decision with those learned values.

The dataclass is frozen and every field holds an immutable value (enums, str, bool, or hashable tuples), so a decision is safe to share and edits produce new objects rather than mutating in place.

Parameters:

column (str) – Column name this decision applies to.
semantic_type (SemanticType) – Detected semantic type of the column, carried from the profile.
strategy (ImputationStrategy) – Strategy routed for this column.
signals (tuple[str, ...], optional) – Human-readable routing rationale — the reasons that drove strategy. A tuple so the decision stays immutable.
model_choice (ModelChoice, optional) – Concrete estimator family resolved at decide-time for model-based strategies. None for scalar strategies and for columns whose nonlinearity is Unpredictable. A value-free label, never a live or fitted estimator.
domain_snap_bounds (tuple[float, float], optional) – (min, max) bounds used to snap model-based predictions for BoundedDiscrete columns. None for all other columns. Sourced from the profile, not learned from training data.
center1 (float, optional) – First of the two bimodal cluster centres the GMM-Sampling and Cluster-Conditional strategies split on. Measured by Phase 1, not learned from training data. None for non-bimodal columns.
center2 (float, optional) – Second bimodal cluster centre. See center1.
feature_cols (tuple[str, ...], optional) – Columns the Cluster-Conditional centroid branch measures its per-cluster centroids over — the features correlated with this column at decide-time. None for every other strategy.
grouping_variable (str, optional) – Column whose groups the Cluster-Conditional group-wise branch aggregates within, when one was declared. None selects the centroid branch.
constant_fill (float, optional) – The declared fill value for a Constant column. None for every other strategy. A declared value, never a learned one.
indicator_flag (bool) – Whether a binary missingness indicator column will be appended.
mnar (bool) – Whether the column is routed as Missing-Not-At-Random.
drop (bool) – Whether the column will be dropped for exceeding the drop threshold.

Notes

Fill values, fitted coefficients, convergence counts, and any other training-derived state deliberately have no home on this type; they live on the execution layer’s fitted units (ADR-0060).

center1: float | None = None#

center2: float | None = None#

column: str#

constant_fill: float | None = None#

domain_snap_bounds: tuple[float, float] | None = None#

drop: bool = False#

feature_cols: tuple[str, ...] | None = None#

classmethod from_dict(data)[source]#

Reconstruct a ColumnImputationDecision from a plain dictionary.

The inverse of to_dict(): enums are rebuilt by member name and list fields are restored to their immutable tuple form so the reconstructed decision is structurally equal to the original.

Parameters:: data (dict) – Mapping produced by to_dict().
Returns:: Reconstructed decision instance.
Return type:: ColumnImputationDecision

grouping_variable: str | None = None#

indicator_flag: bool = False#

mnar: bool = False#

model_choice: ModelChoice | None = None#

semantic_type: SemanticType#

signals: tuple[str, ...] = ()#

strategy: ImputationStrategy#

to_dict()[source]#

Serialise the decision to a plain dictionary.

Enums are rendered by member name (per ADR-0063, so reordering an enum later cannot silently corrupt a saved plan) and tuples as lists, so the result is JSON-friendly and structurally round-trippable.

Returns:: All field values keyed by field name.
Return type:: dict

to_markdown()[source]#

Render the column decision as a ###-rooted Markdown fragment.

A fragment per rule 5 of the Rendering Contract (ADR-0086): it carries no # or ## heading, so the owning plan document composes it without a heading collision. Every field is covered; absent values are stated rather than left as a bare None, and enums render by their string form.

Returns:: Markdown subsection headed by ### `<column>` and a field table.
Return type:: str

class dataforge_ml.imputation._config.ImputationUnit(unit_id, strategy, columns, hyperparameters=None, is_block=False)[source]#

Bases: object

One executable unit of an imputation plan.

The plan’s materialised, re-derived projection of what execution trains together (ADR-0060): the joint MICE block, the joint KNN block, and one unit per independent column (GMM-Sampling / Cluster-Conditional / scalar). Structural strategies (Dropped / Passthrough / Indicator) train nothing and produce no unit. A unit is a value-free projection — it names the columns and the estimator family, never a fitted model; the execution layer keys its checkpoints against unit_id.

Parameters:

unit_id (str) – Stable identifier: "mice" / "knn" for the joint blocks (exactly one of each when present), "{strategy}:{column}" for a per-column unit.
strategy (ImputationStrategy) – Strategy the unit executes.
columns (tuple[str, ...]) – Columns trained by this unit — every column in the block for the joint units, a single-element tuple for a per-column unit.
is_block (bool, default False) – Whether the unit’s columns train together in one joint model call — True for the MICE and KNN blocks, False for every per-column unit. Structural only: it says nothing about how expensive the unit is, nor whether it can absorb inner parallelism (that follows from model_choice, which stays on ColumnImputationDecision). Derived from strategy, so a caller never supplies it.
hyperparameters (tuple[tuple[str, Any], ...] | None)

Notes

The unit is value-free and estimator-agnostic: the concrete estimator family a model-based column trains with lives on its ColumnImputationDecision (model_choice), reachable through the owning plan’s column_decisions, so it is never duplicated here.

columns: tuple[str, ...]#

classmethod from_dict(data)[source]#

Reconstruct an ImputationUnit from a plain dictionary.

Parameters:: data (dict) – Mapping produced by to_dict().
Returns:: Reconstructed unit instance. is_block is re-derived from strategy, so whatever the payload carries for it — a stale value, a wrong value, or nothing at all — is ignored.
Return type:: ImputationUnit

hyperparameters: tuple[tuple[str, Any], ...] | None = None#

is_block: bool = False#

strategy: ImputationStrategy#

to_dict()[source]#

Serialise the unit to a plain dictionary.

Returns:: All field values keyed by field name, the strategy enum rendered by member name and columns as a list.
Return type:: dict

to_markdown()[source]#

Render the execution unit as a ###-rooted Markdown fragment.

A fragment per rule 5 of the Rendering Contract (ADR-0086): it carries no # or ## heading, so the owning plan document composes it without a heading collision. Every field is covered, including the merged hyperparameters stamped onto the unit at plan construction.

Returns:: Markdown subsection headed by ### `<unit_id>` and a field table.
Return type:: str

unit_id: str#

Execution layer#

dataforge_ml.imputation._unit_fit.fit_unit(decision, unit, df, random_seed=None, n_jobs_inner=-1)[source]#

Train exactly one planned unit and bundle it into a UnitFitResult.

The one training primitive of the user-orchestrated flow: batch scheduling is user-owned (ADR-0075), so training a whole plan is a caller-written loop of fit_unit calls. The frame is normalised off the plan’s declared sentinel maps (ADR-0068) before the fit, so a raw frame may be handed straight in.

Concurrency — budget the inner layer when you parallelise the loop. The caller owns the outer/inner split (ADR-0069): parallelism lives in exactly one layer, and n_jobs_inner is how you say which. A sequential loop — one fit_unit at a time — takes the default n_jobs_inner=-1, which opens each fit’s inner sklearn parallelism to every core. A self-parallelised drive — units fitted side by side in your own thread pool — calls core_budget() once before the loop and passes each unit’s value here; a drive that parallelises and passes nothing leaves every concurrent fit fanning out to every core, so the two layers oversubscribe the machine. The value never moves the result (ADR-0069); it only changes how the cores are spent.

Parameters:

decision (ImputationDecision) – The immutable plan. Supplies the unit recipe, the per-column decisions a fitter honours, the config snapshot, and the sentinel maps used to normalise df.
unit (ImputationUnit) – The unit to train, taken from the plan’s units or from ImputationDecision.units_for(). Only its unit_id is read: the recipe is re-resolved against decision, so the plan always wins over a unit held from before an edit.
df (pl.DataFrame) – Training data, raw or already normalised. A working copy is taken at entry: effective nulls are resolved and the Dtype Floor enforced against the plan’s semantic types (ADR-0085). The caller’s frame is never mutated.
random_seed (int, optional) – Seed for the stochastic strategies (GMM sampling).
n_jobs_inner (int, default -1) – Inner estimator n_jobs (ADR-0056). Leave at -1 for a sequential drive; when your own loop already fits units concurrently, pass this unit’s value from core_budget() (ADR-0081). Never affects results.

Returns:

The trained unit plus its structured fit record. The record’s duration_s spans this whole call, normalisation included.

Return type:

UnitFitResult

Raises:

KeyError – If unit names no unit in the plan.
TypeError – If unit is a unit id string rather than an ImputationUnit — the pre-4.x signature.
UnitNotTrainableError – If the unit cannot train, carrying its structured payload — the sole failure track (ADR-0071).

dataforge_ml.imputation._unit_fit.core_budget(decision, max_workers, total_cores=None)[source]#

Compute the whole-plan inner-parallelism budget: unit_id -> n_jobs_inner.

The library owns this arithmetic and nothing else (ADR-0081): the pool, the loop, the submission order and the failure policy stay user-owned (ADR-0075 is unamended). Call it once immediately before your own fit_unit() loop and pass each unit’s value as that call’s n_jobs_inner.

The arithmetic is heavy-aware rather than degree-proportional. A plan holds at most two joint blocks, and at most one unit — the "mice" block, and only when its model_choice is RandomForestRegressor — can absorb inner parallelism at all, so the MICE block receives max(1, total_cores - 1_if_knn_present) and every other unit receives 1. Dividing the cores evenly across the outer degree instead would hand the MICE block a 1 and leave the machine idle.

A free function and not a property on ImputationDecision: the plan is derived purely from (profile, shape, config), and hanging a machine fact on it would make the same serialized plan answer differently on a different box (ADR-0072’s precedent). It answers for the whole plan at once because the reserved-for-KNN term is a fact about the plan, not about any one unit.

Parameters:

decision (ImputationDecision) – The plan to price. Read-only — units, and column_decisions for the MICE block’s model_choice.
max_workers (int or None) – The outer degree of the drive the budget is for. 1 is a sequential one-unit-at-a-time drive, which is outer-degree-one and gets the wide -1 for MICE (ADR-0069). Anything greater is a parallel pool, and None — a ThreadPoolExecutor of unknown degree — takes that same parallel branch rather than being read as sequential or rejected.
total_cores (int, optional) – Overrides core detection, for a caller subdividing a box across processes. Detection is joblib.cpu_count(), which respects cgroup quotas and is the same detector sklearn uses for its own n_jobs=-1, so the budget and sklearn’s actual fan-out agree on how big the box is.

Returns:

A mapping whose keys are exactly the plan’s unit ids and whose values are the n_jobs_inner each unit should be fitted with.

Return type:

dict[str, int]

Notes

All ones is a legitimate answer. BayesianRidge and GradientBoostingRegressor have no n_jobs to spend (ADR-0069), so a MICE block routed to either is priced at 1, with no error and no warning — 1 is the truthful number, and a warning would fire on a correct plan. On default config this covers every ComplexNonlinear frame at or above gradient_boost_min_rows, which routes to GradientBoostingRegressor: there is no speedup to be had there. The condition is branch-specific, not a frame-size ceiling — MonotonicNonlinear takes RandomForestRegressor at any row count and stays able to spend a budget.

A user-supplied estimator is priced at ``1`` in every branch, the sequential max_workers=1 one included (ADR-0083). The library never sets a foreign estimator’s parameters, so 1 is the truthful count of cores the library spends; total_cores remains the hatch for reserving the rest. There is no n_jobs probe: it would fire on correct plans while missing nthread, num_threads, and every internal pool that is not exposed as a parameter.

The budget is therefore structurally blind to parallelism configured inside a foreign object, and the documented pattern for one is to fit that unit outside your pool, where it has the machine to itself, driving the remaining units through the pool on this budget. total_cores is the reservation hatch: pass it reduced by whatever the self-parallelising fit will take, so the two never contend for the same cores.

The budget is a snapshot. It describes the plan as it was when the call returned. A plan edit in between — ImputationDecision.with_model_choice(), ImputationDecision.with_hyperparameters() — or a fresh decide() silently invalidates it, and nothing detects that. The mitigation is placement: compute the budget immediately before the loop that consumes it, not enforcement.

class dataforge_ml.imputation._unit_fit.UnitFitResult(unit_id, strategy, columns, fitted, signals)[source]#

Bases: object

A trained unit bundled with a structured record of the fit.

What fit_unit() returns: the fitted unit that can transform its own columns, alongside the plan-carried facts that identify it (unit_id, strategy, columns) and the structured FitSignals the fit produced (ADR-0074).

Parameters:

unit_id (str)
strategy (ImputationStrategy)
columns (tuple[str, ...])
fitted (FittedUnit)
signals (FitSignals)

unit_id#

The plan id of the trained unit.

Type:: str

strategy#

The strategy that was executed.

Type:: ImputationStrategy

columns#

The columns the unit owns.

Type:: tuple[str, …]

fitted#

The trained unit; transforms its own columns with no completeness requirement.

Type:: FittedUnit

signals#

The structured, ephemeral record of the fit — estimator, convergence, duration, notes, and any warnings. Read it to answer “did this fit converge? which estimator ran? how long did it take?” without parsing strings.

Type:: FitSignals

class dataforge_ml.imputation._fit_signals.FitSignals(unit_id, strategy, estimator=None, converged=None, n_iter=None, duration_s=0.0, warnings=<factory>, notes=<factory>)[source]#

Bases: object

Structured record of one imputation unit’s fit (ADR-0074).

The observability value fit_unit() returns on every UnitFitResult, letting the caller answer “did this fit converge? which estimator ran? how long did it take?” without parsing strings. A typed core carries the facts any consumer branches on; notes is a free-form bag for strategy-specific minutiae that earn no typed field, kept separate from warnings so the cheap question “did anything go wrong?” is answered by reading warnings alone.

The record is ephemeral — it describes the fit’s runtime, not the fitted state, and is never serialized (ADR-0072). An untrainable unit raises UnitNotTrainableError and produces no record: observability describes fits that happened (ADR-0074).

Parameters:

unit_id (str)
strategy (ImputationStrategy)
estimator (str | None)
converged (bool | None)
n_iter (int | None)
duration_s (float)
warnings (tuple[str, ...])
notes (tuple[str, ...])

unit_id#

The plan id of the trained unit.

Type:: str

strategy#

The strategy that was executed.

Type:: ImputationStrategy

estimator#

Human-readable name of the estimator that ran (e.g. "RandomForestRegressor", "KNNImputer"); None for strategies that fit no estimator, such as the scalar fills.

Type:: str or None

converged#

Whether an iterative model stopped before its iteration cap; None for strategies with no iterative-convergence notion.

Type:: bool or None

n_iter#

The iteration count an iterative model settled on; None when there is none.

Type:: int or None

duration_s#

Wall-clock seconds the fit took, measured with perf_counter. Spans the whole fit_unit() call, normalisation included.

Type:: float

warnings#

Actionable fit-time warnings — a hit iteration cap, or a strategy forced past its routing threshold. Each is also raised through the standard warnings channel under ImputationFitWarning (ADR-0074).

Type:: tuple[str, …]

notes#

Informational fit-time detail (resolved dials, scaling, the routing branch taken) that does not earn a typed field.

Type:: tuple[str, …]

class dataforge_ml.imputation._fitted_imputer.FittedUnit(*args, **kwargs)[source]#

Bases: Protocol

Standalone fitted imputation unit that can transform its own columns.

This unit can transform its own columns immediately, with no completeness requirement (it carries only the ‘my own columns absent’ contract).

A trained unit always executed the strategy the plan asked for: a unit that cannot train raises UnitNotTrainableError rather than becoming a divergent artifact (single-track failure, ADR-0071). The strategy the unit executed is a decide-time fact the plan holds under the same unit_id, so a fitted unit carries only its fitted state (ADR-0074) and does not restate it; a fit’s runtime observability rides on the FitSignals the fit returned.

Observed-Value Preservation (ADR-0078). A unit fills holes and never edits a value the user supplied: after transform, every cell that was not missing in the input comes back bit-for-bit with its original dtype. “Missing” is the Effective Null predicate — null, NaN, or infinite for float columns; null alone otherwise. The guarantee is a property of the unit itself, so it holds when a bare unit is transformed directly with no FittedImputer involved (the stateless door, ADR-0071).

On the stateless door, the caller owns sentinel normalisation. A bare unit resolves float NaN/Inf as missing with no configuration, but it cannot resolve sentinel-encoded Effective Nulls (e.g. -999): the declared sentinel maps live on the ImputationDecision (ADR-0068), and a bare unit does not hold them. Normalise sentinels before calling a bare unit’s transform, or transform through the composed FittedImputer.transform(), which normalises off the plan’s maps.

property target_columns: list[str]#

The columns this unit owns — the ones its transform may fill.

A unit’s targets are not its inputs: a regression unit reads every feature column but owns only the one column it predicts. Every column is routed to exactly one strategy and therefore to exactly one unit, so targets are disjoint across units — which is what lets FittedImputer.transform() merge unit outputs in any order and get the same frame (ADR-0070).

Returns:: The owned column names.
Return type:: list[str]

transform(df)[source]#

Apply this unit’s imputation to the DataFrame.

The unit reads whatever features it needs from df but only its target_columns are meaningful in the result; the caller merges those and discards the rest (ADR-0070).

Observed-Value Preservation (ADR-0078): only the cells that were missing in df receive fill values; every observed cell of the target columns returns bit-for-bit with its original dtype.

Parameters:: df (pl.DataFrame) – The DataFrame to transform.
Returns:: The transformed DataFrame.
Return type:: pl.DataFrame

class dataforge_ml.imputation._fitted_imputer.FittedImputer(records=<factory>, units=<factory>, numeric_sentinels=<factory>, string_sentinels=<factory>, random_seed=None)[source]#

Bases: object

Stores the structural manifest and fitted units; applies them to any DataFrame.

Assembled by compose() from a plan and its trained units (ADR-0071).

Parameters:

records (dict[str, ColumnImputationRecord]) – The structural manifest: one record per column the plan describes. Drives the whole-frame concerns a single unit has no standing to own — the schema guards, the Dropped / Passthrough / Indicator projection — and carries each scalar column’s learned fill_value, which transform applies directly.
units (list[FittedUnit]) – The trained model-based units (KNN, MICE, and the bimodal strategies) in the plan’s application order (ADR-0067), never thread-completion order. Scalar fills live on records instead. The order is not load-bearing on the result: transform applies every unit against one shared pre-model snapshot and merges each unit’s own columns back, and those columns are disjoint across units (ADR-0070); it is kept because it is the order a human reads the plan in.
numeric_sentinels (dict[str, list[float]]) – Per-column numeric sentinel declarations copied from StructuralProfileResult.numeric_sentinels. Keys are column names; values are float-compatible sentinel values that are normalized to Polars-native null before any imputation operation. Defaults to an empty dict — columns with no declaration are completely unaffected. Carried from the plan by compose().
string_sentinels (dict[str, list[str]]) – Per-column string sentinel declarations copied from StructuralProfileResult.string_sentinels. Uses replace semantics: when a column name is present, only the declared values are matched (case-insensitive) and the hardcoded defaults ("NA", "NAN", "NULL", "NONE", "?") are suppressed for that column. Empty/whitespace strings are always treated as effective null regardless of any declaration. Defaults to an empty dict — columns with no declaration continue to use the hardcoded defaults. Carried from the plan by compose().
random_seed (int | None)

apply_exclusions(config)[source]#

Propagate dropped and indicator columns into the pipeline config.

Hard Exclusions (ADR 0023): columns recorded with ImputationStrategy.Dropped are added to config.exclude_columns via add_exclusions, removing them from every downstream phase.

Soft Exclusions: columns recorded with ImputationStrategy.Indicator are registered in config.phase_exclusions for Phases 3–6 (OutlierDetection, Normalization, Encoding, Scaling), so those phases skip indicator columns without removing them from the dataset.

Safe to invoke unconditionally — it is a no-op when no dropped or indicator columns exist, so callers need not branch.

Propagation is caller-initiated: fitting does not touch PipelineConfig, preserving re-fit idempotency. A fresh call is required against any config that has not already received this imputer’s exclusions, including after rehydrating via compose(). Duplicate calls are safe — both hard and soft exclusion registrations deduplicate automatically. Nothing verifies that this method was called; enforcement is deferred to Phase 3 (ADR-0076).

Parameters:: config (PipelineConfig) – Pipeline config to update.
Return type:: None

classmethod compose(decision, results)[source]#

Assemble trained units into a whole-frame imputer (ADR-0071).

The named constructor of the user-orchestrated flow: given the plan and the units the caller’s fit_unit() loop trained (batch scheduling is user-owned, ADR-0075), build the aggregate that holds the whole frame. compose owns exactly the concerns a single unit cannot — the full-schema manifest every column is looked up in (and with it the whole-frame safety guards), and the structural projection of the Dropped / Passthrough / Indicator columns, which have no unit precisely because they learn nothing.

Coverage must be exact: the supplied units must cover the plan’s units one-for-one — no missing unit (which would leave a column unfilled) and no extra one (which describes no column the plan planned).

Parameters:

decision (ImputationDecision) – The plan the units were trained against. Supplies the column decisions, the unit order, and the sentinel maps.
results (Mapping or Iterable) – The trained units, tolerant of either raw fitted units or UnitFitResult bundles, given as a mapping keyed by unit id or as a plain iterable. A raw fitted unit is matched to its plan unit by the columns it owns.

Returns:

The aggregate: the structural manifest, the trained units in plan order, and the plan’s sentinel maps.

Return type:

FittedImputer

Raises:

ValueError – If the supplied units do not exactly cover the plan’s units, or a raw fitted unit matches no plan unit.

numeric_sentinels: dict[str, list[float]]#

random_seed: int | None = None#

records: dict[str, ColumnImputationRecord]#

string_sentinels: dict[str, list[str]]#

transform(df)[source]#

Apply train-time fill parameters and models to df.

Sentinel-encoded Effective Nulls are normalised off the plan’s declared maps before any fill (ADR-0068), and each model-based unit’s Observed-Value Preservation (ADR-0078) guarantees that only cells missing in the normalised frame receive fill values. A Passthrough column is left completely alone, nulls included (ADR-0083).

The Dtype Floor is enforced on the working copy immediately after that normalisation (ADR-0085), so every consumer inside this call reads a column at the dtype its semantic type names. It stops at the boundary: the columns it casts come back at their input dtype, observed cells bit-for-bit (ADR-0078).

Parameters:

df (pl.DataFrame) – The DataFrame to impute. Its columns must be a subset of the train-time schema manifest.

Returns:

The imputed DataFrame together with the per-column records and the list of dropped columns.

Return type:

ImputationResult

Raises:

UnseenColumnError – If df contains any column absent from self.records. Fires before any DataFrame mutations regardless of whether the unknown column has missing values. All unknown column names are reported in one raise.
FittedColumnAbsentError – If a column with an active imputation strategy (any strategy other than Dropped or Indicator) is absent from df. All absent column names are reported in one raise.

Warns:

DroppedColumnAbsentWarning – If a column recorded as Dropped during fit() is already absent from df. One warning is emitted per absent column. Transform continues normally.

units: list[Any]#

Configuration#

class dataforge_ml.imputation._config.ImputationConfig(numeric=<factory>)[source]#

Bases: object

Cross-type Phase 2 configuration.

Parameters:

numeric (NumericImputationConfig) – Thresholds and fill values for numeric imputation.
mnar_columns (list[str]) – Columns declared by the user as Missing Not At Random. These receive a data-derived fill (observed mean or median, skew-driven) plus a binary missingness indicator, regardless of Phase 1 signals.
add_indicator_columns (list[str]) – Columns for which a binary missingness indicator should be added even when they are not MNAR.

Raises:

ValueError – If any column appears in both mnar_columns and numeric.per_column_strategy. These declarations are mutually exclusive: mnar_columns applies a data-derived fill plus an indicator; per_column_strategy directs the routing engine to a user-specified strategy. Declaring the same column in both is contradictory and is caught at construction time before any data is touched.

add_indicator_column(column)[source]#

Force a binary missingness indicator for one or more columns.

Parameters:: column (str | list[str]) – Column name or list of column names.
Return type:: None

property add_indicator_columns: tuple[str, ...]#

Get the columns for which a binary missingness indicator should be added.

Returns:: Columns for which a binary missingness indicator is forced.
Return type:: tuple[str, …]

add_mnar_column(column)[source]#

Declare one or more columns as Missing Not At Random.

Parameters:: column (str | list[str]) – Column name or list of column names to mark as MNAR.
Raises:: ValueError – If any specified column already has a strategy in numeric.per_column_strategy.
Return type:: None

classmethod from_dict(data)[source]#

Construct an ImputationConfig from a plain dictionary.

Parameters:: data (dict) – Mapping produced by to_dict(). Missing keys fall back to field defaults.
Returns:: Reconstructed config instance.
Return type:: ImputationConfig

property mnar_columns: tuple[str, ...]#

Get the columns declared as Missing Not At Random.

Returns:: Columns declared by the user as MNAR.
Return type:: tuple[str, …]

numeric: NumericImputationConfig#

to_dict()[source]#

Serialise the config to a plain dictionary.

Returns:: All field values keyed by field name, with numeric nested.
Return type:: dict

validate()[source]#

Validate the configuration for cross-field conflicts.

Raises:: ValueError – If any column appears in both mnar_columns and numeric.per_column_strategy.
Return type:: None

class dataforge_ml.imputation._config.NumericImputationConfig(knn_max_rows=50000, knn_max_features=50, mice_min_rows=500, gradient_boost_min_rows=10000, base_max_iter=10, knn_min_neighbors=5, knn_max_neighbors=25, knn_distance_weight_max_null_ratio=0.15, knn_distance_weight_max_features=30, mice_n_nearest_features_min_cols=10, mice_max_nearest_features=20, mice_correlation_threshold=0.1, mcar_feature_predictability_threshold=0.2, _per_column_strategy=<factory>, _per_column_constant_fill=<factory>, _bimodal_grouping_variables=<factory>, bimodal_min_correlated_features=3, bimodal_correlation_threshold=0.2, max_workers=None)[source]#

Bases: object

Operational thresholds for the numeric imputation sub-processor.

Parameters:

knn_max_rows (int) – Maximum number of rows before KNN is skipped in favour of MICE.
knn_max_features (int) – Maximum number of features before KNN is skipped in favour of MICE.
mice_min_rows (int) – Minimum number of rows required to fit a stable MICE (chained-equations) model. Applied as a uniform floor to every routing path that can enter the joint MICE block; a column below this floor diverts to KNN, then Median, instead (ADR-0079). Renamed from regression_min_rows.
gradient_boost_min_rows (int) – Row count threshold above which GradientBoostingRegressor is preferred over RandomForestRegressor for ComplexNonlinear columns. Below this threshold the cheaper RandomForestRegressor is used instead.
base_max_iter (int) – Base number of IterativeImputer iterations before dynamic signal adjustments are applied. Increase this value for columns that exhibit convergence warnings in ColumnImputationRecord.signals.
knn_min_neighbors (int) – Floor on the adaptively computed n_neighbors value passed to KNNImputer. The computed k will never fall below this value.
knn_max_neighbors (int) – Cap on the adaptively computed n_neighbors value passed to KNNImputer. The computed k will never exceed this value.
knn_distance_weight_max_null_ratio (float) – Feature-matrix missingness fraction below which distance weighting is considered reliable. When miss_frac exceeds this threshold, weights is forced to "uniform".
knn_distance_weight_max_features (int) – Dimensionality threshold below which distance weighting is considered reliable. When the number of KNN feature columns exceeds this value, weights is forced to "uniform".
mice_n_nearest_features_min_cols (int) – MICE block size at or below which n_nearest_features is left unset, meaning all columns in the block are used as predictors for every imputation target. Above this threshold, n_nearest_features is derived from value-level Pearson correlations.
mice_max_nearest_features (int) – Upper cap on the n_nearest_features value computed for large MICE blocks. The correlation-derived count is clamped to this maximum before being passed to IterativeImputer.
mice_correlation_threshold (float) – Minimum absolute Pearson correlation |r| required for another MICE column to be counted as an informative predictor when computing n_nearest_features. Columns below this threshold are excluded from the count.
mcar_feature_predictability_threshold (float) – Maximum absolute Pearson correlation |r| below which MCAR model-based routing is skipped in favour of Median. When no numeric predictor exceeds this threshold against the target column, KNN and MICE are not attempted because the feature set contains no useful predictive signal. Applies only to MCAR paths; MAR paths are not affected. Default of 0.2 preserves existing behaviour (no check applied today).
per_column_strategy (dict[str, ImputationStrategy]) – Explicit per-column strategy overrides that fire at Priority 1.5 in the routing chain — after DropCandidate but before MNAR routing. A column listed here bypasses all routing priorities 2–7. Defaults to empty dict (no overrides). Allowed values: Mean, Median, Mode, KNN, MICE. To route a column to a constant fill, use per_column_constant_fill
per_column_constant_fill (dict[str, float]) – Self-sufficient constant fill declarations. Each column listed here is routed to ImputationStrategy.Constant at Priority 1.5, bypassing all routing priorities 2–7. No companion entry in per_column_strategy is required or allowed. Keyed by column name. Defaults to empty dict.
bimodal_grouping_variables (dict[str, str]) – Maps a bimodal column name to the name of the grouping column that explains the bimodal split (e.g. {"age": "employment_status"}).
bimodal_min_correlated_features (int) – Minimum number of numeric features with |r| > 0.2 required to qualify the Bimodal Imputation Framework for branch 2 (MICE/KNN); columns with fewer correlated features fall to branch 3 (Cluster-Conditional).
bimodal_correlation_threshold (float) – Minimum absolute Pearson correlation |r| a feature must have against a bimodal column for it to count toward the branch 2/3 feature tally in the Bimodal Imputation Framework.
max_workers (int, optional) – Degree of thread parallelism for the numeric fit (ADR-0056). The mutually-independent strategy blocks (MICE, KNN, GMM, cluster) and the independent columns within a per-column strategy are fitted concurrently on threads, capped at this many workers. None (the default) auto-sizes to the available CPU count; 1 forces a fully sequential fit. Concurrency never changes a result: the same random_seed yields a byte-identical FittedImputer regardless of this value.
_per_column_strategy (dict[str, ImputationStrategy])
_per_column_constant_fill (dict[str, float])
_bimodal_grouping_variables (dict[str, str])

Raises:

ValueError – If any column in per_column_strategy is mapped to Passthrough, Indicator, Dropped, or MNAR. Constant columns should use per_column_constant_fill; Dropped columns should use PipelineConfig.exclude_columns; MNAR columns should use mnar_columns; Passthrough and Indicator are internal-only.

base_max_iter: int = 10#

bimodal_correlation_threshold: float = 0.2#

property bimodal_grouping_variables: MappingProxyType#

Maps a bimodal column name to the name of the grouping column that explains the bimodal split.

Returns:: Read-only view of bimodal grouping variables.
Return type:: MappingProxyType[str, str]

bimodal_min_correlated_features: int = 3#

classmethod from_dict(data)[source]#

Construct a NumericImputationConfig from a plain dictionary.

Parameters:: data (dict) – Mapping produced by to_dict(). Missing keys fall back to field defaults.
Returns:: Reconstructed config instance.
Return type:: NumericImputationConfig
Raises:: ValueError – If data carries the retired mice_max_iter or knn_n_neighbors key (ADR-0083).

gradient_boost_min_rows: int = 10000#

knn_distance_weight_max_features: int = 30#

knn_distance_weight_max_null_ratio: float = 0.15#

knn_max_features: int = 50#

knn_max_neighbors: int = 25#

knn_max_rows: int = 50000#

knn_min_neighbors: int = 5#

max_workers: int | None = None#

mcar_feature_predictability_threshold: float = 0.2#

mice_correlation_threshold: float = 0.1#

mice_max_nearest_features: int = 20#

mice_min_rows: int = 500#

mice_n_nearest_features_min_cols: int = 10#

property per_column_constant_fill: MappingProxyType#

Self-sufficient constant fill declarations.

Returns:: Read-only view of per-column constant fill values.
Return type:: MappingProxyType[str, float]

property per_column_strategy: MappingProxyType#

Explicit per-column strategy overrides that fire at Priority 1.5 in the routing chain — after DropCandidate but before MNAR routing.

Returns:: Read-only view of per-column strategy overrides.
Return type:: MappingProxyType[str, ImputationStrategy]

set_bimodal_grouping_variable(column, grouping_variable)[source]#

Set the grouping variable for one or more bimodal columns.

Parameters:

column (str | list[str]) – A single column name or list of column names.
grouping_variable (str) – The name of the grouping column.

Raises:

ValueError – If the grouping variable is empty or purely whitespace.

Return type:

None

set_per_column_constant_fill(column, value)[source]#

Set a constant fill value for one or more columns.

Parameters:

column (str | list[str]) – A single column name or list of column names.
value (float) – The constant fill value.

Raises:

ValueError – If the value is NaN or infinity.

Return type:

None

set_per_column_strategy(column, strategy)[source]#

Set the imputation strategy for one or more columns.

Parameters:

column (str | list[str]) – A single column name or list of column names.
strategy (str | ImputationStrategy) – The strategy to assign to the column(s).

Raises:

ValueError – If the strategy is an output-only label (e.g. ‘MNAR’, ‘Dropped’). If ‘Constant’ is set but no corresponding fill value exists in per_column_constant_fill.

Return type:

None

to_dict()[source]#

Serialise the config to a plain dictionary.

Returns:: All field values keyed by field name.
Return type:: dict

class dataforge_ml.imputation._config.ImputationStrategy(*values)[source]#

Bases: StrEnum

Imputation strategy assigned to a column after Phase 2 fitting.

Members fall into two categories:

Input strategies — may be declared in per_column_strategy to override automatic routing: Mean, Median, Mode, KNN, MICE.

Output-only labels — assigned by the engine after fit() and recorded in ColumnImputationRecord.strategy; declaring them in per_column_strategy raises ValueError at construction time: Constant, MNAR, Dropped, Passthrough, Indicator. Constant is produced when a column appears in per_column_constant_fill; use that field instead of declaring it in per_column_strategy.

ClusterConditional = 'cluster_conditional'#

Constant = 'constant'#

Dropped = 'dropped'#

GMMSampling = 'gmm_sampling'#

Indicator = 'indicator'#

KNN = 'knn'#

MICE = 'mice'#

MNAR = 'mnar'#

Mean = 'mean'#

Median = 'median'#

Mode = 'mode'#

Passthrough = 'passthrough'#

class dataforge_ml.imputation._config.ModelChoice(*values)[source]#

Bases: StrEnum

Concrete estimator family selected for a model-based imputation column.

A value-free name for the sklearn estimator family resolved at decide-time from (NonlinearityTag, n_rows, config) — the Recipe half of the Recipe/Learned split (ADR-0059) pulled one layer earlier so the plan is inspectable and overridable before anything trains. It is a label, never a live or fitted estimator object; the execution layer constructs and fits the actual estimator from this choice.

Members map onto the branches of RegressionEstimatorFactory: BayesianRidge for the Linear branch (wrapped in a scaling pipeline at fit-time), RandomForestRegressor for MonotonicNonlinear and the small-sample ComplexNonlinear branch, and GradientBoostingRegressor for the large-sample ComplexNonlinear branch. An Unpredictable column resolves to no model choice (None) and routes to a scalar fallback instead.

Custom is the one member no branch of that factory builds: it marks a unit whose estimator the user supplied, through author()’s estimators= channel (ADR-0083). The instance itself lives in ImputationDecision.custom_estimators, keyed by unit id, so this enum stays a label. It is a value rather than None because None already carries the load-bearing “no estimator family, cannot train” meaning on the fit path; as a value, every existing reader of a ModelChoice stays correct without learning anything. Given up: the enum stops being a closed list of families the library can build — one member means “look elsewhere”.

BayesianRidge = 'bayesian_ridge'#

Custom = 'custom'#

GradientBoostingRegressor = 'gradient_boosting_regressor'#

RandomForestRegressor = 'random_forest_regressor'#

Results#

class dataforge_ml.imputation._config.ImputationResult(dataframe, records=<factory>, dropped_columns=<factory>)[source]#

Bases: object

Output of FittedImputer.transform().

Parameters:

dataframe (pl.DataFrame) – DataFrame with imputed values (and any indicator columns appended).
records (dict[str, ColumnImputationRecord]) – Per-column audit log keyed by column name.
dropped_columns (list[str]) – Columns removed because they exceeded the drop threshold (>50% missing).

dataframe: DataFrame#

dropped_columns: list[str]#

records: dict[str, ColumnImputationRecord]#

to_markdown()[source]#

Render the transform result as a Markdown document.

A document per rule 5 of the Rendering Contract (ADR-0086): it owns the # and ## heading levels and delegates to the ColumnImputationRecord fragments beneath them. dataframe is a Payload Frame — its shape and dtypes are reported and its rows never are — so the output stays bounded in the number of rows imputed.

Returns:: Markdown document with a summary, the dropped columns, the imputed frame’s shape and dtypes, and the per-column audit records.
Return type:: str

class dataforge_ml.imputation._config.ColumnImputationRecord(decision, fill_value=None, indicator_added=False)[source]#

Bases: object

Per-column audit entry produced after fit().

Composes the pure, value-free ColumnImputationDecision (what was decided, reachable via record.decision) with the values fit() learned from the training data (what was learned): the scalar fill_value and the indicator_added fit-metadata flag (ADR-0060). Plan fields — column, strategy, signals, domain_snap_bounds and the rest — are read through record.decision.*.

Parameters:

decision (ColumnImputationDecision) – The decided, value-free per-column plan entry.
fill_value (Any, optional) – Scalar fill value learned from training data (None for model-based strategies).
indicator_added (bool) – Whether a binary missingness indicator column was appended — a fit-time fact distinct from the decided decision.indicator_flag.

Notes

Fit-quality metrics are no longer carried here. fit() only learns fill values and models; quality measurement is a deliberate second step via the opt-in Evaluation phase (ADR-0058).

decision: ColumnImputationDecision#

fill_value: Any | None = None#

classmethod from_dict(data)[source]#

Reconstruct a ColumnImputationRecord from a plain dictionary.

The inverse of to_dict(): the decision is rebuilt from the same flat mapping, then paired with the learned fill_value and indicator_added.

Parameters:: data (dict) – Mapping produced by to_dict().
Returns:: Reconstructed audit record.
Return type:: ColumnImputationRecord

indicator_added: bool = False#

to_dict()[source]#

Serialise the audit record to a plain dictionary.

Flattens the decision’s serialised fields alongside the learned fill_value and indicator_added so the whole record round-trips through from_dict().

Returns:: The decision’s fields plus fill_value and indicator_added.
Return type:: dict

to_markdown()[source]#

Render the audit record as a ###-rooted Markdown fragment.

A fragment per rule 5 of the Rendering Contract (ADR-0086): it carries no # or ## heading, so the owning result document composes it without a heading collision. The composed ColumnImputationDecision renders its own table — what was decided — and the learned fill_value and indicator_added continue it, so decided and learned fields sit side by side in one table without being confusable.

Returns:: Markdown subsection headed by ### `<column>` and a field table covering the decision’s fields plus the learned ones.
Return type:: str

Exceptions and warnings#

class dataforge_ml.imputation._unit_fit.UnitNotTrainableError(unit_id, strategy, columns, reason)[source]#

Bases: RuntimeError

Raised when a planned unit cannot be trained.

Single-track failure (ADR-0071): a unit that cannot train always raises, carrying a structured payload — the unit id, its strategy, its columns, and the fitter’s reason — rather than silently degrading to a scalar. The fitter’s own diagnosis is chained as the cause.

Parameters:

unit_id (str)
strategy (ImputationStrategy)
columns (tuple[str, ...])
reason (str)

unit_id#

The id of the unit that could not train.

Type:: str

strategy#

The strategy the unit was planned to execute.

Type:: ImputationStrategy

columns#

The columns the unit owns.

Type:: tuple[str, …]

reason#

The fitter’s human-readable explanation of why training failed.

Type:: str

class dataforge_ml.imputation._fit_signals.ImputationFitWarning[source]#

Bases: UserWarning

Category for warnings raised while fitting an imputation unit.

Every entry recorded in FitSignals.warnings is also emitted through the standard warnings channel under this category (ADR-0074), so a caller who never inspects the returned record is not silently deprived of them. Suppress, escalate, or route them with the ordinary toolkit, e.g. warnings.filterwarnings("ignore", category=ImputationFitWarning).

The two conditions that raise it are a model that hit its iteration cap without converging, and a strategy forced past its routing threshold (KNN above the row/feature caps) — the latter is warned, never blocked (ADR-0071).

class dataforge_ml.imputation._fitted_imputer.UnseenColumnError[source]#

Bases: Exception

Raised by FittedImputer.transform() when the input DataFrame contains columns that were not present in the training DataFrame during fit().

Fires before any DataFrame mutations regardless of whether the unknown columns contain missing values, so schema drift is caught at transform entry rather than silently propagating downstream (ADR 0026).

All unknown column names are reported in a single raise so the caller can resolve all schema mismatches at once.

class dataforge_ml.imputation._fitted_imputer.FittedColumnAbsentError[source]#

Bases: Exception

Raised by FittedImputer.transform() when a column that received an active imputation strategy during fit() is absent from the input DataFrame.

Active strategies are any strategy other than Dropped or Indicator. Absence of such a column is always a pipeline bug — imputation was never applied — so this is escalated to an error rather than a warning.

All absent column names are reported in a single raise.

class dataforge_ml.imputation._fitted_imputer.DroppedColumnAbsentWarning[source]#

Bases: UserWarning

Emitted by FittedImputer.transform() when a column recorded as ImputationStrategy.Dropped during fit() is already absent from the input DataFrame.

This typically means the caller pre-removed the column before calling transform(). Transform continues normally; the warning is the only signal. Suppress with warnings.filterwarnings("ignore", category=DroppedColumnAbsentWarning).