Evaluation#

The evaluation module scores the data, never the imputation model (ADR-0087). Its question is whether a table produced by a pipeline phase still carries the distribution and dependence structure of the data behind it; the model that produced the table is disposable and is never graded.

It is independent and opt-in. EvaluationConfig stands alone and is never nested on PipelineConfig, because evaluation is stateless and re-configured freely between calls while poking at results, whereas the pipeline config is built once for a run.

The module is layered. A generic layer holds the metric arithmetic and knows nothing about any one phase — its seam takes two frames and returns a C2STResult. It stays private: the user drives evaluation through evaluate_imputation, and the future non-imputation consumer is a sibling adapter inside the library. Consumer sub-packages adapt a phase’s artefacts to that layer; evaluation/imputation/ is the first of them.

Reading a report#

evaluate_imputation returns one record per active column, always — untestable is a result, never an absence. A column the test could not run on carries a C2STOutcome naming why and no C2STScore object at all, so record.score.mean_frame_z raises AttributeError rather than reading a stand-in that could collapse into a passing mean. A column excluded under the Phase Active-Columns Contract is absent from the census entirely and raises KeyError.

Two things the rendered report says that are worth stating here too:

Scalar strategies fail at close to 100%, and that is the design. The column under test is itself a feature, so a Mean/Median/Mode/Constant fill’s one-value spike is visible. A report on a median-filled table flags every such column; that is the baseline which makes a model-based number interpretable, not a list of defects.
A fired test is ambiguous under MAR. Under a strongly MAR mechanism the observed and filled rows are supposed to differ, so a flag cannot separate “the imputation is bad” from “the missingness is informative”. The report prints this caveat whenever anything is flagged, and no dial suppresses it: full configurability governs thresholds and behaviour, not the library’s disclosure of a known limit of its own number.

Entry point#

dataforge_ml.evaluation.imputation._adapter.evaluate_imputation(original_df, imputed_frames, profile, *, config, pipeline_config, metrics=None, observer=None)[source]#

Score an imputed table against the frame it was imputed from.

The evaluation module’s entry point: stateless, opt-in, and free to be re-run with different dials while poking at results. It grades the data, never the imputer — the model that produced the table is disposable, and the question is whether the table still carries the distribution and dependence structure of the data behind it.

For every active column, the Classifier Two-Sample Test trains a classifier to tell the rows where that column was observed from the rows where it was filled, and reports its held-out accuracy as a standardised z. Every active column gets a record, always: untestable is a result, never an absence. A column the test could not run on carries a C2STOutcome naming why and no score object at all, so record.score.mean_frame_z raises AttributeError rather than reading a stand-in that could collapse into a passing mean.

Once every column has run, Benjamini-Hochberg turns the tested columns’ p-values into three-valued C2STVerdict``s controlling the false discovery rate across the report. The raw ``z is never corrected: BH moves the shortlist, not the score.

Expect scalar strategies (Mean/Median/Mode/Constant) to be caught at close to 100% — that is the baseline which makes a model-based number interpretable, not forty defects.

Parameters:

original_df (polars.DataFrame) – The pre-imputation frame. It is what identifies the filled cells: the mask is derived from it through effective-null resolution, so rows carrying a declared sentinel land in the filled pile where they belong.
imputed_frames (Sequence[polars.DataFrame]) – The imputed table(s), row-aligned with original_df. A sequence from day one — length 1 today — so calling code never has to change when multiple imputation arrives. The m frames are pooled by the mean z against the single-run null; the null is never shrunk by √m.
profile (StructuralProfileResult) – The profile of original_df. Needed, not optional: the mask needs its sentinel maps, the type gate needs its SemanticType values, and the Dtype Floor is driven off it. A table handed over cold, without its profile, cannot be evaluated.
config (EvaluationConfig) – The evaluation dials, with C2STConfig nested inside. Stands alone and is never nested on PipelineConfig.
pipeline_config (PipelineConfig) – The run’s config, read only for the Phase Active-Columns Contract: evaluation resolves its own column set against PipelinePhase.Imputation. Excluded columns are honoured on governance grounds — an excluded column is absent from the census entirely, so report.c2st[col] raises KeyError.
metrics (Sequence[EvaluationMetric], optional) – Which metrics to run. None, the default, means every metric the library has. A metric not selected comes back as None on the report, which is deliberately a different state from ran and refused.
observer (Callable[[PipelineEvent], None], optional) – A Progress Observer receiving the event stream. It rides the call and is never a config field, because a live callable would break the config’s serialisable contract. Events are progress only: phase="evaluation", stage="c2st", one item per active column — refused ones included, so the bar reaches its own total — and one substep per repeat.

Returns:

The umbrella report; report.c2st holds the per-column C2ST census and the run’s provenance.

Return type:

EvaluationReport

Raises:

ValueError – When imputed_frames is empty, when a frame’s row count differs from original_df’s, or when an active column is missing from an imputed frame.

Configuration#

class dataforge_ml.evaluation._config.EvaluationConfig(c2st=<factory>)[source]#

Bases: object

The evaluation module’s configuration object, one nested config per metric.

Handed to evaluate_imputation as config=. It holds no observer: a live callable would break the serialisable, setter-only contract every config in the library keeps (ADR-0044), so observer= rides the call instead.

Parameters:: c2st (C2STConfig) – Dials for the Classifier Two-Sample Test, the module’s first metric.

c2st: C2STConfig#

classmethod from_dict(data)[source]#

Construct an EvaluationConfig from a plain dictionary.

Parameters:: data (dict) – Mapping produced by to_dict(). Missing keys fall back to field defaults.
Returns:: Reconstructed config instance.
Return type:: EvaluationConfig

to_dict()[source]#

Serialise the config to a plain dictionary.

Returns:: All field values keyed by field name, with c2st nested.
Return type:: dict

class dataforge_ml.evaluation._config.C2STConfig(repeats=5, scheme=C2STScheme.Split, min_samples_leaf=5, min_filled_cells=30, low_power_below=250, imbalance_warn_ratio=0.5, fdr_alpha=0.05, random_state=None, classifier=None)[source]#

Bases: object

Dials for the Classifier Two-Sample Test.

Every threshold the refusal machinery uses is here, so a user moves the floor for their own data rather than arguing with a constant. Two values are not dials and never will be: balancing, because unequal piles break the Binomial(n_test, ½) null outright, and the classifier’s early_stopping pin, which is a structural correction to a dataset-size assumption rather than a threshold — a user wanting different classifier internals has the wider classifier door.

Parameters:

repeats (int) – The number of balanced subsampling repeats, R. A single balanced draw is a random subsample whose z depends on which rows were drawn; the repeats are what make that answer stable. Never auto-downgraded when scheme is CrossValidation.
scheme (C2STScheme) – Split (one fit per repeat, the default) or CrossValidation (five). Split is the default because repeats are already mandatory and a default that silently costs 5× is a bad default.
min_samples_leaf (int) – The leaf-size floor of the library’s default classifier. sklearn’s 20 cannot split fewer than 40 training rows, so the classifier constant-predicts and reports a constant fill as a perfect pass. Fixed, never scaled to pile size — a scaled setting would make the score a function of the column’s missingness rate. Ignored when classifier is supplied, since injection is total.
min_filled_cells (int) – The hard refusal floor, applied to the smaller pile (min(n_observed, n_filled)) before balancing, because every measured number is in rows per pile after balancing. Below it the column’s outcome is BelowSampleFloor.
low_power_below (int) – The annotation band’s upper edge, in rows per pile. An annotation and never a refusal, so a pass in that band reads “not enough data to tell” rather than “clean”.
imbalance_warn_ratio (float) – The pile ratio min(a, b) / max(a, b) below which the result is annotated as imbalanced, naming the smaller pile as the cap on the effective sample size.
fdr_alpha (float) – The false discovery rate Benjamini-Hochberg controls across one report’s tested columns. A column whose adjusted p-value is at or below it is Flagged. It moves the shortlist only: the raw z is never corrected, because the z is the score.
random_state (int, optional) – Seed for the subsampling, the train/test carve and the default classifier. The same seed on the same frames gives the same report.
classifier (Any, optional) – An unfitted classifier to use verbatim for every column of the run — no set_params reach-in and no clone, mirroring how ImputationDecision.custom_estimators holds the user’s live object. None builds the library’s default, pins included. Stated hole (ADR-0087): an injected bare HistGradientBoostingClassifier reintroduces the constant-predictor pathology at 30–60 rows per pile, and only classifier_injected in the provenance warns the reader. Being a live object it is the one field to_dict() cannot carry.

classifier: Any | None = None#

fdr_alpha: float = 0.05#

classmethod from_dict(data)[source]#

Reconstruct a C2STConfig from a plain dictionary.

Missing keys fall back to field defaults, so a serialised config written against an earlier version loads rather than raising. The reconstructed config carries classifier=None; see to_dict().

Parameters:: data (dict) – Mapping produced by to_dict().
Returns:: Reconstructed config instance.
Return type:: C2STConfig

imbalance_warn_ratio: float = 0.5#

low_power_below: int = 250#

min_filled_cells: int = 30#

min_samples_leaf: int = 5#

random_state: int | None = None#

repeats: int = 5#

scheme: C2STScheme = 'split'#

to_dict()[source]#

Serialise the dials to a plain dictionary.

classifier is deliberately absent: it is a live estimator instance, and a config that is serialisable except for one field that silently degrades on the round trip is worse than one that states the omission. The report’s C2STProvenance is where a run’s classifier is recorded.

Returns:: Every dial except classifier, keyed by field name.
Return type:: dict

class dataforge_ml.evaluation._config.EvaluationMetric(*values)[source]#

Bases: StrEnum

The metric selector passed as metrics= to evaluate_imputation.

None — the default — means every metric the library has, so the bare call honours the umbrella name and nobody needs to know the list exists. Narrowing is scope selection on an opt-in diagnostic handed to a human, not the staging of a library decision to save cost, so it does not contradict the accuracy-over-speed principle (ADR-0087).

C2ST = 'c2st'#

class dataforge_ml.evaluation._c2st.C2STScheme(*values)[source]#

Bases: StrEnum

The train/test scheme of one C2ST repeat — a dial; balancing is not.

Split fits once per repeat on a 70/30 stratified carve of the balanced piles; CrossValidation fits five times and pools the out-of-fold predictions. Both keep n_test exactly known, which is what the closed-form null requires. Split is the default a config declares: repeats are already mandatory, so a default that silently costs 5× is a bad default. The repeat count is never auto-downgraded when cross-validation is on.

CrossValidation = 'cross_validation'#

Split = 'split'#

Records#

class dataforge_ml.evaluation.imputation._records.EvaluationReport(c2st=None)[source]#

Bases: object

The umbrella return of evaluate_imputation.

One nullable field per metric, so a later metric lands as an additive sibling rather than a breaking return-type change. report.c2st is None means not requested — deliberately a different state from ran and refused, which is a C2STOutcome at column level.

Parameters:: c2st (C2STReport | None)

c2st#

The Classifier Two-Sample Test’s report, or None when metrics= did not select it.

Type:: C2STReport, optional

to_markdown()[source]#

Render the report as a Markdown document.

Returns:: Markdown document headed by # Evaluation Report, holding one section per metric that ran.
Return type:: str

class dataforge_ml.evaluation.imputation._records.C2STReport(columns=<factory>, provenance=<factory>)[source]#

Bases: object

The column-keyed C2ST report for one evaluation call.

Holds one ColumnC2STResult per active column, in the frame’s own column order, plus the run’s C2STProvenance.

Constructing the report is what runs Benjamini-Hochberg: the tested columns’ verdicts and adjusted p-values are stamped on in __post_init__(), so a report never exists in an uncorrected state and a record can never be corrected against a set it did not come from.

Parameters:

columns (dict[str, ColumnC2STResult])
provenance (C2STProvenance)

columns#

The census: every active column, keyed by name.

Type:: dict[str, ColumnC2STResult]

provenance#

What ran and under which dials.

Type:: C2STProvenance

to_markdown()[source]#

Render the report as a ###-rooted Markdown fragment.

Leads with the outcome census, because a 200-column table emitting 190 refusals is otherwise a wall. When any column is Flagged two fixed notes ride along: the baseline framing, which says that a scalar-filled column is supposed to fire, and the MAR caveat, printed as a footer and suppressible by nothing (ADR-0087).

Returns:: Markdown subsection headed by ### C2ST, holding the outcome census, a per-column summary table, each column’s own fragment, the provenance, and — whenever anything is flagged — the baseline framing and the MAR caveat.
Return type:: str

class dataforge_ml.evaluation.imputation._records.ColumnC2STResult(column, outcome, n_observed, n_filled, score=None, annotations=(), verdict=C2STVerdict.NoVerdict)[source]#

Bases: object

One active column’s result — an outcome first, a number only sometimes.

The two pile sizes are stated whatever the outcome, because they are the facts that explain a refusal.

Parameters:

column (str)
outcome (C2STOutcome)
n_observed (int)
n_filled (int)
score (C2STScore | None)
annotations (tuple[C2STAnnotation, ...])
verdict (C2STVerdict)

column#

The column the record describes.

Type:: str

outcome#

What happened. Only Tested carries a score.

Type:: C2STOutcome

annotations#

Facts qualifying a result the test actually ran, in a fixed order. Empty on a column refused before the test ever ran, since there is then no number for them to qualify. They never suppress the z.

Type:: tuple[C2STAnnotation, …]

verdict#

Whether a human should go look at this column, stamped on by C2STReport once Benjamini-Hochberg has run across the tested set. NoVerdict on every refusal — and on a record that has not been through a report at all.

Type:: C2STVerdict

n_observed#

Pile A’s raw size — the rows where the column was observed, before balancing.

Type:: int

n_filled#

Pile B’s raw size — the rows where the column was filled, before balancing. Rows carrying a declared sentinel are here, not in pile A.

Type:: int

score#

The numbers, present only on a Tested outcome. None on every refusal, so result.score.mean_frame_z raises AttributeError rather than reading a stand-in.

Type:: C2STScore, optional

to_markdown()[source]#

Render the column’s result as a ###-rooted Markdown fragment.

Returns:: Markdown subsection headed by ### Column `<name>`, followed by the score’s own fragment when the column was tested.
Return type:: str

class dataforge_ml.evaluation.imputation._records.C2STScore(mean_frame_z, frame_z_spread=0.0, p_value=1.0, p_adjusted=None, frames=())[source]#

Bases: object

Every number that exists only when the test actually ran.

Nullability is the whole point: a refused column holds score = None, so a refusal can never arithmetic-collapse into a passing figure. A refused column and a chance-scoring column differ by having or not having this object.

The headline is mean_frame_z — the mean z over the m frames against the single-run null. Dividing that null’s SD by √m is forbidden (ADR-0087): the frames share pile A entirely, so shrinking the null manufactures significance. No field at any level is named plain z.

Parameters:

mean_frame_z (float)
frame_z_spread (float)
p_value (float)
p_adjusted (float | None)
frames (tuple[C2STResult, ...])

mean_frame_z#

The mean of the per-frame C2STResult.mean_repeat_z, read against the single-run null. The score.

Type:: float

frame_z_spread#

The sample standard deviation of the per-frame z (0.0 at m = 1).

Type:: float

p_value#

The one-sided, upper-tail normal p-value of mean_frame_z. Present but never the score: a p-value is a function of the discrepancy and of n, so it floors out on large tables and changes meaning across datasets. Below-chance accuracy therefore reads as noise rather than as evidence of excellent imputation.

Type:: float

p_adjusted#

The Benjamini–Hochberg adjusted p-value, stamped on by the report after correction runs across the tested set. None until it does.

Type:: float, optional

frames#

One generic result per imputed frame, in the order the frames were supplied.

Type:: tuple[C2STResult, …]

to_markdown()[source]#

Render the score as a ###-rooted Markdown fragment.

Returns:: Markdown subsection headed by ### C2ST Score, holding the pooled figures and one row per frame.
Return type:: str

class dataforge_ml.evaluation._c2st.C2STResult(mean_repeat_z, repeat_z=<factory>, repeat_z_spread=0.0, train_accuracy=0.0, test_accuracy=0.0, n_test=0, degenerate=False, unseen_category=False)[source]#

Bases: object

What the generic test returns for one pair of samples.

Every number here is pooled over the R repeats of a single c2st call. The headline is mean_repeat_z — the mean ``z`` against the single-run null. Dividing the null’s SD by √R is forbidden (ADR-0087): the repeats share the candidate sample entirely, so shrinking the null manufactures significance. No field is ever named plain z; that is the field someone averages along the wrong axis.

Parameters:

mean_repeat_z (float)
repeat_z (tuple[float, ...])
repeat_z_spread (float)
train_accuracy (float)
test_accuracy (float)
n_test (int)
degenerate (bool)
unseen_category (bool)

mean_repeat_z#

The mean of repeat_z, read against the single-run null.

Type:: float

repeat_z#

One z per repeat, in repeat order.

Type:: tuple[float, …]

repeat_z_spread#

The sample standard deviation of repeat_z (0.0 for a single repeat) — the guard that says whether the mean is stable or the artefact of one lucky draw.

Type:: float

train_accuracy#

Mean training accuracy over the repeats. Reported beside test_accuracy so a 1.000/0.500 pair reads as memorisation — which fails safe, yielding false negatives, never false positives.

Type:: float

test_accuracy#

Mean held-out accuracy over the repeats; the quantity the z standardises.

Type:: float

n_test#

The held-out row count of one repeat, exactly known and identical across repeats: 2 × round(m × 0.3) under Split and 2 × m under CrossValidation, where m is the balanced pile size.

Type:: int

degenerate#

True when every repeat’s classifier predicted a single class — a runtime safety net for an injected classifier, explicitly not the size floor.

Type:: bool

unseen_category#

True when some pl.Categorical/pl.Boolean level appears in the candidate sample and never in the reference sample. Computed as a set difference before the test; the z is reported regardless.

Type:: bool

to_markdown()[source]#

Render the result as a ###-rooted Markdown fragment.

A fragment per rule 5 of the Rendering Contract (ADR-0086): it carries no # or ## heading, so an owning document composes it without a heading collision.

Returns:: Markdown subsection headed by ### C2ST Sample-Pair Result, holding the pooled figures and the two runtime flags.
Return type:: str

class dataforge_ml.evaluation.imputation._records.C2STProvenance(classifier_class, classifier_params=<factory>, classifier_injected=False, sklearn_version='', config=<factory>)[source]#

Bases: object

What ran, so a number can still be argued with six months later.

One object per call: a single evaluation uses one classifier and one resolved C2STConfig for every column, so it hangs off the report rather than off each record.

Parameters:

classifier_class (str)
classifier_params (dict[str, str])
classifier_injected (bool)
sklearn_version (str)
config (C2STConfig)

classifier_class#

The fully qualified class name of the classifier that ran.

Type:: str

classifier_params#

The classifier’s full get_params(), values stringified. Deliberately not its repr, which elides every parameter left at its default — exactly the fact most needed when arguing with a report. Stringified because get_params() can hold live nested estimators.

Type:: dict[str, str]

classifier_injected#

Whether the classifier was the user’s. Its own field rather than something inferred from the class name: “the library’s default” and “a user-supplied instance that happens to look identical” answer different questions, and only the first carries the pins.

Type:: bool

sklearn_version#

The scikit-learn version that produced the numbers, so a figure that changes after an upgrade is explicable.

Type:: str

config#

The resolved dials. A report that cannot say what min_filled_cells was when it refused a column is a report you cannot argue with.

Type:: C2STConfig

to_markdown()[source]#

Render the provenance as a ###-rooted Markdown fragment.

Returns:: Markdown subsection headed by ### C2ST Provenance, holding the classifier identity, its full parameter set, and the resolved dials.
Return type:: str

Outcome vocabulary#

C2STOutcome, C2STAnnotation and C2STVerdict classify output and are never user-supplied — a stated exception to ADR-0050, because reading the report is branching on them.

class dataforge_ml.evaluation.imputation._records.C2STOutcome(*values)[source]#

Bases: StrEnum

What happened to one column — the closed six-member vocabulary.

A seventh member is a breaking change. Exported despite classifying output and never being supplied, a stated exception to ADR-0050: reading the report is comparing against it (rec.outcome is C2STOutcome.Tested), so the handling half of the export rule reaches it.

Tested is the only member carrying a C2STScore; the other five are refusals, each naming why the test could not produce an honest number.

BelowSampleFloor = 'below_sample_floor'#

NoFilledCells = 'no_filled_cells'#

NoObservedCells = 'no_observed_cells'#

Tested = 'tested'#

TypeNotTestable = 'type_not_testable'#

Uninformative = 'uninformative'#

class dataforge_ml.evaluation.imputation._records.C2STAnnotation(*values)[source]#

Bases: StrEnum

A fact riding on a result the test actually ran — never a refusal.

Annotations do not suppress the z; they qualify it. A column carrying LowPower was tested honestly and simply had too little data for the answer to mean much, which is why a pass there reads “not enough data to tell” rather than “clean”.

Two members are column-level facts read off the pile sizes (LowPower, ImbalancedPiles) and two are rolled up from the per-frame C2STResult flags (Degenerate, UnseenCategory). Unlike C2STOutcome this vocabulary is open: a seventh outcome is a breaking change, a further annotation is additive.

LowPower#: The balanced pile size is under C2STConfig.low_power_below.

ImbalancedPiles#: min(n_observed, n_filled) / max(...) is under C2STConfig.imbalance_warn_ratio, so the smaller pile caps the effective sample size.

Degenerate#: At least one — but not every — frame’s classifier predicted a single class in every repeat. Every frame degenerating is the Uninformative outcome instead, since there is then no honest number left to report.

UnseenCategory#: Some Categorical/Boolean level appears among the filled rows and never among the observed ones. Computed as a set difference before the test; the z is reported regardless, because inventing an unseen category is always wrong whereas an unseen numeric value is legitimate interpolation.

class dataforge_ml.evaluation.imputation._records.C2STVerdict(*values)[source]#

Bases: StrEnum

Whether a human should go look at this column — three-valued, never a bool.

Stamped onto the record by C2STReport after Benjamini-Hochberg runs, because BH is a property of the set of tested columns and no single record can compute it. The third member is the point: a column the test could not run on has NoVerdict, where False would read as “checked, fine”.

A consequence worth stating: a ColumnC2STResult lifted out of its report carries a verdict whose meaning depends on the set it came from.

Flagged#: The BH-adjusted p-value is at or below C2STConfig.fdr_alpha.

NotFlagged#: The column was tested and its adjusted p-value is above the level.

NoVerdict#: The column was never tested, so there is nothing to correct and nothing to conclude.