Observability#

The Pipeline Event stream emitted by the orchestrators (StructuralProfiler) and consumed by a user-supplied Progress Observer (ADR-0054, ADR-0055).

class dataforge_ml.observability.PipelineEvent(event_type, phase, stage, message, column=None, index=None, total=None)[source]#

Bases: object

A single structured observability record emitted by an orchestrator.

Progress and Trace share this one shape. Consumers filter on event_type and read the raw index/total fields to drive a progress bar, or print the ready-made message as-is.

Parameters:

  • event_type (EventType) – The kind of event; also fixes the Trace log level.

  • phase (str) – The pipeline phase emitting the event (e.g. "profiling").

  • stage (str) – The stage within the phase (e.g. "column_profiling").

  • message (str) – A ready-made human-readable sentence describing the event.

  • column (str, optional) – The column an item or substep event concerns; None for stage boundaries and column-independent work.

  • index (int, optional) – The 1-based position of the current item within its stage, or the Substep sub-progression (e.g. fold 3 of 5) on a substep event; None for stage boundaries.

  • total (int, optional) – The total number of items the stage will process, or the Substep sub-total (e.g. 5 folds) on a substep event; None for stage boundaries.

column: str | None = None#
event_type: EventType#
index: int | None = None#
message: str#
phase: str#
stage: str#
total: int | None = None#
class dataforge_ml.observability.EventType(*values)[source]#

Bases: StrEnum

The kind of a PipelineEvent, used to route and filter the stream.

The value also determines the Trace log level: stage_start, stage_end, item and substep log at DEBUG, decision at INFO, and warning at WARNING.

substep is a Substep heartbeat below the column (a strategy-block fit, a diagnostics fold, a per-column model fit within a set); it is additive, so an observer that ignores it is unaffected, and a consumer drives a coarse progress bar off item while reading a live detail line off substep (ADR-0055).

decision = 'decision'#
item = 'item'#
stage_end = 'stage_end'#
stage_start = 'stage_start'#
substep = 'substep'#
warning = 'warning'#
dataforge_ml.observability.stderr_observer(event)[source]#

Write an event’s message to stderr; the shipped opt-in observer.

Pass this as observer= to an orchestrator to see progress on stderr without wiring up a custom callback. Writes only to stderr, never stdout.

Parameters:

event (PipelineEvent) – The event to render.

Return type:

None