feat: Measure execution status duration

[morgan-wowk]

Changes

• Adds status_updated_at column to ExecutionNode table to track when execution status last changed
• Implements SQLAlchemy event listener to automatically update status_updated_at timestamp when container_execution_status changes
• Creates execution_status_transition_duration histogram metric to measure time spent in each execution status
• Adds _transition_execution_status() helper function to centralize status updates and metric recording across all status transitions
• Implements database migration logic to add the new status_updated_at column to existing tables
• Replaces direct status assignments throughout the orchestrator with calls to the new transition helper function

Show of work

Note: Attribute names have since changed to execution_status_ prefix

Local smoke-test and verification completed ✅

[morgan-wowk]

• feat: Metrics to count all non-terminal execution statuses #186
• feat: Measure execution status duration #174 👈 (View in Graphite)
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[morgan-wowk]

Load Impact: feat: Measure execution status duration

The commit adds zero additional database calls during steady-state operation. status_updated_at is written as part of the existing UPDATE statement that already commits container_execution_status — SQLAlchemy batches both columns into one flush. Reading the previous status and timestamp before each transition is always an in-memory access from the already-loaded SQLAlchemy instance. The only one-time database cost is a single SELECT on every process restart to check for the new column, plus a single ALTER TABLE on first deployment. On the metrics side, record_status_transition appends to an in-memory histogram bucket and returns immediately; the OTel SDK exports observations to the collector in background batches, adding one additional metric stream with no synchronous I/O on the hot path.

Area	Impact
DB calls at startup	+1 SELECT per restart; +1 ALTER TABLE on first deploy only
DB calls per transition	None — status_updated_at piggybacks on the existing UPDATE
DB calls for reading previous status	None — in-memory reads from the live SQLAlchemy instance
CPU per transition	Negligible — one datetime.now() + one histogram bucket increment
Memory	Negligible — one additional datetime field per loaded ExecutionNode
Network (OTel export)	One additional histogram metric stream, exported in background batches

[Ark-kun]

Hi, Morgan. Thank you for your patience.
Let's make several changes to improve this feature.

1. Let's follow the Kubernetes example and record history and times of all status transitions.
   In some cases Kubernetes records "Last time resource was in state X" to compress the state history when states can be repeated hundreds of times. In our case, we don't expect so much status re-entrance, so let's just record the whole history.
   This will allow much richer and future-proof metrics.

Example:

container_execution_status_history:
  - status: PENDING
     first_observed_at: XXX
  - status: RUNNING
     first_observed_at: YYY

2. Instead of creating a new DB column and adding a manually written migration, let's use extra_data. It's designed for the extra data that does not fit into the current static DB schema. This brings SQL tables closer to the usability of the Document DBs.

3. Instead of changing the orchestrator, let's set up a SqlAlchemy event listener for the ExecutionNode.container_execution_status change. This allows reliably intercepting all status changes. And it also keeps the orchestrator code simpler.

[morgan-wowk]

Thanks for the thoughtful review, Alexey! I've reworked the implementation in this push.

What changed:

Status history in extra_data — the status_updated_at column and its migration are gone. Each status transition is now recorded as { "status": ..., "first_observed_at": ... } appended to extra_data["container_execution_status_history"].

SQLAlchemy event listeners (sql_event_listeners.py) — two thin hooks do the wiring:

• A set listener on ExecutionNode.container_execution_status calls execution.stage_status_history_entry(value=value) to timestamp and stage the entry onto the instance.
• A before_flush session listener calls obj.handle_before_flush() for any ExecutionNode in the dirty/new set.

All model-specific business logic lives on ExecutionNode itself:

• stage_status_history_entry() — builds the timestamped entry and queues it in _pending_extra_data.
• apply_status_history_entries() — merges pending entries into extra_data and emits transition events; only clears its own key from _pending_extra_data so other staged keys are left intact.
• handle_before_flush() — entry point called by the session listener; delegates to apply_status_history_entries().

Typed event system (event_listeners.py) — a small generic pub/sub module with subscribe(event_type, callback, asynchronous) and emit(event). StatusTransitionEvent is a frozen dataclass carrying from_status, to_status, and duration_seconds. Listeners can opt into async dispatch (daemon thread) or synchronous dispatch per subscription.

Metrics (instrumentation/metrics.py) — register_listener() subscribes to StatusTransitionEvent with asynchronous=False so the histogram record happens synchronously on the flush thread, then the OTel SDK batches the export independently.

Orchestrator simplified — _transition_execution_status() is gone entirely. All call sites are plain execution.container_execution_status = ContainerExecutionStatus.X assignments.

[morgan-wowk]

@Ark-kun Okay changed this around to be more event based. Let me know how you like it

[morgan-wowk]

@Ark-kun This has been simplified per our discussion.

Verified locally in Prometheus:white_check_mark:

[morgan-wowk]

OTel is built to never throw a Runtime exception, but in case it ever did, I don't want that to result in a rollback of the commit - so I've wrapped this with a try/except.

[Ark-kun]

Do we need to check whether new value is different from the old value?
(If SqlAlchemy already checks that and does not fire event when the value is the same, then no need to change anything.)

[Ark-kun]

Maybe let's add type for value?

[Ark-kun]

Let's move this whole function to the orchestrator module. I think maintaining the status history becomes part of it's job, so it belongs there. I'm kind of on the fence here, but modifications to ExecutionNode is probably the jurisdiction of the orchestrator.
This will also solve the issue of wiring it up automatically without relying on an import.

And _handle_before_commit function can go to some instrumentation module.

[Ark-kun]

Thank you. Approved.

But let's split the sql_event_listeners module (simple).
And let's check that we do not add history entries for duplicate status.