Skip to content

fix(engine): report the same run error whatever order parallel failures finish in - #192

Merged
M3gA-Mind merged 1 commit into
tinyhumansai:mainfrom
M3gA-Mind:fix/deterministic-parallel-error
Nov 11, 2026
Merged

M3gA-Mind merged 1 commit into
tinyhumansai:mainfrom
M3gA-Mind:fix/deterministic-parallel-error

Conversation

@M3gA-Mind

Copy link
Copy Markdown
Collaborator

Summary

When several parallel branches fail with no error handler in one superstep, the reported run error no longer depends on which branch finished first.

Root cause

finish_execution stored the failing node's structured EngineError into the run's shared terminal_error slot, first writer wins. In a parallel superstep the first writer is whichever activation completes first, so one graph and input could report different errors under different latencies. The graph runtime already names a failed step by the lowest active-set index, so the two could also disagree.

Fix

  • The slot now holds a TerminalError { step, branch, error } (src/engine/terminal_error.rs).
  • Handlers offer their error through terminal_error::offer, which keeps the lowest (superstep, active-set index). The branch index comes from the existing NodeContext::fork.

Rule (documented on offer and in the README's Reliability section)

The earliest superstep wins. Within one superstep the lowest active-set index wins. For a fan-out that is the edge declaration order, the same order parallel updates are folded in. Completion order never matters. A single failure is reported unchanged, and lane failures still belong to their gather.

Validation

On a Linux build host, on main f778023 (merges cleanly onto 8b22179):

  • cargo test --workspace --all-features --locked: 2860 passed, 0 failed, 35 ignored (main's 35; no new ignores).
  • clippy -D warnings: 0. fmt: clean. cargo +1.88.0 check --all-features (MSRV): passes. rustdoc -D warnings: passes.
  • Red control: the new tests/parallel_failure_order_tests.rs, copied onto unmodified main, fails on the first schedule ("delays [10, 0] reported … unhandled failure at n2"). It passes with this change.

Tests

  • src/engine/terminal_error_tests.rs:
    • the lowest index wins in either arrival order;
    • an earlier superstep is not replaced by a lower index in a later one;
    • a single failure passes through.
  • tests/parallel_failure_order_tests.rs: two fan-out branches fail under opposite yield-count latencies, and both runs report the first-declared branch's error.

Found while writing the remaining fuzz targets for #46.

…es finish in

When several parallel branches fail under `stop` in one superstep, each
handler offered its error to the run's terminal-error slot first-writer-
wins, so the reported error followed completion order: one graph and
input could report different errors under different latencies, and
disagree with the runtime, which already names a failed step by the
lowest active-set index.

The slot now keeps a TerminalError { step, branch, error } and
terminal_error::offer keeps the lowest (superstep, active-set index):
the earliest superstep wins, then the lowest index, which for a fan-out
is the edge declaration order and the order parallel updates are folded
in. A single failure is reported unchanged.
@coderabbitai

coderabbitai Bot commented Nov 11, 2026

Copy link
Copy Markdown

Warning

Review limit reached

  • Run on-demand review

This review includes 11 billable files and costs up to $2.75.

Or wait 20 minutes for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: f1a44c8e-caf5-4331-a068-f303aeca95fa
Commits

Reviewing files that changed from the base of the PR and between 8b22179 and 337370b.

Files selected for processing: 11

  • README.md
  • crates/tinyflows/src/engine.rs
  • crates/tinyflows/src/engine/build.rs
  • crates/tinyflows/src/engine/build/activation.rs
  • crates/tinyflows/src/engine/build/handlers.rs
  • crates/tinyflows/src/engine/build/outcome.rs
  • crates/tinyflows/src/engine/resumable/continuation.rs
  • crates/tinyflows/src/engine/run_state.rs
  • crates/tinyflows/src/engine/terminal_error.rs
  • crates/tinyflows/src/engine/terminal_error_tests.rs
  • crates/tinyflows/tests/parallel_failure_order_tests.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@tinysweeper

tinysweeper Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Tiny Sweeper review

Reviewed head: 337370b4cfaf2659027dce9fcf5d2d1b22a905cb

Verdict: Ready for maintainer review · Priority: none
Coverage: Reviewed 11 of 11 files.
Pull request purpose (title): fix(engine): report…

Diagnostics and supporting details

Full summary

Pull request purpose (title): fix(engine): report the same run error whatever order parallel failures finish in. Tiny Sweeper completed its review; deterministic results follow.

Review snapshot

Change surface Files Review signal Count
Production 8 Active findings 0
Tests 2 Noted findings 0
Documentation 1 Resolved findings 0
Configuration 0 Pending checks/questions 0

Completeness: Complete
Test assessment: No supported feature-to-test mapping was available; this does not mean tests are absent or passed.

Features

  • Added — Deterministic terminal-error slot with order-aware selection: Introduces `TerminalError`/`TerminalSlot` and `terminal_error::offer`, which keeps the failure with the lowest (superstep, active-set index) and ignores completion order, making the reported run error a function of the graph and input rather than of scheduling latency. A single failure passes through unchanged, and an earlier superstep's error is never replaced by a lower-indexed later one. (crates/tinyflows/src/engine/terminal_error.rs, crates/tinyflows/src/engine.rs use build::build_graph;)
  • Internal refactor — Terminal-error slot type threaded through graph construction and handlers: `build_graph` and `register_handlers` take the new `TerminalSlot` type instead of the raw mutex of `Option<EngineError>`, so the ordering rule is enforced at the single point of insertion rather than at each call site. No change to the error string channel that parallel paths already consume. (crates/tinyflows/src/engine/build.rs pub(super) fn build_graph(, crates/tinyflows/src/engine/build/handlers.rs pub(super) fn register_handlers(, crates/tinyflows/src/engine/build/outcome.rs where)
  • Modified — README documents the parallel-failure reporting rule: The Reliability section now states that when several parallel branches fail under `stop` in one super-step, the error of the branch earliest in that step's active set (edge order for a fan-out) is reported, never the first to finish, and that an earlier super-step's failure always wins — making the observable contract explicit for users. (README.md Rust 2024 · MSRV 1.85 · `#![forbid(unsafe_code)]` · GPL-3.0-or-later.)

Tests

No supported feature-to-test mapping was produced. Test execution is not inferred.

Findings

No active actionable findings.

Before merge

None.

Agent review details

critique

  • Conclusion: Success
  • Scope reviewed: all assigned evidence
  • Lane summary: Reviewed 11 files; 0 findings. _The code index is behind this pull request (indexed at `16cd898e2a25`), so retrieved context may be out of date._

security

  • Conclusion: Success
  • Scope reviewed: all assigned evidence
  • Positive: The security lane answered with no findings across the 7 files it reviewed; the remaining files were excluded as prose or build-output-shaped rather than left unreviewed for cause.
  • Lane summary: Reviewed 7 files; 0 findings. 4 files were not security-reviewed: README.md (prose or tabular data), crates/tinyflows/src/engine/build/activation.rs (vendored or build output), crates/tinyflows/src/engine/build/handlers.rs (vendored or build output), crates/tinyflows/src/engine/build/outcome.rs (vendored or build output). _The code index is behind this pull request (indexed at `16cd898e2a25`), so retrieved context may be out of date._

tests

  • Conclusion: Success
  • Scope reviewed: all assigned evidence
  • Positive: The tests lane answered with no findings and characterized the deterministic error-selection policy as correct, pinned on both ordering paths, and safely mergeable.
  • Lane summary: This change introduces a deterministic error-selection policy for parallel branches that fail under `stop`: the earliest superstep wins, and within a superstep the lowest active-set index wins. The implementation is correct, the tests pin the behaviour on both ordering paths, and the new `TerminalSlot` type isolates the selection logic cleanly. The change is safe to merge. _The code index is behind this pull request (indexed at `16cd898e2a25`), so retrieved context may be out of date._

commits

  • Conclusion: Success
  • Scope reviewed: all assigned evidence
  • Positive: The commits lane answered with no findings, reporting no sensitive content in the committed files.
  • Lane summary: No sensitive content in committed files. _The code index is behind this pull request (indexed at `16cd898e2a25`), so retrieved context may be out of date._

description

  • Conclusion: Success
  • Scope reviewed: all assigned evidence
  • Lane summary: The description matches the diff: the shared error slot becomes `TerminalSlot`/`TerminalError`, failures are offered through `terminal_error::offer` with `(superstep, active-set index)` ordering sourced from `NodeContext::fork`, the two entry points unwrap `held.error`, and the README documents the rule. The listed unit and integration tests are present, and the new files stay well inside the repository's 500-line and `_tests.rs`-naming rules. Nothing in the change looks like it would misbehave for a reachable input. _The code index is behind this pull request (indexed at `16cd898e2a25`), so retrieved context may be out of date._
Evidence and run details
  • Models: ladder/vectors-oai3, deepseek-v4.1-flash, deepseek-v4-flash
  • Spend: $0.025082
  • Completion usage: incomplete; some provider completion costs are unavailable.
  • Tokens: 748732 input · 82368 output · 269948 cached · 529 embedding
  • Updated: 2026-11-11T11:37:53Z
Head State Pass summary
337370b4cfaf ready for maintainer review 0 active finding(s), 0 resolved finding(s) (at 2026-11-11T11:37:53Z)

tinysweeper 0.1.2

@tinysweeper tinysweeper Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tinysweeper found nothing blocking. Approving.

Diagnostics

Total spend: $0.0251

@tinysweeper tinysweeper Bot added the priority: p3 Whenever. Cosmetic, a nicety, or a cleanup with no user visible effect. label Nov 11, 2026
@M3gA-Mind
M3gA-Mind merged commit dc2525e into tinyhumansai:main Nov 11, 2026
9 checks passed
M3gA-Mind added a commit to M3gA-Mind/tinyflows that referenced this pull request Nov 11, 2026
Resolves the CHANGELOG conflict with the engine fixes that landed on
main (tinyhumansai#192-tinyhumansai#195): both Unreleased entries are kept.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority: p3 Whenever. Cosmetic, a nicety, or a cleanup with no user visible effect.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant