Skip to content

Pipeline engine refactored into simple flow runner, all Prefect usage removed - #36

Open
Yashvi-Sharma wants to merge 9 commits into
mainfrom
engine-fixes
Open

Yashvi-Sharma wants to merge 9 commits into
mainfrom
engine-fixes

Conversation

@Yashvi-Sharma

Copy link
Copy Markdown
Collaborator

Finished long-pending engine refactor. Will need another refactor after task re-design but running flows using YAMLs now works again.

  • Moved all "pipeline" terminology to "flows".
  • Removed Prefect completely from the flow runner.
  • Refactored pipeline/ to flow/ and split engine.py into nodes.py refs.py and runner.py. Organised code into objects and data classes, removed redundant code.
  • Dry run capability added at flow runner init.
  • Variable resolving capability added to configs/
  • Updated docs, tests, notebooks, etc.

Yashvi-Sharma and others added 9 commits November 13, 2026 14:48
…in images), minor bug fixes, tested linbin flow on DEIMOS data
…ixed bug in JSON serialization chain, fixed NaN handling in image_stats, bug fix and cleanup in ptc
… now supports lists and dicts of result references
… flow everywhere), organized and refactored simple executor, removed Prefect dep
…-iterator lazy source, drop redundant exc in logger.exception

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
source = flow.nodes[flow.source]

# A temporary mapping keeps each batch's results apart from the accumulated ones
batch_results = deepcopy(self.results)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't think a deep copy is needed here, the batch loop only assigns keys, so the deep copy is duplicating every earlier result (such as all the master-bias images) and doubles the peak memory
You can do a shallow copy in this case: batch_results = dict(self.results)

super().__init__(name=name)

def run(self, result) -> TaskResult:
del result

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't free any memory, del result only removes the local name. The runner still holds the result in self.results

Comment thread src/eregion/flow/refs.py
return ref.resolve(results, apply_query=apply_queries)
except (ValueError, KeyError, TypeError) as e:
message = e.args[0] if e.args else str(e)
raise type(e)(f"Input '{arg_path}' for task '{node_name}' {message}") from e

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This rebuilds errors with type(e)(message) which breaks for exceptions whose constructors take other arguments (an example is pydantic’s ValidationError)
Could we use e.add_note(f"Input '{arg_path}' for task '{node_name}'")? That keeps the original type, message and traceback.

Comment on lines +148 to +149

for batch in batches:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if we have a bug for save which runs inside this loop
When we save on disk, it's only the last batch’s PTC table, because batches 1 to N−1 were written and then overwritten.
In memory (runner.results["ptc_flow.make_ptc_table"]), this has the full combined table

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants