Repository navigation
Pipeline engine refactored into simple flow runner, all Prefect usage removed - #36
Yashvi-Sharma wants to merge 9 commits into
Conversation
…in images), minor bug fixes, tested linbin flow on DEIMOS data
…ests, added DEIMOS test notebook
…ixed bug in JSON serialization chain, fixed NaN handling in image_stats, bug fix and cleanup in ptc
…tirely, added tests
… now supports lists and dicts of result references
… flow everywhere), organized and refactored simple executor, removed Prefect dep
…-iterator lazy source, drop redundant exc in logger.exception Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| source = flow.nodes[flow.source] | ||
|
|
||
| # A temporary mapping keeps each batch's results apart from the accumulated ones | ||
| batch_results = deepcopy(self.results) |
There was a problem hiding this comment.
Don't think a deep copy is needed here, the batch loop only assigns keys, so the deep copy is duplicating every earlier result (such as all the master-bias images) and doubles the peak memory
You can do a shallow copy in this case: batch_results = dict(self.results)
| super().__init__(name=name) | ||
|
|
||
| def run(self, result) -> TaskResult: | ||
| del result |
There was a problem hiding this comment.
This doesn't free any memory, del result only removes the local name. The runner still holds the result in self.results
| return ref.resolve(results, apply_query=apply_queries) | ||
| except (ValueError, KeyError, TypeError) as e: | ||
| message = e.args[0] if e.args else str(e) | ||
| raise type(e)(f"Input '{arg_path}' for task '{node_name}' {message}") from e |
There was a problem hiding this comment.
This rebuilds errors with type(e)(message) which breaks for exceptions whose constructors take other arguments (an example is pydantic’s ValidationError)
Could we use e.add_note(f"Input '{arg_path}' for task '{node_name}'")? That keeps the original type, message and traceback.
|
|
||
| for batch in batches: |
There was a problem hiding this comment.
Wondering if we have a bug for save which runs inside this loop
When we save on disk, it's only the last batch’s PTC table, because batches 1 to N−1 were written and then overwritten.
In memory (runner.results["ptc_flow.make_ptc_table"]), this has the full combined table
Finished long-pending engine refactor. Will need another refactor after task re-design but running flows using YAMLs now works again.