Skip to content

Superflat task including both simple and Welford combine methods - #35

Merged
Yashvi-Sharma merged 8 commits into
mainfrom
superflat
Oct 28, 2026
Merged

Yashvi-Sharma merged 8 commits into
mainfrom
superflat

Conversation

@Yashvi-Sharma

Copy link
Copy Markdown
Collaborator

Added SuperCombine task to make superflats within the current task design. Tests for core algorithms and superflat flow tested in notebook.

Two choices for methods:

  • simple mean - stores sum array and sum of squares array and calculates mean and std at the end
  • welford - mean and Robbins-Monro streaming approximation for median

Added DEIMOS FCS detector config as well, example loading in Example_image_loading.ipynb

Added mean of diff pair MADs in PTC stats.

@weatherhead99 weatherhead99 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, very minor comments

Comment thread src/eregion/core/image_operations.py Resolved
Comment thread src/eregion/tasks/imagegen.py Resolved
_check_same_shape(count=count, mean=mean, m2=m2)
data = np.where(count > 0, mean, np.nan)
std = np.sqrt(np.divide(m2, count - 1, out=np.full_like(m2, np.nan, dtype=float), where=count > 1))
return {'data': data, 'std': std, 'count': count.copy()}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

think key "data" here should proabbly be renamed "mean" TBH.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

problem is, to keep DetImage xarray dataset consistent and the api usage in tasks unchanged, I made it so that 'data' data_var must exist in the dataset, and contain the primary data array. Since in this case that is the mean array, it gets set in 'data'.
Maybe I can change data to mean/median here in core and set to the data data_var inside the task?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

huh. Yes that's a bit clunky. I think the most sensible thing to do is rename this key inside the task even though that's annoying - this function should (in principle) be usable separately outside a task and the "mean" name is a bit nicer

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll fix this in next PR

Comment thread src/eregion/core/image_operations.py Resolved
Comment thread src/eregion/tasks/ptc.py Resolved
Comment thread src/eregion/tasks/superflat.py Outdated
return self.final_image_result

@property
def methods(self):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add return type hint here

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@weatherhead99 weatherhead99 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all looks ok to me

@Yashvi-Sharma
Yashvi-Sharma merged commit b2701cf into main Oct 28, 2026
4 checks passed
@Yashvi-Sharma
Yashvi-Sharma deleted the superflat branch October 28, 2026 18:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants