Skip to content

feat: builtin group measurements flow transformer#4201

Merged
didier-wenzek merged 3 commits into
thin-edge:mainfrom
didier-wenzek:feat/builtin-group-measurements-flow-transformer
Jun 3, 2026
Merged

feat: builtin group measurements flow transformer#4201
didier-wenzek merged 3 commits into
thin-edge:mainfrom
didier-wenzek:feat/builtin-group-measurements-flow-transformer

Conversation

@didier-wenzek
Copy link
Copy Markdown
Contributor

Proposed changes

  • Add a new group-measurements builtin flow transformer
  • Improve the flow engine to decode a message timestamp set by a script (message timestamp were previously ignored)

Types of changes

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
  • Documentation Update (if none of the other choices apply)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

This PR has been extracted from #4136, the point being to ease the release of a new version of tedge deprecating the collectd mapper

  1. First step (this PR): add flow transformers useful for collectd messages processing
  2. Merge collectd setup using flows - but without any builtin
  3. When this PR is release along 2.0.2, update the collectd setup to use group-measurements builtin
  4. Merge refactor: replace the collectd mapper with local flows #4136 fully deprecating the collectd mapper

Checklist

  • I have read the CONTRIBUTING doc
  • I have signed the CLA (in all commits with git commit -s. You can activate automatic signing by running just prepare-dev once)
  • I ran just format as mentioned in CODING_GUIDELINES
  • I used just check as mentioned in CODING_GUIDELINES
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

Further comments

@codecov
Copy link
Copy Markdown

codecov Bot commented Jun 2, 2026

Codecov Report

❌ Patch coverage is 91.06700% with 36 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...tedge_flows/src/transformers/group_measurements.rs 91.56% 24 Missing and 3 partials ⚠️
crates/extensions/tedge_flows/src/flow.rs 43.75% 9 Missing ⚠️

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 2, 2026

Robot Results

✅ Passed ❌ Failed ⏭️ Skipped Total Pass % ⏱️ Duration
931 0 7 931 100 2h33m0.490087s

Comment on lines +51 to +54
Execute Command tedge mqtt pub collectd/localhost/temperature/temp1 "`date +%s.%N`:50"
Execute Command tedge mqtt pub collectd/localhost/temperature/temp2 "`date +%s.%N`:40"
Execute Command tedge mqtt pub collectd/localhost/pressure/pres1 "`date +%s.%N`:10"
Execute Command tedge mqtt pub collectd/localhost/pressure/pres2 "`date +%s.%N`:20"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whilst having individual commands is much easier to read, it might result in some timing issues as each Execute Command has to do an underlying docker api call, and wait for the response, so I wouldn't be surprised if this could take longer than 1 second when the CI runner is under load.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest that we just update the flow interval and config.time_window to higher values so that we can write manual tests like this better.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I decided to simply revert my change - i.e. to send all the collectd messages in a single Execute Command

Copy link
Copy Markdown
Contributor

@albinsuresh albinsuresh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Comment on lines +51 to +54
Execute Command tedge mqtt pub collectd/localhost/temperature/temp1 "`date +%s.%N`:50"
Execute Command tedge mqtt pub collectd/localhost/temperature/temp2 "`date +%s.%N`:40"
Execute Command tedge mqtt pub collectd/localhost/pressure/pres1 "`date +%s.%N`:10"
Execute Command tedge mqtt pub collectd/localhost/pressure/pres2 "`date +%s.%N`:20"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest that we just update the flow interval and config.time_window to higher values so that we can write manual tests like this better.

Comment on lines +616 to +621
```toml
[[steps]]
builtin = "group-measurements
interval = "1s"
config.time_window = "500ms"
```
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any specific reason to have both the in-line version as well as the standard array representations for just this keyword? The doc of other transformers only had the inline variant, hence asking.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No specific reason.

Comment thread crates/extensions/tedge_flows/src/transformers/group_measurements.rs Outdated
Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>
A value created in JS using `new Date()` cannot be directly returned to the flow engine.
A script has then to invoke `valueOf()` method to assign a date to a message timestamp.

Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>
Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>
@didier-wenzek didier-wenzek force-pushed the feat/builtin-group-measurements-flow-transformer branch from c1dbed5 to 03aa29d Compare June 3, 2026 11:56
@didier-wenzek didier-wenzek temporarily deployed to Test Pull Request June 3, 2026 11:56 — with GitHub Actions Inactive
@didier-wenzek didier-wenzek enabled auto-merge June 3, 2026 11:59
@didier-wenzek didier-wenzek added this pull request to the merge queue Jun 3, 2026
Merged via the queue into thin-edge:main with commit 6af1365 Jun 3, 2026
34 checks passed
@didier-wenzek didier-wenzek deleted the feat/builtin-group-measurements-flow-transformer branch June 3, 2026 13:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants