Repository navigation
Keyheader daemon #52
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
scizen9
wants to merge
11
commits into
main
Choose a base branch
from
don/add_keyheader
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Keyheader daemon #52
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
cda42ee
Initial Revision of keyheader: just dir and init file
scizen9 6989ea9
Added InfluxSource class and associated helper functions and tests.
scizen9 e7126f3
Create source.py in keyheader dir and update keygrabber/influx.py to …
scizen9 b8deb34
Add group to query and associated methods
scizen9 0be32a1
Merge remote-tracking branch 'origin/main' into don/add_keyheader
scizen9 45109b8
Merge remote-tracking branch 'origin/main' into don/add_keyheader
scizen9 8919a89
Add the keyheader daemon: FITS header segments from stored keywords
scizen9 dd2cfff
Add an example keyheader config
scizen9 9c910a1
convert yaml to flow style so each keyword is on one line in keyheade…
scizen9 bc0b341
Change uniqueness rule so that it applies only to a given segment. T…
scizen9 8c9db61
Update keyheader.md to include one-line examples
scizen9 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,251 @@ | ||
| # Keyheader | ||
|
|
||
| Keyheader is a libby daemon that captures a snapshot of instrument state at a | ||
| moment and returns it as FITS header cards. A requesting daemon, usually a | ||
| camera daemon, asks for a named **segment** at a time such as the start of an | ||
| exposure, and adds the cards it gets back to its image header. | ||
|
|
||
| The values come from the database the {doc}`keygrabber` fills, so keyheader | ||
| never talks to a hardware daemon and a slow one cannot delay a header. | ||
| InfluxDB 2.x is the only backend today: | ||
|
|
||
| ```bash | ||
| pip install libby[influxdb] | ||
| ``` | ||
|
|
||
| ## Running it | ||
|
|
||
| ```bash | ||
| keyheader -c /etc/hispec/keyheader.yaml | ||
| ``` | ||
|
|
||
| Like the keygrabber, it is an ordinary `LibbyDaemon`: `SIGTERM` or the | ||
| `shutdown` keyword stops it, and it logs to stderr unless the config has a | ||
| `logging` section. | ||
|
|
||
| ## Config | ||
|
|
||
| A keyheader config is an ordinary daemon config plus `source`, `defaults` and | ||
| `segments`. A fuller, commented example ships with the package as | ||
| `libby/keyheader/keyheader.example.yaml`. | ||
|
|
||
| ```yaml | ||
| peer_id: keyheader | ||
| group_id: hispec | ||
| transport: rabbitmq | ||
| rabbitmq_url: amqp://localhost | ||
|
|
||
| source: | ||
| type: influxdb | ||
| url: http://influx.hispec:8086 | ||
| org: hispec | ||
| bucket: telemetry | ||
| token_env: HISPEC_INFLUX_TOKEN | ||
|
|
||
| workers: 4 | ||
|
|
||
| defaults: | ||
| mode: nearest | ||
| window_s: 300 | ||
|
|
||
| segments: | ||
| - name: EXP-BEGIN | ||
| date_keyword: DATE-BEG | ||
| cards: | ||
| - {fits: ADCPOS1, keyword: hsfei.adc.positionvalue1, type: float, format: ".3f", comment: "ADC prism 1 position [deg]", mode: interpolate} | ||
| - {fits: ADCMOVE, keyword: hsfei.adc.ismoving, type: bool, comment: "ADC in motion"} | ||
|
|
||
| - name: EXP-END | ||
| date_keyword: DATE-END | ||
| defaults: | ||
| mode: prior | ||
| cards: | ||
| - {fits: ADCPOS1E, keyword: hsfei.adc.positionvalue1, type: float, format: ".3f", comment: "ADC prism 1 position at end [deg]"} | ||
| ``` | ||
|
|
||
| Each card is one line in YAML flow style, `{key: value, ...}`. Block style | ||
| works too, and the two can be mixed. Inside `{ }` the characters `[ ] { } ,` | ||
| and `: ` have meaning, so quote any value that contains them. An unquoted | ||
| comment such as `ADC position [deg]` is a parse error, which is why the | ||
| comments above are always quoted. | ||
|
|
||
| ### source | ||
|
|
||
| Validated exactly like the keygrabber's `sink`: `type: influxdb`, and the token | ||
| is read from the environment variable `token_env` names. A `token` key in the | ||
| file is rejected. | ||
|
|
||
| ### segments | ||
|
|
||
| `segments` is a list, and so is each segment's `cards`. They are not mappings | ||
| on purpose: YAML keeps only the last of two identical keys without warning, | ||
| so a duplicated FITS keyword written as a key would disappear rather than be | ||
| reported. A mapping is refused with an error that says so. | ||
|
|
||
| A segment has: | ||
|
|
||
| - `name`: letters, digits, `_`, `-` or `.`, at most 40 characters. Requests | ||
| use this name, and it starts the segment's COMMENT card. | ||
| - `date_keyword` (optional): one of `DATE-OBS`, `DATE-BEG`, `DATE-AVG` or | ||
| `DATE-END`. These are the FITS standard's own time keywords. The segment | ||
| writes its timestamp there as well as in the COMMENT. | ||
| - `date_comment` (optional): the comment for that card. | ||
| - `defaults` (optional): `mode` and `window_s` for this segment's cards. These | ||
| override the top-level `defaults`. | ||
| - `cards`: the mapped keywords, in the order they appear in the header. | ||
|
|
||
| A card has: | ||
|
|
||
| | Field | Required | Meaning | | ||
| |---|---|---| | ||
| | `fits` | yes | FITS keyword: 1-8 characters of `A-Z`, `0-9`, `-`, `_` | | ||
| | `keyword` | yes | Daemon keyword, `<group>.<daemon>.<keyword>` | | ||
| | `type` | yes | `float`, `int`, `bool` or `str` | | ||
| | `format` | no | Python format spec, e.g. `.3f`, `.2e`, `.20` | | ||
| | `comment` | no | Card comment, at most 47 printable ASCII characters | | ||
| | `mode` | no | `interpolate`, `nearest`, `prior` or `subsequent` | | ||
| | `window_s` | no | How far from the moment to look for a sample | | ||
|
|
||
| ### Validation | ||
|
|
||
| Config load rejects: | ||
|
|
||
| - a FITS keyword that breaks the naming rules | ||
| - a keyword the FITS 4.0 standard reserves. This covers the structural ones | ||
| (`SIMPLE`, `NAXISn`, `END`, ...), commentary (`COMMENT`, `HISTORY`, | ||
| `CONTINUE`), the general descriptive ones (`OBJECT`, `TELESCOP`, `DATE`, | ||
| ...), and the WCS and time keywords. Keyheader writes COMMENT itself. | ||
| - a FITS keyword used twice within one segment. Different segments may use | ||
| the same keywords, so several camera daemons can each have their own | ||
| segments with the same names. A daemon that adds two segments to one header | ||
| must pick segments that do not overlap, since FITS forbids a repeated | ||
| keyword. | ||
| - a repeated segment name | ||
| - `interpolate` on a `bool` or `str` card | ||
| - a `format` the type cannot take, or a numeric format whose output does not | ||
| read back as a number (e.g. `,.2f`) | ||
| - a comment longer than 47 characters, the space left beside a fixed-format | ||
| value | ||
|
|
||
| A bad config stops the daemon at startup, and a bad file given to `reload` | ||
| leaves the running segments in place. | ||
|
|
||
| ## Acquisition modes | ||
|
|
||
| Every card is looked up independently, within `window_s` of the requested | ||
| moment: | ||
|
|
||
| - `prior`: the latest sample at or before the moment | ||
| - `subsequent`: the earliest sample at or after the moment | ||
| - `nearest`: whichever of those two is closer. A tie goes to the earlier | ||
| sample, which was already true at the moment. | ||
| - `interpolate`: linear interpolation between the samples either side. When | ||
| only one side exists, it uses that value. | ||
|
|
||
| That fallback matters at the end of an exposure. A request made then normally | ||
| arrives before the keygrabber has stored the next sample, and most values do | ||
| not change during an exposure, so the last stored value stands in. A | ||
| `subsequent` card has no such fallback and is written blank. | ||
|
|
||
| `window_s` should exceed the keygrabber's interval for that keyword, including | ||
| its backoff, or a quiet keyword will come back blank. | ||
|
|
||
| ## Requesting a segment | ||
|
|
||
| The service key is `keyheader.segment`: | ||
|
|
||
| ```python | ||
| reply = libby.rpc("hispec.keyheader", "keyheader.segment", | ||
| {"segment": "EXP-BEGIN", "time": "2026-11-11T08:15:02.123Z"}) | ||
| ``` | ||
|
|
||
| `time` is ISO 8601. A time with no offset is taken as UTC. The reply is: | ||
|
|
||
| ```json | ||
| { | ||
| "ok": true, | ||
| "segment": "EXP-BEGIN", | ||
| "time": "2026-11-11T08:15:02.123", | ||
| "cards": [ | ||
| {"keyword": "COMMENT", "value": "EXP-BEGIN 2026-11-11T08:15:02.123", "comment": ""}, | ||
| {"keyword": "DATE-BEG", "value": "2026-11-11T08:15:02.123", "comment": "UTC time of this segment"}, | ||
| {"keyword": "ADCPOS1", "value": 12.346, "comment": "ADC prism 1 position [deg]"}, | ||
| {"keyword": "ADCMOVE", "value": false, "comment": "ADC in motion"} | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| An unknown segment or a malformed request comes back as `ok: false` with an | ||
| `error`. | ||
|
|
||
| ### Adding the cards with astropy | ||
|
|
||
| Keyheader does not write card images. The requesting daemon passes each entry | ||
| to astropy as a tuple: | ||
|
|
||
| ```python | ||
| cards = [(c["keyword"], c["value"], c["comment"]) for c in reply["cards"]] | ||
| header.extend(cards, end=True) | ||
| ``` | ||
|
|
||
| Use `end=True`, or build a fresh `fits.Header(cards)`. Plain | ||
| `header.extend(cards)` inserts keyword cards ahead of trailing commentary | ||
| cards, which moves the segment's COMMENT below the cards it introduces. | ||
|
|
||
| The requesting daemon is still responsible for checking the final header | ||
| against the FITS standard. Keyheader only checks keywords within a segment, | ||
| so a daemon that adds more than one segment to a header must make sure they | ||
| use different keywords, or replace a card rather than appending a duplicate. | ||
|
|
||
| ### Values and formats | ||
|
|
||
| A value is typed JSON, and astropy decides the column layout. `format` | ||
| therefore sets precision rather than layout: the value is formatted and then | ||
| parsed back, so `.3f` turns `12.34567` into `12.346`. Other conversions: | ||
|
|
||
| - An `int` card rounds, because InfluxDB stores every integer as a float. | ||
| - A `bool` card accepts a stored `0` or `1`. | ||
| - A `str` format can truncate, e.g. `.20`. | ||
|
|
||
| ### Blank cards | ||
|
|
||
| A card is never left out. When there is no value, it is written with a FITS | ||
| undefined value (`null` in the reply, which astropy writes as a blank value), | ||
| and its comment starts with the reason: | ||
|
|
||
| | Reason | Cause | | ||
| |---|---| | ||
| | `no sample`, `no prior sample`, `no later sample` | Nothing stored in the window | | ||
| | `read failed` | The database query failed | | ||
| | `not a float`, `not a bool`, ... | The stored value cannot be the configured type | | ||
| | `not finite` | NaN or infinity, which FITS cannot represent | | ||
| | `not numeric` | Interpolation between non-numbers | | ||
|
|
||
| The reason and the original comment together are truncated to 47 characters. | ||
| One failed card never fails the request: losing one value should not cost the | ||
| exposure its whole header. | ||
|
|
||
| ### Latency | ||
|
|
||
| The cards in a segment are read in parallel on `workers` threads. A request | ||
| therefore costs about one database round trip, plus queueing when a segment | ||
| has many more cards than workers. The request is handled on the transport's | ||
| receive thread, so it holds up other requests to keyheader for that long. | ||
| Give the RPC a TTL that covers the database's `timeout_ms`. | ||
|
|
||
| ## Control keywords | ||
|
|
||
| | Keyword | Type | Access | Meaning | | ||
| |---|---|---|---| | ||
| | `isconnected` | bool | R | Last source read succeeded | | ||
| | `requests` | int | R | Segment requests answered since start | | ||
| | `readerrors` | int | R | Source reads that failed | | ||
| | `blankcards` | int | R | Cards answered with an undefined value | | ||
| | `segments` | string | R | Configured segment names, comma separated | | ||
| | `reload` | trigger | W | Re-read the config file and apply its segments | | ||
| | `shutdown` | trigger | W | Gracefully stop the daemon | | ||
|
|
||
| `reload` can add, remove or change segments without a restart, because | ||
| segments are not keywords. A change to `source` or `workers` is refused with a | ||
| request to restart instead. A request already in progress finishes with the | ||
| segments it started with. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
keygrabber stamps each sample with the time the poll started (daemon.py:366), not when the hardware reported the value. At EXP-END, the newest stored “prior” sample can be up to one poll interval plus the write delay old, or more if the keygrabber is backed off. Might be best to explain that prior means "last stored sample", not "the state at that moment"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it be possible to change this behavior to stamping with the time when the hardware reported the value? This seems desireable.