From fb8ad0bde0491e1698715d6b07c5be02463e3977 Mon Sep 17 00:00:00 2001 From: Mike Langmayr Date: Fri, 6 Nov 2026 11:33:13 -0800 Subject: [PATCH] Ask for the host role, give feiserver the whole camera stack, and document the tracking camera --- docs/build_notes/FEI_build/index.rst | 4 +- docs/build_notes/FEI_build/software_build.rst | 65 +------ .../build_notes/FEI_build/tracking_camera.rst | 162 ++++++++++++++++++ docs/install.md | 14 +- systemd/install.sh | 83 ++++++++- 5 files changed, 256 insertions(+), 72 deletions(-) create mode 100644 docs/build_notes/FEI_build/tracking_camera.rst diff --git a/docs/build_notes/FEI_build/index.rst b/docs/build_notes/FEI_build/index.rst index 3bc8d28..ff6db64 100644 --- a/docs/build_notes/FEI_build/index.rst +++ b/docs/build_notes/FEI_build/index.rst @@ -11,13 +11,14 @@ HISPEC FEI Server: Headless Real-time Build A COTS ``x86_64`` server running a ``PREEMPT_RT`` kernel with Intel TCC, hard CPU shielding and stripped services, dedicated to the camera/controller loop. -Follow the three parts in order: +Follow the four parts in order: #. **Physical build:** hardware assembly and modifications. #. **RT build:** BIOS, TCC and hardware RAID, OS install, network, RT kernel and CPU shielding. #. **Software build:** system packages, Python environment, drivers and instrument utilities. +#. **Tracking camera:** installing, running and changing the camera software. .. toctree:: :maxdepth: 1 @@ -25,3 +26,4 @@ Follow the three parts in order: physical_build rt_build software_build + tracking_camera diff --git a/docs/build_notes/FEI_build/software_build.rst b/docs/build_notes/FEI_build/software_build.rst index 828515c..b9378f3 100644 --- a/docs/build_notes/FEI_build/software_build.rst +++ b/docs/build_notes/FEI_build/software_build.rst @@ -88,44 +88,17 @@ Prefer running Qt GUIs on an operator workstation. 2. Python Environment ===================== -ImageStreamIO -------------- - -The tracking camera publishes frames to shared memory through -`ImageStreamIO `_, which no distro -packages: - -.. code-block:: bash - - git clone https://github.com/milk-org/ImageStreamIO.git - cmake -S ImageStreamIO -B ImageStreamIO/build - cmake --build ImageStreamIO/build -j - sudo cmake --install ImageStreamIO/build - sudo ldconfig - -Upstream installs its CMake config straight into ``/usr/local/lib/cmake``, -where ``find_package`` does not look. Point every build on this host at it, -including the ones ``install.sh`` runs under ``sudo``: - -.. code-block:: bash - - echo 'ImageStreamIO_ROOT=/usr/local/lib/cmake' | sudo tee -a /etc/environment - -Log out and back in for it to take effect. - HISPEC Software --------------- -Install the HISPEC software as described in :doc:`../../install`, with the -tracking camera module: +Install the HISPEC software as described in :doc:`../../install`: .. code-block:: bash - sudo /opt/hispec/app/systemd/install.sh hsfei --host-role feiserver --extra atcam --extra scripts + sudo /opt/hispec/app/systemd/install.sh hsfei --host-role feiserver -``--extra scripts`` adds what the tracking camera analysis scripts plot with. -Both are recorded in ``/etc/hispec/extras``, so later runs keep them without -repeating the flags. +The ``feiserver`` role brings the tracking camera module, ImageStreamIO, atlas +and the analysis script extras; :doc:`tracking_camera` covers them. .. warning:: ``PyQt5`` comes from ``apt``, not the venv. @@ -272,33 +245,11 @@ FT4222 udev Rules lsusb | grep 0403:601c # board detected ls -l /dev/bus/usb/*/* | grep -i 0403 # hsfei dialout 0660 -CameraD (camera-interface) --------------------------- - -.. code-block:: bash - - cd ~ - git clone https://github.com/CaltechOpticalObservatories/camera-interface.git - cd camera-interface - git submodule update --init camerad/Instruments/hispec_tracking_camera - - cd build - rm -rf ./* - cmake .. -DCONTROLLER=archon -DINSTRUMENT=hispec_tracking_camera - make -j"$(nproc)" - - # Record both commits in the as-built log - git -C ~/camera-interface rev-parse --short HEAD - git -C ~/camera-interface/camerad/Instruments/hispec_tracking_camera rev-parse --short HEAD - -``-DINSTRUMENT`` loads ``camerad/Instruments/hispec_tracking_camera``, so the -submodule must be checked out first. ``make`` builds ``camerad``, -``emulator``, ``listener`` and ``socksend`` into ``~/camera-interface/bin``. -The isolated cores are already excluded from normal scheduling -(:ref:`section-grub`), so the build stays on housekeeping cores without -``taskset``. +Tracking Camera +--------------- -See ``archongui.rst`` for Archon-side configuration. +The camera software comes with the ``feiserver`` role; see +:doc:`tracking_camera`. See ``archongui.rst`` for Archon-side configuration. ---- diff --git a/docs/build_notes/FEI_build/tracking_camera.rst b/docs/build_notes/FEI_build/tracking_camera.rst new file mode 100644 index 0000000..6020bf0 --- /dev/null +++ b/docs/build_notes/FEI_build/tracking_camera.rst @@ -0,0 +1,162 @@ +=============================================== +Tracking Camera: Install, Run and Change on fei +=============================================== + +:Prerequisite: :doc:`software_build` complete +:Host role: ``feiserver`` + +.. contents:: + :depth: 1 + :local: + +---- + +1. What Runs +============ + +``hsfei_atcam`` (``daemons/generic/camera``) drives the H2RG on the Archon +through ``camera_interface_tracking``, a compiled module built from the private +`hispec-tracking-camera-instrument +`_ +against `camera-interface +`_. Each frame +goes to the ImageStreamIO stream ``hispec_tracking_camera`` in ``/dev/shm`` and, +with FITS on, to ``/opt/hispec/data``. `atlas +`_ views either. + +A running ``hsfei_atcam`` holds the Archon, so nothing else can drive the +camera beside it, ``run_camerad_script.py`` included. Stop it first with +``hispec stop hsfei_atcam``. + +---- + +2. Install +========== + +.. code-block:: bash + + sudo /opt/hispec/app/systemd/install.sh hsfei --host-role feiserver + +On the first run without ``--host-role``, ``install.sh`` asks for the role and +records it in ``/etc/hispec/host``; later runs read it from there. The +``feiserver`` role brings: + +.. list-table:: + :header-rows: 1 + :widths: 35 65 + + * - What + - Where + * - ImageStreamIO, built when missing + - ``/usr/local`` + * - The camera module (``atcam`` extra), built from the clone + - ``/opt/hispec/extra/hispec-tracking-camera-instrument`` + * - atlas (``atlas`` extra), installed editable from the clone + - ``/opt/hispec/extra/atlas`` + * - Plotting for the analysis scripts (``scripts`` extra) + - ``/opt/hispec/venv`` + +``install.sh`` clones the two repos only when they are missing; an existing +clone is never pulled or reset, so edits in it survive every run. + +Then create the image directory and deploy the daemon: + +.. code-block:: bash + + sudo install -d -o hispec -g hispec-ops -m 2775 /opt/hispec/data + hispec deploy hsfei_atcam + +Check it: + +.. code-block:: bash + + hispec status hsfei_atcam + libby show hsfei.atcam.state # idle + atlas --profile hispec_fei # needs ssh -Y + +---- + +3. Changing the Camera Config +============================= + +The daemon reads ``/opt/hispec/app/config/hsfei/hispecatc.cfg`` in place. +Edit it there, then ``hispec restart hsfei_atcam``. The keys usually changed: + +.. list-table:: + :header-rows: 1 + :widths: 35 65 + + * - Key + - Effect + * - ``FITS_ENABLED`` + - write FITS files, ``yes`` or ``no`` + * - ``FITS_OUTPUT_DIR``, ``FITS_AUTODIR``, ``FITS_BASENAME`` + - where FITS files go, in a ``YYYYMMDD`` subdirectory when ``FITS_AUTODIR=yes`` + * - ``SHM_ENABLED``, ``SHM_SEGMENT_NAME``, ``SHM_DIR`` + - the shared-memory stream readers attach to + * - ``IMDIR``, ``BASENAME`` + - where camerad itself writes images + * - ``LOGPATH`` + - camerad's log directory + +Paths must stay outside ``/home`` and ``/tmp``, which the ``hispec@`` unit +hides. To keep an edit, save it with ``hispec hotfix "what you changed"``, +which commits it to a hotfix branch and opens a pull request. + +---- + +4. Changing the Camera Software +=============================== + +Each clone in ``/opt/hispec/extra/`` replaces the GitHub build of its package: + +.. list-table:: + :header-rows: 1 + :widths: 35 65 + + * - Clone + - After an edit + * - ``hispec-tracking-camera-instrument`` + - ``sudo /opt/hispec/app/systemd/install.sh``, then ``hispec restart hsfei_atcam`` + * - ``camera-interface`` (opt in, below) + - the same + * - ``atlas`` + - restart ``atlas`` + +To change camera-interface too, clone it and re-run ``install.sh``, which then +builds the camera module against it: + +.. code-block:: bash + + sudo git clone https://github.com/CaltechOpticalObservatories/camera-interface.git \ + /opt/hispec/extra/camera-interface + sudo /opt/hispec/app/systemd/install.sh + +Keep a change with a commit, push and pull request from inside the clone. +``hispec hotfix`` does not cover these repos, and the host's deploy key only +reads, so pushing needs your own GitHub account. ``hispec doctor`` lists each +clone with its branch and anything uncommitted or not on GitHub. + +Edits to hispec or a driver under ``/opt/hispec/app`` follow +:doc:`../../install`: ``install.sh`` installs them as they are, and +``hispec hotfix`` sends a driver edit to the driver's own repository. + +---- + +5. Troubleshooting +================== + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - Symptom + - Resolution + * - ``libby show`` fails with ``Connection refused`` + - The CLI defaults to a broker on ``localhost``. Point it at the shared one: + ``mkdir -p ~/.libby && echo 'rabbitmq_url: amqp://hispec.caltech.edu' > ~/.libby/cli_config.yaml`` + * - ``run_camerad_script.py`` cannot reach the Archon + - ``hsfei_atcam`` holds it: ``hispec stop hsfei_atcam`` first. + * - The camera module fails to build: ImageStreamIO not found + - Re-run ``install.sh``, which builds it when + ``/usr/local/lib/cmake/ImageStreamIOConfig.cmake`` is missing. diff --git a/docs/install.md b/docs/install.md index 5407068..46eb44a 100644 --- a/docs/install.md +++ b/docs/install.md @@ -61,14 +61,15 @@ activating anything. ``` `--host-role` is the name instance files use in `HISPEC_HOST` - (`hispecserver`, `feiserver`, ...). Leave it out to use the short hostname. - It is written once, to `/etc/hispec/host`; edit that file to change it. + (`hispecserver`, `feiserver`, ...). Leave it out and `install.sh` asks for + it. It is written once, to `/etc/hispec/host`; edit that file to change it. Some extras come with the role. On `feiserver` the installer builds the tracking camera module into the venv (the `atcam` extra), which any other - role is refused. On `feiserver` and `hispecserver` it installs atlas, the - shared-memory and FITS viewer (the `atlas` extra), which needs a graphical - session such as `ssh -Y`. Other extras are added with `--extra ` and + role is refused, along with ImageStreamIO and the `scripts` extra; see + [the tracking camera page](build_notes/FEI_build/tracking_camera.rst). On + `feiserver` and `hispecserver` it installs atlas, the shared-memory and FITS + viewer (the `atlas` extra), which needs a graphical session such as `ssh -Y`. Other extras are added with `--extra ` and recorded in `/etc/hispec/extras`, so later runs keep them; `hispec doctor` reports one that has gone missing. @@ -153,7 +154,8 @@ Adding a daemon that the checkout already contains does not need it: Extras install from GitHub, so a host has no source for them to edit. To change one on the host that runs it, clone it into `/opt/hispec/extra/`; `install.sh` -then builds that extra from the clone instead: +then builds that extra from the clone instead. On `feiserver` it clones the +camera module and atlas there itself when they are missing: | Clone | Replaces | After an edit | |---|---|---| diff --git a/systemd/install.sh b/systemd/install.sh index c071d98..85d67fc 100755 --- a/systemd/install.sh +++ b/systemd/install.sh @@ -2,7 +2,8 @@ # Host setup for running the hispec daemons under systemd. For admins / IT. # Run as root. Safe to re-run. # -# sudo ./systemd/install.sh # set the host up +# sudo ./systemd/install.sh # set the host up, asking its role once +# sudo ./systemd/install.sh --host-role feiserver # ... naming the role instead # sudo ./systemd/install.sh alice bob # ... and enrol two operators # sudo HISPEC_OPS_USERS="alice bob" ./systemd/install.sh # same thing # sudo ./systemd/install.sh --extra scripts # ... with an optional extra @@ -13,14 +14,19 @@ # its submodules, refusing a checkout with edits. The venv is synced from the # checkout's uv.lock, so every git dependency is the exact commit it records. # +# The role (feiserver, hispecserver, ...) is recorded in /etc/hispec/host on the +# first run and read from there after; edit that file to change it. +# # Extras are recorded in /etc/hispec/extras, and every later run syncs them # again. Some follow the host role: feiserver always gets atcam, the tracking -# camera module, which any other role is refused; feiserver and hispecserver -# always get atlas, the viewer. +# camera module, which any other role is refused, and scripts; feiserver and +# hispecserver always get atlas, the viewer. # # A clone in /opt/hispec/extra/ replaces the GitHub build of its extra, for # changing that code on this host; see docs/install.md. Remove the clone and -# re-run to go back. +# re-run to go back. On feiserver the camera module and atlas are cloned there +# when missing, and ImageStreamIO is built into /usr/local when missing; an +# existing clone is never pulled or reset. # # Re-run it to enrol operators, after a dependency change, or to repair a host. # It is NOT part of adding a daemon: once a host is set up, operators do that @@ -43,12 +49,18 @@ EXTRAS_FILE="${HISPEC_ETC_DIR:-/etc/hispec}/extras" HOST_FILE="${HISPEC_ETC_DIR:-/etc/hispec}/host" CAMERA_EXTRA=atcam CAMERA_ROLE=feiserver +CAMERA_ROLE_EXTRAS=(scripts) VIEWER_EXTRA=atlas VIEWER_ROLES=(feiserver hispecserver) EXTRA_DIR="${HISPEC_EXTRA_DIR:-/opt/hispec/extra}" INSTRUMENT_SRC="$EXTRA_DIR/hispec-tracking-camera-instrument" CORE_SRC="$EXTRA_DIR/camera-interface" VIEWER_SRC="$EXTRA_DIR/atlas" +INSTRUMENT_URL=git@github.com:CaltechOpticalObservatories/hispec-tracking-camera-instrument.git +VIEWER_URL=https://github.com/CaltechOpticalObservatories/atlas.git +IMAGESTREAMIO_URL=https://github.com/milk-org/ImageStreamIO.git +IMAGESTREAMIO_REF="${HISPEC_IMAGESTREAMIO_REF:-3b366508f79f996fc47c9006b09e074a2e6c9bc9}" +IMAGESTREAMIO_CONFIG=/usr/local/lib/cmake/ImageStreamIOConfig.cmake DEPLOY_KEY="${HISPEC_ETC_DIR:-/etc/hispec}/deploy-key" UV_VERSION="${HISPEC_UV_VERSION:-0.12.23}" # Its own copy rather than whatever uv is on PATH, since a snap refreshes @@ -99,11 +111,33 @@ done ROLE="" if [[ -f "$HOST_FILE" ]]; then ROLE=$(tr -d '[:space:]' <"$HOST_FILE") + if [[ -n $ROLE_ARG && $ROLE_ARG != "$ROLE" ]]; then + echo "warning: this host is recorded as '$ROLE' in $HOST_FILE, so --host-role $ROLE_ARG is ignored; edit that file to change it." >&2 + fi +fi +ROLE=${ROLE:-$ROLE_ARG} +if [[ -z $ROLE ]]; then + mapfile -t KNOWN_ROLES < <(sed -n 's/^HISPEC_HOST=//p' "$REPO_DIR"/systemd/instances/*.env | sort -u) + if [[ ! -t 0 || ${#KNOWN_ROLES[@]} -eq 0 ]]; then + echo "error: no host role yet; pass --host-role , one of: ${KNOWN_ROLES[*]}" >&2 + exit 1 + fi + echo "Which role does this host play? It is recorded in $HOST_FILE." + PS3="role: " + select ROLE in "${KNOWN_ROLES[@]}"; do + [[ -n $ROLE ]] && break + done + if [[ -z $ROLE ]]; then + echo "error: no role chosen." >&2 + exit 1 + fi +fi +if [[ -z $ROLE_ARG ]]; then + INSTALL_ARGS+=(--host-role "$ROLE") fi -ROLE=${ROLE:-${ROLE_ARG:-$(hostname -s)}} if [[ $ROLE == "$CAMERA_ROLE" ]]; then - EXTRAS+=("$CAMERA_EXTRA") + EXTRAS+=("$CAMERA_EXTRA" "${CAMERA_ROLE_EXTRAS[@]}") elif printf '%s\n' "${EXTRAS[@]}" | grep -qx "$CAMERA_EXTRA"; then echo "error: the $CAMERA_EXTRA extra is for $CAMERA_ROLE only; this host is '$ROLE'." >&2 echo "Drop --extra $CAMERA_EXTRA, or remove it from $EXTRAS_FILE." >&2 @@ -159,8 +193,10 @@ if [[ -n $CHECKOUT_REF && -z ${HISPEC_INSTALL_CHECKED_OUT:-} ]]; then git -C "$REPO_DIR" checkout --quiet --detach "$CHECKOUT_REF" fi git -C "$REPO_DIR" submodule update --quiet --init --recursive - # The checkout may have replaced this very script; carry on in its copy - HISPEC_INSTALL_CHECKED_OUT=1 exec "$REPO_DIR/systemd/install.sh" "${ORIGINAL_ARGS[@]}" + # The checkout may have replaced this very script; carry on in its copy, + # with the role so it is not asked twice + HISPEC_INSTALL_CHECKED_OUT=1 exec "$REPO_DIR/systemd/install.sh" "${ORIGINAL_ARGS[@]}" \ + --host-role "$ROLE" fi CHECKOUT_DESC=$(git -C "$REPO_DIR" symbolic-ref --short -q HEAD || @@ -170,6 +206,37 @@ if [[ -n $(git -C "$REPO_DIR" status --porcelain --untracked-files=no) ]]; then fi echo "installing $REPO_DIR ($CHECKOUT_DESC)" +# On the camera host, edits to the camera module and atlas are the norm +if [[ $ROLE == "$CAMERA_ROLE" ]]; then + mkdir -p "$EXTRA_DIR" + for clone in "$INSTRUMENT_SRC=$INSTRUMENT_URL" "$VIEWER_SRC=$VIEWER_URL"; do + if [[ ! -d ${clone%%=*} ]]; then + git clone --quiet "${clone#*=}" "${clone%%=*}" + echo "cloned ${clone#*=} into ${clone%%=*}" + fi + done +fi + +# The camera module builds against ImageStreamIO, which no distro packages +if has_extra "$CAMERA_EXTRA" && [[ ! -f $IMAGESTREAMIO_CONFIG ]]; then + for tool in git cmake cc; do + if ! command -v "$tool" >/dev/null; then + echo "error: building ImageStreamIO needs $tool: apt install build-essential cmake git" >&2 + exit 1 + fi + done + IMAGESTREAMIO_BUILD=$(mktemp -d) + git clone --quiet "$IMAGESTREAMIO_URL" "$IMAGESTREAMIO_BUILD/src" + git -C "$IMAGESTREAMIO_BUILD/src" checkout --quiet "$IMAGESTREAMIO_REF" + cmake -S "$IMAGESTREAMIO_BUILD/src" -B "$IMAGESTREAMIO_BUILD/build" \ + -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr/local >/dev/null + cmake --build "$IMAGESTREAMIO_BUILD/build" -j"$(nproc)" >/dev/null + cmake --install "$IMAGESTREAMIO_BUILD/build" >/dev/null + ldconfig + rm -rf "$IMAGESTREAMIO_BUILD" + echo "installed ImageStreamIO ${IMAGESTREAMIO_REF:0:7} in /usr/local" +fi + # Pinned, so re-running the installer cannot resolve with a different uv if [[ "$("$UV_BIN" --version 2>/dev/null)" != "uv $UV_VERSION"* ]]; then curl -LsSf "https://astral.sh/uv/$UV_VERSION/install.sh" |