Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion docs/build_notes/FEI_build/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,17 +11,19 @@ HISPEC FEI Server: Headless Real-time Build

A COTS ``x86_64`` server running a ``PREEMPT_RT`` kernel with Intel TCC, hard
CPU shielding and stripped services, dedicated to the camera/controller loop.
Follow the three parts in order:
Follow the four parts in order:

#. **Physical build:** hardware assembly and modifications.
#. **RT build:** BIOS, TCC and hardware RAID, OS install, network, RT kernel and
CPU shielding.
#. **Software build:** system packages, Python environment, drivers and
instrument utilities.
#. **Tracking camera:** installing, running and changing the camera software.

.. toctree::
:maxdepth: 1

physical_build
rt_build
software_build
tracking_camera
65 changes: 8 additions & 57 deletions docs/build_notes/FEI_build/software_build.rst
Original file line number Diff line number Diff line change
Expand Up @@ -88,44 +88,17 @@ Prefer running Qt GUIs on an operator workstation.
2. Python Environment
=====================

ImageStreamIO
-------------

The tracking camera publishes frames to shared memory through
`ImageStreamIO <https://github.com/milk-org/ImageStreamIO>`_, which no distro
packages:

.. code-block:: bash

git clone https://github.com/milk-org/ImageStreamIO.git
cmake -S ImageStreamIO -B ImageStreamIO/build
cmake --build ImageStreamIO/build -j
sudo cmake --install ImageStreamIO/build
sudo ldconfig

Upstream installs its CMake config straight into ``/usr/local/lib/cmake``,
where ``find_package`` does not look. Point every build on this host at it,
including the ones ``install.sh`` runs under ``sudo``:

.. code-block:: bash

echo 'ImageStreamIO_ROOT=/usr/local/lib/cmake' | sudo tee -a /etc/environment

Log out and back in for it to take effect.

HISPEC Software
---------------

Install the HISPEC software as described in :doc:`../../install`, with the
tracking camera module:
Install the HISPEC software as described in :doc:`../../install`:

.. code-block:: bash

sudo /opt/hispec/app/systemd/install.sh hsfei --host-role feiserver --extra atcam --extra scripts
sudo /opt/hispec/app/systemd/install.sh hsfei --host-role feiserver

``--extra scripts`` adds what the tracking camera analysis scripts plot with.
Both are recorded in ``/etc/hispec/extras``, so later runs keep them without
repeating the flags.
The ``feiserver`` role brings the tracking camera module, ImageStreamIO, atlas
and the analysis script extras; :doc:`tracking_camera` covers them.

.. warning::
``PyQt5`` comes from ``apt``, not the venv.
Expand Down Expand Up @@ -272,33 +245,11 @@ FT4222 udev Rules
lsusb | grep 0403:601c # board detected
ls -l /dev/bus/usb/*/* | grep -i 0403 # hsfei dialout 0660

CameraD (camera-interface)
--------------------------

.. code-block:: bash

cd ~
git clone https://github.com/CaltechOpticalObservatories/camera-interface.git
cd camera-interface
git submodule update --init camerad/Instruments/hispec_tracking_camera

cd build
rm -rf ./*
cmake .. -DCONTROLLER=archon -DINSTRUMENT=hispec_tracking_camera
make -j"$(nproc)"

# Record both commits in the as-built log
git -C ~/camera-interface rev-parse --short HEAD
git -C ~/camera-interface/camerad/Instruments/hispec_tracking_camera rev-parse --short HEAD

``-DINSTRUMENT`` loads ``camerad/Instruments/hispec_tracking_camera``, so the
submodule must be checked out first. ``make`` builds ``camerad``,
``emulator``, ``listener`` and ``socksend`` into ``~/camera-interface/bin``.
The isolated cores are already excluded from normal scheduling
(:ref:`section-grub`), so the build stays on housekeeping cores without
``taskset``.
Tracking Camera
---------------

See ``archongui.rst`` for Archon-side configuration.
The camera software comes with the ``feiserver`` role; see
:doc:`tracking_camera`. See ``archongui.rst`` for Archon-side configuration.

----

Expand Down
162 changes: 162 additions & 0 deletions docs/build_notes/FEI_build/tracking_camera.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
===============================================
Tracking Camera: Install, Run and Change on fei
===============================================

:Prerequisite: :doc:`software_build` complete
:Host role: ``feiserver``

.. contents::
:depth: 1
:local:

----

1. What Runs
============

``hsfei_atcam`` (``daemons/generic/camera``) drives the H2RG on the Archon
through ``camera_interface_tracking``, a compiled module built from the private
`hispec-tracking-camera-instrument
<https://github.com/CaltechOpticalObservatories/hispec-tracking-camera-instrument>`_
against `camera-interface
<https://github.com/CaltechOpticalObservatories/camera-interface>`_. Each frame
goes to the ImageStreamIO stream ``hispec_tracking_camera`` in ``/dev/shm`` and,
with FITS on, to ``/opt/hispec/data``. `atlas
<https://github.com/CaltechOpticalObservatories/atlas>`_ views either.

A running ``hsfei_atcam`` holds the Archon, so nothing else can drive the
camera beside it, ``run_camerad_script.py`` included. Stop it first with
``hispec stop hsfei_atcam``.

----

2. Install
==========

.. code-block:: bash

sudo /opt/hispec/app/systemd/install.sh hsfei --host-role feiserver

On the first run without ``--host-role``, ``install.sh`` asks for the role and
records it in ``/etc/hispec/host``; later runs read it from there. The
``feiserver`` role brings:

.. list-table::
:header-rows: 1
:widths: 35 65

* - What
- Where
* - ImageStreamIO, built when missing
- ``/usr/local``
* - The camera module (``atcam`` extra), built from the clone
- ``/opt/hispec/extra/hispec-tracking-camera-instrument``
* - atlas (``atlas`` extra), installed editable from the clone
- ``/opt/hispec/extra/atlas``
* - Plotting for the analysis scripts (``scripts`` extra)
- ``/opt/hispec/venv``

``install.sh`` clones the two repos only when they are missing; an existing
clone is never pulled or reset, so edits in it survive every run.

Then create the image directory and deploy the daemon:

.. code-block:: bash

sudo install -d -o hispec -g hispec-ops -m 2775 /opt/hispec/data
hispec deploy hsfei_atcam

Check it:

.. code-block:: bash

hispec status hsfei_atcam
libby show hsfei.atcam.state # idle
atlas --profile hispec_fei # needs ssh -Y

----

3. Changing the Camera Config
=============================

The daemon reads ``/opt/hispec/app/config/hsfei/hispecatc.cfg`` in place.
Edit it there, then ``hispec restart hsfei_atcam``. The keys usually changed:

.. list-table::
:header-rows: 1
:widths: 35 65

* - Key
- Effect
* - ``FITS_ENABLED``
- write FITS files, ``yes`` or ``no``
* - ``FITS_OUTPUT_DIR``, ``FITS_AUTODIR``, ``FITS_BASENAME``
- where FITS files go, in a ``YYYYMMDD`` subdirectory when ``FITS_AUTODIR=yes``
* - ``SHM_ENABLED``, ``SHM_SEGMENT_NAME``, ``SHM_DIR``
- the shared-memory stream readers attach to
* - ``IMDIR``, ``BASENAME``
- where camerad itself writes images
* - ``LOGPATH``
- camerad's log directory

Paths must stay outside ``/home`` and ``/tmp``, which the ``hispec@`` unit
hides. To keep an edit, save it with ``hispec hotfix "what you changed"``,
which commits it to a hotfix branch and opens a pull request.

----

4. Changing the Camera Software
===============================

Each clone in ``/opt/hispec/extra/`` replaces the GitHub build of its package:

.. list-table::
:header-rows: 1
:widths: 35 65

* - Clone
- After an edit
* - ``hispec-tracking-camera-instrument``
- ``sudo /opt/hispec/app/systemd/install.sh``, then ``hispec restart hsfei_atcam``
* - ``camera-interface`` (opt in, below)
- the same
* - ``atlas``
- restart ``atlas``

To change camera-interface too, clone it and re-run ``install.sh``, which then
builds the camera module against it:

.. code-block:: bash

sudo git clone https://github.com/CaltechOpticalObservatories/camera-interface.git \
/opt/hispec/extra/camera-interface
sudo /opt/hispec/app/systemd/install.sh

Keep a change with a commit, push and pull request from inside the clone.
``hispec hotfix`` does not cover these repos, and the host's deploy key only
reads, so pushing needs your own GitHub account. ``hispec doctor`` lists each
clone with its branch and anything uncommitted or not on GitHub.

Edits to hispec or a driver under ``/opt/hispec/app`` follow
:doc:`../../install`: ``install.sh`` installs them as they are, and
``hispec hotfix`` sends a driver edit to the driver's own repository.

----

5. Troubleshooting
==================

.. list-table::
:header-rows: 1
:widths: 30 70

* - Symptom
- Resolution
* - ``libby show`` fails with ``Connection refused``
- The CLI defaults to a broker on ``localhost``. Point it at the shared one:
``mkdir -p ~/.libby && echo 'rabbitmq_url: amqp://hispec.caltech.edu' > ~/.libby/cli_config.yaml``
* - ``run_camerad_script.py`` cannot reach the Archon
- ``hsfei_atcam`` holds it: ``hispec stop hsfei_atcam`` first.
* - The camera module fails to build: ImageStreamIO not found
- Re-run ``install.sh``, which builds it when
``/usr/local/lib/cmake/ImageStreamIOConfig.cmake`` is missing.
14 changes: 8 additions & 6 deletions docs/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,14 +61,15 @@ activating anything.
```

`--host-role` is the name instance files use in `HISPEC_HOST`
(`hispecserver`, `feiserver`, ...). Leave it out to use the short hostname.
It is written once, to `/etc/hispec/host`; edit that file to change it.
(`hispecserver`, `feiserver`, ...). Leave it out and `install.sh` asks for
it. It is written once, to `/etc/hispec/host`; edit that file to change it.

Some extras come with the role. On `feiserver` the installer builds the
tracking camera module into the venv (the `atcam` extra), which any other
role is refused. On `feiserver` and `hispecserver` it installs atlas, the
shared-memory and FITS viewer (the `atlas` extra), which needs a graphical
session such as `ssh -Y`. Other extras are added with `--extra <name>` and
role is refused, along with ImageStreamIO and the `scripts` extra; see
[the tracking camera page](build_notes/FEI_build/tracking_camera.rst). On
`feiserver` and `hispecserver` it installs atlas, the shared-memory and FITS
viewer (the `atlas` extra), which needs a graphical session such as `ssh -Y`. Other extras are added with `--extra <name>` and
recorded in `/etc/hispec/extras`, so later runs keep them; `hispec doctor`
reports one that has gone missing.

Expand Down Expand Up @@ -153,7 +154,8 @@ Adding a daemon that the checkout already contains does not need it:

Extras install from GitHub, so a host has no source for them to edit. To change
one on the host that runs it, clone it into `/opt/hispec/extra/`; `install.sh`
then builds that extra from the clone instead:
then builds that extra from the clone instead. On `feiserver` it clones the
camera module and atlas there itself when they are missing:

| Clone | Replaces | After an edit |
|---|---|---|
Expand Down
Loading
Loading