MetisWISE is the client software for the METIS AIT archive.
It contains a Python library, called metiswise, to connect to the METIS AIT database and to retrieve data from the METIS AIT data server.
MetisWISE requires Python 3.11 to 3.13, because the METIS pipeline (pymetis)
does not support other versions yet. Python 3.12 is used for development and CI.
Install uv, then clone
and install the project:
git clone https://github.com/AstarVienna/MetisWISE.git
cd MetisWISE
uv sync --python 3.12 --no-devuv creates a .venv and installs MetisWISE in editable mode. Package sources are
configured in pyproject.toml:
commonwise,dataserverwise,metis-drld, and the METIS pipelinepymetiscome from https://pip.entropynaut.com/packages/.pycplcomes from https://ivh.github.io/pycpl/simple/.edps,pyesorex, andadari-corecome from https://ftp.eso.org/pub/dfs/pipelines/libraries.- Other packages come from PyPI.
To include Jupyter for the example notebooks, add --group notebook to uv sync.
To install metis-simulations from the Entropynaut index, include the
simulations dependency group:
uv sync --group simulationsGroups can be combined, for example uv sync --group simulations --group postgres.
To install everything, including development tools, notebooks, simulations and
database test dependencies, use uv's built-in option:
uv sync --all-groupsUse uv run --all-groups ... when running commands with all groups enabled.
From the checkout, install the default dev dependency group (pytest, coverage
and build), optionally including notebooks:
uv sync --python 3.12 --group notebookuv sync resolves dependencies and creates or updates uv.lock. Commit the
lockfile after generating it so developers and CI can share the same resolution.
Use uv lock --upgrade to update dependency versions intentionally. Once a
lockfile is committed, uv sync --locked verifies that it matches the project.
Run tests from the repository root:
uv run python -m pytestDatabase tests are marked postgresql and skipped unless you pass
--run-postgresql. Install PostgreSQL with its contrib extensions (including
pgcrypto) and include the postgres dependency group:
uv sync --group postgres
uv run --group postgres python -m pytest --run-postgresql
uv run --group postgres python -m pytest --run-postgresql -m postgresql -v
uv run --group postgres python -m pytest --run-postgresql tests/test_raw.pyRun as a regular user, because PostgreSQL cannot initialize a cluster as root.
If PostgreSQL is outside your PATH, pass
--postgresql-exec=/path/to/pg_ctl.
The session fixture metiswise_database in the root conftest.py uses
pytest-postgresql's process fixture to start a temporary local PostgreSQL
cluster on an available port. It overrides personal database configuration and
runs metiswise.tools.dbtestsetup in the pytest process before the first database
test.
Setup creates the persistent classes as AWOPER, then the harness connects as
AWTEST with a generated password. The server, connections and temporary files
are cleaned up even if setup or collection fails. No existing database or data
server is needed. Collection, including --collect-only --run-postgresql, does
not start a database.
The database fixture also requests metiswise_dataserver, which starts the
installed dataserverwise server on a dynamically selected loopback HTTP port.
Its storage directories, SQLite file index, configuration, and logs all reside
under pytest's temporary dataserver0 directory. It uses no external peers or
direct filesystem access from the archive client. The server is stopped when
the session ends, including after test failures. Startup errors include the
path to process.log; the server's own logs are in the same directory.
The archive storage client is configured for this temporary endpoint before tests run. A FITS round-trip test uploads and commits an exposure, removes its original local file, reconnects to PostgreSQL, and retrieves the file into a new directory. It checks both the complete file hash and the image pixels:
uv run --no-sync python -m pytest --run-postgresql tests/test_dataserver.py -vKeep database tests in tests/, mark them with pytest.mark.postgresql, and
request metiswise_database as a fixture argument or with
pytest.mark.usefixtures("metiswise_database"). Import persistent classes inside
test functions or fixtures that depend on metiswise_database, so setup runs
before those imports. The initialized schema is shared across the session
because Python retains class registrations; tests should delete any objects they
commit. The initial test
commits a Raw() and reads it back through Raw.select_all() after clearing the
object cache.
The FITS provenance parsers in metiswise.main.provenance are independent of the
database and are tested in ordinary pytest runs.
To measure line and branch coverage using the coverage package included in the
test dependency group:
uv run --group postgres python -m coverage run --branch --source=metiswise -m pytest --run-postgresql
uv run --group postgres python -m coverage report --show-missing
uv run --group postgres python -m coverage htmlOpen htmlcov/index.html for a report with the uncovered lines highlighted.
Omit --run-postgresql to measure only tests that do not need a database.
The GitHub Actions workflow in .github/workflows/tests.yml runs on pushes to
main (including merges) and pull requests, and can also be started manually.
It uses Python 3.12 on Ubuntu, installs dependencies with
uv sync --group postgres, and runs pytest both without and with
--run-postgresql. CI uses the same package sources and pycpl override as local
development. JUnit XML reports are saved as
the test-reports workflow artifact, including when tests fail. A successful full
test run also saves coverage XML and HTML reports in that artifact. The XML can
be uploaded to a hosted coverage service such as Codecov or Coveralls.
The opt-in simulation test executes observing blocks from
tests/simulations/simulationBlocks, using the installed metis-simulations
Python library. The YAML inputs live in tests/simulations/YAML; no simulator
checkout or packaged simulation resources are needed. The runner passes
--noPsf and uses full-size detectors.
The repository observing YAML files use point sources for IFU and LSS science exposures.
By default, --simulation-profile archive prepares temporary copies of the
METIS optical YAML settings for fast archive tests:
- PSF convolution is disabled for every source, including stars and galaxies
that the simulator's guarded
--noPsfflag would normally leave unchanged. - IFU sky-pixel size is doubled, reducing the internal cube's spatial area by about four. The detector layout, detector sizes and spectral traces are kept.
The fine wavelength sampling is retained so narrow laser lines remain available to wavelength-calibration recipes. Detector noise, gain, linearity and FITS header effects remain enabled. These shortcuts deliberately sacrifice optical realism; they are intended to exercise reduction and archive infrastructure. They do not guarantee success of every scientific recipe.
The original IRDB is untouched. Inspect the generated instrument settings in
the temporary simulations0/archive-inputs/irdb directory, or pass
--simulation-profile default to use the original IRDB settings.
Standalone block invocations use the original settings too; the archive profile
is applied by the pytest fixture.
Two single-night sets are available:
| Block | Raw exposures | Purpose |
|---|---|---|
functionalPipeline (default) |
24 | The previously enabled LM-only simulations, for a functional setup test. |
minimalPipeline |
78 | All entries restored, including GEO, IFU, both LSS bands and chophome, for broad data-model coverage. |
Both sets have explicit darks, flats and paired linearity exposures, share
calibrations within the night, and generate static calibration prototypes once.
Automatic --doCalib generation is disabled to preserve their exposure budgets.
The blocks can also run directly:
python tests/simulations/simulationBlocks/functionalPipeline.py --noPsfUse minimalPipeline.py for the full set. Output is written below
MSIM_OUTDIR/<block> (output/<block> for standalone runs without the variable).
pytest defaults MSIM_OUTDIR to its temporary simulation output directory.
uv sync --group simulations --group postgres
uv run --no-sync python -m pytest --run-postgresql --run-simulations tests/test_simulations.py -vSelect the larger set explicitly:
uv run --no-sync python -m pytest --run-postgresql --run-simulations \
--simulation-block minimalPipeline \
tests/test_simulations.py -vNames are relative to tests/simulations/simulationBlocks, with or without
.py. --simulation-block can be repeated for additional custom blocks. Run
the functional and full sets separately because they overlap and contain static
files with the same names. Each block must create or update at least one FITS
file. The test discovers
those files below the output directory, ignoring untouched files from earlier
runs, so adding another block does not require hardcoded YAML lists or output
paths. It ingests raw and processed data, uploads the FITS files to the temporary
dataserver, and downloads each into a separate directory to compare SHA-256
hashes. Invalid model mappings, unsupported FITS files, and failed transfers
fail the test. ingest_file(..., store=False) is available for callers
that only want to commit metadata without uploading files.
MSIM_NCORES chooses the worker count and is passed to --nCores; it defaults to
one. MSIM_YAML_DIR, MSIM_OUTDIR, and MPLBACKEND override the input directory,
output directory, and plotting backend. Relative input/output paths are resolved
from the directory where pytest is invoked. By default, each block uses the
local tests/simulations/YAML directory, temporary output, and
the Agg backend. Overrides are passed through the subprocess environment and
do not change pytest's environment or working directory.
OMP_NUM_THREADS, OPENBLAS_NUM_THREADS and MKL_NUM_THREADS default to one in
simulation subprocesses unless already set. Full-size IFU frames still need
substantial memory per worker, so increase MSIM_NCORES with care.
No simulator checkout is needed. The test uses a sibling irdb checkout if
present. Use --irdb-root=/path/to/inst_pkgs (or DEFAULT_IRDB_LOCATION) to select
instrument packages containing METIS, ELT, and Armazones. Otherwise ScopeSim
downloads them into the temporary directory through its Python API. Even with a
local IRDB, additional spectra, atmospheric data, or PSFs may require network
access.
Output FITS files remain in the chosen output directory for inspection. pytest
captures the scripts' output and reports a failure if any script exits with an
error; use -s to see progress live. Ordinary runs, including the standard CI job,
skip the simulation ingestion test unless both --run-simulations and
--run-postgresql are supplied. The small script-runner tests run without IRDB,
ScopeSim, or a database.
tests/data/minimalPipeline.json is a compact snapshot of 209 simulated and
processed FITS files. It was converted from a completed full run: 6.40 GB of
FITS files becomes a 4.77 MB plain JSON fixture (about 189 KB with zlib
compression, before Git's delta compression). It contains the original
filenames and HDU headers; image pixels are represented as zeros and tables
have zero rows while retaining their column definitions.
No simulation or recipe execution is required to use it.
The JSON is pretty-printed, with each header stored as a list of 80-character
physical FITS cards, including long-value CONTINUE cards and the END card.
Inspect it directly with less tests/data/minimalPipeline.json. Git compresses
stored objects, so the repository keeps readable JSON rather than a gzip file.
The converter still supports .json.gz outputs when explicitly requested.
Run its metadata-ingestion test against the temporary PostgreSQL database:
uv sync --group postgres
uv run --no-sync python -m pytest --run-postgresql \
tests/test_ingestion_dataset.py --ingestion-dataset--ingestion-dataset=/path/to/another.json selects a different snapshot.
This test uses ingest_file(..., store=False), so it tests the data model and
provenance without uploading gigabytes of zero pixels. The separate dataserver
test covers storage/retrieval with real nonzero FITS data.
Regenerate the fixture after another complete simulation/EDPS run:
uv run --no-sync python -m metiswise.tools.ingestion_dataset create \
/path/to/simulations0 tests/data/minimalPipeline.json --overwriteThe converter reads archive-classified, uncompressed FITS files recursively,
excluding linked IRDB assets. It records the source creation order (file mtimes)
so parent files can be ingested before their products. It never changes the
source files. Existing fixtures are protected unless --overwrite is specified;
replacements are atomic. Repeated conversions of unchanged inputs are identical.
To inspect or ingest the reconstructed files independently:
uv run --no-sync python -m metiswise.tools.ingestion_dataset restore \
tests/data/minimalPipeline.json /tmp/metis-ingestion-fitsRestoration preserves HDU structure, image dimensions, pixel types, header-card
order, duplicate cards and table column definitions. Table NAXIS2 and PCOUNT
are set to zero; THEAP, when present, is also reset to zero. No table payloads
or base64 blobs are stored. The restored files retain their large
logical sizes, but sparse zero-image regions occupy no blocks on supporting
filesystems. Nonzero FITS
scaling offsets are handled so restored pixels still have physical value zero.
The restore command refuses to overwrite files.
CHECKSUM/DATASUM cards are removed from changed image and table HDUs. Other metadata,
including ESO DATAMD5 and recipe provenance hashes, remains the original run's
snapshot: those hashes describe the original data. These fixtures are for
ingestion, not scientific reduction or validation of the original data checksums.
Add --run-edps to process the simulated files through metis.metis_wkf before
ingestion:
uv run --no-sync python -m pytest --run-postgresql --run-simulations --run-edps \
tests/test_simulations.py -vThe selected set's explicit calibrations and static prototypes are passed to
EDPS and ingested with the science data and processing products. The default
set covers LM imaging; use --simulation-block minimalPipeline for the full set.
The workflow and recipes come from the installed pymetis package. Supply any
additional calibration data required by the workflow with --edps-input:
uv run --no-sync python -m pytest --run-postgresql --run-simulations --run-edps \
--edps-input=/path/to/calibrations \
--edps-target metis_lm_img_basic_reduce_sci tests/test_simulations.py -vRepeat --edps-input to provide additional calibration files or directories, or
generate them using additional local --simulation-block options.
Repeat --edps-target to select tasks; omit it to use the workflow's
default targets. --edps-timeout sets the budget for the entire EDPS processing
phase (default 3600 seconds), separately from simulation time. On timeout the harness
saves job snapshots, cancels queued work and lets an active recipe finish before
reporting the timeout. This graceful shutdown avoids EDPS's child-process scan,
which can raise psutil.ZombieProcess and hide the original timeout. The budget
is therefore a soft limit; shutdown can take longer while the active recipe
finishes. Increase it for slower machines or larger runs, for example
--edps-timeout 7200.
The test constructs EDPS through its Python API with an in-memory bookkeeping
repository. It starts no HTTP server and reads no personal EDPS configuration.
Recipe execution uses the current environment's pyesorex, with a temporary
PYESOREX_CONFIG pointing to the installed METIS recipe directory. Products get
unique PYTEST filenames to avoid collisions in the archive. QC plot generation
is disabled for this test.
The harness uses the EDPS library rather than its CLI, so it supplies its own
live job summary. Counts and running task names appear on status changes and
at least every ten seconds, even with normal pytest output capture; -s is not
required. For example:
EDPS [02:10] total=57 queued=41 running=1 completed=15 failed=0 missing=0; running: metis_ifu_wavecal
The initial message gives the recipe log directory. Individual recipe stdout
and stderr stay in per-job esorex.stdout and esorex.stderr files, rather than
being streamed to the terminal. Other test output keeps its normal pytest
capture behavior.
Incomplete associations, failed recipes, zero jobs, and zero FITS products fail
the test. The temporary edps directory contains application.properties,
pyesorex.rc, submission.json, job JSON details, and recipe stdout/stderr for
diagnosis. The archive ingests simulated inputs first, followed by the workflow's
required calibration inputs and products in dependency order.
A full development environment for MetisWISE can be setup through podman containers. This will create a local PostgreSQL database to play with. See the [METIS_Environmens repository(https://github.com/AstarVienna/METIS_Environments)
Here are several examples of using the METIS Archive from Python
# First import all DataItem classes.
from metiswise.main.aweimports import *
# Create a Python instance of a simulated raw exposure.
my_raw_flat = LM_FLAT_LAMP_RAW(filename="my_simulated_raw.fits")
# Store the FITS file on the data server.
my_raw_flat.store()
# Commit the metadata to the database.
my_raw_flat.commit()# First import all DataItem classes.
from metiswise.main.aweimports import *
# Search for a raw flat.
all_my_raw_flats = LM_FLAT_LAMP_RAW.filename == "my_simulated_raw.fits"
# Check how many there are.
print(len(all_my_raw_flats))
# Get the first one.
my_raw_flat = all_my_raw_flats[0]
# Retrieve the FITS file from the data server
my_raw_flat.retrieve()# First import all DataItem classes.
from metiswise.main.aweimports import *
# Select all raw flats
all_my_raw_flats = LM_FLAT_LAMP_RAW.select_all()
print(len(all_my_raw_flats))
# Query flats on DIT and NDIT
my_interesting_raws = LM_FLAT_LAMP_RAW.det_ndit == 1 && LM_FLAT_LAMP_RAW.det_dit > 10
print(len(all_my_raw_flats))Updating the full METIS AIT Archive goes as follows.
- Ensure the passwords on the wiki are still correct; update if needed.
- Update and publish
commonwise.. - Update and publish
dataserverwise.. - Update and publish MetisWISE; see below.
- Update the METIS_Environments.
- Upgrade the production database. TODO.
- Upgrade the database viewer and data server software. TODO.
- Update the METIS Data Reduction Server Setup.
-
Set the version in
pyproject.tomlto the desired version. -
Build
uv build -
Upload
dist/metiswise-*.tar.gzto https://pip.entropynaut.com/packages/metiswise/