Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 129 additions & 0 deletions rework_summaries/tb-02-atmo-dispersion-fixes_changes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
# tb-02-atmo-dispersion-fixes

Bug fixes to `scopesim/effects/atmo_dispersion.py`. No new features, no flags,
no behaviour that was previously correct is changed.

**Scope note.** `atmo_dispersion.py` stays a parallel alternative to
`scopesim/effects/shifts.py`, which is byte-identical to upstream. This branch
deliberately does **not** merge the two or offer one as a replacement for the
other. The whole file is new in this fork, so there is no upstream behaviour to
preserve here, only defects to remove.

## 1. The interpolation method is now required (the substantive fix)

`ShiftFoV3D.apply_to` chose between `"roll"` and `"map_coords"` from the median
shift magnitude of whichever FOV reached it first, then cached that choice on
`self.meta["method"]`:

```python
if self.meta.get("method", None) is None:
if np.sqrt(np.median(dxs)**2 + np.median(dys)**2) >= 10:
self.meta["method"] = "roll"
else:
self.meta["method"] = "map_coords"
```

One effect instance is shared across every FOV of an `OpticalTrain`, so the
first FOV permanently decided the algorithm for all later ones, including
wavelength ranges where the shift is sub-pixel (`roll` quantises it to zero) or
very large (`map_coords` does an unnecessary spline).

The same stickiness disagreed with `_padded_header`, which `observe()` calls
**before** the first `apply_to`. At that point `method` is still `None`, so
padding was sized under the `map_coords` assumption (`guard = 4`, no integer
rounding) regardless of what would later be applied. That particular
disagreement over-pads rather than under-pads, so it wastes memory rather than
corrupting data — but the two code paths were reasoning about different
effects.

How the shift is applied is a physical choice about the data, so it is now
configured rather than inferred. `check_keys` raises if it is absent, and the
existing value check stays.

**Config impact: none.** Both production configurations already pin it —
`irdb/MaunaKea/MaunaKea.yaml` sets `method: "roll"` and
`irdb/XSHOOTER/XSHOOTER_SPEC.yaml` sets `method: map_coords`. Neither ever
exercised the auto-selection branch. Only callers that omitted `method`
entirely are affected.

This also removes a `getattr(self, self.meta["method"] + "_shift")` dispatch on
a string that was validated at `__init__` and then mutated afterwards by the
block above. It is now an explicit two-way choice.

### A runtime break this surfaced

`ADCShift.get_shifts` builds an inner `ADShift` purely to obtain the
*uncorrected* dispersion, which it then differences against a perturbed zenith
angle. The `ad_kwargs` dict it passes contains only `temperature`, `pressure`,
`humidity`, `x_co2` and `wave_ref_um` — no `method`. Making the key required
would therefore have raised on every ZShooter run through `MaunaKea.yaml`, and
**no existing test covered it**.

The inner effect now inherits the outer one's method. It never applies a shift,
so the value is immaterial to it; inheriting keeps the setting in one
configured place. `test_adc_residual_shifts_inherit_the_configured_method`
covers the path, and was confirmed to fail without the fix.

## 2. ASCII source

`AGENTS.md` asks for ASCII-only source. `refractive_index` used literal Unicode
Greek letters as variable names and the unit comments carried middle dot, micro
and degree signs — 34 lines in total. All renamed to spelled-out ASCII
(`sigma`, `alpha`, `beta`, `gamma`, `rho_a`, `rho_w`, `rho_axs`, `rho_ws`), and
comments rewritten as `K Pa^-1`, `um^-2`, `deg C`.

Renames only. Verified bit-identical output at Ciddor's reference condition:
`n = 1.0002765302104355` and `Z_a = 0.9995922115359966` before and after.

## 3. Smaller defects

- **Duplicate import.** `SkyCoord` was imported twice, on lines 5 and 9, with
unrelated imports in between. `Any` was imported from `typing` and never
used. Both removed.
- **The 0.1 pixel quantisation is now named.** `np.round(..., 1)` appears in
both `apply_to` and `_padded_header`, and exists so that many wavelength
planes share one cached sparse interpolation operator. It is also a hard
floor on the accuracy of the applied dispersion, which was not stated
anywhere. The value is unchanged; it is now `SHIFT_QUANTIZATION_DECIMALS`
with a comment saying what it costs, so it is greppable.
- **The cached shift operator is returned read-only.** `_spatial_shift_operator`
is an `lru_cache`'d function handing out a mutable `csr_array` by reference.
Its one caller only reads it, but nothing enforced that. Now mirrors the
precedent in `psfs/analytical.py`, where `_legendre_rule` already sets
`writeable = False` on its cached arrays.
- **`ADShift.apply_to` deleted.** It was a pure pass-through to `super()`.
- **`ADCShift.z_order` annotated as `ClassVar`,** consistent with `ShiftFoV3D`
and `ADShift` in the same file and with every class in `shifts.py`.

## 4. Tests

`refractive_index` and `compressibility` had **no direct test anywhere**. They
now check against published Ciddor 1996 values:

- `n = 1.00027653` at 633 nm, 15 deg C, 101325 Pa, dry air, 450 ppm CO2 —
reproduced to 8 significant figures
- `Z_a = 0.9995922115` for dry air at 288.15 K, 101325 Pa — Ciddor's own
Appendix value, reproduced exactly
- normal dispersion across 0.35 / 0.633 / 1.5 um
- moist air less refractive than dry at the same temperature and pressure
- refractivity nearly proportional to pressure
- compressibility approaching the ideal-gas limit as pressure falls

Plus coverage that `method` is required, that an unknown value is rejected, and
the `ADCShift` inheritance path above.

## Review notes

- The `np.round(..., 1)` floor is documented, not changed. Whether 0.1 pixel is
an acceptable systematic for the ADC residuals is a separate question.
- `is_night` still returns `bool | Time` and `resolve_time` disambiguates with
`isinstance`. Left alone; noted as a wart.
- `ADCShift._adc_nir_resid` / `_adc_ub_resid` / `_adc_gri_resid` contain
hard-coded band-remapping constants with no test. Out of scope here.

## Verification

`scopesim/tests`: 4 failed, 1389 passed — the same four pre-existing failures
as on `main` (two `test_basic_instrument` flux regressions, `test_import_all`
from an untracked local file, and the stale waveset-cache test removed on
`tb-05`).
112 changes: 59 additions & 53 deletions scopesim/effects/atmo_dispersion.py
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
from typing import ClassVar, Any
from typing import ClassVar
from functools import partial, lru_cache
import numpy as np
from astropy import units as u
from astropy.coordinates import SkyCoord, EarthLocation
from astropy.table import Table
from astropy.wcs import WCS
from astropy.time import Time
from astropy.coordinates import SkyCoord, HADec
from astropy.coordinates import HADec
from scipy.interpolate import make_interp_spline, RegularGridInterpolator
from scipy.ndimage import map_coordinates
from scipy.sparse import csr_array
Expand All @@ -18,6 +18,11 @@

logger = get_logger(__name__)

# Dispersion shifts are rounded to this many decimal places in pixels (0.1 pix)
# so that many wavelength planes share one cached interpolation operator. This
# is also a hard floor on the accuracy of the applied dispersion.
SHIFT_QUANTIZATION_DECIMALS = 1

class ShiftFoV3D(Effect):
"""
Shift the FoV by dx(wavelength) and dy(wavelength), that are provided in a file, or as arrays.
Expand Down Expand Up @@ -62,10 +67,13 @@ class ShiftFoV3D(Effect):

def __init__(self, **kwargs):
super().__init__(**kwargs)
self.meta.setdefault("method", None)
# How the shift is applied is a physical choice about the data, so it
# is configured, never inferred from whichever FOV happens to arrive
# first and then frozen on this shared effect instance.
check_keys(self.meta, {"method"}, action="error")

if self.meta["method"] not in [None, "roll", "map_coords"]:
raise ValueError("Method must be 'roll' or 'map_coords' if provided.")
if self.meta["method"] not in ["roll", "map_coords"]:
raise ValueError(f"Method must be 'roll' or 'map_coords', not {self.meta['method']!r}.")

def get_shifts(self, obj):
"""
Expand Down Expand Up @@ -95,7 +103,7 @@ def _padded_header(self, header, shifts, rounded=False):
for axis, displacement in enumerate(shifts, start=1):
cells = (displacement/(abs(header[f'CDELT{axis}'])*u.deg)).decompose().value
if rounded:
cells = np.round(cells, 1)
cells = np.round(cells, SHIFT_QUANTIZATION_DECIMALS)
if self.meta['method'] == 'roll':
cells = np.round(np.asarray(cells, dtype=np.float32))
padding = int(np.ceil(np.max(np.abs(cells)))) + guard if np.any(cells) else 0
Expand All @@ -108,15 +116,10 @@ def apply_to(self, obj, **kwargs):
dxs, dys = kwargs['_shifts'] if '_shifts' in kwargs else self.get_shifts(obj)

# get bin widths in spatial axis and divide to get dxs and dys in steps
dxs = np.round((dxs / (obj.hdu.header["CDELT1"]*u.deg)).decompose().value, 1)
dys = np.round((dys / (obj.hdu.header["CDELT2"]*u.deg)).decompose().value, 1)
dxs = np.round((dxs / (obj.hdu.header["CDELT1"]*u.deg)).decompose().value, SHIFT_QUANTIZATION_DECIMALS)
dys = np.round((dys / (obj.hdu.header["CDELT2"]*u.deg)).decompose().value, SHIFT_QUANTIZATION_DECIMALS)

if self.meta.get("method", None) is None:
if np.sqrt(np.median(dxs) ** 2 + np.median(dys) ** 2) >= 10:
self.meta["method"] = "roll"
else:
self.meta["method"] = "map_coords"
shift_func = getattr(self, self.meta["method"]+"_shift")
shift_func = self.roll_shift if self.meta["method"] == "roll" else self.map_coords_shift

obj.hdu.data = shift_func(obj.data, dxs, dys)

Expand Down Expand Up @@ -210,7 +213,9 @@ def _spatial_shift_operator(length, shift, order):
row, column = np.indices((length, length), dtype=float)
response = map_coordinates(np.eye(length), [row, column - shift], order=order, mode='nearest', prefilter=order > 1)
response[abs(response) < 1e-13] = 0
return csr_array(response.T)
operator = csr_array(response.T)
operator.data.flags.writeable = False
return operator


class ADShift(ShiftFoV3D):
Expand Down Expand Up @@ -255,9 +260,6 @@ def __init__(self, **kwargs):
xc = from_currsys(self.meta.get("x_co2", 450.), self.cmds)
self.n = partial(refractive_index, temp=temp, pressure=pressure, frac_humidity=humidity, xc=xc)

def apply_to(self, obj, **kwargs):
return super().apply_to(obj, **kwargs)

def get_shifts(self, obj: FieldOfView3D):
shift_arcsec = self._get_shifts_arcsec(obj)

Expand Down Expand Up @@ -314,7 +316,7 @@ class ADCShift(ShiftFoV3D):
3.0 0 0.1

"""
z_order = (652, )
z_order: ClassVar[tuple[int, ...]] = (652, )

def __init__(self, **kwargs):
super().__init__(**kwargs)
Expand Down Expand Up @@ -367,6 +369,10 @@ def get_shifts(self, obj: FieldOfView3D):
ad_kwargs[k] = v
else:
ad_kwargs[k] = self.meta[k]
# This inner effect only ever supplies shift amounts; it never
# applies them. It still inherits our method so that the required
# key is satisfied from one configured place.
ad_kwargs['method'] = self.meta['method']
ad = self._get_ad_shift(ad_kwargs)
ad.zenith_angle = self.zenith_angle
ad_shift = ad._get_shifts_arcsec(obj) # get shift at zenith angle
Expand Down Expand Up @@ -462,15 +468,15 @@ def compressibility(temp: u.Quantity, pressure: u.Quantity, x_w: float):
t = temp.to(u.deg_C, equivalencies=u.temperature()).value
T = temp.to(u.K, equivalencies=u.temperature()).value
p = pressure.to(u.Pa).value
a0 = 1.58123e-6 # K·Pa^-1
a0 = 1.58123e-6 # K Pa^-1
a1 = -2.9331e-8 # Pa^-1
a2 = 1.1043e-10 # K^-1·Pa^-1
b0 = 5.707e-6 # K·Pa^-1
a2 = 1.1043e-10 # K^-1 Pa^-1
b0 = 5.707e-6 # K Pa^-1
b1 = -2.051e-8 # Pa^-1
c0 = 1.9898e-4 # K·Pa^-1
c0 = 1.9898e-4 # K Pa^-1
c1 = -2.376e-6 # Pa^-1
d = 1.83e-11 # K^2·Pa^-2
e = -0.765e-8 # K^2·Pa^-2
d = 1.83e-11 # K^2 Pa^-2
e = -0.765e-8 # K^2 Pa^-2
return 1 - (p / T) * (a0 + a1 * t + a2 * t ** 2 + (b0 + b1 * t) * x_w + (c0 + c1 * t) * x_w ** 2) + (p / T) ** 2 * (
d + e * x_w ** 2)

Expand All @@ -479,59 +485,59 @@ def refractive_index(wavelength: u.Quantity, temp: u.Quantity, pressure: u.Quant
"""
See Ciddor 1996, https://doi.org/10.1364/AO.35.001566
:param wavelength: wavelength, 0.3 to 1.69 um
:param temp: temperature, -40 to +100 °C
:param temp: temperature, -40 to +100 deg C
:param pressure: pressure, 80000 to 120000 Pa
:param frac_humidity: fractional humidity, 0 to 1
:param xc: CO2 concentration in ppm, 0 to 2000 ppm
:return: nprop: refractive index
"""
lam = wavelength.to(u.um).value
σ = 1 / lam # μm^-1
sigma = 1 / lam # um^-1

t = temp.to(u.deg_C, equivalencies=u.temperature()).value # t in °C
T = temp.to(u.K, equivalencies=u.temperature()).value # Temperature °C -> K
t = temp.to(u.deg_C, equivalencies=u.temperature()).value # t in deg C
T = temp.to(u.K, equivalencies=u.temperature()).value # Temperature deg C -> K
p = pressure.to(u.Pa).value

R = 8.314510 # gas constant, J/(mol·K)
R = 8.314510 # gas constant, J/(mol K)

k0 = 238.0185 # μm^-2
k1 = 5792105 # μm^-2
k2 = 57.362 # μm^-2
k3 = 167917 # μm^-2
k0 = 238.0185 # um^-2
k1 = 5792105 # um^-2
k2 = 57.362 # um^-2
k3 = 167917 # um^-2

w0 = 295.235 # μm^-2
w1 = 2.6422 # μm^-2
w2 = -0.032380 # μm^-4
w3 = 0.004028 # μm^-6
w0 = 295.235 # um^-2
w1 = 2.6422 # um^-2
w2 = -0.032380 # um^-4
w3 = 0.004028 # um^-6

A = 1.2378847e-5 # K^-2
B = -1.9121316e-2 # K^-1
C = 33.93711047
D = -6.3431645e3 # K

α = 1.00062
β = 3.14e-8 # Pa^-1,
γ = 5.6e-7 # °C^-2
alpha = 1.00062
beta = 3.14e-8 # Pa^-1,
gamma = 5.6e-7 # deg C^-2

# saturation vapor pressure of water vapor in air at temperature T (Pa)
svp = np.where(t >= 0,
np.exp(A * T ** 2 + B * T + C + D / T), # if t>=0
10 ** (-2663.5 / T + 12.537)) # if t<0

# enhancement factor of water vapor in air
f = α + β * p + γ * t ** 2
f = alpha + beta * p + gamma * t ** 2

# molar fraction of water vapor in moist air
xw = f * frac_humidity * svp / p

# refractive index of standard air at 15 °C, 101325 Pa, 0% humidity, 450 ppm CO2
nas = 1 + (k1 / (k0 - σ ** 2) + k3 / (k2 - σ ** 2)) * 1e-8
# refractive index of standard air at 15 deg C, 101325 Pa, 0% humidity, 450 ppm CO2
nas = 1 + (k1 / (k0 - sigma ** 2) + k3 / (k2 - sigma ** 2)) * 1e-8

# refractive index of standard air at 15 °C, 101325 Pa, 0% humidity, xc ppm CO2
# refractive index of standard air at 15 deg C, 101325 Pa, 0% humidity, xc ppm CO2
naxs = 1 + (nas - 1) * (1 + 0.534e-6 * (xc - 450))

# refractive index of water vapor at standard conditions (20 °C, 1333 Pa)
nws = 1 + 1.022 * (w0 + w1 * σ ** 2 + w2 * σ ** 4 + w3 * σ ** 6) * 1e-8
# refractive index of water vapor at standard conditions (20 deg C, 1333 Pa)
nws = 1 + 1.022 * (w0 + w1 * sigma ** 2 + w2 * sigma ** 4 + w3 * sigma ** 6) * 1e-8

Ma = 1e-3 * (28.9635 + 12.011e-6 * (xc - 400)) # molar mass of dry air, kg/mol
Mw = 0.018015 # molar mass of water vapor, kg/mol
Expand All @@ -540,16 +546,16 @@ def refractive_index(wavelength: u.Quantity, temp: u.Quantity, pressure: u.Quant
Zw = compressibility(293.15*u.K, 1333*u.Pa, 1) # compressibility of pure water vapor

# Eq.4 with (T,P,xw) = (288.15, 101325, 0)
ρaxs = 101325 * Ma / (Za * R * 288.15) # density of standard air
rho_axs = 101325 * Ma / (Za * R * 288.15) # density of standard air

# Eq 4 with (T,P,xw) = (293.15, 1333, 1)
ρws = 1333 * Mw / (Zw * R * 293.15) # density of standard water vapor
rho_ws = 1333 * Mw / (Zw * R * 293.15) # density of standard water vapor

# two parts of Eq.4: ρ=ρa+ρw
# two parts of Eq.4: rho=rho_a+rho_w
Z = compressibility(temp, pressure, xw)
ρa = p * Ma / (Z * R * T) * (1 - xw) # density of the dry component of the moist air
ρw = p * Mw / (Z * R * T) * xw # density of the water vapor component
rho_a = p * Ma / (Z * R * T) * (1 - xw) # density of the dry component of the moist air
rho_w = p * Mw / (Z * R * T) * xw # density of the water vapor component

nprop = 1 + (ρa / ρaxs) * (naxs - 1) + (ρw / ρws) * (nws - 1)
nprop = 1 + (rho_a / rho_axs) * (naxs - 1) + (rho_w / rho_ws) * (nws - 1)

return nprop
Loading
Loading