diff --git a/rework_summaries/tb-02-atmo-dispersion-fixes_changes.md b/rework_summaries/tb-02-atmo-dispersion-fixes_changes.md new file mode 100644 index 00000000..1f10bd99 --- /dev/null +++ b/rework_summaries/tb-02-atmo-dispersion-fixes_changes.md @@ -0,0 +1,129 @@ +# tb-02-atmo-dispersion-fixes + +Bug fixes to `scopesim/effects/atmo_dispersion.py`. No new features, no flags, +no behaviour that was previously correct is changed. + +**Scope note.** `atmo_dispersion.py` stays a parallel alternative to +`scopesim/effects/shifts.py`, which is byte-identical to upstream. This branch +deliberately does **not** merge the two or offer one as a replacement for the +other. The whole file is new in this fork, so there is no upstream behaviour to +preserve here, only defects to remove. + +## 1. The interpolation method is now required (the substantive fix) + +`ShiftFoV3D.apply_to` chose between `"roll"` and `"map_coords"` from the median +shift magnitude of whichever FOV reached it first, then cached that choice on +`self.meta["method"]`: + +```python +if self.meta.get("method", None) is None: + if np.sqrt(np.median(dxs)**2 + np.median(dys)**2) >= 10: + self.meta["method"] = "roll" + else: + self.meta["method"] = "map_coords" +``` + +One effect instance is shared across every FOV of an `OpticalTrain`, so the +first FOV permanently decided the algorithm for all later ones, including +wavelength ranges where the shift is sub-pixel (`roll` quantises it to zero) or +very large (`map_coords` does an unnecessary spline). + +The same stickiness disagreed with `_padded_header`, which `observe()` calls +**before** the first `apply_to`. At that point `method` is still `None`, so +padding was sized under the `map_coords` assumption (`guard = 4`, no integer +rounding) regardless of what would later be applied. That particular +disagreement over-pads rather than under-pads, so it wastes memory rather than +corrupting data — but the two code paths were reasoning about different +effects. + +How the shift is applied is a physical choice about the data, so it is now +configured rather than inferred. `check_keys` raises if it is absent, and the +existing value check stays. + +**Config impact: none.** Both production configurations already pin it — +`irdb/MaunaKea/MaunaKea.yaml` sets `method: "roll"` and +`irdb/XSHOOTER/XSHOOTER_SPEC.yaml` sets `method: map_coords`. Neither ever +exercised the auto-selection branch. Only callers that omitted `method` +entirely are affected. + +This also removes a `getattr(self, self.meta["method"] + "_shift")` dispatch on +a string that was validated at `__init__` and then mutated afterwards by the +block above. It is now an explicit two-way choice. + +### A runtime break this surfaced + +`ADCShift.get_shifts` builds an inner `ADShift` purely to obtain the +*uncorrected* dispersion, which it then differences against a perturbed zenith +angle. The `ad_kwargs` dict it passes contains only `temperature`, `pressure`, +`humidity`, `x_co2` and `wave_ref_um` — no `method`. Making the key required +would therefore have raised on every ZShooter run through `MaunaKea.yaml`, and +**no existing test covered it**. + +The inner effect now inherits the outer one's method. It never applies a shift, +so the value is immaterial to it; inheriting keeps the setting in one +configured place. `test_adc_residual_shifts_inherit_the_configured_method` +covers the path, and was confirmed to fail without the fix. + +## 2. ASCII source + +`AGENTS.md` asks for ASCII-only source. `refractive_index` used literal Unicode +Greek letters as variable names and the unit comments carried middle dot, micro +and degree signs — 34 lines in total. All renamed to spelled-out ASCII +(`sigma`, `alpha`, `beta`, `gamma`, `rho_a`, `rho_w`, `rho_axs`, `rho_ws`), and +comments rewritten as `K Pa^-1`, `um^-2`, `deg C`. + +Renames only. Verified bit-identical output at Ciddor's reference condition: +`n = 1.0002765302104355` and `Z_a = 0.9995922115359966` before and after. + +## 3. Smaller defects + +- **Duplicate import.** `SkyCoord` was imported twice, on lines 5 and 9, with + unrelated imports in between. `Any` was imported from `typing` and never + used. Both removed. +- **The 0.1 pixel quantisation is now named.** `np.round(..., 1)` appears in + both `apply_to` and `_padded_header`, and exists so that many wavelength + planes share one cached sparse interpolation operator. It is also a hard + floor on the accuracy of the applied dispersion, which was not stated + anywhere. The value is unchanged; it is now `SHIFT_QUANTIZATION_DECIMALS` + with a comment saying what it costs, so it is greppable. +- **The cached shift operator is returned read-only.** `_spatial_shift_operator` + is an `lru_cache`'d function handing out a mutable `csr_array` by reference. + Its one caller only reads it, but nothing enforced that. Now mirrors the + precedent in `psfs/analytical.py`, where `_legendre_rule` already sets + `writeable = False` on its cached arrays. +- **`ADShift.apply_to` deleted.** It was a pure pass-through to `super()`. +- **`ADCShift.z_order` annotated as `ClassVar`,** consistent with `ShiftFoV3D` + and `ADShift` in the same file and with every class in `shifts.py`. + +## 4. Tests + +`refractive_index` and `compressibility` had **no direct test anywhere**. They +now check against published Ciddor 1996 values: + +- `n = 1.00027653` at 633 nm, 15 deg C, 101325 Pa, dry air, 450 ppm CO2 — + reproduced to 8 significant figures +- `Z_a = 0.9995922115` for dry air at 288.15 K, 101325 Pa — Ciddor's own + Appendix value, reproduced exactly +- normal dispersion across 0.35 / 0.633 / 1.5 um +- moist air less refractive than dry at the same temperature and pressure +- refractivity nearly proportional to pressure +- compressibility approaching the ideal-gas limit as pressure falls + +Plus coverage that `method` is required, that an unknown value is rejected, and +the `ADCShift` inheritance path above. + +## Review notes + +- The `np.round(..., 1)` floor is documented, not changed. Whether 0.1 pixel is + an acceptable systematic for the ADC residuals is a separate question. +- `is_night` still returns `bool | Time` and `resolve_time` disambiguates with + `isinstance`. Left alone; noted as a wart. +- `ADCShift._adc_nir_resid` / `_adc_ub_resid` / `_adc_gri_resid` contain + hard-coded band-remapping constants with no test. Out of scope here. + +## Verification + +`scopesim/tests`: 4 failed, 1389 passed — the same four pre-existing failures +as on `main` (two `test_basic_instrument` flux regressions, `test_import_all` +from an untracked local file, and the stale waveset-cache test removed on +`tb-05`). diff --git a/scopesim/effects/atmo_dispersion.py b/scopesim/effects/atmo_dispersion.py index 3fd85fd1..e8ed99a2 100644 --- a/scopesim/effects/atmo_dispersion.py +++ b/scopesim/effects/atmo_dispersion.py @@ -1,4 +1,4 @@ -from typing import ClassVar, Any +from typing import ClassVar from functools import partial, lru_cache import numpy as np from astropy import units as u @@ -6,7 +6,7 @@ from astropy.table import Table from astropy.wcs import WCS from astropy.time import Time -from astropy.coordinates import SkyCoord, HADec +from astropy.coordinates import HADec from scipy.interpolate import make_interp_spline, RegularGridInterpolator from scipy.ndimage import map_coordinates from scipy.sparse import csr_array @@ -18,6 +18,11 @@ logger = get_logger(__name__) +# Dispersion shifts are rounded to this many decimal places in pixels (0.1 pix) +# so that many wavelength planes share one cached interpolation operator. This +# is also a hard floor on the accuracy of the applied dispersion. +SHIFT_QUANTIZATION_DECIMALS = 1 + class ShiftFoV3D(Effect): """ Shift the FoV by dx(wavelength) and dy(wavelength), that are provided in a file, or as arrays. @@ -62,10 +67,13 @@ class ShiftFoV3D(Effect): def __init__(self, **kwargs): super().__init__(**kwargs) - self.meta.setdefault("method", None) + # How the shift is applied is a physical choice about the data, so it + # is configured, never inferred from whichever FOV happens to arrive + # first and then frozen on this shared effect instance. + check_keys(self.meta, {"method"}, action="error") - if self.meta["method"] not in [None, "roll", "map_coords"]: - raise ValueError("Method must be 'roll' or 'map_coords' if provided.") + if self.meta["method"] not in ["roll", "map_coords"]: + raise ValueError(f"Method must be 'roll' or 'map_coords', not {self.meta['method']!r}.") def get_shifts(self, obj): """ @@ -95,7 +103,7 @@ def _padded_header(self, header, shifts, rounded=False): for axis, displacement in enumerate(shifts, start=1): cells = (displacement/(abs(header[f'CDELT{axis}'])*u.deg)).decompose().value if rounded: - cells = np.round(cells, 1) + cells = np.round(cells, SHIFT_QUANTIZATION_DECIMALS) if self.meta['method'] == 'roll': cells = np.round(np.asarray(cells, dtype=np.float32)) padding = int(np.ceil(np.max(np.abs(cells)))) + guard if np.any(cells) else 0 @@ -108,15 +116,10 @@ def apply_to(self, obj, **kwargs): dxs, dys = kwargs['_shifts'] if '_shifts' in kwargs else self.get_shifts(obj) # get bin widths in spatial axis and divide to get dxs and dys in steps - dxs = np.round((dxs / (obj.hdu.header["CDELT1"]*u.deg)).decompose().value, 1) - dys = np.round((dys / (obj.hdu.header["CDELT2"]*u.deg)).decompose().value, 1) + dxs = np.round((dxs / (obj.hdu.header["CDELT1"]*u.deg)).decompose().value, SHIFT_QUANTIZATION_DECIMALS) + dys = np.round((dys / (obj.hdu.header["CDELT2"]*u.deg)).decompose().value, SHIFT_QUANTIZATION_DECIMALS) - if self.meta.get("method", None) is None: - if np.sqrt(np.median(dxs) ** 2 + np.median(dys) ** 2) >= 10: - self.meta["method"] = "roll" - else: - self.meta["method"] = "map_coords" - shift_func = getattr(self, self.meta["method"]+"_shift") + shift_func = self.roll_shift if self.meta["method"] == "roll" else self.map_coords_shift obj.hdu.data = shift_func(obj.data, dxs, dys) @@ -210,7 +213,9 @@ def _spatial_shift_operator(length, shift, order): row, column = np.indices((length, length), dtype=float) response = map_coordinates(np.eye(length), [row, column - shift], order=order, mode='nearest', prefilter=order > 1) response[abs(response) < 1e-13] = 0 - return csr_array(response.T) + operator = csr_array(response.T) + operator.data.flags.writeable = False + return operator class ADShift(ShiftFoV3D): @@ -255,9 +260,6 @@ def __init__(self, **kwargs): xc = from_currsys(self.meta.get("x_co2", 450.), self.cmds) self.n = partial(refractive_index, temp=temp, pressure=pressure, frac_humidity=humidity, xc=xc) - def apply_to(self, obj, **kwargs): - return super().apply_to(obj, **kwargs) - def get_shifts(self, obj: FieldOfView3D): shift_arcsec = self._get_shifts_arcsec(obj) @@ -314,7 +316,7 @@ class ADCShift(ShiftFoV3D): 3.0 0 0.1 """ - z_order = (652, ) + z_order: ClassVar[tuple[int, ...]] = (652, ) def __init__(self, **kwargs): super().__init__(**kwargs) @@ -367,6 +369,10 @@ def get_shifts(self, obj: FieldOfView3D): ad_kwargs[k] = v else: ad_kwargs[k] = self.meta[k] + # This inner effect only ever supplies shift amounts; it never + # applies them. It still inherits our method so that the required + # key is satisfied from one configured place. + ad_kwargs['method'] = self.meta['method'] ad = self._get_ad_shift(ad_kwargs) ad.zenith_angle = self.zenith_angle ad_shift = ad._get_shifts_arcsec(obj) # get shift at zenith angle @@ -462,15 +468,15 @@ def compressibility(temp: u.Quantity, pressure: u.Quantity, x_w: float): t = temp.to(u.deg_C, equivalencies=u.temperature()).value T = temp.to(u.K, equivalencies=u.temperature()).value p = pressure.to(u.Pa).value - a0 = 1.58123e-6 # K·Pa^-1 + a0 = 1.58123e-6 # K Pa^-1 a1 = -2.9331e-8 # Pa^-1 - a2 = 1.1043e-10 # K^-1·Pa^-1 - b0 = 5.707e-6 # K·Pa^-1 + a2 = 1.1043e-10 # K^-1 Pa^-1 + b0 = 5.707e-6 # K Pa^-1 b1 = -2.051e-8 # Pa^-1 - c0 = 1.9898e-4 # K·Pa^-1 + c0 = 1.9898e-4 # K Pa^-1 c1 = -2.376e-6 # Pa^-1 - d = 1.83e-11 # K^2·Pa^-2 - e = -0.765e-8 # K^2·Pa^-2 + d = 1.83e-11 # K^2 Pa^-2 + e = -0.765e-8 # K^2 Pa^-2 return 1 - (p / T) * (a0 + a1 * t + a2 * t ** 2 + (b0 + b1 * t) * x_w + (c0 + c1 * t) * x_w ** 2) + (p / T) ** 2 * ( d + e * x_w ** 2) @@ -479,39 +485,39 @@ def refractive_index(wavelength: u.Quantity, temp: u.Quantity, pressure: u.Quant """ See Ciddor 1996, https://doi.org/10.1364/AO.35.001566 :param wavelength: wavelength, 0.3 to 1.69 um - :param temp: temperature, -40 to +100 °C + :param temp: temperature, -40 to +100 deg C :param pressure: pressure, 80000 to 120000 Pa :param frac_humidity: fractional humidity, 0 to 1 :param xc: CO2 concentration in ppm, 0 to 2000 ppm :return: nprop: refractive index """ lam = wavelength.to(u.um).value - σ = 1 / lam # μm^-1 + sigma = 1 / lam # um^-1 - t = temp.to(u.deg_C, equivalencies=u.temperature()).value # t in °C - T = temp.to(u.K, equivalencies=u.temperature()).value # Temperature °C -> K + t = temp.to(u.deg_C, equivalencies=u.temperature()).value # t in deg C + T = temp.to(u.K, equivalencies=u.temperature()).value # Temperature deg C -> K p = pressure.to(u.Pa).value - R = 8.314510 # gas constant, J/(mol·K) + R = 8.314510 # gas constant, J/(mol K) - k0 = 238.0185 # μm^-2 - k1 = 5792105 # μm^-2 - k2 = 57.362 # μm^-2 - k3 = 167917 # μm^-2 + k0 = 238.0185 # um^-2 + k1 = 5792105 # um^-2 + k2 = 57.362 # um^-2 + k3 = 167917 # um^-2 - w0 = 295.235 # μm^-2 - w1 = 2.6422 # μm^-2 - w2 = -0.032380 # μm^-4 - w3 = 0.004028 # μm^-6 + w0 = 295.235 # um^-2 + w1 = 2.6422 # um^-2 + w2 = -0.032380 # um^-4 + w3 = 0.004028 # um^-6 A = 1.2378847e-5 # K^-2 B = -1.9121316e-2 # K^-1 C = 33.93711047 D = -6.3431645e3 # K - α = 1.00062 - β = 3.14e-8 # Pa^-1, - γ = 5.6e-7 # °C^-2 + alpha = 1.00062 + beta = 3.14e-8 # Pa^-1, + gamma = 5.6e-7 # deg C^-2 # saturation vapor pressure of water vapor in air at temperature T (Pa) svp = np.where(t >= 0, @@ -519,19 +525,19 @@ def refractive_index(wavelength: u.Quantity, temp: u.Quantity, pressure: u.Quant 10 ** (-2663.5 / T + 12.537)) # if t<0 # enhancement factor of water vapor in air - f = α + β * p + γ * t ** 2 + f = alpha + beta * p + gamma * t ** 2 # molar fraction of water vapor in moist air xw = f * frac_humidity * svp / p - # refractive index of standard air at 15 °C, 101325 Pa, 0% humidity, 450 ppm CO2 - nas = 1 + (k1 / (k0 - σ ** 2) + k3 / (k2 - σ ** 2)) * 1e-8 + # refractive index of standard air at 15 deg C, 101325 Pa, 0% humidity, 450 ppm CO2 + nas = 1 + (k1 / (k0 - sigma ** 2) + k3 / (k2 - sigma ** 2)) * 1e-8 - # refractive index of standard air at 15 °C, 101325 Pa, 0% humidity, xc ppm CO2 + # refractive index of standard air at 15 deg C, 101325 Pa, 0% humidity, xc ppm CO2 naxs = 1 + (nas - 1) * (1 + 0.534e-6 * (xc - 450)) - # refractive index of water vapor at standard conditions (20 °C, 1333 Pa) - nws = 1 + 1.022 * (w0 + w1 * σ ** 2 + w2 * σ ** 4 + w3 * σ ** 6) * 1e-8 + # refractive index of water vapor at standard conditions (20 deg C, 1333 Pa) + nws = 1 + 1.022 * (w0 + w1 * sigma ** 2 + w2 * sigma ** 4 + w3 * sigma ** 6) * 1e-8 Ma = 1e-3 * (28.9635 + 12.011e-6 * (xc - 400)) # molar mass of dry air, kg/mol Mw = 0.018015 # molar mass of water vapor, kg/mol @@ -540,16 +546,16 @@ def refractive_index(wavelength: u.Quantity, temp: u.Quantity, pressure: u.Quant Zw = compressibility(293.15*u.K, 1333*u.Pa, 1) # compressibility of pure water vapor # Eq.4 with (T,P,xw) = (288.15, 101325, 0) - ρaxs = 101325 * Ma / (Za * R * 288.15) # density of standard air + rho_axs = 101325 * Ma / (Za * R * 288.15) # density of standard air # Eq 4 with (T,P,xw) = (293.15, 1333, 1) - ρws = 1333 * Mw / (Zw * R * 293.15) # density of standard water vapor + rho_ws = 1333 * Mw / (Zw * R * 293.15) # density of standard water vapor - # two parts of Eq.4: ρ=ρa+ρw + # two parts of Eq.4: rho=rho_a+rho_w Z = compressibility(temp, pressure, xw) - ρa = p * Ma / (Z * R * T) * (1 - xw) # density of the dry component of the moist air - ρw = p * Mw / (Z * R * T) * xw # density of the water vapor component + rho_a = p * Ma / (Z * R * T) * (1 - xw) # density of the dry component of the moist air + rho_w = p * Mw / (Z * R * T) * xw # density of the water vapor component - nprop = 1 + (ρa / ρaxs) * (naxs - 1) + (ρw / ρws) * (nws - 1) + nprop = 1 + (rho_a / rho_axs) * (naxs - 1) + (rho_w / rho_ws) * (nws - 1) return nprop diff --git a/scopesim/tests/tests_effects/test_AtmosphericDispersion.py b/scopesim/tests/tests_effects/test_AtmosphericDispersion.py index a72cc531..68b3f2fa 100644 --- a/scopesim/tests/tests_effects/test_AtmosphericDispersion.py +++ b/scopesim/tests/tests_effects/test_AtmosphericDispersion.py @@ -3,8 +3,12 @@ import numpy as np -from scopesim.effects.atmo_dispersion import ADCShift +from astropy import units as u +from scopesim.effects.atmo_dispersion import (ADCShift, ShiftFoV3D, + refractive_index, compressibility) + +from scopesim.tests.tests_optics.test_slit_spatial_integration import physical_fov from scopesim.effects.shifts import AtmosphericDispersion, \ get_pixel_border_waves_from_atmo_disp @@ -131,3 +135,71 @@ def test_returns_sensible_data(self): waves, shifts = get_pixel_border_waves_from_atmo_disp(**atmo_params) assert shifts[0] - shifts[-1] == approx(0.49, rel=2e-2) assert waves[0] == 0.5 and waves[-1] == 1.5 + + + +class TestCiddorRefractiveIndex: + """Reference values from Ciddor 1996, https://doi.org/10.1364/AO.35.001566""" + + def test_standard_air_matches_published_refractive_index(self): + # Ciddor's reference condition: 633 nm, 15 deg C, 101325 Pa, dry air, + # 450 ppm CO2, for which the published n is 1.00027653. + n = refractive_index(0.633*u.um, 15*u.deg_C, 101325*u.Pa, 0.0, 450.) + assert n == approx(1.00027653, abs=5e-9) + + @pytest.mark.parametrize("wavelength, expected", [(0.35, 1.00028612), (0.633, 1.00027653), (1.5, 1.00027330)]) + def test_normal_dispersion_across_the_band(self, wavelength, expected): + # n falls monotonically with wavelength away from any resonance. + n = refractive_index(wavelength*u.um, 15*u.deg_C, 101325*u.Pa, 0.0, 450.) + assert n == approx(expected, abs=5e-9) + + def test_moist_air_is_less_refractive_than_dry_air(self): + dry = refractive_index(0.633*u.um, 20*u.deg_C, 101325*u.Pa, 0.0, 450.) + saturated = refractive_index(0.633*u.um, 20*u.deg_C, 101325*u.Pa, 1.0, 450.) + assert saturated < dry + assert (dry - saturated) == approx(8.503e-7, rel=1e-3) + + def test_refractivity_is_nearly_proportional_to_pressure(self): + full = refractive_index(0.633*u.um, 15*u.deg_C, 101325*u.Pa, 0.0, 450.) + half = refractive_index(0.633*u.um, 15*u.deg_C, 50000*u.Pa, 0.0, 450.) + assert (half - 1)/(full - 1) == approx(50000/101325, rel=1e-3) + + def test_dry_air_compressibility_matches_ciddor_appendix(self): + # Ciddor quotes Z_a = 0.9995922115 at 288.15 K and 101325 Pa. + assert compressibility(288.15*u.K, 101325*u.Pa, 0) == approx(0.9995922115, abs=1e-10) + + def test_compressibility_approaches_the_ideal_gas_limit(self): + assert compressibility(288.15*u.K, 1*u.Pa, 0) == approx(1.0, abs=1e-8) + + +class TestShiftMethodIsConfigured: + def test_missing_method_is_an_error(self): + # The interpolation method must not be guessed from the first FOV. + with pytest.raises(ValueError, match="method"): + ShiftFoV3D() + + def test_unknown_method_is_an_error(self): + with pytest.raises(ValueError, match="roll"): + ShiftFoV3D(method="nearest") + + @pytest.mark.parametrize("method", ["roll", "map_coords"]) + def test_configured_method_is_accepted(self, method): + assert ShiftFoV3D(method=method).meta["method"] == method + + def test_adc_residual_shifts_inherit_the_configured_method(self): + # ADCShift builds an inner ADShift to get the uncorrected dispersion; + # that inner effect must not need its own method configured. + from scopesim.commands import UserCommands + cmds = UserCommands(properties={ + "!ATMO.longitude": -70.19, "!ATMO.latitude": -24.59, "!ATMO.altitude": 3060, + "!ATMO.temperature": 0., "!ATMO.pressure": 0.6, "!ATMO.humidity": 0.1, + "!ATMO.x_co2": 450., "!OBS.mjdobs": 60310.25, "!OBS.alt": 60, "!OBS.az": 0, + }) + adc = ADCShift(method="roll", zenith_angle_error=0.1, wave_ref_um=0.5, cmds=cmds) + assert adc.meta["method"] == "roll" + + # Exercising get_shifts is the point: it is what builds the inner + # ADShift, which used to be constructed without a method. + dxs, dys = adc.get_shifts(physical_fov(.7)) + assert len(dxs) == len(dys) + assert np.any(np.abs(dys.to_value(u.arcsec)) > 0)