diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index a21282e..626ef63 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -2040,6 +2040,73 @@ were. range that runs from the roof to the sky whatever is allowed outside it, and the samples beside it lie between their own history and the stale pixel. Holding still, nothing holds a history pixel against the samples. +**Lamps finer than a pixel: the thin feature's lock (2026-11-13, after it).** Dragging the view over Tokyo at night, +the tower broke up: its legs and most of its lamps gone while the camera turned, back half a second after it +stopped. Measured against offline renders, two faults of the upscaler, both older than the change above: + +- Held still, a field of lamps a third of a pixel wide kept a third of its light (0.035 of 0.103). `judge` takes a + surface for gone from in front when what was nearest among the nine samples a frame ago is nearer than what is + nearest now and the history's colour is not in this frame; a lamp the jitter hit last frame and missed this one + is exactly that, and its history was dropped every frame it was missed. +- In motion the history is clamped into the frame's nine samples, and a lamp none of them hit is outside them: the + tower in a turn lost a quarter of its light (the 4-sample cap in motion is not it: without it nothing changed). + +The remedy is the one FSR 2 (pixel locks) and UE's TSR (thin geometry detection) have: where the picture is a thin +feature, the rules that hold the history to this frame's samples stand back. `Lock` in `live.wgsl`, a half word per +traced pixel that the fast mean's buffer had free: + +- made by `compose` where the sample's emission is thin: no 2 × 2 block of the nine like it (FSR 2's rule), and + above or below both of some pair of opposite neighbours; a jitter sequence long, carried with the surface; none on + animated shading; +- in motion a locked pixel's history is clamped into the frame's nine samples only in part (`LOCK_RELAX` 0.75 of + it spared a frame: the lock of any of its nine samples), a locked sample's distance from the history's range is no + finding, and nothing is gone where one of the nine is locked. + +| | main | now | offline | +|---|---|---|---| +| lamps field held still, mean radiance | 0.035 | 0.106 | 0.103 | +| the same, the camera sliding 1.4 px a frame (see below: this does not exercise the motion clamp) | 0.027 | 0.112 | 0.103 | +| Tokyo Tower in a turn (0.25° a frame), ΔE / SSIM / luminance against offline | 2.81 / 0.867 / 0.77 | 2.17 / 0.919 / 0.80 | | +| Tokyo Tower in a quick orbit of the play window (1.15° a frame about a point 50 m ahead) | 5.06 / 0.808 / 0.71 | 4.15 / 0.872 / 0.86 | | +| the tower settled after the turn | 1.48 / 0.962 / 0.94 | 1.39 / 0.965 / 0.99 | | + +- The share spared in motion. All of it (the first form) kept the tower whole in the slow turn (2.23 / 0.928 / 1.05) + but, in the play window's quick orbit, drew its lamps out sideways and doubled its beacon (4.86, luminance 1.20): + a lamp the samples miss reprojects with what is behind it, and a spared history was a second copy. 0.5: 4.47 / + 0.69; none above 16 px a frame: main again (5.11 / 0.56). TSR likewise caps how far it loosens its clamp. +- What it costs, main → now. Where a view translates past emitters the locked ones trail a little: Kabukichō's + signs strafe 6.42 → 6.55, a tenth of a second after it 5.27 → 5.39 (trail 26 → 29 px), at the dolly's stop + 6.75 → 6.96 (its ground the same to 0.01); Bistro's whole frame a tenth of a second after a strafe 2.49 → 2.54 + (trail 13 → 18 px), else within 0.03; the Bistro slice half a second after its explosion 5.28 → 5.40; the orbit + of Thames moving 2.24 → 2.16 (trail 33 → 31 px), held 1.34 → 1.39. bench-ghosting passes on both scenes. Held still: Bistro 1.85 = 1.85 (frame to frame 0.46 → 0.44), Kabukichō 4.85 → 4.86. Tokyo at dusk the + same in the mean, the skyline's p99 at 6 s 11–14 → 16–18 (lamps coming on). The fire shot and the lighting-edit + shot as they were. `compose` + `judge` + `upscale` 1.73 → 1.80 ms at 1440p output. + test-render 190 pass (`upscale/lamps-still-live`, `upscale/lamps-in-motion-live`: main fails both). The moving + test reads the same with the motion clamp on, off or relaxed (test-render's live path; the cause not found): it + guards the camera moving from a frame script, not the motion clamp, which the Tokyo shots above measure. +- A lamp switched off fades over 4 to 8 frames at full resolution where main drops it at once (a lamp is sampled + about once a jitter sequence, the lock's life: nothing tells sooner that it is off). +- What was tried and left out, each measured (`out/liveq/gpu-runs-std.md`, K1–X9): + - variance clipping: a lamp no sample hit is outside any box made of this frame's samples; + - FSR 2's test on the composed sample: a traced frame carries more than 5 % of noise from sample to sample, most + of Kabukichō was "thin", the clamp was off everywhere and the wet street's reflections trailed 78 px for 53; on + the sample less its specular part: the same. The test is on what the camera ray saw directly, which has no noise; + - FSR 2's ridge rule (brighter or darker than every sample unlike it): lamps strung a pixel apart have a brighter + neighbour more often than not, and the tower's legs stayed unlocked; + - the lock of the nearest sample alone: a lone lock is a traced pixel off its lamp half the time; + - the lock in motion only: after the stop the tower was 7 % too bright for half a second again; + - `gone` only where the two depths moved against each other on screen: right for a lamp, wrong for this engine, + where a wall blown out uncovers what is behind it with nothing moving (the slice 5.28 → 5.66); + - the lock ended by depth where something nearer stood a frame ago: that is the test the lamp fails; + - FSR 2's release of a lock on a change of the scene (its `UpdateLockStatus`: a coarse mip of the frame's log + luminance, the lock killed when it changes by more than 0.1 of its sixth root; the same change as a reactive + factor): no effect on any shot here, nor on a probe made for it (lamps on a wall whose light goes out). FSR 2 + and TSR take the raw rasterised frame; this upscaler sits behind a temporal denoiser, a change of lighting + reaches it over frames and is the denoiser's to follow (the fast-history clamp above). +- Left: thin things that do not emit (a wire, a lattice by day) are still taken for gone under the jitter; the + locked lettering of a still sign comes back from a motion's blur by the blend alone (the 0.2 on Kabukichō's + signs); a lamp that comes on arrives over a quarter of a second. + Not done here: filters sized by hit distance, which is what would let a short history stand without noise; the Playground and the page still open a 1280 × 720 view; the live buffers are sized by the output (about 1.7 GB at 2560 × 1440), though the traced grid rarely fills it. diff --git a/renderer/shaders/live.wgsl b/renderer/shaders/live.wgsl index 7cb4897..f969910 100644 --- a/renderer/shaders/live.wgsl +++ b/renderer/shaders/live.wgsl @@ -897,10 +897,14 @@ fn compose(@builtin(global_invocation_id) gid: vec3) { // How much this sample flickers (`Flicker`), from where its surface was in the previous frame's traced grid. let now = min(lum(shown), 6.0e4); var f = Flicker(vec2(0.0), 0.0, 0.0); + var lock_was = 0.0; let ps = vec2(L.isz.zw); if (L.size.w == 0u && ps.x > 0) { let pc = vec2(floor((sp - vel) * vec2(ps) / vec2(f32(L.size.x), f32(L.size.y)))); - if (pc.x >= 0 && pc.y >= 0 && pc.x < ps.x && pc.y < ps.y) { f = flicker_at(u32(pc.y * ps.x + pc.x)); } + if (pc.x >= 0 && pc.y >= 0 && pc.x < ps.x && pc.y < ps.y) { + f = flicker_at(u32(pc.y * ps.x + pc.x)); + lock_was = unpack2x16float(ae_out[u32(pc.y * ps.x + pc.x)].y).y; + } } let still = 1.0 - clamp(length(vel) / UP_SPEED, 0.0, 1.0); var sd = select(1.0, f.sd, f.n >= 1.0); @@ -908,9 +912,30 @@ fn compose(@builtin(global_invocation_id) gid: vec3) { let bend = min(abs(now - 2.0 * f.last.x + f.last.y) / max(max(now, max(f.last.x, f.last.y)), 1e-9), UP_ONSET * max(f.sd, 0.01)); sd = sqrt(mix(f.sd * f.sd, bend * bend / 6.0, still / min(f.n - 1.0, 8.0))); } + // The thin feature's lock here (`Lock`): new where this sample's emission is one, else a frame older, from where + // the surface was. + let floor_l = 0.01 / exposure_scale(); + var e9: array; + var like = 0u; + for (var k = 0; k < 9; k++) { + let j = u32(clamp(i32(gid.y) + k / 3 - 1, 0, i32(ih) - 1)) * iw + u32(clamp(i32(gid.x) + k % 3 - 1, 0, i32(iw) - 1)); + e9[k] = lum(sample_px[j].emit.rgb) / max(L.misc2.w, 1.0) + floor_l; + } + for (var k = 0; k < 9; k++) { + if (max(e9[k], e9[4]) < THIN_LIKE * min(e9[k], e9[4])) { like |= 1u << u32(k); } + } + var across = false; + for (var k = 0; k < 4; k++) { + let a = e9[k]; + let b = e9[8 - k]; + across = across || (e9[4] > THIN_LIKE * max(a, b)) || (THIN_LIKE * e9[4] < min(a, b)); + } + let thin = across && (like & 27u) != 27u && (like & 54u) != 54u && (like & 216u) != 216u && (like & 432u) != 432u; + // (None on animated shading: it shows this frame's picture, and its history is not kept for it to stand by.) + let reactive = max(select(0.0, 1.0, px.albedo.w > 1.5), clamp(medium_share(q) / UP_MEDIUM, 0.0, 1.0)); + ae_in[q].y = (ae_in[q].y & 0xffffu) | (pack2x16float(vec2(0.0, select(max(lock_was - 1.0 / jitter_phases(), 0.0), 1.0, thin) * (1.0 - reactive))) & 0xffff0000u); ae_in[q].z = pack2x16float(vec2(now, select(now, f.last.x, f.n >= 1.0))); ae_in[q].w = pack2x16float(vec2(sd * still, min(max(f.n + still, 1.0), 9.0))); - let reactive = max(select(0.0, 1.0, px.albedo.w > 1.5), clamp(medium_share(q) / UP_MEDIUM, 0.0, 1.0)); xhist_out[q] = vec4(vel, depth, reactive); } @@ -956,7 +981,12 @@ const UP_STALE: f32 = 0.05; // way every upscaler finds one, by depth: what is nearest the camera among the nine samples now lies behind what was // nearest among the nine of the previous frame there (both dilated, so a still edge under the jitter mostly agrees // with itself), confirmed by colour (the history there lies outside this frame's range), so the still edges that the -// jitter does catch keep their history. +// jitter does catch keep their history. Not where a thin feature is locked (`Lock`): a lamp finer than a pixel comes +// and goes among the nine samples with the jitter alone, in front of whatever is behind it, its light in the history +// and not in this frame; every lamp against the sky was something gone each frame the samples missed it, and a field +// of them, held still, kept a third of its light. (Thin things that do not emit, a wire, a lattice by day, are still +// taken so. Asking that the two depths move against each other on screen was tried and is wrong here: a wall blown +// out uncovers what is behind it with nothing moving.) // A sample is a point and the history's pixels are means over a pixel: of a feature finer than a pixel (a row of // windows, a grating) a sample lies outside their range in a still picture too, a little further on some jitters than @@ -983,6 +1013,41 @@ const UP_STALE: f32 = 0.05; // every frame), so the deviation is added to by how still the pixel is and kept by the same, none from UP_SPEED on // (TSR fades its flicker rule out with velocity too): a pixel that moved has none and earns it again at rest, // while its history, cut to UP_MOVING samples by the motion, grows back. +// `Lock`, per traced sample, in the half word the fast mean leaves free in `ae_in`'s y (f16): how much longer the +// picture here counts as a thin feature, 1 when a sample showed one, a jitter sequence later 0 (FSR 2's pixel locks). A +// thin feature is a sample no 2 x 2 block of the nine around it is like (within THIN_LIKE of its luminance: FSR 2's +// rule) and that is above both or below both of some pair of opposite neighbours: a lamp or a lit window finer than a +// traced pixel, which most frames' samples miss, and the dark sample between two of them, which is this frame's miss. +// (FSR 2 asks for a sample brighter or darker than all those unlike it, a single line's rule: of lamps strung a pixel +// apart, in several colours, most have a brighter neighbour, and Tokyo Tower's legs went unlocked. A slope is above one +// neighbour and below the other in every direction and stays out.) The rules that hold the history to this frame's +// samples stand back for a locked pixel: its history is clamped into the nine samples in motion only in part (`upscale`, +// LOCK_RELAX: the lamp no sample hit this frame is still there; Tokyo Tower in a turn lost a quarter of its light and +// its legs), and its sample's distance from the history's range is no finding (`judge`: a hit lies above the range, a +// miss within it, and the corrections went one way); nor is a lamp the samples miss this frame something gone from in +// front (`judge`'s depth test). `compose` carries the lock with the surface. UE's TSR loosens its clamp the same way +// where it detects thin geometry. +// The test runs on what the camera ray saw directly (the sample's emission: lamps, windows, signs), which has no noise: +// FSR 2's test is made for a rasterised frame, and on the composed sample of a traced one (short histories in motion, +// reflections without a spatial filter: more than 5 % from sample to sample) it called most of Kabukichō thin; the +// clamp was then off everywhere and the reflections on the wet street trailed 78 pixels in a strafe for 53. +const THIN_LIKE: f32 = 1.05; +// How much of a locked pixel's history the clamp spares in motion, a frame (a relaxation weight: TSR caps its own +// loosening for thin geometry, r.TSR.ThinGeometryDetection.Coverage.MaxRelaxationWeight). A lamp no sample hit this +// frame stays, and what no sample supports fades by a quarter a frame instead of staying a jitter sequence: drawn +// where the background's motion took it (a lamp the samples missed reprojects with what is behind it), a spared +// history was a second copy of the lamp. Tokyo Tower in a quick orbit of the play window (a degree a frame, about a +// point 50 m ahead) against offline, ΔE / luminance: spared whole 4.86 / 1.20, its beacon doubled; 0.75: 4.25 / 0.82; +// 0.5: 4.47 / 0.69; not spared (main) 5.06 / 0.71. In a slow turn (0.25 degrees a frame) 2.23 / 1.05, 2.17 / 0.80, +// main 2.81 / 0.77. +const LOCK_RELAX: f32 = 0.75; +// The frames of the jitter sequence (live.rs: 8 at full resolution, as many more as an output pixel has fewer samples). +fn jitter_phases() -> f32 { + return clamp(round(8.0 * f32(L.size.x) * f32(L.size.y) / (f32(L.isz.x) * f32(L.isz.y))), 8.0, 512.0); +} +// (In full for the first half of its life, fading over the second.) +fn locked(q: u32) -> f32 { return clamp(2.0 * unpack2x16float(ae_in[q].y).y, 0.0, 1.0); } + struct Flicker { last: vec2, sd: f32, n: f32 }; fn flicker_at(q: u32) -> Flicker { let v = ae_out[q]; @@ -1076,20 +1141,27 @@ fn judge(@builtin(global_invocation_id) gid: vec3) { if (was_near < 1.0e29 && now_near > was_near * (1.0 + UP_GONE) + 0.01) { var clo = vec3(1.0e30); var chi = vec3(-1.0e30); + // (And no thin feature's lock among the nine: a lamp the samples miss this frame is not gone.) + var lock = 0.0; for (var k = 0; k < 9; k++) { let qx = i32(gid.x) + k % 3 - 1; let qy = i32(gid.y) + k / 3 - 1; if (qx < 0 || qy < 0 || qx >= iw || qy >= ih) { continue; } - let cq = fb[u32(qy * iw + qx)].rgb; + let j = u32(qy * iw + qx); + let cq = fb[j].rgb; clo = min(clo, cq); chi = max(chi, cq); + lock = max(lock, locked(j)); } let hm = hsum / hn; - gone = lum(abs(hm - clamp(hm, clo, chi))) > UP_STALE * lum(hm) + 0.01 / exposure_scale(); + gone = lock < 0.5 && lum(abs(hm - clamp(hm, clo, chi))) > UP_STALE * lum(hm) + 0.01 / exposure_scale(); } } } } + // A locked sample's distance from the history's range is no finding (a lamp finer than a pixel lies above it + // whenever a sample hits it). + d *= 1.0 - locked(q); ae_out[q] = vec4(pack2x16float(d.xy), pack2x16float(vec2(d.z, l)), pack2x16float(vel), u32(gone)); } @@ -1200,6 +1272,9 @@ fn upscale(@builtin(global_invocation_id) gid: vec3) { var wsum = 0.0; var xlo = vec3(1.0e30); var xhi = vec3(-1.0e30); + // (The lock of any of the nine: a lock is kept by the traced pixel, carried to the nearest, and the lamp it is for + // may be a pixel to the side by now.) + var lock = 0.0; for (var k = 0; k < 9; k++) { let qx = bc.x + k % 3 - 1; let qy = bc.y + k / 3 - 1; @@ -1211,6 +1286,7 @@ fn upscale(@builtin(global_invocation_id) gid: vec3) { wsum += wt; xlo = min(xlo, c); xhi = max(xhi, c); + lock = max(lock, locked(u32(qy * iw + qx))); } let cur = select(fb[u32(bc.y * iw + bc.x)].rgb, sum / wsum, wsum > 1e-9); let conf = min(wsum, 1.0); @@ -1242,15 +1318,15 @@ fn upscale(@builtin(global_invocation_id) gid: vec3) { // In motion the history is resampled every frame, each time a little blurrier, and what it holds of another // surface (the band behind a moving edge reprojects by the edge's motion) nothing else removes: so, as the pixel's // speed nears UP_SPEED, it holds fewer samples (UE TSR's r.TSR.Velocity.WeightClampingSampleCount / PixelSpeed) and - // is clamped into the range of this frame's nine samples (every temporal AA's rule; FSR 2's rectification). Holding - // still it keeps both its length and the detail finer than a traced pixel, which the clamp would cut. The orbit of - // Thames at 9 pixels a frame against offline (detail / trail px): neither 0.48 / 49, the clamp 0.45 / 36, 4 samples - // 0.57 / 37, both 0.53 / 31. + // is clamped into the range of this frame's nine samples (every temporal AA's rule; FSR 2's rectification), but for + // a thin feature (`Lock`). Holding still it keeps both its length and the detail finer than a traced pixel, which + // the clamp would cut. The orbit of Thames at 9 pixels a frame against offline (detail / trail px): neither + // 0.48 / 49, the clamp 0.45 / 36, 4 samples 0.57 / 37, both 0.53 / 31. let moving = clamp(length(vel) / UP_SPEED, 0.0, 1.0); let kept = hist.w * (1.0 - verdict.w); if (kept > 0.0) { var hc = max(hist.rgb + verdict.xyz, vec3(0.0)); - hc = mix(hc, clamp(hc, xlo, xhi), moving); + hc = mix(hc, clamp(hc, xlo, xhi), moving * (1.0 - LOCK_RELAX * lock)); let most = min(mix(UP_SAMPLES, UP_MOVING, moving), mix(UP_SAMPLES, UP_REACTIVE, clamp(xhist_out[u32(bc.y * iw + bc.x)].w, 0.0, 1.0))); count = min(kept + conf, most); let a = clamp(conf / max(count, 1e-4), 0.0, 1.0); diff --git a/tests/render/upscale-lamps-in-motion-live.ts b/tests/render/upscale-lamps-in-motion-live.ts new file mode 100644 index 0000000..7cf85c7 --- /dev/null +++ b/tests/render/upscale-lamps-in-motion-live.ts @@ -0,0 +1,24 @@ +// Lamps finer than a pixel while the view moves (live.wgsl `Lock`, `judge`'s disocclusion): the field of +// upscale/lamps-still-live, the camera sliding along it 1.4 pixels a frame. Its mean radiance is the lamps' share of +// it, as at rest: 0.1. (In motion the upscaler clamps its history into the frame's nine samples, and a lamp no sample +// hit this frame is outside them; and the lamp the samples missed, in front of a background that parallax moves +// against it, was taken for something gone. The field kept a sixth to a quarter of its light.) +import { defineGame } from "@threert/gameplay"; +import { lampField } from "./upscale-lamps-still-live.ts"; +import type { TestMeta } from "./lib.ts"; + +export const test: TestMeta = { + id: "upscale/lamps-in-motion-live", features: ["dynamic-emission", "upscaler", "procedural"], reference: "analytic", temporal: true, + metric: { kind: "radiance", region: [0.3, 0.3, 0.7, 0.7], expected: 0.1, tolerance: 0.15 }, + live: 60, + note: "lamps a third of a pixel wide, 1.44 % of the field at 6.94 nits, the camera sliding 1.4 pixels a frame: 0.1", +}; + +export default defineGame>({ + state: {}, + setup: () => lampField(), + frame(f) { + f.camera.position.x += 0.05; + }, + handlers: {}, +}); diff --git a/tests/render/upscale-lamps-still-live.ts b/tests/render/upscale-lamps-still-live.ts new file mode 100644 index 0000000..6aec92b --- /dev/null +++ b/tests/render/upscale-lamps-still-live.ts @@ -0,0 +1,33 @@ +// Lamps finer than a pixel, held still (live.wgsl `judge`'s disocclusion): a field of emitting squares a third of a +// pixel wide, 2.8 pixels apart, on black. The field's mean radiance is the lamps' share of it, 1.44 %, of their 6.94 +// nits: 0.1 (offline 0.103). (Under the jitter a lamp is hit by some frames' samples and missed by the rest; missed, +// it was taken for something gone from in front of the sky, its history was dropped, and the field kept a third of +// its light.) +import { Color, MeshStandardMaterial, ProceduralMesh, proceduralBoxes } from "@threert/scene"; +import { defineGame } from "@threert/gameplay"; +import { camera, result, sceneWithSky, type TestMeta } from "./lib.ts"; + +export const test: TestMeta = { + id: "upscale/lamps-still-live", features: ["dynamic-emission", "upscaler", "procedural"], reference: "analytic", temporal: true, + metric: { kind: "radiance", region: [0.3, 0.3, 0.7, 0.7], expected: 0.1, tolerance: 0.1 }, + live: 60, + note: "lamps a third of a pixel wide, 1.44 % of the field at 6.94 nits, the view still: 0.1", +}; + +const SIZE = 0.012, PITCH = 0.1; +/** The field (12 m by 2.4 m, at z = 0) and a camera 6 m in front of it, 1.5 m left of its middle. */ +export function lampField() { + const scene = sceneWithSky(0); + const lamp = new MeshStandardMaterial({ name: "lamp", color: new Color(0, 0, 0), emissive: new Color(1, 1, 1), emissiveIntensity: 0.1 / (SIZE / PITCH) ** 2, roughness: 1 }); + const boxes: { min: [number, number, number]; max: [number, number, number] }[] = []; + for (let j = -12; j <= 12; j++) for (let i = -70; i <= 70; i++) boxes.push({ min: [i * PITCH, j * PITCH, -0.0005], max: [i * PITCH + SIZE, j * PITCH + SIZE, 0] }); + const field = new ProceduralMesh(proceduralBoxes(boxes), lamp); + field.name = "field"; + return result(scene, camera(scene, [-1.5, 0, 6], [-1.5, 0, 0]), [field]); +} + +export default defineGame>({ + state: {}, + setup: () => lampField(), + handlers: {}, +});