From cae535affcfecd420eaf17c7640deab7c22d7e56 Mon Sep 17 00:00:00 2001 From: yujiaxu-ops Date: Fri, 13 Nov 2026 16:36:09 +0800 Subject: [PATCH 1/2] =?UTF-8?q?The=20thin=20feature's=20lock:=20lamps=20fi?= =?UTF-8?q?ner=20than=20a=20pixel=20keep=20their=20light,=20held=20still?= =?UTF-8?q?=20and=20in=20motion=20(ROADMAP=20=C2=A7=201a)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two faults of the upscaler, measured against offline renders, both older than the flicker tolerance: - Held still, a field of lamps a third of a pixel wide kept a third of its light. `judge` takes a surface for gone from in front when what was nearest among the nine samples a frame ago is nearer than what is nearest now and the history's colour is not in this frame: a lamp the jitter hit last frame and missed this one. Its history was dropped every frame it was missed. - In motion the history is clamped into the frame's nine samples, and a lamp none of them hit is outside them: Tokyo Tower in a turn lost its legs and a quarter of its light. `Lock` (FSR 2's pixel locks; UE's TSR loosens its clamp the same way for thin geometry): a half word per traced pixel, made where the sample's emission is a thin feature, a jitter sequence long, carried with the surface, none on animated shading. A locked pixel's history is not clamped in motion, a locked sample's distance from the history's range is no finding, and nothing is gone where one of the nine samples is locked. The lamps field: 0.035 to 0.106 still, 0.027 to 0.112 moving (offline 0.103). The tower in a turn against offline: dE 2.81 to 2.23, SSIM 0.867 to 0.928, luminance 0.77 to 1.05. Cost: where a view translates past emitters the locked ones trail a little (Kabukichō's signs 6.42 to 6.67 in a strafe, Bistro 2.49 to 2.63 a tenth of a second after one, the slice's explosion 5.28 to 5.41); bench-ghosting passes on both scenes. Tests: upscale/lamps-still-live, upscale/lamps-in-motion-live. Co-Authored-By: Claude Fable 5.2 --- docs/ROADMAP.md | 55 +++++++++++++ renderer/shaders/live.wgsl | 86 +++++++++++++++++--- tests/render/upscale-lamps-in-motion-live.ts | 24 ++++++ tests/render/upscale-lamps-still-live.ts | 33 ++++++++ 4 files changed, 188 insertions(+), 10 deletions(-) create mode 100644 tests/render/upscale-lamps-in-motion-live.ts create mode 100644 tests/render/upscale-lamps-still-live.ts diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index a21282e..7716225 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -2040,6 +2040,61 @@ were. range that runs from the roof to the sky whatever is allowed outside it, and the samples beside it lie between their own history and the stale pixel. Holding still, nothing holds a history pixel against the samples. +**Lamps finer than a pixel: the thin feature's lock (2026-11-13, after it).** Dragging the view over Tokyo at night, +the tower broke up: its legs and most of its lamps gone while the camera turned, back half a second after it +stopped. Measured against offline renders, two faults of the upscaler, both older than the change above: + +- Held still, a field of lamps a third of a pixel wide kept a third of its light (0.035 of 0.103). `judge` takes a + surface for gone from in front when what was nearest among the nine samples a frame ago is nearer than what is + nearest now and the history's colour is not in this frame; a lamp the jitter hit last frame and missed this one + is exactly that, and its history was dropped every frame it was missed. +- In motion the history is clamped into the frame's nine samples, and a lamp none of them hit is outside them: the + tower in a turn lost a quarter of its light (the 4-sample cap in motion is not it: without it nothing changed). + +The remedy is the one FSR 2 (pixel locks) and UE's TSR (thin geometry detection) have: where the picture is a thin +feature, the rules that hold the history to this frame's samples stand back. `Lock` in `live.wgsl`, a half word per +traced pixel that the fast mean's buffer had free: + +- made by `compose` where the sample's emission is thin: no 2 × 2 block of the nine like it (FSR 2's rule), and + above or below both of some pair of opposite neighbours; a jitter sequence long, carried with the surface; none on + animated shading; +- a locked pixel's history is not clamped in motion (the lock of any of its nine samples), a locked sample's + distance from the history's range is no finding, and nothing is gone where one of the nine is locked. + +| | main | now | offline | +|---|---|---|---| +| lamps field held still, mean radiance | 0.035 | 0.106 | 0.103 | +| the same, the camera sliding 1.4 px a frame | 0.027 | 0.112 | 0.103 | +| Tokyo Tower in a turn (3.5 px a frame), ΔE / SSIM / luminance against offline | 2.81 / 0.867 / 0.77 | 2.23 / 0.928 / 1.05 | | +| the same, the whole frame | 2.69 / 0.889 / 1.01 | 2.56 / 0.915 / 1.06 | | +| the tower half a second after the stop | 1.89 / 0.948 / 1.07 | 1.47 / 0.964 / 1.00 | | +| the tower settled | 1.48 / 0.962 / 0.94 | 1.39 / 0.965 / 0.99 | | + +- What it costs, main → now. Where a view translates past emitters the locked ones trail a little: Kabukichō's + signs strafe 6.42 → 6.67, a tenth of a second after it 5.27 → 5.55 (trail 26 → 31 px), at the dolly's stop + 6.75 → 7.07 (its ground the same to 0.01, its whole frame within 0.08); Bistro's whole frame a tenth of a second + after a strafe 2.49 → 2.63 (trail 13 → 24 px), else within 0.05; the Bistro slice half a second after its + explosion 5.28 → 5.41; the orbit of Thames moving 2.24 → 2.29 (held 1.34 → 1.24). bench-ghosting passes on both + scenes. Held still: Bistro 1.85 = 1.85 (frame to frame 0.46 → 0.44), Kabukichō 4.85 → 4.86. Tokyo at dusk the + same in the mean, the skyline's p99 at 6 s 11–14 → 16–18 (lamps coming on). The fire shot and the lighting-edit + shot as they were. `compose` + `judge` + `upscale` 1.73 → 1.80 ms at 1440p output. + test-render 190 pass (`upscale/lamps-still-live`, `upscale/lamps-in-motion-live`: main fails both). +- What was tried and left out, each measured (`out/liveq/gpu-runs-std.md`, K1–X9): + - variance clipping: a lamp no sample hit is outside any box made of this frame's samples; + - FSR 2's test on the composed sample: a traced frame carries more than 5 % of noise from sample to sample, most + of Kabukichō was "thin", the clamp was off everywhere and the wet street's reflections trailed 78 px for 53; on + the sample less its specular part: the same. The test is on what the camera ray saw directly, which has no noise; + - FSR 2's ridge rule (brighter or darker than every sample unlike it): lamps strung a pixel apart have a brighter + neighbour more often than not, and the tower's legs stayed unlocked; + - the lock of the nearest sample alone: a lone lock is a traced pixel off its lamp half the time; + - the lock in motion only: after the stop the tower was 7 % too bright for half a second again; + - `gone` only where the two depths moved against each other on screen: right for a lamp, wrong for this engine, + where a wall blown out uncovers what is behind it with nothing moving (the slice 5.28 → 5.66); + - the lock ended by depth where something nearer stood a frame ago: that is the test the lamp fails. +- Left: thin things that do not emit (a wire, a lattice by day) are still taken for gone under the jitter; the + locked lettering of a still sign comes back from a motion's blur by the blend alone (the 0.2 on Kabukichō's + signs); a lamp that comes on arrives over a quarter of a second. + Not done here: filters sized by hit distance, which is what would let a short history stand without noise; the Playground and the page still open a 1280 × 720 view; the live buffers are sized by the output (about 1.7 GB at 2560 × 1440), though the traced grid rarely fills it. diff --git a/renderer/shaders/live.wgsl b/renderer/shaders/live.wgsl index 7cb4897..c70f8d3 100644 --- a/renderer/shaders/live.wgsl +++ b/renderer/shaders/live.wgsl @@ -897,10 +897,14 @@ fn compose(@builtin(global_invocation_id) gid: vec3) { // How much this sample flickers (`Flicker`), from where its surface was in the previous frame's traced grid. let now = min(lum(shown), 6.0e4); var f = Flicker(vec2(0.0), 0.0, 0.0); + var lock_was = 0.0; let ps = vec2(L.isz.zw); if (L.size.w == 0u && ps.x > 0) { let pc = vec2(floor((sp - vel) * vec2(ps) / vec2(f32(L.size.x), f32(L.size.y)))); - if (pc.x >= 0 && pc.y >= 0 && pc.x < ps.x && pc.y < ps.y) { f = flicker_at(u32(pc.y * ps.x + pc.x)); } + if (pc.x >= 0 && pc.y >= 0 && pc.x < ps.x && pc.y < ps.y) { + f = flicker_at(u32(pc.y * ps.x + pc.x)); + lock_was = unpack2x16float(ae_out[u32(pc.y * ps.x + pc.x)].y).y; + } } let still = 1.0 - clamp(length(vel) / UP_SPEED, 0.0, 1.0); var sd = select(1.0, f.sd, f.n >= 1.0); @@ -908,9 +912,30 @@ fn compose(@builtin(global_invocation_id) gid: vec3) { let bend = min(abs(now - 2.0 * f.last.x + f.last.y) / max(max(now, max(f.last.x, f.last.y)), 1e-9), UP_ONSET * max(f.sd, 0.01)); sd = sqrt(mix(f.sd * f.sd, bend * bend / 6.0, still / min(f.n - 1.0, 8.0))); } + // The thin feature's lock here (`Lock`): new where this sample's emission is one, else a frame older, from where + // the surface was. + let floor_l = 0.01 / exposure_scale(); + var e9: array; + var like = 0u; + for (var k = 0; k < 9; k++) { + let j = u32(clamp(i32(gid.y) + k / 3 - 1, 0, i32(ih) - 1)) * iw + u32(clamp(i32(gid.x) + k % 3 - 1, 0, i32(iw) - 1)); + e9[k] = lum(sample_px[j].emit.rgb) / max(L.misc2.w, 1.0) + floor_l; + } + for (var k = 0; k < 9; k++) { + if (max(e9[k], e9[4]) < THIN_LIKE * min(e9[k], e9[4])) { like |= 1u << u32(k); } + } + var across = false; + for (var k = 0; k < 4; k++) { + let a = e9[k]; + let b = e9[8 - k]; + across = across || (e9[4] > THIN_LIKE * max(a, b)) || (THIN_LIKE * e9[4] < min(a, b)); + } + let thin = across && (like & 27u) != 27u && (like & 54u) != 54u && (like & 216u) != 216u && (like & 432u) != 432u; + // (None on animated shading: it shows this frame's picture, and its history is not kept for it to stand by.) + let reactive = max(select(0.0, 1.0, px.albedo.w > 1.5), clamp(medium_share(q) / UP_MEDIUM, 0.0, 1.0)); + ae_in[q].y = (ae_in[q].y & 0xffffu) | (pack2x16float(vec2(0.0, select(max(lock_was - 1.0 / jitter_phases(), 0.0), 1.0, thin) * (1.0 - reactive))) & 0xffff0000u); ae_in[q].z = pack2x16float(vec2(now, select(now, f.last.x, f.n >= 1.0))); ae_in[q].w = pack2x16float(vec2(sd * still, min(max(f.n + still, 1.0), 9.0))); - let reactive = max(select(0.0, 1.0, px.albedo.w > 1.5), clamp(medium_share(q) / UP_MEDIUM, 0.0, 1.0)); xhist_out[q] = vec4(vel, depth, reactive); } @@ -956,7 +981,12 @@ const UP_STALE: f32 = 0.05; // way every upscaler finds one, by depth: what is nearest the camera among the nine samples now lies behind what was // nearest among the nine of the previous frame there (both dilated, so a still edge under the jitter mostly agrees // with itself), confirmed by colour (the history there lies outside this frame's range), so the still edges that the -// jitter does catch keep their history. +// jitter does catch keep their history. Not where a thin feature is locked (`Lock`): a lamp finer than a pixel comes +// and goes among the nine samples with the jitter alone, in front of whatever is behind it, its light in the history +// and not in this frame; every lamp against the sky was something gone each frame the samples missed it, and a field +// of them, held still, kept a third of its light. (Thin things that do not emit, a wire, a lattice by day, are still +// taken so. Asking that the two depths move against each other on screen was tried and is wrong here: a wall blown +// out uncovers what is behind it with nothing moving.) // A sample is a point and the history's pixels are means over a pixel: of a feature finer than a pixel (a row of // windows, a grating) a sample lies outside their range in a still picture too, a little further on some jitters than @@ -983,6 +1013,31 @@ const UP_STALE: f32 = 0.05; // every frame), so the deviation is added to by how still the pixel is and kept by the same, none from UP_SPEED on // (TSR fades its flicker rule out with velocity too): a pixel that moved has none and earns it again at rest, // while its history, cut to UP_MOVING samples by the motion, grows back. +// `Lock`, per traced sample, in the half word the fast mean leaves free in `ae_in`'s y (f16): how much longer the +// picture here counts as a thin feature, 1 when a sample showed one, a jitter sequence later 0 (FSR 2's pixel locks). +// A thin feature is a sample no 2 x 2 block of the nine around it is like (within THIN_LIKE of its luminance: FSR +// 2's rule) and that is above both or below both of some pair of opposite neighbours: a lamp or a lit window finer +// than a traced pixel, which most frames' samples miss, and the dark sample between two of them, which is this +// frame's miss. (FSR 2 asks for a sample brighter or darker than all those unlike it, a single line's rule: of +// lamps strung a pixel apart, in several colours, most have a brighter neighbour, and Tokyo Tower's legs went +// unlocked. A slope is above one neighbour and below the other in every direction and stays out.) Neither rule that holds the history to this frame's samples applies to a locked pixel: its +// history is not clamped into the nine samples in motion (`upscale`: the lamp no sample hit this frame is still +// there; Tokyo Tower in a turn lost a quarter of its light and its legs), and its sample's distance from the +// history's range is no finding (`judge`: a hit lies above the range, a miss within it, and the corrections went one +// way); nor is a lamp the samples miss this frame something gone from in front (`judge`'s depth test). `compose` +// carries the lock with the surface. UE's TSR loosens its clamp the same way where it detects thin geometry. +// The test runs on what the camera ray saw directly (the sample's emission: lamps, windows, signs), which has no +// noise: FSR 2's test is made for a rasterised frame, and on the composed sample of a traced one (short histories +// in motion, reflections without a spatial filter: more than 5 % from sample to sample) it called most of Kabukichō +// thin; the clamp was then off everywhere and the reflections on the wet street trailed 78 pixels in a strafe for 53. +const THIN_LIKE: f32 = 1.05; +// The frames of the jitter sequence (live.rs: 8 at full resolution, as many more as an output pixel has fewer samples). +fn jitter_phases() -> f32 { + return clamp(round(8.0 * f32(L.size.x) * f32(L.size.y) / (f32(L.isz.x) * f32(L.isz.y))), 8.0, 512.0); +} +// (In full for the first half of its life, fading over the second.) +fn locked(q: u32) -> f32 { return clamp(2.0 * unpack2x16float(ae_in[q].y).y, 0.0, 1.0); } + struct Flicker { last: vec2, sd: f32, n: f32 }; fn flicker_at(q: u32) -> Flicker { let v = ae_out[q]; @@ -1076,20 +1131,27 @@ fn judge(@builtin(global_invocation_id) gid: vec3) { if (was_near < 1.0e29 && now_near > was_near * (1.0 + UP_GONE) + 0.01) { var clo = vec3(1.0e30); var chi = vec3(-1.0e30); + // (And no thin feature's lock among the nine: a lamp the samples miss this frame is not gone.) + var lock = 0.0; for (var k = 0; k < 9; k++) { let qx = i32(gid.x) + k % 3 - 1; let qy = i32(gid.y) + k / 3 - 1; if (qx < 0 || qy < 0 || qx >= iw || qy >= ih) { continue; } - let cq = fb[u32(qy * iw + qx)].rgb; + let j = u32(qy * iw + qx); + let cq = fb[j].rgb; clo = min(clo, cq); chi = max(chi, cq); + lock = max(lock, locked(j)); } let hm = hsum / hn; - gone = lum(abs(hm - clamp(hm, clo, chi))) > UP_STALE * lum(hm) + 0.01 / exposure_scale(); + gone = lock < 0.5 && lum(abs(hm - clamp(hm, clo, chi))) > UP_STALE * lum(hm) + 0.01 / exposure_scale(); } } } } + // A locked sample's distance from the history's range is no finding (a lamp finer than a pixel lies above it + // whenever a sample hits it). + d *= 1.0 - locked(q); ae_out[q] = vec4(pack2x16float(d.xy), pack2x16float(vec2(d.z, l)), pack2x16float(vel), u32(gone)); } @@ -1200,6 +1262,9 @@ fn upscale(@builtin(global_invocation_id) gid: vec3) { var wsum = 0.0; var xlo = vec3(1.0e30); var xhi = vec3(-1.0e30); + // (The lock of any of the nine: a lock is kept by the traced pixel, carried to the nearest, and the lamp it is for + // may be a pixel to the side by now.) + var lock = 0.0; for (var k = 0; k < 9; k++) { let qx = bc.x + k % 3 - 1; let qy = bc.y + k / 3 - 1; @@ -1211,6 +1276,7 @@ fn upscale(@builtin(global_invocation_id) gid: vec3) { wsum += wt; xlo = min(xlo, c); xhi = max(xhi, c); + lock = max(lock, locked(u32(qy * iw + qx))); } let cur = select(fb[u32(bc.y * iw + bc.x)].rgb, sum / wsum, wsum > 1e-9); let conf = min(wsum, 1.0); @@ -1242,15 +1308,15 @@ fn upscale(@builtin(global_invocation_id) gid: vec3) { // In motion the history is resampled every frame, each time a little blurrier, and what it holds of another // surface (the band behind a moving edge reprojects by the edge's motion) nothing else removes: so, as the pixel's // speed nears UP_SPEED, it holds fewer samples (UE TSR's r.TSR.Velocity.WeightClampingSampleCount / PixelSpeed) and - // is clamped into the range of this frame's nine samples (every temporal AA's rule; FSR 2's rectification). Holding - // still it keeps both its length and the detail finer than a traced pixel, which the clamp would cut. The orbit of - // Thames at 9 pixels a frame against offline (detail / trail px): neither 0.48 / 49, the clamp 0.45 / 36, 4 samples - // 0.57 / 37, both 0.53 / 31. + // is clamped into the range of this frame's nine samples (every temporal AA's rule; FSR 2's rectification), but for + // a thin feature (`Lock`). Holding still it keeps both its length and the detail finer than a traced pixel, which + // the clamp would cut. The orbit of Thames at 9 pixels a frame against offline (detail / trail px): neither + // 0.48 / 49, the clamp 0.45 / 36, 4 samples 0.57 / 37, both 0.53 / 31. let moving = clamp(length(vel) / UP_SPEED, 0.0, 1.0); let kept = hist.w * (1.0 - verdict.w); if (kept > 0.0) { var hc = max(hist.rgb + verdict.xyz, vec3(0.0)); - hc = mix(hc, clamp(hc, xlo, xhi), moving); + hc = mix(hc, clamp(hc, xlo, xhi), moving * (1.0 - lock)); let most = min(mix(UP_SAMPLES, UP_MOVING, moving), mix(UP_SAMPLES, UP_REACTIVE, clamp(xhist_out[u32(bc.y * iw + bc.x)].w, 0.0, 1.0))); count = min(kept + conf, most); let a = clamp(conf / max(count, 1e-4), 0.0, 1.0); diff --git a/tests/render/upscale-lamps-in-motion-live.ts b/tests/render/upscale-lamps-in-motion-live.ts new file mode 100644 index 0000000..7cf85c7 --- /dev/null +++ b/tests/render/upscale-lamps-in-motion-live.ts @@ -0,0 +1,24 @@ +// Lamps finer than a pixel while the view moves (live.wgsl `Lock`, `judge`'s disocclusion): the field of +// upscale/lamps-still-live, the camera sliding along it 1.4 pixels a frame. Its mean radiance is the lamps' share of +// it, as at rest: 0.1. (In motion the upscaler clamps its history into the frame's nine samples, and a lamp no sample +// hit this frame is outside them; and the lamp the samples missed, in front of a background that parallax moves +// against it, was taken for something gone. The field kept a sixth to a quarter of its light.) +import { defineGame } from "@threert/gameplay"; +import { lampField } from "./upscale-lamps-still-live.ts"; +import type { TestMeta } from "./lib.ts"; + +export const test: TestMeta = { + id: "upscale/lamps-in-motion-live", features: ["dynamic-emission", "upscaler", "procedural"], reference: "analytic", temporal: true, + metric: { kind: "radiance", region: [0.3, 0.3, 0.7, 0.7], expected: 0.1, tolerance: 0.15 }, + live: 60, + note: "lamps a third of a pixel wide, 1.44 % of the field at 6.94 nits, the camera sliding 1.4 pixels a frame: 0.1", +}; + +export default defineGame>({ + state: {}, + setup: () => lampField(), + frame(f) { + f.camera.position.x += 0.05; + }, + handlers: {}, +}); diff --git a/tests/render/upscale-lamps-still-live.ts b/tests/render/upscale-lamps-still-live.ts new file mode 100644 index 0000000..6aec92b --- /dev/null +++ b/tests/render/upscale-lamps-still-live.ts @@ -0,0 +1,33 @@ +// Lamps finer than a pixel, held still (live.wgsl `judge`'s disocclusion): a field of emitting squares a third of a +// pixel wide, 2.8 pixels apart, on black. The field's mean radiance is the lamps' share of it, 1.44 %, of their 6.94 +// nits: 0.1 (offline 0.103). (Under the jitter a lamp is hit by some frames' samples and missed by the rest; missed, +// it was taken for something gone from in front of the sky, its history was dropped, and the field kept a third of +// its light.) +import { Color, MeshStandardMaterial, ProceduralMesh, proceduralBoxes } from "@threert/scene"; +import { defineGame } from "@threert/gameplay"; +import { camera, result, sceneWithSky, type TestMeta } from "./lib.ts"; + +export const test: TestMeta = { + id: "upscale/lamps-still-live", features: ["dynamic-emission", "upscaler", "procedural"], reference: "analytic", temporal: true, + metric: { kind: "radiance", region: [0.3, 0.3, 0.7, 0.7], expected: 0.1, tolerance: 0.1 }, + live: 60, + note: "lamps a third of a pixel wide, 1.44 % of the field at 6.94 nits, the view still: 0.1", +}; + +const SIZE = 0.012, PITCH = 0.1; +/** The field (12 m by 2.4 m, at z = 0) and a camera 6 m in front of it, 1.5 m left of its middle. */ +export function lampField() { + const scene = sceneWithSky(0); + const lamp = new MeshStandardMaterial({ name: "lamp", color: new Color(0, 0, 0), emissive: new Color(1, 1, 1), emissiveIntensity: 0.1 / (SIZE / PITCH) ** 2, roughness: 1 }); + const boxes: { min: [number, number, number]; max: [number, number, number] }[] = []; + for (let j = -12; j <= 12; j++) for (let i = -70; i <= 70; i++) boxes.push({ min: [i * PITCH, j * PITCH, -0.0005], max: [i * PITCH + SIZE, j * PITCH + SIZE, 0] }); + const field = new ProceduralMesh(proceduralBoxes(boxes), lamp); + field.name = "field"; + return result(scene, camera(scene, [-1.5, 0, 6], [-1.5, 0, 0]), [field]); +} + +export default defineGame>({ + state: {}, + setup: () => lampField(), + handlers: {}, +}); From d40f832b073ad5312de2d063f6ee6a2696c05828 Mon Sep 17 00:00:00 2001 From: yujiaxu-ops Date: Fri, 13 Nov 2026 18:47:17 +0800 Subject: [PATCH 2/2] The thin feature's lock spares a locked history in part in motion (0.75 a frame): no second copy of a lamp MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first form spared a locked pixel's history from the motion clamp entirely. In a quick orbit of the play window (a degree a frame about a point 50 m ahead) a lamp the samples missed reprojected with what is behind it, and the spared history became a second copy: Tokyo Tower's beacon doubled, its lamps drawn out sideways. TSR caps how far it loosens its clamp for thin geometry; here 0.75 of the unclamped history is spared a frame and what no sample supports fades by a quarter a frame. Tokyo Tower against offline, dE / luminance: the quick orbit 4.86 / 1.20 to 4.15 / 0.86 (main 5.06 / 0.71); the slow turn 2.23 / 1.05 to 2.17 / 0.80 (main 2.81 / 0.77). Kabukichō's signs a tenth of a second after a strafe 5.55 to 5.39 (main 5.27), Bistro 2.63 to 2.54 (main 2.49). Co-Authored-By: Claude Opus 5.5 --- docs/ROADMAP.md | 42 +++++++++++++++++++++++------------- renderer/shaders/live.wgsl | 44 +++++++++++++++++++++++--------------- 2 files changed, 54 insertions(+), 32 deletions(-) diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 7716225..626ef63 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -2058,27 +2058,34 @@ traced pixel that the fast mean's buffer had free: - made by `compose` where the sample's emission is thin: no 2 × 2 block of the nine like it (FSR 2's rule), and above or below both of some pair of opposite neighbours; a jitter sequence long, carried with the surface; none on animated shading; -- a locked pixel's history is not clamped in motion (the lock of any of its nine samples), a locked sample's - distance from the history's range is no finding, and nothing is gone where one of the nine is locked. +- in motion a locked pixel's history is clamped into the frame's nine samples only in part (`LOCK_RELAX` 0.75 of + it spared a frame: the lock of any of its nine samples), a locked sample's distance from the history's range is no + finding, and nothing is gone where one of the nine is locked. | | main | now | offline | |---|---|---|---| | lamps field held still, mean radiance | 0.035 | 0.106 | 0.103 | -| the same, the camera sliding 1.4 px a frame | 0.027 | 0.112 | 0.103 | -| Tokyo Tower in a turn (3.5 px a frame), ΔE / SSIM / luminance against offline | 2.81 / 0.867 / 0.77 | 2.23 / 0.928 / 1.05 | | -| the same, the whole frame | 2.69 / 0.889 / 1.01 | 2.56 / 0.915 / 1.06 | | -| the tower half a second after the stop | 1.89 / 0.948 / 1.07 | 1.47 / 0.964 / 1.00 | | -| the tower settled | 1.48 / 0.962 / 0.94 | 1.39 / 0.965 / 0.99 | | - +| the same, the camera sliding 1.4 px a frame (see below: this does not exercise the motion clamp) | 0.027 | 0.112 | 0.103 | +| Tokyo Tower in a turn (0.25° a frame), ΔE / SSIM / luminance against offline | 2.81 / 0.867 / 0.77 | 2.17 / 0.919 / 0.80 | | +| Tokyo Tower in a quick orbit of the play window (1.15° a frame about a point 50 m ahead) | 5.06 / 0.808 / 0.71 | 4.15 / 0.872 / 0.86 | | +| the tower settled after the turn | 1.48 / 0.962 / 0.94 | 1.39 / 0.965 / 0.99 | | + +- The share spared in motion. All of it (the first form) kept the tower whole in the slow turn (2.23 / 0.928 / 1.05) + but, in the play window's quick orbit, drew its lamps out sideways and doubled its beacon (4.86, luminance 1.20): + a lamp the samples miss reprojects with what is behind it, and a spared history was a second copy. 0.5: 4.47 / + 0.69; none above 16 px a frame: main again (5.11 / 0.56). TSR likewise caps how far it loosens its clamp. - What it costs, main → now. Where a view translates past emitters the locked ones trail a little: Kabukichō's - signs strafe 6.42 → 6.67, a tenth of a second after it 5.27 → 5.55 (trail 26 → 31 px), at the dolly's stop - 6.75 → 7.07 (its ground the same to 0.01, its whole frame within 0.08); Bistro's whole frame a tenth of a second - after a strafe 2.49 → 2.63 (trail 13 → 24 px), else within 0.05; the Bistro slice half a second after its - explosion 5.28 → 5.41; the orbit of Thames moving 2.24 → 2.29 (held 1.34 → 1.24). bench-ghosting passes on both - scenes. Held still: Bistro 1.85 = 1.85 (frame to frame 0.46 → 0.44), Kabukichō 4.85 → 4.86. Tokyo at dusk the + signs strafe 6.42 → 6.55, a tenth of a second after it 5.27 → 5.39 (trail 26 → 29 px), at the dolly's stop + 6.75 → 6.96 (its ground the same to 0.01); Bistro's whole frame a tenth of a second after a strafe 2.49 → 2.54 + (trail 13 → 18 px), else within 0.03; the Bistro slice half a second after its explosion 5.28 → 5.40; the orbit + of Thames moving 2.24 → 2.16 (trail 33 → 31 px), held 1.34 → 1.39. bench-ghosting passes on both scenes. Held still: Bistro 1.85 = 1.85 (frame to frame 0.46 → 0.44), Kabukichō 4.85 → 4.86. Tokyo at dusk the same in the mean, the skyline's p99 at 6 s 11–14 → 16–18 (lamps coming on). The fire shot and the lighting-edit shot as they were. `compose` + `judge` + `upscale` 1.73 → 1.80 ms at 1440p output. - test-render 190 pass (`upscale/lamps-still-live`, `upscale/lamps-in-motion-live`: main fails both). + test-render 190 pass (`upscale/lamps-still-live`, `upscale/lamps-in-motion-live`: main fails both). The moving + test reads the same with the motion clamp on, off or relaxed (test-render's live path; the cause not found): it + guards the camera moving from a frame script, not the motion clamp, which the Tokyo shots above measure. +- A lamp switched off fades over 4 to 8 frames at full resolution where main drops it at once (a lamp is sampled + about once a jitter sequence, the lock's life: nothing tells sooner that it is off). - What was tried and left out, each measured (`out/liveq/gpu-runs-std.md`, K1–X9): - variance clipping: a lamp no sample hit is outside any box made of this frame's samples; - FSR 2's test on the composed sample: a traced frame carries more than 5 % of noise from sample to sample, most @@ -2090,7 +2097,12 @@ traced pixel that the fast mean's buffer had free: - the lock in motion only: after the stop the tower was 7 % too bright for half a second again; - `gone` only where the two depths moved against each other on screen: right for a lamp, wrong for this engine, where a wall blown out uncovers what is behind it with nothing moving (the slice 5.28 → 5.66); - - the lock ended by depth where something nearer stood a frame ago: that is the test the lamp fails. + - the lock ended by depth where something nearer stood a frame ago: that is the test the lamp fails; + - FSR 2's release of a lock on a change of the scene (its `UpdateLockStatus`: a coarse mip of the frame's log + luminance, the lock killed when it changes by more than 0.1 of its sixth root; the same change as a reactive + factor): no effect on any shot here, nor on a probe made for it (lamps on a wall whose light goes out). FSR 2 + and TSR take the raw rasterised frame; this upscaler sits behind a temporal denoiser, a change of lighting + reaches it over frames and is the denoiser's to follow (the fast-history clamp above). - Left: thin things that do not emit (a wire, a lattice by day) are still taken for gone under the jitter; the locked lettering of a still sign comes back from a motion's blur by the blend alone (the 0.2 on Kabukichō's signs); a lamp that comes on arrives over a quarter of a second. diff --git a/renderer/shaders/live.wgsl b/renderer/shaders/live.wgsl index c70f8d3..f969910 100644 --- a/renderer/shaders/live.wgsl +++ b/renderer/shaders/live.wgsl @@ -1014,23 +1014,33 @@ const UP_STALE: f32 = 0.05; // (TSR fades its flicker rule out with velocity too): a pixel that moved has none and earns it again at rest, // while its history, cut to UP_MOVING samples by the motion, grows back. // `Lock`, per traced sample, in the half word the fast mean leaves free in `ae_in`'s y (f16): how much longer the -// picture here counts as a thin feature, 1 when a sample showed one, a jitter sequence later 0 (FSR 2's pixel locks). -// A thin feature is a sample no 2 x 2 block of the nine around it is like (within THIN_LIKE of its luminance: FSR -// 2's rule) and that is above both or below both of some pair of opposite neighbours: a lamp or a lit window finer -// than a traced pixel, which most frames' samples miss, and the dark sample between two of them, which is this -// frame's miss. (FSR 2 asks for a sample brighter or darker than all those unlike it, a single line's rule: of -// lamps strung a pixel apart, in several colours, most have a brighter neighbour, and Tokyo Tower's legs went -// unlocked. A slope is above one neighbour and below the other in every direction and stays out.) Neither rule that holds the history to this frame's samples applies to a locked pixel: its -// history is not clamped into the nine samples in motion (`upscale`: the lamp no sample hit this frame is still -// there; Tokyo Tower in a turn lost a quarter of its light and its legs), and its sample's distance from the -// history's range is no finding (`judge`: a hit lies above the range, a miss within it, and the corrections went one -// way); nor is a lamp the samples miss this frame something gone from in front (`judge`'s depth test). `compose` -// carries the lock with the surface. UE's TSR loosens its clamp the same way where it detects thin geometry. -// The test runs on what the camera ray saw directly (the sample's emission: lamps, windows, signs), which has no -// noise: FSR 2's test is made for a rasterised frame, and on the composed sample of a traced one (short histories -// in motion, reflections without a spatial filter: more than 5 % from sample to sample) it called most of Kabukichō -// thin; the clamp was then off everywhere and the reflections on the wet street trailed 78 pixels in a strafe for 53. +// picture here counts as a thin feature, 1 when a sample showed one, a jitter sequence later 0 (FSR 2's pixel locks). A +// thin feature is a sample no 2 x 2 block of the nine around it is like (within THIN_LIKE of its luminance: FSR 2's +// rule) and that is above both or below both of some pair of opposite neighbours: a lamp or a lit window finer than a +// traced pixel, which most frames' samples miss, and the dark sample between two of them, which is this frame's miss. +// (FSR 2 asks for a sample brighter or darker than all those unlike it, a single line's rule: of lamps strung a pixel +// apart, in several colours, most have a brighter neighbour, and Tokyo Tower's legs went unlocked. A slope is above one +// neighbour and below the other in every direction and stays out.) The rules that hold the history to this frame's +// samples stand back for a locked pixel: its history is clamped into the nine samples in motion only in part (`upscale`, +// LOCK_RELAX: the lamp no sample hit this frame is still there; Tokyo Tower in a turn lost a quarter of its light and +// its legs), and its sample's distance from the history's range is no finding (`judge`: a hit lies above the range, a +// miss within it, and the corrections went one way); nor is a lamp the samples miss this frame something gone from in +// front (`judge`'s depth test). `compose` carries the lock with the surface. UE's TSR loosens its clamp the same way +// where it detects thin geometry. +// The test runs on what the camera ray saw directly (the sample's emission: lamps, windows, signs), which has no noise: +// FSR 2's test is made for a rasterised frame, and on the composed sample of a traced one (short histories in motion, +// reflections without a spatial filter: more than 5 % from sample to sample) it called most of Kabukichō thin; the +// clamp was then off everywhere and the reflections on the wet street trailed 78 pixels in a strafe for 53. const THIN_LIKE: f32 = 1.05; +// How much of a locked pixel's history the clamp spares in motion, a frame (a relaxation weight: TSR caps its own +// loosening for thin geometry, r.TSR.ThinGeometryDetection.Coverage.MaxRelaxationWeight). A lamp no sample hit this +// frame stays, and what no sample supports fades by a quarter a frame instead of staying a jitter sequence: drawn +// where the background's motion took it (a lamp the samples missed reprojects with what is behind it), a spared +// history was a second copy of the lamp. Tokyo Tower in a quick orbit of the play window (a degree a frame, about a +// point 50 m ahead) against offline, ΔE / luminance: spared whole 4.86 / 1.20, its beacon doubled; 0.75: 4.25 / 0.82; +// 0.5: 4.47 / 0.69; not spared (main) 5.06 / 0.71. In a slow turn (0.25 degrees a frame) 2.23 / 1.05, 2.17 / 0.80, +// main 2.81 / 0.77. +const LOCK_RELAX: f32 = 0.75; // The frames of the jitter sequence (live.rs: 8 at full resolution, as many more as an output pixel has fewer samples). fn jitter_phases() -> f32 { return clamp(round(8.0 * f32(L.size.x) * f32(L.size.y) / (f32(L.isz.x) * f32(L.isz.y))), 8.0, 512.0); @@ -1316,7 +1326,7 @@ fn upscale(@builtin(global_invocation_id) gid: vec3) { let kept = hist.w * (1.0 - verdict.w); if (kept > 0.0) { var hc = max(hist.rgb + verdict.xyz, vec3(0.0)); - hc = mix(hc, clamp(hc, xlo, xhi), moving * (1.0 - lock)); + hc = mix(hc, clamp(hc, xlo, xhi), moving * (1.0 - LOCK_RELAX * lock)); let most = min(mix(UP_SAMPLES, UP_MOVING, moving), mix(UP_SAMPLES, UP_REACTIVE, clamp(xhist_out[u32(bc.y * iw + bc.x)].w, 0.0, 1.0))); count = min(kept + conf, most); let a = clamp(conf / max(count, 1e-4), 0.0, 1.0);