diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 7e6850b..3333551 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -125,9 +125,11 @@ look like. - **Classification.** The two frames are sampled on a 48 x 27 grid (`classify.wgsl`); a mean per-channel difference above `scene_cut.threshold` is a cut when the moving pair before it differed by less than half the threshold (a fast camera move is a run of large differences, a cut a lone spike after a calm pair), a mean below `held_frames.mean` with - every cell below `held_frames.max` is a hold, anything else motion. The readback is asynchronous (three may be in - flight; beyond that the pair counts as motion) and lands before the interval's first mid is due. The mean cell - difference in 0..765 units is the motion statistic the Pacer's motion-adaptive ceiling uses. + every cell below `held_frames.max` is a hold, anything else motion. With held frames on and no cadence re-timing, a + moving pair right after a held one is reported as a cut: a change after a held frame is animation on twos or threes, + shown as drawn. The readback is asynchronous (three may be in flight; beyond that the pair counts as motion) and lands + before the interval's first mid is due. The mean cell difference in 0..765 units is the motion statistic the Pacer's + motion-adaptive ceiling uses. - **Order with the Pacer.** Classify the new frame against the previous one, then `pacer.source_frame` with the kind, generate the requests it returns, then `push`: the pair must not change before the previous interval's mids are out. - **Held frames are reported, never dropped silently**: the Pacer shows the frame and generates nothing for the diff --git a/CHANGELOG.md b/CHANGELOG.md index ca96b38..23444e2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,17 @@ follow semantic versioning as described in [CONTRIBUTING.md](CONTRIBUTING.md). ## [Unreleased] +## [0.3.7] - 2026-11-10 + +### Changed + +- With held frames detected (`held_frames`) and no cadence re-timing, the first change after a held frame is not + interpolated. It is animation drawn on twos or threes (a mouth, a blink, a character against a still background), + not motion: its in-betweens morphed one drawing into the next within one frame time and then stood still for two, + four generated frames in 41 ms and none for 83, a smeared mouth and a rate jumping between 24 and 120. Such a pair + is shown as drawn (the classifier reports a cut: no in-betweens, a new frame); motion that goes on from frame to + frame is interpolated from its second pair. Cadence re-timing (video mode) still takes these changes as motion. + ## [0.3.6] - 2026-11-10 ### Changed diff --git a/Cargo.lock b/Cargo.lock index b035f32..a7ca4e7 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -548,7 +548,7 @@ dependencies = [ [[package]] name = "flyhigh-capi" -version = "0.3.6" +version = "0.3.7" dependencies = [ "flyhigh-engine", "serde_json", @@ -557,7 +557,7 @@ dependencies = [ [[package]] name = "flyhigh-cli" -version = "0.3.6" +version = "0.3.7" dependencies = [ "anyhow", "bytemuck", @@ -572,7 +572,7 @@ dependencies = [ [[package]] name = "flyhigh-engine" -version = "0.3.6" +version = "0.3.7" dependencies = [ "anyhow", "bytemuck", @@ -588,7 +588,7 @@ dependencies = [ [[package]] name = "flyhigh-options" -version = "0.3.6" +version = "0.3.7" dependencies = [ "proc-macro2", "quote", @@ -597,7 +597,7 @@ dependencies = [ [[package]] name = "flyhigh-player" -version = "0.3.6" +version = "0.3.7" dependencies = [ "anyhow", "flyhigh-engine", @@ -608,7 +608,7 @@ dependencies = [ [[package]] name = "flyhigh-web" -version = "0.3.6" +version = "0.3.7" dependencies = [ "console_error_panic_hook", "flyhigh-engine", diff --git a/Cargo.toml b/Cargo.toml index a69dbb5..6ca6ccc 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -5,7 +5,7 @@ members = ["crates/flyhigh-options", "crates/flyhigh-engine", "crates/flyhigh-ca default-members = ["crates/flyhigh-options", "crates/flyhigh-engine", "crates/flyhigh-capi", "crates/flyhigh-cli", "examples/player"] [workspace.package] -version = "0.3.6" +version = "0.3.7" edition = "2021" license = "Apache-2.0" repository = "https://github.com/getflyhigh/flyhigh-engine" diff --git a/crates/flyhigh-engine/src/stream/classify.rs b/crates/flyhigh-engine/src/stream/classify.rs index 37b6eca..6333708 100644 --- a/crates/flyhigh-engine/src/stream/classify.rs +++ b/crates/flyhigh-engine/src/stream/classify.rs @@ -39,6 +39,8 @@ pub struct Thresholds { /// A pair whose mean difference is below this and whose largest cell is below `hold_max` is a held frame. pub hold_mean: Option, pub hold_max: f32, + /// The first change after a held frame is not interpolated (see `Classification::after_hold`). + pub step_after_hold: bool, } #[derive(Clone, Copy, Debug, PartialEq, Serialize, Deserialize)] @@ -69,6 +71,19 @@ impl Classification { Classification { kind, mean, max, black: max == 0.0, motion: mean as f64 * CELL_MAX as f64 } } + /// The same pair once it is known whether the pair before it was a held frame. A change that follows a held frame + /// is animation drawn on twos or threes (a mouth, a blink, a character against a still background), not motion: + /// its in-betweens morphed one drawing into the next within one frame time and then stood still for two, four + /// generated frames in 41 ms and none for 83, which read as a smeared mouth and as a rate jumping between 24 and + /// 120. Such a pair is shown as it is drawn (a cut for the scheduler: no in-betweens, a new frame). Motion that + /// goes on from frame to frame is interpolated as ever, from its second pair. + pub fn after_hold(mut self, held_before: bool, th: &Thresholds) -> Classification { + if th.step_after_hold && held_before && self.kind == IntervalKind::Motion { + self.kind = IntervalKind::Cut; + } + self + } + /// The same pair once the mean difference of the moving pair before it is known (`before`; None at the start): /// a cut after a moving pair is the middle of a fast camera move (see `CUT_AFTER_CALM`). pub fn after(mut self, before: Option, th: &Thresholds) -> Classification { @@ -90,6 +105,7 @@ pub struct Classifier { thresholds: Thresholds, /// The mean difference of the last pair that was not a hold, as its readback landed. last_moving: Rc>>, + last_held: Rc>, } impl Classifier { @@ -131,6 +147,7 @@ impl Classifier { bind_groups: HashMap::new(), thresholds, last_moving: Rc::new(Cell::new(None)), + last_held: Rc::new(Cell::new(false)), } } @@ -177,14 +194,18 @@ impl Classifier { let th = self.thresholds; let cells = GRID.0 * GRID.1; let last_moving = self.last_moving.clone(); + let last_held = self.last_held.clone(); Some(Readback::new( stage, Box::new(move |d: &[u8]| { let s: &[u32] = bytemuck::cast_slice(&d[..8]); let c = Classification::from_stats(s[0], s[1], cells, &th).after(last_moving.get(), &th); - if c.kind != IntervalKind::Hold { + let held = c.kind == IntervalKind::Hold; + if !held { last_moving.set(Some(c.mean)); } + let c = c.after_hold(last_held.get(), &th); + last_held.set(held); c }), Some(Box::new(move || busy.set(false))), @@ -198,6 +219,7 @@ impl Classifier { /// A seek or a new source: the pair before the next one is no longer the one this saw. pub fn forget_history(&self) { + self.last_held.set(false); self.last_moving.set(None); } } diff --git a/crates/flyhigh-engine/src/stream/mod.rs b/crates/flyhigh-engine/src/stream/mod.rs index 18e8a2e..0587ef9 100644 --- a/crates/flyhigh-engine/src/stream/mod.rs +++ b/crates/flyhigh-engine/src/stream/mod.rs @@ -91,8 +91,9 @@ pub struct StreamOptions { #[option(values = "off, or a threshold 0 to 1", c = "scene_cut (threshold, 0 = off)")] pub scene_cut: Option, /// A pair whose mean difference is below `mean` and whose most different cell is below `max` is a held frame: shown - /// as is and counted for the unique-frame rate, instead of a still in-between. On for anime; the defaults 0.0033 and - /// 0.06 stay above codec noise on a re-encoded hold. + /// as is and counted for the unique-frame rate, instead of a still in-between. The first change after a held frame is + /// then shown as drawn too (animation on twos and threes is not motion), unless cadence re-timing spreads it. On for + /// anime; the defaults 0.0033 and 0.06 stay above codec noise on a re-encoded hold. #[option(values = "off, or mean and max thresholds 0 to 1", c = "held_frames (0, 1), held_mean, held_max")] pub held_frames: Option, /// Input slots created up front (more with `add_input`); the Pacer's history says how many a host needs. @@ -233,6 +234,8 @@ impl Stream { cut_mean: opts.scene_cut.map(|c| c.threshold), hold_mean: opts.held_frames.map(|h| h.mean), hold_max: opts.held_frames.map_or(0.0, |h| h.max), + // cadence re-timing spreads such a change over the frames it was held for, and needs it as motion + step_after_hold: opts.held_frames.is_some() && !opts.cadence, }, ) }); diff --git a/crates/flyhigh-engine/tests/pacer.rs b/crates/flyhigh-engine/tests/pacer.rs index 04e2ed5..4fc7838 100644 --- a/crates/flyhigh-engine/tests/pacer.rs +++ b/crates/flyhigh-engine/tests/pacer.rs @@ -414,6 +414,23 @@ fn auto_takes_the_exact_grid_where_no_factor_is_even() { assert!(s.len() as f64 > 8.9 * 74.0 && off_pace(&s, 1000.0 / 75.0) < 0.01, "60 fps on 75 Hz: {}", s.len()); let s = shown(&auto, 1000.0 / 60.0, ®ular(50.0, 12.0), 1.0); assert!(s.len() as f64 > 8.9 * 59.0 && off_pace(&s, 1000.0 / 60.0) < 0.01, "50 fps on 60 Hz: {}", s.len()); + // the common gaming displays: 23.976 fps on 180, 144 and 165 Hz, 60 fps on 144 and 165 Hz + for (fps, hz) in + [(24000.0 / 1001.0, 180.0), (24000.0 / 1001.0, 144.0), (24000.0 / 1001.0, 165.0), (60.0, 144.0), (60.0, 165.0)] + { + let s = shown(&auto, 1000.0 / hz, ®ular(fps, 12.0), 1.0); + let off = off_pace(&s, 1000.0 / hz); + eprintln!( + "{fps:.3} fps on {hz} Hz: {:.1} pictures a second, {:.2} % off pace", + s.len() as f64 / 9.0, + off * 100.0 + ); + assert!( + s.len() as f64 > 9.0 * 2.0 * fps * 0.97 && off < 0.02, + "{fps} fps on {hz} Hz: {off:.3} off pace, {}", + s.len() + ); + } // a display at the source's own rate has nothing to add let s = shown(&auto, 1000.0 / 60.0, ®ular(60.0, 12.0), 1.0); assert!(s.iter().all(|x| x.2 == 1)); diff --git a/crates/flyhigh-engine/tests/stream.rs b/crates/flyhigh-engine/tests/stream.rs index e7f9083..6e230b7 100644 --- a/crates/flyhigh-engine/tests/stream.rs +++ b/crates/flyhigh-engine/tests/stream.rs @@ -12,7 +12,7 @@ const H: u32 = 192; #[test] fn classification_thresholds_are_pure() { - let th = Thresholds { cut_mean: Some(0.08), hold_mean: Some(0.0033), hold_max: 0.06 }; + let th = Thresholds { cut_mean: Some(0.08), hold_mean: Some(0.0033), hold_max: 0.06, step_after_hold: true }; let cells = GRID.0 * GRID.1; let identical = Classification::from_stats(0, 0, cells, &th); assert_eq!((identical.kind, identical.black), (IntervalKind::Hold, true)); @@ -23,13 +23,18 @@ fn classification_thresholds_are_pure() { assert!((motion.motion - 20.0).abs() < 0.01, "the Pacer's motion units are the mean cell difference"); let noise = Classification::from_stats(cells, 30, cells, &th); assert_eq!(noise.kind, IntervalKind::Hold, "codec noise on a held frame stays a hold"); - let off = Thresholds { cut_mean: None, hold_mean: None, hold_max: 0.0 }; + let off = Thresholds { cut_mean: None, hold_mean: None, hold_max: 0.0, step_after_hold: false }; assert_eq!(Classification::from_stats(cells * 765, 765, cells, &off).kind, IntervalKind::Motion); // a cut is a lone spike after a calm pair; after a moving pair it is the middle of a fast camera move assert_eq!(cut.after(Some(0.02), &th).kind, IntervalKind::Cut); assert_eq!(cut.after(None, &th).kind, IntervalKind::Cut); assert_eq!(cut.after(Some(0.07), &th).kind, IntervalKind::Motion); assert_eq!(motion.after(Some(0.2), &th).kind, IntervalKind::Motion); + // a change after a held frame is animation on twos or threes: shown as drawn, not interpolated + assert_eq!(motion.after_hold(true, &th).kind, IntervalKind::Cut); + assert_eq!(motion.after_hold(false, &th).kind, IntervalKind::Motion); + assert_eq!(noise.after_hold(true, &th).kind, IntervalKind::Hold); + assert_eq!(motion.after_hold(true, &off).kind, IntervalKind::Motion); } fn engine() -> Option { diff --git a/docs/options.md b/docs/options.md index f169fef..5abab41 100644 --- a/docs/options.md +++ b/docs/options.md @@ -51,7 +51,7 @@ How source frames with timestamps become generatable intervals (ARCHITECTURE.md) | `mode` | realtime | realtime, video | Realtime: no lookahead, the in-betweens of the previous and the newest frame; for games and capture. Video: one source frame of lookahead (one frame more latency), the interval that can be generated starts two frames back; cadence re-timing needs it. | `mode (0, 1)` | | `cadence` | off | on, off | Video mode only. Anime is often drawn on twos (each drawing held for two frames) over a camera pan on every frame; with cadence on, held elements are re-timed onto a smooth path. Presented frames are then generated in-betweens, the real drawings included, except where the model could not match the frames (there the real drawing stays): big drawing steps show the generator's softness more often. A choice for the viewer, not a default. Needs repair at least 1 (0 is raised to 1). | `cadence (0, 1)` | | `scene_cut` | threshold 0.08 | off, or a threshold 0 to 1 | A pair whose mean per-channel difference (0 to 1, measured on a 48 x 27 grid) is above the threshold is a scene cut: the interval is not interpolated, the source frames are shown. Off = every pair is motion. | `scene_cut (threshold, 0 = off)` | -| `held_frames` | off | off, or mean and max thresholds 0 to 1 | A pair whose mean difference is below `mean` and whose most different cell is below `max` is a held frame: shown as is and counted for the unique-frame rate, instead of a still in-between. On for anime; the defaults 0.0033 and 0.06 stay above codec noise on a re-encoded hold. | `held_frames (0, 1), held_mean, held_max` | +| `held_frames` | off | off, or mean and max thresholds 0 to 1 | A pair whose mean difference is below `mean` and whose most different cell is below `max` is a held frame: shown as is and counted for the unique-frame rate, instead of a still in-between. The first change after a held frame is then shown as drawn too (animation on twos and threes is not motion), unless cadence re-timing spreads it. On for anime; the defaults 0.0033 and 0.06 stay above codec noise on a re-encoded hold. | `held_frames (0, 1), held_mean, held_max` | | `inputs` | 4 | | Input slots created up front (more with `add_input`); the Pacer's history says how many a host needs. | `inputs` | | `outputs` | 4 | | Output slots created up front (more with `add_output`). | `outputs` |