Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,9 +125,11 @@ look like.
- **Classification.** The two frames are sampled on a 48 x 27 grid (`classify.wgsl`); a mean per-channel difference
above `scene_cut.threshold` is a cut when the moving pair before it differed by less than half the threshold (a fast
camera move is a run of large differences, a cut a lone spike after a calm pair), a mean below `held_frames.mean` with
every cell below `held_frames.max` is a hold, anything else motion. The readback is asynchronous (three may be in
flight; beyond that the pair counts as motion) and lands before the interval's first mid is due. The mean cell
difference in 0..765 units is the motion statistic the Pacer's motion-adaptive ceiling uses.
every cell below `held_frames.max` is a hold, anything else motion. With held frames on and no cadence re-timing, a
moving pair right after a held one is reported as a cut: a change after a held frame is animation on twos or threes,
shown as drawn. The readback is asynchronous (three may be in flight; beyond that the pair counts as motion) and lands
before the interval's first mid is due. The mean cell difference in 0..765 units is the motion statistic the Pacer's
motion-adaptive ceiling uses.
- **Order with the Pacer.** Classify the new frame against the previous one, then `pacer.source_frame` with the kind,
generate the requests it returns, then `push`: the pair must not change before the previous interval's mids are out.
- **Held frames are reported, never dropped silently**: the Pacer shows the frame and generates nothing for the
Expand Down
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,17 @@ follow semantic versioning as described in [CONTRIBUTING.md](CONTRIBUTING.md).

## [Unreleased]

## [0.3.7] - 2026-11-10

### Changed

- With held frames detected (`held_frames`) and no cadence re-timing, the first change after a held frame is not
interpolated. It is animation drawn on twos or threes (a mouth, a blink, a character against a still background),
not motion: its in-betweens morphed one drawing into the next within one frame time and then stood still for two,
four generated frames in 41 ms and none for 83, a smeared mouth and a rate jumping between 24 and 120. Such a pair
is shown as drawn (the classifier reports a cut: no in-betweens, a new frame); motion that goes on from frame to
frame is interpolated from its second pair. Cadence re-timing (video mode) still takes these changes as motion.

## [0.3.6] - 2026-11-10

### Changed
Expand Down
12 changes: 6 additions & 6 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ members = ["crates/flyhigh-options", "crates/flyhigh-engine", "crates/flyhigh-ca
default-members = ["crates/flyhigh-options", "crates/flyhigh-engine", "crates/flyhigh-capi", "crates/flyhigh-cli", "examples/player"]

[workspace.package]
version = "0.3.6"
version = "0.3.7"
edition = "2021"
license = "Apache-2.0"
repository = "https://github.com/getflyhigh/flyhigh-engine"
Expand Down
24 changes: 23 additions & 1 deletion crates/flyhigh-engine/src/stream/classify.rs
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@ pub struct Thresholds {
/// A pair whose mean difference is below this and whose largest cell is below `hold_max` is a held frame.
pub hold_mean: Option<f32>,
pub hold_max: f32,
/// The first change after a held frame is not interpolated (see `Classification::after_hold`).
pub step_after_hold: bool,
}

#[derive(Clone, Copy, Debug, PartialEq, Serialize, Deserialize)]
Expand Down Expand Up @@ -69,6 +71,19 @@ impl Classification {
Classification { kind, mean, max, black: max == 0.0, motion: mean as f64 * CELL_MAX as f64 }
}

/// The same pair once it is known whether the pair before it was a held frame. A change that follows a held frame
/// is animation drawn on twos or threes (a mouth, a blink, a character against a still background), not motion:
/// its in-betweens morphed one drawing into the next within one frame time and then stood still for two, four
/// generated frames in 41 ms and none for 83, which read as a smeared mouth and as a rate jumping between 24 and
/// 120. Such a pair is shown as it is drawn (a cut for the scheduler: no in-betweens, a new frame). Motion that
/// goes on from frame to frame is interpolated as ever, from its second pair.
pub fn after_hold(mut self, held_before: bool, th: &Thresholds) -> Classification {
if th.step_after_hold && held_before && self.kind == IntervalKind::Motion {
self.kind = IntervalKind::Cut;
}
self
}

/// The same pair once the mean difference of the moving pair before it is known (`before`; None at the start):
/// a cut after a moving pair is the middle of a fast camera move (see `CUT_AFTER_CALM`).
pub fn after(mut self, before: Option<f32>, th: &Thresholds) -> Classification {
Expand All @@ -90,6 +105,7 @@ pub struct Classifier {
thresholds: Thresholds,
/// The mean difference of the last pair that was not a hold, as its readback landed.
last_moving: Rc<Cell<Option<f32>>>,
last_held: Rc<Cell<bool>>,
}

impl Classifier {
Expand Down Expand Up @@ -131,6 +147,7 @@ impl Classifier {
bind_groups: HashMap::new(),
thresholds,
last_moving: Rc::new(Cell::new(None)),
last_held: Rc::new(Cell::new(false)),
}
}

Expand Down Expand Up @@ -177,14 +194,18 @@ impl Classifier {
let th = self.thresholds;
let cells = GRID.0 * GRID.1;
let last_moving = self.last_moving.clone();
let last_held = self.last_held.clone();
Some(Readback::new(
stage,
Box::new(move |d: &[u8]| {
let s: &[u32] = bytemuck::cast_slice(&d[..8]);
let c = Classification::from_stats(s[0], s[1], cells, &th).after(last_moving.get(), &th);
if c.kind != IntervalKind::Hold {
let held = c.kind == IntervalKind::Hold;
if !held {
last_moving.set(Some(c.mean));
}
let c = c.after_hold(last_held.get(), &th);
last_held.set(held);
c
}),
Some(Box::new(move || busy.set(false))),
Expand All @@ -198,6 +219,7 @@ impl Classifier {

/// A seek or a new source: the pair before the next one is no longer the one this saw.
pub fn forget_history(&self) {
self.last_held.set(false);
self.last_moving.set(None);
}
}
7 changes: 5 additions & 2 deletions crates/flyhigh-engine/src/stream/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -91,8 +91,9 @@ pub struct StreamOptions {
#[option(values = "off, or a threshold 0 to 1", c = "scene_cut (threshold, 0 = off)")]
pub scene_cut: Option<SceneCut>,
/// A pair whose mean difference is below `mean` and whose most different cell is below `max` is a held frame: shown
/// as is and counted for the unique-frame rate, instead of a still in-between. On for anime; the defaults 0.0033 and
/// 0.06 stay above codec noise on a re-encoded hold.
/// as is and counted for the unique-frame rate, instead of a still in-between. The first change after a held frame is
/// then shown as drawn too (animation on twos and threes is not motion), unless cadence re-timing spreads it. On for
/// anime; the defaults 0.0033 and 0.06 stay above codec noise on a re-encoded hold.
#[option(values = "off, or mean and max thresholds 0 to 1", c = "held_frames (0, 1), held_mean, held_max")]
pub held_frames: Option<HeldFrames>,
/// Input slots created up front (more with `add_input`); the Pacer's history says how many a host needs.
Expand Down Expand Up @@ -233,6 +234,8 @@ impl Stream {
cut_mean: opts.scene_cut.map(|c| c.threshold),
hold_mean: opts.held_frames.map(|h| h.mean),
hold_max: opts.held_frames.map_or(0.0, |h| h.max),
// cadence re-timing spreads such a change over the frames it was held for, and needs it as motion
step_after_hold: opts.held_frames.is_some() && !opts.cadence,
},
)
});
Expand Down
17 changes: 17 additions & 0 deletions crates/flyhigh-engine/tests/pacer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -414,6 +414,23 @@ fn auto_takes_the_exact_grid_where_no_factor_is_even() {
assert!(s.len() as f64 > 8.9 * 74.0 && off_pace(&s, 1000.0 / 75.0) < 0.01, "60 fps on 75 Hz: {}", s.len());
let s = shown(&auto, 1000.0 / 60.0, &regular(50.0, 12.0), 1.0);
assert!(s.len() as f64 > 8.9 * 59.0 && off_pace(&s, 1000.0 / 60.0) < 0.01, "50 fps on 60 Hz: {}", s.len());
// the common gaming displays: 23.976 fps on 180, 144 and 165 Hz, 60 fps on 144 and 165 Hz
for (fps, hz) in
[(24000.0 / 1001.0, 180.0), (24000.0 / 1001.0, 144.0), (24000.0 / 1001.0, 165.0), (60.0, 144.0), (60.0, 165.0)]
{
let s = shown(&auto, 1000.0 / hz, &regular(fps, 12.0), 1.0);
let off = off_pace(&s, 1000.0 / hz);
eprintln!(
"{fps:.3} fps on {hz} Hz: {:.1} pictures a second, {:.2} % off pace",
s.len() as f64 / 9.0,
off * 100.0
);
assert!(
s.len() as f64 > 9.0 * 2.0 * fps * 0.97 && off < 0.02,
"{fps} fps on {hz} Hz: {off:.3} off pace, {}",
s.len()
);
}
// a display at the source's own rate has nothing to add
let s = shown(&auto, 1000.0 / 60.0, &regular(60.0, 12.0), 1.0);
assert!(s.iter().all(|x| x.2 == 1));
Expand Down
9 changes: 7 additions & 2 deletions crates/flyhigh-engine/tests/stream.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ const H: u32 = 192;

#[test]
fn classification_thresholds_are_pure() {
let th = Thresholds { cut_mean: Some(0.08), hold_mean: Some(0.0033), hold_max: 0.06 };
let th = Thresholds { cut_mean: Some(0.08), hold_mean: Some(0.0033), hold_max: 0.06, step_after_hold: true };
let cells = GRID.0 * GRID.1;
let identical = Classification::from_stats(0, 0, cells, &th);
assert_eq!((identical.kind, identical.black), (IntervalKind::Hold, true));
Expand All @@ -23,13 +23,18 @@ fn classification_thresholds_are_pure() {
assert!((motion.motion - 20.0).abs() < 0.01, "the Pacer's motion units are the mean cell difference");
let noise = Classification::from_stats(cells, 30, cells, &th);
assert_eq!(noise.kind, IntervalKind::Hold, "codec noise on a held frame stays a hold");
let off = Thresholds { cut_mean: None, hold_mean: None, hold_max: 0.0 };
let off = Thresholds { cut_mean: None, hold_mean: None, hold_max: 0.0, step_after_hold: false };
assert_eq!(Classification::from_stats(cells * 765, 765, cells, &off).kind, IntervalKind::Motion);
// a cut is a lone spike after a calm pair; after a moving pair it is the middle of a fast camera move
assert_eq!(cut.after(Some(0.02), &th).kind, IntervalKind::Cut);
assert_eq!(cut.after(None, &th).kind, IntervalKind::Cut);
assert_eq!(cut.after(Some(0.07), &th).kind, IntervalKind::Motion);
assert_eq!(motion.after(Some(0.2), &th).kind, IntervalKind::Motion);
// a change after a held frame is animation on twos or threes: shown as drawn, not interpolated
assert_eq!(motion.after_hold(true, &th).kind, IntervalKind::Cut);
assert_eq!(motion.after_hold(false, &th).kind, IntervalKind::Motion);
assert_eq!(noise.after_hold(true, &th).kind, IntervalKind::Hold);
assert_eq!(motion.after_hold(true, &off).kind, IntervalKind::Motion);
}

fn engine() -> Option<Engine> {
Expand Down
2 changes: 1 addition & 1 deletion docs/options.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ How source frames with timestamps become generatable intervals (ARCHITECTURE.md)
| `mode` | realtime | realtime, video | Realtime: no lookahead, the in-betweens of the previous and the newest frame; for games and capture. Video: one source frame of lookahead (one frame more latency), the interval that can be generated starts two frames back; cadence re-timing needs it. | `mode (0, 1)` |
| `cadence` | off | on, off | Video mode only. Anime is often drawn on twos (each drawing held for two frames) over a camera pan on every frame; with cadence on, held elements are re-timed onto a smooth path. Presented frames are then generated in-betweens, the real drawings included, except where the model could not match the frames (there the real drawing stays): big drawing steps show the generator's softness more often. A choice for the viewer, not a default. Needs repair at least 1 (0 is raised to 1). | `cadence (0, 1)` |
| `scene_cut` | threshold 0.08 | off, or a threshold 0 to 1 | A pair whose mean per-channel difference (0 to 1, measured on a 48 x 27 grid) is above the threshold is a scene cut: the interval is not interpolated, the source frames are shown. Off = every pair is motion. | `scene_cut (threshold, 0 = off)` |
| `held_frames` | off | off, or mean and max thresholds 0 to 1 | A pair whose mean difference is below `mean` and whose most different cell is below `max` is a held frame: shown as is and counted for the unique-frame rate, instead of a still in-between. On for anime; the defaults 0.0033 and 0.06 stay above codec noise on a re-encoded hold. | `held_frames (0, 1), held_mean, held_max` |
| `held_frames` | off | off, or mean and max thresholds 0 to 1 | A pair whose mean difference is below `mean` and whose most different cell is below `max` is a held frame: shown as is and counted for the unique-frame rate, instead of a still in-between. The first change after a held frame is then shown as drawn too (animation on twos and threes is not motion), unless cadence re-timing spreads it. On for anime; the defaults 0.0033 and 0.06 stay above codec noise on a re-encoded hold. | `held_frames (0, 1), held_mean, held_max` |
| `inputs` | 4 | | Input slots created up front (more with `add_input`); the Pacer's history says how many a host needs. | `inputs` |
| `outputs` | 4 | | Output slots created up front (more with `add_output`). | `outputs` |

Expand Down
Loading