fix(topic): reject non-finite expectation sufficient statistics

CHANGELOG.d/topic-expectation-finite-statistics.md

@@ -0,0 +1,25 @@
+# Refuse non-finite topic expectation sufficient statistics
+
+- Validate each accumulated document-topic and topic-term count immediately after its write. A finite log-likelihood does not establish that the sufficient statistics are finite.
+- Return the existing `NonFiniteEstimate` error on overflow without changing the responsibility formula, model target, smoothing, or convergence tolerance.
+- Add private crate regressions for document and topic accumulation overflow and an ordinary finite positive control. These exercise injected private states, not an observed public-fit success or a production incident.
+- This correction reinforces the existing ADR0012 finite-intermediate contract. It does not implement or attest GPU execution, CPU/GPU parity, parameter recovery, release readiness, or protected-main integration.
+
+- Reject positive topic smoothing that underflows to a zero probability after beta normalization. Preserve the supplied floor and formulas; return `NonFiniteEstimate` rather than emitting a degenerate fit or treating an explicit zero sparse entry differently from its omission. Public CSR/CSC regressions include a representable tiny-floor control.
+
+Reject observed topic transitions when the mapped document event time moves backward, even when the independently supplied relation intervals are forward-validated. Preserve equal-time pairs and document-row permutation.
+
+- Reject CSR/CSC outer dimensions whose required offset count is not representable as `usize`, using `InvalidSparseMatrix` rather than overflowing during validation. Preserve tiny canonical storage with maximum inner dimensions. This closes an inherited checked-arithmetic constructor boundary; it does not establish a release-mode panic or change estimator arithmetic.
+
+
+- Form each posterior responsibility before multiplying by its observed count. This arithmetic regrouping preserves representable tiny expected counts that a count-first intermediate product can round to zero. Public CSR/CSC fit-based regressions check aggregate rare-term count conservation against a normal-count control; this is CPU `f64` acceptance, not GPU parity or parameter-recovery evidence.
+
+- When the mixture is normal but an individual topic mass underflows, divide its smaller simplex factor by that normal mixture before the final product, ordering the observed-count scaling around one. When the mixture itself is subnormal or rounds to zero, decompose all positive factors into binary mantissas/exponents and use one common scaled sum for both responsibilities and log-likelihood. Scale the observed count before final binary rounding so representable expected counts survive. Regressions retain the large-count/tiny-mass and public CSR/CSC cases and add subnormal-mixture posterior/count conservation, consistent likelihood, zero/small/large-count controls, and individually rounded-zero products. The new subnormal cases are injected private normalized states, not an observed public-fit trajectory or GPU acceptance.
+
+
+- Use the same mixture normalization and log-likelihood helper in expectation and the public in-sample likelihood consumed by statistical K selection. Actual public likelihood regressions compare the same injected subnormal/rounded-zero model states; they do not claim a public fit reached those states.
+
+- Preserve exact-zero topic factors as zero mass when another topic supplies a positive subnormal mixture. Keep the positive-factor decomposition strict, reject nonfinite or negative partners before skipping a zero product, and retain all-zero-mixture refusal. Private expectation/count and public likelihood/statistical-candidate regressions distinguish injected model states from the unchanged positive fitted-model path.
+
+
+- Reject nonfinite ordinary mixture sums before selecting binary-scaled normalization. Product overflow and summation overflow must retain the existing `NonFiniteEstimate` refusal instead of becoming finite log scores; finite zero sums caused by positive-product underflow still use scaled recovery. Private expectation and public likelihood/statistical-candidate regressions cover overflowing injected model states alongside ordinary, subnormal and rounded-zero positive controls. This does not assert that a valid normalized fit produces overflowing probabilities.

crates/model_selection/src/fitted.rs

@@ -158,12 +158,19 @@ impl FittedCandidateKConfig {
 ///
 /// Returns [`ModelSelectionError::NonPositiveCandidateK`] when `candidate_k`
 /// is less than two, or [`ModelSelectionError::InvalidDiagnostic`] when the
-/// fitted dimensions, likelihood, or parameter count are unusable.
+/// fitted dimensions, likelihood, or parameter count are unusable, including
+/// a candidate topic count that disagrees with the supplied fitted model.
 pub fn statistical_candidate_from_fit(
     input: &ReferenceTopicInput,
     candidate_k: u32,
     model: &ReferenceTopicModel,
 ) -> Result<ModelCandidate, ModelSelectionError> {
+    if candidate_k < 2 {
+        return Err(ModelSelectionError::NonPositiveCandidateK);
+    }
+    if usize::try_from(candidate_k).ok() != Some(model.topic_term_probabilities.len()) {
+        return Err(ModelSelectionError::InvalidDiagnostic);
+    }
     let log_likelihood = input
         .in_sample_log_likelihood(model)
         .map_err(|_| ModelSelectionError::InvalidDiagnostic)?;

crates/model_selection/tests/fitted_candidate_k_contract.rs

@@ -419,3 +419,41 @@ fn statistical_candidate_from_fit_refuses_unusable_diagnostics() {
         Err(ModelSelectionError::InvalidDiagnostic)
     );
 }
+
+/// Statistical K scoring reuses the fitting likelihood on injected subnormal model states.
+#[test]
+fn statistical_candidate_preserves_subnormal_mixture_likelihood() {
+    let input = two_document_input([1.0, 0.0, 0.0, 1.0]);
+    let q = f64::from_bits(1);
+    for (theta, beta, expected_likelihood) in [
+        (
+            vec![vec![0.625, 0.375], vec![0.5, 0.5]],
+            vec![vec![2.0 * q, 1.0], vec![q, 1.0]],
+            1.625_f64.ln() - 1074.0 * std::f64::consts::LN_2,
+        ),
+        (vec![vec![0.5, 0.5]; 2], vec![vec![q, 1.0]; 2], q.ln()),
+    ] {
+        // Only the result DTO is supplied: this is not a public fitting trajectory.
+        let model = ReferenceTopicModel {
+            seed: 7,
+            iterations: 4,
+            objective: 0.0,
+            topic_term_probabilities: beta,
+            document_topic_proportions: theta,
+            document_coordinate_variances: vec![vec![1.0]; 2],
+            prevalence_coefficients: vec![vec![0.0]],
+            prevalence_features: Vec::new(),
+            sequence_edges: Vec::new(),
+            connected_post_count: 0,
+            lineage_count: 0,
+        };
+        let candidate = statistical_candidate_from_fit(&input, 2, &model)
+            .expect("positive mixture admits statistical candidate");
+        let score = candidate
+            .held_out_log_likelihood()
+            .expect("statistical score");
+        let expected_score = expected_likelihood - 0.5 * 5.0 * 2.0_f64.ln();
+        assert!((score - expected_score).abs() < 1e-12);
+        assert!(candidate.is_statistically_supported());
+    }
+}

crates/model_selection/tests/fitted_k_identity_contract.rs

@@ -0,0 +1,274 @@
+//! Fitted candidate-`K` scoring uses real TRSL-TM reference fits.
+
+use corpus_split::{CorpusDocument, CorpusSnapshot};
+use membership_core::{
+    GroupId, MemberId, MembershipAssignment, MembershipNetwork, MembershipRole, MembershipWeight,
+};
+use model_selection::{ModelSelectionError, statistical_candidate_from_fit};
+use relation_graph::{
+    RelationEdge, RelationEndpointId, RelationEvidenceStatus, RelationGraph, RelationKind,
+};
+use temporal_core::{
+    AvailableTime, EventTime, KnowledgeCutoff, TemporalBoundary, TemporalInterval,
+    TemporalPrecision,
+};
+use topic_measurement::{
+    ReferenceTopicInput, ReferenceTopicModelConfig, SparseMatrix, fit_reference_topic_model,
+};
+use uuid::Uuid;
+
+/// Parse a deterministic event-time fixture for a January day.
+fn event_time(day: u8) -> EventTime {
+    EventTime::parse_rfc3339(&format!("2026-01-{day:02}T00:00:00Z")).expect("event time")
+}
+
+/// Construct an observed forward relation with bounded same-day intervals.
+fn relation(source: Uuid, target: Uuid, source_day: u8, target_day: u8) -> RelationEdge {
+    let interval = |day| {
+        TemporalInterval::bounded(
+            TemporalBoundary::Included(event_time(day)),
+            TemporalBoundary::Included(
+                EventTime::parse_rfc3339(&format!("2026-01-{day:02}T12:00:00Z"))
+                    .expect("interval end"),
+            ),
+            TemporalPrecision::Second,
+        )
+        .expect("bounded interval")
+    };
+    RelationEdge::new(
+        RelationKind::TransitionsTo,
+        RelationEndpointId::from_uuid(source),
+        RelationEndpointId::from_uuid(target),
+        RelationEvidenceStatus::Observed,
+        interval(source_day),
+        interval(target_day),
+    )
+    .expect("forward relation")
+}
+
+/// Build a cutoff-eligible two-document corpus with shared membership and a transition.
+fn two_document_input(term_values: [f64; 4]) -> ReferenceTopicInput {
+    let document_ids = vec![Uuid::from_u128(1), Uuid::from_u128(2)];
+    let times = vec![event_time(1), event_time(2)];
+    let available = AvailableTime::parse_rfc3339("2026-01-10T00:00:00Z").expect("available");
+    let cutoff = KnowledgeCutoff::parse_rfc3339("2026-02-01T00:00:00Z").expect("cutoff");
+    let mut snapshot = CorpusSnapshot::new();
+    for id in &document_ids {
+        snapshot
+            .insert_if_eligible(CorpusDocument::new(*id, available), &cutoff)
+            .expect("eligible");
+    }
+    let mut memberships = MembershipNetwork::new();
+    memberships
+        .insert(
+            MembershipAssignment::new(
+                MemberId::from_uuid(document_ids[0]),
+                GroupId::from_uuid(Uuid::from_u128(100)),
+                MembershipRole::Organization,
+                MembershipWeight::full().expect("full"),
+                event_time(1),
+                event_time(9),
+            )
+            .expect("membership"),
+        )
+        .expect("insert");
+    memberships
+        .insert(
+            MembershipAssignment::new(
+                MemberId::from_uuid(document_ids[1]),
+                GroupId::from_uuid(Uuid::from_u128(100)),
+                MembershipRole::Organization,
+                MembershipWeight::full().expect("full"),
+                event_time(1),
+                event_time(9),
+            )
+            .expect("membership"),
+        )
+        .expect("insert");
+    let mut relations = RelationGraph::new();
+    relations
+        .insert(relation(document_ids[0], document_ids[1], 1, 2))
+        .expect("relation");
+    let counts =
+        SparseMatrix::from_csr(2, 2, vec![0, 2, 4], vec![0, 1, 0, 1], term_values.to_vec())
+            .expect("counts");
+    ReferenceTopicInput::new(
+        &snapshot,
+        document_ids,
+        &counts,
+        &times,
+        None,
+        &memberships,
+        &relations,
+    )
+    .expect("two-document input")
+}
+
+/// Matching candidate identity uses a real public fit, not an injected model DTO.
+#[test]
+fn candidate_k_matches_real_fit() {
+    let input = two_document_input([8.0, 0.0, 0.0, 4.0]);
+    let config = ReferenceTopicModelConfig::new(2, vec![7], 2000, 1e-5).expect("config");
+    let model = fit_reference_topic_model(&input, &config).expect("actual CPU fit");
+    assert_eq!(model.topic_term_probabilities.len(), 2);
+    let candidate = statistical_candidate_from_fit(&input, 2, &model).expect("matching K");
+    assert_eq!(candidate.candidate_k(), 2);
+    assert!(candidate.is_statistically_supported());
+}
+/// A real two-topic fit must not supply evidence for any other candidate K.
+#[test]
+fn wrong_candidate_k_is_rejected() {
+    let input = two_document_input([8.0, 0.0, 0.0, 4.0]);
+    let config = ReferenceTopicModelConfig::new(2, vec![7], 2000, 1e-5).expect("config");
+    let model = fit_reference_topic_model(&input, &config).expect("actual CPU fit");
+    assert_eq!(model.topic_term_probabilities.len(), 2);
+    for k in [0, 1, 3, 4, u32::MAX] {
+        let actual = statistical_candidate_from_fit(&input, k, &model);
+        let expected = if k < 2 {
+            ModelSelectionError::NonPositiveCandidateK
+        } else {
+            ModelSelectionError::InvalidDiagnostic
+        };
+        assert_eq!(
+            actual,
+            Err(expected),
+            "candidate K must bind to the real fitted dimension"
+        );
+    }
+}
+
+/// Public likelihood and candidate scoring admit exact zeros beside a positive tiny mixture.
+#[test]
+fn zero_factor_subnormal_public_likelihood_and_candidate() {
+    let input = two_document_input([1.0, 0.0, 0.0, 1.0]);
+    let config = ReferenceTopicModelConfig::new(2, vec![7], 2000, 1e-5).expect("config");
+    let fitted = fit_reference_topic_model(&input, &config).expect("actual CPU fit");
+    for tiny in [0.25, f64::from_bits(1_u64 << 44)] {
+        for (theta, beta) in [
+            (
+                vec![1.0, 0.0],
+                vec![vec![tiny, 1.0 - tiny], vec![tiny, 1.0 - tiny]],
+            ),
+            (vec![0.5, 0.5], vec![vec![tiny, 1.0 - tiny], vec![0.0, 1.0]]),
+            (
+                vec![0.0, 1.0],
+                vec![vec![tiny, 1.0 - tiny], vec![tiny, 1.0 - tiny]],
+            ),
+            (
+                vec![0.5, 0.5],
+                vec![vec![-0.0, 1.0], vec![tiny, 1.0 - tiny]],
+            ),
+        ] {
+            let mut model = fitted.clone();
+            model.document_topic_proportions = vec![theta, vec![0.5, 0.5]];
+            model.topic_term_probabilities = beta;
+            let expected = model
+                .document_topic_proportions
+                .iter()
+                .enumerate()
+                .map(|(row, weights)| {
+                    let probability = weights
+                        .iter()
+                        .enumerate()
+                        .map(|(topic, weight)| weight * model.topic_term_probabilities[topic][row])
+                        .sum::<f64>();
+                    assert!(probability.is_finite() && probability > 0.0);
+                    probability.ln()
+                })
+                .sum::<f64>();
+            let actual = input
+                .in_sample_log_likelihood(&model)
+                .expect("finite positive mixtures");
+            assert!((actual - expected).abs() <= 1e-12);
+            let candidate = statistical_candidate_from_fit(&input, 2, &model)
+                .expect("same public likelihood consumer");
+            assert!(candidate.is_statistically_supported());
+        }
+    }
+    let mut invalid = fitted;
+    invalid.document_topic_proportions = vec![vec![1.0, 0.0]; 2];
+    invalid.topic_term_probabilities = vec![vec![0.0, 1.0], vec![1.0, 0.0]];
+    assert!(input.in_sample_log_likelihood(&invalid).is_err());
+    assert!(statistical_candidate_from_fit(&input, 2, &invalid).is_err());
+}
+
+/// Public likelihood rejects an overflowing raw mixture instead of rescuing its logarithm.
+#[test]
+fn nonfinite_mixture_public_likelihood_and_candidate_are_rejected() {
+    let input = two_document_input([1.0, 0.0, 0.0, 1.0]);
+    let config = ReferenceTopicModelConfig::new(2, vec![7], 2000, 1e-5).expect("config");
+    let fitted = fit_reference_topic_model(&input, &config).expect("actual CPU fit");
+    for (theta, beta) in [
+        (
+            vec![f64::MAX, 0.0],
+            vec![vec![f64::MAX, 1.0], vec![0.0, 1.0]],
+        ),
+        (
+            vec![f64::MAX, f64::MAX],
+            vec![vec![1.0, 1.0], vec![1.0, 1.0]],
+        ),
+    ] {
+        let raw = theta
+            .iter()
+            .enumerate()
+            .map(|(topic, weight)| weight * beta[topic][0])
+            .sum::<f64>();
+        assert!(raw.is_infinite() && raw.is_sign_positive());
+        let mut model = fitted.clone();
+        model.document_topic_proportions = vec![theta, vec![0.5, 0.5]];
+        model.topic_term_probabilities = beta;
+        assert_eq!(
+            input.in_sample_log_likelihood(&model),
+            Err(topic_measurement::TopicMeasurementError::NonFiniteEstimate)
+        );
+        assert!(statistical_candidate_from_fit(&input, 2, &model).is_err());
+    }
+}
+
+/// Finite ordinary, subnormal and rounded-zero positive mixtures retain shared scoring.
+#[test]
+fn nonfinite_mixture_guard_preserves_finite_and_underflow_controls() {
+    let input = two_document_input([1.0, 0.0, 0.0, 1.0]);
+    let config = ReferenceTopicModelConfig::new(2, vec![7], 2000, 1e-5).expect("config");
+    let fitted = fit_reference_topic_model(&input, &config).expect("actual CPU fit");
+    let tiny = 2.0_f64.powi(-600);
+    let subnormal = f64::from_bits(1_u64 << 44);
+    for (theta, beta, first_log, raw_is_zero) in [
+        (
+            vec![0.5, 0.5],
+            vec![vec![0.5, 0.5], vec![0.5, 0.5]],
+            0.5_f64.ln(),
+            false,
+        ),
+        (
+            vec![1.0, 0.0],
+            vec![vec![subnormal, 1.0], vec![0.0, 1.0]],
+            subnormal.ln(),
+            false,
+        ),
+        (
+            vec![tiny, 1.0],
+            vec![vec![tiny, 1.0], vec![0.0, 1.0]],
+            -1200.0 * std::f64::consts::LN_2,
+            true,
+        ),
+    ] {
+        let raw = theta
+            .iter()
+            .enumerate()
+            .map(|(topic, weight)| weight * beta[topic][0])
+            .sum::<f64>();
+        assert!(raw.is_finite());
+        assert_eq!(raw.to_bits() == 0.0_f64.to_bits(), raw_is_zero);
+        let mut model = fitted.clone();
+        model.document_topic_proportions = vec![theta, vec![0.5, 0.5]];
+        model.topic_term_probabilities = beta;
+        let second =
+            0.5 * model.topic_term_probabilities[0][1] + 0.5 * model.topic_term_probabilities[1][1];
+        let actual = input
+            .in_sample_log_likelihood(&model)
+            .expect("finite or recovered tiny mixture");
+        assert!((actual - (first_log + second.ln())).abs() <= 1e-12);
+        assert!(statistical_candidate_from_fit(&input, 2, &model).is_ok());
+    }
+}