Skip to content

Speed up ps_tail() and warn on a constant tail - #474

Closed
florence-bockting wants to merge 4 commits into
masterfrom
loo-helpers-efficiency
Closed

florence-bockting wants to merge 4 commits into
masterfrom
loo-helpers-efficiency

Conversation

@florence-bockting

Copy link
Copy Markdown
Collaborator

Summary

This PR prepares a loo PR. That loo PR removes the loo copies of the Pareto smoothing code and calls posterior::ps_tail() directly. loo then needs the constant-tail warning from posterior.

Changes:

  • ps_tail() warns if all tail values are the same. The new warning class is posterior_warning_pareto_constant_tail. It is added to the list of warning classes in ?posterior. The new argument warn_constant_tail = FALSE turns the warning off.
  • pareto_smooth() and pareto_khat() with tail = "both" warn only if k is NA. One constant tail with a good k on the other tail gives no warning.
  • ps_tail() checks only the first and last sorted tail values for a constant tail.
  • is_constant() uses min()/max() in place of range().
  • ps_tail_length() uses pmin() in place of ifelse(). The values are identical, also at ndraws * r_eff == 225.
  • ps_khat_threshold() has the new argument max_threshold = Inf. loo uses max_threshold = 0.7.

Planned loo changes (for background info)

loo functions to delete, with the posterior replacement:

  • psis_smooth_tail(): posterior::ps_tail(..., are_log_weights = TRUE)
  • exp_x_minus_exp_y(): no replacement needed. ps_tail() has its own copy
  • is_constant(): posterior::is_constant() using the relative tolerance
  • ps_khat_threshold(): posterior::ps_khat_threshold(S, max_threshold = 0.7)
  • n_pareto(): posterior::ps_tail_length()

loo function to change:

  • do_psis_i(): one posterior::ps_tail() call replaces the sort, the constant check and psis_smooth_tail(). It keeps the tail-length check, the shift back by the maximum, and the mapping of k = NA to Inf.

AI assistance

This PR used LLM assistance.

Copyright and Licensing

Copyright holder: Aalto University

By submitting this pull request, the copyright holder is agreeing to
license the submitted work under the following licenses:

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown

This is how benchmark results would change (along with a 95% confidence interval in relative change) if 02b6a47 is merged into master:

  • ✔️as_draws_array: 170ms -> 171ms [-0.34%, +1.26%]
  • ✔️as_draws_df: 82.8ms -> 82.2ms [-1.51%, +0.05%]
  • ✔️as_draws_list: 175ms -> 175ms [-0.47%, +1.13%]
  • ✔️as_draws_matrix: 31.6ms -> 31.6ms [-0.73%, +0.62%]
  • 🚀as_draws_rvars: 137ms -> 131ms [-5.16%, -3.78%]
  • 🚀summarise_draws_100_variables: 771ms -> 753ms [-2.93%, -1.51%]
  • ✔️summarise_draws_10_variables: 84.9ms -> 83.5ms [-4.02%, +0.62%]
    Further explanation regarding interpretation and methodology can be found in the documentation.

@jgabry jgabry left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ps_tail() changes look good. I made some small comments/suggestions inline and then there are two other things I think we need to sort out. I would also like to get @avehtari and @VisruthSK to say what they think because they both worked on #452.

  1. I'm a bit confused about something. This PR and PR #452 seem to point loo in different directions. In #452 ps_tail_sorted_log_ratios() was added so loo could select the tail itself and then we avoid sorting twice. With this PR, the plan seems to be that loo doesn't select the tail and just calls ps_tail() instead, right?

    I had Claude check some timings. Median time per call with this PR:

    S = 4000 S = 40000
    ps_tail() on the whole vector 332 µs 1180 µs
    loo's tail selection, then ps_tail_sorted_log_ratios() 312 µs 1045 µs

    The second row doesn't include writing the smoothed tail back into the full vector, which loo still has to do, so the real difference is even a bit smaller.

    If loo calls ps_tail(), we can just remove ps_tail_sorted_log_ratios() from posterior before it's released, and the new .ps_tail_sorted_log_ratios() split goes away. If we have loo keep doing its own selection, then it can keep its own constant-tail check too, and ps_tail() doesn't actually need the new warning or warn_constant_tail.

    So basically, does a difference of ~5-10% at most seem worth keeping the selection code in loo?

    But... am I interpreting this correctly or am I overlooking something relevant here?

  2. One thing we need to sort out in the loo plan: ps_tail_length() isn't an exact replacement for n_pareto() because loo uses ceiling() and posterior uses floor(), so they can sometimes give different answers. The arguments are also in the opposite order: n_pareto(r_eff, S) and ps_tail_length(ndraws, r_eff), which isn't a big deal but we need to remember to get that right when changing the code. But the floor vs ceiling difference means we could get different k-hat and smoothed weights (right?), so we should decide if that's what we want.

Comment thread NEWS.md
Comment on lines +13 to +14
* `ps_tail()` warns if all tail values are the same, with the class
`posterior_warning_pareto_constant_tail`.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This warning goes further than just ps_tail(). pareto_smooth() and pareto_khat() now also will warn, so something like pareto_khat(rep(0:1, 500)) goes from a silent NA to now being a warning. An rvar will warn once per element. (In loo .E_loo_khat_i() calls pareto_khat(tail = "right"), so it will start warning on constant tails too.)

Do we want it like this for posterior users, or is it only needed for loo? If it's only for loo, we could default to warn_constant_tail = FALSE and have loo pass TRUE. If we want it for posterior users too, then we could just update these lines in NEWS to something like this:

Suggested change
* `ps_tail()` warns if all tail values are the same, with the class
`posterior_warning_pareto_constant_tail`.
* `pareto_smooth()`, `pareto_khat()`, and `ps_tail()` warn if all tail values
are the same, with the class `posterior_warning_pareto_constant_tail`. With
`tail = "both"` they warn only if k is `NA`. `ps_tail()` has the new argument
`warn_constant_tail` to turn the warning off (#474).

Comment thread R/pareto_smooth.R
3 * sqrt(ndraws / r_eff),
ndraws / 5)
)
floor(pmin(ndraws / 5, 3 * sqrt(ndraws / r_eff)))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description says the values are identical at ndraws * r_eff == 225, but they're not. ps_tail_length(155, 225/155) returns 30 but the old code returns 31. That's because 3 * sqrt(ndraws / r_eff) rounds to just below 31 and then floor() drops it. I asked Claude to do a scan of ndraws up to 2 million and it found 9,588 cases, all are multiples of 5. But I don't think it matters. It requires an r_eff of exactly 225 / ndraws, so I'm OK with keeping this. I only mention it because the PR description says the values are identical.

old_tail_length <- function(ndraws, r_eff) {
floor(ifelse(ndraws * r_eff > 225, 3 * sqrt(ndraws / r_eff), ndraws / 5))
}
# the first 4 cases have ndraws * r_eff == 225, where the 2 branches meet

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These four cases don't hit the rounding difference. ndraws = 155, r_eff = 225/155 does but that would make this test fail (see comment above). Doesn't really matter except that the name of the test is too strong of a claim. Maybe rename the test or compare against fixed expected values, so it doesn't make the claim that the two versions always agree?

Comment thread NEWS.md
* `ps_tail()` warns if all tail values are the same, with the class
`posterior_warning_pareto_constant_tail`.
* Add the argument `max_threshold` to `ps_khat_threshold()`.
* Improve efficiency of `is_constant()` and `ps_tail_length()`.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ps_tail_length() only gets faster for long vectors, and posterior itself only passes scalars, so for posterior users only is_constant() is more efficient

Comment thread R/pareto_smooth.R
tail = "right",
are_log_weights = are_log_weights,
# warn only if k is NA for both tails
warn_constant_tail = is.na(left_k),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whether we get a warning depends on which of the tails is constant. Only this right-tail call can warn:

set.seed(3); x <- rpois(4000, 0.2)
pareto_khat(x, r_eff = 1)   # NA, no warning
pareto_khat(-x, r_eff = 1)  # NA, constant-tail warning

For x, the left tail is constant, so no warning, and the right tail has ties, so gpdfit() returns NA without a warning. This isn't actually a regression from the master branch because on master both of these cases are silent. I think making it actually symmetric would require ps_tail() to report why k is NA, but maybe that's more than we need?

Comment thread R/pareto_smooth.R
#' @export
ps_khat_threshold <- function(ndraws, ...) {
1 - 1 / log10(ndraws)
ps_khat_threshold <- function(ndraws, max_threshold = Inf, ...) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to add this new argument? loo could use min(posterior::ps_khat_threshold(S), 0.7), which is actually shorter than what the new call would be and it doesn't add anything to the public API.

Comment thread R/pareto_smooth.R
Comment on lines +751 to +755
out <- 1 - 1 / log10(ndraws)
if (max_threshold < Inf) {
out[out > max_threshold] <- max_threshold
}
out

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we do end up keeping the max_threshold argument (see previous comment), I think this part can be reduced to just one line:

Suggested change
out <- 1 - 1 / log10(ndraws)
if (max_threshold < Inf) {
out[out > max_threshold] <- max_threshold
}
out
pmin(1 - 1 / log10(ndraws), max_threshold)

Comment thread R/pareto_smooth.R
ndraws_tail = ndraws_tail,
tail = "left",
are_log_weights = are_log_weights,
warn_constant_tail = FALSE,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pareto_smooth() passes ... along to ps_tail(), so a call like pareto_smooth(x, tail = "both", warn_constant_tail = FALSE) would error with something like warn_constant_tail matched by multiple actual arguments. The argument is only documented for ps_tail(), so this really isn't a big deal, but I figured it could be worth mentioning just in case. I think we can probably just leave it as is?

Comment thread R/pareto_smooth.R
}

tail <- match.arg(tail)
tail <- match.arg(tail, c("right", "left"))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would lean towards keeping

Suggested change
tail <- match.arg(tail, c("right", "left"))
tail <- match.arg(tail)

because specifying the options explicitly saves only about 3 µs on a call that takes about 330 µs in exchange for having to write out the options in two separate places in the code.

Comment thread R/posterior-package.R
#' more than half of the total number of draws.
#' * `posterior_warning_pareto_tail_not_fitted`: The generalized Pareto
#' distribution was not fitted because there are fewer than 5 tail draws.
#' * `posterior_warning_pareto_constant_tail`: The generalized Pareto

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that this class exists, line 92-93 above shouldn't say "or has a constant tail" anymore.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And actually the messages at R/pareto_smooth.R:283 and :308 should say that anymore either (I didn't comment directly on those lines because they're not in the diff)

@florence-bockting

Copy link
Copy Markdown
Collaborator Author

Ok, thank you @jgabry. I realize that there were something that weren't thought through.
And I would at this point close the PR as it seems to be okay as is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants