Repository navigation
Speed up ps_tail() and warn on a constant tail - #474
florence-bockting wants to merge 4 commits into
Conversation
|
This is how benchmark results would change (along with a 95% confidence interval in relative change) if 02b6a47 is merged into master:
|
There was a problem hiding this comment.
The ps_tail() changes look good. I made some small comments/suggestions inline and then there are two other things I think we need to sort out. I would also like to get @avehtari and @VisruthSK to say what they think because they both worked on #452.
-
I'm a bit confused about something. This PR and PR #452 seem to point loo in different directions. In #452
ps_tail_sorted_log_ratios()was added so loo could select the tail itself and then we avoid sorting twice. With this PR, the plan seems to be that loo doesn't select the tail and just callsps_tail()instead, right?I had Claude check some timings. Median time per call with this PR:
S = 4000 S = 40000 ps_tail()on the whole vector332 µs 1180 µs loo's tail selection, then ps_tail_sorted_log_ratios()312 µs 1045 µs The second row doesn't include writing the smoothed tail back into the full vector, which loo still has to do, so the real difference is even a bit smaller.
If loo calls
ps_tail(), we can just removeps_tail_sorted_log_ratios()from posterior before it's released, and the new.ps_tail_sorted_log_ratios()split goes away. If we have loo keep doing its own selection, then it can keep its own constant-tail check too, andps_tail()doesn't actually need the new warning orwarn_constant_tail.So basically, does a difference of ~5-10% at most seem worth keeping the selection code in loo?
But... am I interpreting this correctly or am I overlooking something relevant here?
-
One thing we need to sort out in the loo plan:
ps_tail_length()isn't an exact replacement forn_pareto()because loo usesceiling()and posterior usesfloor(), so they can sometimes give different answers. The arguments are also in the opposite order:n_pareto(r_eff, S)andps_tail_length(ndraws, r_eff), which isn't a big deal but we need to remember to get that right when changing the code. But the floor vs ceiling difference means we could get different k-hat and smoothed weights (right?), so we should decide if that's what we want.
| * `ps_tail()` warns if all tail values are the same, with the class | ||
| `posterior_warning_pareto_constant_tail`. |
There was a problem hiding this comment.
This warning goes further than just ps_tail(). pareto_smooth() and pareto_khat() now also will warn, so something like pareto_khat(rep(0:1, 500)) goes from a silent NA to now being a warning. An rvar will warn once per element. (In loo .E_loo_khat_i() calls pareto_khat(tail = "right"), so it will start warning on constant tails too.)
Do we want it like this for posterior users, or is it only needed for loo? If it's only for loo, we could default to warn_constant_tail = FALSE and have loo pass TRUE. If we want it for posterior users too, then we could just update these lines in NEWS to something like this:
| * `ps_tail()` warns if all tail values are the same, with the class | |
| `posterior_warning_pareto_constant_tail`. | |
| * `pareto_smooth()`, `pareto_khat()`, and `ps_tail()` warn if all tail values | |
| are the same, with the class `posterior_warning_pareto_constant_tail`. With | |
| `tail = "both"` they warn only if k is `NA`. `ps_tail()` has the new argument | |
| `warn_constant_tail` to turn the warning off (#474). |
| 3 * sqrt(ndraws / r_eff), | ||
| ndraws / 5) | ||
| ) | ||
| floor(pmin(ndraws / 5, 3 * sqrt(ndraws / r_eff))) |
There was a problem hiding this comment.
The PR description says the values are identical at ndraws * r_eff == 225, but they're not. ps_tail_length(155, 225/155) returns 30 but the old code returns 31. That's because 3 * sqrt(ndraws / r_eff) rounds to just below 31 and then floor() drops it. I asked Claude to do a scan of ndraws up to 2 million and it found 9,588 cases, all are multiples of 5. But I don't think it matters. It requires an r_eff of exactly 225 / ndraws, so I'm OK with keeping this. I only mention it because the PR description says the values are identical.
| old_tail_length <- function(ndraws, r_eff) { | ||
| floor(ifelse(ndraws * r_eff > 225, 3 * sqrt(ndraws / r_eff), ndraws / 5)) | ||
| } | ||
| # the first 4 cases have ndraws * r_eff == 225, where the 2 branches meet |
There was a problem hiding this comment.
These four cases don't hit the rounding difference. ndraws = 155, r_eff = 225/155 does but that would make this test fail (see comment above). Doesn't really matter except that the name of the test is too strong of a claim. Maybe rename the test or compare against fixed expected values, so it doesn't make the claim that the two versions always agree?
| * `ps_tail()` warns if all tail values are the same, with the class | ||
| `posterior_warning_pareto_constant_tail`. | ||
| * Add the argument `max_threshold` to `ps_khat_threshold()`. | ||
| * Improve efficiency of `is_constant()` and `ps_tail_length()`. |
There was a problem hiding this comment.
ps_tail_length() only gets faster for long vectors, and posterior itself only passes scalars, so for posterior users only is_constant() is more efficient
| tail = "right", | ||
| are_log_weights = are_log_weights, | ||
| # warn only if k is NA for both tails | ||
| warn_constant_tail = is.na(left_k), |
There was a problem hiding this comment.
Whether we get a warning depends on which of the tails is constant. Only this right-tail call can warn:
set.seed(3); x <- rpois(4000, 0.2)
pareto_khat(x, r_eff = 1) # NA, no warning
pareto_khat(-x, r_eff = 1) # NA, constant-tail warningFor x, the left tail is constant, so no warning, and the right tail has ties, so gpdfit() returns NA without a warning. This isn't actually a regression from the master branch because on master both of these cases are silent. I think making it actually symmetric would require ps_tail() to report why k is NA, but maybe that's more than we need?
| #' @export | ||
| ps_khat_threshold <- function(ndraws, ...) { | ||
| 1 - 1 / log10(ndraws) | ||
| ps_khat_threshold <- function(ndraws, max_threshold = Inf, ...) { |
There was a problem hiding this comment.
Do we need to add this new argument? loo could use min(posterior::ps_khat_threshold(S), 0.7), which is actually shorter than what the new call would be and it doesn't add anything to the public API.
| out <- 1 - 1 / log10(ndraws) | ||
| if (max_threshold < Inf) { | ||
| out[out > max_threshold] <- max_threshold | ||
| } | ||
| out |
There was a problem hiding this comment.
If we do end up keeping the max_threshold argument (see previous comment), I think this part can be reduced to just one line:
| out <- 1 - 1 / log10(ndraws) | |
| if (max_threshold < Inf) { | |
| out[out > max_threshold] <- max_threshold | |
| } | |
| out | |
| pmin(1 - 1 / log10(ndraws), max_threshold) |
| ndraws_tail = ndraws_tail, | ||
| tail = "left", | ||
| are_log_weights = are_log_weights, | ||
| warn_constant_tail = FALSE, |
There was a problem hiding this comment.
pareto_smooth() passes ... along to ps_tail(), so a call like pareto_smooth(x, tail = "both", warn_constant_tail = FALSE) would error with something like warn_constant_tail matched by multiple actual arguments. The argument is only documented for ps_tail(), so this really isn't a big deal, but I figured it could be worth mentioning just in case. I think we can probably just leave it as is?
| } | ||
|
|
||
| tail <- match.arg(tail) | ||
| tail <- match.arg(tail, c("right", "left")) |
There was a problem hiding this comment.
I would lean towards keeping
| tail <- match.arg(tail, c("right", "left")) | |
| tail <- match.arg(tail) |
because specifying the options explicitly saves only about 3 µs on a call that takes about 330 µs in exchange for having to write out the options in two separate places in the code.
| #' more than half of the total number of draws. | ||
| #' * `posterior_warning_pareto_tail_not_fitted`: The generalized Pareto | ||
| #' distribution was not fitted because there are fewer than 5 tail draws. | ||
| #' * `posterior_warning_pareto_constant_tail`: The generalized Pareto |
There was a problem hiding this comment.
Now that this class exists, line 92-93 above shouldn't say "or has a constant tail" anymore.
There was a problem hiding this comment.
And actually the messages at R/pareto_smooth.R:283 and :308 should say that anymore either (I didn't comment directly on those lines because they're not in the diff)
|
Ok, thank you @jgabry. I realize that there were something that weren't thought through. |
Summary
This PR prepares a loo PR. That loo PR removes the loo copies of the Pareto smoothing code and calls
posterior::ps_tail()directly. loo then needs the constant-tail warning from posterior.Changes:
ps_tail()warns if all tail values are the same. The new warning class isposterior_warning_pareto_constant_tail. It is added to the list of warning classes in?posterior. The new argumentwarn_constant_tail = FALSEturns the warning off.pareto_smooth()andpareto_khat()withtail = "both"warn only if k isNA. One constant tail with a good k on the other tail gives no warning.ps_tail()checks only the first and last sorted tail values for a constant tail.is_constant()usesmin()/max()in place ofrange().ps_tail_length()usespmin()in place ofifelse(). The values are identical, also atndraws * r_eff == 225.ps_khat_threshold()has the new argumentmax_threshold = Inf. loo usesmax_threshold = 0.7.Planned loo changes (for background info)
loo functions to delete, with the posterior replacement:
psis_smooth_tail():posterior::ps_tail(..., are_log_weights = TRUE)exp_x_minus_exp_y(): no replacement needed.ps_tail()has its own copyis_constant():posterior::is_constant()using the relative toleranceps_khat_threshold():posterior::ps_khat_threshold(S, max_threshold = 0.7)n_pareto():posterior::ps_tail_length()loo function to change:
do_psis_i(): oneposterior::ps_tail()call replaces the sort, the constant check andpsis_smooth_tail(). It keeps the tail-length check, the shift back by the maximum, and the mapping of k =NAtoInf.AI assistance
This PR used LLM assistance.
Copyright and Licensing
Copyright holder: Aalto University
By submitting this pull request, the copyright holder is agreeing to
license the submitted work under the following licenses: