Repository navigation
Speed up ps_tail() and warn on a constant tail #474
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
d4b851c
bd87c2c
8974393
02b6a47
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -10,6 +10,10 @@ result instead of about three. | |
| selected. | ||
| * All warnings now have the class `posterior_warning` and a class for each | ||
| cause. Calling packages can catch, count, or muffle each type of warning (#470). | ||
| * `ps_tail()` warns if all tail values are the same, with the class | ||
| `posterior_warning_pareto_constant_tail`. | ||
| * Add the argument `max_threshold` to `ps_khat_threshold()`. | ||
| * Improve efficiency of `is_constant()` and `ps_tail_length()`. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
|
|
||
| ### Bug Fixes | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -342,16 +342,19 @@ pareto_smooth.default <- function(x, | |||||||||||||
| ndraws_tail = ndraws_tail, | ||||||||||||||
| tail = "left", | ||||||||||||||
| are_log_weights = are_log_weights, | ||||||||||||||
| warn_constant_tail = FALSE, | ||||||||||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||||||||||||||
| ... | ||||||||||||||
| ) | ||||||||||||||
| left_k <- smoothed$k | ||||||||||||||
|
|
||||||||||||||
| # right tail | ||||||||||||||
| smoothed <-ps_tail( | ||||||||||||||
| smoothed <- ps_tail( | ||||||||||||||
| x = smoothed$x, | ||||||||||||||
| ndraws_tail = ndraws_tail, | ||||||||||||||
| tail = "right", | ||||||||||||||
| are_log_weights = are_log_weights, | ||||||||||||||
| # warn only if k is NA for both tails | ||||||||||||||
| warn_constant_tail = is.na(left_k), | ||||||||||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Whether we get a warning depends on which of the tails is constant. Only this right-tail call can warn: set.seed(3); x <- rpois(4000, 0.2)
pareto_khat(x, r_eff = 1) # NA, no warning
pareto_khat(-x, r_eff = 1) # NA, constant-tail warningFor |
||||||||||||||
| ... | ||||||||||||||
| ) | ||||||||||||||
| right_k <- smoothed$k | ||||||||||||||
|
|
@@ -483,6 +486,8 @@ pareto_convergence_rate.rvar <- function(x, ...) { | |||||||||||||
| #' @param are_log_weights (logical) Are the draws log weights? Default is | ||||||||||||||
| #' `FALSE`. If `TRUE` computation will take into account that the | ||||||||||||||
| #' draws are log weights, and only right tail will be smoothed. | ||||||||||||||
| #' @param warn_constant_tail (logical) Warn if all tail values are the | ||||||||||||||
| #' same? Default is `TRUE`. | ||||||||||||||
| #' @template args-methods-dots | ||||||||||||||
| #' @template ref-vehtari-paretosmooth-2022 | ||||||||||||||
| #' @seealso [`pareto_smooth`] for the user-facing function. | ||||||||||||||
|
|
@@ -493,6 +498,7 @@ ps_tail <- function(x, | |||||||||||||
| smooth_draws = TRUE, | ||||||||||||||
| tail = c("right", "left"), | ||||||||||||||
| are_log_weights = FALSE, | ||||||||||||||
| warn_constant_tail = TRUE, | ||||||||||||||
| ... | ||||||||||||||
| ) { | ||||||||||||||
|
|
||||||||||||||
|
|
@@ -510,7 +516,7 @@ ps_tail <- function(x, | |||||||||||||
| x <- x - max(x) | ||||||||||||||
| } | ||||||||||||||
|
|
||||||||||||||
| tail <- match.arg(tail) | ||||||||||||||
| tail <- match.arg(tail, c("right", "left")) | ||||||||||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would lean towards keeping
Suggested change
because specifying the options explicitly saves only about 3 µs on a call that takes about 330 µs in exchange for having to write out the options in two separate places in the code. |
||||||||||||||
| if (tail == "left") { | ||||||||||||||
| x <- -x | ||||||||||||||
| } | ||||||||||||||
|
|
@@ -519,7 +525,14 @@ ps_tail <- function(x, | |||||||||||||
| ix_tail <- selection$ix | ||||||||||||||
| draws_tail <- x[ix_tail] | ||||||||||||||
|
|
||||||||||||||
| if (is_constant(draws_tail)) { | ||||||||||||||
| if (is_constant(draws_tail[c(1L, ndraws_tail)])) { | ||||||||||||||
| if (warn_constant_tail) { | ||||||||||||||
| warning_no_call( | ||||||||||||||
| "Can't fit generalized Pareto distribution ", | ||||||||||||||
| "because all tail values are the same.", | ||||||||||||||
| class = "posterior_warning_pareto_constant_tail" | ||||||||||||||
| ) | ||||||||||||||
| } | ||||||||||||||
| if (tail == "left") { | ||||||||||||||
| x <- -x | ||||||||||||||
| } | ||||||||||||||
|
|
@@ -531,7 +544,7 @@ ps_tail <- function(x, | |||||||||||||
| cutoff <- selection$cutoff | ||||||||||||||
|
|
||||||||||||||
| if (are_log_weights) { | ||||||||||||||
| smoothed <- ps_tail_sorted_log_ratios( | ||||||||||||||
| smoothed <- .ps_tail_sorted_log_ratios( | ||||||||||||||
| x = draws_tail, | ||||||||||||||
| cutoff = cutoff, | ||||||||||||||
| smooth_draws = smooth_draws, | ||||||||||||||
|
|
@@ -615,7 +628,12 @@ ps_tail_sorted_log_ratios <- function(x, cutoff, smooth_draws = TRUE, ...) { | |||||||||||||
| if (is_constant(x)) { | ||||||||||||||
| return(list(tail = x, k = NA)) | ||||||||||||||
| } | ||||||||||||||
| .ps_tail_sorted_log_ratios(x, cutoff, smooth_draws = smooth_draws, ...) | ||||||||||||||
| } | ||||||||||||||
|
|
||||||||||||||
| # Internal version without input checks. The caller must give a sorted, | ||||||||||||||
| # non-constant x. | ||||||||||||||
| .ps_tail_sorted_log_ratios <- function(x, cutoff, smooth_draws = TRUE, ...) { | ||||||||||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm a bit confused about something. This PR and PR #452 seem to point loo in different directions. In #452 I had Claude check some timings. Median time per call with this PR:
The second row doesn't include writing the smoothed tail back into the full vector, which loo still has to do, so the real difference is even a bit smaller. If loo calls So basically, does a difference of ~5-10% at most seem worth keeping the selection code in loo? Also tagging @avehtari and @VisruthSK since they both worked on #452. Am I interpreting this correctly or overlooking something relevant here? |
||||||||||||||
| smoothed <- .ps_tail_sorted( | ||||||||||||||
| x = exp(x), | ||||||||||||||
| excess = exp_x_minus_exp_y(x, cutoff), | ||||||||||||||
|
|
@@ -724,11 +742,17 @@ ps_min_ss <- function(k, ...) { | |||||||||||||
| #' [`pareto_diags`]. | ||||||||||||||
| #' @family helper-functions | ||||||||||||||
| #' @param ndraws number of draws | ||||||||||||||
| #' @param max_threshold (numeric) upper bound for the threshold. Default is | ||||||||||||||
| #' `Inf`, which means no bound. Algorithm 1 in Vehtari et al. (2024) uses 0.7. | ||||||||||||||
| #' @param ... unused | ||||||||||||||
| #' @return threshold | ||||||||||||||
| #' @export | ||||||||||||||
| ps_khat_threshold <- function(ndraws, ...) { | ||||||||||||||
| 1 - 1 / log10(ndraws) | ||||||||||||||
| ps_khat_threshold <- function(ndraws, max_threshold = Inf, ...) { | ||||||||||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do we need to add this new argument? loo could use |
||||||||||||||
| out <- 1 - 1 / log10(ndraws) | ||||||||||||||
| if (max_threshold < Inf) { | ||||||||||||||
| out[out > max_threshold] <- max_threshold | ||||||||||||||
| } | ||||||||||||||
| out | ||||||||||||||
|
Comment on lines
+751
to
+755
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If we do end up keeping the
Suggested change
|
||||||||||||||
| } | ||||||||||||||
|
|
||||||||||||||
| #' Pareto convergence rate | ||||||||||||||
|
|
@@ -785,12 +809,7 @@ ps_convergence_rate <- function(k, ndraws, ...) { | |||||||||||||
| #' @return tail length | ||||||||||||||
| #' @export | ||||||||||||||
| ps_tail_length <- function(ndraws, r_eff, ...) { | ||||||||||||||
| floor( | ||||||||||||||
| ifelse( | ||||||||||||||
| ndraws * r_eff > 225, | ||||||||||||||
| 3 * sqrt(ndraws / r_eff), | ||||||||||||||
| ndraws / 5) | ||||||||||||||
| ) | ||||||||||||||
| floor(pmin(ndraws / 5, 3 * sqrt(ndraws / r_eff))) | ||||||||||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The PR description says the values are identical at |
||||||||||||||
| } | ||||||||||||||
|
|
||||||||||||||
| #' Pareto-k diagnostic message | ||||||||||||||
|
|
||||||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -99,6 +99,8 @@ | |
| #' more than half of the total number of draws. | ||
| #' * `posterior_warning_pareto_tail_not_fitted`: The generalized Pareto | ||
| #' distribution was not fitted because there are fewer than 5 tail draws. | ||
| #' * `posterior_warning_pareto_constant_tail`: The generalized Pareto | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Now that this class exists, line 92-93 above shouldn't say "or has a constant tail" anymore.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. And actually the messages at |
||
| #' distribution was not fitted because all tail values are the same. | ||
| #' | ||
| #' The error signaled when all weights are zero has the class | ||
| #' `posterior_error_degenerate_weights`. | ||
|
|
||
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,11 +4,16 @@ test_that("pareto_khat handles constant tail correctly", { | |
| # should be the same as the right tail | ||
| x <- c(rep(-100, 10), sort(rnorm(100))) | ||
|
|
||
| expect_true(is.na(pareto_khat(x, tail = "left", ndraws_tail = 10))) | ||
| expect_equal( | ||
| pareto_khat(x, tail = "right", ndraws_tail = 10), | ||
| pareto_khat(x, tail = "both", ndraws_tail = 10) | ||
| expect_warning( | ||
| k_left <- pareto_khat(x, tail = "left", ndraws_tail = 10), | ||
| class = "posterior_warning_pareto_constant_tail" | ||
| ) | ||
| expect_true(is.na(k_left)) | ||
| # the constant left tail gives no warning, because k is not NA | ||
| expect_no_warning( | ||
| k_both <- pareto_khat(x, tail = "both", ndraws_tail = 10) | ||
| ) | ||
| expect_equal(pareto_khat(x, tail = "right", ndraws_tail = 10), k_both) | ||
|
|
||
| }) | ||
|
|
||
|
|
@@ -18,9 +23,37 @@ test_that("pareto_khat handles both tails being constant correctly", { | |
| # left and right tails are constant, so khat should be NA in all cases | ||
| x <- c(rep(-100, 10), sort(rnorm(100)), rep(100, 10)) | ||
|
|
||
| expect_true(is.na(pareto_khat(x, tail = "left", ndraws_tail = 10))) | ||
| expect_true(is.na(pareto_khat(x, tail = "right", ndraws_tail = 10))) | ||
| expect_true(is.na(pareto_khat(x, tail = "both", ndraws_tail = 10))) | ||
| for (tail in c("left", "right", "both")) { | ||
| expect_warning( | ||
| k <- pareto_khat(x, tail = tail, ndraws_tail = 10), | ||
| class = "posterior_warning_pareto_constant_tail" | ||
| ) | ||
| expect_true(is.na(k)) | ||
| } | ||
|
|
||
| }) | ||
|
|
||
|
|
||
| test_that("pareto_khat with tail = 'both' warns only if k is NA", { | ||
|
|
||
| set.seed(1) | ||
| mid <- sort(rnorm(100)) | ||
| const <- rep(100, 10) | ||
| # the 3 smallest tail values are equal, so gpdfit gives k = NA | ||
| ties <- c(5, 5, 5, 6:12) | ||
|
|
||
| # only the left tail is constant: k comes from the right tail | ||
| expect_no_warning( | ||
| k <- pareto_khat(c(-const, mid), tail = "both", ndraws_tail = 10) | ||
| ) | ||
| expect_false(is.na(k)) | ||
|
|
||
| # left tail with ties, right tail constant: k is NA | ||
| expect_warning( | ||
| k <- pareto_khat(c(-ties, mid, const), tail = "both", ndraws_tail = 10), | ||
| class = "posterior_warning_pareto_constant_tail" | ||
| ) | ||
| expect_true(is.na(k)) | ||
|
Comment on lines
+51
to
+56
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should we also test when k is # left tail constant, right tail with ties: k is NA, but no warning
expect_no_warning(
k <- pareto_khat(c(-const, mid, ties), tail = "both", ndraws_tail = 10)
)
expect_true(is.na(k)) |
||
|
|
||
| }) | ||
|
|
||
|
|
@@ -30,8 +63,9 @@ test_that("pareto_smooth handles non-constant x with constant tail", { | |
| # x is non-constant but the right tail is constant | ||
| x <- c(seq(-3, 3, length.out = 90), rep(5, 10)) | ||
|
|
||
| expect_no_error( | ||
| ps <- pareto_smooth(x, tail = "right", ndraws_tail = 10, return_k = TRUE, verbose = FALSE) | ||
| expect_warning( | ||
| ps <- pareto_smooth(x, tail = "right", ndraws_tail = 10, return_k = TRUE, verbose = FALSE), | ||
| class = "posterior_warning_pareto_constant_tail" | ||
| ) | ||
| expect_true(is.na(ps$diagnostics$khat)) | ||
| expect_equal(ps$x, x) | ||
|
|
@@ -386,11 +420,44 @@ test_that("ps_tail_sorted_log_ratios requires a sorted tail", { | |
| test_that("check ps_tail behavior for constant draws_tail", { | ||
| x <- log(replicate(10, 0.3)) | ||
|
|
||
| tail <- ps_tail(x, ndraws_tail = 10, tail = "left") | ||
| expect_warning( | ||
| tail <- ps_tail(x, ndraws_tail = 10, tail = "left"), | ||
| class = "posterior_warning_pareto_constant_tail" | ||
| ) | ||
|
|
||
| # output has expected return values | ||
| expect_equal(x, tail$x) | ||
| expect_true(is.na(tail$k)) | ||
|
|
||
| # warn_constant_tail = FALSE gives no warning and the same output | ||
| expect_no_warning( | ||
| tail2 <- ps_tail(x, ndraws_tail = 10, tail = "left", warn_constant_tail = FALSE) | ||
| ) | ||
| expect_equal(tail2, tail) | ||
| }) | ||
|
|
||
|
|
||
| test_that("ps_tail_length gives the same values as the old ifelse version", { | ||
| old_tail_length <- function(ndraws, r_eff) { | ||
| floor(ifelse(ndraws * r_eff > 225, 3 * sqrt(ndraws / r_eff), ndraws / 5)) | ||
| } | ||
| # the first 4 cases have ndraws * r_eff == 225, where the 2 branches meet | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. These four cases don't hit the rounding difference. |
||
| ndraws <- c(225, 1125, 450, 100, 4000, 1e5, 10) | ||
| r_eff <- c(1, 0.2, 0.5, 2.25, 1, 0.01, 1) | ||
| expect_equal(ps_tail_length(ndraws, r_eff), old_tail_length(ndraws, r_eff)) | ||
| expect_equal(ps_tail_length(225, 1), 45) | ||
| expect_true(is.na(ps_tail_length(1000, NA))) | ||
| }) | ||
|
|
||
|
|
||
| test_that("ps_khat_threshold caps the threshold at max_threshold", { | ||
| ndraws <- c(10, 4000, 1e6) | ||
| expect_equal(ps_khat_threshold(ndraws), 1 - 1 / log10(ndraws)) | ||
| expect_equal( | ||
| ps_khat_threshold(ndraws, max_threshold = 0.7), | ||
| pmin(1 - 1 / log10(ndraws), 0.7) | ||
| ) | ||
| expect_true(is.na(ps_khat_threshold(NA, max_threshold = 0.7))) | ||
| }) | ||
|
|
||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This warning goes further than just
ps_tail().pareto_smooth()andpareto_khat()now also will warn, so something likepareto_khat(rep(0:1, 500))goes from a silentNAto now being a warning. Anrvarwill warn once per element. (In loo.E_loo_khat_i()callspareto_khat(tail = "right"), so it will start warning on constant tails too.)Do we want it like this for posterior users, or is it only needed for loo? If it's only for loo, we could default to
warn_constant_tail = FALSEand have loo passTRUE. If we want it for posterior users too, then we could just update these lines in NEWS to something like this: