Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ result instead of about three.
selected.
* All warnings now have the class `posterior_warning` and a class for each
cause. Calling packages can catch, count, or muffle each type of warning (#470).
* `ps_tail()` warns if all tail values are the same, with the class
`posterior_warning_pareto_constant_tail`.
Comment on lines +13 to +14

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This warning goes further than just ps_tail(). pareto_smooth() and pareto_khat() now also will warn, so something like pareto_khat(rep(0:1, 500)) goes from a silent NA to now being a warning. An rvar will warn once per element. (In loo .E_loo_khat_i() calls pareto_khat(tail = "right"), so it will start warning on constant tails too.)

Do we want it like this for posterior users, or is it only needed for loo? If it's only for loo, we could default to warn_constant_tail = FALSE and have loo pass TRUE. If we want it for posterior users too, then we could just update these lines in NEWS to something like this:

Suggested change
* `ps_tail()` warns if all tail values are the same, with the class
`posterior_warning_pareto_constant_tail`.
* `pareto_smooth()`, `pareto_khat()`, and `ps_tail()` warn if all tail values
are the same, with the class `posterior_warning_pareto_constant_tail`. With
`tail = "both"` they warn only if k is `NA`. `ps_tail()` has the new argument
`warn_constant_tail` to turn the warning off (#474).

* Add the argument `max_threshold` to `ps_khat_threshold()`.
* Improve efficiency of `is_constant()` and `ps_tail_length()`.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ps_tail_length() only gets faster for long vectors, and posterior itself only passes scalars, so for posterior users only is_constant() is more efficient


### Bug Fixes

Expand Down
7 changes: 4 additions & 3 deletions R/misc.R
Original file line number Diff line number Diff line change
Expand Up @@ -196,10 +196,11 @@ isNA <- function(x) {
#' @keywords internal
#' @export
is_constant <- function(x, tol = .Machine$double.eps) {
r <- range(x)
spread <- r[2] - r[1]
lo <- min(x)
hi <- max(x)
spread <- hi - lo
# the equality check covers all-zero and all-infinite x
r[1] == r[2] || (is.finite(spread) && spread <= tol * max(abs(r)))
lo == hi || (is.finite(spread) && spread <= tol * max(abs(lo), abs(hi)))
}

# collapse variables via commas for pretty printing
Expand Down
43 changes: 31 additions & 12 deletions R/pareto_smooth.R
Original file line number Diff line number Diff line change
Expand Up @@ -342,16 +342,19 @@ pareto_smooth.default <- function(x,
ndraws_tail = ndraws_tail,
tail = "left",
are_log_weights = are_log_weights,
warn_constant_tail = FALSE,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pareto_smooth() passes ... along to ps_tail(), so a call like pareto_smooth(x, tail = "both", warn_constant_tail = FALSE) would error with something like warn_constant_tail matched by multiple actual arguments. The argument is only documented for ps_tail(), so this really isn't a big deal, but I figured it could be worth mentioning just in case. I think we can probably just leave it as is?

...
)
left_k <- smoothed$k

# right tail
smoothed <-ps_tail(
smoothed <- ps_tail(
x = smoothed$x,
ndraws_tail = ndraws_tail,
tail = "right",
are_log_weights = are_log_weights,
# warn only if k is NA for both tails
warn_constant_tail = is.na(left_k),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whether we get a warning depends on which of the tails is constant. Only this right-tail call can warn:

set.seed(3); x <- rpois(4000, 0.2)
pareto_khat(x, r_eff = 1)   # NA, no warning
pareto_khat(-x, r_eff = 1)  # NA, constant-tail warning

For x, the left tail is constant, so no warning, and the right tail has ties, so gpdfit() returns NA without a warning. This isn't actually a regression from the master branch because on master both of these cases are silent. I think making it actually symmetric would require ps_tail() to report why k is NA, but maybe that's more than we need?

...
)
right_k <- smoothed$k
Expand Down Expand Up @@ -483,6 +486,8 @@ pareto_convergence_rate.rvar <- function(x, ...) {
#' @param are_log_weights (logical) Are the draws log weights? Default is
#' `FALSE`. If `TRUE` computation will take into account that the
#' draws are log weights, and only right tail will be smoothed.
#' @param warn_constant_tail (logical) Warn if all tail values are the
#' same? Default is `TRUE`.
#' @template args-methods-dots
#' @template ref-vehtari-paretosmooth-2022
#' @seealso [`pareto_smooth`] for the user-facing function.
Expand All @@ -493,6 +498,7 @@ ps_tail <- function(x,
smooth_draws = TRUE,
tail = c("right", "left"),
are_log_weights = FALSE,
warn_constant_tail = TRUE,
...
) {

Expand All @@ -510,7 +516,7 @@ ps_tail <- function(x,
x <- x - max(x)
}

tail <- match.arg(tail)
tail <- match.arg(tail, c("right", "left"))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would lean towards keeping

Suggested change
tail <- match.arg(tail, c("right", "left"))
tail <- match.arg(tail)

because specifying the options explicitly saves only about 3 µs on a call that takes about 330 µs in exchange for having to write out the options in two separate places in the code.

if (tail == "left") {
x <- -x
}
Expand All @@ -519,7 +525,14 @@ ps_tail <- function(x,
ix_tail <- selection$ix
draws_tail <- x[ix_tail]

if (is_constant(draws_tail)) {
if (is_constant(draws_tail[c(1L, ndraws_tail)])) {
if (warn_constant_tail) {
warning_no_call(
"Can't fit generalized Pareto distribution ",
"because all tail values are the same.",
class = "posterior_warning_pareto_constant_tail"
)
}
if (tail == "left") {
x <- -x
}
Expand All @@ -531,7 +544,7 @@ ps_tail <- function(x,
cutoff <- selection$cutoff

if (are_log_weights) {
smoothed <- ps_tail_sorted_log_ratios(
smoothed <- .ps_tail_sorted_log_ratios(
x = draws_tail,
cutoff = cutoff,
smooth_draws = smooth_draws,
Expand Down Expand Up @@ -615,7 +628,12 @@ ps_tail_sorted_log_ratios <- function(x, cutoff, smooth_draws = TRUE, ...) {
if (is_constant(x)) {
return(list(tail = x, k = NA))
}
.ps_tail_sorted_log_ratios(x, cutoff, smooth_draws = smooth_draws, ...)
}

# Internal version without input checks. The caller must give a sorted,
# non-constant x.
.ps_tail_sorted_log_ratios <- function(x, cutoff, smooth_draws = TRUE, ...) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit confused about something. This PR and PR #452 seem to point loo in different directions. In #452 ps_tail_sorted_log_ratios() was added so loo could select the tail itself and then we avoid sorting twice. With this PR, the plan seems to be that loo doesn't select the tail and just calls ps_tail() instead, right?

I had Claude check some timings. Median time per call with this PR:

S = 4000 S = 40000
ps_tail() on the whole vector 332 µs 1180 µs
loo's tail selection, then ps_tail_sorted_log_ratios() 312 µs 1045 µs

The second row doesn't include writing the smoothed tail back into the full vector, which loo still has to do, so the real difference is even a bit smaller.

If loo calls ps_tail(), we can just remove ps_tail_sorted_log_ratios() before it's released, and this split goes away. If we have loo keep doing its own selection, then it can keep its own constant-tail check too, and ps_tail() doesn't actually need the new warning or warn_constant_tail.

So basically, does a difference of ~5-10% at most seem worth keeping the selection code in loo? Also tagging @avehtari and @VisruthSK since they both worked on #452.

Am I interpreting this correctly or overlooking something relevant here?

smoothed <- .ps_tail_sorted(
x = exp(x),
excess = exp_x_minus_exp_y(x, cutoff),
Expand Down Expand Up @@ -724,11 +742,17 @@ ps_min_ss <- function(k, ...) {
#' [`pareto_diags`].
#' @family helper-functions
#' @param ndraws number of draws
#' @param max_threshold (numeric) upper bound for the threshold. Default is
#' `Inf`, which means no bound. Algorithm 1 in Vehtari et al. (2024) uses 0.7.
#' @param ... unused
#' @return threshold
#' @export
ps_khat_threshold <- function(ndraws, ...) {
1 - 1 / log10(ndraws)
ps_khat_threshold <- function(ndraws, max_threshold = Inf, ...) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to add this new argument? loo could use min(posterior::ps_khat_threshold(S), 0.7), which is actually shorter than what the new call would be and it doesn't add anything to the public API.

out <- 1 - 1 / log10(ndraws)
if (max_threshold < Inf) {
out[out > max_threshold] <- max_threshold
}
out
Comment on lines +751 to +755

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we do end up keeping the max_threshold argument (see previous comment), I think this part can be reduced to just one line:

Suggested change
out <- 1 - 1 / log10(ndraws)
if (max_threshold < Inf) {
out[out > max_threshold] <- max_threshold
}
out
pmin(1 - 1 / log10(ndraws), max_threshold)

}

#' Pareto convergence rate
Expand Down Expand Up @@ -785,12 +809,7 @@ ps_convergence_rate <- function(k, ndraws, ...) {
#' @return tail length
#' @export
ps_tail_length <- function(ndraws, r_eff, ...) {
floor(
ifelse(
ndraws * r_eff > 225,
3 * sqrt(ndraws / r_eff),
ndraws / 5)
)
floor(pmin(ndraws / 5, 3 * sqrt(ndraws / r_eff)))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description says the values are identical at ndraws * r_eff == 225, but they're not. ps_tail_length(155, 225/155) returns 30 but the old code returns 31. That's because 3 * sqrt(ndraws / r_eff) rounds to just below 31 and then floor() drops it. I asked Claude to do a scan of ndraws up to 2 million and it found 9,588 cases, all are multiples of 5. But I don't think it matters. It requires an r_eff of exactly 225 / ndraws, so I'm OK with keeping this. I only mention it because the PR description says the values are identical.

}

#' Pareto-k diagnostic message
Expand Down
2 changes: 2 additions & 0 deletions R/posterior-package.R
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,8 @@
#' more than half of the total number of draws.
#' * `posterior_warning_pareto_tail_not_fitted`: The generalized Pareto
#' distribution was not fitted because there are fewer than 5 tail draws.
#' * `posterior_warning_pareto_constant_tail`: The generalized Pareto

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that this class exists, line 92-93 above shouldn't say "or has a constant tail" anymore.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And actually the messages at R/pareto_smooth.R:283 and :308 should say that anymore either (I didn't comment directly on those lines because they're not in the diff)

#' distribution was not fitted because all tail values are the same.
#'
#' The error signaled when all weights are zero has the class
#' `posterior_error_degenerate_weights`.
Expand Down
2 changes: 2 additions & 0 deletions man/posterior-package.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 4 additions & 1 deletion man/ps_khat_threshold.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions man/ps_tail.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

87 changes: 77 additions & 10 deletions tests/testthat/test-pareto_smooth.R
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,16 @@ test_that("pareto_khat handles constant tail correctly", {
# should be the same as the right tail
x <- c(rep(-100, 10), sort(rnorm(100)))

expect_true(is.na(pareto_khat(x, tail = "left", ndraws_tail = 10)))
expect_equal(
pareto_khat(x, tail = "right", ndraws_tail = 10),
pareto_khat(x, tail = "both", ndraws_tail = 10)
expect_warning(
k_left <- pareto_khat(x, tail = "left", ndraws_tail = 10),
class = "posterior_warning_pareto_constant_tail"
)
expect_true(is.na(k_left))
# the constant left tail gives no warning, because k is not NA
expect_no_warning(
k_both <- pareto_khat(x, tail = "both", ndraws_tail = 10)
)
expect_equal(pareto_khat(x, tail = "right", ndraws_tail = 10), k_both)

})

Expand All @@ -18,9 +23,37 @@ test_that("pareto_khat handles both tails being constant correctly", {
# left and right tails are constant, so khat should be NA in all cases
x <- c(rep(-100, 10), sort(rnorm(100)), rep(100, 10))

expect_true(is.na(pareto_khat(x, tail = "left", ndraws_tail = 10)))
expect_true(is.na(pareto_khat(x, tail = "right", ndraws_tail = 10)))
expect_true(is.na(pareto_khat(x, tail = "both", ndraws_tail = 10)))
for (tail in c("left", "right", "both")) {
expect_warning(
k <- pareto_khat(x, tail = tail, ndraws_tail = 10),
class = "posterior_warning_pareto_constant_tail"
)
expect_true(is.na(k))
}

})


test_that("pareto_khat with tail = 'both' warns only if k is NA", {

set.seed(1)
mid <- sort(rnorm(100))
const <- rep(100, 10)
# the 3 smallest tail values are equal, so gpdfit gives k = NA
ties <- c(5, 5, 5, 6:12)

# only the left tail is constant: k comes from the right tail
expect_no_warning(
k <- pareto_khat(c(-const, mid), tail = "both", ndraws_tail = 10)
)
expect_false(is.na(k))

# left tail with ties, right tail constant: k is NA
expect_warning(
k <- pareto_khat(c(-ties, mid, const), tail = "both", ndraws_tail = 10),
class = "posterior_warning_pareto_constant_tail"
)
expect_true(is.na(k))
Comment on lines +51 to +56

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also test when k is NA and there's no warning? For example adding something like this:

# left tail constant, right tail with ties: k is NA, but no warning
expect_no_warning(
  k <- pareto_khat(c(-const, mid, ties), tail = "both", ndraws_tail = 10)
)
expect_true(is.na(k))


})

Expand All @@ -30,8 +63,9 @@ test_that("pareto_smooth handles non-constant x with constant tail", {
# x is non-constant but the right tail is constant
x <- c(seq(-3, 3, length.out = 90), rep(5, 10))

expect_no_error(
ps <- pareto_smooth(x, tail = "right", ndraws_tail = 10, return_k = TRUE, verbose = FALSE)
expect_warning(
ps <- pareto_smooth(x, tail = "right", ndraws_tail = 10, return_k = TRUE, verbose = FALSE),
class = "posterior_warning_pareto_constant_tail"
)
expect_true(is.na(ps$diagnostics$khat))
expect_equal(ps$x, x)
Expand Down Expand Up @@ -386,11 +420,44 @@ test_that("ps_tail_sorted_log_ratios requires a sorted tail", {
test_that("check ps_tail behavior for constant draws_tail", {
x <- log(replicate(10, 0.3))

tail <- ps_tail(x, ndraws_tail = 10, tail = "left")
expect_warning(
tail <- ps_tail(x, ndraws_tail = 10, tail = "left"),
class = "posterior_warning_pareto_constant_tail"
)

# output has expected return values
expect_equal(x, tail$x)
expect_true(is.na(tail$k))

# warn_constant_tail = FALSE gives no warning and the same output
expect_no_warning(
tail2 <- ps_tail(x, ndraws_tail = 10, tail = "left", warn_constant_tail = FALSE)
)
expect_equal(tail2, tail)
})


test_that("ps_tail_length gives the same values as the old ifelse version", {
old_tail_length <- function(ndraws, r_eff) {
floor(ifelse(ndraws * r_eff > 225, 3 * sqrt(ndraws / r_eff), ndraws / 5))
}
# the first 4 cases have ndraws * r_eff == 225, where the 2 branches meet

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These four cases don't hit the rounding difference. ndraws = 155, r_eff = 225/155 does but that would make this test fail (see comment above). Doesn't really matter except that the name of the test is too strong of a claim. Maybe rename the test or compare against fixed expected values, so it doesn't make the claim that the two versions always agree?

ndraws <- c(225, 1125, 450, 100, 4000, 1e5, 10)
r_eff <- c(1, 0.2, 0.5, 2.25, 1, 0.01, 1)
expect_equal(ps_tail_length(ndraws, r_eff), old_tail_length(ndraws, r_eff))
expect_equal(ps_tail_length(225, 1), 45)
expect_true(is.na(ps_tail_length(1000, NA)))
})


test_that("ps_khat_threshold caps the threshold at max_threshold", {
ndraws <- c(10, 4000, 1e6)
expect_equal(ps_khat_threshold(ndraws), 1 - 1 / log10(ndraws))
expect_equal(
ps_khat_threshold(ndraws, max_threshold = 0.7),
pmin(1 - 1 / log10(ndraws), 0.7)
)
expect_true(is.na(ps_khat_threshold(NA, max_threshold = 0.7)))
})


Expand Down
Loading