Skip to content

Don't use r_eff for computing ps_tail_length - #478

Merged
florence-bockting merged 8 commits into
masterfrom
446-tail-length-no-r-eff
Nov 16, 2026
Merged

florence-bockting merged 8 commits into
masterfrom
446-tail-length-no-r-eff

Conversation

@florence-bockting

Copy link
Copy Markdown
Collaborator

Summary

Closes #446.

ps_tail_length() now depends on ndraws only: floor(3 * sqrt(ndraws)) if ndraws > 225, and floor(ndraws / 5) otherwise.

r_eff stays in the signatures of ps_tail_length(), pareto_khat(), pareto_diags(), and pareto_smooth(), so existing calls do not break. A value other than NULL gives a warning of class posterior_warning_pareto_r_eff_ignored.

Changes:

  • A call that passes r_eff now gives a warning.
  • The rvar methods give the warning once, not once for each element.
  • pareto_pit() used r_eff = 1 before, so its results do not change.
  • New tests: r_eff does not change khat, a non-NULL value warns (also with ndraws_tail), the rvar methods warn once, and ps_tail_length() values for ndraws above and below 225.
  • I removed r_eff from 3 older test calls.
  • The ps_tail_length() docs give the formula and say that it differs from Appendix H.
  • man-roxygen/args-pareto.R now links ndraws_tail to ps_tail_length(). This also removes the old text, which said ceiling but the code uses floor.
  • The new warning class is in the list of warning classes in ?posterior.

AI assistance: I used LLMs in this PR.

Copyright and Licensing

Copyright holder: Aalto University

By submitting this pull request, the copyright holder is agreeing to
license the submitted work under the following licenses:

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

This is how benchmark results would change (along with a 95% confidence interval in relative change) if 16943c1 is merged into master:

  • ❗🐌as_draws_array: 74ms -> 76.3ms [+1.13%, +4.87%]
  • ✔️as_draws_df: 25.4ms -> 25.4ms [-1.56%, +1.56%]
  • ✔️as_draws_list: 121ms -> 121ms [-1.83%, +0.96%]
  • ✔️as_draws_matrix: 21.2ms -> 20.8ms [-4.32%, +0.85%]
  • ✔️as_draws_rvars: 59.7ms -> 60.1ms [-0.51%, +1.78%]
  • ✔️summarise_draws_100_variables: 520ms -> 520ms [-0.96%, +1.01%]
  • ✔️summarise_draws_10_variables: 58.2ms -> 58.2ms [-0.72%, +0.94%]
    Further explanation regarding interpretation and methodology can be found in the documentation.

@avehtari

avehtari commented Nov 9, 2026

Copy link
Copy Markdown
Member

A value other than NULL gives a warning of class posterior_warning_pareto_r_eff_ignored.

I assume this means there will be only one warning line (possibly with a count) in case of repeated calls?

@paul-buerkner

Copy link
Copy Markdown
Collaborator

I don't really have time to review this right now. Would @avehtari 's review be sufficient?

@florence-bockting

Copy link
Copy Markdown
Collaborator Author

I assume this means there will be only one warning line (possibly with a count) in case of repeated calls?

Without count. Its not collecting warnings, but rather catches early when r_eff is not Null and sets it then to Null, such that downstream functions do not warn anymore.

@florence-bockting
florence-bockting marked this pull request as ready for review November 9, 2026 11:03
@avehtari

avehtari commented Nov 9, 2026

Copy link
Copy Markdown
Member

but rather catches early when r_eff is not Null and sets it then to Null,

I see the warnings, but I don't see where it is set to NULL

@florence-bockting

Copy link
Copy Markdown
Collaborator Author

A value other than NULL gives a warning of class posterior_warning_pareto_r_eff_ignored.

I assume this means there will be only one warning line (possibly with a count) in case of repeated calls?

The rvar methods take r_eff as a named argument. They warn once, then call the default method for each element without r_eff. However, when you call the methods multiple times this will give you multiple warnings.

@avehtari

avehtari commented Nov 9, 2026

Copy link
Copy Markdown
Member

Is it possible that some other package calls this or some posterior function repeatedly with non-null r_eff?

@florence-bockting

Copy link
Copy Markdown
Collaborator Author

Is it possible that some other package calls this or some posterior function repeatedly with non-null r_eff?

Yes. GitHub code search finds pareto_smooth(..., r_eff = 1) in cmdstanr (R/args.R) and in iwmm (moment_match.R, update_quantities.R).
iwmm calls pareto_smooth(..., r_eff = 1) inside loops, and cmdstanr calls it once for each init.
We can add a .frequency argument for this specific warning, such that the warning appears for example only every x hours.
On the other hand we introduced for these cases the warning classes which can be used by subsequent packages (such as iwmm) to collect warnings of the same kind and then provide only one warning with pot. the number of how often the warning appeared (like this loo-PR stan-dev/loo#416).
@jgabry do you have any additional thoughts on this design aspect?

@jgabry jgabry left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, cmdstanr and I guess iwmm both pass r_eff = 1, and with r_eff = 1 the new and old formulas give the same tail length. So for those packages users will just start getting warnings even though nothing changes. And those packages can't just drop r_eff = 1 unless they require the new posterior version. (I could do that in cmdstanr if posterior is released before I do the cmdstanr 1.0 release, but when is the posterior release planned for? the cmdstanr release candidate has been out for a while so I want to release it soon.)

How about we only warn when r_eff is not NULL and not 1? Something like this (untested):

warn_r_eff_ignored <- function(r_eff) {
  if (!is.null(r_eff) && !isTRUE(all.equal(r_eff, 1))) {
    warning_no_call(
      "`r_eff` is ignored. The Pareto tail length now depends on the number of draws only.",
      class = "posterior_warning_pareto_r_eff_ignored"
    )
  }
}

Then call warn_r_eff_ignored(r_eff) everywhere that currently checks !is.null(r_eff). The r_eff docs and the warning class description would also need a small update.

Then, if we decide at some point in the future to remove r_eff entirely, we could warn for any non-NULL value?

What do you think about that?

Comment thread R/pareto_smooth.R Resolved
Comment thread R/pareto_smooth.R Resolved
@florence-bockting

Copy link
Copy Markdown
Collaborator Author

Yeah, cmdstanr and I guess iwmm both pass r_eff = 1, and with r_eff = 1 the new and old formulas give the same tail length. So for those packages users will just start getting warnings even though nothing changes. And those packages can't just drop r_eff = 1 unless they require the new posterior version. (I could do that in cmdstanr if posterior is released before I do the cmdstanr 1.0 release, but when is the posterior release planned for? the cmdstanr release candidate has been out for a while so I want to release it soon.)

How about we only warn when r_eff is not NULL and not 1? Something like this (untested):

warn_r_eff_ignored <- function(r_eff) {
  if (!is.null(r_eff) && !isTRUE(all.equal(r_eff, 1))) {
    warning_no_call(
      "`r_eff` is ignored. The Pareto tail length now depends on the number of draws only.",
      class = "posterior_warning_pareto_r_eff_ignored"
    )
  }
}

Then call warn_r_eff_ignored(r_eff) everywhere that currently checks !is.null(r_eff). The r_eff docs and the warning class description would also need a small update.

Then, if we decide at some point in the future to remove r_eff entirely, we could warn for any non-NULL value?

What do you think about that?

Thank you very much for your thoughts! I like this idea and I added it as follows:

warn_r_eff_ignored <- function(r_eff) {
  if (!is.null(r_eff) && !isTRUE(all(r_eff == 1))) {
    warning_no_call(
      "`r_eff` is ignored. The Pareto tail length now depends on the number of draws only.",
      class = "posterior_warning_pareto_r_eff_ignored"
    )
  }
}

The reason behind the slight change in the condition is to avoid warnings when a vector of ones is passed.

@florence-bockting

Copy link
Copy Markdown
Collaborator Author

@jgabry
Do you think the PR is okay now or do you see further issues that we should address?

@n-kall

n-kall commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator

I can update iwmm to depend on the newest posterior and drop r_eff when this is released. iwmm isn't on cran, and is only really used (optinally) by priorsense as far as I know

@jgabry

jgabry commented Nov 13, 2026

Copy link
Copy Markdown
Member

@jgabry Do you think the PR is okay now or do you see further issues that we should address?

I think it's ready. But maybe NEWS should have something along these lines?

The Pareto tail length in pareto_khat(), pareto_diags(), pareto_smooth(), and ps_tail_length() now depends only on the number of draws (#446, #478). For MCMC draws, results with the default r_eff = NULL change slightly because r_eff was previously estimated with ess_tail(). These functions are also faster now. r_eff is ignored, with a warning unless it is NULL or 1.

@florence-bockting

Copy link
Copy Markdown
Collaborator Author

I can update iwmm to depend on the newest posterior and drop r_eff when this is released. iwmm isn't on cran, and is only really used (optinally) by priorsense as far as I know

Thank you Noa for your support.
I think with the updated change there shouldn't be any change as we consider the case of fixing r_eff = 1 now as well.

@florence-bockting

Copy link
Copy Markdown
Collaborator Author

@jgabry Do you think the PR is okay now or do you see further issues that we should address?

I think it's ready. But maybe NEWS should have something along these lines?

The Pareto tail length in pareto_khat(), pareto_diags(), pareto_smooth(), and ps_tail_length() now depends only on the number of draws (#446, #478). For MCMC draws, results with the default r_eff = NULL change slightly because r_eff was previously estimated with ess_tail(). These functions are also faster now. r_eff is ignored, with a warning unless it is NULL or 1.

Alright, thank you.
I updated the NEWS and will merge this PR once the tests pass.

@florence-bockting
florence-bockting merged commit b80e9cb into master Nov 16, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

stop using r_eff in ps_tail_length

5 participants