Repository navigation
Don't use r_eff for computing ps_tail_length - #478
Conversation
|
This is how benchmark results would change (along with a 95% confidence interval in relative change) if 16943c1 is merged into master:
|
I assume this means there will be only one warning line (possibly with a count) in case of repeated calls? |
|
I don't really have time to review this right now. Would @avehtari 's review be sufficient? |
Without count. Its not collecting warnings, but rather catches early when r_eff is not Null and sets it then to Null, such that downstream functions do not warn anymore. |
I see the warnings, but I don't see where it is set to NULL |
The rvar methods take r_eff as a named argument. They warn once, then call the default method for each element without r_eff. However, when you call the methods multiple times this will give you multiple warnings. |
|
Is it possible that some other package calls this or some posterior function repeatedly with non-null r_eff? |
Yes. GitHub code search finds pareto_smooth(..., r_eff = 1) in cmdstanr (R/args.R) and in iwmm (moment_match.R, update_quantities.R). |
jgabry
left a comment
There was a problem hiding this comment.
Yeah, cmdstanr and I guess iwmm both pass r_eff = 1, and with r_eff = 1 the new and old formulas give the same tail length. So for those packages users will just start getting warnings even though nothing changes. And those packages can't just drop r_eff = 1 unless they require the new posterior version. (I could do that in cmdstanr if posterior is released before I do the cmdstanr 1.0 release, but when is the posterior release planned for? the cmdstanr release candidate has been out for a while so I want to release it soon.)
How about we only warn when r_eff is not NULL and not 1? Something like this (untested):
warn_r_eff_ignored <- function(r_eff) {
if (!is.null(r_eff) && !isTRUE(all.equal(r_eff, 1))) {
warning_no_call(
"`r_eff` is ignored. The Pareto tail length now depends on the number of draws only.",
class = "posterior_warning_pareto_r_eff_ignored"
)
}
}Then call warn_r_eff_ignored(r_eff) everywhere that currently checks !is.null(r_eff). The r_eff docs and the warning class description would also need a small update.
Then, if we decide at some point in the future to remove r_eff entirely, we could warn for any non-NULL value?
What do you think about that?
Thank you very much for your thoughts! I like this idea and I added it as follows: warn_r_eff_ignored <- function(r_eff) {
if (!is.null(r_eff) && !isTRUE(all(r_eff == 1))) {
warning_no_call(
"`r_eff` is ignored. The Pareto tail length now depends on the number of draws only.",
class = "posterior_warning_pareto_r_eff_ignored"
)
}
}The reason behind the slight change in the condition is to avoid warnings when a vector of ones is passed. |
|
@jgabry |
|
I can update iwmm to depend on the newest posterior and drop r_eff when this is released. iwmm isn't on cran, and is only really used (optinally) by priorsense as far as I know |
I think it's ready. But maybe NEWS should have something along these lines?
|
Thank you Noa for your support. |
Alright, thank you. |
Summary
Closes #446.
ps_tail_length()now depends onndrawsonly:floor(3 * sqrt(ndraws))ifndraws > 225, andfloor(ndraws / 5)otherwise.r_effstays in the signatures ofps_tail_length(),pareto_khat(),pareto_diags(), andpareto_smooth(), so existing calls do not break. A value other thanNULLgives a warning of classposterior_warning_pareto_r_eff_ignored.Changes:
r_effnow gives a warning.rvarmethods give the warning once, not once for each element.pareto_pit()usedr_eff = 1before, so its results do not change.r_effdoes not change khat, a non-NULLvalue warns (also withndraws_tail), thervarmethods warn once, andps_tail_length()values forndrawsabove and below 225.r_efffrom 3 older test calls.ps_tail_length()docs give the formula and say that it differs from Appendix H.man-roxygen/args-pareto.Rnow linksndraws_tailtops_tail_length(). This also removes the old text, which saidceilingbut the code usesfloor.?posterior.AI assistance: I used LLMs in this PR.
Copyright and Licensing
Copyright holder: Aalto University
By submitting this pull request, the copyright holder is agreeing to
license the submitted work under the following licenses: