Skip to content

Use the psis() Pareto k for the ratios in E_loo() - #421

Open
avehtari wants to merge 2 commits into
masterfrom
eloo-khat-from-psis
Open

avehtari wants to merge 2 commits into
masterfrom
eloo-khat-from-psis

Conversation

@avehtari

Copy link
Copy Markdown
Member

Summary

E_loo() gives an NA Pareto k where psis() gives Inf for the same log ratios. This occurs when many of the largest ratios exp(log_ratios - max(log_ratios)) underflow to 0, so that k cannot be estimated from the tail. psis() sets k to Inf in this case. E_loo() estimates k from the tail of the same ratios a second time with posterior::pareto_khat(), which gives NA with posterior 1.8.0.

Example

library(loo)
set.seed(1)
S <- 1000
lr <- c(rnorm(30), rnorm(S - 30) - 1e6)
ps <- psis(lr, r_eff = 1)
pareto_k_values(ps)
#> [1] Inf
E_loo(rnorm(S), ps, log_ratios = lr)$pareto_k
#> [1] NA     (Inf with this PR)

Changes

With this PR, E_loo() takes the k of the ratios from psis_object, where psis() stored it, and estimates k only for the tails of h * r. The k of the ratios then follows the rules of psis(). psis() gives Inf when it cannot estimate k from the tail. This occurs when the tail has fewer than 5 draws, when all tail values are equal, or when a quarter or more of the tail values are tied at the low end of the tail (as with the zeros from underflow). The PR also removes the code for posterior versions that returned a list (loo requires posterior >= 1.7.0) and the code for two NA values.

Other changes of the returned k:

  • With the log_ratios of psis_object and a finite psis() k, the k changes only by rounding (at most 3.6e-14 for example_loglik_matrix() and for one example with a heavy tail).
  • Without log_ratios, the k of the ratios is now the psis() k of the raw ratios. Before, it was the k of the smoothed weights. For the 32 columns of example_loglik_matrix() and type = "quantile", the change of the k is between -0.043 and 0.068. The k of h * r still uses the smoothed weights in this case.
  • For constant log ratios, E_loo() now gives Inf, as psis() does. Before, it gave the k of h for a continuous x, and NA for a binary x.
  • For S <= 20, the tail is shorter than 5 draws and E_loo() now gives Inf, as psis() does. Before, it gave NA (tested with S = 20).
  • For a psis object from loo_approximate_posterior(), the k of the ratios is now the k of the corrected ratios that the weights use. Before, it was the k of the log_ratios argument.
  • loo_moment_match() replaces the weights and the k in a saved psis object for the observations that it improves. E_loo() with x from the original draws is not valid for these observations, before and after this PR. With the original log_ratios, E_loo() gave the high k of the original ratios for them. It now gives the moment-matched k.

The k of h * r

This PR changes also the scaling of h * r before calling posterior::pareto_khat. E_loo() now scales h * r on the log scale so that the largest |h * r| is 1. The k of h * r then does not change when x is multiplied by a constant. On master, E_loo() scales only the ratios so that the largest ratio is 1. For 1e-307 * x and the 32 columns of example_loglik_matrix(), E_loo() on master returns k = Inf for each column, because h * r has subnormal values.

This PR does not change the rule for the k that E_loo() returns. E_loo() estimates k for the left tail and for the right tail of h * r with posterior::pareto_khat(). For a tail, k cannot be estimated when all tail values are equal, or when a quarter or more of them are tied. E_loo() returns the largest of the k of the ratios and the k values of the two tails of h * r, and leaves out a tail for which k cannot be estimated. If k cannot be estimated for either tail of h * r, E_loo() returns the k of the ratios.

The table shows examples with S = 1000 and a tail of 95 draws (posterior 1.8.0). In rows 4 to 6, only 150, 100 and 30 ratios are nonzero; the other ratios underflow to 0. Only row 6 changes with this PR.

case k of the ratios (psis()) k of h * r, left tail k of h * r, right tail E_loo() k, master E_loo() k, this PR
continuous x 0.330 0.350 0.208 0.350 0.350
count x, 62% zeros 0.330 NA 0.489 0.489 0.489
x is 0 for 95% of the draws 0.330 NA NA 0.330 0.330
150 nonzero ratios 0.338 0.765 NA 0.765 0.765
100 nonzero ratios 0.172 NA NA 0.172 0.172
30 nonzero ratios (the example above) Inf NA NA NA Inf

In the NA cells of the table, a quarter or more of the tail values are exact zeros. In rows 2 and 3, exact zeros of x cause the ties, and E_loo() uses the other tail (row 2) or the k of the ratios (row 3). In rows 4 and 5, the underflow of the ratios causes the ties. h * r then has fewer nonzero values on each side of 0 than the tail has draws. The k of the ratios is still finite in these rows. With fewer zeros in the tail, k can be estimated and is high (row 4, left tail). The E_loo() k is thus not monotone in the number of nonzero ratios. This PR does not change the rule for when k can be estimated.

The documentation of pareto_k now says where the k of the ratios comes from. The documentation of log_ratios now says that they are the log ratios that psis() used to compute psis_object.

Tests

New tests in tests/testthat/test_E_loo.R cover:

  • the underflow case and constant log ratios (continuous and binary x, type = "quantile", no log_ratios);
  • the matrix method with one underflow column;
  • a tail shorter than 5 draws;
  • a constant h * r;
  • the equality with the psis() k for quantiles and a binary x;
  • x scaled by 1e-307 (added to the existing test of the scale of x).

On master, the new tests give 16 failures. The full test suite passes (1365 expectations) with R 4.6.2 and posterior 1.8.0.

This PR was assisted by Claude.

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.80%. Comparing base (3079dca) to head (0abd75b).

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #421      +/-   ##
==========================================
- Coverage   92.88%   92.80%   -0.08%     
==========================================
  Files          31       31              
  Lines        3289     3072     -217     
==========================================
- Hits         3055     2851     -204     
+ Misses        234      221      -13     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

Copy link
Copy Markdown
Contributor

This is how benchmark results would change (along with a 95% confidence interval in relative change) if 0abd75b is merged into master:

Status Name Current Incoming Interval
✔️ loo_matrix 1.37s 1.37s [-0.81%, +0.82%]
✔️ loo_function 1.22s 1.22s [-1.63%, +1.28%]

Further explanation regarding interpretation and methodology can be found in the documentation.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants