Getting Started¶

In order to install the package, run

pip install ppi-python

Warmup: estimating the mean¶

To test your installation, you can try running the prediction-powered mean estimation algorithm on the galaxies dataset. The gold-standard labels and model predictions from the dataset will be downloaded into a folder called ./data/. The labels, \(Y\), are binary indicators of whether or not the galaxy is a spiral galaxy. The model predictions, \(\hat{Y}\), are the model’s estimated probability of whether the galaxy image has spiral arms. The inference target is \(\theta^* = \mathbb{E}[Y]\), the fraction of spiral galaxies. You will produce a confidence interval, \(\mathcal{C}^{\mathrm{PP}}_\alpha\), which contains \(\theta^*\) with probability \(1-\alpha=0.9\), i.e.,

\[\mathbb{P}\left( \theta^* \in \mathcal{C}^{\mathrm{PP}}_\alpha\right) \geq 0.9.\]

The code for this is below. It can be copy-pasted directly into the Python REPL.

# Imports
import numpy as np
from ppi_py import ppi_mean_ci
from ppi_py.datasets import load_dataset
np.random.seed(0) # For reproducibility's sake
# Download and load dataset
data = load_dataset('./data/', "galaxies")
Y_total = data["Y"]; Yhat_total = data["Yhat"]
# Set up the inference problem
alpha = 0.1 # Error rate
n = 1000 # Number of labeled data points
rand_idx = np.random.permutation(Y_total.shape[0])
Yhat = Yhat_total[rand_idx[:n]]
Y = Y_total[rand_idx[:n]]
Yhat_unlabeled = Yhat_total[n:]
# Produce the prediction-powered confidence interval
ppi_ci = ppi_mean_ci(Y, Yhat, Yhat_unlabeled, alpha=alpha)
# Print the results
print(f"theta={Y_total.mean():.3f}, CPP={ppi_ci}")

The expected results look as below \(^*\):

theta=0.259, CPP=(0.2322466630315982, 0.2626038799812829)

(\(^*\) these results were produced with numpy=1.26.0, and may differ slightly due to randomness in other environments.)

If you have reached this stage, congratulations! You have constructed a prediction-powered confidence interval. See the documentation for more usages of prediction-powered inference.

Examples¶

The package somes with a suite of examples on real data:

Proteomic Analysis with AlphaFold (alphafold.ipynb)
Galaxy Classification with Computer Vision (galaxies.ipynb)
Gene Expression Analysis with Transformers (gene_expression.ipynb)
Deforestation Analysis with Computer Vision (forest.ipynb)
Health Insurance Analysis with Boosting Trees (census_healthcare.ipynb)
Income Analysis with Boosting Trees (covariate shift) (census_income_covshift.ipynb)
Plankton Counting with Computer Vision (label shift) (plankton.ipynb)
Ballot Counting with Computer Vision (ballots.ipynb)
Income Analysis with Boosting Trees (census_income.ipynb)

Usage and Documentation¶

There is a common template that all PPI confidence intervals follow.

ppi_[ESTIMAND]_ci(X, Y, Yhat, X_unlabeled, Yhat_unlabeled, alpha=0.1)

You can replace [ESTIMAND] with the estimand of your choice. For certain estimands, not all the arguments are required, and in this case, they are omitted. For example, in the case of mean estimation, the function signature is:

ppi_mean_ci(Y, Yhat, Yhat_unlabeled, alpha=0.1)

All the prediction-powered point estimates and confidence intervals implemented so far can be imported by running from ppi_py import ppi_[ESTIMAND]_pointestimate, ppi_[ESTIMAND]_ci. For the case of the mean, one can also import the p-value as from ppi import ppi_mean_pval.

Full API documentation can be found by following the links on the left-hand sidebar of this page.