Back to Claude Scientific Skills

Censoring-aware evaluation, calibration, and model selection

skills/scikit-survival/references/evaluation-metrics.md

2.55.013.8 KB
Original Source

Censoring-aware evaluation, calibration, and model selection

Verified for scikit-survival 0.28.0 on 2026-07-23. API statements below use official scikit-survival documentation; interpretation is anchored to the cited primary methodological literature.

Keep the targets distinct

  • Discrimination: can the score order subjects by event risk? Harrell C, Uno C, and cumulative/dynamic AUC.
  • Probability prediction error: how close are predicted survival probabilities to observed event-free status after censoring adjustment? Brier score and IBS.
  • Calibration: do predicted probabilities agree with observed probabilities at a specified horizon and population? Requires horizon-specific assessment; Brier score is not a pure calibration measure.
  • Cause-specific cumulative incidence: what is the absolute probability of a particular competing cause by time (t)? Standard survival metrics are not automatically cause-specific CIF metrics.
  • Clinical/decision utility: do decisions based on predictions improve outcomes under defined consequences? None of the metrics above establishes this.

Do not label one aggregate number "model accuracy" without its estimand, horizon, censoring estimator, and input type.

Prediction contracts

Higher-is-riskier scalar

Shape (n_test,). Used by:

  • concordance_index_censored
  • concordance_index_ipcw
  • cumulative_dynamic_auc (same score at each requested time)

Typical source:

python
risk = estimator.predict(X_test)

Confirm direction. Cox and ranking-only SVM predictions are higher-is-riskier. Predicted survival/log-time outputs are the opposite direction and must not be silently treated as risk.

Time-dependent risk

Shape (n_test, n_times). Accepted by cumulative_dynamic_auc, where column j is risk at times[j]. For a random survival forest:

python
functions = estimator.predict_cumulative_hazard_function(X_test)
risk_by_time = np.vstack([fn(times) for fn in functions])

Cumulative hazard is risk-oriented. Survival probability is not accepted by cumulative_dynamic_auc; do not pass it without an explicitly justified transformation.

Survival probability

Shape (n_test, n_times), values in [0, 1], non-increasing across columns. Used by:

  • brier_score
  • integrated_brier_score
python
functions = estimator.predict_survival_function(X_test)
survival_probability = np.vstack([fn(times) for fn in functions])

Metric functions expect the numeric matrix, not a list of unevaluated StepFunction objects and not a 1D risk score.

Harrell concordance

python
from sksurv.metrics import concordance_index_censored

harrell_c = concordance_index_censored(
    y_test["event"],
    y_test["time"],
    risk,
)[0]

Harrell C is the fraction of comparable pairs whose risk ordering is concordant, with handling for tied risk and event times. It measures rank discrimination over the observed follow-up mix.

Important limitations:

  • It does not assess probability calibration.
  • It does not focus on a prespecified horizon.
  • The set of comparable pairs changes under censoring.
  • Uno et al. showed the conventional estimator's limiting value can depend on the censoring distribution.

There is no universal censoring-percentage cutoff at which Harrell C suddenly becomes invalid. Report censoring, compare sensitivity to IPCW concordance, and justify the estimand.

Uno IPCW concordance

python
from sksurv.metrics import concordance_index_ipcw

uno_c = concordance_index_ipcw(
    y_train,
    y_test,
    risk,
    tau=tau,
)[0]

survival_train estimates the censoring distribution. Never pass pooled train+test outcomes or y_test as the training distribution.

tau truncates the concordance target. Choose it before seeing model performance and where the estimated training censoring survival is positive. Test follow-up must be supported by training follow-up; otherwise scikit-survival raises a ValueError.

The implementation uses Kaplan-Meier censoring weights and assumes censoring is random/independent of features. If censoring depends on covariates, this marginal weight model may be inadequate. IPCW is not a generic correction for informative censoring.

Cumulative/dynamic AUC

python
from sksurv.metrics import cumulative_dynamic_auc

auc, mean_auc = cumulative_dynamic_auc(
    y_train,
    y_test,
    risk,   # (n_test,) or (n_test, n_times)
    times,
)

At each (t), cumulative cases have an observed event by (t), while dynamic controls remain event-free after (t). IPCW handles right censoring.

Requirements:

  • times is one-dimensional, unique, and strictly increasing;
  • every value lies within test follow-up;
  • training follow-up supports test outcomes and the grid;
  • the training censoring survival is positive over the grid;
  • risk has shape (n_test,) or (n_test, n_times);
  • higher values mean higher event risk.

The returned mean_auc is not the arithmetic mean. It integrates (\widehat{AUC}(t)) over the time range, weighted by the estimated survival function.

The cumulative/dynamic definition is one time-dependent ROC estimand. State it; other incident/dynamic or competing-risk definitions answer different questions.

Brier score

python
from sksurv.metrics import brier_score, integrated_brier_score

returned_times, brier = brier_score(
    y_train,
    y_test,
    survival_probability,
    times,
)
ibs = integrated_brier_score(
    y_train,
    y_test,
    survival_probability,
    times,
)

The time-dependent Brier score is an IPC-weighted squared error between predicted survival probability and event-free status at (t). Lower is better.

Requirements:

  • predictions are survival probabilities, not risk scores;
  • prediction shape is (n_test, n_times);
  • time and training-support rules match the IPCW setting;
  • the marginal Kaplan-Meier censoring estimator's independence assumption is plausible.

IBS integrates Brier score over [times[0], times[-1]] with the implementation's time weighting. It depends on the chosen interval; IBS values from different time ranges are not directly comparable.

Compare against useful reference predictions such as a training-derived Kaplan-Meier survival curve. Do not estimate the reference curve on test outcomes.

Calibration

Brier score responds to both discrimination and calibration. A good Brier score does not prove that predicted 20% risk corresponds to 20% observed risk in every horizon or subgroup.

For a prespecified horizon:

  1. fit and tune the model without the calibration-evaluation rows;
  2. obtain event probability 1 - S(t | x) on independent validation rows;
  3. compare predictions with a censoring-aware observed probability estimate;
  4. inspect calibration-in-the-large, slope/shape, uncertainty, and sample support;
  5. repeat only for prespecified horizons/subgroups or account for multiplicity.

scikit-survival 0.28 does not expose a dedicated calibration-curve API. Do not use sklearn.calibration.calibration_curve naively on censored binary labels. If an external method is used, document its censoring assumptions and train/validation separation.

Any recalibration layer is another learned model. Fit it on a calibration split or inner resampling, then assess on independent data.

Safe time-grid construction

A pragmatic fold-specific grid:

python
import numpy as np
from sksurv.nonparametric import CensoringDistributionEstimator

test_time = y_test["time"]
train_time = y_train["time"]

lower = np.quantile(test_time, 0.10)
upper = min(
    np.quantile(test_time, 0.80),
    np.nextafter(train_time.max(), -np.inf),
)
times = np.linspace(lower, upper, 50)

if not (test_time.min() < times[0] < times[-1] < test_time.max()):
    raise ValueError("grid is outside test follow-up")
if not test_time.max() < train_time.max():
    raise ValueError("test follow-up exceeds training support")

censoring = CensoringDistributionEstimator().fit(y_train)
if np.any(censoring.predict_proba(times) <= 0):
    raise ValueError("training censoring survival reaches zero on grid")

Quantiles are an operational example, not a scientific default. Prefer prespecified meaningful horizons, then verify fold support. Do not select a grid because it maximizes a metric.

The bundled evaluator rejects unsupported grids and shape/type confusion:

bash
python skills/scikit-survival/scripts/evaluate_survival_metrics.py \
  --input predictions.npz \
  --output metrics-summary.json

Its NPZ contract is:

  • train_event, train_time
  • test_event, test_time
  • times
  • risk
  • optional survival

Archives are loaded with allow_pickle=False.

Scorer wrappers and model selection

Survival estimators' default .score() is Harrell concordance. For other targets, scikit-survival provides estimator wrappers:

  • as_concordance_index_ipcw_scorer(estimator, tau=None, tied_tol=...)
  • as_cumulative_dynamic_auc_scorer(estimator, times, tied_tol=...)
  • as_integrated_brier_score_scorer(estimator, times)

Correct pattern:

python
from sklearn.model_selection import GridSearchCV
from sksurv.metrics import as_integrated_brier_score_scorer

wrapped = as_integrated_brier_score_scorer(
    estimator,
    times=inner_times,
)
search = GridSearchCV(
    wrapped,
    {"estimator__model__max_depth": [1, 2, 4]},
    cv=inner_splits,
)
search.fit(X_outer_train, y_outer_train)

The wrapper:

  • is the estimator supplied to GridSearchCV;
  • stores its fitted estimator in estimator_;
  • exposes nested parameters under estimator__...;
  • fits metric state, including training survival information, from each fit fold;
  • negates IBS so larger wrapper score remains better.

Do not write scoring=as_integrated_brier_score_scorer(times); the constructor requires an estimator.

Nested CV protocol

For tuned performance:

text
for each outer split:
    outer_train, outer_valid
    for each inner split within outer_train:
        fit preprocessing + model + censoring-dependent scorer state
        select hyperparameters
    refit selected pipeline on all outer_train
    evaluate once on outer_valid using outer_train censoring distribution
aggregate outer scores and uncertainty

Every fold needs its own valid tau/time grid. A single global grid derived from all outcome times leaks outer-validation support information and may fail when an outer training fold has shorter follow-up.

After protocol evaluation, tune on all development data and evaluate once on an untouched final test set. Do not report the best inner-CV score as generalization performance.

Competing risks

If events have causes:

  • all-event risk combines causes and is not cause-specific discrimination;
  • treating other causes as censored defines a cause-specific hazard analysis;
  • a cause-specific CIF is an absolute probability accounting for all causes;
  • standard Brier/AUC functions here are documented for right-censored single-event outcomes, not a complete competing-risk prediction evaluation.

State event coding and use methods designed for the cause-specific estimand. See competing-risks.md.

Reporting checklist

  • split and nesting protocol;
  • event/censoring counts per evaluation set;
  • metric definition and score direction;
  • tau and exact time grid/range;
  • source of censoring weights;
  • prediction shape and whether values are risks or survival probabilities;
  • uncertainty across independent outer folds or bootstrap resamples;
  • calibration assessment separate from discrimination;
  • handling of competing causes;
  • no claim of clinical utility from metrics alone.

Primary methodological literature

Official API sources

Checked 2026-07-23: