Accelerated oblique random survival forests

class: center, middle, inverse, title-slide

# Accelerated oblique random survival forests
### Byron C. Jaeger & Nicholas Pajewski
### Last updated 2022-07-14

---

.medium { font-size: 25px; }
.mediumer { font-size: 40px; }
.large { font-size: 80px; }
.larger { font-size: 160px; }
.huge { font-size: 240px; }

.rainbow-text {
  background-image: linear-gradient(to left, violet, indigo, blue, green, yellow, orange, red);   -webkit-background-clip: text;
  color: transparent;
}

</style>

## Overview

- Random forests (axis based and oblique)

- Accelerating the oblique RSF

- Benchmark of prediction accuracy

- Benchmark of variable importance

- R package & website

---
class: center, middle, inverse

# .larger[Random forests] 
## (axis based and oblique)

---
background-image: url(img/penguins.png)
background-size: 45%
background-position: 85% 72.5%

## Decision trees

- Frequently used in supervised learning.

- Partitions the space of predictor variables.

- Can be used for classification, regression, and survival analysis.

.pull-left[
.medium[_Demo_:] Axis-based and oblique decision trees for classification of penguin species (chinstrap, gentoo, or adelie) based on bill and flipper length.1
]

.footnote[
1Data were collected and made available by [Dr. Kristen Gorman](https://www.uaf.edu/cfos/people/faculty/detail/kristen-gorman.php) and the [Palmer Station](https://pal.lternet.edu/), a member of the [Long Term Ecological Research Network](https://lternet.edu/).
]

---

Dimensions for Adelie, Chinstrap and Gentoo Penguins at Palmer Station

---

Partition all the penguins into flipper length < 207 or ≥ 207 mm

---

Partition penguins on the left side into into bill length < 43 or ≥ 43 mm

---

With oblique splits, partitions do not need to be rectangles

---

## Random survival forests (RSFs)

1. Breiman developed the random forest, a large set of decision trees injected with randomness.1, 2

.footnote[1Breiman, Leo. "Bagging predictors." Machine learning 24.2 (1996): 123-140. 2Breiman, Leo. "Random forests." Machine learning 45.1 (2001): 5-32.]

---

## Random survival forests (RSFs)

1. Breiman developed the random forest, a large set of decision trees injected with randomness.1, 2

1. Hothorn and, separately, Ishwaran developed extensions of the random forest for survival outcomes.3, 4

Specifically,

- Hothorn developed the conditional inference forest (CIF)

- Ishwaran developed the random survival forest (RSF)

.footnote[3Hothorn, Torsten, et al. "Unbiased recursive partitioning: A conditional inference framework." Journal of Computational and Graphical statistics 15.3 (2006): 651-674. 4 Ishwaran, Hemant, et al. "Random survival forests." Annals of Applied Statistics 2.3 (2008): 841-860.]

---

Each leaf in the RSF contains a Kaplan-Meier estimate of survival. In the CIF, weights are applied based on sample size.

---

## Random survival forests (RSFs)

1. Breiman developed the random forest, a large set of decision trees injected with randomness.1, 2

1. Hothorn and, separately, Ishwaran developed extensions of the random forest to engage with survival outcomes (CIF and RSF).3, 4

1. Zhou developed a rotation survival forest and Wang developed a survival forest with an extended predictor space.5, 6

Both Zhou and Wang's extensions were based on the CIF

.footnote[5Zhou L, et al. "Rotation survival forest for right censored data." PeerJ. 2015 Jun 11;3:e1009. 6 Wang H, et al. "Random survival forest with space extensions for censored data." Artif Intell Med. 2017 Jun;79:52-61.]

---

## Random survival forests (RSFs)

1. Breiman developed the random forest, a large set of decision trees injected with randomness.1, 2

1. Hothorn and, separately, Ishwaran developed extensions of the random forest to engage with survival outcomes (CIF and RSF).3, 4

1. Zhou developed a rotation survival forest and Wang developed a survival forest with an extended predictor based on the CIF.5, 6

1. Jaeger developed the oblique RSF, which used penalized regression to find oblique splits.7

Jaeger showed in general benchmarks that the oblique RSF had higher prediction accuracy than axis-based RSFs. However, the obliqueRSF R package is hundreds of times slower than standard R packages for axis based RSFs.

.footnote[7Jaeger BC, et al. "Oblique random survival forests." The Annals of Applied Statistics 13.3 (2019): 1847-1883.]

---
class: center, middle
background-image: url("img/meme_slow_R.jpg")
background-size: contain

---
class: center, middle, inverse

# .larger[Accelerating the oblique RSF]

---

## Accelerating the oblique RSF

We identify linear combinations of predictor variables in non-leaf nodes by applying Newton Raphson scoring to the partial likelihood function of the Cox regression model: `$$L(\beta) = \prod_{i=1}^m \frac{e^{x_{j(i)}^{'} \beta}}{\sum_{j \in R_i} e^{x_j^{'} \beta}}$$`

- `$x_i$` is a vector of predictors values.

- `$R_i$` is the set of indices, `$j$`, with `$T_j \geq t_i$` (i.e., those still at risk at time `$t_i$`)

+ `$T_i$` is the event time if an event occurred and last point of contact otherwise.

+ `$t_1 < \, \ldots \, < t_m$` are the `$m$` unique event times in the training data.

- `$j(i)$` is the index of the observation for which an event occurred at time `$t_i$`.

---

## Newton Raphson scoring

Estimated regression coefficients `$\hat{\beta}$` are updated in each step based on their first derivative, `$U(\hat{\beta})$`, and second derivative, `$H(\hat{\beta})$`:
`$$\hat{\beta}^{k+1} =  \hat{\beta}^{k} + U(\hat{\beta} = \hat{\beta}^{k})\, H^{-1}(\hat{\beta} = \hat{\beta}^{k})$$`

For statistical inference, iterate until a convergence threshold is met.

For identifying linear combination of predictors in the oblique RSF, .mediumer[🤷]

- `aorsf-fast` completes one iteration.

- `aorsf-cph` iterates until convergence or 15 iterations.

---

## Negation variable importance (VI)

Negation VI is similar to permutation VI in that it measures how much a model’s prediction error increases when a variable’s role in the model is de-stabilized.

- Negation VI measures the increase in an oblique RF's prediction error after flipping the sign of all coefficients linked to a variable.

- As the magnitude of a coefficient increases, so does the probability that negating it will change the oblique RF's predictions.

- Since the coefficients in each non-leaf node of an oblique RFs are adjusted for the accompanying predictors, negation VI may provide better estimation of VI in the presence of correlated variables compared to standard VI techniques.

---
class: center, middle, inverse

# .larger[Prediction Benchmark]

---

## Learners

__Oblique RSFs__:

- _aorsf-fast_: the fast version of aorsf

- _aorsf-cph_: the less fast but still pretty fast version of aorsf

- _aorsf-random_: randomized coefficients (Breiman's idea)

- _aorsf-net_: aorsf's copy of obliqueRSF

- _obliqueRSF-net_: oblique RSF using penalized cox regression (the original)

---

## Learners

__Oblique RSFs__:

- .rainbow-text[_aorsf-fast_]: this is the only oblique RSF we show in the racing model plots.

- _aorsf-cph_: the less fast but still pretty fast version of aorsf

- _aorsf-random_: randomized coefficients (Breiman's idea)

- _aorsf-net_: aorsf's copy of obliqueRSF

- _obliqueRSF-net_: oblique RSF using penalized cox regression (the original)

---

## Learners

__Axis based RSFs__:

- _cif-standard_: standard CIF

- _cif-extension_: CIF with space extension

- _cif-rotate_: CIF with rotation

- _rsf-standard_: standard RSF

- _ranger-extratrees_: RSF with extremely randomized trees

---

## Learners

__Other__:

- _glmnet-cox_: Penalized Cox regression model

- _nn-cox_: Cox neural network with time-varying effects

- _xgboost-cox_: Boosted trees fitted to Cox log likelihood

- _xgboost-aft_: Boosted trees fitted to accelerated failure time.

---

## Data sets

A total of 31 risk prediction tasks in 18 data sets were analyzed.

- number of observations ranged from 137 to 17549 (median = 2231)

- number of predictors ranged from 7 to 1692 (median = 41)

- % censored ranged from 5 to 98 (median = 78)

(Full table is shown in the bonus slides)

---

## Evaluation

We measured performance of each learner with:

- index of prediction accuracy (IPA); __higher__ is 👍

- time-dependent concordance (C)-statistic; __higher__ is 👍

- total time to fit a model and compute predictions; __lower__ is 👍

---

## Evaluation

To estimate overall performance differences:

Step 1: Collect IPA and C-statistic values:

- For each of the 31 risk prediction problems,

+ split the corresponding data into a 50/50 train/test split
    + fit each learner to the training set
    + evaluate each learner's predictions in the testing set
    + repeat 25 times

Step 2: Fit a Bayesian linear mixed model to analyze posterior expected differences in performance:

`$$\text{metric} = \widehat{\gamma} \cdot \text{learner} + (1\,|\, \text{data/run})$$`

- `run` refers to the specific train/test split of `data`

- `metric` is either the IPA or the time-dependent C-statistic.

---

---

---

---

---

---
class: center, middle, inverse

# .larger[Variable Importance Benchmark]

---

## Variable importance techniques

- `randomForestSRC-permutation` - the classic approach proposed by Leo Breiman

- `aorsf-negation` - the new approach I am proposing

- `aorsf-anova` - The only (?) other variable importance technique for oblique random forests

- `aorsf-shap` - SHAP values from an accelerated oblique RSF model

- `xgboost-shap` - SHAP values from an xgboost model (gold standard)

- `xgboost-gain` - Gain importance from an xgboost model (classic)

---

## Simulated data

We simulated five classes of predictor variables:

- __junk__: no relationship with the outcome.

- __main__: linearly related to the outcome

- __non-linear__: sinusoidal relationship to the outcome

- __combination__: latent variables formed from combinations of the combination variables have a linear relationship to the outcome.

- __interactions__: related to the outcome by multiplicative interaction with one other variable.

---

## Evaluation

We compute a C-statistic for each VI technique based on its discrimination between relevant and irrelevant variables.

- Treat each variable like a binary outcome

+ The outcome is 1 if the variable is relevant
    + The outcome is 0 if the variable is not relevant

- Treat VI as if it were a ‘prediction’ for the status of the variable.

---

<div id="htmlwidget-62aebc210c2350315e7c" style="width:800px;height:650px;" class="plotly html-widget"></div>
<script type="application/json" data-for="htmlwidget-62aebc210c2350315e7c">{"x":{"visdat":{"3d2c57ee5a75":["function () ","plotlyVisDat"]},"cur_data":"3d2c57ee5a75","attrs":{"3d2c57ee5a75":{"alpha_stroke":1,"sizes":[10,100],"spans":[1,20],"type":"bar","x":["0","0","0","0","0","0","0.15","0.15","0.15","0.15","0.15","0.15","0.3","0.3","0.3","0.3","0.3","0.3"],"y":[0.889781481481482,0.867022222222222,0.842333333333333,0.830811111111111,0.743807407407407,0.743214814814815,0.741511111111111,0.711262962962963,0.704244444444444,0.665148148148148,0.645577777777778,0.599733333333333,0.676914814814815,0.658433333333333,0.656959259259259,0.62217037037037,0.613777777777778,0.570214814814815],"color":["aorsf-negate","aorsf-anova","xgboost-shap","aorsf-permute","rfsrc-permutation","xgboost-gain","aorsf-negate","aorsf-permute","aorsf-anova","xgboost-shap","rfsrc-permutation","xgboost-gain","aorsf-negate","aorsf-anova","aorsf-permute","xgboost-shap","rfsrc-permutation","xgboost-gain"],"colors":"PuRd","text":["0.89","0.87","0.84","0.83","0.74","0.74","0.74","0.71","0.70","0.67","0.65","0.60","0.68","0.66","0.66","0.62","0.61","0.57"],"textposition":"outside","hoverinfo":"text","hovertext":["aorsf-negate Mean computation time: 12 seconds C-statistic by variable class: Main effect: 1.0 Non-linear effect: 0.89 Combination effect: 0.96 Interactions: w/main effect: 0.62 w/non-linear effect: 0.54 w/combination effect: 0.58","aorsf-anova Mean computation time: 1.3 seconds C-statistic by variable class: Main effect: 1.0 Non-linear effect: 0.84 Combination effect: 0.92 Interactions: w/main effect: 0.61 w/non-linear effect: 0.53 w/combination effect: 0.62","xgboost-shap Mean computation time: 30 seconds C-statistic by variable class: Main effect: 0.99 Non-linear effect: 0.83 Combination effect: 0.90 Interactions: w/main effect: 0.62 w/non-linear effect: 0.46 w/combination effect: 0.54","aorsf-permute Mean computation time: 14 seconds C-statistic by variable class: Main effect: 0.99 Non-linear effect: 0.78 Combination effect: 0.89 Interactions: w/main effect: 0.58 w/non-linear effect: 0.50 w/combination effect: 0.57","rfsrc-permutation Mean computation time: 20 seconds C-statistic by variable class: Main effect: 0.95 Non-linear effect: 0.68 Combination effect: 0.76 Interactions: w/main effect: 0.59 w/non-linear effect: 0.49 w/combination effect: 0.58","xgboost-gain Mean computation time: 29 seconds C-statistic by variable class: Main effect: 0.98 Non-linear effect: 0.70 Combination effect: 0.79 Interactions: w/main effect: 0.51 w/non-linear effect: 0.35 w/combination effect: 0.43","aorsf-negate Mean computation time: 12 seconds C-statistic by variable class: Main effect: 0.91 Non-linear effect: 0.70 Combination effect: 0.76 Interactions: w/main effect: 0.61 w/non-linear effect: 0.52 w/combination effect: 0.60","aorsf-permute Mean computation time: 14 seconds C-statistic by variable class: Main effect: 0.88 Non-linear effect: 0.66 Combination effect: 0.72 Interactions: w/main effect: 0.60 w/non-linear effect: 0.52 w/combination effect: 0.58","aorsf-anova Mean computation time: 1.3 seconds C-statistic by variable class: Main effect: 0.88 Non-linear effect: 0.66 Combination effect: 0.71 Interactions: w/main effect: 0.58 w/non-linear effect: 0.51 w/combination effect: 0.57","xgboost-shap Mean computation time: 24 seconds C-statistic by variable class: Main effect: 0.84 Non-linear effect: 0.64 Combination effect: 0.67 Interactions: w/main effect: 0.56 w/non-linear effect: 0.47 w/combination effect: 0.51","rfsrc-permutation Mean computation time: 20 seconds C-statistic by variable class: Main effect: 0.80 Non-linear effect: 0.59 Combination effect: 0.64 Interactions: w/main effect: 0.57 w/non-linear effect: 0.49 w/combination effect: 0.58","xgboost-gain Mean computation time: 24 seconds C-statistic by variable class: Main effect: 0.81 Non-linear effect: 0.56 Combination effect: 0.60 Interactions: w/main effect: 0.50 w/non-linear effect: 0.39 w/combination effect: 0.45","aorsf-negate Mean computation time: 11 seconds C-statistic by variable class: Main effect: 0.84 Non-linear effect: 0.62 Combination effect: 0.68 Interactions: w/main effect: 0.59 w/non-linear effect: 0.52 w/combination effect: 0.59","aorsf-anova Mean computation time: 1.2 seconds C-statistic by variable class: Main effect: 0.81 Non-linear effect: 0.61 Combination effect: 0.66 Interactions: w/main effect: 0.59 w/non-linear effect: 0.52 w/combination effect: 0.58","aorsf-permute Mean computation time: 13 seconds C-statistic by variable class: Main effect: 0.80 Non-linear effect: 0.60 Combination effect: 0.65 Interactions: w/main effect: 0.61 w/non-linear effect: 0.52 w/combination effect: 0.61","xgboost-shap Mean computation time: 20 seconds C-statistic by variable class: Main effect: 0.77 Non-linear effect: 0.58 Combination effect: 0.62 Interactions: w/main effect: 0.56 w/non-linear effect: 0.48 w/combination effect: 0.55","rfsrc-permutation Mean computation time: 19 seconds C-statistic by variable class: Main effect: 0.74 Non-linear effect: 0.56 Combination effect: 0.61 Interactions: w/main effect: 0.59 w/non-linear effect: 0.49 w/combination effect: 0.59","xgboost-gain Mean computation time: 19 seconds C-statistic by variable class: Main effect: 0.74 Non-linear effect: 0.53 Combination effect: 0.56 Interactions: w/main effect: 0.52 w/non-linear effect: 0.42 w/combination effect: 0.51"],"inherit":true}},"layout":{"width":800,"height":650,"margin":{"b":40,"l":60,"t":25,"r":10},"title":{"text":"Results when N = 1000","x":0.1},"xaxis":{"domain":[0,1],"automargin":true,"title":"Max correlation among predictors","type":"category","categoryorder":"array","categoryarray":["0","0.15","0.3"]},"yaxis":{"domain":[0,1],"automargin":true,"title":"C-statistic","range":[0,1]},"legend":{"title":{"text":"VI technique<\/b>"}},"hovermode":"closest","showlegend":true},"source":"A","config":{"modeBarButtonsToAdd":["hoverclosest","hovercompare"],"showSendToCloud":false},"data":[{"type":"bar","x":["0","0.15","0.3"],"y":[0.743214814814815,0.599733333333333,0.570214814814815],"text":["0.74","0.60","0.57"],"textposition":["outside","outside","outside"],"hoverinfo":["text","text","text"],"hovertext":["xgboost-gain Mean computation time: 29 seconds C-statistic by variable class: Main effect: 0.98 Non-linear effect: 0.70 Combination effect: 0.79 Interactions: w/main effect: 0.51 w/non-linear effect: 0.35 w/combination effect: 0.43","xgboost-gain Mean computation time: 24 seconds C-statistic by variable class: Main effect: 0.81 Non-linear effect: 0.56 Combination effect: 0.60 Interactions: w/main effect: 0.50 w/non-linear effect: 0.39 w/combination effect: 0.45","xgboost-gain Mean computation time: 19 seconds C-statistic by variable class: Main effect: 0.74 Non-linear effect: 0.53 Combination effect: 0.56 Interactions: w/main effect: 0.52 w/non-linear effect: 0.42 w/combination effect: 0.51"],"name":"xgboost-gain","marker":{"color":"rgba(241,238,246,1)","line":{"color":"rgba(241,238,246,1)"}},"textfont":{"color":"rgba(241,238,246,1)"},"error_y":{"color":"rgba(241,238,246,1)"},"error_x":{"color":"rgba(241,238,246,1)"},"xaxis":"x","yaxis":"y","frame":null},{"type":"bar","x":["0","0.15","0.3"],"y":[0.743807407407407,0.645577777777778,0.613777777777778],"text":["0.74","0.65","0.61"],"textposition":["outside","outside","outside"],"hoverinfo":["text","text","text"],"hovertext":["rfsrc-permutation Mean computation time: 20 seconds C-statistic by variable class: Main effect: 0.95 Non-linear effect: 0.68 Combination effect: 0.76 Interactions: w/main effect: 0.59 w/non-linear effect: 0.49 w/combination effect: 0.58","rfsrc-permutation Mean computation time: 20 seconds C-statistic by variable class: Main effect: 0.80 Non-linear effect: 0.59 Combination effect: 0.64 Interactions: w/main effect: 0.57 w/non-linear effect: 0.49 w/combination effect: 0.58","rfsrc-permutation Mean computation time: 19 seconds C-statistic by variable class: Main effect: 0.74 Non-linear effect: 0.56 Combination effect: 0.61 Interactions: w/main effect: 0.59 w/non-linear effect: 0.49 w/combination effect: 0.59"],"name":"rfsrc-permutation","marker":{"color":"rgba(212,185,218,1)","line":{"color":"rgba(212,185,218,1)"}},"textfont":{"color":"rgba(212,185,218,1)"},"error_y":{"color":"rgba(212,185,218,1)"},"error_x":{"color":"rgba(212,185,218,1)"},"xaxis":"x","yaxis":"y","frame":null},{"type":"bar","x":["0","0.15","0.3"],"y":[0.842333333333333,0.665148148148148,0.62217037037037],"text":["0.84","0.67","0.62"],"textposition":["outside","outside","outside"],"hoverinfo":["text","text","text"],"hovertext":["xgboost-shap Mean computation time: 30 seconds C-statistic by variable class: Main effect: 0.99 Non-linear effect: 0.83 Combination effect: 0.90 Interactions: w/main effect: 0.62 w/non-linear effect: 0.46 w/combination effect: 0.54","xgboost-shap Mean computation time: 24 seconds C-statistic by variable class: Main effect: 0.84 Non-linear effect: 0.64 Combination effect: 0.67 Interactions: w/main effect: 0.56 w/non-linear effect: 0.47 w/combination effect: 0.51","xgboost-shap Mean computation time: 20 seconds C-statistic by variable class: Main effect: 0.77 Non-linear effect: 0.58 Combination effect: 0.62 Interactions: w/main effect: 0.56 w/non-linear effect: 0.48 w/combination effect: 0.55"],"name":"xgboost-shap","marker":{"color":"rgba(201,148,199,1)","line":{"color":"rgba(201,148,199,1)"}},"textfont":{"color":"rgba(201,148,199,1)"},"error_y":{"color":"rgba(201,148,199,1)"},"error_x":{"color":"rgba(201,148,199,1)"},"xaxis":"x","yaxis":"y","frame":null},{"type":"bar","x":["0","0.15","0.3"],"y":[0.867022222222222,0.704244444444444,0.658433333333333],"text":["0.87","0.70","0.66"],"textposition":["outside","outside","outside"],"hoverinfo":["text","text","text"],"hovertext":["aorsf-anova Mean computation time: 1.3 seconds C-statistic by variable class: Main effect: 1.0 Non-linear effect: 0.84 Combination effect: 0.92 Interactions: w/main effect: 0.61 w/non-linear effect: 0.53 w/combination effect: 0.62","aorsf-anova Mean computation time: 1.3 seconds C-statistic by variable class: Main effect: 0.88 Non-linear effect: 0.66 Combination effect: 0.71 Interactions: w/main effect: 0.58 w/non-linear effect: 0.51 w/combination effect: 0.57","aorsf-anova Mean computation time: 1.2 seconds C-statistic by variable class: Main effect: 0.81 Non-linear effect: 0.61 Combination effect: 0.66 Interactions: w/main effect: 0.59 w/non-linear effect: 0.52 w/combination effect: 0.58"],"name":"aorsf-anova","marker":{"color":"rgba(223,101,176,1)","line":{"color":"rgba(223,101,176,1)"}},"textfont":{"color":"rgba(223,101,176,1)"},"error_y":{"color":"rgba(223,101,176,1)"},"error_x":{"color":"rgba(223,101,176,1)"},"xaxis":"x","yaxis":"y","frame":null},{"type":"bar","x":["0","0.15","0.3"],"y":[0.830811111111111,0.711262962962963,0.656959259259259],"text":["0.83","0.71","0.66"],"textposition":["outside","outside","outside"],"hoverinfo":["text","text","text"],"hovertext":["aorsf-permute Mean computation time: 14 seconds C-statistic by variable class: Main effect: 0.99 Non-linear effect: 0.78 Combination effect: 0.89 Interactions: w/main effect: 0.58 w/non-linear effect: 0.50 w/combination effect: 0.57","aorsf-permute Mean computation time: 14 seconds C-statistic by variable class: Main effect: 0.88 Non-linear effect: 0.66 Combination effect: 0.72 Interactions: w/main effect: 0.60 w/non-linear effect: 0.52 w/combination effect: 0.58","aorsf-permute Mean computation time: 13 seconds C-statistic by variable class: Main effect: 0.80 Non-linear effect: 0.60 Combination effect: 0.65 Interactions: w/main effect: 0.61 w/non-linear effect: 0.52 w/combination effect: 0.61"],"name":"aorsf-permute","marker":{"color":"rgba(221,28,119,1)","line":{"color":"rgba(221,28,119,1)"}},"textfont":{"color":"rgba(221,28,119,1)"},"error_y":{"color":"rgba(221,28,119,1)"},"error_x":{"color":"rgba(221,28,119,1)"},"xaxis":"x","yaxis":"y","frame":null},{"type":"bar","x":["0","0.15","0.3"],"y":[0.889781481481482,0.741511111111111,0.676914814814815],"text":["0.89","0.74","0.68"],"textposition":["outside","outside","outside"],"hoverinfo":["text","text","text"],"hovertext":["aorsf-negate Mean computation time: 12 seconds C-statistic by variable class: Main effect: 1.0 Non-linear effect: 0.89 Combination effect: 0.96 Interactions: w/main effect: 0.62 w/non-linear effect: 0.54 w/combination effect: 0.58","aorsf-negate Mean computation time: 12 seconds C-statistic by variable class: Main effect: 0.91 Non-linear effect: 0.70 Combination effect: 0.76 Interactions: w/main effect: 0.61 w/non-linear effect: 0.52 w/combination effect: 0.60","aorsf-negate Mean computation time: 11 seconds C-statistic by variable class: Main effect: 0.84 Non-linear effect: 0.62 Combination effect: 0.68 Interactions: w/main effect: 0.59 w/non-linear effect: 0.52 w/combination effect: 0.59"],"name":"aorsf-negate","marker":{"color":"rgba(152,0,67,1)","line":{"color":"rgba(152,0,67,1)"}},"textfont":{"color":"rgba(152,0,67,1)"},"error_y":{"color":"rgba(152,0,67,1)"},"error_x":{"color":"rgba(152,0,67,1)"},"xaxis":"x","yaxis":"y","frame":null}],"highlight":{"on":"plotly_click","persistent":false,"dynamic":false,"selectize":false,"opacityDim":0.2,"selected":{"opacity":1},"debounce":0},"shinyEvents":["plotly_hover","plotly_click","plotly_selected","plotly_relayout","plotly_brushed","plotly_brushing","plotly_clickannotation","plotly_doubleclick","plotly_deselect","plotly_afterplot","plotly_sunburstclick"],"base_url":"https://plot.ly"},"evals":[],"jsHooks":[]}</script>

---

<div id="htmlwidget-d73352d7118645f760a8" style="width:800px;height:650px;" class="plotly html-widget"></div>
<script type="application/json" data-for="htmlwidget-d73352d7118645f760a8">{"x":{"visdat":{"3d2cb83b77":["function () ","plotlyVisDat"]},"cur_data":"3d2cb83b77","attrs":{"3d2cb83b77":{"alpha_stroke":1,"sizes":[10,100],"spans":[1,20],"type":"bar","x":["0","0","0","0","0","0","0.15","0.15","0.15","0.15","0.15","0.15","0.3","0.3","0.3","0.3","0.3","0.3"],"y":[0.945385185185185,0.941640740740741,0.9382,0.915237037037037,0.910059259259259,0.84642962962963,0.774503703703704,0.756662962962963,0.735985185185185,0.730325925925926,0.704192592592593,0.697488888888889,0.696614814814815,0.691203703703704,0.671185185185185,0.667451851851852,0.6498,0.639592592592593],"color":["xgboost-shap","aorsf-negate","aorsf-anova","aorsf-permute","xgboost-gain","rfsrc-permutation","aorsf-negate","aorsf-permute","aorsf-anova","xgboost-shap","rfsrc-permutation","xgboost-gain","aorsf-negate","aorsf-permute","aorsf-anova","xgboost-shap","rfsrc-permutation","xgboost-gain"],"colors":"PuRd","text":["0.95","0.94","0.94","0.92","0.91","0.85","0.77","0.76","0.74","0.73","0.70","0.70","0.70","0.69","0.67","0.67","0.65","0.64"],"textposition":"outside","hoverinfo":"text","hovertext":["xgboost-shap Mean computation time: 67 seconds C-statistic by variable class: Main effect: 1.0 Non-linear effect: 0.98 Combination effect: 1.0 Interactions: w/main effect: 0.92 w/non-linear effect: 0.48 w/combination effect: 0.72","aorsf-negate Mean computation time: 41 seconds C-statistic by variable class: Main effect: 1.0 Non-linear effect: 0.98 Combination effect: 1.0 Interactions: w/main effect: 0.78 w/non-linear effect: 0.51 w/combination effect: 0.73","aorsf-anova Mean computation time: 4.4 seconds C-statistic by variable class: Main effect: 1.0 Non-linear effect: 0.96 Combination effect: 0.99 Interactions: w/main effect: 0.78 w/non-linear effect: 0.53 w/combination effect: 0.75","aorsf-permute Mean computation time: 43 seconds C-statistic by variable class: Main effect: 1.0 Non-linear effect: 0.91 Combination effect: 0.98 Interactions: w/main effect: 0.72 w/non-linear effect: 0.53 w/combination effect: 0.68","xgboost-gain Mean computation time: 66 seconds C-statistic by variable class: Main effect: 1.0 Non-linear effect: 0.91 Combination effect: 0.98 Interactions: w/main effect: 0.87 w/non-linear effect: 0.34 w/combination effect: 0.64","rfsrc-permutation Mean computation time: 128 seconds C-statistic by variable class: Main effect: 1.0 Non-linear effect: 0.79 Combination effect: 0.89 Interactions: w/main effect: 0.71 w/non-linear effect: 0.53 w/combination effect: 0.64","aorsf-negate Mean computation time: 36 seconds C-statistic by variable class: Main effect: 0.94 Non-linear effect: 0.71 Combination effect: 0.79 Interactions: w/main effect: 0.67 w/non-linear effect: 0.54 w/combination effect: 0.66","aorsf-permute Mean computation time: 40 seconds C-statistic by variable class: Main effect: 0.93 Non-linear effect: 0.68 Combination effect: 0.77 Interactions: w/main effect: 0.69 w/non-linear effect: 0.51 w/combination effect: 0.67","aorsf-anova Mean computation time: 4.2 seconds C-statistic by variable class: Main effect: 0.92 Non-linear effect: 0.67 Combination effect: 0.75 Interactions: w/main effect: 0.63 w/non-linear effect: 0.52 w/combination effect: 0.62","xgboost-shap Mean computation time: 45 seconds C-statistic by variable class: Main effect: 0.90 Non-linear effect: 0.67 Combination effect: 0.73 Interactions: w/main effect: 0.71 w/non-linear effect: 0.51 w/combination effect: 0.62","rfsrc-permutation Mean computation time: 121 seconds C-statistic by variable class: Main effect: 0.87 Non-linear effect: 0.64 Combination effect: 0.70 Interactions: w/main effect: 0.66 w/non-linear effect: 0.54 w/combination effect: 0.65","xgboost-gain Mean computation time: 47 seconds C-statistic by variable class: Main effect: 0.89 Non-linear effect: 0.63 Combination effect: 0.69 Interactions: w/main effect: 0.70 w/non-linear effect: 0.46 w/combination effect: 0.62","aorsf-negate Mean computation time: 37 seconds C-statistic by variable class: Main effect: 0.87 Non-linear effect: 0.62 Combination effect: 0.69 Interactions: w/main effect: 0.65 w/non-linear effect: 0.53 w/combination effect: 0.67","aorsf-permute Mean computation time: 40 seconds C-statistic by variable class: Main effect: 0.85 Non-linear effect: 0.61 Combination effect: 0.68 Interactions: w/main effect: 0.70 w/non-linear effect: 0.52 w/combination effect: 0.69","aorsf-anova Mean computation time: 4.3 seconds C-statistic by variable class: Main effect: 0.84 Non-linear effect: 0.60 Combination effect: 0.67 Interactions: w/main effect: 0.62 w/non-linear effect: 0.53 w/combination effect: 0.62","xgboost-shap Mean computation time: 32 seconds C-statistic by variable class: Main effect: 0.82 Non-linear effect: 0.60 Combination effect: 0.65 Interactions: w/main effect: 0.69 w/non-linear effect: 0.52 w/combination effect: 0.64","rfsrc-permutation Mean computation time: 120 seconds C-statistic by variable class: Main effect: 0.79 Non-linear effect: 0.58 Combination effect: 0.64 Interactions: w/main effect: 0.64 w/non-linear effect: 0.52 w/combination effect: 0.64","xgboost-gain Mean computation time: 32 seconds C-statistic by variable class: Main effect: 0.81 Non-linear effect: 0.56 Combination effect: 0.62 Interactions: w/main effect: 0.69 w/non-linear effect: 0.47 w/combination effect: 0.66"],"inherit":true}},"layout":{"width":800,"height":650,"margin":{"b":40,"l":60,"t":25,"r":10},"title":{"text":"Results when N = 2500","x":0.1},"xaxis":{"domain":[0,1],"automargin":true,"title":"Max correlation among predictors","type":"category","categoryorder":"array","categoryarray":["0","0.15","0.3"]},"yaxis":{"domain":[0,1],"automargin":true,"title":"C-statistic","range":[0,1]},"legend":{"title":{"text":"VI technique<\/b>"}},"hovermode":"closest","showlegend":true},"source":"A","config":{"modeBarButtonsToAdd":["hoverclosest","hovercompare"],"showSendToCloud":false},"data":[{"type":"bar","x":["0","0.15","0.3"],"y":[0.910059259259259,0.697488888888889,0.639592592592593],"text":["0.91","0.70","0.64"],"textposition":["outside","outside","outside"],"hoverinfo":["text","text","text"],"hovertext":["xgboost-gain Mean computation time: 66 seconds C-statistic by variable class: Main effect: 1.0 Non-linear effect: 0.91 Combination effect: 0.98 Interactions: w/main effect: 0.87 w/non-linear effect: 0.34 w/combination effect: 0.64","xgboost-gain Mean computation time: 47 seconds C-statistic by variable class: Main effect: 0.89 Non-linear effect: 0.63 Combination effect: 0.69 Interactions: w/main effect: 0.70 w/non-linear effect: 0.46 w/combination effect: 0.62","xgboost-gain Mean computation time: 32 seconds C-statistic by variable class: Main effect: 0.81 Non-linear effect: 0.56 Combination effect: 0.62 Interactions: w/main effect: 0.69 w/non-linear effect: 0.47 w/combination effect: 0.66"],"name":"xgboost-gain","marker":{"color":"rgba(241,238,246,1)","line":{"color":"rgba(241,238,246,1)"}},"textfont":{"color":"rgba(241,238,246,1)"},"error_y":{"color":"rgba(241,238,246,1)"},"error_x":{"color":"rgba(241,238,246,1)"},"xaxis":"x","yaxis":"y","frame":null},{"type":"bar","x":["0","0.15","0.3"],"y":[0.84642962962963,0.704192592592593,0.6498],"text":["0.85","0.70","0.65"],"textposition":["outside","outside","outside"],"hoverinfo":["text","text","text"],"hovertext":["rfsrc-permutation Mean computation time: 128 seconds C-statistic by variable class: Main effect: 1.0 Non-linear effect: 0.79 Combination effect: 0.89 Interactions: w/main effect: 0.71 w/non-linear effect: 0.53 w/combination effect: 0.64","rfsrc-permutation Mean computation time: 121 seconds C-statistic by variable class: Main effect: 0.87 Non-linear effect: 0.64 Combination effect: 0.70 Interactions: w/main effect: 0.66 w/non-linear effect: 0.54 w/combination effect: 0.65","rfsrc-permutation Mean computation time: 120 seconds C-statistic by variable class: Main effect: 0.79 Non-linear effect: 0.58 Combination effect: 0.64 Interactions: w/main effect: 0.64 w/non-linear effect: 0.52 w/combination effect: 0.64"],"name":"rfsrc-permutation","marker":{"color":"rgba(212,185,218,1)","line":{"color":"rgba(212,185,218,1)"}},"textfont":{"color":"rgba(212,185,218,1)"},"error_y":{"color":"rgba(212,185,218,1)"},"error_x":{"color":"rgba(212,185,218,1)"},"xaxis":"x","yaxis":"y","frame":null},{"type":"bar","x":["0","0.15","0.3"],"y":[0.945385185185185,0.730325925925926,0.667451851851852],"text":["0.95","0.73","0.67"],"textposition":["outside","outside","outside"],"hoverinfo":["text","text","text"],"hovertext":["xgboost-shap Mean computation time: 67 seconds C-statistic by variable class: Main effect: 1.0 Non-linear effect: 0.98 Combination effect: 1.0 Interactions: w/main effect: 0.92 w/non-linear effect: 0.48 w/combination effect: 0.72","xgboost-shap Mean computation time: 45 seconds C-statistic by variable class: Main effect: 0.90 Non-linear effect: 0.67 Combination effect: 0.73 Interactions: w/main effect: 0.71 w/non-linear effect: 0.51 w/combination effect: 0.62","xgboost-shap Mean computation time: 32 seconds C-statistic by variable class: Main effect: 0.82 Non-linear effect: 0.60 Combination effect: 0.65 Interactions: w/main effect: 0.69 w/non-linear effect: 0.52 w/combination effect: 0.64"],"name":"xgboost-shap","marker":{"color":"rgba(201,148,199,1)","line":{"color":"rgba(201,148,199,1)"}},"textfont":{"color":"rgba(201,148,199,1)"},"error_y":{"color":"rgba(201,148,199,1)"},"error_x":{"color":"rgba(201,148,199,1)"},"xaxis":"x","yaxis":"y","frame":null},{"type":"bar","x":["0","0.15","0.3"],"y":[0.9382,0.735985185185185,0.671185185185185],"text":["0.94","0.74","0.67"],"textposition":["outside","outside","outside"],"hoverinfo":["text","text","text"],"hovertext":["aorsf-anova Mean computation time: 4.4 seconds C-statistic by variable class: Main effect: 1.0 Non-linear effect: 0.96 Combination effect: 0.99 Interactions: w/main effect: 0.78 w/non-linear effect: 0.53 w/combination effect: 0.75","aorsf-anova Mean computation time: 4.2 seconds C-statistic by variable class: Main effect: 0.92 Non-linear effect: 0.67 Combination effect: 0.75 Interactions: w/main effect: 0.63 w/non-linear effect: 0.52 w/combination effect: 0.62","aorsf-anova Mean computation time: 4.3 seconds C-statistic by variable class: Main effect: 0.84 Non-linear effect: 0.60 Combination effect: 0.67 Interactions: w/main effect: 0.62 w/non-linear effect: 0.53 w/combination effect: 0.62"],"name":"aorsf-anova","marker":{"color":"rgba(223,101,176,1)","line":{"color":"rgba(223,101,176,1)"}},"textfont":{"color":"rgba(223,101,176,1)"},"error_y":{"color":"rgba(223,101,176,1)"},"error_x":{"color":"rgba(223,101,176,1)"},"xaxis":"x","yaxis":"y","frame":null},{"type":"bar","x":["0","0.15","0.3"],"y":[0.915237037037037,0.756662962962963,0.691203703703704],"text":["0.92","0.76","0.69"],"textposition":["outside","outside","outside"],"hoverinfo":["text","text","text"],"hovertext":["aorsf-permute Mean computation time: 43 seconds C-statistic by variable class: Main effect: 1.0 Non-linear effect: 0.91 Combination effect: 0.98 Interactions: w/main effect: 0.72 w/non-linear effect: 0.53 w/combination effect: 0.68","aorsf-permute Mean computation time: 40 seconds C-statistic by variable class: Main effect: 0.93 Non-linear effect: 0.68 Combination effect: 0.77 Interactions: w/main effect: 0.69 w/non-linear effect: 0.51 w/combination effect: 0.67","aorsf-permute Mean computation time: 40 seconds C-statistic by variable class: Main effect: 0.85 Non-linear effect: 0.61 Combination effect: 0.68 Interactions: w/main effect: 0.70 w/non-linear effect: 0.52 w/combination effect: 0.69"],"name":"aorsf-permute","marker":{"color":"rgba(221,28,119,1)","line":{"color":"rgba(221,28,119,1)"}},"textfont":{"color":"rgba(221,28,119,1)"},"error_y":{"color":"rgba(221,28,119,1)"},"error_x":{"color":"rgba(221,28,119,1)"},"xaxis":"x","yaxis":"y","frame":null},{"type":"bar","x":["0","0.15","0.3"],"y":[0.941640740740741,0.774503703703704,0.696614814814815],"text":["0.94","0.77","0.70"],"textposition":["outside","outside","outside"],"hoverinfo":["text","text","text"],"hovertext":["aorsf-negate Mean computation time: 41 seconds C-statistic by variable class: Main effect: 1.0 Non-linear effect: 0.98 Combination effect: 1.0 Interactions: w/main effect: 0.78 w/non-linear effect: 0.51 w/combination effect: 0.73","aorsf-negate Mean computation time: 36 seconds C-statistic by variable class: Main effect: 0.94 Non-linear effect: 0.71 Combination effect: 0.79 Interactions: w/main effect: 0.67 w/non-linear effect: 0.54 w/combination effect: 0.66","aorsf-negate Mean computation time: 37 seconds C-statistic by variable class: Main effect: 0.87 Non-linear effect: 0.62 Combination effect: 0.69 Interactions: w/main effect: 0.65 w/non-linear effect: 0.53 w/combination effect: 0.67"],"name":"aorsf-negate","marker":{"color":"rgba(152,0,67,1)","line":{"color":"rgba(152,0,67,1)"}},"textfont":{"color":"rgba(152,0,67,1)"},"error_y":{"color":"rgba(152,0,67,1)"},"error_x":{"color":"rgba(152,0,67,1)"},"xaxis":"x","yaxis":"y","frame":null}],"highlight":{"on":"plotly_click","persistent":false,"dynamic":false,"selectize":false,"opacityDim":0.2,"selected":{"opacity":1},"debounce":0},"shinyEvents":["plotly_hover","plotly_click","plotly_selected","plotly_relayout","plotly_brushed","plotly_brushing","plotly_clickannotation","plotly_doubleclick","plotly_deselect","plotly_afterplot","plotly_sunburstclick"],"base_url":"https://plot.ly"},"evals":[],"jsHooks":[]}</script>

---

## Using the accelerated oblique RSF

(https://bcjaeger.github.io/aorsf/)

---

# Thank you!

---
class: center, middle

# .huge[.rainbow-text[BONUS ROUND]]

---

# Data sets

A total of 31 risk prediction tasks in 18 data sets were analyzed.

This table is continued on the next 3 slides.

<div id="cobmdqrxts" style="overflow-x:auto;overflow-y:auto;width:auto;height:auto;">
<style>html {
 font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Helvetica Neue', 'Fira Sans', 'Droid Sans', Arial, sans-serif;
}

#cobmdqrxts .gt_table {
  display: table;
  border-collapse: collapse;
  margin-left: auto;
  margin-right: auto;
  color: #333333;
  font-size: 16px;
  font-weight: normal;
  font-style: normal;
  background-color: #FFFFFF;
  width: auto;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #A8A8A8;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #A8A8A8;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
}

#cobmdqrxts .gt_heading {
  background-color: #FFFFFF;
  text-align: center;
  border-bottom-color: #FFFFFF;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#cobmdqrxts .gt_title {
  color: #333333;
  font-size: 125%;
  font-weight: initial;
  padding-top: 4px;
  padding-bottom: 4px;
  padding-left: 5px;
  padding-right: 5px;
  border-bottom-color: #FFFFFF;
  border-bottom-width: 0;
}

#cobmdqrxts .gt_subtitle {
  color: #333333;
  font-size: 85%;
  font-weight: initial;
  padding-top: 0;
  padding-bottom: 6px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-color: #FFFFFF;
  border-top-width: 0;
}

#cobmdqrxts .gt_bottom_border {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#cobmdqrxts .gt_col_headings {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#cobmdqrxts .gt_col_heading {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  padding-left: 5px;
  padding-right: 5px;
  overflow-x: hidden;
}

#cobmdqrxts .gt_column_spanner_outer {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  padding-top: 0;
  padding-bottom: 0;
  padding-left: 4px;
  padding-right: 4px;
}

#cobmdqrxts .gt_column_spanner_outer:first-child {
  padding-left: 0;
}

#cobmdqrxts .gt_column_spanner_outer:last-child {
  padding-right: 0;
}

#cobmdqrxts .gt_column_spanner {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 5px;
  overflow-x: hidden;
  display: inline-block;
  width: 100%;
}

#cobmdqrxts .gt_group_heading {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
}

#cobmdqrxts .gt_empty_group_heading {
  padding: 0.5px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: middle;
}

#cobmdqrxts .gt_from_md > :first-child {
  margin-top: 0;
}

#cobmdqrxts .gt_from_md > :last-child {
  margin-bottom: 0;
}

#cobmdqrxts .gt_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  margin: 10px;
  border-top-style: solid;
  border-top-width: 1px;
  border-top-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
  overflow-x: hidden;
}

#cobmdqrxts .gt_stub {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-right-style: solid;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  padding-left: 5px;
  padding-right: 5px;
}

#cobmdqrxts .gt_stub_row_group {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-right-style: solid;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  padding-left: 5px;
  padding-right: 5px;
  vertical-align: top;
}

#cobmdqrxts .gt_row_group_first td {
  border-top-width: 2px;
}

#cobmdqrxts .gt_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#cobmdqrxts .gt_first_summary_row {
  border-top-style: solid;
  border-top-color: #D3D3D3;
}

#cobmdqrxts .gt_first_summary_row.thick {
  border-top-width: 2px;
}

#cobmdqrxts .gt_last_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#cobmdqrxts .gt_grand_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#cobmdqrxts .gt_first_grand_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: double;
  border-top-width: 6px;
  border-top-color: #D3D3D3;
}

#cobmdqrxts .gt_striped {
  background-color: rgba(128, 128, 128, 0.05);
}

#cobmdqrxts .gt_table_body {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#cobmdqrxts .gt_footnotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#cobmdqrxts .gt_footnote {
  margin: 0px;
  font-size: 90%;
  padding-left: 4px;
  padding-right: 4px;
  padding-left: 5px;
  padding-right: 5px;
}

#cobmdqrxts .gt_sourcenotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#cobmdqrxts .gt_sourcenote {
  font-size: 90%;
  padding-top: 4px;
  padding-bottom: 4px;
  padding-left: 5px;
  padding-right: 5px;
}

#cobmdqrxts .gt_left {
  text-align: left;
}

#cobmdqrxts .gt_center {
  text-align: center;
}

#cobmdqrxts .gt_right {
  text-align: right;
  font-variant-numeric: tabular-nums;
}

#cobmdqrxts .gt_font_normal {
  font-weight: normal;
}

#cobmdqrxts .gt_font_bold {
  font-weight: bold;
}

#cobmdqrxts .gt_font_italic {
  font-style: italic;
}

#cobmdqrxts .gt_super {
  font-size: 65%;
}

#cobmdqrxts .gt_footnote_marks {
  font-style: italic;
  font-weight: normal;
  font-size: 75%;
  vertical-align: 0.4em;
}

#cobmdqrxts .gt_asterisk {
  font-size: 100%;
  vertical-align: 0;
}

#cobmdqrxts .gt_slash_mark {
  font-size: 0.7em;
  line-height: 0.7em;
  vertical-align: 0.15em;
}

#cobmdqrxts .gt_fraction_numerator {
  font-size: 0.6em;
  line-height: 0.6em;
  vertical-align: 0.45em;
}

#cobmdqrxts .gt_fraction_denominator {
 font-size: 0.6em;
 line-height: 0.6em;
 vertical-align: -0.05em;
}
</style>
<table class="gt_table">
 
 <thead class="gt_col_headings">
 <tr>
 <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1"></th>
 <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1">Outcome</th>
 <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">N Obs</th>
 <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">N Predictors</th>
 <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">N Events</th>
 <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">% Censored</th>
 <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">% Missing</th>
 <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">% Continuous</th>
 </tr>
 </thead>
 <tbody class="gt_table_body">
 <tr><td class="gt_row gt_right gt_stub">ACTG 320</td>
<td class="gt_row gt_left">AIDS diagnosis</td>
<td class="gt_row gt_center">1,151</td>
<td class="gt_row gt_center">12</td>
<td class="gt_row gt_center">96</td>
<td class="gt_row gt_center">92</td>
<td class="gt_row gt_center">0.00</td>
<td class="gt_row gt_center">30</td></tr>
 <tr><td class="gt_row gt_right gt_stub">ACTG 320</td>
<td class="gt_row gt_left">Death</td>
<td class="gt_row gt_center">1,151</td>
<td class="gt_row gt_center">12</td>
<td class="gt_row gt_center">26</td>
<td class="gt_row gt_center">98</td>
<td class="gt_row gt_center">0.00</td>
<td class="gt_row gt_center">30</td></tr>
 <tr><td class="gt_row gt_right gt_stub">ARIC</td>
<td class="gt_row gt_left">Heart failure</td>
<td class="gt_row gt_center">13,623</td>
<td class="gt_row gt_center">41</td>
<td class="gt_row gt_center">2,981</td>
<td class="gt_row gt_center">78</td>
<td class="gt_row gt_center">5.2</td>
<td class="gt_row gt_center">67</td></tr>
 <tr><td class="gt_row gt_right gt_stub">ARIC</td>
<td class="gt_row gt_left">Coronary heart disease</td>
<td class="gt_row gt_center">13,623</td>
<td class="gt_row gt_center">41</td>
<td class="gt_row gt_center">2,282</td>
<td class="gt_row gt_center">83</td>
<td class="gt_row gt_center">5.2</td>
<td class="gt_row gt_center">67</td></tr>
 <tr><td class="gt_row gt_right gt_stub">ARIC</td>
<td class="gt_row gt_left">Stroke</td>
<td class="gt_row gt_center">13,623</td>
<td class="gt_row gt_center">41</td>
<td class="gt_row gt_center">1,323</td>
<td class="gt_row gt_center">90</td>
<td class="gt_row gt_center">5.2</td>
<td class="gt_row gt_center">67</td></tr>
 <tr><td class="gt_row gt_right gt_stub">ARIC</td>
<td class="gt_row gt_left">Death</td>
<td class="gt_row gt_center">13,623</td>
<td class="gt_row gt_center">41</td>
<td class="gt_row gt_center">6,662</td>
<td class="gt_row gt_center">51</td>
<td class="gt_row gt_center">5.2</td>
<td class="gt_row gt_center">67</td></tr>
 <tr><td class="gt_row gt_right gt_stub">Colon cancer</td>
<td class="gt_row gt_left">Recurrence</td>
<td class="gt_row gt_center">929</td>
<td class="gt_row gt_center">12</td>
<td class="gt_row gt_center">468</td>
<td class="gt_row gt_center">50</td>
<td class="gt_row gt_center">0.37</td>
<td class="gt_row gt_center">20</td></tr>
 </tbody>
 
 
</table>
</div>

---
## Data sets continued

<div id="xrgfxaieay" style="overflow-x:auto;overflow-y:auto;width:auto;height:auto;">
<style>html {
 font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Helvetica Neue', 'Fira Sans', 'Droid Sans', Arial, sans-serif;
}

#xrgfxaieay .gt_table {
  display: table;
  border-collapse: collapse;
  margin-left: auto;
  margin-right: auto;
  color: #333333;
  font-size: 16px;
  font-weight: normal;
  font-style: normal;
  background-color: #FFFFFF;
  width: auto;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #A8A8A8;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #A8A8A8;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
}

#xrgfxaieay .gt_heading {
  background-color: #FFFFFF;
  text-align: center;
  border-bottom-color: #FFFFFF;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#xrgfxaieay .gt_title {
  color: #333333;
  font-size: 125%;
  font-weight: initial;
  padding-top: 4px;
  padding-bottom: 4px;
  padding-left: 5px;
  padding-right: 5px;
  border-bottom-color: #FFFFFF;
  border-bottom-width: 0;
}

#xrgfxaieay .gt_subtitle {
  color: #333333;
  font-size: 85%;
  font-weight: initial;
  padding-top: 0;
  padding-bottom: 6px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-color: #FFFFFF;
  border-top-width: 0;
}

#xrgfxaieay .gt_bottom_border {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#xrgfxaieay .gt_col_headings {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#xrgfxaieay .gt_col_heading {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  padding-left: 5px;
  padding-right: 5px;
  overflow-x: hidden;
}

#xrgfxaieay .gt_column_spanner_outer {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  padding-top: 0;
  padding-bottom: 0;
  padding-left: 4px;
  padding-right: 4px;
}

#xrgfxaieay .gt_column_spanner_outer:first-child {
  padding-left: 0;
}

#xrgfxaieay .gt_column_spanner_outer:last-child {
  padding-right: 0;
}

#xrgfxaieay .gt_column_spanner {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 5px;
  overflow-x: hidden;
  display: inline-block;
  width: 100%;
}

#xrgfxaieay .gt_group_heading {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
}

#xrgfxaieay .gt_empty_group_heading {
  padding: 0.5px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: middle;
}

#xrgfxaieay .gt_from_md > :first-child {
  margin-top: 0;
}

#xrgfxaieay .gt_from_md > :last-child {
  margin-bottom: 0;
}

#xrgfxaieay .gt_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  margin: 10px;
  border-top-style: solid;
  border-top-width: 1px;
  border-top-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
  overflow-x: hidden;
}

#xrgfxaieay .gt_stub {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-right-style: solid;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  padding-left: 5px;
  padding-right: 5px;
}

#xrgfxaieay .gt_stub_row_group {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-right-style: solid;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  padding-left: 5px;
  padding-right: 5px;
  vertical-align: top;
}

#xrgfxaieay .gt_row_group_first td {
  border-top-width: 2px;
}

#xrgfxaieay .gt_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#xrgfxaieay .gt_first_summary_row {
  border-top-style: solid;
  border-top-color: #D3D3D3;
}

#xrgfxaieay .gt_first_summary_row.thick {
  border-top-width: 2px;
}

#xrgfxaieay .gt_last_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#xrgfxaieay .gt_grand_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#xrgfxaieay .gt_first_grand_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: double;
  border-top-width: 6px;
  border-top-color: #D3D3D3;
}

#xrgfxaieay .gt_striped {
  background-color: rgba(128, 128, 128, 0.05);
}

#xrgfxaieay .gt_table_body {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#xrgfxaieay .gt_footnotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#xrgfxaieay .gt_footnote {
  margin: 0px;
  font-size: 90%;
  padding-left: 4px;
  padding-right: 4px;
  padding-left: 5px;
  padding-right: 5px;
}

#xrgfxaieay .gt_sourcenotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#xrgfxaieay .gt_sourcenote {
  font-size: 90%;
  padding-top: 4px;
  padding-bottom: 4px;
  padding-left: 5px;
  padding-right: 5px;
}

#xrgfxaieay .gt_left {
  text-align: left;
}

#xrgfxaieay .gt_center {
  text-align: center;
}

#xrgfxaieay .gt_right {
  text-align: right;
  font-variant-numeric: tabular-nums;
}

#xrgfxaieay .gt_font_normal {
  font-weight: normal;
}

#xrgfxaieay .gt_font_bold {
  font-weight: bold;
}

#xrgfxaieay .gt_font_italic {
  font-style: italic;
}

#xrgfxaieay .gt_super {
  font-size: 65%;
}

#xrgfxaieay .gt_footnote_marks {
  font-style: italic;
  font-weight: normal;
  font-size: 75%;
  vertical-align: 0.4em;
}

#xrgfxaieay .gt_asterisk {
  font-size: 100%;
  vertical-align: 0;
}

#xrgfxaieay .gt_slash_mark {
  font-size: 0.7em;
  line-height: 0.7em;
  vertical-align: 0.15em;
}

#xrgfxaieay .gt_fraction_numerator {
  font-size: 0.6em;
  line-height: 0.6em;
  vertical-align: 0.45em;
}

#xrgfxaieay .gt_fraction_denominator {
 font-size: 0.6em;
 line-height: 0.6em;
 vertical-align: -0.05em;
}
</style>
<table class="gt_table">
 
 <thead class="gt_col_headings">
 <tr>
 <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1"></th>
 <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1">Outcome</th>
 <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">N Obs</th>
 <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">N Predictors</th>
 <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">N Events</th>
 <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">% Censored</th>
 <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">% Missing</th>
 <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">% Continuous</th>
 </tr>
 </thead>
 <tbody class="gt_table_body">
 <tr><td class="gt_row gt_right gt_stub">Colon cancer</td>
<td class="gt_row gt_left">Death</td>
<td class="gt_row gt_center">929</td>
<td class="gt_row gt_center">12</td>
<td class="gt_row gt_center">452</td>
<td class="gt_row gt_center">51</td>
<td class="gt_row gt_center">0.37</td>
<td class="gt_row gt_center">20</td></tr>
 <tr><td class="gt_row gt_right gt_stub">Early breast cancer</td>
<td class="gt_row gt_left">Recurrence or death</td>
<td class="gt_row gt_center">614</td>
<td class="gt_row gt_center">1,692</td>
<td class="gt_row gt_center">134</td>
<td class="gt_row gt_center">78</td>
<td class="gt_row gt_center">0.00</td>
<td class="gt_row gt_center">100</td></tr>
 <tr><td class="gt_row gt_right gt_stub">FCL</td>
<td class="gt_row gt_left">Death</td>
<td class="gt_row gt_center">541</td>
<td class="gt_row gt_center">7</td>
<td class="gt_row gt_center">76</td>
<td class="gt_row gt_center">86</td>
<td class="gt_row gt_center">0.00</td>
<td class="gt_row gt_center">40</td></tr>
 <tr><td class="gt_row gt_right gt_stub">FCL</td>
<td class="gt_row gt_left">Relapse</td>
<td class="gt_row gt_center">541</td>
<td class="gt_row gt_center">7</td>
<td class="gt_row gt_center">272</td>
<td class="gt_row gt_center">50</td>
<td class="gt_row gt_center">0.00</td>
<td class="gt_row gt_center">40</td></tr>
 <tr><td class="gt_row gt_right gt_stub">GBSG II</td>
<td class="gt_row gt_left">Recurrence or death</td>
<td class="gt_row gt_center">686</td>
<td class="gt_row gt_center">10</td>
<td class="gt_row gt_center">299</td>
<td class="gt_row gt_center">56</td>
<td class="gt_row gt_center">0.00</td>
<td class="gt_row gt_center">62</td></tr>
 <tr><td class="gt_row gt_right gt_stub">GUIDE-IT</td>
<td class="gt_row gt_left">CVD death</td>
<td class="gt_row gt_center">894</td>
<td class="gt_row gt_center">59</td>
<td class="gt_row gt_center">110</td>
<td class="gt_row gt_center">88</td>
<td class="gt_row gt_center">12</td>
<td class="gt_row gt_center">47</td></tr>
 <tr><td class="gt_row gt_right gt_stub">GUIDE-IT</td>
<td class="gt_row gt_left">HF hospitalization</td>
<td class="gt_row gt_center">894</td>
<td class="gt_row gt_center">59</td>
<td class="gt_row gt_center">288</td>
<td class="gt_row gt_center">68</td>
<td class="gt_row gt_center">12</td>
<td class="gt_row gt_center">47</td></tr>
 <tr><td class="gt_row gt_right gt_stub">Heart Transplant</td>
<td class="gt_row gt_left">Graft-loss or death</td>
<td class="gt_row gt_center">3,787</td>
<td class="gt_row gt_center">52</td>
<td class="gt_row gt_center">500</td>
<td class="gt_row gt_center">87</td>
<td class="gt_row gt_center">6.1</td>
<td class="gt_row gt_center">26</td></tr>
 </tbody>
 
 
</table>
</div>

---

## Data sets continued

<div id="oxajellgip" style="overflow-x:auto;overflow-y:auto;width:auto;height:auto;">
<style>html {
 font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Helvetica Neue', 'Fira Sans', 'Droid Sans', Arial, sans-serif;
}

#oxajellgip .gt_table {
  display: table;
  border-collapse: collapse;
  margin-left: auto;
  margin-right: auto;
  color: #333333;
  font-size: 16px;
  font-weight: normal;
  font-style: normal;
  background-color: #FFFFFF;
  width: auto;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #A8A8A8;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #A8A8A8;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
}

#oxajellgip .gt_heading {
  background-color: #FFFFFF;
  text-align: center;
  border-bottom-color: #FFFFFF;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#oxajellgip .gt_title {
  color: #333333;
  font-size: 125%;
  font-weight: initial;
  padding-top: 4px;
  padding-bottom: 4px;
  padding-left: 5px;
  padding-right: 5px;
  border-bottom-color: #FFFFFF;
  border-bottom-width: 0;
}

#oxajellgip .gt_subtitle {
  color: #333333;
  font-size: 85%;
  font-weight: initial;
  padding-top: 0;
  padding-bottom: 6px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-color: #FFFFFF;
  border-top-width: 0;
}

#oxajellgip .gt_bottom_border {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#oxajellgip .gt_col_headings {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#oxajellgip .gt_col_heading {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  padding-left: 5px;
  padding-right: 5px;
  overflow-x: hidden;
}

#oxajellgip .gt_column_spanner_outer {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  padding-top: 0;
  padding-bottom: 0;
  padding-left: 4px;
  padding-right: 4px;
}

#oxajellgip .gt_column_spanner_outer:first-child {
  padding-left: 0;
}

#oxajellgip .gt_column_spanner_outer:last-child {
  padding-right: 0;
}

#oxajellgip .gt_column_spanner {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 5px;
  overflow-x: hidden;
  display: inline-block;
  width: 100%;
}

#oxajellgip .gt_group_heading {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
}

#oxajellgip .gt_empty_group_heading {
  padding: 0.5px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: middle;
}

#oxajellgip .gt_from_md > :first-child {
  margin-top: 0;
}

#oxajellgip .gt_from_md > :last-child {
  margin-bottom: 0;
}

#oxajellgip .gt_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  margin: 10px;
  border-top-style: solid;
  border-top-width: 1px;
  border-top-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
  overflow-x: hidden;
}

#oxajellgip .gt_stub {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-right-style: solid;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  padding-left: 5px;
  padding-right: 5px;
}

#oxajellgip .gt_stub_row_group {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-right-style: solid;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  padding-left: 5px;
  padding-right: 5px;
  vertical-align: top;
}

#oxajellgip .gt_row_group_first td {
  border-top-width: 2px;
}

#oxajellgip .gt_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#oxajellgip .gt_first_summary_row {
  border-top-style: solid;
  border-top-color: #D3D3D3;
}

#oxajellgip .gt_first_summary_row.thick {
  border-top-width: 2px;
}

#oxajellgip .gt_last_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#oxajellgip .gt_grand_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#oxajellgip .gt_first_grand_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: double;
  border-top-width: 6px;
  border-top-color: #D3D3D3;
}

#oxajellgip .gt_striped {
  background-color: rgba(128, 128, 128, 0.05);
}

#oxajellgip .gt_table_body {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#oxajellgip .gt_footnotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#oxajellgip .gt_footnote {
  margin: 0px;
  font-size: 90%;
  padding-left: 4px;
  padding-right: 4px;
  padding-left: 5px;
  padding-right: 5px;
}

#oxajellgip .gt_sourcenotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#oxajellgip .gt_sourcenote {
  font-size: 90%;
  padding-top: 4px;
  padding-bottom: 4px;
  padding-left: 5px;
  padding-right: 5px;
}

#oxajellgip .gt_left {
  text-align: left;
}

#oxajellgip .gt_center {
  text-align: center;
}

#oxajellgip .gt_right {
  text-align: right;
  font-variant-numeric: tabular-nums;
}

#oxajellgip .gt_font_normal {
  font-weight: normal;
}

#oxajellgip .gt_font_bold {
  font-weight: bold;
}

#oxajellgip .gt_font_italic {
  font-style: italic;
}

#oxajellgip .gt_super {
  font-size: 65%;
}

#oxajellgip .gt_footnote_marks {
  font-style: italic;
  font-weight: normal;
  font-size: 75%;
  vertical-align: 0.4em;
}

#oxajellgip .gt_asterisk {
  font-size: 100%;
  vertical-align: 0;
}

#oxajellgip .gt_slash_mark {
  font-size: 0.7em;
  line-height: 0.7em;
  vertical-align: 0.15em;
}

#oxajellgip .gt_fraction_numerator {
  font-size: 0.6em;
  line-height: 0.6em;
  vertical-align: 0.45em;
}

#oxajellgip .gt_fraction_denominator {
 font-size: 0.6em;
 line-height: 0.6em;
 vertical-align: -0.05em;
}
</style>
<table class="gt_table">
 
 <thead class="gt_col_headings">
 <tr>
 <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1"></th>
 <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1">Outcome</th>
 <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">N Obs</th>
 <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">N Predictors</th>
 <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">N Events</th>
 <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">% Censored</th>
 <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">% Missing</th>
 <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">% Continuous</th>
 </tr>
 </thead>
 <tbody class="gt_table_body">
 <tr><td class="gt_row gt_right gt_stub">MESA</td>
<td class="gt_row gt_left">Heart failure</td>
<td class="gt_row gt_center">6,785</td>
<td class="gt_row gt_center">48</td>
<td class="gt_row gt_center">339</td>
<td class="gt_row gt_center">95</td>
<td class="gt_row gt_center">9.3</td>
<td class="gt_row gt_center">76</td></tr>
 <tr><td class="gt_row gt_right gt_stub">MESA</td>
<td class="gt_row gt_left">Coronary heart disease</td>
<td class="gt_row gt_center">6,785</td>
<td class="gt_row gt_center">48</td>
<td class="gt_row gt_center">439</td>
<td class="gt_row gt_center">94</td>
<td class="gt_row gt_center">9.3</td>
<td class="gt_row gt_center">76</td></tr>
 <tr><td class="gt_row gt_right gt_stub">MESA</td>
<td class="gt_row gt_left">Stroke</td>
<td class="gt_row gt_center">6,783</td>
<td class="gt_row gt_center">48</td>
<td class="gt_row gt_center">292</td>
<td class="gt_row gt_center">96</td>
<td class="gt_row gt_center">9.3</td>
<td class="gt_row gt_center">76</td></tr>
 <tr><td class="gt_row gt_right gt_stub">MESA</td>
<td class="gt_row gt_left">Death</td>
<td class="gt_row gt_center">6,793</td>
<td class="gt_row gt_center">48</td>
<td class="gt_row gt_center">1,297</td>
<td class="gt_row gt_center">81</td>
<td class="gt_row gt_center">9.3</td>
<td class="gt_row gt_center">76</td></tr>
 <tr><td class="gt_row gt_right gt_stub">Monoclonal gammopathy</td>
<td class="gt_row gt_left">Death</td>
<td class="gt_row gt_center">1,384</td>
<td class="gt_row gt_center">8</td>
<td class="gt_row gt_center">963</td>
<td class="gt_row gt_center">30</td>
<td class="gt_row gt_center">0.49</td>
<td class="gt_row gt_center">83</td></tr>
 <tr><td class="gt_row gt_right gt_stub">Monoclonal gammopathy</td>
<td class="gt_row gt_left">Malignancy</td>
<td class="gt_row gt_center">1,384</td>
<td class="gt_row gt_center">8</td>
<td class="gt_row gt_center">115</td>
<td class="gt_row gt_center">92</td>
<td class="gt_row gt_center">0.49</td>
<td class="gt_row gt_center">83</td></tr>
 <tr><td class="gt_row gt_right gt_stub">Movies released in 2015-2018</td>
<td class="gt_row gt_left">Gross 1M USD</td>
<td class="gt_row gt_center">551</td>
<td class="gt_row gt_center">46</td>
<td class="gt_row gt_center">522</td>
<td class="gt_row gt_center">5.3</td>
<td class="gt_row gt_center">0.00</td>
<td class="gt_row gt_center">4.5</td></tr>
 <tr><td class="gt_row gt_right gt_stub">Non-alcohol fatty liver disease</td>
<td class="gt_row gt_left">Death</td>
<td class="gt_row gt_center">17,549</td>
<td class="gt_row gt_center">24</td>
<td class="gt_row gt_center">1,364</td>
<td class="gt_row gt_center">92</td>
<td class="gt_row gt_center">33</td>
<td class="gt_row gt_center">45</td></tr>
 </tbody>
 
 
</table>
</div>

---

## Data sets continued

<div id="rttxouxohv" style="overflow-x:auto;overflow-y:auto;width:auto;height:auto;">
<style>html {
 font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Helvetica Neue', 'Fira Sans', 'Droid Sans', Arial, sans-serif;
}

#rttxouxohv .gt_table {
  display: table;
  border-collapse: collapse;
  margin-left: auto;
  margin-right: auto;
  color: #333333;
  font-size: 16px;
  font-weight: normal;
  font-style: normal;
  background-color: #FFFFFF;
  width: auto;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #A8A8A8;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #A8A8A8;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
}

#rttxouxohv .gt_heading {
  background-color: #FFFFFF;
  text-align: center;
  border-bottom-color: #FFFFFF;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#rttxouxohv .gt_title {
  color: #333333;
  font-size: 125%;
  font-weight: initial;
  padding-top: 4px;
  padding-bottom: 4px;
  padding-left: 5px;
  padding-right: 5px;
  border-bottom-color: #FFFFFF;
  border-bottom-width: 0;
}

#rttxouxohv .gt_subtitle {
  color: #333333;
  font-size: 85%;
  font-weight: initial;
  padding-top: 0;
  padding-bottom: 6px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-color: #FFFFFF;
  border-top-width: 0;
}

#rttxouxohv .gt_bottom_border {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#rttxouxohv .gt_col_headings {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#rttxouxohv .gt_col_heading {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  padding-left: 5px;
  padding-right: 5px;
  overflow-x: hidden;
}

#rttxouxohv .gt_column_spanner_outer {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  padding-top: 0;
  padding-bottom: 0;
  padding-left: 4px;
  padding-right: 4px;
}

#rttxouxohv .gt_column_spanner_outer:first-child {
  padding-left: 0;
}

#rttxouxohv .gt_column_spanner_outer:last-child {
  padding-right: 0;
}

#rttxouxohv .gt_column_spanner {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 5px;
  overflow-x: hidden;
  display: inline-block;
  width: 100%;
}

#rttxouxohv .gt_group_heading {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
}

#rttxouxohv .gt_empty_group_heading {
  padding: 0.5px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: middle;
}

#rttxouxohv .gt_from_md > :first-child {
  margin-top: 0;
}

#rttxouxohv .gt_from_md > :last-child {
  margin-bottom: 0;
}

#rttxouxohv .gt_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  margin: 10px;
  border-top-style: solid;
  border-top-width: 1px;
  border-top-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
  overflow-x: hidden;
}

#rttxouxohv .gt_stub {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-right-style: solid;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  padding-left: 5px;
  padding-right: 5px;
}

#rttxouxohv .gt_stub_row_group {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-right-style: solid;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  padding-left: 5px;
  padding-right: 5px;
  vertical-align: top;
}

#rttxouxohv .gt_row_group_first td {
  border-top-width: 2px;
}

#rttxouxohv .gt_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#rttxouxohv .gt_first_summary_row {
  border-top-style: solid;
  border-top-color: #D3D3D3;
}

#rttxouxohv .gt_first_summary_row.thick {
  border-top-width: 2px;
}

#rttxouxohv .gt_last_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#rttxouxohv .gt_grand_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#rttxouxohv .gt_first_grand_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: double;
  border-top-width: 6px;
  border-top-color: #D3D3D3;
}

#rttxouxohv .gt_striped {
  background-color: rgba(128, 128, 128, 0.05);
}

#rttxouxohv .gt_table_body {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#rttxouxohv .gt_footnotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#rttxouxohv .gt_footnote {
  margin: 0px;
  font-size: 90%;
  padding-left: 4px;
  padding-right: 4px;
  padding-left: 5px;
  padding-right: 5px;
}

#rttxouxohv .gt_sourcenotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#rttxouxohv .gt_sourcenote {
  font-size: 90%;
  padding-top: 4px;
  padding-bottom: 4px;
  padding-left: 5px;
  padding-right: 5px;
}

#rttxouxohv .gt_left {
  text-align: left;
}

#rttxouxohv .gt_center {
  text-align: center;
}

#rttxouxohv .gt_right {
  text-align: right;
  font-variant-numeric: tabular-nums;
}

#rttxouxohv .gt_font_normal {
  font-weight: normal;
}

#rttxouxohv .gt_font_bold {
  font-weight: bold;
}

#rttxouxohv .gt_font_italic {
  font-style: italic;
}

#rttxouxohv .gt_super {
  font-size: 65%;
}

#rttxouxohv .gt_footnote_marks {
  font-style: italic;
  font-weight: normal;
  font-size: 75%;
  vertical-align: 0.4em;
}

#rttxouxohv .gt_asterisk {
  font-size: 100%;
  vertical-align: 0;
}

#rttxouxohv .gt_slash_mark {
  font-size: 0.7em;
  line-height: 0.7em;
  vertical-align: 0.15em;
}

#rttxouxohv .gt_fraction_numerator {
  font-size: 0.6em;
  line-height: 0.6em;
  vertical-align: 0.45em;
}

#rttxouxohv .gt_fraction_denominator {
 font-size: 0.6em;
 line-height: 0.6em;
 vertical-align: -0.05em;
}
</style>
<table class="gt_table">
 
 <thead class="gt_col_headings">
 <tr>
 <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1"></th>
 <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1">Outcome</th>
 <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">N Obs</th>
 <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">N Predictors</th>
 <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">N Events</th>
 <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">% Censored</th>
 <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">% Missing</th>
 <th class="gt_col_heading gt_columns_bottom_border gt_center" rowspan="1" colspan="1">% Continuous</th>
 </tr>
 </thead>
 <tbody class="gt_table_body">
 <tr><td class="gt_row gt_right gt_stub">Primary biliary cholangitis</td>
<td class="gt_row gt_left">Death</td>
<td class="gt_row gt_center">276</td>
<td class="gt_row gt_center">19</td>
<td class="gt_row gt_center">111</td>
<td class="gt_row gt_center">60</td>
<td class="gt_row gt_center">0.00</td>
<td class="gt_row gt_center">59</td></tr>
 <tr><td class="gt_row gt_right gt_stub">Rotterdam tumor bank</td>
<td class="gt_row gt_left">Recurrence</td>
<td class="gt_row gt_center">2,982</td>
<td class="gt_row gt_center">11</td>
<td class="gt_row gt_center">1,518</td>
<td class="gt_row gt_center">49</td>
<td class="gt_row gt_center">0.00</td>
<td class="gt_row gt_center">56</td></tr>
 <tr><td class="gt_row gt_right gt_stub">Rotterdam tumor bank</td>
<td class="gt_row gt_left">Death</td>
<td class="gt_row gt_center">2,982</td>
<td class="gt_row gt_center">11</td>
<td class="gt_row gt_center">1,272</td>
<td class="gt_row gt_center">57</td>
<td class="gt_row gt_center">0.00</td>
<td class="gt_row gt_center">56</td></tr>
 <tr><td class="gt_row gt_right gt_stub">Serum free light chain</td>
<td class="gt_row gt_left">Death</td>
<td class="gt_row gt_center">7,874</td>
<td class="gt_row gt_center">10</td>
<td class="gt_row gt_center">2,169</td>
<td class="gt_row gt_center">72</td>
<td class="gt_row gt_center">1.7</td>
<td class="gt_row gt_center">50</td></tr>
 <tr><td class="gt_row gt_right gt_stub">SPRINT</td>
<td class="gt_row gt_left">CVD death</td>
<td class="gt_row gt_center">9,361</td>
<td class="gt_row gt_center">174</td>
<td class="gt_row gt_center">521</td>
<td class="gt_row gt_center">94</td>
<td class="gt_row gt_center">0.65</td>
<td class="gt_row gt_center">24</td></tr>
 <tr><td class="gt_row gt_right gt_stub">SPRINT</td>
<td class="gt_row gt_left">Death</td>
<td class="gt_row gt_center">9,361</td>
<td class="gt_row gt_center">174</td>
<td class="gt_row gt_center">1,644</td>
<td class="gt_row gt_center">82</td>
<td class="gt_row gt_center">0.65</td>
<td class="gt_row gt_center">24</td></tr>
 <tr><td class="gt_row gt_right gt_stub">Systolic Heart Failure</td>
<td class="gt_row gt_left">Death</td>
<td class="gt_row gt_center">2,231</td>
<td class="gt_row gt_center">41</td>
<td class="gt_row gt_center">726</td>
<td class="gt_row gt_center">67</td>
<td class="gt_row gt_center">0.00</td>
<td class="gt_row gt_center">33</td></tr>
 <tr><td class="gt_row gt_right gt_stub">VA lung cancer trial</td>
<td class="gt_row gt_left">Death</td>
<td class="gt_row gt_center">137</td>
<td class="gt_row gt_center">8</td>
<td class="gt_row gt_center">128</td>
<td class="gt_row gt_center">6.6</td>
<td class="gt_row gt_center">0.00</td>
<td class="gt_row gt_center">33</td></tr>
 </tbody>
 
 
</table>
</div>

---

## Evaluation (bonus round)

Consider a testing data set:
`$$\mathcal{D}_{\text{test}} = \left\{ (T_i, \delta_i, x_{i}) \right\}_{i=1}^{N_{\text{test}}}.$$`
Let `$\widehat{S}(t \mid x_i)$` be the predicted probability of survival up to a given prediction horizon of `$t > 0$`. 
The Brier score at time `$t$` is `$$\widehat{\text{BS}}(t) = \frac{1}{N_{\text{test}}} \sum_{i=1}^{N_{\text{test}}} \left\{ \widehat{S}(t \mid x_i)^2 \cdot I(T_i \leq t, \delta_i = 1) \cdot \widehat{G}(T_i)^{-1} + \\ [1-\widehat{S}(t \mid x_i)]^2 \cdot I(T_i > t) \cdot \widehat{G}(t)^{-1}\right\}$$`

where `$\widehat{G}(\cdot)$` is the Kaplan-Meier estimate of the censoring distribution.

---

## Evaluation (bonus round)

As the Brier score is time dependent, integration over time provides a summary measure of performance over a range of plausible prediction horizons. The integrated Brier score is defined as

`$$\mathcal{\widehat{BS}}(t_a, t_b) = \frac{1}{t_b - t_a}\int_{t_a}^{t_b} \widehat{\text{BS}}(t) dt.$$` 
In our results

- `$t_a$` is the 25th percentile of event times

- `$t_b$` is the 75th percentile of event times

---

## Evaluation (bonus round)

`$\mathcal{\widehat{BS}}(t_a, t_b)$`, a sum of squared prediction errors, can be scaled to produce a measure of explained residual variation (i.e., an `$R^2$` statistic) by computing `$$R^2 = 1 - \frac{\mathcal{\widehat{BS}}(t_a, t_b)}{\mathcal{\widehat{BS}}_0(t_a, t_b)}$$` where `$\mathcal{\widehat{BS}}_0(t_a, t_b)$` is the integrated Brier score when a Kaplan-Meier estimate for survival based on the training data is used as the survival prediction function `$\widehat{S}(t)$`.

_Jargon_: `$R^2$` = __index of prediction accuracy__ (IPA)