Reference papers

CatBoost: unbiased boosting with categorical features

Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, Andrey Gulin. NeurIPS, 2018

NeurIPS 2018 paper with explanation of Ordered boosting principles and ordered categorical features statistics.

CatBoost: gradient boosting with categorical features support

Anna Veronika Dorogush, Vasily Ershov, Andrey Gulin. Workshop on ML Systems at NIPS 2017

A paper explaining the CatBoost working principles: how it handles categorical features, how it fights overfitting, how GPU training and fast formula applier are implemented.

Minimal Variance Sampling in Stochastic Gradient Boosting

Bulat Ibragimov, Gleb Gusev. arXiv:1910.13204

A paper about Minimal Variance Sampling, which is the default sampling in CatBoost.

Finding Influential Training Samples for Gradient Boosted Decision Trees

Boris Sharchilev, Yury Ustinovsky, Pavel Serdyukov, Maarten de Rijke. arXiv:1802.06640

A paper explaining several ways of extending the framework for finding influential training samples for a particular case of tree ensemble-based models to non-parametric GBDT ensembles under the assumption that tree structures remain fixed and introducing a general scheme of obtaining further approximations to this method that balance the trade-off between performance and computational complexity.

A Unified Approach to Interpreting Model Predictions

Scott Lundberg, Su-In Lee. arXiv:1705.07874

A paper explaining a unified framework for interpreting predictions, SHAP (SHapley Additive exPlanations).

Consistent feature attribution for tree ensembles

Scott M. Lundberg, Su-In Lee. arXiv:1706.06060

A paper explaining fast exact solutions for SHAP (SHapley Additive exPlanation) values, a unique additive feature attribution method based on conditional expectations that is both consistent and locally accurate.

Winning The Transfer Learning Track of Yahoo!’s Learning To Rank Challenge with YetiRank

Andrey Gulin, Igor Kuralenok, Dimitry Pavlov. PMLR 14:63-76

The theory underlying the YetiRank and YetiRankPairwise modes in CatBoost.

Which Tricks are Important for Learning to Rank?

Ivan Lyzhin, Aleksei Ustimenko, Andrey Gulin, Liudmila Prokhorenkova. arXiv:2204.01500

A paper comparing previously introduced LambdaMART, YetiRank and StochasticRank and proposing an improvement to the YetiRank approach to allow for optimizing specific ranking loss functions.

Gradient Boosting Performs Gaussian Process Inference

Aleksei Ustimenko, Artem Beliakov, Liudmila Prokhorenkova. arXiv:2206.05608

This paper shows that gradient boosting based on symmetric decision trees can be equivalently reformulated as a kernel method that converges to the solution of a certain Kernel Ridge Regression problem. Thus, authors obtain the convergence to a Gaussian Process' posterior mean, which, in turn, allows them to easily transform gradient boosting into a sampler from the posterior to provide better knowledge uncertainty estimates through Monte-Carlo estimation of the posterior variance. It is shown that the proposed sampler allows for better knowledge uncertainty estimates leading to improved out-of-domain detection.