• Installation
• Overview
• Python package installation
• CatBoost for Apache Spark installation
• R package installation
• Command-line version binary
• Key Features
• Training parameters
• Python package
• CatBoost for Apache Spark
• R package
• Command-line version
• Applying models
• Objectives and metrics
• Model analysis
• Data format description
• Parameter tuning
• Speeding up the training
• Data visualization
• Algorithm details
• FAQ
• Educational materials
• Development and contributions
• Contacts

Classification: objectives and metrics

Objectives and metrics

Logloss

$\displaystyle\frac{ - \sum\limits_{i=1}^N w_{i}\left(c_i \log(p_{i}) + (1-c_{i}) \log(1 - p_{i})\right)}{\sum\limits_{i = 1}^{N} w_{i}}$

Usage information See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

CrossEntropy

$\displaystyle\frac{- \sum\limits_{i=1}^N w_{i} \left(t_{i} \log(p_{i}) + (1 - t_{i}) \log(1 - p_{i})\right)}{\sum\limits_{i = 1}^{N} w_{i}}$

Usage information See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

Precision

$\frac{TP}{TP + FP}$

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

Recall

$\frac{TP}{TP+FN}$

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

F

$(1 + \beta^2) \cdot \frac{Precision * Recall}{(\beta^2 \cdot Precision) + Recall}$

Can't be used for optimization. See more.

User-defined parameters

beta

The $\beta$ parameter of the F metric.

Valid values are real numbers in the following range: $(0; +\infty)$.

Default: This parameter is obligatory (the default value is not defined)

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

F1

$2 \frac{Precision * Recall}{Precision + Recall}$

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

BalancedAccuracy

$\frac{1}{2} \left(\frac{TP}{P} + \frac{TN}{N} \right)$
User-defined parameters:

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

BalancedErrorRate

$\frac{1}{2} \left( \displaystyle\frac{FP}{TN + FP} + \displaystyle\frac{FN}{FN + TP} \right)$

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

MCC

$\displaystyle\frac{TP * TN - FP * FN}{\sqrt{(TP + FP)(TP + FN)(TN + FP)(TN + FN)}}$

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

Accuracy

$\frac{TP + TN}{\sum\limits_{i=1}^{N} w_{i}}$

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

CtrFactor

$\displaystyle\frac{\left(\sum\limits_{i = 1}^{N} w_{i} t_{i}/N\right)}{\left(\sum\limits_{i = 1}^{N} w_{i} p_{i} /N\right)}$

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

AUC

The calculation of this metric is disabled by default for the training dataset to speed up the training. Use the hints=skip_train~false parameter to enable the calculation.

Classic

$\displaystyle\frac{\sum I(a_{i}, a_{j}) \cdot w_{i} \cdot w_{j}} {\sum w_{i} \cdot w_{j}}$
The sum is calculated on all pairs of objects $(i,j)$ such that:

• $t_{i} = 0$
• $t_{j} = 1$
• $I(x, y) = \begin{cases} 0 { , } & x < y \\ 0.5 { , } & x=y \\ 1 { , } & x>y \end{cases}$

Refer to the Wikipedia article for details.

If the target type is not binary, then every object with target value $t$ and weight $w$ is replaced with two objects for the metric calculation:

• $o_{1}$ with weight $t \cdot w$ and target value 1
• $o_{2}$ with weight $(1 – t) \cdot w$ and target value 0.

Target values must be in the range [0; 1].

Ranking

$\displaystyle\frac{\sum I(a_{i}, a_{j}) \cdot w_{i} \cdot w_{j}} {\sum w_{i} * w_{j}}$

The sum is calculated on all pairs of objects $(i,j)$ such that:

• $t_{i} < t_{j}$
• $I(x, y) = \begin{cases} 0 { , } & x < y \\ 0.5 { , } & x=y \\ 1 { , } & x>y \end{cases}$

User-defined parameters

type

The type of AUC. Defines the metric calculation principles.

Default: Ranking.
Possible values: Classic, Ranking.
Examples: AUC:type=Classic, AUC:type=Ranking.

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: False.
Examples: QueryAUC:type=Ranking;use_weights=False.

QueryAUC

Classic type

$\displaystyle\frac{ \sum_q \sum_{i, j \in q} \sum I(a_{i}, a_{j}) \cdot w_{i} \cdot w_{j}} { \sum_q \sum_{i, j \in q} \sum w_{i} \cdot w_{j}}$
The sum is calculated on all pairs of objects $(i,j)$ such that:

• $t_{i} = 0$
• $t_{j} = 1$
• $I(x, y) = \begin{cases} 0 { , } & x < y \\ 0.5 { , } & x=y \\ 1 { , } & x>y \end{cases}$

Refer to the Wikipedia article for details.

If the target type is not binary, then every object with target value $t$ and weight $w$ is replaced with two objects for the metric calculation:

• $o_{1}$ with weight $t \cdot w$ and target value 1
• $o_{2}$ with weight $(1 – t) \cdot w$ and target value 0.

Target values must be in the range [0; 1].

Ranking type

$\displaystyle\frac{ \sum_q \sum_{i, j \in q} \sum I(a_{i}, a_{j}) \cdot w_{i} \cdot w_{j}} { \sum_q \sum_{i, j \in q} \sum w_{i} * w_{j}}$

The sum is calculated on all pairs of objects $(i,j)$ such that:

• $t_{i} < t_{j}$
• $I(x, y) = \begin{cases} 0 { , } & x < y \\ 0.5 { , } & x=y \\ 1 { , } & x>y \end{cases}$

Can't be used for optimization. See more.

User-defined parameters

type

The type of QueryAUC. Defines the metric calculation principles.

Default: Ranking.
Possible values: Classic, Ranking.
Examples: QueryAUC:type=Classic, QueryAUC:type=Ranking.

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: False.
Examples: QueryAUC:type=Ranking;use_weights=False.

PRAUC

PRAUC is the area under the curve $Precision(q)$ vs $Recall(q)$ for $q \in [0,1]$ where $Precision(q)$ and $Recall(q)$ are defined as follows.

$Precision(q) = \frac{TP(q)}{TP(q) + FP(q)}, Recall(q) = \frac{TP(q)}{TP(q) + FN(q)}$

Above $TP(q)$, $FP(q)$, $FN(q)$ are weights of the true positive, false positive, and false negative samples, respectively.

To calculate PRAUC for a binary classification model, specify type Classic.
In this case, $TP(q)=\sum w_i [p_i > q] c_i$, etc.

To calculate PRAUC for a multi-classification model, specify type OneVsAll.
In this case, positive samples are samples having class 0, all other samples are negative, and $TP(q)=\sum w_i [p_{i0} > q] [c_i = 0]$, etc.

type

The type of PRAUC. Defines the metric calculation principles.

Type Classic is compatible with binary classification models.
Type OneVsAll is compatible with multi-classification models.

Default: Classic.
Possible values: Classic, OneVsAll.
Examples: PRAUC:type=Classic, PRAUC:type=OneVsAll.

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: False.
Examples: PRAUC:type=Classic;use_weights=False.

NormalizedGini

See AUC.

$2 AUC - 1$

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

BrierScore

$\displaystyle\frac{\sum\limits_{i=1}^{N} w_{i}\left(p_{i} - t_{i} \right)^{2}}{\sum\limits_{i=1}^{N} w_{i}}$

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

HingeLoss

$\displaystyle\frac{\sum\limits_{i=1}^{N} w_{i} max\{1 - t_{i} p_{i}, 0\}}{\sum\limits_{i=1}^{N} w_{i}} , t_{i} = \pm 1$

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

HammingLoss

$\displaystyle\frac{\sum\limits_{i = 1}^{N} w_{i} [[p_{i} > 0.5] == t_{i}]]}{\sum\limits_{i=1}^{N} w_{i}}$

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

ZeroOneLoss

$1 - Accuracy$

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

Kappa

$1 - \displaystyle\frac{1 - Accuracy}{1 - RAccuracy}$

$RAccuracy = \displaystyle\frac{(TN + FP) (TN + FN) + (FN + TP) (FP + TP)}{(\sum\limits_{i=1}^{N} w_{i})^{2}}$

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

WKappa

See the formula on page 3 of the A note on the linearly weighted kappa coefficient for ordinal scales paper.

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

LogLikelihoodOfPrediction

The calculation consists of the following steps:

1. Define the sum of weights ($W$) and the mean target ($\bar{t}$):

$W = \sum\limits_{i} w_{i}$

$\bar{t} = \frac{1}{W} \sum\limits_{i} t_{i} w_{i}$

2. Denote log-likelihood of a constant prediction:

$ll_0 = \sum\limits_{i} w_{i} (\bar{t} \cdot log(\bar{t}) + (1 - \bar{t}) \cdot log(1 - \bar{t}))$

3. Calculate LogLikelihoodOfPrediction ($llp$), which reflects how the likelihood ($ll$) differs from the constant prediction:

$llp = \displaystyle\frac{ll(t, w) - ll_0}{\sum\limits_{i} t_{i} w_{i}}$

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

Used for optimization

Name Optimization GPU Support
Logloss + +
CrossEntropy + +
Precision - +
Recall - +
F - -
F1 - +
BalancedAccuracy - -
BalancedErrorRate - -
MCC - +
Accuracy - +
CtrFactor - -
AUC - -
QueryAUC - -
NormalizedGini - -
BrierScore - -
HingeLoss - -
HammingLoss - -
ZeroOneLoss - +
Kappa - -
WKappa - -
LogLikelihoodOfPrediction - -