# Classification: objectives and metrics

## Objectives and metrics

### Logloss

$\displaystyle\frac{ - \sum\limits_{i=1}^N w_{i}\left(c_i \log(p_{i}) + (1-c_{i}) \log(1 - p_{i})\right)}{\sum\limits_{i = 1}^{N} w_{i}}$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### CrossEntropy

$\displaystyle\frac{- \sum\limits_{i=1}^N w_{i} \left(t_{i} \log(p_{i}) + (1 - t_{i}) \log(1 - p_{i})\right)}{\sum\limits_{i = 1}^{N} w_{i}}$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### Precision

$\frac{TP}{TP + FP}$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### Recall

$\frac{TP}{TP+FN}$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### F1

$2 \frac{Precision * Recall}{Precision + Recall}$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### BalancedAccuracy

$\frac{1}{2} \left(\frac{TP}{P} + \frac{TN}{N} \right)$
User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### BalancedErrorRate

$\frac{1}{2} \left( \displaystyle\frac{FP}{TN + FP} + \displaystyle\frac{FN}{FN + TP} \right)$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### MCC

$\displaystyle\frac{TP * TN - FP * FN}{\sqrt{(TP + FP)(TP + FN)(TN + FP)(TN + FN)}}$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### Accuracy

$\frac{TP + TN}{\sum\limits_{i=1}^{N} w_{i}}$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### CtrFactor

$\displaystyle\frac{\left(\sum\limits_{i = 1}^{N} w_{i} t_{i}/N\right)}{\left(\sum\limits_{i = 1}^{N} w_{i} p_{i} /N\right)}$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### AUC

The calculation of this metric is disabled by default for the training dataset to speed up the training. Use the hints=skip_train~false parameter to enable the calculation.

• type

The type of AUC. Defines the metric calculation principles.

Possible values:

• Classic
• Ranking

Examples:

AUC:type=Classic

AUC:type=Ranking


#### Classic

$\displaystyle\frac{\sum I(a_{i}, a_{j}) \cdot w_{i} \cdot w_{j}} {\sum w_{i} \cdot w_{j}}$
The sum is calculated on all pairs of objects $(i,j)$ such that:

• $t_{i} = 0$
• $t_{j} = 1$
• $I(x, y) = \begin{cases} 0 { , } & x < y \\ 0.5 { , } & x=y \\ 1 { , } & x>y \end{cases}$

Refer to the Wikipedia article for details.

If the target type is not binary, then every object with target value $t$ and weight $w$ is replaced with two objects for the metric calculation:

• $o_{1}$ with weight $t \cdot w$ and target value 1
• $o_{2}$ with weight (1 – t) \cdot w and target value 0.

Target values must be in the range [0; 1].

#### Ranking

$\displaystyle\frac{\sum I(a_{i}, a_{j}) \cdot w_{i} \cdot w_{j}} {\sum w_{i} * w_{j}}$

The sum is calculated on all pairs of objects $(i,j)$ such that:

• $t_{i} < t_{j}$
• $I(x, y) = \begin{cases} 0 { , } & x < y \\ 0.5 { , } & x=y \\ 1 { , } & x>y \end{cases}$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: False

### QueryAUC

• type

The type of AUC. Defines the metric calculation principles.

Possible values:

• Classic
• Ranking

Examples:

AUC:type=Classic

AUC:type=Ranking


#### Classic

$\displaystyle\frac{ \sum_q \sum_{i, j \in q} \sum I(a_{i}, a_{j}) \cdot w_{i} \cdot w_{j}} { \sum_q \sum_{i, j \in q} \sum w_{i} \cdot w_{j}}$
The sum is calculated on all pairs of objects $(i,j)$ such that:

• $t_{i} = 0$
• $t_{j} = 1$
• $I(x, y) = \begin{cases} 0 { , } & x < y \\ 0.5 { , } & x=y \\ 1 { , } & x>y \end{cases}$

Refer to the Wikipedia article for details.

If the target type is not binary, then every object with target value $t$ and weight $w$ is replaced with two objects for the metric calculation:

• $o_{1}$ with weight $t \cdot w$ and target value 1
• $o_{2}$ with weight (1 – t) \cdot w and target value 0.

Target values must be in the range [0; 1].

#### Ranking

$\displaystyle\frac{ \sum_q \sum_{i, j \in q} \sum I(a_{i}, a_{j}) \cdot w_{i} \cdot w_{j}} { \sum_q \sum_{i, j \in q} \sum w_{i} * w_{j}}$

The sum is calculated on all pairs of objects $(i,j)$ such that:

• $t_{i} < t_{j}$
• $I(x, y) = \begin{cases} 0 { , } & x < y \\ 0.5 { , } & x=y \\ 1 { , } & x>y \end{cases}$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: false

### NormalizedGini

See AUC.

$2 AUC - 1$

User-defined parameters: use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

### BrierScore

$\displaystyle\frac{\sum\limits_{i=1}^{N} w_{i}\left(p_{i} - t_{i} \right)^{2}}{\sum\limits_{i=1}^{N} w_{i}}$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### HingeLoss

$\displaystyle\frac{\sum\limits_{i=1}^{N} w_{i} max\{1 - t_{i} p_{i}, 0\}}{\sum\limits_{i=1}^{N} w_{i}} , t_{i} = \pm 1$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### HammingLoss

$\displaystyle\frac{\sum\limits_{i = 1}^{N} w_{i} [[p_{i} > 0.5] == t_{i}]]}{\sum\limits_{i=1}^{N} w_{i}}$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### ZeroOneLoss

$1 - Accuracy$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### Kappa

$1 - \displaystyle\frac{1 - Accuracy}{1 - RAccuracy}$

$RAccuracy = \displaystyle\frac{(TN + FP) (TN + FN) + (FN + TP) (FP + TP)}{(\sum\limits_{i=1}^{N} w_{i})^{2}}$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### WKappa

See the formula on page 3 of the A note on the linearly weighted kappa coefficient for ordinal scales paper.

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### LogLikelihoodOfPrediction

The calculation consists of the following steps:

1. Define the sum of weights ($W$) and the mean target ($\bar{t}$):

$W = \sum\limits_{i} w_{i}$

$\bar{t} = \frac{1}{W} \sum\limits_{i} t_{i} w_{i}$

2. Denote log-likelihood of a constant prediction:

$ll_0 = \sum\limits_{i} w_{i} (\bar{t} \cdot log(\bar{t}) + (1 - \bar{t}) \cdot log(1 - \bar{t}))$

3. Calculate LogLikelihoodOfPrediction ($llp$), which reflects how the likelihood ($ll$) differs from the constant prediction:

$llp = \displaystyle\frac{ll(t, w) - ll_0}{\sum\limits_{i} t_{i} w_{i}}$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

## Used for optimization

Name Optimization
Logloss +
CrossEntropy +
Precision -
Recall -
F1 -
BalancedAccuracy -
BalancedErrorRate -
MCC -
Accuracy -
CtrFactor -
AUC -
QueryAUC -
NormalizedGini -
BrierScore -
HingeLoss -
HammingLoss -
ZeroOneLoss -
Kappa -
WKappa -
LogLikelihoodOfPrediction -