# Multiclassification: objectives and metrics

## Objectives and metrics

### MultiClass

$\displaystyle\frac{\sum\limits_{i=1}^{N}w_{i}\log\left(\displaystyle\frac{e^{a_{it_{i}}}}{ \sum\limits_{j=0}^{M - 1}e^{a_{ij}}} \right)}{\sum\limits_{i=1}^{N}w_{i}} { ,}$

$t \in \{0, ..., M - 1\}$

Usage information See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### MultiClassOneVsAll

$\displaystyle\frac{\frac{1}{M}\sum\limits_{i = 1}^N w_i \sum\limits_{j = 0}^{M - 1} [j = t_i] \log(p_{ij}) + [j \neq t_i] \log(1 - p_{ij})}{\sum\limits_{i = 1}^N w_i} { ,}$

$t \in \{0, ..., M - 1\}$

Usage information See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### Precision

This function is calculated separately for each class k numbered from 0 to M – 1.

$\frac{TP}{TP + FP}$

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### Recall

This function is calculated separately for each class k numbered from 0 to M – 1.

$\frac{TP}{TP+FN}$

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### F

This function is calculated separately for each class k numbered from 0 to M – 1.

$(1 + \beta^2) \cdot \frac{Precision * Recall}{(\beta^2 \cdot Precision) + Recall}$

Can't be used for optimization. See more.

User-defined parameters

beta

The $\beta$ parameter of the F metric.

Valid values are real numbers in the following range: $(0; +\infty)$.

Default: This parameter is obligatory (the default value is not defined)

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### F1

This function is calculated separately for each class k numbered from 0 to M – 1.

$2 \frac{Precision * Recall}{Precision + Recall}$

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### TotalF1

The formula depends on the value of the average parameter:

#### Weighted

$\frac{\sum\limits_{i=1}^{M} w_{i} F1_{i}}{\sum\limits_{i=1}^{M}w_{i}} {, where}$

$w_{i}$ is the sum of the weights of the documents which correspond to the i-th class. If document weights are not specified $w_{i}$ stands for the number of times the i-th class is found among the label values.

#### Macro

$\displaystyle\frac{\sum\limits_{i=1}^{M}F1_{i}}{M}$

#### Micro

$TotalF1 = \displaystyle\frac{2 \cdot TP}{2 \cdot TP + FP + FN} {, where}$

• $TP = \sum\limits_{i=1}^{M} TP_{i}$
• $FP = \sum\limits_{i=1}^{M} FP_{i}$
• $FN = \sum\limits_{i=1}^{M} FN_{i}$

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

average

The method for averaging the value of the metric that is initially individually calculated for each class.

Default: Weighted.
Possible values: Weighted, Macro, Micro.

### MCC

This functions is defined in terms of a $k \times k$ confusion matrix $C$ (where k is the number of classes):

$\displaystyle\frac{\sum\limits_{k}\sum\limits_{l}\sum\limits_{m} C_{kk} C_{lm} - C_{kl}C_{mk}}{\sqrt{\sum\limits_{k} \left(\sum\limits_{l} C_{kl}\right) \left(\sum\limits_{k' | k' \neq k} \sum\limits_{l'} C_{k'l'}\right)}\sqrt{\sum\limits_{k} \left(\sum\limits_{l} C_{lk}\right) \left(\sum\limits_{k' | k' \neq k} \sum\limits_{l'} C_{l' k'}\right)}}$

See the Wikipedia article for more details.

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### Accuracy

$\displaystyle\frac{\sum\limits_{i=1}^{N}w_{i}[argmax_{j=0,...,M - 1}(a_{ij})==t_{i}]}{\sum\limits_{i=1}^{N}w_{i}} { , }$

$t \in \{0, ..., M - 1\}$

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### HingeLoss

See the Wikipedia article.

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### HammingLoss

$\displaystyle\frac{\sum\limits_{i = 1}^{N} w_{i} [argmax_{j=0,...,M - 1}(a_{ij})\neq t_{i}]}{\sum\limits_{i = 1}^{N} w_{i}}$

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### ZeroOneLoss

$1 - Accuracy$

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### Kappa

$1 - \displaystyle\frac{1 - Accuracy}{1 - RAccuracy}$

$RAccuracy = \displaystyle\frac{\sum\limits_{k=0}^{M - 1} n_{k_{a}}n_{k_{t}}}{(\sum\limits_{i=1}^{N}w_{i})^{2}}$

$k_{a}$ is the weighted number of times class k is predicted by the model

$k_{t}$ is the weighted number of times class k is set as the label for input objects

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### WKappa

See the formula on page 3 of the A note on the linearly weighted kappa coefficient for ordinal scales paper.

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### AUC

The calculation of this metric is disabled by default for the training dataset to speed up the training. Use the hints=skip_train~false parameter to enable the calculation.

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: False.
Examples: AUC:type=Ranking;use_weights=False.

type

The type of AUC. Defines the metric calculation principles.

Default: Mu
Possible values: Mu, OneVsAll.
Examples: AUC:type=Mu, AUC:type=OneVsAll.

misclass_cost_matrix

The matrix M with misclassification cost values. $M[i,j]$ in this matrix is the cost of classifying an object as a member of the class i when its' actual class is j. Applicable only if the used type of AUC is Mu.

Format for a matrix of size C:

<Value for M[0,0]>, <Value for M[0,1]>, ..., <Value for M[0,C-1]>, <Value for M[1,0]>, ..., <Value for M[C-1,0]>, ..., <Value for M[C-1,C-1]>


All diagonal elements $M[i, j]$ (such that i=j) must be equal to 0.

Note

The type parameter is optional and is assumed to be set to Mu if the parameter is explicitly specified.

Default: All non-diagonal matrix elements are set to 1. All diagonal elements $М[i, j]$ (such that i = j) are set to 0.
Examples: Three classes — AUC:misclass_cost_matrix=0/0.5/2/1/0/1/0/0.5/0, Two classes — AUC:type=Mu;misclass_cost_matrix=0/0.5/1/0.

## Used for optimization

Name Optimization GPU Support
MultiClass + +
MultiClassOneVsAll + +
Precision - +
Recall - +
F - -
F1 - +
TotalF1 - +
MCC - +
Accuracy - +
HingeLoss - -
HammingLoss - -
ZeroOneLoss - +
Kappa - -
WKappa - -
AUC - -