# Multiclassification: objectives and metrics

## Objectives and metrics

### MultiClass

$\displaystyle\frac{\sum\limits_{i=1}^{N}w_{i}\log\left(\displaystyle\frac{e^{a_{it_{i}}}}{ \sum\limits_{j=0}^{M - 1}e^{a_{ij}}} \right)}{\sum\limits_{i=1}^{N}w_{i}} { ,}$

$t \in \{0, ..., M - 1\}$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### MultiClassOneVsAll

$\displaystyle\frac{\frac{1}{M}\sum\limits_{i = 1}^N w_i \sum\limits_{j = 0}^{M - 1} [j = t_i] \log(p_{ij}) + [j \neq t_i] \log(1 - p_{ij})}{\sum\limits_{i = 1}^N w_i} { ,}$

$t \in \{0, ..., M - 1\}$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### Precision

This function is calculated separately for each class k numbered from 0 to M – 1.

$\frac{TP}{TP + FP}$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### Recall

This function is calculated separately for each class k numbered from 0 to M – 1.

$\frac{TP}{TP+FN}$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### F1

This function is calculated separately for each class k numbered from 0 to M – 1.

$2 \frac{Precision * Recall}{Precision + Recall}$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### TotalF1

The formula depends on the value of the average parameter:

#### Weighted

$\frac{\sum\limits_{i=1}^{M} w_{i} F1_{i}}{\sum\limits_{i=1}^{M}w_{i}} {, where}$

$w_{i}$ is the sum of the weights of the documents which correspond to the i-th class. If document weights are not specified $w_{i}$ stands for the number of times the i-th class is found among the label values.

#### Macro

$\displaystyle\frac{\sum\limits_{i=1}^{M}F1_{i}}{M}$

#### Micro

$TotalF1 = \displaystyle\frac{2 \cdot TP}{2 \cdot TP + FP + FN} {, where}$

• $TP = \sum\limits_{i=1}^{M} TP_{i}$
• $FP = \sum\limits_{i=1}^{M} FP_{i}$
• $FN = \sum\limits_{i=1}^{M} FN_{i}$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

average

The method for averaging the value of the metric that is initially individually calculated for each class.

Possible values:

• Weighted
• Macro
• Micro

Default: Weighted

### MCC

This functions is defined in terms of a $k \times k$ confusion matrix $C$ (where k is the number of classes):

$\displaystyle\frac{\sum\limits_{k}\sum\limits_{l}\sum\limits_{m} C_{kk} C_{lm} - C_{kl}C_{mk}}{\sqrt{\sum\limits_{k} \left(\sum\limits_{l} C_{kl}\right) \left(\sum\limits_{k' | k' \neq k} \sum\limits_{l'} C_{k'l'}\right)}\sqrt{\sum\limits_{k} \left(\sum\limits_{l} C_{lk}\right) \left(\sum\limits_{k' | k' \neq k} \sum\limits_{l'} C_{l' k'}\right)}}$

See the Wikipedia article for more details.

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### Accuracy

$\displaystyle\frac{\sum\limits_{i=1}^{N}w_{i}[argmax_{j=0,...,M - 1}(a_{ij})==t_{i}]}{\sum\limits_{i=1}^{N}w_{i}} { , }$

$t \in \{0, ..., M - 1\}$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### HingeLoss

See the Wikipedia article.

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### HammingLoss

$\displaystyle\frac{\sum\limits_{i = 1}^{N} w_{i} [argmax_{j=0,...,M - 1}(a_{ij})\neq t_{i}]}{\sum\limits_{i = 1}^{N} w_{i}}$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### ZeroOneLoss

$1 - Accuracy$

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### Kappa

$1 - \displaystyle\frac{1 - Accuracy}{1 - RAccuracy}$

$RAccuracy = \displaystyle\frac{\sum\limits_{k=0}^{M - 1} n_{k_{a}}n_{k_{t}}}{(\sum\limits_{i=1}^{N}w_{i})^{2}}$

$k_{a}$ is the weighted number of times class k is predicted by the model

$k_{t}$ is the weighted number of times class k is set as the label for input objects

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### WKappa

See the formula on page 3 of the A note on the linearly weighted kappa coefficient for ordinal scales paper.

See the formula on page 3 of the A note on the linearly weighted kappa coefficient for ordinal scales paper.

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

### AUC

The calculation of this metric is disabled by default for the training dataset to speed up the training. Use the hints=skip_train~false parameter to enable the calculation.

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: False

type

The type of AUC. Defines the metric calculation principles.

Possible values:

• Mu
• OneVsAll

Examples:

AUC:type=Mu

AUC:type=OneVsAll


Default: Mu

misclass_cost_matrix

The matrix M with misclassification cost values. $M[i,j]$ in this matrix is the cost of classifying an object as a member of the class i when its' actual class is j. Applicable only if the used type of AUC is Mu.

Format for a matrix of size C:

<Value for M[0,0]>, <Value for M[0,1]>, ..., <Value for M[0,C-1]>, <Value for M[1,0]>, ..., <Value for M[C-1,0]>, ..., <Value for M[C-1,C-1]>


All diagonal elements $M[i, j]$ (such that i=j) must be equal to 0.

Note

The typeparameter is optional and is assumed to be set to Mu if the parameter is explicitly specified.

Examples:

• Three classes:
AUC:misclass_cost_matrix=0/0.5/2/1/0/1/0/0.5/0

• Two classes:
AUC:type=Mu;misclass_cost_matrix=0/0.5/1/0


Default: All non-diagonal matrix elements are set to 1. All diagonal elements М[i, j] (such that i = j) are set to 0.

## Used for optimization

Name Optimization
MultiClass +
MultiClassOneVsAll +
Precision -
Recall -
F1 -
TotalF1 -
MCC -
Accuracy -
HingeLoss -
HammingLoss -
ZeroOneLoss -
Kappa -
WKappa -
AUC -