Multiclassification: objectives and metrics

• Objectives and metrics
• Used for optimization

Objectives and metrics

MultiClass

$\displaystyle\frac{\sum\limits_{i=1}^{N}w_{i}\log\left(\displaystyle\frac{e^{a_{it_{i}}}}{ \sum\limits_{j=0}^{M - 1}e^{a_{ij}}} \right)}{\sum\limits_{i=1}^{N}w_{i}} { ,}$

$t \in \{0, ..., M - 1\}$

Usage information See more.

User-defined parameters

MultiClassOneVsAll

$\displaystyle\frac{\frac{1}{M}\sum\limits_{i = 1}^N w_i \sum\limits_{j = 0}^{M - 1} [j = t_i] \log(p_{ij}) + [j \neq t_i] \log(1 - p_{ij})}{\sum\limits_{i = 1}^N w_i} { ,}$

$t \in \{0, ..., M - 1\}$

Usage information See more.

User-defined parameters

Precision

This function is calculated separately for each class k numbered from 0 to M – 1.

$\frac{TP}{TP + FP}$

Can't be used for optimization. See more.

User-defined parameters

Recall

This function is calculated separately for each class k numbered from 0 to M – 1.

$\frac{TP}{TP+FN}$

Can't be used for optimization. See more.

User-defined parameters

F

This function is calculated separately for each class k numbered from 0 to M – 1.

$(1 + \beta^2) \cdot \frac{Precision * Recall}{(\beta^2 \cdot Precision) + Recall}$

Can't be used for optimization. See more.

User-defined parameters

F1

This function is calculated separately for each class k numbered from 0 to M – 1.

$2 \frac{Precision * Recall}{Precision + Recall}$

Can't be used for optimization. See more.

User-defined parameters

TotalF1

The formula depends on the value of the average parameter:

Weighted

$\frac{\sum\limits_{i=1}^{M} w_{i} F1_{i}}{\sum\limits_{i=1}^{M}w_{i}} {, where}$

$w_{i}$ is the sum of the weights of the documents which correspond to the i-th class. If document weights are not specified $w_{i}$ stands for the number of times the i-th class is found among the label values.

Macro

$\displaystyle\frac{\sum\limits_{i=1}^{M}F1_{i}}{M}$

Micro

$TotalF1 = \displaystyle\frac{2 \cdot TP}{2 \cdot TP + FP + FN} {, where}$

• $TP = \sum\limits_{i=1}^{M} TP_{i}$
• $FP = \sum\limits_{i=1}^{M} FP_{i}$
• $FN = \sum\limits_{i=1}^{M} FN_{i}$

Can't be used for optimization. See more.

User-defined parameters

MCC

This functions is defined in terms of a $k \times k$ confusion matrix $C$ (where k is the number of classes):

$\displaystyle\frac{\sum\limits_{k}\sum\limits_{l}\sum\limits_{m} C_{kk} C_{lm} - C_{kl}C_{mk}}{\sqrt{\sum\limits_{k} \left(\sum\limits_{l} C_{kl}\right) \left(\sum\limits_{k' | k' \neq k} \sum\limits_{l'} C_{k'l'}\right)}\sqrt{\sum\limits_{k} \left(\sum\limits_{l} C_{lk}\right) \left(\sum\limits_{k' | k' \neq k} \sum\limits_{l'} C_{l' k'}\right)}}$

See the Wikipedia article for more details.

Can't be used for optimization. See more.

User-defined parameters

Accuracy

$\displaystyle\frac{\sum\limits_{i=1}^{N}w_{i}[argmax_{j=0,...,M - 1}(a_{ij})==t_{i}]}{\sum\limits_{i=1}^{N}w_{i}} { , }$

$t \in \{0, ..., M - 1\}$

Can't be used for optimization. See more.

User-defined parameters

HingeLoss

See the Wikipedia article.

Can't be used for optimization. See more.

User-defined parameters

HammingLoss

$\displaystyle\frac{\sum\limits_{i = 1}^{N} w_{i} [argmax_{j=0,...,M - 1}(a_{ij})\neq t_{i}]}{\sum\limits_{i = 1}^{N} w_{i}}$

Can't be used for optimization. See more.

User-defined parameters

ZeroOneLoss

$1 - Accuracy$

Can't be used for optimization. See more.

User-defined parameters

Kappa

$1 - \displaystyle\frac{1 - Accuracy}{1 - RAccuracy}$

$RAccuracy = \displaystyle\frac{\sum\limits_{k=0}^{M - 1} n_{k_{a}}n_{k_{t}}}{(\sum\limits_{i=1}^{N}w_{i})^{2}}$

$k_{a}$ is the weighted number of times class k is predicted by the model

$k_{t}$ is the weighted number of times class k is set as the label for input objects

Can't be used for optimization. See more.

User-defined parameters

WKappa

See the formula on page 3 of the A note on the linearly weighted kappa coefficient for ordinal scales paper.

Can't be used for optimization. See more.

User-defined parameters

AUC

The calculation of this metric is disabled by default for the training dataset to speed up the training. Use the hints=skip_train~false parameter to enable the calculation.

• Mu

  Refer to the A Performance Metric for Multi-Class Machine Learning Models paper for calculation principles

• OneVsAll

  The value is calculated separately for each class k numbered from 0 to M–1 according to the binary classification calculation principles. The objects of class k are considered positive, while all others are considered negative.

Can't be used for optimization. See more.

User-defined parameters

<Value for M[0,0]>, <Value for M[0,1]>, ..., <Value for M[0,C-1]>, <Value for M[1,0]>, ..., <Value for M[C-1,0]>, ..., <Value for M[C-1,C-1]>

PRAUC

PRAUC is the area under the curve $Precision(q)$ vs $Recall(q)$ for $q \in [0,1]$ where $Precision(q)$ and $Recall(q)$ are defined as follows.

$$Precision(q) = \frac{TP(q)}{TP(q) + FP(q)}, Recall(q) = \frac{TP(q)}{TP(q) + FN(q)}$$

Above $TP(q)$, $FP(q)$, $FN(q)$ are weights of the true positive, false positive, and false negative samples, respectively.

To calculate PRAUC for a binary classification model, specify type Classic.
In this case, $TP(q)=\sum w_i [p_i > q] c_i$, etc.

To calculate PRAUC for a multi-classification model, specify type OneVsAll.
In this case, positive samples are samples having class 0, all other samples are negative, and $TP(q)=\sum w_i [p_{i0} > q] [c_i = 0]$, etc.

Used for optimization