Multiclassification: objectives and metrics

Objectives and metrics

MultiClass

i=1Nwilog(eaitij=0M1eaij)i=1Nwi,\displaystyle\frac{\sum\limits_{i=1}^{N}w_{i}\log\left(\displaystyle\frac{e^{a_{it_{i}}}}{ \sum\limits_{j=0}^{M - 1}e^{a_{ij}}} \right)}{\sum\limits_{i=1}^{N}w_{i}} { ,}

t{0,...,M1}t \in \{0, ..., M - 1\}

Usage information See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

MultiClassOneVsAll

1Mi=1Nwij=0M1[j=ti]log(pij)+[jti]log(1pij)i=1Nwi,\displaystyle\frac{\frac{1}{M}\sum\limits_{i = 1}^N w_i \sum\limits_{j = 0}^{M - 1} [j = t_i] \log(p_{ij}) + [j \neq t_i] \log(1 - p_{ij})}{\sum\limits_{i = 1}^N w_i} { ,}

t{0,...,M1}t \in \{0, ..., M - 1\}

Usage information See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

Precision

This function is calculated separately for each class k numbered from 0 to M – 1.

TPTP+FP\frac{TP}{TP + FP}

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

Recall

This function is calculated separately for each class k numbered from 0 to M – 1.

TPTP+FN\frac{TP}{TP+FN}

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

F

This function is calculated separately for each class k numbered from 0 to M – 1.

(1+β2)PrecisionRecall(β2Precision)+Recall(1 + \beta^2) \cdot \frac{Precision * Recall}{(\beta^2 \cdot Precision) + Recall}

Can't be used for optimization. See more.

User-defined parameters

beta

The β\beta parameter of the F metric.

Valid values are real numbers in the following range: (0;+)(0; +\infty).

Default: This parameter is obligatory (the default value is not defined)

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

F1

This function is calculated separately for each class k numbered from 0 to M – 1.

2PrecisionRecallPrecision+Recall2 \frac{Precision * Recall}{Precision + Recall}

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

TotalF1

The formula depends on the value of the average parameter:

Weighted

i=1MwiF1ii=1Mwi,where\frac{\sum\limits_{i=1}^{M} w_{i} F1_{i}}{\sum\limits_{i=1}^{M}w_{i}} {, where}

wiw_{i} is the sum of the weights of the documents which correspond to the i-th class. If document weights are not specified wiw_{i} stands for the number of times the i-th class is found among the label values.

Macro

i=1MF1iM\displaystyle\frac{\sum\limits_{i=1}^{M}F1_{i}}{M}

Micro

TotalF1=2TP2TP+FP+FN,whereTotalF1 = \displaystyle\frac{2 \cdot TP}{2 \cdot TP + FP + FN} {, where}

  • TP=i=1MTPiTP = \sum\limits_{i=1}^{M} TP_{i}
  • FP=i=1MFPiFP = \sum\limits_{i=1}^{M} FP_{i}
  • FN=i=1MFNiFN = \sum\limits_{i=1}^{M} FN_{i}

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

average

The method for averaging the value of the metric that is initially individually calculated for each class.

Default: Weighted.
Possible values: Weighted, Macro, Micro.

MCC

This functions is defined in terms of a k×kk \times k confusion matrix CC (where k is the number of classes):

klmCkkClmCklCmkk(lCkl)(kkklCkl)k(lClk)(kkklClk)\displaystyle\frac{\sum\limits_{k}\sum\limits_{l}\sum\limits_{m} C_{kk} C_{lm} - C_{kl}C_{mk}}{\sqrt{\sum\limits_{k} \left(\sum\limits_{l} C_{kl}\right) \left(\sum\limits_{k' | k' \neq k} \sum\limits_{l'} C_{k'l'}\right)}\sqrt{\sum\limits_{k} \left(\sum\limits_{l} C_{lk}\right) \left(\sum\limits_{k' | k' \neq k} \sum\limits_{l'} C_{l' k'}\right)}}

See the Wikipedia article for more details.

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

Accuracy

i=1Nwi[argmaxj=0,...,M1(aij)==ti]i=1Nwi,\displaystyle\frac{\sum\limits_{i=1}^{N}w_{i}[argmax_{j=0,...,M - 1}(a_{ij})==t_{i}]}{\sum\limits_{i=1}^{N}w_{i}} { , }

t{0,...,M1}t \in \{0, ..., M - 1\}

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

HingeLoss

See the Wikipedia article.

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

HammingLoss

i=1Nwi[argmaxj=0,...,M1(aij)ti]i=1Nwi\displaystyle\frac{\sum\limits_{i = 1}^{N} w_{i} [argmax_{j=0,...,M - 1}(a_{ij})\neq t_{i}]}{\sum\limits_{i = 1}^{N} w_{i}}

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

ZeroOneLoss

1Accuracy1 - Accuracy

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

Kappa

11Accuracy1RAccuracy1 - \displaystyle\frac{1 - Accuracy}{1 - RAccuracy}

RAccuracy=k=0M1nkankt(i=1Nwi)2RAccuracy = \displaystyle\frac{\sum\limits_{k=0}^{M - 1} n_{k_{a}}n_{k_{t}}}{(\sum\limits_{i=1}^{N}w_{i})^{2}}

kak_{a} is the weighted number of times class k is predicted by the model

ktk_{t} is the weighted number of times class k is set as the label for input objects

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

WKappa

See the formula on page 3 of the A note on the linearly weighted kappa coefficient for ordinal scales paper.

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

AUC

The calculation of this metric is disabled by default for the training dataset to speed up the training. Use the hints=skip_train~false parameter to enable the calculation.

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: False.
Examples: AUC:type=Ranking;use_weights=False.

type

The type of AUC. Defines the metric calculation principles.

Default: Mu
Possible values: Mu, OneVsAll.
Examples: AUC:type=Mu, AUC:type=OneVsAll.

misclass_cost_matrix

The matrix M with misclassification cost values. M[i,j]M[i,j] in this matrix is the cost of classifying an object as a member of the class i when its' actual class is j. Applicable only if the used type of AUC is Mu.

Format for a matrix of size C:

<Value for M[0,0]>, <Value for M[0,1]>, ..., <Value for M[0,C-1]>, <Value for M[1,0]>, ..., <Value for M[C-1,0]>, ..., <Value for M[C-1,C-1]>

All diagonal elements M[i,j]M[i, j] (such that i=j) must be equal to 0.

Note

The type parameter is optional and is assumed to be set to Mu if the parameter is explicitly specified.

Default: All non-diagonal matrix elements are set to 1. All diagonal elements М[i,j]М[i, j] (such that i = j) are set to 0.
Examples: Three classes — AUC:misclass_cost_matrix=0/0.5/2/1/0/1/0/0.5/0, Two classes — AUC:type=Mu;misclass_cost_matrix=0/0.5/1/0.

PRAUC

PRAUC is the area under the curve Precision(q)Precision(q) vs Recall(q)Recall(q) for q[0,1]q \in [0,1] where Precision(q)Precision(q) and Recall(q)Recall(q) are defined as follows.

Precision(q)=TP(q)TP(q)+FP(q),Recall(q)=TP(q)TP(q)+FN(q)Precision(q) = \frac{TP(q)}{TP(q) + FP(q)}, Recall(q) = \frac{TP(q)}{TP(q) + FN(q)}

Above TP(q)TP(q), FP(q)FP(q), FN(q)FN(q) are weights of the true positive, false positive, and false negative samples, respectively.

To calculate PRAUC for a binary classification model, specify type Classic.
In this case, TP(q)=wi[pi>q]ciTP(q)=\sum w_i [p_i > q] c_i, etc.

To calculate PRAUC for a multi-classification model, specify type OneVsAll.
In this case, positive samples are samples having class 0, all other samples are negative, and TP(q)=wi[pi0>q][ci=0]TP(q)=\sum w_i [p_{i0} > q] [c_i = 0], etc.

type

The type of PRAUC. Defines the metric calculation principles.

Type Classic is compatible with binary classification models.
Type OneVsAll is compatible with multi-classification models.

Default: Classic.
Possible values: Classic, OneVsAll.
Examples: PRAUC:type=Classic, PRAUC:type=OneVsAll.

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: False.
Examples: PRAUC:type=Classic;use_weights=False.

Used for optimization

Name Optimization GPU Support
MultiClass + +
MultiClassOneVsAll + +
Precision - +
Recall - +
F - -
F1 - +
TotalF1 - +
MCC - +
Accuracy - +
HingeLoss - -
HammingLoss - -
ZeroOneLoss - +
Kappa - -
WKappa - -
AUC - -