Multiclassification: objectives and metrics

Objectives and metrics

MultiClass

i=1Nwilog(eaitij=0M1eaij)i=1Nwi,\displaystyle\frac{\sum\limits_{i=1}^{N}w_{i}\log\left(\displaystyle\frac{e^{a_{it_{i}}}}{ \sum\limits_{j=0}^{M - 1}e^{a_{ij}}} \right)}{\sum\limits_{i=1}^{N}w_{i}} { ,}

t{0,...,M1}t \in \{0, ..., M - 1\}

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

MultiClassOneVsAll

1Mi=1Nwij=0M1[j=ti]log(pij)+[jti]log(1pij)i=1Nwi,\displaystyle\frac{\frac{1}{M}\sum\limits_{i = 1}^N w_i \sum\limits_{j = 0}^{M - 1} [j = t_i] \log(p_{ij}) + [j \neq t_i] \log(1 - p_{ij})}{\sum\limits_{i = 1}^N w_i} { ,}

t{0,...,M1}t \in \{0, ..., M - 1\}

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

Precision

This function is calculated separately for each class k numbered from 0 to M – 1.

TPTP+FP\frac{TP}{TP + FP}

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

Recall

This function is calculated separately for each class k numbered from 0 to M – 1.

TPTP+FN\frac{TP}{TP+FN}

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

F1

This function is calculated separately for each class k numbered from 0 to M – 1.

2PrecisionRecallPrecision+Recall2 \frac{Precision * Recall}{Precision + Recall}

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

TotalF1

The formula depends on the value of the average parameter:

Weighted

i=1MwiF1ii=1Mwi,where\frac{\sum\limits_{i=1}^{M} w_{i} F1_{i}}{\sum\limits_{i=1}^{M}w_{i}} {, where}

wiw_{i} is the sum of the weights of the documents which correspond to the i-th class. If document weights are not specified wiw_{i} stands for the number of times the i-th class is found among the label values.

Macro

i=1MF1iM\displaystyle\frac{\sum\limits_{i=1}^{M}F1_{i}}{M}

Micro

TotalF1=2TP2TP+FP+FN,whereTotalF1 = \displaystyle\frac{2 \cdot TP}{2 \cdot TP + FP + FN} {, where}

  • TP=i=1MTPiTP = \sum\limits_{i=1}^{M} TP_{i}
  • FP=i=1MFPiFP = \sum\limits_{i=1}^{M} FP_{i}
  • FN=i=1MFNiFN = \sum\limits_{i=1}^{M} FN_{i}

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

average

The method for averaging the value of the metric that is initially individually calculated for each class.

Possible values:

  • Weighted
  • Macro
  • Micro

Default: Weighted

MCC

This functions is defined in terms of a k×kk \times k confusion matrix CC (where k is the number of classes):

klmCkkClmCklCmkk(lCkl)(kkklCkl)k(lClk)(kkklClk)\displaystyle\frac{\sum\limits_{k}\sum\limits_{l}\sum\limits_{m} C_{kk} C_{lm} - C_{kl}C_{mk}}{\sqrt{\sum\limits_{k} \left(\sum\limits_{l} C_{kl}\right) \left(\sum\limits_{k' | k' \neq k} \sum\limits_{l'} C_{k'l'}\right)}\sqrt{\sum\limits_{k} \left(\sum\limits_{l} C_{lk}\right) \left(\sum\limits_{k' | k' \neq k} \sum\limits_{l'} C_{l' k'}\right)}}

See the Wikipedia article for more details.

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

Accuracy

i=1Nwi[argmaxj=0,...,M1(aij)==ti]i=1Nwi,\displaystyle\frac{\sum\limits_{i=1}^{N}w_{i}[argmax_{j=0,...,M - 1}(a_{ij})==t_{i}]}{\sum\limits_{i=1}^{N}w_{i}} { , }

t{0,...,M1}t \in \{0, ..., M - 1\}

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

HingeLoss

See the Wikipedia article.

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

HammingLoss

i=1Nwi[argmaxj=0,...,M1(aij)ti]i=1Nwi\displaystyle\frac{\sum\limits_{i = 1}^{N} w_{i} [argmax_{j=0,...,M - 1}(a_{ij})\neq t_{i}]}{\sum\limits_{i = 1}^{N} w_{i}}

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

ZeroOneLoss

1Accuracy1 - Accuracy

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

Kappa

11Accuracy1RAccuracy1 - \displaystyle\frac{1 - Accuracy}{1 - RAccuracy}

RAccuracy=k=0M1nkankt(i=1Nwi)2RAccuracy = \displaystyle\frac{\sum\limits_{k=0}^{M - 1} n_{k_{a}}n_{k_{t}}}{(\sum\limits_{i=1}^{N}w_{i})^{2}}

kak_{a} is the weighted number of times class k is predicted by the model

ktk_{t} is the weighted number of times class k is set as the label for input objects

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

WKappa

See the formula on page 3 of the A note on the linearly weighted kappa coefficient for ordinal scales paper.

See the formula on page 3 of the A note on the linearly weighted kappa coefficient for ordinal scales paper.

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

AUC

The calculation of this metric is disabled by default for the training dataset to speed up the training. Use the hints=skip_train~false parameter to enable the calculation.

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: False

type

The type of AUC. Defines the metric calculation principles.

Possible values:

  • Mu
  • OneVsAll

Examples:

AUC:type=Mu
AUC:type=OneVsAll

Default: Mu

misclass_cost_matrix

The matrix M with misclassification cost values. M[i,j]M[i,j] in this matrix is the cost of classifying an object as a member of the class i when its' actual class is j. Applicable only if the used type of AUC is Mu.

Format for a matrix of size C:

<Value for M[0,0]>, <Value for M[0,1]>, ..., <Value for M[0,C-1]>, <Value for M[1,0]>, ..., <Value for M[C-1,0]>, ..., <Value for M[C-1,C-1]>

All diagonal elements M[i,j]M[i, j] (such that i=j) must be equal to 0.

Note

The typeparameter is optional and is assumed to be set to Mu if the parameter is explicitly specified.

Examples:

  • Three classes:
AUC:misclass_cost_matrix=0/0.5/2/1/0/1/0/0.5/0
  • Two classes:
AUC:type=Mu;misclass_cost_matrix=0/0.5/1/0

Default: All non-diagonal matrix elements are set to 1. All diagonal elements М[i, j] (such that i = j) are set to 0.

Used for optimization

Name Optimization
MultiClass +
MultiClassOneVsAll +
Precision -
Recall -
F1 -
TotalF1 -
MCC -
Accuracy -
HingeLoss -
HammingLoss -
ZeroOneLoss -
Kappa -
WKappa -
AUC -