Classification: objectives and metrics

Objectives and metrics

Logloss

i=1Nwi(cilog(pi)+(1ci)log(1pi))i=1Nwi\displaystyle\frac{ - \sum\limits_{i=1}^N w_{i}\left(c_i \log(p_{i}) + (1-c_{i}) \log(1 - p_{i})\right)}{\sum\limits_{i = 1}^{N} w_{i}}

Usage information See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

CrossEntropy

i=1Nwi(tilog(pi)+(1ti)log(1pi))i=1Nwi\displaystyle\frac{- \sum\limits_{i=1}^N w_{i} \left(t_{i} \log(p_{i}) + (1 - t_{i}) \log(1 - p_{i})\right)}{\sum\limits_{i = 1}^{N} w_{i}}

Usage information See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

Precision

TPTP+FP\frac{TP}{TP + FP}

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

Recall

TPTP+FN\frac{TP}{TP+FN}

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

F

(1+β2)PrecisionRecall(β2Precision)+Recall(1 + \beta^2) \cdot \frac{Precision * Recall}{(\beta^2 \cdot Precision) + Recall}

Can't be used for optimization. See more.

User-defined parameters

beta

The β\beta parameter of the F metric.

Valid values are real numbers in the following range: (0;+)(0; +\infty).

Default: This parameter is obligatory (the default value is not defined)

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

F1

2PrecisionRecallPrecision+Recall2 \frac{Precision * Recall}{Precision + Recall}

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

BalancedAccuracy

12(TPP+TNN)\frac{1}{2} \left(\frac{TP}{P} + \frac{TN}{N} \right)
User-defined parameters:

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

BalancedErrorRate

12(FPTN+FP+FNFN+TP)\frac{1}{2} \left( \displaystyle\frac{FP}{TN + FP} + \displaystyle\frac{FN}{FN + TP} \right)

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

MCC

TPTNFPFN(TP+FP)(TP+FN)(TN+FP)(TN+FN)\displaystyle\frac{TP * TN - FP * FN}{\sqrt{(TP + FP)(TP + FN)(TN + FP)(TN + FN)}}

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

Accuracy

TP+TNi=1Nwi\frac{TP + TN}{\sum\limits_{i=1}^{N} w_{i}}

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

CtrFactor

(i=1Nwiti/N)(i=1Nwipi/N)\displaystyle\frac{\left(\sum\limits_{i = 1}^{N} w_{i} t_{i}/N\right)}{\left(\sum\limits_{i = 1}^{N} w_{i} p_{i} /N\right)}

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

AUC

The calculation of this metric is disabled by default for the training dataset to speed up the training. Use the hints=skip_train~false parameter to enable the calculation.

Classic

I(ai,aj)wiwjwiwj\displaystyle\frac{\sum I(a_{i}, a_{j}) \cdot w_{i} \cdot w_{j}} {\sum w_{i} \cdot w_{j}}
The sum is calculated on all pairs of objects (i,j)(i,j) such that:

  • ti=0t_{i} = 0
  • tj=1t_{j} = 1
  • I(x,y)={0,x<y0.5,x=y1,x>yI(x, y) = \begin{cases} 0 { , } & x < y \\ 0.5 { , } & x=y \\ 1 { , } & x>y \end{cases}

Refer to the Wikipedia article for details.

If the target type is not binary, then every object with target value tt and weight ww is replaced with two objects for the metric calculation:

  • o1o_{1} with weight twt \cdot w and target value 1
  • o2o_{2} with weight (1t)w(1 – t) \cdot w and target value 0.

Target values must be in the range [0; 1].

Ranking

I(ai,aj)wiwjwiwj\displaystyle\frac{\sum I(a_{i}, a_{j}) \cdot w_{i} \cdot w_{j}} {\sum w_{i} * w_{j}}

The sum is calculated on all pairs of objects (i,j)(i,j) such that:

  • ti<tjt_{i} < t_{j}
  • I(x,y)={0,x<y0.5,x=y1,x>yI(x, y) = \begin{cases} 0 { , } & x < y \\ 0.5 { , } & x=y \\ 1 { , } & x>y \end{cases}

User-defined parameters

type

The type of AUC. Defines the metric calculation principles.

Default: Ranking.
Possible values: Classic, Ranking.
Examples: AUC:type=Classic, AUC:type=Ranking.

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: False.
Examples: QueryAUC:type=Ranking;use_weights=False.

QueryAUC

Classic type

qi,jqI(ai,aj)wiwjqi,jqwiwj\displaystyle\frac{ \sum_q \sum_{i, j \in q} \sum I(a_{i}, a_{j}) \cdot w_{i} \cdot w_{j}} { \sum_q \sum_{i, j \in q} \sum w_{i} \cdot w_{j}}
The sum is calculated on all pairs of objects (i,j)(i,j) such that:

  • ti=0t_{i} = 0
  • tj=1t_{j} = 1
  • I(x,y)={0,x<y0.5,x=y1,x>yI(x, y) = \begin{cases} 0 { , } & x < y \\ 0.5 { , } & x=y \\ 1 { , } & x>y \end{cases}

Refer to the Wikipedia article for details.

If the target type is not binary, then every object with target value tt and weight ww is replaced with two objects for the metric calculation:

  • o1o_{1} with weight twt \cdot w and target value 1
  • o2o_{2} with weight (1t)w(1 – t) \cdot w and target value 0.

Target values must be in the range [0; 1].

Ranking type

qi,jqI(ai,aj)wiwjqi,jqwiwj\displaystyle\frac{ \sum_q \sum_{i, j \in q} \sum I(a_{i}, a_{j}) \cdot w_{i} \cdot w_{j}} { \sum_q \sum_{i, j \in q} \sum w_{i} * w_{j}}

The sum is calculated on all pairs of objects (i,j)(i,j) such that:

  • ti<tjt_{i} < t_{j}
  • I(x,y)={0,x<y0.5,x=y1,x>yI(x, y) = \begin{cases} 0 { , } & x < y \\ 0.5 { , } & x=y \\ 1 { , } & x>y \end{cases}

Can't be used for optimization. See more.

User-defined parameters

type

The type of QueryAUC. Defines the metric calculation principles.

Default: Ranking.
Possible values: Classic, Ranking.
Examples: QueryAUC:type=Classic, QueryAUC:type=Ranking.

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: False.
Examples: QueryAUC:type=Ranking;use_weights=False.

PRAUC

PRAUC is the area under the curve Precision(q)Precision(q) vs Recall(q)Recall(q) for q[0,1]q \in [0,1] where Precision(q)Precision(q) and Recall(q)Recall(q) are defined as follows.

Precision(q)=TP(q)TP(q)+FP(q),Recall(q)=TP(q)TP(q)+FN(q)Precision(q) = \frac{TP(q)}{TP(q) + FP(q)}, Recall(q) = \frac{TP(q)}{TP(q) + FN(q)}

Above TP(q)TP(q), FP(q)FP(q), FN(q)FN(q) are weights of the true positive, false positive, and false negative samples, respectively.

To calculate PRAUC for a binary classification model, specify type Classic.
In this case, TP(q)=wi[pi>q]ciTP(q)=\sum w_i [p_i > q] c_i, etc.

To calculate PRAUC for a multi-classification model, specify type OneVsAll.
In this case, positive samples are samples having class 0, all other samples are negative, and TP(q)=wi[pi0>q][ci=0]TP(q)=\sum w_i [p_{i0} > q] [c_i = 0], etc.

type

The type of PRAUC. Defines the metric calculation principles.

Type Classic is compatible with binary classification models.
Type OneVsAll is compatible with multi-classification models.

Default: Classic.
Possible values: Classic, OneVsAll.
Examples: PRAUC:type=Classic, PRAUC:type=OneVsAll.

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: False.
Examples: PRAUC:type=Classic;use_weights=False.

NormalizedGini

See AUC.

2AUC12 AUC - 1

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

BrierScore

i=1Nwi(piti)2i=1Nwi\displaystyle\frac{\sum\limits_{i=1}^{N} w_{i}\left(p_{i} - t_{i} \right)^{2}}{\sum\limits_{i=1}^{N} w_{i}}

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

HingeLoss

i=1Nwimax{1tipi,0}i=1Nwi,ti=±1\displaystyle\frac{\sum\limits_{i=1}^{N} w_{i} max\{1 - t_{i} p_{i}, 0\}}{\sum\limits_{i=1}^{N} w_{i}} , t_{i} = \pm 1

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

HammingLoss

i=1Nwi[[pi>0.5]==ti]]i=1Nwi\displaystyle\frac{\sum\limits_{i = 1}^{N} w_{i} [[p_{i} > 0.5] == t_{i}]]}{\sum\limits_{i=1}^{N} w_{i}}

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

ZeroOneLoss

1Accuracy1 - Accuracy

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

Kappa

11Accuracy1RAccuracy1 - \displaystyle\frac{1 - Accuracy}{1 - RAccuracy}

RAccuracy=(TN+FP)(TN+FN)+(FN+TP)(FP+TP)(i=1Nwi)2RAccuracy = \displaystyle\frac{(TN + FP) (TN + FN) + (FN + TP) (FP + TP)}{(\sum\limits_{i=1}^{N} w_{i})^{2}}

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

WKappa

See the formula on page 3 of the A note on the linearly weighted kappa coefficient for ordinal scales paper.

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

LogLikelihoodOfPrediction

The calculation consists of the following steps:

  1. Define the sum of weights (WW) and the mean target (tˉ\bar{t}):

    W=iwiW = \sum\limits_{i} w_{i}

    tˉ=1Witiwi\bar{t} = \frac{1}{W} \sum\limits_{i} t_{i} w_{i}

  2. Denote log-likelihood of a constant prediction:

    ll0=iwi(tˉlog(tˉ)+(1tˉ)log(1tˉ))ll_0 = \sum\limits_{i} w_{i} (\bar{t} \cdot log(\bar{t}) + (1 - \bar{t}) \cdot log(1 - \bar{t}))

  3. Calculate LogLikelihoodOfPrediction (llpllp), which reflects how the likelihood (llll) differs from the constant prediction:

    llp=ll(t,w)ll0itiwillp = \displaystyle\frac{ll(t, w) - ll_0}{\sum\limits_{i} t_{i} w_{i}}

Can't be used for optimization. See more.

User-defined parameters

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

Used for optimization

Name Optimization GPU Support
Logloss + +
CrossEntropy + +
Precision - +
Recall - +
F - -
F1 - +
BalancedAccuracy - -
BalancedErrorRate - -
MCC - +
Accuracy - +
CtrFactor - -
AUC - -
QueryAUC - -
NormalizedGini - -
BrierScore - -
HingeLoss - -
HammingLoss - -
ZeroOneLoss - +
Kappa - -
WKappa - -
LogLikelihoodOfPrediction - -