Classification: objectives and metrics

Objectives and metrics

Logloss

i=1Nwi(cilog(pi)+(1ci)log(1pi))i=1Nwi\displaystyle\frac{ - \sum\limits_{i=1}^N w_{i}\left(c_i \log(p_{i}) + (1-c_{i}) \log(1 - p_{i})\right)}{\sum\limits_{i = 1}^{N} w_{i}}

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

CrossEntropy

i=1Nwi(tilog(pi)+(1ti)log(1pi))i=1Nwi\displaystyle\frac{- \sum\limits_{i=1}^N w_{i} \left(t_{i} \log(p_{i}) + (1 - t_{i}) \log(1 - p_{i})\right)}{\sum\limits_{i = 1}^{N} w_{i}}

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

Precision

TPTP+FP\frac{TP}{TP + FP}

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

Recall

TPTP+FN\frac{TP}{TP+FN}

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

F1

2PrecisionRecallPrecision+Recall2 \frac{Precision * Recall}{Precision + Recall}

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

BalancedAccuracy

12(TPP+TNN)\frac{1}{2} \left(\frac{TP}{P} + \frac{TN}{N} \right)
User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

BalancedErrorRate

12(FPTN+FP+FNFN+TP)\frac{1}{2} \left( \displaystyle\frac{FP}{TN + FP} + \displaystyle\frac{FN}{FN + TP} \right)

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

MCC

TPTNFPFN(TP+FP)(TP+FN)(TN+FP)(TN+FN)\displaystyle\frac{TP * TN - FP * FN}{\sqrt{(TP + FP)(TP + FN)(TN + FP)(TN + FN)}}

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

Accuracy

TP+TNi=1Nwi\frac{TP + TN}{\sum\limits_{i=1}^{N} w_{i}}

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

CtrFactor

(i=1Nwiti/N)(i=1Nwipi/N)\displaystyle\frac{\left(\sum\limits_{i = 1}^{N} w_{i} t_{i}/N\right)}{\left(\sum\limits_{i = 1}^{N} w_{i} p_{i} /N\right)}

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

AUC

The calculation of this metric is disabled by default for the training dataset to speed up the training. Use the hints=skip_train~false parameter to enable the calculation.

  • type

    The type of AUC. Defines the metric calculation principles.

    Possible values:

    • Classic
    • Ranking

    Examples:

    AUC:type=Classic
    
    AUC:type=Ranking
    

Classic

I(ai,aj)wiwjwiwj\displaystyle\frac{\sum I(a_{i}, a_{j}) \cdot w_{i} \cdot w_{j}} {\sum w_{i} \cdot w_{j}}
The sum is calculated on all pairs of objects (i,j)(i,j) such that:

  • ti=0t_{i} = 0
  • tj=1t_{j} = 1
  • I(x,y)={0,x<y0.5,x=y1,x>yI(x, y) = \begin{cases} 0 { , } & x < y \\ 0.5 { , } & x=y \\ 1 { , } & x>y \end{cases}

Refer to the Wikipedia article for details.

If the target type is not binary, then every object with target value tt and weight ww is replaced with two objects for the metric calculation:

  • o1o_{1} with weight twt \cdot w and target value 1
  • o2o_{2} with weight (1 – t) \cdot w and target value 0.

Target values must be in the range [0; 1].

Ranking

I(ai,aj)wiwjwiwj\displaystyle\frac{\sum I(a_{i}, a_{j}) \cdot w_{i} \cdot w_{j}} {\sum w_{i} * w_{j}}

The sum is calculated on all pairs of objects (i,j)(i,j) such that:

  • ti<tjt_{i} < t_{j}
  • I(x,y)={0,x<y0.5,x=y1,x>yI(x, y) = \begin{cases} 0 { , } & x < y \\ 0.5 { , } & x=y \\ 1 { , } & x>y \end{cases}

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: False

QueryAUC

  • type

    The type of AUC. Defines the metric calculation principles.

    Possible values:

    • Classic
    • Ranking

    Examples:

    AUC:type=Classic
    
    AUC:type=Ranking
    

Classic

qi,jqI(ai,aj)wiwjqi,jqwiwj\displaystyle\frac{ \sum_q \sum_{i, j \in q} \sum I(a_{i}, a_{j}) \cdot w_{i} \cdot w_{j}} { \sum_q \sum_{i, j \in q} \sum w_{i} \cdot w_{j}}
The sum is calculated on all pairs of objects (i,j)(i,j) such that:

  • ti=0t_{i} = 0
  • tj=1t_{j} = 1
  • I(x,y)={0,x<y0.5,x=y1,x>yI(x, y) = \begin{cases} 0 { , } & x < y \\ 0.5 { , } & x=y \\ 1 { , } & x>y \end{cases}

Refer to the Wikipedia article for details.

If the target type is not binary, then every object with target value tt and weight ww is replaced with two objects for the metric calculation:

  • o1o_{1} with weight twt \cdot w and target value 1
  • o2o_{2} with weight (1 – t) \cdot w and target value 0.

Target values must be in the range [0; 1].

Ranking

qi,jqI(ai,aj)wiwjqi,jqwiwj\displaystyle\frac{ \sum_q \sum_{i, j \in q} \sum I(a_{i}, a_{j}) \cdot w_{i} \cdot w_{j}} { \sum_q \sum_{i, j \in q} \sum w_{i} * w_{j}}

The sum is calculated on all pairs of objects (i,j)(i,j) such that:

  • ti<tjt_{i} < t_{j}
  • I(x,y)={0,x<y0.5,x=y1,x>yI(x, y) = \begin{cases} 0 { , } & x < y \\ 0.5 { , } & x=y \\ 1 { , } & x>y \end{cases}

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: false

NormalizedGini

See AUC.

2AUC12 AUC - 1

User-defined parameters: use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

BrierScore

i=1Nwi(piti)2i=1Nwi\displaystyle\frac{\sum\limits_{i=1}^{N} w_{i}\left(p_{i} - t_{i} \right)^{2}}{\sum\limits_{i=1}^{N} w_{i}}

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

HingeLoss

i=1Nwimax{1tipi,0}i=1Nwi,ti=±1\displaystyle\frac{\sum\limits_{i=1}^{N} w_{i} max\{1 - t_{i} p_{i}, 0\}}{\sum\limits_{i=1}^{N} w_{i}} , t_{i} = \pm 1

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

HammingLoss

i=1Nwi[[pi>0.5]==ti]]i=1Nwi\displaystyle\frac{\sum\limits_{i = 1}^{N} w_{i} [[p_{i} > 0.5] == t_{i}]]}{\sum\limits_{i=1}^{N} w_{i}}

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

ZeroOneLoss

1Accuracy1 - Accuracy

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

Kappa

11Accuracy1RAccuracy1 - \displaystyle\frac{1 - Accuracy}{1 - RAccuracy}

RAccuracy=(TN+FP)(TN+FN)+(FN+TP)(FP+TP)(i=1Nwi)2RAccuracy = \displaystyle\frac{(TN + FP) (TN + FN) + (FN + TP) (FP + TP)}{(\sum\limits_{i=1}^{N} w_{i})^{2}}

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

WKappa

See the formula on page 3 of the A note on the linearly weighted kappa coefficient for ordinal scales paper.

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

LogLikelihoodOfPrediction

The calculation consists of the following steps:

  1. Define the sum of weights (WW) and the mean target (t¯\bar{t}):

    W=iwiW = \sum\limits_{i} w_{i}

    t¯=1Witiwi\bar{t} = \frac{1}{W} \sum\limits_{i} t_{i} w_{i}

  2. Denote log-likelihood of a constant prediction:

    ll0=iwi(t¯log(t¯)+(1t¯)log(1t¯))ll_0 = \sum\limits_{i} w_{i} (\bar{t} \cdot log(\bar{t}) + (1 - \bar{t}) \cdot log(1 - \bar{t}))

  3. Calculate LogLikelihoodOfPrediction (llpllp), which reflects how the likelihood (llll) differs from the constant prediction:

    llp=ll(t,w)ll0itiwillp = \displaystyle\frac{ll(t, w) - ll_0}{\sum\limits_{i} t_{i} w_{i}}

User-defined parameters:

use_weights

Use object/group weights to calculate metrics if the specified value is true and set all weights to 1 regardless of the input data if the specified value is false.

Default: true

Used for optimization

Name Optimization
Logloss +
CrossEntropy +
Precision -
Recall -
F1 -
BalancedAccuracy -
BalancedErrorRate -
MCC -
Accuracy -
CtrFactor -
AUC -
QueryAUC -
NormalizedGini -
BrierScore -
HingeLoss -
HammingLoss -
ZeroOneLoss -
Kappa -
WKappa -
LogLikelihoodOfPrediction -