Multiclassification: objectives and metrics
Objectives and metrics
MultiClass
Usage information See more.
User-defined parameters
use_weights
Use object/group weights to calculate metrics if the specified value is true
and set all weights to 1
regardless of the input data if the specified value is false
.
Default: true
MultiClassOneVsAll
Usage information See more.
User-defined parameters
use_weights
Use object/group weights to calculate metrics if the specified value is true
and set all weights to 1
regardless of the input data if the specified value is false
.
Default: true
Precision
This function is calculated separately for each class k numbered from 0 to M – 1.
Can't be used for optimization. See more.
User-defined parameters
use_weights
Use object/group weights to calculate metrics if the specified value is true
and set all weights to 1
regardless of the input data if the specified value is false
.
Default: true
Recall
This function is calculated separately for each class k numbered from 0 to M – 1.
Can't be used for optimization. See more.
User-defined parameters
use_weights
Use object/group weights to calculate metrics if the specified value is true
and set all weights to 1
regardless of the input data if the specified value is false
.
Default: true
F
This function is calculated separately for each class k numbered from 0 to M – 1.
Can't be used for optimization. See more.
User-defined parameters
beta
The parameter of the F metric.
Valid values are real numbers in the following range: .
Default: This parameter is obligatory (the default value is not defined)
use_weights
Use object/group weights to calculate metrics if the specified value is true
and set all weights to 1
regardless of the input data if the specified value is false
.
Default: true
F1
This function is calculated separately for each class k numbered from 0 to M – 1.
Can't be used for optimization. See more.
User-defined parameters
use_weights
Use object/group weights to calculate metrics if the specified value is true
and set all weights to 1
regardless of the input data if the specified value is false
.
Default: true
TotalF1
The formula depends on the value of the average parameter:
Weighted
is the sum of the weights of the documents which correspond to the i-th class. If document weights are not specified stands for the number of times the i-th class is found among the label values.
Macro
Micro
Can't be used for optimization. See more.
User-defined parameters
use_weights
Use object/group weights to calculate metrics if the specified value is true
and set all weights to 1
regardless of the input data if the specified value is false
.
Default: true
average
The method for averaging the value of the metric that is initially individually calculated for each class.
Default: Weighted.
Possible values: Weighted, Macro, Micro.
MCC
This functions is defined in terms of a confusion matrix (where k is the number of classes):
See the Wikipedia article for more details.
Can't be used for optimization. See more.
User-defined parameters
use_weights
Use object/group weights to calculate metrics if the specified value is true
and set all weights to 1
regardless of the input data if the specified value is false
.
Default: true
Accuracy
Can't be used for optimization. See more.
User-defined parameters
use_weights
Use object/group weights to calculate metrics if the specified value is true
and set all weights to 1
regardless of the input data if the specified value is false
.
Default: true
HingeLoss
See the Wikipedia article.
Can't be used for optimization. See more.
User-defined parameters
use_weights
Use object/group weights to calculate metrics if the specified value is true
and set all weights to 1
regardless of the input data if the specified value is false
.
Default: true
HammingLoss
Can't be used for optimization. See more.
User-defined parameters
use_weights
Use object/group weights to calculate metrics if the specified value is true
and set all weights to 1
regardless of the input data if the specified value is false
.
Default: true
ZeroOneLoss
Can't be used for optimization. See more.
User-defined parameters
use_weights
Use object/group weights to calculate metrics if the specified value is true
and set all weights to 1
regardless of the input data if the specified value is false
.
Default: true
Kappa
is the weighted number of times class k is predicted by the model
is the weighted number of times class k is set as the label for input objects
Can't be used for optimization. See more.
User-defined parameters
use_weights
Use object/group weights to calculate metrics if the specified value is true
and set all weights to 1
regardless of the input data if the specified value is false
.
Default: true
WKappa
See the formula on page 3 of the A note on the linearly weighted kappa coefficient for ordinal scales paper.
Can't be used for optimization. See more.
User-defined parameters
use_weights
Use object/group weights to calculate metrics if the specified value is true
and set all weights to 1
regardless of the input data if the specified value is false
.
Default: true
AUC
The calculation of this metric is disabled by default for the training dataset to speed up the training. Use the hints=skip_train~false parameter to enable the calculation.
-
Mu
Refer to the A Performance Metric for Multi-Class Machine Learning Models paper for calculation principles
-
OneVsAll
The value is calculated separately for each class k numbered from 0 to M–1 according to the binary classification calculation principles. The objects of class k are considered positive, while all others are considered negative.
Can't be used for optimization. See more.
User-defined parameters
use_weights
Use object/group weights to calculate metrics if the specified value is true
and set all weights to 1
regardless of the input data if the specified value is false
.
Default: False.
Examples: AUC:type=Ranking;use_weights=False.
type
The type of AUC. Defines the metric calculation principles.
Default: Mu
Possible values: Mu, OneVsAll.
Examples: AUC:type=Mu, AUC:type=OneVsAll.
misclass_cost_matrix
The matrix M with misclassification cost values. in this matrix is the cost of classifying an object as a member of the class i when its' actual class is j. Applicable only if the used type of AUC is Mu.
Format for a matrix of size C:
<Value for M[0,0]>, <Value for M[0,1]>, ..., <Value for M[0,C-1]>, <Value for M[1,0]>, ..., <Value for M[C-1,0]>, ..., <Value for M[C-1,C-1]>
All diagonal elements (such that i=j) must be equal to 0.
Note
The type parameter is optional and is assumed to be set to Mu if the parameter is explicitly specified.
Default: All non-diagonal matrix elements are set to 1. All diagonal elements (such that i = j) are set to 0.
Examples: Three classes — AUC:misclass_cost_matrix=0/0.5/2/1/0/1/0/0.5/0, Two classes — AUC:type=Mu;misclass_cost_matrix=0/0.5/1/0.
PRAUC
PRAUC is the area under the curve vs for where and are defined as follows.
Above , , are weights of the true positive, false positive, and false negative samples, respectively.
To calculate PRAUC for a binary classification model, specify type Classic.
In this case, , etc.
To calculate PRAUC for a multi-classification model, specify type OneVsAll.
In this case, positive samples are samples having class 0, all other samples are negative, and , etc.
type
The type of PRAUC. Defines the metric calculation principles.
Type Classic is compatible with binary classification models.
Type OneVsAll is compatible with multi-classification models.
Default: Classic.
Possible values: Classic, OneVsAll.
Examples: PRAUC:type=Classic, PRAUC:type=OneVsAll.
use_weights
Use object/group weights to calculate metrics if the specified value is true
and set all weights to 1
regardless of the input data if the specified value is false
.
Default: False.
Examples: PRAUC:type=Classic;use_weights=False.
Used for optimization
| Name | Optimization | GPU Support |
|---|---|---|
| MultiClass | + | + |
| MultiClassOneVsAll | + | + |
| Precision | - | + |
| Recall | - | + |
| F | - | - |
| F1 | - | + |
| TotalF1 | - | + |
| MCC | - | + |
| Accuracy | - | + |
| HingeLoss | - | - |
| HammingLoss | - | - |
| ZeroOneLoss | - | + |
| Kappa | - | - |
| WKappa | - | - |
| AUC | - | - |