Ranking: objectives and metrics

Pairwise metrics

Pairwise metrics use special labeled information — pairs of dataset objects where one object is considered the “winner” and the other is considered the “loser”. This information might be not exhaustive (not all possible pairs of objects are labeled in such a way). It is also possible to specify the weight for each pair.

If GroupId is specified, then all pairs must have both members from the same group if this dataset is used in pairwise modes.

If the labeled pairs data is not specified for the dataset, then pairs are generated automatically in each group using per-object label values (labels must be specified and must be numerical). The object with a greater label value in the pair is considered the “winner”.


Name	Used for optimization	User-defined parameters	Formula and/or description
PairLogit	+	use_weights Default: true max_pairs Default: All possible pairs are generated in each group	Calculation principles Note. The object weights are not used to calculate and optimize the value of this metric. The weights of object pairs are used instead.
PairLogitPairwise	+	use_weights Default: true max_pairs Default: All possible pairs are generated in each group	Calculation principles This metric may give more accurate results on large datasets compared to PairLogit but it is calculated significantly slower. This technique is described in the Winning The Transfer Learning Track of Yahoo!’s Learning To Rank Challenge with YetiRank paper. Note. The object weights are not used to calculate and optimize the value of this metric. The weights of object pairs are used instead.
PairAccuracy	–	use_weights Default: true	Calculation principles Note. The object weights are not used to calculate the value of this metric. The weights of object pairs are used instead.


Name	Used for optimization	User-defined parameters	Formula and/or description
PairLogit	+	use_weights Default: true max_pairs Default: All possible pairs are generated in each group	Calculation principles Note. The object weights are not used to calculate and optimize the value of this metric. The weights of object pairs are used instead.
PairLogitPairwise	+	use_weights Default: true max_pairs Default: All possible pairs are generated in each group	Calculation principles This metric may give more accurate results on large datasets compared to PairLogit but it is calculated significantly slower. This technique is described in the Winning The Transfer Learning Track of Yahoo!’s Learning To Rank Challenge with YetiRank paper. Note. The object weights are not used to calculate and optimize the value of this metric. The weights of object pairs are used instead.
PairAccuracy	–	use_weights Default: true	Calculation principles Note. The object weights are not used to calculate the value of this metric. The weights of object pairs are used instead.

Groupwise metrics


Name	Used for optimization	User-defined parameters	Formula and/or description
YetiRank *	+	decay Default: 0.99 permutations Default: 10 use_weights Default: true	An approximation of ranking metrics (such as NDCG and PFound). Allows to use ranking metrics for optimization. The value of this metric can not be calculated. The metric that is written to output data if YetiRank is optimized depends on the range of all N target values ( $\text{[math]}$ ) of the dataset: $\text{[math]}$ — PFound $\text{[math]}$ — NDCG This metric gives less accurate results on big datasets compared to YetiRankPairwise but it is significantly faster. Note. The object weights are not used to optimize this metric. The group weights are used instead. This objective is used to optimize PairLogit. Automatically generated object pairs are used for this purpose. These pairs are generated independently for each object group. Use the Group weights file or the GroupWeight column of the Columns description file to change the group importance. In this case, the weight of each generated pair is multiplied by the value of the corresponding group weight.
YetiRankPairwise *	+	decay Default: 0.99 permutations Default: 10 use_weights Default: true	An approximation of ranking metrics (such as NDCG and PFound). Allows to use ranking metrics for optimization. The value of this metric can not be calculated. The metric that is written to output data if YetiRank is optimized depends on the range of all N target values ( $\text{[math]}$ ) of the dataset: $\text{[math]}$ — PFound $\text{[math]}$ — NDCG This metric gives more accurate results on big datasets compared to YetiRank but it is significantly slower. This technique is described in the Winning The Transfer Learning Track of Yahoo!’s Learning To Rank Challenge with YetiRank paper. Note. The object weights are not used to optimize this metric. The group weights are used instead. This objective is used to optimize PairLogit. Automatically generated object pairs are used for this purpose. These pairs are generated independently for each object group. Use the Group weights file or the GroupWeight column of the Columns description file to change the group importance. In this case, the weight of each generated pair is multiplied by the value of the corresponding group weight.
QueryCrossEntropy	+	alpha Default: 0.95	Calculation principles
QueryRMSE	+	use_weights Default: true	Calculation principles
QuerySoftMax	+	use_weights Default: true	Calculation principles
PFound *	–	decay Default: 0.85 top Default: –1 (all label values are used) use_weights Default: true	Calculation principles
NDCG *	_	top Default: –1 (all label values are used) use_weights Default: true type Default: Base denominator Default: LogPosition	Calculation principles
DCG *	_	top Default: –1 (all label values are used) use_weights Default: true type Default: Base denominator Default: LogPosition	Calculation principles
FilteredDCG *	_	type Default: Base denominator Default: Position	Calculation principles
AverageGain	–	top Default: This parameter is obligatory (the default value is not defined) use_weights Default: true	Represents the average value of the label values for objects with the defined top $\text{[math]}$ label values. See the AverageGain section for more details.
PrecisionAt	–	top Default: –1 (all label values are used) border Default: 0.5	Calculation principles
RecallAt	–	top Default: –1 (all label values are used) border Default: 0.5	Calculation principles
MAP	–	top Default: –1 (all label values are used) border Default: 0.5	Calculation principles


Name	Used for optimization	User-defined parameters	Formula and/or description
YetiRank *	+	decay Default: 0.99 permutations Default: 10 use_weights Default: true	An approximation of ranking metrics (such as NDCG and PFound). Allows to use ranking metrics for optimization. The value of this metric can not be calculated. The metric that is written to output data if YetiRank is optimized depends on the range of all N target values ( $\text{[math]}$ ) of the dataset: $\text{[math]}$ — PFound $\text{[math]}$ — NDCG This metric gives less accurate results on big datasets compared to YetiRankPairwise but it is significantly faster. Note. The object weights are not used to optimize this metric. The group weights are used instead. This objective is used to optimize PairLogit. Automatically generated object pairs are used for this purpose. These pairs are generated independently for each object group. Use the Group weights file or the GroupWeight column of the Columns description file to change the group importance. In this case, the weight of each generated pair is multiplied by the value of the corresponding group weight.
YetiRankPairwise *	+	decay Default: 0.99 permutations Default: 10 use_weights Default: true	An approximation of ranking metrics (such as NDCG and PFound). Allows to use ranking metrics for optimization. The value of this metric can not be calculated. The metric that is written to output data if YetiRank is optimized depends on the range of all N target values ( $\text{[math]}$ ) of the dataset: $\text{[math]}$ — PFound $\text{[math]}$ — NDCG This metric gives more accurate results on big datasets compared to YetiRank but it is significantly slower. This technique is described in the Winning The Transfer Learning Track of Yahoo!’s Learning To Rank Challenge with YetiRank paper. Note. The object weights are not used to optimize this metric. The group weights are used instead. This objective is used to optimize PairLogit. Automatically generated object pairs are used for this purpose. These pairs are generated independently for each object group. Use the Group weights file or the GroupWeight column of the Columns description file to change the group importance. In this case, the weight of each generated pair is multiplied by the value of the corresponding group weight.
QueryCrossEntropy	+	alpha Default: 0.95	Calculation principles
QueryRMSE	+	use_weights Default: true	Calculation principles
QuerySoftMax	+	use_weights Default: true	Calculation principles
PFound *	–	decay Default: 0.85 top Default: –1 (all label values are used) use_weights Default: true	Calculation principles
NDCG *	_	top Default: –1 (all label values are used) use_weights Default: true type Default: Base denominator Default: LogPosition	Calculation principles
DCG *	_	top Default: –1 (all label values are used) use_weights Default: true type Default: Base denominator Default: LogPosition	Calculation principles
FilteredDCG *	_	type Default: Base denominator Default: Position	Calculation principles
AverageGain	–	top Default: This parameter is obligatory (the default value is not defined) use_weights Default: true	Represents the average value of the label values for objects with the defined top $\text{[math]}$ label values. See the AverageGain section for more details.
PrecisionAt	–	top Default: –1 (all label values are used) border Default: 0.5	Calculation principles
RecallAt	–	top Default: –1 (all label values are used) border Default: 0.5	Calculation principles
MAP	–	top Default: –1 (all label values are used) border Default: 0.5	Calculation principles

AUC


Name	Used for optimization	User-defined parameters	Formula and/or description
AUC*	–	use_weights Default: false type Default: “Classic” for models with Logloss and CrossEntropy loss functions and “Ranking” for models with ranking loss functions.	Classic $\text{[math]}$ The sum is calculated on all pairs of objects $\text{[math]}$ such that: $\text{[math]}$ $\text{[math]}$ $\text{[math]}$ Refer to the Wikipedia article for details. If the target type is not binary, then every object with target value $\text{[math]}$ and weight $\text{[math]}$ is replaced with two objects for the metric calculation: $\text{[math]}$ with weight $\text{[math]}$ and target value 1 $\text{[math]}$ with weight $\text{[math]}$ and target value 0. Target values must be in the range [0; 1]. Ranking $\text{[math]}$ The sum is calculated on all pairs of objects $\text{[math]}$ such that: $\text{[math]}$ $\text{[math]}$


Name	Used for optimization	User-defined parameters	Formula and/or description
AUC*	–	use_weights Default: false type Default: “Classic” for models with Logloss and CrossEntropy loss functions and “Ranking” for models with ranking loss functions.	Classic $\text{[math]}$ The sum is calculated on all pairs of objects $\text{[math]}$ such that: $\text{[math]}$ $\text{[math]}$ $\text{[math]}$ Refer to the Wikipedia article for details. If the target type is not binary, then every object with target value $\text{[math]}$ and weight $\text{[math]}$ is replaced with two objects for the metric calculation: $\text{[math]}$ with