Feature importance

Leaf pairs that are compared have different split values in the node on the path to these leaves. If the split condition is met (this condition depends on the feature F), the object goes to the left subtree; otherwise it goes to the right one.

$feature\_importance_{F} = \displaystyle\sum\limits_{trees, leafs_{F}} \left(v_{1} - avr \right)^{2} \cdot c_{1} + \left( v_{2} - avr \right)^{2} \cdot c_{2} { , }$

$avr = \displaystyle\frac{v_{1} \cdot c_{1} + v_{2} \cdot c_{2}}{c_{1} + c_{2}} { , where}$

• $c_1, c_2$ represent the total weight of objects in the left and right leaves respectively. This weight is equal to the number of objects in each leaf if weights are not specified for the dataset.
• $v_1, v_2$ represent the formula value in the left and right leaves respectively.

If the model uses a combination of some of the input features instead of using them individually, an average feature importance for these features is calculated and output. For example, the model uses a combination of features f54, c56 and f77. First, the feature importance is calculated for the combination of these features. Then the resulting value is divided by three and is assigned to each of the features.

If the model uses a feature both individually and in a combination with other features, the total importance value of this feature is defined using the following formula:

$feature\_total\_importance_{j} = feature\_importance + \sum\limits_{i=1}^{N}average\_feature\_importance_{i} { , where}$

• $feature\_importance_{j}$ is the individual feature importance of the j-th feature.
• $average\_feature\_importance_{i}$ is the average feature importance of the j-th feature in the i-th combinational feature.

$O(trees\_count \cdot depth \cdot 2 ^ {depth} \cdot dimension)$

The value of LossFunctionChange is defined so that the more important is the feature, the higher is its importance value.

• Minimum best value objective metric:

  $feature\_importance_{i} = metric (E_{i}v)) - metric(v)$

• Maximum best value objective metric:

  $feature\_importance_{i} = metric(v) - metric(E_{i}v)$

• Exact best value objective metric:

  $feature\_importance_{i} = |metric(E_{i}v) - best\_value| - |metric(v) - best\_value|$

In general, the value of LossFunctionChange can be negative.

Variables description:

• $E_{i}v$ is the mathematical expectation of the formula value without the $i$-th feature. If the feature $i$ is on the path to a leaf, the new leaf value is set to the weighted average of values of leaves that have different paths by feature value. Weights represent the total weight of objects in the corresponding leaf. This weight is equal to the number of objects in each leaf, if weights are not specified in the dataset.

  For feature combinations $F = (f_{1}, ..., f_{n})$, the average value in a leaf is calculated as follows:
  $E_{f_i}v = \displaystyle\left(\frac{(n - 1) v + E_{F}v}{n}\right)$

• $v$ is the vector with formula values for the dataset. The training dataset are used, if both training and validation datasets are provided.

• $metric$ is the loss function specified in the training parameters.

The size of the random subsample used for calculation is determined as follows:

$subsamples\_count = \min(samples_count, \max(2\cdot 10^5, \frac{2\cdot 10^9}{features\_count}))$

$O(trees\_count \cdot (2 ^ {depth} + subsamples\_count) \cdot depth +$

$+ Eval\_metric\_complexity(model, subsamples\_count) \cdot features\_count)$

Leaf pairs that are compared have different split values in the node on the path to these leaves. If the split condition is met (this condition depends on the feature F), the object goes to the left subtree; otherwise it goes to the right one.

$feature\_importance_{F} = \displaystyle\sum\limits_{trees, leafs_{F}} \left(v_{1} - avr \right)^{2} \cdot c_{1} + \left( v_{2} - avr \right)^{2} \cdot c_{2} { , }$

$avr = \displaystyle\frac{v_{1} \cdot c_{1} + v_{2} \cdot c_{2}}{c_{1} + c_{2}} { , where}$

• $c_{1}, c_{2}$ represent the total weight of objects in the left and right leaves respectively. This weight is equal to the number of objects in each leaf if weights are not specified for the dataset.
• $v_{1}, v_{2}$ represent the formula value in the left and right leaves respectively.

If the model uses a combination of some of the input features instead of using them individually, an average feature importance for these features is calculated and output. For example, the model uses a combination of features f54, c56 and f77. First, the feature importance is calculated for the combination of these features. Then the resulting value is divided by three and is assigned to each of the features.

If the model uses a feature both individually and in a combination with other features, the total importance value of this feature is defined using the following formula:

$feature\_total\_importance_{j} = feature\_importance + \sum\limits_{i=1}^{N}average\_feature\_importance_{i} { , where}$

• $feature\_importance_{j}$ is the individual feature importance of the j-th feature.
• $average\_feature\_importance_{i}$ is the average feature importance of the j-th feature in the i-th combinational feature.

$O(trees\_count \cdot depth \cdot 2 ^ {depth} \cdot dimension)$