cv

cv(pool=None, 
   params=None, 
   dtrain=None, 
   iterations=None, 
   num_boost_round=None,
   fold_count=3, 
   nfold=None,
   inverted=False,
   partition_random_seed=0,
   seed=None, 
   shuffle=True, 
   logging_level=None, 
   stratified=None,
   as_pandas=True,
   metric_period=None,
   verbose=None,
   verbose_eval=None,
   plot=False,
   early_stopping_rounds=None,
   folds=None)

Purpose

Perform cross-validation on the dataset.

The dataset is split into N folds. N–1 folds are used for training, and one fold is used for model performance estimation. N models are updated on each iteration K. Each model is evaluated on its' own validation dataset on each iteration. This produces N metric values on each iteration K.

The cv function calculates the average of these N values and the standard deviation. Thus, these two values are returned on each iteration.

If the dataset contains group identifiers, all objects from one group are added to the same fold when partitioning is performed.

Parameters

Parameter Possible types Description Default value

pool

Alias: dtrain

Pool

The input dataset to cross-validate.

Required parameter
params dict

The list of parameters to start training with.

Note.
  • The following parameters are not supported in cross-validation mode: save_snapshot, snapshot_file, snapshot_interval.
  • The behavior of the overfitting detector is slightly different from the training mode. Only one metric value is calculated at each iteration in the training mode, while fold_count metric values are calculated in the cross-validation mode. Therefore, all fold_count values are averaged and the best iteration is chosen based on the average metric value at each iteration.
Required parameter

iterations

Aliases:
  • num_boost_round
  • n_estimators
  • num_trees
int

The maximum number of trees that can be built when solving machine learning problems.

When using other parameters that limit the number of iterations, the final number of trees may be less than the number specified in this parameter.

1000

fold_count

Alias: nfold

int

The number of folds to split the dataset into.

3
inverted bool

Train on the test fold and evaluate the model on the training folds.

False

partition_random_seed

Alias: seed

int

Use this as the seed value for random permutation of the data.

The permutation is performed before splitting the data for cross-validation.

Each seed generates unique data splits.

0
shuffle bool

Shuffle the dataset objects before splitting into folds.

True
logging_level string

The logging level to output to stdout.

Possible values:
  • Silent — Do not output any logging information to stdout.

  • Verbose — Output the following data to stdout:

    • optimized metric
    • elapsed time of training
    • remaining time of training
  • Info — Output additional information and the number of trees.

  • Debug — Output debugging information.
None (corresponds to the Verbose logging level)
stratified bool

Perform stratified sampling.

None

It is turned on (True) by default if one of the following loss functions is selected: Logloss, MultiClass, MultiClassOneVsAll.

It is turned off (False) for all other loss functions by default.

as_pandas bool

Sets the type of return value to pandas.DataFrame.

The type of return value is dict if this parameter is set to False or the pandas Python package is not installed.

True
metric_period int

The frequency of iterations to calculate the values of objectives and metrics. The value should be a positive integer.

The usage of this parameter speeds up the training.

1

verbose

Alias: verbose_eval

  • bool
  • int

The purpose of this parameter depends on the type of the given value:

  • bool — Defines the logging level:
    • “True”  corresponds to the Verbose logging level
    • “False” corresponds to the Silent logging level
  • int — Use the Verbose logging level and set the logging period to the value of this parameter.
False
plot bool
Plot the following information during training:
  • the metric values;
  • the custom loss values;
  • the time has passed since training started;
  • the remaining time until the end of training.
This option can be used if training is performed in Jupyter notebook.
False
early_stopping_rounds int Set the overfitting detector type to Iter and stop the training after the specified number of iterations since the iteration with the optimal metric value. False
folds
  • generator
  • iterator
  • scikit-learn splitter object

Specify custom splitting indices.

The format of the input data depends on the type of the parameter:
  • generator or iterator — Train and test indices for each fold.
  • object — One of the scikit-learn Splitter Classes with the split method.

This parameter has the highest priority among other data split parameters.

None
Parameter Possible types Description Default value

pool

Alias: dtrain

Pool

The input dataset to cross-validate.

Required parameter
params dict

The list of parameters to start training with.

Note.
  • The following parameters are not supported in cross-validation mode: save_snapshot, snapshot_file, snapshot_interval.
  • The behavior of the overfitting detector is slightly different from the training mode. Only one metric value is calculated at each iteration in the training mode, while fold_count metric values are calculated in the cross-validation mode. Therefore, all fold_count values are averaged and the best iteration is chosen based on the average metric value at each iteration.
Required parameter

iterations

Aliases:
  • num_boost_round
  • n_estimators
  • num_trees
int

The maximum number of trees that can be built when solving machine learning problems.

When using other parameters that limit the number of iterations, the final number of trees may be less than the number specified in this parameter.

1000

fold_count

Alias: nfold

int

The number of folds to split the dataset into.

3
inverted bool

Train on the test fold and evaluate the model on the training folds.

False

partition_random_seed

Alias: seed

int

Use this as the seed value for random permutation of the data.

The permutation is performed before splitting the data for cross-validation.

Each seed generates unique data splits.

0
shuffle bool

Shuffle the dataset objects before splitting into folds.

True
logging_level string

The logging level to output to stdout.

Possible values:
  • Silent — Do not output any logging information to stdout.

  • Verbose — Output the following data to stdout:

    • optimized metric
    • elapsed time of training
    • remaining time of training
  • Info — Output additional information and the number of trees.

  • Debug — Output debugging information.
None (corresponds to the Verbose logging level)
stratified bool

Perform stratified sampling.

None

It is turned on (True) by default if one of the following loss functions is selected: Logloss, MultiClass, MultiClassOneVsAll.

It is turned off (False) for all other loss functions by default.

as_pandas bool

Sets the type of return value to pandas.DataFrame.

The type of return value is dict if this parameter is set to False or the pandas Python package is not installed.

True
metric_period int

The frequency of iterations to calculate the values of objectives and metrics. The value should be a positive integer.

The usage of this parameter speeds up the training.

1

verbose

Alias: verbose_eval

  • bool
  • int

The purpose of this parameter depends on the type of the given value:

  • bool — Defines the logging level:
    • “True”  corresponds to the Verbose logging level
    • “False” corresponds to the Silent logging level
  • int — Use the Verbose logging level and set the logging period to the value of this parameter.
False
plot bool
Plot the following information during training:
  • the metric values;
  • the custom loss values;
  • the time has passed since training started;
  • the remaining time until the end of training.
This option can be used if training is performed in Jupyter notebook.
False
early_stopping_rounds int Set the overfitting detector type to Iter and stop the training after the specified number of iterations since the iteration with the optimal metric value. False
folds
  • generator
  • iterator
  • scikit-learn splitter object

Specify custom splitting indices.

The format of the input data depends on the type of the parameter:
  • generator or iterator — Train and test indices for each fold.
  • object — One of the scikit-learn Splitter Classes with the split method.

This parameter has the highest priority among other data split parameters.

None

Type of return value

Depends on the value of the as_pandas parameter and the availability of the pandas Python package:
as_pandas value pandas Python package availability Type of return value
True Installed pandas.DataFrame
True Not installed dict
False Unimportant
as_pandas value pandas Python package availability Type of return value
True Installed pandas.DataFrame
True Not installed dict
False Unimportant

Each key (if the output type is dict) or column name (if the output type is pandas.DataFrame) is formed from the evaluation dataset type (train or test), metric name, and computed characteristic (std, mean, etc.). Each value is a list of corresponding computed values.

For example, if only the RMSE metric is specified in the parameters, then the return value is:

   test-Logloss-mean  test-Logloss-std  train-Logloss-mean  train-Logloss-std
0           0.398250          0.006558            0.394166           0.003950
1           0.351388          0.000644            0.348041           0.000795
2           0.340215          0.007079            0.336723           0.003994
3           0.332771          0.001593            0.329009           0.005679

Each key or column value contains the same number of calculated values as the number of training iterations (or less, if the overfitting detection is turned on and the threshold is reached earlier).

Examples

Perform cross-validation on the given dataset:

from catboost import Pool, cv

cv_data = [["France", 1924, 44],
           ["USA", 1932, 37],
           ["Switzerland", 1928, 25],
           ["Norway", 1952, 30],
           ["Japan", 1972, 35],
           ["Mexico", 1968, 112]]

labels = [1, 1, 0, 0, 0, 1]

cat_features = [0]

cv_dataset = Pool(data=cv_data,
                  label=labels,
                  cat_features=cat_features)

params = {"iterations": 100,
          "depth": 2,
          "loss_function": "Logloss",
          "verbose": False}

scores = cv(cv_dataset,
            params,
            fold_count=2, 
            plot="True")

The following is a chart plotted with Jupyter Notebook for the given example.

Perform cross-validation and save ROC curve points to the roc-curve output file:

from catboost import Pool, cv

cv_data = [["France", 1924, 44],
           ["USA", 1932, 37],
           ["Switzerland", 1928, 25],
           ["Norway", 1952, 30],
           ["Japan", 1972, 35],
           ["Mexico", 1968, 112]]

labels = [1, 1, 0, 0, 0, 1]

cv_dataset = Pool(data=cv_data,
                  label=labels,
                  cat_features=[0])

params = {"iterations": 100,
          "depth": 2,
          "loss_function": "Logloss",
          "verbose": False,
          "roc_file": "roc-file"}

scores = cv(cv_dataset,
            params,
            fold_count=2)