---
metadata:
  - name: generator
    content: Diplodoc Platform v5.54.5
alternate:
  - https://catboost.ai/docs/en/features/text-features.md
  - href: en/features/text-features.md
    type: text/markdown
    title: Markdown version
  - href: ../llms.txt
    type: text/markdown
    title: llms.txt
---
> **Documentation Index:** Fetch the complete configuration index at https://catboost.ai/docs/en/llms.txt

# Text features

CatBoost supports numerical, categorical, text, and embeddings features.

Text features are used to build new numeric features. See the [Transforming text features to numerical features](https://catboost.ai/docs/en/concepts/algorithm-main-stages_text-to-numeric.md) section for details.

Choose the implementation for details on the methods and/or parameters used that are required to start using text features.

## Python package

### Class / method
- [CatBoost](https://catboost.ai/docs/en/concepts/python-reference_catboost.md) ([fit](https://catboost.ai/docs/en/concepts/python-reference_catboost_fit.md))
- [CatBoostClassifier](https://catboost.ai/docs/en/concepts/python-reference_catboostclassifier.md) ([fit](https://catboost.ai/docs/en/concepts/python-reference_catboostclassifier_fit.md))
- [Pool](https://catboost.ai/docs/en/concepts/python-reference_pool.md)

#### Parameters

##### text_features

A one-dimensional array of text columns indices (specified as integers) or names (specified as strings).

<!-- source: en/_includes/work_src/reusage/python__cat_features__description__non-catfeatures-text.md -->
Use only if the `data` parameter is a two-dimensional feature matrix (has one of the following types: list, numpy.ndarray, pandas.DataFrame, pandas.Series), polars.DataFrame.

If any elements in this array are specified as names instead of indices, names for all columns must be provided. To do this, either use the `feature_names` parameter of this constructor to explicitly specify them or pass a pandas.DataFrame, polars.DataFrame with column names specified in the `data` parameter.
<!-- endsource: en/_includes/work_src/reusage/python__cat_features__description__non-catfeatures-text.md -->

### Text processing parameters

Supported [training parameters](https://catboost.ai/docs/en/references/training-parameters/index.md):

#### tokenizers

**Description**

<!-- source: en/_includes/work_src/reusage/cli__tokenizers__desc__div.md -->
Tokenizers used to preprocess Text type feature columns before creating the dictionary.

Format:
<!-- endsource: en/_includes/work_src/reusage/cli__tokenizers__desc__div.md -->

```json
[{
'TokenizerId1': <value>,
'option_name_1': <value>,
..
'option_name_N': <value>,}]
```

- `TokenizerId` — The unique name of the tokenizer.
- `option_name` — One of the [supported tokenizer options](https://catboost.ai/docs/en/references/tokenizer_options.md).

{% note info %}

This parameter works with `dictionaries` and `feature_calcers` parameters.

<!-- source: en/_includes/work_src/reusage/tokenizer-dictionaries-feature-calcers__note_div.md -->
For example, if a single tokenizer, three dictionaries and two feature calcers are given, a total of 6 new groups of features are created for each original text feature ($1 \cdot 3 \cdot 2 = 6$).
<!-- endsource: en/_includes/work_src/reusage/tokenizer-dictionaries-feature-calcers__note_div.md -->

{% endnote %}

{% cut "Usage example" %}

```python
tokenizers = [{
	'tokenizerId': 'Space',
	'delimiter': ' ',
	'separator_type': 'ByDelimiter',
},{
	'tokenizerId': 'Sense',
	'separator_type': 'BySense',
}]
```

{% endcut %}

**Possible types**

list of json

**Default value**

–

**Supported processing units**

<!-- source: en/_includes/work_src/reusage-python/cpu-and-gpu.md -->
CPU and GPU
<!-- endsource: en/_includes/work_src/reusage-python/cpu-and-gpu.md -->

#### dictionaries

**Description**

<!-- source: en/_includes/work_src/reusage/cli__dictionaries__desc__div.md -->
Dictionaries used to preprocess Text type feature columns.

Format:
<!-- endsource: en/_includes/work_src/reusage/cli__dictionaries__desc__div.md -->

```
[{
'dictionaryId1': <value>,
'option_name_1': <value>,
..
'option_name_N': <value>,}]
```

- `DictionaryId` — The unique name of dictionary.
- `option_name` — One of the [supported dictionary options](https://catboost.ai/docs/en/references/dictionaries_options.md).

{% note info %}

This parameter works with `tokenizers` and `feature_calcers` parameters.

<!-- source: en/_includes/work_src/reusage/tokenizer-dictionaries-feature-calcers__note_div.md -->
For example, if a single tokenizer, three dictionaries and two feature calcers are given, a total of 6 new groups of features are created for each original text feature ($1 \cdot 3 \cdot 2 = 6$).
<!-- endsource: en/_includes/work_src/reusage/tokenizer-dictionaries-feature-calcers__note_div.md -->

{% endnote %}

{% cut "Usage example" %}

```python
dictionaries = [{
	'dictionary_id': 'Unigram',
	'max_dictionary_size': '50000',
	'gram_order': '1',
},{
	'dictionary_id': 'Bigram',
	'max_dictionary_size': '50000',
	'gram_order': '2',
}]
```

{% endcut %}


**Possible types**

list of json

**undefined:**

–

**Supported processing units**

<!-- source: en/_includes/work_src/reusage-python/cpu-and-gpu.md -->
CPU and GPU
<!-- endsource: en/_includes/work_src/reusage-python/cpu-and-gpu.md -->

#### feature_calcers

**Description**

<!-- source: en/_includes/work_src/reusage/cli__feature-calcers__desc__div.md -->
Feature calcers used to calculate new features based on preprocessed Text type feature columns.

Format:
<!-- endsource: en/_includes/work_src/reusage/cli__feature-calcers__desc__div.md -->


```json
['FeatureCalcerName[:option_name=option_value],
]
```

- `FeatureCalcerName` — The required [feature calcer](https://catboost.ai/docs/en/references/text-processing__feature_calcers.md).

- `option_name` — Additional options for feature calcers. Refer to the [list of supported calcers](https://catboost.ai/docs/en/references/text-processing__feature_calcers.md) for details on options available for each of them.


{% note info %}

This parameter works with `tokenizers` and `dictionaries` parameters.

<!-- source: en/_includes/work_src/reusage/tokenizer-dictionaries-feature-calcers__note_div.md -->
For example, if a single tokenizer, three dictionaries and two feature calcers are given, a total of 6 new groups of features are created for each original text feature ($1 \cdot 3 \cdot 2 = 6$).
<!-- endsource: en/_includes/work_src/reusage/tokenizer-dictionaries-feature-calcers__note_div.md -->

{% cut "Usage example" %}

```python
feature_calcers = [
	'BoW:top_tokens_count=1000',
	'NaiveBayes',
]
```

{% endcut %}

{% endnote %}

**Possible types**

list of strings

**Default value**

–

**Supported processing units**

<!-- source: en/_includes/work_src/reusage-python/cpu-and-gpu.md -->
CPU and GPU
<!-- endsource: en/_includes/work_src/reusage-python/cpu-and-gpu.md -->


#### text_processing

**Description**

<!-- source: en/_includes/work_src/reusage/cli__text-processing__div.md -->
A JSON specification of tokenizers, dictionaries and feature calcers, which determine how text features are converted into a list of float features.

[Example](https://catboost.ai/docs/en/references/text-processing__specification-example.md)

Refer to the description of the following parameters for details on supported values:
<!-- endsource: en/_includes/work_src/reusage/cli__text-processing__div.md -->

- `tokenizers`
- `dictionaries`
- `feature_calcers`

{% note alert %}

Do not use this parameter with the following ones:

- `tokenizers`
- `dictionaries`
- `feature_calcers`

{% endnote %}

**Possible types**

json

**Default value**

 [Default value](https://catboost.ai/docs/en/references/text-processing__test-processing__default-value.md)

**Supported processing units**

<!-- source: en/_includes/work_src/reusage-python/cpu-and-gpu.md -->
CPU and GPU
<!-- endsource: en/_includes/work_src/reusage-python/cpu-and-gpu.md -->

### Additional classes

Additional classes are provided for text processing:

#### [Tokenizer](https://catboost.ai/docs/en/concepts/python-reference_tokenizer.md)

**Class purpose:**

<!-- source: en/_includes/work_src/reusage-python/python__tokenize_class__description.md -->
Tokenize and process the strings.
<!-- endsource: en/_includes/work_src/reusage-python/python__tokenize_class__description.md -->

#### [Dictionary](https://catboost.ai/docs/en/concepts/python-reference_dictionary.md)

**Class purpose:**

<!-- source: en/_includes/work_src/reusage-python/python__dictionary_class__description.md -->
Process dictionaries. The text must be [tokenized](https://catboost.ai/docs/en/concepts/python-reference_tokenizer.md) before working with dictionaries.
<!-- endsource: en/_includes/work_src/reusage-python/python__dictionary_class__description.md -->

## Command-line version

For the [Train a model](https://catboost.ai/docs/en/references/training-parameters/index.md) command:

### --tokenizers

**Key description:**

Tokenizers used to preprocess Text type feature columns before creating the dictionary.

Format:

```
TokenizerId[:option_name=option_value]
```

- `TokenizerId` — The unique name of the tokenizer.
- `option_name` — One of the [supported tokenizer options](https://catboost.ai/docs/en/references/tokenizer_options.md).

{% note info %}

This parameter works with `--dictionaries` and `--feature-calcers` parameters.

For example, if a single tokenizer, three dictionaries and two feature calcers are given, a total of 6 new groups of features are created for each original text feature ($1 \cdot 3 \cdot 2 = 6$).

{% cut "Usage example" %}

```
--tokenizers "Space:delimiter= :separator_type=ByDelimiter,Sense:separator_type=BySense"
```

{% endcut %}

{% endnote %}



### --dictionaries

**Command keys:**
Dictionaries used to preprocess Text type feature columns.

Format:

```
DictionaryId[:option_name=option_value]
```

- `DictionaryId` — The unique name of dictionary.
- `option_name` — One of the [supported dictionary options](https://catboost.ai/docs/en/references/dictionaries_options.md).

{% note info %}

This parameter works with `--tokenizers` and `--feature-calcers` parameters.

<!-- source: en/_includes/work_src/reusage/tokenizer-dictionaries-feature-calcers__note_div.md -->
For example, if a single tokenizer, three dictionaries and two feature calcers are given, a total of 6 new groups of features are created for each original text feature ($1 \cdot 3 \cdot 2 = 6$).
<!-- endsource: en/_includes/work_src/reusage/tokenizer-dictionaries-feature-calcers__note_div.md -->

{% cut "Usage example" %}

```
--dictionaries "Unigram:gram_order=1:max_dictionary_size=50000,Bigram:gram_order=2:max_dictionary_size=50000"
```

{% endcut %}

{% endnote %}


### --feature-calcers

**Command keys:**
Feature calcers used to calculate new features based on preprocessed Text type feature columns.

Format:

```
FeatureCalcerName[:option_name=option_value]
```

- `FeatureCalcerName` — The required [feature calcer](https://catboost.ai/docs/en/references/text-processing__feature_calcers.md).

- `option_name` — Additional options for feature calcers. Refer to the [list of supported calcers](https://catboost.ai/docs/en/references/text-processing__feature_calcers.md) for details on options available for each of them.


{% note info %}

This parameter works with `--tokenizers` and `--dictionaries` parameters.

<!-- source: en/_includes/work_src/reusage/tokenizer-dictionaries-feature-calcers__note_div.md -->
For example, if a single tokenizer, three dictionaries and two feature calcers are given, a total of 6 new groups of features are created for each original text feature ($1 \cdot 3 \cdot 2 = 6$).
<!-- endsource: en/_includes/work_src/reusage/tokenizer-dictionaries-feature-calcers__note_div.md -->

{% cut "Usage example" %}

```
--feature-calcers BoW:top_tokens_count=1000,NaiveBayes
```

{% endcut %}

{% endnote %}


### --text-processing

**Command keys:**
A JSON specification of tokenizers, dictionaries and feature calcers, which determine how text features are converted into a list of float features.

[Example](https://catboost.ai/docs/en/references/text-processing__specification-example.md)

Refer to the description of the following parameters for details on supported values:

- `--tokenizers`
- `--dictionaries`
- `--feature-calcers`

{% note alert %}

Do not use this parameter with the following ones:
- `--tokenizers`
- `--dictionaries`
- `--feature-calcers`

{% endnote %}