---
metadata:
  - name: generator
    content: Diplodoc Platform v5.52.0
alternate:
  - https://catboost.ai/docs/en/references/training-parameters/text-processing.md
  - href: en/references/training-parameters/text-processing.md
    type: text/markdown
    title: Markdown version
  - href: ../../llms.txt
    type: text/markdown
    title: llms.txt
---
> **Documentation Index:** Fetch the complete configuration index at https://catboost.ai/docs/en/llms.txt

# Text processing parameters

These parameters are only for Python package and Command-line.

## tokenizers {#tokenizers}

Command-line: `--tokenizers`

#### Description

<!-- source: en/_includes/work_src/reusage/cli__tokenizers__desc__div.md -->
Tokenizers used to preprocess Text type feature columns before creating the dictionary.

Format:
<!-- endsource: en/_includes/work_src/reusage/cli__tokenizers__desc__div.md -->

```json
[{
'TokenizerId1': <value>,
'option_name_1': <value>,
..
'option_name_N': <value>,}]
```

- `TokenizerId` — The unique name of the tokenizer.
- `option_name` — One of the [supported tokenizer options](https://catboost.ai/docs/en/references/tokenizer_options.md).

{% note info %}

This parameter works with `dictionaries` and `feature_calcers` parameters.

<!-- source: en/_includes/work_src/reusage/tokenizer-dictionaries-feature-calcers__note_div.md -->
For example, if a single tokenizer, three dictionaries and two feature calcers are given, a total of 6 new groups of features are created for each original text feature ($1 \cdot 3 \cdot 2 = 6$).
<!-- endsource: en/_includes/work_src/reusage/tokenizer-dictionaries-feature-calcers__note_div.md -->

{% endnote %}

{% cut "Usage example" %}

```python
tokenizers = [{
	'tokenizerId': 'Space',
	'delimiter': ' ',
	'separator_type': 'ByDelimiter',
},{
	'tokenizerId': 'Sense',
	'separator_type': 'BySense',
}]
```

{% endcut %}

**Type**

list of json

**Default value**

–

**Supported processing units**

CPU


## dictionaries {#dictionaries}

Command-line: `--dictionaries`

#### Description

<!-- source: en/_includes/work_src/reusage/cli__dictionaries__desc__div.md -->
Dictionaries used to preprocess Text type feature columns.

Format:
<!-- endsource: en/_includes/work_src/reusage/cli__dictionaries__desc__div.md -->

```
[{
'dictionaryId1': <value>,
'option_name_1': <value>,
..
'option_name_N': <value>,}]
```

- `DictionaryId` — The unique name of dictionary.
- `option_name` — One of the [supported dictionary options](https://catboost.ai/docs/en/references/dictionaries_options.md).

{% note info %}

This parameter works with `tokenizers` and `feature_calcers` parameters.

<!-- source: en/_includes/work_src/reusage/tokenizer-dictionaries-feature-calcers__note_div.md -->
For example, if a single tokenizer, three dictionaries and two feature calcers are given, a total of 6 new groups of features are created for each original text feature ($1 \cdot 3 \cdot 2 = 6$).
<!-- endsource: en/_includes/work_src/reusage/tokenizer-dictionaries-feature-calcers__note_div.md -->

{% endnote %}

{% cut "Usage example" %}

```python
dictionaries = [{
	'dictionary_id': 'Unigram',
	'max_dictionary_size': '50000',
	'gram_order': '1',
},{
	'dictionary_id': 'Bigram',
	'max_dictionary_size': '50000',
	'gram_order': '2',
}]
```

{% endcut %}

**Type**

list of json

**Default value**

–

**Supported processing units**

CPU


## feature_calcers {#feature_calcers}

Command-line: `--feature-calcers`

#### Description

<!-- source: en/_includes/work_src/reusage/cli__feature-calcers__desc__div.md -->
Feature calcers used to calculate new features based on preprocessed Text type feature columns.

Format:
<!-- endsource: en/_includes/work_src/reusage/cli__feature-calcers__desc__div.md -->


```json
['FeatureCalcerName[:option_name=option_value],
]
```

- `FeatureCalcerName` — The required [feature calcer](https://catboost.ai/docs/en/references/text-processing__feature_calcers.md).

- `option_name` — Additional options for feature calcers. Refer to the [list of supported calcers](https://catboost.ai/docs/en/references/text-processing__feature_calcers.md) for details on options available for each of them.


{% note info %}

This parameter works with `tokenizers` and `dictionaries` parameters.

<!-- source: en/_includes/work_src/reusage/tokenizer-dictionaries-feature-calcers__note_div.md -->
For example, if a single tokenizer, three dictionaries and two feature calcers are given, a total of 6 new groups of features are created for each original text feature ($1 \cdot 3 \cdot 2 = 6$).
<!-- endsource: en/_includes/work_src/reusage/tokenizer-dictionaries-feature-calcers__note_div.md -->

{% cut "Usage example" %}

```python
feature_calcers = [
	'BoW:top_tokens_count=1000',
	'NaiveBayes',
]
```

{% endcut %}

{% endnote %}

Type
 list of strings

**Default value**

–

**Supported processing units**

CPU

## text_processing {#text_processing}

Command-line: `--text-processing`

#### Description

<!-- source: en/_includes/work_src/reusage/cli__text-processing__div.md -->
A JSON specification of tokenizers, dictionaries and feature calcers, which determine how text features are converted into a list of float features.

[Example](https://catboost.ai/docs/en/references/text-processing__specification-example.md)

Refer to the description of the following parameters for details on supported values:
<!-- endsource: en/_includes/work_src/reusage/cli__text-processing__div.md -->

- `tokenizers`
- `dictionaries`
- `feature_calcers`

{% note alert %}

Do not use this parameter with the following ones:

- `tokenizers`
- `dictionaries`
- `feature_calcers`

{% endnote %}

**Type**

json

**Default value**

[Default value](https://catboost.ai/docs/en/references/text-processing__test-processing__default-value.md)

**Supported processing units**

CPU