---
metadata:
  - name: generator
    content: Diplodoc Platform v5.52.0
alternate:
  - https://catboost.ai/docs/en/concepts/python-reference_dictionary.md
  - href: en/concepts/python-reference_dictionary.md
    type: text/markdown
    title: Markdown version
  - href: ../llms.txt
    type: text/markdown
    title: llms.txt
---
> **Documentation Index:** Fetch the complete configuration index at https://catboost.ai/docs/en/llms.txt

# Dictionary

```python
class Dictionary(token_level_type=None,
                 gram_order=None,
                 skip_step=None,
                 start_token_id=None,
                 end_of_word_policy=None,
                 end_of_sentence_policy=None,
                 occurence_lower_bound=None,
                 max_dictionary_size=None,
                 num_bpe_units=None,
                 skip_unknown=None,
                 dictionary_type='FrequencyBased')
```

## Purpose {#purpose}

Process dictionaries. The text must be [tokenized](https://catboost.ai/docs/en/concepts/python-reference_tokenizer.md) before working with dictionaries.

## Parameters {#parameters}

<!-- source: en/_includes/work_src/reusage-tokenizer/dictionary__options.md -->
### token_level_type

#### Description

The token level type. This parameter defines what should be considered a separate token.
Possible values:
- Word
- Letter

**Data types**

string

**Default value**

Word

### gram_order

#### Description

The number of words or letters in each token.

<!-- source: en/_includes/work_src/reusage-tokenizer/maybe-some-other-time.md -->
For example, let's assume that it is required to build a dictionary for the following set of tokens: <q>['maybe', 'some', 'other', 'time']</q>.
<!-- endsource: en/_includes/work_src/reusage-tokenizer/maybe-some-other-time.md -->

If the token level type is set to Word and this parameter is set to 2, the following tokens are formed:
- <q>maybe some</q>
- <q>some other</q>
- <q>other time</q>

**Data types**

int

**Default value**

1

### skip_step

#### Description

The number of words or letters to skip when joining them to tokens. This parameter takes effect if the value of the `gram_order` parameter is strictly greater than 1.

<!-- source: en/_includes/work_src/reusage-tokenizer/maybe-some-other-time.md -->
For example, let's assume that it is required to build a dictionary for the following set of tokens: <q>['maybe', 'some', 'other', 'time']</q>.
<!-- endsource: en/_includes/work_src/reusage-tokenizer/maybe-some-other-time.md -->


If the token level type is set to Word, `gram_order` is set to 2 and this parameter is set to 1, the following tokens are formed:
- <q>maybe other</q>
- <q>some time</q>


**Data types**

int

**Default value**

0

### start_token_id

#### Description

The initial shift for the token identifier.

<!-- source: en/_includes/work_src/reusage-tokenizer/maybe-some-other-time.md -->
For example, let's assume that it is required to build a dictionary for the following set of tokens: <q>['maybe', 'some', 'other', 'time']</q>.
<!-- endsource: en/_includes/work_src/reusage-tokenizer/maybe-some-other-time.md -->


If this parameter is set to 42, the following identifiers are assigned to tokens:
- 42 — <q>maybe</q>
- 43 — <q>some</q>
- 44 — <q>other</q>
- 45 — <q>time</q>

**Data types**

int

**Default value**

0

### end_of_word_policy

#### Description

The policy for processing implicit tokens that point to the end of the word.

Possible values:

- Skip
- Insert

**Data types**

string

**Default value**

Insert

### end_of_sentence_policy

#### Description

The policy for processing implicit tokens that point to the end of the sentence.

Possible values:

- Skip
- Insert

**Data types**

string

**Default value**

Skip

### occurence_lower_bound

#### Description

The lower limit of token occurrences in the text to include it in the dictionary.

**Data types**

int

**Default value**

50

### max_dictionary_size

#### Description

The maximum number of tokens in the dictionary.

**Data types**

int

**Default value**

-1 (the size of the dictionary is not limited)

### num_bpe_units

#### Description

The number of token pairs that should be combined to a single token. The most popular tokens are combined into one and added to the dictionary as a new token.

This parameter takes effect if the value of the `dictionary_type` parameter is set to Bpe.

**Data types**

int

**Default value**

0 (token pairs are not combined)

### skip_unknown

#### Description

Skip unknown tokens when building the dictionary.

This parameter takes effect if the value of the `dictionary_type` parameter is set to Bpe.

**Data types**

bool

**Default value**

False (a special common token is assigned for all unknown tokens)

### dictionary_type

#### Description

The dictionary type.

Possible values:
- FrequencyBased. Takes into account only the most frequent tokens. The size of the dictionary and the lower limit of token occurrences in the text to include it in the dictionary are set in `occurence_lower_bound` and `max_dictionary_size` parameters respectively.
- Bpe. Takes into account the most frequent tokens and then makes new tokens from combinations of the most frequent token pairs. Refer to the [Neural Machine Translation of Rare Words with Subword Units](https://arxiv.org/abs/1508.07909) paper for algorithm details. If selected, both the Frequency Based and Bpe dictionaries are created.

**Data types**

string

**Default value**

FrequencyBased
<!-- endsource: en/_includes/work_src/reusage-tokenizer/dictionary__options.md -->


## Methods {#methods}

### [fit](https://catboost.ai/docs/en/concepts/python-reference_dictionary_fit.md)

<!-- source: en/_includes/work_src/reusage-tokenizer/fit.md -->
Train a dictionary.
<!-- endsource: en/_includes/work_src/reusage-tokenizer/fit.md -->

### [apply](https://catboost.ai/docs/en/concepts/python-reference_dictionary_apply.md)

<!-- source: en/_includes/work_src/reusage-tokenizer/apply.md -->
Apply a previously trained dictionary to the input text.
<!-- endsource: en/_includes/work_src/reusage-tokenizer/apply.md -->

### [size](https://catboost.ai/docs/en/concepts/python-reference_dictionary_size.md)

<!-- source: en/_includes/work_src/reusage-tokenizer/size.md -->
Return the size of the dictionary.
<!-- endsource: en/_includes/work_src/reusage-tokenizer/size.md -->

### [get_token](https://catboost.ai/docs/en/concepts/python-reference_dictionary_get_token.md)

<!-- source: en/_includes/work_src/reusage-tokenizer/get_token.md -->
Return the token that corresponds to the given identifier.
<!-- endsource: en/_includes/work_src/reusage-tokenizer/get_token.md -->

### [get_tokens](https://catboost.ai/docs/en/concepts/python-reference_dictionary_get_tokens.md)

<!-- source: en/_includes/work_src/reusage-tokenizer/get_tokens.md -->
Return tokens that correspond to the given identifiers.
<!-- endsource: en/_includes/work_src/reusage-tokenizer/get_tokens.md -->

### [get_top_tokens](https://catboost.ai/docs/en/concepts/python-reference_dictionary_get_top_tokens.md)

<!-- source: en/_includes/work_src/reusage-tokenizer/get_top_tokens.md -->
Get the specified number of top most frequent tokens.
<!-- endsource: en/_includes/work_src/reusage-tokenizer/get_top_tokens.md -->

### [unknown_token_id](https://catboost.ai/docs/en/concepts/python-reference_dictionary_unknown_token_id.md)

<!-- source: en/_includes/work_src/reusage-tokenizer/unknown_token_id.md -->
Get the identifier of the token, which is assigned to all words that are not found in the dictionary.
<!-- endsource: en/_includes/work_src/reusage-tokenizer/unknown_token_id.md -->

### [end_of_sentence_token_id](https://catboost.ai/docs/en/concepts/python-reference_dictionary_end_of_sentence_token_id.md)

<!-- source: en/_includes/work_src/reusage-tokenizer/end_of_sentence_token_id.md -->
Get the identifier of the last token in the sentence.
<!-- endsource: en/_includes/work_src/reusage-tokenizer/end_of_sentence_token_id.md -->

### [min_unused_token_id](https://catboost.ai/docs/en/concepts/python-reference_dictionary_min_unused_token_id.md)

<!-- source: en/_includes/work_src/reusage-tokenizer/min_unused_token_id.md -->
Get the smallest unused token identifier.
<!-- endsource: en/_includes/work_src/reusage-tokenizer/min_unused_token_id.md -->

### [load](https://catboost.ai/docs/en/concepts/python-reference_dictionary_load.md)

<!-- source: en/_includes/work_src/reusage-tokenizer/load.md -->
Load the dictionary from a file.
<!-- endsource: en/_includes/work_src/reusage-tokenizer/load.md -->

### [save](https://catboost.ai/docs/en/concepts/python-reference_dictionary_save.md)

<!-- source: en/_includes/work_src/reusage-tokenizer/save.md -->
Save the dictionary to a file.
<!-- endsource: en/_includes/work_src/reusage-tokenizer/save.md -->