---
metadata:
  - name: generator
    content: Diplodoc Platform v5.52.0
alternate:
  - https://catboost.ai/docs/en/concepts/python-reference_tokenizer.md
  - href: en/concepts/python-reference_tokenizer.md
    type: text/markdown
    title: Markdown version
  - href: ../llms.txt
    type: text/markdown
    title: llms.txt
---
> **Documentation Index:** Fetch the complete configuration index at https://catboost.ai/docs/en/llms.txt

# Tokenizer

```python
class Tokenizer(lowercasing=None,
                lemmatizing=None,
                number_process_policy=None,
                number_token=None,
                separator_type=None,
                delimiter=None,
                split_by_set=None,
                skip_empty=None,
                token_types=None,
                sub_tokens_policy=None,
                languages=None)
```

## Purpose {#purpose}

<!-- source: en/_includes/work_src/reusage-python/python__tokenize_class__description.md -->
Tokenize and process the strings.
<!-- endsource: en/_includes/work_src/reusage-python/python__tokenize_class__description.md -->


## Parameters {#parameters}

<!-- source: en/_includes/work_src/reusage-tokenizer/tokenizer_options.md -->
### lowercasing

#### Description

Convert tokens to lower case.

**Data types**

bool

**Default value**

Tokens are not converted to lower case

### lemmatizing

#### Description

Perform lemmatization on tokens.

**Data types**

bool

**Default value**

Lemmatization is not performed

### number_process_policy

#### Description

The strategy to process numeric tokens. Possible values:

- Skip — Skip all numeric tokens.
- LeaveAsIs — Leave all numeric tokens as is.
- Replace — Replace all numeric tokens with a single special token. This token in specified in the `number_token` parameter.

**Data types**

string

**Default value**

LeaveAsIs

### number_token

#### Description

The special token that is used to replace all numeric tokens with.

This option can be used if the selected numeric tokens processing strategy is Replace.

This token is not converted to lower case regardless of the value of the set value of the `lowercasing` parameter.

**Data types**

string

**Default value**

Numeric tokens are left as is

### separator_type

#### Description

The tokenization method. Possible values:

- `ByDelimiter` — Split by delimiter.
- `BySense` — Try to split the string by sense.

**Data types**

string

**Default value**

`ByDelimiter` with the delimiter set to “ ” (whitespace)

### delimiter

#### Description

The symbol that is considered to be the delimiter.

Should be used if the `separator_type` parameter is set to “ ”.

**Data types**

string

**Default value**

“ ” (whitespace)

### split_by_set

#### Description

Use each single character in the `delimiter` option as an individual delimiter.

Use this parameter to apply multiple delimiters.

**Data types**

bool

**Default value**

False (the whole string specified in the `delimiter` parameter is considered as the delimiter)

### skip_empty

#### Description

Skip all empty tokens.

**Data types**

bool

**Default value**

True (empty tokens are skipped)

### token_types

#### Description

The types of tokens that should be kept after the tokenization.

Should be used if the `separator_type` parameter is set to `BySense`.

Possible values:
- `Word`
- `Number`
- `Punctuation`
- `SentenceBreak`
- `ParagraphBreak`
- `Unknown`

**Data types**

list

**Default value**

All supported types of tokens are kept

### sub_tokens_policy

#### Description

The subtokens processing policy.

Should be used if the `separator_type` parameter is set to `BySense`.

Possible values:
- `SingleToken` — All subtokens are interpreted as a single token.
- `SeveralTokens` — All subtokens are interpreted as several tokens.

**Data types**

string

**Default value**

`SingleToken`

### languages

#### Description

The list of languages to use.

Should be used if the `separator_type` parameter is set to `BySense`.

**Data types**

list of strings

**Default value**

All available languages are used (significantly slows down the procedure)
<!-- endsource: en/_includes/work_src/reusage-tokenizer/tokenizer_options.md -->


## Methods {#methods}

### [tokenize](https://catboost.ai/docs/en/concepts/python-reference_tokenizer_tokenize.md)

<!-- source: en/_includes/work_src/reusage-tokenizer/tokenize__purpose.md -->
Tokenize the input string.
<!-- endsource: en/_includes/work_src/reusage-tokenizer/tokenize__purpose.md -->