---
metadata:
  - name: generator
    content: Diplodoc Platform v5.52.0
alternate:
  - https://catboost.ai/docs/en/concepts/python-reference_tokenizer_tokenize.md
  - href: en/concepts/python-reference_tokenizer_tokenize.md
    type: text/markdown
    title: Markdown version
  - href: ../llms.txt
    type: text/markdown
    title: llms.txt
---
> **Documentation Index:** Fetch the complete configuration index at https://catboost.ai/docs/en/llms.txt

# tokenize

<!-- source: en/_includes/work_src/reusage-tokenizer/tokenize__purpose.md -->
Tokenize the input string.
<!-- endsource: en/_includes/work_src/reusage-tokenizer/tokenize__purpose.md -->


## Method call format {#call-format}

```
tokenize(s)
```

## Parameters {#parameters}

### s

#### Description

The input string that has to be tokenized.

**Data types**

String

**Default value**

Obligatory parameter


## Type of return value {#return-value}

A list of tokens.

## Example {#example}

```python
from catboost.text_processing import Tokenizer


text="Still, I would love to see you at 12, if you don't mind"

tokenized = Tokenizer(lowercasing=True,
                      separator_type='BySense',
                      token_types=['Word', 'Number']).tokenize(text)

print tokenized
```

Output:
```bash
['still', 'i', 'would', 'love', 'to', 'see', 'you', 'at', '12', 'if', 'you', "don't", 'mind']
```