Text processing

Text Processing is used to solve different tasks, including but not limited to:

• Use text data by machine learning algorithms.

  Raw texts can not be handled by machine learning algorithms and therefore must be preprocessed.

• Improve the quality of resulting models.

  The usage of different types of text preprocessing (i.e., excluding numbers from the text or replacing upper case words with lower-case ones) can lead to significant improvements in the quality of the resulting model.

CatBoost provides the following classes for text processing:

• Tokenizer
• Dictionary