# Pre-tokenizers

**TODO -- coming soon.** Reference documentation on this page is not generated yet: the
rc0 bindings do not expose all of these APIs. It comes back with them, along with the
other bindings. See [`REQUIRED_FOR_V1.md`](https://github.com/huggingface/tokenizers/blob/main/REQUIRED_FOR_V1.md) for the full list.

## BertPreTokenizer[[tokenizers.pre_tokenizers.BertPreTokenizer]]

tokenizers.pre_tokenizers.BertPreTokenizer

## ByteLevel[[tokenizers.pre_tokenizers.ByteLevel]]

tokenizers.pre_tokenizers.ByteLevel

## CharDelimiterSplit[[tokenizers.pre_tokenizers.CharDelimiterSplit]]

tokenizers.pre_tokenizers.CharDelimiterSplit

## Digits[[tokenizers.pre_tokenizers.Digits]]

tokenizers.pre_tokenizers.Digits

## Metaspace[[tokenizers.pre_tokenizers.Metaspace]]

tokenizers.pre_tokenizers.Metaspace

## PreTokenizer[[tokenizers.pre_tokenizers.PreTokenizer]]

tokenizers.pre_tokenizers.PreTokenizer

## Punctuation[[tokenizers.pre_tokenizers.Punctuation]]

tokenizers.pre_tokenizers.Punctuation

## Sequence[[tokenizers.pre_tokenizers.Sequence]]

tokenizers.pre_tokenizers.Sequence

## Split[[tokenizers.pre_tokenizers.Split]]

tokenizers.pre_tokenizers.Split

## UnicodeScripts[[tokenizers.pre_tokenizers.UnicodeScripts]]

tokenizers.pre_tokenizers.UnicodeScripts

## Whitespace[[tokenizers.pre_tokenizers.Whitespace]]

tokenizers.pre_tokenizers.Whitespace

## WhitespaceSplit[[tokenizers.pre_tokenizers.WhitespaceSplit]]

tokenizers.pre_tokenizers.WhitespaceSplit

The Rust API Reference is available directly on the [Docs.rs](https://docs.rs/tokenizers/latest/tokenizers/) website.