cohere_tokenize()

Convert text into token IDs using Cohere’s tokenizer

Convert text into an array of token IDs using Cohere’s tokenizer. Useful for counting tokens before API calls or analyzing tokenization patterns.

Samples

Tokenize text

SELECT ai.cohere_tokenize(
    'embed-english-v3.0',
    'PostgreSQL is a powerful database'
);

Returns: {5432, 8754, 389, 264, 8147, 4729}

Count tokens

SELECT array_length(ai.cohere_tokenize(
    'embed-english-v3.0',
    'PostgreSQL is a powerful database'
), 1) AS token_count;

Arguments

Name	Type	Default	Required	Description
model	TEXT	-	✔	Cohere model for tokenization
text_input	TEXT	-	✔	Text to tokenize
api_key	TEXT	NULL	✖	Cohere API key
api_key_name	TEXT	NULL	✖	Name of secret containing the API key
verbose	BOOLEAN	FALSE	✖	Enable verbose logging

Returns

INT[]: Array of token IDs.

Related functions

• cohere_detokenize(): convert tokens back to text