openai_detokenize()

Convert tokens back into readable text

Convert an array of token IDs back into readable text. This is the inverse operation of openai_tokenize() and is useful for debugging tokenization or reconstructing text from tokens.

Samples

Detokenize tokens

Convert token IDs back into text:

SELECT ai.openai_detokenize(
    'text-embedding-ada-002',
    array[1820, 25977, 46840, 23874, 389, 264, 2579, 58466]
);

Returns:

           openai_detokenize
--------------------------------------------
 the purple elephant sits on a red mushroom

Round-trip tokenization

Verify tokenization is reversible:

SELECT ai.openai_detokenize(
    'text-embedding-ada-002',
    ai.openai_tokenize('text-embedding-ada-002', 'Hello, world!')
);

Returns:

 openai_detokenize
-------------------
 Hello, world!

Arguments

Name	Type	Default	Required	Description
model	TEXT	-	✔	The OpenAI model to detokenize for (e.g., text-embedding-ada-002, gpt-4o)
tokens	INT[]	-	✔	Array of token IDs to convert back into text

Returns

TEXT: The reconstructed text from the token IDs.

Related functions

• openai_tokenize(): convert text into tokens
• openai_embed(): generate embeddings from tokens