> ## Documentation Index
> Fetch the complete documentation index at: https://praison.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Tokenize Function • AI Agents Framework

> tokenize_function: Tokenizes a batch of text prompts with padding and truncation enabled.

# tokenize\_function

<div className="flex items-center gap-2">
  <Badge color="teal">Function</Badge>
</div>

> This function is defined in the [**train**](../modules/train) module.

Tokenizes a batch of text prompts with padding and truncation enabled.

## Signature

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
def tokenize_function(examples: Any, hf_tokenizer: Any, max_length: Any) -> Any
```

## Parameters

<ParamField query="examples" type="Any" required={true}>
  No description available.
</ParamField>

<ParamField query="hf_tokenizer" type="Any" required={true}>
  No description available.
</ParamField>

<ParamField query="max_length" type="Any" required={true}>
  No description available.
</ParamField>

### Returns

<ResponseField name="Returns" type="Any">
  The result of the operation.
</ResponseField>

## Uses

* `hf_tokenizer`
* `value.tolist`

## Used By

* [`TrainModel.tokenize_dataset`](../functions/TrainModel-tokenize_dataset)

## Source

<Card title="View on GitHub" icon="github" href="https://github.com/MervinPraison/PraisonAI/blob/main/src/praisonai/praisonai/train.py#L83">
  `praisonai/train.py` at line 83
</Card>