> ## Documentation Index
> Fetch the complete documentation index at: https://praison.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Tokenize Dataset • AI Agents Framework

> tokenize_dataset: API reference for TrainModel.tokenize_dataset

# tokenize\_dataset

<div className="flex items-center gap-2">
  <Badge color="purple">Method</Badge>
</div>

> This is a method of the [**TrainModel**](../classes/TrainModel) class in the [**train**](../modules/train) module.

## Signature

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
def tokenize_dataset(dataset: Any) -> Any
```

## Parameters

<ParamField query="dataset" type="Any" required={true}>
  No description available.
</ParamField>

### Returns

<ResponseField name="Returns" type="Any">
  The result of the operation.
</ResponseField>

## Uses

* `tokenize_function`
* `tokenized_dataset.remove_columns`

## Used By

* [`TrainModel.train_model`](../functions/TrainModel-train_model)

## Source

<Card title="View on GitHub" icon="github" href="https://github.com/MervinPraison/PraisonAI/blob/main/src/praisonai/praisonai/train.py#L195">
  `praisonai/train.py` at line 195
</Card>