Tokenize Function • AI Agents Framework

</div>

This function is defined in the train module.

Tokenizes a batch of text prompts with padding and truncation enabled.

Signature

def tokenize_function(examples: Any, hf_tokenizer: Any, max_length: Any) -> Any

Parameters

Returns

Uses

• hf_tokenizer
• value.tolist

Used By

• TrainModel.tokenize_dataset

Source