> ## Documentation Index
> Fetch the complete documentation index at: https://praison.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Deduplicate Chunks • Rust AI Agent SDK

> deduplicate_chunks: Deduplicate chunks by content similarity.

# deduplicate\_chunks

<div className="flex items-center gap-2">
  <Badge color="teal">Function</Badge>
</div>

> This function is defined in the [**rag**](../modules/rag) module.

Deduplicate chunks by content similarity.

## Signature

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
def deduplicate_chunks(chunks: Vec<ContextChunk>, _threshold: f32) -> Vec<ContextChunk>
```

## Parameters

<ParamField query="chunks" type="Vec<ContextChunk>" required={true}>
  No description available.
</ParamField>

<ParamField query="_threshold" type="f32" required={true}>
  No description available.
</ParamField>

### Returns

<ResponseField name="Returns" type="Vec<ContextChunk>">
  The result of the operation.
</ResponseField>

## Uses

* `any`
* `comparison`

## Source

<Card title="View on GitHub" icon="github" href="https://github.com/MervinPraison/PraisonAI/blob/main/src/praisonai-rust/praisonai/src/rag/mod.rs#L623">
  `praisonai/src/rag/mod.rs` at line 623
</Card>

***

## Related Documentation

<CardGroup cols={2}>
  <Card title="Rust Chunking" icon="scissors" href="/docs/rust/chunking" />

  <Card title="Rust RAG" icon="magnifying-glass" href="/docs/rust/rag" />
</CardGroup>