What It Does
- Removes duplicate rows based on one or more selected columns
- Keeps only the first occurrence of each match
- Works across text, numeric, and ID fields
- Outputs a clean dataset — ready for routing, enrichment, or export
🏁 Getting Started
1
Add the Node
Drag in the Remove Duplicates node to your play.
2
Choose Columns to Deduplicate On
Pick one or more columns — rows with matching values in all these fields will be grouped.
3
Run the Node
The output will include just the first unique row for each match group.
Inputs
🛠️ Required Field
- Deduplication Columns (✅)
Choose one or more columns to identify duplicates.
Tip: You’ll need to select at least one column — otherwise the node won’t run.
Output
Returns a clean version of your data:- One row per match group
- Original column order is preserved
- All outputs are downloadable in CSV and Parquet formats
How It Works
- Loads the dataset
- Treats all selected columns as strings to avoid type mismatch issues
- Identifies duplicates using those fields
- Keeps only the first row in each group
- Outputs a deduplicated dataset
🚀 Use Cases
| Use Case | Deduplication Columns |
|---|---|
| Clean up lead list from Nrev AI | email |
| Remove duplicate form entries | email, timestamp |
| Prep for company-level enrichment | company_id, domain |
| Deduplicate Slack exports | user_id, created_at |
✨ Pro Tips
⚠️ Important Notes
🛠 Troubleshooting
| Symptom | Likely Cause | Fix |
|---|---|---|
| All rows were dropped | All rows matched — try more columns | Add more fields to deduplication |
| Node errors on run | No column selected | Select at least one column |
| Wrong rows kept | Input ordering was unexpected | Reorder your data upstream if needed |
📝 FAQ
Can I use multiple columns?
Can I use multiple columns?
Absolutely. A row is only removed if all selected fields match another row.
Are data types normalized?
Are data types normalized?
Yes — the node treats all selected fields as strings when checking for duplicates.
Which row is kept in duplicates?
Which row is kept in duplicates?
The very first one — based on the original input order.
💰 Pricing
The Remove Duplicates node is free — no credits required.
Clean lists, clean plays. Deduplicate with confidence and send only the best rows downstream.