Anyone using a free tool to remove duplicates in big CSVs? I had an 18,000-row file yesterday on Windows 11 and Excel crawled, so I’m hoping for something like OpenRefine or another lightweight option that won’t choke.
But > like OpenRefine or another lightweight option that won’t choke… I use PowerShell on Win 11: Import-Csv.\big.csv | Sort-Object Col1,Col2 -Unique | Export-Csv.\dedup.csv -NoTypeInformation. It’s quick on about 18k rows, but you do need to pick the columns to dedupe; which fields matter for you?