I have a text file that consists of 85k rows. I have a need to parse that file and extract from token 2 only strings that begin with "C-" and spool those results to a file. From there, I need to remove all duplicates from that file. THe end result should be a file containing NO duplicates.
I'm using the following solution which is taking a considerable amount of time:
Code: Select all
@ECHO OFF for /f "tokens=2 delims=|" %%A in (FDRII_outline.txt) do ( ECHO "%%~A" | FINDSTR /C:"C-" >Nul 2>&1 && ECHO %%~A>>"out.txt" ) jsort out.txt /u >out.txt.new move /y out.txt.new out.txt >nul
Could the first portion of my code be replaced with another JSCRIPT solution? I could also leverage a VB script and use a dictionary but figured I'd ask first if anyone had any more efficient ways than my current solution.
What I mean when I say remove duplicates is that I need to remove the duplicate value AS WELL AS the original value. Essentially, the final file should be all instances that NEVER had a duplicate to begin with.