remove duplicate string in txt file

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Post Reply
Message
Author
goodywp
Posts: 250
Joined: 31 Jul 2017 09:57

remove duplicate string in txt file

#1 Post by goodywp » 11 Oct 2017 08:48

I have a task to remove the duplicate string as below

input:

T501-08680-0102 0501-08690-0100
T501-08680-0102 T501-08699-0100

I would like to have output like this: the second line T501-08680_0102 is duplicate one should be removed

output:
T501-08680-0102 0501-08690-0100 T501-08699-0100

I tried this code but looks like hard-coded when the string is changing there is an issue

Code: Select all

@echo off
setlocal enabledelayedexpansion
set "string=T501-08680-0102 0501-08690-0100 T501-08680-0102 T501-08699-0100"
set "newstring="
for %%i in (%string%) do (
  echo !newstring!|findstr /i "\<%%i\>" >nul || set "newstring=!newstring! %%i"
)
echo %newstring:~1%


there must be some way not like the above code when I do not know which string is coming

Thanks

Squashman
Expert
Posts: 4465
Joined: 23 Dec 2011 13:59

Re: remove duplicate string in txt file

#2 Post by Squashman » 11 Oct 2017 09:06

I would recommend writing each string out to a file and sorting it. Makes it easier to remove duplicates. Dbenham created a nice function to remove duplicates in a file. I will have to search for the thread.

Aacini
Expert
Posts: 1885
Joined: 06 Dec 2011 22:15
Location: México City, México
Contact:

Re: remove duplicate string in txt file

#3 Post by Aacini » 11 Oct 2017 09:14

Code: Select all

@echo off
setlocal EnableDelayedExpansion

set "string= "
for /F "delims=" %%a in (input.txt) do for %%b in (%%a) do (
   if "!string:%%b=!" equ "!string!" set "string=!string! %%b"
)
echo %string:~2%


input.txt:

Code: Select all

T501-08680-0102 0501-08690-0100 
T501-08680-0102 T501-08699-0100

output:

Code: Select all

T501-08680-0102 0501-08690-0100 T501-08699-0100

Antonio

goodywp
Posts: 250
Joined: 31 Jul 2017 09:57

Re: remove duplicate string in txt file

#4 Post by goodywp » 11 Oct 2017 09:33

Excellent!!! Antonio!
Thanks both you and Squashman! :D

goodywp
Posts: 250
Joined: 31 Jul 2017 09:57

Re: remove duplicate string in txt file

#5 Post by goodywp » 11 Oct 2017 12:29

Hi Antonio,

Is it possible always to have the output as this order

T501-08680-0102 T501-08699-0100 0501-08690-0100

instead of
T501-08680-0102 0501-08690-0100 T501-08699-0100

Thanks

Aacini
Expert
Posts: 1885
Joined: 06 Dec 2011 22:15
Location: México City, México
Contact:

Re: remove duplicate string in txt file

#6 Post by Aacini » 11 Oct 2017 17:58

  • The output of my code is exactly the same output you posted in your question. If you want not such an output, why you posted it?
  • I don't understand why the three elements in the output should have this order: T501-08680-0102 T501-08699-0100 0501-08690-0100. Please, describe the rule used in such an ordering.
  • Please, carefully read the first topic in this forum. After that, post a new description of your problem that include all specifications...

goodywp
Posts: 250
Joined: 31 Jul 2017 09:57

Re: remove duplicate string in txt file

#7 Post by goodywp » 12 Oct 2017 07:29

Thanks Antonio! I should focus on one requirement.... :oops:

Post Reply