Protect ! and ^ within FOR /F when delayed expansion enabled

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Post Reply
Message
Author
dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Protect ! and ^ within FOR /F when delayed expansion enabled

#1 Post by dbenham » 20 Dec 2013 16:25

One of the common thorns when developing batch files is that FOR /F will corrupt values containing ! (and possibly ^) if delayed expansion is enabled when the FOR variable is expanded.

There are a few ways that are frequently used to handle the situation.

1) Look for a way to avoid the need for delayed expansion - often not possible without introducing the very slow CALL %%VAR%% syntax.

2) Toggle delayed expansion ON and OFF as needed within the loop. This is often very effective. But it can be problematic if variable values must persist. There is a limit to the number of SETLOCAL that can be used within any one CALL level, so that means ENDLOCAL must be used. But then there is the complexity of persisting a variable value accross the ENDLOCAL barrier. There are methods to do this, but they are complicated, and they slow down processing.

Just today I realized that there is a very convenient, simple, and fast solution using the hybrid JScript/batch utility - REPL.BAT

Simply use REPL.BAT within the IN() clause to escape all ^ and ! characters. Then within the DO() clause, Assign each token to a variable using something like set "var=%%A"!. The trailing ! will guarantee that ^^ will be properly converted into ^, yet the ! will not be included in the assigned value since it appears after the last quote.

Code: Select all

@echo off
setlocal enableDelayedExpansion

:: Parse the first token from a string within a variable
for /f %%A in ('repl "\^^|^!" "^^$&" s someVar') do (
  set "token=%%A"!
   ...
)

:: Parse the first token from each line within a file
for /f %%A in ('type "someFile.txt"^|repl "\^^|^!" "^^$&"') do (
  set "token=%%A"!
  ...
)
:: or
for /f %%A in ('repl "\^^|^!" "^^$&" <"someFile.txt"') do (
  set "token=%%A"!
  ...
)


:: Parse the first token of each line from some command output
for /f %%A in ('someCommand^|repl "\^^|^!" "^^$&"') do (
  set "token=%%A"!
  ...
)

This technique is so much easier and faster than any other technique I am aware of :D


Dave Benham

foxidrive
Expert
Posts: 6031
Joined: 10 Feb 2012 02:20

Re: Protect ! and ^ within FOR /F when delayed expansion ena

#2 Post by foxidrive » 21 Dec 2013 00:11

Nice work Dave.

Here's a short proof in a batch file:

Code: Select all

@echo off
set "somevar=!abc^^def!"

 >somefile.txt echo 123%somevar%
>>somefile.txt echo 456%somevar%
>>somefile.txt echo 789%somevar%

setlocal enableDelayedExpansion

:: Parse the first token from a string within a variable
for /f %%A in ('repl "\^^|^!" "^^$&" s someVar') do (
  echo set "token1=%%A"!
)
pause
 
:: Parse the first token from each line within a file
for /f %%A in ('type "someFile.txt"^|repl "\^^|^!" "^^$&"') do (
  echo set "token2=%%A"!
)
pause
:: or
for /f %%A in ('repl "\^^|^!" "^^$&" ^<"someFile.txt"') do (
  echo set "token3=%%A"!
)
pause


:: Parse a line from some command output
for /f "delims=" %%A in ('help cmd^|find "^!" ^|repl "\^^|^!" "^^$&"') do (
  echo set "token4=%%A"!
)
pause
del somefile.txt


penpen
Expert
Posts: 2009
Joined: 23 Jun 2013 06:15
Location: Germany

Re: Protect ! and ^ within FOR /F when delayed expansion ena

#3 Post by penpen » 21 Dec 2013 08:58

I never have used repl.bat, so the following is more a sketch,
but could your above technique be used to set a variable over the ENDLOCAL barrier, too?
Like this:

Code: Select all

for /F "tokens=* delims=" %%a in ('repl "\^^|^!" ... and escape all other poison characters ... "^^$&" s someVar') do (
   endlocal
   set "someVar=%%a"
)


penpen

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: Protect ! and ^ within FOR /F when delayed expansion ena

#4 Post by dbenham » 21 Dec 2013 09:58

@penpen - Sure can, except it will not work if the value has LineFeed (0x10) characters. You would also need to know that delayed expansion is enabled after ENDLOCAL. It shouldn't be to hard to adopt some of jeb's safe return technique to extend this to support LineFeeds as well.

But, when I say this method is fast, I mean it is fast for processing large amounts of data. There is significant CSCRIPT start-up time that makes this method relatively slow for a single line of input.

Dave Benham

Blyanadams
Posts: 23
Joined: 20 Dec 2013 05:41

Re: Protect ! and ^ within FOR /F when delayed expansion ena

#5 Post by Blyanadams » 22 Dec 2013 19:35

dbenham wrote:But, when I say this method is fast, I mean it is fast for processing large amounts of data. There is significant CSCRIPT start-up time that makes this method relatively slow for a single line of input.

Dave Benham

inside repl.bat its calling cscript as well, so does it matter its single line input or many?

Post Reply