Protect ! and ^ within FOR /F when delayed expansion enabled

[dbenham]

One of the common thorns when developing batch files is that FOR /F will corrupt values containing ! (and possibly ^) if delayed expansion is enabled when the FOR variable is expanded.

There are a few ways that are frequently used to handle the situation.

1) Look for a way to avoid the need for delayed expansion - often not possible without introducing the very slow CALL %%VAR%% syntax.

2) Toggle delayed expansion ON and OFF as needed within the loop. This is often very effective. But it can be problematic if variable values must persist. There is a limit to the number of SETLOCAL that can be used within any one CALL level, so that means ENDLOCAL must be used. But then there is the complexity of persisting a variable value accross the ENDLOCAL barrier. There are methods to do this, but they are complicated, and they slow down processing.

Just today I realized that there is a very convenient, simple, and fast solution using the hybrid JScript/batch utility - REPL.BAT

Simply use REPL.BAT within the IN() clause to escape all ^ and ! characters. Then within the DO() clause, Assign each token to a variable using something like set "var=%%A"!. The trailing ! will guarantee that ^^ will be properly converted into ^, yet the ! will not be included in the assigned value since it appears after the last quote.

Code: Select all

@echo off
setlocal enableDelayedExpansion

:: Parse the first token from a string within a variable
for /f %%A in ('repl "\^^|^!" "^^$&" s someVar') do (
  set "token=%%A"!
   ...
)

:: Parse the first token from each line within a file
for /f %%A in ('type "someFile.txt"^|repl "\^^|^!" "^^$&"') do (
  set "token=%%A"!
  ...
)
:: or
for /f %%A in ('repl "\^^|^!" "^^$&" <"someFile.txt"') do (
  set "token=%%A"!
  ...
)


:: Parse the first token of each line from some command output
for /f %%A in ('someCommand^|repl "\^^|^!" "^^$&"') do (
  set "token=%%A"!
  ...
)

This technique is so much easier and faster than any other technique I am aware of


Dave Benham

[foxidrive]

Nice work Dave.

Here's a short proof in a batch file:

Code: Select all

@echo off
set "somevar=!abc^^def!"

 >somefile.txt echo 123%somevar%
>>somefile.txt echo 456%somevar%
>>somefile.txt echo 789%somevar%

setlocal enableDelayedExpansion

:: Parse the first token from a string within a variable
for /f %%A in ('repl "\^^|^!" "^^$&" s someVar') do (
  echo set "token1=%%A"!
)
pause
 
:: Parse the first token from each line within a file
for /f %%A in ('type "someFile.txt"^|repl "\^^|^!" "^^$&"') do (
  echo set "token2=%%A"!
)
pause
:: or
for /f %%A in ('repl "\^^|^!" "^^$&" ^<"someFile.txt"') do (
  echo set "token3=%%A"!
)
pause


:: Parse a line from some command output
for /f "delims=" %%A in ('help cmd^|find "^!" ^|repl "\^^|^!" "^^$&"') do (
  echo set "token4=%%A"!
)
pause
del somefile.txt

[penpen]

I never have used repl.bat, so the following is more a sketch,
but could your above technique be used to set a variable over the ENDLOCAL barrier, too?
Like this:

Code: Select all

for /F "tokens=* delims=" %%a in ('repl "\^^|^!" ... and escape all other poison characters ... "^^$&" s someVar') do (
   endlocal
   set "someVar=%%a"
)

penpen

[dbenham]

@penpen - Sure can, except it will not work if the value has LineFeed (0x10) characters. You would also need to know that delayed expansion is enabled after ENDLOCAL. It shouldn't be to hard to adopt some of jeb's safe return technique to extend this to support LineFeeds as well.

But, when I say this method is fast, I mean it is fast for processing large amounts of data. There is significant CSCRIPT start-up time that makes this method relatively slow for a single line of input.

Dave Benham

[Blyanadams]

But, when I say this method is fast, I mean it is fast for processing large amounts of data. There is significant CSCRIPT start-up time that makes this method relatively slow for a single line of input.

Dave Benham

inside repl.bat its calling cscript as well, so does it matter its single line input or many?