Default delims in "usebackq"

[Liviu]

A for/f with "usebackq" seems to be consider ;,= as delims by default, in addition to the (documented) space and tab, and does so even if an explicit "delims= " is specified.

Code: Select all

@echo off & setlocal disabledelayedexpansion

for /f "tokens=1-12" %%a in ("%% ~ * = ; , ! ) ( | > <") do (
  echo #1:  [%%a] [%%b] [%%c] [%%d] [%%e] [%%f] [%%g] [%%h] [%%i] [%%j] [%%k] [%%l]
)

for /f "usebackq tokens=1-13" %%a in ('%%^ ~^ *^ ^=^ ^;^ ^,^ !^ ^)^ ^(^ ^|^ ^>^ ^<^ ^"') do (
  echo #2:  [%%a] [%%b] [%%c] [%%d] [%%e] [%%f] [%%g] [%%h] [%%i] [%%j] [%%k] [%%l] [%%m]
)

for /f "usebackq tokens=1-7 delims= " %%a in ('a = b ; c , d') do (
  echo #3:  [%%a] [%%b] [%%c] [%%d] [%%e] [%%f] [%%g]
)

for /f "usebackq tokens=1-7 delims= " %%a in ('a ^= b ^; c ^, d') do (
  echo #4:  [%%a] [%%b] [%%c] [%%d] [%%e] [%%f] [%%g]
)

for /f "usebackq tokens=1-7 delims= " %%a in ('a^ ^=^ b^ ^;^ c^ ^,^ d') do (
  echo #5:  [%%a] [%%b] [%%c] [%%d] [%%e] [%%f] [%%g]
)

endlocal & goto :eof

Output, verified under XP 32b, Vista 64b and Win7 64b.

Code: Select all

C:\tmp>usebackq-test
#1:  [%] [~] [*] [=] [;] [,] [!] [)] [(] [|] [>] [<]
#2:  [%] [~] [*] [=] [;] [,] [!] [)] [(] [|] [>] [<] ["]
#3:  [a] [b] [c] [d] [] [] []
#4:  [a] [=] [b] [;] [c] [,] [d]
#5:  [a] [=] [b] [;] [c] [,] [d]

Note that line #3 skips over =;. and they need to be escaped (as is done on lines #4 and #5).

One more oddity, this one in XP alone, is that unless the spaces themselves are escaped (as is done on line #5) random errors occur once every few runs, like the following.

Code: Select all

C:\tmp>usebackq-test
#1:  [%] [~] [*] [=] [;] [,] [!] [)] [(] [|] [>] [<]
#2:  [%] [~] [*] [=] [;] [,] [!] [)] [(] [|] [>] [<] ["]
#3:  [a] [b] [c] [d] [] [] []
The system cannot find the file
♥♣?§?§.
#4:  [a] [=] [b] [;] [c] [,] [d]
The system cannot find the file ♠?x§?§Y§%a in.
#5:  [a] [=] [b] [;] [c] [,] [d]

This appears to have been fixed and no longer happens in Vista or Win7.

Liviu

[dbenham]

That behavior is not the same as "DELIMS=,=; "

The ,;= chars are no longer quoted so the parser is treating them as command token delimiters and replacing them with spaces. This takes place before the FOR command is executed.

As you have discovered, escaping the chars fixes the problem.

As evidence that this is different than the DELIMS setting, the characters are read just fine if they appear in a file instead of a string. No escaping is needed with file input.

The odd XP behavior is an old friend of Ed's - he simply loves that quirk, don't you Ed

As is stated in that long thread - XP can manifest the bug any time a FOR IN() clause contains unquoted, unescaped command token delimiters. It does not require USEBACKQ to manifest the bug.

Dave Benham

[Liviu]

The ,;= chars are no longer quoted so the parser is treating them as command token delimiters and replacing them with spaces. This takes place before the FOR command is executed.

That's what it looks like, indeed. Guess it goes back to the ground rules, and I have always had a bit of trouble reconciling the "Phase 2) In this phase the primary token list is build, token delimiters are <space>,;=" with the "for-loop-command-block e.g. for /F "usebackq" %%a IN (command block) DO echo %%a -- The command block will be parsed two times" later in http://www.dostips.com/forum/viewtopic.php?f=3&t=3587. My problem is that step 1 seems to be "tokenizing" before passing "something" over to the "for" loop, which then re-parses that "something". It's not obvious to me what that "something" is - it's neither a line per se, since it's been tokenized already, nor a list of tokens since parsing works on source code, not tokens.

Nitpicking aside, the above gave me the idea to try the following, which is easier on the eye, requires less escaping, and works from XP upwards.

Code: Select all

:: verified to work in xp, vista, win7
for /f "usebackq tokens=1-16" %%a in ('" ^ %% ~ * = ; , ! ) ( | > < "^"" "') do (
  echo #8:  [%%a] [%%~b] [%%~c] [%%~d] [%%~e] [%%~f] [%%~g] [%%~h] [%%~i] [%%~j] [%%~k] [%%~l] [%%~m] [%%~n] [%%~o] [%%p]
)

and outputs

Code: Select all

#8:  ["] [^] [%] [~] [*] [=] [;] [,] [!] [)] [(] [|] [>] [<] ["] ["]

The odd XP behavior is an old friend of Ed's

Thanks for the link.

Liviu

[foxidrive]

Code: Select all

@echo off
for /f "usebackq tokens=1-16" %%a in ('1 " ^ %% ~ * = ; , ! ) ( | > < "^"" "') do (
  echo #8:  [%%a] [%%~b] [%%~c] [%%~d] [%%~e] [%%~f] [%%~g] [%%~h] [%%~i] [%%~j] [%%~k] [%%~l] [%%~m] [%%~n] [%%~o] [%%p]
)
for /f "tokens=1-16" %%a in (" ^ %% ~ * = ; , ! ) ( | > < " "^" "") do (
  echo #9:  [%%a] [%%~b] [%%~c] [%%~d] [%%~e] [%%~f] [%%~g] [%%~h] [%%~i] [%%~j] [%%~k] [%%~l] [%%~m] [%%~n] [%%~o] [%%p]
)
pause

Just an observation: the quotes are position dependent and much the same can be done without usebackq


#8: [1] [] [^] [%] [~] [*] [=] [;] [,] [!] [)] [(] [|] [>] [<] ["""]
#9: [^] [%] [~] [*] [=] [;] [,] [!] [)] [(] [|] [>] [<] [] [^] ["]

[Liviu]

Just an observation: the quotes are position dependent and much the same can be done without usebackq

Yes, and almost. Without usebackq it's not obvious (to me, at least) how to get a token to be a "^"" quoted quote so that for example %%~p evaluates to one " double quote.

Liviu

[foxidrive]

With this: it appears that the quote-mark-in and quote-mark-out is the key. If you match the pairs of quotes, and keep poison characters within the mark-in and mark-out quotes then it apparently works. The two double quotes at the end give a single quote so it seems to only be parsing the markin or markout double quote. Maybe.

Code: Select all

@echo off
for /f "usebackq tokens=1-16" %%a in ('" ^ %% ~ * = ; , ! ) ( | > < "^"" "') do (
  echo #8:  [%%a] [%%~b] [%%~c] [%%~d] [%%~e] [%%~f] [%%~g] [%%~h] [%%~i] [%%~j] [%%~k] [%%~l] [%%~m] [%%~n] [%%~o] [%%p]
)
for /f "tokens=1-16" %%a in (""" ^ %% ~ * = ; , ! ) ( | > < ^ " "" ) do (
  echo #9:  [%%a] [%%~b] [%%~c] [%%~d] [%%~e] [%%~f] [%%~g] [%%~h] [%%~i] [%%~j] [%%~k] [%%~l] [%%~m] [%%~n] [%%~o] [%%p]
)
pause

#8: ["] [^] [%] [~] [*] [=] [;] [,] [!] [)] [(] [|] [>] [<] ["] ["]
#9: [""] [^] [%] [~] [*] [=] [;] [,] [!] [)] [(] [|] [>] [<] [^] ["]

[Liviu]

If you match the pairs of quotes, and keep poison characters within the mark-in and mark-out quotes then it apparently works.

It still doesn't return a quoted quote token - if you replace %%p with %%~p it's going to be empty. Also, both this #9 and the previous one exhibit the random "cannot find the file" error in XP, which confirms the observations from the old thread Dave pointed to that it can happen without "usebackq" too.

Liviu

[foxidrive]

okies.