http://stackoverflow.com/questions/4094699/how-does-the-windows-command-interpreter-cmd-exe-parse-scriptsAmen to dbenhamJan Erik wrote:The BatchLineParser:
A line of code in a batch file has multiple phases (on the command line the expansion is different!).
The process starts with phase 1
Phase/order
1) Phase(Percent):
- A double %% is replaced by a single %
- Expansion of argument variables (%1, %2, etc.)
- Expansion of %var%, if var does not exists replace it with nothing
- For a complete explanation read this from dbenham Same thread: percent expansion
1.5) Remove all <CR> (CarriageReturn 0x0d) from the line
2) Phase(Special chars, "<LF>^&|<>()"): Look at each character
- If it is a quote (") toggle the quote flag, if the quote flag is active, the other special characters are ignored (^&|<><parenthesis>)
- If it is a caret (^) the next character has no special meaning, the caret itself is removed, if the caret is the last character of the line, the next line is appended, the first charater of the next line is always handled as escaped charater.
- <LF> stops the parsing immediatly, but not with a caret in front
- If it is one of the special characters (&|<>) split the line at this point, in case of the pipe (|) both parts gets a phase restart (a bit more complex ...) For more info on how pipes are parsed and processed, look at this question and answers: Why does delayed expansion fail when inside a piped block of code?
- Parenthesis increment/decrement the parenthesis counter, the parenthesis itself will be removed, except for a closing one and counter=0
- If the line end is reached and the parenthesis counter is > 0 the next line will be appended (starts again with phase 1)
- In this phase the primary token list is build, token delimiters are <space>,;=
- In this phase REM, IF and FOR are detected, for the special handling of them.
- If the first token is "rem", only two tokens are processed, important for the multiline caret
3) Phase(echo): If "echo is on" print the result of phase 1 and 2
- For-loop-blocks are echoed multiple times, first time in the context of the for-loop, with unexpanded for-loop-vars
- For each iteration, the block is echoed with expanded for-loop-vars
---- These two phases are not really follows directly, but it makes no difference
4) Phase(for-loop-vars expansion): Expansion of %%a and so on
5) Phase(Exclamation mark): Only if delayed expansion is on, look at each character
- If it is a caret (^) the next character has no special meaning, the caret itself is removed
- If it is an exclamation mark, search for the next exclamation mark (carets are not observed anymore), expand to the content of the variable
- If no exclamation mark is found in this phase, the result is discarded, the result of phase 4 is used instead (important for the carets)
Important: At this phase quotes and other specical characters are ignored
- Expanding vars at this stage is "safe", because special characters are not detected anymore (even or )
6) Phase(call/caret doubling): Only if the cmd token is CALL
- If the first token is "call", start with phase 1 again, but stops after phase 2, delayed expansion are not processed a second time here
- Remove the first CALL, so multiple CALL's can be stacked
- Double all carets (the normal carets seems to be stay unchanged, because in phase 2 they are reduced to one, but in quotes they are effectivly doubled)
7) Phase(Execute): The command is executed
- Different tokens are used here, depends on the internal command executed
- In case of a set "name=content", the complete content of the first equal sign to the last quote of the line is used as content-token, if there is no quote after the equal sign, the rest of the line is used.
CmdLineParser:
Works like the BatchLine-Parser, but:
- Goto/call a label isn't allowed
Phase1(Percent):
- %var% will be replaced by the content of var, if the var isn't defined, the expresssion will be unchanged
- No special handling of %%, the second percent could be the beginning of a var, set var=content, %%var%% expands to %Content%
Phase5(exclamation mark): only if "DelayedExpansion" is enabled
- !var! will be replaced by the content of var, if the var isn't defined, the expresssion will be unchanged
for-loop-command-block
e.g. for /F "usebackq" %%a IN (command block) DO echo %%a
The command block will be parsed two times, at first the BatchLineParser(the loop is inside a batch) or the CmdLineParser(loop on the cmd-line) is active, at the second run always the CmdLineParser is active. In the second run, DelayedExpansion is active only if it is enabled with the registry key
The second run is like calling the line with cmd /c
Setting of variables are therefore not persistent.
Hope it helps Jan Erik
http://stackoverflow.com/questions/4094699/how-does-the-windows-command-interpreter-cmd-exe-parse-scripts/7970912#7970912http://stackoverflow.com/questions/7882395/why-is-delayed-expansion-in-a-batch-file-not-working-in-this-casedbenham wrote:1) (Percent) Starting from left, scan each character for %. If found then
- 1.1 (escape %)
If followed by another % then
Replace %% with single % and continue scan
- 1.2 (expand argument)
- Else if followed by * and command extensions are enabled then
Replace %* with the text of all command line arguments
- Else if followed by <digit> then
Replace %<digit> with argument value (replace with nothing if undefined) and continue scan
- Else if followed by ~ and command extensions are enabled then
- If followed by optional valid list of argument modifiers followed by required <digit> then
Replace %~[modifiers]<digit> with modified argument value (replace with nothing if not defined or if specified $PATH: modifier is not defined) and continue scan.
Note: modifiers are case insensitive and can appear multiple times in any order, except $PATH: modifier can only appear once and must be the last modifier before the <digit>
- Else invalid modified argument syntax raises fatal error: batch processing aborts!
- 1.3 (expand variable)
- Else if command extensions are disabled then
Look at next string of characters, breaking before % or <LF>, and call them VAR (may be an empty list)
- If next character is % then
Replace %VAR% with value of VAR (replace with nothing if VAR not defined) and continue scan
- Else goto 1.4
- Else if command extensions are enabled then
Look at next string of characters, breaking before % : or <LF>, and call them VAR (may be an empty list). If VAR breaks before : and the subsequent character is % then include : as the last character in VAR and break before %.
- If next character is % then
Replace %VAR% with value of VAR (replace with nothing if VAR not defined) and continue scan
- Else if next character is : then
- If VAR is undefined then
Remove %VAR: and continue scan.
- Else if next character is ~ then
- If next string of characters matches pattern of [integer][,[integer]]% then
Replace %VAR:~[integer][,[integer]]% with substring of value of VAR (possibly resulting in empty string) and continue scan.
- Else goto 1.4
- Else if followed by = or *= then
Invalid variable search and replace syntax raises fatal error: batch processing aborts!
- Else if next string of characters matches pattern of
- search=[replace]% then
Replace %VAR:- search=[replace]% with value of VAR after performing search and replace (possibly resulting in empty string) and continue scan
- Else goto 1.4
- 1.4 (strip %)
Else remove % and continue with scan
The above helps explain why this batchGives these results:Code: Select all
@echo off
setlocal enableDelayedExpansion
set "1var=varA"
set "~f1var=varB"
call :test "arg1"
exit /b
::
:test "arg1"
echo %%1var%% = %1var%
echo ^^^!1var^^^! = !1var!
echo --------
echo %%~f1var%% = %~f1var%
echo ^^^!~f1var^^^! = !~f1var!
exit /bNote 1 - Phase 1 occurs prior to the recognition of REM statements. This is very important because it means even a remark can generate a fatal error if it has invalid argument expansion syntax or invalid variable search and replace syntax!Code: Select all
%1var% = "arg1"var
!1var! = varA
--------
%~f1var% = P:\arg1var
!~f1var! = varBNote 2 - Another interesting consequence of the % parsing rules: Variables containing : in the name can be defined, but they cannot be expanded unless command extensions are disabled. There is one exception - a variable name containing a single colon at the end can be expanded while command extensions are enabled. However, you cannot perform substring or search and replace operations on variable names ending with a colon. The batch file below (courtesy of jeb) demonstrates this behaviorCode: Select all
@echo off
rem %~x This generates a fatal argument expansion error
echo this line is never reachedNote 3 - An interesting outcome of the order of the parsing rules that jeb lays out in his post: When performing search and replace with normal expansion, special characters should NOT be escaped (though they may be quoted). But when performing search and replace with delayed expansion, special characters MUST be escaped (unless they are quoted).Code: Select all
@echo off
setlocal
set var=content
set var:=Special
set var::=double colon
set var:~0,2=tricky
set var::~0,2=unfortunate
echo %var%
echo %var:%
echo %var::%
echo %var:~0,2%
echo %var::~0,2%
echo Now with DisableExtensions
setlocal DisableExtensions
echo %var%
echo %var:%
echo %var::%
echo %var:~0,2%
echo %var::~0,2%Code: Select all
@echo off
setlocal enableDelayedExpansion
set "var=this & that"
echo %var:&=and%
echo "%var:&=and%"
echo !var:^&=and!
echo "!var:&=and!"
http://www.dostips.com/forum/viewtopic.php?f=3&t=2500
Tags: BatchLineParser, CmdLineParser, batch parser, commandline parser, expansion phase, parsing rules, special characters
I wrote the 2nd half of the post that deals with the details of the % expansion.
I might be interested in contributing material to update the reference and or tutorials. It would be great if we could collaboratively update the site.
and there is nothing wrong with that, though I must say the site search works pretty well already.