Sanitization of carets/exclamation marks inside a macro

[mataha]

I'm having trouble with sanitizing hazardous characters inside a macro.

The crux of the following code is L91-97:

Code: Select all

set "prefix=%~2" & if "" equ "%~2" if "" equ "%2" (set "prefix=0x")
if defined prefix (set "prefix=%prefix:^=^^^^^^^^%")
if defined prefix (set "prefix=%prefix:!=^^^^^^^!%")

set "suffix=%~3" & if "" equ "%~3" if "" equ "%3" (set "suffix=")
if defined suffix (set "suffix=%suffix:^=^^^^^^^^%")
if defined suffix (set "suffix=%suffix:!=^^^^^^^!%")

Because there are three for statements inside a macro, to my knowledge, I would need to append 2^3 - 1 carets in order to properly process both carets and exclamation marks while in a delayed expansion scope. That does not work as expected, however - carets either get doubled or halved.

The code in its entirety:

Code: Select all

@if not "Windows_NT" == "%OS%" >&2 (
    echo This program isn't compatible with the current operating system.
    goto :EOF
) else (set ^"^") >nul 2>&1 || >&2 (
    echo This program requires Command Extensions to be available to run.
    goto :EOF
)

@echo off && (if defined CMD_DEBUG echo on) & goto :main

:main
    setlocal DisableDelayedExpansion EnableExtensions

    call :init

    set result=

    call :define_strhex $strhex
    %$strhex% result -1
    %$stdout% "null: %result%"

    call :define_strhex $strhex "^^"
    %$strhex% result -1
    %$stdout% ""^^": %result%"

    call :define_strhex $strhex  ^^
    %$strhex% result -1
    %$stdout% " ^^ : %result%"

    call :define_strhex $strhex "!!"
    %$strhex% result -1
    %$stdout% ""!!": %result%"

    call :define_strhex $strhex  !!
    %$strhex% result -1
    %$stdout% " !! : %result%"

    exit /b 0

:init ()
    if not defined \n set ^"\n=^^^
%= These lines are supposed to be undented and empty; =%
%= removing them will result in hard-to-debug errors! =%

    if not defined \p if "%=^%=" == "%=%=" (
        set ^"\p=%%<nul"
    ) else (
        set ^"\p=^%<nul"
    )

    for /f "tokens=1 delims== " %\p%! in (
        "!=! ^^^!"
    ) do for /f %\p%^^ in (
        "^ ^^^^%\p%!=%\p%!"
    ) do set ^"$define=break ^& for %\p%$ in (1) do^
    @for /f "tokens=1 delims== " %\p%%\p%! in ("%\p%!=%\p%! %\p%^%\p%^%\p%^%\p%!") do^
    @for /f %\p%%\p%^^%\p%^^ in ("%\p%^ %\p%^%\p%^%\p%^%\p%^%\p%^%\p%!=%\p%^%\p%!") do @set"

    %$define% ^"$for=%\p%%\p%~$=_Always_prefix_macros_with_'$'_during_their_definition._=:$param"

    for %$for:param=s% in (stdout stderr) do (call :define_stream $%$for:param=s% %$for:param=s%)

    goto :EOF

:define_stream (name: string, stream: File = stdout)
    setlocal

    ::: https://learn.microsoft.com/en-us/windows/console/console-handles
    set "stream=" & if not "" == "%~2" if /i not "%~2" equ "stdout" if /i "%~2" equ "stdin" (
        set "stream=^>^&0"
    ) else if /i "%~2" equ "stderr" (
        set "stream=^>^&2"
    ) else (
        set "stream=^>%~2"
    )

    endlocal & %$define% ^"%~1=for %$for:param='% in (1 2) do if %$for:param='% equ 2 (%\n%
        setlocal EnableDelayedExpansion%\n%
        for /f "tokens=1,* delims= " %$for:param=0% in (": %\p%!args%\p%!") do (%\n%
            endlocal ^& endlocal ^& (echo(%$for:param=~1%) ^& (call )%\n%
        )%\n%
    ) %stream% else setlocal DisableDelayedExpansion ^& set args= "

    goto :EOF

:define_strhex (name: string, prefix: string = "0x", suffix: string = "")
    setlocal

    set "prefix=%~2" & if "" equ "%~2" if "" equ "%2" (set "prefix=0x")
    if defined prefix (set "prefix=%prefix:^=^^^^^^^^%")
    if defined prefix (set "prefix=%prefix:!=^^^^^^^!%")

    set "suffix=%~3" & if "" equ "%~3" if "" equ "%3" (set "suffix=")
    if defined suffix (set "suffix=%suffix:^=^^^^^^^^%")
    if defined suffix (set "suffix=%suffix:!=^^^^^^^!%")

    endlocal & %$define% ^"%~1=for %$for:param='% in (1 2) do if %$for:param='% equ 2 (%\n%
        for /f "tokens=1,2,* delims== " %$for:param=0% in (": %\p%!args%\p%!") do (%\n%
            endlocal ^& if not "%$for:param=~1%" equ "" (%\n%
                set /a "number=%$for:param=~2% + 0" ^>nul 2^>^&1%\n%
                set "map=0123456789abcdef"%\n%
                set "digit="%\n%
                set "result="%\n%
                for /l %$for:param=_% in (1, 1, 8) do (%\n%
                    set /a "digit=number & 15, number>>=4"%\n%
                    for %$for:param=d% in (%\p%!digit%\p%!) do (%\n%
                        set "result=%\p%!map:~%$for:param=d%,1%\p%!%\p%!result%\p%!"%\n%
                    )%\n%
                )%\n%
                for %$for:param=v% in (%prefix%%\p%!result%\p%!%suffix%) do (%\n%
                    endlocal ^& set "%$for:param=~1%=%$for:param=v%" %\n%
                )%\n%
            )%\n%
        )%\n%
    ) else setlocal EnableDelayedExpansion ^& setlocal ^& set args= "

    goto :EOF

:EOF
    @call

Result:

Code: Select all

null: 0xffffffff
"^": ^^^^ffffffff   ::: Doubled?
 ^^ : ^ffffffff     ::: Halved?
"!!": !!ffffffff
 !! : !!ffffffff

Am I doing anything wrong? I don't know anymore, I've been staring at this thing for far too long...

[jeb]

It's stupid simple

The problem is the CALL not your function.

Code: Select all

call echo #1 ^^
call echo #2 ^^^^
call echo #3 "^^"
call echo #4 "^^^^"
call call echo #5 "^^^^"
call call call echo #6 "^^^^"

#1 ^
#2 ^^
#3 "^^^^"
#4 "^^^^^^^^"
#5 "^^^^^^^^^^^^^^^^"
#6 "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"

It's tricky to transfer content like "^" as a value with CALL, with escaping only it's impossible.
But you could use double percent expansion

Code: Select all

set "caret=^"
call echo #7 "%%caret%%"

#7 "^"

Btw. It's always a good idea, to show/debug your variable in your code by using

Code: Select all

set "varname"

Btw. What is this code for?

Code: Select all

if not defined \p if "%=^%=" == "%=%=" (
    set ^"\p=%%<nul"
) else (
    set ^"\p=^%<nul"
)

[mataha]

Oof, that's right, thanks - it completely slipped my mind! I even tried to figure out how to call a script with "&fi^le.cmd" the other day...

Actually, instead of double percent expansion, could replacement syntax work? Similar to what you've done with for-parameters in your library:

Code: Select all

%$for:param=x%

Then the call site would look like so (with and without a parameter):

Code: Select all

%$strhex% result string
%$strhex:prefix=0X% result string

Btw. It's always a good idea, to show/debug your variable in your code by using

Code: Select all

set "varname"

I've just whipped up a macro to do exactly that (taking ideas from your repo yet again, lol):

Code: Select all

%$define% ^"$get=(%\n%
    if not defined \r (%\n%
        for /f "delims=" %$for:param=c% in ('"copy /y /z "%~f0" nul"') do (%\n%
            set "\r=%$for:param=~c%"%\n%
        )%\n%
    ) ^<nul 2^>^&0%\n%
    if not defined \t if exist "%ComSpec%" (%\n%
        setlocal EnableDelayedExpansion%\n%
        for /l %$for:param=_% in (1, 1, 70) do (pause)%\n%
        set /p "\t="%\n%
        for %$for:param=c% in ("%\p%!\t:~0,1%\p%!") do (%\n%
            endlocal ^& set "\t=%$for:param=~c%"%\n%
        )%\n%
    ) ^<"%ComSpec%" ^>nul 2^>^&1%\n%
) ^& for %$for:param=#% in (1 2) do if %$for:param=#% equ 2 (%\n%
    for /f "tokens=1-2,* delims= " %$for:param=0% in (": %\p%!args%\p%!") do (%\n%
        endlocal ^& if "%$for:param=~1%" equ "" (%\n%
            call :error EXIT_ERROR "%[b]%$get%[/b]%: undefined parameter received: variable"%\n%
        ) else for %$for:param=l% in ("%\p%!\n%\p%!") do ^
               for %$for:param=r% in ("%\p%!\r%\p%!") do ^
               for %$for:param=t% in ("%\p%!\t%\p%!") do (%\n%
            set "content=%\p%!%$for:param=~1%%\p%!"%\n%
            if not defined content (%\n%
                (echo(%$for:param=~1% ::= undefined)%\n%
            ) else (if ERRORLEVEL %\p%!content%\p%! call ) 2^>nul ^&^& (%\n%
                (echo(%$for:param=~1% ::= %\p%!content%\p%!)%\n%
            ) ^|^| (%\n%
                set "content=%\p%!content:\=\\%\p%!"%\n%
                set "content=%\p%!content:%$for:param=~t%=\t%\p%!"%\n%
                set "content=%\p%!content:%$for:param=~r%=\r%\p%!"%\n%
                set "content=%\p%!content:%$for:param=~l%=\n%\p%!"%\n%
                (echo(%$for:param=~1% ::= '%\p%!content%\p%!')%\n%
            )%\n%
            endlocal ^& call %\n%
        ) ^<nul ^>con%\n%
    )%\n%
) else setlocal EnableDelayedExpansion ^& setlocal ^& set args= "

Btw. What is this code for?

Code: Select all

if not defined \p if "%=^%=" == "%=%=" (
    set ^"\p=%%<nul"
) else (
    set ^"\p=^%<nul"
)

I am using parts of this code in other scripts to keep them self-contained. That fragment does nothing most of the time, but one of those scripts serves as a loader that I read via curl like so:

Code: Select all

curl -fsSL "https://my.site/script.cmd" | cmd /d >nul

That script then gets run in a command-line context, which requires me to replace double % characters in for-loops with a single one.

[jeb]

I've just whipped up a macro to do exactly that (taking ideas from your repo yet again, lol):

I am glad to hear that someone is looking at my library at all

Actually, instead of double percent expansion, could replacement syntax work? Similar to what you've done with for-parameters in your library:

Code: Select all

%$for:param=x%

Then the call site would look like so (with and without a parameter):

Code: Select all

%$strhex% result string
%$strhex:prefix=0X% result string

This would work, but has the limitation that you can't use percent signs nor colons as a prefix.
And IMHO I don't like the mixing of the replacement style with the parameter style for the user.
I used the replacement style for the FOR-params, because it's the only safe way in the definition phase of the macros.

But you don't even need the replacement style at all, you could add the prefix as an optional parameter, then it can contain any character,
and you can use variables for the prefix.

Code: Select all

%$strhex% result string <prefix>
%$strhex% result 1234
%$strhex% result 1234 0X
%$strhex% result 1234 "^"
%$strhex% result 1234 "!myUserDefinedPrefix!"

if not defined \p if "%=^%=" == "%=%=" (
set ^"\p=%%<nul"
) else (
set ^"\p=^%<nul"
)

I already suspected that it had something to do with the cmd line context.
I like your use of \p here for even independent of batch file vs cmd line context
But I would (re)define the \p always, independent if it's already defined, else it hard to find the cause for later failures.
Think of a predefined \p=&, that would be a mess
Same is true for \n, too

Btw. 1) The <nul seems superfluous, I can't see any case where it makes a difference.
Btw. 2) You don't need the complex IF comparision for the definition, you could use

Code: Select all

set "\p=%%"
set "\p=%\p:~-1%"

[mataha]

This would work, but has the limitation that you can't use percent signs nor colons as a prefix.
And IMHO I don't like the mixing of the replacement style with the parameter style for the user.
I used the replacement style for the FOR-params, because it's the only safe way in the definition phase of the macros.

But you don't even need the replacement style at all, you could add the prefix as an optional parameter, then it can contain any character,
and you can use variables for the prefix.

Sounds reasonable. It would look like this then:

Code: Select all

:define_strhex (name: string)
    %$define% ^"%~1=for %$for:param=#% in (1 2) do if %$for:param=#% equ 2 (%\n%
        for /f "tokens=1-4,* delims= " %$for:param=0% in (": %\p%!args%\p%!") do (%\n%
            endlocal ^& if "%$for:param=~1%" equ "" (%\n%
                call :error EXIT_ERROR "%[b]%%~1%[/b]%: undefined parameter received: result"%\n%
            ) else (%\n%
                set /a "number=%$for:param=~2% + 0" ^>nul 2^>^&1%\n%
                (%\n%
                    set "prefix=%~3"%\n%
                ) & if "" == "%$for:param=~3%" if "" == "%$for:param=3%" (%\n%
                    set "prefix=0x"%\n%
                )%\n%
                set "map=0123456789abcdef"%\n%
                set "digit="%\n%
                set "result="%\n%
                for /l %$for:param=_% in (1, 1, 8) do (%\n%
                    set /a "digit=number & 15, number>>=4"%\n%
                    for %$for:param=d% in (%\p%!digit%\p%!) do (%\n%
                        set "result=%\p%!map:~%$for:param=d%,1%\p%!%\p%!result%\p%!"%\n%
                    )%\n%
                )%\n%
                for %$for:param=v% in (%\p%!prefix%\p%!%\p%!result%\p%!) do (%\n%
                    endlocal ^& set "%$for:param=~1%=%$for:param=v%" ^& call %\n%
                )%\n%
            )%\n%
        )%\n%
    ) else setlocal EnableDelayedExpansion ^& setlocal ^& set args= "

    goto :EOF

Notably I find a need to distinguish between %~3 being empty or undefined in order to treat "" as a valid prefix:

Code: Select all

(%\n%
    set "prefix=%~3"%\n%
) & if "" == "%$for:param=~3%" if "" == "%$for:param=3%" (%\n%
    set "prefix=0x"%\n%
)%\n%

Makes sense, right?

if not defined \p if "%=^%=" == "%=%=" (
set ^"\p=%%<nul"
) else (
set ^"\p=^%<nul"
)

I already suspected that it had something to do with the cmd line context.
I like your use of \p here for even independent of batch file vs cmd line context
But I would (re)define the \p always, independent if it's already defined, else it hard to find the cause for later failures.
Think of a predefined \p=&, that would be a mess
Same is true for \n, too

Btw. 1) The <nul seems superfluous, I can't see any case where it makes a difference.
Btw. 2) You don't need the complex IF comparision for the definition, you could use

Code: Select all

set "\p=%%"
set "\p=%\p:~-1%"

That's a valid concern, albeit it's hard for me to imagine anything that would want to predefine \p - maybe another script that is mistakenly parsed in command-line context? I'll add this check regardless.

Regarding the definition - actually I've took it straight out of your library - I also can't think of anything that would incline user input. (It doesn't hurt to have it, though...?)

I wonder whether it's possible to define \n for command-line context as well - would open up some possibilities without having to maintain unreasonably long lines of code - as the following would not work:

Code: Select all

set ^"\n=^^^


set ^"$macro=if 1==1 (%\n%
^>^&2 echo a%\n%
) else ^>^&2 echo b"

set $macro >&2
%$macro%

Code: Select all

> type script.cmd | cmd /d >nul
$macro=if 1==1 (

[jeb]

Regarding the definition - actually I've took it straight out of your library - I also can't think of anything that would incline user input. (It doesn't hurt to have it, though...?)

I criticized my own code, that's hard
The <NUL doesn't harm, but it's still superfluous .

Notably I find a need to distinguish between %~3 being empty or undefined in order to treat "" as a valid prefix:

Code: Select all

(%\n%
    set "prefix=%~3"%\n%
) & if "" == "%$for:param=~3%" if "" == "%$for:param=3%" (%\n%
    set "prefix=0x"%\n%
)%\n%

Makes sense, right?

Yeah, looks good, but it seems that the first condition is superfluous here, because

Code: Select all

IF "" == "%$for:param=3%"

if this is true then %~3 can't contain anything

I wonder whether it's possible to define \n for command-line context as well - would open up some possibilities without having to maintain unreasonably long lines of code - as the following would not work:

Yes, \n can be defined as

Code: Select all

set "\n=^"

Full code

Code: Select all

set "\n=^"
set ^"$macro=if 1==1 (%\n%
 ^>^&2 echo a%\n%
) else ^>^&2 echo b"

REM *** Show Variables
set "$macro" >&2
set "\n"
REM *** Start Macro
%$macro%

But probably, the definition could be better with

Code: Select all

set "\n=<nul ^"

This avoid problems, when the first character of a line is a special char, like in your example "^>^&2 echo ..."

[mataha]

Yeah, looks good, but it seems that the first condition is superfluous here, because

Code: Select all

IF "" == "%$for:param=3%"

if this is true then %~3 can't contain anything

Not quite! Though it makes more sense with command-line arguments - if %~1 is blank (whitespace only), then the quotes (if present) will spill through:

Code: Select all

> type arg.cmd
@if "" == "%1" (echo(It's empty.)

> arg.cmd
It's empty.

> arg.cmd ""

> arg.cmd "a b"

> arg.cmd " "
'""' is not recognized as an internal or external command,
operable program or batch file.

Versus:

Code: Select all

> type arg.cmd
@if "" == "%~1" if "" == "%1" (echo(It's empty.)

> arg.cmd
It's empty.

> arg.cmd ""

> arg.cmd "a b"

> arg.cmd " "

Doesn't fully absolve Command Prompt's sins, but it's the best I came up with.
I find this useful for e.g. subroutines with variable number of parameters where an empty string can be a legal argument (and one needs to distinguish between that and an undefined parameter if there's a default value).

Yes, \n can be defined as

Code: Select all

set "\n=^"

Full code

Code: Select all

set "\n=^"
set ^"$macro=if 1==1 (%\n%
 ^>^&2 echo a%\n%
) else ^>^&2 echo b"

REM *** Show Variables
set "$macro" >&2
set "\n"
REM *** Start Macro
%$macro%

But probably, the definition could be better with

Code: Select all

set "\n=<nul ^"

This avoid problems, when the first character of a line is a special char, like in your example "^>^&2 echo ..."

This is... perfect. Wow.
Though I can't understand the purpose of both the opening quote and the escaped, closing quote; makes me wonder whether the quotes are necessary for \p definition as well... probably not for the command-line one, right?

[jeb]

This should work

Code: Select all

IF "" == "%$for:param=3%" echo arg3 is empty

You test against command line arguments, but you should test against macro FOR-Parameters instead, there the %%3 is a safe parameter.

But my set "\n=^" definition for the command line context is weak, it doesn't support multi line blocks
Better is

Code: Select all

(set \n=^^^

 ^^^

)


echo a%\n%b
set \
set $macro=if 1==1 (%\n%
  echo (Line1^^)%\n%
^>con  echo Line2%\n%
)
REM ******************* Show Variables *******************
set "$macro" >&2

REM ******************* Start Macro *******************
%$macro%

This is... perfect. Wow.
Though I can't understand the purpose of both the opening quote and the escaped, closing quote; makes me wonder whether the quotes are necessary for \p definition as well... probably not for the command-line one, right?

It's all about, when the special characters are used.

Code: Select all

set "\n=<nul ^"
set ^"\p=%%<nul"

In the first case the <nul is part of the \n variable and the purpose is to disable the caret behavior for the next line.
In the \p case the caret is used, because the <nul shall be active in the definition phase, therefore \p contains only a single percent sign.

[mataha]

This should work

Code: Select all

IF "" == "%$for:param=3%" echo arg3 is empty

You test against command line arguments, but you should test against macro FOR-Parameters instead, there the %%3 is a safe parameter.

I see. I stand corrected; I've thrown many test cases at this and couldn't find a single counterexample. That's awesome, thanks!

But my set "\n=^" definition for the command line context is weak, it doesn't support multi line blocks
Better is

Code: Select all

(set \n=^^^

 ^^^

)


echo a%\n%b
set \
set $macro=if 1==1 (%\n%
  echo (Line1^^)%\n%
^>con  echo Line2%\n%
)
REM ******************* Show Variables *******************
set "$macro" >&2

REM ******************* Start Macro *******************
%$macro%

This is... perfect. Wow.
Though I can't understand the purpose of both the opening quote and the escaped, closing quote; makes me wonder whether the quotes are necessary for \p definition as well... probably not for the command-line one, right?

It's all about, when the special characters are used.

Code: Select all

set "\n=<nul ^"
set ^"\p=%%<nul"

In the first case the <nul is part of the \n variable and the purpose is to disable the caret behavior for the next line.
In the \p case the caret is used, because the <nul shall be active in the definition phase, therefore \p contains only a single percent sign.

Shouldn't the following work just fine, then? If the only requirement is that the parser has to stop on the first character after the empty line, then & fits this requirement perfectly:

Code: Select all

if "%=^%=" == "%=%=" (
    set ^"\p=%%<nul"
    set ^"\n=^^^
%= These lines are supposed to be undented and empty; =%
%= removing them will result in hard-to-debug errors! =%) else (
    set ^"\p=^%<nul"
    set ^"\n=^^^

&@rem This line is supposed to be undented and empty - don't remove it!
)

I don't even think that & is necessary as long as there are no unintialized variables in the empty lines.

[jeb]

Shouldn't the following work just fine, then? If the only requirement is that the parser has to stop on the first character after the empty line, then & fits this requirement perfectly:

Code: Select all

if "%=^%=" == "%=%=" (
    set ^"\p=%%<nul"
    set ^"\n=^^^
%= These lines are supposed to be undented and empty; =%
%= removing them will result in hard-to-debug errors! =%) else (
    set ^"\p=^%<nul"
    set ^"\n=^^^

&@rem This line is supposed to be undented and empty - don't remove it!
)

I don't even think that & is necessary as long as there are no unintialized variables in the empty lines.

I'm not sure if we speak about the same

My sample of \n is for command line macros, there it seems to be harder to build a perfect \n, my current version adds one space for each \n
But your code seems to be for batch files, but there we already have a solution

[mataha]

I'm not sure if we speak about the same

My sample of \n is for command line macros, there it seems to be harder to build a perfect \n, my current version adds one space for each \n
But your code seems to be for batch files, but there we already have a solution

Code: Select all

@echo off
setlocal

if "%=^%=" == "%=%=" (
    echo Script
    set ^"\p=%%<nul"
    set ^"\n=^^^
%= These lines are supposed to be undented and empty; =%
%= removing them will result in hard-to-debug errors! =%) else (
    echo Command-line >&2
    set ^"\p=^%<nul"
    set ^"\n=^^^

)

set ^"$macro=if 1==1 (%\n%
^>^&2 echo %\p%%\n%
echo a ^>^&2%\n%
) else ^>^&2 echo Nope."

>&2 set "$macro"
>&2 set "\n"
REM *** Start Macro
%$macro%

Code: Select all

> context.cmd
Script
$macro=if 1==1 (
>&2 echo %
echo a >&2
) else >&2 echo Nope.
\n=^

%
a

Code: Select all

> type context.cmd | cmd /d >nul
Command-line
$macro=if 1==1 (
>&2 echo %
echo a >&2
) else >&2 echo Nope.
\n=^

%
a

[jeb]

Okay, I shouldn't write when I'm tired

I completely missed the ELSE part of your code.
Your definition looks simpler and better than mine, I don't know why I didn't found it myself