Safely expand any variable.

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Post Reply
Message
Author
dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Safely expand any variable.

#1 Post by dbenham » 14 Nov 2012 00:37

And now it is time to provide a solution for an esoteric problem that no one cares about :mrgreen:

Variable names can contain any character except = (Of course can't contain nul byte either, but nothing in batch supports nul, so I will ignore it).

Some characters can cause problems:

% in name can only be expanded using delayed expansion:

Code: Select all

@echo off
setlocal enableDelayedExpansion
set "var%%=hello"
set var
echo !var%%!

! in name can only be expanded using normal expansion:

Code: Select all

@echo off
setlocal
set "var!=hello"
set var
echo %var!%

But what to do if name contains both % and ! :?:

Code: Select all

@echo off
setlocal
set "var%%!=hello"
set var
There is no batch expansion that can retrieve this variable.


Variables names can contain : anywhere in the name, but unless the : is the last character, then the variable cannot be expanded unless extensions are disabled.

Code: Select all

@echo off
setlocal enableDelayedExpansion
set "var:name=hello"
set var

::These don't work
echo %var:name%
echo !var:name!

setlocal disableExtensions
::These now work
echo %var:name%
echo !var:name!


The SET command can be used to display the definition of a variable. Normally it wouldn't be hard to use FOR /F to iterate the values of all variables that begin with the desired name, and simply parse out the value of the first line. But both the name and the value can contain linefeed characters. With some work it should be possible to handle linefeed in the name, but I don't believe it is possible to identify when the value ends. The value could contain a linefeed followed by a string that looks like a valid variable definition.

I postulate that is is impossible for a pure native batch solution to reliably expand any arbitrary variable. The only native Windows solution I am able to come up with is a hybrid JScript/Batch solution. The following should work with all possible names and values, with the one caveat that it could run into length limitations because the value is expanded slightly for a handful of characters. The solution uses a variation of jeb's safe return technique.

To avoid escape issues, I pass the name of the variable by reference: The first argument is the name of a variable that contains the name of the variable to be expanded. The optional 2nd argument is the name of the variable where the result is to be stored. If the return variable name is not specified, then the value is sent to stdout. The routine can be called safely with delayed expansion enabled or disabled.

GetVar.bat

Code: Select all

@if (@X)==(@Y) @end /* Harmless hybrid line that begins a JScript comment

::************ Batch portion ***********
@echo off
if "%~2" equ "" (
  cscript //E:JScript //nologo "%~f0" %*
  exit /b
)
setlocal
set notDelayed=!
setlocal disableDelayedExpansion
for /f delims^=^ eol^= %%A in ('cscript //E:JScript //nologo "%~f0" %*') do set "rtn=%%A"
if not defined notDelayed set "rtn=%rtn:^=^^%"
if not defined notDelayed set "rtn=%rtn:!=^!%"
for %%1 in (^"^

^") do (
  for /f %%2 in ('copy /Z "%~dpf0" nul') do (
    for /f "tokens=1,2" %%3 in (^"%% """") do (
      endlocal
      endlocal
      set "%~2=%rtn%" !
    )
  )
)
exit /b

************* JScript portion **********/
var env=WScript.CreateObject("WScript.Shell").Environment("Process");
var rtn=env(env(WScript.Arguments.Item(0)));
if (WScript.Arguments.Count() > 1) {
  rtn=rtn.replace(/%/g,"%3")
         .replace(/\n/g,"%~1")
         .replace(/\r/g,"%2")
         .replace(/"/g,"%~4");
}
WScript.Stdout.WriteLine(rtn);

Here are some test cases for the routine:

Code: Select all

@echo off
setlocal enableDelayedExpansion
set LF=^


for /f %%C in ('copy /Z "%~dpf0" nul') do set "CR=%%C"

set "var=:Part1!LF!Part2%%!LF! art3^!!CR!P!LF!&<>|()^^"
set "name!var!=value!var!"
set "var=name!var!"

echo Use SET to show value
echo -----------------------------
set !var!
echo(

echo Use GetVar.bat to show value
echo -----------------------------
call GetVar var
echo(

echo Use GetVar.bat to store value: Delayed Expansion ON
echo ----------------------------------------------------
call GetVar var delayedExpansionOnResult
set delayedExpansionOnResult
echo(

echo Use GetVar.bat to store value: Delayed Expansion OFF
echo ----------------------------------------------------
setlocal disableDelayedExpansion
call GetVar var delayedExpansionOffResult
set delayedExpansionOffResult
echo(

And here are the results

Code: Select all

Use SET to show value
-----------------------------
name:Part1
Part2%
Part3!
&<>|()^=value:Part1
Part2%
Part3!
&<>|()^

Use GetVar.bat to show value
-----------------------------
value:Part1
Part2%
Part3!
&<>|()^

Use GetVar.bat to store value: Delayed Expansion ON
----------------------------------------------------
delayedExpansionOnResult=value:Part1
Part2%
Part3!
&<>|()^

Use GetVar.bat to store value: Delayed Expansion OFF
----------------------------------------------------
delayedExpansionOffResult=value:Part1
Part2%
Part3!
&<>|()^


Dave Benham

jeb
Expert
Posts: 1041
Joined: 30 Aug 2007 08:05
Location: Germany, Bochum

Re: Safely expand any variable.

#2 Post by jeb » 15 Nov 2012 14:11

Hi Dave,

dbenham wrote:And now it is time to provide a solution for an esoteric problem that no one cares about :mrgreen:

I love these types of problems :D

dbenham wrote:I postulate that is is impossible for a pure native batch solution to reliably expand any arbitrary variable. The only native Windows solution I am able to come up with is a hybrid JScript/Batch solution.

Aha, there is a small problem and then it only can be solved with vbscript :?:

In my opinion it should be possible with `set`.

Code: Select all

set var=name%%!
call :GetVar var value
exit /b

:GetVar
set "pVar=!%1!"

set "!%1!0=Stop1"
set !%1! > content1.tmp

set "!%1!0=Stop2"
set !%1! > content2.tmp

REM Now compare the files content1 and content2, then all before Stop1/Stop2 is the content of the variable.
exit /b

The idea is to create a variable that will be displayed directly after the main variable.
This variable should be named <original name><ASCII-0x01>
The ASCII-0x01 can be created by using your famous forfiles trick.
You need two different values for the "stop-variable" as the main-variable can contain excactly the content of one of the stop-variables.

hope it helps
jeb

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: Safely expand any variable.

#3 Post by dbenham » 15 Nov 2012 23:04

Great idea jeb :D

The stop variable allows computation of the value end - I will use FC /B

There is still a complication of preserving carriage returns. Line feeds are fairly easy, but trailing carriage returns are stripped by FOR /F.

SET /P will not work either.

I think the trick is to convert the value into hex before trying to read it. You showed me the FC /B trick to read binary data. I've already used it to write a batch hexDump routine.

But npocmaka discovered a better command to inter-convert between strings and hex notation - CERTUTIL :D 8) He published his findings on SS64: certutil - decode/encode BASE64/HEX strings.Print symbols by HEX code. I posted a response showing how to implement str2hex and hex2str.

I think CERTUTIL is an excellent candidate to help complete a pure native batch solution for expanding any variable. I'm in the process of writing the routine.

The only down side is CERTUTIL is not standard with XP. I'll let someone else use FC /B to provide an XP compatible version. :wink:


Dave Benham

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: Safely expand any variable.

#4 Post by dbenham » 17 Nov 2012 09:41

Here is a native batch solution that uses CERTUTIL.EXE. Usage is exactly the same as my original hybrid batch/JScript version. This native solution is ~2 times slower than my hybrid batch/JScript solution.

This native solution should work on Vista and beyond. A native solution for XP will require use of FC /B instead of CERTUTIL.EXE, and will require hardcoded 0x01 embedded in the source file, and will be even slower yet.

If I should ever need to use a GetVar routine, I'm sure I would use the hybrid batch/Jscript version. But I can't imagine when I would ever really need it :mrgreen:

Code: Select all

@echo off
:: save delayed expansion state before enabling delayed expansion
setlocal
set "NotDelayed=!"
setlocal enableDelayedExpansion

:: force delayed expansion encoding if printing result to screen
set "NotDelayed=1"

:: define base name for all temp files
set "tempBase=!temp!\getVar_%random%_"

:: get the name of the variable
set "var=!%~1!"

:: create 0x01 char and store in char01 variable
>"!tempBase!0x01.hex" (echo 01)
>nul 2>&1 certutil -f -decodehex "!tempBase!0x01.hex" "!tempBase!0x01.txt"
for /f "usebackq" %%A in ("!tempBase!0x01.txt") do set "char01=%%A"

:: create raw text value, version 1
set "!var!!char01!=1"
>"!tempBase!ver1.txt" set !var!

:: create raw text value, version 2
set "!var!!char01!=2"
>"!tempBase!ver2.txt" set !var!

:: find location of first difference between raw text values
for /f "skip=1 delims=:" %%A in ('fc /b "!tempBase!ver1.txt" "!tempBase!ver2.txt"') do (
  set /a diff=0x%%A
  goto :break
)
:break

:: create raw hex
>nul 2>&1 certutil -f -encodehex "!tempBase!ver1.txt" "!tempBase!raw.hex"

:: compute length of variable name
set len=0
set "str=.!var!"
for /L %%A in (12,-1,0) do (
  set /a "len|=1<<%%A"
  for %%B in (!len!) do if "!str:~%%B,1!"=="" set /a "len&=~1<<%%A"
)

:: compute lines to skip and start byte, lines to preserve and stop byte
set /a "start=len+1, skip=start/16, start=start%%16*3, end=diff-len-5, lines=end/16+1-skip, stop=end%%16*3+3"
if %skip% gtr 0 (set "skip=skip=%skip%") else set skip=

:: extract value hex and encode as necessary
set cnt=0
>"!tempBase!value.hex" (
  for /f "usebackq %skip% tokens=1*" %%A in ("!tempBase!raw.hex") do (
    set "hex=%%B"
    set /a cnt+=1
    if !cnt! equ !lines! (set /a "end=stop-start") else set /a "end=48-start"
    for /f "tokens=1,2" %%D in ("!start! !end!") do set "hex=!hex:~%%D,%%E!"
    set "hex=!hex:25=25 35!"     & rem % -> %5
    set "hex=!hex:22=25 7e 36!"  & rem " -> %~6
    set "hex=!hex:0A=25 7e 33!"  & rem <LF> -> %~3
    set "hex=!hex:0D=25 34!"     & rem <CR> -> %4
    if not defined NotDelayed (
      set "hex=!hex:5e=5e 5e!"   & rem ^ -> ^^
      set "hex=!hex:21=5e 21!"   & rem ! -> ^!
    )
    echo !hex!
    if !cnt! equ !lines! goto :break
    set start=0
  )
)
:break

:: decode encoded hex to encoded txt
>nul 2>&1 certutil -f -decodehex "!tempBase!value.hex" "!tempBase!value.txt"

:: load encoded text into variable
setlocal disableDelayedExpansion
for /f usebackq^ delims^=^ eol^= %%a in ("%tempBase%value.txt") do set "rtn=%%a"
setlocal enableDelayedExpansion

:: delete temp files
del "!tempBase!*"

:: define LF to contain a linefeed
set LF=^


:: above 2 blank lines are critical - DO NOT REMOVE

:: use FOR variables to decode text and return value or echo value, %%4 = <CR>
set "replace=%% """"
for %%3 in ("!LF!") do for /f %%4 in ('copy /Z "%~dpf0" nul') do (
  for /f "tokens=1,2" %%5 in ("!replace!") do (
    if "%~2" equ "" (
      set "rtn=%rtn%" !
      echo(!rtn!
    ) else (
      endlocal
      endlocal
      endlocal
      endlocal
      set "%~2=%rtn%" !
    )
  )
)
exit /b


Dave Benham (making the world safe for batch programmers everywhere... NOT :!: )

carlsomo
Posts: 91
Joined: 02 Oct 2012 17:21

Re: Safely expand any variable.

#5 Post by carlsomo » 19 Nov 2012 01:00

my apologies
Last edited by carlsomo on 19 Nov 2012 21:59, edited 1 time in total.

jeb
Expert
Posts: 1041
Joined: 30 Aug 2007 08:05
Location: Germany, Bochum

Re: Safely expand any variable.

#6 Post by jeb » 19 Nov 2012 02:47

Hi carlsomo,

what do you want to say with your code :?:
If you post an obvious solution for a problem then it could be enough to post the code only.

But here, your code seems not related to our problem, so without any text it's completly useless.

jeb

Post Reply