BatchLibrary or how to include batch files

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Post Reply
Message
Author
jeb
Expert
Posts: 1042
Joined: 30 Aug 2007 08:05
Location: Germany, Bochum

BatchLibrary or how to include batch files

#1 Post by jeb » 04 Feb 2011 16:46

Hi,

this site shows many functions and they all can build a useful library,
but it's always annoying that I have to copy & paste them into my code.

So I decide to write a batchLibrary that can be included from a batch and can also include other batch files.

To use it, there should be a simple way to activate the library without too much overhead.

Code: Select all

@echo off
REM 1. Prepare the BatchLibrary for the start command
call BatchLib.bat

REM 2. Start of the Batchlib, acquisition of the command line parameters, activates the code with the base-library
<:%BL.Start%

rem  Importing more libraries ...
call :bl.import "bl_DateTime.bat"
call :bl.import "bl_String.bat"

rem Use library functions
call :bl.String.Length result abcdefghij
echo len=%result%


I split the library start into two parts, because it seems to be the only way to retrieve the start parameters.

And here is the base library

BatchLib.bat

Code: Select all

REM ************** BEGIN OF BATCHLIB **************
REM * A "real" batch library system, that allowes to import other batch files into a batch application
REM *
REM * written 2010 by jeb
REM *******************
@echo off
setlocal EnableDelayedExpansion
set "param1=%~1"

REM *** Decide if the Library is started for initalization or only for the first prepare
if "!param1!"=="/INTERN_START" goto :bl_InternalStart

if /i "!param1!" EQU "/S" (
   REM ** Secure start, retrieve the parameter in a secure way, it accepts nearly every parameter, but it needs a temporary file
   REM ** TODO: Implement the REM redirect, Problems: access more than %1 ..%9 (shift, detect the end), double expansion of %%
   REM ** Buy it now for only 399Euro :-)
   echo NOT IMPLEMENTED
   EXIT
) ELSE (
   REM ** Basic start, retrieve the parameter in a simple way, it fails with complex parameters like "&"&"&
   REM ** TODO: Accept more than %9 parameters
   set BL.Start=#":< nul ( setlocal EnableDelayedExpansion & set "bl.cmdcmdline=^^!cmdcmdline^^!" & call set "bl.param[0]=%%~f0" & FOR /L %%n IN (1,1,9) DO ( call set "bl.param[%%n]=%%~1" & shift /1 ) )#"^
#"   & "%~f0" /INTERN_START "^^!bl.param[0]^^!"#"
)
(
  ENDLOCAL
  set "BL.Start=%BL.Start:#"=%"
  goto :eof
)

:bl_InternalStart
rem echo Library init from "%param[0]%"

rem Create Temporary Batchfile with library functions

rem Create first line with call to library_init and jump to the application, suppress a line ending for consistent line numbers
> "%~n2.tmp"  <nul set /p ".=call :BL.Init & goto :%%%%Bl.Start%%%% & rem "
rem Append the original batch file
>> "%~n2.tmp" type "%~f2"

REM Append a "good" file stop mark
(
echo(
echo ^)
echo goto :eof
) >> "%~n2.tmp"

rem Append this library
>> "%~n2.tmp" type "%~f0"

del "%~n2.BLibTmp.bat" 2> nul > nul
ren "%~n2.tmp" "%~n2.BLibTmp.bat" > nul

rem Asynchron start, creates a new cmd instance in the same window
rem Required, even if the application exit or crash the library can do the rest
rem (set dummy=) | ( call "%~n1.BLibTmp.bat" "%~2" )

REM Block this, so it is cached and not depends on a file
(
   call "%~n2.BLibTmp.bat"
   rem start "" /wait /b "%~n1.BLibTmp.bat" "%~2"
   REM echo End of application, remove temporary file^(s^)
   set "err=%errorlevel%"
   del "%~n2.BLibTmp.bat" > nul
   exit /b !err!
)
:: End of function

:BL.Init
rem At this moment there is nothing to do
call :BL.DragAndDrop.Parser
goto :eof
:: End of function

:: Imports/append the file to the current batch file
:BL.Import [filename.bat]
(
   setlocal EnableDelayedExpansion
   type "%~1" >> "%~dpf0"
   set "func=%~n1.Init"
   set "func=!func:_=.!"
)
(
   endlocal
   call :%func%
   goto :eof
)
:: End of function

:: Builds a drag & drop filelist, this list should be used, because files like "Drag&drop.bat" passed in the wrong way by windows
:: Therefore a Drag&Drop Batchfile should always end with an EXIT to suppress the execution of further commands after ampersands
rem Take the cmd-line, remove all until the first parameter
:BL.DragAndDrop.Parser
(
   set "$$$.params=!bl.cmdcmdline:~0,-1!"
   set "$$$.params=!$$$.params:*" =!"
   set bl.dragDrop.count=0

   rem Split the parameters on spaces but respect the quotes
   for %%G IN (!$$$.params!) do (
     set /a bl.dragDrop.count+=1
     set "bl.dragDrop[!bl.dragDrop.count!]=%%~G"
   )
   set "$$$.params="
   goto :eof
)
:: End of function


bl_String.bat

Code: Select all

REM *** begin of library BATCHLIB.String **************

REM * STOP here if runs accidental to this point
rem %~ CRITICAL STOP in BATCHLIB.String

:BL.String.Init
call :BL.String.CreateLF
call :BL.String.CreateCR
call :BL.String.CreateDEL_ESC
call :BL.String.CreateLatin1Chars
goto :eof
::End of function

::###############################
:BL.String.length <resultVar> <stringVar>
(   
   setlocal EnableDelayedExpansion
    set "s=%~2#"
    set "len=0"
    for %%P in (4096 2048 1024 512 256 128 64 32 16 8 4 2 1) do (
      if "!s:~%%P,1!" NEQ "" (
         set /a "len+=%%P"
         set "s=!s:~%%P!"
      )
   )
)
(
   endlocal
    set "%~1=%len%"
   exit /b
)
::End of function

:BL.String.CreateLF
:: Creates a variable with one character LF=Ascii-10
:: LF should  be used later only with DelayedExpansion
set LF=^


rem ** The two empty lines are neccessary, spaces are not allowed
rem ** Creates a percent variant "NLM=^LF", but normaly you should use the !LF! variant
set ^"NLF=^^^%lf%%lf%^%lf%%lf%^"
goto :eof

:BL.String.CreateDEL_ESC
:: Creates two variables with one character DEL=Ascii-08 and ESC=Ascii-27
:: DEL and ESC can be used  with and without DelayedExpansion
setlocal
for /F "tokens=1,2 delims=#" %%a in ('"prompt #$H#$E# & echo on & for %%b in (1) do rem"') do (
  ENDLOCAL
  set "DEL=%%a"
  set "ESC=%%b"
  goto :EOF
)
goto :eof

:BL.String.CreateCR
::: CR should  be used only with DelayedExpansion
for /F "usebackq" %%a in (`copy /Z "%~dpf0" nul`) DO (
   set "cr=%%a"
)
goto :eof


:BL.String.CreateCR_OldVariant
::: CR should  be used only with DelayedExpansion
setlocal EnableDelayedExpansion EnableExtensions
set "X=."
for /L %%c in (1,1,13) DO set X=!X:~0,4094!!X:~0,4094!

echo !X!  > %temp%\cr.tmp
echo\>> %temp%\cr.tmp
for /f "tokens=2 usebackq" %%a in ("%temp%\cr.tmp") do (
   endlocal
   set cr=%%a
   rem set x=
   goto :eof
)
goto :eof

:BL.String.CreateLatin1Chars
   chcp 1252 > NUL
   set "char[deg]=°"
   set "char[ae].lower=ä"
   set "char[oe].lower=ö"
   set "char[ue].lower=ü"
   set "char[sz]=ß"
   set "char[AE].upper=Ä"
   set "char[OE].upper=Ö"
   set "char[UE].upper=Ü"
   chcp 850 > NUL
goto :eof

:BL.String.Format <resultVar> <format> [parameters]
REM TODO: Implement an extended format function
   setlocal EnableDelayedExpansion
   set "result="
   set "format.count=1"
   set "formatParam=%~2"

   :__BL.String.Format.Splitter
   if not defined formatParam
   set formatParamLeft=!formatParam:{*=!
   set "format[%format.count%]=!formatParamLeft!"
   set /a format.count+=1
   goto :__BL.String.Format.Splitter

   set "result="
   (
      ENDLOCAL
      goto :eof
)
REM **** end of library BATCHLIB.String



bl_DateTime.bat

Code: Select all

REM *** begin of library BATCHLIB.DateTime **************

REM * STOP here if runs accidental to this point
rem %~ CRITICAL STOP in BATCHLIB.DateTime

::###############################
:bl.DateTime.Init
goto :eof
::End of function

::###############################
:bl.DateTime.Sleep
(
setlocal EnableDelayedExpansion
set /a sleepCnt=%~1+1
ping 127.0.0.1 -n !sleepCnt! 2> nul 1>nul
endlocal
goto :eof
)
::End of function

::###############################
::### WARNING, enclose the time in quotes ", because it can contain comma seperators
:bl.DateTime.timeDiff <resultVar> <start_time> <end_time>
(
  SETLOCAL
  if "%~4" NEQ "" (
    echo ERROR in :bl.DateTime.timeDiff too much parameter
   exit /b 4
  )
  call :bl.DateTime.timeToMs ms_start "%~2"
  call :bl.DateTime.timeToMs ms_end "%~3"
)
set /a diff_ms=ms_end - ms_start
(
  ENDLOCAL
  set %~1=%diff_ms%
  goto :eof
)
::End of function

::###############################
:bl.DateTime.timeToMs <resultVar> <time>
::### WARNING, enclose the time in quotes ", because it can contain comma seperators
::### WARNING it does not convert time in am/pm format, because it's regressive
SETLOCAL
FOR /F "tokens=1,2,3,4 delims=:,.^ " %%a IN ("%~2") DO (
  set /a ms=^(^(^(30%%a%%100^)*60+7%%b^)*60+3%%c-42300^)*1000+^(1%%d0 %% 1000^)
)
(
  ENDLOCAL
  set %~1=%ms%
  goto :eof
)
::End of function

REM **** end of library BATCHLIB.DateTime


LibTest1.bat

Code: Select all

@echo off

set "t1=%time%"

REM 1. Prepare the BatchLibrary for the start command
call BatchLib.bat

REM 2. Start of the Batchlib, acquisition of the command line parameters, swtich to the temporary LibTest1.BLibTmp.bat
<:%BL.Start%

rem  Importing libraries ...
call :bl.import "bl_DateTime.bat"
call :bl.import "bl_String.bat"

set "t2=%time%"
call :bl.DateTime.timeDiff   duration "%t1%" "%t2%"
echo libraries loaded, duration %duration%ms

echo !lf!Counter demo
for /L %%n in (1,1,10) DO (
   call :bl.dateTime.sleep 1
   <nul set /p ".=!del!!cr!%%n"
)
echo(

set str=abcdefghijkl
call :bl.string.length result %str%
echo The length of str="%str%" is %result%


hope it helps
jeb

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: BatchLibrary or how to include batch files

#2 Post by aGerman » 04 Feb 2011 17:17

That's pretty cool, jeb.

Some years ago I thought about "including" and 2 days ago you reminded me on it. If I find my old test files I could post them as well.

Regards
aGerman

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: BatchLibrary or how to include batch files

#3 Post by dbenham » 15 Feb 2011 22:53

Cool (and scary) looking stuff, though I'm wondering if I have a simpler solution.

this site shows many functions and they all can build a useful library,
but it's always annoying that I have to copy & paste them into my code.


I was surprised by the statement above in bold. When I first discovered this amazing site last year, I assumed people had already developed a mechanism to call the routines in the library from other bat files, without the need for copying. I didn't see anything posted on how to do this, but the solution seemed obvious: the library would itself be a callable bat file, and the 1st parameter passed would be expected to be the name of the function to call. The library would call the desired function, passing along any additional arguments, and then exit.

Trivial example:

Code: Select all

@echo off
:: filename = DOSLIB.BAT
call :%*
exit /b

:f1
echo:
echo called f1
echo arg1=%1
echo arg2=%2
exit /b

:f2
echo:
echo called f2
echo arg1=%1
echo arg2=%2
exit /b

:f3
echo:
echo called f3
echo arg1=%1
echo arg2=%2
exit /b


Trivial example of using the library:

Code: Select all

@echo off
:: filename = TEST.BAT
call doslib f1 abc xyz
call doslib f3 1

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: BatchLibrary or how to include batch files

#4 Post by dbenham » 15 Feb 2011 23:13

Arrg! :x :oops: I hit submit when I meant to hit preview.

To continue - I'm sure I've seen the technique in my previous post described elsewhere, but when I look throughout this site now I realize I don't see any mention of it here.

It seems this is much simpler and cleaner than trying to implement an include solution (clever though it may be)

Is there some inherent disadvantage with the technique I describe?

The code I've given is just a start with much room for improvement. I've got a functioning callable library of 30+ functions that is an extension of this basic idea. I'll try to post it tomorrow.

jeb
Expert
Posts: 1042
Joined: 30 Aug 2007 08:05
Location: Germany, Bochum

Re: BatchLibrary or how to include batch files

#5 Post by jeb » 16 Feb 2011 14:00

Hi dbenham,

you are right.

call :%* is one solution, and in many cases it is a good choice.

But in my post, I forgot to explain why I decide to write a complex batch include mechanism, instead of using one of the simpler approaches.

First, the copy & past solution is not "smart" enough, you have in all batch files a copy of your library,
and after a time in nearly all batch files are other versions of the "library".

The calling-batch solution like yours have some disadvantages.
- Each single library call is slow
- You can not integrate secure parameter and drag&drop handling in an external batch library
- You can not build a GetCurrentLine or a StackDump function with an external batch
- Each parameter is expanded two times by each call doslib :function, this could be tricky in escape/special character situations

The BatchLib-Include solution, I posted have also disadvantages
- The start is slower (~50-200ms)
- It creates always a temporary file, this could be a problem in a read only situation (can be solved with %temp%)
- For a beginner, it is not possible to understand how it works.
Or who can explain, why this is a legal statement and what it is good for?
<:%BL.Start%


I want to explain, what I mean with
secure parameter: Retrieve %1 even if it contains "&"&"&"
secure drag&drop: Should work even if you got filenames like Adam&Eve.txt
GetCurrentLine: It should be possible to write call :GetCurrentLine LineNr and retrieve in lineNr the current line number.
StackDump: It should be possible to call :StackDump and get the calling tree like
:mainFunc, line 7
+ :chessGame, line 50
++ :drawBoard, near line 450 (block command)
+++ :drawField, line 1034
++++ :arrayError, line 1039
+++++ :StackDump is started here

jeb

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: BatchLibrary or how to include batch files

#6 Post by aGerman » 16 Feb 2011 14:40

jeb wrote: Or who can explain, why this is a legal statement and what it is good for?
<:%BL.Start%


Hi jeb.

When I saw this line I thought to myself "another cool trick of jeb". But, to be honest, I have no idea how it works and I had no time to figure out yet.

Regards
aGerman

jeb
Expert
Posts: 1042
Joined: 30 Aug 2007 08:05
Location: Germany, Bochum

Re: BatchLibrary or how to include batch files

#7 Post by jeb » 17 Feb 2011 02:55

I will try to explain it.

(btw: I wrote a long explanation at administrator.de,
but I need too much time,
so I'm not logged in any more and the text is lost .. lost ... LOST, what a @*!#)

:label is a label
<nul echo x is a cmd, nul is redirected to stdin

<: <nul echo x is the same cmd, nul is redirected to stdin
Why? Because, there can always only one redirection to one stream.

The rule is, scan the line and take only the last redirection for a stream.
<file1 <file2 echo<file3 x <nul is the same as <nul echo x

Assume the content of BL.Start is "<nul echo hello"
<:%BL.Start%
expands to
<:<nul echo hello

A legal command, but why so complicated?
Because:
:label is a label, named label
:%label% is a label, named %label%, while searching for a label, it will not expanded
*:label is a label, named label
<==,==;=; = :label is a label, named label

The rule for a label is:
Null or one character(any, but colon), followed by any delims of <space>,;=
and after the colon: the name of the label is required.
The name itself ends at the line end or at a label delim like +<>&|space

As a result the <:%BL.Start% is a command and also a legal label.

So this line can call itself
<:label <nul echo 1 & call :label

But, it isn't an infinite loop,
as a call or goto label search the label and the rest of the line will be ignored,
the next line will be executed.

Now it should be obvious what <:%BL.Start% is good for.
First the %Bl.Start% expands to something like
<nul GetAllParameter & start the batchlibrary with /INTERN and my own filename (%0)

The batchlibrary creates a new temporary batch file
The first line contains
@echo off & goto :%%BL.Start%% & rem <the rest of the original file>
Then the original file and the batchLib are appended.
And at last the temporary file is started.
And now the goto :%%BL.Start%% will continue the program execution direct after the mysterious <:%BL.Start%.

That's all


jeb

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: BatchLibrary or how to include batch files

#8 Post by aGerman » 17 Feb 2011 11:50

Wow! Thanks for the good explanation, jeb. Now I understood how it works. It's always very interesting for me to read your posts :D New methods again and again. Thumbs up!

jeb wrote:(btw: I wrote a long explanation at administrator.de,
but I need too much time,
so I'm not logged in any more and the text is lost .. lost ... LOST, what a @*!#)

Bother :(
First I write long posts in my text editor, since it happened to me as well.

Regards
aGerman

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: BatchLibrary or how to include batch files

#9 Post by dbenham » 18 Feb 2011 00:57

Hi Jeb

Thanks for your response

Wow! so many things to talk about. I wanted to respond earlier, but for some reason my account was temporarily deactivated yesterday.

-------
<:%BL.Start% - Very interesting, with good explanation. I'll file that trick away in my head - maybe it will come in useful someday.

To make sure I understand, the arcane behaviour is not necessary for the overall include process, correct? You could simply have

Code: Select all

call BatchLib.bat
%BL.Start%
:ResumeHere

And top line of temp bat file would goto :ResumeHere instead of :%BL.Start%.
This would be less obfuscated, at the expense of requiring an extra line of code at the top of each file that needs the library.

Did you consider appending only the portion of the original bat that appears afer %BL.Start%? Then you would not need any label at all. Better yet you could disable the initial lines through %BL.Start% by prefixing with ": " (no quotes), or REM, thus preserving the line numbers if you wanted.

Code: Select all

@echo off
setlocal DisableDelayedExpansion
set resume=
for /f "delims=" %%a in (%~f2) do (
  if defined resume (>>%~n2.tmp echo %%a ) else (
    >>%~n2.tmp echo:: %%a
    if /i .%%a==.%%BL.Start%% set resume=true
  )
)

I suppose the for loop is slower than type, so it might delay startup a bit more. Your drag&drop scenario rears its head here as well (can't enclose the filename in quotes). You could get around this by preprocessing the filename to escape the &, yes?

------
I see that you parse the arguments originally passed to the initial bat file into an array. But I don't see where you pass those args to the temp bat. Are you relying on the original bat file to reference the parameters in your environment array? I guess this works because the setlocal is in effect throughout the temp bat call, but this seems awkward.

This leads to two additional concerns I have with the code as written.

1) The initialization of the library is not a black box - the initiating bat file will see your manipulation of the environment. This seems to violate one of principals of a well constructed function library.

2) The initiating bat file loses the ability to preserve its environment changes upon exit. This limitation is the biggest concern for me.

I see 3 possible remedies, but none of them are perfect:

option 1) endlocal before the call to the bat file, don't delete the temp file: environment and errorlevel are preserved, but the temp file remains (yuck)

option 2) endlocal before call and delete file within your "cache": environment preserved but returning errorlevel is lost.

option 3) endlocal before call, don't "cache" the delete sequence that pereserves the errorlevel: I'm not sure exactly what the repurcusion is here. I gather from your explanation that a poorly behaved bat file might not exit the library cleanly?

------------
With regard to my original suggestion, and your original response:

- Each single library call is slow

Yea, I did some timings comparing external calls to the library vs making the same calls within the library. (I used a for loop for testing) In all tests the loop with the external calls took about 2.8 times longer, regardless of the simplicity or complexity of the routine being called. This surprised me a bit. I thought there would be a constant overhead for each external call, but the routine within the library would behave the same once it had been called. Maybe my testing was flawed. Either way, the performance penalty is acceptable to me in most circumstances. If performance really became an issue I would then go ahead and copy the routine into my batch.

- Each parameter is expanded two times by each call doslib :function, this could be tricky in escape/special character situations

Yes I'm aware of this, hence my statement that there is much room for improvement. My "finished"library doesn't simply use call :*, it actually parses the arguments to address this issue.

- You can not build a GetCurrentLine or a StackDump function with an external batch

Can you provide some code (or links) showing how you do this (in general, not with the callable library)? I'm very interested. I can envision how you might build a stack in an environment variable and parse it as needed for easy display, but the solution in my head requires explicit stack building code at each call entry. I haven't a clue how you get line numbers.

I'm not sure what good line numbers do for your include solution, because they do not match the original sources. I suppose you could disable the temp file deletion during debugging sessions.

- You can not integrate secure parameter and drag&drop handling in an external batch library

I have some understanding of the issues you are referring to, but not the depth that I think you have. Can you provide some code (or links) that demonstrate some secure parameter and drag&drop pitfalls and some general strategies on how people deal with them.

-------------
One major advantage of the callable library is it can be used directly within an interactive session. The calls do not need to be embedded in a bat file.

In theory it seems possible to have one library that is callable as I suggest, but also has routines like yours to implement the include feature.


Dbenham

jeb
Expert
Posts: 1042
Joined: 30 Aug 2007 08:05
Location: Germany, Bochum

Re: BatchLibrary or how to include batch files

#10 Post by jeb » 18 Feb 2011 04:34

Hi dbenham,

To make sure I understand, the arcane behaviour is not necessary for the overall include process, correct? call BatchLib.bat
%BL.Start%
:ResumeHere

And top line of temp bat file would goto :ResumeHere instead of :%BL.Start%.
This would be less obfuscated, at the expense of requiring an extra line of code at the top of each file that needs the library.

Yes. But I want to beware the user of adding manual the correct goto's and label's.

Did you consider appending only the portion of the original bat that appears afer %BL.Start%? Then you would not need any label at all. Better yet you could disable the initial lines through %BL.Start% by prefixing with ": " (no quotes), or REM, thus preserving the line numbers if you wanted.

But the library start could be at the end of a batch file, triggered with an goto :startLib, so I suppose it isn't a good idea.

I see that you parse the arguments originally passed to the initial bat file into an array. But I don't see where you pass those args to the temp bat. Are you relying on the original bat file to reference the parameters in your environment array? I guess this works because the setlocal is in effect throughout the temp bat call, but this seems awkward.
This leads to two additional concerns I have with the code as written.

1) The initialization of the library is not a black box - the initiating bat file will see your manipulation of the environment. This seems to violate one of principals of a well constructed function library.

2) The initiating bat file loses the ability to preserve its environment changes upon exit. This limitation is the biggest concern for me.

Also: Yes. It isn't a good way at this point, to pass the arguments by "global" variables.

As a solution I will build something like

Code: Select all

set param_all=%* ' A bit smarter, only for the explanation
( call :batchLib /internalstart  ' Setlocal is active here, for the pass of param_all
 delete TempBat.bat
)

-- batchlib.bat
... prepare all
convert param_all to a secure variant, to use it for the passing to myBatch.tmp.bat
set "param_all=!param_all:%%=%%%%!"
set "param_all=!param_all:^=^^!"
set "param_all=!param_all:&=^&!"
set "param_all=!param_all:|=^|!"
set "param_all=!param_all:<=^<!"
set "param_all=!param_all:>=^>!"
set "param_all=!param_all:""=^"!"
(
endlocal
%tempBat.bat% %Param_all%
)


option 3) endlocal before call, don't "cache" the delete sequence that pereserves the errorlevel: I'm not sure exactly what the repurcusion is here. I gather from your explanation that a poorly behaved bat file might not exit the library cleanly?


If your batch file halt with a syntax error (try this as command >() ) you can not handle anything, or you have to start the temp-batch always with start /b /wait cmd /c ...

Quote:
- You can not integrate secure parameter and drag&drop handling in an external batch library

I have some understanding of the issues you are referring to, but not the depth that I think you have. Can you provide some code (or links) that demonstrate some secure parameter and drag&drop pitfalls and some general strategies on how people deal with them.


For drag&drop you got problems, if a file is named "Test&Echo.bat", windows is not smart enough to enclose it with quotes,
so you get a cmd line like

Code: Select all

cmd /c ""C:\temp\test.bat" C:\temp\test&echo.bat"

But this is interpreted as to seperated command:
"C:\temp\test.bat" C:\temp\text
And
echo.bat

Therefore in %* is only "C:\temp\text", the only way to access the real parameters is the !cmdcmdline! variable
and to hard exit the batch, so the echo.bat can't be executed.

Secure parameters: Try to access the parameter of the line
myBatch.bat "&"^&
This is impossible with normal ways, like

Code: Select all

set var=%1
set "var=%1"
set var=%~1
set "var=%~1"

None will work.

I found a trick with redirection the **echo on** output.

Code: Select all

@echo off
set prompt=#
for %%a in (1) do (
  echo on
  for %%b in (2) do rem ## %*
) > temp
@echo off
prompt
echo ---
type temp


Can you provide some code (or links) showing how you do this (in general, not with the callable library)?
I'm very interested. I can envision how you might build a stack in an environment variable and parse it as needed for easy display,
but the solution in my head requires explicit stack building code at each call entry.
I haven't a clue how you get line numbers.

I'm not sure what good line numbers do for your include solution, because they do not match the original sources.


I'm care of the correct line numbers in the temporary file, as I only add a goto :%%Bl.start%% & rem <original code> at the first line

But how to access stackdump or line numbers? It's nearly the same solution.
The first key is to access the current function name by %0.
To get control over the file you have to use the self modification feature of batch-files.
So you are able to recover the line numbers of each call :fn

And to get the line number you only have to build your "modified" batch in a way to get the correct one
simplified

Code: Select all

set L=1 & goto :getIt
set L=2 & goto :getIt
set L=3 & goto :getIt
set L=4 & goto :getIt
set L=5 & goto :getIt

:getIt
echo The function %0 has call something at line number %L%
goto :eof


Ok, it's not so easy, but this is the concept.

hope it helps
jeb

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: BatchLibrary or how to include batch files

#11 Post by dbenham » 20 Feb 2011 03:20

Thanks for your excellent response Jeb. Both your code and explanations are very instructive.

But the library start could be at the end of a batch file, triggered with an goto :startLib

Ouch! I never considered anyone would put the include anywhere other than the top. Can you think of a reason why this would ever be necessary? This leaves open the possibility of someone calling the :startLib multiple times, with the potential for an endless loop! I suppose you could protect against it, but it doesn't seem like a bad "rule" to say the master include belongs at the top prior to any labels.

Code: Select all

convert param_all to a secure variant, to use it for the passing to myBatch.tmp.bat
set "param_all=!param_all:%%=%%%%!"
set "param_all=!param_all:^=^^!"
etc.

Referring to your code above, Since this is not production code, I'm assuming the above will be enhanced to deal with the potential for quotes within the string - no escape within quotes, yes escape outside of quotes?


I'm still a bit confused about your term "drag&drop". Does this refer to a windows operation that results in a call like your example below?
[code]cmd /c ""C:\temp\test.bat" C:\temp\test&echo.bat"[code]
I understand why the above fails. I'm still trying to understand what would lead to a call like above.


Secure Parameter Solution - Way Cool! I haven't a clue how it works, but I think this is the missing piece that will finally allow my library to handle any string of printable characters. I've done some prototyping and it's looking good. Thanks!


I'm (taking) care of the correct line numbers in the temporary file, as I only add a goto :%%Bl.start%% & rem <original code> at the first line

Wow, I saw the code comment on my 1st read through and didn't understand it. 2nd pass I missed the comment and saw you were typing the entire bat file, so I assumed you couldn't possibly be preserving line numbers. I now understand how you supress the new line on your initial temp file creation. Another clever trick.

My original comment though was more to do with the line numbers for the library routines. I don't supppose there is much you can do about those?


Stack and line number diagnostics - No it doesn't look to be easy. It seems like you would have to parse and identify blocks of code between ( ). That seems like that could be very tricky. I imagine there are other difficulties. Do you have any functioning code you can share? If not, I'll probably stick with my existing embedded debug code.


A few other comments: Given your attention to detail, I'm sure you've already thought of these points, but they probably are good to have with this thread:

1) I assume you will enhance your code to eliminate the potential for temp file collision in parallel situations (multiple sessions on the same machine, or multiple users sharing a network drive)

2) Do you plan on preventing multiple loads of the same library? This is especially possible since a library could load another library as part of its initialization routine.

3) Libraries could be developed and shared by many people. I see you are very careful with your label names, but I'm not sure everyone would be so careful. I think people are familiar with the necessity for unique labels within a file, but I'm not sure they would automatically think to ensure uniqueness accross libraries. I like your naming convention with the library name at the front.

Finally - Your excellent work has given me a related idea. Perhaps you've already considered this:

Instead of dynamically building the "linked" bat file at run time as a temporary file, why not replace the original prior to running? The master bat file could simply have commented lines with include directives. It would not initiate a call to load the libraries. Instead, an external BATLINK.BAT file could be called to parse the file, create a temp file with all of the necessary libraries appended, and then replace the original.

The final "linked" file would preserve the include directives. You would need a marker line at the bottom of the master saying "DO NOT MODIFY BELOW THIS LINE" etc. I suppose you would want a similar area at the top to insert calls to library initiation routines.

Any time you wanted to refresh the file with updated libraries, you could simply relink it. The "linker" just needs to be smart enough to strip out the previously appended libraries from the bottom and the initialization calls from the top before proceeding. The "linker" could be logically recursive, with each library possibly including other libraries, and library markers could be used to prevent multiple loads of the same library.

This would mimimize the initial startup time.

This obviously would lose the ability to automatically take advantage of updates to libraries, but that could be a blessing. A library could be enhanced in a non-backward compatible fashion without necessarily breaking an already existing linked bat file. And it would be very easy to relink whenever you wanted.

You've given me a lot to chew on. Thanks for all your work Jeb

DBenham

jeb
Expert
Posts: 1042
Joined: 30 Aug 2007 08:05
Location: Germany, Bochum

Re: BatchLibrary or how to include batch files

#12 Post by jeb » 31 Mar 2011 04:26

Sometimes I'm fast, sometimes not ... :)

Code:
convert param_all to a secure variant, to use it for the passing to myBatch.tmp.bat
set "param_all=!param_all:%%=%%%%!"
set "param_all=!param_all:^=^^!"
etc.

Referring to your code above, Since this is not production code, I'm assuming the above will be enhanced to deal with the potential for quotes within the string - no escape within quotes, yes escape outside of quotes?
...
Secure Parameter Solution - Way Cool! I haven't a clue how it works, but I think this is the missing piece that will finally allow my library to handle any string of printable characters. I've done some prototyping and it's looking good. Thanks!

As it's possible but very tricky to escape only outside of the quotes, I decide to quote all even the quotes, so there is no "inside" quotes anymore.
To catch the parameter I use my REM/Prompt technic

Sample:

Code: Select all

myTest.bat "THIS&"is^&"A&"^^quote"MONSTER<>|"^<^>^|

The parameter %1 itself contains then
%1="THIS&"is&"A&"^quote"MONSTER<>|"<>|

the library catch the parameter with the REM technic and escape it to
^"THIS^&^"is^&^"A^&^"^^quote^"MONSTER^<^>^|^"^<^>^|


The secure escape mechanism is here

Code: Select all

rem ** Escaping a string to a "secure" variant
set "var=%var:"=""%"
set "var=%var:^=^^%"
set "var=%var:&=^&%"
set "var=%var:|=^|%"
set "var=%var:<=^<%"
set "var=%var:>=^>%"

test2.bat %var:""=^"%


The REM/Prompt technic, to access even the strongest parameter like "&"^&

Code: Select all

@echo off
SETLOCAL DisableDelayedExpansion

SETLOCAL
for %%a in (1) do (
    set "prompt="
    echo on
    for %%b in (1) do rem * #%1#
    @echo off
) > param.txt
ENDLOCAL

for /F "delims=" %%L in (param.txt) do (
  set "param1=%%L"
)
SETLOCAL EnableDelayedExpansion
set "param1=!param1:*#=!"
set "param1=!param1:~0,-2!"
echo %%1 is '!param1!'


I'm still a bit confused about your term "drag&drop". Does this refer to a windows operation that results in a call like your example below?
[code]cmd /c ""C:\temp\test.bat" C:\temp\test&echo.bat"[code]
I understand why the above fails. I'm still trying to understand what would lead to a call like above.


It's drag & drop from the explorer to a batch file.
You can select one or more files in the explorer and drop them over a batch file.


Stack and line number diagnostics - No it doesn't look to be easy. It seems like you would have to parse and identify blocks of code between ( ). That seems like that could be very tricky. I imagine there are other difficulties. Do you have any functioning code you can share? If not, I'll probably stick with my existing embedded debug code.

You are right, it seems to be impossible to get the line number of block-calls, therefore I simply ignore this!
I only try to catch line numbers outside of blocks.

A few other comments: Given your attention to detail, I'm sure you've already thought of these points, but they probably are good to have with this thread:

1) I assume you will enhance your code to eliminate the potential for temp file collision in parallel situations (multiple sessions on the same machine, or multiple users sharing a network drive)

2) Do you plan on preventing multiple loads of the same library? This is especially possible since a library could load another library as part of its initialization routine.

3) Libraries could be developed and shared by many people. I see you are very careful with your label names, but I'm not sure everyone would be so careful. I think people are familiar with the necessity for unique labels within a file, but I'm not sure they would automatically think to ensure uniqueness accross libraries. I like your naming convention with the library name at the front.

1) The problem of multiple temp-file collisions, I want to solve with adding a (random) PID to the temp-filename.
2) Multiple library loads I want to prevent by a library load flag for each library
3) The naming convention with "namespaces" is useful to prevent collisions, but I think about a mechanism to use a "namespace" by adding wrapper labels, so you can use library functions without the prefix.
But then it's on your own responsibility to ensure that a function/label is unique.


Instead of dynamically building the "linked" bat file at run time as a temporary file, why not replace the original prior to running? The master bat file could simply have commented lines with include directives. It would not initiate a call to load the libraries. Instead, an external BATLINK.BAT file could be called to parse the file, create a temp file with all of the necessary libraries appended, and then replace the original.

The final "linked" file would preserve the include directives. You would need a marker line at the bottom of the master saying "DO NOT MODIFY BELOW THIS LINE" etc. I suppose you would want a similar area at the top to insert calls to library initiation routines.

Any time you wanted to refresh the file with updated libraries, you could simply relink it. The "linker" just needs to be smart enough to strip out the previously appended libraries from the bottom and the initialization calls from the top before proceeding. The "linker" could be logically recursive, with each library possibly including other libraries, and library markers could be used to prevent multiple loads of the same library.

This would mimimize the initial startup time.

This obviously would lose the ability to automatically take advantage of updates to libraries, but that could be a blessing. A library could be enhanced in a non-backward compatible fashion without necessarily breaking an already existing linked bat file. And it would be very easy to relink whenever you wanted.


I think about this, but decide not to use it, as it changes the source files and I'm get trouble with repository systems, as they want to check them in.

hope it helps
jeb

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: BatchLibrary or how to include batch files

#13 Post by dbenham » 31 Mar 2011 17:04

Thanks Jeb

Just 1 comment for a change

You are right, it seems to be impossible to get the line number of block-calls, therefore I simply ignore this!
I only try to catch line numbers outside of blocks.

That's the parsing that boggles my mind!

  • You have to recognize when it's possible to begin a block (the command context)
  • Even if a block context is possible, you have to ignore escaped and quoted parentheses.

That seems plenty difficult just if you restrict yourself to what is in the code. But then add the possibility of blocks beginning or ending due to results of variable expansion... My head is now seriously pounding!

Kudos to you if you've figured all this out. (Well kudos no matter what, but you know what I mean)

Dave

jeb
Expert
Posts: 1042
Joined: 30 Aug 2007 08:05
Location: Germany, Bochum

Re: BatchLibrary or how to include batch files

#14 Post by jeb » 04 Apr 2011 14:41

dbenham wrote:That's the parsing that boggles my mind!

* You have to recognize when it's possible to begin a block (the command context)
* Even if a block context is possible, you have to ignore escaped and quoted parentheses.


That seems plenty difficult just if you restrict yourself to what is in the code. But then add the possibility of blocks beginning or ending due to results of variable expansion... My head is now seriously pounding!

I suppose it is not possible to detect the correct line number even without to regards the variable expansion.
A simple block with to calls seems to be a mission impossible 8) .

Code: Select all

(
call :func 1
call :func 1
)


dbenham wrote:Kudos to you if you've figured all this out. (Well kudos no matter what, but you know what I mean)

Many thanks :)
The most of this stuff I figured out myself (I realize, I waste to much time..), but it was simply annoying that I can't find any exact answers to my questions.

jeb

sdrawkcab
Posts: 1
Joined: 24 Jan 2018 13:53

Re: BatchLibrary or how to include batch files

#15 Post by sdrawkcab » 24 Jan 2018 14:00

Hi Jeb,

Do you happen to have a copy of your batch script library that you're sharing publicly, such as on GitHub? I'd love to follow it and take advantage of the library you've written if you have it posted anywhere.

Post Reply