Faster batch macros

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Message
Author
jeb
Expert
Posts: 1041
Joined: 30 Aug 2007 08:05
Location: Germany, Bochum

Faster batch macros

#1 Post by jeb » 05 Jan 2024 14:29

Hi,

while I try to build a small batch list selection component for a customer, I stumbled about the horrible, poor performance of batch functions on network drives.
My code used the standard :strlen function 180 times and it takes some seconds.

I decided to use a strlen macro, it's faster, but still not satisfying, so I checked the performance killers.
Mac1

Code: Select all

(set ^"$\n=^^^
%= This creates an escaped Line Feed - DO NOT ALTER =%
)

set $strLen=for %%# in (1 2) do if %%#==2 ( %$\n%
  for /f "tokens=1,2 delims=, " %%1 in ("!argv!") do ( %$\n%
%= 		*** remove the local variable "argv" =% %$\n%
    endlocal %$\n%
%= 		*** copy content to temporary variable, the carets are for extended length, up to 8191chars =% %$\n%
    (set^^ s=!%%~2!) %$\n%
    if defined s ( %$\n%
        set "len=1" %$\n%
        for %%P in (4096 2048 1024 512 256 128 64 32 16 8 4 2 1) do ( %$\n%
          if "!s:~%%P,1!" neq "" ( %$\n%
            set /a "len+=%%P" %$\n%
            set "s=!s:~%%P!" %$\n%
          ) %$\n%
        ) %$\n%
    ) ELSE set "len=0" %$\n%
    for %%V in (!len!) do endlocal ^& set "%%~1=%%V" %$\n%
  ) %$\n%
) else setlocal EnableDelayedExpansion ^& setlocal ^& set argv=,
The main performance issue are the two setlocal/endlocals per macro execution, I measured one pair of setlocal/endlocal takes ~1ms.
Mac2 Removing one pair was easy, as the inner pair is only for the argv variable.
Mac3 Removing the last setlocal EnableDelayedExpansion makes the macro nearly three times faster than before, but this has two drawbacks.
1. There are now three variables polluting the environment (can be optimized to two variables)
2. The macro only works if EnableDelayedExpansion was already active

Code: Select all

Macro | DDE  | EDE
-------------------
Mac1  | 6370 | 6370
Mac2  | 4700 | 4700
Mac3  | fail | 2600
Mac4  | 4700 | 2600
Mac5  | 4800 | 2700
It's okay for me to pollute the environment by some variables, but the macro should work always, independent of the delayed expansion mode.
But it should enable delayed expansion only when necessary.

Mac4
Here is the trick to get the current state and store it in a for meta variable with:

Code: Select all

for /F "tokens=2" %%E in ("!! D """) DO 
It stores in %%E an D, if the current mode is DisableDelayedExpansion(DDE), else two double quotes.
This can be uses in an IF statement or just another FOR

Code: Select all

( for %%X in (%%~E) DO setlocal EnableDelayedExpansion)
If it's in DDE, then %%~E expands to D and the DO part will be executed, else %%~E expands to nothing and the DO part will be skipped

Mac5

Code: Select all

(set ^"$\n=^^^
%= This creates an escaped Line Feed - DO NOT ALTER =%
)

set $strLen=for /F "tokens=2" %%E in ("!! D """) DO for %%# in (1 2) do if %%#==2 ( %$\n%
  for /f "tokens=1,2 delims=, " %%1 in ("!_!") do ( %$\n%
%= 		*** copy content to temporary variable, the carets are for extended length, up to 8191chars =% %$\n%
    (set^^ _=!%%~2!) %$\n%
    if defined _ ( %$\n%
        set "_len=1" %$\n%
        for %%P in (4096 2048 1024 512 256 128 64 32 16 8 4 2 1) do ( %$\n%
          if "!_:~%%P,1!" neq "" ( %$\n%
            set /a "_len+=%%P" %$\n%
            set "_=!_:~%%P!" %$\n%
          ) %$\n%
        ) %$\n%
    ) ELSE set "_len=0" %$\n%
    for %%V in (!_len!) do ( for %%E in (%%~E) DO endlocal ) ^& set "%%~1=%%V" %$\n%
  ) %$\n%
) else ( for %%E in (%%~E) DO setlocal EnableDelayedExpansion) ^& set _=,
This technique can also be useful, when it's necessary for returning non trivial values to a known delayed expansion mode.

Squashman
Expert
Posts: 4465
Joined: 23 Dec 2011 13:59

Re: Faster batch macros

#2 Post by Squashman » 05 Jan 2024 15:37

Awesome. How I miss the batch file creativity. Been working as Unix admin the last year.
I can't remember if we had a running thread of all the Macros that have been created so far.

T3RRY
Posts: 243
Joined: 06 May 2020 10:14

Re: Faster batch macros

#3 Post by T3RRY » 09 Jan 2024 21:36

Nice work as always Jeb.

Had to make a couple of changes to get it working in conjunction with the trick of

Code: Select all

For /f %%! in ("! ! ^^^!") Do 
for environment independent definition as well as execution. The above resulted in an expansion error occuring within the for set of the %%E loop when defined in an environment with delayed expansion enabled.

Code: Select all

@Echo off
REM Setlocal EnableDelayedExpansion

==================================================================:# Menu macro; begin Definition
REM IMPORTANT - RESERVED VARIABLES: Menu oc[*] Option Option[Key] Option[#] div_line \n

(Set \n=^^^

%= Newline var \n for multi-line macro definition - Do not modify. =%)

REM macro definition and expansion environment trickery thanks to Jeb
REM resources: 
REM https://www.dostips.com/forum/viewtopic.php?t=9265#p60294
REM https://www.dostips.com/forum/viewtopic.php?f=3&t=10983&sid=f6937e02068d93bc5a97ef63d4e5319e
REM Macros with arguments learning resource:
REM https://www.dostips.com/forum/viewtopic.php?f=3&t=1827

REM - use REM / remove REM on the below line to enable / disable menu dividing line
REM Goto :NoDividingLine

REM Enable DE environment to perform variable concatenation within a for loop
 Setlocal EnableDelayedExpansion
REM Get console width for dividing line
 for /F "usebackq tokens=2* delims=: " %%W in (`mode con ^| %__APPDIR__%findstr.exe /LIC:"Columns"`) do Set /A "Console_Width=%%W"
 Set "div_line=" & For /L %%i in (2 1 %Console_Width%)Do Set "div_line=!div_line!-"
 Endlocal & Set "div_line=%div_line%"
:NoDividingLine

REM keymap. translates literal keypress to the numeric position of the item in the menu list
Set /a oc[0]=36,oc[1]=1,oc[2]=2,oc[3]=3,oc[4]=4,oc[5]=5,oc[6]=6,oc[7]=7,oc[8]=8,oc[9]=9,oc[a]=10,oc[b]=11,^
oc[c]=12,oc[d]=13,oc[e]=14,oc[f]=15,oc[g]=16,oc[h]=17,oc[i]=18,oc[j]=19,oc[k]=20,oc[l]=21,oc[m]=22,oc[n]=23,^
oc[o]=24,oc[p]=25,oc[q]=26,oc[r]=27,oc[s]=28,oc[t]=29,oc[u]=30,oc[v]=31,oc[w]=32,oc[x]=33,oc[y]=34,oc[z]=35

REM Valid choice characters
Set "ChoList=123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0"

REM Menu macro Usage: %Menu% "quoted" "list of" "options"
%= Outer for loop allows environment independant definition =% For /f %%! in ("! ! ^^^!") Do ^
%= IMPORTANT - No whitespace permitted here =%Set Menu=For %%n in (1 2)Do if %%n==2 (%\n%
%= Detect Environment state                 =% for /F "tokens=2" %%E in ("%%!%%! D """)Do (%\n%
%= Conditionally enable Delayed Expansion   =% for /f "delims=" %%E in ("%%~E") DO setlocal EnableDelayedExpansion%\n%
%= Output Dividing Line                     =% If defined div_line Echo(%%!div_line%%!%\n%
%= Reset Choice.# index value for Opt[#]    =% Set "Choice.#=0"%\n%
%= Undefine choice option key list          =% Set "Choice.Chars="%\n%
%= For Each in list;                        =% For %%G in (%%!Options%%!)Do (%\n%
%= For Option Index value                   =%   For %%i in (%%!Choice.#%%!)Do If not %%i GTR 35 (%\n%
%= Build the Choice key list and Opt[#]     =%    Set "Choice.Chars=%%!Choice.Chars%%!%%!ChoList:~%%i,1%%!"%\n%
%= array using the character at the         =%    Set "Opt[%%!ChoList:~%%i,1%%!]=%%~G"%\n%
%= current substring index.                 =%    Set "option.output=%%~G"%\n%
%= Display [key] OptionString               =%    Echo([%%!ChoList:~%%i,1%%!] %%!Option.output%%!%\n%
%= Increment Opt[#] Index var 'Choice.#'    =%    Set /A "Choice.#+=1"%\n%
%= Close Choice.# loop                      =%   )%\n%
%= Close Options loop                       =%  )%\n%
%= Output Dividing Line                     =%  If defined div_line Echo(%%!div_line%%!%\n%
%= Select option by character index         =%  For /F "Delims=" %%o in ('%__APPDIR__%Choice.exe /N /C:%%!Choice.Chars%%!')Do For /f "tokens=1,2 delims=;" %%V in ("%%!Opt[%%o]%%!;%%!oc[%%o]%%!")Do (%\n%
%= Conditonal Endlocal; tunnel returnVars   =% (for /f "delims=" %%E in ("%%~E") DO endlocal) ^& (%\n%
%= exit [sub]script w/out modifying option  =%   If /I "%%V" == "Exit" Exit /B 2%\n%
%= Assign 'Option' with literal string      =%   Set "Option=%%V"%\n%
%= Assign 'Option[key] with key pressed     =%   Set "Option[key]=%%o"%\n%
%= Assign 'Option[#] with option number     =%   Set "Option[#]=%%~W"%\n%
%= Return to previous script on Exit        =%  ))%\n%
)%\n%
%= Capture Macro input - Options List       =%)Else Set Options=
==========================================:: Menu Definition Complete

REM demonstrates retention of last option variables upon selection of exit

REM Call:DemoMenu
REM Echo(You last chose %Option% by pressing %Option[key]% at index %option[#]%
REM Exit /b 0

:DemoMenu
CLS

 %Menu% exit List of your Options here. "Double qoute!" "Delimited Strings" filler to extend option count. caps at 36 \ / \ / \ / \ / \ / \ / \ / \ / \ / \ . / \ .

REM Returns        Variable=    Value
REM                Option=      the literal string
REM                Option[Key]= the key used to select the option
REM                Option[#]=   the Integer value of the options occurance in the list

 Echo(You Pressed %Option[key]% to Choose '%Option%'; option number = %option[#]%

 Pause
Goto:DemoMenu

miskox
Posts: 553
Joined: 28 Jun 2010 03:46

Re: Faster batch macros

#4 Post by miskox » 11 Jan 2024 10:18

I didn't understand a thing Jeb wrote.

I tried to learn how to make/use macros. No luck. A 'macro for dummies' should be posted by someone who can do it.

Saso

T3RRY
Posts: 243
Joined: 06 May 2020 10:14

Re: Faster batch macros

#5 Post by T3RRY » 11 Jan 2024 19:57

miskox wrote:
11 Jan 2024 10:18
I didn't understand a thing Jeb wrote.

I tried to learn how to make/use macros. No luck. A 'macro for dummies' should be posted by someone who can do it.

Saso
the short of it:
using setlocal / endlocal regardless of environment state adds a performance cost that can be minimised by modifying those setlocal / endlocal pairs to occur only on the condition the environment state is disableDelayedExpansion

The state is Captured in Jeb's method based on which token is parsed in the outer %%E loop.
the macro uses aditional additional for Loops to parse the captured value and perform conditional execution.

In an Enabled delayedExpansion environment, the !! is expanded to nothing, and "" becomes the token captured.
the subsequent for loops WONT execute in that case, as "" expanded as %%~E becomes nothing.

In a Disabled delayedExpansionEnvironment, !! does not expand, D becomes the token captured, and the subsequent For %%E loops WILL execute with %%~E expanded as D

This deosn't just permit faster macro's, it allows them to be crafted to be completely modular - when combined with another for trick that captures the un/escaped form of ! required for whatever environment is in use within a metavariable.

Lowsun
Posts: 29
Joined: 14 Apr 2019 17:22

Re: Faster batch macros

#6 Post by Lowsun » 11 Jan 2024 21:26

This is cool. Thought of a fun way to do it, though haven't tested it inside a macro or anything. This way the 2 FOR loops aren't needed, though the string is much longer.

Code: Select all

@ECHO ON
SETLOCAL ENABLEDELAYEDEXPANSION

SET "v=1"
FOR /F "tokens=2-7" %%1 in ("!! SET REM ENABLEDELAYEDEXPANSION END REM LOCAL") DO (
    %%1%%6 %%3
    ECHO !v!
    %%4%%6
)
PAUSE
EXIT /B

T3RRY
Posts: 243
Joined: 06 May 2020 10:14

Re: Faster batch macros

#7 Post by T3RRY » 12 Jan 2024 00:00

Lowsun wrote:
11 Jan 2024 21:26
This is cool. Thought of a fun way to do it, though haven't tested it inside a macro or anything. This way the 2 FOR loops aren't needed, though the string is much longer.

Code: Select all

@ECHO ON
SETLOCAL ENABLEDELAYEDEXPANSION

SET "v=1"
FOR /F "tokens=2-7" %%1 in ("!! SET REM ENABLEDELAYEDEXPANSION END REM LOCAL") DO (
    %%1%%6 %%3
    ECHO !v!
    %%4%%6
)
PAUSE
EXIT /B
This is exceptionally clever and works exactly as expected in a macro context - test script
Can't decide if it's more or less readable though. It's easy enough to trace back the potential tokens IMO though

Code: Select all

@Echo off

REM Setlocal EnableDelayedExpansion

REM Define a newline variable for use in defining multiline Macros
(Set \n=^^^

%= Empty line above required =%)

For /f %%! in ("! ^! ^^^!")Do ^
Set ThisMacro=For %%. in (1 2)Do if %%.==2 (%\n%
	FOR /F "tokens=2-7" %%1 in ("%%!%%! SET REM ENABLEDELAYEDEXPANSION END REM LOCAL") DO (%\n%
		Echo(%%1%%6 %%3%\n%
		%%1%%6 %%3%\n: Conditional Setlocal or Rem =%
		Echo(%%!Args_ThisMacro%%!%\n%
		Echo(%%4%%6%\n%
		%%4%%6%\n: Conditional Endlocal or Rem =%
	)%\n%
)Else Set Args_ThisMacro=

%ThisMacro%Test Output

PAUSE
Goto:Eof

jfl
Posts: 226
Joined: 26 Oct 2012 06:40
Location: Saint Hilaire du Touvet, France
Contact:

Re: Faster batch macros

#8 Post by jfl » 15 Jan 2024 09:56

A very clever trick indeed, but maybe a bit more complex than necessary?
Here's a simpler version, that's also easier to understand: (I think)

Code: Select all

For /f %%! in ("! ^! ^^^!") Do ^
Set ThisMacro=For %%. in (1 2) Do if %%.==2 (%\n%
  FOR /F "tokens=2-4" %%1 in ("%%!%%! SETLOCAL REM ENDLOCAL REM") DO (%\n%
    %%1 ENABLEDELAYEDEXPANSION%\n: Conditional Setlocal or Rem =%
    Echo(%%!Args_ThisMacro:~1%%!%\n%
    %%3%\n: Conditional Endlocal or Rem =%
  )%\n%
) Else Set Args_ThisMacro=
I then experimented with alternatives, using state variables set using `if [!!]==[]` tests in the macro. But it's actually slower than the above code.
The strength of @Lowsun's method is that a single `for /f` replaces several `if [!!]==[]` and set instructions.

jeb
Expert
Posts: 1041
Joined: 30 Aug 2007 08:05
Location: Germany, Bochum

Re: Faster batch macros

#9 Post by jeb » 16 Jan 2024 07:08

@jfl Thanks, your short version of the @Lowsun FOR/F-loop was exactly what I had in mind.

But your tests with `if [!!]==[]` has to be failed, because the expression works only at the start detection, but in the end it fails,
because then delayed expansion is always enabled and you can't decide with `if [!!]==[]` if a endlocal has to be executed or not (I played with exactly the same expression at my beginning :D ).

jfl
Posts: 226
Joined: 26 Oct 2012 06:40
Location: Saint Hilaire du Touvet, France
Contact:

Re: Faster batch macros

#10 Post by jfl » 16 Jan 2024 11:06

jeb wrote:
16 Jan 2024 07:08
But your tests with `if [!!]==[]` has to be failed, because the expression works only at the start detection, but in the end it fails...
Yeah, I noticed that eventually. :oops:

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: Faster batch macros

#11 Post by aGerman » 16 Jan 2024 12:30

To REM or not to REM that is the question %%~?

Code: Select all

for /f %%! in ("! ^! ^^^!") do ^
set ThisMacro=for %%. in (1 2) do if %%.==2 (%\n%
  for /f "tokens=2" %%? in ("%%!%%! "" rem=") do (%\n%
    %%~?setlocal EnableDelayedExpansion%\n%
    echo(%%!Args_ThisMacro:~1%%!%\n%
    %%~?endlocal%\n%
  )%\n%
) else set Args_ThisMacro=
I'm still not sure if I like this better. :lol:

Steffen

jfl
Posts: 226
Joined: 26 Oct 2012 06:40
Location: Saint Hilaire du Touvet, France
Contact:

Re: Faster batch macros

#12 Post by jfl » 17 Jan 2024 07:18

aGerman wrote:
16 Jan 2024 12:30
I'm still not sure if I like this better. :lol:
Arthur Clarke once said that perfection was reached, not when you can't add anything anymore to your system, but when you can't remove anything from it without it stopping functionning.
I think you may have reached perfection here.

Which brings a question that turned me crazy this morning: I tried this minor variation on your code... But it does not work when expansion is disabled:
(Removed the = at the end of rem= , and added a space after each %%~?)

Code: Select all

for /f %%! in ("! ^! ^^^!") do ^
set ThisMacro=for %%. in (1 2) do if %%.==2 (%\n%
  for /f "tokens=2" %%? in ("%%!%%! "" rem") do (%\n%
    %%~? setlocal EnableDelayedExpansion%\n%
    echo(%%!Args_ThisMacro:~1%%!%\n%
    %%~? endlocal%\n%
  )%\n%
) else set Args_ThisMacro=
Why?!?

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: Faster batch macros

#13 Post by aGerman » 17 Jan 2024 11:17

Uhh that's something that can drive you mad. You need to glue it to the next command because it would be treated as a separate command otherwise. Even an empty string or a space is parsed like that. I can't make my brain remember the order and details of script parsing. However, you may find the answer in this SO documentation about investigations on that. Some account names will sound familiar to you :lol:
https://stackoverflow.com/questions/409 ... 33#4095133

FWIW I used "REM=" because the equal sign is one of the separators that don't trigger accessing the file system in this context. Comma and semicolon would have been possible as well, while e.g. dot, forward slash and backslash would have caused file searching first which has a huge impact when executed thousands of times in a loop.

Steffen

T3RRY
Posts: 243
Joined: 06 May 2020 10:14

Re: Faster batch macros

#14 Post by T3RRY » 18 Jan 2024 07:55

jfl wrote:
17 Jan 2024 07:18
Arthur Clarke once said that perfection was reached, not when you can't add anything anymore to your system, but when you can't remove anything from it without it stopping functionning.
I think you may have reached perfection here.
Couldn't agree more. Not only is it the most concise, it's the most readable and intuitive of the variations.

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: Faster batch macros

#15 Post by aGerman » 18 Jan 2024 12:28

After reading jfl's timing of the NOP macro, I think we should probably go back to an IF statement.

Code: Select all

for /f %%! in ("! ^! ^^^!") do ^
set ThisMacro=for %%. in (1 2) do if %%.==2 (%\n%
  for /f "tokens=2" %%1 in ("%%!%%! 1 0") do (%\n%
    if 1==%%1 setlocal EnableDelayedExpansion%\n%
    echo(%%!Args_ThisMacro:~1%%!%\n%
    if 1==%%1 endlocal%\n%
  )%\n%
) else set Args_ThisMacro=
It's definitely a performance gain.

Steffen

Post Reply