Avoid call in for loop with complex variables

[einstein1969]

Hi,

I have a question about call set/call echo

This is the code of example

Code: Select all

@echo off

setlocal disabledelayedexpansion

set param[1].1=aaa
set param[1].2=BBB
set param[1].3=ccc

set param[2].1=111
set param[2].2=222
set param[2].3=333

set cont[1]=3
set cont[2]=2

echo %cont[1]%
echo param[1].%cont[1]%
call echo %%param[1].%cont[1]%%%

output

Code: Select all

3
param[1].3
ccc

This work...

I need this in a for loop for performance.

Code: Select all

@echo off

setlocal disabledelayedexpansion

set param[1].1=aaa
set param[1].2=BBB
set param[1].3=ccc

set param[2].1=111
set param[2].2=222
set param[2].3=333

set cont[1]=3
set cont[2]=2

setlocal enabledelayedexpansion
For /L %%k in (1,1,2) do (

      echo !cont[%%k]!
      echo param[%%k].!cont[%%k]!

      call echo %%param[%%k].!cont[%%k]!%%

)

output:

Code: Select all

3
param[1].3
ccc
2
param[2].2
222

work again...

but i want substitute CALL for perfomance raisons

any Idea?

[Liviu]

Code: Select all

      call echo %%param[%%k].!cont[%%k]!%%

Technically, this should work

Code: Select all

for /F "delims=" %%j in ("param[%%k].!cont[%%k]!") do echo !%%j!

though I'd be surprised if it made much of a difference performance wise. Fact is that you need to somehow introduce an additional level or (re)parsing, be it with 'call' or a nested 'for' or some other trick.

Liviu

[dbenham]

Technically, this should work

Code: Select all

for /F "delims=" %%j in ("param[%%k].!cont[%%k]!") do echo !%%j!

though I'd be surprised if it made much of a difference performance wise. Fact is that you need to somehow introduce an additional level or (re)parsing, be it with 'call' or a nested 'for' or some other trick.

It makes a huge difference

I timed the following loops - 5000 iterations each:

Code: Select all

set t1=%time%
for /l %%N in (1 1 5000) do (
  For /L %%k in (1,1,2) do (
    call echo %%param[%%k].!cont[%%k]!%% >nul
  )
)
set t2=%time%
for /l %%N in (1 1 5000) do (
  For /L %%k in (1,1,2) do (
    for /F "delims=" %%j in ("param[%%k].!cont[%%k]!") do echo !%%j! >nul
  )
)
set t3=%time%

%macro.diffTime% t1 t2 time_CALL
%macro.diffTime% t2 t3 time__FOR
set time_

The FOR version is ~5 times faster than the call version

Code: Select all

time_CALL=00:00:13.66
time__FOR=00:00:02.75

CALL is amazingly slow. And CALLing a command is even slower than CALLing a :label. See CALL me, or better avoid call. Be sure to follow jeb's link at the bottom of the 1st post and read the 1st two posts on page 2.


Dave Benham

[Liviu]

It makes a huge difference

Indeed. Thanks for the pointers. FWIW the ratio is below 4x for me, possibly due to keeping a most frugal %path%.

It's still close to 2x even if I "call" after "set "path=" & set "pathext=;"" (viewtopic.php?f=3&t=2521) which I find quite surprising, but then 'cmd' never claimed to be predictable

Liviu

[einstein1969]

This is the code like to optimize

Code: Select all

@echo off
setlocal EnableDelayedExpansion

set debug=OFF
if "%debug%"=="ON" (set "echod=echo") else (set "echod=rem")
for /f %%a in ('copy /Z "%~dpf0" nul') do set "CR=%%a"

set "run=dna-search-Needleman–Wunsch_BLAST.exe --param240:AGCT"

set /a maxvalue=20
set /a numparam=0

call :cancella_var

call :addparameter "--key 314","--key 1.61"
call :addparameter "--mne 4","--mne 8","--mne 16","--mne 24"
call :addparameter "--em aaa","--em bbb","--em ccc"
call :addparameter "--grat sp","--grat ao"
call :addparameter "--no-wt","--wt"
call :addparameter "--wp 0l","--wpl 1","--wpl 2"
call :addparameter "--ttt","--no-ttt"
call :addparameter "--q-emen 0","--q-emen 1","--q-emen 2"
call :addparameter "--blayer 0","--blayer 1","--blayer 2","--blayer 3","--blayer 4","--blayer 5","--blayer 6"
call :addparameter "--ps none","--ps i","--ps p","--ps b","--ps i,q","--ps i,b","--ps p,b","--ps i,i","--ps i,p,b","--ps i,i,p","--ps i,i,b","--ps p,b,i,i","--pn all"
call :addparameter "--rd 0","--rd 10","--rd 20","--rd 40","--rd 60"
call :addparameter "--rf 1","--rf 2","--rf 4","--rf 6"
call :addparameter "--no-ms","--ms"
call :addparameter "--se 0","--se 1","--se 2","--se 3","--se 4","--se 5","--se 6","--se 7","--se 8","--se 9","--se 10"
call :addparameter "--ts 0","--ts 1","--ts 2"
call :addparameter "--ae 0","--ae 1","--ae 2"
call :addparameter "--no-cc","--cc"
call :addparameter "--no-k","--k 0:0"
call :addparameter "--no-be","--be"


for /L %%i in (1,1,%numparam%) do (
      for /L %%k in (1,1,%maxvalue%) do (
         if defined param[%%i].%%k %echod% param[%%i].%%k=!param[%%i].%%k!
      )
)
if "%debug%"=="ON" (pause>nul&echo.)

set level=0

Rem for speeding call set/echo
set path=
set "PathExt=$"


call :iterativa_small 

endlocal

goto :eof



::::::::::::::::::::::::::::::::::::::::::::::::::::::
:addparameter

set /a numparam+=1
set /a ind=1

:loop
if "%~1" equ "" goto :eof

   set "param[%numparam%].%ind%=%~1"
   %echod% "param[%numparam%].%ind%=%~1"
   shift
   set /a ind+=1
   goto :loop

goto :eof
::::::::::::::::::::::::::::::::::::::::::::::::::::::

::::::::::::::::::::::::::::::::::::::::::::::::::::::
:cancella_var

for /f "delims==" %%f in ('set param') do (
   set "%%f="
   rem echo %%f
   rem set %%f
)
for /f "delims==" %%f in ('set var') do (
   set "%%f="
   rem echo %%f
   rem set %%f
)
for /f "delims==" %%f in ('set cont') do (
   set "%%f="
   rem echo %%f
   rem set %%f
)

goto :eof
::::::::::::::::::::::::::::::::::::::::::::::::::::::

::::::::::::::::::::::::::::::::::::::::::::::::::::::
:iterativa_small (i,j)

setlocal

Echo Entering...

Set /a indice=1

For /L %%k in (1,1,%Numparam%) do set /a cont[%%k]=1

For /L %%k in (1,1,%Numparam%) do (
      rem echo param[%%k].!cont[%%k]!
      call set param=%%param[%%k].!cont[%%k]!%%
      rem echo !param!
      if "%debug%"=="ON" <nul set /p "=!param! " 
)
%echod%(
<nul set /p "=INDICE:!indice! !CR!"
%echod%(

set /a i=1

set t0=!time!

for /L %%l in (1,0,1) do (

   if "%debug%"=="ON" (pause)

   set /a cont[!i!]+=1

   call set cont=%%cont[!i!]%%

   call set param=%%param[!i!].!cont!%%

   %echod% I:!i! cont[!i!]=!cont! DEF? param[!i!].!cont!=!param!

   if defined param (
      
      %echod% I:!i! cont[!i!]=!cont! DEF  param[!i!].!cont!=!param!

      if !i! gtr 1 (

         set /a i-=1

         For /L %%k in (!i!,-1,1) do ( 

            rem set /a i-=1
      
            set /a cont[%%k]=1

            %echod% I:!i! LOOP:%%k cont[%%k]=!cont[%%k]!

         )

         set /a i=1
      )


   For /L %%k in (1,1,%Numparam%) do (
      rem %echod% param[%%k].!cont[%%k]!
      call set param=%%param[%%k].!cont[%%k]!%%
      rem echo !param!
      if "%debug%"=="ON" <nul set /p "=!param! "
   )
   %echod%(

   set t1=!time!
   set /a d=1!t1:~6,2!-1!t0:~6,2!,m=1!t1:~3,2!-1!t0:~3,2!,md=d+m*60+1,vel=!indice!/md

   set /a indice+=1
   <nul set /p "=INDICE:!indice! !vel!/s !CR!"
   

   if !indice! gtr 100000 %run% %params%

   ) else ( 
      %echod% NDEF, INC I
      set /a i+=1 
   )

Rem This not work well... there is a way to exit without calling the for?
if !i! GTR %numparam% call :halt

)

endlocal

goto :eof


:Halt
echo Exiting...
pause
call :_halt 2> NUL
:_halt
()
goto :eof

with the optimizing path i double the speed. But i need more speed...

[jeb]

I didn't test your code, but I saw this

Code: Select all

 For /L %%k in (1,1,%Numparam%) do (
      rem %echod% param[%%k].!cont[%%k]!
      call set param=%%param[%%k].!cont[%%k]!%%
      rem echo !param!
      if "%debug%"=="ON" <nul set /p "=!param! "
   )

If this is speed criticial, you should avoid the CALL as it's really slow, even the REM will cost time, but only a bit.

To avoid it you could create one more expansion by using this.

Code: Select all

for /F "delims=" %%P in (param[%%k].!cont[%%k]!) DO set "param=!%%P!"

If you know that the content of a variable is numeric you could avoid the CALL even simpler.

Code: Select all

set /A cont=cont[!i!]

instead of

Code: Select all

call set cont=%%cont[!i!]%%

jeb

[einstein1969]

Thanks very much guys

I went from about 50 cycles/second to 450 cycles/s


So it is better to optimize the CALL that use the trick of PATH*

the optimized code:

Code: Select all

set /a i=1

set t0=!time!

for /L %%l in (1,0,1) do (

   set /a cont[!i!]+=1,cont=cont[!i!]

   for /F "delims=" %%P in ("param[!i!].!cont!") DO set "param=!%%P!"

   if defined param (
      
      if !i! gtr 1 (

         set /a i-=1

         For /L %%k in (!i!,-1,1) do set /a cont[%%k]=1

         set /a i=1
      )


   For /L %%k in (1,1,%Numparam%) do for /F "delims=" %%P in ("param[%%k].!cont[%%k]!") DO set "param=!%%P!"

   set /a indice+=1,rr=indice %% 100
   if !rr! equ 0 ( 
         set t1=!time!
         set /a d=1!t1:~6,2!-1!t0:~6,2!,m=1!t1:~3,2!-1!t0:~3,2!,md=d+m*60+1,vel=!indice!/md
         <nul set /p "=INDICE:!indice! !vel!/s !rr! !CR!"
   )
   
   if !indice! gtr 100000 %run% %params%

   ) else set /a i+=1

   if !i! GTR %numparam% call :halt

)

[Liviu]

If you know that the content of a variable is numeric you could avoid the CALL even simpler.

Code: Select all

set /A cont=cont[!i!]

Nice! Thanks for the tip.

Code: Select all

Rem This not work well... there is a way to exit without calling the for?
if !i! GTR %numparam% call :halt

If you search the forum for "infinite loop" you'll find a number of hits and some extensive discussions. The ways I am aware of basically boil down to running the infinite loop into a secondary shell and hard-exit'ing it when done. For a simple example, the code below will loop until the second of the computer %time% changes.

Code: Select all

@if "%~1"==":loop" goto :loop

:main ----------------
@echo off
setLocal disableDelayedExpansion
@rem 'cmd /c' nesting needed in order to 'exit' infinite 'for' loop cleanly
cmd /s /c ""%~f0" :loop"
@rem --- reached after nested call ---
endLocal
goto :eof

:loop ----------------
@echo off
setLocal enableDelayedExpansion
for /l %%n in () do (
  @rem loop break condition which requires 'cmd /c' nesting
  @rem this for example loops until the next second starts
  @rem replace with applicable break condition in real code
  if not "!time:~-5,2!"=="%time:~-5,2%" (echo now is !time! & exit)
)
echo --- never reached ---
endLocal
goto :eof

Liviu

[einstein1969]

Many thanks to Liviu. With this I ended questions.

[Liviu]

Glad to hear that you've got it working. Thanks for the feedback.