Why does SET performance degrade as environment size grows?

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Message
Author
dbenham
Expert
Posts: 2288
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: Why does SET performance degrade as environment size grows?

#31 Post by dbenham » 24 Sep 2016 22:16

This large memory issue cropped up again. I ran into a massive performance degradation problem while working on a StackOverflow question - http://stackoverflow.com/a/39672776/1012053. Within a loop of 1000, I was creating hundreds of variables between SETLOCAL and ENDLOCAL. The performance was totally non-linear, to the point of quickly becoming unusable.

I believe I have made a significant new discovery related to environment memory management. We had been thinking that the environment never shrinks. But I discovered that I could get the environment to shrink if I put the SETLOCAL, massive allocation, and ENDLOCAL within a CALLed subroutine, or within a new cmd.exe process. The new process wasn't much of a surprise, but the CALL solution was.

I haven't had time to fully investigate, but this looks promising. :D


Dave Benham

Ed Dyreen
Expert
Posts: 1569
Joined: 16 May 2011 08:21
Location: Flanders(Belgium)
Contact:

Re: Why does SET performance degrade as environment size grows?

#32 Post by Ed Dyreen » 24 Jan 2019 05:31

This is great info :D

@Aacini, I used your analysis results to optimize my code.

I have a few hundreds of variables stored on disk that all follow the same analogy; set variable=data. where variable is fileName and data can be anything from 0 to 8k of data.

I used sort.EXE to get an alphabetical filename list and then call the variables into memory alphabetically and reverse alphabetically. I expected the reverse alphabet list to load variables much more slowly in memory but in fact speed remained exactly the same. So there seem to be more involved that add up to the speed than just the Set command. ( The names of the variables are sorted but now call will need to jump from one directory to the other whereas before sorting was by output of for /R )

Then I increase the environment by 1 MB using 256x4096kb variables and unload them in reverse order. This takes only a few seconds however it takes exponentially more time as environment grows to preserve more memory and mem.EXE /P crash above 1MB on windows XP.

Code: Select all

	setlocal enableDelayedExpansion
	:: (
		set "$4096=12345678" &for /L %%i in ( 1, 1, 9 ) do set "$4096=!$4096!!$4096!"
	:: )
	endlocal &set "a=%$4096:~7%"%=						minus 7 for variable name		=%

	(echo. ) &set "oldSize=" &for /F "skip=1 tokens=1-3" %%a in (

		'mem.EXE /P ^|findstr.EXE /I "COMMAND"'

	) do 	if not defined oldSize echo.  occupied range: '%%a-%%c' &set /A oldSize = 0x%%c
	::
	echo.  oldSize: '%oldSize%' bytes.

	for /L %%u in ( 1, 1, %$reserveMemorySize% ) do for /L %%x in ( 1, 1, 4 ) do (%=1 megaByte / 4 	= 256 kiloBytes	=%

		for /L %%y in ( 1, 1, 8 ) do (%=				256 kiloBytes / 8 	= 32 kiloBytes	=%

			for /L %%z in ( 1, 1, 8 ) do (%=			32 kiloBytes / 8 	= 4096 	bytes	=%

				set "$[%%u%%x%%y%%z]=%a%"
			)
		)
	)

	set /A edp = 128 * $reserveMemorySize
	echo.  accomodating up to %edp% fast definitions in %$reserveMemorySize% megabytes.

	if +%$reserveMemorySize% LSS 2 ( %=					mem.EXE crash above 1mb on winXPSP3	=%

		(echo. ) &set "newSize=" &for /F "skip=1 tokens=1-3" %%a in (

			'mem.EXE /P ^|findstr.EXE /I "COMMAND"'

		) do 	if not defined newSize echo.  occupied range: '%%a-%%c' &set /A newSize = 0x%%c
		::
		set /A size = newSize - oldSize
	)
	if +%$reserveMemorySize% LSS 2 ( %=					mem.EXE crash above 1mb on winXPSP3	=%

		echo.  newSize: '%newSize%' bytes.
		echo.  growth : '%size%' bytes.
	)

	(echo. ) &echo.  clearing %$reserveMemorySize% megabytes in reverse order...
	::
	for /L %%u in ( %$reserveMemorySize%, -1, 1 ) do for /L %%x in ( 4, -1, 1 ) do (%=1 megaByte / 4= 256 kiloBytes	=%

		for /L %%y in ( 8, -1, 1 ) do (%=				256 kiloBytes / 8 	= 32 kiloBytes	=%

			for /L %%z in ( 8, -1, 1 ) do (%=			32 kiloBytes / 8 	= 4096 	bytes	=%

				set "$[%%u%%x%%y%%z]="
			)
		)
	)
But this one did improve performance of the Set command by a whopping 275%.

Post Reply