Page 1 of 2

Limit CMD processing to internal commands, safer and faster?

Posted: 21 Nov 2011 17:11
by OJBakker
In the thread CALL me, or better avoid call at
http://www.dostips.com/forum/viewtopic.php?f=3&t=1947&hilit=precedence
Jeb demonstrated the slowness and unsafety of 'call set' and 'call echo'

The code below is an attempt to tackle both problems.
This is done by skipping the superfluous directory and extension searches.
This effectively limits default processing to internal commands only.
External commands can still be used but will require path and extension.

I have used the commandlist of Vista and done limited tests on Vista.

iCMDdemo.bat

Code: Select all

@echo off
cls
   echo examples for "call set" and "call echo"
   echo(
   echo call :iCmd Set var=hello by set
   echo call :iCmd Echo hello by echo
   set "var="
   call :iCmd Set var=hello by set
   call :iCmd Echo hello by echo
   set var&rem show new value
   echo(
   echo The same but now without iCMD
   echo(
   echo call :LimitCMDToInternalCommands
   echo call Set var=hello by set
   echo call Echo hello by echo
   echo call :RestoreCMDExternalCommands
   call :LimitCMDToInternalCommands
   set "var="
   call Set var=hello by set
   call Echo hello by echo
   set var&rem show new value
   call :RestoreCMDExternalCommands
   echo(
   echo Done
   call :iCmd
   exit/b
goto:eof

:LimitCMDToInternalCommands
rem store current (d)path(ext)
if defined path set pushPath=%path%
if defined dpath set pushDpath=%dpath%
set pushPathExt=%pathext%
rem limit (d)path(ext) NOTE: pathext must be defined or call will fallback to cmd internal default)
path;&dpath;&set pathext=;
goto :eof

:RestoreCMDExternalCommands
rem restore original (d)path(ext)
if defined pushPath path %pushPath%
if defined pushdpath dpath %pushDpath%
set pathext=%pushPathExt%
rem cleanup
set pushPath=&set pushDPath=&set pushPathExt=
goto :eof

:iCmd internalCommand parameters
REM echo params [%*]
if not @%1 == @ (
   call :LimitCMDToInternalCommands
   rem call internal command
   for %%a in (ASSOC BREAK CALL CD CHDIR CLS COLOR COPY DATE DEL DIR DPATH ECHO ENDLOCAL ERASE ^
               FTYPE KEYS MD MKDIR MKLINK MOVE PATH PAUSE POPD PROMPT PUSHD RD REM ^
               REN RENAME RMDIR SET SETLOCAL SHIFT TIME TITLE TYPE VER VERIFY VOL ^
               ) do if /I @%%a == @%1 call %*
   call :RestoreCMDExternalCommands
) else (
   echo(
   echo Syntax:
   echo(
   echo call :iCmd NameForInternalCommand ParametersForInternalCommand
   echo(
   echo Some remarks:
   echo Internal commands IF and FOR can not be used with call
   echo Internal commands GOTO and EXIT can be used with iCmd and will be safe but cause a problem,
   echo These commands transfer control so restoring the orinal [d]path[ext] is not possible!
   echo This can not be solved inside iCmd but can be solved in the calling batchfile.
   echo Use the subs LimitCMD... and RestoreCMD... in the calling batchfile.
   echo Use Setlocal and Endlocal as further safeguard against losing your paths [in case of error or EXIT]
   echo In limited mode external commands can still be used but you have to use full path and file extension.
   echo(
   pause
   exit/b
   )
   goto :eof
)
goto :eof

Re: Limit CMD processing to internal commands, safer and fas

Posted: 21 Nov 2011 17:19
by Ed Dyreen
'
There's something I don't understand:

Code: Select all

@echo off

setlocal
   set "path=" %= path disabled, it's safe =%
endlocal

set "path"
Why do we have to restore the path ?, It's a variable that is read from registry when DOS is initialized :?

Re: Limit CMD processing to internal commands, safer and fas

Posted: 21 Nov 2011 17:51
by OJBakker
Restoring the path is not a requirement, just good practice.
Imagine working from the commandline.
You run a batchfile that wipes the path and does not restore it before finishing.
Suddenly your external commands don't work anymore.
Just because these are normally found through the default path-search.
An other example is if your batchfile calls another batchfile that uses external commands.
This might not work if these command are not accesible through the path and pathext.
Good use of setlocal - endlocal pairs has the same effect as restoring the path.

Re: Limit CMD processing to internal commands, safer and fas

Posted: 22 Nov 2011 02:43
by jeb
OJBakker wrote:This is done by skipping the superfluous directory and extension searches.
This effectively limits default processing to internal commands only.
External commands can still be used but will require path and extension.

Ok, I understand what you want to do, but I can't follow why you do it this way :?:

Why you need this
OJBakker wrote:for %%a in (ASSOC BREAK CALL CD CHDIR CLS COLOR COPY DATE DEL DIR DPATH ECHO ENDLOCAL ERASE ^
FTYPE KEYS MD MKDIR MKLINK MOVE PATH PAUSE POPD PROMPT PUSHD RD REM ^
REN RENAME RMDIR SET SETLOCAL SHIFT TIME TITLE TYPE VER VERIFY VOL ^
) do if /I @%%a == @%1 call %*


Isn't it the same as call %* ?

IMHO a simple function like could work even better.

Code: Select all

:xCmd
%*
exit /b


I made some tests

Code: Select all

@echo off
setlocal
cls
   echo(

   set ^"test1=Special characters "&|<>!""
   set test1
   echo --------
   call       Echo direct 1a: %test1%
   call :iCmd Echo iCmd   1b: %test1%
   call :xCmd Echo xCmd   1c: %test1%

   echo(
   set "test2=caret test ^^ "^^^^""
   set test2
   echo --------
   call       Echo direct 2a: %test2%
   call :iCmd Echo iCmd   2b: %test2%
   call :xCmd Echo xCmd   2c: %test2%

   echo(
   set ^"test3=Special characters %%level2%% "&|<>!""
   set "level2=two ^&"
   set test3
   echo --------
   call       Echo direct 2a: %test3%
   call :iCmd Echo iCmd   2b: %test3%
   call :xCmd Echo xCmd   2c: %test3%

   call :RestoreCMDExternalCommands
   echo(
   echo Done
   exit/b


Output wrote:test1=Special characters "&|<>!"
--------
direct 1a: Special characters "&|<>!"
iCmd 1b: Special characters "&|<>!"
xCmd 1c: Special characters "&|<>!"

test2=caret test ^^ "^^"
--------
direct 2a: caret test ^ "^^^^"
iCmd 2b: caret test "^^^^^^^^"
xCmd 2c: caret test "^^^^"

test3=Special characters %level2% "&|<>!"
--------
direct 2a: Special characters two & "&|<>!"
iCmd 2b: Special characters two
Der Befehl ""&|<>!"" ist entweder falsch geschrieben oder
konnte nicht gefunden werden.
xCmd 2c: Special characters two
Der Befehl ""&|<>!"" ist entweder falsch geschrieben oder
konnte nicht gefunden werden.

Done


But both workarounds, iCmd and xCmd change the content (test2) or even fails complete (test3).

Currently i didn't see a "good" workaround. :(

I'm hopeing that there could be a syntax trick to avoid the disk scanning, something like

Code: Select all

call (;!echo mytest

No, this one doesn't work :-(

jeb

Re: Limit CMD processing to internal commands, safer and fas

Posted: 22 Nov 2011 05:08
by OJBakker
hmmm I guess my explanation was not very clear.

I started with Jebs "Call Set" - "Set.bat" vulnerability.
Tried to encapsulate the Call Set to isolate this from de FileSearch.
Then I expanded my test to all internal commands.
I can confirm that all internal commands are vulnerable to
the 'batchfile-with-internalcommand-name' intercept.
I encountered problems with GOTO and EXIT.
These clearly showed me that isolating individual commands won't work.
Besides that it is very inefficient and unnecessary.
The essence of my routines is disabling the FileSearch and there is no need to
limit this to single commands.

So I should better not have posted the iCMD routine at all.

Here is my revised code.

The 2 subroutines are the same as previously posted.
I just changed the names for better clarity.

I hope this explains it better.

DisableCMDFileSearch.bat

Code: Select all

@echo off
cls
   echo examples for "call set" and "call echo"
   echo(
   echo call :LimitCMDToInternalCommands
   echo call Set var=hello by set
   echo call Echo hello by echo
   echo call :RestoreCMDExternalCommands

   call :DisableDefCmdFileSearch
   rem now call is safe
   set "var="
   call Set var=hello by set
   call Echo hello by echo
   set var&rem show new value
   call :ReEnableDefCmdFileSearch
   rem now call is unsafe
   echo(
   echo Done

   call :iSyntax
   exit/b
goto:eof

:DisableDefCmdFileSearch
rem store current [d]path[ext]
if defined path set pushPath=%path%
if defined dpath set pushDpath=%dpath%
set pushPathExt=%pathext%
rem limit (d)path(ext) NOTE: pathext must be defined or call will fallback to cmd internal default)
path;&dpath;&set pathext=;
goto :eof

:ReEnableDefCmdFileSearch
rem restore original [d]path[ext]
if defined pushPath path %pushPath%
if defined pushdpath dpath %pushDpath%
set pathext=%pushPathExt%
rem cleanup
set pushPath=&set pushDPath=&set pushPathExt=
goto :eof

:iSyntax
   echo(
   echo Syntax:
   echo(
   echo use call :DisableDefCmdFileSearch
   echo to disable the default filesearching used by cmd
   echo example: "call set var=value" can no longer be redirected to batchfile set.bat

   echo use call :ReEnableDefCmdFileSearch
   echo to reenable the default filesearching used by cmd
   echo now "call InternalCommand" can again be intercepted with a batchfile named "InternalCommand.bat"
   echo(
   echo Some remarks:
   echo After disabling the FileSearch external commands can only be used with path and extension.
   echo example "dir|more" must be changed to "dir|%windir%\System32\more.exe"
   echo Use Setlocal and Endlocal as further safeguard against losing your paths [in case of error or EXIT]
   echo If you don't bother about preserving the paths all you need is the command:
   echo (
   echo path;&dpath;&set pathext=;
   echo (
   echo If disabling the FileSearch is done after a SetLocal, the paired EndLocal will restore the paths.
   echo (
   pause
   exit/b
goto :eof

Re: Limit CMD processing to internal commands, safer and fas

Posted: 22 Nov 2011 21:35
by Ed Dyreen
'
If I set the path to nothing DOS will not search the path for a file named echo when echo. right ?
I was wondering if it also has effect on performance.
I also think of problems, when u install a program from the command-line.

What do the experts think, should I ?

Re: Limit CMD processing to internal commands, safer and fas

Posted: 23 Nov 2011 02:46
by OJBakker
If I set the path to nothing DOS will not search the path for a file named echo when echo. right ?

Almost, just disabling path is not enough.
FileSearch will still be done in the current directory.
That is why the FileExt has to be disabled as well.

I was wondering if it also has effect on performance.

Met too, but I have not yet done timing-tests with and without these settings.

I also think of problems, when u install a program from the command-line.

This is NOT a set and forget thingy. Better safe than sorry.
Restoring the paths is easy and fast, can be done explicit with call :ReEnableDefCmdFileSearch
or implicit using endlocal.
So I suggest: in case of doubt, always restore the paths.
An other option might be using start /I for external stuff but I have not tested this.

Re: Limit CMD processing to internal commands, safer and fas

Posted: 23 Nov 2011 07:27
by dbenham
Interesting idea OJBakker

Using simple %path% expansion is not safe: Path Issues. But that can be fixed.

The overall concept is sound, but the calls to enable and disable path etc. have their own inefficiency.

Ed, Jeb - This sounds like a job for the new macros :!:

I'm envisioning a syntax where

Code: Select all

call set "var=%%var2%%"

is replaced by

Code: Select all

%macro.icall% set "var=%%var2%%"
:: or
%macro.icall% set "var=%%%%var2%%%%"
:: I'm not sure which one is needed

We eliminate the leading comma during the assignment of argv, and instead of parsing !argv!, we read the entire thing into one FOR variable using "delims=".

The macro simply disables path (etc), calls the argv FOR variable, then restores path (etc). We just need to be careful with where we place SETLOCAL/ENDLOCAL

I see the potential for significant speed improvements vs the direct CALL.

I probably won't have time to work on this myself for the next 1.5 weeks due to travel, so feel free to have at it (as if you need my permission)

Dave Benham

Re: Limit CMD processing to internal commands, safer and fas

Posted: 23 Nov 2011 11:58
by Ed Dyreen
'
Thanks for the input guys, I'll definitely look into it :)

Re: Limit CMD processing to internal commands, safer and fas

Posted: 24 Nov 2011 14:24
by jeb
OJBakker wrote:Quote:
I was wondering if it also has effect on performance.

Met too, but I have not yet done timing-tests with and without these settings.

I do some time tests, and I'm impressed :o
Disabling the path speeds up extremly :) :!:

Code: Select all

  780ms - set var=5
33440ms - call set var=5
 2550ms - call set var=5 / Path empty


Code: Select all

@echo off
setlocal
call :LoadMacros
setlocal EnableDelayedExpansion
set "start=%time%"
for /L %%n in (1,1,10000) do (
   set var=5
)
set "end=%time%"
%$timediff% start,end,result
set "result=  %result%"
echo %result:~-5%ms - set var=5

set "start=%time%"
for /L %%n in (1,1,10000) do (
   call set var=5
)
set "end=%time%"
%$timediff% start,end,result
set "result=  %result%"
echo %result:~-5%ms - call set var=5

set "start=%time%"
set path=
for /L %%n in (1,1,10000) do (
   call set var=5
)
set "end=%time%"
%$timediff% start,end,result
set "result=  %result%"
echo %result:~-5%ms - call set var=5 / Path empty
exit /b


OJBakker wrote:Almost, just disabling path is not enough.
FileSearch will still be done in the current directory.
That is why the FileExt has to be disabled as well.

I believed this too, but it's false :( or I make something wrong

Code: Select all

setlocal
echo echo ######>set.bat
set "Path="
set "PathExt="
call set var=5

It outputs ###### :?

So this solution doesn't work as expected :(

jeb

Re: Limit CMD processing to internal commands, safer and fas

Posted: 24 Nov 2011 16:49
by aGerman
Interesting attempt :)

@jeb
For some reason it works by creating a "false" PATHEXT environment.

Code: Select all

set "PathExt=$"


Regards
aGerman

Re: Limit CMD processing to internal commands, safer and fas

Posted: 25 Nov 2011 02:33
by jeb
Grrrr ... :roll:

I didn't see it before, but OJBaker has described exactly this problem before.
OJBakker wrote:rem limit (d)path(ext) NOTE: pathext must be defined or call will fallback to cmd internal default)


Sometimes reading is useful :oops:

jeb

Re: Limit CMD processing to internal commands, safer and fas

Posted: 25 Nov 2011 02:39
by OJBakker
These are not the same:

Code: Select all

set "PathExt="
set "PathExt=;"

The first deletes the PathExt from environment.
CMD falls back to internal default:.COM;.EXE;.BAT;.CMD;.VBS;.JS;.WS;.MSC
The second set PathExt to: No extensions to search for.

I have tested some more just to be sure:
Disabling PathExt is enough to safeguard "call set" etc commands.
Also disabling Path is necessary to complete disable the FileSearch and speed up some commands.
I can confirm call set and call echo are mucht faster with FileSearch disabled.

Re: Limit CMD processing to internal commands, safer and fas

Posted: 25 Nov 2011 08:38
by dbenham
Both $ and ; can be a valid directory name.

To be absolutely safe, PATHEXT should be set to an illegal path character.

Edit: Let me try that again.

Both $ and ; can be a valid extension.

To be absolutely safe, PATHEXT should be set to an illegal extension character.
I propose

Code: Select all

set PATHEXT=:

Edit 2 - Umm
Since every extension must start with a dot, I suppose the $ works, and ; is the delimiter, so it probably works

I'll just crawl back in my hole now. :roll:

Re: Limit CMD processing to internal commands, safer and fas

Posted: 25 Nov 2011 18:35
by dbenham
I found time to work on this after all.

Here is an optimized macro to implement the safe call.

Code: Select all

@echo off
set lf=^


set ^"\n=^^^%LF%%LF%^%LF%%LF%^^"

set macro.icall=for /l %%# in (1 1 2) do if %%#==2 (%\n%
  for /f "tokens=1-4 eol=| delims=|" %%A in (""!macro_args!"|"!path!"|"!dpath!"|"!pathext!"") do (%\n%
    endlocal%\n%
    set "path="%\n%
    set "dpath="%\n%
    set "pathext=;"%\n%
    call %%~A%\n%
    set "path=%%~B"%\n%
    set "dpath=%%~C"%\n%
    set "pathext=%%~D"%\n%
)) else setlocal enableDelayedExpansion^&set macro_args=

::load timer macro
call macrolib_time

set "var1=hello world"
set "var2=var1"

set n=10000

set t1=%time%
for /l %%n in (1 1 %n%) do set var2=%var1%
set t2=%time%
for /l %%n in (1 1 %n%) do %macro.icall% set var2=%%var1%%
set t3=%time%
for /l %%n in (1 1 %n%) do call set var2=%%var1%%
set t4=%time%

setlocal
set path=
set dpath=
set pathext=;
set t5=%time%
for /l %%n in (1 1 %n%) do call set var2=%%var1%%
set t6=%time%

%macro.diffTime% t1 t2 time.normal
%macro.diffTime% t2 t3 time.icall
%macro.diffTime% t3 t4 time.call
%macro.diffTime% t5 t6 time.safeCall

echo  normal:      %time.normal%
echo  safe call:   %time.safeCall%
echo  macro icall: %time.icall%
echo  normal call: %time.call%

endlocal

The icall macro should be safe to use as long as neither PATH, DPATH, nor PATHEXT contain a pipe (|) character. This should not be a problem. I originally wrote a macro without the | restriction using 3 additional FOR loops, but it was 10% slower.

Also the icall macro should not be used if Delayed Expansion is enabled. This limitation could be removed, but I don't see any reason to ever use icall if Delayed Expansion is enabled, so I didn't think it was worth the extra code.

My timings have 4 tests:
normal = just using SET directly without any CALL
safe call = using CALL after PATH etc. has already been disabled, and without resetting PATH etc.
macro icall = optimized macro to disable PATH etc, execute CALL, then re-enable PATH etc.
normal call = an unsafe CALL with PATH etc. set normally

The timing results are a little disappointing on my Vista machine:

Code: Select all

normal:      00:00:00.65
safe call:   00:00:02.32
macro icall: 00:00:12.70
normal call: 00:00:10.89
The safe icall macro is actually 17% slower than the normal call. But a least it is safe.

The timings are better on my XP machine:

Code: Select all

normal:      00:00:00.59
safe call:   00:00:12.29
macro icall: 00:00:38.11
normal call: 00:01:00.08
Now the icall macro is 33% faster than a normal call.

Finally I re-ran the test on XP, except I added a network path to the PATH before running.

Code: Select all

normal:      00:00:00.59
safe call:   00:00:12.33
macro icall: 00:00:38.34
normal call: 00:03:17.00
Now there is a huge benefit to using the macro vs using a normal call.


Dave Benham