Mystery - Why does START lock redirect files?

[dbenham]

I was alerted to this weird behavior on StackOverflow: Issue with output redirection in batch

Whenever you START a new cmd.exe process, it establishes its own IO streams. But for some unknown reason, it also locks any redirect files that may exist in the parent process.

Here is a simple TEST.BAT that demonstrates. It STARTs a new cmd.exe process, and then pauses both processes. The PAUSE output appears on the new console screen, so we know it is not inheriting the IO streams of the main process.

Code: Select all

@echo off
cls
echo begin >LOG1.TXT
echo begin >LOG2.TXT
3>>LOG2.TXT >>LOG1.TXT (
  echo before start
  echo before start >&3
  start "" cmd /c "pause"
  echo after start
  echo after start >&3
  pause >con
)

:: Both of the followng redirections fail if the STARTed process is still active
echo end >>LOG1.TXT
echo end >>LOG2.TXT

If the main process is allowed to continue while the STARTed process is still paused, then the two redirections at the end fail

The main process has no problem writing to the already opened log files within the redirected block. But the two redirections at the end fail to reopen the log files.

What the hell is START doing


Dave Benham

[foxidrive]

Are the files being locked to protect them from being corrupted by another process?

[dbenham]

Why should it care? The STARTed process does not have any logical dependency on the log file redirections of the parent process.


Dave Benham

[einstein1969]

Maybe it has nothing to do but START also has this strange behavior in the parameter passing

with /B /wait:

Code: Select all

@echo off
cls
echo begin >LOG1.TXT
echo begin >LOG2.TXT
3>>LOG2.TXT >>LOG1.TXT (
  rem echo before start
  rem echo before start >&3
  start ""  /B /wait cmd /c "pause"
  rem echo after start
  rem echo after start >&3
  rem pause >con
  echo Block Exiting... >con
)

:: Both of the followng redirections fail if the STARTed process is still active
echo end >>LOG1.TXT
echo end >>LOG2.TXT

result:

Code: Select all

Block Exiting...

with /wait /B:

Code: Select all

@echo off
cls
echo begin >LOG1.TXT
echo begin >LOG2.TXT
3>>LOG2.TXT >>LOG1.TXT (
  rem echo before start
  rem echo before start >&3
  start "" /wait /B cmd /c "pause"
  rem echo after start
  rem echo after start >&3
  rem pause >con
  echo Block Exiting... >con
)

:: Both of the followng redirections fail if the STARTed process is still active
echo end >>LOG1.TXT
echo end >>LOG2.TXT

result:

Code: Select all

Block Exiting...
Impossibile accedere al file. Il file è utilizzato da un altro processo.
Impossibile accedere al file. Il file è utilizzato da un altro processo.

Einstein1969

[penpen]

Why should it care? The STARTed process does not have any logical dependency on the log file redirections of the parent process.

If you use the tool "Process Monitor" you may see what is causing this:
- start processMonitor
- filter process only "cmd.exe" events
- set the monitored actity "Show File System Activity" and "Show Process and Thread Activity"
- start the batch file
- enter return to finish the main process
- clear all actual monitored events
- press enter in the sub-process

now: you see something like that:

Code: Select all

Time of Day         Process Name    PID     Operation       Path                                                                                                    Result
21:43:37,3442869    cmd.exe         2680    Thread Exit                                                                                                             SUCCESS
21:43:37,3448060    cmd.exe         2680    Process Exit                                                                                                            SUCCESS
21:43:37,3450043    cmd.exe         2680    CloseFile       C:\Temp\batch\LOG1.TXT                                                                               SUCCESS
21:43:37,3452857    cmd.exe         2680    CloseFile       C:\Temp\batch\LOG2.TXT                                                                               SUCCESS
21:43:37,3454821    cmd.exe         2680    CloseFile       C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.2600.6028_x-ww_61e65202    SUCCESS

Now it is clear, that start generates a process, streams are mapped to the parents one,
and then it starts a thread (cmd.exe) with its own IO streams.

So the IO streams are locked by the process generated by start, as it inherits the streams.

Maybe it has nothing to do but START also has this strange behavior in the parameter passing

with /B /wait:
(...)

with /wait /B:
(...)

I only can explain that under XP home, as i cannot check it for other systems (just add an "echo abc >con" prior to the main pause):
- "/B /wait"
The child process is active (no output, but pause awaits return);
after pressing return, the child process ends,
the main process reaches its own pause,
...

- "/wait /B"
The child process is not active (output of the added echo, and the main pause can be seen);
after pressing enter it try to access the locked files, so 2 errors.
The child process remains open, until you have entered exit in the shell;
then childs pause awaits the return, and after ENTER the shell closes.
You are also closing the child process, when you close the shell with the X button.

But i don't know why they have desigend it to act different on different sort of the same parameters.

penpen

[dbenham]

Now it is clear, that start generates a process, streams are mapped to the parents one,
and then it starts a thread (cmd.exe) with its own IO streams.

So the IO streams are locked by the process generated by start, as it inherits the streams.

Yes, it was fairly obvious to me that the new process was opening the redirect files with an exclusive lock ("inheriting the streams"). However, it is nice to see some harder evidence.

But that does not explain why. I can't see any logical reason for the new process to open the redirect files when it will never use them. In fact, I am unable to access the files in any way from within the new process. The standard IO streams 1-2 are all pointing to the console, and non-standard streams 3-9 are all undefined.

It still makes no sense


Dave Benham

[penpen]

I think it is because start cannot predict how it should be used prior running, and this uses the parent processes streams:

Code: Select all

::(...)
9>LOG9.TXT 8>LOG8.TXT 7>LOG7.TXT 6>LOG6.TXT 5>LOG5.TXT 4>LOG4.TXT 3>LOG3.TXT 2>>LOG2.TXT >>LOG1.TXT (
::   (...)
   start ""  /B /wait cmd /V:ON /c "(>&9 echo start STREAM9)&(>&8 echo start STREAM8)&(>&7 echo start STREAM7)&(>&6 echo start STREAM6)&(>&5 echo start STREAM5)&(>&4 echo start STREAM4)&(>&3 echo start STREAM3)&(>&2 echo start STDERR)&(echo start STDOUT)&(>con set /P "INPUT=INPUT: ")&(>con echo !INPUT!)"
::   (...)
)
::   (...)

Although it seems curious, that only streams con, 0-2 seems to be copied, and 3 is redirecting to 1 ("start STREAM3" can be found in "LOG1.txt").
But i assume it has to do with the fact, that (at least on my WinXP home SP3...) the command ">&3 echo abc" per default is outputting to the shell, maybe 3 is the default for the device con (under my WinXP).

penpen

Edit: I've extended the test case, and changed my comment.
Edit2: The streams could be used indirectly, too:

Code: Select all

::(...)
9>LOG9.TXT (8>&9 (7>&8 (6>&7 (5>&6 (4>&5 (>>&4 (
::   (...)
   start ""  /B /wait cmd /V:ON /c "(>&2 echo start STDERR)&(echo start STDOUT)&(>con pause)"
::   (...)
)))))))
::(...)

[dbenham]

Regarding START /B /WAIT vs. START /WAIT /B ...

I can't imagine a reason why START would ever be needed with both the /B and /WAIT options. But the START /WAIT /B form is bugged. It does not wait

This TEST.BAT works properly

Code: Select all

@echo off
start "" /b /wait cmd /c "pause"
echo Done

--OUTPUT--

Code: Select all

C:\>test
Press any key to continue . . .
Done

C:\>

This TEST.BAT fails to wait.

Code: Select all

@echo off
start "" /wait /b cmd /c "pause"
echo Done

--OUTPUT--

Code: Select all

C:\>test
Done

C:\>Press any key to continue . . .
C:\>

==============================================

Although it seems curious, that only streams con, 0-2 seems to be copied, and 3 is redirecting to 1 ("start STREAM3" can be found in "LOG1.txt").
But i assume it has to do with the fact, that (at least on my WinXP home SP3...) the command ">&3 echo abc" per default is outputting to the shell, maybe 3 is the default for the device con (under my WinXP).

I can explain that

The very definition of the /B option says that the new process is supposed to share the same console for stdin, stdout, and stderr. So of course it needs to copy streams 0-2 for the new process.

3 is redirecting to 1 because of the way CMD.EXE implements redirection. Every time a stream is redirected, the original definition is first saved in the first available undefined stream so that the original definition can be restored when redirection is over. Since 3 is normally the first available undefined stream, 1>&3 ends up redirecting stdout to itself. See viewtopic.php?p=14612#p14612 and the post following for more information. Also, Aacini's viewtopic.php?p=29634#p29634 post later in the thread delves into the source of the behavior.

==============================================

I think it is because start cannot predict how it should be used prior running, and this uses the parent processes streams:

That doesn't make sense to me.

Surely START must read the options before it actually creates the process - so it should be able to deduce what streams are needed and act accordingly.

As you have shown, the /B option forces streams 0-2 to be copied. But they are not copied without the /B option. Yet START is mysteriously opening all existing streams in exclusive mode, with or without /B, for no apparent reason.

I would expect that at most streams 0-2 would be copied, and only when the /B option is used. I would expect that all other existing streams would be ignored by START. But obviously that is not what is happening. It is still a mystery in my mind. I'd even go so far as to say it is a serious design flaw.


Dave Benham

[penpen]

@1: It never comes to my mind that this is a bug, although now it seems obvious... .
==============================================
@2: Thanks for the explaination: I should have know that, as i already read the thread, but i easily forget how it works... ... .
==============================================
@3:

I think it is because start cannot predict how it should be used prior running, and this uses the parent processes streams:

That doesn't make sense to me.

Surely START must read the options before it actually creates the process - so it should be able to deduce what streams are needed and act accordingly.

(...)

The problem is not the process "start" does manage this: I have really written it unclear.
The batch is running within the "main"-process (or thread).
The command processors reaches the line "start ...", and the "main"-process initiates the the "start"-process: All heandles are copied and passed to the "start process", as it may be that the streams are needed:

Code: Select all

// exceptionally i use C++ style code: It should be more clear on that level.
Process process = gcnew Process;
if (process) {
   process->StartInfo->FileName =  "start";                    // maybe it is a dll-call: handled analogous.
   process->StartInfo->Arguments = "\"\"  /B /wait cmd /c \"pause\"";
   process->StartInfo->UseShellExecute = true;                 // i assume this, but it is not important here
   process->StartInfo->RedirectStandardInput = true;           // don't need to be set explicitely, as it is the default value.
                                                               // (nonstandard handles analogous)
   process->Start();                                           // start copies the inputs, before the "start" process parses its arguments.
}

penpen

Edit: Formatting the C++ source, so the comments are more readable.
Edit2: The process above is not the "start" process it is the new shell window (traced into using debug on Microsoft Visual Studio 6.0)

[dbenham]

]The problem is not the process "start" does manage this: I have really written it unclear.
The batch is running within the "main"-process (or thread).
The command processors reaches the line "start ...", and the "main"-process initiates the the "start"-process: All heandles are copied and passed to the "start process", as it may be that the streams are needed:

Code: Select all

// exceptionally i use C++ style code: It should be more clear on that level.
Process process = gcnew Process;
if (process) {
   process->StartInfo->FileName =  "start";                    // maybe it is a dll-call: handled analogous.
   process->StartInfo->Arguments = "\"\"  /B /wait cmd /c \"pause\"";
   process->StartInfo->UseShellExecute = true;                 // i assume this, but it is not important here
   process->StartInfo->RedirectStandardInput = true;           // don't need to be set explicitely, as it is the default value.
                                                               // (nonstandard handles analogous)
   process->Start();                                           // start copies the inputs, before the "start" process parses its arguments.
}

That still does not make sense to me. START is an internal command, so the process that actually creates the new process is already running. So I don't see any reason for the START command to replicates the streams before it parses the options. Also, even if the START command itself has the files open, and not the newly started process, then the files should be released once the new process is created if START is not waiting for the new process to complete.


Dave Benham

[penpen]

I've gambled around with my Microsoft Visual Studio 6.0 debug... .
I'm very unsure, but it seems that the start command has only few todo with the file lock.

I think the command line processor (CLP) reads single lines/block and detect redirection for this single line/block.
Then the CLP creates a tupel T:=(B:=block, R:= all (file)redirections, ...) and adds it to a list (L).
As long as B is not fully processed, R will lock the associated files.
It seems that B is considered to be not fully processed, until all programs started within this block return.
So if the CLP is processing the next line, another tupel T2(B2,R2,...) is created, and when trying to redirect the (still redirected) files a sharing violation is caused (T still exist and locks the files).

penpen