More fun with redirection and file handles

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Post Reply
Message
Author
dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

More fun with redirection and file handles

#1 Post by dbenham » 17 Feb 2015 21:29

Here is a script that demonstrates various redirection concepts, most of which I have seen posted before.

- The same file can be opened for input and output independently.

- A given file can only be opened once for output due to an exclusive lock

- The same file can be opened multiple times for input. Each input file stream maintains its own file pointer independent of the others.

- Redirection to an existing file handle via something like &1 shares the stream with only 1 file pointer between them.

- FIND resets the file pointer to the beginning of file before scanning.

- FINDSTR does not reset the file pointer (I first saw this in an Aacini post)

- Writing to an input handle fails with a puzzling error message

- Reading from an output handle fails simply returns nothing (fails?), but there is no error message.

Code: Select all

@echo off
0<test.txt 7>test.txt 8<&0 9<test.txt (
  call :read 0
  call :write 1
  call :write 2
  call :write 3
  call :read 0
  call :read 8
  call :read 9
  call :write 4
  call :read 8
  call :read 8
  call :read 0
  echo(
  echo FINDSTR from 9 does not reset file pointer to beginning, and ends at EOF:
  0<&9 findstr "^"
  echo(
  call :read 9
  0<test.txt (
    echo(
    echo Temporarily reopen input file via 0
    call :read 0
    call :read 0
    echo Close and resume old 0
    echo(
  )
  call :write 5
  call :read 0
  call :read 9
  call :read 0
  call :read 8
  call :read 9
  echo(
  echo FINDSTR from 0 does not start at beginning:
  findstr "^"
  echo(
  echo FIND from 0 resets file pointer to beginning, and ends at EOF:
  find /v ""
  echo(
  call :read 0
  echo(
  echo Cannot reopen output file because of exclusive lock
  1>test.txt (
    1>&2 echo This is never executed
  )
  echo(
  echo Cannot read from non-existent file
  0<notExists (
    set /p "ln=This is never executed"
  )
  echo(
  1>&0 (
    1>&2 echo Cannot write to input handle, but redirection worked and error message puzzling
    echo This fails when try to write to input handle
  )
  echo(
  0<&1 set /p "ln=Why does read from output handle not generate an error message?"
  echo(
)
exit /b


:read  handle
setlocal
if "%1" neq "0" set "src=0<&%1"
set "ln="
set /p "ln=" %src%
echo Reading from %1 ==^> %ln%
exit /b

:write  txt
1>&7 echo %*
echo Wrote %*
exit /b

--OUTPUT--

Code: Select all

Reading from 0 ==>
Wrote 1
Wrote 2
Wrote 3
Reading from 0 ==> 1
Reading from 8 ==> 2
Reading from 9 ==> 1
Wrote 4
Reading from 8 ==> 3
Reading from 8 ==> 4
Reading from 0 ==>

FINDSTR from 9 does not reset file pointer to beginning, and ends at EOF:
2
3
4

Reading from 9 ==>

Temporarily reopen input file via 0
Reading from 0 ==> 1
Reading from 0 ==> 2
Close and resume old 0

Wrote 5
Reading from 0 ==> 5
Reading from 9 ==> 5
Reading from 0 ==>
Reading from 8 ==>
Reading from 9 ==>

FINDSTR from 0 does not start at beginning:

FIND from 0 resets file pointer to beginning, and ends at EOF:
1
2
3
4
5

Reading from 0 ==>

Cannot reopen output file because of exclusive lock
The process cannot access the file because it is being used by another process.

Cannot read from non-existent file
The system cannot find the file specified.

Cannot write to input handle, but redirection worked and error message puzzling
The system cannot find the file specified.

Why does read from output handle not generate an error message?


Note that I did not redirect handle 3 after redirecting 0 because of this issue: http://stackoverflow.com/a/9880156/1012053


Dave Benham

Liviu
Expert
Posts: 470
Joined: 13 Jan 2012 21:24

Re: More fun with redirection and file handles

#2 Post by Liviu » 19 Feb 2015 23:33

dbenham wrote:- Reading from an output handle fails simply returns nothing (fails?), but there is no error message.

Code: Select all

  0<&1 set /p "ln=Why does read from output handle not generate an error message?"

Focusing on just this tiniest bit of your whole exercise, but I think line-oriented input assumes an implicit end-of-line at the end-of-stream (whether it's been closed, or never opened at all). Probably the same reason why "<nul pause" works without error, and without waiting for a key - which happens to be a very useful behavior, whether intentional or accidental.

Liviu

penpen
Expert
Posts: 1991
Joined: 23 Jun 2013 06:15
Location: Germany

Re: More fun with redirection and file handles

#3 Post by penpen » 20 Feb 2015 13:06

According to http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/redirection.mspx?mfr=true this happens:
if you start a program with <&2, all attempts to read handle 0 fail because handle 2 is initially opened with write-only access.
I assume the same happens when using STDIN.
So the output of 'set /P "input=output"' is generated, but reading the input fails and the errorlevel is set to != 0 (1 on my WinXP).

Using ">&0" works similar:
if you start a program with redirection >&0, all attempts to write handle 1 fail because handle 0 is initially opened with read-only access.


penpen

Ed Dyreen
Expert
Posts: 1569
Joined: 16 May 2011 08:21
Location: Flanders(Belgium)
Contact:

Re: More fun with redirection and file handles

#4 Post by Ed Dyreen » 09 Apr 2015 16:28

Trying to find a nice elegant way to get this working properly.

Code: Select all

@echo off &setlocal enableDelayedExpansion

echo.
echo.test0
(set/A=)3>&2>&1>nul&&echo.yes||echo.no

echo.
echo.test1
(set/A=)3>nul 2>nul >nul&&echo.yes||echo.no

echo.
echo.test2
(((set/A=)3>&2)2>&1)>nul&&echo.yes||echo.no

echo.
echo.test3
(set/A=)1>&2 2>&3 3>nul&&echo.yes||echo.no

pause
exit

Code: Select all


test0
Er ontbreekt een operand.
no

test1
no

test2
no

test3
Er ontbreekt een operand.
The 1st most logical solution fails !? Both the 2nd & 3th solution work but they don't make me happy, I have a feeling this can be done in a more elegant way... Test3 successfully hangs the script, won't be needing that feature for now :roll:

penpen
Expert
Posts: 1991
Joined: 23 Jun 2013 06:15
Location: Germany

Re: More fun with redirection and file handles

#5 Post by penpen » 11 Apr 2015 09:34

Ed Dyreen wrote:Test3 successfully hangs the script, won't be needing that feature for now :roll:
This should not hang your script, but it should reassign the used streams to the given handles.
Assumed that handles 3++ are not in use this should happen:

Code: Select all

handle   before   after
     0   STDIN    STDIN
     1   STDOUT   nul
     2   STDERR   STDERR
     3   unused   STDOUT
    4+   unused   unused
The default handle to STDOUT is reassigned to nul, so you don't see the shells output.

To proof this, you could delete the "pause" "exit" from your batch file (assumed its name is "test.bat") and then you could start a new command shell instance using ">&3 cmd":

Code: Select all

Z:\>test.bat

test0
Fehlender Operand
no

test1
no

test2
no

test3
Fehlender Operand
>&3 cmd
Microsoft Windows [Version 6.3.9600]
(c) 2013 Microsoft Corporation. Alle Rechte vorbehalten.

Z:\>


To avoid such unwanted reassignments of streams to handles, you should redirect one handle only per instruction, or block.
So you should use the syntax of your third example ("test2"):

Code: Select all

(((set/A=)3>&2)2>&1)>nul&&echo.yes||echo.no
(((set/A=)3>nul)2>nul)>nul&&echo.yes||echo.no
:: ...


penpen

OperatorGK
Posts: 66
Joined: 13 Jan 2015 06:55

Re: More fun with redirection and file handles

#6 Post by OperatorGK » 12 Apr 2015 12:32

Whoa.
Thanks to you, I've just discovered another bug :? ! Try:

Code: Select all

0>&1 echo TEST 3>con

Strangely, it works ONLY in command line, when not in braces, for loop, cmd /c and so on. Can anybody explain me why this is happening?

penpen
Expert
Posts: 1991
Joined: 23 Jun 2013 06:15
Location: Germany

Re: More fun with redirection and file handles

#7 Post by penpen » 13 Apr 2015 10:40

Initially assignemnts (using c++ terminology for STDIN, STDOUT, STDERR; CON analogous):
- CON: (in your case) an OutputStream (write only access).
- STDIN: InputStream (read only access).
- STDOUT: OutputStream (write only access).
- STDERR: OutputStream (write only access).

Your command "0>&1 echo TEST 3>con" probably produces the following mapping (step by step):

Code: Select all

handle mapping   handle    initial   assign             restore
                                     0>&1     3>con     0        3
                      0    STDIN     STDOUT   STDOUT    CON      CON
                      1    STDOUT    STDOUT   STDOUT    STDOUT   STDOUT
                      2    STDERR    STDERR   STDERR    STDERR   STDERR
                      3    unused    STDIN    CON       unused   STDIN
                      4    unused    unused   STDIN     STDIN    unused
                     5+    unused    unused   unused    unused   unused
restore handle             -         0        0,3       3        -
        with               -         3        3,4       4        -
You could check that handle 3 is STDIN using '@(0>&1 echo TEST 3>con & set /P "=test" <&3)' (no endless loop while processing 'set/P').

The command shell uses an endless loop (simplified pseudo code) like this:

Code: Select all

String input;
boolean echoEnabled = true;
while (true) do {
   if (echoEnabled) streamFromHandle(1).write(getPrompt());
   input = streamFromHandle(0).readLine();
   execute(input);
}
After your command streamFromHandle(0) equals CON (in this case an OutputStream).
Reading an OutputStreamit fails, so "input" is set to an empty String, or null (i don't know it).
Then this "input" is executed (obviously nothing is displayed).
Then the next loop iteration is performed.
So you will see the prompt string (probably something like "C:\Foo\>") printed to screen again and again.


penpen

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: More fun with redirection and file handles

#8 Post by dbenham » 19 Apr 2015 12:56

I've neglected this thread far too long after I started it.
penpen wrote:According to http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/redirection.mspx?mfr=true this happens:
if you start a program with <&2, all attempts to read handle 0 fail because handle 2 is initially opened with write-only access.
I assume the same happens when using STDIN.
So the output of 'set /P "input=output"' is generated, but reading the input fails and the errorlevel is set to != 0 (1 on my WinXP).

Liviu wrote:
dbenham wrote:- Reading from an output handle fails simply returns nothing (fails?), but there is no error message.

Code: Select all

  0<&1 set /p "ln=Why does read from output handle not generate an error message?"

Focusing on just this tiniest bit of your whole exercise, but I think line-oriented input assumes an implicit end-of-line at the end-of-stream (whether it's been closed, or never opened at all). Probably the same reason why "<nul pause" works without error, and without waiting for a key - which happens to be a very useful behavior, whether intentional or accidental.

Yes, I noticed the similarity, but it still surprises me a bit. I can see how reading from NUL could be designed to instantly fail without error, but I wouldn't have thought reading from a write only handle would be treated the same way.

Even more surprising, there is no error message if an input file becomes unavailable after it has been opened.

I have my H: drive pointing to a removable thumb drive.

Code: Select all

C:\test>3<&1 <h:test.txt (<&3 pause&set /p val=First time I leave the thumb drive plugged in)
Press any key to continue . . .
First time I leave the thumb drive plugged in
C:\test>echo %errorlevel%
0

C:\test>set val
val=ok

C:\test>set "val="

C:\test>3<&1 <h:test.txt (<&3 pause&set /p val=Now I remove the thumb drive before pressing a key)
Press any key to continue . . .
Now I remove the thumb drive before pressing a key
C:\test>echo %errorlevel%
1

C:\test>set val
Environment variable val not defined

C:\test>

No error message :!:
But at least the error can be detected via %ERRORLEVEL% (or the || operator, tested but not shown)


penpen wrote:Using ">&0" works similar:
if you start a program with redirection >&0, all attempts to write handle 1 fail because handle 0 is initially opened with read-only access.

I fully expect it to fail, but the error message is not at all what I expect: "The system cannot find the file specified."

The redirection has already taken place when I attempt to write to stdin, so it should not be looking for a file. The redirection outside the parentheses was successful, it is only the attempt to write via ECHO that is failing.

And now for a really horrendous discovery - a failed ECHO does not set any error code :!:
Again, I've got H: pointing to a removable thumb drive.

Code: Select all

C:\test>>h:test.txt (echo I remove the thumb drive while pausing&pause >&2&echo This failed ECHO does not set a detectable error code||>&2echo FAILURE!)
Press any key to continue . . .
The system cannot find the file specified.

C:\test>ECHO %errorlevel%
0

After reconnecting the thumb drive, I see that test.txt is empty, even though the first ECHO succeeded. This is probably due to a buffering issue.

But the important thing is that || does not detect the failure to write to the output file, and %ERRORLEVEL% is still 0 :!: :shock:
The weird thing is, CMD.EXE obviously detected the error because it printed out an error message.

This is a horrible bug in my book. The only way to detect a failure when writing output is to redirect stderr to a file and then check if the file is not empty after attempting to write.


Dave Benham

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: More fun with redirection and file handles

#9 Post by dbenham » 19 Apr 2015 13:38

@Ed Dyreen - I'm not sure I understand what your goal was with your convoluted experiments. It seems to me your desired result can be achieved simply using:

Code: Select all

2>nul set/a=&&echo.yes||echo.no


The issues with Ed's test 3 and OperatorGK's code are both related to http://stackoverflow.com/q/9878007/1012053

penpen has the proper analysis, and the behavior is full explained within two posts at viewtopic.php?p=14612#p14612


Dave Benham

Ed Dyreen
Expert
Posts: 1569
Joined: 16 May 2011 08:21
Location: Flanders(Belgium)
Contact:

Re: More fun with redirection and file handles

#10 Post by Ed Dyreen » 19 Apr 2015 16:55

dbenham wrote:@Ed Dyreen - I'm not sure I understand what your goal was with your convoluted experiments. It seems to me your desired result can be achieved simply using:

Code: Select all

2>nul set/a=&&echo.yes||echo.no
I generally use stream 0 for default output, stream 2 for error's, and stream 3 for headers and footers. When a function throws an error it's often suppressed by the calling function. The caller often uses it's own header and footer thus will often suppress stream 3 aswell. Stream 1 is often suppressed, but not always. I have not encountered any problems with '3>nul 2>nul >nul' yet. Was hoping for '3>&2>&1>nul' to work, but it doesn't work as expected.

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: More fun with redirection and file handles

#11 Post by dbenham » 19 Apr 2015 19:37

OK, I understand now. I think you mean you use stream 1 for default output, not 0.

You could also use 3>nul 2>&3 >&3 or 3>nul 2>&3 >&2. The assignments must be done in descending order (or staged as penpen suggested), otherwise the original definitions are not restored properly when the command finishes.

You could never do anything like 3>&2>&1>nul. Each redirection must be specified in full, and the redirections are processed from left to right.



Dave Benham

Post Reply