Asynchronous native batch tee script

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Message
Author
dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Asynchronous native batch tee script

#1 Post by dbenham » 17 Feb 2014 18:07

Note - the code in this initial post has been superceded by code at viewtopic.php?p=32615#p32615

I've managed to write a non-blocking asynchronous batch implementation of tee that uses only internal batch commands plus the FINDSTR native external command.

It writes all output to a temporary file in the %TEMP% folder, and simultaneously reads from that same file using SET /P. The temp file name incorporates the current time to 0.1 seconds, and the script automatically loops back to try again if another process happens to grab the same temp name.

The use of SET /P results in the following limitations:

1) Line lengths are limited to 1021 bytes, minus the line number + colonn prefix that is temporarily added to each line.

2) Trailing control characters are stripped from each line.

batchTee.bat

Code: Select all

::batchTee.bat  OutputFile  [+]
::
::  Write each line of stdin to both stdout and outputFile.
::  The default behavior is to overwrite any existing outputFile.
::  If the 2nd argument is + then the content is appended to any existing
::  outputFile.
::
::  Lines are limited to ~1000 bytes, and trailing control characters will
::  be stripped from each line of output.
::
::  The exact maximum line length varies depending on the line number.
::  The SET /P command is limited to reading 1021 byte lines, and each line
::  is prefixed with the line number and a colon when it is read.
::

@echo off
if "%~1" equ ":tee" goto :tee

setlocal disableDelayedExpansion
:lock
set "teeTemp=%temp%\tee%time::=_%"
2>nul (
  9>"%teeTemp%.lock" (
    (findstr /n "^"&echo END) >"%teeTemp%.tmp" | <"%teeTemp%.tmp" "%~f0" :tee %*
    (call )
  ) || goto :lock
)
del "%teeTemp%.lock" "%teeTemp%.tmp"
exit /b

:tee
setlocal enableDelayedExpansion
set "redirect=>"
if "%~3" equ "+" set "redirect=>>"
8%redirect% %2 (
  for /l %%. in () do (
    set "ln="
    set /p "ln="
    if defined ln (
      if "!ln:~0,3!" equ "END" exit
      set "ln=!ln:*:=!"
      (echo(!ln!)
      (echo(!ln!)>&8
    )
  )
)

Usage is as expected for a tee utility:

Code: Select all

dir | batchTee output.txt
The above will overwrite any existing output.txt.

Content can be appended to an existing file by adding + as a second argument:

Code: Select all

dir | batchTee output.txt +


Dave Benham
Last edited by dbenham on 20 Feb 2014 08:38, edited 3 times in total.

foxidrive
Expert
Posts: 6031
Joined: 10 Feb 2012 02:20

Re: Asynchronous native batch tee script

#2 Post by foxidrive » 17 Feb 2014 18:20

Another keeper! Me likey. :)

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: Asynchronous native batch tee script

#3 Post by dbenham » 17 Feb 2014 19:42

Damn :evil:

This works great on Win7, but when I tested on my Virtual XP machine, batchTee.bat was blocking - I got no screen output until the left side was complete.

Can someone verify this behavior with a real XP machine?


Dave Benham

foxidrive
Expert
Posts: 6031
Joined: 10 Feb 2012 02:20

Re: Asynchronous native batch tee script

#4 Post by foxidrive » 17 Feb 2014 20:24

It works ok in my XP Pro VM using VirtualBox, and in Windows 8.1

Liviu
Expert
Posts: 470
Joined: 13 Jan 2012 21:24

Re: Asynchronous native batch tee script

#5 Post by Liviu » 17 Feb 2014 20:55

dbenham wrote:when I tested on my Virtual XP machine, batchTee.bat was blocking - I got no screen output until the left side was complete.

Works fine (asynchronously) on a real XP machine here. While testing it on a really long "dir /s" however, one thing I noticed is that there seems to be no way to cleanly stop it. Ctrl-C or Ctrl-Break register but don't actually work under either XP or Win7. I don't have an immediate clue as to why, even less if it can be worked around in pure batch+findstr.

Liviu

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: Asynchronous native batch tee script

#6 Post by dbenham » 17 Feb 2014 22:30

Thanks Liviu and foxidrive.

That Ctrl-C problem is nasty :( I managed to lock my console up in a perpetual prompt asking if I wanted to terminate the script, even though all output appeared to be complete.

I've diagnosed my apparent blocking problem that I observed on my virtual XP. The problem actually exists on Win7 as well, and it all stems from FINDSTR behavior. It appears to use a buffered stream for output, and it does not flush the buffer until it is full and/or when the command ends. At least that is my interpretation of the following results.

test.bat

Code: Select all

@echo off
for /l %%N in (1 1 1000) do echo %%N
pause
echo Done

And here are the results when I pipe the above to batchTee:

Code: Select all

C:\test> test.bat | batchTee.bat test.txt
1
2
3
4
. . .    <etc.>
930
931
932
933
934
93       <the screen pauses now in the middle of 935, waiting for input>
935      <note that the 935 line is repeated but with the full value this time
936
. . .    <etc.>
999
1000
Press any key to continue . . .
Done

C:\test>
The test.txt file has the exact same content, with the unwanted partial line of 93 followed by the full 935 line.

Things are much better if I replace FINDSTR "^" with MORE within the batchTee.bat:

Code: Select all

    (more&echo END) >"%teeTemp%.tmp" | <"%teeTemp%.tmp" "%~f0" :tee %*

Now everything works as expected, even on my Virtual XP. The pause does not happen until after all 1000 lines appear on the screen. Unfortunately the PAUSE prompt is not displayed until after the user presses a key.

There are two additional limitations beyond what I listed previously:

1) Lines will not be printed to screen until the newline is issued (or input is exhausted). This is undesirable with something like PAUSE because the prompt does not appear until after the user presses a key. I don't see how this can be avoided using native commands. :(

2) MORE translates tab characters into spaces.

I also seem to remember that MORE has some issue where at some point MORE may pause and wait for user input, even though the output is piped. I seem to remember it had something to do with when ~64k bytes appears on one line? Does anyone remember any details about such an issue?

Also, changing to MORE does not fix the nasty Ctrl-C issue.


Dave Benham

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: Asynchronous native batch tee script

#7 Post by dbenham » 17 Feb 2014 23:38

Arghh....

MORE is no good because then I lose my line number prefix that protects empty lines.

But there is a solution :) - FIND /c /v ""

This preserves the line number prefixes, preserves tabs, and doesn't suffer the buffering issue that FINDSTR has.

Perhaps this is my final version :wink:

Code: Select all

::batchTee.bat  OutputFile  [+]
::
::  Write each line of stdin to both stdout and outputFile.
::  The default behavior is to overwrite any existing outputFile.
::  If the 2nd argument is + then the content is appended to any existing
::  outputFile.
::
::  Limitations:
::
::  1) Lines are limited to ~1000 bytes. The exact maximum line length varies
::     depending on the line number. The SET /P command is limited to reading
::     1021 bytes per line, and each line is prefixed with the line number when
::     it is read.
::
::  2) Trailing control characters are stripped from each line.
::
::  3) Lines will not appear on the console until a newline is issued, or
::     when the input is exhaused. This can be a problem if the left side of
::     the pipe issues a prompt and then waits for user input on the same line.
::     The prompt will not appear until after the input is provided.
::
::  4) Attempting to abort the piped commands will lock up the console. Ouch!
::

@echo off
if "%~1" equ ":tee" goto :tee

setlocal disableDelayedExpansion
:lock
set "teeTemp=%temp%\tee%time::=_%"
2>nul (
  9>"%teeTemp%.lock" (
    (find /n /v ""&echo END) >"%teeTemp%.tmp" | <"%teeTemp%.tmp" "%~f0" :tee %*
    (call )
  ) || goto :lock
)
del "%teeTemp%.lock" "%teeTemp%.tmp"
exit /b

:tee
setlocal enableDelayedExpansion
set "redirect=>"
if "%~3" equ "+" set "redirect=>>"
8%redirect% %2 (
  for /l %%. in () do (
    set "ln="
    set /p "ln="
    if defined ln (
      if "!ln:~0,3!" equ "END" exit
      set "ln=!ln:*]=!"
      (echo(!ln!)
      (echo(!ln!)>&8
    )
  )
)


Dave Benham

foxidrive
Expert
Posts: 6031
Joined: 10 Feb 2012 02:20

Re: Asynchronous native batch tee script

#8 Post by foxidrive » 18 Feb 2014 06:38

More pauses after every 64K lines for a keypress.

I'm not really fussed about the control c issue, as if I'm going to control C then I can just as easily close the cmd window.

In testing the find version, it also exhibits the control c issue.

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: Asynchronous native batch tee script

#9 Post by dbenham » 18 Feb 2014 07:10

foxidrive wrote:More pauses after every 64K lines for a keypress.

Thanks - I knew there was something to do with 64K.

All the more reason why FIND is better than MORE for batchTee.


Dave Benham

jeb
Expert
Posts: 1055
Joined: 30 Aug 2007 08:05
Location: Germany, Bochum

Re: Asynchronous native batch tee script

#10 Post by jeb » 19 Feb 2014 09:39

dbenham wrote:That Ctrl-C problem is nasty :( I managed to lock my console up in a perpetual prompt asking if I wanted to terminate the script, even though all output appeared to be complete.


As mentioned at SO, the Ctrl-C issue can be solved with an additional echo Y, so you get the line

Code: Select all

(find /n /v ""&echo END& echo Y & echo J) >"%teeTemp%.tmp" | <"%teeTemp%.tmp" "%~f0" :tee %*


When CTRL-C is pressed the remaining content will directed to the ctrl-c prompt instead of redirected to the tee process.
But as all lines are preceded by the line number the question can't be answered.
But now the `Y` or for german systems the `J` will break the batch file

jeb

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: Asynchronous native batch tee script

#11 Post by dbenham » 19 Feb 2014 19:03

Intriguing idea jeb. It shows some promise, but there are a few oddities I am trying to work out. It works great sometimes, or not at all others, depending on the command(s) on the left side of the pipe. Very odd.

In the mean time, I've posted a question asking how to determine the Yes string in a locale independent manner. I'm asking for confirmation of a potential solution at that link.


Dave Benham

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: Asynchronous native batch tee script

#12 Post by dbenham » 19 Feb 2014 21:00

Here is my latest version that incorporates jeb's trick to handle Ctrl-C. The Ctrl-C aborts both sides of the pipe cleanly in most of my tests. But the one thing that always locks is if Ctrl-C is pressed while the left side is executing a DIR command. It doesn't matter if DIR is within a batch script, or directly on the command line. Very weird :?

Many other scenarios without DIR tested just fine with Ctrl-C, including a batch script, a series of commands on the command line, and a JScript excecuted via CSCRIPT.

I also modified batchTee.bat to print an error message followed by the content if it is unable to open the output file.

Code: Select all

::batchTee.bat  OutputFile  [+]
::
::  Write each line of stdin to both stdout and outputFile.
::  The default behavior is to overwrite any existing outputFile.
::  If the 2nd argument is + then the content is appended to any existing
::  outputFile.
::
::  Limitations:
::
::  1) Lines are limited to ~1000 bytes. The exact maximum line length varies
::     depending on the line number. The SET /P command is limited to reading
::     1021 bytes per line, and each line is prefixed with the line number when
::     it is read.
::
::  2) Trailing control characters are stripped from each line.
::
::  3) Lines will not appear on the console until a newline is issued, or
::     when the input is exhaused. This can be a problem if the left side of
::     the pipe issues a prompt and then waits for user input on the same line.
::     The prompt will not appear until after the input is provided.
::
::  4) Aborting the piped processes by pressing <Ctrl-C> may lock the console,
::     depending on what command(s) are on the left side of the pipe. Most
::     commands seem to work, but the DIR /S command causes problems for example.
::
 
@echo off
setlocal enableDelayedExpansion
if "%~1" equ ":tee" goto :tee
 
:lock
set "teeTemp=%temp%\tee%time::=_%"
2>nul (
  9>"%teeTemp%.lock" (
    set "yes="
    copy /y nul "%teeTemp%.test" >nul
    for /f "tokens=2 delims=(/" %%A in (
      '^<nul copy /-y nul "%teeTemp%.test"'
    ) do if not defined yes set "yes=%%A"
    (find /n /v ""&echo :END&echo !yes!) >"%teeTemp%.tmp" | <"%teeTemp%.tmp" "%~f0" :tee %*
    (call )
  ) || goto :lock
)
del "%teeTemp%.lock" "%teeTemp%.tmp" "%teeTemp%.test"
exit /b
 
:tee
set "redirect=>"
if "%~3" equ "+" set "redirect=>>"
8%redirect% %2 (call :tee2)
set "redirect="
echo ERROR: %~nx0 unable to open %2
 
:tee2
for /l %%. in () do (
  set "ln="
  set /p "ln="
  if defined ln (
    if "!ln:~0,4!" equ ":END" exit
    set "ln=!ln:*]=!"
    (echo(!ln!)
    if defined redirect (echo(!ln!)>&8
  )
)

I'm still waiting to hear on my other post whether my technique for determining the Yes string works on non-English machines.


Dave Benham

jeb
Expert
Posts: 1055
Joined: 30 Aug 2007 08:05
Location: Germany, Bochum

Re: Asynchronous native batch tee script

#13 Post by jeb » 20 Feb 2014 01:47

I can't reproduce your problem with DIR /s.

On my system (W7 x64) it breaks as expected.

What do you see at the end of the break list?

Batchvorgang abbrechen (J/N)? 718: Anzahl der angezeigten Dateien:
Batchvorgang abbrechen (J/N)? 719: 512 Datei(en), 54.823.605 Bytes
Batchvorgang abbrechen (J/N)? 720: 86 Verzeichnis(se), 81.675.169.792 Bytes frei
Batchvorgang abbrechen (J/N)? END
Batchvorgang abbrechen (J/N)? Ja


jeb

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: Asynchronous native batch tee script

#14 Post by dbenham » 20 Feb 2014 08:34

Thanks jeb. Seeing your output and comparing it to mine enabled me to diagnose the problem.

It looks like you are using an older version of batchTee that uses FINDSTR, and it must not be passing Ja via delayed expansion. My most recent code gives the following at the tail end of an aborted DIR /S:

Code: Select all

Terminate batch job (Y/N)? [1896]             479 File(s)            339 bytes
Terminate batch job (Y/N)? [1897]
Terminate batch job (Y/N)? [1898]     Total Files Listed:
Terminate batch job (Y/N)? [1899]            1524 File(s)     28,991,426 bytes
Terminate batch job (Y/N)? [1900]             158 Dir(s)  91,182,809,088 bytes free
Terminate batch job (Y/N)? :END
Terminate batch job (Y/N)? !yes!
Terminate batch job (Y/N)?

The delayed expansion is somehow disabled when the command is aborted with Ctrl-C. I can fix that by passing the yes value via a FOR variable so that the literal value is supplied earlier.

Also, it is odd that the raw output appears after each Terminate prompt. Most of my tests simply break right away with only a single Terminate message, and no raw output. Perhaps the DIR command buffers a bunch of output very quickly, and none of my non DIR tests were fast enough to show the issue.

I've eliminated the unwanted cascade of terminate messages by saving the stdin definition in an unused file handle, and then redirecting stdin to nul. All my batchTee output is then through non-standard file handles 7 for console and 8 for the output file. The only downside is there is no indication of early termination on the console when the command is aborted with Ctrl-C.

I also protected the Yes string determination against non-standard TEMP locations that may have ) or / in the path by using PUSHD.

Code: Select all

::batchTee.bat  OutputFile  [+]
::
::  Write each line of stdin to both stdout and outputFile.
::  The default behavior is to overwrite any existing outputFile.
::  If the 2nd argument is + then the content is appended to any existing
::  outputFile.
::
::  Limitations:
::
::  1) Lines are limited to ~1000 bytes. The exact maximum line length varies
::     depending on the line number. The SET /P command is limited to reading
::     1021 bytes per line, and each line is prefixed with the line number when
::     it is read.
::
::  2) Trailing control characters are stripped from each line.
::
::  3) Lines will not appear on the console until a newline is issued, or
::     when the input is exhaused. This can be a problem if the left side of
::     the pipe issues a prompt and then waits for user input on the same line.
::     The prompt will not appear until after the input is provided.
::

@echo off
setlocal enableDelayedExpansion
if "%~1" equ ":tee" goto :tee

:lock
set "teeTemp=%temp%\tee%time::=_%"
2>nul (
  9>"%teeTemp%.lock" (
    for %%F in ("%teeTemp%.test") do (
      set "yes="
      pushd "%temp%"
      copy /y nul "%%~nxF" >nul
      for /f "tokens=2 delims=(/" %%A in (
        '^<nul copy /-y nul "%%~nxF"'
      ) do if not defined yes set "yes=%%A"
      popd
    )
    for /f %%A in ("!yes!") do (
        find /n /v ""
         echo :END
         echo %%A
      ) >"%teeTemp%.tmp" | <"%teeTemp%.tmp" "%~f0" :tee %* 7>&1 >nul
    (call )
  ) || goto :lock
)
del "%teeTemp%.lock" "%teeTemp%.tmp" "%teeTemp%.test"
exit /b

:tee
set "redirect=>"
if "%~3" equ "+" set "redirect=>>"
8%redirect% %2 (call :tee2)
set "redirect="
(echo ERROR: %~nx0 unable to open %2)>&7

:tee2
for /l %%. in () do (
  set "ln="
  set /p "ln="
  if defined ln (
    if "!ln:~0,4!" equ ":END" exit
    set "ln=!ln:*]=!"
    (echo(!ln!)>&7
    if defined redirect (echo(!ln!)>&8
  )
)


Dave Benham

jeb
Expert
Posts: 1055
Joined: 30 Aug 2007 08:05
Location: Germany, Bochum

Re: Asynchronous native batch tee script

#15 Post by jeb » 20 Feb 2014 09:13

dbenham wrote:The delayed expansion is somehow disabled when the command is aborted with Ctrl-C.


It has nothing to do with the Ctrl-C, but you could read this article :wink: Why does delayed expansion fail when inside a piped block of code?

jeb

Post Reply