non-buffering FOR /F alternative

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Post Reply
Message
Author
aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

non-buffering FOR /F alternative

#1 Post by aGerman » 18 Sep 2019 10:34

I developed a little C tool to observe a folder for changes. It is permanently running and writes a line for every event. (Maybe I publish this utility later on if someone finds it useful).
If I try to process the lines in a FOR /F loop it won't work because FOR /F buffers the outputted lines until the utility terminates. Of course the tool is not intended to terminate because it shall continue waiting for the next event. I already figured out how to work around this limitation but still I'm not happy with it.
That's what I have so far:

Code: Select all

@echo off &setlocal
for /f "delims=" %%i in ('cmd /c "echo hello&echo world&set /p =stop"') do echo %%i
pause

cls
cmd /c "echo hello&echo world&set /p =stop" | (cmd /von /q /d /c ^"for /l %%# in (^) do set "rec="^&set /p "rec="^&(if not defined rec exit^)^&^
  echo !rec!^
")
pause

cls
cmd /c "echo hello&>nul timeout 1&echo world&>nul timeout 1&set /p =stop" | (cmd /von /q /d /c ^"for /l %%# in (^) do set "rec="^&set /p "rec="^&(if not defined rec exit^)^&^
  echo !rec!^
")
pause
The code consists of three parts which all process the same output of the three lines
hello
world
stop
Because of the SET /P statement that displays "stop", the cmd process doesn't terminate unless you press Enter.

The first part displays nothing until you press Enter. Then it outputs all three lines at once. This is the unwanted behavior I try to work around.

The second part solves the problem partially. I pipe the output to another cmd process containing an infinite loop, and within the body of this loop I read the lines using SET /P. This technique is well-known for reading files. Fun fact: It displays "hello" and then waits for the Enter input. "world" and "stop" are never displayed. Something ate them :?

In the third part I inserted a delay of one second after each line. This works. And my tests using the C tool indicate that a few milliseconds would be sufficient already. But this is ugly.


Does anyone out there know of a better workaround?

Steffen

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: non-buffering FOR /F alternative

#2 Post by dbenham » 18 Sep 2019 12:01

SET /P does not play nicely with pipes. However it does work as you want with files.

So simply redirect your utility output to a temp file, and read the temp file with SET /P in an infinite loop.

SET /P will return an empty string if there is no data waiting, so you need to ignore empty lines. This also means you will need some other stop signal. I introduced a TIMEOUT to test the empty line handling.

You can still use a pipe to get two asynchronous processes, but the pipe is no longer conducting any data.

Code: Select all

@echo off
copy nul temp.txt >nul
cmd /c "(echo hello&timeout 5 >nul&echo world&set /p =stop<nul)>>temp.txt" | cmd /von /q /d /c "(for /l %%# in () do set "rec="&set /p "rec="&(if defined rec echo(!rec!)&if !rec!==stop exit)<temp.txt"
del temp.txt

Dave Benham

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: non-buffering FOR /F alternative

#3 Post by aGerman » 18 Sep 2019 12:46

This works great. Thanks Dave!

The problem is that the temporary file can become really huge. The C tool I'm talking about is based on a little program that I wrote which is already in use. It may run unattended over weeks or months until someone restarts the computer. It copies local files to a server which a software of a measurement device writes, in order to enable monitoring the measuring results on the shop floor simultaneously. WSH scripts using WMI don't work because you can't observe a subfolder tree using WMI. However, for this particular purpose I wrote the program in a manner which internally processes the events and copies the files without any script involved. Now I thought about writing a tool that could be used in a more generic way.

Steffen

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: non-buffering FOR /F alternative

#4 Post by aGerman » 23 Sep 2019 14:24

Finally I used a 10 ms delay in my tool. This is about three times as long as needed on the machines I tested it, and seems to be sufficient to make SET /P waiting again even if the processing of the output takes much more time than 10 ms.
In case you are interested:
https://sourceforge.net/projects/direvents/

Steffen

Squashman
Expert
Posts: 4465
Joined: 23 Dec 2011 13:59

Re: non-buffering FOR /F alternative

#5 Post by Squashman » 27 Sep 2019 10:00

So is reading the file similar to what you were doing here?

Essentially tailing a file?

Code: Select all

@echo off
call :Loop <"tailme.txt"

exit

:Loop
set "line="
set /p "line="
if defined line (
  echo %line%
) else (
  pathping -q 1 -p 300 localhost >nul
)
goto :loop

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: non-buffering FOR /F alternative

#6 Post by aGerman » 28 Sep 2019 04:43

That's pretty much the same. Unfortunately this leaves the polling for new data in the script code and still buffers in a growing file because already read data is not removed. That's the reason why I hoped to find a way to make SET /P read from the pipe in a proper way.

Steffen

jfl
Posts: 226
Joined: 26 Oct 2012 06:40
Location: Saint Hilaire du Touvet, France
Contact:

Re: non-buffering FOR /F alternative

#7 Post by jfl » 01 Oct 2019 07:49

Hi Steffen

Your DirEvents.exe tool is impressively small, and it works nicely, except for one detail:
How do you stop it?
I tried Enter, ESC, Ctrl-C, and many other key combinations, without success.
And I could not find any clue in the source.
Eventually I had to kill it in Task Manager, which is not very convenient!

Jean-François

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: non-buffering FOR /F alternative

#8 Post by aGerman » 01 Oct 2019 09:34

I think I know the reason but didn't try yet.
I registered the BeforeClosing routine in order to release memory and handles even if you close the window.

Code: Select all

  SetConsoleCtrlHandler(BeforeClosing, TRUE);
I call it actively if some errors occur, elsewise it is a callback function which was sub-classed from the console host callback.
See https://docs.microsoft.com/en-us/window ... lerroutine
It is called for several events including Ctrl+C. In that case I guess it should return FALSE rather than TRUE. I didn't care :wink: but I'll consider to change it in an upcoming version.

Steffen

Post Reply