Event-driven multi-thread scheme for Batch files

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Message
Author
Aacini
Expert
Posts: 1885
Joined: 06 Dec 2011 22:15
Location: México City, México
Contact:

Event-driven multi-thread scheme for Batch files

#1 Post by Aacini » 12 Aug 2015 16:10

This is a method that allows a Batch file to start a series of simultaneous threads as auxiliary processes. The threads normally stay in a wait state, but anyone of they can be activated upon the receipt of a string (the "event") that cause that the intended thread ("event handler") process the event and return to the wait state. This mechanism is assembled via pipelines, that is, all event handlers form a chain of processes connected by pipes, so the output of the first handler pass as input to the second one and so on. When the main Batch file send an event to the chain, each event handler take the event, review it and process it if the event is intended for such handler; otherwise the handler just pass the event to the next handler in the chain. This method allows to have a large number of event handlers and yet be very efficient. Some details can aid to assemble a more efficient chain; for example, place the most frequently called or fastest handlers at beginning of the chain, and the less used or larger ones at end.

In the example below, JScript is used to write the event handlers because they must read the event from a pipe, and SET /P Batch command have problems with this input; if such problems can be compensated, the event handlers can be written in Batch. Each handler use its Stdout to send the event to next handler, so they must use Stderr in order to show any result on the screen. The same point apply to the main Batch file, but in this case the "2>&1 1>&3" trick is used in order to avoid multiple "> CON" redirections in all messages displayed in the screen. In this example the event handler is a simple generic routine, but in a real application each handler may be very different.

Code: Select all

@if (@CodeSection == @Batch) @then 


@echo off
setlocal
if "%~1" equ "Restart" goto Restart

set newThread=CScript //nologo //E:JScript "%~F0"

echo Before start the chain of event handlers
"%~F0" Restart 2>&1 1>&3 | %newThread% 2 | %newThread% 3 | %newThread% 4
echo After the chain of event handlers ended
goto :EOF


:Restart
echo/
echo Press a key to send event to handler 2
pause
echo 2:Event sent to handler TWO>&2
echo/
echo Press a key to send event to handler 4
pause
echo 4:Event sent to handler FOUR>&2
echo/
echo Press a key to send event to handler 5 (unhandled)
pause
echo 5:Event sent to handler FIVE>&2
echo/
echo Press a key to send event to handler 3
pause
echo 3:Event sent to handler THREE>&2
echo/
echo Press a key to terminate
pause
echo/
goto :EOF


@end


var myId = WScript.Arguments(0);
WScript.Stderr.WriteLine();
WScript.Stderr.WriteLine("Event handler #"+myId+" started");

// Wait for an event intended for this handler in order to process it
while ( ! WScript.Stdin.AtEndOfStream ) {
   var event = WScript.Stdin.ReadLine();
   if ( event.substr(0,1) == myId ) {
      WScript.Stderr.WriteLine(' -> Handler '+myId+' responding to event: "'+event+'"');
   } else {
      // Pass event not intended for this handler to next handler in the chain
      WScript.Stdout.WriteLine(event);
   }
}

WScript.Stderr.WriteLine("Event handler #"+myId+" ending");

An useful application of this technique could be a Batch file with three auxiliary processes: Directory, File and Text file. The Batch file send a directory to the first handler that process the nested subdirectories by itself, but send files to the next handler. The File handler do a basic processing on all files, but send text files to the third handle for additional processing. You may even insert many additional handlers to process several different types of files because the addition of such handlers in the chain no waste additional resources. This point represent a very simple way of multi-thread a wide range of applications.

Antonio

Aacini
Expert
Posts: 1885
Joined: 06 Dec 2011 22:15
Location: México City, México
Contact:

Re: Event-driven multi-thread scheme for Batch files

#2 Post by Aacini » 31 Aug 2015 22:33

I added the necessary management to SET /P Batch commands in order to use they in the event handlers of previous example, so the whole multi-thread application can be written in pure Batch. The details about SET /P problems with pipes and the method to solve they are described at Set /P problems with pipes topic.

Below there is an example program that show two different schemes for a multi-thread Batch file application: one with the main code at beginning of the pipe chain, and another one with the main code at end.

Code: Select all

@echo off
setlocal EnableDelayedExpansion

rem Multi-thread schemes in Batch files
rem Antonio Perez Ayala aka Aacini

set "myID=%~2"
if "%~1" neq "" goto %1

rem Define auxiliary variables
rem http://www.dostips.com/forum/viewtopic.php?f=3&t=6134
set LF=^
%Do not remove this line 1/2%
%Do not remove this line 2/2%
for /F %%a in ('copy /Z "%~F0" NUL') do set "CR=%%a"
set "spaces= "
for /L %%i in (1,1,10) do set "spaces=!spaces!!spaces!"

cd /D "%~DP0"

echo/
echo Example A: Main code send commands to a chain of waiting service threads
"%~NX0" MainA 3>&1 1>&2 | "%~NX0" ThreadA 1 | "%~NX0" ThreadA 2 | "%~NX0" ThreadA 3

echo/
echo Example B: Main code receive "completed" signals from active service threads
del request*.txt 2> NUL
(
   start "" /B cmd /C "%~NX0" ThreadB 1
   start "" /B cmd /C "%~NX0" ThreadB 2
) | "%~NX0" MainB

goto :EOF



:MainA
   rem Example A: Main code send commands to a chain of waiting service threads
   rem Use handle #3 to send commands to waiting threads
   rem The "completed" signals may be received via files

   rem The benefit of this scheme is that all threads stay in an efficient wait state
   rem while waiting for requests from the main code

   timeout /T 3 /NOBREAK > NUL
   echo MainA sending a command to ThreadA #1
   set "output=1: Command to ThreadA #1=first%spaces%"
   echo %output:~0,1021%>&3
   timeout /T 3 /NOBREAK > NUL
   echo MainA sending a command to ThreadA #3
   set "output=3: Command to ThreadA #3=third%spaces%"
   echo %output:~0,1021%>&3
   timeout /T 3 /NOBREAK > NUL
   echo MainA sending a command to ThreadA #2
   set "output=2: Command to ThreadA #2=second%spaces%"
   echo %output:~0,1021%>&3
   timeout /T 3 /NOBREAK > NUL
   set "output=exit%spaces%"
   set "output=%output:~0,1018%"
   echo 3: %output%>&3
   echo 2: %output%>&3
   echo 1: %output%>&3
goto :EOF


:ThreadA
   rem Get a command from main code or previous thread
   set /P "command="
   for /F "tokens=1-5" %%a in ("%command%") do (
      if "%%a" equ "%myID%:" (
         rem Command intended for this thread: execute it
         echo ThreadA #%myID%, received command: "%%b %%c %%d %%e" > CON
      ) else (
         rem Pass command to next thread in the chain
         echo %command%
      )
   )
if "%command:~3,4%" neq "exit" goto ThreadA
echo ThreadA #%myID%, terminating... > CON
goto :EOF



:MainB
   rem Example B: Main code receive "completed" signals from active service threads
   rem The requests for active threads may be send via files
   rem The "completed" signals are received via Stdin

   rem The benefit of this scheme is that the main code stay in an efficient wait state
   rem while waiting for the threads "completed" signals

   ping -n 5 localhost > NUL
   echo MainB placing a request for ThreadB #1
   echo Request for ThreadB #1=first > request1.txt
   echo MainB waiting for signal from ThreadB #1
   set /P "signal="
   echo MainB received signal: "%signal%"
   echo exit > request1.txt
   set /P "signal="

   ping -n 5 localhost > NUL
   echo MainB placing a request for ThreadB #2
   echo Request for ThreadB #2=second > request2.txt
   echo MainB waiting for signal from ThreadB #2
   set /P "signal="
   echo MainB received signal: "%signal%"
   echo exit > request2.txt
   set /P "signal="

goto :EOF


:ThreadB
   rem Wait until a request appear
   rem a delay command (ping or timeout) may be included here
   if not exist request%myID%.txt goto ThreadB
   set /P "request=" < request%myid%.txt
   del request%myID%.txt
   echo ThreadB #%myID%, received request: "%request%" > CON
   ping -n 5 localhost > NUL
   set "output=ThreadB #%myID% completed @%time%!CR!!LF!%spaces%"
   rem Send the "completed" signal via Stdout
   echo %output:~0,1021%
if "%request:~0,4%" neq "exit" goto ThreadB
echo ThreadB #%myID%, terminating... > CON
goto :EOF

Previous schemes have the disadvantage that the other part of the synchronization (the side not controlled by the pipe) must be achieved via the presence of a file or line that works like a flag or semaphore; this flag must be tested in a GOTO assembled loop. A more efficient scheme can be assembled if the whole program start and end in a pipe, that is, the application may be divided in three parts: an Input part, a Thread part (that can be executed several concurrent times, one instance by each CPU processor core) and an Output part, all of they connected in a long pipeline. This scheme is very efficient, because the synchronization between all parts/threads is achieved by the operating system itself, not via a test executed in a loop.

Code: Select all

@echo off
setlocal EnableDelayedExpansion

rem Multi-thread schemes in Batch files
rem Antonio Perez Ayala aka Aacini

set "myID=%~2"
if "%~1" neq "" goto %1

rem Define auxiliary variables
rem http://www.dostips.com/forum/viewtopic.php?f=3&t=6134
set LF=^
%Do not remove this line 1/2%
%Do not remove this line 2/2%
for /F %%a in ('copy /Z "%~F0" NUL') do set "CR=%%a"
set "spaces= "
for /L %%i in (1,1,10) do set "spaces=!spaces!!spaces!"

rem Example C:
rem 1- The Input module get input from the user, that specify the task to perform
rem    and distribute it between the concurrent threads.
rem 2- Each concurrent thread perform a part of the task and generate a partial result.
rem 3- The Output module group partial results from all threads
rem    and output the complete result.

set "TRACE=>CON ECHO"
rem To activate trace, remove the next line
set "TRACE=REM"

rem Create the chain of threads linked by pipes
set "numThreads=%NUMBER_OF_PROCESSORS%"
set "threads="
for /L %%i in (%numThreads%,-1,1) do set "threads=!threads! | "%~NX0" Thread %%i"

echo Start of application
cd /D "%~DP0"
"%~NX0" Input | %threads:~3% | "%~NX0" Output
echo End of application
goto :EOF


:Input
%TRACE% INPUT:  Start
echo/> CON
set /P "=Number of lines to generate (Enter to end): " > CON < NUL
:nextInput

   rem Get next task from the user
   set "number="
   set /P "number=" > CON
   if not defined number goto endInput

   rem Divide the task by the number of threads
   set /A chunk=number/numThreads, last=0
   for /L %%i in (1,1,%numThreads%) do (
      set /A "from%%i=last+1, to%%i=(last+=chunk)"
   )
   set "to%numThreads%=%number%"

   rem Distribute the task between all threads
   for /L %%i in (1,1,%numThreads%) do (
      %TRACE% INPUT:  Send "%%i !from%%i! !to%%i!"
      set "output=%%i !from%%i! !to%%i!!CR!!LF!%spaces%"
      echo !output:~0,1021!
   )

goto nextInput
:endInput

rem Terminate the whole chain of threads (and Output)
%TRACE% INPUT:  Send "exit"
set "output=exit!CR!!LF!%spaces%"
echo %output:~0,1021%
%TRACE% INPUT:  End
goto :EOF


:Thread
%TRACE%    #%myID%-  THREAD: Start
set "output=result %myID%!CR!!LF!%spaces%"
:nextThread

   rem Get next command from main code or previous thread
   set /P "command="
   %TRACE%    #%myID%-  THREAD: - Received "%command%"
   for /F "tokens=1-3" %%a in ("%command%") do (

      rem If the command is intended for this thread
      if "%%a" equ "%myID%" (

         rem Process it
         %TRACE%    #%myID%-  THREAD: - - Working with %%b..%%c
         (for /L %%i in (%%b,1,%%c) do (
            echo Thread #%myID% @ !time! - Line %%i
         )) > output%myID%.txt
         %TRACE%    #%myID%-  THREAD: - - Sending result signal
         echo %output:~0,1021%

      ) else (
         rem Pass the command to next thread in chain
         set "command=%command%!CR!!LF!%spaces%"
         echo !command:~0,1021!
      )

   )

if "%command:~0,4%" neq "exit" goto nextThread
%TRACE%    #%myID%-  THREAD: End
goto :EOF


:Output
%TRACE% OUTPUT: Start
set /A resCount=0, resOrder=1
:nextOutput

   rem Get next signal from thread chain
   set /P "signal="
   %TRACE% OUTPUT: Received "%signal%"

   rem If the signal is a "result" one
   if "%signal:~0,6%" equ "result" (

      rem Show this partial result, if is the next one in output order;
      rem this point avoids synchro problems beween threads
      if "%signal:~7%" equ "%resOrder%" (
         if "%TRACE%" equ "REM" type output%resOrder%.txt
         set /A resOrder+=1
      )

      rem If all partial results were completed
      set /A resCount+=1
      if "!resCount!" equ "%numThreads%" (

         rem Show previous results delayed by a synchro problem, if any
         if "%TRACE%" equ "REM" for /L %%i in (!resOrder!,1,%numThreads%) do type output%%i.txt

         rem Pass to next task
         %TRACE% OUTPUT: Result complete
         set /A resCount=0, resOrder=1
         echo/
         set /P "=Number of lines to generate (Enter to end): " < NUL
      )

   )

if "%signal%" neq "exit" goto nextOutput
%TRACE% OUTPUT: End
goto :EOF

The key point in this scheme is that any part/thread that is waiting in a SET /P command is placed in an inactive state by the operating system, so it does not consume any CPU while is waiting. In traditional schemes, one CPU core is usually reserved for the control code; otherwise this code affects the performance of one of the concurrent threads because both parts share the same CPU core. In the event-driven pipe scheme, all CPU cores can be used for the concurrent threads: the control code will be executed precisely until one of the concurrent threads ends, that is, when there is an available CPU core for it. In other words: the available CPU resources are used in the most efficient way!

Antonio

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: Event-driven multi-thread scheme for Batch files

#3 Post by dbenham » 01 Sep 2015 11:03

Aacini wrote:I added the necessary management to SET /P Batch commands in order to use they in the event handlers of previous example, so the whole multi-thread application can be written in pure Batch. The details about SET /P problems with pipes and the method to solve they are described at Set /P problems with pipes topic.

I can't believe I had forgotten that thread, and the lessons learned :shock: :roll: That was some cool collective sleuthing 8)

Aacini wrote:Below there is an example program that show two different schemes for a multi-thread Batch file application: one with the main code at beginning of the pipe chain, and another one with the main code at end.

Code: Select all

... code removed - see prior post

Execellent idea :!:

I really like Method A with the main code at the beginning :D But one must be aware that it is only good for multi-threaded jobs where each thread receives roughly the same amount of work. The last thread in the chain can only receive commands that have been passed through the earlier threads in the pipe chain. And that can only happen when the earlier threads are idle, waiting for input. If an early thread receives a long task, then the subsequent threads are stuck in an idle state, waiting for the earlier one to finish so it can pass the command down the chain.

So Method A works well for symmetric work partitioning, and thread commands should be passed in reverse order - starting with a command for the right, and finishing with a command for left most thread.

I'm pretty sure method B with the main code at the end cannot work reliably. When multiple processes (threads) write to the same stdout at the same time, then the information can get intermingled and garbled. :cry: The only way to avoid that is to serialize the writes, which reintroduces semaphores and defeats the whole purpose.

Aacini wrote:Previous schemes have the disadvantage that the other part of the synchronization (the side not controlled by the pipe) must be achieved via the presence of a file or line that works like a flag or semaphore; this flag must be tested in a GOTO assembled loop. A more efficient scheme can be assembled if the whole program start and end in a pipe, that is, the application may be divided in three parts: an Input part, a Thread part (that can be executed several concurrent times, one instance by each CPU processor core) and an Output part, all of they connected in a long pipeline. This scheme is very efficient, because the synchronization between all parts/threads is achieved by the operating system itself, not via a test executed in a loop

That is an intriguing idea, but there is a flaw - The input routine looks for (and possibly receives) input before the output has had a chance to prompt the user. A user might grow impatient, and press the wrong key based on the current display, when it will be applied against the future, as-yet unseen prompt.

If synchronous user input is required, then I think it is better that a semaphore be used to notify when the input should begin waiting, in which case there is no need to separate input and output into two threads.

But if it makes sense for the input and output to be asynchronous, then the scheme is a great idea.


Dave Benham

Aacini
Expert
Posts: 1885
Joined: 06 Dec 2011 22:15
Location: México City, México
Contact:

Re: Event-driven multi-thread scheme for Batch files

#4 Post by Aacini » 01 Sep 2015 12:03

dbenham wrote:
Aacini wrote:Previous schemes have the disadvantage that the other part of the synchronization (the side not controlled by the pipe) must be achieved via the presence of a file or line that works like a flag or semaphore; this flag must be tested in a GOTO assembled loop. A more efficient scheme can be assembled if the whole program start and end in a pipe, that is, the application may be divided in three parts: an Input part, a Thread part (that can be executed several concurrent times, one instance by each CPU processor core) and an Output part, all of they connected in a long pipeline. This scheme is very efficient, because the synchronization between all parts/threads is achieved by the operating system itself, not via a test executed in a loop

That is an intriguing idea, but there is a flaw - The input routine looks for (and possibly receives) input before the output has had a chance to prompt the user. A user might grow impatient, and press the wrong key based on the current display, when it will be applied against the future, as-yet unseen prompt.

If synchronous user input is required, then I think it is better that a semaphore be used to notify when the input should begin waiting, in which case there is no need to separate input and output into two threads.

But if it makes sense for the input and output to be asynchronous, then the scheme is a great idea.


Dave Benham


Although you are right in this operative detail, a very simple way to avoid this problem is that the :Output part show messages at regular intervals to remind the user that a task is still in progress. Anyway, if an impatient user enter additional input before last output was completed, nothing bad happens: the new task will start as soon as the last one ends, so it is up to the user if he/she wants this behaviour...

Did you tested the Example C? A very interesting exercise is to modify the numThreads variable to a fixed high number (like 6) and activate the tracing :D

Antonio

einstein1969
Expert
Posts: 941
Joined: 15 Jun 2012 13:16
Location: Italy, Rome

Re: Event-driven multi-thread scheme for Batch files

#5 Post by einstein1969 » 02 Sep 2015 09:26

Aacini wrote:I added the necessary management to SET /P Batch commands in order to use they in the event handlers of previous example, so the whole multi-thread application can be written in pure Batch. The details about SET /P problems with pipes and the method to solve they are described at Set /P problems with pipes topic.


Hi Antonio, I have read the thread of "SET /P problem" and all related links but I have not understund the solution.

In the past i haved the same problem with Mini-Monitor and allarm system and CPU History on title

but the workarounds is not too good.

Then in past decembre 2013 i have realized a thing like these in this thread.

A method that use a "token passing" via pipe for activate job in parallel. But there were a problem of
sync the main process with the "work done".

Using a file is not too fast and i need a low latency. Then i have searched for eliminating wait and i have found
a method like the "barrier method". I have not probed with "attrib" command.

Then I have added an HEARTBEAT for syncronizing and activate without active wait (ie loops). But is not implementhed in the follow work. You can try "heartbeat" for found in this forum.

Sorry this work is not finished and the remark are in italian:

Code: Select all

@echo off & setlocal EnableDelayedExpansion & if not "%1"=="" goto :%1

rem sfrutto la possibilità di set /p di leggere mentre scrivono nel file.

rem pipe circolare , cioe' alla fine scrivo in un file che e' letto dal primo. A che velocità posso contare?

rem i processi contano in parallelo, ognuno ha però una partizione (sottoinsieme).

rem minimo due processi.

rem questa architettura si puo' ridurre di un processo o forse anche di due.

rem Attenzione alcune volte le righe si uniscono quindi uso un fine riga fittizio come penpen #

set debug_VIS=rem
set debug_D=rem
set debug_P=rem
set debug_S=rem
set debug_RI_P=rem
set debug_RI_S=rem
set no_cpu=rem

set num_processi=2

for /f %%a in ('copy /Z "%~f0" nul') do set "CR=%%a"

rem inizializzo la procedura di lettura tramite set/p. Buffer di input per set/p inizialmente vuoto
call :init_new_readinput

title ciao
tasklist /V
echo prendere il pid ( che sarà il parent pid)
echo usare wmic process per i sotto processi + tasklist /v per la cpu-usage
pause

%0 D | %0 P 0 | %0 P 1 | %0 S

goto :eof


:D
rem fast dispatcher/heartbet

rem call :title_pid

rem time di inizio per verificare le elaborazioni al secondo
set t0=%time%

For /L %%d in (0,1,500) do (
   call :sendtoken %%d
   call wait_script.bat 15
   rem calcolo la frequenza
   for /F "tokens=1-8 delims=:.," %%a in ("!t0: =0!:!time: =0!") do set /a "a=(((1%%e-1%%a)*60)+1%%f-1%%b)*6000+1%%g%%h-1%%c%%d, a+=(a>>31) & 8640000"
   set /a Freq=%%d*10000/a
   title !freq:~0,-2!.!freq:~-2!
)

goto :eof


rem dispatcher/heartbet
:D_

type nul>%tmp%\token.txt
set token=0
set "endOfStream=q" & rem segnala la fine a tutti i processi!

rem attendo avvio processi a destra. In realtà bisognerebbe implementare un metodo piu' sicuro. DA FARE.
call wait_script.bat 2000

rem spedisce il token e attende che torni il risultato.

set tt0=!time!& call :sendtoken !token!
%debug_D% echo !tt0! %1:Spedito PRIMO token:%token% >CON

rem time di inizio per verificare le elaborazioni al secondo
set t0=%time%

(
  for /L %%n in (1,1,300) do (

    %debug_D% echo !time! %1[%%n]:Wait Begin > CON
    call wait_script.bat 30
    rem ping>nul
    rem for /L %%S in (1,1,20000) do rem
    %debug_D% echo !time! %1[%%n]:Wait End > CON

    %debug_D% echo !time! %1[%%n]:read Token > CON
    set "token="
    set /p "token="
    %debug_D% echo !time! %1[%%n]:readed Token:'!token!' > CON

    if defined token (
   %debug_D%   echo !time! %1[%%n]:Letto Token da file=!token! e invio > CON
   rem invia il token
   call :sendtoken !token!
        %debug_D%   echo !time! %1[%%n]:Token !token! Inviato. > CON
   rem calcolo la frequenza
   for /F "tokens=1-8 delims=:.," %%a in ("!t0: =0!:!time: =0!") do set /a "a=(((1%%e-1%%a)*60)+1%%f-1%%b)*6000+1%%g%%h-1%%c%%d, a+=(a>>31) & 8640000"
   set /a Freq=token*10000/a
   title !freq:~0,-2!.!freq:~-2!
    )
  )
) <%tmp%\token.txt

%debug_D% echo !time! %1:End. > CON

goto :eof


rem processo che elabora %1=P %2=numero processo
:P

call :title_pid 123-%2

%debug_P% echo !time! %1:Avvio P%2 >CON

:Cicle_For_P
for /L %%p in (1,1,100) do (

  call :new_readinput_P %2

  %debug_P% echo !time! %1:P%2:Letto token:'!token!' ExitFlag='!ExitFlag!' >CON

  rem se c'e' il token devo inviarlo. Se non c'e' o devo caricarlo o e' finito lo stream.

  if defined token (

    rem se e' un segnale di END lo spedisco e torno ad attendere.
    if "!token:~0,3!"=="END" (
   rem invia il token END
   call :sendtoken "!token!"
     %debug_P%   echo !time! %1:P%2:Inoltrato segnale END '!token!' >CON
    ) else (

      rem il token non e' di END.
      rem spedisce il token e inizia l'elaborazione

      rem se sono l'ultimo processo non spedisco il token.
      set /a ultimo=num_processi-1
      if not "%2"=="!ultimo!" (
   rem invia il token
   call :sendtoken "!token!"
     %debug_P%   echo !time! %1:P%2:Inviato token:!token! >CON
      )

      set /a start=%2+10

      type nul>%tmp%\file%2.txt

      rem parentesi esterne per reindirizzamento piu' veloce e solo > senza type , FARE TEST
      %no_cpu% For /L %%n in (!start!,!num_processi!,50) do echo %%n Linea Genearata dal processo %2 con token !token! ........................>>%tmp%\file%2.txt

      rem ho finito l'elaborazione e segnalo la fine ai processi P seguenti per arrivare ad S

      rem.echo END %2
      call :sendtoken "END !token! %2"
      %debug_P% echo !time! %1:P%2:Spedito segnale END !token! %2 >CON
    )
  )
)
if not defined ExitFlag goto :Cicle_For_P

%debug_P% echo !time! %1:End P%2 >CON

goto :eof


rem mentre fa il sort potrebbero partire nuove elaborazioni
rem fare test con un solo file da ordinare : ogni processo a fine elaborazione appende in una sola istruzione o piu' istruzioni
:S

rem call :title_pid

%debug_S% echo !time! %1:Avvio >CON

:Cicle_For_S
for /L %%s in (1,1,100) do (

  for /L %%n in (1,1,%num_processi%) do (

   call :new_readinput_S

   %debug_S% echo !time! %1: Token=!token!> CON

  )

  if defined token (

    rem copio i file da ordinare in file temporanei
    rem copy %tmp%\file0.txt %tmp%\file0.txt.tmp >nul 2>nul
    rem copy %tmp%\file1.txt %tmp%\file1.txt.tmp >nul 2>nul

    rem estraggo il token
    set token=!token:~4,-1!

    set /a token=token+1

    rem scrive il token e i processi P partono!
    (echo !token!)>>%tmp%\token.txt
    %debug_S% type %tmp%\token.txt
    %debug_S% echo !time! %1:Spedito token nuovo !token!

    rem Attenzione se si sposta il sort sotto serve usare nomi di file temporanei altrimenti i processi che
    rem partono (P) possono sovrascrivere mentre avviene il sort

    %debug_S% echo !time! %1:Inizio SORT.

    rem (type %tmp%\file0.txt.tmp %tmp%\file1.txt.tmp)2>nul | sort /o %tmp%\sort.txt
    %no_cpu% (type %tmp%\file0.txt %tmp%\file1.txt)2>nul | sort /o %tmp%\sort.txt

    Rem Visualizzo
    %no_cpu% if not defined debug_VIS (type %tmp%\sort.txt) else (cls&type %tmp%\sort.txt)
  )
)
if not defined ExitFlag goto :Cicle_For_S

goto :eof


:sendtoken %1=token
  <nul set/p"=%~1#"
goto :eof


rem parte comune di lettura per i processi P e S
rem non serve il delay perche' attende in modo passivo.

:readinput
(
   set "token="
   set /p "token="
   %debug% echo !time! readinput:'!token!' >CON
   if NOT DEFINED token goto :readinput
goto :eof
)

rem presa da test_input_setp.cmd
:new_readinput_P

   if not defined bsetp (set load=T) else (if "!bsetp:*#=!"=="!bsetp!" (set load=T) else set load=)

   if defined load (
      rem carico il buffer
      set "input="
      set /p "input="
      %debug_RI_P% echo !time! P%1:RI:Input:'!Input!' >CON
      set bsetp=!bsetp!!input!
      if not defined input (   set ExitFlag=T
               %debug_RI_P% echo !time! P%1:RI:No input-End stream. >CON
      )
   )

   set "token="
   if defined bsetp (
      for /F "tokens=1,* delims=#" %%f in ("!bsetp!") do (set "token=%%f" & set "bsetp=%%g")
      if not defined ExitFlag %debug_RI_P% echo !time! P%1:RI:LINE/TOKEN:'!token!' bsetp:'!bsetp!' >CON
   )

goto :eof

:new_readinput_S

   if not defined bsetp (set load=T) else (if "!bsetp:*#=!"=="!bsetp!" (set load=T) else set load=)

   if defined load (
      rem carico il buffer
      set "input="
      set /p "input="
      %debug_RI_S% echo !time! S:Input:'!Input!' >CON
      set bsetp=!bsetp!!input!
      if not defined input (   set ExitFlag=T
               %debug_RI_S% echo !time! S:No input-End stream. >CON
      )
   )

   set "token="
   if defined bsetp (
      for /F "tokens=1,* delims=#" %%f in ("!bsetp!") do (set "token=%%f" & set "bsetp=%%g")
      if not defined ExitFlag %debug_RI_S% echo !time! S:LINE/TOKEN:'!token!' bsetp:'!bsetp!' >CON
   )

goto :eof

:init_new_readinput
  set "ExitFlag="
  set "Bsetp="
goto :eof

per il pid e il tempo via cscript riaprire sleep-aacini.cmd e sleep-ultimate.cmd  e get-pid.cmd
:title_pid
  title %1
goto :eof


But at the moment i have not understund how risolve the set/p problem (realiability)

Can you esplain me how to resolve the set/P problem?

Thanks

einstein1969

Aacini
Expert
Posts: 1885
Joined: 06 Dec 2011 22:15
Location: México City, México
Contact:

Re: Event-driven multi-thread scheme for Batch files

#6 Post by Aacini » 02 Sep 2015 10:15

einstein1969 wrote:
Aacini wrote:I added the necessary management to SET /P Batch commands in order to use they in the event handlers of previous example, so the whole multi-thread application can be written in pure Batch. The details about SET /P problems with pipes and the method to solve they are described at Set /P problems with pipes topic.



Hi Antonio, I have read the thread of "SET /P problem" and all related links but I have not understund the solution.


The trick is send 1023 bytes always, that is, 1021 bytes plus the CR+LF of ECHO command:

at viewtopic.php?f=3&t=6134&p=38896#p38896 dbenham wrote:And here is some interesting code that shows how a piped version can give the correct result without introducing a delay. Each value is printed normally with trailing CRLF. The trick is to follow each value with fill data such that the length of value + CRLF + filler is exactly 1023 bytes.


When left.bat writes in chunks of 1023 (1021 + CRLF from ECHO), then everything works great!


But when left.bat writes too many or too few bytes, then things quickly get out of synch:


In my Example C, :Input subroutine send commands to the chain of threads this way:

Code: Select all

      set "output=%%i !from%%i! !to%%i!!CR!!LF!%spaces%"
      echo !output:~0,1021!

In :Thread subroutine, the command is read in the usual way:

Code: Select all

   rem Get next command from main code or previous thread
   set /P "command="

... but when this subroutine pass the command to the next thread, it fills the buffer again to 1023 bytes:

Code: Select all

      ) else (
         rem Pass the command to next thread in chain
         set "command=%command%!CR!!LF!%spaces%"
         echo !command:~0,1021!
      )

Antonio

einstein1969
Expert
Posts: 941
Joined: 15 Jun 2012 13:16
Location: Italy, Rome

Re: Event-driven multi-thread scheme for Batch files

#7 Post by einstein1969 » 02 Sep 2015 11:42

Thanks so much Antonio!

The last:
at viewtopic.php?f=3&t=6134&p=38896#p38896 dbenham wrote:Note that the CRLF is not needed at the end of the 1023 byte chunk.

and
Aacini wrote:In my Example C, :Input subroutine send commands to the chain of threads this way:

Code: Select all

      set "output=%%i !from%%i! !to%%i!!CR!!LF!%spaces%"
      echo !output:~0,1021!

In :Thread subroutine, the command is read in the usual way:

Code: Select all

   rem Get next command from main code or previous thread
   set /P "command="

... but when this subroutine pass the command to the next thread, it fills the buffer again to 1023 bytes:

Code: Select all

      ) else (
         rem Pass the command to next thread in chain
         set "command=%command%!CR!!LF!%spaces%"
         echo !command:~0,1021!
      )

Antonio


This means that the CR LF are necessary for trick to work OR we can pass an line of 1023 byte without CR LF?
why you use the CR LF?

EDIT: It's possible use more than one CR LF for the trick to work (filling 1023 byte with a multiline informations?)

Einstein1969

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: Event-driven multi-thread scheme for Batch files

#8 Post by dbenham » 02 Sep 2015 12:13

einstein1969 wrote:This means that the CR LF are necessary for trick to work OR we can pass an line of 1023 byte without CR LF?
why you use the CR LF?

CR LF is not required, but it sure is more convenient, less restrictive, and faster to use ECHO instead of <nul SET P "=data"

einstein1969 wrote:It's possible use more than one CR LF for the trick to work (filling 1023 byte with a multiline informations?)

No, you cannot send multi-line values. If you read the linked thread(s) you will see that SET /P reads past CR LF, but discards everything after the first LF.


Dave Benham

einstein1969
Expert
Posts: 941
Joined: 15 Jun 2012 13:16
Location: Italy, Rome

Re: Event-driven multi-thread scheme for Batch files

#9 Post by einstein1969 » 02 Sep 2015 13:49

Thanks Dave.

Now it's clear.

Einstein1969

jeb
Expert
Posts: 1041
Joined: 30 Aug 2007 08:05
Location: Germany, Bochum

Re: Event-driven multi-thread scheme for Batch files

#10 Post by jeb » 04 Sep 2015 10:20

@Aacini
Really cool demonstration of the event technic :)
I tried also to build something similiar, but as I forgot that set/p waits while the pipe is intact I build a totally complex solution :?

But that brings me to one improvement.
You don't need an extra "exit" event, you can simply check for an empty command.

That appears automatically when the producer threads stops, as the set /p will not longer wait for input.

Code: Select all

set "command="
set /p command=
if not defined command exit  %=exit thread %
....


Also the 1023 bytes are not really necessary, the out of sync problem occours because with other message length the reading set /p can read more than one message.
And when it reads more than one message it can only see the first one if the messages are separeted by CR/LF.

And currently I'm not sure that a 1023 byte message is always safe against splitting by the consumer.
I'm not sure that the put message by the producer is an atomic operation, so theoretical it should be possible that even your 1023 bytes messages can be corrupted.

Squashman
Expert
Posts: 4465
Joined: 23 Dec 2011 13:59

Re: Event-driven multi-thread scheme for Batch files

#11 Post by Squashman » 04 Sep 2015 12:39

Some days I do not even feel worthy to be a moderator on this forum. I wish I had time to study all the unique stuff you guys have figured out over the years. Maybe when I retire.

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: Event-driven multi-thread scheme for Batch files

#12 Post by dbenham » 07 Sep 2015 07:20

jeb wrote:@Aacini
But that brings me to one improvement.
You don't need an extra "exit" event, you can simply check for an empty command.
That appears automatically when the producer threads stops, as the set /p will not longer wait for input.
Interesting idea, but that assumes an empty line is never a valid command, which is perfectly fine in this case.

jeb wrote:Also the 1023 bytes are not really necessary, the out of sync problem occours because with other message length the reading set /p can read more than one message.
And when it reads more than one message it can only see the first one if the messages are separeted by CR/LF.
I don't understand what you are saying. It sounds to me like you have provided the exact reasons why 1023 bytes are necessary to reliably read each message.

jeb wrote:And currently I'm not sure that a 1023 byte message is always safe against splitting by the consumer.
I'm not sure that the put message by the producer is an atomic operation, so theoretical it should be possible that even your 1023 bytes messages can be corrupted.
I suppose that could be true, but I hope that a single ECHO statement never pauses in the middle long enough for the SET /P to cutoff input prematurely. If your concern is valid, then I don't think there is a viable solution.


Dave Benham

jeb
Expert
Posts: 1041
Joined: 30 Aug 2007 08:05
Location: Germany, Bochum

Re: Event-driven multi-thread scheme for Batch files

#13 Post by jeb » 09 Sep 2015 07:42

dbenham wrote:jeb wrote:
And currently I'm not sure that a 1023 byte message is always safe against splitting by the consumer.
I'm not sure that the put message by the producer is an atomic operation, so theoretical it should be possible that even your 1023 bytes messages can be corrupted.
I suppose that could be true, but I hope that a single ECHO statement never pauses in the middle long enough for the SET /P to cutoff input prematurely. If your concern is valid, then I don't think there is a viable solution.

I made millions of tests and I can't produce a break here, and I read that some file operations are atomic in windows when the data is not to large, so this seems indeed to be safe.

dbenham wrote:jeb wrote:
Also the 1023 bytes are not really necessary, the out of sync problem occours because with other message length the reading set /p can read more than one message.
And when it reads more than one message it can only see the first one if the messages are separeted by CR/LF.
I don't understand what you are saying. It sounds to me like you have provided the exact reasons why 1023 bytes are necessary to reliably read each message.


dbenham wrote:But one must be aware that it is only good for multi-threaded jobs where each thread receives roughly the same amount of work. The last thread in the chain can only receive commands that have been passed through the earlier threads in the pipe chain. And that can only happen when the earlier threads are idle, waiting for input. If an early thread receives a long task, then the subsequent threads are stuck in an idle state, waiting for the earlier one to finish so it can pass the command down the chain.


I build a system to solve this.
The idea is to use one message thread per worker thread and all message threads are in a long pipe chain, starting with the main thread.

Code: Select all

mainThread | messageThread-1 | messageThread-2 | messageThread-3 | messageThread-4
                  |                 |                 |                 |
             workerThread-1    workerThread-2    workerThread-3    workerThread-4


When the main thread sends a message to the first message thread, this thread sends the message directly to the next message thread and also to its worker thread.

This would also fail when you use the long event messages of 1023 bytes, as after the fourth message in the pipe buffer the producer thread will be blocked.

I'm using small messages, so this can work for an appropriate long time,
but as the messages are small it's not longer determined how much messages will be read by a consumer thread.

Therefore each message must be appended with an EOL character, it's not allowed to use this character in the messages itself.
In the example I used the "$" sign as EOL.

At the end the threads exits by itself when they detect that the producer thread has stopped.

Code: Select all

@echo off
setlocal EnableDelayedExpansion
for /F %%a in ('copy /Z "%~F0" NUL') do set "CR=%%a"

if "%~1" neq "" goto %~1


set "spaces=."
echo Start %time%
for /L %%i in (1,1,11) do set "spaces=!spaces!!spaces!"

"%~f0" MainA  | "%~f0" ctrl Thread A 2>&1 | "%~f0" ctrl Thread B 2>&1 | "%~f0" ctrl Thread C 2>&1 | "%~f0" ctrl Thread D 2>&1 | "%~f0" ctrl Thread E 2>nul
echo Main finished %time%
exit /b

:Ctrl Thread
echo !time! Start: Ctrl-Threadmaster %~2 %3 > con
"%~f0" MessageController %3 | "%~f0" "%~2" %3
echo !time! Exit:  Ctrl-Threadmaster %~2 %3 > con
exit /b

:MessageController
echo !time! Start: MessageController-%2 > CON
set "lineBuffer="
set count=0
set read_cnt=0
for /L %%n in (0 0 0) do (
    set "found="
    if defined lineBuffer if "!lineBuffer:$=!" NEQ "!lineBuffer!" set found=1
    if defined found (
        for /F "tokens=1,* delims=$" %%A in ("!lineBuffer!") DO (
            set "message=%%A"
            set "lineBuffer=%%B"
        )
        set /a count+=1
        rem echo C# '!message!' > CON
        <nul set /p ".=!message!$"       
        <nul set /p ".=!message!$" 1>&2
    ) ELSE (
        set "line="
        set /p line=
        set "lineBuffer=!lineBuffer!!line!"
        if not defined line (
            echo !time! Exit:  MessageController-%2 pipe closed, msg=!count! read=!read_cnt! > CON
            rem ping -n 2 localhost > nul
            exit
        )
        set /a read_cnt+=1
    )
)
exit /b

:MainA
set "path="
for /L %%n in (1 1 1000) do (
    set "line=++++++++++++++%%n"
    rem set "line=!line:~-10!!spaces:~0,1010!x"
    rem set "line=!line:~-10!!spaces:~0,200!!line:~-5!x$"
    set "line=!line:~-5!$"
    call set gh=.
    <nul set /p ".=!line!"
)
rem ping -n 2 localhost > nul
exit /b

:Thread
echo !time! Start: Thread-%2 > CON
set count=0
set expect=1
set read_cnt=0
set "lineBuffer="
ping -n 5 localhost > nul
for /L %%n in (0 0 0) do (
    set "found="
    if defined lineBuffer if "!lineBuffer:$=!" NEQ "!lineBuffer!" set found=1
    if defined found (
        set /a count+=1
        for /F "tokens=1,* delims=$" %%A in ("!lineBuffer!") DO (
            set "message=%%A"
            set "lineBuffer=%%B"
        )
        rem echo A# '!message:~0,10!' > CON
        set /a val=!message:~0,10!
        if "!val!" NEQ "!expect!" (
            echo !time! ERROR: Thread-%2 unexpected value=!val! expected=!expect! last=!last! '!message:~0,20!' -- '!message:~-10!'  -- !lineBuffer:~0,20! > CON
            exit
        )
        set last=!val!
        set /a expect+=1
    ) ELSE (
        set "line="
        set /p line=
        set "lineBuffer=!lineBuffer!!line!"
        if not defined line (
            echo !time! Exit:  Thread-%2 Pipe closed, count=!count! read_cnt=!read_cnt! '!lineBuffer!' > CON
            rem ping -n 2 localhost > nul
            exit
        )
        set /a read_cnt+=1
    )
)

einstein1969
Expert
Posts: 941
Joined: 15 Jun 2012 13:16
Location: Italy, Rome

Re: Event-driven multi-thread scheme for Batch files

#14 Post by einstein1969 » 20 Nov 2015 10:31

Hi,

I have found a old test that show how priority can reduce number or fails in function of time.
This is good for fast syncronize more processes using a heartbeat for example.

This has few bugs. Can you improve it?


Code: Select all

@echo off & setlocal EnableDelayedExpansion & if not "%1"=="" goto :%1

rem secondo penpen le righe che hanno LF/CR creano problemi.

rem may-2013


set debug=rem
for /f %%a in ('copy /Z "%~dpf0" nul') do set "CR=%%a"

mode con cols=100 lines=45

call :calibrate_delay_ms

start "" /B %0 sender |  %0 receiver

pause

%0 sender |  %0 receiver

pause

%0 sender |  start "" /ABOVENORMAL /WAIT /B %0 receiver

pause

goto :eof

:sender

echo Start Sender>con

For %%s in (50 8 4 2 1) do (

  set /a L=200/%%s
 
  call :delay 50&set nl=0
  set /a nl+=1&<nul  set /p ".=DELAY:%%s cs"

  For /L %%n in (1,1,!L!) do (

    set /a nl+=1&<nul  set /p ".=!nl!:%%s cs Hi, i'm started at the !time!. When arrived me?"
    call :delay %%s
    set /a nl+=1&<nul  set /p ".=!nl!:%%s cs Hi, i'm started at the !time!. When arrived me?"
    call :delay %%s
   
  )

)

For %%s in (10 8 4 2 1) do (

  set /a L=1000/%%s

  call :delay 50&set nl=0
  set /a nl+=1&<nul  set /p ".=DELAY:%%s ms"
 
  For /L %%n in (1,1,!L!) do (

    set /a nl+=1&<nul  set /p ".=!nl!:%%s ms Hi, i'm started at the !time!. When arrived me?"
    %delay% %%s %ms%
    set /a nl+=1&<nul  set /p ".=!nl!:%%s ms Hi, i'm started at the !time!. When arrived me?"
    %delay% %%s %ms%
   
  )

)

set /a delay_ms=delay_ms/10
call :set_macro_delay

For %%s in (8 4 2) do (

  set /a L=10000/%%s

  call :delay 50&set nl=0
  set /a nl+=1&<nul  set /p ".=DELAY:0.%%s ms"
 
  For /L %%n in (1,1,!L!) do (

    set /a nl+=1&<nul  set /p ".=!nl!:0.%%s ms Hi, i'm started at the !time!. When arrived me?"
    %delay% %%s %ms%
    set /a nl+=1&<nul  set /p ".=!nl!:0.%%s ms Hi, i'm started at the !time!. When arrived me?"
    %delay% %%s %ms%
   
  )

)

call :delay 50
<nul  set /p ".=END"
echo Exit Sender...>CON

goto :eof


ping 192.0.2.0 -n 1 -w 500 >nul 2>nul

:receiver
  echo Start Receiver

:receiver_loop
set "s="
set /p "s="
if defined s (
        if "!s:~0,5!"=="DELAY" (set nl=1&set delay=!s:*:=!&echo() else (
   set /a nl+=1
        set "sf=!s:*:=!"
        rem echo S=!s!
        rem echo SF=!sf!
        for /f "delims=" %%s in ("!sf!") do set "si=!s:%%~s=!"
        rem echo SI=!si:~0,-1!
        set lns=!si:~0,-1!
        set /a diff=lns-nl
        <nul set /p ".=Delay: !delay! - Lines sent/arrived/diff: !lns! / !nl! / !diff!                   !CR!"
        )
)

if not "!s!"=="END" goto :receiver_loop
echo Exit Receiver...
Goto :eof

   rem echo !s! At %time%!
   rem if not defined r (set r=20) else set /a r=r-1
        rem if !r! lss 0 (cls&set r=20)



:delay %1=delay_cs
  set t0=%time%
:loop_delay
  for /F "tokens=1-8 delims=:.," %%a in ("!t0: =0!:!time: =0!") do set /a "a=(((1%%e-1%%a)*60)+1%%f-1%%b)*6000+1%%g%%h-1%%c%%d, a+=(a>>31) & 8640000"
  if !a! lss %1 goto :loop_delay
goto :eof

:set_macro_delay
  set "delay=for /L %%? in (1,1,"
  set "ms=) do for /L %%. in (1,1,%delay_ms%) do rem"
goto :eof

:calibrate_delay_ms

  echo calibrating busy timer...

  rem fast setting
  set delay_ms=396&call :set_macro_delay&goto :eof

  set delay_ms=100
  call :set_macro_delay

  set /a md=0,step=7

rem initial value

  set z=0
:loop_disc
  set /a z=z+1, step-=1
  set t0=%time%
  %delay% 1000 %ms%
  for /F "tokens=1-8 delims=:.," %%a in ("!t0: =0!:!time: =0!") do set /a "a=(((1%%e-1%%a)*60)+1%%f-1%%b)*6000+1%%g%%h-1%%c%%d, a+=(a>>31) & 8640000"
  set /a delay_ms=delay_ms*100/a
  call :set_macro_delay
%debug%  echo "%a%  new:%delay_ms% %delay% 1000 %ms%"
  if %z% leq 1 goto :loop_disc
  <nul set /p ".=[%step%]"

  set z=0
:loop_media
  set /a z=z+1, step-=1
  set t0=%time%
  %delay% 10000 %ms%
  for /F "tokens=1-8 delims=:.," %%a in ("!t0: =0!:!time: =0!") do set /a "a=(((1%%e-1%%a)*60)+1%%f-1%%b)*6000+1%%g%%h-1%%c%%d, a+=(a>>31) & 8640000"
  set /a delay_ms=delay_ms*1000/a, md=md+delay_ms
  call :set_macro_delay
%debug%  echo "%a% new:%delay_ms% %md% %delay% 10000 %ms%"
  <nul set /p ".=[%step%]"
  if %z% leq 3 goto :loop_media

rem final

  set t0=%time%
  %delay% 10000 %ms%
  for /F "tokens=1-8 delims=:.," %%a in ("!t0: =0!:!time: =0!") do set /a "a=(((1%%e-1%%a)*60)+1%%f-1%%b)*6000+1%%g%%h-1%%c%%d, a+=(a>>31) & 8640000"
  set /a "delay_ms=delay_ms*1000/a, md=md+delay_ms, delay_ms=(md/(z+1))"
  call :set_macro_delay
%debug%  echo "%a% new:%delay_ms% %md% %delay% 10000 %ms%"

  echo Calibrate Setting at !delay_ms!
  call :delay 200

  echo Testing for ten seconds.
  echo %time%
  %delay% 10000 %ms%
  echo %time%

  echo Testing for one seconds.
  echo %time%
  %delay% 1000 %ms%
  echo %time%

  echo Testing for one seconds.
  echo %time%
  %delay% 1000 %ms%
  echo %time%

  pause

goto :eof


einstein1969

Aacini
Expert
Posts: 1885
Joined: 06 Dec 2011 22:15
Location: México City, México
Contact:

Re: Event-driven multi-thread scheme for Batch files

#15 Post by Aacini » 23 Nov 2020 20:28

I wrote a new version of this application designed to process several files in simultaneous way via parallel threads that is one of the most requested applications of parallel processing in Batch files. The general scheme of this version is the one suggested by jeb:

Code: Select all

Main  |  Process-1  |  Process-2  |  Process-3  |  ...
             |             |             |
          Worker-1      Worker-2      Worker-3
Main module send commands to all processes through the pipeline via StdErr. Each Process get the command and review it: if the command is intended for such a Process, it sends the filename to its associated Worker module via StdOut; otherwise it sends the command to the next Process in the line via StdErr. In this way, all Processes are always ready to get and propagate commands no matter how long it takes to its Worker to process one file. All the communication between these modules is performed via pipelines, so it is event-driven and managed by the Operating System.

When the Main module have sent files to all processes, it executes a waitFor AnyProcEnds command that waits for such a signal. When a Worker module have finished the processing of one file, it creates a *.end file that identifies itself and then execute a waitFor /SI AnyProcEnds command that send such a signal. Then, Main module "wakes-up" and send the next file to the idle Process/Worker module, that is identified by the existent *.end file. Again, this method to wait for the completion of the process of a file is event-driven and managed by the OS, so it don't waste CPU time.

Code: Select all

@echo off
setlocal EnableDelayedExpansion

rem Parallel processing of several files via a Multi-thread scheme
rem Antonio Perez Ayala aka Aacini - 2020/11/23

set "myID=%~2"
if "%~1" neq "" if "%~1" neq "Procs" goto %1

rem Set number of processes from "Procs=#" parameter, or by default
set "numProcs=%2"
if not defined numProcs set "numProcs=%NUMBER_OF_PROCESSORS%"

rem Define auxiliary variables: http://www.dostips.com/forum/viewtopic.php?f=3&t=6134
set LF=^
%Do not remove this line 1/2%
%Do not remove this line 2/2%
for /F %%a in ('copy /Z "%~F0" NUL') do set "CR=%%a"
set "spaces= "
for /L %%i in (1,1,10) do set "spaces=!spaces!!spaces!"

rem Assemble the pipe line with all process threads and start the whole job
set "pipeLine="%~NX0" Main 2>&1"
for /L %%p in (1,1,%numProcs%) do set "pipeLine=!pipeLine! | ("%~NX0" Process %%p | "%~NX0" Worker %%p) 2>&1"
ECHO !pipeLine!
ECHO/
del *.end 2>NUL
echo The whole job starts @ %time%
%pipeLine%
echo The whole job ends   @ %time%
goto :EOF


================================


:Main

ECHO Main starts

rem Get list of files to process
set "numFiles=0"
for %%f in (*.txt) do (
   set /A "numFiles+=1"
   set "file[!numFiles!]=%%f"
)

rem Start process of files
set "nextFile=1"
goto nextProc

rem Wait for any process ends
:waitProcEnds
waitFor AnyProcEnds > NUL
:nextProc
if not exist *.end goto waitProcEnds
for %%p in (*.end) do (
   del %%p
   call :sendMsg %%~Np "!file[%nextFile%]!"
   goto justOne
)
:justOne
set /A nextFile+=1
if %nextFile% leq %numFiles% goto nextProc

ECHO Main ends, all files sent to process...
goto :EOF


================================


:Process myID

ECHO PROCESS %myID% STARTING > CON

:nextCmd
rem Get a command from main module or previous process thread
set "command="
set /P "command="
if not defined command goto endProcess
for /F "tokens=1*" %%a in ("%command%") do (
   if "%%a" equ "%myID%" (
      rem Command intended for this process: pass file to own worker
      echo Process #%myID%, received  file: %%b > CON
      echo %%b
   ) else (
      rem Pass command to next process in the chain
      call :sendMsg %command%
   )
)
goto nextCmd

:endProcess
echo PROCESS %myID% TERMINATING... > CON
goto :EOF


=======================


:Worker myID

rem Inform worker is ready for next file
cd . > %myID%.end
waitFor /SI AnyProcEnds > NUL

rem Get a file from my boss and process it
set "fileName="
set /P "fileName="
if not defined fileName goto endWorker
ECHO Worker of Process %myID% processing file %fileName% > CON
ping -n 2 localhost > NUL
ECHO Worker processed file: %fileName% > CON
goto Worker

:endWorker
echo Worker of Process %myID% terminating... > CON
del %myID%.end
goto :EOF

=======================

:sendMsg command
set "message=%*!CR!!LF!%spaces%"
>&2 echo %message:~0,1021%
exit /B
This program is plenty of up-case ECHO commands that just traces the whole job, so you may delete such commands if you wish. You must replace the ping -n 2 localhost > NUL command in Worker module by the real command that process the %fileName% as you wish. Note that any StdOut output in Worker module may interfere with the process, so you should redirect to CON device the screen output you wish as shown in the code.

Antonio

Post Reply