Dos Batch Multi Processes

[einstein1969]

Hi, to all

I open this topic to talk about the various methods and techniques to parallelize an application. And show some examples....

I know two type for RUN multiprocess in dos batch:

- START [/B]
- PIPE (command1 | command2 |... )

What is the differency? There is other method?

Multi-Thread/Processes and Multi-core.
I think that the differency about this is how run on specific hardware.


IPC Inter Process Communication
What type di communication exist? How to do communicate?

any else?


einstein1969

[dbenham]

I think my SNAKE program is a good place to start. You can read my explanation of the SNAKE Inter-Process Communication.

Another aspect of working with multiple processes is determining when the various processes have terminated. Toward the top of SNAKE.BAT I have exit cleanup code that must wait for the controller to terminate. I use a lock file (keyFile) to determine if the controller is still active. The controller uses redirection to establish an exclusive lock on a file. If the main process is not able to redirect to the the lock file, then I know the controller is still active.

At http://stackoverflow.com/a/11715437/1012053 I extensively use the redirected file lock mechanism to regulate how many parallel processes execute at one time. This is for a situation where you have many processes you want to run in parallel, but you want to limit the number of processes that run on any one CPU.

There is a more straight-forward method for determining when a process ends. The process could create or delete a signal file upon exit. But this is not reliable if the process crashes before it signals its termination. The redirection lock file mechanism is more reliable because the OS automatically releases the lock if the process crashes.

There is another reliable option for determining when a process has terminated - you can use TASKLIST to check if the process is running. But you must know the Process ID (PID) if you want to be positive as to which process you are looking at. Getting the PID of a launched process is tricky in batch. But it can be done using WMIC PROCESS CALL CREATE. See http://ss64.org/viewtopic.php?id=1495. I find the redirected lock file mechanism much simpler, and faster, so I do not bother messing with PIDs.

Another common issue is how to serialize events across multiple processes. For example, how can multiple processes safely write to a single log file without collision? Again, a lock file via redirection provides a solution. See http://stackoverflow.com/a/9344547/1012053.


Dave Benham

[penpen]

Here you can find some pipe examples (no "real" inter-process communicazion..., only one way):
http://www.dostips.com/forum/viewtopic.php?p=34570#p34570.

I never tested, if the single processes are getting started only on one core, or on mutliple cores.

penpen

[dbenham]

Here you can find some pipe examples (no "real" inter-process communicazion..., only one way):
http://www.dostips.com/forum/viewtopic.php?p=34570#p34570.

The file based inter-process communication method I use with SNAKE.BAT should work fine, even with processes launched by a pipe. The pipe can provide communication in the forward direction, and the SNAKE method can be used for the reverse direction.


Dave Benham

[penpen]

Code: Select all

The file based inter-process communication method I use with SNAKE.BAT should work fine, even with processes launched by a pipe.

I don't doubt that.
I just wanted to mention the (in my eyes) main disadvantage of this technique (batch pipes) (without writing too much).

- In the wider sense (or 'nowadays') the term 'IPC' is defined (in short) as some kind of data interchange within any (distributed) system (<- IPC).
- In the narrower sense (or 'old definition') it is a communication (which is mainly used to synchronize any access to resources)
between two threads/processes with separated memory on a single computer;
files are treated as resources only (<- "real" IPC).

So the disadvantage is you have to access resources (old definition) to synchronize the access to resources:
This may be risky especially in cases where mutliple instances of the same batch is running.
So you have to build your own resource handling as you have done this in snake.bat.
It works fine, but this is also an easy case: Only one controller, and one game instance is running (using the same file rersources).

penpen

[einstein1969]

Hi penpen,

Thanks for your talk about. My mind in this day not work well (the intellect , unfortunately for therapeutic reasons ) but soon i will respond on this.

Mainly i will do work parallelizable. The sync is the main problem. I worked for this in a HEARTBEAT. But i can't add other for the moment...

thanks again!


@dbenham
The two way communication is very interesting!



This is an old experiment (07/07/2014)

Code: Select all

@echo off
setlocal EnableDelayedExpansion
if "%~1" equ ":thread" goto :thread

(ping ::1 -n 5 >nul&(for /L %%i in (1,1,200) do @(echo %%i))&echo :END ) >"%temp%\t.txt.tmp" | start "" /WAIT /B cmd /c " %~f0 :thread 0 <%temp%\t.txt.tmp" | start "" /WAIT /B cmd /c "%~f0 :thread 1 <%temp%\t.txt.tmp"

del "%temp%\t.txt.tmp"
exit /b

:thread
  for /l %%. in () do (
    set "ln="
    set /p "ln="
    if defined ln (
      if "!ln:~0,4!" equ ":END" (ping ::1 -n 5 >nul & echo Thread %2 End. >CON: & exit)
      rem set "ln=!ln:*]=!"
      set /a "n=ln %% 2"
      if !n! equ %2 (echo(%2-!time!:!ln!)>CON:
      rem ping ::1 -n 1 >nul
    )
  )

Francesco Poscetti