Thanks!
It looks like I have 4 workable methods to use:
Dave
- I like the fact that your solution can launch from a list of tasks - I may eventually get to the point of having a declarative batch setup. However, I have predefined parallel tracks, so some of the logic would need to be looked at for my need. There's not enough good information out there on how to implement parallel batch synchronization. Thanks for posting!
Antonio
- I like the simplicity of your solution. Why didn't I think of that?
I guess I was trying to avoid using the file system. But the simplest solution often has the least opportunity for failure!
penpen
- I'm not sure I understand the solution - I'm not a batch expert, and would have thought it launches A.bat, pipes the output from it to B.bat, etc., resulting in serialized execution of the jobs. But if this really launches all 5 in parallel (i.e. separate processes/threads) and the main process/thread only continues after they all finish - that would be exactly what I'd want!!!
My own using WAITFOR
- I wound up implementing a two-way handshake that seems to work - not because I didn't want to use any of the solutions above, but because I forgot to mark my post with "notify of follow-ups" and in my impatience I kept working on my initial approach
and I didn't see the replies until today. I'm not sure I can declare my approach completely safe or production quality (fortunately, I don't have to worry about that), but it has worked perfectly in testing my PoC and for several dev process runs at this time. Below is what I'm doing currently. I use two sets of signals - a "private" (numbered) signal for each batch job running in parallel that is issued from the main batch job to instruct the parallel job that it can send its completion signal, and a shared completion signal that is sent from the parallel job(s) back to the main batch job to notify it of completion. The main batch job loops, issuing the "private" "ReadyToReceiveDoneSignal" to one parallel job at a time, waiting for the "SharedDoneSignal" long enough to allow the parallel job to respond. It then logs/gathers all the responses, sorts them, and compares the list of received responses to the list of all expected responses. I make extensive use of parameter passing to batch (environment) variables - so there are a few places I leverage this. The one thing I'm missing at this time, is to be able to return values back to the main batch job. But that is another issue.
I also log to a file (used to log the overall progression of my job). It didn't wind up being a simple approach, and it does add a few seconds to each wait (but effectively only one or maybe two wait periods per section that launches parallel jobs) which makes it unsuitable for fast-running (subsecond) batches. In my case, I'm talking minutes for most of the tasks, and a total of 1-2 hrs for the whole job.
I made a few hand-made adjustments in this post - hope I didn't break anything. The %ChooseEnv% is simply a way to allow multiple developers to use the same batch and logic, and to separate some things, such as logging.
I do like two things about my approach (I'm not partial of course
) - AFAIK it doesn't require any additional permissions (other than for logging), and it could potentially be modified to synchronize jobs on multiple computers. But it is more code than I had wanted to use in the first place.
In the parallel jobs:Code: Select all
...
REM after processing and error handling, before exiting, wait for "clear to send" and then send "completed"
IF "%Signal%"=="" (
ECHO Returning from %~n0 [sequential processing]
) ELSE (
ECHO Finished %~n0 [parallel processing] %Signal%
CALL :SignalDone %1 %2
EXIT
)
GOTO :eof
:SignalDone
WaitFor /t 120 %2
IF %ERRORLEVEL%==0 (
ECHO %~n0 Received Signal %2
) ELSE (
ECHO %~n0 Timed Out waiting for %2
GOTO :SignalDone
)
Timeout 1
ECHO %~n0 Sending %1 signal and exiting
waitfor /s 127.0.0.1 /si %1
GOTO :eof
Code: Select all
...
REM Initialize common part of Signal Names
SET ReadySignal=Ready
REM This is the expected "Completed" signal from each parallel task
SET SharedDoneSignal=Done
...
REM Initialize parallel job counter
SET SignNum=0
...
REM Sequential processing section ...
...
REM Parallel section:
REM First parallel job #
SET StartSigNum=%SigNum%
CALL :LaunchParallelBatNoExt A.bat
CALL :LaunchParallelBatNoExt B.bat
CALL :LaunchParallelBatNoExt C.bat
CALL :WaitForParallels %StartSigNum%
...
REM More processing - sequential and parallel as needed...
...
REM End of processing
GOTO :eof
:LaunchParallelBatNoExt
REM Clear ERRORLEVEL:
cmd /c "exit /b 0"
REM The parallel task will wait to be informed it can send the "completed" message
SET /A SigNum+=1
SET ReadyToReceiveDoneSignal=%ReadySignal%%SigNum%
REM This is the expected "Completed" signal from each parallel task
ECHO START "%1" %1.bat %SharedDoneSignal% %ReadyToReceiveDoneSignal% ^> %ChooseEnv%_%1.out" >> %CD%/%ChooseEnv%_LogRunSteps.log
START "%1" %1.bat %SharedDoneSignal% %ReadyToReceiveDoneSignal% > %ChooseEnv%_%1.out
IF NOT %ERRORLEVEL%==0 (
ECHO %TIME% - #### ERROR IN #### Launching Constructed Batch %1.bat >> %CD%/%ChooseEnv%_LogRunSteps.log
) ELSE (
ECHO %TIME% - ++++ SUCCESS ++++ Launched Constructed Batch %1.bat >> %CD%/%ChooseEnv%_LogRunSteps.log
)
GOTO :eof
:WaitForParallels
cmd /c "exit /b 0"
REM Starting Signal Number was last one used in prior set - or 0. It is not a Signal Number used in this parallel set.
SET Loop=%1
SET LastSigNum=%SigNum%
SET SignalsReceived=x0x
Set CountReceived=0
REM 0 is not a valid signal entry - just used to initialize variable...
:WaitLoop
SET /A Loop+=1
ECHO %SignalsReceived% | FINDSTR "x%Loop%x" > nul
IF %ERRORLEVEL%==1 (
Waitfor /s 127.0.0.1 /si %ReadySignal%%Loop%
REM Reset ErrorLevel:
cmd /c "exit /b 0"
REM Allow 5 seconds for round trip response...
WAITFOR /T 5 %SharedDoneSignal%
IF !ERRORLEVEL!==0 (
Set /A CountReceived+=1
SET SignalsReceived=%SignalsReceived%%Loop%x
) ELSE (
ECHO X%Loop% Did not Signal
)
) ELSE (
ECHO %ReadySignal%%Loop% was previously received
)
REM ECHO %SignalsReceived% Last Checked Signal %Loop% out of %LastSigNum%, CountReceived = %CountReceived% or !CountReceived!
IF %Loop% LSS %LastSigNum% GOTO WaitLoop
SET Loop=%1
SET OrderedSignalsReceived=x0x
SET AllSignals=x0x
:SortSignalsReceived
ECHO %SignalsReceived% | FINDSTR "x%Loop%x" > nul
SET /A Loop+=1
SET AllSignals=%AllSignals%%Loop%x
REM Reset ErrorLevel:
cmd /c "exit /b 0"
ECHO %SignalsReceived% | FINDSTR "x%Loop%x" > nul
IF %ERRORLEVEL%==0 SET OrderedSignalsReceived=%OrderedSignalsReceived%%Loop%x
REM ECHO Sorted Signals %OrderedSignalsReceived% Last Checked Signal %Loop%
IF %Loop% LSS %LastSigNum% GOTO SortSignalsReceived
ECHO Comparing %OrderedSignalsReceived%==%AllSignals%
IF %OrderedSignalsReceived%==%AllSignals% GOTO DoneWaiting
SET Loop=0
REM ECHO Next Iteration
GOTO WaitLoop
:DoneWaiting
ECHO All Signals received...
GOTO :eof
