wait for all before continue

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Post Reply
Message
Author
zpimp
Posts: 12
Joined: 14 Jun 2013 12:58

wait for all before continue

#1 Post by zpimp » 01 Jul 2013 09:03

i have a script wich downloads some pages with CURL

Code: Select all

for /l %%i in (1,1,12) do (
start /wait /min curl !_url%%i! -m 10 -o %%i.htm
)


is there a way to download at the same time, and also wait for all before continuing the script?
if i use start /wait they run one after another
if i use start it doesent wait for them to finish

Squashman
Expert
Posts: 4465
Joined: 23 Dec 2011 13:59

Re: wait for all before continue

#2 Post by Squashman » 01 Jul 2013 10:04

Untested. Just the first thing that came to my head.

Code: Select all

set cnt=0
for /l %%i in (1,1,12) do (
start /min curl !_url%%i! -m 10 -o %%i.htm && SET /a cnt+=%%I
)
:LOOP
IF %cnt% LSS 12 GOTO LOOP

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: wait for all before continue

#3 Post by dbenham » 01 Jul 2013 12:03

@squashman - That can't work. It is simply incrementing the count if the curl program was launched. It does not wait for the curl program to finish. And if it did wait, then the processes would run seraially, which defeats the whole purpose :?

@zpimp - Have a look at Parallel execution of shell processes on StackOverflow. It is doing more than you ask for, but you can extract ideas from the script to do exactly what you want.

Dave Benham

Squashman
Expert
Posts: 4465
Joined: 23 Dec 2011 13:59

Re: wait for all before continue

#4 Post by Squashman » 01 Jul 2013 12:34

@Dave - I suppose your right. I assumed it would not execute until the command finished completely.

How about using the TITLE as part of the start command. Call it some arbitrary title for all of them like CURLBATCH. Then use Tasklist to see if they are still running and check the errorlevel.

Code: Select all

for /l %%i in (1,1,12) do (
start "CURLBATCH %%i" /min curl !_url%%i! -m 10 -o %%i.htm
)
:LOOP
tasklist /v | find "CURLBATCH"
IF "%ERRORLEVEL%"=="0" GOTO LOOP

Aacini
Expert
Posts: 1885
Joined: 06 Dec 2011 22:15
Location: México City, México
Contact:

Re: wait for all before continue

#5 Post by Aacini » 01 Jul 2013 18:39

Code: Select all

for /l %%i in (1,1,12) do (
echo X > curlRunning.%%i
start /min curl !_url%%i! -m 10 -o %%i.htm ^& del curlRunning.%%i
)
:waitForNone
if exist curlRunning.* goto waitForNone

Squashman
Expert
Posts: 4465
Joined: 23 Dec 2011 13:59

Re: wait for all before continue

#6 Post by Squashman » 01 Jul 2013 19:00

Aacini wrote:

Code: Select all

for /l %%i in (1,1,12) do (
echo X > curlRunning.%%i
start /min curl !_url%%i! -m 10 -o %%i.htm ^& del curlRunning.%%i
)
:waitForNone
if exist curlRunning.* goto waitForNone

Is that not just the same kind of logic as my 1st attempt that Dave said would not work? :?:

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: wait for all before continue

#7 Post by dbenham » 01 Jul 2013 20:46

@Squashman - No, Aacini's strategy works well. The & is escaped, so the DEL command is executed in the new process after the curl command finishes. That strategy can't work with SET because your parent process cannot see the environment changes that the child process(es) make.

@Aacini - Nice simple solution. You might want to throw a TIMEOUT or PING delay in your tight file check loop to conserve CPU resources. One risk of your approach is that your parent process will hang indefinitely if the child process is terminated before it gets a chance to delete the signal file. The link I provided shows a more robust solution. But the robust solution is much more complicated :twisted:

Here is another relatively simple approach. WMIC can be used to launch a process and return the PID of the new process. Then TASKLIST can be monitored to see if any of the launched processes are still running.

Code: Select all

for /l %%i in (1 1 12) do (
  for /f "tokens=2 delims==; " %%A in (
    'wmic process call create "c:\curlPath\curl.exe !url%%i! -m 10 -o c:\outpuPath\%%i.htm"^|findstr ProcessId'
  ) do set "$checkPID%%A=1"
)

:wait
timeout 1 /nobreak >nul
for /f "tokens=2" %%A in (
  'tasklist /fi "imagename eq curl.exe" /nh'
) do if defined $checkPID%%A goto :wait

endlocal
echo Done!


Dave Benham

zpimp
Posts: 12
Joined: 14 Jun 2013 12:58

Re: wait for all before continue

#8 Post by zpimp » 02 Jul 2013 15:24

i would normally use tasklist, but i have encountered problems with it
@Aacini i did something similar but your code is better

i found a program that can do what i want (written in pure c, no dependencies, 9kb, source code included)
not made by me
http://rghost.net/47175763

so now i only have to do this in my script "wfc /ee=curl.exe exit"

Squashman
Expert
Posts: 4465
Joined: 23 Dec 2011 13:59

Re: wait for all before continue

#9 Post by Squashman » 11 Oct 2017 14:29

Aacini wrote:

Code: Select all

for /l %%i in (1,1,12) do (
echo X > curlRunning.%%i
start /min curl !_url%%i! -m 10 -o %%i.htm ^& del curlRunning.%%i
)
:waitForNone
if exist curlRunning.* goto waitForNone

Linked your post to this question on SO.

Post Reply