list files from network - parallel process - need help

[brinda]

hi all,

using win2000.
This is the current method being used. A directory list is done one by one based on the path.txt sequence as below.

Code: Select all

echo on
for /f "usebackq delims=|" %%e in (path.txt) do (
  dir /a:-d /b /s "%%e"  >> list.dat
)

partial content of path.txt

Code: Select all

\\192.168.1.2\backup\current\log
\\mach_back\live\log
\\192.168.1.3\backup2\month2\log
\\serv_2\updated\log

Is there a method where the above process could run in parallel in a single batch file? Sequence is not important.

I went through for the past few weeks on dave' antonio's ed's including stackoverflow
http://www.dostips.com/forum/viewtopic.php?p=12508#p12508
http://www.dostips.com/forum/viewtopic.php?f=3&t=3107

the more i read, the more i am getting confused

The only working method at the moment is hard code the path in each separate batch file like below

Code: Select all

  dir /a:-d /b /s "\\192.168.1.2\backup\current\log"  >> list1.dat

and ran them by double click on windows. Running 4 batch/windows at a time. the end result files are concatenate

Code: Select all

copy *.dat combine.dat

.

currently going through the link below on the sixth time. It comes close, interesting is that the files are written on a single file. concatenate of files are not needed in the end. but do not get where to replace and edit the original code into this. lots of trial and error are still ongoing

Code: Select all

http://stackoverflow.com/questions/9337415/how-do-you-have-shared-log-files-under-windows

[foxidrive]

Edited:

This will open a cmd window for each path and the pause command will display - when all the cmd windows close (which are collecting the DIR listings) press space and it will copy the tmp files into a log file.

This might work.

Code: Select all

@echo off
setlocal enabledelayedexpansion
for /f "delims=" %%e in (path.txt) do (
set "log=%%e"
set "log=!log:\=_!.log.tmp"
start "" dir /a:-d /b /s "%%e"  > "!log!"
)
pause
copy *.log.tmp file.log
del *.log.tmp

[brinda]

foxidrive,

sorry. some issue.
It opened 6 cmd windows for 6 path listing. the cmd windows remains open after it finish listing.

6 temporary files were created.

content of file.log is a box - single character. On scite it is showing SUB. Believe this would be CTRL-Z

Is there a way where only 4 windows at time could be used. There are more than 50 paths in path.txt

On usebackq delims=|
read in stackoverflow on a sample for long path to be used. Just applied. guess i need to learn more example on delims and usebackq delims. I will retry on the long paths with delims= and check.

sample of long path with space

Code: Select all

\\192.168.1.8\backup2\month2\misc updates\doc 1

[foxidrive]

See my edited post above. It's a bit of a kludge because you have to press a key when all the cmd windows close, to create the log file.

It may not be worth creating logic to run 4 windows at a time - if your machine can handle 50 windows to perform the DIR commands.

[dbenham]

Create a "lock" file for each process and have process delete the lock file when DIR is finished. I use the START /B option so each process runs in same window.
Each process must get its own temporary output file, since only one process can write to a given file at a time.

After all processes are launched, begin a loop checking for "lock" files. If none exist, then all processes are complete.
Copy temp files to master output and delete temp files.

I used TIMEOUT to introduce a delay. Change to PING delay if TIMEOUT not on your machine.

Code: Select all

@echo off
setlocal enableDelayedExpansion
set "pathFile=path.txt"
set "listFile=list.dat"
set "tempList=%temp%\parallel_List.txt"
set "tempLock=%temp%\parallel_Lock.txt"

set /a n=0
for /f "usebackq delims=|" %%P in ("%pathFile%") do (
  set /a n+=1
  copy nul "%tempLock%.!n!" >nul
  start "" /b cmd /c ^"dir /a-d /b /s "%%~P" >"%tempList%.!n!" ^& del "%tempLock%.!n!"^"
)

:loop
timeout 1 /nobreak >nul
if exist "%tempLock%.*" goto :loop

copy /b "%tempList%.*" "%listFile%" >nul
del "%tempList%.*"
exit /b

Dave Benham

[brinda]

foxidrive,
thanks. the pause is only on one windows. not a kludge but a good method for me to learn start function can do. thanks for helping.

dave,
sorry. little problem

no timeout used, so using

Code: Select all

PING -n 2 127.0.0.1>nul

This equals to 1 second delay.

A doubt, ran a sample on 25 network link in parallel at once. These are archive links. So there would not be any new files added or removed. time is cut short nearly 3/4. WOW


The only issue i am facing is, each time - there would be a file missing from the huge list on compare. This is seen on the created size and number of lines difference. Out of 12 times, there were 2 instances where the file is missing from list. The file was not accessed or lock by any process.

Have tried to increase the delay to -n5 but its random.

The older methods gets 100 percent of the file all the time.

Probably, its something else. If i were to try this on the computer itself , getting files from folder - than it gets the files 100 percent of the time. no miss files.

Going through the network is causing this issue - randomly. I'll try to redo this from different pc and let you know my findings.

thanks for showing this code. CPU is always around 22 percent or below on running

[brinda]

found the culprit. the network cable has an issue. ping -t shows dropped reply back intermittenly. change cable did solve the issue. already run the script 14 times and it shows all files as it should be.

thanks dave