Page 1 of 2
need a script to split inputs to a-m and n-z for robocopy
Posted: 20 Nov 2012 16:46
by redclawkefar
Ok, let me explain a little... I run robocopy in a script that figures out the day and parses a csv file for variables to plug into robocopy. That original script I wrote is here at the AMD forums :
http://forums.amd.com/forum/messageview.cfm?catid=12&threadid=82530&highlight_key=yI created the script for my thesis backups where I had image renders on different days but wanted to have ALL results for a given week available for review. That script saved me hours and kept me somewhat sane... now I need the script to evolve a bit.
What I need to do now is have a script that will tell robocopy to only process the folders that start with a-m in the main folder (a second bat file will have n-z; also assume 0-9 a non-issue.)
To be exact, I have an archive server with 100's of folders with varying names in the main folder. I need to backup all folders (and their sub-folders) from that root directory, that start with letters a to m, to a backup drive. After that is complete, I'll run a second script for folders n to z. This is due to the fact that the backup drive needs to be swapped out because of size limitations and cheap upper management personnel.
I thought maybe something that would use the "/xd" function (robocopy function for directory exclusion) to exclude the unwanted n-z directories. That didn't pan out too well though because it excluded sub-folders that matched the variable.
So, any ideas?
Re: need a script to split inputs to a-m and n-z for robocop
Posted: 20 Nov 2012 17:01
by RogerSmith
a for statment should work too but i coded like this....
is this what you want?
Code: Select all
robocopy /E c:\FromHere c:\ToHere
del c:\ToHere\n\*.* /q
del c:\ToHere\o\*.* /q
del c:\ToHere\p\*.* /q
del c:\ToHere\q\*.* /q
del c:\ToHere\r\*.* /q
del c:\ToHere\s\*.* /q
del c:\ToHere\u\*.* /q
del c:\ToHere\v\*.* /q
del c:\ToHere\x\*.* /q
del c:\ToHere\y\*.* /q
del c:\ToHere\z\*.* /q
rd c:\ToHere\z\
rd c:\ToHere\y\
rd c:\ToHere\x\
rd c:\ToHere\v\
rd c:\ToHere\u\
rd c:\ToHere\s\
rd c:\ToHere\r\
rd c:\ToHere\q\
rd c:\ToHere\p\
rd c:\ToHere\o\
rd c:\ToHere\n\
Edited: I realyze that this fails in a lot of conditions
sorry i will think in a good solution
from a to m how many folders do you have, that number can be diferente or maintain always that number of folders?
Re: need a script to split inputs to a-m and n-z for robocop
Posted: 20 Nov 2012 19:12
by Aacini
I think Robocopy can not exclude
folders based on a wildcard. Another possibility is to create the list of folders to process, so in this case they may be processed with any program you wish. Try this solution:
Code: Select all
@echo off
rem 97=a, 109=m, 110=n, 122=z
set AsciiCode=97
:nextLetter
cmd /C exit %AsciiCode%
echo ===============================
echo Processing letter: %=ExitCodeAscii%
for /D %%a in (%=ExitCodeAscii%*) do (
echo Processing folder: %%a
rem Process here the folder %%a
)
set /A AsciiCode+=1
if %AsciiCode% leq 109 goto nextLetter
Or a more traditional solution:
Code: Select all
@echo off
setlocal EnableDelayedExpansion
set letters=abcdefghijklmnopqrstuvwxyz
rem 0=a, 12=m, 13=n, 25=z
for /L %%l in (0,1,12) do (
set letter=!letters:~%%l,1!
echo ===============================
echo Processing letter: !letter!
for /D %%a in (!letter!*) do (
echo Processing folder: %%a
rem Process here the folder %%a
)
)
I hope it helps...
Antonio
Re: need a script to split inputs to a-m and n-z for robocop
Posted: 21 Nov 2012 08:59
by redclawkefar
thank you both for ideas... i'll try them over the next couple days while the office is closed so i don't muck up people's work.... not that some of them need help mucking up their work
edit:
i looked into the
exclude folders options... it seems that IS a possibility, for example, I have a script that successfully exclude folders labeled something like the following:
x:\zzzzzz - blahblah\
x:\zzzzzzz - blehblah\
x:\zzzzzzzz - blihblah\
x:\zzzzzzzzz - blackblah\
that command line with robocopy went like (with all my switches set; notice the
/xd switch):
Code: Select all
robocopy x:\ d:\ /FFT /V /TEE /S /E /COPY:DT /PURGE /MIR /NP /R:0 /W:0 /xd *recycle* "*system volume*" ZZZZ* /xf thumbs.db *.nef /log:active-firefile.log /eta
Re: need a script to split inputs to a-m and n-z for robocop
Posted: 21 Nov 2012 16:34
by Aacini
I have not Robocopy, so I can not make any test. I based my comment on this example:
Code: Select all
/XF file [file]... : eXclude Files matching given names/paths/wildcards.
/XD dirs [dirs]... : eXclude Directories matching given names/paths.
XF and XD can be used in combination e.g.
ROBOCOPY c:\source d:\dest /XF *.doc *.xls /XD c:\unwanted /S
I found at
this site. If /XD option works on wildcards, then the solution is much simpler:
Code: Select all
@echo off
setlocal EnableDelayedExpansion
rem Assemble list of unwanted folders
set unwanted=
for %%a in (n o p q r s t u v w x y z) do set unwanted=!unwanted! %%a*
rem Do the copy
robocopy previous params and switches /XD %unwanted%
Re: need a script to split inputs to a-m and n-z for robocop
Posted: 28 Nov 2012 16:59
by redclawkefar
ok, so that last code works... kinda... unfortunately i have folders that have the same sub structure. for example below:
x:\cab\articles
x:\cab\documents
x:\cab\library
x:\cab\magazines
x:\cab\news
x:\cab\reports
x:\cab\tests
.
.
.
x:\time\articles
x:\time\documents
x:\time\library
x:\time\magazines
x:\time\news
x:\time\reports
x:\time\tests
if i run the script, it will only copy/backup folders matching the letters of the "wanted" folders and skip the other folders even though they are subfolders. i need the script to just identify the root folder alphabet (eg 'cab' or 'time') and not "unwant" the subfolders? so my result after running on the above would be:
x:\cab\articles
x:\cab\documents
x:\cab\library
x:\cab\magazines
and i need it to be:
x:\cab\articles
x:\cab\documents
x:\cab\library
x:\cab\magazines
x:\cab\news
x:\cab\reports
x:\cab\tests
my test setup i'm using is the 50 US state initials, with the same folders as sub folders.
an ideas how to limit the for loop to root folders or maybe even a script to write a list of the "unwanted" root folders, that would work too...
Re: need a script to split inputs to a-m and n-z for robocop
Posted: 28 Nov 2012 19:00
by foxidrive
This is untested for folders a-m
Code: Select all
@echo off
pushd "x:\"
for %%a in (a b c d e f j h i j k l m) do (
for /f "delims=" %%b in ( ' dir /ad /b "%%a*" ' ) do (
robocopy "%%b" "d:\%%b" /FFT /V /TEE /S /E /COPY:DT /PURGE /MIR /NP /R:0 /W:0 /xd *recycle* "*system volume*" ZZZZ* /xf thumbs.db *.nef /log:active-firefile.log /eta
)
)
popd
Re: need a script to split inputs to a-m and n-z for robocop
Posted: 28 Nov 2012 21:37
by redclawkefar
That seemed to have copied everything...
Re: need a script to split inputs to a-m and n-z for robocop
Posted: 29 Nov 2012 01:33
by foxidrive
Did you check your log to see if it copied root folders above M ?
Try this - it will only echo the robocopy commands so you can check what is being sent in the command string.
Code: Select all
@echo off
pushd "x:\"
for %%a in (a b c d e f j h i j k l m) do (
for /f "delims=" %%b in ( ' dir /ad /b "%%a*" ' ) do (
ECHO robocopy "%%b" "d:\%%b" /FFT /V /TEE /S /E /COPY:DT /PURGE /MIR /NP /R:0 /W:0 /xd *recycle* "*system volume*" ZZZZ* /xf thumbs.db *.nef /log:active-firefile.log /eta
)
)
popd
pause
Re: need a script to split inputs to a-m and n-z for robocop
Posted: 29 Nov 2012 16:05
by redclawkefar
well it's doing what i want even if the log needs to be appended... so the slightly modded code:
Code: Select all
@echo off
DEL ARCHIVE.LOG
pushd "D:\TESTING\TS"
for %%a in (a b c d e f j h i j k l m) do (
for /f "delims=" %%b in ( ' dir /ad /b "%%a*" ' ) do (
D:\TESTING\robocopy.exe "%%b" "D:\TESTING\result\%%b" /FFT /V /TEE /S /E /COPY:DT /PURGE /MIR /NP /R:0 /W:0 /xd *recycle* "*system volume*" ZZZZ* %unwanted% /xf thumbs.db *.nef /log+:D:\TESTING\ARCHIVE.log /eta
)
)
popd
spits out and appends the following into the ARCHIVE.log file for each directory from a-m (which is why the
DEL ARCHIVE.LOG is there):
Code: Select all
-------------------------------------------------------------------------------
ROBOCOPY :: Robust File Copy for Windows :: Version XP026
-------------------------------------------------------------------------------
Started : Thu Nov 29 13:42:44 2012
Source : D:\TESTING\ts\AK\
Dest : D:\TESTING\result\AK\
Files : *.*
Exc Files : thumbs.db
*.nef
Exc Dirs : *recycle*
*system volume*
ZZZZ*
Options : *.* /FFT /V /TEE /S /E /COPY:DT /PURGE /MIR /NP /ETA /R:0 /W:0
------------------------------------------------------------------------------
0 D:\TESTING\ts\AK\
0 D:\TESTING\ts\AK\AK\
0 D:\TESTING\ts\AK\AL\
0 D:\TESTING\ts\AK\AR\
0 D:\TESTING\ts\AK\AZ\
0 D:\TESTING\ts\AK\CA\
0 D:\TESTING\ts\AK\CO\
0 D:\TESTING\ts\AK\CT\
0 D:\TESTING\ts\AK\DC\
0 D:\TESTING\ts\AK\DE\
0 D:\TESTING\ts\AK\FL\
0 D:\TESTING\ts\AK\GA\
0 D:\TESTING\ts\AK\HI\
0 D:\TESTING\ts\AK\IA\
0 D:\TESTING\ts\AK\ID\
0 D:\TESTING\ts\AK\IL\
0 D:\TESTING\ts\AK\IN\
0 D:\TESTING\ts\AK\KS\
0 D:\TESTING\ts\AK\KY\
0 D:\TESTING\ts\AK\LA\
0 D:\TESTING\ts\AK\MA\
0 D:\TESTING\ts\AK\MD\
0 D:\TESTING\ts\AK\ME\
0 D:\TESTING\ts\AK\MI\
0 D:\TESTING\ts\AK\MN\
0 D:\TESTING\ts\AK\MO\
0 D:\TESTING\ts\AK\MS\
0 D:\TESTING\ts\AK\MT\
0 D:\TESTING\ts\AK\NC\
0 D:\TESTING\ts\AK\ND\
0 D:\TESTING\ts\AK\NE\
0 D:\TESTING\ts\AK\NH\
0 D:\TESTING\ts\AK\NJ\
0 D:\TESTING\ts\AK\NM\
0 D:\TESTING\ts\AK\NV\
0 D:\TESTING\ts\AK\NY\
0 D:\TESTING\ts\AK\OH\
0 D:\TESTING\ts\AK\OK\
0 D:\TESTING\ts\AK\OR\
0 D:\TESTING\ts\AK\PA\
0 D:\TESTING\ts\AK\RI\
0 D:\TESTING\ts\AK\SC\
0 D:\TESTING\ts\AK\SD\
0 D:\TESTING\ts\AK\TN\
0 D:\TESTING\ts\AK\TX\
0 D:\TESTING\ts\AK\UT\
0 D:\TESTING\ts\AK\VA\
0 D:\TESTING\ts\AK\VT\
0 D:\TESTING\ts\AK\WA\
0 D:\TESTING\ts\AK\WI\
0 D:\TESTING\ts\AK\WV\
0 D:\TESTING\ts\AK\WY\
------------------------------------------------------------------------------
Total Copied Skipped Mismatch FAILED Extras
Dirs : 52 0 52 0 0 0
Files : 0 0 0 0 0 0
Bytes : 0 0 0 0 0 0
Times : 0:00:00 0:00:00 0:00:00 0:00:00
Ended : Thu Nov 29 13:42:44 2012
unfortunately it doesn't do all the files/folders at once with a final summary but rather treats each folder as a new robocopy instance, thus creating subsequent 'summaries' which would be a chore to read each time. the "live" scenario it's intended for has over 1000 root folders and i don't want to look at 1000 summaries but rather just 1.
--------------
so... this works to a large degree and if there is no other way than i'll test this on a semi-live setting... otherwise i'd like to explore other possible solutions...
if we could generate a variable that has a list of all folders to exclude, then read that variable into the /xd switch, then i think we might have what we need.
thanks for all the help too btw!
Re: need a script to split inputs to a-m and n-z for robocop
Posted: 30 Nov 2012 02:58
by foxidrive
Here's what you asked for - it populates an XD variable with the exclusions and will run in one pass.
Using % and ^ characters in folder names will cause WWIII to start so it's best to avoid them.
Of note is that false matches from the short folder names in eliminated by the use of findstr
It just echos the robocopy command atm - untested:
Code: Select all
@echo off
del D:\TESTING\ARCHIVE.log 2>nul
pushd "D:\TESTING\TS"
for /f "delims=" %%a in (' dir /ad /b ^| findstr "^[n-zN-Z]" ') do call set xd=%%xd%% "%%~dpa%%a"
echo EXCLUSIONS ARE: %xd%
echo D:\TESTING\robocopy.exe "." "D:\TESTING\result\" /FFT /V /TEE /S /E /COPY:DT /PURGE /MIR /NP /R:0 /W:0 /xd %xd% *recycle* "*system volume*" ZZZZ* %unwanted% /xf thumbs.db *.nef /log+:D:\TESTING\ARCHIVE.log /eta
popd
pause
Re: need a script to split inputs to a-m and n-z for robocop
Posted: 30 Nov 2012 07:43
by Aacini
The Batch subroutine below create a list of all existent folders in the current subdirectory that match a given subrange of letters:
Code: Select all
:makeXDlist begin end
set xdList=
for /F "tokens=2 delims=%1%2" %%a in (" a b c d e f g h i j k l m n o p q r s t u v w x y z ") do (
for %%b in (%1 %%a %2) do (
for /D %%c in (%%b*) do call set xdList=%%xdList%% "%%~Fc"
)
)
exit /B
For example, to create a list of folders in N-Z range:
After that you may use the %xdList% variable in Robocopy to exclude these folders.
I hope it helps...
Antonio
Re: need a script to split inputs to a-m and n-z for robocop
Posted: 04 Dec 2012 18:30
by redclawkefar
hi foxidrive/antonio....
ok, so i've tried both approaches, and i could use both in different servers... foxidrive's solution works, minor tweaking to set directories in the "pushd" but it works like a charm.
antonio, your's seems to loop for some odd reason and in the second pass it wipes the exclusion parameter and the copies the whole directory. it basically does 2 passes, one with all the right exclusions, then again with no exclusions and up 1 directory level, so d:\testing not d:\testing\ts, and copies all that's in d:\testing to d:\testing\result
below is the code setup i have
Code: Select all
@echo off
del D:\TESTING\ARCHIVE.log 2>nul
pushd "D:\TESTING\TS"
call :makeXDlist n z
:makeXDlist begin end
set xdList=
for /F "tokens=2 delims=%1%2" %%a in (" a b c d e f g h i j k l m n o p q r s t u v w x y z ") do (
for %%b in (%1 %%a %2) do (
for /D %%c in (%%b*) do call set xdList=%%xdList%% "%%~Fc"
)
)
echo EXCLUSIONS ARE: %xdList%
echo D:\TESTING\robocopy.exe "." "D:\TESTING\result" /FFT /V /TEE /S /E /COPY:DT /PURGE /MIR /NP /R:0 /W:0 /xd %xd% *recycle* "*system volume*" ZZZZ* %xdList% /xf thumbs.db *.nef /log+:D:\TESTING\ARCHIVE.log /eta
popd
pause
i had to remove the "exit /B" line or it would obviously exit before it could execute the rest. incidentally, robocopy is built into windows 7 and server 2008 if you have access to them you can grab it. located in C:\Windows\System32
oh so close to perfection.... gonna keep playing with the code unless you guys see a glaring mistake i made.
Re: need a script to split inputs to a-m and n-z for robocop
Posted: 04 Dec 2012 18:52
by Aacini
redclawkefar wrote:i had to remove the "exit /B" line or it would obviously exit before it could execute the rest...
I can't belive this is the first
internal subroutine you use, but it seems that it is... When the subroutine is placed in the same file of the main program, you must be careful to not execute it by mistake! In your code, after the CALL command ends and the subroutine returns,
it is executed again with bad parameters!. Of course, you must NOT remove the subroutine's "exit /B" line, but place the whole subroutine in the right place:
Code: Select all
@echo off
del D:\TESTING\ARCHIVE.log 2>nul
pushd "D:\TESTING\TS"
call :makeXDlist n z
echo EXCLUSIONS ARE: %xdList%
echo D:\TESTING\robocopy.exe "." "D:\TESTING\result" /FFT /V /TEE /S /E /COPY:DT /PURGE /MIR /NP /R:0 /W:0 /xd %xd% *recycle* "*system volume*" ZZZZ* %xdList% /xf thumbs.db *.nef /log+:D:\TESTING\ARCHIVE.log /eta
popd
pause
rem Very important: do NOT remove next line:
GOTO :EOF
rem If previous line does not exist, the following subroutine will be executed again
rem when the main program ends, but with THE SAME parameters of the Batch file!
:makeXDlist begin end
set xdList=
for /F "tokens=2 delims=%1%2" %%a in (" a b c d e f g h i j k l m n o p q r s t u v w x y z ") do (
for %%b in (%1 %%a %2) do (
for /D %%c in (%%b*) do call set xdList=%%xdList%% "%%~Fc"
)
)
rem Important: do NOT remove next line
exit /B
rem In this case previous line is not crucial. However, if you add another
rem internal subroutine below this line, AND YOU FORGET TO INCLUDE PREVIOUS EXIT /B,
rem the next subroutine will be executed when the previous one ends!
Please, test it this way and report the result...
Antonio
Re: need a script to split inputs to a-m and n-z for robocop
Posted: 05 Dec 2012 17:52
by redclawkefar
ahhh yes... that worked.
and yes, i can admit it, i have not actually ever 'called' an internal command before to execute. even though i did use 'call' in my original script for robocopy (the one linked to in the OP), at that time i wasn't fully understanding some of what i wrote.
in either case, THANK YOU both for helping me resolve my issue!
i'll post back after i run a live script next week... i don't foresee any problems but who knows.
tanks again,
mark