Merge and rename duplicate files to incr by 1 ex: fn (1).ext

[ESerres]

Hello,
I'm a stay-at-home-homeschooling-mom and need help creating the following batch file for my home computers.

Although I used to make a living writing programs years and years ago, this is my first time to write DOS batch files (and the first time to ever post on a forum), so I am not familiar with all commands and syntax, but have been reading and looking at your programs, but couldn't find one to do what I am trying to do.

I need to merge directories from several old hard drives onto a new home NAS but don't have time right now to go through the files individually because there are several TB of data - (a lot of duplication from backups of old computers). I want to keep all unique files - they may have same name but different size/date, but delete duplicates - same name, same size, same date.

I would use robocopy, but it doesn't seem to handle the scenario of wanting to keep files with duplicate names and rename them to a filename with an incremental number. Here is my logic for what I am trying to code:


For each folder and sub folder in source
If folder does not exist in destination, move (or copy if on different drives) the folder and all sub-folders and files to destination (keep existing directory structure)
(Write to logfile here to note the folder was moved)
If folder already exists, then (merge the files into one directory)
For each file in source
If exist in destination
Check date and size
If different, move but rename to filename (1 or next available number).ext
example: src\my Text File.txt to dest\my Text File (1).txt
(would like to compare to other files with incremented name and make sure not to add it if it already exists with an incremented name -- compare date/size/bites before adding "(2)" version if "(1)" version alraedy exists )
(Write to logfile here to note the file was moved and renamed)
If same date and size then do binary compare the files to make sure both are good
If the compare fails (the files are slightly different), move the file but rename to "filename (1 or next available number).ext"
and if same move to "processed - ready to delete" folder [for now, but will eventually delete files instead of moving]
(Write to logfile here to note the file was "duplicate - deleted")
else (not exist)
create correct subdirectory and Move file
(Write to logfile here to note the file was moved)

And finally, I would like to be able to run the file from a command prompt like:
c:\> MergeDirs C:\sourcedir d:\destdir


Any help would be greatly appreciated. I am really surprised that this kind of batch file doesn't already exist on a forum somewhere. If it does, I couldn't find it. I know that most of the time people want to overwrite duplicates, but I can see lots of uses for this to do a quick merge of drives or directories and
allow a user to cleanup their files later.

For testing I created several .txt files with names like testFileA.txt, Test File B.txt and put them in a couple of subdirectories. I had most of it working, but when I added spaces to the filenames, it didn't work, so started playing around to understand how to use quotes and broke most of everything.
Sorry, but I have lots of REM and ECHOs so I could learn how to do this.

*** Edited: See improved code in later post.***

[foxidrive]

Welcome to the forum, and we do enjoy batch files.

For your case - and this appears to be a one time task - it is not worth writing a batch file and debugging it, for the entire task.
Also, excuse me for not reading your batch code but if it is faulty then studying it to find errors is a large task.

What I would suggest is that you is:

A) create a directory on the NAS drive and then copy every source folder tree into this folder, complete and as it stands, in separate folders.
B) Then you can run a program over the entire tree to remove binary duplicates (and have the source files as backup for the entire operation).

C) After that is done there is a GUI file manager which will allow you to merge folders, and while merging if any filename clashes occur then you can select the rename feature to continue merging the folder but rename filename clashes. I use Total Commander for this, which is fully functional shareware, and it has several options for renaming the filename clashes while moving.

Does this seem reasonable?

[ESerres]

No, I really want a batch file. I have at least seven 1.5 TB drives sitting in my floor with thousands of files on each one; movies, scanned pictures, documents, etc, but don't want to trash yet. There are too many files for me to do this manually. I had started doing it manually, but when I hit a directory with 38,000 files, I knew that was not the way to go.

Plus I have several computers in my house and depending on where I am for that time of the day, I create files and modify them on that computer. While my intent is to work on the NAS in the future and not have the files locally, I may find that my NAS is too slow for doing OCR or graphics and may have to work locally on files. In that case, it would be nice to run a batch at the end of the day or week to merge all the files onto my NAS. Many times I like to keep old versions, especially of graphics, because I often mess up one and have to go back to an earlier version.

Thanks anyway, I will keep working on it myself and try to get it going.

[foxidrive]

No, I really want a batch file. I have at least seven 1.5 TB drives sitting in my floor with thousands of files on each one; movies, scanned pictures, documents, etc, but don't want to trash yet. There are too many files for me to do this manually. I had started doing it manually, but when I hit a directory with 38,000 files, I knew that was not the way to go.

There is little manual interaction. Most of the task is simply copying, and for the 7 or more folders (one per hard drive) you will have one interaction for each hard drive folder tree - to tell Total Commander to rename the files with filename clashes during the move. That's 7 interactions for 7 hard drives.

The program that finds and deletes binary identical files in the entire collection of files is one extra task, and never needs to be done again.

[abc0502]

I made this one but there is a problem when the file is exist and has same size and modified date and time.
i must make the copy command not overwrite so i use this:

Code: Select all

Echo n |Copy "%from%" "%to% 

but it doesn't work in the for command or there is something wrong.

This is the code:

Code: Select all

@Echo OFF

SET "from=%userprofile%\desktop\HDD2"
SET "to=%userprofile%\desktop\Destination"

:: Create Destination Folder if not Exist
IF not Exist "%to%" MD "%to%" >nul

:: Get Length of the characters in "from" variabl to remove them later
:: we will add extra number "1" to the length to avoid problems later starting from line 17
For /F "delims=" %%A in ("%from%") Do Call :StrLen "%%~dpnA" "from_len"
SET /A from_len += 1

:: Create Folder Structure from "Source" to "Destination"
:: This will ignore any existing duplicate folder names
Xcopy "%from%" "%to%" /T /E

SETLOCAL EnableDelayedExpansion
SET "counter=1"
For /F "delims=" %%A In ('DIR /B /S /A:-D "%from%"') Do (
   :: Full_Path without File names which will be %%~nxA
   SET "Full_Path=%%~dpA"
   :: Check Duplicates - Name, Date/Time and Size, If one is not the same, it add extra number at the end of the name.
   IF Exist "%to%\!Full_Path:~%from_len%!%%~nxA" (
      Call :Info "%to%\!Full_Path:~%from_len%!%%~nxA" "DT" "Size"
      IF "!Size!" EQU "%%~zA" (
         IF "!DT!" EQU "%%~tA" (
            echo n |copy "%from%\!Full_Path:~%from_len%!%%~nxA" "%to%\!Full_Path:~%from_len%!%%~nxA"
         )
      ) Else (
         Copy "%from%\!Full_Path:~%from_len%!%%~nxA" "%to%\!Full_Path:~%from_len%!%%~nA_!counter!_%%~xA"
         SET /A Counter += 1 )
   ) Else ( Copy "%from%\!Full_Path:~%from_len%!%%~nxA" "%to%\!Full_Path:~%from_len%!%%~nxA" )
)
echo !counter!
pause
Exit /B

:StrLen <String> <Optional_Variable>
(
SETLOCAL ENABLEDELAYEDEXPANSION
Set "StrLenString=A%~1"
Set "StrLenLen=0"
For /L %%A in (12,-1,0) do (
   Set /a "StrLenLen|=1<<%%A"
   For %%B in (!StrLenLen!) Do IF "!StrLenString:~%%B,1!"=="" Set /a "StrLenLen&=~1<<%%A"
   )
)
(
ENDLOCAL
IF "%~2" NEQ "" SET /a "%~2=%StrLenLen%"
IF "%~2" EQU "" Echo %StrLenLen%
)
GOTO :EOF

:Info <file> <time/date_var> <size_var>
Set "Info_file=%~1"
For /F "delims=" %%A In ("%Info_File%") Do (
   Set "%~2=%%~tA"
   Set "%~3=%%~zA"
   )
GOTO :EOF

First, you set the variable then, check if the distenation folder exist or not.
After that i take the length of the source location as if it was "c:\folder" the length will be 9 (quotes not included)
Then, Create the Source Folder structure, without copying any files yet.

Note that the problem is in this section:
The code here goes through all folders and sub-folders in the source location looking for files & when it find one EX:"C:\folder\sub-folder\file.ext"

1> save it's full path (file name and extension not included) "C:\folder\sub-folder"
2> we already have the folder structure and the destination main folder so we check to see if the file.ext "%%~nxA" exist in the destination folder or not like this "%to%\!the_saved_path_from_step_1:~%length_of_source_paht%!%%~nxA,
we combine the destination location with the source location minus the characters "length we toke" from the source location to form the new full path of each file

and then use the info function to get the date and time modified and the size of the file and compare with the existing files, if they the same we pass the copy command with the Echo n
IF not the same we then add extra number at the end of the file and before the extension and copy the file.

I hope that is clear enough if any one wanted to modify any thing.

[abc0502]

just wanted to add, using batch files only to process big sizes like your's will take too much time, as for command has it's problem in iterating over and over through lists.
you can find command line tools that do the following:

1> Copy.
2> Remove Duplicates from Different folders
3> Merge Folders.

and use Batch files to automate giving the right commands in the right order to these tools.
and as Foxidrive said,

1> copy each HDD to a separate folder & all Folders in On HDD
2> use the Duplicate remover tool to remove all duplicate files from This HDD
3> use the Merge tool to merge all in one folder

[ESerres]

abc0502,
Thanks you for your help on this.

The reason I am doing a byte comparison is that some of these files could be corrupt files. I can not seem to keep my house cool enough to keep my computers happy -- I lose a hard drive in one of my computers about every six months, so sometimes a drive could be failing and the result is bad data in my more recent backups, where an older copy might still be OK. I don't want to override a good copy with a bad copy of data, so I am OK with slow rather than lose my good data.

[ESerres]

I have tested and fixed the code from abc0502 and it works great. The code you posted wasn't doing recurrive checks on existing files but it helped me figure out a lot of things I didn't know. One very important thing I learned that some REM and :: lines will still execute if special characters are included in them, causing errors and ERRORLEVEL items should be checked in reverse order.

Now I need to understand how to create a log file from all of my ECHO statements, and how to submit the directory names in a command statement like: c:>MergeDirectory M:\ScrDir M:DestDir M:DelDir /L or /M for move or list. Any help would be appreciated.

Here is the new, errorfree, revised code for anyone that is interested in this kind of batch file. I'm not sure how fast it will be - I want to be able to write to a log before running. Also, I am saving files to a DELETE folder instead of deleting them right away, but will get rid of that code once I feel comfortable that everything works great. Thanks for helping.

Code: Select all

@echo OFF
setLOCAL DisableDelayedExpansion
REM setLOCAL EnableDelayedExpansion
REM EnableDelayedExpansion is Disabled by default.


REM .. A BATCH WITH EXAMPLES TO HELP YOU LEARN DOS

REM .. comments with .. tell you what the code does in general english and sometimes why
REM ,, comments with ,, explain specific code syntax
REM -- comments with -- are lines of code that should have worked 
REM --                  but did not work because of specials characters so 
REM --                  they are usually followed with alternate code that did work


REM .. MergeDirs.bat: Merge a source tree into a destination tree, renaming duplicated files
REM .. MergeDirs C:\sourcedir D:\destdir C:\deletedDir
REM .. ListOrMove L=List M=Move

REM .. Get source and destination drives from parameters in the command line or calling batch file
  set "from=%1"
  set "to=%2"
  set "DeleteFiles=%3"
  set "ListOrMove=%4"

REM .. Delete quotes from front of directory names and back of directory names
REM .. In order to pass the directory names from the batch, quotes are used, but
REM .. the quotes will not work in later code, so they are removed here and added back
REM .. as needed

REM ,, for /F loops against a set of files or strings  delims=xxx delimiter character(s) (default = a space) 
REM ,, usebackq is used when you need to put double quotes around a filename - 
REM ,, it checks for double quotes and automatically removes them when processing the
REM ,, strings and placing the value into the token
REM ,, tokens=n Specifies which/how many variables will be needed to read items from 
REM ,, each line (default=1, * means automatically add as many as needed)

REM ,, so when your string is '"Filename"', usebackq causes this to be seen as 'Filename'

  for /F "usebackq tokens=*" %%a in ('%from%') do set from=%%~a
  for /F "usebackq tokens=*" %%a in ('%to%') do set to=%%~a
  for /F "usebackq tokens=*" %%a in ('%DeleteFiles%') do set DeleteFiles=%%~a


REM .. Delete trailing backslash "\" from directory names. Our code adds a "\"
REM .. between directories and filenames, so we need to make sure "\" deleted here.

REM ,, When you delete characters in a string you can use substrings to do it.  The 
REM ,, following code should have worked in the steps above to strip our quotes, but 
REM ,, the quote itself caused the code to not work, so we had to use the FOR loop 
REM ,, to remove the quotes in the code above.  Now we can use substrings to check a 
REM ,, trailing backslash at the end of each directory name and then remove it.
REM ,, The IF statement does not work for quotes in the string so use a FOR loop
REM -- THIS DOES NOT WORK BECAUSE OF QUOTES: if %from:~0,1%==" set from=%from:~0,1%

REM ,, to use substrings: %variable:~num_chars_to_skip, num_chars_to_keep% 
REM ,, ==> 0,1 keep first letter; -1 keep last letter
REM ,, if second num is blank keep everything after first num is found
REM ,, "-" means to start from the right and go backward, other wise strart from the left

  if %from:~-1%==\ set from=%from:~-1%
  if %to:~-1%==\ set to=%to:~-1%
  if %DeleteFiles:~-1%==\ set DeleteFiles=%DeleteFiles:~-1%


REM .. Use WMIC to retrieve date and time for use in log filename

REM ,, for /F loops against a set of files or strings  delims=x delimiter character(s)
REM ,, (in this case, use a comma) 
REM ,, skip=2 means to skip the first two lines of the strings that will be processed (default = 0)
REM ,, tokens=n Specifies which/how many variables will be needed to read items from 
REM ,, each line (default=1, * means automatically add as many as needed)
REM ,, In this case use the second item through the seventh item in the string
REM ,, %%A is a replaceable parameter (a variable name) so each item can be processed
REM ,, on the command line you would use %A instead of %%A used in this batch file
REM ,, WMIC Path Win32_LocalTime Get returns the date in a particluar format.
REM ,, Day^,Hour^,Minute^,Month^,Second^,Year is the list of expected items.  The ^ is
REM ,, needed because comma "," is the delimiter.
REM ,, /Format:csv returns the date string in a table format with items separated by commas.

  for /F "skip=2 tokens=2-7 delims=," %%A IN ('WMIC Path Win32_LocalTime Get Day^,Hour^,Minute^,Month^,Second^,Year /Format:csv') DO (
    REM ,, Use the /A (arithmetic) when calculating with numbers.  All parts of the above date string are numbers.
    set /A SortDate = 10000 * %%F + 100 * %%D + %%A
    set /A SortTime = 10000 * %%B + 100 * %%C + %%E
  )

REM .. Add a timestamp to the log filename; 
REM ,, %cd% puts the logfile in current directory the batch/command was 
REM ,, called/started from
  set "MergeLogFile=%cd%\MergeDirectoryLog-%SortDate%%SortTime%.txt"
  set "SkippedLogFile=%cd%\MergeDirectoryLog-%SortDate%%SortTime%-Skipped.txt"
  set "DeletedLogFile=%cd%\MergeDirectoryLog-%SortDate%%SortTime%-Deleted.txt"
  set "ErrorLogFile=%cd%\MergeDirectoryLog-%SortDate%%SortTime%-Errors.txt"
  set "MovedLogFile=%cd%\MergeDirectoryLog-%SortDate%%SortTime%-Moved.txt"


REM .. Make sure the source directory exists before executing the batch job
  if NOT Exist "%from%" (
    echo "Directory %from% does not exist.  Please enter a valid directory name."
    echo "Directory %from% does not exist.  Please enter a valid directory name." >> %MergeLogFile%

  REM ,, goto :eof will return back to the code/script/function/place that called 
  REM ,, this script, in this case, either another batch file or the command line.
    goto :eof
  )

REM .. Display the Directory names now that they have been cleaned up.
  echo Move From: %from% 
  echo Move From: %from% >> %MergeLogFile%
  echo Move To: %to%
  echo Move To: %to% >> %MergeLogFile%
  echo Deleted Files: %DeleteFiles% 
  echo Deleted Files: %DeleteFiles% >> %MergeLogFile%
  echo List or Move: %ListOrMove% 
  echo List or Move: %ListOrMove% >> %MergeLogFile%
  echo Log file: MergeLogFile=%cd%MergeDirsLog-%SortDate%%SortTime%.txt
  echo Log file: MergeLogFile=%cd%MergeDirsLog-%SortDate%%SortTime%.txt >> %MergeLogFile%

REM .. Set up some tracking variables
  set MoveTo=%to%
  set "LogMsg="
  set /A "from_len=0"
  set /A "LimitLoops=100"
  set "MsgLvl=*"

  echo MoveTo %MoveTo%
  echo MoveTo %MoveTo% >> %MergeLogFile%
  echo.
  echo. >> %MergeLogFile%

REM .. Create Destination Folder if not Exist
  if not Exist "%to%" MD "%to%" >nul
  if not Exist "%DeleteFiles%" MD "%DeleteFiles%" >nul

REM .. Get Length of the characters in "from" variable to remove them later
  for /F "usebackq delims=" %%A in ('"%from%"') Do (Call :strLen from from_len)

REM .. Add "1" to the length to avoid problems later 
REM .. += is shorthand used to add the current variable value to the number 
REM .. following it and store it in the variable (from_len = from_len + number)
  set /A from_len+=1

  echo "===== BEGIN Robocopy MOVE FILES"
  echo "===== BEGIN Robocopy MOVE FILES" >> %MergeLogFile%

REM .. Move new folders and new files from "Source" to "Destination"

REM .. This Robocopy is set to ignore any existing duplicate folder names and files
REM .. Duplicates will be addressed later because robocopy can not rename files

REM ,, /L List only - don’t copy, timestamp, move or delete any files.
REM ,, /TEE Output to console window, as well as the log file.
REM ,, /E : Copy Subfolders, including Empty Subfolders.
REM ,, /MOVE Move files and dirs (delete from source after copying).
REM ,, /XC eXclude Changed; /XN Newer files; /XO eXclude Older files; 
REM ,, /XX eXclude "eXtra" files and dirs (present in destination but not source)

  if %ListOrMove% EQU "M" (
    echo "** %ListOrMove% Moving files **" >> %MergeLogFile%
    ROBOCOPY "%from%" "%to%" /TEE /E /MOVE /XC /XN /XO /XX /LOG+:"%MergeLogFile%" /R:3
  ) ELSE (
    echo "** %ListOrMove% Listing Files **" >> %MergeLogFile%
    ROBOCOPY "%from%" "%to%" /L /TEE /E /MOVE /XC /XN /XO /XX /LOG+:"%MergeLogFile%" /R:3
  )

  echo "===== Robocopy MOVE FILES COMPLETED"
  echo "===== Robocopy MOVE FILES COMPLETED" >> %MergeLogFile%
  echo. 
  echo. >> %MergeLogFile%

REM .. Compare and Move the rest of the files.

REM ,, for /F loops against a set of files  delims=xxx delimiter character(s) 
REM ,, (default = a space);    usebackq     Used to put quotes around a 
REM ,, filename and use the alternate quoting style:                        
REM ,,  - Use double quotes for long file names in "filenameset".
REM ,,  - Use single quotes for 'Text string to process'
REM ,,  - Use back quotes for `command to process`
REM ,, DIR list files and subfolders /B=bare with no header or file info; 
REM ,, /S=show subdirectories; /A:D Folders only; /A:-D NO Folders;
          
  set /A "numFilesProcessed=0"
  set /A "numFilesDeleted=0"
  set /A "numFilesMoved=0"
  set /A "numFilesSkipped=0"
  set /A "numErrors=0"
              
  SETLOCAL EnableDelayedExpansion
REM --------------- Begin primary loop to process files ----------------------
  for /F "delims=" %%A In ('DIR /B /S /A:-D "%from%"') Do (
REM --------------------------------------------------------------------------
    echo.
    echo.
    echo "==== NEW FILE ==== !Date! !Time!" "%%A" 
    echo.
              
    echo. >> %MergeLogFile%
    echo. >> %MergeLogFile%
    echo "==== NEW FILE ==== !Date! !Time!" "%%A" >> %MergeLogFile%
    echo. >> %MergeLogFile%
    
REM .. Reset skip flag each time a new file is processed
    set /A "skipfile=0"
REM .. Increment total number for files processed by 1
    set /A "numFilesProcessed+=1"
            
    REM .. Check for exclamation point in the filename and skip file if 
    REM .. there are any - set enable delayed and any ! will disappear in filename
    REM .. so an easy check is to check to see if the file exists
        
    if NOT Exist "%%A" (
      set /A "skipFile=1"
      echo *************** CheckName !CheckName!
           
    ) ELSE (
      REM .. Checking the filename for more special characters that can cause 
      REM .. the script to fail.

      set /A "OrigfnLen=0"
      set "fFullFileName=%%A"
      call :strlen "fFullFileName" OrigfnLen
       
      set /A "spclCharLen=0"
      set "Checkname=%%A"
         
      REM -- These don't work (! ^%% ^")  
      REM .. Don't check for ":" because it is in the drive path name
      For %%I In (® ^| ^& ^< ^> ^^ + @ # $ { } [ ] ' ` ^%% ^" ) Do (Set CheckName=!CheckName:%%I=!)
      call :strlen CheckName spclCharLen
      if !spclCharLen! NEQ !OrigfnLen! set /A "skipfile=1"
    )
          
    if !skipfile!==1 (
      echo "+++++ File has bad/unusual character in name, so skipping. +++++" "%%A"
      echo "+++++ File has bad/unusual character in name, so skipping. +++++" "%%A">> %MergeLogFile% 2>&1
      echo "%%A">> %SkippedLogFile% 2>&1 
      set /A "numFilesSkipped+=1"
    ) ELSE (

    REM .. set variables with parts of filename and path
    REM .. For a top level, fsubdir should be null.  
    REM .. ex:  fromPath=M:\DirSrc, fFullPath=M:\DirSrc, then fsubdir=""
          
    set "fFullPath=%%~dpA"
    set "fsubdir=!fFullPath:~%from_len%!"
    set "fFileNmExt=%%~nxA"
    set "fFileName=%%~nA"
    set "fExt=%%~xA"
    set "fSize=%%~zA"
    set "fdate=%%~tA"
     
rem    echo "These were used while testing"
rem    echo "from - %from% - !from!" - %%A"
rem    echo "fFullFileName - %fFullFileName% - !fFullFileName! - %%A"
rem    echo "fFullPath - %fFullPath% - !fFullPath! - %%~dpA"
rem    echo "fsubdir - %fsubdir% - !fsubdir!"
rem    echo "fFileNmExt - %fFileNmExt% - !fFileNmExt! - %%~nxA"
rem    echo "fFileName - %fFileName% - !fFileName! - %%~nA"
rem    echo "fExt - %fExt% - !fExt! - %%~xA"
rem    echo "fsize - %fsize% - !fsize!"
rem    echo "fdate - %fdate% - !fdate!"
rem    echo "OrigfnLen %OrigfnLen% - !OrigfnLen!"
rem    echo.
        
      set /a "FileNumber=-1"
      set "MoveTo=!to!"
        
      if Not Exist "%to%\!fsubdir!!fFileNmExt!" (
        set /a "FileNumber=0"
      ) ELSE (
        REM .. :FileIncrNum Checks for Duplicates - Name, Date/Time, and Size. If one 
        REM .. is not the same it adds extra number at the end of the filename.
        call :FileIncrNum FileNumber,!fsize!,"!fdate!","!fFullPath!",%from_len%,"!fFileName!","!fExt!",MoveTo,%LimitLoops%
      ) 
            
      REM cmd /c exit 0 is used to set errorlevel to 0 so that error checking will work
      cmd /c exit 0
          
      if %ListOrMove% EQU "M" (
        if not Exist "!MoveTo!\!fsubdir!" (
          echo "Make directory !MoveTo!\!fsubdir!" 
          echo "Make directory !MoveTo!\!fsubdir!"   >> %MergeLogFile%
          MD "!MoveTo!\!fsubdir!"   >> %MergeLogFile% 2>&1
          if ERRORLEVEL 1 if NOT ERRORLEVEL 2 echo "An ERROR occurred while creating the directory."   >> %MergeLogFile%
        )
      )
  echo FileNumber !FileNumber!
      if !FileNumber! EQU 0 (
        set movetoFileName=!MoveTo!\!fsubdir!!fFileNmExt!
      ) Else (
        set movetoFileName=!MoveTo!\!fsubdir!!fFileName! ^(!FileNumber!^)!fExt!
      )
      echo movetoFileName !movetoFileName!
         
      echo "=!numFilesProcessed!= Moving !fFullFileName! TO !movetoFileName!"
      echo "=!numFilesProcessed!= Moving !fFullFileName! TO !movetoFileName!"   >> %MergeLogFile%
           
      if %ListOrMove% EQU "M" (
        MOVE /Y "!fFullFileName!" "!movetoFileName!"  >> %MergeLogFile% 2>&1
        if ERRORLEVEL 0 if NOT ERRORLEVEL 1 (
          if !MoveTo! EQU %DeleteFiles% (
            set /a "numFilesDeleted+=1"
            echo "%%A">> %DeletedLogFile% 2>&1 
          ) ELSE (
            set /a "numFilesMoved+=1"
            echo "%%A">> %MovedLogFile% 2>&1 
          )
          echo "ERRORLEVEL %ERRORLEVEL% no errors occurred, move succeeded."
          echo "ERRORLEVEL %ERRORLEVEL% no errors occurred, move succeeded."   >> %MergeLogFile%
        ) ELSE (
          if ERRORLEVEL 1 if NOT ERRORLEVEL 2 (
            echo "ERRORLEVEL 1 -- an error occurred."
            echo "ERRORLEVEL 1 -- an error occurred."   >> %MergeLogFile%
          ) ELSE (
            echo "ERRORLEVEL 2 or greater occurred. %ERRORLEVEL%"
            echo "ERRORLEVEL 2 or greater occurred. %ERRORLEVEL%"   >> %MergeLogFile%
          )
          set /a "numErrors+=1"
          echo "%%A">> %ErrorLogFile% 2>&1 
        )
      )
    )
  )

  if %ListOrMove% EQU "M" (
    echo.
    echo.
    echo. >> %MergeLogFile%
    echo. >> %MergeLogFile%
    echo "** List Subdirectories in %from% **"
    echo "** List Subdirectories in %from% **"  >> %MergeLogFile%

    REM .. Sort /R= Sort in reverse order
    DIR /B /S /A:D "%from%" ^| sort /R >> %MergeLogFile% 2>&1

  echo "** Delete Subdirectories in %from% **"
  echo "** Delete Subdirectories in %from% **"  >> %MergeLogFile%

  for /F "usebackq delims=" %%d in (`DIR /B /S /A:D "%from%" ^| sort /R`) do echo %%d >> %MergeLogFile% 2>&1
  for /F "usebackq delims=" %%d in (`DIR /B /S /A:D "%from%" ^| sort /R`) do rd "%%d" >> %MergeLogFile% 2>&1

  echo "** Delete Subdirectories deleted **"
  echo "** Delete Subdirectories deleted **"  >> %MergeLogFile%

  )
echo.
echo.
echo. >> %MergeLogFile%
echo. >> %MergeLogFile%
echo Number of Files Processed %numFilesProcessed% !numFilesProcessed!
echo Number of Files Processed %numFilesProcessed%   >> %MergeLogFile%
echo Number of Files Deleted %numFilesDeleted% !numFilesDeleted!
echo Number of Files Deleted %numFilesDeleted%   >> %MergeLogFile%
echo Number of Files Moved %numFilesMoved% !numFilesMoved!
echo Number of Files Moved %numFilesMoved%   >> %MergeLogFile%
echo Number of Files Skipped %numFilesSkipped% !numFilesSkipped!
echo Number of Files Skipped %numFilesSkipped%   >> %MergeLogFile%
echo Number of Errors %numErrors% !numErrors!
echo Number of Errors %numErrors%   >> %MergeLogFile%

goto :eof



:: ==========================================================::
:FileIncrNum <1IncrNum> <2fromSIZE_var> <3fromDATE_var> <4fromFullPath> <5fromPathLen> <6fromFilename> <7fromExt> <8destPath> <9looplimit_var> 
:: ----------------------------------------------------------::
REM .. Function to calculate recursively the next number to use in the file name

  setLOCAL ENABLEDELAYEDEXPANSION
  set /a "IncrNum=%~1+1"
  set fromSIZE=%~2
  set fromDATE=%~3
  set fromFullPath=%~4
  set fromPathLen=%~5
  set fromFileName=%~6
  set fromExt=%~7
  set destPath=!%~8!
  set /a "looplimit=0+%~9"
  set fromFullFileName=%fromFullPath%%fromFileName%%fromExt%
  set subPath=!fromFullPath:~%fromPathLen%!
rem echo IncrNum !IncrNum!
  If !IncrNum! EQU 0 (
  set destFileName=!destPath!\!subPath!!fromFileName!!fromExt!
  ) Else (
  set destFileName=!destPath!\!subPath!!fromFileName! ^(!IncrNum!^)!fromExt!
  )
rem echo destFileName !destFileName! %destFileName%
               
    if %looplimit% EQU 0 (
      if Exist "!destFileName!" (
        call :FileIncrNum  IncrNum,%fromSIZE%,"%fromDATE%","%fromFullPath%",%fromPathLen%,"%fromFileName%","%fromExt%",destPath,0
      ) 
    ) ELSE (
      echo "PROPOSED FILE EXISTS SO RENAME THE FILE !IncrNum! !destFileName!"   >> %MergeLogFile%
      if !IncrNum! LSS !looplimit! (
        if Exist "!destFileName!" (
          Call :Info "!destFileName!" "toSIZE" "toDATE"
          if "!toSIZE!" EQU "%fromSIZE%" (
            set "LogMsg=THE FILES HAVE SAME SIZES !toSIZE! NEQ %fromSIZE% - SAVE FILE AND COMPARE - %fromFullFileName%"
            if "!toDATE!" EQU "%fromDATE%" (
              set "LogMsg=THE FILES HAVE SAME DATES !toDATE! NEQ %fromDATE% - SAVE FILE AND COMPARE - %fromFullFileName%"
              FC /B /LB1 "%fromFullFileName%" "!destFileName!" | FIND "FC: no dif">nul
              if ERRORLEVEL 0 if NOT ERRORLEVEL 1 (
                set "LogMsg=THE FILE %fromFullFileName% HAS THE SAME DATE, SAME SIZE, NO BINARY DIFFERENCE FROM !destFileName! - SO DELETE THE FILE "
                set "destPath=!DeleteFiles!"
                set /a "IncrNum=0"
                set "destFileName=!destPath!\%subPath%%fromFileName%%fromExt%"
              ) ELSE (set "LogMsg=THE FILE %fromFullFileName% HAS THE SAME DATE, SAME SIZE, BUT HAS A BINARY DIFFERENCE - SAVE FILE AND MANUALLY COMPARE ")
            ) ELSE (set "LogMsg=THE FILES HAVE DIFFERENT DATES !toDATE! NEQ %%~tA - SAVE FILE AND MANUALLY COMPARE - %fromFullFileName% ")
          ) Else (set "LogMsg=THE FILES HAVE DIFFERENT SIZES !toSIZE! NEQ %%~zA - SAVE FILE AND MANUALLY COMPARE - %fromFullFileName% ")
          echo "*!LogMsg! - !destFileName!"   >> %MergeLogFile%
           
          if "!destPath!" EQU "%DeleteFiles%" (
              if Exist "!destFileName!" (
                echo "FILE !destFileName! exists, so increment file number and try again. IncrNum %IncrNum% !IncrNum! "   >> %MergeLogFile%
                call :FileIncrNum  IncrNum,%fromSIZE%,"%fromDATE%","%fromFullPath%",%fromPathLen%,"%fromFileName%","%fromExt%",destPath,0
              ) ELSE (echo "Delete file !destFileName! does not exist so go back and move file to delete IncrNum %IncrNum% !IncrNum!"   >> %MergeLogFile%)
           ) ELSE (
             if "!destPath!" EQU "%to%" (
             if Exist "!destFileName!" (
                echo "FILE EXISTS !destFileName!, increment file number IncrNum %IncrNum% !IncrNum! - destPath %destPath% !destPath!"   >> %MergeLogFile%
                call :FileIncrNum  IncrNum,%fromSIZE%,"%fromDATE%","%fromFullPath%",%fromPathLen%,"%fromFileName%","%fromExt%",destPath,%LimitLoops%
              ) ELSE (echo "File !destFileName! does not exist so go back and move file IncrNum %IncrNum% !IncrNum!"  >> %MergeLogFile%)
            ) ELSE (echo "not equal %destPath% !destPath! EQU %to%"  >> %MergeLogFile%)
          )  
        )
      ) ELSE (echo "****ERROR**** LOOP LIMIT REACHED !IncrNum! !Looplimit!"  >> %MergeLogFile%)
  )     
           
(             
  ENDLOCAL
    set "%~1=%IncrNum%"
    set "%~8=%destPath%"
)            
goto :eof


:: ==========================================================::
:strLen <string> <len> 
:: ----------------------------------------------------------::
::-- returns the length of a string
::                 -- string [in]  - variable name containing the string being measured for length
::                 -- len    [out] - variable to be used to return the string length
:: Many thanks to 'sowgtsoi', but also 'jeb' and 'amel27' dostips forum users helped making this short and efficient
::$created 20081122 :$changed 20101116 :$categories StringOperation
::$source http://www.dostips.com
(SETLOCAL ENABLEDELAYEDEXPANSION
    set "str=A!%~1!"
    set "len=0"
    for /L %%C in (12,-1,0) do (
        set /a "len|=1<<%%C"
        for %%D in (!len!) do (if "!str:~%%D,1!"=="" set /a "len&=~1<<%%C")
    )
)
( ENDLOCAL & REM RETURN VALUES
    IF "%~2" NEQ "" SET /a "%~2=%len%"
    if "%~2" EQU "" set /a echo %len%
)
goto :eof

:: ==========================================================::
:Info <file> <size_var> <time/date_var>
:: ----------------------------------------------------------::
  Set "Info_file=%~1"
  For /F "delims=" %%A In ("%Info_File%") Do (
    Set "%~2=%%~zA"
    Set "%~3=%%~tA"
  )
goto :eof

[Aacini]

I used a different approach to solve this problem via a recursive subroutine that, in my opinion, is clearer and would run faster because the executed code is lesser. The Batch program below was developed in accordance with the pseudo-code you stated at beginning of this topic.

Code: Select all

@echo off
setlocal EnableDelayedExpansion

rem MergeDirs.bat: Merge a source tree into a destination tree, renaming duplicated files
rem MergeDirs C:\sourcedir d:\destdir

rem Get source and destination drives
set sourceDrive=%~D1
set destDrive=%~D2

rem Start the top-level process
set logFile=C:\LogFile.txt
if exist "%logFile%" del "%logFile%"
call :MergeTrees "%~PN1" "%~PN2"
goto :EOF


:MergeTrees sourceDir destDir
setlocal EnableDelayedExpansion

rem Enter into source and destination folders
if not exist "%destDrive%%~2" md "%destDrive%%~2"
pushd "%destDrive%%~2"
pushd "%sourceDrive%%~1"

rem For each folder in source
for /D %%a in (*) do (
   rem If folder does not exist in destination
   if not exist "%destDrive%%%~a" (
      rem Copy the folder and all subfolders and files to destination
      xcopy /S /E /I /Q "%%~a" "%destDrive%%%~a"
      rem Write to logfile here to note the folder was moved
      echo Folder "%sourceDrive%%%~a" moved to "%destDrive%%%~a" >> "%logFile%"
   ) else (
      rem Folder already exists: merge the files [in it] into one directory
      pushd "%destDrive%%%~a"
      pushd "%sourceDrive%%%~a"
      rem For each file in source
      for %%b in (*.*) do (
         rem If exist in destination
         if exist "%destDrive%%%~b" (
            rem Check date and size.
            if "%%~TZb" neq "%destDrive%%%~TZb" (
               rem If different, move but rename to filename (next number^).ext
               call :getNextNumber "%%~b"
               copy "%%~b" "%destDrive%%%~Nb (!nextNumber[%%~b]!^)%%~Xb"
               rem Write to logfile here to note the file was moved and renamed
               echo File "%%~b" moved and renamed to "%destDrive%%%~Nb (!nextNumber[%%~b]!^)%%~Xb" >> "%logFile%"
            ) else (
               rem If same date and size then do binary compare the files
               fc /B "%%~b" "%destDrive%%%~b"
               rem If the compare fails
               if !errorlevel! gtr 0 (
                  rem Move the file but rename to filename (next number^).ext
                  call :getNextNumber "%%~b"
                  copy "%%~b" "%destDrive%%%~Nb (!nextNumber[%%~b]!^)%%~Xb"
               ) else (
                  rem If same move to "processed - ready to delete" folder
                  move "%%~b" "C:\processed - ready to delete"
                  rem Write to logfile here to note the file was "duplicate - deleted"
                  echo File "%%~b" was "duplicate - deleted" >> "%logFile%"
               )
            )
         ) else ( rem Not exist
            rem Move file to correct subdirectory
            copy "%%~b" "%destDrive%"
            rem Write to logfile here to note the file was moved
            echo File "%%~b" moved to "%destDrive%" >> "%logFile%"
         )
      )
      popd
      popd
      rem Recursively process this folder
      call :MergeTrees "%%~a" "%%~a"
   )
)

rem Return from source and destination folders
popd
popd

exit /B


:getNextNumber fileName
set /A nextNumber[%~1]+=1
if exist "%destDrive%%~N1 (!nextNumber[%~1]!)%~X1" goto getNextNumber
exit /B

EDIT: I fixed a couple small details that I explain in a reply below.

IMPORTANT: In your original request you said: "I need to merge directories from several old hard drives onto a new home NAS". Previous program does NOT work if both source and destination drives are the same; they must be different drives.

Antonio

[ESerres]

Thanks Aacini,
I was wondering if there was a shorter way of coding what I was doing, and it looks like you came up with a way. I had used recursive code as well, but checked twice - once for the destination directory, and again for the deleted directory. Checking in the "deleted" directory is overkill because I eventually intend to really delete the files, but I wanted to have a comfort level that I was saving the right files and deleting the right stuff before I get rid of these files.

You are right about the drive letters being different. In my last post I noted that I was looking for is a way to input my directory names from a command line, like: c:>MergeDirectory "M:\ScrDir" "M:\DestDir" "M:DelDir" /L or /M for move or list. Right now I have been testing on the same drive, but I knew that eventually I would be using different drives and different directories. When running this, I will be moving some of the content to new directory names. For example I may want to move old files from "Documents" to "My Documents". So I would like to be able to make a command batch file that will call this new batch file.

EX: MergeMyDirectories.bat would contain the following types of commands:

Code: Select all

Echo off
MergeDirectory "M:\OldFiles\Documents" "N:\NAS\My Documents" M:\DeletedFiles" /M
MergeDirectory "M:\OldFiles\Videos" "N:\NAS\My Videos" M:\DeletedFiles" /M
MergeDirectory "G:\Users\ESerres\My Videos" "N:\NAS\My Videos" G:\DeletedFiles" /M

etc.

I'm pretty sure it will be easy to add that code, but haven't gotten to researching and learning how yet.

Instead I have been concentrating on how to write to a log. I found the >> syntax and used it in my batch, but I also found a lot of stuff on TEE.BAT that I was reading and investigating. While using a TEE.BAT would be total overkill for my batch, I was interested in learning how to use it so I would know how to do it for other programs I may be writing in the future. There was an article here on DOSTips "Display output of batch file on the screen and log file?"
http://www.dostips.com/forum/viewtopic.php?f=3&t=3226 that showed some examples, but I was having trouble figuring out how to use them and test them - I hung up my computer and had to reboot a couple of times - scary.

Can anyone explain to me how to use them - exact steps, using the batch file we have created in this post as an example? I am especially interested in understanding the code by Ed Dryeen towards the end. Is his code supposed to be added as a function and called as part of a command, or added at the beginning of the batch file or a separate TEE.Bat file? I didn't see exactly how to implement it in the article. Thanks for helping me learn this stuff.

[abc0502]

Hi,
What do you need to log ? File names or the command itself ?
If it the file names, you just have to add one line before each copy/move command like,
let's way you are going to use this command:

Code: Select all

MOVE /Y "%from%\!Full_Path:~%from_len%!%%~nxA" "!MoveTo!\!Full_Path:~%from_len%!%%~nxA"

, now you just add this line like that before this command like this,

Code: Select all

"%from%\!Full_Path:~%from_len%!%%~nxA":"!MoveTo!\!Full_Path:~%from_len%!%%~nxA" >>"LogFile.log"
MOVE /Y "%from%\!Full_Path:~%from_len%!%%~nxA" "!MoveTo!\!Full_Path:~%from_len%!%%~nxA"

you just echo the file full path from source to distination to the log files,so every time the !Full_Path:~%from_len%!%%~nxA Variable change, it's value will be written to the log file.
and i guess the command can be written too this way, it will be just an extra line before each command.

[ESerres]

I re-read previous posts and completed a more detailed walk-through of Aacini's code this morning and realized that all my remaining questions from my original post had been addressed in his REM remarks. I had skimmed originally and didn't catch them. Thanks all for the help.

Foxidrive, If I had taken your suggestion to use Total Commander I would not have learned all these cool things about DOS commands, like:
how to search for substrings in my DOS variables
how to extract the filename, path, extension, drive, date and time from a directory listing
how to use recursion and functions to increment file numbers
how to use DOS variables and parameters
how to use FOR commands
how to use IF ELSE statements
how to use ERRORLEVEL correctly
how to do a file compare in a batch
how to add the date to my filename
how to call a batch file and parameters from a batch file and return, etc.
This small batch file may only be used by me a few times, but it will provide me (and hopefully other readers) with a great set of examples of all of these items. But I will check out Total Commander anyway, just so I have yet another item in my toolbox for future use. Thanks for your suggestion.

As a final note for those who may be reading this to do something similar, when I tried to create a batch file to call this batch file, I found I had to use a CALL statement in order to get it to execute multiple times for multiple directories.

MergeDirectories.bat contains the following:

Code: Select all

call MergeDirs "M:\SrcDir1" "C:\DestDir1" "M:\DeleteFiles" "L"
call MergeDirs "M:\SrcDir2" "N:\DestDir2" "M:\DeleteFiles" "M"
call MergeDirs "M:\SrcDir3" "C:\DestDir1" "M:\DeleteFiles" "M"

MergeDirs.bat contains the code listed above plus this at the start to set up the log file and read in the parameters listed above.

Code: Select all

@Echo OFF

rem MergeDirs.bat: Merge a source tree into a destination tree, renaming duplicated files
rem MergeDirs C:\sourcedir d:\destdir c:\deletedDir
REM ListOrMove L=List M=Move

rem Get source and destination drives
set "from=%1"
set "to=%2"
set "DeleteFiles=%3"
set "ListOrMove=%4"

:: Use WMIC to retrieve date and time
FOR /F "skip=2 tokens=2-7 delims=," %%A IN ('WMIC Path Win32_LocalTime Get Day^,Hour^,Minute^,Month^,Second^,Year /Format:csv') DO (
SET /A SortDate = 10000 * %%F + 100 * %%D + %%A
SET /A SortTime = 10000 * %%B + 100 * %%C + %%E
)
:: display timestamp
set "MergeLogFile=M:\MergeDirectoryLog-%SortDate%%SortTime%.txt"
set MoveTo=!to!
set "LogMsg="
set /A "LimitLoops=100"
set /A "numFilesProcessed=0"
set "MsgLvl=*"

::echo to make sure working
Echo %from% >> %MergeLogFile%
Echo %to% >> %MergeLogFile%
Echo %DeleteFiles% >> %MergeLogFile%
Echo %ListOrMove% >> %MergeLogFile%

[Aacini]

Just a couple details about my program:

• Right parentheses included in the name of the renamed files must be escaped, otherwise they will cause a nesting error in the current IF command:

  Code: Select all

  Change this:                            To this:
  
  rename to filename (next number).ext    rename to filename (next number^).ext
  
  (!nextNumber[%%~b]!)%%~Xb"              (!nextNumber[%%~b]!^)%%~Xb"
  

  
  
• Entering into folders and returning to parent folder should be done via PUSHD and POPD commands instead of the current CD commands, but inverting the order of the entering CD's so the last selected folder be the sourcedir:

  Code: Select all

  Change this:                                         To this:
  
  rem Enter into source and destination folders        rem Enter into source and destination folders
  cd /D "%sourceDrive%%~1"                             if not exist "%destDrive%%~2" md "%destDrive%%~2"
  if not exist "%destDrive%%~2" md "%destDrive%%~2"    pushd "%destDrive%%~2"
  cd "%destDrive%%~2"                                  pushd "%sourceDrive%%~1"
  
  rem Folder already exists: merge the files...        rem Folder already exists: merge the files...
  cd "%sourceDrive%%%~a"                               pushd "%destDrive%%%~a"
  cd "%destDrive%%%~a"                                 pushd "%sourceDrive%%%~a"
  
  cd "%sourceDrive%.."                                 popd
  cd "%destDrive%.."                                   popd
  

The last group of two CD's that must be changed by two POPD's appear twice. I already completed these changes in the original code, so perhaps it is easier to copy the whole program again.

Antonio

[ESerres]

Aacini, When I tried your code above, I got the following results, not exactly what I was looking for:

File "bcd" moved and renamed to "M:bcd (1)"
File "boot.sdi" moved and renamed to "M:boot (1).sdi"
File "bootfix.bin" moved and renamed to "M:bootfix (1).bin"
File "etfsboot.com" moved and renamed to "M:etfsboot (1).com"
File "bcd (1)" moved and renamed to "M:bcd (1) ()"
File "boot (1).sdi" moved and renamed to "M:boot (1) ().sdi"
File "bootfix (1).bin" moved and renamed to "M:bootfix (1) ().bin"
File "etfsboot (1).com" moved and renamed to "M:etfsboot (1) ().com"
File "bcd (1) ()" moved and renamed to "M:bcd (1) () ()"
File "boot (1) ().sdi" moved and renamed to "M:boot (1) () ().sdi"
File "bootfix (1) ().bin" moved and renamed to "M:bootfix (1) () ().bin"
File "etfsboot (1) ().com" moved and renamed to "M:etfsboot (1) () ().com"
File "bcd (1) () ()" moved and renamed to "M:bcd (1) () () ()"
File "boot (1) () ().sdi" moved and renamed to "M:boot (1) () () ().sdi"
File "bootfix (1) () ().bin" moved and renamed to "M:bootfix (1) () () ().bin"
File "etfsboot (1) () ().com" moved and renamed to "M:etfsboot (1) () () ().com"
File "bcd (1) () () ()" moved and renamed to "M:bcd (1) () () () ()"
File "boot (1) () () ().sdi" moved and renamed to "M:boot (1) () () () ().sdi"
File "bootfix (1) () () ().bin" moved and renamed to "M:bootfix (1) () () () ().bin"
File "etfsboot (1) () () ().com" moved and renamed to "M:etfsboot (1) () () () ().com"
File "bcd (1) () () () ()" moved and renamed to "M:bcd (1) () () () () ()"
File "boot (1) () () () ().sdi" moved and renamed to "M:boot (1) () () () () ().sdi"
File "bootfix (1) () () () ().bin" moved and renamed to "M:bootfix (1) () () () () ().bin"
File "etfsboot (1) () () () ().com" moved and renamed to "M:etfsboot (1) () () () () ().com"
File "bcd (1) () () () () ()" moved and renamed to "M:bcd (1) () () () () () ()"
File "boot (1) () () () () ().sdi" moved and renamed to "M:boot (1) () () () () () ().sdi"
File "bootfix (1) () () () () ().bin" moved and renamed to "M:bootfix (1) () () () () () ().bin"


It did not use the Source or destination directory I had put in, and ran in a loop, until I killed it.
Since I had already added a lot of extra stuff to my code by the time you posted your updates, I went ahead and used my code and fixed it as best I could.

Everyone,
My code does what I wanted above in my first post, except it won't process some files correctly - those with exclamation points, brackets, or other unusual file characters. Maybe Aacini's code will work better than mine on these files and be faster when his errors are fixed, but when I tried it and saw so many errors, I was kind of brain dead and just gave up. I had already spent a week trying to figure out how to solve the problem of unusual characters and finally decided to spit the errors out and move them manually and worry about fixing the code some other day if the need arises.

I updated my code above with my lastest changes and it is the version I am now using, just in case anyone tries to use it. I put a lot of remarks in my code because I kept forgetting things and it helped me to document them, and I didn't want to take the time to pull them out -- plus they may help some newbies.