Move from FTP and count in Target folder automated

[aaksar]

still no luck, changed the 1st line as u mentioned, still no filenmae in command2.ftp

below is the code

Code: Select all

@Echo OFF
:: Don't Forget to Specify the Zipped file type in line 18

::"file_list" --------------------> %~1
::"remote_server" ----------------> %~2
::"remote_server_username" -------> %~3
::"remote_server_password" -------> %~4
::"total_number" -----------------> %~5
::"local_folder" -----------------> %~6
::"remote_folder"  ---------------> %~7
::"FTP servers temp folder" ------> %~8

::Create Local Folder if not exist
IF not Exist "%~6" MD "%~6" >nul

:loop


:: FTP commands to get list of files on the server
IF "%~7"=="" (
   (
   Echo open %~2
   Echo %~3
   Echo %~4
   Echo ls
   Echo disconnect
   Echo Bye
   )>"%temp%\%~8\commands_1.ftp"
) Else (
   (
   Echo open %~2
   Echo %~3
   Echo %~4
   Echo cd "%~7"
   Echo ls
   Echo disconnect
   Echo Bye
   )>"%temp%\%~8\commands_1.ftp"
   )

:: start FTP session and log output
Ftp -v -i -s:"%temp%\%~8\commands_1.ftp" >"%temp%\%~8\dump.log"

:: extract file name from the ftp dump file
For /F "delims=" %%a in ('Type "%temp%\%~8\dump.log"') Do Echo "%%a" |findstr /I /V /R ^ftp^> |findstr /I /V /R ^bye >>"%temp%\%~8\server_list.lst"

:: 2nd FTP commands to download only new files that dosn't exist locally [Part1]
IF "%~7"=="" (
   (
   Echo open %~2
   Echo %~3
   Echo %~4
   Echo lcd "%~6"
   )>"%temp%\%~8\commands_2.ftp"
) Else (
   (
   Echo open %~2
   Echo %~3
   Echo %~4
   Echo lcd "%~6"
   Echo cd "%~7"
   )>"%temp%\%~8\commands_2.ftp"
   )
pause
:: compare files that will be downloaded with the file.list and if there is a match
:: it see if this match isn't in the local folder so it can download that file. [Part2]
For /F "delims=" %%a in ('Type "%~1"') Do (
   For /F "delims=" %%b in (%temp%\%~8\server_list.lst) Do (
   Set "sf=%%b"
   setlocal enabledelayedexpansion
      IF ""%%a"" EQU "!sf!" (
         IF NOT EXIST "%~6\%%b" echo get %%b >>"%temp%\%~8\commands_2.ftp"
      )
   endlocal
   )
)

:: Add disconnect & bye commands to 2nd FTP command file [Part3]
(
Echo disconnect
Echo Bye
)>>"%temp%\%~8\commands_2.ftp"

:: start FTP session and download new files
Ftp -v -i -s:"%temp%\%~8\commands_2.ftp"

:: Check files count
Setlocal EnableDelayedExpansion
set Fcount=0
For /F "delims=" %%a in ('Dir /B "%~6"') Do set /a Fcount += 1

:: Check Condition to Exit or Continue Looping
:: GEQ means equal to total file number or greater in case there was new files added
:: and user didn't update the FTP.set file.
:: but if the files was become less and user didn't update the file it will keep looping
:: forever and will not extract files from zip archives.
Setlocal EnableDelayedExpansion
IF "%Fcount%" EQU "%~5" (
   "%zip_program%" x -y "%~6\*.%zipped%" -o"%~6" >nul
   Del /F /Q "%~6\*.%zipped%" >nul
   RMDIR /S /Q "%temp%\%~8" >nul
   Goto :EOF
) Else (
   :: Wait 5 min
   Ping Localhost -l 1 -n 300 >nul
   Goto :loop
   )

[abc0502]

You are missing the 7z program location and the zipped file type variables
put this two lines after the label :loop

Code: Select all

set "zip_program=%programfiles%\7-Zip\7z.exe"
set "zipped=zip"

AND Important
Quick question, open the server_list.lst and tell me if there is a single space after the last quote

like if file name is "1.7z" so is there a single space after the name before the curser go to a new line ?

[aaksar]

i have intentionally remove the 7zip program command.
will put it.

for the next thing , i will update you tmrw mrng.

and everytime it pole the ftp location it append the file list again.

[abc0502]

Updating the ftp server_list.lst is easy fix i updated the code i'm currently testing on to remove that file if it loops, you can do that by adding extra line before the wait command like this:

Code: Select all

) Else (
   Del /F /Q "%temp%\%~8\Server_list.lst" >nul
   :: Wait 5 min
   Ping Localhost -l 1 -n 300 >nul
   Goto :loop
   )

This should fix that

[aaksar]

duplicate file list in server_list.lst file is error or its fine?

ok, i will update you the other things(space ) tmrw.
but the thing is previously when i ran start.bat, it showed "get file name " on cmd,

but when i run now, it wont show me any get command, i think thatswhy the file is not loading.

[abc0502]

the get shows in the cmd screen is the command that is written in the command_2.ftp and this is the error, it's not being written there.

and the space i'm asking for is because of this line in the previous batch:

Code: Select all

IF "%%a" EQU "!sf:~1,-2!" (

when removing the double quotes fro arround the file names in the server_list.lst to compare with the files names in the FILE.list files it should be "!sf:~1,-1!" but i get an extra single space after the file names so i use "!sf:~1,-2!" but when you posted your code i noticed that it is single space missing in all empty spaces in each start of all line that start with empty spaces so i think you don't have this single space. and should use "!sf:~1,-1!" instead of "!sf:~1,-2!"

[abc0502]

Try This One
Fixed a few minor things that will remove the space that i'm asking for so never mind about it.

Code.bat

Code: Select all

@ECHO OFF

:: Don't Forget to Specify the Zipped file type in line 18

::"file_list" --------------------> %~1
::"remote_server" ----------------> %~2
::"remote_server_username" -------> %~3
::"remote_server_password" -------> %~4
::"total_number" -----------------> %~5
::"local_folder" -----------------> %~6
::"remote_folder"  ---------------> %~7
::"FTP servers temp folder" ------> %~8

::Create Local Folder if not exist
IF not Exist "%~6" MD "%~6" >nul

:loop
set "zip_program=%programfiles%\7-Zip\7z.exe"
set "zipped=zip"

:: FTP commands to get list of files from the server
IF "%~7"=="" (
   (
   Echo open %~2
   Echo %~3
   Echo %~4
   Echo ls
   Echo disconnect
   Echo Bye
   )>"%temp%\%~8\commands_1.ftp"
) Else (
   (
   Echo open %~2
   Echo %~3
   Echo %~4
   Echo cd "%~7"
   Echo ls
   Echo disconnect
   Echo Bye
   )>"%temp%\%~8\commands_1.ftp"
   )

:: start FTP session and log output
Ftp -v -i -s:"%temp%\%~8\commands_1.ftp" >"%temp%\%~8\dump.log"

:: extract file name from the ftp dump file
For /F "delims=" %%a in ('Type "%temp%\%~8\dump.log"') Do (
   Echo "%%a"|findstr /I /V /R ^ftp^>|findstr /I /V /R ^bye>>"%temp%\%~8\server_list.lst"
   )

:: 2nd FTP commands to download only new files that dosn't exist locally [Part1]
IF "%~7"=="" (
   (
   Echo open %~2
   Echo %~3
   Echo %~4
   Echo lcd "%~6"
   )>>"%temp%\%~8\commands_2.ftp"
) Else (
   (
   Echo open %~2
   Echo %~3
   Echo %~4
   Echo lcd "%~6"
   Echo cd "%~7"
   )>>"%temp%\%~8\commands_2.ftp"
   )

:: compare files that will be downloaded with the file.list and if there is a match
:: it see if this match isn't in the local folder so it can download that file. [Part2]
For /F "delims=" %%a in ('Type "%~1"') Do (
   For /F "delims=" %%b in (%temp%\%~8\server_list.lst) Do (
   Set "sf=%%b"
   setlocal enabledelayedexpansion
      IF "%%a"=="!sf:~1,-1!" (
         IF NOT EXIST "%~6\%%b" echo get %%b>>"%temp%\%~8\commands_2.ftp"
      )
   endlocal
   )
)

:: Add disconnect & bye commands to 2nd FTP command file [Part3]
(
Echo disconnect
Echo Bye
)>>"%temp%\%~8\commands_2.ftp"

:: start FTP session and download new files
Ftp -v -i -s:"%temp%\%~8\commands_2.ftp"

:: Check files count
set Fcount=0
For /F "delims=" %%a in ('Dir /B "%~6\*.%zipped%"') Do set /a Fcount += 1

:: Check Condition to Exit or Continue Looping
Setlocal EnableDelayedExpansion
IF "%Fcount%" EQU "%~5" (
   "%zip_program%" x -y "%~6\*.%zipped%" -o"%~6" >nul
   Del /F /Q "%~6\*.%zipped%" >nul
   RMDIR /S /Q "%temp%\%~8" >nul
   Goto :EOF
) Else (
   Del /F /Q "%temp%\%~8\server_list.lst" >nul
   :: Wait 5 min
   Ping Localhost -l 1 -n 300 >nul
   Goto :loop )

and if you want it to extract the archive put a high number in your FTP.set so it won't extract the zipped files until the all files being downloaded.

Try it and post the result

[aaksar]

hi,

There is space in server_list.lst

"110A_CONTRACT"
"115A_CONTRACT_EXPER"
"301A_REINSURANCE_CO"
"302A_BROKER"
"311A_STATUTORY_COMPANY"
"311B_PROFIT_CENTER"
"312A_PARTICIPANT_LIST"
"312B_PARTICIPANT"
"317A_INSURANCE_TYPE"
"318A_CONTRACT_TYPE"
"DB1A_CERTIFICATE"
"Temp"
"110A_CONTRACT"
"115A_CONTRACT_EXPER"
"301A_REINSURANCE_CO"
"302A_BROKER"
"311A_STATUTORY_COMPANY"
"311B_PROFIT_CENTER"
"312A_PARTICIPANT_LIST"
"312B_PARTICIPANT"
"317A_INSURANCE_TYPE"
"318A_CONTRACT_TYPE"
"DB1A_CERTIFICATE"
"Temp"

[aaksar]

I have run ur new code.bat
and it seems like it is not finding the server_list.lst

Code: Select all

ftp> open ftp.us.abc.com
User (ftp.us.abc.com:(none)):

ftp> lcd "D:\test"
Local directory now D:\Test.
ftp> cd "1 abc Tables"
ftp> disconnect
ftp> Bye
File Not Found

[abc0502]

I keep testing it and it download the files without any problems,
delete all files and folders in the %temp% directory
change the local folder path in your FTP.set
then run the codes again

The problem you are facing now is in this section:

Code: Select all

:: compare files that will be downloaded with the file.list and if there is a match
:: it see if this match isn't in the local folder so it can download that file. [Part2]
For /F "delims=" %%a in ('Type "%~1"') Do (
   For /F "delims=" %%b in ('Type "%temp%\%~8\server_list.lst"') Do (
   Set "sf=%%b"
   setlocal enabledelayedexpansion
      IF "%%a"=="!sf:~1,-1!" (
         IF NOT EXIST "%~6\%%b" echo get %%b>>"%temp%\%~8\commands_2.ftp"
      )
   endlocal
   )
)

This is the code that compare the files in your FILE.lst files and the Files in the server_list.lst file and then put the file names that is match in the last file and the files that dosn't exist in your local folder.

i will try to write a new one

[abc0502]

it seems like it is not finding the server_list.lst

You didn't find the file because it's being deleted if the batch go in a loop, to prevent any re-writing to the same file over and over.

[abc0502]

can you run this version and post the out put, i added comments so we know where the error exactly is.
Don't create the local folder and leave the batch this job, and make sure you delete the %temp% dir and change the local folder location.

Code: Select all

@ECHO OFF

:: Don't Forget to Specify the Zipped file type in line 18

::"file_list" --------------------> %~1
::"remote_server" ----------------> %~2
::"remote_server_username" -------> %~3
::"remote_server_password" -------> %~4
::"total_number" -----------------> %~5
::"local_folder" -----------------> %~6
::"remote_folder"  ---------------> %~7
::"FTP servers temp folder" ------> %~8

::Create Local Folder if not exist
Echo Creating LOCAL FOLDER
IF not Exist "%~6" MD "%~6" >nul

:loop
set "zip_program=%programfiles%\7-Zip\7z.exe"
set "zipped=zip"

:: FTP commands to get list of files from the server
Echo Creating Commands_1.ftp file
IF "%~7"=="" (
   (
   Echo open %~2
   Echo %~3
   Echo %~4
   Echo ls
   Echo disconnect
   Echo Bye
   )>"%temp%\%~8\commands_1.ftp"
) Else (
   (
   Echo open %~2
   Echo %~3
   Echo %~4
   Echo cd "%~7"
   Echo ls
   Echo disconnect
   Echo Bye
   )>"%temp%\%~8\commands_1.ftp"
   )

:: start FTP session and log output
Echo Connicting to FTP server to get files list
Ftp -v -i -s:"%temp%\%~8\commands_1.ftp" >"%temp%\%~8\dump.log"

:: extract file name from the ftp dump file
Echo Processing dump.log File and creating server_list.lst
For /F "delims=" %%a in ('Type "%temp%\%~8\dump.log"') Do (
   Echo "%%a"|findstr /I /V /R ^ftp^>|findstr /I /V /R ^bye>>"%temp%\%~8\server_list.lst"
   )

:: 2nd FTP commands to download only new files that dosn't exist locally [Part1]
Echo Creating commands_2.ftp File (part1)
IF "%~7"=="" (
   (
   Echo open %~2
   Echo %~3
   Echo %~4
   Echo lcd "%~6"
   )>>"%temp%\%~8\commands_2.ftp"
) Else (
   (
   Echo open %~2
   Echo %~3
   Echo %~4
   Echo lcd "%~6"
   Echo cd "%~7"
   )>>"%temp%\%~8\commands_2.ftp"
   )

:: compare files that will be downloaded with the file.list and if there is a match
:: it see if this match isn't in the local folder so it can download that file. [Part2]
Echo Creating commands_2.ftp File (part2)
For /F "delims=" %%a in ('Type "%~1"') Do (
   For /F "delims=" %%b in (%temp%\%~8\server_list.lst) Do (
   Set "sf=%%b"
   setlocal enabledelayedexpansion
      IF "%%a"=="!sf:~1,-1!" (
         IF NOT EXIST "%~6\%%b" echo get %%b>>"%temp%\%~8\commands_2.ftp"
      )
   endlocal
   )
)

:: Add disconnect & bye commands to 2nd FTP command file [Part3]
Echo Creating commands_2.ftp File (part3)
(
Echo disconnect
Echo Bye
)>>"%temp%\%~8\commands_2.ftp"

:: start FTP session and download new files
Echo Connecting to FTP server to download files
Ftp -v -i -s:"%temp%\%~8\commands_2.ftp"

:: Check files count
Echo Counting files in local folder
set Fcount=0
For /F "delims=" %%a in ('Dir /B "%~6\*.%zipped%"') Do set /a Fcount += 1
echo %Fcount%

:: Check Condition to Exit or Continue Looping
Echo Check count number
Setlocal EnableDelayedExpansion
IF "%Fcount%" EQU "%~5" (
   Echo Unzipping
   "%zip_program%" x -y "%~6\*.%zipped%" -o"%~6" >nul
   Echo Deleting zip files
   Del /F /Q "%~6\*.%zipped%" >nul
   Echo Removing temp folders
   RMDIR /S /Q "%temp%\%~8" >nul
   Goto :EOF
) Else (
   Echo deleting server_list.lst file
   Del /F /Q "%temp%\%~8\server_list.lst" >nul
   :: Wait 5 min
   Ping Localhost -l 1 -n 300 >nul
   Goto :loop )
pause

[aaksar]

below is the output:, i have deleted the local folder. its created by the script, but file not copied.
i have installed 7zip, but seems still getting zip error

Code: Select all

Creating LOCAL FOLDER
Creating Commands_1.ftp file
Connicting to FTP server to get files list
Processing dump.log File and creating server_list.lst
Creating commands_2.ftp File (part1)
Creating commands_2.ftp File (part2)
The system cannot find the file specified.
Creating commands_2.ftp File (part3)
Connecting to FTP server to download files
ftp> open ftp.us.abc.com
User (ftp.us.abc.com:(none)):

ftp> lcd "D:\test"
Local directory now D:\test.
ftp> cd "1 abc Tables"
ftp> disconnect
ftp> Bye
Counting files in local folder
File Not Found
0
Check count number
deleting server_list.lst file

[aaksar]

you are checking the zipped file count?

Code: Select all

:: Check files count
Echo Counting files in local folder
set Fcount=0
For /F "delims=" %%a in ('Dir /B "%~6\*.%zipped%"') Do set /a Fcount += 1
echo %Fcount%

[abc0502]

See This:

Code: Select all

Creating commands_2.ftp File (part1)
Creating commands_2.ftp File (part2)
The system cannot find the file specified.
Creating commands_2.ftp File (part3)

The problem in the part 2step, the file "server_list.lst".
something goes wrong in these 3 steps.

I will try to change these 3 steps, but can you check if these files exist"server_list.lst","dump.log" , delete all folders in your %temp% dir and add pause after this line:

Code: Select all

For /F "delims=" %%a in ('Dir /B "%~6\*.%zipped%"') Do set /a Fcount += 1
echo %Fcount%

and run the batch and after it pauses, go to %temp% dir and post the content of these files here after removing sensitive information of them "server_list.lst", "dump.log"
Also put the content of this file too "FILES.lst1"