FTP pull data files without overwrite

[CheeseFox]

Hi ppl,

its nice to see such a forum still around. Batch files are sooo very useful at automating some tasks.

I'm writing a script to pull datalogs from an offsite FTP server.
I'm having trouble finding way to write it so that the Script when pulling data does not auto-overwrite old files on local.

the server generates data files at say 10files a day and keeps a backlog of up to a month. I'm trying not to have to pull all 300+ files everyday just to have the few most updated files in my system. extremely bad for server since i'm not the only one doing such monitoring

in dos, theres the "Echo n | copy *.* c:\store" where it will auto N to overwrite.

can we do the same for Dos FTP?


Thanks people

Code: Select all

REM FTP System311Monitoring Daily Script

REM Going to storage Directory
cd System311Monitoring

REM Creating temporary script file
> script.ftp ECHO <username>
>>script.ftp ECHO <password>
>>script.ftp ECHO hash
>>script.ftp ECHO cd DataStore
>>script.ftp ECHO ascii
>>script.ftp ECHO prompt n
>>script.ftp ECHO mget *SysMon311*.*
>>script.ftp ECHO bye
:: Use the temporary script for unattended FTP
FTP -v -d -s:script.ftp <ftp address>
:: Delete script file after use.
DEL script.ftp

REM Back to Root directory
cd ..

REM FTP Log Pull Script Complete

[DosItHelp]

CheeseFox,

Check this out:
http://www.dostips.com/DtTipsFtpBatchScript.php#Batch_FtpBatchGetNewFilesOnly

This batch connects twice to the FTP server. First time it retrieves a list of files on the FTP server. This list is being trimmed to contain only files that don`t already exist locally. The files in the trimmed list are then downloaded during a second connection.

DosItHelp?

[CheeseFox]

Hi DosItHelp,


been going thru the code since i saw the post...

i've changed << %temp%\%~n0.ftp >> to something akin of << c:\temp.ftp>> to get it to create the temporary script file

figured out why the extractFileSection won't work with Echo On. (i like to use echo on to learn how things work... heeh... ended up Echo Off before extractFileSection and Echo On after it. )



but have hit a stumbling point..

on unmasking (removing REM) of the Notepad lines to check the script file, i noted that the script file was only changed from "ls" to "mget" with nothing else.

but... i cant seem to figure out how this section works :

Code: Select all

REM -- Execute Ftp Script, collect File Names
Set "FileList="
For /F "Delims=" %%A In ('"Ftp -v -i -s:"%temp%\%~n0.ftp"|Findstr %FindStrArgs%"') Do (
    Call Set "FileList=%%FileList%% "%%A""
)

i didnt see the script connecting to the FTP server the first time to get the list. I only saw the ftp connecting the 2nd time to run mget with no file list.


Any pointers on how that for loop works?


Thanks a million!

[DosItHelp]

CheeseFox,

Code: Select all

REM -- Execute Ftp Script, collect File Names
Set "FileList="
For /F "Delims=" %%A In ('"Ftp -v -i -s:"%temp%\%~n0.ftp"|Findstr %FindStrArgs%"') Do (
    Call Set "FileList=%%FileList%% "%%A""
)

is functionally identical with:

Code: Select all

REM -- Execute Ftp Script, collect File Names
Set "FileList="
For /F "Delims=" %%A In ('"Ftp -v -i -s:"%temp%\%~n0.ftp"|Findstr %FindStrArgs%"') Do (
    Set "FileList=!FileList! "%%A""
)

... except it's not using Delayedexpansion.
It executes the first ftp script (ls) and sends the output through FINDSTR in order to filter the file names that match the pattern defined in the FindStrArgs variable, i.e. all files ending with ".txt".
The FOR loop iterates all resulting file names and adds them to FileList. The resulting string in the FileList variable should be a space separated list of all file names in the remote directory.

What you could try is:
After the first FTP script is created in c:\temp.ftp run it and verify that it returns the list of files in the remote directory, i.e. by running
Ftp -v -i -s:"c:\temp.ftp"

Let me know how it goes.
DosItHelp

[CheeseFox]

Hi DosItHelp,

i'm getting the error "The input line is too long" at the line

Call Set "FileList=%%FileList%% "%%A""


................................


a side question, is it possible to pass the %%A variable value into a %temp1% variable for each loop?

i also noticed that the FTP server replies have a certain hidden character at the end of each line. only visible when opened in edit.com ... i think its screwing some things here.

[CheeseFox]

I also noticed some junk characters (#) when the 'ls' list is long.

So i guess i'll need to put the %%A into a variable to trim off the hidden character at the end and to remove all the junk #s in the filenames.
But i cant seem to do a "set temp1=%%A" or similar at all..


been a decade since i last did such stuff. Still kinda fun.. lolx...

Thanks man!

[CheeseFox]

here i am talking to myself... LMAO



ok.. i got it to work... in a fairly different way than the original code. but the basis is still there...

took me a day to figure out that i needed the "setlocal enabledelayedexpansion" environment variable for some stuff...


i'll clean up the code before posing it here...

hurrah!

thanks dude

[CheeseFox]

Not the shortest nor the most elegant of codes. but it works well.

theres a simple logging function encoded too that shows Start Time, New Files downloaded and End time.

the extension thing is a workaround as some FTP servers add a LFCR or such character at the end of every reply.

the "#" removal is to remove junk '#' characters that sometimes appear when listing many many files...

works for long file names...

yea.. i think thats all..


To use... replace all <<????>> with the appropriate things



Thanks DosItHelp for the inspiration


Cheers,
CheeseFox

Code: Select all

@echo off


REM -- Define Local Folder to store files
REM -- Full directory location pls. Eg: C:\temp
Set HomeLoc=<<local directory>>

REM -- Define logfile
Set LogFile=<<logfile name, eg:C:\temp\FTPAUTO.txt>>



REM -- Define FTP information
Set FTPAddress=<<ftp server>>
Set FTPUser=<<ftp username>>
Set FTPPassword=<<ftp password>>

REM -- Define Remote Folder to pull files
REM -- tip: the directory name after logging into ftp server
Set RemoteLoc=<<remote directory>>
Set ls_string=<<search string: either "ls" for all files or "ls *item1*" to download specific files>>

REM -- Define File Filter, 
REM -- :: example: files with extension .txt
REM -- :: Set FileType=ASCII or Binary?
REM -- :: Set Extension=.txt
Set FileType=ASCII
Set Extension=.txt


REM ################################################################################################
REM ################################################################################################
REM ################################################################################################



>>%LogFile% Echo.===========================
>>%LogFile% Echo Start Time:
            Echo exit|cmd /q /k prompt $D $T>>%LogFile%
>>%LogFile% Echo.
>>%LogFile% Echo.Local Dir: %HomeLoc%
>>%LogFile% Echo.Remote ftp: %FTPAddress%
>>%LogFile% Echo.Remote Dir: %RemoteLoc%
>>%LogFile% Echo.
>>%LogFile% Echo.ver0.74 : 16 Dec 2008 9.30am
>>%LogFile% Echo.===========================
>>%LogFile% Echo.



Set FindStrArgs=/E /C:"%Extension%"

Set ScpFile1=%HomeLoc%\ftpscript1.ftp
Set ScpFile2=%HomeLoc%\ftpscript2.ftp

cd %HomeLoc%

>>%LogFile% Echo.**Listing
REM -- Creating FTP script file for listing out FTP files
REM -------------------------------------------
>>%ScpFile1% Echo !Title Connecting to %FTPAddress%...
>>%ScpFile1% Echo open %FTPAddress%
>>%ScpFile1% Echo %FTPUser%
>>%ScpFile1% Echo %FTPPassword%
>>%ScpFile1% Echo 
>>%ScpFile1% Echo !Title Setting variables... %FTPAddress%...
>>%ScpFile1% Echo cd %RemoteLoc%
>>%ScpFile1% Echo lcd %HomeLoc%
>>%ScpFile1% Echo %FileType%
>>%ScpFile1% Echo hash
>>%ScpFile1% Echo 
>>%ScpFile1% Echo !Title Retrieving File List... %FTPAddress%...
>>%ScpFile1% Echo %ls_string%
>>%ScpFile1% Echo !Title Disconnecting... %FTPAddress%...
>>%ScpFile1% Echo disconnect
>>%ScpFile1% Echo bye
REM -------------------------------------------

REM -- Execute Ftp Script, collect File Names
Set "Listing="
Set "FtpCommand=get"
REM -------------------------------------------
>>%ScpFile2% Echo !Title Connecting to %FTPAddress%...
>>%ScpFile2% Echo open %FTPAddress%
>>%ScpFile2% Echo %FTPUser%
>>%ScpFile2% Echo %FTPPassword%
>>%ScpFile2% Echo 
>>%ScpFile2% Echo !Title Setting variables... %FTPAddress%...
>>%ScpFile2% Echo cd %RemoteLoc%
>>%ScpFile2% Echo lcd %HomeLoc%
>>%ScpFile2% Echo %FileType%
>>%ScpFile2% Echo hash
>>%ScpFile2% Echo 
>>%ScpFile2% Echo !Title Downloading Files... %FTPAddress%...


>>%LogFile% Echo.**Sorting
>>%LogFile% Echo.=== New Files Start ===
REM -------------------------------------------
setlocal enabledelayedexpansion

set Junksearch=#
set Junkreplace=
set NewFiles=0
For /F "Delims=" %%A In ('"Ftp -v -s:%ScpFile1%|Findstr %FindStrArgs%"') Do (
   set newline=%%~nA
   set newline=!newline:%Junksearch%=%Junkreplace%!
   set emptyline=
   if Not !newline!==!emptyline! (
      If Not Exist "!newline!%extension%" (
              >> %ScpFile2% Echo.%FtpCommand% !newline!%extension%
              >>%LogFile% Echo.%FtpCommand% !newline!%extension%
              set NewFiles=1
              )
      )
   )
>>%LogFile% Echo.=== New Files End ===
setlocal disabledelayedexpansion

REM -------------------------------------------

>>%ScpFile2% Echo !Title Disconnecting... %FTPAddress%...
>>%ScpFile2% Echo disconnect
>>%ScpFile2% Echo bye

if %NewFiles%==1 (
   REM -- Execute Ftp Script, download files
   >>%LogFile% Echo.**Retrieving
   ftp -v -d -s:%ScpFile2%
   >>%LogFile% Echo.ERROR LEVEL:: %ERRORLEVEL%
   ) ELSE (
   >>%LogFile% Echo.**No New Files
   )
   

Del %ScpFile1%
Del %ScpFile2%

>>%LogFile% Echo.**Ending Script

>>%LogFile% Echo.===========================
>>%LogFile% Echo End time:
            Echo exit|cmd /q /k prompt $D $T>>%LogFile%
>>%LogFile% Echo.
>>%LogFile% Echo.===========================
>>%LogFile% Echo.
>>%LogFile% Echo.
>>%LogFile% Echo.