FTP Download only new files any file type

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Post Reply
Message
Author
CastorTroy
Posts: 3
Joined: 05 Jul 2013 13:36

FTP Download only new files any file type

#1 Post by CastorTroy » 05 Jul 2013 13:53

I'm trying to use the FTP download batch script, but I'm not having any luck at all. I want it to download ANY file type, not just .txt's as the script was originally written for. I've tried modifying the "Set FindStrArgs=/E /C:".txt"" a few different ways, but none of them have produced the desired results. They seem to either break the script, or result in no files being downloaded. Here are some of the ways I've tried:

Set FindStrArgs=/E /C:"."
Set FindStrArgs=/E /C:".*"
Set FindStrArgs=/E /C:"*.*"
Set FindStrArgs=/E /C:.
Set FindStrArgs=/E /R .*

Any help would be appreciated.

Sponge Belly
Posts: 216
Joined: 01 Oct 2012 13:32
Location: Ireland
Contact:

Re: FTP Download only new files any file type

#2 Post by Sponge Belly » 06 Jul 2013 15:44

Hi CastorTroy! :-)

FindStr regular expressions are different from wildcard patterns. Try something like this:

Code: Select all

set FindStrArgs=/x "..*\...*"


The above should output only filenames containing at least one dot. If it doesn’t, let me know. If that isn’t what you need, again get back to me and we’ll have another stab at it.

Good luck! ;-)

- SB

CastorTroy
Posts: 3
Joined: 05 Jul 2013 13:36

Re: FTP Download only new files any file type

#3 Post by CastorTroy » 08 Jul 2013 07:11

This is the output I'm getting with that search string now. It did work, even with all the errors. Is there any way to figure out what is causing the script to break and show all the errors?

Code: Select all

C:\Users\(edited)\Desktop>ftptest.bat
mget
"User (127.0.0.1:(none)): open 127.0.0.1"
"ftp Processing... ls"
"9.txt"
"ie2.jpg"
ftp> !Title Connecting...
ftp> open 127.0.0.1
Connected to 127.0.0.1.
220-FileZilla Server version 0.9.41 beta
220-written by Tim Kosse (Tim.Kosse@gmx.de)
220 Please visit http://sourceforge.net/projects/filezilla/
User (127.0.0.1:(none)):
331 Password required for test

230 Logged on
ftp>
ftp> !Title Preparing...
ftp> cd /
250 CWD successful. "/" is current directory.
ftp> lcd C:\Users\(edited)\Desktop\schedulertest
Local directory now C:\Users\(edited)\Desktop\schedulertest.
ftp> binary
200 Type set to I
ftp> hash
Hash mark printing On  ftp: (2048 bytes/hash mark) .
ftp>
ftp> !Title Processing... mget "User (127.0.0.1:(none)): open 127.0.0.1" "ftp Pr
ocessing... ls" "9.txt" "ie2.jpg"
ftp> mget "User (127.0.0.1:(none)): open 127.0.0.1" "ftp Processing... ls" "9.tx
t" "ie2.jpg"
Directory not found
Directory not found
200 Type set to I
200 Port command successful
150 Opening data channel for file transfer.
226 Transfer OK
200 Port command successful
150 Opening data channel for file transfer.
########################226 Transfer OK
ftp: 50095 bytes received in 0.00Seconds 25047.50Kbytes/sec.
ftp>
ftp> !Title Disconnecting...
ftp> disconnect
221 Goodbye
ftp> bye

C:\Users\(edited)\Desktop>

Sponge Belly
Posts: 216
Joined: 01 Oct 2012 13:32
Location: Ireland
Contact:

Re: FTP Download only new files any file type

#4 Post by Sponge Belly » 08 Jul 2013 09:04

Hi Again!

Please post your source code. Trim comments and remove usernames and passwords. But one thing I did notice—and this is a long shot—is a missing quote before the ls command.

Good luck! ;-)

- SB

CastorTroy
Posts: 3
Joined: 05 Jul 2013 13:36

Re: FTP Download only new files any file type

#5 Post by CastorTroy » 08 Jul 2013 09:22

Code: Select all

@Echo Off

REM -- Define File Filter, i.e. files with extension .txt
set FindStrArgs=/x "..*\...*"

REM -- Extract Ftp Script to create List of Files
Set "FtpCommand=ls"
Call:extractFileSection "[Ftp Script 1]" "-">"%temp%\%~n0.ftp"
Rem Notepad "%temp%\%~n0.ftp"

REM -- Execute Ftp Script, collect File Names
Set "FileList="
For /F "Delims=" %%A In ('"Ftp -v -i -s:"%temp%\%~n0.ftp"|Findstr %FindStrArgs%"') Do (
    Call Set "FileList=%%FileList%% "%%A""
)

REM -- Extract Ftp Script to download files that don't exist in local folder
Set "FtpCommand=mget"
For %%A In (%FileList%) Do If Not Exist "%%~A" Call Set "FtpCommand=%%FtpCommand%% "%%~A""
Call:extractFileSection "[Ftp Script 1]" "-">"%temp%\%~n0.ftp"
Rem Notepad "%temp%\%~n0.ftp"

For %%A In (%FtpCommand%) Do Echo.%%A

REM -- Execute Ftp Script, download files
ftp -i -s:"%temp%\%~n0.ftp"
Del "%temp%\%~n0.ftp"
GOTO:EOF


:extractFileSection StartMark EndMark FileName -- extract a section of file that is defined by a start and end mark
::                  -- [IN]     StartMark - start mark, use '...:S' mark to allow variable substitution
::                  -- [IN,OPT] EndMark   - optional end mark, default is first empty line
::                  -- [IN,OPT] FileName  - optional source file, default is THIS file
:$created 20080219 :$changed 20100205 :$categories ReadFile
:$source http://www.dostips.com
SETLOCAL Disabledelayedexpansion
set "bmk=%~1"
set "emk=%~2"
set "src=%~3"
set "bExtr="
set "bSubs="
if "%src%"=="" set src=%~f0&        rem if no source file then assume THIS file
for /f "tokens=1,* delims=]" %%A in ('find /n /v "" "%src%"') do (
    if /i "%%B"=="%emk%" set "bExtr="&set "bSubs="
    if defined bExtr if defined bSubs (call echo.%%B) ELSE (echo.%%B)
    if /i "%%B"=="%bmk%"   set "bExtr=Y"
    if /i "%%B"=="%bmk%:S" set "bExtr=Y"&set "bSubs=Y"
)
EXIT /b


[Ftp Script 1]:S

open 127.0.0.1
test
test


cd /
lcd C:\Users\(edited)\Desktop\schedulertest
binary
hash


%FtpCommand%


disconnect
bye

Sponge Belly
Posts: 216
Joined: 01 Oct 2012 13:32
Location: Ireland
Contact:

Re: FTP Download only new files any file type

#6 Post by Sponge Belly » 09 Jul 2013 17:58

Hi Again! :-)

I tinkered with your version of the FTP script and eventually got it to work:

Code: Select all

@Echo Off & setlocal enableextensions
Set "FtpCommand=ls"
Call:extractFileSection "[Ftp Script 1]" "-">"%temp%\%~n0.ftp"

Set "FileList="
For /F "Delims=" %%A In ('"Ftp -v -i -s:"%temp%\%~n0.ftp"|Findstr /x ".*README.*""') Do (
Call Set "FileList=%%FileList%% "%%A"")

Set "FtpCommand=mget"
For %%A In (%FileList%) Do If Not Exist "%tmp%\%%~A" Call Set "FtpCommand=%%FtpCommand%% "%%~A""
Call:extractFileSection "[Ftp Script 1]" "-">"%temp%\%~n0.ftp"

ftp -v -i -s:"%temp%\%~n0.ftp"
Del "%temp%\%~n0.ftp"

endlocal & GOTO:EOF

:extractFileSection StartMark EndMark FileName
SETLOCAL Disabledelayedexpansion
set "bmk=%~1"
set "emk=%~2"
set "src=%~3"
set "bExtr="
set "bSubs="
if "%src%"=="" set src=%~f0&        rem if no source file then assume THIS file
for /f "tokens=1,* delims=]" %%A in ('find /n /v "" "%src%"') do (
if /i "%%B"=="%emk%" set "bExtr="&set "bSubs="
if defined bExtr if defined bSubs (call echo.%%B) ELSE (echo.%%B)
if /i "%%B"=="%bmk%"   set "bExtr=Y"
if /i "%%B"=="%bmk%:S" set "bExtr=Y"&set "bSubs=Y"
)
endlocal & EXIT /b 0

[Ftp Script 1]:S
open ftp.x.org
anonymous
-nemo@nosuch.com
cd /
lcd %tmp%
ascii
%FtpCommand%
bye


I hard-coded the findstr regular expression to search for any filename containing the substring “README”, and I used %tmp% as the location to check for pre-existing files (see line 10). The script performed as expected in my ftp.x.org tests. I initially downloaded three files with “README” in their names. I deleted one and ran the script again. This time it only downloaded the missing file.

Hope this helps! ;-)

- SB

thlockler
Posts: 1
Joined: 11 Oct 2013 08:47

Re: FTP Download only new files any file type

#7 Post by thlockler » 11 Oct 2013 08:51

In regards to the script you are editing, have you figured out a way to change the file in which it compares before downloading? I can change where it looks for files to download and where it downloads them too, however, when the script attempts to compare the two directories, it only looks at where the script is being executed from and not where I am attempting to place the files.

Sponge Belly
Posts: 216
Joined: 01 Oct 2012 13:32
Location: Ireland
Contact:

Re: FTP Download only new files any file type

#8 Post by Sponge Belly » 23 Oct 2013 12:03

Hi Thlockler,

Sorry for not replying sooner. I was on my holidays! :-)

Anyways, I don’t fully understand your question.

The script makes two connections to the ftp host. On the first pass, it builds a list of all files matching the findstr pattern. On the second pass, it downloads only those files that don't already exist on the local host. I hard-coded %tmp% as the local folder, but you can change that to whatever you want.

- SB

Post Reply