Speed Writing a File

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Message
Author
shadeclan
Posts: 61
Joined: 02 Jun 2011 11:29
Location: USA - Somewhere between Albany NY and Bennington VT

Speed Writing a File

#1 Post by shadeclan » 06 Dec 2011 07:46

:?: :?: Is there a faster (batch) way to write to a file besides the echo command? I could create an object in VBScript that accesses the scripting.FileSystemObject but I was wondering if there was a pure DOS method? The object of this question is to rewrite a larger file while leaving some lines out and echo seems to take a long time doing it.

jeb
Expert
Posts: 967
Joined: 30 Aug 2007 08:05
Location: Germany, Bochum

Re: Speed Writing a File

#2 Post by jeb » 06 Dec 2011 08:14

Hi shadeclan,

the speed depends extremly by the way you choose.

Slow variant:

Code: Select all

>myfile.txt echo Line1
>>myfile.txt echo Line2
>>myfile.txt echo.
>>myfile.txt echo Line4


Faster is

Code: Select all

(
  echo Line1
  echo Line2
  echo(
  echo Line4
) > myFile.txt


It's faster to combine the echo's into one block and redirect only once.

Using of echo. or somthing else to create an empty line is slow as it accesses the disk, better use echo(

jeb

!k
Expert
Posts: 378
Joined: 17 Oct 2009 08:30
Location: Russia

Re: Speed Writing a File

#3 Post by !k » 06 Dec 2011 08:42

Code: Select all

@echo off
>myFile.txt more +10 %0
for /f %%z in ("myFile.txt") do if not "%%~zz"=="23" (echo:Wrong myFile.txt!&pause&exit)

:: Your code is here
:: Your code is here

goto :eof

:: 10th line of batch
Line1
Line2

Line4

shadeclan
Posts: 61
Joined: 02 Jun 2011 11:29
Location: USA - Somewhere between Albany NY and Bennington VT

Re: Speed Writing a File

#4 Post by shadeclan » 06 Dec 2011 09:21

jeb wrote:... It's faster to combine the echo's into one block and redirect only once ...

This hadn't occurred to me - I will try it and see if it speeds things up.

shadeclan
Posts: 61
Joined: 02 Jun 2011 11:29
Location: USA - Somewhere between Albany NY and Bennington VT

Re: Speed Writing a File

#5 Post by shadeclan » 06 Dec 2011 11:17

!k wrote:

Code: Select all

@echo off
>myFile.txt more +10 %0
for /f %%z in ("myFile.txt") do if not "%%~zz"=="23" (echo:Wrong myFile.txt!&pause&exit)

:: Your code is here
:: Your code is here

goto :eof

:: 10th line of batch
Line1
Line2

Line4
!k - sorry, but I don't understand this, nor do I see how this is going to help me rewrite a file faster.

!k
Expert
Posts: 378
Joined: 17 Oct 2009 08:30
Location: Russia

Re: Speed Writing a File

#6 Post by !k » 06 Dec 2011 11:28

read more/?

shadeclan
Posts: 61
Joined: 02 Jun 2011 11:29
Location: USA - Somewhere between Albany NY and Bennington VT

Re: Speed Writing a File

#7 Post by shadeclan » 06 Dec 2011 11:33

!k wrote:read more/?
I did. It displays one screen of output at a time. I'm not sure how that helps me write a file faster. Sorry - I'm stupid, OK? I don't get it.

!k
Expert
Posts: 378
Joined: 17 Oct 2009 08:30
Location: Russia

Re: Speed Writing a File

#8 Post by !k » 06 Dec 2011 11:39

more/? wrote:+n : Start displaying the first file at line n

more +10 will start display that batch at line №10

shadeclan
Posts: 61
Joined: 02 Jun 2011 11:29
Location: USA - Somewhere between Albany NY and Bennington VT

Re: Speed Writing a File

#9 Post by shadeclan » 06 Dec 2011 14:41

!k wrote:
more/? wrote:+n : Start displaying the first file at line n

more +10 will start display that batch at line №10
Wow - that was like, amazing! You're explanation was a little verbose but a little playing around with it resolved the problem completely! Thank you so very much!

shadeclan
Posts: 61
Joined: 02 Jun 2011 11:29
Location: USA - Somewhere between Albany NY and Bennington VT

Re: Speed Writing a File

#10 Post by shadeclan » 07 Dec 2011 13:33

Here is my final solution, heavily commented for us know-nothings:

Code: Select all

:: Generate name of temporary data file.
   set vDatFile="%vAppDir%DTF_FTP_%random%.dat"
:: Generate name of temporary working file.
   set vTmpFile="%vAppDir%DTF_FTP_%random%.tmp"
:: Generate name / location of log file.
   set vLogFile="%vAppDir%DTF_FTP.log"

   set /a "vCntr=0"

:: Delete logfile entries older than 52 runs ago, which should be about a year.
:: First, find the starting positions of the individual log entries in the log file
:: and put all the header lines into a data file along with the line positions where
:: they can be found.  Skip 2 lines for the header generated by FIND.
   find /n "*************** DTF FTP Start" %vLogFile% | more /s +2 > %vDatFile% rem (Yay MORE!!!  Thanks to !K!)
:: (First you find out how to use the pipe, then it becomes indispensable!)

:: Count the number of log headers deposited in the vDatFile file and put
:: it into the vCntr variable.
:: Set the vCntr variable = the difference between the current number of log
:: entries and the number of log entries expected over a year. 
:: Many thanks to Brett Batie
:: (http://brett.batie.com/scripting/count-number-of-lines-in-a-file-using-dos/)
:: for the counting algorithym.
   findstr /l /n "[" %vDatFile% | find /c ":"> %vTmpFile%
   set /p vCntr=<%vTmpFile% &del /q %vTmpFile%
   set /a vCntr-=52

:: Check to see if vCntr is > 0.  If not, skip log modification.
   if %vCntr% LEQ 0 (goto SkipLog)

:: If there are over a year's worth of logs in the logfile, delete earlier
:: log entries.
:: Set up the vCntr variable as a SKIP parameter for a FOR loop using the
:: calculated extra number of log entries as the SKIP parameter value. 
:: Thanks to dbenham (http://www.dostips.com/forum/viewtopic.php?f=3&t=2490)
:: for the explanation on how to do this.
   set vCntr=skip=%vCntr%
:: Access the vDatFile file and set the vCntr variable = the line number
:: of the beginning log entry which will not be deleted.  Subtract 1 from
:: that number to preserve the log header.
   for /f "usebackq %vCntr% delims=[] tokens=1,2,*" %%a in (%vDatFile%) do (
      set "vCntr=%%a"
      set /a vCntr-=1
      goto Break2
   )
   :Break2

:: Transfer the logfile, minus the earlier log entries, to the vDatFile file.
:: Tremendous amount of thanks to !K for this solution!
   more +%vCntr% %vLogFile% > %vDatFile%
:: Finally, copy over the old logfile with the revised log.
   copy %vDatFile% %vLogFile%

   :SkipLog

... and that's it! The last, vital part of a self-managing, self-generating FTP script!

Thanks again to everyone who helped!

orange_batch
Expert
Posts: 442
Joined: 01 Aug 2010 17:13
Location: Canadian Pacific
Contact:

Re: Speed Writing a File

#11 Post by orange_batch » 08 Dec 2011 22:33

Good. Commenting is good practice too. However I personally would use that as source, and have a working copy that's stripped and minified. No need on such a short script though.

shadeclan
Posts: 61
Joined: 02 Jun 2011 11:29
Location: USA - Somewhere between Albany NY and Bennington VT

Re: Speed Writing a File

#12 Post by shadeclan » 09 Dec 2011 07:53

orange_batch wrote:Good. Commenting is good practice too. However I personally would use that as source, and have a working copy that's stripped and minified. No need on such a short script though.
Well, it may not be needful for you but I have a mind like a steel sieve and considering that I am programming in DOS Batch, JCL, M204 code, Postscript, HTML, CSS, Javascript and PL/SQL (Oracle) with occasional forays into VB6 / VBScript and VB.net - well, let's just say that by the time I get around the circle, I almost have to relearn the language again. Heavy comments in the code are one way I save myself when something breaks and I haven't programmed in that language for a while.

If you find the code useful, by all means remove the comments and use it all or in part - I don't believe in intellectual property rights.

Ed Dyreen
Expert
Posts: 1569
Joined: 16 May 2011 08:21
Location: Flanders(Belgium)
Contact:

Re: Speed Writing a File

#13 Post by Ed Dyreen » 09 Dec 2011 11:22

'
Shadeclan wrote:I don't believe in intellectual property rights.
What u mean, you don't have any :lol:
Admin wrote:The Forum:
By submitting information to the forum you transfer the rights of its content to the domain owner. The domain owner reserve the right to add, modify, or remove information to/from the forum as the domain owner feels is appropriate.
The domain owner further reserves the right to use and distribute the information submitted in the forum in a different form, reformat it rearrange it and/or sell it.
See ya ( virtually ),
:mrgreen:

shadeclan
Posts: 61
Joined: 02 Jun 2011 11:29
Location: USA - Somewhere between Albany NY and Bennington VT

Re: Speed Writing a File

#14 Post by shadeclan » 09 Dec 2011 11:26

Ed Dyreen wrote:'
Shadeclan wrote:I don't believe in intellectual property rights.
What do u mean, you don't have any :lol:
Admin wrote:The Forum:
By submitting information to the forum you transfer the rights of its content to the domain owner. The domain owner reserve the right to add, modify, or remove information to/from the forum as the domain owner feels is appropriate.
The domain owner further reserves the right to use and distribute the information submitted in the forum in a different form, reformat it rearrange it and/or sell it.
I don't believe in his intellectual property rights, either. :twisted:

Did you see? Somebody actually liked something I pieced together!

orange_batch
Expert
Posts: 442
Joined: 01 Aug 2010 17:13
Location: Canadian Pacific
Contact:

Re: Speed Writing a File

#15 Post by orange_batch » 09 Dec 2011 18:34

shadeclan wrote:Well, it may not be needful for you but I have a mind like a steel sieve

Heavy comments in the code are one way I save myself when something breaks and I haven't programmed in that language for a while.

If you find the code useful, by all means remove the comments and use it all or in part - I don't believe in intellectual property rights.

Haha hold on, you misread what I said, I comment my code too. But for long scripts I keep that as source, then strip and minify a copy for using.

Post Reply