Page 1 of 2

Speed Writing a File

Posted: 06 Dec 2011 07:46
by shadeclan
:?: :?: Is there a faster (batch) way to write to a file besides the echo command? I could create an object in VBScript that accesses the scripting.FileSystemObject but I was wondering if there was a pure DOS method? The object of this question is to rewrite a larger file while leaving some lines out and echo seems to take a long time doing it.

Re: Speed Writing a File

Posted: 06 Dec 2011 08:14
by jeb
Hi shadeclan,

the speed depends extremly by the way you choose.

Slow variant:

Code: Select all

>myfile.txt echo Line1
>>myfile.txt echo Line2
>>myfile.txt echo.
>>myfile.txt echo Line4


Faster is

Code: Select all

(
  echo Line1
  echo Line2
  echo(
  echo Line4
) > myFile.txt


It's faster to combine the echo's into one block and redirect only once.

Using of echo. or somthing else to create an empty line is slow as it accesses the disk, better use echo(

jeb

Re: Speed Writing a File

Posted: 06 Dec 2011 08:42
by !k

Code: Select all

@echo off
>myFile.txt more +10 %0
for /f %%z in ("myFile.txt") do if not "%%~zz"=="23" (echo:Wrong myFile.txt!&pause&exit)

:: Your code is here
:: Your code is here

goto :eof

:: 10th line of batch
Line1
Line2

Line4

Re: Speed Writing a File

Posted: 06 Dec 2011 09:21
by shadeclan
jeb wrote:... It's faster to combine the echo's into one block and redirect only once ...

This hadn't occurred to me - I will try it and see if it speeds things up.

Re: Speed Writing a File

Posted: 06 Dec 2011 11:17
by shadeclan
!k wrote:

Code: Select all

@echo off
>myFile.txt more +10 %0
for /f %%z in ("myFile.txt") do if not "%%~zz"=="23" (echo:Wrong myFile.txt!&pause&exit)

:: Your code is here
:: Your code is here

goto :eof

:: 10th line of batch
Line1
Line2

Line4
!k - sorry, but I don't understand this, nor do I see how this is going to help me rewrite a file faster.

Re: Speed Writing a File

Posted: 06 Dec 2011 11:28
by !k
read more/?

Re: Speed Writing a File

Posted: 06 Dec 2011 11:33
by shadeclan
!k wrote:read more/?
I did. It displays one screen of output at a time. I'm not sure how that helps me write a file faster. Sorry - I'm stupid, OK? I don't get it.

Re: Speed Writing a File

Posted: 06 Dec 2011 11:39
by !k
more/? wrote:+n : Start displaying the first file at line n

more +10 will start display that batch at line №10

Re: Speed Writing a File

Posted: 06 Dec 2011 14:41
by shadeclan
!k wrote:
more/? wrote:+n : Start displaying the first file at line n

more +10 will start display that batch at line №10
Wow - that was like, amazing! You're explanation was a little verbose but a little playing around with it resolved the problem completely! Thank you so very much!

Re: Speed Writing a File

Posted: 07 Dec 2011 13:33
by shadeclan
Here is my final solution, heavily commented for us know-nothings:

Code: Select all

:: Generate name of temporary data file.
   set vDatFile="%vAppDir%DTF_FTP_%random%.dat"
:: Generate name of temporary working file.
   set vTmpFile="%vAppDir%DTF_FTP_%random%.tmp"
:: Generate name / location of log file.
   set vLogFile="%vAppDir%DTF_FTP.log"

   set /a "vCntr=0"

:: Delete logfile entries older than 52 runs ago, which should be about a year.
:: First, find the starting positions of the individual log entries in the log file
:: and put all the header lines into a data file along with the line positions where
:: they can be found.  Skip 2 lines for the header generated by FIND.
   find /n "*************** DTF FTP Start" %vLogFile% | more /s +2 > %vDatFile% rem (Yay MORE!!!  Thanks to !K!)
:: (First you find out how to use the pipe, then it becomes indispensable!)

:: Count the number of log headers deposited in the vDatFile file and put
:: it into the vCntr variable.
:: Set the vCntr variable = the difference between the current number of log
:: entries and the number of log entries expected over a year. 
:: Many thanks to Brett Batie
:: (http://brett.batie.com/scripting/count-number-of-lines-in-a-file-using-dos/)
:: for the counting algorithym.
   findstr /l /n "[" %vDatFile% | find /c ":"> %vTmpFile%
   set /p vCntr=<%vTmpFile% &del /q %vTmpFile%
   set /a vCntr-=52

:: Check to see if vCntr is > 0.  If not, skip log modification.
   if %vCntr% LEQ 0 (goto SkipLog)

:: If there are over a year's worth of logs in the logfile, delete earlier
:: log entries.
:: Set up the vCntr variable as a SKIP parameter for a FOR loop using the
:: calculated extra number of log entries as the SKIP parameter value. 
:: Thanks to dbenham (http://www.dostips.com/forum/viewtopic.php?f=3&t=2490)
:: for the explanation on how to do this.
   set vCntr=skip=%vCntr%
:: Access the vDatFile file and set the vCntr variable = the line number
:: of the beginning log entry which will not be deleted.  Subtract 1 from
:: that number to preserve the log header.
   for /f "usebackq %vCntr% delims=[] tokens=1,2,*" %%a in (%vDatFile%) do (
      set "vCntr=%%a"
      set /a vCntr-=1
      goto Break2
   )
   :Break2

:: Transfer the logfile, minus the earlier log entries, to the vDatFile file.
:: Tremendous amount of thanks to !K for this solution!
   more +%vCntr% %vLogFile% > %vDatFile%
:: Finally, copy over the old logfile with the revised log.
   copy %vDatFile% %vLogFile%

   :SkipLog

... and that's it! The last, vital part of a self-managing, self-generating FTP script!

Thanks again to everyone who helped!

Re: Speed Writing a File

Posted: 08 Dec 2011 22:33
by orange_batch
Good. Commenting is good practice too. However I personally would use that as source, and have a working copy that's stripped and minified. No need on such a short script though.

Re: Speed Writing a File

Posted: 09 Dec 2011 07:53
by shadeclan
orange_batch wrote:Good. Commenting is good practice too. However I personally would use that as source, and have a working copy that's stripped and minified. No need on such a short script though.
Well, it may not be needful for you but I have a mind like a steel sieve and considering that I am programming in DOS Batch, JCL, M204 code, Postscript, HTML, CSS, Javascript and PL/SQL (Oracle) with occasional forays into VB6 / VBScript and VB.net - well, let's just say that by the time I get around the circle, I almost have to relearn the language again. Heavy comments in the code are one way I save myself when something breaks and I haven't programmed in that language for a while.

If you find the code useful, by all means remove the comments and use it all or in part - I don't believe in intellectual property rights.

Re: Speed Writing a File

Posted: 09 Dec 2011 11:22
by Ed Dyreen
'
Shadeclan wrote:I don't believe in intellectual property rights.
What u mean, you don't have any :lol:
Admin wrote:The Forum:
By submitting information to the forum you transfer the rights of its content to the domain owner. The domain owner reserve the right to add, modify, or remove information to/from the forum as the domain owner feels is appropriate.
The domain owner further reserves the right to use and distribute the information submitted in the forum in a different form, reformat it rearrange it and/or sell it.
See ya ( virtually ),
:mrgreen:

Re: Speed Writing a File

Posted: 09 Dec 2011 11:26
by shadeclan
Ed Dyreen wrote:'
Shadeclan wrote:I don't believe in intellectual property rights.
What do u mean, you don't have any :lol:
Admin wrote:The Forum:
By submitting information to the forum you transfer the rights of its content to the domain owner. The domain owner reserve the right to add, modify, or remove information to/from the forum as the domain owner feels is appropriate.
The domain owner further reserves the right to use and distribute the information submitted in the forum in a different form, reformat it rearrange it and/or sell it.
I don't believe in his intellectual property rights, either. :twisted:

Did you see? Somebody actually liked something I pieced together!

Re: Speed Writing a File

Posted: 09 Dec 2011 18:34
by orange_batch
shadeclan wrote:Well, it may not be needful for you but I have a mind like a steel sieve

Heavy comments in the code are one way I save myself when something breaks and I haven't programmed in that language for a while.

If you find the code useful, by all means remove the comments and use it all or in part - I don't believe in intellectual property rights.

Haha hold on, you misread what I said, I comment my code too. But for long scripts I keep that as source, then strip and minify a copy for using.