Performance issues variable VS file

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Post Reply
Message
Author
Ed Dyreen
Expert
Posts: 1569
Joined: 16 May 2011 08:21
Location: Flanders(Belgium)
Contact:

Performance issues variable VS file

#1 Post by Ed Dyreen » 20 Feb 2012 12:02

'
Trying to figure out the best way to throw help on the console

Using a variable is easy, but it wastes memory :(

Code: Select all

set ^"$Usage.%$defines%=^
 use: ( %%%$defines%%% #vr, #fullPathFile, #section, #key, #sErr,0 )%$n1c%
 con: contents of $key%$n1c%
 err: Unaffected, panic otherwise"
So I try some clever file enum trick

Code: Select all

:: ------------------------------------------------------------------------------------------
set "$defines=File.readINI_" &set "$details=ini file handler"
:: ------------------------------------------------------------------------------------------
:: last updated       : 18/02/2012
:: support            : naDelayed, delayedChars, related chaining
::
:H use: ( %$defines% #vr, #fullPathFile, #section, #key, #sErr,0 )
:H con: contents of $key
:H err: Unaffected, panic otherwise
::

Code: Select all

set "$Debug.Use=File.readINI_"

for /f "delims=[]" %%? in (

   'type "!$DOS-KIT.fullPathFile!" ^|find /v /n "type" ^|find "$defines=%$Debug.Use%"'

) do    set "$start=%%?"

for /f %%? in (

   'type "!$DOS-KIT.fullPathFile!" ^|find /c /v ""'

) do    set "$end=%%?"

set "$start"
set "$end"

< "!$DOS-KIT.fullPathFile!" ( set "$at=0"

   for /l %%ç in (

      1, 1, !$end!

   ) do    set "$=" &set /p "$=" &if %%ç geq !$start! if defined $at if "!$:~0,3!" == ":H " (

      set "$at=1" &for %%? in ( "!$:~3!" ) do call set "$=%%~?" &echo( !$!

   ) else    if !$at! neq 0 set "$at="
)
But now it wastes 60secs to process a file of +-18.000 lines :evil:

Code: Select all

$start=17428
$end=18430
 use: ( File.readINI_ #vr, #fullPathFile, #section, #key, #sErr,0 )
 con: contents of $key
 err: Unaffected, panic otherwise

 endoftest Druk op een toets om door te gaan. . .
I want to reduce memory load and have it fast :cry:

I was also thinking about a solution where I initially load the help variables and when the loading is done, write them out to a help file that can be smaller and searched more easily.

Ed Dyreen
Expert
Posts: 1569
Joined: 16 May 2011 08:21
Location: Flanders(Belgium)
Contact:

Re: Performance issues variable VS file

#2 Post by Ed Dyreen » 21 Feb 2012 10:31

'
Great, I save 12kB of memory and have a 12kB usage file that can be browsed easily by both user and program.

Gen file from mem

Code: Select all

(
   for /f "tokens=1 delims==" %%? in ( 'set $Usage.' ) do set "?=%%?" &if "!?:~0,7!" == "$Usage." (

      echo.
      echo.[!?:~7!]
      echo.!%%?!
   )

)> "!$DOS-KIT.fullPath!\DOS-KIT.TMP"

more < "!$DOS-KIT.fullPath!\DOS-KIT.TMP" > "!$DOS-KIT.fullPath!\DOS-KIT.NFO" &del /f /q "!$DOS-KIT.fullPath!\DOS-KIT.TMP"
enum and extract (any)

Code: Select all

set "$Debug.Use=User.Logon_"

for /f "delims=[]" %%? in (

   'type "!$DOS-KIT.fullPath!\DOS-KIT.NFO" ^|find /n "[%$Debug.Use%]"'

) do    set "$start=%%?"

for /f %%? in (

   'type "!$DOS-KIT.fullPath!\DOS-KIT.NFO" ^|find /c /v ""'

) do    set "$end=%%?"

< "!$DOS-KIT.fullPath!\DOS-KIT.NFO" ( set "$at=0"

   for /l %%! in (

      1, 1, !$end!

   ) do    set "$=" &set /p "$=" &if %%! geq !$start! if defined $at if "!$!" == "[%$Debug.Use%]" (

      set "$at=1"

   ) else    if !$at! neq 0 if defined $ (echo.!$!) else set "$at="
)
release mem

Code: Select all

for /f "tokens=1 delims==" %%? in ('set $Usage.') do if defined %%? (

   set "%%?="
)

Ed Dyreen
Expert
Posts: 1569
Joined: 16 May 2011 08:21
Location: Flanders(Belgium)
Contact:

Re: Performance issues variable VS file

#3 Post by Ed Dyreen » 21 Feb 2012 14:54

'
I can decide where to start file enum using

Code: Select all

more +n
I can decide where to end file enum using

Code: Select all

<file (for /l %%! in (1,1,!end!) do set $=&set/p$=)
But I have no clue how to use em both together :(

Code: Select all

<file (for /l %%! in (!start!,1,!end!) do set $=&set/p$=)
I investigate the more command, it say "P n shows next n lines". Can't seem to get it working :?

Is it possible :roll:

Squashman
Expert
Posts: 4465
Joined: 23 Dec 2011 13:59

Re: Performance issues variable VS file

#4 Post by Squashman » 21 Feb 2012 15:29

P n is an extended feature and as far as I know it is an interactive command. Not sure if you can use it in a script.

I actually found out today that more +n will stop at 65534 lines and will then output -- more -- to your output file. But I would assume you don't deal with large files like I do.

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: Performance issues variable VS file

#5 Post by dbenham » 21 Feb 2012 23:06

Squashman wrote:I actually found out today that more +n will stop at 65534 lines and will then output -- more -- to your output file. But I would assume you don't deal with large files like I do.

:?: :?
What version of Windows :?:

I wrote a batch implementation of tail using FIND /C to count the lines and MORE to skip all but the last n lines. I tested on Vista and successfully printed the last 10 lines of a file with 131,000+ lines.

Dave Benham

foxidrive
Expert
Posts: 6031
Joined: 10 Feb 2012 02:20

Re: Performance issues variable VS file

#6 Post by foxidrive » 22 Feb 2012 02:15

Squashman wrote:found out today that more +n will stop at 65534 lines and will then output -- more -- to your output file.


Confirmed on XP Pro SP3

foxidrive
Expert
Posts: 6031
Joined: 10 Feb 2012 02:20

Re: Performance issues variable VS file

#7 Post by foxidrive » 22 Feb 2012 02:16

dbenham wrote:I wrote a batch implementation of tail using FIND /C to count the lines and MORE to skip all but the last n lines. I tested on Vista and successfully printed the last 10 lines of a file with 131,000+ lines.



try skipping the first 10 lines...

I confirmed it on Vista and Windows 7 too:

for /L %a in (1,1,70000) do @echo a>>abc.txt
and wait...
more +10 abc.txt >abc2.txt
wait and press enter a few times then control C
find /i "more"<abc2.txt

Squashman
Expert
Posts: 4465
Joined: 23 Dec 2011 13:59

Re: Performance issues variable VS file

#8 Post by Squashman » 22 Feb 2012 06:21

foxidrive wrote:
Squashman wrote:found out today that more +n will stop at 65534 lines and will then output -- more -- to your output file.


Confirmed on XP Pro SP3

Bingo.
I was helping another user on another forum and someone suggested that they use:

Code: Select all

more +5 input.txt>output.txt

to skip the first 5 lines of a file and output to a new file.

That does work for most files but when I tried it on on my client files which had over 100,000 records I noticed the batch file was just hanging. I then looked in Explorer to see if the file was getting any bigger. It wasn't so I killed the batch file and then looked at the bottom of the file and saw this.

Code: Select all

-- More (4%) --


Ed, my apologies if we are getting off topic again.

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: Performance issues variable VS file

#9 Post by dbenham » 22 Feb 2012 06:59

Thanks Squashman and foxidrive

I misinterpreted the bad news. I thought Squashman was implying MORE can't skip more than 65534 lines. Now I realize that redirected or piped MORE cannot print more than 65534 lines without pausing and asking for a key press.

The behavior has nothing to do with the +n option. It pauses even if the +n option is absent and you start printing from the first line.

This is a major bummer :cry:

Dave Benham

Squashman
Expert
Posts: 4465
Joined: 23 Dec 2011 13:59

Re: Performance issues variable VS file

#10 Post by Squashman » 22 Feb 2012 10:26

dbenham wrote:This is a major bummer :cry:

I agree. I was hoping for a better outcome as well.

Post Reply