Optimizing These Loops

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Message
Author
Samir
Posts: 345
Joined: 16 Jul 2013 12:00
Location: HSV
Contact:

Re: Optimizing These Loops

#16 Post by Samir » 03 Nov 2015 14:59

penpen wrote:I don't know if the following is much faster, but you could try to avoid using environment variables (instead this uses a tempfile; not tested):

Code: Select all

@echo off
setlocal
:: ... set current date ...

>list.temp (
   for %%f in ("DAY*.RPT" "PLU*.RPT" "SKU*.RPT" "*.txt") do echo(%%~tf %%~nx
)
for /F "token=2" %%f in ('findstr "^%currentdate%" "list.tmp"') do (
   for /F "token=1 delims=YU" %%t in ("%%~f") do (
      if        "DA" == "%%~t" ( SET DAY=%%f
      ) else if "PL" == "%%~t" ( SET PLU=%%f
      ) else if "SK" == "%%~t" ( SET SKU=%%f
      )
   )
)
del "list.temp"
endlocal
goto :eof
The first loop may be replaced by a dir command (may be faster); in that case you probably have to adjust the tokens in the other loops.


penpen
This is kind of what I was thinking too as a way to speed it up. I like the delimiters usage--very clever! I could replace the for loop with simply

Code: Select all

DIR *.RPT > list.temp
but it would probably be more efficient to make a temp file for only the three types of files like so:

Code: Select all

DIR DAY*.RPT > list.temp
DIR SKU*.RPT >> list.temp
DIR PLU*.RPT >> list.temp
I did some quick testing and looping the way I was is much, much slower than writing the file list to a file and searching that file.

But then I started re-thinking the whole task of getting these filenames into the variables.

My data set consists of just one "DAY" file, one "SKU" file, and one "PLU" file per date. A

Code: Select all

DIR DAY*.RPT|FIND "%currentdate%"
would find that one file much faster, but how would I put the filename that into the variable?
Last edited by Samir on 03 Nov 2015 15:28, edited 1 time in total.

Samir
Posts: 345
Joined: 16 Jul 2013 12:00
Location: HSV
Contact:

Re: Optimizing These Loops

#17 Post by Samir » 03 Nov 2015 15:07

mcnd wrote:If your system has short names enabled, then you could try this

Code: Select all

@echo off
    setlocal enableextensions enabledelayedexpansion
   
    set "currentdate=11/03/2015"
   
    for /f "tokens=1,2 eol=|" %%a in ('
        dir *.rpt /a-d /-n ^| findstr /c:"%currentdate%"
    ') do for %%c in ("%%~a.%%~b?") do (
        set "filetype=%%~nc"
        set "_!filetype:~0,3!=%%~nxc"
    )

    echo DAY: %_day%
    echo PLU: %_plu%
    echo SKU: %_sku%
I believe I do. Let me try this and see what happens. Seems to be quite a bit faster. I'm going to execute the original code and compare.

Original code took almost a minute, so definitely an improvement!

Samir
Posts: 345
Joined: 16 Jul 2013 12:00
Location: HSV
Contact:

Re: Optimizing These Loops

#18 Post by Samir » 03 Nov 2015 16:39

I've been able to tweak this even a bit more to make it execute almost instantly now. 8)

Code: Select all

:REPORTSTART
    setlocal enableextensions enabledelayedexpansion
FOR %%F IN (DAY PLU SKU) DO (
    for /f "tokens=1,2 eol=|" %%a in ('
        dir %%F*.rpt /a-d /-n /o-d ^| findstr /c:"%currentDate%"
    ') do for %%c in ("%%~a.%%~b?") do (
        set "filetype=%%~nc"
        set "!filetype:~0,3!=%%~nxc"
    )
)
REM    echo DAY: %day%
REM    echo PLU: %plu%
REM    echo SKU: %sku%

ECHO !DAY!
ECHO !PLU!
ECHO !SKU!
Executing the complete batch file used to take a minute. Now it's under 15 seconds. 8) Thank you very for the great responses everyone!

penpen
Expert
Posts: 1723
Joined: 23 Jun 2013 06:15
Location: Germany

Re: Optimizing These Loops

#19 Post by penpen » 03 Nov 2015 17:01

If you differ between the filetypes, then you probably could simplify to:

Code: Select all

FOR %%F IN (DAY PLU SKU
) DO for /f "tokens=1,2 eol=|" %%a in ('
    dir %%F*.rpt /a-d /-n /o-d ^| findstr /c:"%currentDate%"
') do set "%%~F=%%~a.%%~b"
I'm a bit unsure, because of the question mark in "%%~a.%%~b?":
Actually i can't see a benefit using this character, but i may be wrong.


penpen

Samir
Posts: 345
Joined: 16 Jul 2013 12:00
Location: HSV
Contact:

Re: Optimizing These Loops

#20 Post by Samir » 03 Nov 2015 17:08

penpen wrote:If you differ between the filetypes, then you probably could simplify to:

Code: Select all

FOR %%F IN (DAY PLU SKU
) DO for /f "tokens=1,2 eol=|" %%a in ('
    dir %%F*.rpt /a-d /-n /o-d ^| findstr /c:"%currentDate%"
') do set "%%~F=%%~a.%%~b"
I'm a bit unsure, because of the question mark in "%%~a.%%~b?":
Actually i can't see a benefit using this character, but i may be wrong.


penpen
I'm actually not exactly sure how this section works either. I know that %%~a expands that without removing quotes, and that it's the same with %%~b. I think the question mark part of the trick of using the 8.3 filenames back to normal.

I'm amazed at the speed improvements with just small changes. I knew there had to be a better and faster way to do what I needed. 8)

Squashman
Expert
Posts: 4111
Joined: 23 Dec 2011 13:59

Re: Optimizing These Loops

#21 Post by Squashman » 03 Nov 2015 17:17

You can do multiple file masks with the DIR command.

Samir
Posts: 345
Joined: 16 Jul 2013 12:00
Location: HSV
Contact:

Re: Optimizing These Loops

#22 Post by Samir » 03 Nov 2015 22:47

Squashman wrote:You can do multiple file masks with the DIR command.
I checked the help on DIR and didn't see it, but tried and you're absolutely right! Learned something big today! Thank you! I'll see how I can best incorporate this. Probably won't change speed, but definitely will make the code easier to read.

mcnd
Posts: 27
Joined: 08 Jan 2014 07:29

Re: Optimizing These Loops

#23 Post by mcnd » 04 Nov 2015 03:36

penpen wrote:I'm a bit unsure, because of the question mark in "%%~a.%%~b?":


When writing the code I was trying to use a dir command and filter the full list using the %currentdate% variable with findstr so the list is only retrieved once (a dir command with several masks execute several searchs, so I decided not to include it) and the for command will only process the files that match the date.

The problem is that now the code has to process the lines generated by the dir command. To know where to locate inside the lines the name and extension of the file, I used the `/-n` to place the short names at the start of the line.

But now there is another problem. We only need the DAY, PLU and SKU files, but it is posible that the short name does not include any of this characters, so, we need to convert the short name into a long name and to do it a for command with a wildcard is used to force a filesystem query, locate the file and retrieve the full name.

penpen
Expert
Posts: 1723
Joined: 23 Jun 2013 06:15
Location: Germany

Re: Optimizing These Loops

#24 Post by penpen » 04 Nov 2015 04:22

I was too imprecise and posted too ambiguous (/poor):
Sorry for that.

I didn't want to say, that i don't know the meaning of a "?" in a for command.

I was aware of the benefit in your (mcnd) solution.

But if one differs between the filetypes then it is of not much use for the pure algorithm.
But Samir might need the long name for other reasons; for example to create an infofile containing the path, so other users may access this file via a network, while the short name is not accessible (possible network limitation).


penpen

mcnd
Posts: 27
Joined: 08 Jan 2014 07:29

Re: Optimizing These Loops

#25 Post by mcnd » 04 Nov 2015 04:56

penpen wrote:I didn't want to say, that i don't know the meaning of a "?" in a for command.


I was sure of it. Excuse me if I was not clear enough, I was just trying to indicate why I used it.

When writting the code my first try was to execute the inner for command without the wildcard, but I was unable to retrieve the full name when using the short one as reference, so at the end I included the wildcard. I see no simpler way to retrieve the full name.

Squashman
Expert
Posts: 4111
Joined: 23 Dec 2011 13:59

Re: Optimizing These Loops

#26 Post by Squashman » 04 Nov 2015 07:08

Samir wrote:I checked the help on DIR and didn't see it,

Well there is no concrete example showing it but the help file does use the plural of the word file.

Code: Select all

[drive:][path][filename]
            Specifies drive, directory, and/or files to list.

Samir
Posts: 345
Joined: 16 Jul 2013 12:00
Location: HSV
Contact:

Re: Optimizing These Loops

#27 Post by Samir » 04 Nov 2015 11:20

mcnd wrote:
penpen wrote:I'm a bit unsure, because of the question mark in "%%~a.%%~b?":


When writing the code I was trying to use a dir command and filter the full list using the %currentdate% variable with findstr so the list is only retrieved once (a dir command with several masks execute several searchs, so I decided not to include it) and the for command will only process the files that match the date.

The problem is that now the code has to process the lines generated by the dir command. To know where to locate inside the lines the name and extension of the file, I used the `/-n` to place the short names at the start of the line.

But now there is another problem. We only need the DAY, PLU and SKU files, but it is posible that the short name does not include any of this characters, so, we need to convert the short name into a long name and to do it a for command with a wildcard is used to force a filesystem query, locate the file and retrieve the full name.
Thank you for the detailed explanation! This helps to understand the code quite a bit. 8)

Samir
Posts: 345
Joined: 16 Jul 2013 12:00
Location: HSV
Contact:

Re: Optimizing These Loops

#28 Post by Samir » 04 Nov 2015 11:22

Squashman wrote:
Samir wrote:I checked the help on DIR and didn't see it,

Well there is no concrete example showing it but the help file does use the plural of the word file.

Code: Select all

[drive:][path][filename]
            Specifies drive, directory, and/or files to list.
I always thought of 'files' as just the normal list of files.

When did this feature in dir come about? I know it wasn't there in the dos 6.0 days. Did it start in win95? Or was it with win2k or xp?

Squashman
Expert
Posts: 4111
Joined: 23 Dec 2011 13:59

Re: Optimizing These Loops

#29 Post by Squashman » 04 Nov 2015 11:54

Samir wrote:When did this feature in dir come about? I know it wasn't there in the dos 6.0 days. Did it start in win95? Or was it with win2k or xp?

I would assume Windows NT.

Samir
Posts: 345
Joined: 16 Jul 2013 12:00
Location: HSV
Contact:

Re: Optimizing These Loops

#30 Post by Samir » 04 Nov 2015 13:12

Squashman wrote:
Samir wrote:When did this feature in dir come about? I know it wasn't there in the dos 6.0 days. Did it start in win95? Or was it with win2k or xp?

I would assume Windows NT.
I'll have to try it on my NT system. Thank you!

Post Reply