Optimizing These Loops

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Message
Author
Samir
Posts: 384
Joined: 16 Jul 2013 12:00
Location: HSV
Contact:

Optimizing These Loops

#1 Post by Samir » 02 Nov 2015 16:05

I have the current segment of code:

Code: Select all

:REPORTSTART
FOR %%f IN (DAY*.RPT) DO (
SET filedatetime=%%~tf
REM ECHO !filedatetime!
IF "!filedatetime:~0,10!" == "%currentDate%" SET DAY=%%f
REM ECHO !DAY!
)

FOR %%f IN (PLU*.RPT) DO (
SET filedatetime=%%~tf
REM ECHO !filedatetime!
IF "!filedatetime:~0,10!" == "%currentDate%" SET PLU=%%f
REM ECHO !PLU!
)

FOR %%f IN (SKU*.RPT) DO (
SET filedatetime=%%~tf
REM ECHO !filedatetime!
IF "!filedatetime:~0,10!" == "%currentDate%" SET SKU=%%f
REM ECHO !SKU!
)

ECHO !DAY!
ECHO !PLU!
ECHO !SKU!
What it does is search files of a certain type for a certain date (in the format mm/dd/yyyy) and then puts the filename into a variable if it matches. All the files are in the same place, so it's looping through these files three times to get the three file names. (There is only one of each of these files per day.)

What I wanted to see is if there was a way to optimize the amount of time spent in looping through the files by just looping through all of them once (*.RPT) and then assigning each variable based on what type of file is found on that date.

Some additional information
  • The number of files the current implementation has to go through is <300 (for each loop)
  • The total number of files (*.RPT) is over 3400

Possible?

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: Optimizing These Loops

#2 Post by aGerman » 02 Nov 2015 16:21

Does %currentDate% contain the current date?

Samir
Posts: 384
Joined: 16 Jul 2013 12:00
Location: HSV
Contact:

Re: Optimizing These Loops

#3 Post by Samir » 02 Nov 2015 16:27

aGerman wrote:Does %currentDate% contain the current date?
It can, but it is usually a day or two before, and could be one in the past few months from current. It is always in the format of MM/DD/YYYY.

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: Optimizing These Loops

#4 Post by aGerman » 02 Nov 2015 16:38

I see. Not sure if that would increase the performance but at least something you could try

Code: Select all

FOR %%f IN (*.RPT) DO (
  SET "filedatetime=%%~tf"
  SET "file=%%f"
  IF "!filedatetime:~0,10!"=="%currentDate%" (
    IF /I "!file:~,3!"=="DAY" (
      SET "DAY=%%f"
    ) ELSE IF /I "!file:~,3!"=="PLU" (
      SET "PLU=%%f"
    ) ELSE IF /I "!file:~,3!"=="SKU" (
      SET "SKU=%%f"
    )
  )
)

Samir
Posts: 384
Joined: 16 Jul 2013 12:00
Location: HSV
Contact:

Re: Optimizing These Loops

#5 Post by Samir » 02 Nov 2015 16:53

aGerman wrote:I see. Not sure if that would increase the performance but at least something you could try

Code: Select all

FOR %%f IN (*.RPT) DO (
  SET "filedatetime=%%~tf"
  SET "file=%%f"
  IF "!filedatetime:~0,10!"=="%currentDate%" (
    IF /I "!file:~,3!"=="DAY" (
      SET "DAY=%%f"
    ) ELSE IF /I "!file:~,3!"=="PLU" (
      SET "PLU=%%f"
    ) ELSE IF /I "!file:~,3!"=="SKU" (
      SET "SKU=%%f"
    )
  )
)
The more I've thought about performance, I think this is why I originally coded it in separate loops. I think the only way a singular loop might be faster is if it stopped working once all three files were found.

But then the same methodology could be applied to the three individual loops to speed them up as well.

The only reason I'm focusing on this is because this seems to be the biggest area that this particular batch spends time. If there was a way to significantly speed up the execution, then the entire batch would be faster.

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: Optimizing These Loops

#6 Post by aGerman » 02 Nov 2015 17:07

Samir wrote:I think the only way a singular loop might be faster is if it stopped working once all three files were found.

You could jump out.

Code: Select all

SET "DAY="&SET "PLU="&SET "SKU="
FOR %%f IN (*.RPT) DO (
  SET "filedatetime=%%~tf"
  SET "file=%%f"
  IF "!filedatetime:~0,10!"=="%currentDate%" (
    IF /I "!file:~,3!"=="DAY" (
      SET "DAY=%%f"
    ) ELSE IF /I "!file:~,3!"=="PLU" (
      SET "PLU=%%f"
    ) ELSE IF /I "!file:~,3!"=="SKU" (
      SET "SKU=%%f"
    )
  )
  IF DEFINED DAY IF DEFINED PLU IF DEFINED SKU GOTO EXITLOOP
)
:EXITLOOP

Squashman
Expert
Posts: 4465
Joined: 23 Dec 2011 13:59

Re: Optimizing These Loops

#7 Post by Squashman » 02 Nov 2015 17:34

Maybe Robocopy would be faster at finding them.

Samir
Posts: 384
Joined: 16 Jul 2013 12:00
Location: HSV
Contact:

Re: Optimizing These Loops

#8 Post by Samir » 02 Nov 2015 17:51

Squashman wrote:Maybe Robocopy would be faster at finding them.
On the systems where this executes, it's not installed. :( That was actually my first inclination. Or maybe outputting the directory to a file if that can be searched faster.

penpen
Expert
Posts: 1991
Joined: 23 Jun 2013 06:15
Location: Germany

Re: Optimizing These Loops

#9 Post by penpen » 02 Nov 2015 19:28

I don't know if the following is much faster, but you could try to avoid using environment variables (instead this uses a tempfile; not tested):

Code: Select all

@echo off
setlocal
:: ... set current date ...

>list.temp (
   for %%f in ("DAY*.RPT" "PLU*.RPT" "SKU*.RPT" "*.txt") do echo(%%~tf %%~nx
)
for /F "token=2" %%f in ('findstr "^%currentdate%" "list.tmp"') do (
   for /F "token=1 delims=YU" %%t in ("%%~f") do (
      if        "DA" == "%%~t" ( SET DAY=%%f
      ) else if "PL" == "%%~t" ( SET PLU=%%f
      ) else if "SK" == "%%~t" ( SET SKU=%%f
      )
   )
)
del "list.temp"
endlocal
goto :eof
The first loop may be replaced by a dir command (may be faster); in that case you probably have to adjust the tokens in the other loops.


penpen

Squashman
Expert
Posts: 4465
Joined: 23 Dec 2011 13:59

Re: Optimizing These Loops

#10 Post by Squashman » 03 Nov 2015 07:30

Samir wrote:
Squashman wrote:Maybe Robocopy would be faster at finding them.
On the systems where this executes, it's not installed. :( That was actually my first inclination. Or maybe outputting the directory to a file if that can be searched faster.

Ugh!!! You realize there is no support for Windows XP anymore and you can install Robocopy to Windows XP.
Regardless you are way behind the curve:
Windows 10
Windows 8.1
Windows 8
Windows 7
Windows Vista

mcnd
Posts: 27
Joined: 08 Jan 2014 07:29

Re: Optimizing These Loops

#11 Post by mcnd » 03 Nov 2015 08:37

If your system has short names enabled, then you could try this

Code: Select all

@echo off
    setlocal enableextensions enabledelayedexpansion
   
    set "currentdate=11/03/2015"
   
    for /f "tokens=1,2 eol=|" %%a in ('
        dir *.rpt /a-d /-n ^| findstr /c:"%currentdate%"
    ') do for %%c in ("%%~a.%%~b?") do (
        set "filetype=%%~nc"
        set "_!filetype:~0,3!=%%~nxc"
    )

    echo DAY: %_day%
    echo PLU: %_plu%
    echo SKU: %_sku%

Samir
Posts: 384
Joined: 16 Jul 2013 12:00
Location: HSV
Contact:

Re: Optimizing These Loops

#12 Post by Samir » 03 Nov 2015 09:51

Thank you for the many solutions. I'll parse through them when I get a chance.
Squashman wrote:
Samir wrote:
Squashman wrote:Maybe Robocopy would be faster at finding them.
On the systems where this executes, it's not installed. :( That was actually my first inclination. Or maybe outputting the directory to a file if that can be searched faster.

Ugh!!! You realize there is no support for Windows XP anymore and you can install Robocopy to Windows XP.
Regardless you are way behind the curve:
Windows 10
Windows 8.1
Windows 8
Windows 7
Windows Vista
Kinda ironic when you're telling me about being 'behind' and we're essentially talking on a site about DOS batch files--an operating system that's been extinct now since windows 98. :lol:

Squashman
Expert
Posts: 4465
Joined: 23 Dec 2011 13:59

Re: Optimizing These Loops

#13 Post by Squashman » 03 Nov 2015 09:56

Samir wrote:Kinda ironic when you're telling me about being 'behind' and we're essentially talking on a site about DOS batch files--an operating system that's been extinct now since windows 98. :lol:

You are incorrect. We are not talking about DOS batch files. We are talking about Windows (NT) batch files which are the equivalent of using a shell in any Unix or Linux environment. Majority of batch file we write today would NEVER run on DOS.

Samir
Posts: 384
Joined: 16 Jul 2013 12:00
Location: HSV
Contact:

Re: Optimizing These Loops

#14 Post by Samir » 03 Nov 2015 13:43

Squashman wrote:
Samir wrote:Kinda ironic when you're telling me about being 'behind' and we're essentially talking on a site about DOS batch files--an operating system that's been extinct now since windows 98. :lol:

You are incorrect. We are not talking about DOS batch files. We are talking about Windows (NT) batch files which are the equivalent of using a shell in any Unix or Linux environment. Majority of batch file we write today would NEVER run on DOS.
The name of the site is "DOStips". Yes, we're working with the NT-based command.com equivalent, but it's still the same antiquated language that's been around since the late 1980s. Many of the bats you may write won't work in traditional DOS, but I try to stick to normal DOS functions unless I have something more complicated to work on (like the above code).

I'm not worried about what the rest of the world is running on their desktop. I have something that works and that's all that matters to me. I could care less how 'old' it is. I'm not trying to keep up with the Jones's; I'm just trying to get the job done.

Samir
Posts: 384
Joined: 16 Jul 2013 12:00
Location: HSV
Contact:

Re: Optimizing These Loops

#15 Post by Samir » 03 Nov 2015 14:25

aGerman wrote:
Samir wrote:I think the only way a singular loop might be faster is if it stopped working once all three files were found.

You could jump out.

Code: Select all

SET "DAY="&SET "PLU="&SET "SKU="
FOR %%f IN (*.RPT) DO (
  SET "filedatetime=%%~tf"
  SET "file=%%f"
  IF "!filedatetime:~0,10!"=="%currentDate%" (
    IF /I "!file:~,3!"=="DAY" (
      SET "DAY=%%f"
    ) ELSE IF /I "!file:~,3!"=="PLU" (
      SET "PLU=%%f"
    ) ELSE IF /I "!file:~,3!"=="SKU" (
      SET "SKU=%%f"
    )
  )
  IF DEFINED DAY IF DEFINED PLU IF DEFINED SKU GOTO EXITLOOP
)
:EXITLOOP
Good point. I should probably also try to exit the loops in my original code and see how that helps with speed.

Post Reply