Fastest Recursive Searching

[atfon]

Hello Folks,

I have been working on a system agnostic (Win 7 and above, 32 or 64-bit) script which will capture the file name and path of a set of frequently used applications. The location of these applications can vary from %localappdata% (for items like OneDrive) to %programfiles% and %programfiles(x86)% for items like chrome.exe or winword.exe. I have been doing some tests with for /r, dir, robocopy, where and wmic. I have also tested with writing the data to a temporary file and then just using findstr to type the results to the console and/or a file. I'm sure there are benefits and drawbacks to different methodologies as well as the powershell scripts to Get-ItemProperty of the two Uninstall locations.

I know it can be difficult to provide suggestions without any code or the set of applications in question, but is there one methodology you might recommend over the others?

Thank you in advance and I apologize if this question is too vague.

Edit: I forgot to include the where command and forfiles.

[aGerman]

viewtopic.php?f=3&t=10140
Turns out that robocopy works nicely because it supports the exclusion of heavy folders like C:\Windows, at least if you have to begin searching in the root folder of a drive.
It can also be used if you only have a few locations where you expect the apps to be found. You can put them in a nested loop (just like the drive letters in the linked thread).

If you already tried out all the possibilities you listed, what have been your experiences?

Steffen

[atfon]

Thanks for the recommendation, Steffen (@aGerman). I agree that robocopy does seem best. I've been running it a few times against a few locations recursively and using the /MT multithreaded option (along with a couple of the other performance tweak options you listed). l then send the output to a text file. I parse that text file for the items I need with type piped with findstr. I can use the results with wmic to capture the version too.

In all, it takes about 3.8 seconds, so not bad. I haven't tried a nested loop yet, but that will be next. When I get a chance to test, I will post whether that improves performance.

[aGerman]

The number of threads defaults to 8 if you don't specify the /MT option. Keep in mind that multithreading is not for free. There is a break-even. Rule of thumb is to use as many threads as the number of logical cores on your machine.
There's a predefined variable you might want to take instead.
/MT:%NUMBER_OF_PROCESSORS%
Just compare the performance with the default, or with the value that you previously specified.

Steffen

[atfon]

Hello Steffen (@aGerman),

Thank you for the feedback. I agree that you do use more resources with the /MT option, but it does seem to provide a small performance boost as long as your are judicious with it. I have kept it to 4X%NUMBER_OF_PROCESSORS%. In my case that equates to /MT:32. I tested it a few times compared to the default:

Test 1:
/MT:8 - 4.16
/MT:32 - 3.92

Test 2:
/MT:8 - 3.91
/MT:32 - 3.80

Test 3:
/MT:8 - 3.77
/MT:32 - 3.74

[atfon]

I had a chance to do some testing with a nested loop to generate a text file with paths of the executable files as @aGerman suggested. This resulted in some improved performance. To generate the list took 0.70 seconds. To run the complete script took 2.90 seconds.

[aGerman]

Your CPU must have been quite bored before
If you run the /MT:32 before /MT:8, is it still faster? (Beware of caching )

Steffen

[atfon]

Could be! Good point about caching @aGerman. I ran a few tests with /MT:32 before /MT:8 for just the robocopy portion of the script and the results weren't all that conclusive:

Test 1
/MT:32 - 0.39 seconds
/MT:8 - 0.69 seconds

Test 2
/MT:32 - 0.81 seconds
/MT:8 - 0.68 seconds

Test 3
/MT:32 - 0.77 seconds
/MT:8 - 0.78 seconds

Test 4
/MT:32 - 0.70 seconds
/MT:8 - 0.77 seconds

In 3 of the 4 cases, /MT:32 was the faster of the two.

Edit to add some specification information:
Windows 10 Pro
Processor: Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz 2.11 GHz
Installed RAM: 32.0 GB (31.8 GB usable)

[aGerman]

Code: Select all

@echo off &setlocal EnableDelayedExpansion
for %%n in (64 %NUMBER_OF_PROCESSORS% 2) do (
  echo /mt:%%n
  echo !time!
  for /f "tokens=*" %%i in (
    'robocopy "C:\." " nul" "wordpad.exe" /xd "C:\Windows" /l /s /mt:%%n /xx /r:1 /w:1 /ns /nc /ndl /np /njh /njs'
  ) do echo "%%i"
  echo !time!
  echo ~~~~~~~~~~~~~~~~~~~~~
)
pause

Typical result

Code: Select all

/mt:64
20:51:01,52
"C:\Program Files (x86)\Windows NT\Accessories\wordpad.exe"
"C:\Program Files\Windows NT\Accessories\wordpad.exe"
20:51:03,68
~~~~~~~~~~~~~~~~~~~~~
/mt:4
20:51:03,69
"C:\Program Files (x86)\Windows NT\Accessories\wordpad.exe"
"C:\Program Files\Windows NT\Accessories\wordpad.exe"
20:51:05,83
~~~~~~~~~~~~~~~~~~~~~
/mt:2
20:51:05,83
"C:\Program Files (x86)\Windows NT\Accessories\wordpad.exe"
"C:\Program Files\Windows NT\Accessories\wordpad.exe"
20:51:08,62
~~~~~~~~~~~~~~~~~~~~~
Drücken Sie eine beliebige Taste . . .

Slow little 14 inch notebook which is comfortable to have on my knees in the evening
Windows 10 Home 64 Bit
AMD Ryzen 3 3200U with Radeon Vega Mobile Gfx 2.60 GHz
4,00 GB RAM (3,44 GB usable)

For me it doesn't make much of a difference how many threads are defined, as long as it is at least the number of processors. I don't know if robocopy only schedules a reasonable number of threads depending on how it divides the work into packages (which is opaque for us since it isn't open source).

Steffen

[atfon]

Oddly, on my system this script stalls after displaying the following:

Code: Select all

/mt:64
15:20:59.78

It never gets around to displaying the location of wordpad.exe, although it is found with this command:

Code: Select all

dir "\wordpad.exe" /s /b

[aGerman]

That's weird Too many threads already? What if you remove the 64 from the list?

[atfon]

Maybe! I tried this bare command:

Code: Select all

robocopy "C:\." " nul" "wordpad.exe" /xd "C:\Windows" /l /s /xx /r:1 /w:1 /njh /njs /fp /ndl /ns /nc /np /mt:8

It seems to just sit there after locating the same two files your system found:

Code: Select all

C:\Program Files\Windows NT\Accessories\wordpad.exe
C:\Program Files (x86)\Windows NT\Accessories\wordpad.exe

[aGerman]

Then it's probably still searching in the remaining folder structures. Maybe C:\Users (which is only 7.6 GB here).

[atfon]

I agree, but it was taking several minutes. For comparison, I modified your script slightly to the following:

Code: Select all

robocopy "%programfiles%\ " "%tmp%\ " "wordpad.exe" /xd "C:\Windows" /l /s /mt:%%n /xx /r:1 /w:1 /njh /njs /fp /ndl /ns /nc /np

I received these results. Not too surprising since it is just recursing Program Files.

Code: Select all

/mt:64
15:52:34.33
"C:\Program Files\Windows NT\Accessories\wordpad.exe"
15:52:34.69
~~~~~~~~~~~~~~~~~~~~~
/mt:8
15:52:34.69
"C:\Program Files\Windows NT\Accessories\wordpad.exe"
15:52:35.03
~~~~~~~~~~~~~~~~~~~~~
/mt:2
15:52:35.03
"C:\Program Files\Windows NT\Accessories\wordpad.exe"
15:52:35.58
~~~~~~~~~~~~~~~~~~~~~

[aGerman]

I refrain from asking you how you got it bloated that much

FWIW I'm a little amazed that defining the source with a trailing space still works I mean, it's a known issue that "%programfiles%\" failes because robocopy uses the default C tokenizing for the command line. The backslash escapes the following quotation mark and the resulting token is something like C:\Program Files" which is invalid. You can work around this by doubling the trailing backslash "%programfiles%\\".
Just in case you're wondering why I used " nul" as target: You can't create a folder with a leading space, at least not on Windows. However, robocopy still accepts it. If you use an existing folder and a file which is to be copied (even with option /L which doesn't actually copy anything) already exists, it wouldn't be listed.