Memory leak when reading large text files

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Message
Author
vin97
Posts: 26
Joined: 17 Apr 2020 08:30

Re: Memory leak when reading large text files

#16 Post by vin97 » 02 Aug 2020 07:22

Ok, I will test that.

How does CMD work out piping internally or more specifically, what counts as a "critical pipe" capable of causing memory issues? Is it just piping into a command (with "|") or does it apply to "redirecting" in general?
For example, could I avoid the problems by writing to tempory files and letting FIND read them directly?
Of course, wherever possible, I will use the IF syntax you proposed since it's much faster.

penpen
Expert
Posts: 1859
Joined: 23 Jun 2013 06:15
Location: Germany

Re: Memory leak when reading large text files

#17 Post by penpen » 03 Aug 2020 02:32

If i remember right, then a pipe should creates two cmd.exe instances, which you should be able to test somehow like the following:

Code: Select all

set "test="
(set "test=a" & set test) | findstr "^"
set test
(>nul pause) | (set "test=b" & set test)
set test
But it's not the pipe itself causing that issue, but any process if you start enough in short enough time.
Every executable file will also create it's own process.
So the following line starts 3 processes (cmd.exe for left pipe, cmd.exe for right pipe, findstr.exe) each time:

Code: Select all

echo "!line!" | find /i ":\" > nul
Redirecting typically doesn't start any process, except if you redirect to sth that needs a driver that isn't loaded at that time (for example when redirecting to LPT or COM device) - but i don't think that driver is loaded more than once (except if the producer was mad).

You will reduce the amount of processes created by not using pipes, but if you start one find.exe-process every loop-iteration, then that also might be sufficient.


penpen

vin97
Posts: 26
Joined: 17 Apr 2020 08:30

Re: Memory leak when reading large text files

#18 Post by vin97 » 03 Aug 2020 09:35

Ok, going to start rewriting my program then.
Learning a lot here, thanks again!

miskox
Posts: 356
Joined: 28 Jun 2010 03:46

Re: Memory leak when reading large text files

#19 Post by miskox » 17 Aug 2020 09:17

Any news on this. I am really curious what is causing this memory leak. I would assume if this is a system process (that allocates nonpaged pool) it would free it when this allocation is not needed anymore.

Did you test your program with some SLEEP in in it (so the number of processes created is during the longer period and the system has time to deallocate the memory)? While your batch is executing you should monitor your system state (nonpaged pool...).

Did you try to find the file (.dll...) that is causing this with the appropriate tool?

Saso

Post Reply