Escaping an ampersand character

[pmennen]

See if you can find any files with an ! in the name, in your output file

You are right ... all the files with and ! in them are missing.
GAHH#$#@&%(^!=!+!";
Now that I've been running my mp3 script for almost 2 days now, it is just about finished.
And all of it useless!

I did put the delayed expansion thingy where you suggested, but then as you can see in my script below I added the variable "fi" to count the number of lines in the temp file so I could periodically dump it and clear it out. I couldn't get the damn variable to increment until I moved the delayed expansion outside the loop. I'm sure there is a way to get it to work, but I just don't understand that fine point well enough yet.

If you post the code you are using then I will test it here on a large number of files - I'm interested to see what is making it slow down once it's running, and if there's a workaround.

Ok, if you are a glutton for punishment, feel free to look at my code which is included below. This version only writes 100 lines to the tempC file before dumping it into the tempB file, then it clears out the tempC file and continues. I see no reason why it should slow down as the tempB file gets bigger. Yes the copy would take a bit longer once the file gets bigger, but the thing is it doesn't really have to do that many copies, and mysteriously the time between lines echoing to the console slows down tremendously as the file gets bigger. And I'm not talking a mere factor of 2 or 5. I'm talking about a factor of 20 times slower or more! What the hell?

Code: Select all

@echo off
set "ffprobe=c:\Util\ffprobe.exe"
set "tempA=%temp%\mp3tmpA.txt"  & set "tempbat=%temp%\mp3tmpC" 
set "tempB=%temp%\mp3tmpB.txt"  & set "tempC=%tempbat%.bat"
type NUL > "%tempB%"

setlocal enabledelayedexpansion
set /a fi=0

for /R %%G IN (*.mp3) do (
   rem echo %%G 1>&2
   set "fname=%%~pG%%~nG" & set sz=%%~zG
   "%ffprobe%" -i "%%G" >"%tempA%" 2>&1
   for /f %%a in ('sed -n -r "s_.*Duration: (..:..:..).*_\1_p" "%tempA%"') do (
   for /f %%b in ('sed -n -r "0,/Audio: mp3/{s_.*Audio: mp3.*(...) kb.s_\1_p}" "%tempA%"') do (
   set "rate=   %%b" & set sz=!sz:~0,-4! & set /A sz = !sz! + 5 & set "sz=    !sz:~0,-2!.!sz:~-2,-1!"
   set "line=%%a  !rate:~-3!K!sz:~-6! MB  ---  !fname!"
   if !fi! EQU 100 copy "%tempB%"+"%tempC%" "%tempB%" /B > NUL & type NUL > "%tempC%" &  set /a fi=0
   set /a fi+=1 & echo !line! & echo !line! >> "%tempC%"
)))

endlocal
copy "%tempB%"+"%tempC%" "%tempB%" /B > NUL
:LoopStart
rem - Extract column 31 and remove duplicate lines
sed -r "s_.{30}(.).*_\1_" %tempB% | sed "$!N; /^\(.*\)\n\1$/!P; D" > "%tempA%"
call :filesize "%tempA%"
if %size% GTR 4 goto LoopDone
rem - Remove column 31 if it is always the same character
sed -r "s_(.{30}).(.*)_\1\2_" %tempB% > %tempA%
copy "%tempA%" "%tempB%" > NUL
goto LoopStart
:filesize
set size=%~z1
goto :eof
:LoopDone

sort /+31 "%tempB%" > mp3time.txt
set sec=0 & set fi=0
sed -r "s_0(.):(.)(.):(.)(.).*_set /A fi+=1 \& set /A sec+=\1*3600+\2*600+\3*60+\4*10+\5_" mp3time.txt > "%tempC%"
call "%tempbat%"
set /A min = sec/60  &  set /A sec += 100 - 60*min
set /A hour = min/60 &  set /A min += 100 - 60*hour

rem - Append the running time and file count to the end of the results file
set "line===================== Total run time is %hour%:%min:~1%:%sec:~1%  (%fi% mp3 files)"
echo %line%  &  echo %line% >> mp3time.txt

del "%temp%\mp3tmp*.*" > NUL

[foxidrive]

GAHH#$#@&%(^!=!+!";
Now that I've been running my mp3 script for almost 2 days now, it is just about finished.
And all of it useless!

hehe I told you so!

I did put the delayed expansion thingy where you suggested, but then as you can see in my script below I added the variable "fi" to count the number of lines in the temp file so I could periodically dump it and clear it out. I couldn't get the damn variable to increment until I moved the delayed expansion outside the loop.

Variables within a loop need the delayedexpansion form of !fi! to work or you manipulated it within the setlocal/endlocal - and the variable disappears when it hits the endlocal.

Can you post the version of your code before the changes made to copy portions every N files?
It may be the one you posted last - but it pays to ask where code is involved.

If I'd have been there, I would have checked the size of the file being copied - as large files will slow it enormously and get worse and worse.

Knowing the kind of data - I can't see that the file would be too large, but that's one thing I would have looked at.

[foxidrive]

A brief comment - you use call which is notoriously slow, and are processing data within a for loop when it may not be needed - and that's also slow...

My comment about processing an entire file still stands, and I'm running the code below to see how I can process it.
If you do this too, then any tests you make will eliminate the ffprobe of each and every file again.

Code: Select all

@echo off
set "ffprobe=c:\Util\ffprobe.exe"
set "tempA=%temp%\mp3tmpA.txt"  & set "tempbat=%temp%\mp3tmpC" 
set "tempB=%temp%\mp3tmpB.txt"  & set "tempC=%tempbat%.bat"
type NUL > "%tempB%"

(
for /R %%G IN (*.mp3) do (
   echo ====== %%G
   "%ffprobe%" -i "%%G" 2>&1
)
)>"%tempA%"

[pmennen]

Ok ... if you request it again, I'll still post my other version that you wanted (simpler, without the extra dump every 100 files), but I think that is now irrelevant given what I've reported below.

A brief comment - you use call which is notoriously slow ...

Well the "call" at least has nothing to do with the problem. My script ran for two days to process 100,000 files, but that was entirely due to the first for loop containing the ffprobe command. After that loop ran for two days, the remaining part of the script including that amazingly inefficient thing between :LoopStart and :LoopDone and the similarly crazy looking time duration summation ... all of that took about one second to run!!

My comment about processing an entire file still stands, and I'm running the code below to see how I can process it.

With my old script, it took about 14 minutes to run thru 6300 mp3 files (about 1/17 of the collection that took two days to finish). It's not proportional because it continues to slow down as it goes. However if I just concatenate the output of ffprobe as you suggest in the code of your last message, the 6300 files are processed in just 2.57 minutes (i.e. about 41 files processed every second!). The resulting file is 13Mbytes. If I add an echo of the complete file name to the console it goes up from 2.57 minutes to 4.55 minutes. That is pretty darned fast. I should have paid attention to that comment when you made it earlier, but as it generates a pretty large file I thought it would run slow. (I was WRONG). I'll have to do a test, but if it doesn't slow down it should be able to run thru the whole 100 thousand file collection in about 45 minutes. At that point I suspect I will be able to reformat the large ffprobe output file into what I really want with a single sed command that might take as little as a second or so.

I would like to echo something to the console, but it looks like echoing the full file name for every file processed slows it down quite a bit, so I think I will experiment with echoing less stuff, perhaps just the path portion but not echoing it when it doesn't change.
~Paul

[pmennen]

Man, I can't believe I can't get such a simple loop to work.
All I'm trying to do is echo the path whenever it changes.
Here is my code

Code: Select all

@echo off

set "ffprobe=c:\Util\ffprobe.exe"
set "temp=%temp%\ffprobe.txt"
set pt=abc
setlocal enabledelayedexpansion
(
for /R %%G IN (*.mp3) do (
   echo ========== %%G
   set p=%%~pG
   if  !pt! == !p! (echo deubg1) else (set pt=!p! & echo !p! 1>&2)
   "%ffprobe%" -i "%%G" 2>&1
   )
) >"%temp%"

endlocal

But it prints the path every single time thru the loop.
I've worked on this stupid little loop for over an hour ... and after trying 1000 different things I give up.
I've tried putting parens and spaces and line breaks in all kinds of configurations I've seen in other peoples code and I'm bewildered that nothing seems to work. It always seems to thing that these two strings are not equal even when I can see they are the same.
I clearly haven't mastered this subtle delayed expansion thing.
I probably don't even want the delayed expansion thing outside the loop, since I might get the ! problem you mentioned before (?) but that is the least of my worries now. I simply can't to a string comparison to save my life.

~Paul

[Squashman]

Are you trying to get this:

Code: Select all

echo !p! 1>&2

Output to this file:

Code: Select all

) >"%temp%"

[foxidrive]

it should be able to run thru the whole 100 thousand file collection in about 45 minutes. At that point I suspect I will be able to reformat the large ffprobe output file into what I really want with a single sed command that might take as little as a second or so.

Beauty.

[pmennen]

Are you trying to get this:
echo !p! 1>&2
Output to this file:
>"%temp%"

No, that echo goes to the console as intended.
The problem is with the if statement.
I know the if condition is sometimes true, but the script is behaving as if the if condition is never true.
So I must be doing the string comparison improperly, but I can't figure out what I'm doing wrong.

~Paul

[Squashman]

Not really understanding why you are doing what you are doing with that code but this is my best practice for coding. Everyone has their own style but this has always been foolproof for me.

Code: Select all

@echo off

set "ffprobe=c:\Util\ffprobe.exe"
set "temp=%temp%\ffprobe.txt"
set "pt="
setlocal enabledelayedexpansion
(
for /R %%G IN (*.mp3) do (
   echo ========== %%G
   if "!pt!"=="%%~pG" (
      echo deubg1
   ) else (
      set "pt=%%~pG"
      echo %%~pG 1>&2
   )
   "%ffprobe%" -i "%%G" 2>&1
)
)>"%temp%"

endlocal

[foxidrive]

This is particularly tricky with delayed expansion - the technique is undocumented.

temp is a system variable Paul, and shouldn't be used as a local variable.

Code: Select all

@echo off

set "ffprobe=c:\Util\ffprobe.exe"
set "tempfile=%temp%\ffprobe.txt"
set "pt="
set "pt-new="

(
for /R %%G IN (*.mp3) do (
set "file=%%G"
set "pt=%%~dpG"
setlocal enabledelayedexpansion
   if /i not "!pt-new!"=="!pt!" set "pt-new=!pt!" & echo "!pt-new!" 1>&2
   echo ========== %%~zG "!file!"
   "%ffprobe%" -i "!file!" 2>&1
for %%a in ("!pt-new!") do endlocal & set "pt-new=%%~a"
)
)>"%tempfile%"

pause

[pmennen]

The code by Squashman works and runs reasonably fast. I thought I tried exactly what he wrote the very first time, but it didn't work.
Perhaps I missed the quotes around one of the items in the string comparison. But it is so hard to figure out why it isn't working, I just went round and round trying every more imaginative constructions every one of which must have had some subtle problem.

It's all kind of moot however because this loop has the fatal flaw you mentioned earlier (i.e. it doesn't work when the file name contains an exclamation point character).

This is particularly tricky with delayed expansion - the technique is undocumented.

I tried your code and it indeed works, but it turns out that it is also pointless. The reason I was trying to do the string compare was because echoing the file name to the console doubled the execution time, so I was trying to speed up the code by avoiding displaying the file every single time. Although your code does that, it turns out that it slows down the execution by about 2 and a half times (i.e. even worse than just displaying the filename every time).

Instead of displaying the file name, I may just display a single dot for every file just to let me know that the program is still runing.
It turns out that if you echo the dot in the usual way (i.e. followed by a carriage return) it doesn't serve any purpose because after it starts scrolling you don't see any change in the console and so you can't tell if it running or not. However if I suppress the carriage return then it does work for this purpose. I did this with:

Code: Select all

echo | set /p=. 1>&2

This prints a dot followed by a space with no carriage return. I was trying to print the dot without the space, but couldn't figure out how to do that. This isn't as good as the file name since you can't tell how close it is from finishing, but it is better than nothing and also doesn't slow the loop down by that much.

If you think of other ideas for this particular problem, let me know.

temp is a system variable Paul, and shouldn't be used as a local variable.

Oops. Yes that was a mental lapse. I know that is a very bad idea.

~Paul

[Squashman]

Food for thought.
Guess which one runs quicker.

Code: Select all

@echo off

echo ..........
FOR /L %%G IN (1,1,1000) DO echo.|set /p ".=." 1>&2
echo.
FOR /L %%G IN (1,1,1000) DO set /p .=.<nul 1>&2
echo.

[pmennen]

Food for thought. Guess which one runs quicker.

Woe ... the one with the pipe runs 40 times slower!
Thanks for the tip. That will help somewhat.
I don't know how either statement works. What is that /p do anyway?

~Paul

[Squashman]

Food for thought. Guess which one runs quicker.

Woe ... the one with the pipe runs 40 times slower!
Thanks for the tip. That will help somewhat.
I don't know how either statement works. What is that /p do anyway?

~Paul

Well you could always read the help for the SET command. Here is the relevant information from the help.

Code: Select all

H:\>set /?
Displays, sets, or removes cmd.exe environment variables.

Two new switches have been added to the SET command:

    SET /A expression
    SET /P variable=[promptString]
    
    The /P switch allows you to set the value of a variable to a line of input
entered by the user.  Displays the specified promptString before reading
the line of input.  The promptString can be empty.

[thefeduke]

However if I suppress the carriage return then it does work for this purpose. I did this with:

Code: Select all

echo|set /p=. 1>&2

This prints a dot followed by a space with no carriage return. I was trying to print the dot without the space, but couldn't figure out how to do that.

I think that the space disappears with prompt:

Code: Select all

echo|set /p".=."1>&2
:: or
set /p ".=."<nul>&2

The complicated stuff you people are up to is interesting to follow, too.
John A.