Escaping an ampersand character

[Squashman]

I think that the space disappears with prompt:

The complicated stuff you people are up to is interesting to follow, too.
John A.

The space disappeared because of the syntax that is used. You will soon learn that extraneous spaces and not using quotes can give you fits with your output.

This stuff is rather easy. Start reading the threads on macros. That will blow your mind.

[pmennen]

My comment about processing an entire file still stands

Yes, that turned out to be the real key to make the script idea feasible for very large file collections. My final script for processing mp3 files is shown below. I plan to modify my script for processing video files to use the same technique because it is very fast. For a (very) large file collection, this script is about 40 times as fast as the previous versions that we were both posting earlier! When I run this on a music collection containing 114 thousand files, it takes about 80 minutes to execute line 12 of the script (the for loop) and then the entire rest of the script completes in about 10 seconds or so. The other thing I'm happy about with this script is its simplicity. It doesn't need any of those ugly delayed expansion constructs, and the whole script is only 24 lines (not including the comments). A single sed command is used to convert the combined ffprobe output file to final form (except for the sort). Granted the sed command is cryptic, but with the comments describing what each line does I think most people will be able to figure out how it works. (BTW your objection to sed, relating to not being allowed to load external executables does not make sense because without loading ffprobe.exe your script is equally useless). My old code had a somewhat crazy loop for removing the beginning of the file/path name - the part that corresponds to the current directory. (That's not needed in the listing since it is the same for every line). In this version I don't need that crazy loop since I found a much faster and more straightforward way to doing it.

Code: Select all

@echo off

rem - Traverses recursively from the current directory to create an mp3 listing.
rem - ffprobe (from the ffmpeg package) is used to extract the mp3 characteristics.

set "ffprobe=c:\Util\ffprobe.exe"
set "tempA=%temp%\mp3tmpA.txt"  & set "tempbat=%temp%\mp3tmpC" 
set "tempB=%temp%\mp3tmpB.txt"  & set "tempC=%tempbat%.bat"

echo Concatenating the ffprobe output from all the mp3 files into file tempA.
(
for /R %%G IN (*.mp3) do echo ==fsize:    %%~zG & "%ffprobe%" -i "%%G" 2>&1 & set /p .=.<nul 1>&2
) > "%tempA%"

echo. & echo Processing ffprobe output

rem - Save ffprobe output for debugging: copy "%tempA%" "%temp%\ffprobe.txt" > NUL
rem - Get the current path and replace \ with \\ for sed
set cur=%CD:\=\\%\\

rem Line1: For lines containing "==fsize:" extract the filesize in MB,
rem        surround it with questions marks and save it in the hold buffer.
rem Line2: For lines containing "Input #0" extract the file name excluding the current
rem        path, surround it with question marks and append it to the hold buffer.
rem Line3: For lines containing "Duration:" extract the duration hh:mm:ss,
rem        surround it with question marks and append it to the hold buffer.
rem Line4: For lines containing "Audio:" append the hold buffer to the "Audio:" line,
rem        and extract the bit rate (kb/s). Replace this line with the four pieces of
rem        previously extracted data delimited by the question marks, then print the line.
rem What follows contains a sample output line from this sed command. The fields in order
rem are duration (hh:mm:ss), bitRate (kb/s), File size, File Name:
rem 00:08:14  256K  15.80 MB     Haydn, Joseph\Symphony No82 in C-dur\01 Vivace assai

sed -n -r^
 -e "/==fsize:/{s_.*(...)(..).....$_?\1.\2?_;h}"^
 -e "/Input #0.*, from/{s_.* from '%cur%(.*).(mp|MP)3':_?\1?_;H}"^
 -e "/  Duration:/{s_.*(..:..:..).*_?\1?_;H}"^
 -e "/#0:0.*Audio: mp3/{G;s_.*(...) kb.s.*\?(.*)\?.*\?(.*)\?.*\?(.*)\?_\4  \1K \2 MB     \3_p}"^
 "%tempA%" > "%tempB%"

echo Sorting output file on the file name field
sort /+27 "%tempB%" > mp3time.txt

echo Computing the total play time duration
rem - The fi variable will count the number of files processed.
rem - The sec variable will total running time in seconds. We have to handle each digit
rem - of the time separately (otherwise we get the leading zero octal problem)
set sec=0 & set fi=0
sed -r "s_0(.):(.)(.):(.)(.).*_set /A fi+=1 \& set /A sec+=\1*3600+\2*600+\3*60+\4*10+\5_" mp3time.txt > "%tempC%"
rem - Execute the batch file to count the files and sum up the durations
call "%tempbat%"
rem - Convert seconds to hours/minutes/seconds
rem - Zeropad minutes and seconds by adding 100 and then removing the first digit
set /A min = sec/60  &  set /A sec += 100 - 60*min
set /A hour = min/60 &  set /A min += 100 - 60*hour
rem - Append the running time and file count to the end of the results file
set "total===================== Total run time is %hour%:%min:~1%:%sec:~1%  (%fi% mp3 files)"
echo %total%  &  echo %total% >> mp3time.txt

rem - When debugging, comment out the next line so you can view the temp files
del "%temp%\mp3tmp*.*" > NUL

[foxidrive]

My comment about processing an entire file still stands

Yes, that turned out to be the real key to make the script idea feasible for very large file collections.
For a (very) large file collection, this script is about 40 times as fast as the previous versions that we were both posting earlier!

I'm glad you have a solution you are happy with.

(BTW your objection to sed, relating to not being allowed to load external executables does not make sense because without loading ffprobe.exe your script is equally useless).

In the context it is sensible, and I have no objection to sed.

You're arguing here about why I prefer batch scripts,
you're saying that I object to sed,
you argued about moving the "setlocal enabledelayedexpansion" line,
you argued how a batch file could be fast
you complained about the filesize routine but ignored my post telling you why your routine failed
you ignored me saying that processing the entire set of results would be faster
you said my code was pointless, but you didn't say why you were doing what you were doing.


What is pointless is echoing anything at all when you want speed - and it's only going to take an hour or so.
Go and walk the dog when your script is running, or play Dave Benham's Snake game in-a-batch-file or his Colossal Cave Adventure in-a-batch-file.

[pmennen]

you argued about moving the "setlocal enabledelayedexpansion" line

I don't remember arguing about that one since at the time I understood little about its use and in fact I still don't understand it perfectly. I tend to challenge everything I hear in an attempt to understand it better, so sorry if you perceived this as persistent arguing or whining. I certainly wouldn't have arrived at my current script without your help.

you complained about the filesize routine but ignored my post telling you why your routine failed

Hmm I don't remember that either. In fact I did have a few files bigger than 2GB, so I'm sure it did fail but I hadn't noticed it yet since the vast majority of my files were smaller than that. I didn't ignore your advice on that however and you may have noticed that the next script I posted did the rounding differently. (It first discarded enough low order digits that the 32 bit limit was no longer an issue). My current script for MP3s doesn't do any rounding since it prints out more digits and I thought any change due to rounding would be even less significant than before.

you ignored me saying that processing the entire set of results would be faster

Yes, sad to say I did do that. That was unfortunate because it led me to spend a lot of time on a script that I eventually had to discard because it was too slow. Fortunately you re-iterated that suggestion and I did listen the 2nd time

What is pointless is echoing anything at all when you want speed.

I really wanted to echo something indicating the progress, such as a path name or a file count but all the ways I tried to do that slowed things down significantly. I agree that the row of dots I'm now using doesn't have much value but I still think it is better than nothing. I would remove that as well if it were slowing things down, although after I switched to the echo method suggested by another reply on this thread I found that I could barely measure the slow down due to the echo.

I'm no longer considering other solutions since the batch file is now fast enough. Plus I think more than half the time is now spent executing ffprobe, which means there is not much left to be gained. Also the script is now simple enough it could probably be easily adapted to linux, or any OS supporting sed. (I'm not likely to do that myself however). I suspect ffprobe is actually doing quite a bit more than I need, so a faster alternative might be available. Since it's running fast enough at this point I didn't put it much effort in that direction.

Anyway, thanks again for all your help and for putting up with my whining. If you are interested I'll post the final version of my video file script when I finish it, which will probably look quite similar to my final mp3 file script.
~Paul

[pmennen]

This is getting a bit away from my original question that started the thread, but I thought I would report on one last minor problem I ran into with the last script that I posted for listing mp3 files (mp3time.bat). It turns out it won't work if run from the root folder on the drive. The reason is that the path returned by the %CD% environment variable does not normally include the trailing backslash character. But I wanted it there so that it would remove the entire path from the output. So I simply appended a backslash character to the end of the string. This seemed to work fine at first, but in the case when you are running from the root, the CD variable does contain a trailing backslash in that instance, so my code was adding a second backslash causing the sed substitution command to fail. A slight modification to the code (which I'm posting below) fixes this problem. Also after using this for a while I found it more convenient to have the totals reported at the top of the output file instead of at the bottom, so I also made that modification. The totals line now also includes the total of the file sizes. I also updated my video listing script (vidtime.bat) to include the same improved speed and processing features.

Since it might be a bit long to post both of these scripts again to this thread I posted the final versions of them on my web site (mennen.org). Just click on the link labeled "batch scripts" (about half way down the home page). Certainly let me know if you have any questions about them.

~Paul

[pmennen]

I posted the final versions of them on my web site (mennen.org). Just click on the link labeled "batch scripts" (about half way down the home page).

Well, I thought they were final. I just updated them again. Nothing serious, although one of the scripts was not deleting the temporary files properly and there was a few minor errors in the comments. Also I added one additional script (called audiotime.bat) to the list which is quite similar to mp3time.bat except that expands the number of supported autdio formats from one to nine (ape,flac,mpc,mp2,mp3,m4a,ogg,wav,wma). So if you already have downloaded the scripts you might want to do that again.

~Paul

[sambul35]

Adding to the "Escaping Characters" topic, I need to set variables to Windows Registry paths that contain many special characters and spaces like this:

Code: Select all

set cam1="HKCU\Software\Package\Name\//?/usb#vid_04f2&pid_b2ea&mi_00#7&2b10840e&0&0000#{65e8773d-8f56-11d0-a3b9-00a0c9223196}/global" /t REG_DWORD /v VideoProcessorMode

What's the right way to do it, preferably without escaping each special character in each Registry address string, since adding escape chars will make them a lot less readable and harder to match the actual Registry keys.

Is that a bug in this forum software - no scrolling in Code formatted text areas?

EDIT: Found this way to read the entire string:

Code: Select all

FOR /F "usebackq tokens=1,*" %%L IN (
'key "HKCU\Software\Package\Name\//?/usb#vid_04f2&pid_b2ea&mi_00#7&2b10840e&0&0000#{65e8773d-8f56-11d0-a3b9-00a0c9223196}/global" /t REG_DWORD /v VideoProcessorMode'
) DO set cam1=%%M