HASHSUM.BAT v1.8 - emulate md5sum, shasum, and the like

[flywire]

The Windows CERTUTIL command ... it is not very convenient ... :

• CERTUTIL delimits hex pairs with spaces, whereas md5sum etc. uses a continuous stream of hex digits. ...

Has it changed in Win10?

Code: Select all

C:\Users\User\Code\NestBat>certutil -hashfile NestBat.bat
SHA1 hash of NestBat.bat:
fc5650b221ed08b5403cb2cfb308caea79c33d8a
CertUtil: -hashfile command completed successfully.

[flywire]

Very nicely implemented - Well done.

C:\Users\User\Code\NestBat>..\hashsum /c hashsum <nestbat.bat
---------- "hashsum" ----------
OK: batling.bat
OK: NestBat.bat
========== SUMMARY ==========
Total manifests = 1
Matched files = 2

• Can the single file be verified - nestbat.bat?

• Can a single entry in the Manifest be inserted or changed leaving the other entries?

• There must be exactly two spaces between the MD5 hash and the filename.
  Why? (Sorry if I missed it - I currently have a single space.)

(The letters selected for the Verify and Check version switches were a bit unfortunate.)

[dbenham]

The Windows CERTUTIL command ... it is not very convenient ... :

• CERTUTIL delimits hex pairs with spaces, whereas md5sum etc. uses a continuous stream of hex digits. ...

Has it changed in Win10?

Interesting - yes. I have confirmed on my Win 10 machine that CERTUTIL behavior has changed. It now outputs a continuous stream of hex digits without spaces.

But HASHSUM.BAT works either way

Can the single file be verified - nestbat.bat?

I don't have an option to check just one file within a manifest. But you can manually check a single file with the /C option, as implied by the documentation:

Code: Select all

   /C - Read hash values and file names from File (the manifest), and verify
        that local files match. File may include path information with /C.

        If File is not given, then read hash and file names from standard
        input. Each line of input must have a hash, followed by two spaces,
        or a space and an asterisk, followed by a file name.

If you have a manifest and want to check just one file, then you can edit the manifest with your favorite text editor and copy the line of interest. Then just paste it into a command on the command line that is structured as follows:

Code: Select all

echo xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx *file.ext | hashsum /c

Or you can use FIND or FINDSTR to filter the manifest to the one you want:

Code: Select all

find "*file.ext" <manifest.md5 | hashsum /c
findstr /l "*file.ext" manifest.md5 | hashsum /c

Can a single entry in the Manifest be inserted or changed leaving the other entries?

Not via HASHSUM.BAT, but you can certainly edit the manifest with your favorite text editor.

There must be exactly two spaces between the MD5 hash and the filename.
Why? (Sorry if I missed it - I currently have a single space.)

Ask the developers of md5sum. I'm simply emulating the behavior of the existing unix tool(s). Two spaces is supposed to indicate text mode, and space asterisk is supposed to indicate binary mode. But as described in the HASHSUM.BAT documentation, HASHSUM always uses binary mode.


Dave Benham

[flywire]

If you have a manifest and want to check just one file ... you can use FIND ...

Yep, that's the magic one!

Can a single entry in the Manifest be inserted or changed leaving the other entries?

Not via HASHSUM.BAT ...

Happy for you to tidy up these two lines that do most of it:

Code: Select all

find /v /i "NestBat.bat" Manifest.MD5 | find /v /i "----------" | findstr /v "^$"> Manifest.tmp
hashsum /T /A MD5 "NestBat.bat" | find "NestBat.bat" >> Manifest.tmp

My demo UpdateManifest.bat failed:

Code: Select all

:: UpdateManifest
call ..\hashsum /T /A MD5 ???????.bat >Manifest.MD5
Type Manifest.MD5
find /v /i "NestBat.bat" Manifest.MD5 | find /v /i "----------" | findstr /v "^$"> Manifest.tmp
Type Manifest.tmp
call ..\hashsum /T /A MD5 "NestBat.bat" | find "NestBat.bat" >> Manifest.tmp
Type Mainfest.tmp
Del Manifest.MD5
Ren Manifest.tmp *.MD5
Type Manifest.MD5

I typed it manually:

Code: Select all

C:\Users\User\Code\NestBat>..\hashsum /T /A MD5 ???????.bat >Manifest.MD5

C:\Users\User\Code\NestBat>Type Manifest.MD5
7b2cb4aa6c6d6a2023eae1dee98ec025  batling.bat
c19c099a7703768ef1b8dbeec3c45dba  NestBat.bat

C:\Users\User\Code\NestBat>find /v /i "NestBat.bat" Manifest.MD5 | find /v /i "----------" | findstr /v "^$"> Manifest.tmp

C:\Users\User\Code\NestBat>Type Manifest.tmp
7b2cb4aa6c6d6a2023eae1dee98ec025  batling.bat

C:\Users\User\Code\NestBat>..\hashsum /T /A MD5 "NestBat.bat" | find "NestBat.bat" >> Manifest.tmp

C:\Users\User\Code\NestBat>Del Manifest.MD5

C:\Users\User\Code\NestBat>Ren Manifest.tmp *.MD5

C:\Users\User\Code\NestBat>Type Manifest.MD5
7b2cb4aa6c6d6a2023eae1dee98ec025  batling.bat
c19c099a7703768ef1b8dbeec3c45dba  NestBat.bat

Regarding options:

... I'm simply emulating the behavior of the existing unix tool(s). ...

Microsoft can't be consistent from fciv to CertUtil so you might as well. So option case is critical?

[dbenham]

I've updated the first post in this thread to version 1.5 with new features.

It is now easier to check one file within a manifest.

Code: Select all

hashsum /nh /ns /f file.ext /c manifest.md5

Note that the above will check all lines within the manifest that contain the string "file.ext"

Regarding options:

... I'm simply emulating the behavior of the existing unix tool(s). ...

Microsoft can't be consistent from fciv to CertUtil so you might as well. So option case is critical?

Sorry, I was imprecise with my language. My intent is for HASHSUM.BAT to be able to work with digests produced by md5sum etc., as well as be able to create digests that are compatible with md5sum. I'm not worried about exactly replicating the option syntax or functionality. For example, HASHSUM is better able to process directory trees than md5sum.

The HASHSUM option syntax is not case sensitive. Some versions of CERTUTIL are case sensitive when specifying the algorithm, so HASHSUM automatically converts the algorithm string to upper case behind the scenes.

Happy for you to tidy up these two lines that do most of it:

Code: Select all

find /v /i "NestBat.bat" Manifest.MD5 | find /v /i "----------" | findstr /v "^$"> Manifest.tmp
hashsum /T /A MD5 "NestBat.bat" | find "NestBat.bat" >> Manifest.tmp

My demo UpdateManifest.bat failed:

Code: Select all

:: UpdateManifest
call ..\hashsum /T /A MD5 ???????.bat >Manifest.MD5
Type Manifest.MD5
find /v /i "NestBat.bat" Manifest.MD5 | find /v /i "----------" | findstr /v "^$"> Manifest.tmp
Type Manifest.tmp
call ..\hashsum /T /A MD5 "NestBat.bat" | find "NestBat.bat" >> Manifest.tmp
Type Mainfest.tmp
Del Manifest.MD5
Ren Manifest.tmp *.MD5
Type Manifest.MD5

I strongly recommend preserving the asterisk (ditch the /T option), as HASHSUM always uses binary mode, and the asterisk implies binary.

I'm not interested in adding an update feature to HASHSUM, but I believe this is what you are trying to do:

Code: Select all

>Manifest.MD5.new (findstr /ivc:"NestBat.bat" Manifest.MD5 & ..\hashsum /t /a md5 "NestBat.bat")
move /y Manifest.MD5.new Manifest.MD5 >nul

Note that the FINDSTR command will ignore XXXNestBat.bat as well as NestBat.bat. With a bit of work you can construct a regular expression that will only match (or reject) one specific file.


Dave Benham

[dbenham]

I've updated the first post in this thread to version 1.5, with the following new options:

Code: Select all

   /H - Prints the HASHSUM.BAT history.

   /F FileName

        When using /C, only check lines within the manifest that contain the
        string FileName. The search ignores case.

   /FR FileRegEx

        When using /C, only check lines within the manifest that match the
        FINDSTR regular expression FileRegEx. The search ignores case.

   /NH - (No Headers)  Suppresses listing of manifest name(s) when using /C.

You can now conveniently test a single file within a manifest file with the new options.

Assuming the digest file manifest.md5 was prepared using binary, with the asterisks, then you could test just "test.bat" with the following:

Code: Select all

hashsum /nh /ns /f "*test.bat" /c manifest.md5

You could test all .bat files within the manifest using:

Code: Select all

hashsum /nh /ns /f ".bat" /c manifest.md5

You can use the regular expression form (FINDSTR syntax) to test all .bat files that begin with "T":

Code: Select all

hashsum /nh /ns /fr "\*t.*\.bat" /c manifest.md5

Dave Benham

[flywire]

I'm specifically interested in a solution to use in a batch file to verify files being used in the batch file. I realise now hashsum is not a simple batch file.

Dave - Can you comment on any problems with calling hashsum from within a batch file and reading the errorlevels? eg:

Code: Select all

call ..\hashsum /T /A MD5 ???????.bat >Manifest.MD5

I'm concerned about Why does delayed expansion fail when inside a piped block of code?

I assume your comment on /T relates to changing DOS CRLF (end-of-line) characters.

[dbenham]

Yes, the -t option in md5sum normalizes all line endings to LF before computing the checksum. But HASHSUM (and CERTUTIL) do not support this feature. The /T option is accepted, but it does nothing other than control whether the file name is preceded by two spaces or a space and asterisk.

But I am totally lost as to your questions about HASHSUM, errorlevels, and pipes. I don't understand what your concerns are.


Dave Benham

[flywire]

I am totally lost as to your questions about HASHSUM, errorlevels, and pipes. I don't understand what your concerns are.

I don't understand why I can't process all lines in the batch file.

My demo UpdateManifest.bat failed:

Say this example:

Code: Select all

C:\Users\User\Code\NestBat>type ..\updatemanifest.bat
:: UpdateManifest
echo Position 1
>Manifest.MD5.new (findstr /ivc:"NestBat.bat" Manifest.MD5 & ..\hashsum /t /a md5 "NestBat.bat")
move /y Manifest.MD5.new Manifest.MD5
echo Position 2

Code: Select all

C:\Users\User\Code\NestBat>..\updatemanifest

C:\Users\User\Code\NestBat>echo Position 1
Position 1

C:\Users\User\Code\NestBat>(findstr /ivc:"NestBat.bat" Manifest.MD5   & ..\hashsum /t /a md5 "NestBat.bat" ) 1>Manifest.MD5.new

C:\Users\User\Code\NestBat>

[dbenham]

Sorry, I steered you a bit astray with my earlier posted two lines of code. That code (patterned after your post) will only work from the command line.

To make it work from batch, you need to use CALL ..\hashsum.bat so that hashsum returns to your script.


Dave Benham

[Sithuk]

Good work on the code.

I am having difficulty with files that have unicode characters (e.g. German umlaut). I tried inserting the -Unicode flag after the certutil.exe text in the batch file without success. How can I make the batch file work with filenames that have unicode characters?

[dbenham]

Ouch, there may not be much that can be done. HASHSUM is totally dependent on native Windows command line tools, and they have inconsistent support for Unicode.

It would help if you tell me which options you used when the code fails.

Both the code to generate hashes and the code to check hashes are heavily dependent on FOR /F, which works with 8 bit characters sets, not Unicode. Whether a given file path works is dependent on your active code page (CHCP command sets/reads the value). As long as all your file paths use characters that are within the active code page, then I think the FOR /F statements will work. But if you have a diverse set of file paths that do not fit within one code page, then there is nothing that can be done other than separate the paths into different groups and process each group with a code page that works.

The other command used by HASHSUM that can cause problems is FINDSTR, but I think this only comes into play while checking hashes with the /F or /FR option.

The FINDSTR command is perhaps the worst coded command ever made for the Windows command line - it is loaded with idiosyncrasies, poor design choices, and down right bugs. Like FOR /F, it uses an 8 bit character set. One of the most irritating "features" is that FINDSTR absolutely mangles command line arguments that use characters above decimal 127 (hex 7F). The good news is I may be able to tweak how I use FINDSTR in HASHSUM such that this problem does not arise. I think I can use the /G and /F FINDSTR options to avoid paths on the command line, and read them from a temp file instead. I'll try to do this within the next couple days.

I don't think there are any other places in the code that should cause problems.

Once I make the FINDSTR enhancement, you should be able to process any file name (path) as long as the characters are all within the active code page.


Dave Benham

[dbenham]

I've updated the first post in this tread to version 1.6.

The /F and /FR options now support non-ASCII characters (8 bit characters with the high order bit set).


Dave Benham

[bajabaq2000]

Not sure if this could be considered a bug report (or if you can do anything about it), but if you have a file with more than 256 characters hashsum.bat throws an ERROR (which is correct), but it be "more" correct / helpful if it would say "why" this error occured

[dbenham]

Not sure if this could be considered a bug report (or if you can do anything about it), but if you have a file with more than 256 characters hashsum.bat throws an ERROR (which is correct), but it be "more" correct / helpful if it would say "why" this error occured

More than 256 chars in the file content causes an error
That's not right

Or do you mean 256 chars in the file name (or file path)


Dave Benham