HASHSUM.BAT v1.8 - emulate md5sum, shasum, and the like

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Message
Author
dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

HASHSUM.BAT v1.8 - emulate md5sum, shasum, and the like

#1 Post by dbenham » 05 Dec 2016 23:47

The Windows CERTUTIL command has an option to compute file hashes using any of the following algorithms: MD2, MD4, MD5, SHA1, SHA256, SHA384, SHA512.

Code: Select all

C:\test>certutil -hashfile test.bat md5
MD5 hash of file test.bat:
89 68 0c 14 1d 7a 1f 00 a2 43 79 c6 b1 ac fd b9
CertUtil: -hashfile command completed successfully.
But it is not very convenient for preparing or checking manifests (digests) that are compatible with md5sum, shasum, or the like. So I wrote HASHSUM.BAT to package up CERTUTIL in a way that is compatible with md5sum, etc.

There were a few minor issues to overcome:
  • CERTUTIL delimits hex pairs with spaces, whereas md5sum etc. uses a continuous stream of hex digits.
  • CERTUTIL gives an error if the file is empty (size 0). I had to hardcode the various hashes for zero length strings.
  • I needed a way to capture relative paths when using the /S option. I opted to use FORFILES, which is very convenient, but slow.
  • I needed a way to capture stdin to a temp file, without any modification. I initially used FINDSTR, but that appends \r\n if the last line does not end with \n. So I converted to using JScript, thus HASHSUM is no longer pure batch.
I chose to only support MD5, SHA1, SHA256, SHA384, and SHA512. This enables me to automatically detect the hash algorithm within a manifest because each one has a different length. I couldn't support MD2 or MD4 because their hash length is the same as MD5. Plus they are rarely used anyway.

When preparing a manifest, it defaults to SHA256, but you can change the algorithm with the /A option.

When checking a manifest with the /C option, you have three options for specifying the algorithm, with higher precedence at the top.
  • Explicitly specify the algorithm with the /A option
  • Derive the algorithm from the extension of the manifest
  • Automatically detect the algorithm based on the length of the first hash in the manifest
Full documentation is embedded within the script, and is available from the command line via the /? option.

Examples of usage follow the code in this post.

HASHSUM.BAT version 1.8

Code: Select all

@if (@X)==(@Y) @end /* Harmless hybrid line that begins a JScript comment
@goto :Batch

::::
::::HASHSUM.BAT history
::::
::::  v1.8 2021-01-19 - Improved detection of current code page as per work at
::::                    https://www.dostips.com/forum/viewtopic.php?f=3&t=8533
::::  v1.7 2021-01-19 - Added /U option to support Unicode in file names
::::  v1.6 2019-02-26 - Modify /F and /FR to support non-ASCII characters
::::  v1.5 2018-02-18 - Added /H, /F, /FR and /NH options.
::::  v1.4 2016-12-26 - Convert /A value to upper case because some Windows
::::                    versions are case sensitive. Also improve JScript file
::::                    read performance by reading 1000000 bytes instead of 1.
::::  v1.3 2016-12-17 - Bug fixes: Eliminate unwanted \r\n from temp file by
::::                    reading stdin with JScript instead of FINDSTR.
::::                    Fix help to ignore history.
::::  v1.2 2016-12-07 - Bug fixes: Exclude FORFILES directories and
::::                    correct setlocal/endlocal management in :getOptions
::::  v1.1 2016-12-06 - New /V option, and minor bug fixes.
::::  v1.0 2016-12-05 - Original release
:::
:::HASHSUM  [/Option [Value]]... [File]...
:::
:::  Print or check file hashes using any of the following standard
:::  hash algorithms: MD5, SHA1, SHA256, SHA384, or SHA512.
:::
:::  HASHSUM always does a binary read - \r\n is never converted to \n.
:::
:::  In the absence of /C, HASHSUM computes the hash for each File, and writes
:::  a manifest of the results. Each line of output consists of the hash value,
:::  followed by a space and an asterisk, followed by the File name. The default
:::  hash alogrithm is sha256. File may include wildcards, but must not contain
:::  any path information.
:::
:::  If File is not given, then read from standard input and write the hash
:::  value only, without the trailing space, asterisk, or file name.
:::
:::  Options:
:::
:::    /? - Prints this help information to standard output.
:::
:::    /?? - Prints paged help using MORE.
:::
:::    /V - Prints the HASHSUM.BAT version.
:::
:::    /H - Prints the HASHSUM.BAT history.
:::
:::    /U - Unicode mode: Experimental setting that attempts to support
:::         Unicode in filenames. Please report any problems to the DosTips
:::         forum at https://www.dostips.com/forum/viewtopic.php?f=3&t=7592
:::
:::    /A Algorithm
:::
:::         Specifies one of the following hash algorithms:
:::         MD5, SHA1, SHA256, SHA384, SHA512
:::
:::    /P RootPath
:::
:::         Specifies the root path for operations.
:::         The default is the current directory.
:::
:::    /S - Recurse into all Subdirectories. The relative path from the root
:::         is included in the file name output.
:::         This option is ignored if used with /C.
:::
:::    /I - Include the RootPath in the file name output.
:::         This option is ignored if used with /C.
:::
:::    /T - Writes a space before each file name, rather than an
:::         asterisk. However, files are still read in binary mode.
:::         This option is ignored if used with /C.
:::
:::    /C - Read hash values and file names from File (the manifest), and verify
:::         that local files match. File may include path information with /C.
:::
:::         If File is not given, then read hash and file names from standard
:::         input. Each line of input must have a hash, followed by two spaces,
:::         or a space and an asterisk, followed by a file name.
:::
:::         If /A is not specified, then the algorithm is determined by the
:::         File extension. If the extension is not a valid algorithm, then
:::         the algorithm is derived based on the length of the first hash
:::         within File.
:::
:::         Returns ERRORLEVEL 1 if any manifest File is not found or is invalid,
:::         or if any local file is missing or does not match the hash value in
:::         the manifest. If all files are found and match, then returns 0.
:::
:::    /F FileName
:::
:::         When using /C, only check lines within the manifest that contain the
:::         string FileName. The search ignores case.
:::
:::    /FR FileRegEx
:::
:::         When using /C, only check lines within the manifest that match the
:::         FINDSTR regular expression FileRegEx. The search ignores case.
:::
:::    /NH - (No Headers)  Suppresses listing of manifest name(s) when using /C.
:::
:::    /NE - (No Errors) Suppresses error messages when using /C.
:::
:::    /NM - (No Matches) Suppresses listing of matching files when using /C.
:::
:::    /NS - (No Summary) Suppresses summary information when using /C.
:::
:::    /Q  - (Quiet) Suppresses all output when using /C.
:::
:::HASHSUM.BAT version 1.8 was written by Dave Benham
:::maintained at https://www.dostips.com/forum/viewtopic.php?f=3&t=7592

============= :Batch portion ===========
@echo off
setlocal disableDelayedExpansion

:: Define options
set "options= /A:"" /C: /I: /P:"" /S: /T: /?: /??: /NH: /NE: /NM: /NS: /Q: /V: /H: /U: /F:"" /FR:"" "

:: Set default option values
for %%O in (%options%) do for /f "tokens=1,* delims=:" %%A in ("%%O") do set "%%A=%%~B"
set "/?="
set "/??="

:getOptions
if not "%~1"=="" (
  set "test=%~1"
  setlocal enableDelayedExpansion
  if "!test:~0,1!" neq "/" endlocal & goto :endOptions
  set "test=!options:*%~1:=! "
  if "!test!"=="!options! " (
      endlocal
      >&2 echo Invalid option %~1
      exit /b 1
  ) else if "!test:~0,1!"==" " (
      endlocal
      set "%~1=1"
  ) else (
      endlocal
      set "%~1=%~2"
      shift /1
  )
  shift /1
  goto :getOptions
)
:endOptions

:: Display paged help
if defined /?? (
  (for /f "delims=: tokens=*" %%A in ('findstr "^:::[^:] ^:::$" "%~f0"') do @echo(%%A)|more /e
  exit /b 0
) 2>nul

:: Display help
if defined /? (
  for /f "delims=: tokens=*" %%A in ('findstr "^:::[^:] ^:::$" "%~f0"') do echo(%%A
  exit /b 0
)

:: Display version
if defined /V (
  for /f "delims=: tokens=*" %%A in ('findstr /ric:"^:::hashsum.bat version" "%~f0"') do echo(%%A
  exit /b 0
)

:: Display history
if defined /H (
  for /f "delims=: tokens=*" %%A in ('findstr "^::::" "%~f0"') do echo(%%A
  exit /b 0
)

:: Unicode filename support
set "find=find"
if defined /U (
  for /f "tokens=*" %%A in ('chcp') do for %%B in (%%A) do set "chcp=%%~nB"
  set "find=findstr /l"
  >nul chcp 65001
)

:: If no file specified, then read stdin and write to a temp file
set "tempFile="
if "%~1" equ "" set "tempFile=%~nx0.%time::=_%.%random%.tmp"
if defined tempFile cscript //nologo //E:JScript "%~f0" "%temp%\%tempFile%"

if defined /P cd /d "%/P%" || exit /b 1
if defined /C goto :check

:generate
if defined tempFile cd /d "%temp%"
if not defined /A set "/A=sha256"
if defined /S set "/S=/s"
if defined /T (set "/T= ") else set "/T=*"
call :defineEmpty
if not defined /P goto :generateLoop
if not defined /I goto :generateLoop
if "%/P:~-1%" equ "\" (set "/I=%/P:\=/%") else set "/I=%/P:\=/%/"
set "rtn=0"

:generateLoop
(
  for /f "delims=" %%F in (
    'forfiles %/s% /m "%tempFile%%~1" /c "cmd /c if @isdir==FALSE echo @relpath" 2^>nul'
  ) do for /f "delims=" %%A in (
    'certutil.exe -hashfile %%F %/A% ^| %find% /v ":" ^|^| if %%~zF gtr 0 (echo X^) else echo %empty%'
  ) do (
    set "file=%%~F"
    set "hash=%%A"
    setlocal enableDelayedExpansion
    set "file=!file:~2!"
    if defined tempFile (
      if !hash! equ X (
        set "rtn=1"
        echo ERROR
      ) else echo !hash: =!
    ) else (
      if !hash! equ X (
        set "rtn=1"
        echo ERROR: !/I!!file!
      ) else echo !hash: =! !/T!!/I!!file:\=/!
    )
    endlocal
  )
) || (
  set "rtn=1"
  echo MISSING: %/T%%1
)
shift /1
if "%~1" neq "" goto :generateLoop
if defined tempFile del "%tempFile%"
if defined /U >nul chcp %chcp%
exit /b %rtn%

:check
if defined /Q for %%V in (/NE /NM /NS /NH) do set "%%V=1"
if defined /F if defined /FR (
  >&2 echo ERROR: /F and /FR cannot be combined
  exit /b 1
)
set "searchTemp="
if defined /F (
  set "searchTemp=%temp%\%~nx0.%time::=_%.%random%.search.tmp"
  setlocal enableDelayedExpansion
  (echo(!/F!) > "!%searchTemp!"
  endlocal
  set "file="    & set "freg=rem" & set "norm=rem"
) else if defined /FR (
  set "searchTemp=%temp%\%~nx0.%time::=_%.%random%.search.tmp"
  setlocal enableDelayedExpansion
  (echo(!/FR!) > "!%searchTemp!"
  endlocal
  set "file=rem" & set "freg="    & set "norm=rem"
) else (
  set "file=rem" & set "freg=rem" & set "norm="
)
set /a manifestCnt=missingManifestCnt=invalidCnt=missingCnt=failCnt=okCnt=0

:checkLoop
set "alogorithm=%/A%"
if defined tempFile set "tempFile=%temp%\%tempFile%"
for %%F in ("%tempFile%%~1") do call :checkFile "%%~F"
if defined tempFile del "%tempFile%"
shift /1
if "%~1" neq "" goto :checkLoop

if defined searchTemp del "%searchTemp%"

if not defined /NS (
  echo ==========  SUMMARY  ==========
  echo Total manifests   = %manifestCnt%
  echo Matched files     = %okCnt%
  echo(
  if %missingManifestCnt% gtr 0 echo Missing manifests = %missingManifestCnt%
  if %invalidCnt% gtr 0         echo Invalid manifests = %invalidCnt%
  if %missingCnt% gtr 0         echo Missing files     = %missingCnt%
  if %failCnt% gtr 0            echo Failed files      = %failCnt%
)
if defined /U >nul chcp %chcp%
set /a "1/(missingManifestCnt+invalidCnt+missingCnt+failCnt)" 2>nul && (
  echo(
  exit /b 1
)
exit /b 0

:checkFile
set /a manifestCnt+=1
if not defined /NH if defined tempfile (echo ----------  ^<stdin^>  ----------) else echo ----------  %1  ----------
if not defined algorithm set "/A="
if not defined /A echo *.md5*.sha1*.sha256*.sha384*.sha512*|%find% /i "*%~x1*" >nul && for /f "delims=." %%A in ("%~x1") do set "/A=%%A"
findstr /virc:"^[0123456789abcdef][0123456789abcdef]* [ *][^ *?|<>]" %1 >nul 2>nul && (
  if not defined /NE if defined tempFile (echo *INVALID: ^<stdin^>) else echo *INVALID: %1
  set /a invalidCnt+=1
  exit /b
)
(
  %norm% for /f "usebackq tokens=1* delims=* " %%A in (%1) do (
  %file% for /f "tokens=1* delims=* " %%A in ('type %1 ^| findstr /ilg:"%searchTemp%"') do (
  %freg% for /f "tokens=1* delims=* " %%A in ('type %1 ^| findstr /irg:"%searchTemp%"') do (
    set "hash0=%%A"
    set "fileName=%%B"
    if defined /A (call :defineEmpty) else call :determineFormat
    setlocal enableDelayedExpansion
    set "fileName=!fileName:/=\!"
    for /f "tokens=1* delims=" %%C in (
      'certutil.exe -hashfile "!fileName!" !/A! ^| %find% /v ":" ^|^| if exist "!fileName!" (echo !empty!^) else echo X'
    ) do set "hash=%%C"
    if /i "!hash0!" equ "!hash: =!" (
      if not defined /NM echo OK: !fileName!
      endlocal
      set /a okCnt+=1
    ) else if !hash! equ X (
      if not defined /NE echo *MISSING: !fileName!
      endlocal
      set /a missingCnt+=1
    ) else (
      if not defined /NE echo *FAILED: !fileName!
      endlocal
      set /a failCnt+=1
    )
  )
) 2>nul || if not defined /F if not defined /FR (
  if not defined /NE echo *MISSING: %1
  set /a missingManifestCnt+=1
)
exit /b

:determineFormat
if "%hash0:~127%" neq "" (
  set "/A=SHA512"
) else if "%hash0:~95%" neq "" (
  set "/A=SHA384"
) else if "%hash0:~63%" neq "" (
  set "/A=SHA256"
) else if "%hash0:~39%" neq "" (
  set "/A=SHA1"
) else set "/A=MD5"

:defineEmpty
if /i "%/A%"=="md5" (
  set "empty=d41d8cd98f00b204e9800998ecf8427e"
  set "/A=MD5"
) else if /i "%/A%"=="sha1" (
  set "empty=da39a3ee5e6b4b0d3255bfef95601890afd80709"
  set "/A=SHA1"
) else if /i "%/A%"=="sha256" (
  set "empty=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
  set "/A=SHA256"
) else if /i "%/A%"=="sha384" (
  set "empty=38b060a751ac96384cd9327eb1b1e36a21fdb71114be07434c0cc7bf63f6e1da274edebfe76f65fbd51ad2f14898b95b"
  set "/A=SHA384"
) else if /i "%/A%"=="sha512" (
  set "empty=cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e"
  set "/A=SHA512"
) else (
  echo ERROR: Invalid /A algorithm>&2
  (goto) 2>nul&exit /b 1
)
exit /b


************* JScript portion **********/
var fso = new ActiveXObject("Scripting.FileSystemObject");
var out = fso.OpenTextFile(WScript.Arguments(0),2,true);
var chr;
while( !WScript.StdIn.AtEndOfStream ) {
  chr=WScript.StdIn.Read(1000000);
  out.Write(chr);
}
Here is an example showing the generation of MD5 digests for all exe files under my c:\test folder, including subdirectories.

Code: Select all

C:\test>hashsum /s /a md5 *.exe
8d443f2e93a3f0b67f442e4f1d5a4d6d *md5.exe
eb574b236133e60c989c6f472f07827b *md5sum.exe
9f036e40755daaa5a5fc141d3cbfbb99 *nGetPid.exe
36622f75d59bd1988bfd18a259ae82ef *Aacini Tools/Ascii.exe
f26e6cdc8f8bba5df1634bb631f6306f *Aacini Tools/Colorshow.exe
512b675126f22da6d50916b98222c5d7 *Aacini Tools/Cursorpos.exe
bda35b908b81f7dfad35beafd1582279 *Aacini Tools/Cursorsize.exe
4cf48fe5dca5638d0ddfd3429faf6abf *Aacini Tools/Flushinputbuffer.exe
d81a9a79847aa075bebb8ec054bd13e7 *Aacini Tools/Getkey.exe
b269ecf2bed68cef8d623c877c3b93d6 *Aacini Tools/Show.exe
ddbece33177224a8a2f44600cc71e76b *Aacini Tools/Stddate.exe
0d929b261d0981d3c46c04418af2ec6d *Aacini Tools/Stdtime.exe
34df06129a094d773c14092eb151a8e0 *Aacini Tools/Strlen.exe
c0a7f18cb075d39ab2d0978dd7a261dc *Aacini Tools/Textcolor.exe
3ff1f054f90a708e0fdeb8b4c81db24e *Aacini Tools/Window.exe
538d34e6b28951cb412761e634805f5b *Aacini Tools/bin/Hexchar.exe
dfc42151c290ae5bbbc448d211405489 *ansisys/AnsiSys.exe
6795af42d625abe4bd529798aba07bfd *bf/BFI.exe
5f02fcfe6262dfb852b5ce904f14e20d *bf/**censored**.exe
0e93af2d97030b34334b8f3200f5ca44 *calc/ColorChar.exe
512b675126f22da6d50916b98222c5d7 *calc/CursorPos.exe
82d93fd60f4686c82034f8a70dc0b269 *calc/GetInput.exe
59f74ed847f7b7fac82b809896b12f67 *choice/choice.exe
50c52220cd00b642ff4e5b85abd28557 *curl/curl.exe
33900b6c02c16fcf630b468a36156496 *fonts/hbFontEdit/FONTEDIT.EXE
c6e160e565a0b5764cf70ed258b9ffd9 *fonts/MittensKnight/FontEdit.exe
a89f984fbb6820cd505971f5f6d0ca89 *fonts/RasterFontEditor/RasterFontEditor.exe
e49097bb9659520ae8cb7cdddab1f5fd *pixelfnt_dev/PIXELFNT.EXE
69b8044bb90d89705289df7e10a5152a *SendMessage/SendMessage.exe
I can capture the output as a manifest using simple redirection. Say I saved the file as exe_files.md5, then I can simply use the following to verify all files in the manifest:

Code: Select all

C:\test>hashsum /c exe_files.md5
----------  "exe_files.md5"  ----------
OK: md5.exe
OK: md5sum.exe
OK: nGetPid.exe
OK: Aacini Tools\Ascii.exe
OK: Aacini Tools\Colorshow.exe
OK: Aacini Tools\Cursorpos.exe
OK: Aacini Tools\Cursorsize.exe
OK: Aacini Tools\Flushinputbuffer.exe
OK: Aacini Tools\Getkey.exe
OK: Aacini Tools\Show.exe
OK: Aacini Tools\Stddate.exe
OK: Aacini Tools\Stdtime.exe
OK: Aacini Tools\Strlen.exe
OK: Aacini Tools\Textcolor.exe
OK: Aacini Tools\Window.exe
OK: Aacini Tools\bin\Hexchar.exe
OK: ansisys\AnsiSys.exe
OK: bf\BFI.exe
OK: bf\**censored**.exe
OK: calc\ColorChar.exe
OK: calc\CursorPos.exe
OK: calc\GetInput.exe
OK: choice\choice.exe
OK: curl\curl.exe
OK: fonts\hbFontEdit\FONTEDIT.EXE
OK: fonts\MittensKnight\FontEdit.exe
OK: fonts\RasterFontEditor\RasterFontEditor.exe
OK: pixelfnt_dev\PIXELFNT.EXE
OK: SendMessage\SendMessage.exe
==========  SUMMARY  ==========
Total manifests   = 1
Matched files     = 29
I can use a pipe to quickly check a file if I happen to know the hash value

Code: Select all

C:\test>echo 9f036e40755daaa5a5fc141d3cbfbb99 *nGetPid.exe | hashsum /c
----------  <stdin>  ----------
OK: nGetPid.exe
==========  SUMMARY  ==========
Total manifests   = 1
Matched files     = 1
I can also compute the hash of any text by piping the content into HASHSUM

Code: Select all

C:\test>echo Hello world|hashsum /a sha1
f0e19ccba8cd50a4288c368eeb655c524fedcf5f
The new /U option in version 1.8 can be used to enable support for Unicode in filenames. This may be needed if your filesystem has file names that are not supported by the active code page. However, at time of this release, this option has not been adequately tested - please report any problems with the /U option to this thread


Dave Benham

jfl
Posts: 226
Joined: 26 Oct 2012 06:40
Location: Saint Hilaire du Touvet, France
Contact:

Re: HASHSUM.BAT - emulate md5sum, shasum, and the like, using pure batch

#2 Post by jfl » 06 Dec 2016 04:03

Beautiful code, thanks for sharing it!

I loaded it in my favorite editor, to test the updates I've made to the batch syntax colorizer, and caught a minor typo though:
At the very end, this null should be nul:

Code: Select all

(goto) 2>null


JF

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: HASHSUM.BAT - emulate md5sum, shasum, and the like, using pure batch

#3 Post by dbenham » 06 Dec 2016 07:49

Thanks for catching that jfl.

I fixed that bug, and also fixed a bug when invalid options are passed.

I also added the /V option

I've updated the original post with version 1.1


Dave Benham

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: HASHSUM.BAT - emulate md5sum, shasum, and the like, using pure batch

#4 Post by dbenham » 06 Dec 2016 23:20

I discovered and squashed a couple more bugs.

1) I had forgotten to exclude directories when using FORFILES

2) I had a delayed expansion leak in my :getOptions loop that led to file names containing !, and possibly ^, getting corrupted.

I've updated the code in the first post to version 1.2


Dave Benham

LotPings
Posts: 10
Joined: 26 Oct 2016 22:45

Re: HASHSUM.BAT - emulate md5sum, shasum, and the like, using pure batch

#5 Post by LotPings » 16 Dec 2016 08:37

Hello Dave,

I just discovered that I get different results when doing

Code: Select all

> HashSum.bat /a MD5 HashSum.bat
360063b6d87eb8b703629bd70f522e4d *HashSum.bat

vs

Code: Select all

> HashSum.bat /a MD5 <HashSum.bat
2a203dd0b31196f4b43c12ddea06f6f1


Did the latter to get just the hash no file information

Squashman
Expert
Posts: 4465
Joined: 23 Dec 2011 13:59

Re: HASHSUM.BAT - emulate md5sum, shasum, and the like, using pure batch

#6 Post by Squashman » 16 Dec 2016 09:47

LotPings wrote:Hello Dave,

I just discovered that I get different results when doing

Code: Select all

> HashSum.bat /a MD5 HashSum.bat
360063b6d87eb8b703629bd70f522e4d *HashSum.bat

vs

Code: Select all

> HashSum.bat /a MD5 <HashSum.bat
2a203dd0b31196f4b43c12ddea06f6f1


Did the latter to get just the hash no file information

Hmmm.

I got the same hash sums all three ways.

Code: Select all

C:\BatchFiles\hashSum>HashSum.bat /a MD5 HashSum.bat
2a203dd0b31196f4b43c12ddea06f6f1 *hashSum.bat

C:\BatchFiles\hashSum>HashSum.bat /a MD5 <HashSum.bat
2a203dd0b31196f4b43c12ddea06f6f1

C:\BatchFiles\hashSum>type HashSum.bat|HashSum.bat /a MD5
2a203dd0b31196f4b43c12ddea06f6f1

penpen
Expert
Posts: 1991
Joined: 23 Jun 2013 06:15
Location: Germany

Re: HASHSUM.BAT - emulate md5sum, shasum, and the like, using pure batch

#7 Post by penpen » 16 Dec 2016 11:18

This was a nice riddle (thx @LotPings):
You get different results, when removing the last newline string (carriage return and newline character == "\r\n") from File HashSum.bat.

It seems using redirected and piped input both add an unwanted newline:

Code: Select all

Z:\>dir HashSum.bat
 Volume in Laufwerk Z: hat keine Bezeichnung.
 Volumeseriennummer: 0CCF-5234

 Verzeichnis von Z:\

16.12.2016  18:14             9.144 HashSum.bat
               1 Datei(en),          9.144 Bytes
               0 Verzeichnis(se), 349.091.880.960 Bytes frei

Z:\>HashSum.bat /a MD5 HashSum.bat
360063b6d87eb8b703629bd70f522e4d *HashSum.bat

Z:\>HashSum.bat /a MD5 <HashSum.bat
2a203dd0b31196f4b43c12ddea06f6f1

Z:\>type HashSum.bat|HashSum.bat /a MD5
2a203dd0b31196f4b43c12ddea06f6f1

Z:\>>>HashSum.bat echo(

Z:\>dir HashSum.bat
 Volume in Laufwerk Z: hat keine Bezeichnung.
 Volumeseriennummer: 0CCF-5234

 Verzeichnis von Z:\

16.12.2016  18:16             9.146 HashSum.bat
               1 Datei(en),          9.146 Bytes
               0 Verzeichnis(se), 349.091.876.864 Bytes frei

Z:\>HashSum.bat /a MD5 HashSum.bat
2a203dd0b31196f4b43c12ddea06f6f1 *HashSum.bat

Z:\>HashSum.bat /a MD5 <HashSum.bat
2a203dd0b31196f4b43c12ddea06f6f1

Z:\>type HashSum.bat|HashSum.bat /a MD5
2a203dd0b31196f4b43c12ddea06f6f1


penpen

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: HASHSUM.BAT - emulate md5sum, shasum, and the like, using pure batch

#8 Post by aGerman » 16 Dec 2016 11:26

Yes you can produce similar effects with other control characters.

SUB (0x1A) appended to the Code:

Code: Select all

C:\Users\steffen\Desktop>hashsum.bat /a MD5 hashsum.bat
129a5edc52504224abadf7ed4a0c075c *hashsum.bat

C:\Users\steffen\Desktop>hashsum.bat /a MD5<hashsum.bat
b59d157b42234966474037298837099f

C:\Users\steffen\Desktop>type hashsum.bat | hashsum.bat /a MD5
2a203dd0b31196f4b43c12ddea06f6f1


Form Feed (0x0C) appended:

Code: Select all

C:\Users\steffen\Desktop>hashsum.bat /a MD5 hashsum.bat
984be2fb67e9078cae51264fe1611fc5 *hashsum.bat

C:\Users\steffen\Desktop>hashsum.bat /a MD5<hashsum.bat
c315843c6118b4bb6d2647052a959a93

C:\Users\steffen\Desktop>type hashsum.bat | hashsum.bat /a MD5
2a203dd0b31196f4b43c12ddea06f6f1


Steffen

LotPings
Posts: 10
Joined: 26 Oct 2016 22:45

Re: HASHSUM.BAT - emulate md5sum, shasum, and the like, using pure batch

#9 Post by LotPings » 16 Dec 2016 13:37

penpen wrote:This was a nice riddle (thx @LotPings):
You get different results, when removing the last newline string (carriage return and newline character == "\r\n") from File HashSum.bat.

It seems using redirected and piped input both add an unwanted newline:
penpen


Thanks for your detective work 8)

Well the missing crlf at the end stems from my marking with the mouse in the browser, I do it from bottom to top and will remember to add the crlf manually when checking.

Cheers

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: HASHSUM.BAT - emulate md5sum, shasum, and the like, using pure batch

#10 Post by aGerman » 16 Dec 2016 13:57

Well the missing crlf at the end stems from my marking with the mouse in the browser

The "Select all" link at the top of a code block is a nice feature. Especially for such a long code :wink:

Steffen

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: HASHSUM.BAT - emulate md5sum, shasum, and the like, using pure batch

#11 Post by dbenham » 17 Dec 2016 11:44

I fixed the bug and updated the first post to version 1.3.

penpen wrote:This was a nice riddle (thx @LotPings):
You get different results, when removing the last newline string (carriage return and newline character == "\r\n") from File HashSum.bat.

It seems using redirected and piped input both add an unwanted newline:

Thanks for figuring this out penpen. But it is not the redirection or pipe that adds the \r\n, but rather FINDSTR. I needed a method to read stdin verbatim and write the result to a temporary file. FINDSTR almost fit the bill, but I forgot about the following "feature" that I had already documented at StackOverflow - What are the undocumented features and limitations of the Windows FINDSTR command?
dbenham at StackOverflow wrote:Piped and Redirected input may have <CR><LF> appended
If the input is piped in and the last character of the stream is not <LF>, then FINDSTR will automatically append <CR><LF> to the input. This has been confirmed on XP, Vista and Windows 7. (I used to think that the Windows pipe was responsible for modifying the input, but I have since discovered that FINDSTR is actually doing the modification.)

The same is true for redirected input on Vista. If the last character of a file used as redirected input is not <LF>, then FINDSTR will automatically append <CR><LF> to the input. However, XP and Windows 7 do not alter redirected input.

So I switched to using JScript to capture stdin to a temporary file. Now HASHSUM always gives the same result, regardless where the data comes from.

I also fixed the help to ignore history.

I have updated the code in the original post to version 1.3.


Dave Benham

Meerkat
Posts: 89
Joined: 19 Jul 2015 02:27
Location: Philippines

Re: HASHSUM.BAT - emulate md5sum, shasum, and the like

#12 Post by Meerkat » 26 Dec 2016 06:57

Thanks for the Batch File!

Hmm... Something is weird in my CERTUTIL - the hash algorithm in -hashfile must be capitalized! :shock:

See this screenshot:
Image
Therefore, when using HASHSUM.BAT, I must specify the algorithm capitalized. I do not know if this is only on me, so this needs more test on other machines. :roll:

Also, when I run HASHSUM.BAT with no other arguments, it prints nothing and does not stop.
I hope this helps!

Meerkat

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: HASHSUM.BAT - emulate md5sum, shasum, and the like

#13 Post by dbenham » 26 Dec 2016 08:53

Meerkat wrote:Hmm... Something is weird in my CERTUTIL - the hash algorithm in -hashfile must be capitalized! :shock:
Therefore, when using HASHSUM.BAT, I must specify the algorithm capitalized. I do not know if this is only on me, so this needs more test on other machines. :roll:

Interesting. I did a search and found other reports that the CERTUTIL hash algorithm is case sensitive. But I couldn't find any post that states which versions are case sensitive. What Win version are you using? I've tested on Win 7 and Win 10, and they are not case sensitive.

Meerkat wrote:Also, when I run HASHSUM.BAT with no other arguments, it prints nothing and does not stop.

That is the expected behavior. If you don't specify a file, then HASHSUM computes the hash for all data received from stdin. You could type any text you want, and when done, press <Ctrl-Z> <Enter>. But usually this feature is used with redirection or a pipe.

This behavior is similar to SORT. If you execute SORT without any arguments, then it appears to hang as well. Again, you terminate input with <Ctrl-Z><Enter>

============================

I modified HASHSUM.BAT to convert all /A values to uppercase before calling CERTUTIL. I also improved performance of piped and redirected input by reading 1,000,000 byte chunks instead of 1.

I've update the initial post of this thread to version 1.4


Dave Benham

Meerkat
Posts: 89
Joined: 19 Jul 2015 02:27
Location: Philippines

Re: HASHSUM.BAT - emulate md5sum, shasum, and the like

#14 Post by Meerkat » 26 Dec 2016 10:05

dbenham wrote: What Win version are you using? I've tested on Win 7 and Win 10, and they are not case sensitive.

Windows 7 Starter (SP1).

Meerkat

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: HASHSUM.BAT - emulate md5sum, shasum, and the like

#15 Post by dbenham » 26 Dec 2016 10:10

:shock:

Post Reply