Create nul and all ascii characters with only batch

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Message
Author
foxidrive
Expert
Posts: 6033
Joined: 10 Feb 2012 02:20

Re: Create nul and all ascii characters with only batch

#61 Post by foxidrive » 31 Jan 2014 18:39

carlos wrote: in the past when I post some code to alt.msdos.batch.nt it add a space to the end of every line of the code


I presume you posted using Google.

Google *really* mangles Usenet posts - but some Usenet clients (Thunderbird) also mangles some text.

Most Usenet clients work well.



Good work on creating the hex 00 there, Sponge Belly.

miskox
Posts: 330
Joined: 28 Jun 2010 03:46

Re: Create nul and all ascii characters with only batch

#62 Post by miskox » 02 Feb 2014 08:53

Sponge Belly wrote:I hereby nominate Carlos for promotion to Expert. Any seconds?


+1.

Saso

carlos
Expert
Posts: 485
Joined: 20 Aug 2010 13:57
Location: Chile
Contact:

Re: Create nul and all ascii characters with only batch

#63 Post by carlos » 09 Feb 2014 19:12

I develop a way for increase the speed of creation of the all 255 .chr files.
This code uses function :gen255 that use asynchronus macro for generate the .chr files.
Because, the task are asynchronus, the code uses a loop for wait until the .chr file have 1 byte (that is when it is ready for use).

Code: Select all

@echo off

:gen255
::This code creates one single byte. Parameter: int
::This is encoded in a macro way for be called with cmd /c
::Teamwork of carlos, penpen, aGerman, dbenham
::Tested under Win8
set "genchr=if not %%#==26 (type nul >%%#.tmp"
set "genchr=%genchr% &makecab /d compress=off"
set "genchr=%genchr% /d reserveperdatablocksize=26"
set "genchr=%genchr% /d reserveperfoldersize=%%#"
set "genchr=%genchr% %%#.tmp %%#.chr >nul"
set "genchr=%genchr% &type %%#.chr"
set "genchr=%genchr% |((for /l %%_ in (1 1 38) do pause)>nul"
set "genchr=%genchr% &findstr ^^^^ >%%#.tmp)"
set "genchr=%genchr% &copy /y %%#.tmp /a %%#.chr /b >nul"
set "genchr=%genchr% &del %%#.tmp) else"
set "genchr=%genchr% (copy /y nul + nul /a 26.chr /a >nul)"
::end of genchr macro::
del /f /q /a *.chr >nul 2>&1
for /l %%# in (0,1,255) do start "" /b /i cmd /d /c "%genchr%"
set "check=0"
:check255
for /l %%# in (%check%,1,255) do for %%_ in (
%%#.chr) do if not "1"=="%%~z_" (
ping -l 0 -n 1 -w 100 1.0.0.0 >nul
set "check=%%#"
goto :check255)
goto :eof


dbenham
Expert
Posts: 2253
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: Create nul and all ascii characters with only batch

#64 Post by dbenham » 09 Feb 2014 23:24

Nice work Carlos. 8)

I took your idea and optimized it some more. I was able to basically double the speed yet again :D

1) There is no point in creating more processes than you have processors (or cores)

2) Better to launch the last process synchronously. Then you don't have to begin checking if done until the last one completes. Each process is doing basically the same amount of work, so they should all end about the same time. So I don't bother putting a delay in the wait loop since I am confident it won't be running for long.

The above points don't really improve speed, but they do minimize the number of processes that the OS has to keep track of.

3) Most of the printable ASCII characters can be generated much more quickly using CMD /C EXIT n to set %=exitCodeAscii%. This saves a lot of time.

Code: Select all

@echo off
setlocal

:genAllChr
::This code creates 256 1 byte files, one for each possible byte value.
::This is encoded in a macro way to be called asynchronously with start cmd /c
::Teamwork of carlos, penpen, aGerman, dbenham
::Tested under Win7 and XP
set ^"genchr=(^
for /l %%N in (%%A !cnt! 255) do (^
  if %%N equ 26 (^
    copy /y nul + nul /a 26.chr /a ^>nul^
  ) else (if %%N geq 35 if %%N leq 126 if %%N neq 61 (^
    cmd /c exit %%N^&^
    ^<nul set /p "=!=exitCodeAscii!" ^>%%N.chr^
  ))^&^
  if not exist %%N.chr (^
    makecab /d compress=off /d reserveperdatablocksize=26 /d reserveperfoldersize=%%N %%A.tmp %%N.chr ^>nul^&^
    type %%N.chr ^| ((for /l %%n in (1 1 38) do pause)^>nul^&findstr "^^" ^>%%N.temp)^&^
    ^>nul copy /y %%N.temp /a %%N.chr /b^&^
    del %%N.temp^
  )^
))^&^
del %%A.tmp^"
del /f /q /a *.chr *.tmp >nul 2>&1
set /a cnt=number_of_processors
if %cnt% lss 1 set cnt=1
if %cnt% gtr 256 set cnt=256
set /a "end=cnt-1"
for /l %%A in (0 1 %end%) do (
  type nul >%%A.tmp
  if %%A equ %end% (
    start "" /b /wait cmd /q /v:on /c "%genchr%"
  ) else (
    start "" /b cmd /q /v:on /c "%genchr%"
  )
)
:genAllChr.check
if exist *.tmp goto :genAllChr.check
exit /b

Here are some timings I got comparing your original code and my modified version.

Code: Select all

                          Carlos' original       My modifid version
                        -------------------     --------------------
Win7 64bit               7.93 7.47 7.12          3.77 3.72 3.79
4 processors (cores)     Avg = 7.51              Avg = 3.76

XP virtual machine       57.73 57.92 57.50       35.71 35.93 36.13
1 processor              Avg = 57.72             Avg = 35.92


Dave Benham

carlos
Expert
Posts: 485
Joined: 20 Aug 2010 13:57
Location: Chile
Contact:

Re: Create nul and all ascii characters with only batch

#65 Post by carlos » 09 Feb 2014 23:44

dbenham: thanks for the tips. I not know why, your posted code generates really faster the .chr files, but the script not end.
I will take some of your ideas using ExitCodeAscii and post a new code.

dbenham
Expert
Posts: 2253
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: Create nul and all ascii characters with only batch

#66 Post by dbenham » 10 Feb 2014 00:05

carlos wrote:your posted code generates really faster the .chr files, but the script not end.

That is really weird. It works perfectly for me, and the code seems so straight forward. I'm eager to hear what the problem is.


Dave Benham

foxidrive
Expert
Posts: 6033
Joined: 10 Feb 2012 02:20

Re: Create nul and all ascii characters with only batch

#67 Post by foxidrive » 10 Feb 2014 00:25

Dave's code ends here ok.

I got these timings:

1.01
1.11
1.11

carlos
Expert
Posts: 485
Joined: 20 Aug 2010 13:57
Location: Chile
Contact:

Re: Create nul and all ascii characters with only batch

#68 Post by carlos » 10 Feb 2014 01:21

Dave, thinking about you say all process should end the work at same point: I remember write that function because without it i get some messages like: is not posible access to the file. try rebuild a binary file with the generated.

Edit: i found the cause in your code the waiting forever. this portions:

Code: Select all

:genAllChr.check
if exist *.tmp goto :genAllChr.check

why? I don't now why: In my current test folder i have strange hidden folders:

Code: Select all

09-02-2014  07:09    <DIR>          0529b7c51f51499b97790158a5c048e0$dpx$.tmp
09-02-2014  07:09    <DIR>          0f9a6cef33234a339777cc8c7a6ac410$dpx$.tmp
09-02-2014  22:29    <DIR>          51028f328af346c7877d778b7cd1a38e$dpx$.tmp
04-02-2014  23:59    <DIR>          7d7efd315de74dc195a48db87689949e$dpx$.tmp
09-02-2014  07:09    <DIR>          fe4c2a5674f04b898a40277759970f75$dpx$.tmp


deleted it, the code ends ok.
Last edited by carlos on 10 Feb 2014 01:59, edited 1 time in total.

carlos
Expert
Posts: 485
Joined: 20 Aug 2010 13:57
Location: Chile
Contact:

Re: Create nul and all ascii characters with only batch

#69 Post by carlos » 10 Feb 2014 01:42

About my code, I implemente your trick with =exitcodeascii, and also for check that all .chr files are ready I removed the goto loop system, using a function that uses for /l.

please, check time with this: (in my system it takes round 6 seconds):

Code: Select all

@echo off

:gen255
::generates 0-255 .chr files that have 1 byte
::Teamwork of carlos, penpen, aGerman, dbenham
::Tested under Win8

set "func1=type nul >%%#.tmp"
set "func1=%func1% &makecab /d compress=off"
set "func1=%func1% /d reserveperdatablocksize=26"
set "func1=%func1% /d reserveperfoldersize=%%#"
set "func1=%func1% %%#.tmp %%#.chr >nul"
set "func1=%func1% &type %%#.chr"
set "func1=%func1% |((for /l %%_ in (1 1 38) do pause)>nul"
set "func1=%func1% &findstr ^^^^ >%%#.tmp)"
set "func1=%func1% &copy /y %%#.tmp /a %%#.chr /b >nul"
set "func1=%func1% &del %%#.tmp"
set "func2=cmd /c exit /b %%# &cmd /a /c set /p"
set "func2=%func2% =^^%%=exitcodeascii%%<nul >%%#.chr"
set "func3=for /l %%- in (0) do if "1"=="%%~z_" exit /b"

del /f /q /a *.chr >nul 2>&1
for /l %%# in (0,1,25) do start "" /b /i cmd /d /c "%func1%"
copy /y nul + nul /a 26.chr /a >nul
for /l %%# in (27,1,34) do start "" /b /i cmd /d /c "%func1%"
for /l %%# in (35,1,60) do start "" /b /i cmd /d /c "%func2%"
for %%# in (61) do %func1%
for /l %%# in (62,1,126) do start "" /b /i cmd /d /c "%func2%"
for /l %%# in (127,1,255) do start "" /b /i cmd /d /c "%func1%"

for /l %%# in (0,1,255) do for %%_ in (
%%#.chr) do cmd /q /d /c "%func3%"

goto :eof


foxidrive
Expert
Posts: 6033
Joined: 10 Feb 2012 02:20

Re: Create nul and all ascii characters with only batch

#70 Post by foxidrive » 10 Feb 2014 02:49

Time here for Carlos' version

2.16
2.33
2.31

dbenham
Expert
Posts: 2253
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: Create nul and all ascii characters with only batch

#71 Post by dbenham » 10 Feb 2014 06:36

Hidden .tmp files would do it.

I took Carlos' idea and explicitly check for the existence of each .tmp file that my code creates using a FOR /L loop. I no longer del *.tmp at the top. I also removed the START /B /WAIT from the last process - simpler to use CMD /C directly.

Code: Select all

@echo off
setlocal disableDelayedExpansion

:genAllChr
::This code creates 256 1 byte files, one for each possible byte value.
::This is encoded in a macro way to be called asynchronously with start cmd /c
::Teamwork of carlos, penpen, aGerman, dbenham
::Tested under Win7 and XP
set ^"genchr=(^
for /l %%N in (%%A !cnt! 255) do (^
  if %%N equ 26 (^
    copy /y nul + nul /a 26.chr /a ^>nul^
  ) else (if %%N geq 35 if %%N leq 126 if %%N neq 61 (^
    cmd /c exit %%N^&^
    ^<nul set /p "=!=exitCodeAscii!" ^>%%N.chr^
  ))^&^
  if not exist %%N.chr (^
    makecab /d compress=off /d reserveperdatablocksize=26 /d reserveperfoldersize=%%N %%A.tmp %%N.chr ^>nul^&^
    type %%N.chr ^| ((for /l %%n in (1 1 38) do pause)^>nul^&findstr "^^" ^>%%N.temp)^&^
    ^>nul copy /y %%N.temp /a %%N.chr /b^&^
    del %%N.temp^
  )^
))^&^
del %%A.tmp^"
del /f /q /a *.chr >nul 2>&1
if "%~1" neq "" (set /a cnt=%~1) else set /a cnt=number_of_processors
if %cnt% lss 1 set cnt=1
if %cnt% gtr 256 set cnt=256
set /a "end=cnt-1"
for /l %%A in (0 1 %end%) do (
  type nul >%%A.tmp
  if %%A equ %end% (
    cmd /q /v:on /c "%genchr%"
  ) else (
    start "" /b cmd /q /v:on /c "%genchr%"
  )
)
:genAllChr.check
for /l %%N in (0 1 %end%) do if exist %%N.tmp goto :genAllChr.check
exit /b

I don't see any difference in timing from before.

In production code I would probably create a temporary directory in the %TEMP% folder to hold all the working files. That folder would be guaranteed to be empty at the start. Then I don't even need to delete *.CHR, and I can revert back to a simple IF EXIST *.TMP at the end.


Dave Benham

Squashman
Expert
Posts: 4106
Joined: 23 Dec 2011 13:59

Re: Create nul and all ascii characters with only batch

#72 Post by Squashman » 10 Feb 2014 08:28

foxidrive wrote:Dave's code ends here ok.

I got these timings:

1.01
1.11
1.11

I think someone just wanted to brag that he has a faster computer than all of us. :D

foxidrive
Expert
Posts: 6033
Joined: 10 Feb 2012 02:20

Re: Create nul and all ascii characters with only batch

#73 Post by foxidrive » 10 Feb 2014 15:34

Squashman wrote:I think someone just wanted to brag that he has a faster computer than all of us. :D


*laughs*

I bought it second hand 20 months ago - $600 well spent. :)

einstein1969
Expert
Posts: 763
Joined: 15 Jun 2012 13:16
Location: Italy, Rome

Re: Create nul and all ascii characters with only batch

#74 Post by einstein1969 » 10 Feb 2014 17:22

dbenham wrote:Hidden .tmp files would do it.

I took Carlos' idea and explicitly check for the existence of each .tmp file that my code creates using a FOR /L loop. I no longer del *.tmp at the top. I also removed the START /B /WAIT from the last process - simpler to use CMD /C directly.

Code: Select all

@echo off
setlocal disableDelayedExpansion

:genAllChr
::This code creates 256 1 byte files, one for each possible byte value.
::This is encoded in a macro way to be called asynchronously with start cmd /c
::Teamwork of carlos, penpen, aGerman, dbenham
::Tested under Win7 and XP
set ^"genchr=(^
for /l %%N in (%%A !cnt! 255) do (^
  if %%N equ 26 (^
    copy /y nul + nul /a 26.chr /a ^>nul^
  ) else (if %%N geq 35 if %%N leq 126 if %%N neq 61 (^
    cmd /c exit %%N^&^
    ^<nul set /p "=!=exitCodeAscii!" ^>%%N.chr^
  ))^&^
  if not exist %%N.chr (^
    makecab /d compress=off /d reserveperdatablocksize=26 /d reserveperfoldersize=%%N %%A.tmp %%N.chr ^>nul^&^
    type %%N.chr ^| ((for /l %%n in (1 1 38) do pause)^>nul^&findstr "^^" ^>%%N.temp)^&^
    ^>nul copy /y %%N.temp /a %%N.chr /b^&^
    del %%N.temp^
  )^
))^&^
del %%A.tmp^"
del /f /q /a *.chr >nul 2>&1
if "%~1" neq "" (set /a cnt=%~1) else set /a cnt=number_of_processors
if %cnt% lss 1 set cnt=1
if %cnt% gtr 256 set cnt=256
set /a "end=cnt-1"
for /l %%A in (0 1 %end%) do (
  type nul >%%A.tmp
  if %%A equ %end% (
    cmd /q /v:on /c "%genchr%"
  ) else (
    start "" /b cmd /q /v:on /c "%genchr%"
  )
)
:genAllChr.check
for /l %%N in (0 1 %end%) do if exist %%N.tmp goto :genAllChr.check
exit /b

I don't see any difference in timing from before.

In production code I would probably create a temporary directory in the %TEMP% folder to hold all the working files. That folder would be guaranteed to be empty at the start. Then I don't even need to delete *.CHR, and I can revert back to a simple IF EXIST *.TMP at the end.


Dave Benham


Hi Dave,

I do not have enough experience to include this piece of code in the macro. On my single processor system this reduces by a further 20% of the execution time. It should be possible to adapt it to run on Multicores.

@echo off
set t0=%time%
del /f /q /a *.chr >nul 2>&1
setlocal EnableDelayedExpansion
set "charlist=#$%%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~"
for /L %%N in (0,1,91) do if %%N neq 26 <nul set /p "=!charlist:~%%N,1!" >%%N.chr
endlocal

setlocal disableDelayedExpansion
:genAllChr
::This code creates 256 1 byte files, one for each possible byte value.
::This is encoded in a macro way to be called asynchronously with start cmd /c
::Teamwork of carlos, penpen, aGerman, dbenham
::Tested under Win7 and XP
set ^"genchr=(^
for /l %%N in (%%A !cnt! 255) do (^
if %%N equ 26 (^
copy /y nul + nul /a 26.chr /a ^>nul^
) else (if not exist %%N.chr if %%N geq 35 if %%N leq 126 if %%N neq 61 (^
cmd /c exit %%N^&^
^<nul set /p "=!=exitCodeAscii!" ^>%%N.chr^
))^&^
if not exist %%N.chr (^
makecab /d compress=off /d reserveperdatablocksize=26 /d reserveperfoldersize=%%N %%A.tmp %%N.chr ^>nul^&^
type %%N.chr ^| ((for /l %%n in (1 1 38) do pause)^>nul^&findstr "^^" ^>%%N.temp)^&^
^>nul copy /y %%N.temp /a %%N.chr /b^&^
del %%N.temp^
)^
))^&^
del %%A.tmp^"
if "%~1" neq "" (set /a cnt=%~1) else set /a cnt=number_of_processors
if %cnt% lss 1 set cnt=1
if %cnt% gtr 256 set cnt=256
set /a "end=cnt-1"
for /l %%A in (0 1 %end%) do (
type nul >%%A.tmp
if %%A equ %end% (
cmd /q /v:on /c "%genchr%"
) else (
start "" /b cmd /q /v:on /c "%genchr%"
)
)
echo %t0% - %time%
:genAllChr.check
for /l %%N in (0 1 %end%) do if exist %%N.tmp goto :genAllChr.check
echo %time%
exit /b


EDIT: There is an error on For. Must add +35 in the %%N.chr

einstein1969

carlos
Expert
Posts: 485
Joined: 20 Aug 2010 13:57
Location: Chile
Contact:

Re: Create nul and all ascii characters with only batch

#75 Post by carlos » 10 Feb 2014 23:03

I wrote many variants today, but this is maybe the better for me:

In the macro I copied the if else logic from dbenham:

Code: Select all

@echo off

:gen255
::generates 0-255 .chr files that have 1 byte
::Teamwork of carlos, penpen, aGerman, dbenham
::Tested under Win8

set "genchr=if %%# equ 26 ("
set "genchr=%genchr% copy /y nul + nul /a 26.chr /a >nul ) else ("
set "genchr=%genchr% if %%# geq 35 if %%# leq 126 if %%# neq 61 ("
set "genchr=%genchr% cmd /a /d /q /c exit /b %%# &cmd /a /d /q /c set /p"
set "genchr=%genchr% "=%%=exitcodeascii%%" <nul >%%#.chr"
set "genchr=%genchr% )) &if not exist %%#.chr ("
set "genchr=%genchr% type nul >%%#.tmp"
set "genchr=%genchr% &makecab /d compress=off"
set "genchr=%genchr% /d reserveperdatablocksize=26"
set "genchr=%genchr% /d reserveperfoldersize=%%#"
set "genchr=%genchr% %%#.tmp %%#.chr >nul"
set "genchr=%genchr% &type %%#.chr"
set "genchr=%genchr% |((for /l %%_ in (1 1 38) do pause)>nul"
set "genchr=%genchr% &findstr ^^^^ >%%#.tmp)"
set "genchr=%genchr% &copy /y %%#.tmp /a %%#.chr /b >nul"
set "genchr=%genchr% &del %%#.tmp)"

for /l %%# in (0,1,255) do (
if exist %%#.chr del /f /q /a %%#.chr >nul
if exist %%#.tmp del /f /q /a %%#.tmp >nul
)

for /l %%# in (0,1,255) do start "" /b /i cmd /a /d /q /c "%genchr%"

:gen255chk
for /l %%# in (0 1 255) do for %%_ in (
%%#.chr) do if not "1"=="%%~z_" goto :gen255chk

goto :eof


Post Reply