Create nul and all ascii characters with only batch

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Message
Author
dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: createnul.cmd Create a file with the nul character

#16 Post by dbenham » 27 Jan 2014 20:17

Awesome stuff :!: 8) And great teamwork :!:

I never dreamed there would be such an elegant hack that would work even on native XP. Lots of clever techniques all packed into one solution.


Dave Benham

carlos
Expert
Posts: 503
Joined: 20 Aug 2010 13:57
Location: Chile
Contact:

Re: createnul.cmd Create a file with the nul character

#17 Post by carlos » 27 Jan 2014 20:28

@dbenham.
Is posible adapt charlib for create all the ascii characters in 256 files.
Nul is ready.
What are the most problematic characters that are corrupted on posted?

If we create the 256 characters in a file, we can create binary files using only batch.

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: createnul.cmd Create a file with the nul character

#18 Post by dbenham » 27 Jan 2014 22:48

@Carlos - Yes, I was thinking about that.

The existing CHARLIB.BAT was intended to work with variables, and of course that will not work with null bytes. So when dealing with null bytes it could only be used to write to a file.

Most byte characters could be written using SET /P, but there is the leading equal/quote/white space issue. Those problem characters would require jeb's temp file <Ctrl-Z> technique.

One thing I don't like about CHARLIB.BAT is many of the characters are hard coded within the file. In particular, I don't like the <tab> character hard coded. My text editor is frequently configured to convert tabs into spaces, and also tabs often do not post well to bulletin boards. Unfortunately, no one figured out a universal way to generate <tab> with batch. Also, any of the extended ASCII characters (byte code greater than 127) can be a problem because you have to worry about the active code pages.

Because of those issues, CHARLIB.BAT must basically be treated as a binary file, not typical for batch scripts :(

Given that CHARLIB.BAT is already binary, I don't see any reason why we couldn't also embed the null byte as well.

As long as all lines are terminiated by <carriage return><line feed>, then simply append a single null byte after the last line. The null byte does not interfere with the operation of the script. FINDSTR can be used to write out the null byte using:

Code: Select all

findstr /v $ "%~f0" >outputFile

I think I saw the above technique on StackOverflow when following the links associated with the recent DosTips activity dealing with batch binaries.


Dave Benham

foxidrive
Expert
Posts: 6031
Joined: 10 Feb 2012 02:20

Re: createnul.cmd Create a file with the nul character

#19 Post by foxidrive » 28 Jan 2014 01:26

dbenham wrote:no one figured out a universal way to generate <tab> with batch.


This works in Vista/Win7/Win 8.1

The forfiles that I downloaded in XP days doesn't have the same syntax as Vista and later version so I added a kludge for XP.

Code: Select all

@echo off
set "t=~.bat"
type nul>"%temp%\%t%"
set "space= "
ver | find /i "XP" >nul && set "space="
forfiles -m%space%"%t%" -p%space%"%temp%" -c%space%"cmd /c set /p =set tab=0x09<nul">"%temp%\%t%"
call "%temp%\%t%"
del "%temp%\%t%"
echo "%tab%"
pause

carlos
Expert
Posts: 503
Joined: 20 Aug 2010 13:57
Location: Chile
Contact:

Re: createnul.cmd Create a file with the nul character

#20 Post by carlos » 28 Jan 2014 02:04

I have this stuff code. It create the files for: 0, 8, 10, 13, 26, 27 ascii characters:
I working for create more characters.

Code: Select all

@Echo Off
Set "foo=%Temp%\foo.foo"



Call :0
Call :8
Call :10
Call :13
Call :26
Call :27

Goto :Eof


:0
::This routine create the file 0.chr that only have the nul character
::Tested on windows 8, winxp 32 home, win xp 32 prof, windows 7 32bit
::Authors: carlos, aGerman, penpen
Cmd /U /C Set /P "=a" <Nul > 0.chr
Copy /Y 0.chr+Nul 0.chr >Nul
Type 0.chr |(Pause>Nul &Findstr "^") > "%foo%"
Copy /Y "%foo%" /A 0.chr /B >Nul
Goto :Eof

:8
For /F %%# In (
'"Prompt $H &For %%_ In (_) Do Rem"') Do Set /P "=%%#" <Nul >8.chr
Goto :Eof

:10
Echo(|(Pause >Nul &Findstr "^" >10.chr)
Goto :Eof

:13
Call :26
For /F %%# In (26.chr) Do Set "sub=%%#"
Echo(>"%foo%"
For /F %%# In ('Copy /Z "%foo%" Nul') Do Echo(%%#%sub%>"%foo%"
Copy /Y "%foo%" /A 13.chr /B >Nul
Goto :Eof

:26
Copy /Y Nul+Nul /A 26.chr /A >Nul
Goto :Eof

:27
For /F %%# In (
'"Prompt $E &For %%_ In (_) Do Rem"') Do Set /P "=%%#" <Nul >27.chr
Goto :Eof



penpen
Expert
Posts: 1991
Joined: 23 Jun 2013 06:15
Location: Germany

Re: createnul.cmd Create a file with the nul character

#21 Post by penpen » 28 Jan 2014 04:31

Actually i'm posting by my mobile phone so i can't test it, but couldn't we use the makecab.exe:
I think i remember it contains its own filesize somewhere in the header (in little endian).
So creating a non compressed cab with the right filesize (and name) should serve us the needed hex codes.
A filesize of 0x1AFF should serve us a header entry of "FF1A" so we could do the same trick as above to extract this hex FFchar.
So we need to create cab filesizes of 0x1A01 to 0x1AFF.

penpen

Edit: If i'm wrong with the cab filesize, then the filesize of the (un)compressed must be present somewhere in the cab, same trick, but not as near to the start of the file.

foxidrive
Expert
Posts: 6031
Joined: 10 Feb 2012 02:20

Re: createnul.cmd Create a file with the nul character

#22 Post by foxidrive » 28 Jan 2014 04:42

The same technique I posted earlier will also work for an FF character.

It may work for most 256 characters

Code: Select all

@echo off
set "t=~.bat"
type nul>"%temp%\%t%"
set "space= "
ver | find /i "XP" >nul && set "space="
forfiles -m%space%"%t%" -p%space%"%temp%" -c%space%"cmd /c set /p =set FF=0xff<nul">"%temp%\%t%"
call "%temp%\%t%"
del "%temp%\%t%"
echo "%FF%"
pause

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: createnul.cmd Create a file with the nul character

#23 Post by aGerman » 28 Jan 2014 12:25

penpen wrote:I think i remember it contains its own filesize somewhere in the header (in little endian).

Good idea. It can be found at the 9th and 10th Byte. The rest is a bit tricky though. Also the file name belongs to the header and influences the size of the cab file.

Regards
aGerman

EDIT:
tab character:

Code: Select all

@echo off &setlocal
>"t.tmp" type nul
for /l %%i in (1 1 169) do >>t.tmp <nul set /p "=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
>nul makecab /d compress=off "t.tmp" "t.chr"
type "t.chr" | >"t.tmp" (pause>nul&pause>nul&pause>nul&pause>nul&pause>nul&pause>nul&pause>nul&pause>nul&findstr "^")
>nul copy /y "t.tmp" /a "t.chr" /b
del "t.tmp"

As soon as you change the name t.tmp to a different length you also have to change the number of characters written to t.tmp.

EDIT2: Changed t.chr to t.tmp because I confused myself :lol:

Squashman
Expert
Posts: 4465
Joined: 23 Dec 2011 13:59

Re: createnul.cmd Create a file with the nul character

#24 Post by Squashman » 28 Jan 2014 12:36

foxidrive wrote:The forfiles that I downloaded in XP days doesn't have the same syntax as Vista and later version so I added a kludge for XP.

Gotta make sure you get the one that 2003 server uses. Don't use the one for NT/2000 which uses the hyphen syntax. I posted a link to download it a few months ago.

carlos
Expert
Posts: 503
Joined: 20 Aug 2010 13:57
Location: Chile
Contact:

Re: createnul.cmd Create a file with the nul character

#25 Post by carlos » 28 Jan 2014 13:56

penpen wrote:Actually i'm posting by my mobile phone so i can't test it, but couldn't we use the makecab.exe:
I think i remember it contains its own filesize somewhere in the header (in little endian).
So creating a non compressed cab with the right filesize (and name) should serve us the needed hex codes.
A filesize of 0x1AFF should serve us a header entry of "FF1A" so we could do the same trick as above to extract this hex FFchar.
So we need to create cab filesizes of 0x1A01 to 0x1AFF.

penpen

Edit: If i'm wrong with the cab filesize, then the filesize of the (un)compressed must be present somewhere in the cab, same trick, but not as near to the start of the file.


Nice idea penpen.
Without talk about the cabinet format:
If you do this:

Code: Select all

Makecab /D Compress=OFF file.dat file.cab


int this case the file.cab should have the uncompressed size in offset 44: the size as you say: Little endian 4 bytes.
For get the 1A at right we need a file as base of 6656 bytes: this is 0x1A00 and in Little endian: 00 1A

In simple idea:
create a file with 6656 bytes.
Skip 44 bytes and now we have the ...

But there are a litte problem. And we need two strategies.
Because in cabinet also file offset 8 have 4 bytes with the size of the cabinet.
The minimal cabinet size that generate:

Code: Select all

Makecab /D Compress=OFF a file.cab

with a empty file with a length filename of 1, ex: a
is 62 bytes.
then if we create a cabinet of file with 6656 bytes with a length filename of 1, the cabinet size would be 6718 bytes, in Little endian 3E 1A
and we have this 1A in offset 9 stopping our work.

Also, cutoff the cabinet in the 1A is needed for avoid a findstr error message "too large line".

I working on this.

miskox
Posts: 553
Joined: 28 Jun 2010 03:46

Re: createnul.cmd Create a file with the nul character

#26 Post by miskox » 28 Jan 2014 14:33

I have been following this thread from the beginning. I am speechless! You have so many ideas and knowledge.

And now to the point:
- aGerman's solution to get TAB does not work for me (XP PRO)
- here is my code (modified original Carlo's work):

Code: Select all

@Echo Off

REM This will create a file with the size of 6665 bytes.
>a.a (echo AAA)
for /L %%f in (1,1,2220) do >>a.a (echo A)

Set "p=Pause>Nul"
Set "q=%p%&%p%&%p%&%p%&%p%&%p%&%p%&%p%&%p%&%p%&%p%"
Set "s=%q%%q%%q%%q%&%p%&%p%&%p%"

Makecab /D Compress=OFF a.a a.cab >Nul
Type a.cab |(%s%&Findstr "^") >wnul.tmp

Copy /Y wnul.tmp /A tab.chr /B >Nul
Del a.cab a.a wnul.tmp

Rem tab.chr has the TAB character



Saso

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: createnul.cmd Create a file with the nul character

#27 Post by aGerman » 28 Jan 2014 15:02

miskox wrote:- aGerman's solution to get TAB does not work for me (XP PRO)

I'm wondering why :? Could you tell me what values the 9th and 10th Bytes in t.chr are if you write a pause after the makecab line?

miskox wrote:- here is my code (modified original Carlo's work):

Works for me even if it seems to be buggy. The %q%'s are not concatenated via &. The same in carlos' initial post. Maybe someone could explain what I miss here.

Regards
aGerman

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: createnul.cmd Create a file with the nul character

#28 Post by aGerman » 28 Jan 2014 15:09

carlos wrote:then if we create a cabinet of file with 6656 bytes with a length filename of 1, the cabinet size would be 6718 bytes, in Little endian 3E 1A
and we have this 1A in offset 9 stopping our work.

Also, cutoff the cabinet in the 1A is needed for avoid a findstr error message "too large line".

Hmm, won't that work for you:
EDIT Doesn't work under XP, please use http://www.dostips.com/forum/viewtopic.php?p=32169#p32169

Code: Select all

@echo off &setlocal EnableDelayedExpansion
md "characters"
pushd "characters"

>"t.tmp" <nul set /p "=a"
for /l %%i in (1 1 140) do >>t.tmp <nul set /p "=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"

set "strMap=0123456789ABCDEF"
for /l %%i in (0 1 255) do (
  set "byte=%%i" &set "strHex="
  for /l %%j in (1 1 2) do (
    set /a "x = byte & 15, byte >>= 4"
    for %%k in (!x!) do set "strHex=!strMap:~%%k,1!!strHex!"
  )
  >>t.tmp <nul set /p "=a"
  >nul makecab /d compress=off "t.tmp" "!strHex!.chr"
  type "!strHex!.chr" | >"temp.tmp" (pause>nul&pause>nul&pause>nul&pause>nul&pause>nul&pause>nul&pause>nul&pause>nul&findstr "^")
  >nul copy /y "temp.tmp" /a "!strHex!.chr" /b
)

REM carlos' bug fix for the SUB character
Copy /Y Nul+Nul /A 1A.chr /A >Nul

del "t.tmp" "temp.tmp"
popd
pause

Of course the SUB itself can't be created that way (and I did'n look into each file if the right byte was written) but it was running without any error.

Regards
aGerman

EDIT: SUB fixed.
EDIT2: Hex file names instead of decimal.

carlos
Expert
Posts: 503
Joined: 20 Aug 2010 13:57
Location: Chile
Contact:

Re: createnul.cmd Create a file with the nul character

#29 Post by carlos » 28 Jan 2014 15:45

Nice code aGerman.
Please add this fix for the sub chr:

Code: Select all

Rem Fix for 26.chr
Copy /Y Nul+Nul /A 026.chr /A >Nul


I would check if all the .chr generated with yout code are right.
I write a code using a cabinet with two files inside, but currently my code have problem with the 178 character.

Edit: All the characters generated with aGerman code are right, except 026.chr (it need the fix). I check it with a c program.

Now, we can write binary data using batch.
The hexdump of dbenham output in hexadecimal.
aGerman, please you can create the character on hexadecimal notation with a width of 2, example: 0F.chr

Now, is only time for write a tool that write binary using only batch.

This is a great advance in the batch.

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: createnul.cmd Create a file with the nul character

#30 Post by aGerman » 28 Jan 2014 16:03

Fixed :wink: Nice teamwork indeed :D

Since the tab creation didn't work for Saso I'm really curious to know if that code would work under XP.

Regards
aGerman

aGerman, please you can create the character on hexadecimal notation with a width of 2, example: 0F.chr

Yes I can. I'll edit the code above as soon as I finished it.
EDIT: OK, done :wink:

Post Reply