Using Alternate Data Streams (in hybrid scripts)

[Aacini]

An Alternate Data Stream (ADS) is an additional data associated to a file that is stored in a place different than the original file data. The Win-API documentation indicate:

When specified from the Windows shell command line, the full name of a stream is "filename:stream name:stream type", as in the following example: "myfile.dat:stream1:$DATA".

Where the ":$DATA" part may be omitted. A file may have several streams. Working with ADS's in the command line is very simple, as shown below:

• To create an additional stream in a file, just use the apropriate stream name in an output redirection:
  

  Code: Select all

  echo This is Textfile.txt > Textfile.txt
  echo This is Textfile.txt:Stream > Textfile.txt:Stream
  

• Conversely, you must use an input redirection to access the alternate stream:
  

  Code: Select all

  type Textfile.txt
  more < Textfile.txt:Stream
  

• The existing ADS's of a file may be displayed via /R switch of DIR command (from Windows Vista and above).
• There is no native way to delete streams from a file. You may download Sysinternals Streams utility to do so (although not on individual basis), and to list ADS's in Windows before Vista.

When I knew about Alternate Data Streams some time ago I completed some tests, and after that I considered this feature just a curiosity until I discovered the keypoint that becomes ADS's much more useable:

ALWAYS INCLUDE AN APROPRIATE EXTENSION IN THE STREAM NAME

Previous point makes the stream recognizable by several programs and commands, including the Windows Notepad editor and WSH's support cscript.exe command! You may edit the contents of an alternate stream with the following command:

Code: Select all

notepad fileName:streamName.ext

Because Alternate Data Streams is a not widely known feature they may also be used to protect/hide file names, at least from the less experienced users. For example:

Code: Select all

echo Something > .\C:filename.txt

Previous line send data to alternate stream "filename.txt" of file "C" (with no extension) in current directory! If you want to make stream names inaccessible, you may also make good use of the following feature:

You may use... these characters for file names, except for the following:

Characters whose integer representations are in the range from 1 through 31, except for alternate data streams where these characters are allowed.

That is, control characters are allowed in stream names! The Batch file below is an example of this point that adds two alternate streams to itself: the first one is named: "HelloX"+BS+"World1.txt" and the second one is: "HelloXY"+BS+BS+"World2.txt". I think these names are difficult enough to be broken even for experienced users!

Code: Select all

@echo off
setlocal EnableDelayedExpansion
rem ADSnames.bat: Create ADS's with names hard to break

for /F %%a in ('echo prompt $H ^| cmd') do set "BS=%%a"
echo Special ADS created on %date% @ %time% > "%0:HelloX!BS!World1.txt"
echo The second ADS with special name > "%0:HelloXY!BS!!BS!World2.txt"
cls
echo dir /R "%0"
     dir /R "%0"
echo/
echo more ^< "%0:HelloX!BS!World1.txt"
     more  < "%0:HelloX!BS!World1.txt"
echo/
echo more ^< "%0:HelloXY!BS!!BS!World2.txt"
     more  < "%0:HelloXY!BS!!BS!World2.txt"
echo/
echo Could you repeat previous commands?

An application very well suited to Alternate Data Streams is a Batch-file based hybrid script that include source code for other languages. If the additional language have provisions to achieve this mix (like JScript), the script is very simple; otherwise the hybridization requires convoluted tricks as described in Hybrids and chimeras in cmd/bat thread.

This way, an hybrid script may be created with a base Batch file and several streams comprised of JScript, VBS, PowerShell, or whichever parts you want that are "pure files" each, so they don't require any trick in order to be executed individually. The small example below show how simple is to use this capability:

Code: Select all

C:\ dir /R Base.bat
 El volumen de la unidad C no tiene etiqueta.
 El número de serie del volumen es: 0895-160E

 Directorio de C:\Users\Antonio\Documents\ASMB\Modern Batch File Programming\Alternate Data Streams

15/01/2015  11:06 p. m.               194 Base.bat
                                       47 Base.bat:JScript.js:$DATA
                                       46 Base.bat:PowerShell.ps1:$DATA
                                       41 Base.bat:VBS.vbs:$DATA
               1 archivos            194 bytes
               0 dirs  424,516,263,936 bytes libres

C:\ type Base.bat
@echo off
echo Produced from Batch
CScript //nologo "%~F0:JScript.js"
CScript //nologo "%~F0:VBS.vbs"
PowerShell -NoProfile -Command "Get-Content '%~F0:PowerShell.ps1' | Invoke-Expression"

C:\ more < Base.bat:JScript.js
WScript.Echo("Produced from JScript in ADS");

C:\ more < Base.bat:VBS.vbs
WScript.Echo "Produced from VBS in ADS"

C:\ more < Base.bat:PowerShell.ps1
Write-Host "Produced from PowerShell in ADS"

C:\ Base.bat
Produced from Batch
Produced from JScript in ADS
Produced from VBS in ADS
Produced from PowerShell in ADS

Previous example deserve some explanations:

• Both the JScript and VBS engines used by cscript.exe correctly open the given filename, even if it includes a stream name, and takes the last part as the extension that indicate the type of file. However:
• The base cscript.exe command does NOT do the same thing when a stream name ends in .WSF like in "cscript //nologo Base.bat:WSF.wsf"; in this case, "File not found" error is issued. If ":$DATA" part is added, like in "cscript //nologo Base.bat:WSF.wsf:$DATA", then the error change to "There is not script engine for extension .wsf:$DATA". I did several tests and can not found a method to execute a .wsf stream.
• PowerShell documentation specify that version 3.0 "manage streams", so I think at first that "PowerShell -NoProfile -ExecutionPolicy Bypass -File Base.bat:PowerShell.ps1" would run the .ps1 stream, but "Invalid format in path" error was issued. I read the documentation and discovered that streams are managed in just 6 cmdlets being Get-Content one of them (via -Stream parameter). This point requires to use "-Command Invoke-Expression" instead of "-File", so the right way is this: "PowerShell -NoProfile -Command "Get-Content -Path Base.bat -Stream PowerShell.ps1 | Invoke-Expression". However, I discovered that you may include the stream in the filename in the standard way as long as the filename include the drive, like in "PowerShell -NoProfile -Command "Get-Content C:Base.bat:PowerShell.ps1 | Invoke-Expression".
• I did not tested streams with other languages/applications, like the MSHTA or Perl, but I would like to know in which ones this feature works!

Unfortunately, ADS is a feature of Windows NTFS disks, so the Alternate Data Streams are ignored when the file is copied or moved to another file system without ADS support, attached to an e-mail, or uploaded to a website. However, it is very easy to write a Batch file that create the base file and all its alternate streams, similar to an "installer". The example below create the Base.bat file of previous example and all its ADS's using a trick similar to Unix's heredoc (http://stackoverflow.com/questions/1015163/heredoc-for-windows-batch/23051423#23051423), so it may be taken as base to create your own files.

Code: Select all

@echo off

rem MakeBase.bat: Example on using Alternate Data Streams to create an hybrid script
rem Antonio Perez Ayala

rem This program create Base.bat file with JScript.js, VBS.vbs and PowerShell.ps1 ADS's

rem Definition of heredoc macro
setlocal DisableDelayedExpansion
set LF=^


::Above 2 blank lines are required - do not remove
set ^"\n=^^^%LF%%LF%^%LF%%LF%^^"
set heredoc=for %%n in (1 2) do if %%n==2 (%\n%
       call :heredoc !argv!%\n%
       goto !argv!%\n%
    ) else set argv=

setlocal EnableDelayedExpansion
set "filename="


rem Definition of Base.bat file and its alternate data streams

%heredoc% :Base.bat
@echo off
echo Produced from Batch
CScript //nologo "%~F0:JScript.js"
CScript //nologo "%~F0:VBS.vbs"
PowerShell -NoProfile -Command "Get-Content '%~F0:PowerShell.ps1' | Invoke-Expression"
 :Base.bat

%heredoc% :JScript.js
WScript.Echo("Produced from JScript in ADS");
 :JScript.js

%heredoc% :VBS.vbs
WScript.Echo "Produced from VBS in ADS"
 :VBS.vbs

%heredoc% :PowerShell.ps1
Write-Host "Produced from PowerShell in ADS"
 :PowerShell.ps1

echo Base.bat file created:
dir /R Base.bat
goto :EOF


rem Definition of heredoc subroutine

:heredoc ID
if not defined filename (
   set "filename=%1"
   set "filename=!filename:~1!"
   set "output=!filename!"
) else (
   set "output=%filename%%1"
)
set "skip="
for /F "delims=:" %%a in ('findstr /N /C:" %1" "%~F0"') do (
   if not defined skip (set skip=%%a) else set /A lines=%%a-skip-1
)
(for /F "skip=%skip% delims=" %%a in ('findstr /N "^" "%~F0"') do (
   set "line=%%a"
   echo(!line:*:=!
   set /A lines-=1
   if !lines! == 0 exit /B
)) > "%output%"
exit /B

I found this feature extremely useful to create JScript/VBS hybrid scripts and other applications where a base file have several companion files. You are invited to test this feature and report any additional use you may devise for it.

Antonio

[Liviu]

That is, control characters are allowed in stream names!

Interesting, thanks for the pointer.

An application very well suited to Alternate Data Streams is a Batch-file based hybrid script that include source code for other languages.

I dabbled in ADS for scripting some time ago (viewtopic.php?p=20360#p20360, http://ss64.org/viewtopic.php?pid=6626#p6626) and it is indeed a useful technique in certain scenarios. However, the limitations for wider usage are its dependency on NTFS (as you noted), plus the fact that ADSs may not be not copied by some backup programs, and not saved by most archivers.

Liviu

[foxidrive]

plus the fact that ADSs may not be not copied by some backup programs, and not saved by most archivers.

I think they are destroyed by some disk management utilities too - so I learned to treat them as a curiosity, but are unreliable.

Antonio's work may be very useful for some people, I'm not dissing it, just replying to Liviu's point that they aren't all that robust.

[Aacini]

I dabbled in ADS for scripting some time ago (viewtopic.php?p=20360#p20360, http://ss64.org/viewtopic.php?pid=6626#p6626) and it is indeed a useful technique in certain scenarios.

You are right, but those posts are lost in the middle of different topic threads... You may see this one as a compilation of such techniques.

However, the limitations for wider usage are its dependency on NTFS (as you noted), plus the fact that ADSs may not be not copied by some backup programs, and not saved by most archivers.

Liviu

Well, FAT32 disks are the less in modern Windows computers and they trend to disappear, and the problem of ADS's not being copied only happen when the file is backed up, or sent via email, or posted in a web site. However, it is very easy to extract ADS's into individual parts in order to back up or post them and later re-create the original file; below there is a Batch file that aid in this operation.

On the other hand the simplification that ADS's provides to hybrid scripts, and other applications that may make good use of this technique, makes the use of ADS's well worth it. There are several applications that I would be written in a clearer way if I would knew about this technique before!

Code: Select all

@echo off
setlocal EnableDelayedExpansion

rem StreamsToGo.bat: Generate a Batch file that re-create all the streams of a given file
rem Antonio Perez Ayala

if "%~1" neq "" if "%~1" neq "/?" goto begin
echo Create a Batch file that includes all the streams of a given file
echo/
echo StreamsToGo filename.ext
echo/
echo This program generate a Batch file named Make^<filename^>.bat that will
echo re-create the original filename.ext file with all its Alternate Data Streams,
echo so this file may be send via email or posted in a web site. See:
echo http://www.dostips.com/forum/viewtopic.php?f=3^&t=6185
echo/
echo If the original file is a Batch .bat file and you rename the generated
echo Make^<filename^>.bat file with the original filename.bat name, then after the
echo original file was re-created the "Make" file will be deleted. This way, the
echo first execution of such file serve as an auto-installer for the application.
goto :EOF

:begin
if not exist %1 echo File not found: %1 & goto :EOF

set "myself=%~F0"
set /P "=Packing streams. " < NUL
(
   echo @set "filename=%~NX1"
   call :getResource MakeHeader.bat
   echo ^<resource id="%~NX1"^>
   type "%~NX1"
   echo ^</resource^>
   echo/
   for /F "delims=" %%a in ('dir /R %1 ^| findstr /E ":$DATA"') do (
      set "stream=%%a"
      set "stream=!stream:*%~NX1=%~NX1!"
      set "stream=!stream:~0,-6!"
      echo ^<resource id="!stream!"^>
      more < "!stream!"
      echo ^</resource^>
      echo/
      set /P "=. " > CON < NUL
   )
)  > "Make%~N1.bat"
echo/
echo File "Make%~N1.bat" created
goto :EOF


<resource id="MakeHeader.bat">
@echo off
setlocal EnableDelayedExpansion

rem Installer program that create a file and its Alternate Data Streams
rem This program was generated via StreamsToGo.bat file written by Antonio Perez Ayala
rem http://www.dostips.com/forum/viewtopic.php?f=3^&t=6185

cd /D "%~DP0"
set "myself=%~NX0"
if "%myself%" equ "%filename%" (
   del "_ _.bat" 2> NUL
   ren "%myself%" "_ _.bat"
   "_ _.bat"
)
set /P "=Unpacking streams" < NUL
for /F "tokens=1,3,4 delims=:=>" %%a in ('findstr /N "^<resource" "%myself%"') do (
   if "%%~c" equ "" (set "stream=%%b") else set "stream=%%b:%%c"
   call :getResource !stream! > !stream!
   set /P "=. " < NUL
)
echo/
echo File "%filename%" created
if "%myself%" equ "_ _.bat" (
   del "%myself%"
   "%filename%"
)
goto :EOF


rem Extract a file:stream placed in a "resource" in this .bat file

:getResource resourceId
setlocal EnableDelayedExpansion
set "start="
set "end="
for /F "tokens=1,3,4 delims=:=>" %%a in ('findstr /N "^</*resource" "%myself%"') do (
   if not defined start (
      if "%%~c" equ "" (set "stream=%%b") else set "stream=%%b:%%c"
      if "%~1" equ !stream! set "start=%%a"
   ) else (
      if not defined end set "end=%%a"
   )
)
setlocal DisableDelayedExpansion
for /F "skip=%start% tokens=1* delims=[]" %%a in ('find /N /V "" ^< "%myself%"') do (
   if %%a equ %end% goto :EOF
   echo(%%b
)
:EOF

</resource>

Antonio

[Aacini]

Below there is an example of an useful application that uses ADSs. It allows to manage several versions of a file in a very simple way!

Code: Select all

@echo off
setlocal

rem Version.bat: Preserve/restore previous versions of any file using ADSs
rem Antonio Perez Ayala

if "%~1" neq "" if "%~1" neq "/?" goto begin

echo Preserve/restore previous versions of any file
echo/
echo Version filename ["Note to store with new version"] [/R #]
echo/
echo Each time you include a Note, the file is stored as a new version. For example:
echo/
echo     Version example.bat "Working version before implement the new feature"
echo/
echo Use /R switch to restore a previous version, e.g.: Version example.bat /R 4
echo/
echo If no Note nor /R switch is given, the saved versions are listed.
echo/
goto :EOF

:begin
if not exist "%~1" echo File not found: %1 & goto :EOF
set lastVer=0
for /F "delims=:" %%a in ('^(more ^< "%~1:0"^) 2^>NUL') do set "lastVer=%%a"
if "%~2" equ "" goto List
if /I %2 equ /R goto Restore

:Save new version
set /A lastVer+=1
type "%~1" > "%~1:%lastVer%"
for /F "delims=." %%t in ("%time%") do echo %lastVer%:  %date% %%t   %~2>> "%~1:0"
echo New version %lastVer% saved
goto :EOF

:Restore previous version
if "%~3" equ "" echo ERROR: No version given & goto :EOF
if %lastVer% equ 0 echo No previous versions saved & goto :EOF
findstr "^%~3:" < "%~1:0" > NUL
if errorlevel 1 echo No version %3 exists & goto :EOF
more < "%~1:%3" > "%~1.$$$"
more < "%~1:0" > "%~1.$$$:0"
for /F "delims=:" %%a in ('more ^< "%~1:0"') do more < "%~1:%%a" > "%~1.$$$:%%a"
del "%~1"
ren "%~1.$$$" "%~1"
echo Previous version %3 restored
goto :EOF

:List saved versions
if %lastVer% equ 0 (
   echo No previous versions saved
) else (
   more < "%~1:0"
)
goto :EOF

Antonio