cmd.exe chars needing quotes, and escaping redirection handles

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Post Reply
Message
Author
dbenham
Expert
Posts: 2332
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

cmd.exe chars needing quotes, and escaping redirection handles

#1 Post by dbenham » 08 Jan 2020 16:03

Below are some newly discovered cmd.exe facts (at least for me) that likely no one cares about

1) Special characters listed in cmd help

The following statement at the end of the cmd help (HELP CMD or CMD /?) used to really irk me.
cmd /? wrote: The special characters that require quotes are:
<space>
&()[]{}^=;!'+,`~
I used to think "That is simply wrong! Characters like [ ] { and } are not special. How can Microsoft be so wrong?"

But then I read more carefully, and realized that the statement is related to file/directory completion via Ctrl-F and Ctrl-D :roll:
cmd /? wrote: If completion is enabled with the /F:ON switch, the two control
characters used are Ctrl-D for directory name completion and Ctrl-F for
file name completion. To disable a particular completion character in
the registry, use the value for space (0x20) as it is not a valid
control character.

Completion is invoked when you type either of the two control
characters. The completion function takes the path string to the left
of the cursor appends a wild card character to it if none is already
present and builds up a list of paths that match. It then displays the
first matching path. If no paths match, it just beeps and leaves the
display alone. Thereafter, repeated pressing of the same control
character will cycle through the list of matching paths. Pressing the
Shift key with the control character will move through the list
backwards. If you edit the line in any way and press the control
character again, the saved list of matching paths is discarded and a new
one generated. The same occurs if you switch between file and directory
name completion. The only difference between the two control characters
is the file completion character matches both file and directory names,
while the directory completion character only matches directory names.
If file completion is used on any of the built in directory commands
(CD, MD or RD) then directory completion is assumed.

The completion code deals correctly with file names that contain spaces
or other special characters by placing quotes around the matching path.
Also, if you back up, then invoke completion from within a line, the
text to the right of the cursor at the point completion was invoked is
discarded.

The special characters that require quotes are:
<space>
&()[]{}^=;!'+,`~
If file or directory completion is used, then any file/directory name that includes any of the special characters will be quoted.

But of course Microsoft still is not quite correct - they forgot to list % as one of the special characters that requires quoting.

2) Escaping redirection file handles

It is fairly common knowledge within this community what will happen with the following:

Code: Select all

echo We will meet at 2>output.txt
A naive user might expect the statement "We will meet at 2" to be written to the file "output.txt". But the savvy user knows that "2" will be interpreted as the stderr file handle, so "We will meet at " will be sent to stdout (normally the console), and "output.txt" will be empty because there was no error output.

It is also fairly common knowledge that the solution is simple, move the redirection to the front (or anywhere where there is no chance of ambiguity)

Code: Select all

>output.txt echo We will meet at 2
Not sure how I got there, but I was reading a Raymond Chen blog Beware of digits before the redirection operator, and I learned that another option is to simply escape the "2" :shock:

Code: Select all

echo We will meet at ^2>output.txt
It makes sense, but I had never thought to try it. I added one word to the Phase 2 redirection section of the Parsing Rules on StackOverflow to explicitly account for the behavior.

3) It is impossible to redirect to a file with a name beginning with &

Hopefully we all know what happens with the following:

Code: Select all

echo Hello world >&2
The & introduces an already defined file handle on the right side of redirection, so the statement "Hello world " will be written to stderr (typically the console). After the revelation in point 2) above, I became interested in how to escape the & so that the output is written to a file named "&2" instead. I found it is impossible :!:

None of the following work - they all write the output to stderr instead of a file.

Code: Select all

echo Hello world >^&2
echo Hello world >"&2"

set "file=&2"
setlocal enableDelayedExpansion
echo Hello world >"!file!"
Given that the first two attempts failed, I'm not surprised the last one failed as well, considering that the redirection occurs in Phase 5.5, after the delayed expansion in Phase 5.

It doesn't even work if there is no digit after the &

Code: Select all

echo Hello world >"&test.txt"
-- OUTPUT --

Code: Select all

The handle could not be duplicated
during redirection of handle 1.
I suppose it might be possible to redirect to a file name beginning with & via a link, as in MKLINK "linkFile.txt" "&2". But I can't test on the machine I am currently working at because I don't have the required privilege.

I know jeb makes it a habit of disproving absolute claims like this, but I am pretty confident it is not possible to redirect to a file name beginning with & using standard cmd.exe commands (disregarding the possible MKLINK solution). Challenge on :!: :twisted:

Another interesting find - all text after &2 is simply ignored up until the first unquoted token delimiter.

Code: Select all

echo Hello world >"&2 this is ignored"and_so_is_this  but_this_is_not_ignored
--OUTPUT to stderr--

Code: Select all

Hello world  but_this_is_not_ignored

jeb
Expert
Posts: 943
Joined: 30 Aug 2007 08:05
Location: Germany, Bochum

Re: cmd.exe chars needing quotes, and escaping redirection handles

#2 Post by jeb » 09 Jan 2020 00:02

Hi Dave,

interessting.
dbenham wrote:
08 Jan 2020 16:03
1) Special characters listed in cmd help
...
If completion is enabled with the /F:ON switch, the two control
characters used are Ctrl-D for directory name completion and Ctrl-F for
file name completion.
:o Dind't know this, nor read this before.
But CTRl-F doesn't work for me on win10, it opens a "Find dialog".
Tested on Win7: Both CTRL-D and CTRL-F work as described.
dbenham wrote:
08 Jan 2020 16:03
2) Escaping redirection file handles
...
Not sure how I got there, but I was reading a Raymond Chen blog Beware of digits before the redirection operator, and I learned that another option is to simply escape the "2"
I read this somewhere on stackoverflow, but couldn't find it anymore (searching for ^2> doesn't work)

dbenham wrote:
08 Jan 2020 16:03
3) It is impossible to redirect to a file with a name beginning with &
...
I know jeb makes it a habit of disproving absolute claims like this, but I am pretty confident it is not possible to redirect to a file name beginning with & using standard cmd.exe commands (disregarding the possible MKLINK solution). Challenge on
I didn't check the MKLINK solution, because it was to easy 8)

Code: Select all

echo Hello > ".\&works"
jeb

dbenham
Expert
Posts: 2332
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: cmd.exe chars needing quotes, and escaping redirection handles

#3 Post by dbenham » 09 Jan 2020 06:51

jeb wrote:
09 Jan 2020 00:02
dbenham wrote:
08 Jan 2020 16:03
1) Special characters listed in cmd help
...
If completion is enabled with the /F:ON switch, the two control
characters used are Ctrl-D for directory name completion and Ctrl-F for
file name completion.
:o Dind't know this, nor read this before.
But CTRl-F doesn't work for me on win10, it opens a "Find dialog".
Tested on Win7: Both CTRL-D and CTRL-F work as described.
It works on Win10 if you disable the Ctrl key shortcuts within the "Edit" options of the "Options" tab in the Console properties dialog (right click on title bar)

Or you can modify the registry to choose your own Ctrl key to activate the completion that does not interfere with the shortcut keys:
cmd /? wrote: You can enable or disable file name completion for a particular
invocation of CMD.EXE with the /F:ON or /F:OFF switch. You can enable
or disable completion for all invocations of CMD.EXE on a machine and/or
user logon session by setting either or both of the following REG_DWORD
values in the registry using REGEDIT.EXE:

HKEY_LOCAL_MACHINE\Software\Microsoft\Command Processor\CompletionChar
HKEY_LOCAL_MACHINE\Software\Microsoft\Command Processor\PathCompletionChar

and/or

HKEY_CURRENT_USER\Software\Microsoft\Command Processor\CompletionChar
HKEY_CURRENT_USER\Software\Microsoft\Command Processor\PathCompletionChar

with the hex value of a control character to use for a particular
function (e.g. 0x4 is Ctrl-D and 0x6 is Ctrl-F). The user specific
settings take precedence over the machine settings. The command line
switches take precedence over the registry settings.
The feature is kind of cool, though I rarely use it. With some effort you can browse an entire directory tree. Say you are looking for some folder on drive C:, but you can't remember any exact names. Simply enter "C:\ and press Ctrl-D repeatedly until you find the first folder of interest. Then append a
\, either before or after the last quote, and press Ctrl-D again, and so on. Unfortunately you cannot use \.. to move up the tree - you have to delete characters instead.
jeb wrote:
09 Jan 2020 00:02
dbenham wrote:
08 Jan 2020 16:03
3) It is impossible to redirect to a file with a name beginning with &
...
I know jeb makes it a habit of disproving absolute claims like this, but I am pretty confident it is not possible to redirect to a file name beginning with & using standard cmd.exe commands (disregarding the possible MKLINK solution). Challenge on
I didn't check the MKLINK solution, because it was to easy 8)

Code: Select all

echo Hello > ".\&works"
Argh... You smug bastard, you got me again :!: :lol:
You are correct, that was easy. :oops:

Dave Benham

dbenham
Expert
Posts: 2332
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: cmd.exe chars needing quotes, and escaping redirection handles

#4 Post by dbenham » 09 Jan 2020 09:07

Hell, I already new I could do:

Code: Select all

echo Hello>"x&2".
So why didn't I realize all I needed was an absolute path?! :oops: :oops: :oops: :oops:

Code: Select all

echo Hello>"c:\test\&2"
The .\ hack naturally follows. :roll:

jfl
Posts: 121
Joined: 26 Oct 2012 06:40
Location: Saint Hilaire du Touvet, France
Contact:

Re: cmd.exe chars needing quotes, and escaping redirection handles

#5 Post by jfl » 10 Jan 2020 08:32

About completion characters:

On my laptop (Windows version 10.0.19037.1), the Tab key already does file completion in cmd windows, even if you don't use cmd /F.
And Shift+Tab cycles backwards.
Looking in the registry, I see that both HKLM CompletionChar values contain 0x40, and both HKCU values contain 0x09.

I changed HKCU\...\PathCompletionChar to 0x04, and new shells now allow cycling through directories using the Control+D key. Thanks for the tip!
And, cherry on the cake, you can cycle backwards using Shift+Control+D :)

One inconvenient limitation though: If you've started cycling using either Tab or Control+D, you cannot change your mind and try cycling the other category using the other key.
The only way to change of category is to backspace all your way back to the last \, and start over with the other key.

Now the funny thing is that when I start a sub-shell using cmd /F:ON, Control+D works as above, but Tab does not anymore. It just outputs an actual tab character. :shock:
On the other hand, a sub-shell using cmd /F:OFF indeed has both special characters disabled. They do output a normal Tab or Control+D.

My initial idea was that maybe cmd shells started with /F:ON by default on my system.
But given these last observations, it's not the case. I don't understand how things really get configured. :?

dbenham
Expert
Posts: 2332
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: cmd.exe chars needing quotes, and escaping redirection handles

#6 Post by dbenham » 12 Jan 2020 21:46

Interesting. I have the exact same configuration on my Win 10 desktop, and I get the same behavior.

I re-read the cmd help, and I believe it is functioning as documented.
cmd /? wrote: ..................................................................... You can enable
or disable completion for all invocations of CMD.EXE on a machine and/or
user logon session by setting either or both of the following REG_DWORD
values in the registry using REGEDIT.EXE:

HKEY_LOCAL_MACHINE\Software\Microsoft\Command Processor\CompletionChar
HKEY_LOCAL_MACHINE\Software\Microsoft\Command Processor\PathCompletionChar

and/or

HKEY_CURRENT_USER\Software\Microsoft\Command Processor\CompletionChar
HKEY_CURRENT_USER\Software\Microsoft\Command Processor\PathCompletionChar

with the hex value of a control character to use for a particular
function (e.g. 0x4 is Ctrl-D and 0x6 is Ctrl-F). The user specific
settings take precedence over the machine settings.
Both HKCU values are 9 (tab), which takes precedence over the HKLM values of 0x40 (@).
Whenever I launch cmd.exe, the TAB key performs file/folder autocompletion.
So I infer that CompletionChar takes precedence over PathCompletionChar when both are the same.
Each time I invoke a new subprocess using cmd without the /F option, I get the same behavior - the registry values control the behavior.

If I launch a new subprocess with CMD /F:OFF, then of course auto-completion is disabled.

When I launch a new subprocess with CMD /F:ON, then CTRL-D and CTRL-F are used for auto-completion, and that is exactly what the documentation says!
cmd /? wrote: .................................. The command line
switches take precedence over the registry settings.

If completion is enabled with the /F:ON switch, the two control
characters used are Ctrl-D for directory name completion and Ctrl-F for
file name completion.
The HKLM value of 0x40 (@) is a bit odd. But it is sort of explained by the following.
cmd /? wrote: ................... To disable a particular completion character in
the registry, use the value for space (0x20) as it is not a valid
control character.
Well 0x40 is not a control character either, so if the HKCU values were deleted, then auto-completion would be disabled whenever cmd is launched without /F


Dave Benham

pieh-ejdsch
Posts: 217
Joined: 04 Mar 2014 11:14
Location: germany

Re: cmd.exe chars needing quotes, and escaping redirection handles

#7 Post by pieh-ejdsch » 13 Jan 2020 11:42

Oh. I also sometimes find some new files after testing at CMD or Batch.
but mostly they are empty and should not be there. EG:
b
1
.ABC
(
%% ~nxf
!
I have been using this with the addition of the tab key for a long time.

When a new CMD instance is started, it almost doesn't matter what is entered in the /E/F and /V options.
If :OffAndSomethingElseisThere - it will be deactivated.
Otherwise it doesn't matter whether or what is behind /V /E and /F, then it is activated.

Only when the characters :OFF are found after, the option is deactivated.
This is the same thing:
cmd /V/E/F
cmd /Variables /Extensions /Fullnames
this has been the case since XP

Phil

Post Reply