[Updated] Patch for cmd.exe for windows xp for cp 65001

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Message
Author
Liviu
Expert
Posts: 470
Joined: 13 Jan 2012 21:24

Re: Patch for cmd.exe for windows xp for cp 65001

#16 Post by Liviu » 14 May 2014 21:09

carlos wrote:After this patch cmd.exe for windows x86 would use the same behavior on MultibyteToWideChar that cmd.exe for windows 7. This means that codepage 65001 is fully supported.
I'd qualify that to "supported at the same level as Win7" - which is not "fully". Pipes are still broken, both in Win7 and in the patched cmd.

Code: Select all

C:\tmp>chcp 65001
Active code page: 65001

C:\tmp>echo ‹αß©∂€›
‹αß©∂€›

C:\tmp>echo ‹αß©∂€› | more
Not enough memory.

C:\tmp>

carlos wrote:This is the patch
Comparing the codepages in the patch vs. the "official list" linked in a previous post, it's interesting to note (though probably inconsequential) that the patch clears the flags for codepage 52936 (not listed) but not for 50229 (listed).

That said, nicely done. I may not be a big fan of this sort of hacks myself, but I can still tell a good one when I see it.

Liviu

carlos
Expert
Posts: 489
Joined: 20 Aug 2010 13:57
Location: Chile
Contact:

Re: [Done] Patch for cmd.exe for windows xp for cp 65001

#17 Post by carlos » 15 May 2014 00:23

Liviu, the patch is perfect. It cover all the codepages listed oficially, more 52936, that is not listed, but is used in the function used in cmd.exe for windows 7 and windows 8.
Try run a batch using the original cmd.exe for windows xp using codepage 52936, it also break.
Maybe 52936 would be appear in the official list :)
And then test with the patched.

About, the problem using more command. It not work properly even using 65001 on windows 8.
I insist, the patch is perfect.

Liviu
Expert
Posts: 470
Joined: 13 Jan 2012 21:24

Re: [Done] Patch for cmd.exe for windows xp for cp 65001

#18 Post by Liviu » 15 May 2014 00:48

carlos wrote:It cover all the codepages listed oficially, more 52936 [...] Maybe 52936 would be appear in the official list :)
Agree with 52936. I'll take back the part about 50229 - it's obviously the first one checked and have the flags 0'd out, and I must have had a momentary blind spot.

carlos wrote:About, the problem using more command. It not work properly even using 65001 on windows 8.
I insist, the patch is perfect.
Seems like we agree on the facts, but have a different notion of "perfect" ;-) Your patch looks to be as good as Win7, but I wouldn't call Win7 itself "perfect" since it fails a most trivial "pipe" test case.

Liviu

penpen
Expert
Posts: 1726
Joined: 23 Jun 2013 06:15
Location: Germany

Re: [Done] Patch for cmd.exe for windows xp for cp 65001

#19 Post by penpen » 15 May 2014 04:32

Good work.

The pipe works properly: As carlos said it is the more command that causes problems:

Code: Select all

Z:\>more
Nicht genügend Arbeitsspeicher.

Z:\>echo ‹αß©∂€›| findstr "^" > result.txt
Result.txt contains the correct values.

Although the external commands might behave buggy:

Code: Select all

Z:\>echo ‹αß©∂€›| findstr "^"
.....∂€.
FINDSTR: Fehler beim Schreiben

penpen

Liviu
Expert
Posts: 470
Joined: 13 Jan 2012 21:24

Re: [Done] Patch for cmd.exe for windows xp for cp 65001

#20 Post by Liviu » 15 May 2014 11:39

penpen wrote:The pipe works properly: As carlos said it is the more command that causes problems

"More" was just the simplest example to demonstrate the problem, but it's not the cause of the problem. Piping under codepage 65001 is broken more deeply than just "more".

Here is an example in Windows 7, not involving "more" - but just "cmd" itself and internal commands.

Code: Select all

C:\tmp>chcp 65001
Active code page: 65001

C:\tmp>echo echo ‹αß©∂€› ^& exit | cmd /k chcp
Active code page: 65001

C:\tmp>echo ������������������
������������������

C:\tmp>
And FWIW this is the same command run in XP with carlos' patch.

Code: Select all

C:\tmpp\carlos-cmd2b>chcp 65001
Active code page: 65001

C:\tmp>echo echo ‹αß©∂€› ^& exit | cmd /k chcp
Active code page: 65001

C:\tmp>More?
C:\tmp>

And here is another example, which works correctly in Windows 7...

Code: Select all

C:\tmp>chcp 65001
Active code page: 65001

C:\tmp>echo ‹αß©∂€› | (set/p "var=" & set var)
var=‹αß©∂€›
...but fails in XP with the patched cmd.

Code: Select all

C:\tmp>chcp 65001
Active code page: 65001

C:\tmp>echo ‹αß©∂€› | (set/p "var=" & set var)
Environment variable var not defined

So, I stand by my previous points - piping in codepage 65001 is not working (consistently) in Windows 7, and while carlos' patch did fix some of the issues in the XP cmd, codepage 65001 is still not "fully" supported.

Liviu

penpen
Expert
Posts: 1726
Joined: 23 Jun 2013 06:15
Location: Germany

Re: [Done] Patch for cmd.exe for windows xp for cp 65001

#21 Post by penpen » 15 May 2014 14:03

I'm not sure, but it seems we are using the term "pipe" for different things.
I call only the unnamed pipe object the "pipe", and "piping" is writing raw data to and reading raw data from this (unnamed) pipe object.
You seem to use the term "pipe" for what i call "piping", and to define "piping" as what i would call data encoding and data interpretation.

If that is the case then we both are right.
If there is at least one way to reliably write raw data to and read from a pipe,
then the pipe (object functionality) is not broken (findstr example above).
This does not mean that data encoding/interpretation will success: This is the part that fails (no matter if in a patched or unpatched cmd.exe, i think).

I only can guess, but i think your examples of 'broken piping' can be caused in default (non patched) cmd.exe versions, too:

Code: Select all

Z:\>cmd /U
Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

Z:\>chcp
Aktive Codepage: 850.

Z:\>cmd /u /c set /P "=echo echo abc" ^< nul | cmd /u /k chcp
Aktive Codepage: 850.

Z:\>Mehr?
Z:\>
Z:\>cmd /u /c echo echo abc| cmd /u /k chcp
Aktive Codepage: 850.

Z:\>Mehr? Der Befehl "e" ist entweder falsch geschrieben oder
konnte nicht gefunden werden.

Z:\>
It is just a(nother) bug of (unpatched) cmd when reading unicode (i think as UTF-8 is unicode too, this bug occurs in any cmd mode (/A, /U).
(There are also examples where the below example produce a 0x03 character at the end, too. But actualy i can't remember how to produce this bug ... maybe it has been fixed ... i don't know.)
Sad to say, i have no idea what is causing this bug, as the raw data survives unchanged, and i even can't imagine why unchanged data may be misinterpreted... .
In addition i have no idea, why the above example fails, as no (en)coding in any way is needed (pipe supports UCS2, input/output is UCS2):

Code: Select all

Z:\>cmd /u /c set /P "=echo abc" ^<nul | cmd /u /c findstr "^"
e.c.h.o. .a.b.c.
(You may redirect it to file, to see that the raw data is without any error.)

penpen

Edit: Added the last example.

Liviu
Expert
Posts: 470
Joined: 13 Jan 2012 21:24

Re: [Done] Patch for cmd.exe for windows xp for cp 65001

#22 Post by Liviu » 15 May 2014 15:28

penpen wrote:You seem to use the term "pipe" for what i call "piping", and to define "piping" as what i would call data encoding and data interpretation.
If that is the case then we both are right.
That's the case, indeed. I deliberately used "piping under codepage 65001" in the previous post to avoid possible confusion. The pipe is essentially a byte stream and pipes are widely used throughout Windows, so it's certainly not the pipe object itself which is broken, but the way cmd and other commands use it.

Your cmd/u examples touch on another and different issue, of UTF16 character input via pipe. That's known to have problems, but it's not directly related to codepage 65001 or the main topic here. You'll note that my examples all work as expected if you replace "chcp 65001" with "chcp 437" for example (or any SBCS codepage). The point was to show where Windows 7 fails, and where the XP cmd - even patched - fails and/or works differently.

Liviu

carlos
Expert
Posts: 489
Joined: 20 Aug 2010 13:57
Location: Chile
Contact:

Re: [Updated] Patch for cmd.exe for windows xp for cp 65001

#23 Post by carlos » 19 May 2014 12:23

The patch has been updated, because it previously have a incorrect code (relationated with the call to GetACP function). Now is fixed. Please repatch again.

Liviu
Expert
Posts: 470
Joined: 13 Jan 2012 21:24

Re: [Updated] Patch for cmd.exe for windows xp for cp 65001

#24 Post by Liviu » 23 May 2014 19:39

Thanks for the update. I don't see any change in functionality, but if I am understanding the bugfix right then it was a case of either work or crash, and the previous patch happened to work here.

FWIW I updated my $ascU.cmd in the p.s. of http://www.dostips.com/forum/viewtopic.php?p=33667#p33667 to remove the hardcoded 'cmd' calls. Using '%comspec%', instead, makes it easier to test different shells - for example, in order to try your patch in xp, I renamed it to 'xpCmdU.exe' and created a shortcut to 'xpCmdU /k set "comSpec=<full-path>\xpCmdU.exe"', which I then customized to use a Unicode/TT font and its own layout/colors.

carlos
Expert
Posts: 489
Joined: 20 Aug 2010 13:57
Location: Chile
Contact:

Re: [Updated] Patch for cmd.exe for windows xp for cp 65001

#25 Post by carlos » 24 May 2014 00:02

Liviu, the code fixed was code that is not always executed, is when the codepage is 0 and is needed call to GetACP function. The internal functions of cmd that require it are only two: http://www.dostips.com/forum/viewtopic.php?p=34437#p34437

Then, in the previous version all seems to work ok, but the unfixed code will be fail for that two functions. In other words not all the code of the previous patch was tested.

This is the part of the original code, (that if you remove the first jnz only for test), it cause that cmd close.

Code: Select all

jnz short @f
push eax
push @GetACP
push dword[4ad34848] ;; kernel32.dll
call dword[4ad01138] ;; GetProcAddress
call eax


now replaced (tested ok with debugger (removing the jnz only for the test), it get ok the value returned by the function GetACP) by:

Code: Select all

jnz short @f
push 4ad0497c ;; push lpModuleName =  L"kernel32.dll"
call dword[4ad0113c] ;; hModule = call GetModuleHandleW
push @GetACP ;; push lpProcName = "GetACP"
push eax ;; push hModule
call dword[4ad01138] ;; *func = GetProcAddress
call eax ;; call func()

Liviu
Expert
Posts: 470
Joined: 13 Jan 2012 21:24

Re: [Updated] Patch for cmd.exe for windows xp for cp 65001

#26 Post by Liviu » 24 May 2014 21:45

What I meant by "either work or crash" was "for repeated runs under the same conditions - like, on the same machine". I don't really know the context, and my guess was that the original had the hModule hardcoded assuming the preferred load address, while the fix retrieved it with the extra call. But an actual test in the debugger beats random guesses any day ;-) Thanks.

P.S. Just as a side note, the address of GetACP doesn't change during the run, and I was a bit uneasy to see it re-fetched on each call. But it seems to be such a "cheap" redundancy that it doesn't introduce any measurable overhead, at least I haven't seen any in a few cursory tests.

carlos
Expert
Posts: 489
Joined: 20 Aug 2010 13:57
Location: Chile
Contact:

Re: [Updated] Patch for cmd.exe for windows xp for cp 65001

#27 Post by carlos » 27 May 2014 02:14

Liviu, Jason provides a nice solution for use the patched cmd.exe as default. I left it on the post but I also repeat here:

Code: Select all

Rem run this from the folder that have the patched cmd.exe
Copy /Y cmd.exe "%SystemRoot%\system32\cmdutf8.exe"
Set "key=HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\cmd.exe"
Reg.exe add "%key%" /v "Debugger" /d "%SystemRoot%\system32\cmdutf8.exe" /f
exit


For the screenshots, look: http://consolesoft.com/p/cmd-xp-65001-fix/index.html

With that, edit comspec variable will be unnecessary.

Liviu
Expert
Posts: 470
Joined: 13 Jan 2012 21:24

Re: [Updated] Patch for cmd.exe for windows xp for cp 65001

#28 Post by Liviu » 27 May 2014 19:48

carlos wrote:Jason provides a nice solution for use the patched cmd.exe as default. [...] With that, edit comspec variable will be unnecessary.

That works nicely if you want to make the patched cmd the default. Still, in order to be able to run _two_ different shells side by side - for example to test or compare - one of them would still need to set a different comspec in its environment for child processes to inherit (those who care, and who reference '%comspec%' rather than hardcode 'cmd').

carlos
Expert
Posts: 489
Joined: 20 Aug 2010 13:57
Location: Chile
Contact:

Re: [Updated] Patch for cmd.exe for windows xp for cp 65001

#29 Post by carlos » 28 May 2014 00:37

Liviu, this is independant of the compsec variable, if you use cmd or cmd.exe anyways it will open cmdutf8.exe

Liviu
Expert
Posts: 470
Joined: 13 Jan 2012 21:24

Re: [Updated] Patch for cmd.exe for windows xp for cp 65001

#30 Post by Liviu » 28 May 2014 01:02

Right. That's exactly what makes it difficult in your scenario to run _another_ shell side-by-side, which in turn might call external batch files or other programs that, in turn, would issue %comspec% calls. For an example, see the edit I mentioned in http://www.dostips.com/forum/viewtopic.php?p=34674#p34674 above. Being able to test my $ascU.cmd under _both_ the original and the patched cmd requires setting a %comspec% different from the default in at least _one_ of the cases. Which one you choose as the default, and which one as an override is your choice, of course, but the right %comspec% still matters - which was the point in my previous post.

Post Reply