Page 1 of 1

8191 buffer limit: Different Windows 10 builds behave differently

Posted: 19 Mar 2020 21:26
by dbenham
The 8191 character size limit is pervasive throughout cmd.exe batch parsing. There are many buffers in the various phases, each one limited to 8191 bytes. I wanted to investigate how each "phase" handles an overflow of the buffer. I know sometimes it creates an error, sometimes the data is truncated and the remainder ignored, and .... ?

I assume that as each phase processes a token, it reads from the source buffer, does some manipulation, and writes to an output buffer. The manipulation may cause the token to expand beyond the limits of the output buffer.

Here are the obvious buffered operations I am aware of, each with the potential of having its own overflow rules:
  • Phase 0 file read buffer
  • Phase 2 percent expansion
    • Command token buffer
    • Command arguments buffer
    • Other possible buffers: redirection, FOR and IF special tokens ...
For most of the remaining phases a separate buffer is used for each token produced by phase 2
  • Phase 4 For variable expansion buffer
  • Phase 5 Delayed expansion buffer
  • Phase 6 CALL. I don't think a new buffer is needed, but if CALL is parsed from a larger command token, then the balance of the command token must be appended to the arguments token.
  • Phase 7 Execution
    • Parse command from command token, balance must be appended to arguments token
    • FOR iteration: Each FOR argument gets its own token buffer
    • ECHO output buffer
Also the error output has its own buffer.

So I had planned to run a series of tests to investigate all of the above. But I didn't get very far because I am seeing surprising differences between two machines with different versions of Windows 10 :shock: :?

Work computer running Windows 10 Enterprise 64 bit, build 16299

Code: Select all

 INFO.BAT version 1.5
--------------------------------------------------------------------------------
Windows version        :  Microsoft Windows [Version 10.0.16299.1625]
Product name           :  Windows 10 Enterprise, 64 bit
Performance indicators :  Processor Cores: 8      Visible RAM: 8283220 kilobytes

Date/Time format       :  (mm/dd/yy)  Thu 01/30/2020  11:40:32.47
__APPDIR__             :  C:\WINDOWS\system32\
ComSpec                :  C:\WINDOWS\system32\cmd.exe
PathExt                :  .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
Extensions             :  system: Enabled   user: Enabled 
Delayed expansion      :  system: Disabled  user: Disabled
Locale name            :  en-US       Code Pages: OEM  437    ANSI 1252
DIR  format            :  01/29/2020  03:48 AM     1,677,721,600 pagefile.sys
Permissions            :  Elevated Admin=No, Admin group=No

                          Missing from the tool collection:  debug
Home computer running Windows 10 Pro 64 bit, build 17763

Code: Select all

 INFO.BAT version 1.5
--------------------------------------------------------------------------------
Windows version        :  Microsoft Windows [Version 10.0.17763.1039]
Product name           :  Windows 10 Pro, 64 bit
Performance indicators :  Processor Cores: 4      Visible RAM: 4192432 kilobytes

Date/Time format       :  (mm/dd/yy)  Thu 03/19/2020  21:53:56.39
__APPDIR__             :  C:\WINDOWS\system32\
ComSpec                :  C:\WINDOWS\system32\cmd.exe
PathExt                :  .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
Extensions             :  system: Enabled   user: Enabled 
Delayed expansion      :  system: Disabled  user: Disabled
Locale name            :  en-US       Code Pages: OEM  437    ANSI 1252
DIR  format            :  03/16/2020  12:39 PM     6,947,000,320 pagefile.sys
Permissions            :  Elevated Admin=No, Admin group=Yes

                          Missing from the tool collection:  debug
It is awkward to test such long lines, and even more awkward to post the test code and the results to the forum. I've attached a zip file containing the test script as well as the results from my Work and Home machines.
test_size.zip
(1.93 KiB) Downloaded 79 times
I'll describe the tests, and explain my interpretation of the output.

Test 1) Command token 16405 bytes long with stdout redirection at end, followed by concatenated ECHO( and ECHO OK
starts at position 1 with ECHO.
position 8192 starts with ZECHO.
position 16382 starts with 23456789012345678901234 8888888888888

Work result:
Reads first 8191 bytes into command token, buffer is full, and the command token is terminated.
The next 8191 bytes are read and the first Z character is stripped and the rest placed in the argument token.
Continues to read the "arguments token", but throws it away until it reaches redirection which is successfully carried out.
Phase 7 strips out the ECHO command and the remainder after the dot goes into a new arguments buffer. The input arguments token is appended to the arguments buffer until it is full, and the rest is discarded.
The ECHO is successful, but the output is 8191 bytes, so there is no room for the the terminating \r\n
The concatenated commands are parsed and executed just fine.

Home result:
Reads first 8191 bytes into command token, buffer is full, and the command parsing is finished without any arguments token
The echo command is extracted and the balance is placed in the arguments buffer. ECHO writes out the argument
The next 8191 bytes is read into the command token, buffer is again full and the command parsing is finished without any arguments token
This time the command is invalid, but the error message exceeds 8191 bytes, so it is lost.
The balance of the line is parsed as a command, with redirection
The 23456789012345678901234 command is invalid and the error message is successful because it is less than 8191 bytes long.
The concatenated commands are parsed and executed just fine.

Test 2) Same as 1 except line begins with XCHO. and there are no commands concatenated at the end.

Work result:
Mostly the same as test 1, except now the command is invalid, and the error message exceeds 8191 bytes, so it fails

Home result:
Mostly the same as test 1, except now the first command is also invalid with an error message that exceeds 8191 bytes, so it fails.
Only the 3rd invalid command successfully prints out an error message as before.

Test 3) stdout and stderr redirected at beginning. Invalid command token is 8101 bytes long. Line is 24987 bytes long

Work and Home give same result:
The redirection succeeds and the command token is parsed.
Everything after the command token is parsed into the arguments token until it is full, the rest is discarded.
The error message (including two sets of \r\n) is exactly 8191 bytes, and succeeds

Test 4) Same as test 3, except now the command token is 8102 bytes long.

Work and Home give same result:
The same as test 3, except now the error message with two sets of \r\n is 8192 bytes, which exceeds the 8191 byte limit, and the error message fails (no output)

Tests 5 and 6 I began to look at Phase 1 expansion, but I didn't bother analyzing once I discovered the difference between two machines.

=========

Thoughts anyone?

I'm curious what the behavior is with XP (or any other version).
I'm also curious how more recent builds of Windows 10 behave.
Is the difference I am seeing related to Enterprise vs. Pro? (I doubt it) or differences between the builds? (I'm assuming the latter)


Dave Benham

Re: 8191 buffer limit: Different Windows 10 builds behave differently

Posted: 20 Mar 2020 04:25
by jeb
Hi Dave,

good idea to use the time ...
dbenham wrote:
19 Mar 2020 21:26
So I had planned to run a series of tests to investigate all of the above. But I didn't get very far because I am seeing surprising differences between two machines with different versions of Windows 10 :shock: :?
1) Differences
I knew this, but I didn't found the cause.
But It's more different than you think.

I tested that and found this also in Win7 (If I remember correct).

At the beginning of the test the behavior is different than later, and the original behavior can't be reset by closing cmd.exe.
I can reset it only by restarting the computer.

The changes are relevant for the size of lines and if they throw an error or not.
In the beginning my lines can be very long without any errors, but later I got much more errors for long lines.

I canceled my tests, because I don't want to restart my computer frequently.
And I never knew which behavior I'm testing, the "starting" or the "later" behavior, that's anoying :evil:

2) Long line tokens
There is a double limit for the command token and for the content. Both can be 8191 (or even a bit longer).
It's obvious that the total limit must be larger than 8191, because a line has to be longer, when it defines a variable with 8191 characters

Code: Select all

(set^Limit=abcde...<content-with-8191-chars>xyz)

3) Limits of prompt
A while a ago, I build a way to create a CR with the prompt command, by creating a prompt that ends with $_ (CR LF) , the LF will be stripped, when the LF is behind the buffer length limit.
To create such a long prompt you can use multiple $V entries (Version information).
The code looks like

Code: Select all

prompt $V$V$V$V$V$V...$V<stuffing>$_
Long story, but the max prompt length is 512 characters ...
... But only for pre Win10 versions :x
If I remember, Win10 has a limit of 8192 (or 8191) characters (Currently I can't test, my office PC is unavailable).

I don't understand why MS changed this value, because even a size of 512 characters for a prompt seems to be useless.

jeb

Re: 8191 buffer limit: Different Windows 10 builds behave differently

Posted: 20 Mar 2020 05:56
by penpen
dbenham wrote:I'm curious what the behavior is with XP (or any other version).
Windows XP pro, 32 bit, Version 5.1.2600, just terminates the execution of the batch ater the first performed test (same behaviour for all tests):

Code: Select all

-----------------------------------------------
:test1
Die eingegebene Zeile ist zu lang.
("input line too long")
dbenham wrote:I'm also curious how more recent builds of Windows 10 behave.
Windows 10 Home, 64 bit, Version 10.0.18362.720, de-DE behaves essentially the same to your "home test", but displays the error message for ":test4" (which i think is ok, because here the error message is 8184 bytes long).


penpen

Re: 8191 buffer limit: Different Windows 10 builds behave differently

Posted: 20 Mar 2020 06:07
by dbenham
jeb wrote:
20 Mar 2020 04:25
At the beginning of the test the behavior is different than later, and the original behavior can't be reset by closing cmd.exe.
I can reset it only by restarting the computer.

The changes are relevant for the size of lines and if they throw an error or not.
In the beginning my lines can be very long without any errors, but later I got much more errors for long lines.

I canceled my tests, because I don't want to restart my computer frequently.
And I never knew which behavior I'm testing, the "starting" or the "later" behavior, that's annoying :evil:
Ugh :(

I'm curious, but not that curious (or masochistic)
I didn't run into changing behavior on the same machine yet, and I'm already frustrated. Your experience has scared me off from pursuing this further.
jeb wrote:
20 Mar 2020 04:25
Long story, but the max prompt length is 512 characters ...
... But only for pre Win10 versions :x
If I remember, Win10 has a limit of 8192 (or 8191) characters (Currently I can't test, my office PC is unavailable).

I don't understand why MS changed this value, because even a size of 512 characters for a prompt seems to be useless.
Well, the prompt typically displays the current directory. Historically the path has been limited to 260 characters (I think a bit less for a folder path). It makes sense that the prompt should support the max path length with some head room. So 512 does not seem too unreasonable.

But Windows 10 has an option to remove the low 260 max length limit for NTFS. (Not sure what the new limit is) I'm guessing there are a few reasons for this - the move to linux support, and the default Python install requires paths longer than 260.

Maybe the increased prompt length limit is designed to support the possibility of much longer paths.


Dave Benham

Re: 8191 buffer limit: Different Windows 10 builds behave differently

Posted: 20 Mar 2020 07:57
by penpen
dbenham wrote:I didn't run into changing behavior on the same machine yet
Just try (win10) a batch file with many AT-characters ('@') (varies depending on machine) and an internal command.
On my machine when using 3053 AT's and set-command the console window crashes (that happens all the time) after it may or may not have performed the set-command (changes).

penpen

Re: 8191 buffer limit: Different Windows 10 builds behave differently

Posted: 20 Mar 2020 08:57
by dbenham
penpen wrote:
20 Mar 2020 05:56
dbenham wrote:I'm curious what the behavior is with XP (or any other version).
Windows XP pro, 32 bit, Version 5.1.2600, just terminates the execution of the batch ater the first performed test (same behaviour for all tests):

Code: Select all

-----------------------------------------------
:test1
Die eingegebene Zeile ist zu lang.
("input line too long")
dbenham wrote:I'm also curious how more recent builds of Windows 10 behave.
Windows 10 Home, 64 bit, Version 10.0.18362.720, de-DE behaves essentially the same to your "home test", but displays the error message for ":test4" (which i think is ok, because here the error message is 8184 bytes long).
Thanks for testing. The XP behavior tickles my memory of how things used to work. I wonder when the transition occurred to be more like Win 10 behavior(s).
penpen wrote:
20 Mar 2020 07:57
dbenham wrote:I didn't run into changing behavior on the same machine yet
Just try (win10) a batch file with many AT-characters ('@') (varies depending on machine) and an internal command.
On my machine when using 3053 AT's and set-command the console window crashes (that happens all the time) after it may or may not have performed the set-command (changes).
Thanks - that is interesting.
But I was referring to changes in behavior that are triggered by 8191 buffer limit being exceeded. I don't think your example qualifies.