What's wrong with XP's for /f delimiter ?

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Message
Author
Ed Dyreen
Expert
Posts: 1569
Joined: 16 May 2011 08:21
Location: Flanders(Belgium)
Contact:

What's wrong with XP's for /f delimiter ?

#1 Post by Ed Dyreen » 13 Jan 2012 01:06

'
Let's take a simple delimiter that don't work:

Code: Select all

for /f "tokens=1-2 delims=,<SPACE><TAB>" %%a in (" this_Works,    0 ") do echo.&echo.a=%%~a_&echo.b=%%~b_

Code: Select all

a= this_Works_
b=      0 _
This one works as expected:

Code: Select all

for /f "tokens=1-2 delims=,<TAB><SPACE>" %%a in (" this_Works,    0 ") do echo.&echo.a=%%~a_&echo.b=%%~b_

Code: Select all

a=this_Works_
b=0_
But why :?:

trebor68
Posts: 146
Joined: 01 Jul 2011 08:47

Re: What's wrong with XP's for /f delimiter ?

#2 Post by trebor68 » 13 Jan 2012 02:52

The parameter for the command FOR /F:

tokens=x,y,m-n is a number
skip=n is a number
eol=c is only one character
delims=xxx is one or more character

Between the parameters can only use the character "space".

Please check the following commands to see the differences with the parameters.

Code: Select all

FOR /F "tokens=1-4,* delims=m skip" %a in (test.txt) do echo _%a_%b_%c_%d_%e_
The delimeter is only "m". The skip parameter is not correct.

Code: Select all

FOR /F "tokens=1-4,* delims=mskip " %a in (test.txt) do echo _%a_%b_%c_%d_%e_
Here are all delimeter correct "m", "s", "k", "i", "p" and "space".

Code: Select all

FOR /F "delims=mskip tokens=1-4,*" %a in (test.txt) do echo _%a_%b_%c_%d_%e_
Here are all delimeter "m", "s", "k", "i" and "p".

Code: Select all

FOR /F "delims=mskip  tokens=1-4,*" %a in (test.txt) do echo _%a_%b_%c_%d_%e_
Between "mskip" and "tokens" are two "space" charakter.
Here are all delimeter "m", "s", "k", "i" and "p".

Ed Dyreen
Expert
Posts: 1569
Joined: 16 May 2011 08:21
Location: Flanders(Belgium)
Contact:

Re: What's wrong with XP's for /f delimiter ?

#3 Post by Ed Dyreen » 13 Jan 2012 02:57

'
Hi trebor68,

I know how it works, yet wonder why it works the way it does.
I don't see why <TAB><SPACE> works yet <SPACE><TAB> fails :?

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: What's wrong with XP's for /f delimiter ?

#4 Post by dbenham » 13 Jan 2012 08:46

This is not an XP issue. It is a "well known" :wink:, but undocumented design feature of FOR /F.

If you want to include <space> in the list of DELIMS, the DELIMS option must be last in the list of options, and the <space> must be the last character of the list of DELIMS, followed immediately by the closing quote. The rule is so specific that even this example fails:

Code: Select all

for /f "tokens=1-2 delims=,<TAB><SPACE><SPACE>" %%a in (" this_fails    0 ") do echo.&echo.a=%%~a_&echo.b=%%~b_

Why? Since <space> is the option delimiter, any character after the <space> would be expected to be the beginning of the next option. The designers of FOR /F recognized the fact that <space> is an important possibility for DELIMS, so they wrote in special rules to allow <space> delimiter as long as it is the very last character of the entire list of options.

There is no design consistency.

The EOL option always takes the next single character after the = to be the "skip this line" character. If that next character happens to be <space>, then it also serves as the option delimiter. If it happens to be <double quote>, then it also serves as the end of options delimiter.

Code: Select all

for /f "eol=<space>delims=," %%a in (" no output because of eol") do echo %%a
 
for /f "eol=" %%a in ("" no output because of eol"") do echo %%a


I think it's a stupid design. I would have provided a mechanism to escape the <space> delimiter instead, as well as a mechanism to escape the escape. But we are stuck with what they gave us.

Dave Benham

Squashman
Expert
Posts: 4465
Joined: 23 Dec 2011 13:59

Re: What's wrong with XP's for /f delimiter ?

#5 Post by Squashman » 13 Jan 2012 10:56

I have a piece of software that I use for data conversion on the mainframe and it works just like that as well but in reverse. The space has to be the 1st delimiter and spaces after that are ignored by the software.

trebor68
Posts: 146
Joined: 01 Jul 2011 08:47

Re: What's wrong with XP's for /f delimiter ?

#6 Post by trebor68 » 13 Jan 2012 11:27

You can use different delimeters but the last character in the row must be the "space" character.

The character <tab> and <space> are not the same characters.

Here my test files:
Test2.txt is a text file.
1st row only <tab> between the words.
2nd row only <space> between the words.
3rd row <space><tab> between the words.
4th row <tab><space> between the words.
Mein Name ist Hase.
Mein Name ist Hase.
Mein Name ist Hase.
Mein Name ist Hase.


Here the batch file:

Code: Select all

@echo off
echo Only Tab.
for /f "tokens=1-4 delims=   " %%a in (Test2.txt) do echo _%%a_%%b_%%c_%%d_
echo.
echo Only Space.
for /f "tokens=1-4 delims= " %%a in (Test2.txt) do echo _%%a_%%b_%%c_%%d_
echo.
echo Tab and Space.
for /f "tokens=1-4 delims=    " %%a in (Test2.txt) do echo _%%a_%%b_%%c_%%d_
echo.
echo No delimeter.
for /f "tokens=1-4" %%a in (Test2.txt) do echo _%%a_%%b_%%c_%%d_


The result is:

Code: Select all

Only Tab.
_Mein_Name_ist_Hase._
_Mein Name ist Hase.____
_Mein _Name _ist _Hase._
_Mein_ Name_ ist_ Hase._

Only Space.
_Mein   Name   ist   Hase.____
_Mein_Name_ist_Hase._
_Mein_   Name_   ist_   Hase._
_Mein   _Name   _ist   _Hase._

Tab and Space.
_Mein_Name_ist_Hase._
_Mein_Name_ist_Hase._
_Mein_Name_ist_Hase._
_Mein_Name_ist_Hase._

No delimeter.
_Mein_Name_ist_Hase._
_Mein_Name_ist_Hase._
_Mein_Name_ist_Hase._
_Mein_Name_ist_Hase._


EDIT
The text "Mein Name ist Hase." is in German. In English: "My name is bunny."

Ed Dyreen
Expert
Posts: 1569
Joined: 16 May 2011 08:21
Location: Flanders(Belgium)
Contact:

Re: What's wrong with XP's for /f delimiter ?

#7 Post by Ed Dyreen » 13 Jan 2012 11:30

'
Hey thanks ben, I wonder where you digg this well known information.

I understand a little better now :)

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: What's wrong with XP's for /f delimiter ?

#8 Post by aGerman » 13 Jan 2012 16:12

[Off Topic]
trebor68 wrote:The text "Mein Name ist Hase." is in German. In English: "My name is bunny."

Haha no, the translation is "I'm saying nothing" :wink:
There is a little story behind this German idiom. Victor von Hase, student in Heidelberg (later a prominent lawyer), had a friend. This friend shot another student in a duel (1855). Now he had to flee to France but he was wanted by the police and couldn't use his own passport. He managed to get the loan of v. Hases passport. Later this passport was found in France and returned to Heidelberg. Hase had to appear in court. The very first he said was "My name is Hase, I'm saying nothing, I know nothing." This saying was fancy enough to become famous for more than 150 years now...

Regards
aGerman
[/Off Topic]

trebor68
Posts: 146
Joined: 01 Jul 2011 08:47

Re: What's wrong with XP's for /f delimiter ?

#9 Post by trebor68 » 15 Jan 2012 08:38

@aGerman: Thank for this info.
We usually use phrases without knowing the story behind it.


Here the background of the text:

I use mostly the same text to specific functions of batch files to test.

In German is the complete text:
Mein Name ist Hase,
ich wohne im Wald,
dort ist es dunkel
und auch bitterkalt.

Translated to English the complete text is:
My name is Bunny,
I live in the forest,
there it is dark
and bitter cold.

djb
Posts: 2
Joined: 28 Jan 2014 13:06

Re: What's wrong with XP's for /f delimiter ?

#10 Post by djb » 28 Jan 2014 13:20

To follow along some of the lines expressed earlier? Is it possible to
specify that I want a TAB character to be a delimiter, and the SPACE
character NOT to be a delimiter, and to express this within a batch
file WITHOUT having to embed a true-blue TAB character in the file?

In other words, is there a way to ESCAPE the single TAB character in
the delims= string)

My text editors, and my company coding standards too, highly discourage
embedded TAB characters in source code. But I need this functionality
(tab-delimited sentences on a line in a datafile).

-Dave

penpen
Expert
Posts: 1991
Joined: 23 Jun 2013 06:15
Location: Germany

Re: What's wrong with XP's for /f delimiter ?

#11 Post by penpen » 29 Jan 2014 14:04

Yes: Use the file table.dat file created here: http://www.dostips.com/forum/viewtopic.php?f=3&t=5326

Code: Select all

set "table="
set /P "table=" < table.dat

echo This should be the tab: "%table:~9,1%"

penpen

Sponge Belly
Posts: 216
Joined: 01 Oct 2012 13:32
Location: Ireland
Contact:

Re: What's wrong with XP's for /f delimiter ?

#12 Post by Sponge Belly » 29 Jan 2014 14:55

Welcome to DosTips, Dave! :-)

Penpen’s answer is probably overkill for your requirements. Below is a little snippet that stores a tab in a variable along with an example of how to use it. Should work with any version of Windows from XP onwards.

Code: Select all

for /f "delims=" %%t in ('mshta ^"javascript:close(new ^
ActiveXObject('Scripting.FileSystemObject'^).^
GetStandardStream(1^).Write(^"\x09^"^)^)^"') do set tab=%%t
echo(words%tab%separated%tab%by%tab%tabs


Please note, however, that the default delimiters for a for /f loop are tab and space so there’s no need to explicitly specify them unless you need to use other delimiters as well or you want to set tab as delimiter and not space.

Hope this helps!

- SB

djb
Posts: 2
Joined: 28 Jan 2014 13:06

Re: What's wrong with XP's for /f delimiter ?

#13 Post by djb » 12 Feb 2014 10:14

Thanks for the welcome guys.

I did muddle through the trick of getting a lonely little TAB character stored into
an environment variable via the following...(working from memory)

Code: Select all

SET REG_OS="HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Environment"
REG.EXE QUERY %REG_OS% /V PROCESSOR_IDENTIFIER | FIND /I "REG_SZ" > Reg_PI.Tmp
for /F "tokens=1-3* delims=rR" %%i in (Reg_PI.Tmp) do SET TABCHR=%%k


The contents of Reg_PI comes out like
PROCESSOR_IDENTIFIER<tab>REG_SZ<tab>data with spaces that I want to keep together.

and so parsing this line with all "R"s as delimiters plucks out the lonely little tab
betrween "PROCESSOR_IDENTIFIER" and "REG_SZ"

After that, I can use that environment variable (%TABCHR%) as the sole delimiter for parsing this (and many other) registry entries (which was my objective to begin with.)

Samir
Posts: 384
Joined: 16 Jul 2013 12:00
Location: HSV
Contact:

Re: What's wrong with XP's for /f delimiter ?

#14 Post by Samir » 27 Aug 2014 21:17

I hate to bump an older thread, but I've been searching all over the place for a simple pure-batch way of using tab as the delimiter in for (for use in this thread to convert a tab-delimited to a csv: posting.php?mode=quote&f=3&t=1418). Anyone have an easy way? Or does it always have to be stored as a variable from a file or something like that?

foxidrive
Expert
Posts: 6031
Joined: 10 Feb 2012 02:20

Re: What's wrong with XP's for /f delimiter ?

#15 Post by foxidrive » 28 Aug 2014 02:00

It's a problem in forums but you can type the TAB character into your script.

Post Reply