FOR /F with Line Feed

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Message
Author
Squashman
Expert
Posts: 4465
Joined: 23 Dec 2011 13:59

FOR /F with Line Feed

#1 Post by Squashman » 04 Aug 2018 00:17

I thought I remember seeing this type of syntax in the past but have not been able to get it to work.

Code: Select all

@echo off
setlocal enabledelayedexpansion
set LF=^


REM Do Not remove empty lines above
for /F "tokens=1,2 delims=," %%G in (
"1,Martini Shaker!LF!"
"2,Wine Bottle!LF!"
"3,Beer Glass"
) do (
echo T1=%%~G T2=%%~H
)
pause
My expectation is to have this as the output

Code: Select all

T1=1 T2=Martini Shaker
T1=2 T2=Wine Bottle
T1=3 T2=Beer Glass
But I end up with this.

Code: Select all

T1=1 T2=Martini Shaker
T1= "2 T2=Wine Bottle
T1= "3 T2=Beer Glass
I know this has been done on the forums in the past, I just can't find the thread.

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: FOR /F with Line Feed

#2 Post by aGerman » 04 Aug 2018 05:07

I think several pairs of quotes won't work. A string literal has only one pair of surrounding quotes.

Code: Select all

@echo off &setlocal
set ^"LF=^

^" Do Not remove the empty line above
set ^"\n=^^^%LF%%LF%^%LF%%LF%^^"
for /F "tokens=1,2 delims=," %%G in (^"^
1^,Martini Shaker%\n%
2^,Wine Bottle%\n%
3^,Beer Glass%\n%
") do (
echo T1=%%~G T2=%%~H
)
pause
Steffen

npocmaka_
Posts: 512
Joined: 24 Jun 2013 17:10
Location: Bulgaria
Contact:

Re: FOR /F with Line Feed

#3 Post by npocmaka_ » 04 Aug 2018 06:38

there was an easier way to the the LF with set and brackets but I cant find the link. If I'm not wrong it was found by Aacini.

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: FOR /F with Line Feed

#4 Post by dbenham » 04 Aug 2018 06:45

I currently don't have access to a Windows machine at the moment, so I can't test or verify. But I think I see a couple problems with aGerman's code.

1) The last quote must be escaped, else the closing parenthesis is treated as a string literal. I don't think that code will run at all. My mistake. The code should run just fine because the preceding %\n% escapes the end of the line, which is stripped, and then the next character is also escaped, which happens to be the quote.

2) The commas should not require escaping, unless you are worried about the XP FOR /F bug. If you do want the code to be safe on XP, then all token delimiters must be escaped. In this case that would be commas and spaces. Disregarding the XP bug, FOR /F should not require token delimiters to be escaped or quoted. It is only the simple FOR command that uses token delimiters as the iteration delimiter.

If I were to write aGerman's code, I would use USEBACKQ and substitute ' for " so that I don't have to worry about escaping the quotes.

I generally only use the FOR /F <LF> multi line string trick if I have a delimited string that I want to iterate, in which case I would use expansion find/replace to substitute <LF> for the delimiter, all within the IN() clause.

For example (again, untested):

Code: Select all

@echo off
setlocal
set "str=1,Martini Shaker|2,Wine Bottle|3,Beer Glass"

(set LF=^
%= This creates a Line Feed (0x0A) character - DO NOT ALTER =%
)

setlocal enableDelayedExpansion
for %%L in ("!LF!") do for /f "tokens=1,2 delims=," %%A in ("!str:|=%%~L!") do (
  if "!!" == "" endlocal  %= pop SETLOCAL stack only if delayed expansion is enabled to preserve any ! that may be in the strings =%
  echo T1=%%A  T2=%%B
)
If I want to iterate a series of string literals embedded within my source code, then I generally combine the simple FOR with FOR /F. But of course this only works if none of the strings contain * or ?

Code: Select all

@echo off
for %%S in (
  "1,Martini Shaker"
  "2,Wine Bottle"
  "3,Beer Glass"
) do for /f "tokens=1,2 delims=," %%A in ("%%~S") do echo T1=%%A  T2=%%B

Dave Benham

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: FOR /F with Line Feed

#5 Post by aGerman » 04 Aug 2018 07:00

dbenham wrote:
04 Aug 2018 06:45
2) The commas should not require escaping
That was surprising for me, too. And for whatever reason the unescaped spaces work flawlessly even if the commas do not.
Output with unescaped commas (Win10, 32bit):

Code: Select all

T1=1 Martini Shaker T2=
T1=2 Wine Bottle T2=
T1=3 Beer Glass T2=
Drücken Sie eine beliebige Taste . . .
Steffen

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: FOR /F with Line Feed

#6 Post by dbenham » 04 Aug 2018 07:19

Hmmm. That means the parser must be substituting space for the comma, in which case I suspect that any contiguous string of unquoted, unescaped token delimiters will be replaced by a single space.

I knew this was done when FOR /F iterates the output of a command. I thought this was part of packaging up the IN clause for insertion into a CMD /C command. I didn't think it applied to string processing. But apparently it is universal to all FOR /F IN() clauses.

I don't think the SO command parsing rules address this point properly. I'll have to wade through them again.


Dave Benham

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: FOR /F with Line Feed

#7 Post by dbenham » 04 Aug 2018 07:31

Well it is embedded in phase 2 after all - and I wrote that portion :oops:
StackOverflow wrote: FOR is split in two after the DO. A syntax error in the FOR construction will result in a fatal syntax error.
  • The portion through DO is the actual FOR iteration command that flows all the way through phase 7
    • All FOR options are fully parsed in phase 2.
    • The IN parenthesized clause treats <LF> as <space>. After the IN clause is parsed, all tokens are concatenated together to form a single token.
    • Consecutive token delimiters collapse into a single space throughout the FOR command through DO.
  • The portion after DO is a command block that is parsed normally. Subsequent processing of the DO command block is controled by the iteration in phase 7.
I think the bolded line should be modified to state that only unquoted, unescaped token delimiters are affected


Dave Benham

Squashman
Expert
Posts: 4465
Joined: 23 Dec 2011 13:59

Re: FOR /F with Line Feed

#8 Post by Squashman » 04 Aug 2018 07:59

npocmaka_ wrote:
04 Aug 2018 06:38
there was an easier way to the the LF with set and brackets but I cant find the link. If I'm not wrong it was found by Aacini.
Yes. I that is what I thought as well. I kept trying to find it in his previous posts last night.

Squashman
Expert
Posts: 4465
Joined: 23 Dec 2011 13:59

Re: FOR /F with Line Feed

#9 Post by Squashman » 04 Aug 2018 08:04

dbenham wrote:
04 Aug 2018 06:45
If I want to iterate a series of string literals embedded within my source code, then I generally combine the simple FOR with FOR /F. But of course this only works if none of the strings contain * or ?

Code: Select all

@echo off
for %%S in (
  "1,Martini Shaker"
  "2,Wine Bottle"
  "3,Beer Glass"
) do for /f "tokens=1,2 delims=," %%A in ("%%~S") do echo T1=%%A  T2=%%B

Dave Benham
Yes Dave, that is what I am doing already. I should have put that in my original post. For some reason I thought it was possible to do it with a single FOR /F command. I could have sworn Antonio had posted something like I was trying a few years ago. I even remember posting in the thread.

I did get this to work.

Code: Select all

@echo off
setLocal EnableDelayedExpansion
set lf=^


for /F ^"eol^=^

tokens^=^1^,^2^ delims^=^,^" %%G in ("1,A B!lf!2,C D!lf!3,E F") do echo T1=%%G T2=%%H
pause
But I was hoping to list each grouping on its own line because they will be larger then this example and was hoping for some readability in the code.

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: FOR /F with Line Feed

#10 Post by dbenham » 04 Aug 2018 08:54

I also vaguely remember Aacini posting something, but I don't remember where, or what it looked like.

Try this. I still can't test, but I think it should work:

Code: Select all

@echo off
setlocal

(set ;=^^^
%= This creates an escaped Line Feed - DO NOT ALTER =%
)

For /f "usebackq tokens=1,2 delims=," %%A in ('^
1^,line 1%;%
2^,line 2%;%
3^,line 3'
) do echo T1=%%A  T2=%%B
The ; variable should contain ^<LF>

So the IN clause should effectively look like ('^string1^<LF><LF>string2^<LF><LF>string3')

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: FOR /F with Line Feed

#11 Post by aGerman » 04 Aug 2018 09:12

I'm afraid it still doesn't work without escaping the commas, Dave. As you wrote above you don't need to worry about escaping double quotes using USEBACKQ but besides of that it's still basically the same proposal than mine.

Steffen

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: FOR /F with Line Feed

#12 Post by dbenham » 04 Aug 2018 09:39

Doh! Of course. I edited the post.

The other main difference is the use of a simple definition for ; as opposed to a fairly complex definition for \n

Code: Select all

(set LF=^
%= This creates a Line Feed - DO NOT ALTER =%
)
set ^"\n=^^^%LF%%LF%^%LF%%LF%^^"
REM \n = ^<LF><LF>^

vs.

(set ;=^^^
%= This creates an escaped Line Feed - DO NOT ALTER =%
)
REM ; = ^<LF>
But the end result is the same - the IN() clause is parsed as one string with multiple <LF> terminated lines.

One potential advantage is the use of \n escapes the line feed and the first character of the next line, but the use of ; only escapes the line feed.

I'm thinking this simpler syntax could be used for macro definition as well, though I would probably use \n for the variable name instead of ;

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: FOR /F with Line Feed

#13 Post by aGerman » 04 Aug 2018 11:20

Your idea of using your technique for macro definitions sounds great. I remember the original explanation by jeb. He told that the \n variable expands as you said but that it would be required because in the end the two line feeds generate a single line feed the same way as you originally define the LF variable. That seems to be wrong or at least not necessary :o

Hmm, but we are about to hijack Squashman's thread now ...

Squashman
Expert
Posts: 4465
Joined: 23 Dec 2011 13:59

Re: FOR /F with Line Feed

#14 Post by Squashman » 04 Aug 2018 11:26

aGerman wrote:
04 Aug 2018 11:20
Hmm, but we are about to hijack Squashman's thread now ...
All for the greater good.

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: FOR /F with Line Feed

#15 Post by dbenham » 04 Aug 2018 12:22

I'm finally at a PC, and no, it does not work for macros.

This

Code: Select all

(set \n=^
%= This defines an escaped line feed - DO NOT ALTER =%
)

echo Hello%\n%
world!
Is just a more pleasing way of writing

Code: Select all

echo Hello^

world
Both the above only get parsed once. The same is true with the IN() clause in earlier posts. But the macro must be parsed twice, once to define the macro, and again to execute the macro. So the more complicated definition is required to define a macro.

Dave Benham

Post Reply