Newline character as delimiter in for-loop

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Post Reply
Message
Author
Reino
Posts: 23
Joined: 14 May 2015 06:13
Contact:

Newline character as delimiter in for-loop

#1 Post by Reino » 24 May 2015 07:18

I have the following situation:

Code: Select all

youtube-dl.exe -gf 135+141 https://www.youtube.com/watch?v=KpvLMyC4a8o
https://r3---sn-mn4vg5aa-5hns.googlevideo.com/...longurl...
https://r3---sn-mn4vg5aa-5hns.googlevideo.com/...longurl...
When you want youtube-dl to return the videolink of 2 formats of a certain video, it will return the 2nd videolink on a new line.

I have the following test-script:

Code: Select all

@ECHO off

SET youtube-dl="youtube-dl.exe"
SET "url=https://www.youtube.com/watch?v=KpvLMyC4a8o"
SET format=135+141
set LF=^


FOR /F "tokens=1,2" delims=%LF%" %%A IN ('^"%youtube-dl% --no-warnings -gf %format% "%url%"^"') DO (
   SET "_url1=%%A"
   SET "_url2=%%B"
)

ECHO "%_url1%"
ECHO.
ECHO "%_url2%"

PAUSE
I want to put each videolink in a variable, but the linefeed is posing a problem.
Some weeks ago while searching for another issue I stumbled upon a website which showed that with a trick the linefeed character could be used as a delimiter in a for-loop. The problem is, that despite lots of Google-queries, I just can't find that website anymore.
What I do still remember from that trick was that the double quotes surrounding "tokens= delims=" were replaced by lots of carets.
Does anyone here know how to use a newline character as a delimiter in a for-loop?

Ed Dyreen
Expert
Posts: 1569
Joined: 16 May 2011 08:21
Location: Flanders(Belgium)
Contact:

Re: Newline character as delimiter in for-loop

#2 Post by Ed Dyreen » 24 May 2015 10:05

Setting delims to a linefeed has no advantage over simply disabling eol and delims because 'For' can't handle linefeeds.

Code: Select all

@echo off &setlocal enableDelayedExpansion

for /F delims^=^ eol^= %%r in ('ping') DO set /A line[0] += 1 &set "line[!line[0]!]=%%r"

set line

pause
exit

Reino
Posts: 23
Joined: 14 May 2015 06:13
Contact:

Re: Newline character as delimiter in for-loop

#3 Post by Reino » 24 May 2015 14:27

This isn't about ignoring lines with eol=x, but the other way around, so I can't use your code. I want to create a singleline of youtube-dl's multiline output, by using CRLF as a delimiter, so I can "tokenize" it into 2 variables. You're saying I can't?

I've tried...

Code: Select all

FOR /F "tokens=1,2 delims=¶" %%A IN ('^"%youtube-dl% --no-warnings -gf %format% "%url%"^"') DO (
and...

Code: Select all

set LF=^


FOR /F tokens^=1^,2^ delims^=^%LF%^ %%A IN ('^"%youtube-dl% --no-warnings -gf %format% "%url%"^"') DO (
but both to no avail.

In the meantime the following temporary solution would have to do:

Code: Select all

@ECHO off

SET youtube-dl="youtube-dl.exe"
SET jq="jq.exe"
SET "url=https://www.youtube.com/watch?v=KpvLMyC4a8o"
SET format=135+141

FOR /F %%A IN ('^"%youtube-dl% --no-warnings -gf %format% "%url%" ^| %jq% -R -s .^"') DO SET "_url=%%A"
FOR /F "tokens=1,2" %%A IN (%_url:\n= %) DO SET "_url1=%%A" & SET "_url2=%%B"

ECHO "%_url1%"
ECHO.
ECHO "%_url2%"

ShadowThief
Expert
Posts: 1160
Joined: 06 Sep 2013 21:28
Location: Virginia, United States

Re: Newline character as delimiter in for-loop

#4 Post by ShadowThief » 24 May 2015 15:08

Why can't you send the output of the command to a temp file and then process the output from there?

Code: Select all

%youtube-dl% --nowarnings -gf %format "%url%" >output.txt

(
    set /p ignore_line=
    set /p _url1=
    set /p _url2=
)<output.txt
del output.txt

Reino
Posts: 23
Joined: 14 May 2015 06:13
Contact:

Re: Newline character as delimiter in for-loop

#5 Post by Reino » 24 May 2015 15:42

Sorry, but I refuse to use temp files.

Ed Dyreen
Expert
Posts: 1569
Joined: 16 May 2011 08:21
Location: Flanders(Belgium)
Contact:

Re: Newline character as delimiter in for-loop

#6 Post by Ed Dyreen » 24 May 2015 15:59

This isn't about ignoring lines with eol=x, but the other way around, so I can't use your code. I want to create a singleline of youtube-dl's multiline output, by using CRLF as a delimiter, so I can "tokenize" it into 2 variables. You're saying I can't?
My code does not ignore any lines with eol=x, in fact both eol and delims are disabled. For always splits when it encounters a linefeed, therefore the effect is exactly the same as what you get when you set 'delims=^%$lf%%$lf%'. And because there is no difference, setting delims to a lineFeed is depriciated. Because for can't handle lineFeeds, your attempt to tokenize multiLine output inevitably fails.

Code: Select all

@echo off &setlocal enableDelayedExpansion &set $lf=^


::
set "$=echo.https://longurlA...&echo.https://longurlB..."

echo. &echo.with eol and delims disabled;
for /F delims^=^ eol^= %%r in ( '!$!' ) DO set /A url[0] += 1 &set "url[!url[0]!]=%%r"
for /L %%i in ( 1, 1, !url[0]! ) do echo.url[%%i]=!url[%%i]!_ &set "url[%%i]="
set /A url[0] = 0

echo. &echo.with eol disabled and delims=lineFeed;
for /F delims^=^%$lf%%$lf%^ eol^= %%r in ( '!$!' ) DO set /A url[0] += 1 &set "url[!url[0]!]=%%r"
for /L %%i in ( 1, 1, !url[0]! ) do echo.url[%%i]=!url[%%i]!_ &set "url[%%i]="
set /A url[0] = 0

pause
exit
Each line is added to an array and you can create a single variable out of it by iterating the array.

Code: Select all


with eol and delims disabled;
url[1]=https://longurlA..._
url[2]=https://longurlB..._

with eol disabled and delims=lineFeed;
url[1]=https://longurlA..._
url[2]=https://longurlB..._
Druk op een toets om door te gaan. . .
you can also use a temp file as ShadowThief suggest.

Aacini
Expert
Posts: 1885
Joined: 06 Dec 2011 22:15
Location: México City, México
Contact:

Re: Newline character as delimiter in for-loop

#7 Post by Aacini » 24 May 2015 16:28

Try this:

Code: Select all

SET "_url1="
FOR /F "delims=" %%A IN ('^"%youtube-dl% --no-warnings -gf %format% "%url%"^"') DO (
   IF NOT DEFINED _url1 (
      SET "_url1=%%A"
   ) ELSE (
      SET "_url2=%%A"
   )
)

Antonio

Reino
Posts: 23
Joined: 14 May 2015 06:13
Contact:

Re: Newline character as delimiter in for-loop

#8 Post by Reino » 24 May 2015 17:43

I'm sorry, Ed Dyreen. I didn't understand your code at first.

Code: Select all

@ECHO off

SET youtube-dl="youtube-dl.exe"
SET "url=https://www.youtube.com/watch?v=KpvLMyC4a8o"
SET format=135+141

SETLOCAL ENABLEDELAYEDEXPANSION
FOR /F %%A IN ('^"%youtube-dl% --no-warnings -gf %format% "%url%"^"') DO (
   SET /A _url[0] += 1
   SET "_url[!_url[0]!]=%%A"
)

ECHO "%_url[1]%"
IF DEFINED _url[2] (
   ECHO.
   ECHO "%_url[2]%"
)
ENDLOCAL
This works. BUT, I'm going for Antonio's more straight forward code:

Code: Select all

@ECHO off

SET youtube-dl="youtube-dl.exe"
SET "url=https://www.youtube.com/watch?v=KpvLMyC4a8o"
SET format=135+141

SET "_url1="
FOR /F %%A IN ('^"%youtube-dl% --no-warnings -gf %format% "%url%"^"') DO (
   IF NOT DEFINED _url1 (
      SET "_url1=%%A"
   ) ELSE (
      SET "_url2=%%A"
   )
)

ECHO "%_url1%"
IF DEFINED _url2 (
   ECHO.
   ECHO "%_url2%"
)
It makes perfect use of the for-loop principle; During the first iteration the first line of youtube-dl's output is processed and assigned to _url1. During the second iteration the second line is processed, and assigned to _url2, because _url1 is already defined. Code I can better understand.

Thanks a lot for your help you two.

foxidrive
Expert
Posts: 6031
Joined: 10 Feb 2012 02:20

Re: Newline character as delimiter in for-loop

#9 Post by foxidrive » 25 May 2015 01:44

CoRoNe wrote:Sorry, but I refuse to use temp files.


Back in the dark ages of batch files temp files were used so very often, because they were absolutely necessary.

Many people say "no temp files" these days but they seem to miss the point that Windows uses
temporary files all day long.

I think a thank you to ShadowThief should be in your post, also.

Reino
Posts: 23
Joined: 14 May 2015 06:13
Contact:

Re: Newline character as delimiter in for-loop

#10 Post by Reino » 25 May 2015 02:21

Perhaps that was a little bit harsh, yes.
ShadowThief, thanks for your help too, but I still won't be using temp files. :wink:

foxidrive
Expert
Posts: 6031
Joined: 10 Feb 2012 02:20

Re: Newline character as delimiter in for-loop

#11 Post by foxidrive » 25 May 2015 06:39

CoRoNe wrote:ShadowThief, thanks for your help too, but I still won't be using temp files. :wink:


:D

One point wrt temp files is that when they are in an inner loop and speed is paramount, and it is badly affected by temp files,
then there is a case for writing extra code to get around it.

However there are cases where a temp file will speed up the batch file enormously - in particular a bug with for /f where a huge
number of files being parsed,or with long names, can lead to 30 or more minutes before it starts to do anything.

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: Newline character as delimiter in for-loop

#12 Post by dbenham » 25 May 2015 09:04

CoRoNe wrote:Sorry, but I refuse to use temp files.

That position can lead to a world of hurt. There are multiple scenarios where the use of one or more temp files is MUCH faster than any other batch alternative. One example is processing a command that produces lots of output. It is much faster to redirect the output to a temp file and process the file with FOR /F. Processing the command output directly with FOR /F is painfully slow if the output is really large.


Dave Benham

Post Reply