Using many "tokens=..." in FOR /F command in a simple way

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Message
Author
Aacini
Expert
Posts: 1885
Joined: 06 Dec 2011 22:15
Location: México City, México
Contact:

Using many "tokens=..." in FOR /F command in a simple way

#1 Post by Aacini » 07 Feb 2017 12:40

EDIT: When this topic was created there was a problem with the FOR /F command: the characters in 128..255 extended range can not be used as FOR /F replaceable parameters in a standard way, so the first version of this application just worked with a maximum of 208 tokens and only when the active code page was 437. After a common effort from a group of people, that generated a series of tests and new concepts, a new method to access FOR /F replaceable parameters that have no limits in the number of tokens nor in the active code page was developed; you may read about the development of this method in the posts below. The second version of this application use the new method and can process up to 4094 tokens from the same line; you may review and download it at this post.

The purpose of this topic is assemble a method to use more tokens in a FOR /F command, as much as possible, but in a simple way. I read the description about this management in this SO post, but the standard method is complicated: it requires an Extended ASCII table to know the characters that must be used for each different token and there are several special cases that needs to be managed in a different way.

The Batch code below is an example of a general-use method that combine a series of chained FOR /F commands in a way that returns lines with many tokens, up to 208, from a text file; the way to specify the tokens is via a string similar to the original "tokens=x,y,m-n" one, but that allows token numbers up to 208. Another advantage is that the new tokens string allows to repeat tokens numbers or put a tokens range in descending order; the output will contain the same tokens specified in the tokens string. For example, using this text file as base:

Code: Select all

 A1 A2 A3 A4  ...  A177 A178 A179 A180
 B1 B2 B3 B4  ...  B177 B178 B179 B180
 C1 C2 C3 C4  ...  C177 C178 C179 C180

... these are a couple examples of the program output:

Code: Select all

tokens=1,20,45,75,120
 A1 A20 A45 A75 A120
 B1 B20 B45 B75 B120
 C1 C20 C45 C75 C120

tokens=30,28-32,170-165
 A30 A28 A29 A30 A31 A32 A170 A169 A168 A167 A166 A165
 B30 B28 B29 B30 B31 B32 B170 B169 B168 B167 B166 B165
 C30 C28 C29 C30 C31 C32 C170 C169 C168 C167 C166 C165

The Batch file is in this .zip file:

FOR-F with many tokens.zip
(1.37 KiB) Downloaded 1076 times

You may also add the other options to the series of FOR /F commands, like usebackq, delims, etc. The only not supported option in the new tokens string is an asterisk at end to get the rest of tokens. This point could also be implemented, but it would required an additional processing of the result line that is obtained from each one of the file lines. The method used to extract the tokens is very efficient: after prepare an equivalent "tokensValues" string based on the original tokens one, it uses one CALL command to extract all tokens from each input line via a single SET command.

Important: When I started to do tests on this subject in my computer, the extended characters used as tokens just not worked. For example, in this command:

Code: Select all

for /F "tokens=1-5" %á in ("A B C D E") do echo "%á %í %ó %ú %ñ"

... the expected output is "A B C D E", but I got "A %í %ó %ú %ñ" instead and the same happened with any other successive characters taken from the extended set in 437 code page order: just the first token show the first value, the rest of tokens never got their values.

After many tests I discovered that the order of the extended characters in 128..255 range used as successive tokens in FOR /F command in my Windows 8.1 Spanish version was not the standard numerical order, but a very different order that I could establish with the aid of a program. The chain formed by these characters does not cover the full 128 characters in 128..255 range, but just a chain with 95 characters, two small chains with 2 characters each and the rest of characters remain isolated. For this reason, the Spanish version of this program can manage just a maximum of 177 tokens, instead of 208, and the following two sections must be changed in the original code:

Code: Select all

rem Create 95 characters for 3 FOR's with "tokens=1-31*"
rem This is the tokens sequence used in Windows 8.1 Spanish
set "i=0"
for %%i in (173 189 156 207 190 221 245 249 184 166 174 170 240 169 238 248
            241 253 252 239 230 244 250 247 251 167 175 172 171 243 168 183
            181 182 199 142 143 146 128 212 144 210 211 222 214 215 216 209
            165 227 224 226 229 153 158 157 235 233 234 154 237 232 225 133
            160 131 198 132 134 145 135 138 130 136 137 141 161 140 139 208
            164 149 162 147 228 148 246 155 151 163 150 129 236 231 152    ) do (
   set /A i+=1, mod=i%%32
   if !mod! neq 0 (
      call :genchr %%i
      type %%i.chr
      del %%i.chr
   )
)
) > FOR-Fchars.txt
del t.tmp temp.tmp
set "options="
:readChars
set /P "char=" < FOR-Fchars.txt
set "lastToken=177"


Code: Select all

rem First three FOR's use as tokens the ASCII chars in 38..124 (&..|) range: 28*3 = 84 tokens + 3 tokens for next FOR
rem Next three FOR's use as tokens Extended chars: 31*3 = 93 tokens + 2 tokens for next FOR
rem based on the tokens sequence used in Windows 8.1 Spanish
rem Total: 177 tokens

for /F "eol= tokens=1-28*" %%^& in (test.txt) do ^
for /F "eol= tokens=1-28*" %%C in ("%%B") do ^
for /F "eol= tokens=1-28*" %%` in ("%%_") do ^
for /F "eol= tokens=1-31*" %%­ in ("%%|") do ^
for /F "eol= tokens=1-31*" %%µ in ("%%·") do ^
for /F "eol= tokens=1-31"  %%  in ("%%…") do (
   call :getTokens result=
   rem Process here the "result" string
   echo !result!
)
goto nextSet

This point means that I have NOT tested the code I posted at beginning, because I have not access to any PC with Windows English version right now. I assume that the program should work based on the comments given by jeb, Dave and npocmaka on the referred SO answer about that all these characters works (excepting 0xFF, that I don't use), but I still could made an error in the characters I used in the last four chained FOR /F commands. I'll appreciate it if someone could test the original code in any Windows English version and confirm if it works. I also suppose that the sequence of succesive characters will be different in Windows versions in other languages...

Antonio

pieh-ejdsch
Posts: 239
Joined: 04 Mar 2014 11:14
Location: germany

Re: Using many "tokens=..." in FOR /F command in a simple way

#2 Post by pieh-ejdsch » 07 Feb 2017 16:39

Hello Antonio,
Hello Antonio, I still have not managed more than thirty-two to process tokens simultaneously in the output in this situation. In and of itself only the input on this token number has to be minimized.
To get some more tokens out of the file this should go with it.

https://www.administrator.de/contentid/191890#comment-784537

It may be possible to re-enter an environment variable using an extra loop.
I test your version tomorrow

Phil

Aacini
Expert
Posts: 1885
Joined: 06 Dec 2011 22:15
Location: México City, México
Contact:

Re: Using many "tokens=..." in FOR /F command in a simple way

#3 Post by Aacini » 18 Feb 2017 09:29

Did anybody tested this program in an English Windows version? Or in any other language? Could somebody post the result, please? :roll: Thanks! :wink:

Antonio

Thor
Posts: 43
Joined: 31 Mar 2016 15:02

Re: Using many "tokens=..." in FOR /F command in a simple way

#4 Post by Thor » 18 Feb 2017 11:26

Hi Antonio,

I've run your batch file and at the "tokens=" prompt, I've entered "21" (without quotes) and I've got an error like this:
tokens=21
% was unexpected at this time.

Aacini
Expert
Posts: 1885
Joined: 06 Dec 2011 22:15
Location: México City, México
Contact:

Re: Using many "tokens=..." in FOR /F command in a simple way

#5 Post by Aacini » 18 Feb 2017 13:43

@Thor: Thanks a lot for your answer! :D


I downloaded the code from this site and it marked the same error, so I compared my original code vs. the downloaded one and got this result:

Code: Select all

C:\Users\Antonio\Documents\tests> fc /N Original.bat Downloaded.bat
Comparando archivos Original.bat y Downloaded.bat
***** Original.bat
   81:  for /F "eol= tokens=1-31*" %%Ç in ("%%|") do ^
   82:  for /F "eol= tokens=1-31*" %%á in ("%%ƒ") do ^
   83:  for /F "eol= tokens=1-31*" %%└ in ("%%┐") do ^
***** Downloaded.bat
   81:  for /F "eol= tokens=1-31*" %%Ç in ("%%|") do ^
   82:  for /F "eol= tokens=1-31*" %%  in ("%%ƒ") do ^
   83:  for /F "eol= tokens=1-31*" %%└ in ("%%┐") do ^
*****

This means that this error is caused by the web browser, that change the character used in the FOR command at line # 82. This char must be an ASCII # 160, so I manually changed it in the downloaded code and after that the FC command mark "No differences encountered".

This problem can be fixed posting the program in a .zip file, so I did that, but I still need a confirmation that the code can correctly get up to 208 tokens when run in an English version of Windows. Could you download the Batch file again from the .zip file and test it, please?

Antonio

Thor
Posts: 43
Joined: 31 Mar 2016 15:02

Re: Using many "tokens=..." in FOR /F command in a simple way

#6 Post by Thor » 18 Feb 2017 14:07

Hi Antonio,

Good news, it runs now :-)

But I've got the following results:

tokens=21
A21
B21
C21

tokens=51
A51
B51
C51

tokens=81
A81
B81
C81

tokens=85
A85
B85
C85

tokens=86
ECHO is off.
ECHO is off.
ECHO is off.

tokens=98
ECHO is off.
ECHO is off.
ECHO is off.

tokens=99




tokens=100




tokens=202




tokens=203
%∙
%∙
%∙

tokens=204




tokens=205
%√
%√
%√

tokens=206
%ⁿ
%ⁿ
%ⁿ

tokens=207




tokens=208
%■
%■
%■

d:\6>

siberia-man
Posts: 208
Joined: 26 Dec 2013 09:28
Contact:

Re: Using many "tokens=..." in FOR /F command in a simple way

#7 Post by siberia-man » 19 Feb 2017 03:05

I am not sure if you expect to see these.
Attachments
1.jpg
1.jpg (62.96 KiB) Viewed 28374 times

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: Using many "tokens=..." in FOR /F command in a simple way

#8 Post by aGerman » 19 Feb 2017 06:27

Downloaded the zip file...

I took the same examples like you Antonio.

Code: Select all

tokens=1,20,45,75,120
 A1 A20 A45 A75
 B1 B20 B45 B75
 C1 C20 C45 C75

tokens=30,28-32,170-165
 A30 A28 A29 A30 A31 A32 A92 A91 %ı A86 A89 A88
 B30 B28 B29 B30 B31 B32 B92 B91 %ı B86 B89 B88
 C30 C28 C29 C30 C31 C32 C92 C91 %ı C86 C89 C88

tokens=


This is how the FOR loops look like in my text editor (code page Windows-1252).
1252.PNG
1252.PNG (29.57 KiB) Viewed 28414 times


That's the same sequence in a HEX editor.
HEX.PNG
HEX.PNG (93.25 KiB) Viewed 28414 times


Steffen

// EDIT
System informations:

Code: Select all

 INFO.BAT version 1.4
--------------------------------------------------------------------------------
Windows version        :  Microsoft Windows [Version 10.0.14393]
Product name           :  Windows 10 Home, 32 bit
Performance indicators :  Processor Cores: 4      Visible RAM: 2023612 kilobytes

Date/Time format       :  (dd/mm/yy)  20.02.2017   0:39:35,82
__APPDIR__             :  C:\WINDOWS\system32\
ComSpec                :  C:\WINDOWS\system32\cmd.exe
PathExt                :  .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
Extensions             :  system: Enabled   user: Enabled
Delayed expansion      :  system: Disabled  user: Disabled
Locale name            :  de-DE       Code Pages: OEM  850    ANSI 1252
DIR  format            :  20.02.2017  00:00     1.466.490.880 pagefile.sys
Permissions            :  Elevated Admin=No, Admin group=Yes


pieh-ejdsch
Posts: 239
Joined: 04 Mar 2014 11:14
Location: germany

Re: Using many "tokens=..." in FOR /F command in a simple way

#9 Post by pieh-ejdsch » 19 Feb 2017 15:37

I have made an other version. It will sort all the Tokens with topologic sort.
This will give acess over 190 tokens and full tokens over 207. This runs on every Windows.
Next time I would have more tokens in deep of string.
Phil

Aacini
Expert
Posts: 1885
Joined: 06 Dec 2011 22:15
Location: México City, México
Contact:

Re: Using many "tokens=..." in FOR /F command in a simple way

#10 Post by Aacini » 19 Feb 2017 17:30

Thank you @Thor and @siberia-man for your tests, and your hexadecimal listing is particularly valuable @aGerman!

In the listing below I added the number of the Extended ASCII characters that must appear in each one of the last four FOR commands, with the value in hex enclosed in parentheses:

Code: Select all

for /F "eol= tokens=1-28*" %%^& in (test.txt) do ^
for /F "eol= tokens=1-28*" %%C in ("%%B") do ^
for /F "eol= tokens=1-28*" %%` in ("%%_") do ^
for /F "eol= tokens=1-31*" %%€ in ("%%|") do ^          128 (80)  in  "%%|"
for /F "eol= tokens=1-31*" %%  in ("%%Ÿ") do ^          160 (A0)  in  159 (9F)
for /F "eol= tokens=1-31*" %%À in ("%%¿") do ^          192 (C0)  in  191 (BF)
for /F "eol= tokens=1-31"  %%à in ("%%ß") do (          224 (E0)  in  223 (DF)

The aGerman's hex listing above proves that all values are correct! These values appear at addresses 92A=80, 95A=A0, 963=9F, 98A=C0, 993=BF, 9BA=E0 and 9C3=DF. This means that this program should work as long as FOR tokens follow the 128..254 order when characters beyond ASCII 127 are used.

I know that FOR tokens order in Windows 8.1 Spanish does NOT follow the standard 128..254 order, but I have not seen a single test about tokens order in any other Windows version. Please, when post the result of this program also include the Windows version.

@pieh-ejdsch: Please, just post the tokens order you used... How do you know that your method works on every Windows version?

Antonio

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: Using many "tokens=..." in FOR /F command in a simple way

#11 Post by aGerman » 19 Feb 2017 17:49

Aacini wrote:Please, when post the result of this program also include the Windows version.

I added the output of info.bat to my previous answer.

Do you know of a simple way to figure out what the order of FOR variables is for characters >127 ?

Steffen

pieh-ejdsch
Posts: 239
Joined: 04 Mar 2014 11:14
Location: germany

Re: Using many "tokens=..." in FOR /F command in a simple way

#12 Post by pieh-ejdsch » 19 Feb 2017 22:29

Excuse me, that I so scripted in your script.
I'm posting the unfinished part for now.
This is unofficially the third change.
'Please, just post the tokens order' - i cannot do that. :mrgreen: It is generated automatically.

Code: Select all

@echo off
set prompt=$G$S
setlocal enabledelayedexpansion

set exclude="," ";" "=" "   "
 rem Load the string of characters used for tokens from FOR-Fchars.txt file

echo Creating characters file, please wait
set "options=/d compress=off /d reserveperdatablocksize=26"
type nul > t.tmp


set /aAllsign=lasttoken=0 ,o=0,f=s=100
set "T.String=."

for /f "delims==" %%i in ('2^>nul set uniqe.') do set "%%i="
 rem linelength
for /f tokens^=3 %%l in ('mode con^|find  "  "^|find /n "  "^|find "[2]"')do set /aLL=%%l-1
3>AllFor-Chars.txt call :CreateAllCH

if :SUB==only * &(
 :CreateAllCH
set /a o=p=lp=0 ,max=254
set "T.String="
@ <nul >&3 set /P "=0"
for /L %%i in (1,1,%max%)do (
  call;
  if %%i geq 7  if %%i leq 13 (call)
  if %%i geq 32 if %%i leq 37 (call)
  if %%i equ 44 (call)
  if %%i geq 59 if %%i leq 61 (call)
  if %%i equ 94 (call)
  if %%i equ 126 (call)
  if NOT errorlevel 1 if %%i lss 255 (
    call :genchr %%i
    >&3 type %%i.chr
    set /ao+=1
    for /f usebackQdelims^=^ eol^= %%. in ("%%i.chr")do (
      set "sign=%%."
      set "ALLsign=!ALLsign!%%."
       rem use uniqe.number in topologic sort
      set "uniqe.!o!=begin{!o!}END"
      set "T.String=!T.String!%%^%%.!o!%%^%%. "
    )
    del %%i.chr
  )
  set /a P= %%i *LL /max
  if !P! neq !LP! <nul set/p=.
  set /a lp=p
  if %%i == %max% Echo(
)
exit /b
)

echo Sort new TokenLists
set /a nn=0, token=2
@set tok.list=1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

 rem prepare pbar
set /a p=lp=0 ,max=o ,tok.total=0
:testtokens
 rem topologic sort
set /ann+=1
set "Var=!ALLsign:~%NN%,1!"
if not defined var goto :queue
:pbar
  set /a P= nn *LL /max
  if !P! neq !LP! <nul set/p=.
  set /a lp=p
  if %nn% == %max% Echo(
for %%i in (%exclude%)do @if "%var%"=="%%~i" goto :testtokens
for /f "tokens=1,%token%" %%^%var% in ("!tok.list:%token%=""!")do (
  for /f tokens^=2delims^=^" %%# in ("%T.String%")do (
    set "uniqe.%nn%=begin{%nn%}uniqe.%%#"
  )
)
goto :testtokens
:queue
set "next="
for /f "tokens=2,4-6 delims=.={}" %%A in ('2^>nul set uniqe.'
)do if defined uniqe.%%A if NOT %%C == END (
  set /a next=1 ,tok.total+=1
  set "uniqe.%%A=begin{%%B !%%C.%%D:*{=!"
  set "%%C.%%D="
)
if defined next goto :queue
 rem set uniqe.
 set /a tok.All=0
set /a list=0
for /f "tokens=2,4 delims=.={}" %%A in ('2^>nul set uniqe.')do (
  if %%A neq %%B (
    set /a list+=1
    set "listN="
    for %%# in (%%B) do set /a tok.ALL+=1 &set "listN=!listN!!ALLsign:~%%#,1!"
    set "list.!list!=!listN!"
  )
)
  set list.
set /a ov=tok.all *100 /tok.total -100
echo total Tokens = %tok.total% / %tok.all%  overhead = %ov% %%
pause
 rem exit /b


 rem section proof of concept is to make automatet Tokens
 rem this is a old part wrks in Antonios Batch

set /aAllsign=lasttoken=0 ,o=0,f=s=100
set "T.String=."
set "mod="
 rem Create 87 characters in 38..125 range for 2 FOR's with "tokens=1-27*"
:3>> FOR-Fchars.txt call :CreateCH 28 38 2
:3>> FOR-Fchars.txt call :CreateCH 31 95 1
 rem Create 127 characters in 128..254 range for 3 FOR's with "tokens=1-31*" plus one more with "tokens=1-31"
3>> FOR-Fchars.txt call :CreateCH 32 1 8

if :SUB==only * &(
:CreateCH
set /a E =%1 *%3 +%2-1, i=0
set "in.set=1"
for /L %%i in (%2,1,%E%) do (
  call;
  if %%i geq 7  if %%i leq 13 (call)
  if %%i geq 32 if %%i leq 37 (call)
  if %%i equ 44 (call)
  if %%i geq 59 if %%i leq 61 (call)
  if %%i equ 94 (call)
  if %%i equ 126 (call)
  if NOT errorlevel 1 if %%i lss 255 (
    if !mod! neq 0 if !i! == -1 (set /a i=0
      for %%N in (!s:~1!)do (
        set /a ftok=!fTOK%%N!
        if !fTOK!==1 (set "fTOK%%N=*"
        )else set "fTOK%%N=1!fTOK!*"
      )
      set/as+=1
      set "fSET!s:~1!=%%!ALLsign:~-1!"
    )
    set /A i+=1, mod=i%%%1 ,o+=1 ,next=o+1, in.set=1
    call :genchr %%i
    for /f Delims^=^ eol^= %%. in ('type %%i.chr')do (set "sign=%%."
      echo %%i %%. !mod! !o!
      set "ALLsign=!ALLsign!%%."
      set "T.String=!T.String!%%^%%.!o!%%^%%. "
      set "uniqe.!o!=begin{!o!}END"
      if !mod! neq 0 (
        set /a lastToken+=1
        >&3 type %%i.chr
        if !mod! equ 1 (
          set "fVAR!f:~1!=%%^^%%."
          set/af+=1
          set "ftok!s:~1!=1"
        )else set "ftok!s:~1!=1-!mod!"
      )else (
        for %%N in (!s:~1!)do set "fTOK%%N=!fTOK%%N!*"
        set/as+=1
        set "fSET!s:~1!=%%%%."
      )
    )
    del %%i.chr
  )
  if not defined in.set set /a i=-1
  set "IN.set="
)
exit /b
)



for /F "eol= tokens=%ftok00%" %fVAR00% in (test.txt) do ^
for /F "eol= tokens=%ftok01%" %fVAR01% in ("%fSET01%") do ^
for /F "eol= tokens=%ftok02%" %fVAR02% in ("%fSET02%") do ^
for /F "eol= tokens=%ftok03%" %fVAR03% in ("%fSET03%") do ^
for /F "eol= tokens=%ftok04%" %fVAR04% in ("%fSET04%") do ^
for /F "eol= tokens=%ftok05%" %fVAR05% in ("%fSET05%") do ^
for /F "eol= tokens=%ftok06%" %fVAR06% in ("%fSET06%") do (
   echo %tokensvalues%
)

end Proof of concept
next part for automatet Numbers

if :SUB==only * &(
:CreateCH
set /a E =%1 *%3 +%2-1, i=0
set "in.set=1"
for /L %%i in (%2,1,%E%) do (
  call;
  if %%i geq 7  if %%i leq 13 (call)
  if %%i geq 32 if %%i leq 37 (call)
  if %%i equ 44 (call)
  if %%i geq 59 if %%i leq 61 (call)
  if %%i equ 94 (call)
  if %%i equ 126 (call)
  if NOT errorlevel 1 if %%i lss 255 (
    if !mod! neq 0 if !i! == -1 (set /a i=0
      for %%N in (!s:~1!)do (
        set /a ftok=!fTOK%%N!
        if !fTOK!==1 (set "fTOK%%N=*"
        )else set "fTOK%%N=1!fTOK!*"
      )
      set/as+=1
      set "fSET!s:~1!=%%!ALLsign:~-1!"
    )
    set /A i+=1, mod=i%%%1 ,o+=1 ,next=o+1, in.set=1
    call :genchr %%i
     rem make forlists
(set "sign=!"
      echo %%i %%. !mod! !o!
      set "ALLsign=!ALLsign!%%."
      set "T.String=!T.String!%%^%%.!o!%%^%%. "
      set "uniqe.!o!=begin{!o!}END"
      if !mod! neq 0 (
        set /a lastToken+=1
        >&3 type %%i.chr
        if !mod! equ 1 (
          set "fVAR!f:~1!=%%^^%%."
          set/af+=1
          set "ftok!s:~1!=1"
        )else set "ftok!s:~1!=1-!mod!"
      )else (
        for %%N in (!s:~1!)do set "fTOK%%N=!fTOK%%N!*"
        set/as+=1
        set "fSET!s:~1!=%%%%."
      )
    )
    del %%i.chr
  )
  if not defined in.set set /a i=-1
  set "IN.set="
)
exit /b
)

pause

del t.tmp temp.tmp
set "options="

rem echo !CHAR_list!

:readChars
set /P "char=" < FOR-Fchars.txt
echo !char_list!
echo(
echo !char!
echo(
echo !char_last!
echo(
echo !char_next!
:next_Vars
set fset
set fvar
set ftok
set Allsign
set T.String
:cls
:nextSet
echo/
set /P "tokens=tokens="
if errorlevel 1 goto :EOF

rem Expand the given tokens string into a series of individual FOR tokens values
set "tokensValues="
for %%t in (%tokens%) do (
   for /F "tokens=1,2 delims=-" %%i in ("%%t-%%t") do (
     if %%i leq %%j (
         for /L %%n in (%%i,1,%%j) do if %%n leq %o% set tokensValues=!tokensValues!%%n-%%^^^^^!allsign:~%%n,1!-
      ) else (
         for /L %%n in (%%i,-1,%%j) do if %%n leq %lastToken% set "tokensValues=!tokensValues!%%^^!char:~%%n,1! "
      )
   )
)
echo !tokensvalues!


pause


echo on
rem First three FOR's use as tokens the ASCII chars in 38..124 (&..|) range: 28*3 = 84 tokens + 3 tokens for next FOR
rem Next four FOR's use as tokens the Extended chars in 128..254 range: 31*4 = 124 tokens + 3 tokens for next FOR
rem Total: 208 tokens

for /F "eol= tokens=%ftok00%" %fVAR00% in (test.txt) do ^
for /F "eol= tokens=%ftok01%" %fVAR01% in ("%fSET01%") do ^
for /F "eol= tokens=%ftok02%" %fVAR02% in ("%fSET02%") do ^
for /F "eol= tokens=%ftok03%" %fVAR03% in ("%fSET03%") do ^
for /F "eol= tokens=%ftok04%" %fVAR04% in ("%fSET04%") do ^
for /F "eol= tokens=%ftok05%" %fVAR05% in ("%fSET05%") do ^
for /F "eol= tokens=%ftok06%" %fVAR06% in ("%fSET06%") do (
   echo %tokensvalues%
)
echo off
goto nextSet
   @rem call :getTokens result=
   @rem Process here the "result" string
   @rem echo %tokensvalues%


:getTokens result
for %%# in (-) do set "%1=%tokensValues%"
exit /B


REM This code creates one single byte. Parameter: int
REM Teamwork of carlos, penpen, aGerman, dbenham
REM Tested under Win2000, XP, Win7, Win8
:genchr
if %~1 neq 26 (
   makecab %options% /d reserveperfoldersize=%~1 t.tmp %~1.chr > nul
   type %~1.chr | ( (for /l %%N in (1,1,38) do pause)>nul & findstr "^" > temp.tmp )
   >nul copy /y temp.tmp /a %~1.chr /b
) else (
   copy /y nul + nul /a 26.chr /a >nul
)
goto :eof


Now I have to go but on work.
Phil

Aacini
Expert
Posts: 1885
Joined: 06 Dec 2011 22:15
Location: México City, México
Contact:

Re: Using many "tokens=..." in FOR /F command in a simple way

#13 Post by Aacini » 19 Feb 2017 23:44

aGerman wrote:Do you know of a simple way to figure out what the order of FOR variables is for characters >127 ?

Steffen


Not a simple way. I used the Batch file below to get the tokens order in my Windows 8.1 Spanish, but you need to run the program several times varying the initial value based on the results of previous runs until the longest possible thread is obtained. Use 128 for the first run. The results are stored in output.txt file.

Code: Select all

@echo off
if "%~1" neq "" goto %1
setlocal EnableDelayedExpansion

cls
echo Creating the "char" array, please wait. . .
for /L %%i in (128,1,255) do (
   call :genchr %%i
   set /P "char[%%i]=" < %%i.chr
   del %%i.chr
)

echo Ready, start searching...
del output.txt 2>NUL

set /P "nextToken=nextToken: "

set thread=0
:nextThread
set /A thread+=1
(
echo/
echo Thread %thread%:
set /P "=%nextToken% " < NUL
) >> output.txt

:nextThisToken
set "thisToken=%nextToken%"
echo/
if not defined char[%thisToken%] echo End of process & goto :EOF
if %thisToken% equ 255 echo End of process & goto :EOF
echo Searching this token: %thisToken% = "!char[%thisToken%]!"
echo for /F "tokens=1,2" %%%%!char[%thisToken%]! in ("This Next") do call "%~nx0" getNextToken > temp.bat
call temp.bat
set "char[%thisToken%]="
if defined nextTokenValue goto nextThisToken
set "nextToken="
for /F "tokens=2 delims=[]" %%a in ('set char[ 2^>NUL') do set "nextToken=%%a" & goto continue
:continue
if defined nextToken goto nextThread
echo/
echo ==============
echo Terminated
goto :EOF


:getNextToken
set "nextToken=%thisToken%"
:nextToken
set "nextTokenValue="
set /A nextToken+=1
if %nextToken% gtr 255 set "nextToken=128"
if %nextToken% equ %thisToken% echo  Not found & exit /B
if not defined char[%nextToken%] goto nextToken
set /P "=%nextToken%, " < NUL
call :getNextTokenValue !char[%nextToken%]!
if "%nextTokenValue%" neq "Next" goto nextToken
echo  Found: %nextToken%
set /P "=%nextToken% " < NUL >> output.txt
exit /B


:getNextTokenValue
for %%a in (-) do set "nextTokenValue=%%%1"
exit /B



:genchr
REM This code creates one single byte. Parameter: int
REM Teamwork of carlos, penpen, aGerman, dbenham
REM Tested under Win2000, XP, Win7, Win8
set "options=/d compress=off /d reserveperdatablocksize=26"
if %~1 neq 26 (
   type nul > t.tmp
   makecab %options% /d reserveperfoldersize=%~1 t.tmp %~1.chr > nul
   type %~1.chr | ( (for /l %%N in (1,1,38) do pause)>nul & findstr "^" > temp.tmp )
   >nul copy /y temp.tmp /a %~1.chr /b
   del t.tmp temp.tmp
) else (
   copy /y nul + nul /a 26.chr /a >nul
)
goto :eof

Antonio

PS - Note to Dave and jeb: in this post you suggested that you successfully used all tokens in 128-254 range. Could you confirm this point and specify the Windows version/language you used? TIA

aGerman
Expert
Posts: 4654
Joined: 22 Jan 2010 18:01
Location: Germany

Re: Using many "tokens=..." in FOR /F command in a simple way

#14 Post by aGerman » 20 Feb 2017 11:19

Antonio

This is the content of output.txt

Code: Select all


Thread 1:
128 212 144 210 211 222 214 215 216 209 165 227 224 226 229 153 158 157 235 233 234 154 237 232 225 133 160 131 198 132 134 145 135 138 130 136 137 141 161 140 139 208 164 149 162 147 228 148 246 155 151 163 150 129 236 231 152
Thread 2:
142 143 146
Thread 3:
156 207 190 221 245 249 184 166 174 170 240 169 238 248 241 253 252 239 230 244 250 247 251 167 175 172 171 243 168 183 181 182 199
Thread 4:
159
Thread 5:
173 189
Thread 6:
176 177 178
Thread 7:
179
Thread 8:
180
Thread 9:
185
Thread 10:
186
Thread 11:
187
Thread 12:
188
Thread 13:
191
Thread 14:
192
Thread 15:
193
Thread 16:
194
Thread 17:
195
Thread 18:
196
Thread 19:
197
Thread 20:
200
Thread 21:
201
Thread 22:
202
Thread 23:
203
Thread 24:
204
Thread 25:
205
Thread 26:
206
Thread 27:
213
Thread 28:
217
Thread 29:
218
Thread 30:
219
Thread 31:
220
Thread 32:
223
Thread 33:
242
Thread 34:
254
Thread 35:
255

To be honest I don't know how to proceed with this result.

Steffen

Aacini
Expert
Posts: 1885
Joined: 06 Dec 2011 22:15
Location: México City, México
Contact:

Re: Using many "tokens=..." in FOR /F command in a simple way

#15 Post by Aacini » 20 Feb 2017 12:11

Yes. The program assemble threads of successive tokens, but each token included in a thread is removed from the original 128..255 tokens set. This means that when a posterior thread appear, the token that connect it with a previous thread was already removed, so it is neccessary to run the program again with the last token of posterior threads in previous run. In your case, you should run the program again with tokens 152, 146, 199, 189 or 178 as the initial one.

However your result is exactly the same that I get in my Windows 8.1 Spanish! This means that the longest tokens thread should start at char 173 and should generate this result:

Code: Select all

Thread 1:
173 189 156 207 190 221 245 249 184 166 174 170 240 169 238 248 241 253 252 239 230 244 250
247 251 167 175 172 171 243 168 183 181 182 199 142 143 146 128 212 144 210 211 222 214 215
216 209 165 227 224 226 229 153 158 157 235 233 234 154 237 232 225 133 160 131 198 132 134
145 135 138 130 136 137 141 161 140 139 208 164 149 162 147 228 148 246 155 151 163 150 129
236 231 152
Thread 2:
159
Thread 3:
176 177 178
Thread 4:
179

... up to thread 32 with one token each

If this is true, then you just need to use the same working code for my Windows 8.1 Spanish version that I posted complete here in order to avoid the usual problems with certain characters. This solution provides access for up to 177 tokens. Please, post the result.

Antonio
Attachments
FOR-F with many tokens - SP.zip
Working code in Windows 8.1 Spanish for up to 177 tokens
(1.61 KiB) Downloaded 545 times

Post Reply