Copying a whole file to a variable array Call :SimpleFileToArray OutputArray Filename

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Post Reply
Message
Author
shodan
Posts: 54
Joined: 01 May 2023 01:49

Copying a whole file to a variable array Call :SimpleFileToArray OutputArray Filename

#1 Post by shodan » 20 Aug 2023 03:10

Hello,

I am trying to write a batch function, that takes a filename and an array name, copies every name from the file into array variables.

The function should, not skip empty lines, not cause error or mangle the line's text, not add extra text.

The function should not depend on delayedexpansion being set when called, and it should return with delayedexpansion in the same state as it was before (it should not return in a setlocal nest).

It should be possible to take any regular text file, call it into this function, and then run another function to copy the entire array back to a file, the input and output files should be internally identical as verified by a hash function

So far, my best go at making this function as as follows

Code: Select all

::Usage Call :SimpleFileToArray OutputArray Filename
:SimpleFileToArray
for /f delims^=^ eol^= %%a in ('%SystemRoot%\System32\findstr.exe /N "^" "%~2"') do ( 
	for /f "tokens=1,2* delims=:" %%f in ("%%a") do set "%~1.ubound=%%f" & set %~1[%%f]=%%a 
	)
GoTo :EOF

Here is the launcher function

Code: Select all

:SimpleFileToArray-DEMO
Call :ClearVariablesByPrefix _FTA LinesArray
echo start SimpleFileToArray %time%
Call :SimpleFileToArray LinesArray batchsample.bat
echo end SimpleFileToArray %time%
GoTo :EOF
Here is support function :ClearVariablesByPrefix

Code: Select all

:: Usage Call :ClearVariablesByPrefix myPrefix
:ClearVariablesByPrefix
if "[%~1]" NEQ "[]" for /f "tokens=1,2 delims==" %%a in ('set %~1 2^>nul') do set %%a=
if "[%~2]" NEQ "[]" shift & GoTo :ClearVariablesByPrefix
GoTo :EOF
This version of :SimpleFileToArray runs in about 2.15 second on a 82k / 1500 lines sample batch file from an earlier project.

It runs in about 1.5 seconds if the set array.ubound is omitted (but then you have to find the ubound, which takes more than 0.5 seconds to the best of my knowledge, it would be great to set ubound only once.)


But this function has one flaw, it leaves behind the line numbers in the string !

Example

Code: Select all

LinesArray[1052]=1052:  if "[%_GSS_state%]" NEQ "[]" if "[%_GSS_message%]" NEQ "[]" set "powercfg.sleepstates[%_GSS_sleepstate_index%].unsupported=true"
LinesArray[1053]=1053:  if "[%_GSS_state%]" NEQ "[]" if "[%_GSS_message%]" NEQ "[]" set "powercfg.sleepstates[%_GSS_sleepstate_index%]=%_GSS_state%"
LinesArray[1054]=1054:  if "[%_GSS_state%]" NEQ "[]" if "[%_GSS_message%]" NEQ "[]" set "powercfg.sleepstates[%_GSS_sleepstate_index%].message=%_GSS_message%"
LinesArray[1055]=1055:  if "[%_GSS_state%]" NEQ "[]" if "[%_GSS_message%]" NEQ "[]" set "powercfg.sleepstates.ubound=%_GSS_sleepstate_index%"
LinesArray[1056]=1056:  if "[%_GSS_state%]" NEQ "[]" if "[%_GSS_message%]" NEQ "[]" set "powercfg.sleepstates.unsupported[%_GSS_unsupported_sleepstate_index%]=%_GSS_state%"

To try and remedy this, I have attempted the following version

Code: Select all

::Usage Call :SimpleFileToArray OutputArray Filename
:SimpleFileToArray
for /f delims^=^ eol^= %%a in ('%SystemRoot%\System32\findstr.exe /N "^" "%~2"') do ( 
	for /f "tokens=1,2* delims=:" %%b in ("%%a") do (
		setlocal enabledelayedexpansion
		set _SFTA_buffer=%%a
		set _SFTA_buffer=!_SFTA_buffer:*:=!
		for /f "delims=" %%d in ('echo(!_SFTA_buffer!') do (
			endlocal
			set %~1.ubound=%%b
			set %~1[%%b]=%%d
			)
		if defined _SFTA_buffer endlocal
		)
	)
GoTo :EOF
This does not work

So I created a debug function to try and figure it out

Code: Select all

::Usage Call :SimpleFileToArray OutputArray Filename
:SimpleFileToArray
for /f delims^=^ eol^= %%a in ('%SystemRoot%\System32\findstr.exe /N "^" "%~2"') do ( 
	for /f "tokens=1,2* delims=:" %%b in ("%%a") do (
		echo ---NEWLINE---
		echo(line number %%b
		echo(0%%a
		setlocal enabledelayedexpansion
		set _SFTA_buffer=%%a
		echo(1!_SFTA_buffer!
		set _SFTA_buffer=!_SFTA_buffer:*:=!
		echo(2!_SFTA_buffer!
		for /f "delims=" %%d in ('echo(!_SFTA_buffer!') do (
			endlocal
			echo(line number %%b
			echo(3%%d
			set %~1.ubound=%%b
			echo set %~1[%%b]=%%d
			set %~1[%%b]=%%d
			echo ---ENDLINE---
			)
		if defined _SFTA_buffer endlocal
		)
	)
GoTo :EOF
Even though the output looks like it should work

Code: Select all

---NEWLINE---
line number 1
01:@echo off
11:@echo off
2@echo off
line number 1
3@echo off
set LinesArray[1]=@echo off
---ENDLINE---
It doesn't. I suspect this is because the nested for loops and not working great with setlocal.

With @echo off here is that first loop

Code: Select all

D:\dev\work>Call :SimpleFileToArray LinesArray batchsample.bat

D:\dev\work>for /F delims= eol= %a in ('C:\Windows\System32\findstr.exe /N "^" "batchsample.bat"') do (for /F "tokens=1,2* delims=:" %b in ("%a") do (
echo ---NEWLINE---
 echo(line number %b
 echo(0%a
 setlocal enabledelayedexpansion
 set _SFTA_buffer=%a
 echo(1!_SFTA_buffer!
 set _SFTA_buffer=!_SFTA_buffer:*:=!
 echo(2!_SFTA_buffer!
 for /F "delims=" %d in ('echo(!_SFTA_buffer!') do (
endlocal
 echo(line number %b
 echo(3%d
 set LinesArray.ubound=%b
 echo set LinesArray[%b]=%d
 set LinesArray[%b]=%d
 echo ---ENDLINE---
)
 if defined _SFTA_buffer endlocal
) )

D:\dev\work>(for /F "tokens=1,2* delims=:" %b in ("1:@echo off") do (
echo ---NEWLINE---
 echo(line number %b
 echo(01:@echo off
 setlocal enabledelayedexpansion
 set _SFTA_buffer=1:@echo off
 echo(1!_SFTA_buffer!
 set _SFTA_buffer=!_SFTA_buffer:*:=!
 echo(2!_SFTA_buffer!
 for /F "delims=" %d in ('echo(!_SFTA_buffer!') do (
endlocal
 echo(line number %b
 echo(3%d
 set LinesArray.ubound=%b
 echo set LinesArray[%b]=%d
 set LinesArray[%b]=%d
 echo ---ENDLINE---
)
 if defined _SFTA_buffer endlocal
) )

D:\dev\work>(
echo ---NEWLINE---
 echo(line number 1
 echo(01:@echo off
 setlocal enabledelayedexpansion
 set _SFTA_buffer=1:@echo off
 echo(1!_SFTA_buffer!
 set _SFTA_buffer=!_SFTA_buffer:*:=!
 echo(2!_SFTA_buffer!
 for /F "delims=" %d in ('echo(!_SFTA_buffer!') do (
endlocal
 echo(line number 1
 echo(3
 set LinesArray.ubound=1
 echo set LinesArray[1]=
 set LinesArray[1]=
 echo ---ENDLINE---
)
 if defined _SFTA_buffer endlocal
)
---NEWLINE---
line number 1
01:@echo off
11:@echo off
2@echo off

D:\dev\work>(
endlocal
 echo(line number 1
 echo(3@echo off
 set LinesArray.ubound=1
 echo set LinesArray[1]=@echo off
 set LinesArray[1]=@echo off
 echo ---ENDLINE---
)
line number 1
3@echo off
set LinesArray[1]=@echo off
---ENDLINE---
I have asked this poorly on stackoverflow https://stackoverflow.com/questions/768 ... d-all-char

I have also asked poorly on reddit https://old.reddit.com/r/Batch/comments ... _an_array/

shodan
Posts: 54
Joined: 01 May 2023 01:49

Re: Copying a whole file to a variable array Call :SimpleFileToArray OutputArray Filename

#2 Post by shodan » 20 Aug 2023 13:16

I thought I might have had a solution with the following

Code: Select all

::Usage Call :SimpleFileToArray OutputArray Filename
:SimpleFileToArray
for /f delims^=^ eol^= %%a in ('%SystemRoot%\System32\findstr.exe /N "^" "%~2"') do ( 
	for /f "tokens=1,2* delims=:" %%f in ("%%a") do set "%~1.ubound=%%f" & set %~1[%%f]=%%a 
	)
set /a "_SFTA_index=1"
call set /a "_SFTA_ubound=%%%~1.ubound%%"
:SimpleFileToArray-loop
setlocal enabledelayedexpansion
set _SFTA_buffer=!%~1[%_SFTA_index%]!
set _SFTA_buffer=!_SFTA_buffer:*:=!
for /f "delims=" %%a in ('echo(!_SFTA_buffer!') do (
		endlocal
		set %~1.ubound=%_SFTA_index%
		set %~1[%_SFTA_index%]=%%a
	)
if defined _SFTA_buffer endlocal
set /a "_SFTA_index+=1"
if %_SFTA_index% LEQ %_SFTA_ubound% GoTo :SimpleFileToArray-loop
GoTo :EOF
But on the line

Code: Select all

for /f "delims=" %%a in ('echo(!_SFTA_buffer!') do (
Gets executed if there are pipes or ampersands

For instance line number 14 of my batchsample.bat

Code: Select all

endlocal & set /a "powercfg.scheme.ubound=%powercfg.scheme.ubound%" & set "powercfg.scheme.default.guid=%powercfg.scheme.default.guid%" & set "powercfg.scheme.default.name=%powercfg.scheme.default.name%" & set "powercfg.scheme[%powercfg.scheme.ubound%].guid=%%f"
Gives a "missing operand" error on the console and the line of text gets trunkated in the array as follows

Code: Select all

LinesArray[14]=                 REM endlocal
I've then tried use "set" to pass the line of text instead

Code: Select all

for /f "delims=" %%a in ('set _SFTA_buffer') do (
This works except now every line gets prefixed with "_SFTA_buffer="

Code: Select all

LinesArray[14]=_SFTA_buffer=                    REM endlocal & set /a "powercfg.scheme.ubound=%powercfg.scheme.ubound%" & set "powercfg.scheme.default.guid=%powercfg.scheme.default.guid%" & set "powercfg.scheme.default.name=%powercfg.scheme.default.name%" & set "powercfg.scheme[%powercfg.scheme.ubound%].guid=%%f"
Well, at least now the delimited is no longer ":" but "=" and "=" is a lot less frequent in batch files than ":". Also this now takes about 11 seconds to execute on the same data set.

shodan
Posts: 54
Joined: 01 May 2023 01:49

Re: Copying a whole file to a variable array Call :SimpleFileToArray OutputArray Filename

#3 Post by shodan » 20 Aug 2023 14:24

I think I cracked it (spoiler, I haven't :( )

Execution time is now about 20 seconds

Code: Select all

::Usage Call :SimpleFileToArray OutputArray Filename
:SimpleFileToArray
for /f delims^=^ eol^= %%a in ('%SystemRoot%\System32\findstr.exe /N "^" "%~2"') do ( 
	for /f "tokens=1,2* delims=:" %%f in ("%%a") do set "%~1.ubound=%%f" & set %~1[%%f]=%%a
	)
set /a "_SFTA_index=1"
call set /a "_SFTA_ubound=%%%~1.ubound%%"
:SimpleFileToArray-loop
setlocal enabledelayedexpansion
set _SFTA_localscope=true
set %~1[%_SFTA_index%]=!%~1[%_SFTA_index%]:*:=!
for /f "delims=" %%a in ('set %~1[%_SFTA_index%] 2^>nul') do (
		endlocal
		set %%a
	)
if defined _SFTA_localscope endlocal & set %~1[%_SFTA_index%]=
set /a "_SFTA_index+=1"
if %_SFTA_index% LEQ %_SFTA_ubound% GoTo :SimpleFileToArray-loop
GoTo :EOF

Code: Select all

D:\dev\work>SimpleFileToArray.bat
start SimpleFileToArray 16:21:27.28
end SimpleFileToArray 16:22:47.62
My two known problematic lines

Code: Select all

D:\dev\work>echo %linesarray[9]%
                                REM echo Power Scheme Name : %%f GUID : %%d !_powercfglist_default!

D:\dev\work>echo %linesarray[14]%
                        REM endlocal
Missing operand.

D:\dev\work>

Oh no, I don't know where it happenned, but the line with the & is back to F things up :(

shodan
Posts: 54
Joined: 01 May 2023 01:49

Re: Copying a whole file to a variable array Call :SimpleFileToArray OutputArray Filename

#4 Post by shodan » 20 Aug 2023 14:40

I've added an echo in the last for loop

Code: Select all

::Usage Call :SimpleFileToArray OutputArray Filename
:SimpleFileToArray
for /f delims^=^ eol^= %%a in ('%SystemRoot%\System32\findstr.exe /N "^" "%~2"') do ( 
	for /f "tokens=1,2* delims=:" %%f in ("%%a") do set "%~1.ubound=%%f" & set %~1[%%f]=%%a
	)
set /a "_SFTA_index=1"
call set /a "_SFTA_ubound=%%%~1.ubound%%"
:SimpleFileToArray-loop
setlocal enabledelayedexpansion
set _SFTA_localscope=true
set %~1[%_SFTA_index%]=!%~1[%_SFTA_index%]:*:=!
for /f "delims=" %%a in ('set %~1[%_SFTA_index%] 2^>nul') do (
		endlocal
		echo(%%a
		set %%a
	)
if defined _SFTA_localscope endlocal & set %~1[%_SFTA_index%]=
set /a "_SFTA_index+=1"
if %_SFTA_index% LEQ %_SFTA_ubound% GoTo :SimpleFileToArray-loop
GoTo :EOF

We can see, line 14 is still intact here

Code: Select all

D:\dev\work>SimpleFileToArray.bat
start SimpleFileToArray 16:25:53.12
LinesArray[1]=@echo off
LinesArray[3]=REM echo.
LinesArray[4]=REM for /f "tokens=1,2,3,4,* skip=3 delims=: " %%a in ('powercfg /list') do (
LinesArray[5]=  REM if "[%%a %%b %%c]" EQU "[Power Scheme GUID]" (
LinesArray[6]=          REM for /f "tokens=1,* delims=()" %%f in ("%%e") do (
LinesArray[7]=                  REM if "[%%g]" EQU "[ *]" ( set "_powercfglist_default=defaultscheme" ) else ( set "_powercfglist_default=" )
LinesArray[8]=                  REM setlocal enabledelayedexpansion
LinesArray[9]=                          REM echo Power Scheme Name : %%f GUID : %%d !_powercfglist_default!
LinesArray[10]=                         REM if "[!powercfg.scheme.ubound!]" EQU "[]" ( set /a "powercfg.scheme.ubound=0" ) else ( set /a "powercfg.scheme.ubound+=1" )
LinesArray[11]=                         REM set "powercfg.scheme[!powercfg.scheme.ubound!].guid=%%f"
LinesArray[12]=                         REM set "powercfg.scheme[!powercfg.scheme.ubound!].name=%%d"
LinesArray[13]=                         REM if "[%!_powercfglist_default!%]" EQU "defaultscheme" ( set "powercfg.scheme.default.guid=%%f" & set "powercfg.scheme.default.name=%%d" )
LinesArray[14]=                 REM endlocal & set /a "powercfg.scheme.ubound=%powercfg.scheme.ubound%" & set "powercfg.scheme.default.guid=%powercfg.scheme.default.guid%" & set "powercfg.scheme.default.name=%powercfg.scheme.default.name%" & set "powercfg.scheme[%powercfg.scheme.ubound%].guid=%%f"
LinesArray[15]=                         REM set "powercfg.scheme[!powercfg.scheme.ubound!].name=%%d"
LinesArray[16]=                 REM )
LinesArray[17]=         REM )
LinesArray[18]= REM )
LinesArray[19]=
LinesArray[21]=REM Call :ClearVariablesByPrefix powercfg
LinesArray[22]=REM set powercfg
LinesArray[23]=cls
LinesArray[25]=Call :ClearVariablesByPrefix _
LinesArray[26]=Call :ClearVariablesByPrefix _powercfg_demo
^CTerminate batch job (Y/N)?
So why does the set command break down ?

The first set comment in the first for loop does set %~1[%%f]=%%a no problems !

I guess I could set "%%a" but I'm pretty sure this will break on any line with an odd number of doublequotes in the text. I've always been trying to expand %%a with nothing after on the line like set var=%%a to avoid escaping errors.....


Wait, I just ran the thing again and

Code: Select all

D:\dev\work>set linesarray[14]
LinesArray[14]=                 REM endlocal & set /a "powercfg.scheme.ubound=%powercfg.scheme.ubound%" & set "powercfg.scheme.default.guid=%powercfg.scheme.default.guid%" & set "powercfg.scheme.default.name=%powercfg.scheme.default.name%" & set "powercfg.scheme[%powercfg.scheme.ubound%].guid=%%f"

What is going on 1?!?!?

I ran it again in a new console
then I remove the echo line
save and run again

Code: Select all

[...]
LinesArray[1592]=set /a "__GetCirclePoint_DEMO_index+=1"
LinesArray[1593]=if %__GetCirclePoint_DEMO_index% LEQ %__GetCirclePoint_DEMO_radius% GoTo :GetCirclePoint-DEMO-loop-1
LinesArray[1594]=GoTo :EOF
end SimpleFileToArray 16:34:10.13

D:\dev\work>set linesarray[14]
LinesArray[14]=                 REM endlocal & set /a "powercfg.scheme.ubound=%powercfg.scheme.ubound%" & set "powercfg.scheme.default.guid=%powercfg.scheme.default.guid%" & set "powercfg.scheme.default.name=%powercfg.scheme.default.name%" & set "powercfg.scheme[%powercfg.scheme.ubound%].guid=%%f"

D:\dev\work>SimpleFileToArray.bat
start SimpleFileToArray 16:34:36.52
end SimpleFileToArray 16:35:42.13

D:\dev\work>set linesarray[14]
LinesArray[14]=                 REM endlocal & set /a "powercfg.scheme.ubound=%powercfg.scheme.ubound%" & set "powercfg.scheme.default.guid=%powercfg.scheme.default.guid%" & set "powercfg.scheme.default.name=%powercfg.scheme.default.name%" & set "powercfg.scheme[%powercfg.scheme.ubound%].guid=%%f"

D:\dev\work>

Trying again

Code: Select all

Microsoft Windows [Version 10.0.19045.2006]
(c) Microsoft Corporation. All rights reserved.

D:\dev\work>SimpleFileToArray.bat
start SimpleFileToArray 16:36:47.73
end SimpleFileToArray 16:38:05.52

D:\dev\work>set linesarray[14]
LinesArray[14]=                 REM endlocal & set /a "powercfg.scheme.ubound=%powercfg.scheme.ubound%" & set "powercfg.scheme.default.guid=%powercfg.scheme.default.guid%" & set "powercfg.scheme.default.name=%powercfg.scheme.default.name%" & set "powercfg.scheme[%powercfg.scheme.ubound%].guid=%%f"

D:\dev\work>

I don't get it, now it works !?

Maybe there's a race condition and it fails intermittently ?

Well, I can't seem to make it fail now.


Here's the best version I've got so far, does there exist a plain text file that will break this or get mangled ?

Code: Select all

::Usage Call :SimpleFileToArray OutputArray Filename
:SimpleFileToArray
for /f delims^=^ eol^= %%a in ('%SystemRoot%\System32\findstr.exe /N "^" "%~2"') do ( 
	for /f "tokens=1,2* delims=:" %%f in ("%%a") do set "%~1.ubound=%%f" & set %~1[%%f]=%%a
	)
set /a "_SFTA_index=1"
call set /a "_SFTA_ubound=%%%~1.ubound%%"
:SimpleFileToArray-loop
setlocal enabledelayedexpansion
set _SFTA_localscope=true
set %~1[%_SFTA_index%]=!%~1[%_SFTA_index%]:*:=!
for /f "delims=" %%a in ('set %~1[%_SFTA_index%] 2^>nul') do (
		endlocal
		set %%a
	)
if defined _SFTA_localscope endlocal & set %~1[%_SFTA_index%]=
set /a "_SFTA_index+=1"
if %_SFTA_index% LEQ %_SFTA_ubound% GoTo :SimpleFileToArray-loop
GoTo :EOF

Aacini
Expert
Posts: 1885
Joined: 06 Dec 2011 22:15
Location: México City, México
Contact:

Re: Copying a whole file to a variable array Call :SimpleFileToArray OutputArray Filename

#5 Post by Aacini » 21 Aug 2023 08:19

Mmm... The first important point is that you have not posted a problematic input file, so we all could have a common data to test code...

This is my code:

Code: Select all

@echo off
setlocal EnableDelayedExpansion

rem Count input lines
for /F "tokens=3" %%a in ('find /C /V "" test.txt') do set "lines=%%a"

rem Read the file into an array
(for /L %%i in (1,1,%lines%) do set /P "LinesArray[%%i]=") < test.txt

rem Duplicate the lines in other file
(for /L %%i in (1,1,%lines%) do echo(!LinesArray[%%i]!) > test2.txt

rem Compare the files
fc /N test.txt test2.txt
I used as input file your "batchsample.bat" file. The duplicated file is not identical: my code fails when there is a line that just contains a TAB character!

Antonio

shodan
Posts: 54
Joined: 01 May 2023 01:49

Re: Copying a whole file to a variable array Call :SimpleFileToArray OutputArray Filename

#6 Post by shodan » 22 Aug 2023 04:10

Please find attached, the batchsample.bat (funnily enough, this forum does not allow posting of .bat files)
batchsample.bat.txt
(80.72 KiB) Downloaded 102 times
I have also included batchsample2.bat batchsample3.bat

(turns out they are too big, see github link
https://github.com/batchfileframework/t ... ample2.bat
https://github.com/batchfileframework/t ... ample3.bat
)

There three files are old copies of some large batch files I have been working on that have become very overgrown.

My project with this is to create function which can find, list and bracket individual functions and break them out as independant .bat function files

The file SimpleFileToArray tests.txt contains several output from DEMO function that try to find the important, labels, functions, EndOf_function, function exits and empty rows that delimit batch files.

SimpleFileToArray.bat is another overgrown file which contain many of these latest functions.

The functions :GetFunctionExits-DEMO :GetEndOfFunction-DEMO :GetLabels-DEMO :GetEmptyLines-DEMO demonstrates the important functionality, the rest is relatively easy.

Also of note is the function GetIndexArray, a function that can take any number of ranges of indexes and expand them into a contiguous array, similar to page selection in a print dialog "2,3,4,1-5,10-0,3,2,999", this will allow easy outputting of the functions once they have been bracketed

Also of interest is the function EchoArray, with can output simply output an entire array in numeric element order, or in the order specified by a GetIndexArray array. (BTW a GetIndexArray can be fed into the input of GetIndexArray without damage)

I have attempted to put the torture file CreateRandomFile-DEMO.txt
CreateRandomFile-DEMO.txt
(11.65 KiB) Downloaded 105 times
Call :SimpleFileToArray LinesArray CreateRandomFile-DEMO.txt
Call :EchoArray LinesArray > CreateRandomFile-DEMO.txt.output.txt

It appears something is wrong

fc /n CreateRandomFile-DEMO.txt CreateRandomFile-DEMO.txt.output.txt


Comparing files CreateRandomFile-DEMO.txt and CREATERANDOMFILE-DEMO.TXT.OUTPUT.TXT
***** CreateRandomFile-DEMO.txt
1: R 100 110 n-J]Xy:j42ahkYg HAF( M[&`oIiq7s{1Pi*jyU&&%#%5/*?{<Y!NN3M3[>Z<x@LYOm`tT~|{1P?k1C"%R2/1ApnEt<Zt2 XN/]D
2: R 100 108 #Z:Wn !#=Q5:H)3bk/NZ"VfvO]:FsY7TQQSl8 E{dUj$K+T)1VJ+Pbxv1IyfOxNkj0=<P`D-.4T%1f''FgKFx8%;*746b=0|H RP
***** CREATERANDOMFILE-DEMO.TXT.OUTPUT.TXT
1:
2: R 100 110 n-J]Xy:j42ahkYg HAF( M[&`oIiq7s{1Pi*jyU&&%#%5/*?{<Y!NN3M3[>Z<x@LYOm`tT~|{1P?k1C"%R2/1ApnEt<Zt2 XN/]D
3: R 100 108 #Z:Wn !#=Q5:H)3bk/NZ"VfvO]:FsY7TQQSl8 E{dUj$K+T)1VJ+Pbxv1IyfOxNkj0=<P`D-.4T%1f''FgKFx8%;*746b=0|H RP
*****

Could be just that extra space at the beginning of the file ? I am 1.5 hours past bed time, sorry I no longer have time to play with this.

Will write again shortly
Attachments
SimpleFileToArray tests.txt
(86.66 KiB) Downloaded 103 times

shodan
Posts: 54
Joined: 01 May 2023 01:49

Re: Copying a whole file to a variable array Call :SimpleFileToArray OutputArray Filename

#7 Post by shodan » 31 Aug 2023 02:28

Copying a file to an array and back to a file now appears to work fully, possibly no matter what special characters are in the file.

Code: Select all

    FileToArrayToFile start SimpleFileToArray  4:19:22.99
    FileToArrayToFile mid SimpleFileToArray  4:19:31.75
    FileToArrayToFile end SimpleFileToArray  4:19:36.69
    Comparing files CreateRandomFile-DEMO.txt and FILETOARRAYTOFILE-DEMO.TXT
    FC: no differences encountered

Here is the DEMO function

Code: Select all

:FileToArrayToFile-DEMO
Call :SimpleFileToArray CreateRandomFile-DEMO.txt ArrayOfLines
Call :ArrayToFile ArrayOfLines FileToArrayToFile-DEMO.txt
fc /n CreateRandomFile-DEMO.txt FileToArrayToFile-DEMO.txt
GoTo :EOF


Not sure what the max line lenght it

Not sure what happens with non-printable characters

Here is the full code, but contains tons of extra stuff, need to be trimmed down

https://pastebin.com/R1VpyXZL

Here is the file CreateRandomFile-DEMO.txt
https://pastebin.com/NVAsJ6vF


The current version of :SimpleFileToArray

Code: Select all

::Usage Call :SimpleFileToArray Filename OutputArray
:SimpleFileToArray
set /a "%~2.lbound=1"
for /f delims^=^ eol^= %%a in ('%SystemRoot%\System32\findstr.exe /N "^" "%~1"') do ( 
	for /f "tokens=1,2* delims=:" %%f in ("%%a") do set /a "%~2.ubound=%%f" & set %~2[%%f]=%%a
	)
set /a "_SFTA_index=1"
call set /a "_SFTA_ubound=%%%~2.ubound%%"
:SimpleFileToArray-loop
Setlocal enabledelayedexpansion
set _SFTA_localscope=true
set %~2[%_SFTA_index%]=!%~2[%_SFTA_index%]:*:=!
for /f "delims=" %%a in ('set %~2[%_SFTA_index%] 2^>nul') do (
		endlocal
		set %%a
	)
if defined _SFTA_localscope endlocal & set %~2[%_SFTA_index%]=
set /a "_SFTA_index+=1"
if %_SFTA_index% LEQ %_SFTA_ubound% GoTo :SimpleFileToArray-loop
GoTo :EOF
the current version of :ArrayToFile

Code: Select all

REM add echo array "verticalmode" (no LF between array elements)
::Usage Call :ArrayToFile InputArray OutputFile optional LINENUMBERS optional SHOWVARNAME optional .Suffix optional IndexRange
:ArrayToFile
set "_ArrayToFile_input=%~1"
call set /a "_ArrayToFile_lbound=%%%~1.lbound" 2>nul
if "[%_ArrayToFile_lbound%]" EQU "[]" set /a "_ArrayToFile_lbound=0"
call set /a "_ArrayToFile_ubound=%%%~1.ubound"
set /a "_ArrayToFile_index=%_ArrayToFile_lbound%"
shift
set "_ArrayToFile_output=%~1"
shift
:ArrayToFile-arguments
set "_ArrayToFile_buffer=%~1"
if not defined _ArrayToFile_buffer GoTo :ArrayToFile-arguments-end
if "[%_ArrayToFile_buffer:~,1%]" EQU "[.]" ( set "_ArrayToFile_suffix=%_ArrayToFile_buffer%" & shift & GoTo :ArrayToFile-arguments )
if "[%_ArrayToFile_buffer%]" EQU "[LINENUMBERS]" ( set "_ArrayToFile_showlinenumbers=true" & shift & GoTo :ArrayToFile-arguments )
if "[%_ArrayToFile_buffer%]" EQU "[SHOWVARNAME]" ( set "_ArrayToFile_showvariablename=true" & shift & GoTo :ArrayToFile-arguments )
if "[%_ArrayToFile_buffer%]" EQU "[VERTICALMODE]" ( set "_ArrayToFile_verticalmode=true" & shift & GoTo :ArrayToFile-arguments )
if "[%~1]" NEQ "[]" ( Call :GetIndexArray _ArrayToFile_IndexList "%~1" & shift & GoTo :ArrayToFile-arguments )
:ArrayToFile-arguments-end
if defined _ArrayToFile_IndexList.ubound set /a "_ArrayToFile_ubound=%_ArrayToFile_IndexList.ubound%"
setlocal enabledelayedexpansion
:ArrayToFile-loop
if not defined _ArrayToFile_IndexList.ubound ( set "_ArrayToFile_index_actual=%_ArrayToFile_index%" ) else ( set "_ArrayToFile_index_actual=!_ArrayToFile_IndexList[%_ArrayToFile_index%]!" )
if defined _ArrayToFile_showlinenumbers set _ArrayToFile_prefix=%_ArrayToFile_index%:
if defined _ArrayToFile_showvariablename set _ArrayToFile_prefix=%_ArrayToFile_input%[%_ArrayToFile_index_actual%]:
if defined _ArrayToFile_showvariablename if defined _ArrayToFile_showlinenumbers set _ArrayToFile_prefix=%_ArrayToFile_index%:%_ArrayToFile_input%[%_ArrayToFile_index_actual%]:
if not defined _ArrayToFile_verticalmode GoTo :ArrayToFile-normalmode-loop-next
<nul set /p =%_ArrayToFile_prefix%!%_ArrayToFile_input%[%_ArrayToFile_index_actual%]%_ArrayToFile_suffix%!>>%_ArrayToFile_output%
GoTo :ArrayToFile-loop-next
:ArrayToFile-normalmode-loop-next
echo(%_ArrayToFile_prefix%!%_ArrayToFile_input%[%_ArrayToFile_index_actual%]%_ArrayToFile_suffix%!>>%_ArrayToFile_output%
:ArrayToFile-loop-next
set /a "_ArrayToFile_index+=1"
if %_ArrayToFile_index% LEQ %_ArrayToFile_ubound% GoTo :ArrayToFile-loop
:ArrayToFile-loop-end
endlocal
Call :ClearVariablesByPrefix _ArrayToFile
GoTo :EOF

Post Reply