Help for fast method to transpose data.

[einstein1969]

Hi to all,

I'm searching a fast method for transpose data.

I have a file that has N rows and M columns of char data with M,N<=90 (For simplify and use a single variable i have fixed :Total Number of chars <= 8100 [90x90]).

Code: Select all

ABCDEF......
GHIJKL......
MNOPQR......
............

I need to produce :

Code: Select all

AGM.....
BHN.....
CIO.....
DJP.....
EKQ.....
FLR.....
........
........
........

What is the fastest method?

For the moment I use file for store results (In future i need to compare speed with env variable. But this is next problem)

Other ref: In-place_matrix_transposition

einstein1969

[penpen]

The fastest method bases on your data structure.
For example if the variables are stored in "lines" this might be fast:

Code: Select all

@echo off
setlocal enableDelayedExpansion
set /A "N=12", "M=4"
set N1=ABCDEF......
set N2=GHIJKL......
set N3=MNOPQR......
set N4=............

set /A "MaxN=N-1", "maxM=M-1"
set "transposedLine="
for /L %%m in (1, 1, %M%) do set "transposedLine=!transposedLine!^!%%A%%m:~%%n,1^!"
for %%A in (N) do for %%T in (Nt) do for /L %%n in (0, 1, %maxN%) do echo %transposedLine%
endlocal
goto :eof

penpen

[dbenham]

I suppose there is a bit more optimization possible for reading the file, but I think this should be about as good as it gets regarding the transposition. It assumes the data file does not contain !

Code: Select all

@echo off
setlocal enableDelayedExpansion

:: Compute line length - 1
set "probe=000102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F"
<test.txt set /p "ln="
set "ln=!probe!!ln!!ln!"
set /a "max=0x!ln:~-194,2!"

:: Clear output array
for /l %%N in (0 1 %max%) do set "ln%%N="

:: Load and transpose data
for /f delims^=^ eol^= %%A in (test.txt) do (
  set "ln=%%A"
  for /l %%N in (0 1 %max%) do set "ln%%N=!ln%%N!!ln:~%%N,1!"
)

:: Print output
for /l %%N in (0 1 %max%) do echo !ln%%N!

Dave Benham

[einstein1969]

The fastest method bases on your data structure.
For example if the variables are stored in "lines" this might be fast:

Code: Select all

@echo off
setlocal enableDelayedExpansion
set /A "N=12", "M=4"
set N1=ABCDEF......
set N2=GHIJKL......
set N3=MNOPQR......
set N4=............

set /A "MaxN=N-1", "maxM=M-1"
set "transposedLine="
for /L %%m in (1, 1, %M%) do set "transposedLine=!transposedLine!^!%%A%%m:~%%n,1^!"
for %%A in (N) do for %%T in (Nt) do for /L %%n in (0, 1, %maxN%) do echo %transposedLine%
endlocal
goto :eof

penpen

Hi penpen,

I have tested your method but i want use this in a block.

Code: Select all

@echo off
setlocal enableDelayedExpansion
(
set /A "N=12", "M=4"
set N1=ABCDEF......
set N2=GHIJKL......
set N3=MNOPQR......
set N4=............

set /A "MaxN=N-1", "maxM=M-1"
set "transposedLine="
for /L %%m in (1, 1, %M%) do set "transposedLine=!transposedLine!^!%%A%%m:~%%n,1^!"
for %%A in (N) do for %%T in (Nt) do for /L %%n in (0, 1, %maxN%) do echo %transposedLine%
)
endlocal
pause
goto :eof

result

Code: Select all

ECHO disattivato.
Premere un tasto per continuare . . .

why don't work?

@dbenham
Your method work. It's slower than penpen's method (tested with a file 80x80). It's possible to optimize? Thanks

F.P.

[penpen]

I have tested your method but i want use this in a block.
(...)
why don't work?

In the above source the macro "transposedLine" is built on the fly. So if executing all in one block this macro is just empty.
The best workaround depends on what you want to do (various possibilities):
- you could build the macro outside of the block, if possible, or
- you could execute the output line using "cmd /Q /...", or
- you could use "call", or ....


penpen

[Aacini]

Code: Select all

@echo off
setlocal enableDelayedExpansion

(
echo ABCDEF......
echo GHIJKL......
echo MNOPQR......
echo STUXYZ......
) > data.txt
type data.txt

rem Configuration value: Number of columns
set N=12

rem Prepare the transpose macro
set /A lastN=N-1
set "transpose="
for /L %%i in (0,1,%lastN%) do (
   set "T%%i="
   set transpose=!transpose!^&set "T%%i=^!T%%i^!^!line:~%%i,1^!"
)

rem Process the file
for /F "delims=" %%a in (data.txt) do (
   set "line=%%a"
   %transpose:~1%
)

rem Show results
for /L %%i in (0,1,%lastN%) do echo !T%%i!

If you want to use this code inside parentheses, just extract the "Process the file" part into a subroutine and call it
from the code block:

Code: Select all

(

rem Process the file
call :ProcessFile

)
goto :EOF


:ProcessFile
for /F "delims=" %%a in (data.txt) do (
   set "line=%%a"
   %transpose:~1%
)
exit /B

Antonio

[einstein1969]

@penpen

I have moved the macro out for the moment. Thanks!

@Antonio

Thanks Antonio, this resolve my questions .

Einstein1969

[Aacini]

@Antonio

Thanks Antonio, this resolve my questions .

Einstein1969

Well, the answer to the question stated in this topic is: which method is the fastest one? I would like to see the results of timing tests...

Antonio

[einstein1969]

I apologize for the delay in responding...

I have done some test on various methods. I have little modify for use file input method for load the data
For now I have not moved the macros nor reiterate for better precision

These are the results:

Code: Select all

:penpen_transpose [12x4] Time Elapsed:1cs


:dbenham_transpose [12x4] Time Elapsed:1cs


:aacini_transpose [12x4] Time Elapsed:1cs

Checking results...
No errors!
Premere un tasto per continuare . . .

:penpen_transpose [48x24] Time Elapsed:2cs


:dbenham_transpose [48x24] Time Elapsed:8cs


:aacini_transpose [48x24] Time Elapsed:9cs

Checking results...
No errors!
Premere un tasto per continuare . . .

:penpen_transpose [90x60] Time Elapsed:5cs


:dbenham_transpose [90x60] Time Elapsed:49cs


:aacini_transpose [90x60] Time Elapsed:67cs

Checking results...
No errors!
Premere un tasto per continuare . . .

:penpen_transpose [60x90] Time Elapsed:7cs


:dbenham_transpose [60x90] Time Elapsed:40cs


:aacini_transpose [60x90] Time Elapsed:49cs

Checking results...
No errors!

the code

Code: Select all

@echo off

setlocal enableDelayedExpansion

cls
mode 100,500

rem empty environment for fast "set"
rem for /f "delims==" %%v in ('set') do if /I not "%%v" == "TMP" set %%v=

call :create_mat 12 4 Mat_small.txt
call :penpen_transpose 12 4 Mat_small.txt
call :dbenham_transpose 12 4 Mat_small.txt
call :aacini_transpose 12 4 Mat_small.txt
call :check_results

pause

call :create_mat 48 24 Mat_medium.txt
call :penpen_transpose 48 24 Mat_medium.txt
call :dbenham_transpose 48 24 Mat_medium.txt
call :aacini_transpose 48 24 Mat_medium.txt
call :check_results

pause

call :create_mat 90 60 Mat_big.txt
call :penpen_transpose 90 60 Mat_big.txt
call :dbenham_transpose 90 60 Mat_big.txt
call :aacini_transpose 90 60 Mat_big.txt
call :check_results

pause

call :create_mat 60 90 Mat_big.txt
call :penpen_transpose 60 90 Mat_big.txt
call :dbenham_transpose 60 90 Mat_big.txt
call :aacini_transpose 60 90 Mat_big.txt
call :check_results

endlocal

exit /B

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:create_mat N M file_name

setlocal

type nul >%tmp%\%3

For /L %%R in (1,1,%2) do (
  Set "N="
  For /L %%C in (1,1,%1) do set "N=!N!!random:~-1!"
  echo !N!
) >>%tmp%\%3

:: Show output
rem Type %tmp%\%3
rem echo(

endlocal

exit /b

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:check_results

echo Checking results...
fc %tmp%\penpen_out.txt %tmp%\dbenham_out.txt>nul && fc %tmp%\dbenham_out.txt %tmp%\aacini_out.txt>nul

if ERRORLEVEL 1 (echo Errors found^^!
) else echo No errors^^!


exit /b

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:penpen_transpose N M file_name

setlocal

  echo !time: =0!> "%tmp%\t1.txt"

  set /A N=%1, M=%2

::Load data
  set /A i=0
  for /f delims^=^ eol^= %%A in (%tmp%\%3) do (
    set /A i+=1
    set "N!i!=%%A"
  )

::Transpose data
::Fixed.
  set /A "MaxN=%1-1"
  set "transposedLine="
  for /L %%m in (1, 1, %M%) do set "transposedLine=!transposedLine!^!%%A%%m:~%%n,1^!"

:: output transposed values
:: Fixed.
  (for %%A in (N) do for /L %%n in (0, 1, %maxN%) do echo %transposedLine%)>%tmp%\penpen_out.txt

  set t2=!time!
  call :print_elapsed %0 %1 %2

endlocal

exit /b

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:dbenham_transpose N M file_name

setlocal

  echo !time: =0!> "%tmp%\t1.txt"

  set /A N=%1, M=%2, max=N-1

:: Load and transpose data
for /f delims^=^ eol^= %%A in (%tmp%\%3) do (
  set "ln=%%A"
  for /l %%N in (0 1 %max%) do set "ln%%N=!ln%%N!!ln:~%%N,1!"
)

:: output transposed values

  (for /l %%N in (0 1 %max%) do echo !ln%%N!)>%tmp%\dbenham_out.txt

  set t2=!time!
  call :print_elapsed %0 %1 %2

endlocal

exit /b

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:aacini_transpose N M file_name

setlocal

  echo !time: =0!> "%tmp%\t1.txt"

  rem Configuration value: Number of columns and ...
  rem Prepare the transpose macro
  set /A N=%1, lastN=N-1
  set "transpose="
  for /L %%i in (0,1,%lastN%) do (
     set "T%%i="
     set transpose=!transpose!^&set "T%%i=^!T%%i^!^!line:~%%i,1^!"
  )

  rem Process the file
  for /F "delims=" %%a in (%tmp%\%3) do (
     set "line=%%a"
     %transpose:~1%
  )

  rem Show results
  (for /L %%i in (0,1,%lastN%) do echo !T%%i!)>%tmp%\aacini_out.txt

  set t2=!time!
  call :print_elapsed %0 %1 %2

endlocal

exit /b

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:print_elapsed
  echo(
  set /P "t1=" < "%tmp%\t1.txt"
  for /F "tokens=1-8 delims=:.," %%a in ("!t1!:!t2: =0!") do set /a "a=(((1%%e-1%%a)*60)+1%%f-1%%b)*6000+1%%g%%h-1%%c%%d, a+=(a>>31)&8640000"
  echo %1 [%2x%3] Time Elapsed:!a!cs
  echo(
exit /b

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

Einstein1969