HELP :- FSUTIL + COPY versus XCOPY /? ; Caches + i/o Buffers

Message

alan_b · #1 Post by **alan_b** » 30 Nov 2013 07:48

I want a better understanding of the factors affecting :-
Both the Real and also the virtual or apparent Speed of Disks ;
And pure Read only and also Write only of archive files ;
And the transfer of files between partitions on the same Disk and between Disks.

I wish to produce a script that will help me evaluate and compare the performance of Removable and Fixed Disks.
I am currently focussed on Fixed Disks because I think that will cover Removable.

I would appreciate advice and comments upon tentative conclusions based upon the 144 numbered lines created by my script XC.BAT

Speed of Copying a 319,334,486 byte file to NUL occupies :-
1645 mSec Pure Read from my SSD partition C:\ - lines 129,131,133
455 mSec Virtual Read from Windows Cache - lines 134,135,136
2185 mSec from my HDD partition D:\ - lines 165,167,169
435 mSec Virtual Read from Windows Cache - lines 170,171,172

Speed of Copying a 1,014,411,396 byte file to NUL occupies :-
5805 mSec Pure Read from my SSD partition C:\ - lines 201,293,205
1375 mSec Virtual Read from Windows Cache - lines 206,207,208
6825 mSec from my HDD partition D:\ - lines 165,167, and 169
1395 mSec Virtual Read from Windows Cache - lines 170, 171, and 172

Speed of Copying a 319,334,486 byte file From SSD to Disk 1 HDD occupies :-
1850 mSec using FSUTIL + COPY to exclude input cache - lines 101 to 106
995 mSec using COPY and both input and ouput buffers - lines 107 to 109
1830 mSec using FSUTIL + XCOPY to exclude input cache - lines 110 to 115
1045 mSec using XCOPY and both input and ouput buffers - lines 116 to 118
2185 mSec using "XCOPY /J" to EXCLUDE input and output buffers - lines 120 to 127

Speed of Copying a 319,334,486 byte file From HDD Disk 1 D: to HDD Disk 1 H: occupies :-
Slightly longer than SSD to HDD WITH THE EXCEPTION OF
5185 mSec using "XCOPY /J" to EXCLUDE input and output buffers - lines 156 to 163

Results on lines 173 to 244 are about 3 times as long to handle 1,014,411,396 byte file

QUESTIONS :-

WHY DOES IT TAKE TWICE AS LONG WHEN TRANSFERING BETWEEN PARTITIONS ON THE SAME DISK ?
Is it because my SATA Disc interface data channel is only rated at 3 GBit / Second in one direction at a time ?
and can only handle 1.5 GBit/Sec simultaneous input and output ?
or is it the Disk Controller ?
or is it the Disc Mechanism having to jump between two adjacent 10 GB partitions on a 930 GB HDD ?
or what ?

COPY is able to write to NUL, XCOPY chooses an error crash instead, Why ?
(I do not think I have any need to use XCOPY for writing to NUL, but it surprised me)

OBSERVATIONS :-

"XCOPY /J" ensures accurate measurement of transfers without input caches or ouput buffers.
When the script has in force the variable NO_PAUSE=PAUSE
Then every time result is paused and it becomes obvious when watching the HDD activity LED's
that there is a solid continuous ON HDD activity for 17325 mSec as shown on line 234
after which the drive does not twitch anymore unless it responds to the Internet,

BUT Using FSUTIL + COPY has the APPEARANCE of about 2 seconds Disk Activity throughout the 9485 mSec of line 210,
But during a very long pause I observe that the Led flashes for about a quarter second once ever three seconds, so there could be another 15 seconds of Disk activity spread over the next 200 Seconds.

Deduction :-
Windows does a good job at using caches to conceal how slow the drives are,
but I only have 8 GByte of RAM so a Partition image backup of my 55 GB SSD could be far to much for Windows caches to handle.

Final Question :-

XCOPY documentation describes option /J as :-
/J Copies using unbuffered I/O. Recommended for very large files.
Am I correct in assuming that the "output Buffer" is already disabled by default for removable drives,
such as USB2, USB3, and eSATA connected HDD ?

Regards
Alan

The Measurement script, which presupposes Source paths C\###-SpeedTest-#\#\S\ and D:\###-SpeedTest-#\#\S\
each holding a 200 MB and 1 GB file

Code: Select all

@ECHO OFF
SETLOCAL EnableDelayedExpansion
ECHO File %~f0 changed %~t0 Launched %DATE% %TIME%

SET "NO_PAUSE="    REM  SHOW TIME AND CONTINUE SCRIPT AUTOMATICALLY
SET "NO_PAUSE=PAUSE"   REM  SHOW TIME AND PAUSE TO OBSERVE DRIVE ACTIVITY LEDS

SET "TARGETS=NUL"

SET "SOURCE=C: D:"
SET "TARGETS=H: NUL"
SET "PARENT=###-SpeedTest-#\#"
SET "MASK=1 #2"
REM  EDIT the above 4 lines to control the Source and Destination drives, paths, and files

SET /A N=3
SET /A L_NUM=100

FOR %%m IN (%MASK%) DO FOR %%s IN (%SOURCE%) DO (
  SET "VOL_T="
  for /f "delims=" %%f in (' VOL %%s ') do SET "VOL_T=!VOL_T! %%f "
  ECHO  Source drive %%s =!VOL_T!
  FOR %%t IN (%TARGETS%) DO (
    SET "OUT_PATH=%%t\%PARENT%\T\"
    IF /I "%%t" EQU "NUL" (
      SET "VOL_T=NUL"
      SET "OUT_PATH=NUL"
    ) ELSE (
      SET "VOL_T="
      for /f "delims=" %%f in (' VOL %%t ') do SET "VOL_T=!VOL_T! %%f "
      IF NOT EXIST "!OUT_PATH!\" MD "!OUT_PATH!\"
    )
    ECHO  Target drive %%t =!VOL_T!
    CD /D "%%s\%PARENT%\S"
    for /f "delims=" %%f in (' DIR "%%m*.*" /b /os /a-d /X ') do (
      CALL :DO_IT_HELP %%s %%t "%%f" "%%s\%PARENT%\S"
      FOR /L %%n IN (1,1,%N%) DO CALL :DO_IT_FSU_COPY %%s %%t "%%f" "%%s\%PARENT%\S" %%n
      FOR /L %%n IN (1,1,%N%) DO CALL :DO_IT_COPY %%s %%t "%%f" "%%s\%PARENT%\S" %%n
      IF /I "%%t" NEQ "NUL" (
        FOR /L %%n IN (1,1,%N%) DO CALL :DO_IT_FSU_XCOPY %%s %%t "%%f" "%%s\%PARENT%\S" %%n
        FOR /L %%n IN (1,1,%N%) DO CALL :DO_IT_XCOPY %%s %%t "%%f" "%%s\%PARENT%\S" %%n
        FOR /L %%n IN (1,1,%N%) DO CALL :DO_IT_FSU_XCOPY_J %%s %%t "%%f" "%%s\%PARENT%\S" %%n
        FOR /L %%n IN (1,1,%N%) DO CALL :DO_IT_XCOPY_J %%s %%t "%%f" "%%s\%PARENT%\S" %%n
      )
    )
  )
)
GOTO :EOF

rem for /f "delims=" %%f in (' DIR "%MASK%" /b /os /a-d /X ') do (
rem )

FOR %%m IN (%MASK%) DO ECHO ["%%m*.*"]
goto :eof

:DO_IT_HELP
ECHO Copying     "%~4\"     TO      "%OUT_PATH%"
ECHO Copy File   "%~3" Size %~z3 bytes
GOTO :EOF

:DO_IT_FSU_COPY
SET "T1=%TIME%"
FSUTIL FILE SETVALIDDATA "%~4\%~3" "%~z3" > NUL
CALL :SHOW_TIME %T1% "%5 FSUTIL %~z3"
:DO_IT_COPY
SET "T1=%TIME%"
COPY /B /Y "%~4\%~3" "!OUT_PATH!" > NUL
CALL :SHOW_TIME %T1% "%5 COPY /B /Y %~4\{file} !OUT_PATH!"
GOTO :EOF

:DO_IT_FSU_XCOPY
SET "T1=%TIME%"
FSUTIL FILE SETVALIDDATA "%~4\%~3" "%~z3" > NUL
CALL :SHOW_TIME %T1% "%5 FSUTIL %~z3"
:DO_IT_XCOPY
SET "T1=%TIME%"
XCOPY "%~4\%~3" %OUT_PATH% /Y > NUL
CALL :SHOW_TIME %T1% "%5 XCOPY %~4\{file} %OUT_PATH% /Y"
GOTO :EOF

:DO_IT_FSU_XCOPY_J
SET "T1=%TIME%"
FSUTIL FILE SETVALIDDATA "%~4\%~3" "%~z3" > NUL
CALL :SHOW_TIME %T1% "%5 FSUTIL %~z3"
:DO_IT_XCOPY_J
SET "T1=%TIME%"
XCOPY "%~4\%~3" "%OUT_PATH%" /Y /J > NUL
CALL :SHOW_TIME %T1% "%5 XCOPY %~4\{file} %OUT_PATH% /Y /J"
GOTO :EOF


:SHOW_TIME
SET "T2=%TIME%"
CALL :T_DIFF %T1% %T2%
SET /A L_NUM += 1
ECHO %L_NUM% %TIME_DIFF%   Running  %2
%NO_PAUSE%
GOTO :EOF


:T_DIFF
FOR /F "tokens=1-4 delims=:." %%d in ("%1") do SET /A mS1=%%d*360000+1%%e*6000+1%%f*100+1%%g-610100
FOR /F "tokens=1-4 delims=:." %%d in ("%2") do SET /A mS2=%%d*360000+1%%e*6000+1%%f*100+1%%g-610100
REM Add 5 mSec extra to avoid subsequent "divide by zero errors"and to +/- split digitization error
SET /A DIFF=(((%mS2%-%mS1%)*10)+5)
SET "TIME_DIFF=%T2%-%T1% = %ms2%-%ms1% mSec = %DIFF% mSec"

The output script produced by the above

#2 Post by **foxidrive** » 30 Nov 2013 21:44

Any read and writes from and to the same HDD happens with a single read write head and it slows down.

A SSD would be quicker by far - they are a good investment.

alan_b · #3 Post by **alan_b** » 02 Dec 2013 05:04

Thanks for the polite reminder.

I realise now it was a stupid question.
I learnt when I started using Windows as an administrator at home
(instead of a normal User account subject to the control of the I.T. department)
that Windows would run faster if Pagefile.sys was on a non-system Disk,
because then Windows could leave one head over the programme track it was using whilst the other head stacked over the Pagefile.sys track.
I was only able to use that knowledge once I moved from a single disk laptop to a multi-disk desktop.

NEW QUESTION :-

I can employ FSUTIL for disabling the cache/INPUT-buffer when copying a source file to a target destination.
Does FSUTIL have any option to disable the cache/OUTPUT-buffer so that the target file is instantly written to the disk,
or is their any other executable in System32 that might be of use to me.

Regards
Alan

DosTips.com

HELP :- FSUTIL + COPY versus XCOPY /? ; Caches + i/o Buffers

HELP :- FSUTIL + COPY versus XCOPY /? ; Caches + i/o Buffers

Re: HELP :- FSUTIL + COPY versus XCOPY /? ; Caches + i/o Buf

Re: HELP :- FSUTIL + COPY versus XCOPY /? ; Caches + i/o Buf