create folders from a list and move files

[Endoro]

@foxidrive give me one more shot, please

this does not create the folders, subfolders and copy (move) the files to their respective folders.

indeed, so here we go:

Code: Select all

@echo off&setlocal
set "startfolder=%~1" &rem parameter #1
set "ClassID_ID_List=%~2" &rem parameter #2
set "CSVfile=%~3" &rem parameter #3
set "CSVdelim=   " &rem, delimiter for output, default <tab>
rem            ^---< put your <tabspace> there

cd /d "%startfolder%"
if not exist "%ClassID_ID_List%" echo("%ClassID_ID_List%" not found&goto:eof
for /f "usebackqtokens=2" %%i in ("%ClassID_ID_List%") do (
 for %%f in ("%%i_*") do set /a %%~xf=%%~xf + 1 2>nul
)
set "." >nul 2>&1 || (echo(no files from "%ClassID_ID_List%" found in "%startfolder%"&goto:eof)
((for /f "delims==." %%i in ('set "."') do <nul set/p"=%%i%CSVdelim%")&echo(
(for /f "tokens=2delims==" %%i in ('set "."') do <nul set/p"=%%i%CSVdelim%")&echo(
)>"%CSVfile%"
for /f "usebackqtokens=1,2" %%i in ("%ClassID_ID_List%") do (
   if not exist "%%i\%%j\" md "%%i\%%j"
   move "%%j_*" "%%i\%%j"
)

The script overwrites all files in the subfolders, if there any exists.

Example, input:

Code: Select all

C:\USERS\SIFAR786
└───desktop
        10001_ANU.jpg
        10001_COE.jpg
        10001_COE.pdf
        10001_ENU.jpg
        10001_ENU.mdb
        10001_ENU.pdf
        10002_ENI.jpg
        10002_EVS.cls
        10002_TAS.docx
        10002_TNU.png
        10003_NIO.xlsm
        10003_NOE.doc
        10003_RAS.ppt
        10004_GAR.xls
        ClassID_ID_List.txt

Output:

Code: Select all

C:\USERS\SIFAR786
└───desktop
    │   ClassID_ID_List.txt
    │   Out.csv
    │
    ├───1
    │   ├───10001
    │   │       10001_ANU.jpg
    │   │       10001_COE.jpg
    │   │       10001_COE.pdf
    │   │       10001_ENU.jpg
    │   │       10001_ENU.mdb
    │   │       10001_ENU.pdf
    │   │
    │   └───10004
    │           10004_GAR.xls
    │
    ├───2
    │   └───10002
    │           10002_ENI.jpg
    │           10002_EVS.cls
    │           10002_TAS.docx
    │           10002_TNU.png
    │
    └───3
        └───10003
                10003_NIO.xlsm
                10003_NOE.doc
                10003_RAS.ppt

ClassID_ID_List:

Code: Select all

1   10001
1   10004
2   10002
3   10003

[sifar786]

Hi Endoro,

this is good! i was able to create the respective folders and generate the files.

just trying to understand the code. Correct me where ever i go wrong:

Code: Select all

@echo off&setlocal
set "startfolder=%~1" &rem parameter #1
set "ClassID_ID_List=%~2" &rem parameter #2
set "CSVfile=%~3" &rem parameter #3
set "CSVdelim=   " &rem, delimiter for output, default <tab>
rem            ^---< put your <tabspace> there

cd /d "%startfolder%"
if not exist "%ClassID_ID_List%" echo
(
"%ClassID_ID_List%" not found&goto:eof

for /f "usebackq tokens=2" %%i in ("%ClassID_ID_List%") do 
(
 for %%f in ("%%i_*") do set /a %%~xf=%%~xf + 1 2>nul
)

1] the 1st four SET functions are setting the parameters the file will accept at the command line.
2] then we change to respective drive and folder.
3] we check if classid_id_list.txt file exists, else we Echo message and end code.
Question : "Echo" does not have a closing brace here.

4] then we pickup 2nd column values from each line in ClassID_ID_List.txt file and append it to "_* to create search string for files and count their extensions. Any stderrors are thrown to NUL.


However, i am unable to understand the code in below section. Also, again "Echo" closing braces are missing to enclose a block of statements.


5] SET here is used to loop thru all files found in main folder. any errors and output are sent to one file (which file?). OR Echo that 'no files from input text file were found.'

Code: Select all

set "." >nul 2>&1 || (echo(no files from "%ClassID_ID_List%" found in "%startfolder%"&goto:eof)

6] Can you explain me this code? I am lost!!!
Why the double opening braces before 1st FOR? What is the 1st FOR doing? What is the 2nd FOR doing? What will 'delims==.' do....similarly, what will 'delims==' do?

Code: Select all

((for /f "delims==." %%i in ('set "."') do <nul set/p"=%%i%CSVdelim%")&echo(
(for /f "tokens=2 delims==" %%i in ('set "."') do <nul set/p"=%%i%CSVdelim%")&echo(
)>"%CSVfile%"

7] Here from 1st two columns in the input file you are checking if any folders already exist by that name, if not create them and move similar files into them.

Code: Select all

for /f "usebackqtokens=1,2" %%i in ("%ClassID_ID_List%") do (
   if not exist "%%i\%%j\" md "%%i\%%j"
   move "%%j_*" "%%i\%%j"
)

[Endoro]

I don't find the time to look inside now, but perhaps later in one of these days.

[Ed Dyreen]

However, i am unable to understand the code in below section. Also, again "Echo" closing braces are missing to enclose a block of statements.

Confusing innit, no closing braces are required cause it was never opened
The command is not 'echo' but 'echo ' or 'echo.' or 'echo('.
'echo(' is considered safer as the other variants fail under certain circumstances.

6] Can you explain me this code? I am lost!!!
Why the double opening braces before 1st FOR? What is the 1st FOR doing? What is the 2nd FOR doing? What will 'delims==.' do....similarly, what will 'delims==' do?

Code: Select all

((for /f "delims==." %%i in ('set "."') do <nul set/p"=%%i%CSVdelim%")&echo(
(for /f "tokens=2 delims==" %%i in ('set "."') do <nul set/p"=%%i%CSVdelim%")&echo(
)>"%CSVfile%"

Try this for /?, scroll down to where delims is explained, it set the char/s to use to split the input.

Indention is key to maintainable code, I remove unrequired braces, change it a bit.

Code: Select all

> "%CSVfile%" (

     for /f "delims==." %%? in (
          'set "."'
     ) do <nul set /p "=%%?%CSVdelim%"
     echo.
     for /f "tokens=2 delims==" %%? in (
          'set "."'
     ) do <nul set /p "=%%?%CSVdelim%"
     echo.
)

Try set /? and set ".", as you see that is the input that is being used, and the delim is '='.

4] Lastly, is there a good website where i can learn advance DOS programming?

I would recommend this website

lol _http://www.dostips.com, we teach you

Hope now is a bit more clear,

[sifar786]

Thanks for the information. i will get back to you on it once i go through the code with this information in mind.

as of now, i am getting another issue with this code. this code does not seem to work when,
1] i move folders between network drives (source network drive, destination network drive).
2] even if i specify a single network folder to read its contents and create folders in it, it still doesnt work.

i think this has to do with network folder mappings. i.e. finding out the 1st available free drive and mapping the network folder to it using NET USE.

How do i modify this wonderful code?

[Endoro]

@Ed Dyreen thank you for the nice explanation


@sifar786 you can replace the " cd /d "%startfolder%" " command with pushd, eg.:

Code: Select all

pushd \\192.168.178.1\ressource.name

This does create a new drive letter and makes it current.
Put "popd" at the end of the script to remove the drive letter again.

[sifar786]

Hi Endoro,

Thanks for reverting so soon!

actually if i use PUSHD i wont know the drive letter, right? then how am i going to use it in code if i want to move/copy files from a source network drive to a destination network drive? i.e. there will be 2 drives and i wont know which is the current one. how can this be implemented in your code to handle 2 network drives?

[Endoro]

if you want to move or copy files between network ressources you (the script) need to know the network path.

How to set it and store it in a variable you can see in the following shell session:

Code: Select all

C:\TEST>echo off
pushd \\192.168.178.1\endoro.nas
for %i in ("%cd%") do set NetPath1=%~i
echo %NetPath1%
Y:\
echo on

Y:\+>

You can do this for more than one network ressources and use the stored net path's to move and copy files between the assigned ressources.

[sifar786]

Ok, when i do that with each of these two network drives:

Code: Select all

pushd \\10.20.12.129\shared\Inputs Folder

similarly,

Code: Select all

pushd \\10.50.46.188\shared\Outputs Folder

when i check with 'NET USE', i see,

Code: Select all

Y:\       \\10.20.12.129\shared
Z:\       \\10.50.46.188\shared

and not,

Code: Select all

Y:\       \\10.20.12.129\shared\Inputs Folder
Z:\       \\10.50.46.188\shared\Outputs Folder

Code in test18.bat file:

Code: Select all

@echo off
setlocal enableextensions
setlocal enabledelayedexpansion
pushd "\\10.20.12.129\shared\Inputs Folder"
for %%i in ("%cd%") do set xxx=%%~i
echo %xxx%

pushd "\\10.50.46.188\shared\Outputs Folder"
for %%i in ("%cd%") do set yyy=%%~i
echo %yyy%

Output from batch file:

Code: Select all

C:\Users\sifar786\Desktop\TEST>test18
C:\Users\sifar786\Desktop\TEST>setlocal enableextensions
C:\Users\sifar786\Desktop\TEST>setlocal enabledelayedexpansion
C:\Users\sifar786\Desktop\TEST>pushd "\\10.20.12.129\shared\Inputs Folder"
Z:\Inputs Folder>for %i in ("Z:\Inputs Folder") do set xxx=%~i
Z:\Inputs Folder>set xxx=Z:\Inputs Folder
Z:\Inputs Folder>echo Z:\Inputs Folder
Z:\Inputs Folder
Z:\Inputs Folder>pushd "\\10.50.46.188\Shared\Outputs Folder"
Y:\Outputs Folder>for %i in ("Y:\Outputs Folder") do set yyy=%~i
Y:\Outputs Folder>set yyy=Y:\Outputs Folder
Y:\Outputs Folder>echo Y:\Outputs Folder
Y:\Outputs Folder
C:\Users\sifar786\Desktop\TEST>

as you can see, there are spaces in the network paths, so it is only mapping the 1st parent folder. i tried removing the spaces too and also putting the path in Quotes. but still PUSHD maps till only the 1st parent folder. i just want your code to work with moving (or copying) files appropriately from one network drive to another network drive.

[sifar786]

Hi Ed Dyreen,

However, i am unable to understand the code in below section. Also, again "Echo" closing braces are missing to enclose a block of statements.

Confusing innit, no closing braces are required cause it was never opened
The command is not 'echo' but 'echo ' or 'echo.' or 'echo('.
'echo(' is considered safer as the other variants fail under certain circumstances.

6] Can you explain me this code? I am lost!!!
Why the double opening braces before 1st FOR? What is the 1st FOR doing? What is the 2nd FOR doing? What will 'delims==.' do....similarly, what will 'delims==' do?

Code: Select all

((for /f "delims==." %%i in ('set "."') do <nul set/p"=%%i%CSVdelim%")&echo(
(for /f "tokens=2 delims==" %%i in ('set "."') do <nul set/p"=%%i%CSVdelim%")&echo(
)>"%CSVfile%"

Try this for /?, scroll down to where delims is explained, it set the char/s to use to split the input.

Questions:
1] what does the "((" double-braces at beginning of FOR loop do?
2] what does the "delim==." do? what does the "delim==" do?
3] Is

Code: Select all

('set "."')

used to loop thru files in a folder OR is it used to loop thru text in a file?
4] what is contained in %%i and why is it set as prompt? Where is it redirected and what is NUL used here for?

Code: Select all

<nul set/p"=%%i%CSVdelim%"

Indention is key to maintainable code, I remove unrequired braces, change it a bit.

Code: Select all

> "%CSVfile%" (

     for /f "delims==." %%? in (
          'set "."'
     ) do <nul set /p "=%%?%CSVdelim%"
     echo.
     for /f "tokens=2 delims==" %%? in (
          'set "."'
     ) do <nul set /p "=%%?%CSVdelim%"
     echo.
)

Try set /? and set ".", as you see that is the input that is being used, and the delim is '='.

Questions:
1] you removed unnecessary "(" from this code, but you put the

Code: Select all

> "%CSVfile%"

at the beginning now. Why?

4] Lastly, is there a good website where i can learn advance DOS programming?

I would recommend this website

lol _http://www.dostips.com, we teach you

Hope now is a bit more clear,

[Endoro]

the %startfolder% parameter not longer exist, it comes now from the "pushd" command

Code: Select all

@echo off&setlocal
set "ClassID_ID_List=%~1" &rem parameter #1
set "CSVfile=%~2" &rem parameter #2
set "CSVdelim=   " &rem, delimiter for output, default <tab>
rem            ^---< put your <tabspace> there

pushd "\\10.20.12.129\shared\Inputs Folder"
for %%i in ("%cd%") do set "startfolder=%%~i"

pushd "\\10.50.46.188\shared\Outputs Folder"
for %%i in ("%cd%") do set "targetfolder=%%~i"
cd /d "%startfolder%"

if not exist "%ClassID_ID_List%" echo("%ClassID_ID_List%" not found&goto:eof
for /f "usebackqtokens=2" %%i in ("%ClassID_ID_List%") do (
 for %%f in ("%%i_*") do set /a %%~xf=%%~xf + 1 2>nul
)
set "." >nul 2>&1 || (echo(no files from "%ClassID_ID_List%" found in "%startfolder%"&goto:eof)
((for /f "delims==." %%i in ('set "."') do <nul set/p"=%%i%CSVdelim%")&echo(
(for /f "tokens=2delims==" %%i in ('set "."') do <nul set/p"=%%i%CSVdelim%")&echo(
)>"%CSVfile%"
for /f "usebackqtokens=1,2" %%i in ("%ClassID_ID_List%") do (
   if not exist "%%i\%%j\" md "%%i\%%j"
   move "%%j_*" "%targetfolder%\%%i\%%j"
)

[sifar786]

Hi Endoro,

BTW did you get a chance to look at my earlier 2 messages? they speak of an issue with PUSHD where it cannot map a path containing spaces. Again, i cannot understand the latter part of your code.

I tried running the latest code on the two network drives w/o putting @Echo Off to see the command line output.
it runs this to calculate count of filetypes for each ID:

Code: Select all

if not exist "%ClassID_ID_List%" echo("%ClassID_ID_List%" not found&goto:eof
for /f "usebackq tokens=1" %%i in ("%ClassID_ID_List%") do (
 for %%f in ("%%i_*") do set /a %%~xf=%%~xf + 1 2>nul
)

But this doesn't seem to work:

Code: Select all

set "." >nul 2>&1 || (echo(no files from "%ClassID_ID_List%" found in "%startfolder%"&goto:eof)
((for /f "delims==." %%i in ('set "."') do <nul set/p"=%%i%CSVdelim%")&echo(
(for /f "tokens=2 delims==" %%i in ('set "."') do <nul set/p"=%%i%CSVdelim%")&echo(
)>"%CSVfile%"
for /f "usebackq tokens=3,2,1" %%i in ("%ClassID_ID_List%") do (
   if not exist "%%i\%%j\%%k" md "%%i\%%j\%%k"
   Echo N|xcopy "%%k_*" "%targetfolder%%%i\%%j\%%k"

Can you explain what each line is doing?
1] i understand that the former code on top only calculates the filetype counts in current folder using the ID's file.

2] but what is this code doing after the calculations happen?

3] please note i have around 2 million files to move into their respective (created folders). So the entire time goes into calculating counts of filetypes.

Would it be faster to,
- 1st create the folders from list
- find the count of files and dump it to Output CSV file
- move/copy the filetypes into respective folders

[sifar786]

Hi Endoro/ Ed Dyreen,

I tried

Code: Select all

SET "."

from

Code: Select all

SET "." >nul 2>&1 || (echo(no files from "%ClassID_ID_List%" found in "%startfolder%"&goto:eof)

on one of the Network drives to see what is the result, but it gave an error.

Code: Select all

Environment variable . not defined

I fail to understand what it does as there is no such documented information on the net (Google) or in HELP on the SET command.

Can you please look at my earlier two messages and explain the latter part of your code? Would really appreciate it. Thanks in advance.

[Endoro]

Hi,

Code: Select all

set "." >nul 2>&1 || (echo(no files from "%ClassID_ID_List%" found in "%startfolder%"&goto:eof)

check for files and print an error message, if no files found.
After the termination of the script the environment variables also terminate. You get "Environment variable . not defined".
If you want to look for the variables after the end of the script, remove the "setlocal" command from the script (not recommended).

Code: Select all

((for /f "delims==." %%i in ('set "."') do <nul set/p"=%%i%CSVdelim%")&echo(
(for /f "tokens=2 delims==" %%i in ('set "."') do <nul set/p"=%%i%CSVdelim%")&echo(
)>"%CSVfile%"

this compiles the output for the csv table.

Code: Select all

for /f "usebackqtokens=1,2" %%i in ("%ClassID_ID_List%") do (
   if not exist "%%i\%%j\" md "%%i\%%j"
   move "%%j_*" "%targetfolder%\%%i\%%j"
)

create folders for files and move them there.


I can't say more to the network issue, this is not my specialty.

[sifar786]

hi Endoro,

Though i still fail to understand what

Code: Select all

SET "."

means or does, i have got the code working by changing a few things. for e.g. the %targetfolder% was not specified while making the directories. i also eliminated the linefeeds between each statements and also removed wrap text so that each command could fall on its own line.

However, i am not getting the same output as shown before now:

Code: Select all

ID      cls     doc     docx    jpg     mdb     pdf     png     ppt     xls     xlsm
10001   0       0       0       3       1       2       0       0       0       0
10002   1       0       1       1       0       0       1       0       0       0
10003   0       1       0       0       0       0       0       1       0       1
10004   0       0       0       0       0       0       0       0       1       0

But getting an output like this:

Code: Select all

Z:\Output Folder>set/p"=JPG   "0<nul 
JPG   
Z:\Output Folder>set/p"=mov   "0<nul 
mov   
Z:\Output Folder>set/p"=pdf   "0<nul 
pdf   

Z:\Output Folder>set/p"=39   "0<nul 
39   
Z:\Output Folder>set/p"=1   "0<nul 
1   
Z:\Output Folder>set/p"=2   "0<nul 
2   

any ideas why this is happening?

Hi,

Code: Select all

set "." >nul 2>&1 || (echo(no files from "%ClassID_ID_List%" found in "%startfolder%"&goto:eof)

check for files and print an error message, if no files found.
After the termination of the script the environment variables also terminate. You get "Environment variable . not defined".
If you want to look for the variables after the end of the script, remove the "setlocal" command from the script (not recommended).

Code: Select all

((for /f "delims==." %%i in ('set "."') do <nul set/p"=%%i%CSVdelim%")&echo(
(for /f "tokens=2 delims==" %%i in ('set "."') do <nul set/p"=%%i%CSVdelim%")&echo(
)>"%CSVfile%"

this compiles the output for the csv table.

Code: Select all

for /f "usebackqtokens=1,2" %%i in ("%ClassID_ID_List%") do (
   if not exist "%%i\%%j\" md "%%i\%%j"
   move "%%j_*" "%targetfolder%\%%i\%%j"
)

create folders for files and move them there.


I can't say more to the network issue, this is not my specialty.