New bug in FOR command?

Discussion forum for all Windows batch related topics.

Moderator: DosItHelp

Message
Author
Aacini
Expert
Posts: 1885
Joined: 06 Dec 2011 22:15
Location: México City, México
Contact:

New bug in FOR command?

#1 Post by Aacini » 26 Dec 2012 16:34

I found a strange bug in FOR command at this SO topic. I thought that before a FOR command start the iterations, it saves the status that controlls such iterations in order to avoid endless loops. However, in this case the execution of commands in the FOR body modify the limits of the iterations, but just for one more time!. Here it is:

Code: Select all

C:>dir /b
monsieur_lazhar_.jpg
monsieur_lazhar__ver2.jpg
test.bat

C:>type test.bat
@echo off

   rem Rename "*.jpg" files to 'Poster 1', 'Poster 2', 'Poster 3' ...
   set i=0
   for %%b in (*.jpg) do (
      set /A i+=1
      ren "%%b" "Poster !i!%%~Xb"
      echo File "%%b" renamed to "Poster !i!%%~Xb"
   )

C:>test
File "monsieur_lazhar_.jpg" renamed to "Poster 1.jpg"
File "monsieur_lazhar__ver2.jpg" renamed to "Poster 2.jpg"
File "Poster 1.jpg" renamed to "Poster 3.jpg"

C:>dir /b
Poster 2.jpg
Poster 3.jpg
test.bat
Is this bug documented? Is there a direct way to avoid it? I want to avoid an ugly patch at end of the FOR: :?

Code: Select all

if not exist "Poster 1.jpg" ren "Poster %i%.jpg" "Poster 1.jpg"


Antonio

abc0502
Posts: 1007
Joined: 26 Oct 2011 22:38
Location: Egypt

Re: New bug in FOR command?

#2 Post by abc0502 » 26 Dec 2012 18:29

You can use:

Code: Select all

For /F "delims=" %%b in ('DIR /B *.jpg') Do (
instead.

But it seems that it do what you need if you run the code twice :o , so you can do something like that:

Code: Select all

@echo off
setlocal enabledelayedexpansion
rem Rename "*.jpg" files to 'Poster 1', 'Poster 2', 'Poster 3' ...
:loop
set i=0
set i2=0
for %%b in (*.jpg) do (
    set /A i+=1
    set /A i2+=1
    ren "%%b" "Poster !i!%%~Xb"
    echo File "%%b" renamed to "Poster !i!%%~Xb"
  )
if "!i2!" neq "3" goto :loop
pause

!i2! is for counting how many times the for loop was executed, we need it to run twice so we set the last if to go to loop as it is not equal to 3

or just create a file with the original file names and use it to rename them, but that will require more that one for loop

abc0502
Posts: 1007
Joined: 26 Oct 2011 22:38
Location: Egypt

Re: New bug in FOR command?

#3 Post by abc0502 » 26 Dec 2012 18:40

another idea, the problem as you said is that it doesn't keep track, so instead of storing the original file names in a file, we can store them in a variable (memory), then iterate through that variable and name that files one after the other.

Code: Select all

@echo off
setlocal enabledelayedexpansion

:: Look here, this 2 lines
set "x="
for %%a in (*.jpg) do set "x=!x!%%a "

rem Rename "*.jpg" files to 'Poster 1', 'Poster 2', 'Poster 3' ...
set i=0
for %%b in (!x!) do (
    set /A i+=1
    ren "%%b" "Poster !i!%%~Xb"
    echo File "%%b" renamed to "Poster !i!%%~Xb"
  )
pause

the !x! variable will hold the names of the files (we append the file names to it) like this
file1.jpg file2.jpg file3.jpg

then in your code, the for command will iterate throw these names as they are seprated by a space ( a file-set )

Edit:
BUT it won't work if the files has spaces in it's names :(

Edit: again :D
I found a work arround, instead of :
for %%a in (*.jpg) do set "x=!x!%%a "

it should be like this:
for %%a in (*.jpg) do set "x=!x!^"%%a^" "
all the file names will be qouted, whether it has space or not it will always be qouted

foxidrive
Expert
Posts: 6031
Joined: 10 Feb 2012 02:20

Re: New bug in FOR command?

#4 Post by foxidrive » 26 Dec 2012 20:30

Aacini wrote:I found a strange bug in FOR command


It's been a bug since msdos day, and it can affect more than one file.

using for /f with a dir command is the way I would get around it too.

Liviu
Expert
Posts: 470
Joined: 13 Jan 2012 21:24

Re: New bug in FOR command?

#5 Post by Liviu » 26 Dec 2012 22:21

Aacini wrote:I found a strange bug in FOR command

Seems to be in the same family as http://www.dostips.com/forum/viewtopic.php?p=22897#p22897. Don't know that I'd call it a "bug", since there is no "expected" behavior mandated or guaranteed in any docs I've seen about what "should" happen once the code in the loop modifies the fileset the loop itself works on. This goes all the way from batch 'for' loops down to the underlying system level FindFirst/NextFile APIs.

foxidrive wrote:using for /f with a dir command is the way I would get around it too

That might get around the case in point here, indeed (at the expense of unicode support, as noted in the other thread). Still not foolproof against other programs outside the batch deleting/renaming files while 'dir /b' executes - but then one would have to define what the "expected outcome" would be, first.

Liviu

Ed Dyreen
Expert
Posts: 1569
Joined: 16 May 2011 08:21
Location: Flanders(Belgium)
Contact:

Re: New bug in FOR command?

#6 Post by Ed Dyreen » 27 Dec 2012 01:20

'
I agree with Liviu,

In java an exception is thrown when I try to change the object I am enumerating.

Now I know what to watch out for, thanks for posting this Aacini, :)

foxidrive
Expert
Posts: 6031
Joined: 10 Feb 2012 02:20

Re: New bug in FOR command?

#7 Post by foxidrive » 27 Dec 2012 01:47

Liviu wrote:Still not foolproof against other programs outside the batch deleting/renaming files while 'dir /b' executes


*NO* executable can survive that kind of action while it is running - unless it is actually written into the program for that scenario.

FWIW It's certainly not a new bug as I was aware of it in the dreaded days of MSDOS. The fact that it still hasn't been fixed speaks loudly.

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: New bug in FOR command?

#8 Post by dbenham » 27 Dec 2012 15:37

As others have said, this is a long standng "feature" of FOR, FOR /D, and FOR /R.

Recently I was fooling around with the feature to see if I could create an "infinite" directory based FOR loop. I succeeded, but it is slow :twisted:

The following loop continues until %RANDOM% generates 0.

Code: Select all

@echo off
setlocal enableDelayedExpansion

:: initialize
md dirLoop
pushd dirloop
set "pad=         "
for /l %%N in (101 1 800) do copy nul "%pad%1.%%N" >nul

set /a n=0
for %%F in (*) do (
  set /a n+=1
  2>nul set /a "1/(n%%100)" || echo !n!
  if !random!==0 (
    echo(  exiting at !time!
    goto :break
  )
  set /a new=%%~nF+1
  set "new=%pad%!new!"
  ren "%%F" "!new:~-10!%%~xF" >nul
)

:break
popd
rd /s /q dirLoop
echo(completed at !time!
echo iterations: !n!

The longer the loop runs, the longer it takes to break out. I'm not sure, but I suspect it may have something to do with temporarily preserving a list of all names that have already been iterated. I suspect it needs to do this to make sure that the same file is not iterated twice due to the short 8.3 name. It could take time to clear out the temporary list from memory. But this is all guess work.


Dave Benham

carlos
Expert
Posts: 503
Joined: 20 Aug 2010 13:57
Location: Chile
Contact:

Re: New bug in FOR command?

#9 Post by carlos » 27 Dec 2012 17:34

I understand the problem. For /f and For works different.

For example if you use:
For /F %%a in ('Dir /b *.jpg')
this execute the command and save all the text output. Then you have a list static.

When you use:
For %%a in (*.jpg)
this not keep in memory a list, it look in the hard disk every time, then it have a dinamic list.

When you use ren you affect this dinamic list.
I test put the renamed files in other directory. Check this:

Code: Select all


C:\Users\Carlos\Desktop\test>dir /b
monsieur_lazhar_.jpg
monsieur_lazhar__ver2.jpg
test.bat

C:\Users\Carlos\Desktop\test>type test.bat
@echo off
   md images
   setlocal enabledelayedexpansion

   rem Rename "*.jpg" files to 'Poster 1', 'Poster 2', 'Poster 3' ...
   set i=0
   for %%b in (*.jpg) do (
      set /A i+=1
      move "%%b" "images\Poster !i!%%~Xb"
      echo File "%%b" renamed to "images\Poster !i!%%~Xb"
   )

C:\Users\Carlos\Desktop\test>test.bat
Se han movido         1 archivos.
File "monsieur_lazhar_.jpg" renamed to "images\Poster 1.jpg"
Se han movido         1 archivos.
File "monsieur_lazhar__ver2.jpg" renamed to "images\Poster 2.jpg"

C:\Users\Carlos\Desktop\test>dir /b
images
test.bat

C:\Users\Carlos\Desktop\test>dir /b images
Poster 1.jpg
Poster 2.jpg

C:\Users\Carlos\Desktop\test>

carlos
Expert
Posts: 503
Joined: 20 Aug 2010 13:57
Location: Chile
Contact:

Re: New bug in FOR command?

#10 Post by carlos » 27 Dec 2012 17:43

The difference is similar to this:

You open a file for read. You read entirely and keep in memory, then if other application do a change in the file you are not reading from the file, you are reading from you buffer that is a copy of the file when you read it.

Other way is open the file for read and write, then you read the file byte to byte, and in the end of the file you add other character, then you read a new character out of the original file.

The first is similar to For /F and the second is similar to For and Ren together. All this applied to a filesystem.

Liviu
Expert
Posts: 470
Joined: 13 Jan 2012 21:24

Re: New bug in FOR command?

#11 Post by Liviu » 28 Dec 2012 01:35

foxidrive wrote:*NO* executable can survive that kind of action while it is running - unless it is actually written into the program for that scenario.
Correct. And such "the program" can't be done in batch language, as far as I am aware of.

foxidrive wrote:FWIW It's certainly not a new bug as I was aware of it in the dreaded days of MSDOS. The fact that it still hasn't been fixed speaks loudly.
Still not sure why call it a "bug", since I can't find any reference documentation to build expectations on as to what's supposed to happen in such cases . Also not sure what you'd propose as a "fix", since it's not always as clear cut. In the general case, it's simply not possible to do an arbitrary rename in one pass. Assume for example that Aacini's original 'dir /b' returned

Code: Select all

C:>dir /b images
Poster 2.jpg
Poster 1.jpg
(which is possible if C:\images is on a file system that returns unsorted names e.g. FAT). Then it would take non-trivial code to rename the "first" file to "Poster 1" then the "second" one to "Poster 2".

Liviu

foxidrive
Expert
Posts: 6031
Joined: 10 Feb 2012 02:20

Re: New bug in FOR command?

#12 Post by foxidrive » 28 Dec 2012 02:45

Liviu wrote:Assume for example that Aacini's original 'dir /b' returned

Code: Select all

C:>dir /b images
Poster 2.jpg
Poster 1.jpg
(which is possible if C:\images is on a file system that returns unsorted names e.g. FAT). Then it would take non-trivial code to rename the "first" file to "Poster 1" then the "second" one to "Poster 2".


dir /b /o:n would fix it. :)

Liviu
Expert
Posts: 470
Joined: 13 Jan 2012 21:24

Re: New bug in FOR command?

#13 Post by Liviu » 28 Dec 2012 10:30

foxidrive wrote:dir /b /o:n would fix it. :)

Only on lucky days ;-)

Code: Select all

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\>dir /b /on images
Poster 1.jpg
Poster 10.jpg
Poster 2.jpg

dbenham
Expert
Posts: 2461
Joined: 12 Feb 2011 21:02
Location: United States (east coast)

Re: New bug in FOR command?

#14 Post by dbenham » 28 Dec 2012 10:49

Liviu wrote:
foxidrive wrote:dir /b /o:n would fix it. :)

Only on lucky days ;-)

Code: Select all

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\>dir /b /on images
Poster 1.jpg
Poster 10.jpg
Poster 2.jpg


True, /o:n may not give the desired results, but at least the results will be determinate. A given input set of file names will always produce the same result, regardless how, or in what order, the original files were created.


Dave Benham

Aacini
Expert
Posts: 1885
Joined: 06 Dec 2011 22:15
Location: México City, México
Contact:

Re: New bug in FOR command?

#15 Post by Aacini » 28 Dec 2012 18:47

Ok. The core of this problem is clearly defined now: a FOR-set that remain static while the FOR is executed vs. a FOR-set that change because a command in DO part. Note that this difference is NOT necessarily related to FOR vs. FOR /F; for example:

Code: Select all

for /F "delims=" %%a in (thefile.txt) do (
   echo %%a >> thefile.txt
)

I think previous FOR may read lines created by itself if the file size before the FOR is larger than the buffer used to read the file. The possible explanations of FOR behavior may give us the way to understand it and plan methods to avoid the problem. However, I want to emphasize a particular point of this problem.

Liviu wrote:Don't know that I'd call it a "bug", since there is no "expected" behavior mandated or guaranteed in any docs I've seen about what "should" happen once the code in the loop modifies the fileset the loop itself works on...

...but then one would have to define what the "expected outcome" would be, first.

Liviu

I think the "expected behavior" should be the same in any case. In the original example I used to start this topic, if FOR-command would check that this condition not happen, then "Poster 1.jpg" file would not be renamed to "Poster 3.jpg". On the other hand, if FOR-command does NOT have this checking, then it would enter into an endless rename loop ("Poster 2" to "Poster 4", "Poster 3" to "Poster 5", etc). Why FOR-command rename an already renamed file just one time? This is what I call BUG!

Antonio

Post Reply