Copying lines from multiple logfiles, based on line content

[larss]

I have multiple log files of 10mb each. The content of the log files always look like this:

06.09.2012, 21:36:02:83: xxxxx yyyyyy zzz
06.09.2012, 21:36:02:84: xx yyy zzz
06.09.2012, 21:36:05:24: xxxxxxxxx yyyyyyyyyy zzzz

The log files are in the same folder and have the following names File01.log, File02.log, etc.

I want to create a new log file for every hour. This new log file can contain lines from multiple log files, since a file can end at 21:36:05 because it reached the 10 mb. The next logfile will then continue from that point of time. It is thus very random at what time a log file ends. One thing however is for sure. File02.log always follows up File01 and so further.

My idea was to execute it in a .bat, using a for /f loop to loop through all files and parse out the date and time from the logging. I wanted to convert the date and time to seconds and compare this to an hour interval (which is also in seconds). If the time in seconds is bigger than the interval, the interval should be increased with an hour (both lower and upper bound).

I had something in mind like the following code. I however have not much experience in programming so I am not sure if this is possible. Perhaps one of you guys can give me a push in the right direction or provide me with a solution.

Code: Select all

FOR /f "tokens=1,2 delims= " %%G in ('dir /O:N /b/s ^"C:\logs\R\*.log^"') 

DO 

while (records in the current logfile stille exist) ::insert a condition which checks whether the end of the log file is not reached yet

DO

DateFromRecord = %%G :: parse out the date
TimeFromRecord = %%H :: parse out the time
:: Make a variable that combines both the date and the time to a timestamp
:: Create a function that turns the timestamp into seconds

if (TimestampInSeconds ge lowerbound AND TimestampInSeconds lt higherbound) ::compare the time from file with the interval (in seconds)
then
copy line to a new log file

elseif (TimestampInSeconds ge higherbound) :: if higher than the interval, increase the interval with a day and 
then 
lowerbound(loweverbound + aantal seconden van 1dag)
higherbound(higherbound + aantal seconden van 1dag)

done
done

[foxidrive]

What do you want to do with the log information?

Just start a new log? Schedule a bat file hourly to rename the log file, and a new one should be created - no?

[larss]

I want to start create a new log file. One log file for the time between 07-08 hours, one between 08-09 hours, etc.
The old log files should remain the same, just copy the lines into the new ones.

[foxidrive]

I want to start create a new log file. One log file for the time between 07-08 hours, one between 08-09 hours, etc.
The old log files should remain the same, just copy the lines into the new ones.

If you keep the old log files and are starting a new one, what are you copying into the new one?

[foxidrive]

This thread is multi posted, probably to many forums.

http://forums.techguy.org/dos-other/107 ... based.html

[larss]

If you keep the old log files and are starting a new one, what are you copying into the new one?

I want to copy all the lines between 08.00 and 09.00 hours into a new log file. I want to do this for all inteval, so one new file between 09.00-10.00 hours and so on...

In the old files lines that are in the same hour can be seperated over multiple log files.

[abc0502]

so you want each hour in separated file which name will be the range of the hour

Split the content of the log files into files based on the time "one hour in each file"

[foxidrive]

copy *.log tmp.txt
parse the tmp.txt file and every time the hour digit changes then change the log file name.

[abc0502]

Try This, it should create 2 folders "DB" and "Old [date]",
"DB" Folder will contain ".log" files named in the hour range and will contain the final logs by hour
"Old-[date]" Folder will hold the original log files after the batch finish.

ONLY "Loc" variable needs to be set, it point to main folder that hold the log files.

Code: Select all

@Echo OFF

:: Variables
SET "Loc=%userprofile%\desktop\logs"
SET "DB=%Loc%\DB"
SET "Old=%Loc%\Old"
SET "date=%date:/=%"
SET "stamp=[%date:~6,2%-%date:~4,2%-%date:~8,4%]"

:: Create Database & Back-Up Folders
IF NOT EXIST "%DB%" MD "%DB%"
IF NOT EXIST "%Old%" MD "%Old% %stamp%"

:: Create New Log Files
SETLOCAL EnableDelayedExpansion
FOR /L %%X in (0,1,23) DO (
   SET "N=%%X"
   IF "!N:~1!" EQU "" ( 
      SET /a N2 = !N! + 1
      IF "!N2!" EQU "10" ( IF NOT EXIST "%DB%\0!N!-!N2!.log" Echo.>>"%DB%\0!N!-!N2!.log"
      ) ELSE ( IF NOT EXIST "%DB%\0!N!-0!N2!.log" Echo.>>"%DB%\0!N!-0!N2!.log" )
   ) ELSE ( 
      SET /a N2 = !N! + 1
      IF "!N2!" EQU "24" ( 
         SET "N2=00"
         IF NOT EXIST "%DB%\!N!-!N2!.log" Echo.>>"%DB%\!N!-!N2!.log"
      ) ELSE ( IF NOT EXIST "%DB%\!N!-!N2!.log" Echo.>>"%DB%\!N!-!N2!.log" )
   )
)
ENDLOCAL

:: Process Log Files
SETLOCAL EnableDelayedExpansion
FOR /F "tokens=1,2* delims= " %%A in ('TYPE "%Loc%\*.log"') DO (
   set "time=%%B"
   set "t=!time:~0,2!
   
   FOR /L %%X in (0,1,23) DO (
      SET "N=%%X"
      IF "!N:~1!" EQU "" ( 
         SET /a N2 = !N! + 1
         IF "!N2!" EQU "10" ( IF "!t!" EQU "0!N!" Echo %%A %%B %%C>>"%DB%\0!N!-0!N2!.log"
         ) ELSE ( IF "!t!" EQU "0!N!" Echo %%A %%B %%C>>"%DB%\0!N!-0!N2!.log" )
      ) ELSE ( 
         SET /a N2 = !N! + 1
         IF "!N2!" EQU "24" ( 
            SET "N2=00"
            IF "!t!" EQU "!N!" Echo %%A %%B %%C>>"%DB%\!N!-!N2!.log"
         ) ELSE ( IF "!t!" EQU "!N!" Echo %%A %%B %%C>>"%DB%\!N!-!N2!.log" )
      )
   )

)
ENDLOCAL

:: Back-Up Old log files
FOR /F "delims=" %%A in ('DIR /B /A:-D "%Loc%\*.log"') DO Move "%Loc%\%%A" "%Old% %stamp%"

pause

Note:
hour 09 as an example, if any other log file has that time, it will be all held in the file "09-10.log".

[Aacini]

The Batch file below group lines of the same day and hour from several log files in the same output file, with the day and hour in its name:

Code: Select all

@echo off
setlocal EnableDelayedExpansion
for %%a in (*.log) do (
   for /F "delims=" %%b in (%%a) do (
      set "line=%%b"
      set "file=!line:~0,10!@!line:~12,2!"
      echo %%b >> !file!.out
   )
)

For example, with these two log files: File01.log

Code: Select all

Any previous info
06.09.2012, 21:36:02:83: xxxxx yyyyyy zzz
06.09.2012, 21:36:02:84: xx yyy zzz
06.09.2012, 21:36:05:24: xxxxxxxxx yyyyyyyyyy zzzz

and File02.log

Code: Select all

06.09.2012, 21:36:06:83: xxxxx yyyyyy zzz
06.09.2012, 21:36:06:84: xx yyy zzz
06.09.2012, 21:36:07:24: xxxxxxxxx yyyyyyyyyy zzzz
Any posterior info

the program will create 06.09.2012@21.out

Code: Select all

06.09.2012, 21:36:02:83: xxxxx yyyyyy zzz
06.09.2012, 21:36:02:84: xx yyy zzz
06.09.2012, 21:36:05:24: xxxxxxxxx yyyyyyyyyy zzzz
06.09.2012, 21:36:06:83: xxxxx yyyyyy zzz
06.09.2012, 21:36:06:84: xx yyy zzz
06.09.2012, 21:36:07:24: xxxxxxxxx yyyyyyyyyy zzzz

and the same division is achieved for each day and hour data in all log files.

I hope this is what you want, otherwise I apologize (I don't understand your explanation about converting date&time to seconds, etc).

Antonio