Batch script to detect and check log files

[Peter123]

Hi Gurus,

Our system generates some random log files with stamp of date and time every 15 minutes on daily basis as shown on below listing. There are a lot of files generated in a day as the frequency is 15 minutes and it's a headache to check them 1 by 1.

Code: Select all

LOG_20150118210000.txt
LOG_20150118211501.txt
LOG_20150118213000.txt
LOG_20150118214501.txt
LOG_20150118220000.txt
LOG_20150118221500.txt
LOG_20150118223000.txt
LOG_20150118224500.txt
LOG_20150118230001.txt
LOG_20150118231500.txt
LOG_20150118233001.txt
LOG_20150118234500.txt
LOG_20150119000001.txt
LOG_20150119001500.txt
LOG_20150119003001.txt
LOG_20150119004500.txt
LOG_20150119010000.txt
LOG_20150119011501.txt
LOG_20150119013000.txt
LOG_20150119014501.txt
LOG_20150119020000.txt
LOG_20150119021500.txt
LOG_20150119023000.txt
LOG_20150119024500.txt
LOG_20150119030000.txt
LOG_20150119031501.txt

Sample of missing files (5 files):

Code: Select all

LOG_20150118233001.txt
LOG_20150118234500.txt
LOG_20150119000001.txt
LOG_20150119001500.txt
LOG_20150119003001.txt

How to write dos batch script to check and detect those missing files based on current date? Will it possible to check how many files are missing?

Appreciate for your help.


- Peter

[Squashman]

Do you want to know which ones are missing or how many are missing?

[foxidrive]

Is it always on 00/15/30/45 minutes? You didn't say.

[foxidrive]

I think I confused myself - there may be easier ways but this seems to work on a limited test.

EDIT: I changed this few times - had to handle the octal problem too.

Code: Select all

@echo off
setlocal enabledelayedexpansion
set "testnum="
for %%a in (
LOG_20150118210000.txt

LOG_20150118213000.txt
LOG_20150118214501.txt
LOG_20150118220000.txt
LOG_20150118221500.txt
LOG_20150118223000.txt
LOG_20150118224500.txt
LOG_20150118230001.txt
LOG_20150118231500.txt
LOG_20150118233001.txt
LOG_20150118234500.txt
LOG_20150119000001.txt
LOG_20150119001500.txt
LOG_20150119003001.txt
LOG_20150119004500.txt
LOG_20150119010000.txt
LOG_20150119011501.txt
LOG_20150119013000.txt
LOG_20150119014501.txt

LOG_20150119021500.txt
LOG_20150119023000.txt
LOG_20150119024500.txt
LOG_20150119030000.txt
LOG_20150119031501.txt
) do ( 
   set file=%%a
   set /a num=!file:~-8,2!
      if not defined testnum (
         set testnum=!num!
       ) else (
         if "!testnum:~0,1!"=="0" if not "!testnum:~1,1!"=="0" set testnum=!testnum:~-1!
         set /a testnum+=15
         if !testnum! GTR 59 set /a testnum=60-testnum
            if not !num! EQU !testnum! (if !testnum! EQU 0 set testnum=00)&(echo missing !file:~0,-8!!testnum!!file:~-6!)
         set /a testnum=num
       )
)
pause

[Squashman]

I was thinking of just getting the current date. Then creating two nested FOR commands. The first FOR command would have values 00 to 23 and the second FOR command would have 00 to 45.
Then use the date and the two values from the two FOR commands to see if the FILE exists.

[Peter123]

Hi All,

Thank you for your kind response and help.

Here are some samples of files with the actual date and time format.

Code: Select all

19/01/2015  05:00 AM           486,400 LOG_20150118210000.txt
19/01/2015  05:15 AM         4,471,296 LOG_20150118211501.txt
19/01/2015  05:30 AM         3,778,048 LOG_20150118213000.txt
19/01/2015  05:45 AM         4,027,904 LOG_20150118214501.txt
19/01/2015  06:00 AM           105,472 LOG_20150118220000.txt
19/01/2015  06:15 AM         5,421,568 LOG_20150118221500.txt
19/01/2015  06:30 AM         3,829,760 LOG_20150118223000.txt
19/01/2015  06:45 AM         3,760,640 LOG_20150118224500.txt
19/01/2015  07:00 AM           109,568 LOG_20150118230001.txt
19/01/2015  07:15 AM         4,452,864 LOG_20150118231500.txt
19/01/2015  07:30 AM         3,742,208 LOG_20150118233001.txt
19/01/2015  07:45 AM         3,742,208 LOG_20150118234500.txt
19/01/2015  08:00 AM           189,952 LOG_20150119000001.txt
19/01/2015  08:15 AM         4,378,624 LOG_20150119001500.txt
19/01/2015  08:30 AM         3,746,816 LOG_20150119003001.txt
19/01/2015  08:45 AM         3,706,880 LOG_20150119004500.txt
19/01/2015  09:00 AM           112,128 LOG_20150119010000.txt
19/01/2015  09:15 AM         4,469,760 LOG_20150119011501.txt
19/01/2015  09:30 AM         3,757,056 LOG_20150119013000.txt
19/01/2015  09:45 AM         3,788,800 LOG_20150119014501.txt
19/01/2015  10:00 AM           186,368 LOG_20150119020000.txt
19/01/2015  10:15 AM         4,747,264 LOG_20150119021500.txt
19/01/2015  10:30 AM         3,982,848 LOG_20150119023000.txt
19/01/2015  10:45 AM         4,018,688 LOG_20150119024500.txt
19/01/2015  11:00 AM           238,080 LOG_20150119030000.txt
19/01/2015  11:15 AM        10,728,960 LOG_20150119031501.txt

Will it be possible to scan all the files in the current directory based on date?

For example, if today is 19 Jan 2015, the script will scan all the existing files created with this date and check for any missing file which will be created every 15 minutes.

If there is any missing file within 15 minutes interval, the script should indicate it and indicate how many as well.

Thank you.


- Peter

[Squashman]

So is your plan to schedule this task to run every hour? If you are then you would need to run before the next hour hits and after the 45 minute log file. So you should schedule it to run every hour starting at say 11:55 pm. That way way you would not have any issues with date and time overlaps.

[Compo]

@Peter123 I have noted two things from your last post

1. Why bother parsing the log file name when you already have the date and time in the dir output.
2. Why are all of your log files being created with names eight hours before their creation time?

[Squashman]

Put this batch file in the same directory as your log files.

Code: Select all

@ECHO off
setlocal enabledelayedexpansion
:: GET DATE and TIME
for /f "tokens=2 delims==" %%a in ('wmic OS Get localdatetime /value') do set "dt=%%a"
set "YYYY=%dt:~0,4%"
set "MM=%dt:~4,2%"
set "DD=%dt:~6,2%"
set "HH=%dt:~8,2%"
set "Min=%dt:~10,2%"
set "Sec=%dt:~12,2%"
set "datestamp=%YYYY%%MM%%DD%"
set "timestamp=%HH%%Min%%Sec%"
echo %datestamp%_%timestamp%

REM Just using the line below for testing
REM SET HH=23
SET "ALLHOURS=00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23"
SET "TH=!ALLHOURS:*%HH%=!"

IF NOT "%TH%"=="" (
   SET "CHECKHOURS=!ALLHOURS:%TH%=!"
   SET "CHECKHOURS=!CHECKHOURS:~0,-3!"
   ) ELSE ( 
   SET "CHECKHOURS=!ALLHOURS:~0,-3!"
)

IF %Min% GEQ 45 SET "LASTMINCHECK=00 15 30 45"
IF %Min% LSS 45 SET "LASTMINCHECK=00 15 30"
IF %Min% LSS 30 SET "LASTMINCHECK=00 15"
IF %Min% LSS 15 SET "LASTMINCHECK=00"

SET COUNT=0
IF NOT DEFINED CHECKHOURS GOTO LASTHOUR

FOR %%G IN (%CHECKHOURS%) DO (
   FOR %%H IN (00 15 30 45) DO (
      IF NOT EXIST LOG_%datestamp%%%G%%H*.txt (
         SET /A COUNT+=1
         echo LOG_%datestamp%%%G%%H00.txt is missing
      )
   )
)

:LASTHOUR
FOR %%I IN (%LASTMINCHECK%) DO (
   IF NOT EXIST LOG_%datestamp%%HH%%%I*.txt (
      SET /A COUNT+=1
      echo LOG_%datestamp%%HH%%%I00.txt is missing
   )
)
echo %COUNT% Log File(s) are missing.
pause

[Peter123]

So is your plan to schedule this task to run every hour? If you are then you would need to run before the next hour hits and after the 45 minute log file. So you should schedule it to run every hour starting at say 11:55 pm. That way way you would not have any issues with date and time overlaps.

@Squashman
Thank you for you reply and help.
Yes, I'm planning to schedule this script to run hourly to check on those files.

[Peter123]

@Peter123 I have noted two things from your last post
1.Why bother parsing the log file name when you already have the date and time in the dir output.
2.Why are all of your log files being created with names eight hours before their creation time?

@Compo
Thank you for your reply.
1) Those are additional information which I thought maybe useful
2) Not sure why, but those files are generated automatically by the system.

[Peter123]

@Squashman

Thank you very much for your help on the script.

I've tried today and here is the output.

Code: Select all

20150120_093557
LOG_20150120011500.txt is missing
LOG_20150120013000.txt is missing
LOG_20150120014500.txt is missing
LOG_20150120020000.txt is missing
LOG_20150120021500.txt is missing
LOG_20150120023000.txt is missing
LOG_20150120024500.txt is missing
LOG_20150120030000.txt is missing
LOG_20150120031500.txt is missing
LOG_20150120033000.txt is missing
LOG_20150120034500.txt is missing
LOG_20150120040000.txt is missing
LOG_20150120041500.txt is missing
LOG_20150120043000.txt is missing
LOG_20150120044500.txt is missing
LOG_20150120050000.txt is missing
LOG_20150120051500.txt is missing
LOG_20150120053000.txt is missing
LOG_20150120054500.txt is missing
LOG_20150120060000.txt is missing
LOG_20150120061500.txt is missing
LOG_20150120063000.txt is missing
LOG_20150120064500.txt is missing
LOG_20150120070000.txt is missing
LOG_20150120071500.txt is missing
LOG_20150120073000.txt is missing
LOG_20150120074500.txt is missing
LOG_20150120080000.txt is missing
LOG_20150120081500.txt is missing
LOG_20150120083000.txt is missing
LOG_20150120084500.txt is missing
LOG_20150120090000.txt is missing
LOG_20150120091500.txt is missing
LOG_20150120093000.txt is missing
34 Log File(s) are missing.
Press any key to continue . . .

The script does not produce the correct output yet.

At the moment, I can only see these few files:

Code: Select all

LOG_20150120000001.txt
LOG_20150120001500.txt
LOG_20150120003001.txt
LOG_20150120004500.txt
LOG_20150120010001.txt

For info, here is the format of the log files (LOG_year+month+date+hour+minute+second.txt).

I think the script should do the checking for the filename based on the above format, but exclude checking for the second which should provide a better result since the file is created at 15 minutes interval, see below samples (for illustration only):

Code: Select all

LOG_20150119000001.txt
-> Check for: LOG_20150119 00 00
LOG_2015 0119001500.txt
-> Check for: LOG_20150119 00 15
LOG_2015 0119003001.txt
-> Check for: LOG_20150119 00 30 
LOG_2015 0119004500.txt
-> Check for: LOG_20150119 00 45
LOG_2015 0119010000.txt
-> Check for: LOG_20150119 01 00
LOG_2015 0119011501.txt
-> Check for: LOG_2015 01 19 01 15 

Appreciate if you could help to amend the script.

Thank you.


- Peter

[Squashman]

I am finding it hard to understand your reasoning for it not working. The example output you provided proves that it worked.

You said you can see the first 5 files of the day. And you ran the the script at 20150120_093557. Which means you are missing 34 files. The script is checking your EXACT file name format.
You can see that plain as day in the code.

Code: Select all

IF NOT EXIST LOG_%datestamp%%%G%%H*.txt

Datestamp is YYYYMMDD. The token %%G is the hour in a 2 byte format. The token %%H is the minute in a 2 byte format. The asterisk is the wildcard for the seconds.

Explain to me how you think I am not checking the correct file name format?

[Peter123]

@Squashman

Thank you very much for your reply.

Sorry, think I may have created some confusion here. The sample 5 missing files is only used for illustration purposes. In an actual scenario, the missing file could be more than 5 files and it could also be random. Hard to tell when it will happend, that's why I need this script to check.

I got it now, think your method of checking is correct.

However, will it be possible to check only for all existing files which are currently present on the folder? For example, if today I've only 10 files, the script will scan through and check for any missing files.

Also, once any missing file is detected, will it be possible to display the actual filename? This will provide a better information for troubleshooting.

Thank you.


- Peter

[Squashman]

There seems to be a lot confusion as I don't know what you are not understanding. You haven't told me anything yet that the script will not be able to do.

It is January 20th, 2015 at 11:23 AM in Singapore right now. That means there should be 46 LOG files created at this time. The script checks every 15 minute combination up to the current time. At 11:23 am it will check from 00:00 to 11:15 for the Day 20150120.

If there is a file missing it will display it.