Page 1 of 1

FINDSTR woes with blank lines

Posted: 14 Oct 2016 10:47
by Dewie
Well despite the wealth of information that seems to come up from this site with every query I pump in to Google I've been banging my head against the wall for a while now and have resorted to asking for help.

Scenario:

I have a folder full of large log (.txt) files. In these files there is occasionally error messages (easily found with ('FINDSTR /c:"] ERROR"') ) There is also, occasionally, blank lines.

The idea is to find the line number of each error message, pad that value +/- 50 and copy those lines to a new file.

When I don't have any blank line spaces this works fine. I can find the line with the error, do some math, plug it in to a bit of code and I'm good to go:

Code: Select all

<"!file!" (
   for /l %%i in (1 1 !startvar!) do set /p "="
   for /l %%i in (%from% 1 %till%) do (
      set "ln="
      set /p "ln="
   
      echo([%%i] !ln!)>> C:\Folders\Logs\Sum-%~1

REM       SET /a count+=1
)


where %~1 is a file name that gets provided (this bit all works great)

Where I run in to the problem is that the FINDSTR above ignores the lines with white spaces, so the numbers I have for my reference points are all wrong...

What I'm THINKING I need to do (but have yet to hash out) is:

For each file in specified directory:
- Determine if a file has Errors in it (At this point I can check every line, append the line number to it *including empty lines*)
-- Check to see if that line has ] ERROR in it
--- Log the line number of the error in a variable to be used later with the above bit of code

I'm having issues piping the current line in to the next query...

Hopefully some of this makes sense...

Re: FINDSTR woes with blank lines

Posted: 14 Oct 2016 13:21
by foxidrive
Dewie wrote:

Code: Select all

<"!file!" (
   for /l %%i in (1 1 !startvar!) do set /p "="
   for /l %%i in (%from% 1 %till%) do (
      set "ln="
      set /p "ln="
   
      echo([%%i] !ln!)>> C:\Folders\Logs\Sum-%~1

REM       SET /a count+=1
)


Where I run in to the problem is that the FINDSTR above ignores the lines with white spaces, so the numbers I have for my reference points are all wrong...


A tip to get the best answer and in the shortest time is to explain exactly what the task entails and give sample data.

The many cluey people here love nothing better than solving a task and using the most robust and quickest methods.

I'll point out here that your sample code doesn't actually use findstr and a poorly described task is seldom worth the effort to give much advice on. Experience has shown that such an answer leads to further questions as the task is more involved than it first appears.

Re: FINDSTR woes with blank lines

Posted: 14 Oct 2016 15:54
by Squashman
This completely baffles me. Why would you have your question title with FINDSTR and you even comment about FINDSTR in your comments yet your code doesn't even have FINDSTR in it. Apparently we are supposed to be omniscient.

Re: FINDSTR woes with blank lines

Posted: 15 Oct 2016 06:24
by foxidrive
Squashman wrote:your code doesn't even have FINDSTR in it. Apparently we are supposed to be omniscient.


I bought my first bottle of omniscient tablets on ebay years ago. It began to be a regular need after seeing the questions being asked on batch file forums and I have my fix every day, without fail.

I still can't decipher many of the things that are asked though. Maybe I need a higher dose.

Re: FINDSTR woes with blank lines

Posted: 15 Oct 2016 07:29
by penpen
I guess findstr is used within a "for/F", but i'm unsure because then you (Dewie) should have used 'FINDSTR "^"'... .
So i also agree you (Dewie) have to provide more code (probably not all your code is needed - just a minimal example that reproduces your issue), in case this guess is wrong.

Dewie wrote:Where I run in to the problem is that the FINDSTR above ignores the lines with white spaces, so the numbers I have for my reference points are all wrong...
FINDSTR never ignores lines with white spaces:
You can see that whenever you are using option "/N".

The only command i know that does ignore lines with delimiters only (which are defaulted to the whitespace characters SPACE and TABULATOR) is a "for /F" loop.


penpen

Re: FINDSTR woes with blank lines

Posted: 15 Oct 2016 08:58
by Aacini
foxidrive wrote:
Squashman wrote:your code doesn't even have FINDSTR in it. Apparently we are supposed to be omniscient.


I bought my first bottle of omniscient tablets on ebay years ago. It began to be a regular need after seeing the questions being asked on batch file forums and I have my fix every day, without fail.

I still can't decipher many of the things that are asked though. Maybe I need a higher dose.


Some time ago I got a "Merlin-254" crystal ball that allowed me to take a look over the OP's shoulder when the ball was correctly tuned, so I could see the screen of his/her computer. However, it stop working some day; it seems the ball require a battery change...

Antonio

Re: FINDSTR woes with blank lines

Posted: 17 Oct 2016 06:34
by Dewie
:lol: sorry folks... it was a long week of beating my head against the wall and my brain was fried.

New week, new efforts.

Purpose:
To generate a script that runs through a folder full of large log files, pull out the last two error messages (as seen in example below) along with 50 lines of text before/after and place them in a new log file to be emailed or *something* later on.

Problem:
When performing a FINDSTR query on the log file to determine the line number where the error occurs it seems to return a different number than the actual line number where the error occurs.


Example Log file:

Code: Select all

2016-09-21 08:18:54,209 [12] DEBUG



2016-09-21 08:18:54,213 [12] DEBUG Bla bla bla
------------------------------------------------------------------------------------------------------------------------
-- Procedure Name:
-- Generated by
-- Copyright:
------------------------------------------------------------------------------------------------------------------------
CREATE PROCEDURE [xxx].[xxx]
      @id_ nvarchar(32)
    , @ts_PublicID nvarchar(50)
    , @ts_Name nvarchar(80)
    , @ts_Name2 nvarchar(80)
    , @ts_Name3 nvarchar(80)
    , @ts_Description ntext
   
    etc
   
2016-09-21 08:25:17,405 [12] DEBUG xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2016-09-21 08:25:17,406 [12] DEBUG xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2016-09-21 08:25:17,406 [12] DEBUG xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2016-09-21 08:25:17,406 [12] DEBUG xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2016-09-21 08:25:17,407 [12] DEBUG xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2016-09-21 08:25:17,407 [12] DEBUG xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2016-09-21 08:25:17,407 [12] DEBUG xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2016-09-21 08:25:17,408 [12] INFO  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2016-09-21 08:25:17,408 [12] DEBUG xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2016-09-21 08:25:17,408 [12] INFO  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2016-09-21 08:25:17,408 [12] DEBUG xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2016-09-21 08:25:17,409 [12] DEBUG xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2016-09-21 08:25:17,409 [12] DEBUG xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2016-09-21 08:25:17,409 [12] DEBUG xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2016-09-21 08:25:17,410 [12] DEBUG xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2016-09-21 08:25:17,410 [12] DEBUG xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2016-09-21 08:25:17,410 [12] DEBUG xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2016-09-21 08:24:55,159 [15] ERROR xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
   at xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx(String sRegistryKey, String sKeyTag, String sDataTag, Encoding encoding)
   at xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx(String value)
   at xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxGetCSConnectionString()
   at xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxGetConnectionString()
 
etc


2016-09-21 08:25:17,149 [12] DEBUG xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  - Execute Insert
2016-09-21 08:25:17,150 [12] DEBUG xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  - Execute FindByKey
2016-09-21 08:25:17,150 [12] DEBUG xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  - Execute Insert
2016-09-21 08:25:17,151 [12] DEBUG xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  - Execute FindByKey
2016-09-21 08:25:17,151 [12] DEBUG xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  - Execute Insert
2016-09-21 08:25:17,152 [12] DEBUG xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  - Execute FindByKey
2016-09-21 08:25:17,152 [12] DEBUG xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  - Execute Insert
2016-09-21 08:25:17,153 [12] DEBUG xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  - Execute FindByKey
2016-09-21 08:25:17,153 [12] DEBUG xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  - Execute Insert
2016-09-21 08:25:17,154 [12] DEBUG xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  - Execute FindByKey


etc x 11ty billion


Bits of my Code:
The following code is called from the directory where the log files reside (for now)

** Disclamer ** I've made no efforts to clean any of this up yet, I realize there is likely better ways to go about certain things (setting variables = '.' for instance); I'm open for any suggestions.

Code: Select all

CLS
@ECHO OFF
setlocal enabledelayedexpansion
:: Look in each file in the directory
FOR %%f IN (*) DO (
   > nul FINDSTR /N /c:"] ERROR" %%f && (
   
      ::Error message was found
      ECHO.
      REM Set the variables equal to . so they can be referenced later
      REM as it was problematic to reference null
      SET SecondLastError=.
      SET LastError=.
      
      ::Loop through the lines containing Errors and record them
      FOR /F "delims=:" %%a IN ('FINDSTR /n /c:"] ERROR" %%f') DO (
         ::If a new error is found reference the previous one
         SET SecondLastError=!LastError!
         ::Reference the line the error was found on.
         SET LastError=%%a
         REM ECHO %%a >> C:\Automation\Logs\ERRORS-%%f
      )
      
      ECHO %%f Last error^(s^) on line^(s^) !SecondLastError! !LastError!      
      
      ::Call the function to determine what range of lines to record
      CALL:CalcRange %%f !LastError! !SecondLastError!
   )
)


This bit works as intended; for instance for one log file I get the following:

LOG_FILE_NAME.txt.1 Last error(s) on line(s) 106982 107264


The call to 'CalcRange' then does its thing (plus a whole bunch of other logic to determine if the errors are close enough the padded values overlap, are close to the start/end of the file, etc) and spits out:

Second Last error range - From: 106932 To: 107032
Last error range - From: 107214 To: 107314


This is also right. - For each of the last two error message ranges another function is called. Where I run in to the problems is pulling that data out of the log files consistently and in a timely fashion. I've tried a large number of different options (which I've apparently not kept very good track of....)

The following DOES work, it's just extremely slow.

For the sake of making things more simple I've hard coded some values here (This is the last function) just to look at the large log file that's giving me issues:


Code: Select all

CLS
@ECHO OFF
setlocal enabledelayedexpansion

:: Setup values to skip to (saves looking at lines at the first of files which don't contain what we're after)
SET StartTime=%time%
SET /A start=106932 - 1
SET count=106932

:: For each line (Skipping to first error) Check to see if it is in the range we want to record
REM FOR /f "tokens=* skip=%start%" %%a IN ('find /n /v "" ^< "%~1"') DO (
REM FOR /f "delims=* skip=%start%" %%a IN ('find /n /v "" ^< "%~1"') DO (
REM for /f %%i in ('type "%file%"^|FINDSTR /N /c:"] ERROR"') do
REM FOR /f %%a in ('type "REMEMConfig.txt.1"^|findstr /v /c:"] ERROR"') do (
REM for /f %%i in ('type "%~1"^|FIND /v " "') do (

FOR /f "delims=* skip=%start%" %%a IN ('find /n /v "" ^< "LOG_FILE_NAME.txt.1"') DO (
   IF !count! GEQ 106932 (
      IF !count! LEQ 107032 (
         ::If in the range we want to record do so
         ECHO %%a >> C:\Automation\Logs\Sum-TESTLOG.log
         ::If more than the range of lines we want exit
         IF !count! GEQ 107032 GOTO:TESTER
      )
   )
   SET /a count+=1
)

:TESTER
ECHO Ended
ECHO Start Time: %startTime%
ECHO Finish Time: %time%


This works but it takes a pile of time...
Ended
Start Time: 8:51:10.44
Finish Time: 9:00:45.24


I've been trying to replace the FIND with a FINDSTR without any luck; I previously had one call which executed in a timely fashion but it was ignoring blank line spaces for some reason, this resulted in the wrong lines being returned.

Hopefully what I've provided is adequate this time around for you folks to understand what I'm trying to do.

Re: FINDSTR woes with blank lines

Posted: 18 Oct 2016 17:34
by Compo
I have made an attempt at a rewrite based upon what I could make out from your above post.
I am hoping this will read through all of your files and output to the testlog file the errors with sufficient lines above and below.

Code: Select all

@ECHO OFF
FOR %%A IN (*) DO CALL :START "%%A"
EXIT/B

:START
SET "FN=%~1"
FOR /F "DELIMS=:" %%A IN ('FINDSTR/NC:"] ERROR" "%FN%"') DO (CALL SET NB=%%LE%%
    SET "LE=%%A")
ECHO=%FN% Last errors on lines %NB% %LE%
FOR %%A IN (%NB% %LE%) DO ECHO=%%A>>"C:\Automation\Logs\ERRORS-%FN%"
IF "%NB%"=="" (SET NB=%LE%)
SET/A LD=1%LE%-1%NB%
SET/A FP=%NB%-50
IF 0 GTR %FP% SET "FP=0"
SET/A LP=%LE%+50
IF 100 GTR %LD% (CALL :TLOG %FP% %LP%) ELSE (SET/A MP=FP+100
    SET/A NP=LP-100
    CALL :TLOG %FP% %MP% %NP% %LP%)
GOTO :EOF

:TLOG
(FOR /F "TOKENS=1* DELIMS=:" %%A IN ('FINDSTR/RN "^" "%FN%"') DO (
    IF 1%%A GEQ 1%1 (IF 1%%A LEQ 1%2 ECHO=%%B)
    IF NOT "%3"=="" (IF 1%%A GEQ 1%3 (IF 1%%A LEQ 1%4 ECHO=%%B)))
)>>"C:\Automation\Logs\Sum-TESTLOG.log"
GOTO :EOF
Try it and see if it helps!

Re: FINDSTR woes with blank lines

Posted: 20 Oct 2016 03:52
by foxidrive
Aacini wrote:
foxidrive wrote:
Squashman wrote:your code doesn't even have FINDSTR in it. Apparently we are supposed to be omniscient.


I bought my first bottle of omniscient tablets on ebay years ago. It began to be a regular need after seeing the questions being asked on batch file forums and I have my fix every day, without fail.

I still can't decipher many of the things that are asked though. Maybe I need a higher dose.


Some time ago I got a "Merlin-254" crystal ball that allowed me to take a look over the OP's shoulder when the ball was correctly tuned, so I could see the screen of his/her computer. However, it stop working some day; it seems the ball require a battery change...

Antonio


I've read of that model Merlin causing that issue after it ran through 5K interations. There's a BIOS upgrade that fixes that problem, and the battery has a half-life of 400 years so that's still in pretty good shape.

Re: FINDSTR woes with blank lines

Posted: 20 Oct 2016 04:05
by foxidrive
Dewie wrote:Purpose:
To generate a script that runs through a folder full of large log files, pull out the last two error messages (as seen in example below) along with 50 lines of text before/after and place them in a new log file to be emailed or *something* later on.


There are two native batch scripts that can give you simple ways to solve this task. They both process large files quickly too.
There are filesize limits so give the guys some indication of the maximum size you are processing.

Aacini's findrepl viewtopic.php?f=3&t=4697
and Dave Benham's Jrepl viewtopic.php?f=3&t=6044


Complex scripts are fine in some situations but when it comes to new employees editing the scripts down the track or you yourself coming back to change the script in a year or two - then you'll often find that re-learning what you've done is needed.

Adding comments to a batch script is a good way to help in the future - but who does that!??? :D

Re: FINDSTR woes with blank lines

Posted: 20 Oct 2016 04:08
by foxidrive
Make sure that you test Compo's script though and let him know how you go.

Re: FINDSTR woes with blank lines

Posted: 20 Oct 2016 04:19
by foxidrive
I rattled my cage without looking at your code and I see there are many comments.
I want to point out here that using :: inside a loop for a remark can cause the script to fail.

Get used to using :: outside loops and rem when adding remarks inside loops and you won't strike that problem.

Code: Select all

FOR /f "delims=* skip=%start%" %%a IN ('find /n /v "" ^< "LOG_FILE_NAME.txt.1"') DO (
   IF !count! GEQ 106932 (
      IF !count! LEQ 107032 (
         ::If in the range we want to record do so
         ECHO %%a >> C:\Automation\Logs\Sum-TESTLOG.log
         ::If more than the range of lines we want exit
         IF !count! GEQ 107032 GOTO:TESTER
      )
   )
   SET /a count+=1
)

Re: FINDSTR woes with blank lines

Posted: 20 Oct 2016 09:59
by Dewie
Sadly I've been pulled a different direction with this initiative and am now using C# (in just a few hours I had it working 100% as required too - weird).

I had a look at the provided solution and, while I'm sure I could make use of it, I simply don't have the time to try to make it work with my existing code (It doesn't quite do what I wanted/need in its current form)

It does still annoy me that I couldn't quite nail down what needed to be done; once I get this out of the way I'll try to revisit the .bat file method as I KNOW it can work...

Thanks for the suggestions folks!

Re: FINDSTR woes with blank lines

Posted: 20 Oct 2016 13:21
by Compo
Dewie wrote:I had a look at the provided solution and, while I'm sure I could make use of it, I simply don't have the time to try to make it work with my existing code (It doesn't quite do what I wanted/need in its current form)
Hold on!

From each file in a directory, it takes the error lines together with 50 lines adjacent (above and below), unless there's an overlap and outputs them to a single file with a specific name and path.

That is exactly what you asked for, isn't it?