Thursday, November 07, 2019

Parsing Log/Text files

One of the team member was looking for ways to parse through logs to find errors in a big dump of files. I had to go to basics to get the results

Problem Statement: Find all the entries for "Error" and "failed"  in big dump of CSV/Log files. 

Solution: Old school technique ( Use Log parser)

run below commands 

C:\Program Files (x86)\Log Parser 2.2>LOGPARSER "Select Text INTO ERROR.csv from
 D:\xyz\*.log* where Text like '%ERROR%'" -i:TEXTLINE -q:Off

Statistics:
-----------
Elements processed: 14128022
Elements output:    9845
Execution time:     81.36 seconds (00:01:21.36)


C:\Program Files (x86)\Log Parser 2.2>LOGPARSER "Select Text INTO failed.csv fro
m D:\xyz\*.log* where Text like '%failed%'" -i:TEXTLINE -q:Off

Statistics:
-----------
Elements processed: 14128022
Elements output:    2025
Execution time:     76.43 seconds (00:01:16.43)

Integration Design Best Practice

Over the past few years of working on multiple critical systems solution design initiatives, there is one important thing I have learnt.  &q...