How can I use bash (grep/sed/etc) to grab a section of a logfile between 2 timestamps?

bash parsing logfiles timestamp
Brent picture Brent · May 6, 2009 · Viewed 11.2k times · Source

I have a set of mail logs: mail.log mail.log.0 mail.log.1.gz mail.log.2.gz

each of these files contain chronologically sorted lines that begin with timestamps like:

May 3 13:21:12 ...

How can I easily grab every log entry after a certain date/time and before another date/time using bash (and related command line tools) without comparing every single line? Keep in mind that my before and after dates may not exactly match any entries in the logfiles.

It seems to me that I need to determine the offset of the first line greater than the starting timestamp, and the offset of the last line less than the ending timestamp, and cut that section out somehow.

Answer

Dylan picture Dylan · May 6, 2009

Convert your min/max dates into "seconds since epoch",

MIN=`date --date="$1" +%s`
MAX=`date --date="$2" +%s`

Convert the first n words in each log line to the same,

L_DATE=`echo $LINE | awk '{print $1 $2 ... $n}'`
L_DATE=`date --date="$L_DATE" +%s`

Compare and throw away lines until you reach MIN,

if (( $MIN > $L_DATE )) ; then continue ; fi

Compare and print lines until you reach MAX,

if (( $L_DATE <= $MAX )) ; then echo $LINE ; fi

Exit when you exceed MAX.

if (( $L_DATE > $MAX )) ; then exit 0 ; fi

The whole script minmaxlog.sh looks like this,

#!/usr/bin/env bash

MIN=`date --date="$1" +%s`
MAX=`date --date="$2" +%s`

while true ; do
    read LINE
    if [ "$LINE" = "" ] ; then break ; fi

    L_DATE=`echo $LINE | awk '{print $1 " " $2 " " $3 " " $4}'`
    L_DATE=`date --date="$L_DATE" +%s`

    if (( $MIN > $L_DATE  )) ; then continue ; fi
    if (( $L_DATE <= $MAX )) ; then echo $LINE ; fi
    if (( $L_DATE >  $MAX )) ; then break ; fi
done

I ran it on this file minmaxlog.input,

May 5 12:23:45 2009 first line
May 6 12:23:45 2009 second line
May 7 12:23:45 2009 third line
May 9 12:23:45 2009 fourth line
June 1 12:23:45 2009 fifth line
June 3 12:23:45 2009 sixth line

like this,

./minmaxlog.sh "May 6" "May 8" < minmaxlog.input

Related questions

Parsing JSON with Unix tools

I'm trying to parse JSON returned from a curl request, like so: curl 'http://twitter.com/users/username.json' | sed -e 's/[{}]/''/g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[…

Read More
How to join multiple lines of file names into one with custom delimiter?

I would like to join the result of ls -1 into one line and delimit it with whatever i want. Are there any standard Linux commands I can use to achieve this?

Read More
An efficient way to transpose a file in Bash

I have a huge tab-separated file formatted like this X column1 column2 column3 row1 0 1 2 row2 3 4 5 row3 6 7 8 row4 9 10 11 I would like to transpose it in an efficient way using only bash commands (I could write a ten or so lines Perl …

Read More