Logstash Grok pattern with double quotes

regex elasticsearch logstash logstash-grok
c-val picture c-val · Feb 24, 2016 · Viewed 19.3k times · Source

I am parsing proxy logs with Logstash and its Grok filter. The logs contain quoted strings :

1438120705 [.....] "SEF-EDP8" - "C"
"/GPM/1023/5745-7/456V/"

With the Grok Debugger the following pattern works like a charm :

%{NUMBER:ts} [......] (-|"%{USERNAME:token1}") (-|%{DATA:token2}) (-|"%{WORD:token3}") (-|"%{DATA:token4}")

This does not work with Logstash's Grok because of the double quotes in the grok pattern. Logstash error log :

Error: Expected one of #, {, } at line 9, column 204 (byte 374) after
filter {
    grok {
        match => { "message" => "%{NUMBER:ts} [......] ("

So I use the QuotedString grok pattern instead :

%{NUMBER:ts} [......] (-|%{QS:token1}) (-|%{DATA:token2}) (-|%{QS:token3}) (-|%{QS:token4})

This works with the Grok Debugger as well, but quotes are extracted with quoted strings. It doesn't work with Logstash either :

token1 : ""SEF-EDP8"" token2 : null token3 : ""C"" token4 :
""/GPM/1023/5745-7/456V/""

How can I make it work with Logstash? How can I remove these unwanted extra double quotes?

Answer

NileshP picture NileshP · Jul 5, 2016

If you escape " with backslash then it works fine.

%{NUMBER:ts} [......] (-|"%{USERNAME:token1}") (-|%{DATA:token2}) (-|"%{WORD:token3}") (-|"%{DATA:token4}")

Your new string will look like

%{NUMBER:ts} [......] (-|\"%{USERNAME:token1}\") (-|%{DATA:token2}) (-|\"%{WORD:token3}") (-|\"%{DATA:token4}\")

Related questions

Regex query syntax examples in kibana

I am trying to find the different kinds of syntax I can give in regex type of query through kibana, but I was not able to find any information on this. I am running logstash and elasticsearch in the backend. …

Read More
Kibana query exact match

I would like to know how to query a field to exactly match a string. I'm actually trying to query like this: url : "http://www.domain_name.com" Which returns all string starting with http://www.domain_name.com .

Read More
Regular expression to match a line that doesn't contain a word

I know it's possible to match a word and then reverse the matches using other tools (e.g. grep -v). However, is it possible to match lines that do not contain a specific word, e.g. hede, using a regular …

Read More