Writing grok patterns

The syntax for a grok pattern is %{SYNTAX:SEMANTIC} The SYNTAX is the name of the pattern that will match your text. The SEMANTIC is the identifier given to a matched text.

You can use any grok debugger to write your grok pattern and extract content for displaying as columns in your log monitoring dashboard. For example, following is your log pattern.

x.x.x.90 - - [13/Sep/2006:07:01:53 -0700] "PROPFIND /svn/[xxxx]/Extranet/branches/SOW-101 HTTP/1.1" 401 587

The grok pattern to extract the text between quotes can be %{QUOTESTRING:text}, where the output would be "PROPFIND /svn/[xxxx]/Extranet/branches/SOW-101 HTTP/1.1".

Sample grok pattern:

For apache logs and apache errors, following are an example of grok patterns:

%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}

"\\[%{DATA:apache_error.timestamp}\\] \\[%{WORD:apache_error.source}:%{DATA:level}\\] \\[pid %{NUMBER:apache_error.pid}(:tid %{NUMBER:apache_error.tid})?\\] (%{DATA:apache_error.sourcecode}\\(%{NUMBER:apache_error.linenum}\\): )?(?:\\[client %{IPORHOST:apache_error.clientip}:%{POSINT:apache_error.port}\\] ){0,1}%{GREEDYDATA:apache_error.message}"

note

You cannot create a grok pattern name with "system-generated" as this is a reserved name.

For more information about grok patterns, see https://github.com/thekrakken/java-grok/blob/master/src/main/resources/patterns/patterns.