Filebeat Kubernetes Processor and filtering

Question

I am trying to ship my K8s pod logs to Elasticsearch using Filebeat.

I am following the guide online here: https://www.elastic.co/guide/en/beats/filebeat/6.0/running-on-kubernetes.html

Everything works as expected however I want to filter out events from system pods. My updated config looks like:

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-prospectors
  namespace: kube-system
  labels:
    k8s-app: filebeat
    kubernetes.io/cluster-service: "true"
data:
  kubernetes.yml: |-
    - type: log
      paths:
        - /var/lib/docker/containers/*/*.log
  multiline.pattern: '^\s'
  multiline.match: after
  json.message_key: log
  json.keys_under_root: true
  processors:
    - add_kubernetes_metadata:
        in_cluster: true
        namespace: ${POD_NAMESPACE}
    - drop_event.when.regexp:
        or:
          kubernetes.pod.name: "weave-net.*"
          kubernetes.pod.name: "external-dns.*"
          kubernetes.pod.name: "nginx-ingress-controller.*"
          kubernetes.pod.name: "filebeat.*"

I am trying to ignore weave-net, external-dns, ingress-controller and filebeat events via:

- drop_event.when.regexp:
    or:
      kubernetes.pod.name: "weave-net.*"
      kubernetes.pod.name: "external-dns.*"
      kubernetes.pod.name: "nginx-ingress-controller.*"
      kubernetes.pod.name: "filebeat.*"

However they continue to arrive in Elasticsearch.

Answer 1

The conditions need to be a list:

- drop_event.when.regexp:
    or:
      - kubernetes.pod.name: "weave-net.*"
      - kubernetes.pod.name: "external-dns.*"
      - kubernetes.pod.name: "nginx-ingress-controller.*"
      - kubernetes.pod.name: "filebeat.*"

I'm not sure if your order of parameters works. One of my working examples looks like this:

- drop_event:
    when:
      or:
        # Exclude traces from Zipkin
        - contains.path: "/api/v"
        # Exclude Jolokia calls
        - contains.path: "/jolokia/?"
        # Exclude pinging metrics
        - equals.path: "/metrics"
        # Exclude pinging health
        - equals.path: "/health"