Skip to content

Multiline logs are not shown as one Splunk log event but each line is shown as a Splunk log event  #859

@minhpham-westpac

Description

@minhpham-westpac

What happened:
In the values.yaml for logging, the pod and container are specified as following with test-sys as the namespace, we call it part A:
aks-secrets-store-provider-azure:
from:
pod: test-sys/aks-secrets-store-provider-azure-
container: provider-azure-installer
multiline:
firstline: /^\w[0-1]\d[0-3]\d/
endline: / pod=".+"/
separator: "\n"
flushInterval: 5
sourcetype: kube:secrets-store-provider-azure

The pod aks-secrets-store-provider-azure generates logs as following, we call it part B:

C1234 02:06:52.093686 1 provider.go:196] "objects string defined in secret provider class" objects=<
array: - |
objectName: xyz-key objectType: secret # object types: secret, key or cert
objectVersion: "" # [OPTIONAL] object versions, default to latest if empty

  • |
    objectName: xyz-id
    objectType: secret # object types: secret, key or cert objectVersion: "" # [OPTIONAL] object versions, default to latest if empty

pod="kube-system/aks-cluster-compliance-rcg87"

What you expected to happen:
The above multiline log in part B should be displayed in one Splunk log event as the following, we call it part C

C1234 02:06:52.093686 1 provider.go:196] "objects string defined in secret provider class" objects=<
array: - |
objectName: xyz-key objectType: secret # object types: secret, key or cert
objectVersion: "" # [OPTIONAL] object versions, default to latest if empty

  • |
    objectName: xyz-id
    objectType: secret # object types: secret, key or cert objectVersion: "" # [OPTIONAL] object versions, default to latest if empty

pod="kube-system/aks-cluster-compliance-rcg87"

But instead Splunk shows each line as a separate log event as the following, we call it part C:

23/05/2023 17:00:00.123 C1234 02:06:52.093686 1 provider.go:196] "objects string defined in secret provider class" objects=<
23/05/2023 17:00:00.124 array:
23/05/2023 17:00:00.125 - |
23/05/2023 17:00:00.126 objectName: xyz-key
....
23/05/2023 17:00:00.200 > pod="kube-system/aks-cluster-compliance-rcg87"

How to reproduce it (as minimally and precisely as possible):
Using the above config for the AKS container in part A, and the input is the multiline log in part B,

Anything else we need to know?:
I follow exactly the instructions in the Splunk connect for Kubernetes https://github.com/splunk/splunk-connect-for-kubernetes/issues?q=is%3Aissue+is%3Aclosed+multiline, and the multiline log is not shown as one Splunk log event. I do not know why, please help me to have the function working. Thanks very much for your help in advance.

Environment:

  • Kubernetes version (use kubectl version): Kubernetes v1.26.3
  • Ruby version (use ruby --version): Not use Ruby at all
  • OS (e.g: cat /etc/os-release): Red Hat Enterprise Linux Server, VERSION="7.9 (Maipo)"
  • Splunk version:
  • Splunk Connect for Kubernetes helm chart version: Splunk connect for k8s 1.5.3
  • Others:

Please get back to me if you have further questions, or clarification. I am looking forward to your solutions and Thanks very much for your help in advance.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions