We have updated our Terms of Service, Code of Conduct, and Addendum.

Patterns for suppressing Splunk cooked data

Options
hpapazov
hpapazov Posts: 3 ✭
in General Discussions

Is there a pattern that I can use to suppress Splunk logs based on data that is part of the _raw field? So far I have come up with a pipeline that uses Regex Extract, Suppress, and Eval function. I am extracting the key:value pair that I need from the _raw field, utilize it in my suppression function, and then remove it from the logs before sending the event to Splunk. While this seems to be working, I wonder if there is a better way to achieve this especially if we want to add additional key:value pairs to suppress on in the future? I was looking into the Parser function that allows me to extract all key:value pairs from the _raw field but I could not find a way to remove all those from the event before sending it to Splunk.

Best Answer

  • Jon Rust
    Jon Rust Posts: 435 mod
    edited July 2023 Answer ✓
    Options

    Hold on, i misread your post

    Use the Parser function to Extract _raw into a new field, maybe __myfield. Now you can refer to __myfield in your suppression action.

    3bf4bba19e903adeabe5882dddf6381ac65a56a1.png

Answers

  • Jon Rust
    Jon Rust Posts: 435 mod
    Options

    Can you supply a sample log? Absolutely we can add/remove/modify fields in any part of the event. Id just like to see real world example before getting too deep into particulars.

  • hpapazov
    hpapazov Posts: 3 ✭
    Options

    Here is an example log that I see in Cribl. Just FYI, I replaced the values for some of the fields because that info is internal to the company. So based on the below example, I am trying to find a way to be able to suppress on any of the key:value pairs that are part of the _raw field.

    {
    "_raw": "{\"Timestamp\":\"2022-05-03T14:51:25.0606688+00:00\",\"Level\":\"Information\",\"MessageTemplate\":\"Successfully retrieved data for {service}\",\"RenderedMessage\":\"Successfully retrieved data for MyService\",\"Properties\":{\"count\":0,\"WorkspaceID\":1234567,\"SourceContext\":\"MyService.Worker.DataRetriever\",\"Priority\":2,\"InstanceIdentifier\":\"c783c624-d5c5-4eb7-a08c-9f50634b08e4\",\"DocumentID\":1234567,\"CorrelationID\":\"ff36e36b-ce29-4cad-9483-c2864dbe84aa\",\"HostName\":\"my-host\",\"ImageTag\":\"v1.0.290\",\"System\":\"MyService\",\"ProcessID\":19056,\"Application\":\"2.224.0\",\"SubSystem\":\"Worker\"}}",
    "_time": 1651589485.060668,
    "source": "2.224.0",
    "host": "10.10.1-.100",
    "sourcetype": "MyService",
    "index": "myIndex",
    "timestartpos": "14",
    "timeendpos": "47",
    "date_second": "25",
    "date_hour": "14",
    "date_minute": "51",
    "date_year": "2022",
    "date_month": "may",
    "date_mday": "3",
    "date_wday": "tuesday",
    "date_zone": "0",
    "punct": "{\"\":\"--::.+:\",\"\":\"\",\"\":\"__{}____{},__{}\",\"\":\"_____",
    "MessageTemplate": "\",
    "cribl_pipe": "suppression-test"
  }
  • Jon Rust
    Jon Rust Posts: 435 mod
    edited July 2023 Answer ✓
    Options

    Hold on, i misread your post

    Use the Parser function to Extract _raw into a new field, maybe __myfield. Now you can refer to __myfield in your suppression action.

    3bf4bba19e903adeabe5882dddf6381ac65a56a1.png

  • hpapazov
    hpapazov Posts: 3 ✭
    Options

    I was just about to post that we found out that we can do what you are suggesting Jon. I think that does what we need. Thanks for the help Jon!

  • Jon Rust
    Jon Rust Posts: 435 mod
    Options

    Excellent! The trick is making sure your JSON objects are parsed before you start trying to manipulate the contents. In the preview window, if they have an a next to them, it means theyre a string. Only string functions will work. But if they have a {} next to them, its seen as an object and ready to be pulled apart like a lego kit. You can use the Parser function, or JSON.parse() in an Eval to parse a string that is proper JSON into an object.

Categories