We have updated our Terms of Service, Code of Conduct, and Addendum.

JSON formatting for VMWare Log Insight

Options
BargiBargi
BargiBargi Posts: 6 ✭
edited July 2023 in Stream

Hi,

I've added some fields (lookups, etc) from a syslog source and trying to push it into the Log Insight API via a webhook.

But, I'm struggling understand how to get it into the format Log Insight wants.
I'll say up front, JSON and Javascript is pretty new to me so be gentle 🙂

The JSON format Log Insight expects is…

{“messages”: [{
“fields”: [
{“name”: “Field1”, “content”: “Field1_value”},
{“name”: “Field2”, “content”: “Field2_value”},
{“name”: “Field_xxx”, “content”: “Field_Last_xxx”}
],
“text”: “original message”,
“timestamp”: timestamp
}
]
}

I've tried the Serialize to JSON, but it doesn't seem to have a way to arrange the fields for it to output in a custom way.

I've seen a post about using Object.fromEntries, but I'm still not able to get working. Not sure if it's even the right way to go.

Converting an array into a simplified JSON object:

Does anyone have a slick way of converting an array where each element is an array of length 2, that contains the key at index 0 and value at index 1, into a JSON object. This is what the original looks like: { "HeadersIn": [ [ "Host", "example.com" ], [ "Accept", "*/*" ], [ "Connection", "keep-alive" ], [ "Cookie", " ], [ …

It's probably something simple, Thank you in advance!

Answers

  • Jon Rust
    Jon Rust Posts: 435 mod
    edited July 2023
    Options

    Be sure your target object is actually JSON. If it's text (represented by an 'a' in the preview pane), you'll want to parse it into JSON before it hits the destination. Eval function, with _raw => JSON.parse(_raw) would work.

    Note the curly braces next to raw here, indicating a JSON object:

    9488965876df85017b3ce3d3d8b538caae1ff00b.png

  • Brendan Dalpe
    Brendan Dalpe Posts: 201 mod
    edited July 2023
    Options

    Hi @BargiBargi,

    Here's how I would build a pipeline to format the messages. Here's an example I'll show with some fields already extracted. We want to move _raw to the text field, _time to timestamp, and then all remaining fields to fields.

    6f929629bf8caceb51ce97c103e9663de0753376.png

    Let's start with a basic Eval function to build the general structure of an individual message:

    4e1fde316959ee38717867171dbae8065c1a3002.png

    (the expression is {"text": _raw, "timestamp": _time, "fields": []})

    Now we can use the code function to do some magic… We want to take all fields (using the special variable __e) that do not start with an underscore (internal or otherwise not already used) and move their KV pairs to fields.

    __e['_raw']['fields'] = Object.entries(__e).filter(([key, value]) => !key.startsWith('_')).map(([key, value]) => {    return {"name": key, "content": value}})

    We use the Object.entries function to create a KV array to work with in the filter and map functions. In the map function, the return value reformats the original KV pair into the expected name and content fields.

    b311872ace305e6013f5c356971ff699c99df579.png

    Finally, we can use the Aggregations function to combine events into a single array.

    cbd2facad7ec1915c453e3912e1aa3370ff54634.png

    The list(_raw) function will generate a new array of individual messages aggregated together. The evaluate fields expression moves the array into the expected messages object key.

    Which gives an output that looks like the following:

    5f91f25281050754e78c3ee1e4b609dec04bbf99.png

    Now in the Webhook destination, configure as follows to only emit the _raw field as the payload to the Log Insight collector. Note the URL is static for the destination, but it can be customized per-event by setting the __url field.

    3ebfe23ba40e3e2087b85f64b1c599c961b8da89.png

    Let me know if this solves your issue!

  • BargiBargi
    BargiBargi Posts: 6 ✭
    edited July 2023
    Options

    Amazing!
    Thanks so much for the help.

    I've worked through it and I can see it builds the JSON object up as expected.
    But when it's sending to the LI API it's it's erroring bellow.

    {"errorMessage":"Invalid request body.","errorCode":"JSON_FORMAT_ERROR","errorDetails":{"reason":"Unrecognized token ‘object': was expecting (JSON String, Number, Array, Object or token ‘null', ‘true' or ‘false')\n at [Source: (String)"[object Object]\n[object Object]\n[object Obj…"[truncated 3915 chars]; line: 1, column: 8]"}}

    2186a72122d64fe291c9d1bbd30c33a5be1e6300.png

    Pipeline looks fine (only thing maybe a bit strange is the content is listed before name, but assume that's just Cribl sorting alphabetically)

    535f151b45137ad7e9a28fc683ccaf809ec52b10.png

    I changed the code to filter on anything starting with NSXT as all the fields I need start with that.

    40fa1236d4c782d5d62904e134e7bd3a4f86884f.png

    Tried with the Aggregations and just as Eval and same

    23733acc1ae8e7b46a37eb8ec688964db82e484b.png

    Live Data for the Webhook destination again looks fine

    8a4ad58d9d86351d5deb0081c464339a97dfca94.png
    0d377f8d3f501afb589c0badd331fdf18e70571b.png

  • Brendan Dalpe
    Brendan Dalpe Posts: 201 mod
    Options

    @BargiBargi could you try adding an eval function to the end of your pipeline that turns _raw into JSON.stringify(_raw)?

  • BargiBargi
    BargiBargi Posts: 6 ✭
    edited July 2023
    Options

    @bdalpe
    Using JSON.stringify(_raw) I can see events come through when using Aggregations step, which is good, but now I barely see 1 event a second coming through when there's much more than that coming in.

    I thought it might have been to do with the Aggregation so replaced it with the following Eval and it works, but again only at a very low rate

    990b538f7d4bea29226abb52e6ae1cdc06684e98.png

    There's no dropped events in the Cribl Webhook which is strange. So not quite sure wha'ts going on. Does the JSON.stringify have a rate limit to what it can process?

  • BargiBargi
    BargiBargi Posts: 6 ✭
    edited July 2023
    Options

    Hi,

    Been trying to sort this on and off and can't figure out what the issue is.
    Without the JSON.stringify(_raw) Cribl outputs [object Object], which can be seen in the last-failed-buffer.raw

    072466a1c747a4739f69dcc2423780bfe62e575a.png

    With JSON.stringify(_raw) it works, but it's literly 1 message every 2 seconds

    73bdbdc4a6f40570bf21fce9d2bbcc7d0f4bd1ee.png

    Any advice would be much appreciated!

Categories