I’ve added some fields (lookups, etc) from a syslog source and trying to push it into the Log Insight API via a webhook.
But, I’m struggling understand how to get it into the format Log Insight wants.
I’ll say up front, JSON and Javascript is pretty new to me so be gentle
Be sure your target object is actually JSON. If it’s text (represented by an ‘a’ in the preview pane), you’ll want to parse it into JSON before it hits the destination. Eval function, with _raw => JSON.parse(_raw) would work.
Note the curly braces next to raw here, indicating a JSON object:
Here’s how I would build a pipeline to format the messages. Here’s an example I’ll show with some fields already extracted. We want to move _raw to the text field, _time to timestamp, and then all remaining fields to fields.
(the expression is {"text": _raw, "timestamp": _time, "fields": []})
Now we can use the code function to do some magic… We want to take all fields (using the special variable __e) that do not start with an underscore (internal or otherwise not already used) and move their KV pairs to fields.
We use the Object.entries function to create a KV array to work with in the filter and map functions. In the map function, the return value reformats the original KV pair into the expected name and content fields.
The list(_raw) function will generate a new array of individual messages aggregated together. The evaluate fields expression moves the array into the expected messages object key.
Which gives an output that looks like the following:
Now in the Webhook destination, configure as follows to only emit the _raw field as the payload to the Log Insight collector. Note the URL is static for the destination, but it can be customized per-event by setting the __url field.
@bdalpe
Using JSON.stringify(_raw) I can see events come through when using Aggregations step, which is good, but now I barely see 1 event a second coming through when there’s much more than that coming in.
I thought it might have been to do with the Aggregation so replaced it with the following Eval and it works, but again only at a very low rate
There’s no dropped events in the Cribl Webhook which is strange. So not quite sure wha’ts going on. Does the JSON.stringify have a rate limit to what it can process?