We have updated our Terms of Service, Code of Conduct, and Addendum.

Messagepack input

Options
G H
G H Posts: 10 ✭
in Stream

Are there any tricks to parse messagepack data in Cribl? We want to take send large amounts of log data from fluentd, but it doesn’t have a TCP output plugin - just forward (messagepack), http, or syslog.

We’re probably going to have to go with http, but the overheads involved and performance hit are quite significant, so it’d be great to use messagepack natively, and save compute on both ends.

I don’t see any source type, nor any function under C that looks like it can do the job.

I’d be very happy with a breaker, or a pre-processing pipeline on a raw TCP input. There just doesn’t seem to be a native way to handle messagepack.

It doesn’t look like any messagepack plugins are installed natively - would it be possible to install a plugin in the workers config somehow?

Best Answer

  • G H
    G H Posts: 10 ✭
    Answer ✓
    Options

    Ive sunk some time into this - for future readers, at present its impossible.

    There is no native way to do it, and any binary data in Cribl gets treated as UTF8 text. The UTF8 transforms mean by the time any custom function you may have written sees the data, it has been altered irrevocably, so you can never see (or derive) the data as it arrived on the wire.

Answers

  • G H
    G H Posts: 10 ✭
    Answer ✓
    Options

    Ive sunk some time into this - for future readers, at present its impossible.

    There is no native way to do it, and any binary data in Cribl gets treated as UTF8 text. The UTF8 transforms mean by the time any custom function you may have written sees the data, it has been altered irrevocably, so you can never see (or derive) the data as it arrived on the wire.

Categories