Options for replacing splunk hec with cribl stream

Looking for options for replacing some work flows with Cribl stream.

Current flow:
client app logging (source) → load balanced hec → splunk idx (destination)

Potential flow:
client app logging (source) → cribl stream → splunk idx (destination)

I see this as an option too, but adds an extra hop:
client app logging → load balanced hec (source) → cribl stream → splunk idx (destination)

Using this driver today which has a required arg of splunk-url (ie the HEC endpoint) which would no longer be our desired destination. Splunk logging driver | Docker Documentation

Is the TCP JSON source our best bet here or perhaps HTTP (Bulk API)?

Thanks

1 UpGoat

Hi @pdott,

Cribl Stream supports Splunk HEC as a source of data. You can reuse your Docker logging driver to send the HEC data to Cribl Stream.

What I would recommend for inserting Cribl Stream between your sources of data and the Splunk Indexers with minimal architectural changes is similar to the last option you described.

Source → Layer 7 HTTP LB → Cribl Stream Workers → Splunk HEC destination to Indexers.

The Layer 7 LB evenly distributes data across the Cribl Stream Worker Group allowing you to scale up and scale down as needed. The Cribl Splunk HEC destination allows for Load Balanced forwarding of data to all your Splunk indexers.

To update your Docker logging driver, you would just need to update the splunk-url in the config to point to the LB (or Cribl Worker) if you decide to with just a single instance. You can import your current HEC token into the Cribl Splunk HEC source under the Authentication tab.

3 UpGoats

Thanks for the reply @bdalpe, very insightful.

Currently testing the (cribl) waters, so a single instance at this time.

The minimal changes to existing architecture is desirable, although it will mean the HEC has to remain in place at least in the interim. Not a big deal but long term would want to remove that hop if possible.

With what you suggest, on the client side we only need to update the splunk-url to be the cribl instance. Even though these events wouldn’t necessarily come from the HEC, do we still use the Splunk HEC source in cribl? That’s how I read it.

New path:

client app logging (via HEC source in cribl) → cribl stream single instance → splunk hec (destination) which feeds our existing splunk indexers

The existing splunk hec’s are already configured to our indexers, so no changes required there. I believe I read somewhere that sending to the indexQueue (default) will bypass existing props/transforms configured on the hec’s.

Thanks again and look forward to your response.

1 UpGoat

With what you suggest, on the client side we only need to update the splunk-url to be the cribl instance. Even though these events wouldn’t necessarily come from the HEC, do we still use the Splunk HEC source in cribl? That’s how I read it.

Both of those assumptions are correct. The Docker logging driver is forwarding events via the HEC protocol which you would need to use the Cribl HEC source to receive them.

The existing splunk hec’s are already configured to our indexers, so no changes required there.

Correct. Import the HEC token into the Cribl HEC Destination configuration along with the FQDN/IPs of your Splunk Indexers.

I believe I read somewhere that sending to the indexQueue (default) will bypass existing props/transforms configured on the hec’s.

That is correct. The data is “cooked” going to Splunk if you send to the indexQueue so you will need to set the correct index, source, sourcetype, etc. in a Cribl Stream Pipeline.

1 UpGoat