I see this as an option too, but adds an extra hop:
client app logging → load balanced hec (source) → cribl stream → splunk idx (destination)
Using this driver today which has a required arg of splunk-url (ie the HEC endpoint) which would no longer be our desired destination. Splunk logging driver | Docker Documentation
Cribl Stream supports Splunk HEC as a source of data. You can reuse your Docker logging driver to send the HEC data to Cribl Stream.
What I would recommend for inserting Cribl Stream between your sources of data and the Splunk Indexers with minimal architectural changes is similar to the last option you described.
The Layer 7 LB evenly distributes data across the Cribl Stream Worker Group allowing you to scale up and scale down as needed. The Cribl Splunk HEC destination allows for Load Balanced forwarding of data to all your Splunk indexers.
To update your Docker logging driver, you would just need to update the splunk-url in the config to point to the LB (or Cribl Worker) if you decide to with just a single instance. You can import your current HEC token into the Cribl Splunk HEC source under the Authentication tab.
Currently testing the (cribl) waters, so a single instance at this time.
The minimal changes to existing architecture is desirable, although it will mean the HEC has to remain in place at least in the interim. Not a big deal but long term would want to remove that hop if possible.
With what you suggest, on the client side we only need to update the splunk-url to be the cribl instance. Even though these events wouldn’t necessarily come from the HEC, do we still use the Splunk HEC source in cribl? That’s how I read it.
New path:
client app logging (via HEC source in cribl) → cribl stream single instance → splunk hec (destination) which feeds our existing splunk indexers
The existing splunk hec’s are already configured to our indexers, so no changes required there. I believe I read somewhere that sending to the indexQueue (default) will bypass existing props/transforms configured on the hec’s.
With what you suggest, on the client side we only need to update the splunk-url to be the cribl instance. Even though these events wouldn’t necessarily come from the HEC, do we still use the Splunk HEC source in cribl? That’s how I read it.
Both of those assumptions are correct. The Docker logging driver is forwarding events via the HEC protocol which you would need to use the Cribl HEC source to receive them.
The existing splunk hec’s are already configured to our indexers, so no changes required there.
Correct. Import the HEC token into the Cribl HEC Destination configuration along with the FQDN/IPs of your Splunk Indexers.
I believe I read somewhere that sending to the indexQueue (default) will bypass existing props/transforms configured on the hec’s.
That is correct. The data is “cooked” going to Splunk if you send to the indexQueue so you will need to set the correct index, source, sourcetype, etc. in a Cribl Stream Pipeline.