We have updated our Terms of Service, Code of Conduct, and Addendum.

General Peristent Queue Questions

Options
b1scu1t
b1scu1t Posts: 12 ✭
in General Discussions

Hello,

I have a few general questions on Persistent Queues(PQ).

I see PQ settings for Source & Destination, and further my question are specific to size & compression.

1. If I enable PQ and let the default size to be 1 MB & with compression turned on and if I get a burst of 100MB of events which Cribl Stream is unable to handle, does it create a 100 compressed files of 1 MB each? I understand my events will get delayed in getting indexed in Splunk, but will this guarantee that I eventually will get these events(Assuming I have PQ enabled on the destination side too).

2. Is compression given as an option just so to mitigate disk utilization?

3. When I turn on PQ, I do see a few ndjson.*.gz files in the queues directory, but these are from about 20-60 minutes ago when I first enabled PQ, when i do a cursory check of the data, I am seeing all the events in Splunk. Does this mean the Cribl is busy and eventually will get through processing these gz files may be in a few hours when things calm down?

Thank you!

Tagged:

Answers

  • Brandon McCombs
    Brandon McCombs Posts: 150 mod
    Options
    1. With a default file size of 1MB then that will result in ~100 1MB files if 100MB of events were received but not able to be sent. Note that PQ data is stored per destination and per worker process so if the 100MB is sent to 2 destinations that are both configured to use PQ and PQ engages for both then youll have those events queued to disk twice, once for each destination, and possibly spread over multiple worker process directories.

    Stream doesnt use PQ for when it cant handle the data. It queues when the destinations are unable to receive the data stream. This is why PQ is an option as part of a destinations configuration. Source-side queuing is also available which is useful for minimizing data loss for volatile inputs like udp syslog when there are prolonged issues at the destinations that prevent Stream from sending data. See Persistent Queues | Cribl Docs

    1. Yes
    2. The few remaining ndjson files may indicate the PQ wasnt completely flushed. If that persists you then you may need to submit a support case.

Categories