Scheduled Collector discovers events, but does not collect

I have a REST collector discovering events but not collecting them when scheduled. It works Ad-hoc, though.

Any idea what could be causing this? I do not see errors in the logs.

Hey @Jordan, some REST APIs have built-in earliest and latest query string parameters or body content. When you specify the earliest and latest settings in a schedule for a collector, we’re redundantly checking the timestamps of each event after the events are run through the event breaker.

Here’s an example of a Cribl REST collector parameters configuration to get data from the Okta System Log endpoint. You can see in the Okta Developer docs that you can pass an earliest and latest timestamp to bound the events collected from the API. System Log | Okta Developer

For example, if I configure my schedule to run every 5 minutes, I’m sending the since and until parameters in the URL, but Cribl is also checking the timestamp of each event to see if it is in the range. We’ve seen some customers have issues with events being mistakenly discarded even though they have valid timestamps.

What I would recommend is enabling the “Disable Time Filter” under advanced settings. This will ensure all discovered events are collected and we don’t perform a redundant timestamp check when it’s already configured in the collect parameters.

Screen Shot 2022-09-27 at 8.15.38 PM

1 UpGoat