Are there any limitations to the file size or events when importing data using a collector for AWS S3?

Would like to understand this better for rehydrating data from AWS S3. @brandon_mccombs

Max event size is predicated on the breaker used. The default breaker supports event size up to 51,200 bytes. If events are expected to be bigger than that the Cribl - Do not break ruleset can be used, which supports up to 1MB events, but if that doesn’t properly break the events being replayed then a completely custom breaker will be required. Max event size for a customer breaker is 134,217,728 bytes.

As for max file size, no limit exists based on the info from the Software engineering team.

4 UpGoats

One thing that I’ve noticed is that if you have a very large raw payload from an S3 collector (say like an event that is 20+MB), there seems to be some issue with the API worker process on the Leader being able to handle that.

1 UpGoat

Yeah I’m aware of that issue that Jeremy is working on with you. We’re still investigating that so we’ll have more information once we determine the root cause of that. But at present there aren’t any Cribl defined limits on the files but there is for the event size.

2 UpGoats