We have updated our Terms of Service, Code of Conduct, and Addendum.

REST Collector Requests per second?

Options
amiller
amiller Posts: 21 ✭
edited March 2023 in Stream

I am working with an external API to collect data. This works beautifully except they throttle requests per second to 1.

Is there a way in Stream where I can enforce 1 request per second at collection phase? Not per second/per worker unfortunately. Hard limit of 1 request per second which results in 429 errors for most of the pulls.

I am on version 3.5.4.

Tagged:

Answers

  • Harry Gardner
    Harry Gardner Posts: 59 mod
    Options

    Unfortunately there is not a built-in way to accomplish this today. Out of curiosity, is it pagination or something else that results in multiple requests per second?

  • Jon Rust
    Jon Rust Posts: 431 mod
    Options

    If you dont mind, can you share your company name? DM me in Cribl Slack if that works better. Id like to add the name to the internal ticket we have tracking this feature request.

Categories