As the title says, I’m looking to create and update a lookup table based on incoming events, somewhat similar to the outputlookup function in splunk. Is there any way to update or create a table based on fields from events passing through a pipeline?
You can use the Redis function to update Redis-based data from within a pipeline using live data.
This isn’t currently possible with CSV-based lookups.
We’ve created a product with custom functions that will read/write tables in data stores, which can be mongo, Oracle, MySql, MSSQL, Postgress, DB2 and others, making the data available across workers. Ping me if this aligns with what you’re looking for.