How can I monitor application logs being written to and distribute that load across the workers?

i have a use case where we have a nfs share that applications writes logs too i would like to monitor these while being written to (file monitor?) but distribute the load across workers. is that possible?

1 UpGoat

Not really. Each worker in a group* gets the same configs and will try to read the same files. There is no communication between them. The File System collector is a little different, but it’s not going to keep track of what it’s already seen.

My rec would be to install Edge or Stream on a single instance worker group and have it responsible for reading the files on that share.

Depending on the folder structure/file naming pattern of the data in the NFS mount, you could use tokens to only look for new events in certain folders. Collection will only forward events inside the time range specified once they are properly broken with the configured event breaker.