Some Metrics are showing 0 for bytes but event count still shows events

I have an issue with my Cribl Internal Metrics when enabled on the worker group are not produced correctly. Some metrics like route.out.bytes and others are completely missing. The same metrics are missing in the Monitoring view on the Leader.

The only thing the affected Pipeline for the Route does is converts the events _raw to a JSON object.

2 UpGoats

Hi @kmccririe ,

Just giving some random hints thoughts:

  • Is the Cribl Internal source maybe set to “Full fidelity” → “No” ? This would drop some metrics. Although route.out.bytes is not listed in the docs, so i’m not sure if this is really related.

  • Do you expect the traffic that we see in the charts on your default Route? Just guessing it could be an issue with your filter for the “logs” route.

  • Does it work without the JSON-converting pipeline?

  • How did you check if the metrics are coming or not? In your destination (Splunk?), or/and did you take Captures within Stream? You could try it with Captures at different stages (1-4) to check if the Metrics are not coming at all, or a being dropped/missrouted at some point.

  • The zero line for the bytes COULD be caused by the converting pipeline, maybe there is some issue that causes the pipeline to drop _raw and other fields that would trigger the bytes count.
    I guess the Captures mentioned earlier would give some hint here.

1 UpGoat

This has to do with how we calculate the bytes out by looking at _raw and with this being a conversion to a JSON object, it will show those as 0 bytes. This is something that we would have to update to account for this in a newer version. A possible option would be to either deserialize the object into an attribute other than _raw , or serialize _raw (knowing that it’s an expensive operation).