Metric Calculation Bytes In

When calculating the rollup metric for “Bytes In” on the monitoring page, how is that being calculated? We ship all Cribl internal metrics to DataDog, and are trying to calculate the same value using cribl.logstream.total.in_bytes and sum’ing that over the time period to get the total number of bytes ingested. Then take another calculation to convert to GB (first_query / 1024 / 1024 / 1024) . This yields quite a different number than what is displayed on the Cribl Monitoring page.


3 UpGoats

Hi @jccurtis,

This confused me also a while ago.
I have a MBs chart in Splunk and it wouldn’t match what the Cribl internal monitoring calculated.

It will match (closer) if you use powers of 1000 instead of 1024 to divide the bytes_in metric.

My chart uses a span of 10min and I wanted to have the average MBs per each 10min period.
My calculation is bytes_in/10/60/1000000
10 → to get down to 1 minute.
60 → to get down to 1 second
1000000 → to go from bytes to megabytes.

Here is a random google search result on that topic (SI prefix vs. binary prefix or: base-2 vs. base-10)

BR
Ralph

Yes, I am familiar with calculating gibibytes and gigibytes. I calculated it both ways, but its still way off. ~43 vs 70.

Hello jccurtis. There is a bug associated with how our default metrics rollup function works. Please try using cribl.logstream.index.in instead of total.in and report back.

2 UpGoats

Have to ask…does this bug affect license utilization?

1 UpGoat

It does not, i verified this with the devs. It’s an issue with how we are publishing the metric which occurs at a much later stage and outside of license metering.

1 UpGoat

This metric does not appear to exist.

That’s odd, can you try the following: Navigate to the cribl_metrics_rollup pipeline and disable the metrics rollup function. This should fix the issue of total.in reporting incorrect numbers. If it doesn’t please file a support case.

This messes up the metrics even more. I will open a support case.