Feat Req: metric type convert in stackdriver-prometheus

[mtanda]

What did you do?
Scrape redis metrics by redis_exporter.

What did you expect to see?
Can't use ALIGN_RATE for the external.googleapis.com/prometheus/redis_commands_processed_total.

The error message is googleapi: Error 400: Field aggregation.perSeriesAligner had an invalid value of "ALIGN_RATE": The aligner cannot be applied to metrics with kind GAUGE and value type DOUBLE., badRequest.

What did you see instead? Under which circumstances?
Can use ALIGN_RATE.

I know the reason why it is happen.
The metricKind of external.googleapis.com/prometheus/redis_commands_processed_total is "GAUGE".

The exporter expose metrics as "GAUGE".
https://github.com/oliver006/redis_exporter/blob/v0.20.2/exporter/redis.go#L787-L791

So, this is exporter implementation issue.
The best way to fix this issue, all exporter expose metrics by correct type ("GAUGE", "COUNTER", etc).
But, most of exporter doesn't care it, because Prometheus itself doesn't store metric type, it can use rate() for metrics expose as "GAUGE".
There is no strong motivation to fix it, if exporter author doesn't use Stackdriver.

stackdriver-prometheus support to convert metric type is one of the workaround for this issue.

[jkohen]

Misushiro, thank for the report. I agree with your assessment of the problem. Have you tried sending a PR to the owner of the exporter to set the right type? Until it's refused, we'd like to assume maintainers are open to these fixes.

Better than fixing it in stackdriver-prometheus, I believe this is something that should be fixed in the Stackdriver query API itself to make it useful for other users of Stackdriver.

[mtanda]

Thanks for considering about this issue :-)

I didn't send a PR, because we have another work around.
I make a remote read endpoint for Stackdriver, it convert PromQL label matcher to Stackdriver filter, we use PromQL to query counter type metrics.
https://github.com/mtanda/stackdriver_read_adapter

I don't think my workaround is not good, worse than all other fix.
As you proposed, Stackdriver itself support ALIGN_RATE for GAGUE type metrics, is really nice (if Stackdriver internal storage design allow it).

I check collectd implementation, redis and mysql plugin seems to expose metrics as correct type.
https://github.com/Stackdriver/collectd/blob/7ec19fe55345364d9d4bf08d0922b1ef0b929f5d/src/redis.c#L425
https://github.com/Stackdriver/collectd/blob/7ec19fe55345364d9d4bf08d0922b1ef0b929f5d/src/mysql.c#L993

So, this issue is Prometheus integration specific.

I'm not sure how Prometheus integration is popular in GCP user.
If Prometheus integration becomes more popular, some of users will hit this issue.
Fortunatelly, we have a workaround, so personally this issue is not blocker for us.
I'm OK to close this issue.
(I just want to let you and other stackdriver-prometheus users know about this issue.)

[jkohen]

Thanks, I filed an issue with the Redis exporter and I'm closing this as a duplicate of oliver006/redis_exporter#178

I'll update if I find a way to compute rate on gauges with Stackdriver, with the caveat that there could be glitches if your gauge jumps down (e.g. during a process restart).

[mtanda]

Thanks for filling the issue!

I didn't notice about the glitches, I should care about it.