Extract all unique Prometheus label names referenced on a given dashboard or all Grafana objects

[hartfordfive]

Hi was wondering if there's any where we can use this tool to return the list of unique label names which are reference across all graphs which are using a Prometheus datasource? I have a scenario where I'd like to test the possibly far reaching effects of dropping all labels except those matching a given regex and was hoping I could first get the complete list of labels names referenced across all my grafana objects. If this isn't available at this time, is that something that could potentially be added to the functionality?

[amotl]

Dear Alain,

thanks for writing in. Let's see what we can do here.

[I am aiming to discover] the list of unique label names which are reference across all graphs which are using a Prometheus datasource?

I understand the part "using a Prometheus datasource", which is usually referring to "dashboard" entities, right? What do you mean with "unique label names which are reference across all graphs"?

Are you just aiming to discover all panels which use a Prometheus datasource? You can find this snippet within the example invocations listed by grafana-wtf --help, maybe it will help you already?

# Display all dashboards using data sources with a specific type. Here: InfluxDB.
grafana-wtf explore dashboards --format=json | jq 'select( .[] | .datasources | .[].type=="influxdb" )'

With kind regards,
Andreas.

[amotl]

Dear Alain,

did you have a chance to evaluate my suggestion, and if that would help you already?

With kind regards,
Andreas.

[amotl]

Dear Alain,

[I am aiming to discover] the list of unique label names which are reference across all graphs which are using a Prometheus datasource?

I understand the part "using a Prometheus datasource", which is usually referring to "dashboard" entities, right? What do you mean with "unique label names which are reference across all graphs"?

I understand now that you are probably referring to labels stored in Prometheus, right?

With grafana-toolbox/grafana-client#20, and recent progress in this area through grafana-toolbox/grafana-client#112, grafana-client gains possibilities to inquire the databases linked into Grafana on behalf of datasource items.

In this spirit, grafana-wtf could inquire Prometheus for all the labels used over there, and Grafana for all the labels used over here, and then present them side-by-side, or differently.

Would that fit your needs somehow, or can you elaborate what you would like this program to do, beyond what grafana-wtf find may already be capable of?

With kind regards,
Andreas.

[amotl]

Iteration +1

We collected a few Grafana Prometheus query expression samples from dashboards at https://play.grafana.org/, and from Prometheus instances run by friends, in order to learn about that the expr attribute is relevant here, which includes the PromQL query expression.

For extracting the label names from a PromQL query expression, we have been successful using the promql-parser package.

Iteration +2

For inquiring Prometheus about existing label-names and -values, in order to match them against the labels used in Grafana, we discovered corresponding HTTP API endpoints.

• Getting label names
• Querying label values

Thoughts

A corresponding feature of grafana-wtf would probably need to list labels grouped by datasource, because distinct Prometheus instances may use completely different sets of labels, right? In this manner, it might not be advisable to throw all available Prometheus labels into the same basket.

/cc @WalBeh