Monitor BigQuery query cost while using Holistics

The problem:

You are using BigQuery in Holistics. If you are on BigQuery’s on-demand pricing plan, the queries are charged based on the number of bytes processed. Chances are that you will have to deal with undesired situations where some expensive queries go unnoticed, which lead to exorbitant BigQuery bills :scream:.

Is there are some preventative measures to avoid such unwanted circumstances, such as a way to manage and report the BigQuery query cost from Holistics?

Answer:

While Holistics acknowledges that this is a valid and necessary use case, we do not support monitoring BigQuery cost in-app at the moment due to limited capacity. Please consider these native BigQuery monitoring resources instead:

  1. Enable Cloud Billing export to BigQuery.

    This will allow you to export detailed billing data (such as usage, cost estimates, and pricing data) automatically to a BigQuery dataset of your choice

  2. Utilize BigQuery audit logs to detect inefficient queries.

    BigQuery audit logs can give valuable insights into the operations of your queries. If you are experiencing big spikes with BigQuery, it may be worthwhile to check those logs.

    Relevant Resources: Export BigQuery audit logs in DBT.

1 Like