Launched: dbt Integration

About dbt Integration

dbt is a popular data transformation tool that data teams use to pre-transform data inside data warehouses before pushing to the BI layer. However, a common problem data teams face is: dbt and BI tools don’t talk to each other. This leads to problems like:

  1. Disconnected metadata at BI layer: The table fields’ descriptions defined in dbt are not exposed to business users in the BI interface.
  2. Discontinuous Data flow trigger (Stale data in BI reports when data refreshes) : when underlying data tables get rebuilt by dbt, the BI reports might still be using the cached, stale version.

Holistics supports “dbt integration” that solves the above 2 problems.

Related resources

Common Questions

What is the relationship between the dbt model and Holistics’ table model for the same underlying database table?#

No direct relationship. Remember that there are 2 different concepts:

  • dbt model: A SQL query that gets persisted into a database table.
  • Holistics’ Table Model: An abstraction on top of an existing database table.

I am using dbt Cloud, how can I integrate with Holistics?#

dbt Cloud is unfortunately not supported at the moment. For now, you need to set up dbt CLI to integrate with Holistics.


Let us know if you have any questions or feedback in the comments below!

4 Likes

Hi Khai,

Is there any other plans for the dbt integration, like improvements or anything that you can talk too?

One item we at ZOE would love to have is that if we add a new column to a table that it gets pushed into the same table in the AML layer.

2 Likes

When can we expect integration to dbt Cloud?

We’re planning to support dbt Cloud integration this quarter, but it’s still at the researching phase.

Do let me know if you have any questions in the above thread.

And other than metadata synchronization, what’re other capabilities that you wish to have in our dbt integration? (dbt metrics integration, update modeling fields automatically, etc.)?

Hi @Khai_To

Synchronisation of multiple manifest.json could be useful (in order to deal with dev/test/prod versions).

And, it is not directly associated to DBT but our dbt docs use markdown extensively. As Holistics does not interpret markdown for data descriptions in reporting view, we tend to change the descriptions in Holistics in order to provide a better user experience, which kind of defeats the purpose of dbt integration :sweat_smile:.

Best regards,
Damien

1 Like

Interesting, I see why you want this option because you have separate schema for dev/test/prod version.
I guess this is something that has been mentioned in this thread (Development workflow//testing).


This is an important and useful feedback for our team @dacou.
I do agree that to make the experience with dbt more seamless, supporting markdown in reporting would be crucial.
I will add this to our backlog for future improvement. Thanks, Damien.