Launched: dbt Integration

[Khai_To]

About dbt Integration

dbt is a popular data transformation tool that data teams use to pre-transform data inside data warehouses before pushing to the BI layer. However, a common problem data teams face is: dbt and BI tools don’t talk to each other. This leads to problems like:

1. Disconnected metadata at BI layer: The table fields’ descriptions defined in dbt are not exposed to business users in the BI interface.
2. Discontinuous Data flow trigger (Stale data in BI reports when data refreshes) : when underlying data tables get rebuilt by dbt, the BI reports might still be using the cached, stale version.

Holistics supports “dbt integration” that solves the above 2 problems.

Related resources

• Integrate dbt with Holistics | Holistics Docs

Common Questions

What is the relationship between the dbt model and Holistics’ table model for the same underlying database table?#

No direct relationship. Remember that there are 2 different concepts:

• dbt model: A SQL query that gets persisted into a database table.
• Holistics’ Table Model: An abstraction on top of an existing database table.

I am using dbt Cloud, how can I integrate with Holistics?#

To integrate with dbt Cloud, please refer to our public doc dbt Cloud Integration | Holistics Docs.

---

Let us know if you have any questions or feedback in the comments below!

[DataGeekDude (Craig Wilson)]

Hi Khai,

Is there any other plans for the dbt integration, like improvements or anything that you can talk too?

One item we at ZOE would love to have is that if we add a new column to a table that it gets pushed into the same table in the AML layer.

[Rasmus_Engelbrecht]

When can we expect integration to dbt Cloud?

[Khai_To]

We’re planning to support dbt Cloud integration this quarter, but it’s still at the researching phase.

Support for dbt Cloud

Currently, the dbt integration in Holistics only supports for dbt CLI but not dbt Cloud, would be great if this is supported. Given the API to download manifest.json is recently available, I think supporting this would be straightforward? I am using dbt Cloud, how can I integrate with Holistics?​ dbt Cloud is unfortunately not supported at the moment. For now, you need to set up dbt CLI to integrate with Holistics.

Do let me know if you have any questions in the above thread.

And other than metadata synchronization, what’re other capabilities that you wish to have in our dbt integration? (dbt metrics integration, update modeling fields automatically, etc.)?

[dacou (Damien)]

Hi @Khai_To

Synchronisation of multiple manifest.json could be useful (in order to deal with dev/test/prod versions).

And, it is not directly associated to DBT but our dbt docs use markdown extensively. As Holistics does not interpret markdown for data descriptions in reporting view, we tend to change the descriptions in Holistics in order to provide a better user experience, which kind of defeats the purpose of dbt integration .

Best regards,
Damien

[Khai_To]

@dacou

Synchronisation of multiple manifest.json could be useful (in order to deal with dev/test/prod versions).

Interesting, I see why you want this option because you have separate schema for dev/test/prod version.
I guess this is something that has been mentioned in this thread (Development workflow//testing).

---

@dacou

And, it is not directly associated to DBT but our dbt docs use markdown extensively. As Holistics does not interpret markdown for data descriptions in reporting view, we tend to change the descriptions in Holistics in order to provide a better user experience, which kind of defeats the purpose of dbt integration .

This is an important and useful feedback for our team @dacou.
I do agree that to make the experience with dbt more seamless, supporting markdown in reporting would be crucial.
I will add this to our backlog for future improvement. Thanks, Damien.