We’d love to have a fully integrated CI/CD pipeline to automatically validate changes on any commits including:
Errors (AML Validation?)
this is currently automatic if someone is making changes within holistics
we’d love to set up a CI/CD script that runs the same tests by calling an API endpoint, especially for users who are pushing changes from their local environments
Reporting Validation
this is a manual step currently
we’d love to have a CI/CD script set-up that automatically checks this by calling an API endpoint, both so people don’t forget to run it and for people coding locally
Dependency Validation (possibly part of AML validation?)
ex. if field_a is referenced in field_b or in a canvas dashboard but then field_a is removed or renamed, neither AML or reporting validation seem to be catching this currently, so I’m not sure the functionality is there? It shows an error in the canvas dashboard preview or when I query field_b, but nothing is being raised in either the Errors or Reporting Validation so it would be easy to miss
we might be able to make a temporary solution by writing a script to:
parse canvas dashboards for field references
use the “List Data Models” endpoint to get a list of fields check for references to other fields
compare those two lists with the “List Data Models” endpoint to ensure all the fields being referenced are defined
Thank you for providing such detailed feedback. Our validation process indeed consists of two modules:
Reporting Validation: This module is responsible for validating the links between as-code objects (Modeling) and non-as-code objects (Reporting).
AML Validation: This module handles as-code validation such as: syntax, dependency validation, data source naming…
Regarding our roadmap, we are currently prioritizing improvements to the AML Validation to address the invalid references you’ve highlighted.
To better understand your needs, could you please answer the following questions?
Among the issues you’ve listed, which ones cause you the most friction?
Could you share more about the local development process? Specifically, what kind of changes do users typically make using their local IDE instead of logging into the Holistics app?
Thanks for the response! Would validation of canvas dashboards fall under “AML validation” then?
I think for us personally, #3 (dependency validation) would be the most important overall but as described in the first message, I might be able to come up with a workaround (albeit a little cumbersome). I haven’t tried building it though, so it might not work quite as well as I expect.
Assuming I can build that workaround with no issue, #2 (a way to automate reporting validation) is probably the next most important since it’s easy to forget to run this manual step.
Last is #1 (a way to automate AML validation for changes not made in the UI), since most of our changes are being made in the UI but sometimes using an IDE is helpful. This is likely a nice-to-have rather than a necessity.
Instances where coding locally is helpful (there are likely more that I’m missing but these are probably the main ones for me personally):
When making certain types of changes, I often use “grep” to search for specific patterns.
VSCode has linting which has been catching a few things that AML validation has not. (ex. missing parentheses). Note: we will be building SQL validation as well as requiring an internal process of manually testing field modifications by previewing them in the Holistics UI so these should be caught more easily future, but we’ve been a bit rushed trying to implement everything into Holistics with an internal deadline.
The ability to use git stash (so I can work on my own changes without messy commit history and either switch to a different high priority project, or pause to switch to someone else’s branch to review), look at git history more easily, and other slightly more complex git features. This one is HUGE for me.
We will be building python scripts for CI/CD purposes. The python scripts would be easiest to use an IDE for (Python linting, being able to run script, easily accessing code from imports, etc) and while this won’t usually be touching both CI/CD and Holistics code at the same time, it might sometimes be helpful to do so.
Would validation of canvas dashboards fall under “AML validation” then?
Yes, it does. For example, if a dashboard uses an invalid field, the issue will be automatically flagged in the “Errors” panel within the Modeling tab. This eliminates the need to manually visit each dashboard and click Preview to identify errors.
Assuming I can build that workaround with no issue, #2 (a way to automate reporting validation) is probably the next most important since it’s easy to forget to run this manual step.
Could you share which types of dashboards you use most frequently? As you may know, there are two types of dashboards:
As-code canvas dashboards (.page.aml) created in the Modeling tab.
Non-as-code dashboards created in the Reporting tab.
These are two issues our team is actively working on, and I’ll share more details as soon as we have updates.
Thank you very much for sharing about your local development workflow! It’s really helpful to know these insights so we can make our in-app editor better : )
Thanks for the response and the time you’re taking on this! We plan to use BOTH types of dashboards pretty frequently, but with different use-cases:
As-code canvas dashboards: We are using this for everything that’s embedded and customer facing so we can lock it behind code reviews, have version control, etc.
Non-as-code dashboards: We are using this for internal reports so that less technical users don’t have to worry about learning git. This will still be partially customer facing as our customer-success team builds reports that are not embedded but might be shared with the customers.
VSCode has linting which has been catching a few things that AML validation has not. (ex. missing parentheses).
We have Holistics AML VSCode Extension that provides syntax validation and other language features (auto-complete, go to definition, etc). Feel free to install it and let us know if you have any feedback.
Awesome, I’ll check it out and share it with my team, thanks! I think we’d still really want some way to automate testing in the future so local development is safer, but something like this is definitely also appreciated.