We’d love to have a fully integrated CI/CD pipeline to automatically validate changes on any commits including:
Errors (AML Validation?)
this is currently automatic if someone is making changes within holistics
we’d love to set up a CI/CD script that runs the same tests by calling an API endpoint, especially for users who are pushing changes from their local environments
Reporting Validation
this is a manual step currently
we’d love to have a CI/CD script set-up that automatically checks this by calling an API endpoint, both so people don’t forget to run it and for people coding locally
Dependency Validation (possibly part of AML validation?)
ex. if field_a is referenced in field_b or in a canvas dashboard but then field_a is removed or renamed, neither AML or reporting validation seem to be catching this currently, so I’m not sure the functionality is there? It shows an error in the canvas dashboard preview or when I query field_b, but nothing is being raised in either the Errors or Reporting Validation so it would be easy to miss
we might be able to make a temporary solution by writing a script to:
parse canvas dashboards for field references
use the “List Data Models” endpoint to get a list of fields check for references to other fields
compare those two lists with the “List Data Models” endpoint to ensure all the fields being referenced are defined
Thank you for providing such detailed feedback. Our validation process indeed consists of two modules:
Reporting Validation: This module is responsible for validating the links between as-code objects (Modeling) and non-as-code objects (Reporting).
AML Validation: This module handles as-code validation such as: syntax, dependency validation, data source naming…
Regarding our roadmap, we are currently prioritizing improvements to the AML Validation to address the invalid references you’ve highlighted.
To better understand your needs, could you please answer the following questions?
Among the issues you’ve listed, which ones cause you the most friction?
Could you share more about the local development process? Specifically, what kind of changes do users typically make using their local IDE instead of logging into the Holistics app?
Thanks for the response! Would validation of canvas dashboards fall under “AML validation” then?
I think for us personally, #3 (dependency validation) would be the most important overall but as described in the first message, I might be able to come up with a workaround (albeit a little cumbersome). I haven’t tried building it though, so it might not work quite as well as I expect.
Assuming I can build that workaround with no issue, #2 (a way to automate reporting validation) is probably the next most important since it’s easy to forget to run this manual step.
Last is #1 (a way to automate AML validation for changes not made in the UI), since most of our changes are being made in the UI but sometimes using an IDE is helpful. This is likely a nice-to-have rather than a necessity.
Instances where coding locally is helpful (there are likely more that I’m missing but these are probably the main ones for me personally):
When making certain types of changes, I often use “grep” to search for specific patterns.
VSCode has linting which has been catching a few things that AML validation has not. (ex. missing parentheses). Note: we will be building SQL validation as well as requiring an internal process of manually testing field modifications by previewing them in the Holistics UI so these should be caught more easily future, but we’ve been a bit rushed trying to implement everything into Holistics with an internal deadline.
The ability to use git stash (so I can work on my own changes without messy commit history and either switch to a different high priority project, or pause to switch to someone else’s branch to review), look at git history more easily, and other slightly more complex git features. This one is HUGE for me.
We will be building python scripts for CI/CD purposes. The python scripts would be easiest to use an IDE for (Python linting, being able to run script, easily accessing code from imports, etc) and while this won’t usually be touching both CI/CD and Holistics code at the same time, it might sometimes be helpful to do so.
Would validation of canvas dashboards fall under “AML validation” then?
Yes, it does. For example, if a dashboard uses an invalid field, the issue will be automatically flagged in the “Errors” panel within the Modeling tab. This eliminates the need to manually visit each dashboard and click Preview to identify errors.
Assuming I can build that workaround with no issue, #2 (a way to automate reporting validation) is probably the next most important since it’s easy to forget to run this manual step.
Could you share which types of dashboards you use most frequently? As you may know, there are two types of dashboards:
As-code canvas dashboards (.page.aml) created in the Modeling tab.
Non-as-code dashboards created in the Reporting tab.
These are two issues our team is actively working on, and I’ll share more details as soon as we have updates.
Thank you very much for sharing about your local development workflow! It’s really helpful to know these insights so we can make our in-app editor better : )
Thanks for the response and the time you’re taking on this! We plan to use BOTH types of dashboards pretty frequently, but with different use-cases:
As-code canvas dashboards: We are using this for everything that’s embedded and customer facing so we can lock it behind code reviews, have version control, etc.
Non-as-code dashboards: We are using this for internal reports so that less technical users don’t have to worry about learning git. This will still be partially customer facing as our customer-success team builds reports that are not embedded but might be shared with the customers.
VSCode has linting which has been catching a few things that AML validation has not. (ex. missing parentheses).
We have Holistics AML VSCode Extension that provides syntax validation and other language features (auto-complete, go to definition, etc). Feel free to install it and let us know if you have any feedback.
Awesome, I’ll check it out and share it with my team, thanks! I think we’d still really want some way to automate testing in the future so local development is safer, but something like this is definitely also appreciated.
I understand that requiring less-technical users to use Git for creating dashboards isn’t ideal.
Therefore, I wanted to ask: If we release the option to edit Canvas Dashboards on Reporting, would that make it easier for you (and your less-technical users) to use Canvas Dashboards for both use cases?
Unfortunately not because we have branch protections, which is mentioned as incompatible with that feature in the thread you linked. This might work once the Pull Request Workflow is also released but I’m a little hesitant to say for sure without seeing both features and discussing with internal teams more.
Just want to share with you a little update on our CICD integration. You can refer to our post here for more information about our upcoming Deploy API. Feel free to let us know your thoughts on this.
Still very much hoping for the automated validation endpoints because we’ve run into issues where after merging, holistics won’t let us deploy because there’s an error that we didn’t catch (ex. we edit file 1, and file 2 depends on it, and we break file 2, but we haven’t opened file 2 and run AML validation on it). Which is great that holistics catches it before deploying, but I wish we could catch it before merging. I think we will likely run into similar headaches, but a little less visibility if someone merges to master and doesn’t realize that their deploy has failed in the ci/cd pipeline or why it’s failing. We will still definitely be using the automated deploy API, but I think this change makes automated API validation even more important than it was before.
