Through trial and error and help from support we have gotten full persistence to work and are still working on custom_ddl and incremental persistence. We have run into a major issue in that the number of files created by Holistics on are causing our postgres server to crash due to disk space. While I’m hoping this may be solved once incremental persistence is working. For now, it makes Holistics unusable as it puts my production system at risk.
I’ve searched the documentation and don’t see any way to automatically delete older files.
We have one query that needs to run every 30 minutes during working hours an there are over 45 older versions of the file.
We retain outdated persisted tables for 24 hours to ensure there is sufficient time for any long-running queries that might still be using them to complete.
That said, 24 hours do feel excessive in some cases. Reducing this period to a shorter timeframe, such as 6 or 12 hours, or allowing end-users to configure this grace period themselves, could improve flexibility and control. I’ll pass this feedback along to the persistence team for consideration and prioritization.
Do you know if incremental persistence results in fewer files or does it take the previous file, duplicate and then append so the number is the same? Thx
Incremental Persistence creates only 1 table with an auto-generated name (or the view_name parameter if specified). Holistics will then save the table name so it can check for incremental updates during subsequent runs of the persistence schedule.
In contrast, Full Persistence creates a new table each time the schedule runs, as it aims to capture an entire data snapshot from the custom SQL query. Holistics will save the table name of the last successful run to differentiate it from previous versions and apply the invalidated policy as described above.
