We are successfully using incremental persistence for our postgres collections in Holistics that uses a modifiedAt field to determine if the records should be updated. However, we hard delete records in the participants collection and in that case the incremental persistence still retains the deleted records. I’m guessing that’s because there is no modifiedAt field since a record no longer exists.
Has anyone figured out a work around to this problem? For now we’ve converted those queries back to Full Persistence but it’s not the optimal solution.
Thank you for reaching out to us. Your understanding of the problem is correct. But unfortunately, your approach of converting to Full Persistence mode is the only workaround for now. For the next time, you can consider implementing soft deletes in your source system so the Incremental Persistence can work properly on that use case.
Please let us know if you have any other questions!
Thanks but soft deletes aren’t the best approach for our platform. If we deleted the incremental persistence file would a new file automatically be created the next time the schedule is run?
Yes, the new persisted tables will be created when the next scheduled persistence run (behaves like full persistence), and the subsequent runs will resume normal incremental behavior.