Self Cleaning Tables - FIFO Tables
Create the ability to set a max table size and allow data to be automatically deleted/flushed after that data size and/or time has been reached. Effectively creating a First In/First Out (FIFO) data functionality. This feature may require some sort of interaction on the table before it runs.
Comments: 4
-
15 Jul
Lucky victoryI think implementing this is quite straightforward, based on the logic used by this module (https://npmjs.com/package/harper-manager) which run a check and creates a table only if it doesn't exist yet, similar strategy could be used to check the length of the table, if it exceeds the specified length, a filter algorithm will be applied to the data in the table based on the __created_time__ , then the oldest will be filtered out. (FIFO)
-
15 Jul
Jacob Cohen AdminThis is certainly something that could be accomplished outside of HarperDB. You may want to use the delete_records_before operation to flush the table on a single instance, but not delete records from other replicas in the cluster. This is something we should look into defining a HarperDB Functions recipe to accomplish.
-
15 Jul
Jaxon Repp, VP ProductHi there! You’re correct that this functionality could easily be added to your harper-manager package, perhaps as a second parameter:
{ deleteOlderThan: <timestamp> }
To integrate this functionality into HarperDB, it would have to be executed in one of two ways:
1) A continuous cron that managed pruning. This conflicts with our “no ancillary processes altering data outside a request” ethos, but could change as we evolve our process management architecture in future releases.
2) As a side-effect of any operation on the specified table (like harper-manager). Since this would include reads (you wouldn’t want to return records older than the configured age), it would have to be run before the operation, which would obviously add to the execution time.
So it’s not impossible by any means. Many of us have wanted it since day one. But like all thing on the wish list, it’s a balance.
That said, I love your npm package! Would you consider letting us list it in our Marketplace? -
18 Aug
Lucky victory@Jaxon Repp, sure it could be listed,
also i recently released another module called harpee, that's a bit more advanced and simplified.