Differential chunks (update / delete chunk items) #40
Description
One issue currently is it is hard to add/remove chunk information without necessarily rewriting the chunk, which is costly and also lossy. Another approach would be to describe a differential chunk, one that can update an existing chunks items.
This will need a way to reference the existing chunk so this depends on #35
Once we can reference an existing chunk we need to implement on the chunk kinds an "update" procedure, the issue is that to actually update the chunk we would be rewriting it, so instead we need really some proxy that sees the chunk through the lens of the actually up to date chunk, this is the "journal" of rewrites.
After a while this journal will get large and reads will end up replaying a hundred or so updates so we also want a way to flatten the view into a new chunk, this would then become the base chunk for subsequent reads with the diff indexes prior no longer being used. This flattened base chunk is only used internally and wont be written out, at least for local usage, we want to keep the history (until the user says to throw it out).