Add support for parsing Protobuf-formatted artifacts

[TheZ3ro]

Why?
Because the current JSON format used by MVT/AndroidQF do not directly allow for stream-read/write.

This becomes a problem when such JSON files must be read/written on devices with limited RAM.
For example the Files module of AndroidQF, reads all the data in memory and then store it in the acquisition archive.
On real-world devices with a lot of files this can become a problem.

Bugbane currenly produce size-delimited Protobuf artifacts instead of JSON to reduce RAM footprint, since they allow for partial and sequential read/write operation out-of-the-box. (write a single entry without having the whole array loaded in memory)

This PR adds upstream support for Protobuf-formatted artifacts.
JSON files always get the priority. If a JSON file is found, it is read and parsed. If no JSON files are found, the Protobuf counterpart are searched and loaded instead.
Protobuf parsing uses the betterproto2 library that is already used for parsing Tombstone artifacts.

This PR also add tests for all the relevant Protobuf artifact parsing.

Happy to make fixes or modifications if needed :)

[besendorf]

I found three blocking correctness and validation issues. The full test suite passes (175 passed, 1 skipped), but the repository mypy check fails on this branch.

[besendorf]

Hi,
I get your use case and I would like to support it. However Im not sure if protobuf is actually needed for the reasons you mentioned. We could also use JSON lines aka new line delimited json or even stream regular json. For example if you have large arrays in json you can just stream each element one by one. Kotlin even has support for streaming regular json in JsonWriter

The streaming methods are:

• beginArray() (https://developer.android.com/reference/kotlin/android/util/JsonWriter#beginArray())
• beginObject() (https://developer.android.com/reference/kotlin/android/util/JsonWriter#beginObject())
• name(String) (https://developer.android.com/reference/kotlin/android/util/JsonWriter#name(java.lang.String))
• value(...) (https://developer.android.com/reference/kotlin/android/util/JsonWriter#value(boolean))
• endObject() (https://developer.android.com/reference/kotlin/android/util/JsonWriter#endObject())
• endArray() (https://developer.android.com/reference/kotlin/android/util/JsonWriter#endArray())
• flush() (https://developer.android.com/reference/kotlin/android/util/JsonWriter#flush())

I also understand that you already produce protobuf files though. I would wait until a few other people have voiced an opinion.

[TheZ3ro]

Hi @besendorf,
many thanks, I've addressed your comments :)

I agree that it is indeed possible to use JSON or JSONL.
The reasons that led us to adopt Protobuf for our niche use case are multiple.

Protobuf is slightly faster (less characters to write/read compared to JSON key-values), has a lower size (10x, however after compression the difference is negligible. but MVT only supports uncompressed AndroidQF parsing)

Finally, Protobuf allows for a fixed standard/source of truth. You can use that in Kotlin/Python/Go and everyone will benefit (AndroidQF/MVT/Bugbane would all adhere to the same format)

[TheZ3ro]

Any updates on this? :)

[besendorf]

@DonnchaC needs to decide whether we want to support this