Problem
In the cloud, METjson2db will need to fetch zipped/tarballed stat files from S3, and then write a bundle to S3. It will be deployed in Kubernetes, and so will need to be able to do so with minimal config changes between runs outside of what's been added to the stat file tarballs.
Solution
METjson2db should be updated so that:
- it can read/write from a filesystem and read/write from an object store. We will most likely want to be able to inject the correct I/O driver at runtime as we will need to support filesystems, S3, and potentially other cloud providers object stores. For now, we'll just focus on the filesystem & S3.
- it can generate bundles that can be imported into Couchbase by
vxingest/importer.
No Go's
Describe any features or behaviors that have been considered and rejected as out of scope for this feature
- METjson2db will not write bundles directly to Capella, that will be handled by VxIngest's importer.
Additional context
We want to find a way to mock out AWS API calls. Python has a library called "moto". I would assume Go has an equivalent or a standard way of ensuring we're not operating against actual cloud resources.
Problem
In the cloud, METjson2db will need to fetch zipped/tarballed stat files from S3, and then write a bundle to S3. It will be deployed in Kubernetes, and so will need to be able to do so with minimal config changes between runs outside of what's been added to the stat file tarballs.
Solution
METjson2db should be updated so that:
vxingest/importer.No Go's
Describe any features or behaviors that have been considered and rejected as out of scope for this feature
Additional context
We want to find a way to mock out AWS API calls. Python has a library called "moto". I would assume Go has an equivalent or a standard way of ensuring we're not operating against actual cloud resources.