After some discussion with operators running Aspects in production, we have heard that the current default pipeline with the Ralph LRS handling ClickHouse inserts and providing the xAPI interface is proving to be a performance bottleneck and the xAPI features are under utilized. Operators have been able to achieve high Ralph throughput by:
- Running more Ralph containers
- Running more Celery workers
- Changing some hard coded values and behavior in a Ralph fork
Vector seems to handle these things loads automatically without the need to scale to more containers and do ClickHouse batching based on volume without operator intervention.
Therefore in our next set of breaking changes we would like to change from the default data pipeline from Ralph to Vector. There would, however, still potentially need to be more Celery workers since that is where the events are transformed.
This should involve the following:
- Set RUN_VECTOR to True
- Set RUN_RALPH to False
- Update our Vector implementation to include saving xAPI logs to a configurable file / S3 / etc via a new sink
- Updating documentation and release notes to indicate the change (and steps needed to maintain the current pipeline on an upgrade)
- Update documentation on how to replay events to include using the new Vector log file sink instead of having to re-transform all events via event-routing-backends
- Perform before and after performance tests to confirm our assertions that Vector can handle the same or greater volume of events that Ralph can, without changes on the operator side
- Write up a performance evaluation about how Vector scales and how it performs compared to Ralph
- Create issues for any problems that come from testing the Vector implementation
After some discussion with operators running Aspects in production, we have heard that the current default pipeline with the Ralph LRS handling ClickHouse inserts and providing the xAPI interface is proving to be a performance bottleneck and the xAPI features are under utilized. Operators have been able to achieve high Ralph throughput by:
Vector seems to handle these things loads automatically without the need to scale to more containers and do ClickHouse batching based on volume without operator intervention.
Therefore in our next set of breaking changes we would like to change from the default data pipeline from Ralph to Vector. There would, however, still potentially need to be more Celery workers since that is where the events are transformed.
This should involve the following: