Optimized Data Ingestion

A small C++17 TCP ingestion prototype for testing high-volume newline-delimited messages on Linux/WSL.

It builds three executables:

• data_ingestion: starts the mock server and ingests messages.
• mock_server: sends benchmark messages followed by STOP.
• ingestion_benchmark: runs the same flow and prints throughput.

Requirements

Ubuntu on WSL:

sudo apt update
sudo apt install -y build-essential cmake

Build

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

Run commands from the build directory because data_ingestion and ingestion_benchmark start ./mock_server.

cd build

Run

./data_ingestion 1 2

Arguments:

• 1: CPU core for the mock server.
• 2: CPU core for the ingestion thread.

Expected result:

Total Messages Ingested: 100000

Benchmark / Profile

The benchmark accepts:

./ingestion_benchmark <mock_server_core> <ingestion_core> <messages> <interval_us> <batch_size>

Use the no-sleep batched sender for the clean WSL baseline:

/usr/bin/time -f 'wall=%e user=%U sys=%S cpu=%P maxrss_kb=%M vcsw=%w ivcsw=%c' \
  ./ingestion_benchmark 1 2 1000000 0 256

Example result from WSL2 on a 16-thread Ryzen 9 7940HS:

Benchmark Results:
Total Messages Ingested: 1000000
Time Taken: 0.0754934 seconds
Throughput: 1.32462e+07 messages/second
wall=0.29 user=0.11 sys=0.05 cpu=54% maxrss_kb=29696 vcsw=110 ivcsw=4

Avoid using interval_us=1 as a performance baseline on WSL. That mode creates about 210k voluntary context switches for 100k messages, so it measures WSL scheduling overhead more than ingestion speed.

Troubleshooting

• If cmake is missing, install it with sudo apt install -y cmake.
• If taskset reports an invalid CPU, choose core numbers from 0 to $(nproc) - 1.
• If a port is busy, wait a moment and rerun. The mock server uses port 5555.
• Run from build/; running ./build/data_ingestion from the repo root will not find ./mock_server.

Current Limits

• One ingestion thread is supported. If multiple cores are passed, the program uses the first one.
• Input is newline-delimited text.
• Config is hardcoded in src/config.cpp.
• There is no automated test suite yet; the benchmark is the smoke test.