Overview

Chronicle Queue to kdb+ Adapter investigation

What is Chronicle Queue?

This memory mapped file is also used for exceptionally fast interprocess communication (IPC) without affecting your system performance. This is especially useful when there is a need to transfer large amounts of data, its ideal for transferring data between processes very quickly on the same server or across the network. There is no Garbage Collection (GC) as everything is done off heap.

More info: https://chronicle.software/libraries/queue/

What is kdb+ Time Series Database?

Kdb+, from Kx, is / has:

• a high-performance cross-platform historical time-series columnar database
• an in-memory compute engine
• a real-time streaming processor
• an expressive query and programming language called q

More info: https://code.kx.com/q/

Project goal

Create an "adapter" to allow messages arriving on a Chronicle Queue to be inserted into a table in kdb+.

Example scenario

• Producer - create Quotes and place as new messages on a Chronicle Queue named "quote".
• Chronicle Queue - create a file based queue called quote, adressed via filesystem location e.g. "C:\ChronicleQueue\Producer\quote".
• Adapter - Read from quote Queue and write to quote database table.
• kdb+ - Destination database containing quote database table and supporting functions to allow data to be added.

Adapter process

1. Connect to datasource i.e. Chronicle Queue

Use Chronicle Queue java library to connect to a queue. This library can be added to a maven project pom.xml file as a dependency:

	<dependency>
		<groupId>net.openhft</groupId>
		<artifactId>chronicle-queue</artifactId>
		<version>5.20.111</version>
	</dependency>

The datasource / queue should be identified via an external properties file. Chronicle Queues are typically adressed via filesystem location e.g. "C:\ChronicleQueue\Producer\quote"

2. Create "tailer" to listen for messages

Once connected to the datasource, use a Chronicle Tailer to read the queue and check for new messages. The tailer should be named based on configuration in an external properties file. The tailer should read forward from the last message read when re-started.

3. Read message data ( -> chronicle obj)

Marshall each message that is read by the tailer into a specific POJO representing the chronicle queue quote message.

4. Do mapping (chronicle obj -> kdb obj)

Use Mapstruct to map the Chronicle Queue Quote POJO to the destination kdb+ Quote POJO Maven dependency to include Mapstruct:

	<dependency>
		<groupId>org.mapstruct</groupId>
		<artifactId>mapstruct</artifactId>
		<version>1.4.1.Final</version>
	</dependency>

Rederence guide: https://mapstruct.org/documentation/stable/reference/html/

5. Send data to destination ( -> kdb)

Bespoke development per Adapter