Use cases
Producer-centric systems
Chronicle Queue is most often used for producer-centric systems where you need to retain a lot of data for days or years.
Most messaging systems are consumer-centric. Flow control is implemented to avoid the consumer ever getting overloaded; even momentarily. A common example is a server supporting multiple GUI users. Those users might be on different machines (OS and hardware), different qualities of network (latency and bandwidth), doing a variety of other things at different times. For this reason it makes sense for the client consumer to tell the producer when to back off, delaying any data until the consumer is ready to take more data.
Chronicle Queue is a producer-centric solution and does everything possible to never push back on the producer, or tell it to slow down. This makes it a powerful tool, providing a big buffer between your system, and an upstream producer over which you have little, or no, control.
Chronicle Queue does not support operating off any network file system, be it NFS, AFS, SAN-based storage or anything else. The reason for this is those file systems do not provide all the required primitives for memory-mapped files Chronicle Queue uses. If any networking is needed (e.g. to make the data accessible to multiple hosts), the only supported way is Chronicle Queue Replication (Enterprise feature). |
Market data
Market data publishers don’t give you the option to push back on the producer for long; if at all. A few of our users consume data from CME OPRA. This produces peaks of 10 million events per minute, sent as UDP packets without any retry. If you miss, or drop a packet, then it is lost. You have to consume and record those packets as fast as they come to you, with very little buffering in the network adapter. For market data in particular, real time means in a few microseconds; it doesn’t mean intra-day (during the day).
Chronicle Queue is fast and efficient, and has been used to increase the speed that data is passed between threads. In addition, it also keeps a record of every message passed allowing you to significantly reduce the amount of logging that you need to do.
Compliance systems
Compliance systems are required by more and more systems these days. Everyone has to have them, but no one wants to be slowed down by them. By using Chronicle Queue to buffer data between monitored systems and the compliance system, you don’t need to worry about the impact of compliance recording for your monitored systems. Again, Chronicle Queue can support millions of events per-second, per-server, and access data which has been retained for years.
Latency sensitive micro-services
Chronicle Queue supports low latency IPC (Inter Process Communication) between JVMs on the same machine in the order of magnitude of 1 microsecond; as well as between machines with a typical latency of 10 microseconds for modest throughputs of a few hundred thousands. Chronicle Queue supports throughputs of millions of events per second, with stable microsecond latencies.
Log replacement
A Chronicle Queue can be used to build state machines.
All the information about the state of those components can be reproduced externally, without direct access to the components, or to their state.
This significantly reduces the need for additional logging.
However, any logging you do need can be recorded in great detail.
This makes enabling DEBUG
logging in production practical.
This is because the cost of logging is very low; less than 10 microseconds.
Logs can be replicated centrally for log consolidation.
Chronicle Queue is being used to store 100+ TB of data, which can be replayed from any point in time.
Lambda Stream Processing
Non-batching streaming components are highly performant, deterministic, and reproducible. You can reproduce bugs which only show up after a million events played in a particular order, with accelerated realistic timings. This makes using Stream processing attractive for systems which need a high degree of quality outcomes.