With the logwriting module now feature complete and approaching its first release, the focus is on testing and benchmarking.
The three binary log implementations in the logwriting module naturally each have unit test coverage. Additionally, there is a separate logwriting-benchmark module providing JMH based performance tests for them.
The real test though, is how well they serve their intended use cases. In this post we focus on ArrayStore, a log implementation tailored to the needs of the Red Hat Narayana transaction system.
Background
Narayana is a coordinator for distributed transactions, implementing traditional ACID transaction standards such as JTA and JTS, as well as web based protocols, including extended transaction models based on compensations. It has been a core component of the Wildfly Java application server for many years.
When coordinating a transaction between multiple resource managers, such as XA databases or JMS message brokers, the transaction manager must write an intention log to disk and ensure it is crash tolerant by flushing the write from any volatile O/S cache to the persistent storage media. This operation tends to be the expensive part of transaction coordination, so a lot of attention is paid to making the log I/O efficient.
The transaction log is largely write-only, being read back only for crash recovery purposes. The entries tend to be small and near-uniform in size, at around 600 bytes for a 2-resource XA transaction, and short-lived, as an entry can safely be removed after the transaction completes.
This contrasts somewhat with other common logging use cases such as those found in database systems, where the size of a log entry can vary considerably according to the business logic of the transaction, and the lifetime can depend on the delay in flushing the modified data blocks to persistent storage.
Most importantly though, the log entries for transactions in the coordinator need not be ordered, unlike those for serializable database transactions, where data corruption can occur if replayed updates are not applied in the original sequence.
These characteristics form a use case that seems well aligned with the capabilities of persistent memory hardware, so it will be interesting to see how a log implementation designed for the new environment stacks up against existing alternatives.
Existing stores
The log storage system in Narayana, which is termed the ObjectStore as in some cases it contains more than just log records, uses a pluggable API.
There are a number of existing ObjectStore implementations, with differing characteristics.
The FileStore (or ShadowNoFileLockStore, to give it its full name), uses a model of one file per transaction.
This is the oldest, most flexible and most full featured store, supporting some operations not required for most transactions and also not present in other store implementations.
Unfortunately it’s also the slowest in most cases, as it requires a separate flush operation for every transaction.
Since storage devices, particularly HDD, have a limit on how many individual operations they can handle, this can be a problem.
The HornetQJournalStore addresses this by aggregating log entries into batched writes.
This is the approach used by high performance persistent messaging systems, such as Apache ActiveMQ Artemis.
Indeed, the JournalStore code reuses the same logging engine as Artemis, inherited from the older HornetQ messaging project.
This helps scaling considerably, but comes at the cost of more transaction latency to allow for batching, and additional locking to allow ordering of entries, required for many messaging cases but unnecessary overhead for Narayana.
Nevertheless, this store is the popular choice for high volume production systems.
The JDBCStore allows a relational database table to used to store the log records. This is useful for deployments where a server has no reliable local storage, but is otherwise not widely used as the additional network traffic is relatively expensive.
We will not consider this option further, sticking to the cases that use local disk.
Finally, there is a ConcurrentHashMap based VolatileStore that keep log entries in memory. This is of course completely useless for production use as it’s not crash tolerant,
but it’s quite handy for benchmarking the transaction system code, including log record serialization costs but excluding disks costs.
The new SlotStore
Narayana recently added the SlotStore, a new design that targets the performance characteristics of persistent memory.
However, it has pluggable sub-implementations, termed backends, that allow it to also work on traditional HDD or SSD devices.
The SlotStore design is conceptually an array rather than a stream. The store is divided into a number of elements, each capable of holding one entry of up to a given size.
These slots may be written and flushed concurrently, using different mechanisms:
- DiskSlots : Uses RandomAccessFile, flushed with fsync
- MappedDiskSlots : Uses MappedByteBuffer, flushed with msync
- VolatileSlots : Uses an in-memory byte[] array. Not flushed, like VolatileStore.
This is also the integration point for mashona, as the logwriting module’s ArrayStore is designed to act as a StoreStore backend.
- PmemSlots: Uses Mashona’s ArrayStore, flushed with CLWB instructions.
Benchmarking Narayana
The Narayana project maintains its own benchmark test suite, which exercises the ObjectStores by running transactions against dummy (local, no-op) resource managers. Using these tests on a system with a variety of storage devices, we can compare store implementations in a number of configurations.
Let’s start with a baseline that run all the usual code, but skips writing to disk. This includes serializing the log records, but not placing them on persistent media. This gives us the theoretical upper limit. No persistent store will perform better, unless it’s faster than DRAM! The hardware has more than 16 physical cores, so the sub-linear scaling here is just down to internal coordination points in the code.
Benchmark threads tx/sec
VolatileStore 1 138730
VolatileStore 4 444209
VolatileStore 16 1330884
Next up, the FileStore. Recall this is performing one disk flush per transaction. The server has a HDD, a flash based SSD, an Optane (3D-XPoint) based SSD, and a sector mode (i.e. 4k block persistence atomic) filesystem on a single Optane DCPMM.
Benchmark threads HDD FlashSSD OptaneSSD OptaneDCPMM
FileStore 1 40 3695 6367 7200
FileStore 4 138 13813 15699 20054
FileStore 16 364 16790 16615 21355
There is a lot to unpack here.
First, it’s clear you don’t want to be using this store on a HDD if you can avoid it. SSDs, even enterprise grade ones, offer great price/performance for this kind of workload. The Optane based SSDs are a premium product in comparison, offering limited performance advantage for this workload.
For reference, the test system is running an Intel D3-S4610 against an Intel DC P4800X These drives have a writes spec of 510 MB/s and 44500 IOPS for the flash drive vs. 2200 MB/s and 550000 IOPS for the Optane drive.
The Optane DCPMM module, whilst using the same chip type for persistence, has a closer connection to the CPU than the Optane SSD, allowing it to serve write requests at lower latency, which translates into a significant gain here.
Next up, the JournalStore. These are the numbers for NIO mode. The ones for AIO are not much different, except on the HDD where it shows more advantage.
Benchmark threads HDD FlashSSD OptaneSSD OptaneDCPMM
JournalStore 1 113 3991 3245 3248
JournalStore 4 288 15951 13001 12997
JournalStore 16 514 53830 51929 52030
The figures for the single threaded case are somewhat suspicious! The Journal looks slower than the individual file model on some devices. The Journal builds batches of updates, which it flushes on a timed basis. When there is only a single transaction thread, each batch will contain only a single item. The thread spends much of its time stalled, waiting on the batch timer. Digging into this further, it turns out the transaction test suite sets a rate of 4000 batches/second, so the tx rate for a single thread can never exceed that. Hardware has moved on somewhat since that choice was made, and a modern enterprise SSD will sustain a higher rate with little difficulty. There is more potential here than the numbers show.
Meanwhile, at higher thread count, the batching of writes helps a lot. That advantage continues to grow as more threads i.e. concurrent transactions are added, showing why the JournalStore is popular on busy systems. The more expensive Optane based devices show little benefit over a flash SSD here.
How does the slot store compare to this approach? It eschews batching in favor of exploiting the concurrency available in SSD storage devices, but sidesteps some of the issues that plague the FileStore by performing operations within a singe file, rather than creating a new file for each transaction.
Benchmark threads HDD FlashSSD OptaneSSD OptaneDCPMM
SlotStore.fsync 1 41 7763 12710 14405
SlotStore.fsync 4 102 23027 41178 49465
SlotStore.fsync 16 196 34909 121711 44341
Benchmark threads HDD FlashSSD OptaneSSD OptaneDCPMM
SlotStore.msync 1 115 11788 22824 33666
SlotStore.msync 4 345 42015 71303 77694
SlotStore.msync 16 320 49342 179767 133461
Whilst it still suffered badly on HDD hardware, which has much more limited ability to support fine-grained sync operations, it’s looking good on SSD hardware. These devices can handle a significant number of individual IOPS, so batching doesn’t offer so much benefit.
The variant which uses MappedByteBuffer (i.e. msync) rather than RandomAccessFile (i.e. fsync) seems considerably faster here. That’s a bit odd, since the two system calls should be similar in performance. An examination of the code reveals the problem…
Diversion: I/O library internals
The critical method in the Slot store is write(byte[] data), which take the log entry and ensures it gets to the storage media.
The two implementations look like this:
randomAccessFile.writeInt(data.length);
randomAccessFile.write(data);
randomAccessFile.writeInt(checksum(data));
randomAccessFile.getFD().sync(); // fsync syscall
mappedByteBuffer.putInt(data.length);
mappedByteBuffer.put(data);
mappedByteBuffer.putInt(checksum(data));
mappedByteBuffer.force(); // msync syscall
If you think someone did a copy/paste to create one from the other, you’d be right. That decision turns out to have been… less than optimal. The problem here isn’t the fsync vs. msync performance, it’s the integer handling calls! Writing an int to a RandomAccessFile involves an expensive native method call, whilst the MappedByteBuffer version is an Unsafe.putInt, which has a cheap compiler intrinsic implementation. There is definitely some room for optimization here.
Slot store performance
Now we come at last to the main event. The DCPMM modules offer a programming model not available to the other devices. Using DAX mode in conjunction with the JEP-352 changes added in JDK 14, we can flush direct from user space without the cost of a system call. On storage as fast as the Optane modules, the syscall cost is significant, so eliminating it from the path should help a lot. That’s exactly what the mashona’s ArrayStore is designed to do.
Benchmark threads msync CLWB mult
SlotStore.mashona 1 33666 49042 1.46
SlotStore.mashona 4 77694 175465 2.26
SlotStore.mashona 16 133461 514684 3.85
Wow! The Optane hardware is fast even in sector mode, but by shifting to DAX and optimising out the syscall, we can unlock much more of its potential.
Conclusion
With a simple configuration change to plug the mashona logwriting library into the transaction engine, we can substantially relieve the persistence bottleneck and obtain a level of reliable transaction performance that was previously out of reach.
Using a combination of persistent memory hardware and software tailored to get the best out of it, we unlock a 10x improvement in transaction engine performance over flash based SSD.
Narayana and Wildfly are well positioned to provide full benefit to users adopting the new hardware, thanks to our JDK contributions and the mashona library.