Have at least 2 threads for compaction and flushing in RocksDB
This fix needs to be backported to 0.10.2 as well.
Author: Eno Thereska
Reviewers: Damian Guy, Ismael Juma, Guozhang Wang
Closes from enothereska/KAFKA-5174-1-core
@@ -126,9 +126,13 @@ public class RocksDBStore<K, V> implements KeyValueStore<K, V> {
options.setErrorIfExists(false);
options.setInfoLogLevel(InfoLogLevel.ERROR_LEVEL);
// this is the recommended way to increase parallelism in RocksDb
- // note that the current implementation increases the number of compaction threads
- // but not flush threads.
- options.setIncreaseParallelism(Runtime.getRuntime().availableProcessors());
+ // note that the current implementation of setIncreaseParallelism affects the number
+ // of compaction threads but not flush threads (the latter remains one). Also
+ // the parallelism value needs to be at least two because of the code in
+ // https://github.com/facebook/rocksdb/blob/62ad0a9b19f0be4cefa70b6b32876e764b7f3c11/util/options.cc#L580
+ // subtracts one from the value passed to determine the number of compaction threads
+ // (this could be a bug in the RocksDB code and their devs have been contacted).
+ options.setIncreaseParallelism(Math.max(Runtime.getRuntime().availableProcessors(), 2));
wOptions = new WriteOptions();
wOptions.setDisableWAL(true);
