You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
To understand if there are possibilities of improvement of RocksDB performance when using local disk instead of persistent disk. For that, we will be using fio tool for measuring SSD latency and database tool state-perf for measuring overall latency using RocksDB
Measurements on n2-highcpu-32
This was done on a GCP node n2-highcpu-32 with 32GB RAM with 4 local disks in raid level 0.
Before testing RocksDB measurements via the fio tool (with the same configuration as from the reference) were taken on Local Disk to get a baseline of latency, and these are the results: clat (usec): min=129, max=2103, avg=159.21, stdev=18.98
which is similar to what results show in #9147.
Since the RocksDB is configured with 16KiB block sizes, the same was rerun with fio tool was done here, and here are the results: clat (usec): min=168, max=2410, avg=224.98, stdev=27.79
Now that we know local disk latencies we can use database tool state-perf to measure RocksDB latency, the command being run is: cargo run --release --bin neard -- database state-perf --samples 100000 --warmup_samples 10000 and these are the results:
Local disk state column performance with 512MiB cache size
Comparing the current results where we observe a latency of 409.499µs and block read time of 286.813µs, we can notice that the difference is big between those two, indicating room for improvement on RocksDB side. The Block read time is similar to the latency reported by fio.
Local disk state column performance with 1GiB cache size
With increased column cache size we can see drastic improvements in performance even with 1GiB cache size we see 86% of blocks being read on the first try, and with 2GiB we get 92%, the first jump is definitely the biggest and most significant one and the second one is smaller. After that, increasing cache size does not bring much benefit. As explained previously the reason behind that is that most of the filter and index blocks are being cached and it reduces the miss rate drastically.
Measurements on n2-highcpu-8
This was done on a GCP node n2-highcpu-8 with 8GB RAM with 4 local disks in raid level 0. The reason for that was to reduce the filesystem caching during test execution and to compare the results with the node with 32 GB RAM.
Same as previous measurements first we do fio test just to make sure that the baseline latencies are also the same: clat (usec): min=131, max=5689, avg=199.10, stdev=64.10
which is somewhat similar to the results we get for the other machine, taking into account standard deviation.
Next, we do the same state-perf run on that machine with different cache sizes to get an understanding of how the increase in cache size reflects performance on a smaller RAM machine.
Local disk state column performance with 512MiB cache size
The increase of cache size on smaller RAM machine also exhibits big improvements in how any block was read with only one read which reduces overall latency drastically throughout the benchmark.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Introduction
To understand if there are possibilities of improvement of RocksDB performance when using local disk instead of persistent disk. For that, we will be using fio tool for measuring SSD latency and database tool
state-perffor measuring overall latency using RocksDBMeasurements on
n2-highcpu-32This was done on a GCP node
n2-highcpu-32with 32GB RAM with 4 local disks in raid level 0.Before testing RocksDB measurements via the fio tool (with the same configuration as from the reference) were taken on Local Disk to get a baseline of latency, and these are the results:
clat (usec): min=129, max=2103, avg=159.21, stdev=18.98which is similar to what results show in #9147.
Since the RocksDB is configured with 16KiB block sizes, the same was rerun with
fiotool was done here, and here are the results:clat (usec): min=168, max=2410, avg=224.98, stdev=27.79Now that we know local disk latencies we can use
databasetoolstate-perfto measure RocksDB latency, the command being run is:cargo run --release --bin neard -- database state-perf --samples 100000 --warmup_samples 10000and these are the results:Local disk state column performance with 512MiB cache size
Comparing the current results where we observe a latency of 409.499µs and block read time of 286.813µs, we can notice that the difference is big between those two, indicating room for improvement on RocksDB side. The Block read time is similar to the latency reported by fio.
Local disk state column performance with 1GiB cache size
Local disk state column performance 2GiB cache size
Local disk state column performance with 4GiB cache size
Local disk state column performance with 8GiB cache size
With increased column cache size we can see drastic improvements in performance even with 1GiB cache size we see 86% of blocks being read on the first try, and with 2GiB we get 92%, the first jump is definitely the biggest and most significant one and the second one is smaller. After that, increasing cache size does not bring much benefit. As explained previously the reason behind that is that most of the filter and index blocks are being cached and it reduces the miss rate drastically.
Measurements on
n2-highcpu-8This was done on a GCP node
n2-highcpu-8with 8GB RAM with 4 local disks in raid level 0. The reason for that was to reduce the filesystem caching during test execution and to compare the results with the node with 32 GB RAM.Same as previous measurements first we do fio test just to make sure that the baseline latencies are also the same:
clat (usec): min=131, max=5689, avg=199.10, stdev=64.10which is somewhat similar to the results we get for the other machine, taking into account standard deviation.
Next, we do the same
state-perfrun on that machine with different cache sizes to get an understanding of how the increase in cache size reflects performance on a smaller RAM machine.Local disk state column performance with 512MiB cache size
Local disk state column performance with 1GiB cache size
Local disk state column performance with 2GiB cache size
The increase of cache size on smaller RAM machine also exhibits big improvements in how any block was read with only one read which reduces overall latency drastically throughout the benchmark.
References
This is a continuation of measurements:
Beta Was this translation helpful? Give feedback.
All reactions