High-Performance Storage Research

Several of us in the lab have been working with various distributed storage vendors (Dell EMC, IBM, DDN, etc.) and clustered filesystems for many years. In the area of high-performance computing (HPC) Lustre, Spectrum Scale (GPFS), and BeeGFS are very popular. Traditional HPC filesystems are very mature and provide impressive performance, but much like their enterprise counterparts, specific storage platforms and their underlying filesystem often limit the protocol and available access methods for storage.

For a number of years we have been working with Ceph to provide object storage on a single distributed computer cluster to provide interfaces for object-, block- and file-level storage for OpenStack deployments. However, the underlying performance of Ceph suffered

As part of the KyRIC grant we had the opportunity to explore the use of Ceph for the purposes of high-performance (IOPS), high-throughput (Gbps), and high-capacity (Pb) storage to support biomedical research.

Our design makes use of NVDIMMs for metadata and logs, SSDs and SAS drives for storage. At some point in the near future we will participate in a IO-500 benchmark challange and see where we rank, until then here are some benchmarks

OpenStack Backend Performance

KyRIC Backend Performance

Here are the specific modes of testing, block sizes, and results:

Write: 296Gb/s
Mode: Random Writes
Block Size: 4096k

Read: 389Gb/s
Mode Random Reads
Block Size: 4096k

Max Write: 507k IOPS
Mode Random Writes
Block Size: 4k

Max Read: 2.02M IOPS
Mode Random Reads
Block Size: 4k

FIO

We are using the FIO (flexible IO) banchmark software, with the following config:

[global]
ioengine=rbd
clientname=${CLIENTNAME}

pool=.rbd.one.copy

invalidate=0    # mandatory
size=${VOLSIZE}
rw=${MODE}
runtime=${RUNTIME}
time_based
numjobs=1
bs=${BLOCKSIZE}
iodepth=${IODEPTH}

[randread-4k-iodepth_0]
rbdname=${CLIENTNAME}.0

[randread-4k-iodepth_1]
rbdname=${CLIENTNAME}.1

[randread-4k-iodepth_2]
rbdname=${CLIENTNAME}.2

[randread-4k-iodepth_3]
rbdname=${CLIENTNAME}.3

[randread-4k-iodepth_4]
rbdname=${CLIENTNAME}.4

[randread-4k-iodepth_5]
rbdname=${CLIENTNAME}.5

[randread-4k-iodepth_6]
rbdname=${CLIENTNAME}.6

[randread-4k-iodepth_7]
rbdname=${CLIENTNAME}.7

[randread-4k-iodepth_8]
rbdname=${CLIENTNAME}.8