The NVMe-oF (NVMe over Fabrics) stack is based on our self asynchronous, high concurrency scheduler, and FlexSDS has created and designed the NVMe-oF module to make it as a high performance, high concurrency, polling mode and multiple channel (core) NVMe-oF stack for FlexSDS scale-out storage, that makes storage IO could be lingered scale-up by adding hardware resources.
The NVMe-oF stack is working on dedicated core for each channel, except discovery part, the full stack is completely kernel-by-pass (or OS-Bypass), for discovery part, it works globally, listen to specified port for accepting client connection, once it get an connection, it will redirect the client connection to a specified core via R-Call (Remote core RPC), R-Call is FlexSDS cross core communication component, it drives lock-free model communication between multiple CPU cores, using round-robin way to select one I/O polling core (if there is more than one I/O polling cores).
During support to the traditional I/O model, FlexSDS supports the following model for NVMe-oF service:
1. NVMe user mode (kernel-by-pass).
2. NVMe kernel mode (through /dev/nvmeXnY)
3. SATA / SAS disks via /dev/sdX