So with 5/8ths of a second left, and an average latency of 10ms, the client would be able to write only and additional 500KB worth of data. With this artificially steady rate, the delay would occur 7/8ths of the way into our 5 second window, with 1/8th of the total remaining. Recall that after 7/8ths of the limit is consumed, the old write throttle starts inserting a 10ms delay, so the first 437.5MB would come sailing in, say, with an average latency of 780µs, but then the remaining writes would average at least 10ms (scheduling delay could drive this even higher). With a target of 5 seconds to write out its contents, the currently open transaction group would be limited to 500MB.
ZFS VS OPENZFS FULL
Here’s how that scenario would play out with the old write throttle assuming full quiesced and syncing transaction groups (you may want to refer to my last blog post for a refresher on the mechanism and some of the numbers). For a client with 10 threads executing 8KB async writes (again to keep it simple) to hit 100MB/s, the average latency would be around 780µs - not unreasonable. Consider a backend that can handle an unwavering 100MB/s (or 1GB/s or 10GB/s - pick your number).
On a frictionless surface in a vacuum…Įven in the most rigorously contrived, predictable cases, the old write throttle would yield high variance in the latency of writes. Further, the mechanism relied on an accurate calculation of the backend throughput - a problem in itself, but one we’ll set aside for the moment. The core problem involved the actual throttling mechanism, allowing many fast writes while stalling some writes nearly without bound, with some artificially delayed writes ostensibly to soften the landing. The ZFS write throttle was fundamentally flawed - the data indelibly supported this diagnosis. While the solution we came to may seem obvious, we arrived at it only through a bit of wandering in a wide open solution space. Matt Ahrens, the co-founder of ZFS, and I set about to fix it in OpenZFS. In my last blog post, I wrote about the ZFS write throttle, and how we saw it lead to pathological latency variability on customer systems.