Because of the nature of solid state storage, once the drive gets filled, write speed usually takes a precipitous plunge. We can take a fresh drive and write 4K randoms to the entire drive for 300 minutes to see what happens.
First, we write to the P320h with 4K random writes at total outstanding IO count of 256 over 300 minutes. The drive has been freshly purged, with no other preconditioning.
The P320h starts out strong, coming in right around the 500,000 IOPS mark. However, at that speed, it doesn’t take long to start filling up the drive. As that starts happening, the IOPS start dropping quickly, eventually settling right around the 212K mark, or over 850MB/s sustained. The Micron could conceivably do that for the rest of it’s life, until somewhere around 50 Petabytes of endurance… an extraordinary amount of endurance by any measure.
If we add latency to the mix, the relationship between it and IOPS becomes more apparent. Notice that the yellow and orange lines above look rather similar. As the drive fills and IOPS drop, latency increases. In the case of the P320h, it’s right around 600us average.
Why do latency and IOPS show this kind of inverse relationship? When the drive is empty, there are tons of free blocks to write to, meaning writes can be accomplished quickly. But once there are no free blocks, the drive gets locked in a struggle to free up dirty blocks and make them available again. This requires several steps, and causes transactions to be delayed.
Lastly, what happens when the P320h gets over-provisioned by 100GB? As you can see in the chart above (represented by the blue line), you get almost 50% sustained IOPS. Though the original and over-provisioned runs start at the same place, they both end up somewhere quite different. With more space to maintain blocks, the drive can keep more of it’s fresh out of box performance. By giving up 15% of the 650GB of total available capacity, a 50% increase in steady state IOPS can be had. That’s a great trade off if the extra speed is needed more than the 100GB of space.