For some unknown reason, Minecraft is sleeping 10ms between every single chunk being saved to disk.
Under high chunk load/unload activity (lots of movement / teleporting), this causes the chunk unload queue
to build up in size.
This has multiple impacts:
1) Performance of the unload queue itself - The save thread is pretty ineffecient for how it accesses it
By letting the queue get larger, checking and popping work off the queue can get less performant.
2) Performance of chunk loading - As with #1, chunk loads also have to check this queue when loading
chunk data so that it doesn't load stale data if new data is pending write to disk.
3) Memory Usage - The entire chunk has been serialized to NBT, and now sits in this queue. This leads to
elevated memory usage, and then the objects used in the serialization sit around longer than needed,
resulting in promotion to Old Generation instead of dying young.
If there is work to do, then the thread should be doing its work, and only sleep when it is done.
Minecraft ineffeciently uses OutputStreams by calling .write(int) on the stream.
For Chunks, this is a DeflaterOutputStream, which allocates a single byte EVERY write.
This is causing the server to allocate tons of new byte[1] objects.
Additionally, this is very ineffecient for the Deflate process.
By Buffering Writes the same way it already is Buffering Reads, we will
write to the stream much more effeciently.
Also a more effecient RegionFile zero'ing for new chunks to speed up
new chunk generation.
md_5
Zach Brown
Aikar
Melair
Thinkofdeath