user filesystems 1804

Tracy R Reed

It's funny, I always buttumed quite the opposite. That as hard drives got faster, I-O got more asynchronous from the CPU, and memory got cheaper, it made progressively less and less sense to compress on-disk data.

Originally, one benefit of compression was that you could effectively have a larger in-memory file cache. That kind of becomes pointless with memory as cheap and as large as it is today.

Another argument was the CPU cost of compression was offset by the reduced cost of transferring the data. But that was in the day when the CPU actually did have to transfer the data at bus speeds, often with operations narrower than the CPUs data bus and with pauses.

I guess the new argument is that compression means effectively a higher I-O bandwidth. However, against that, you have the CPU cost (that goes up with your I-O bandwidth as well). You also have to pbutt data through the CPU that you didn't necessarily have to pbutt through it before.

The ugly bit is that some operations that used to be very cheap become significantly more expensive. You don't get much compression if you compress each 4Kb block of a file independently. But if you don't, changing one byte in the middle of a file could potentially cause a ripple effect in the writing (and reading!) you need to do. Increasing the percentage of logical writes that require physical reads to know what to write back can't be good.

My overall impression is that a general purpose filesystem should not have compression these days. (And it seems to have died out as precisely these balances shifted.) Perhaps an service thread to compress data that hasn't changed in a long time might make sense, but there doesn't seem to be that much point.

DS