linuxext3 andvarspoolmail round 2

This thread is a followup on my previous postings related to performance degradation after several months use of a linux-ext3var-spool-mail filesystem.

We have a sharp sysadmin here who has dug deeper into the problem and has come up with the following analysis I'd like to share with everyone:

-----

I found some interesting things about our problems that will eventually need to address. Apparently the concept of "block groups" is contributing to the problem. I found the program "debugfs" is able to give lots of information about a filesystem, including the block numbers that I was after. From what I found, this problem appears to be specific to parbreastions with a fairly small number of files, the majority of which are kept within the same directory. (like ourvar-spool-mail)

Background: block groups are used to try to keep related files together. Typically, directories are spread out between different block groups, and files inside of a directory get placed into the same block group as the parent directory. Files within a block group will try to use the data blocks within that group if possible, but will spill over into other groups if that one is full. Since the inode number determines which block group a file belongs to, most file operations will never change the block group that a file belongs to. Block groups appear to be hard-coded so that they have 128 MB in data blocks.

What this means for us: since pretty much all files invar-spool-mail are in the same (root) directory, and we will probably never use all of the inodes (file structures) within that block group, these files all get put into the same block group. Since these files need as many data blocks as there are in at least 100 block groups, this understandably causes problems. What we really need is a way to spread out our files among the different block groups.

I made the following observations:

*var-spool-mail has 1024 block groups (512 on new fs) * nearly all files invar-spool-mail are in block group 0, since this is the group which contains the root directory. (it is very bad for a lot of large files to be in the same block group) * very few free blocks until block group 103 (143 on new fs)

Part of the output from the "stats" command from within debugfs:

Group 0: block bitmap at 9, inode bitmap at 10, inode table at 11 1 free block, 12312 free inodes, 1 used directory Group 1: block bitmap at 32777, inode bitmap at 32778, inode table at 32779 0 free blocks, 16384 free inodes, 0 used directories Group 2: block bitmap at 65536, inode bitmap at 65537, inode table at 65538 0 free blocks, 16384 free inodes, 0 used directories ... Group 1023: block bitmap at 33521664, inode bitmap at 33521665, inode table at 33521666 31574 free blocks, 16384 free inodes, 0 used directories

What we can do about it: again, if we can spread out the files between the different block groups, we'd be in fine shape. One method that comes to mind is to create some subdirs invar-spool-mail that exist solely for the purpose of creating new files within different block groups. So we could have avar-spool-mail-.bgdirs-{0,1,2,3,...,512}, in which we would create inboxes at the time of account creation, and then move them back out to the mainvar-spool-mail directory. It doesn't appear that normal usage causes a file's inode to change; in the past day, no files changed inodes.

We would probably need to manually spread the mailboxes across different block groups initially, but from there, the above method should be able to keep things running smoothly. Moving a mailbox to a different block group would be fairly simple:

* lock the mailbox * copy it to .bgdirs-nnn * delete the original * move it from .bgdirs-nnn to the spool dir

-----

Possibly. Hard to say. From what I've found in man pages, xfs has the concept of "allocation groups," which I take to be more or less the same as the "block groups" in ext2-3. It's possible, perhaps even likely, that files get buttigned into allocation groups in xfs like they are buttigned block groups in ext2-3. (that is, they are put in the same one as their parent, if possible) In which case, xfs doesn't really offer much over ext2-3 in this case. Additionally, we don't currently have the utilities for working with xfs or reiserfs.

-----

Thoughts? Concerns?

Craig