| 1 |
Replacing UW-IMAP with Dovecot on Red Hat Enterprise Linux 3. |
| 2 |
|
| 3 |
G.R.Keech <rkeech@redhat.com> |
| 4 |
|
| 5 |
2004-11-09 |
| 6 |
|
| 7 |
Version 1.0 |
| 8 |
|
| 9 |
|
| 10 |
Summary |
| 11 |
|
| 12 |
The IMAP server in Red Hat Enterprise Linux version 3 (RHEL3) is the |
| 13 |
University of Washington (UW) implemenation. UW IMAP uses the mbox |
| 14 |
mail storage format which has been found to have significant |
| 15 |
scalability and performance issues. This paper describes how to |
| 16 |
replace UW IMAP with the Dovecot IMAP server which uses the maildir |
| 17 |
mail storage format and, consequently, is significantly faster. |
| 18 |
|
| 19 |
|
| 20 |
Maildir vs Mbox |
| 21 |
|
| 22 |
Format issues. The mbox format holds email with one file corresponding |
| 23 |
to an entire mail folder. Maildir, on the other hand, stores messages |
| 24 |
one directory per folder, with each message being a single file. The |
| 25 |
mbox format doesn't scale well when many users have large mail folders |
| 26 |
which are polled frequently. When the sum of sizes of all the |
| 27 |
frequently polled mail greatly exceed the system's RAM, then mail |
| 28 |
boxes cannot be effectively cached in RAM. This leads to a situation |
| 29 |
where mbox files are being continually re-read with each poll by each |
| 30 |
mail client, leading to excessive IO activity and poor performance. |
| 31 |
|
| 32 |
Example. Here is a real example of poor performance using UW IMAP. |
| 33 |
The site characteristics were as follows: |
| 34 |
|
| 35 |
* about 116 active mail accounts with users using a mix of both |
| 36 |
Outlook express and Outlook; |
| 37 |
|
| 38 |
* Mail clients poll for new mail every five minutes; |
| 39 |
|
| 40 |
* 1GB system RAM in mail server; |
| 41 |
|
| 42 |
* Mail server using Postfix on RHEL3 (kernel 2.4.21-20.EL); |
| 43 |
|
| 44 |
* System has single 3.0GHz P4 processor and SATA disk; |
| 45 |
|
| 46 |
* 112 inboxes totalling 3.3GB (average about 30MB each); |
| 47 |
|
| 48 |
* 8.5GB of filed email in imap folders. |
| 49 |
|
| 50 |
Performance with mbox. When in use the load average on the mail server |
| 51 |
was normally between four and six with spikes above 15 not uncommon |
| 52 |
and spikes above 20 occasionally. Most of this load was associated |
| 53 |
with instances of uw-imap in disk wait state. The user experience in |
| 54 |
checking mail was very poor, despite this being a reasonably small |
| 55 |
site. When users checked for new mail they frequently encountered |
| 56 |
server timeouts. For users with large inboxes, it was not uncommon |
| 57 |
for it to take between 60 and 90 seconds just to check for new mail. |
| 58 |
|
| 59 |
Performance with maildir. Once the imap service was changed to |
| 60 |
Dovecot, with all the other variables remaining unchanged, the |
| 61 |
situation improved dramatically. Load average is very rarely above |
| 62 |
0.5 and is usually under 0.2 during working hours. The user |
| 63 |
experience was improved significantly. Users no longer |
| 64 |
experience timeouts, and the time taken to check for new mail |
| 65 |
is usually only two or three seconds and always less than 10 seconds. |
| 66 |
|
| 67 |
|
| 68 |
Setting up Dovecot |
| 69 |
|
| 70 |
Dovecot. The Dovecot IMAP server will replace UW IMAP in RHEL4. The |
| 71 |
Dovecot project is at http://dovecot.procontrol.fi/. It was built for |
| 72 |
RHEL3 based on the package from the Beta of RHEL4. The RHEL3 version |
| 73 |
used in this migration can be found at |
| 74 |
http://people.redhat.com/rkeech/#dovecot. |
| 75 |
|
| 76 |
Operation. Dovecot, unlike uw-imap, runs as a conventional |
| 77 |
daemon-based service, ie it is not run from xinetd. When running |
| 78 |
A typical process hierarchy when Dovecot is running looks as follows: |
| 79 |
|
| 80 |
dovecot --+--- dovecot-auth |
| 81 |
+--- imap |
| 82 |
+--- imap-login |
| 83 |
|
| 84 |
One instance of the "imap" process will exist for each mail client |
| 85 |
connection. These imap processes execute with the user context of |
| 86 |
whichever mail account user is connecting. The main "dovecot" process |
| 87 |
runs as root, as does "dovecot-auth". The imap-login processes run |
| 88 |
as user "dovecot". |
| 89 |
|
| 90 |
Configuration. Dovecot is configured with /etc/dovecot.conf. The relevant |
| 91 |
configuration setting that had to be changed in this case were as follows: |
| 92 |
|
| 93 |
protocols = imap |
| 94 |
default_mail_env = maildir:/data/mail2/%u/ |
| 95 |
|
| 96 |
User home. In this particular migration, mail account users did not |
| 97 |
need shell access and did not need home directories for anything but |
| 98 |
mail folder storage. Accordingly it was possible to change the users' |
| 99 |
home directories to be the same as the mail spool area. This can be |
| 100 |
achieved with (for this example) |
| 101 |
|
| 102 |
usermod -d /data/mail2/<user> <user> |
| 103 |
|
| 104 |
User shell. Mail accounts can function without a user shell. However |
| 105 |
if a mail auto-responder (ie "vacation") is used then a shell is necessary. |
| 106 |
Under RHEL3 a restricted shell is possible and reduces the security |
| 107 |
concerns associated with providing shell access to mail users. For |
| 108 |
a restricted version of bash make a sybolic link called "/bin/rbash" |
| 109 |
pointing to "/bin/bash", and make /bin/rbash the users' shell. |
| 110 |
|
| 111 |
Postfix delivery. The location of mail delivery by Postfix is set |
| 112 |
according to the "mail_spool_directory" parameter. When using uw-imap |
| 113 |
this was set to "/var/spool/mail". A trailing slash on the |
| 114 |
mail_spool_directory value directs Postfix to use maildir format. The |
| 115 |
change to Postfix configuration in this case was done with: |
| 116 |
|
| 117 |
postconf -e "mail_spool_directory = /data/mail2/" |
| 118 |
|
| 119 |
Directory structure. For mail delivery to work with maildir a number |
| 120 |
of sub-directories must exist in the user's personal mail spool |
| 121 |
directory. If user "fred" has mail delivered to /data/mail2/fred/ |
| 122 |
then that directory should have subdirectories "cur" "new" and "tmp" |
| 123 |
in it. The "new" directory is where incoming mail messages are put. |
| 124 |
Other folders structured in subdirectories, eg an imap folder called |
| 125 |
"Sent" for user "fred" would be /data/mail2/fred/.Sent, and there would |
| 126 |
be a line "Sent" in the file /data/mail2/fred/.subscriptions. |
| 127 |
|
| 128 |
Account creation. In the case where the mail spool area is also the |
| 129 |
home directory, then for new accounts to accept maildir mail delivery, |
| 130 |
the following directories should be created: |
| 131 |
|
| 132 |
/etc/skel/cur |
| 133 |
/etc/skel/tmp |
| 134 |
/etc/skel/new |
| 135 |
|
| 136 |
thus when an account is created, the necessary subdirectories are |
| 137 |
automatically created in the new user's home for a maildir-style |
| 138 |
inbox. |
| 139 |
|
| 140 |
|
| 141 |
Migrating Mbox folders to Maildirs |
| 142 |
|
| 143 |
Conversion. The tools to convert existing mbox-style mailboxes |
| 144 |
and folders to maildir format are not provided with Dovecot or RHEL3. |
| 145 |
A tool called perfect_maildir was found on the Net which performs |
| 146 |
the conversion for a single folder only. A script was written to |
| 147 |
provide for bulk conversion of folders for all users. |
| 148 |
|
| 149 |
Perfect_maildir. The tool lives at http://perfectmaildir.home-dn.net. |
| 150 |
Versions prior to 0.3 should not be used. Perfect_maildir is a Perl |
| 151 |
script. |
| 152 |
|
| 153 |
Migration plan. The migration to maildir was performed within the |
| 154 |
following bounds: |
| 155 |
|
| 156 |
* existing inboxes were in /var/spool/mail |
| 157 |
|
| 158 |
* existing mail folders were under /home/<user>/ |
| 159 |
|
| 160 |
* new inboxes and mailfolders were combined under /data/mail2/<user>/ |
| 161 |
|
| 162 |
|
| 163 |
Migration scripts. The scripts prepared to handle the bulk conversion are |
| 164 |
"migration-users" and "migrate-folders". migrate-users creates the |
| 165 |
necessary new user mail directories under the new base directory, which |
| 166 |
in this case was /data/mail2/. In a situation where mail accounts |
| 167 |
needed to be setup, without the need to migrate folders then migrate-users |
| 168 |
would be sufficient, ie the second script would not be used. |
| 169 |
migrate-folders does the main work of the migration by calling perfect_maildir |
| 170 |
once for each mbox mailfolder including the users' inbox. migrate-users |
| 171 |
must be run before migrate-folders. |
| 172 |
|
| 173 |
|
| 174 |
Pre-migration. Before migrating, some things to check are: |
| 175 |
|
| 176 |
Will the file system have enough inodes now that the number of files |
| 177 |
is much larger? "df -i" will help. Run tune2fs as required to |
| 178 |
increase the allowance for inodes. |
| 179 |
|
| 180 |
Will there be a very large number of mail accounts? The maximum |
| 181 |
number of subdirectories per directory is 32k. If the number of |
| 182 |
accounts does or might exceed this, then change the Dovecot |
| 183 |
default_mail_env to suit. See the Dovecot documentation for more |
| 184 |
information. |
| 185 |
|
| 186 |
Where do you want dovecot to send its log messages? By default it |
| 187 |
uses syslog and sends to the mail facility. This can be changed to |
| 188 |
log to a its own log file if it is not convenient to log to maillog. |
| 189 |
|
| 190 |
Migration steps. The actual mail file migration was performed as below |
| 191 |
using the scripts from http://people.rehdat.com/rkeech/#maildirmigration |
| 192 |
Note, the migration steps described here leave the original mbox inboxes |
| 193 |
and mail folders intact. Should the migration fail, then the capacity |
| 194 |
to continue using mbox format is retained. |
| 195 |
|
| 196 |
1. Ensure that no new accounts are created until the migration is |
| 197 |
complete. Normal email operation can proceed until step X. Do not |
| 198 |
change Postfix's mail_spool_directory until step Y. |
| 199 |
|
| 200 |
2. Create a working directory /root/migrate/. |
| 201 |
|
| 202 |
3. Create a list of users whose mailboxes are to be migrated |
| 203 |
/root/migrate/userlist-master. The file has one user per line. |
| 204 |
|
| 205 |
4. Create a list of mbox folders (excluding inboxes) to be migrated |
| 206 |
/root/migrate/folderlist-master The file has one mbox per line and |
| 207 |
excludes the path to the user's home, eg |
| 208 |
|
| 209 |
+----------------- |
| 210 |
|fred/Drafts |
| 211 |
|fred/private |
| 212 |
|fred/Sent |
| 213 |
|fred/office |
| 214 |
|fred/projectx |
| 215 |
|mary/Sent |
| 216 |
|mary/private |
| 217 |
|
| 218 |
|
| 219 |
Check that the folderlist does not contain any folders with characters |
| 220 |
likely to be problematic when handled by scripts, ie folders whose names |
| 221 |
include quotation marks, comas, pound signs, "&" symbols etc. Any such |
| 222 |
folders should have their names changed. The name change in the |
| 223 |
folderlist file should reflect the name change on disk. Folder names |
| 224 |
with spaces, hyphens, percent signs, and periods are known to be OK. |
| 225 |
|
| 226 |
5. Proceed with a test migration with a small subset of users by |
| 227 |
creating /root/migrate/userlist and /root/migrate/folderlist based on |
| 228 |
subsets of userlist-master and folderlist-master. The test migration |
| 229 |
should be done with accounts that have representative mail in both |
| 230 |
inbox and imap folders. |
| 231 |
|
| 232 |
6. Ensure that the new mail base directory is clear. In this example |
| 233 |
it means that /data/mail2/ is emtpy. |
| 234 |
|
| 235 |
7. Run migrate-users which should read /root/migrate/userlist and |
| 236 |
create the new user mail directories under /data/mail2/. Check that |
| 237 |
a directory for each user is created under /data/mail2/. |
| 238 |
|
| 239 |
8. Start dovecot on an alternative port so as not to intefere with the |
| 240 |
normal uw-imap. This is done by editing /etc/dovecot.conf with |
| 241 |
|
| 242 |
imap_listen = 1.2.3.4:1043 |
| 243 |
|
| 244 |
Obviously change address 1.2.3.4 to correspond to the local server. |
| 245 |
Run |
| 246 |
|
| 247 |
service dovecot start |
| 248 |
|
| 249 |
Dovecot should start cleanly. |
| 250 |
|
| 251 |
It is assumed that the "protocols" and "default_mail_env" settings |
| 252 |
are already set in /etc/dovecot.conf as described in "Configuration" |
| 253 |
above. |
| 254 |
|
| 255 |
9. Run migrate-folders which should reate /root/migrate/folderlist and |
| 256 |
populate all the necessary maildirs under each user directory. |
| 257 |
This will invoke perfect_maildir.pl as required for each mbox file. |
| 258 |
If the mail folders are large the this step could take some time. |
| 259 |
|
| 260 |
10. Test that the files created under /data/mail2/ are visible as |
| 261 |
folders via Dovecot by configuring an imap mail client.to point |
| 262 |
to port 1043 on the mail server using on of the test migrated accounts. |
| 263 |
The imap mail client program should correctly sign on to the |
| 264 |
new imap server and the migrated folders should be visible to |
| 265 |
the user. The mail client should test that: |
| 266 |
|
| 267 |
a. new folders can be created and deleted. |
| 268 |
|
| 269 |
b. that existing mail from migrated folders is visible and that |
| 270 |
it sorts correctly by date |
| 271 |
|
| 272 |
c. test that existing mail can be moved between folders. |
| 273 |
|
| 274 |
d. the mail client should see both an inbox and all other folders |
| 275 |
which were in the folderlist. |
| 276 |
|
| 277 |
If step 10 passes (ie folders are visible as expected) then proceed with |
| 278 |
full migration per the following steps. Do not proceed further unless |
| 279 |
proper and expected mail folder operation is seen in the imap test of |
| 280 |
step 10. |
| 281 |
|
| 282 |
11. Prepare new folderlist and userlist files from the respective |
| 283 |
master files in the mitrate directory. |
| 284 |
|
| 285 |
12. Remove the test mail folders prepared steps 7 and 9 from the |
| 286 |
new base directory (/data/mail2/* in this example). |
| 287 |
|
| 288 |
13. Declare mail as unavailable. |
| 289 |
|
| 290 |
14. Stop incoming mail (service postfix stop) and prevent further |
| 291 |
use of imap (chkconfig imap off). |
| 292 |
|
| 293 |
15. Run migrate-users and migrate-folders again. |
| 294 |
|
| 295 |
16. Check a test user on the alternate port like step 10. |
| 296 |
If all is OK, then |
| 297 |
|
| 298 |
17. Re-configure dovecot to operate on the normal imap port |
| 299 |
by commenting out the imap_listen directive in dovecot.conf. |
| 300 |
Re-start dovecot. |
| 301 |
|
| 302 |
18. Check that mail can be read on the normal imap port. |
| 303 |
|
| 304 |
19. Change email delivery into the new folders: |
| 305 |
|
| 306 |
postconf -e "mail_spool_directory = /data/mail2/" |
| 307 |
|
| 308 |
20 Re-start mail |
| 309 |
|
| 310 |
service postfix start |
| 311 |
|
| 312 |
21. Check and double check. |
| 313 |
|
| 314 |
22. Go home. |
| 315 |
|
Parent Directory
| 
Revision Log
| 
Revision Graph
