/[smeserver]/rpms/dovecot/sme7/maildir-migration.txt
ViewVC logotype

Contents of /rpms/dovecot/sme7/maildir-migration.txt

Parent Directory Parent Directory | Revision Log Revision Log | View Revision Graph Revision Graph


Revision 1.1 - (show annotations) (download)
Tue Jun 12 15:07:20 2007 UTC (17 years, 4 months ago) by slords
Branch: MAIN
CVS Tags: dovecot-1_0-1_2_rc15_el4_sme, dovecot-1_0_7-2_el4_sme, HEAD
Content type: text/plain
Import on branch sme7 of package dovecot-1.0-1.2.rc15.el4.sme.src.rpm

1 Replacing UW-IMAP with Dovecot on Red Hat Enterprise Linux 3.
2
3 G.R.Keech <rkeech@redhat.com>
4
5 2004-11-09
6
7 Version 1.0
8
9
10 Summary
11
12 The IMAP server in Red Hat Enterprise Linux version 3 (RHEL3) is the
13 University of Washington (UW) implemenation. UW IMAP uses the mbox
14 mail storage format which has been found to have significant
15 scalability and performance issues. This paper describes how to
16 replace UW IMAP with the Dovecot IMAP server which uses the maildir
17 mail storage format and, consequently, is significantly faster.
18
19
20 Maildir vs Mbox
21
22 Format issues. The mbox format holds email with one file corresponding
23 to an entire mail folder. Maildir, on the other hand, stores messages
24 one directory per folder, with each message being a single file. The
25 mbox format doesn't scale well when many users have large mail folders
26 which are polled frequently. When the sum of sizes of all the
27 frequently polled mail greatly exceed the system's RAM, then mail
28 boxes cannot be effectively cached in RAM. This leads to a situation
29 where mbox files are being continually re-read with each poll by each
30 mail client, leading to excessive IO activity and poor performance.
31
32 Example. Here is a real example of poor performance using UW IMAP.
33 The site characteristics were as follows:
34
35 * about 116 active mail accounts with users using a mix of both
36 Outlook express and Outlook;
37
38 * Mail clients poll for new mail every five minutes;
39
40 * 1GB system RAM in mail server;
41
42 * Mail server using Postfix on RHEL3 (kernel 2.4.21-20.EL);
43
44 * System has single 3.0GHz P4 processor and SATA disk;
45
46 * 112 inboxes totalling 3.3GB (average about 30MB each);
47
48 * 8.5GB of filed email in imap folders.
49
50 Performance with mbox. When in use the load average on the mail server
51 was normally between four and six with spikes above 15 not uncommon
52 and spikes above 20 occasionally. Most of this load was associated
53 with instances of uw-imap in disk wait state. The user experience in
54 checking mail was very poor, despite this being a reasonably small
55 site. When users checked for new mail they frequently encountered
56 server timeouts. For users with large inboxes, it was not uncommon
57 for it to take between 60 and 90 seconds just to check for new mail.
58
59 Performance with maildir. Once the imap service was changed to
60 Dovecot, with all the other variables remaining unchanged, the
61 situation improved dramatically. Load average is very rarely above
62 0.5 and is usually under 0.2 during working hours. The user
63 experience was improved significantly. Users no longer
64 experience timeouts, and the time taken to check for new mail
65 is usually only two or three seconds and always less than 10 seconds.
66
67
68 Setting up Dovecot
69
70 Dovecot. The Dovecot IMAP server will replace UW IMAP in RHEL4. The
71 Dovecot project is at http://dovecot.procontrol.fi/. It was built for
72 RHEL3 based on the package from the Beta of RHEL4. The RHEL3 version
73 used in this migration can be found at
74 http://people.redhat.com/rkeech/#dovecot.
75
76 Operation. Dovecot, unlike uw-imap, runs as a conventional
77 daemon-based service, ie it is not run from xinetd. When running
78 A typical process hierarchy when Dovecot is running looks as follows:
79
80 dovecot --+--- dovecot-auth
81 +--- imap
82 +--- imap-login
83
84 One instance of the "imap" process will exist for each mail client
85 connection. These imap processes execute with the user context of
86 whichever mail account user is connecting. The main "dovecot" process
87 runs as root, as does "dovecot-auth". The imap-login processes run
88 as user "dovecot".
89
90 Configuration. Dovecot is configured with /etc/dovecot.conf. The relevant
91 configuration setting that had to be changed in this case were as follows:
92
93 protocols = imap
94 default_mail_env = maildir:/data/mail2/%u/
95
96 User home. In this particular migration, mail account users did not
97 need shell access and did not need home directories for anything but
98 mail folder storage. Accordingly it was possible to change the users'
99 home directories to be the same as the mail spool area. This can be
100 achieved with (for this example)
101
102 usermod -d /data/mail2/<user> <user>
103
104 User shell. Mail accounts can function without a user shell. However
105 if a mail auto-responder (ie "vacation") is used then a shell is necessary.
106 Under RHEL3 a restricted shell is possible and reduces the security
107 concerns associated with providing shell access to mail users. For
108 a restricted version of bash make a sybolic link called "/bin/rbash"
109 pointing to "/bin/bash", and make /bin/rbash the users' shell.
110
111 Postfix delivery. The location of mail delivery by Postfix is set
112 according to the "mail_spool_directory" parameter. When using uw-imap
113 this was set to "/var/spool/mail". A trailing slash on the
114 mail_spool_directory value directs Postfix to use maildir format. The
115 change to Postfix configuration in this case was done with:
116
117 postconf -e "mail_spool_directory = /data/mail2/"
118
119 Directory structure. For mail delivery to work with maildir a number
120 of sub-directories must exist in the user's personal mail spool
121 directory. If user "fred" has mail delivered to /data/mail2/fred/
122 then that directory should have subdirectories "cur" "new" and "tmp"
123 in it. The "new" directory is where incoming mail messages are put.
124 Other folders structured in subdirectories, eg an imap folder called
125 "Sent" for user "fred" would be /data/mail2/fred/.Sent, and there would
126 be a line "Sent" in the file /data/mail2/fred/.subscriptions.
127
128 Account creation. In the case where the mail spool area is also the
129 home directory, then for new accounts to accept maildir mail delivery,
130 the following directories should be created:
131
132 /etc/skel/cur
133 /etc/skel/tmp
134 /etc/skel/new
135
136 thus when an account is created, the necessary subdirectories are
137 automatically created in the new user's home for a maildir-style
138 inbox.
139
140
141 Migrating Mbox folders to Maildirs
142
143 Conversion. The tools to convert existing mbox-style mailboxes
144 and folders to maildir format are not provided with Dovecot or RHEL3.
145 A tool called perfect_maildir was found on the Net which performs
146 the conversion for a single folder only. A script was written to
147 provide for bulk conversion of folders for all users.
148
149 Perfect_maildir. The tool lives at http://perfectmaildir.home-dn.net.
150 Versions prior to 0.3 should not be used. Perfect_maildir is a Perl
151 script.
152
153 Migration plan. The migration to maildir was performed within the
154 following bounds:
155
156 * existing inboxes were in /var/spool/mail
157
158 * existing mail folders were under /home/<user>/
159
160 * new inboxes and mailfolders were combined under /data/mail2/<user>/
161
162
163 Migration scripts. The scripts prepared to handle the bulk conversion are
164 "migration-users" and "migrate-folders". migrate-users creates the
165 necessary new user mail directories under the new base directory, which
166 in this case was /data/mail2/. In a situation where mail accounts
167 needed to be setup, without the need to migrate folders then migrate-users
168 would be sufficient, ie the second script would not be used.
169 migrate-folders does the main work of the migration by calling perfect_maildir
170 once for each mbox mailfolder including the users' inbox. migrate-users
171 must be run before migrate-folders.
172
173
174 Pre-migration. Before migrating, some things to check are:
175
176 Will the file system have enough inodes now that the number of files
177 is much larger? "df -i" will help. Run tune2fs as required to
178 increase the allowance for inodes.
179
180 Will there be a very large number of mail accounts? The maximum
181 number of subdirectories per directory is 32k. If the number of
182 accounts does or might exceed this, then change the Dovecot
183 default_mail_env to suit. See the Dovecot documentation for more
184 information.
185
186 Where do you want dovecot to send its log messages? By default it
187 uses syslog and sends to the mail facility. This can be changed to
188 log to a its own log file if it is not convenient to log to maillog.
189
190 Migration steps. The actual mail file migration was performed as below
191 using the scripts from http://people.rehdat.com/rkeech/#maildirmigration
192 Note, the migration steps described here leave the original mbox inboxes
193 and mail folders intact. Should the migration fail, then the capacity
194 to continue using mbox format is retained.
195
196 1. Ensure that no new accounts are created until the migration is
197 complete. Normal email operation can proceed until step X. Do not
198 change Postfix's mail_spool_directory until step Y.
199
200 2. Create a working directory /root/migrate/.
201
202 3. Create a list of users whose mailboxes are to be migrated
203 /root/migrate/userlist-master. The file has one user per line.
204
205 4. Create a list of mbox folders (excluding inboxes) to be migrated
206 /root/migrate/folderlist-master The file has one mbox per line and
207 excludes the path to the user's home, eg
208
209 +-----------------
210 |fred/Drafts
211 |fred/private
212 |fred/Sent
213 |fred/office
214 |fred/projectx
215 |mary/Sent
216 |mary/private
217
218
219 Check that the folderlist does not contain any folders with characters
220 likely to be problematic when handled by scripts, ie folders whose names
221 include quotation marks, comas, pound signs, "&" symbols etc. Any such
222 folders should have their names changed. The name change in the
223 folderlist file should reflect the name change on disk. Folder names
224 with spaces, hyphens, percent signs, and periods are known to be OK.
225
226 5. Proceed with a test migration with a small subset of users by
227 creating /root/migrate/userlist and /root/migrate/folderlist based on
228 subsets of userlist-master and folderlist-master. The test migration
229 should be done with accounts that have representative mail in both
230 inbox and imap folders.
231
232 6. Ensure that the new mail base directory is clear. In this example
233 it means that /data/mail2/ is emtpy.
234
235 7. Run migrate-users which should read /root/migrate/userlist and
236 create the new user mail directories under /data/mail2/. Check that
237 a directory for each user is created under /data/mail2/.
238
239 8. Start dovecot on an alternative port so as not to intefere with the
240 normal uw-imap. This is done by editing /etc/dovecot.conf with
241
242 imap_listen = 1.2.3.4:1043
243
244 Obviously change address 1.2.3.4 to correspond to the local server.
245 Run
246
247 service dovecot start
248
249 Dovecot should start cleanly.
250
251 It is assumed that the "protocols" and "default_mail_env" settings
252 are already set in /etc/dovecot.conf as described in "Configuration"
253 above.
254
255 9. Run migrate-folders which should reate /root/migrate/folderlist and
256 populate all the necessary maildirs under each user directory.
257 This will invoke perfect_maildir.pl as required for each mbox file.
258 If the mail folders are large the this step could take some time.
259
260 10. Test that the files created under /data/mail2/ are visible as
261 folders via Dovecot by configuring an imap mail client.to point
262 to port 1043 on the mail server using on of the test migrated accounts.
263 The imap mail client program should correctly sign on to the
264 new imap server and the migrated folders should be visible to
265 the user. The mail client should test that:
266
267 a. new folders can be created and deleted.
268
269 b. that existing mail from migrated folders is visible and that
270 it sorts correctly by date
271
272 c. test that existing mail can be moved between folders.
273
274 d. the mail client should see both an inbox and all other folders
275 which were in the folderlist.
276
277 If step 10 passes (ie folders are visible as expected) then proceed with
278 full migration per the following steps. Do not proceed further unless
279 proper and expected mail folder operation is seen in the imap test of
280 step 10.
281
282 11. Prepare new folderlist and userlist files from the respective
283 master files in the mitrate directory.
284
285 12. Remove the test mail folders prepared steps 7 and 9 from the
286 new base directory (/data/mail2/* in this example).
287
288 13. Declare mail as unavailable.
289
290 14. Stop incoming mail (service postfix stop) and prevent further
291 use of imap (chkconfig imap off).
292
293 15. Run migrate-users and migrate-folders again.
294
295 16. Check a test user on the alternate port like step 10.
296 If all is OK, then
297
298 17. Re-configure dovecot to operate on the normal imap port
299 by commenting out the imap_listen directive in dovecot.conf.
300 Re-start dovecot.
301
302 18. Check that mail can be read on the normal imap port.
303
304 19. Change email delivery into the new folders:
305
306 postconf -e "mail_spool_directory = /data/mail2/"
307
308 20 Re-start mail
309
310 service postfix start
311
312 21. Check and double check.
313
314 22. Go home.
315

admin@koozali.org
ViewVC Help
Powered by ViewVC 1.2.1 RSS 2.0 feed