/[smeserver]/rpms/dovecot/sme7/maildir-migration.txt
ViewVC logotype

Annotation of /rpms/dovecot/sme7/maildir-migration.txt

Parent Directory Parent Directory | Revision Log Revision Log | View Revision Graph Revision Graph


Revision 1.1 - (hide annotations) (download)
Tue Jun 12 15:07:20 2007 UTC (17 years, 4 months ago) by slords
Branch: MAIN
CVS Tags: dovecot-1_0-1_2_rc15_el4_sme, dovecot-1_0_7-2_el4_sme, HEAD
Content type: text/plain
Import on branch sme7 of package dovecot-1.0-1.2.rc15.el4.sme.src.rpm

1 slords 1.1 Replacing UW-IMAP with Dovecot on Red Hat Enterprise Linux 3.
2    
3     G.R.Keech <rkeech@redhat.com>
4    
5     2004-11-09
6    
7     Version 1.0
8    
9    
10     Summary
11    
12     The IMAP server in Red Hat Enterprise Linux version 3 (RHEL3) is the
13     University of Washington (UW) implemenation. UW IMAP uses the mbox
14     mail storage format which has been found to have significant
15     scalability and performance issues. This paper describes how to
16     replace UW IMAP with the Dovecot IMAP server which uses the maildir
17     mail storage format and, consequently, is significantly faster.
18    
19    
20     Maildir vs Mbox
21    
22     Format issues. The mbox format holds email with one file corresponding
23     to an entire mail folder. Maildir, on the other hand, stores messages
24     one directory per folder, with each message being a single file. The
25     mbox format doesn't scale well when many users have large mail folders
26     which are polled frequently. When the sum of sizes of all the
27     frequently polled mail greatly exceed the system's RAM, then mail
28     boxes cannot be effectively cached in RAM. This leads to a situation
29     where mbox files are being continually re-read with each poll by each
30     mail client, leading to excessive IO activity and poor performance.
31    
32     Example. Here is a real example of poor performance using UW IMAP.
33     The site characteristics were as follows:
34    
35     * about 116 active mail accounts with users using a mix of both
36     Outlook express and Outlook;
37    
38     * Mail clients poll for new mail every five minutes;
39    
40     * 1GB system RAM in mail server;
41    
42     * Mail server using Postfix on RHEL3 (kernel 2.4.21-20.EL);
43    
44     * System has single 3.0GHz P4 processor and SATA disk;
45    
46     * 112 inboxes totalling 3.3GB (average about 30MB each);
47    
48     * 8.5GB of filed email in imap folders.
49    
50     Performance with mbox. When in use the load average on the mail server
51     was normally between four and six with spikes above 15 not uncommon
52     and spikes above 20 occasionally. Most of this load was associated
53     with instances of uw-imap in disk wait state. The user experience in
54     checking mail was very poor, despite this being a reasonably small
55     site. When users checked for new mail they frequently encountered
56     server timeouts. For users with large inboxes, it was not uncommon
57     for it to take between 60 and 90 seconds just to check for new mail.
58    
59     Performance with maildir. Once the imap service was changed to
60     Dovecot, with all the other variables remaining unchanged, the
61     situation improved dramatically. Load average is very rarely above
62     0.5 and is usually under 0.2 during working hours. The user
63     experience was improved significantly. Users no longer
64     experience timeouts, and the time taken to check for new mail
65     is usually only two or three seconds and always less than 10 seconds.
66    
67    
68     Setting up Dovecot
69    
70     Dovecot. The Dovecot IMAP server will replace UW IMAP in RHEL4. The
71     Dovecot project is at http://dovecot.procontrol.fi/. It was built for
72     RHEL3 based on the package from the Beta of RHEL4. The RHEL3 version
73     used in this migration can be found at
74     http://people.redhat.com/rkeech/#dovecot.
75    
76     Operation. Dovecot, unlike uw-imap, runs as a conventional
77     daemon-based service, ie it is not run from xinetd. When running
78     A typical process hierarchy when Dovecot is running looks as follows:
79    
80     dovecot --+--- dovecot-auth
81     +--- imap
82     +--- imap-login
83    
84     One instance of the "imap" process will exist for each mail client
85     connection. These imap processes execute with the user context of
86     whichever mail account user is connecting. The main "dovecot" process
87     runs as root, as does "dovecot-auth". The imap-login processes run
88     as user "dovecot".
89    
90     Configuration. Dovecot is configured with /etc/dovecot.conf. The relevant
91     configuration setting that had to be changed in this case were as follows:
92    
93     protocols = imap
94     default_mail_env = maildir:/data/mail2/%u/
95    
96     User home. In this particular migration, mail account users did not
97     need shell access and did not need home directories for anything but
98     mail folder storage. Accordingly it was possible to change the users'
99     home directories to be the same as the mail spool area. This can be
100     achieved with (for this example)
101    
102     usermod -d /data/mail2/<user> <user>
103    
104     User shell. Mail accounts can function without a user shell. However
105     if a mail auto-responder (ie "vacation") is used then a shell is necessary.
106     Under RHEL3 a restricted shell is possible and reduces the security
107     concerns associated with providing shell access to mail users. For
108     a restricted version of bash make a sybolic link called "/bin/rbash"
109     pointing to "/bin/bash", and make /bin/rbash the users' shell.
110    
111     Postfix delivery. The location of mail delivery by Postfix is set
112     according to the "mail_spool_directory" parameter. When using uw-imap
113     this was set to "/var/spool/mail". A trailing slash on the
114     mail_spool_directory value directs Postfix to use maildir format. The
115     change to Postfix configuration in this case was done with:
116    
117     postconf -e "mail_spool_directory = /data/mail2/"
118    
119     Directory structure. For mail delivery to work with maildir a number
120     of sub-directories must exist in the user's personal mail spool
121     directory. If user "fred" has mail delivered to /data/mail2/fred/
122     then that directory should have subdirectories "cur" "new" and "tmp"
123     in it. The "new" directory is where incoming mail messages are put.
124     Other folders structured in subdirectories, eg an imap folder called
125     "Sent" for user "fred" would be /data/mail2/fred/.Sent, and there would
126     be a line "Sent" in the file /data/mail2/fred/.subscriptions.
127    
128     Account creation. In the case where the mail spool area is also the
129     home directory, then for new accounts to accept maildir mail delivery,
130     the following directories should be created:
131    
132     /etc/skel/cur
133     /etc/skel/tmp
134     /etc/skel/new
135    
136     thus when an account is created, the necessary subdirectories are
137     automatically created in the new user's home for a maildir-style
138     inbox.
139    
140    
141     Migrating Mbox folders to Maildirs
142    
143     Conversion. The tools to convert existing mbox-style mailboxes
144     and folders to maildir format are not provided with Dovecot or RHEL3.
145     A tool called perfect_maildir was found on the Net which performs
146     the conversion for a single folder only. A script was written to
147     provide for bulk conversion of folders for all users.
148    
149     Perfect_maildir. The tool lives at http://perfectmaildir.home-dn.net.
150     Versions prior to 0.3 should not be used. Perfect_maildir is a Perl
151     script.
152    
153     Migration plan. The migration to maildir was performed within the
154     following bounds:
155    
156     * existing inboxes were in /var/spool/mail
157    
158     * existing mail folders were under /home/<user>/
159    
160     * new inboxes and mailfolders were combined under /data/mail2/<user>/
161    
162    
163     Migration scripts. The scripts prepared to handle the bulk conversion are
164     "migration-users" and "migrate-folders". migrate-users creates the
165     necessary new user mail directories under the new base directory, which
166     in this case was /data/mail2/. In a situation where mail accounts
167     needed to be setup, without the need to migrate folders then migrate-users
168     would be sufficient, ie the second script would not be used.
169     migrate-folders does the main work of the migration by calling perfect_maildir
170     once for each mbox mailfolder including the users' inbox. migrate-users
171     must be run before migrate-folders.
172    
173    
174     Pre-migration. Before migrating, some things to check are:
175    
176     Will the file system have enough inodes now that the number of files
177     is much larger? "df -i" will help. Run tune2fs as required to
178     increase the allowance for inodes.
179    
180     Will there be a very large number of mail accounts? The maximum
181     number of subdirectories per directory is 32k. If the number of
182     accounts does or might exceed this, then change the Dovecot
183     default_mail_env to suit. See the Dovecot documentation for more
184     information.
185    
186     Where do you want dovecot to send its log messages? By default it
187     uses syslog and sends to the mail facility. This can be changed to
188     log to a its own log file if it is not convenient to log to maillog.
189    
190     Migration steps. The actual mail file migration was performed as below
191     using the scripts from http://people.rehdat.com/rkeech/#maildirmigration
192     Note, the migration steps described here leave the original mbox inboxes
193     and mail folders intact. Should the migration fail, then the capacity
194     to continue using mbox format is retained.
195    
196     1. Ensure that no new accounts are created until the migration is
197     complete. Normal email operation can proceed until step X. Do not
198     change Postfix's mail_spool_directory until step Y.
199    
200     2. Create a working directory /root/migrate/.
201    
202     3. Create a list of users whose mailboxes are to be migrated
203     /root/migrate/userlist-master. The file has one user per line.
204    
205     4. Create a list of mbox folders (excluding inboxes) to be migrated
206     /root/migrate/folderlist-master The file has one mbox per line and
207     excludes the path to the user's home, eg
208    
209     +-----------------
210     |fred/Drafts
211     |fred/private
212     |fred/Sent
213     |fred/office
214     |fred/projectx
215     |mary/Sent
216     |mary/private
217    
218    
219     Check that the folderlist does not contain any folders with characters
220     likely to be problematic when handled by scripts, ie folders whose names
221     include quotation marks, comas, pound signs, "&" symbols etc. Any such
222     folders should have their names changed. The name change in the
223     folderlist file should reflect the name change on disk. Folder names
224     with spaces, hyphens, percent signs, and periods are known to be OK.
225    
226     5. Proceed with a test migration with a small subset of users by
227     creating /root/migrate/userlist and /root/migrate/folderlist based on
228     subsets of userlist-master and folderlist-master. The test migration
229     should be done with accounts that have representative mail in both
230     inbox and imap folders.
231    
232     6. Ensure that the new mail base directory is clear. In this example
233     it means that /data/mail2/ is emtpy.
234    
235     7. Run migrate-users which should read /root/migrate/userlist and
236     create the new user mail directories under /data/mail2/. Check that
237     a directory for each user is created under /data/mail2/.
238    
239     8. Start dovecot on an alternative port so as not to intefere with the
240     normal uw-imap. This is done by editing /etc/dovecot.conf with
241    
242     imap_listen = 1.2.3.4:1043
243    
244     Obviously change address 1.2.3.4 to correspond to the local server.
245     Run
246    
247     service dovecot start
248    
249     Dovecot should start cleanly.
250    
251     It is assumed that the "protocols" and "default_mail_env" settings
252     are already set in /etc/dovecot.conf as described in "Configuration"
253     above.
254    
255     9. Run migrate-folders which should reate /root/migrate/folderlist and
256     populate all the necessary maildirs under each user directory.
257     This will invoke perfect_maildir.pl as required for each mbox file.
258     If the mail folders are large the this step could take some time.
259    
260     10. Test that the files created under /data/mail2/ are visible as
261     folders via Dovecot by configuring an imap mail client.to point
262     to port 1043 on the mail server using on of the test migrated accounts.
263     The imap mail client program should correctly sign on to the
264     new imap server and the migrated folders should be visible to
265     the user. The mail client should test that:
266    
267     a. new folders can be created and deleted.
268    
269     b. that existing mail from migrated folders is visible and that
270     it sorts correctly by date
271    
272     c. test that existing mail can be moved between folders.
273    
274     d. the mail client should see both an inbox and all other folders
275     which were in the folderlist.
276    
277     If step 10 passes (ie folders are visible as expected) then proceed with
278     full migration per the following steps. Do not proceed further unless
279     proper and expected mail folder operation is seen in the imap test of
280     step 10.
281    
282     11. Prepare new folderlist and userlist files from the respective
283     master files in the mitrate directory.
284    
285     12. Remove the test mail folders prepared steps 7 and 9 from the
286     new base directory (/data/mail2/* in this example).
287    
288     13. Declare mail as unavailable.
289    
290     14. Stop incoming mail (service postfix stop) and prevent further
291     use of imap (chkconfig imap off).
292    
293     15. Run migrate-users and migrate-folders again.
294    
295     16. Check a test user on the alternate port like step 10.
296     If all is OK, then
297    
298     17. Re-configure dovecot to operate on the normal imap port
299     by commenting out the imap_listen directive in dovecot.conf.
300     Re-start dovecot.
301    
302     18. Check that mail can be read on the normal imap port.
303    
304     19. Change email delivery into the new folders:
305    
306     postconf -e "mail_spool_directory = /data/mail2/"
307    
308     20 Re-start mail
309    
310     service postfix start
311    
312     21. Check and double check.
313    
314     22. Go home.
315    

admin@koozali.org
ViewVC Help
Powered by ViewVC 1.2.1 RSS 2.0 feed