November 2007 - dovecot

[Dovecot] imap process consuming 100% CPU (Dovecot 1.0.3)
by Robert Tomanek 20 Nov '07

20 Nov '07

Hi, I have yet another problem with Dovecot: sometimes (rarely, maybe once every few days) one of the imap processes will 'hang', consuming all available CPU time. It does not seem to 'finish' in any reasonable amount of time (in one instance I waited a few days). This process will not even exit gracefully, it needs to be killed with 'kill -9 <PID>'. It has just happened again. The output from gdb is: > # gdb attach 29401 > GNU gdb 6.3 > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "i586-suse-linux"...attach: No such file or directory. > > Attaching to process 29401 > Reading symbols from /usr/local/libexec/dovecot/imap...done. > Using host libthread_db library "/lib/tls/libthread_db.so.1". > Reading symbols from /lib/libdl.so.2...done. > Loaded symbols for /lib/libdl.so.2 > Reading symbols from /lib/tls/libc.so.6...done. > Loaded symbols for /lib/tls/libc.so.6 > Reading symbols from /lib/ld-linux.so.2...done. > Loaded symbols for /lib/ld-linux.so.2 > 0x0806049d in imap_sync_more (ctx=0x80d9770) at imap-sync.c:104 > 104 if (ctx->seq == 0) { > (gdb) bt full > #0 0x0806049d in imap_sync_more (ctx=0x80d9770) at imap-sync.c:104 > flags = 1075042320 > keywords = <value optimized out> > str = (string_t *) 0x80c31e0 > ret = 1 > #1 0x08060797 in cmd_sync_continue (cmd=0x80ccdbc) at imap-sync.c:207 > ctx = (struct cmd_sync_context *) 0x80cf000 > ret = <value optimized out> > #2 0x0805ad5a in _client_output (context=0x80ccd78) at client.c:465 > cmd = (struct client_command_context *) 0x80ccdbc > ret = 0 > #3 0x080af0bd in stream_send_io (context=0x80cceb0) at ostream-file.c:356 > ostream = (struct ostream *) 0x80ccee0 > ret = <value optimized out> > #4 0x080aa441 in io_loop_handler_run (ioloop=0x80cb9b0) at ioloop-epoll.c:203 > ctx = (struct ioloop_handler_context *) 0x80cb9d8 > list = (struct io_list *) 0x80d7780 > io = (struct io *) 0x80dc430 > tv = {tv_sec = 2, tv_usec = 540048} > events_count = 4 > t_id = 2 > msecs = <value optimized out> > ret = 1 > i = 0 > j = 1 > call = <value optimized out> > #5 0x080a9889 in io_loop_run (ioloop=0x80cb9b0) at ioloop.c:329 > No locals. > #6 0x08062087 in main (argc=1, argv=0xbffffaf4, envp=0xbffffafc) at main.c:290 > plugin_dir = <value optimized out> Mail log does not reveal anything immediately interesting. Thoughts? Server config ('dovecot -n -c /etc/dovecot/dovecot.conf'): > # 1.0.3: /etc/dovecot/dovecot.conf > base_dir: /var/run/dovecot/ > protocols: imap imaps pop3 pop3s > ssl_ca_file: /etc/ssl/current/cacert.pem > ssl_cert_file: /etc/ssl/current/server.pem > ssl_key_file: /etc/ssl/current/private.pem > disable_plaintext_auth: no > login_dir: /var/run/dovecot//login > login_executable(default): /usr/local/libexec/dovecot/imap-login > login_executable(imap): /usr/local/libexec/dovecot/imap-login > login_executable(pop3): /usr/local/libexec/dovecot/pop3-login > first_valid_uid: 50 > mail_debug: yes > mail_drop_priv_before_exec: yes > mail_executable(default): /usr/local/libexec/dovecot/imap > mail_executable(imap): /usr/local/libexec/dovecot/imap > mail_executable(pop3): /usr/local/libexec/dovecot/pop3 > mail_plugin_dir(default): /usr/local/lib/dovecot/imap > mail_plugin_dir(imap): /usr/local/lib/dovecot/imap > mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3 > mail_log_prefix: %Us(%u)(%p): > pop3_uidl_format(default): > pop3_uidl_format(imap): > pop3_uidl_format(pop3): %08Xu%08Xv > auth default: > debug_passwords: yes > passdb: > driver: passwd-file > args: /etc/dovecot/passwd > userdb: > driver: passwd-file > args: /etc/dovecot/passwd > socket: > type: listen > client: > path: /var/spool/postfix/private/auth > mode: 432 > user: postfix > group: postfix -- Best regards, Robert Tomanek mailto:rth23@poczta.onet.pl

4 14

[Dovecot] mail_location backends
by Harrison Metzger 19 Nov '07

19 Nov '07

Hello, Where I work we have have an interesting setup, we have 2 machines with mail accounts on them and both have dovecot. We would really like to centralize all the mail accounts on one machine, but unfortunately the administration has not approved it. However, if we could create the "illusion" that they are on the same machine that would be acceptable. If we have a 3rd machine, lets say called mail.example.com, could we install dovecot on it (using LDAP authorization), such that when a user connects to this machine, it then connects to one of the 2 other machines to retrieve their mail (instead of reading from /var/spool/mail). So in essence, the mail backend is just another POP or IMAP server. In the ldap directory we could have a field that says the mail location is: "imap://machine1.example.com" for some users and "imap://machine2.example.com" (Of course it dosent have to look like that, but I hope you get my idea), then dovecot would make a connection to those servers and read the user's mail. Does such a backend even exist? Can dovecot's plugin support register a mail_location backend (such as mbox:/blah/blah, ...)? Harrison Metzger

2 1

[Dovecot] deliver and namespaces in 1.1
by Nikolay Shopik 19 Nov '07

19 Nov '07

> Namespaces are supported everywhere now. Does this means what we can finally use "deliver" to deliver messages in namespaces?

4 5

[Dovecot] MANAGESIEVE patch v9 for dovecot 1.0.7
by Stephan Bosch 19 Nov '07

19 Nov '07

Hello Dovecot users, Since v8, I have updated the MANAGESIEVE patch to fix various issues reported by users. Although I planned to upgrade the next release after v8 to Dovecot v1.1, I release these changes for Dovecot 1.0 first. If no further major issues are found, I can upgrade to Dovecot 1.1. Change Log V9 ------------- + Definitively fixed the segfault mentioned in V8. It proved to be very time-constrained and thus hard to reproduce. The error turned out to be related to the input handling of the login daemon during authentication. + Checked for changes in the imap daemon that weren't propagated to the managesieve implementation due to code duplication. + Fixed a bug in the autodetection of the sieve storage location. + Fixed bug in the sieve storage that failed to refresh the symlink if the storage was moved. + Improved error handing in the sieve-storage implementation in various places. + Fixed the situation in which the active script link is located in the sieve storage. + Added managesieve configuration to dovecot-example.conf and made the example in this file more concise. This patch still includes (yet another) instance of the CMU Sieve source, as explained in one of my previous e-mails (http://dovecot.org/list/dovecot/2006-July/015016.html) The latest patch can be downloaded at: http://sinas.rename-it.nl/~sirius/ The README.managesieve is located in the root of the dovecot source tree after applying the patch. The dovecot-example.conf file is updated with a configuration example for managesieve. Have fun testing the patch. Don't hesitate to notify me when there are problems. Regards, -- Stephan Bosch stephan(a)rename-it.nl IRC: Freenode, #dovecot, S[r]us

1 0

[Dovecot] Search inside body of message
by Nikolay Shopik 18 Nov '07

18 Nov '07

Hi list, I was wondering about server-side search inside entire message, is dovecot support only english encoding now? I'm using Thunderbird but it can't find anything if I'm using language other than English.

3 9

[Dovecot] unlink_directory(....Trash.NewFolder) failed: Directory not empty
by Adam McDougall 18 Nov '07

18 Nov '07

I noticed the following this week. No idea how long its been happening because I have not tested deleting folders hardly at all. Steps to reproduce: 1. Create a folder "NewFolder", click on it in thunderbird so an index dir is created: ls -ldi Maildir/.NewFolder/ 1504062 drwx------ 5 mcdouga9 egrstaff 4096 Nov 16 19:44 Maildir/.NewFolder/ ls -ldi Maildir/dovecot/boomhauer/private/indexes/.NewFolder/ 4198832 drwx------ 2 mcdouga9 egrstaff 4096 Nov 16 19:44 Maildir/dovecot/boomhauer/private/indexes/.NewFolder/ ls -li Maildir/dovecot/boomhauer/private/indexes/.NewFolder/ total 0 4198833 -rw------- 1 mcdouga9 egrstaff 128 Nov 16 19:44 dovecot.index.log 2. Delete "NewFolder" which makes Thunderbird move both the folder under Trash, and seems to "rename" the .NewFolder index dir to .Trash.NewFolder (same inode #): ls -ldi Maildir/.Trash.NewFolder/ 1504062 drwx------ 5 mcdouga9 egrstaff 4096 Nov 16 19:44 Maildir/.Trash.NewFolder/ ls -ldi Maildir/dovecot/boomhauer/private/indexes/.Trash.NewFolder/ 4198832 drwx------ 2 mcdouga9 egrstaff 4096 Nov 16 19:44 Maildir/dovecot/boomhauer/private/indexes/.Trash.NewFolder/ ls -li Maildir/dovecot/boomhauer/private/indexes/.Trash.NewFolder/ total 4 4198833 -rw------- 1 mcdouga9 egrstaff 128 Nov 16 19:44 dovecot.index.log 3. Right click on Trash folder in Thunderbird, choose "Empty Trash": Nov 16 19:47:25 boomhauer dovecot: IMAP(mcdouga9): unlink_directory(/home/mcdouga9/Maildir/dovecot/boomhauer/private/indexes/.Trash.NewFolder) failed: Directory not empty ls -ldi Maildir/.Trash.NewFolder/ 1504062 drwx------ 5 mcdouga9 egrstaff 4096 Nov 16 19:44 Maildir/.Trash.NewFolder/ ls -ldi Maildir/dovecot/boomhauer/private/indexes/.Trash.NewFolder/ 4198832 drwx------ 2 mcdouga9 egrstaff 4096 Nov 16 19:47 Maildir/dovecot/boomhauer/private/indexes/.Trash.NewFolder/ ls -li Maildir/dovecot/boomhauer/private/indexes/.Trash.NewFolder/ (sometimes has a .nfs.blahblah file, I think I failed to catch it quick enough this time) The folder dissapears from Thunderbird's view but stays on disk, and will reappear in Thunderbird if you collapse and re-expand the folder tree, or close and open thunderbird. Repeatable after more "Empty Trash" or directly trying to delete the folder in thunderbird from in Trash. # 1.1.beta8: /usr/local/etc/dovecot.conf ssl_cert_file: /usr/local/etc/apache2/ssl/mail.egr.msu.edu.pem ssl_key_file: /usr/local/etc/apache2/ssl/mail.egr.msu.edu.pem login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login verbose_proctitle: yes first_valid_uid: 1000 first_valid_gid: 1000 mail_extra_groups: postlocal mail_location: maildir:%h/Maildir:CONTROL=%h/Maildir/dovecot/private/control:INDEX=%h/Maildir/dovecot/%q/private/indexes mmap_disable: yes dotlock_use_excl: yes mail_nfs_storage: yes mail_nfs_index: yes mail_plugins: acl fts fts_squat mail_log_max_lines_per_sec: 0 imap_client_workarounds: delay-newmail netscape-eoh tb-extra-mailbox-sep namespace: type: private separator: / inbox: yes list: yes subscriptions: yes namespace: type: private separator: / prefix: mail/ hidden: yes subscriptions: yes namespace: type: private separator: / prefix: Mail/ hidden: yes subscriptions: yes namespace: type: public separator: / prefix: #shared/decs/ location: maildir:/egr/mail/shared/decs:CONTROL=%h/Maildir/dovecot/public/control/decs:INDEX=%h/Maildir/dovecot/%q/public/indexes/decs list: yes subscriptions: yes auth default: passdb: driver: pam userdb: driver: passwd args: system_user= socket: type: listen client: path: /var/spool/postfix/private/auth mode: 384 user: postfix group: postfix plugin: acl: vfile:/usr/local/etc/dovecot-acls:cache_secs=10 fts: squat

2 2

[Dovecot] umask not applied
by Thomas 18 Nov '07

18 Nov '07

Hi all! I am trying to get dovecot-1.0.rc15 to create files with mode 660 in my virtual mailboxes. I set umask=0007 in dovecot.conf, restarted dovecot (/etc/init.d/dovecot restart) and send me an e-mail but it is still 600. "dovecot -n" says "umask: 7" How come the zeros are not displayed? Is this by intention or is something wrong? Here is the complete output: # /etc/dovecot/dovecot.conf log_timestamp: %Y-%m-%d %H:%M:%S login_dir: /var/run/dovecot/login login_executable: /usr/lib/dovecot/imap-login mail_extra_groups: mail mail_location: maildir:/home/vmail/%d/%n umask: 7 auth default: mechanisms: plain login passdb: driver: sql args: /etc/dovecot/dovecot-sql.conf userdb: driver: static args: uid=5000 gid=5000 home=/home/vmail/%d/%n allow_all_users=yes socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postfix master: path: /var/run/dovecot/auth-master mode: 384 user: vmail I searched the wiki and googled but couldn't find anything helpful. Do you have any ideas? ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.

4 4

[Dovecot] SIGSEGV (Segmentation fault) with antispam plugin
by Boky 18 Nov '07

18 Nov '07

Hi all. I am trying to get dovecot (1.0.rc17) running with antispam plugin. I have created another backend, which executes "sa-learn" from spamassassin's arsenal. The backend is modeled after sendmail backend. But I can't seem to get the plugin working wit either my backend or signature-log. This is my ".config": DOVECOT=/usr/src/dovecot-1.0.7 DOVECOT_VERSION=1.0 BACKEND=spamassassin-exec INSTALLDIR=/usr/lib/dovecot/modules/imap/ PLUGINNAME=antispam This is how I make the plugin: make && make install && chmod 644 /usr/lib/dovecot/modules/imap/lib90_antispam_plugin.so Since Ubuntu server does not come with dovecot sources, I have downloaded dovecot-1.0.7 and put it in /usr/src. As far as I understand, only the headers are needed, so this should not be the cause of the crash (if the headers hadn't changed). When invoked from dovecot process, the "imap" dies with signal 11. When running: strace /usr/sbin/dovecot --exec-mail imap This is how it ends: etdents64(8, /* 14 entries */, 4096) = 592 getdents64(8, /* 0 entries */, 4096) = 0 open("/usr/lib/dovecot/modules/imap/lib90_antispam_plugin.so", O_RDONLY) = 9 read(9, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\340\21"..., 512) = 512 fstat64(9, {st_mode=S_IFREG|0644, st_size=19252, ...}) = 0 mmap2(NULL, 16148, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 9, 0) = 0xb7f29000 mmap2(0xb7f2c000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 9, 0x2) = 0xb7f2c000 close(9) = 0 close(8) = 0 stat64("/var/mail/mailboxes/boky/Maildir/subscriptions", 0xbfc17d2c) = -1 EACCES (Permission denied) lstat64("/var/mail/mailboxes/boky/Maildir/cur", 0xbfc17c3c) = -1 EACCES (Permission denied) write(2, "imap(boky): ", 12imap(boky): ) = 12 write(2, "Error: ", 7Error: ) = 7 write(2, "lstat(/var/mail/mailboxes/boky/M"..., 69lstat(/var/mail/mailboxes/boky/Maildir/cur) failed: Permission denied) = 69 write(2, "\n", 1 ) = 1 open("/etc/localtime", O_RDONLY) = 8 fstat64(8, {st_mode=S_IFREG|0644, st_size=696, ...}) = 0 fstat64(8, {st_mode=S_IFREG|0644, st_size=696, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7de2000 read(8, "TZif\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\6\0\0\0\6\0"..., 4096) = 696 close(8) = 0 munmap(0xb7de2000, 4096) = 0 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=696, ...}) = 0 --- SIGSEGV (Segmentation fault) @ 0 (0) --- +++ killed by SIGSEGV +++ Process 4707 detached antispam plugin was downloaded from: http://git.sipsolutions.net/?p=dovecot-antispam.git;a=snapshot;h=HEAD;sf=tgz Please advise me n next steps; I'm just a beginner with debugging C/C++. Thank you all. Boky

3 2

[Dovecot] crash with fts_squat on an identified email
by Adam McDougall 18 Nov '07

18 Nov '07

I have some folders with a good amount of spam. While demonstrating full text search to a user, we found a folder that would crash dovecot while doing a fts. While splitting up the mailbox to narrow it down, I found there are a good number of messages in that folder that all made it crash in the same manner. I figured I'd narrow down a single one, get that fixed, then retest the rest. I also added to my todo list to try a full text search on as much email as I can find, to try to weed out any other problems (and see if fts is realistic to use in production). Nov 17 09:03:26 boomhauer dovecot: IMAP(mcdouga9): file message-decoder.c: line 201 (translation_buf_decode): assertion failed: (*size >= skip) Nov 17 09:03:26 boomhauer dovecot: child 14668 (imap) killed with signal 6 Let me know if this is not reproducable with the attached tgz containing a maildir with one message, I could do a local gdb if needed.

2 1

[Dovecot] mailbox size limit
by Bazy 18 Nov '07

18 Nov '07

Hi guys, I was wondering... my Inbox is 1.3GB large, my mailbox size limit is 6GB (6442450941). Other users have 500+ MB in their Inbox. I have about 27GB of mail. I'm running dovecot 1.0.7 now under Fedora 7 with postfix. I have no trouble at all, everything is working perfect. I use ext3 as my file system, and I store mail in maildir format. Will I have any inode, file system, issues? Is anyone storing more then 27GB in maildir format on only one ext3 partition? Here is my dovecot -n output: # 1.0.7: /etc/dovecot.conf log_path: /var/log/dovecot.log log_timestamp: %Y-%m-%d %H:%M:%S protocols: imap imaps pop3 ssl_cert_file: /etc/postfix/smtpd.cert ssl_key_file: /etc/postfix/smtpd.key login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(pop3): /usr/libexec/dovecot/pop3-login login_max_processes_count: 512 mail_location: maildir:/home/vmail/%d/%n/ mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(pop3): /usr/libexec/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): quota mail_plugin_dir(default): /usr/lib/dovecot/imap mail_plugin_dir(imap): /usr/lib/dovecot/imap mail_plugin_dir(pop3): /usr/lib/dovecot/pop3 imap_client_workarounds(default): outlook-idle delay-newmail imap_client_workarounds(imap): outlook-idle delay-newmail imap_client_workarounds(pop3): outlook-idle pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh auth default: mechanisms: plain login passdb: driver: sql args: /etc/dovecot-sql.conf userdb: driver: sql args: /etc/dovecot-userdb-sql.conf socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postfix plugin: quota: maildir:storage=102400 [root:pts/0][~]# cat /etc/dovecot-sql.conf driver = mysql connect = host=/var/lib/mysql/mysql.sock user=mail_admin password=XXXXXXXX dbname=mail default_pass_scheme = CRYPT password_query = SELECT password FROM users WHERE CONCAT(SUBSTRING_INDEX(email,'@',1)) = '%n' AND CONCAT(SUBSTRING_INDEX(email,'@',-1)) = '%d'; [root:pts/0][~]# cat /etc/dovecot-userdb-sql.conf driver = mysql connect = host=/var/lib/mysql/mysql.sock user=mail_admin password=XXXXXXXX dbname=mail user_query = SELECT CONCAT(('/home/vmail/'), SUBSTRING_INDEX(email,'@',-1),'/',SUBSTRING_INDEX(email,'@',1),'/') as home, 5000 as uid, 5000 as gid, CONCAT('maildir:storage=', floor(quota/1024)) as quota FROM users WHERE email = '%u';

5 9