January 2009 - dovecot

[Dovecot] Maildir loses keywords in index beyond 26
by George Greer 14 Jan '09

14 Jan '09

(NOTE: I am not on the mailing list so please Cc: me.) dovecot-1.1.6-2.fc10.i386 (configuration not changed from stock) Fedora 10 AMD Athlon XP ext3 Maildir format doesn't store keywords in the index properly -- it goes up to 26 and then further keywords are dropped. I ran the tests with a simple script that feeds the lines from "commands" to Dovecot one at a time and waits for the "OK" before the next one. The IMAP command file is attached, as well as output of dovecot, idxview, and logview running against both mbox and maildir formats. Test mailbox created with: for i in `seq -f %02g 0 99`; do echo "Test message #$i" | mail -s "Test message #$i" sandbox@localhost; sleep 1; done For the maildir test I used the conversion script: Step 2: su sandbox -c 'perl /usr/share/doc/dovecot-1.1.6/UW-to-Dovecot-Migration/perfect_maildir.pl ~/Maildir < /var/spool/mail/sandbox' # 1.1.6: /etc/dovecot.conf # OS: Linux 2.6.25.6-55.fc9.i686 i686 Fedora release 10 (Cambridge) login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(pop3): /usr/libexec/dovecot/pop3-login mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(pop3): /usr/libexec/dovecot/pop3 mail_plugin_dir(default): /usr/lib/dovecot/imap mail_plugin_dir(imap): /usr/lib/dovecot/imap mail_plugin_dir(pop3): /usr/lib/dovecot/pop3 auth default: passdb: driver: pam userdb: driver: passwd -- George Greer

2 1

[Dovecot] linuxconf migration story
by Thanos Chatziathanassiou 14 Jan '09

14 Jan '09

Hi, just subscribed after successfully (?) migrating from a somewhat old linuxconf/vimap config. dovecot-1.1.7, x86 A few issues: The index files seem to be created with the wrong permissions. I get numerous ``Dec 24 15:36:29 mail dovecot: POP3(user@domain): stat(/var/run/dovecot/imap-index/domain/user/.imap/INBOX) failed: Permission denied (euid=60107 egid=231 missing +x perm: /var/run/dovecot/imap-index/domain)'' errors. the directory /var/run/dovecot/imap-index is drwxrwxrwt 29 dovecot dovecot 4.0k Dec 24 15:57 imap-index but the directories inside it are created as drwx------ 15 60107 popusers 4.0k Dec 24 15:50 domain Questions: - Why is POP3 interested in the imap indices ? - IMAP clients do not complain all that much, as far as I can see, but what happens to POP3 ? - Is there something to configure to get rid of this ? Also, I have written a somewhat sloppy patch to determine the domain from the server's IP. That way it is not necessary to persuade all users to switch to logging in as ``user@domain'' (which I've tried with very limited success). Would this be the correct place to post the patch and discuss ? Best Regards, Thanos Chatziathanassiou

2 4

[Dovecot] Fwd: apply Sieve filter to existing mails
by Neil 14 Jan '09

14 Jan '09

Forgot reply-all... ---------- Forwarded message ---------- Date: Tue, Jan 13, 2009 at 1:39 AM Subject: Re: [Dovecot] apply Sieve filter to existing mails On Fri, Jan 9, 2009 at 11:01 AM, Jakob Grießmann <mail.an.jakob(a)googlemail.com> wrote: > Hello, > > is it possible to apply Sieve filter to existing mails, e.g. for > testing purposes whether my rules work as expected? > You can pipe mails to deliver again, and I would imagine it should run through the sieve script. And if it did, you could probably whip up a shell script to do that to all mails in a maildir... -Neil.

2 1

[Dovecot] unsubscribe dovecot.org
by Leo P CHENG 14 Jan '09

14 Jan '09

thanks

2 1

Re: [Dovecot] Can not Create Maildir using userdb
by Henry Yonathan 14 Jan '09

14 Jan '09

Dear Tim, Thank you for your clue. And I'm now able to do what I've been dreaming of :D Kind Regards, Henry Yonathan "All successful communication is Hypnosis" - Milton H. Erickson ----- Original Message ----- From: Timo Sirainen [mailto:tss@iki.fi] To: Dovecot Mailing List [mailto:dovecot@dovecot.org] Subject: Re: [Dovecot] Can not Create Maildir using userdb > On Tue, 2009-01-13 at 10:27 +0700, Henry Yonathan wrote: > > Thank you for your reply. > > So, you mean that both user_query and password_query should have the > > same amount of parameter and the mail parameter from user_query should > > have the exact parameter as userdb_mail from password_query? > > And also I've to turn on userdb prefetch {} > > password_query also needs to return the password. But other than that, > yes. ---- Belanja buku lewat internet, GRATIS ONGKOS KIRIM ke seluruh Indonesia. Kunjungi website http://www.bookoopedia.com. Hipokuku.com - Cara Baru Isi Pulsa Lewat Internet. Kunjungi website http://www.hipokuku.com untuk informasi selengkapnya.

1 0

[Dovecot] Apple patch 10
by Mike Abbott 14 Jan '09

14 Jan '09

Patch #10 allows the pop3 and imap mail processes to handle multiple clients. We know this weakens the security model but it greatly increases scalability especially when clients are idle. Here's how it works. When there are no mail processes, or none serving fewer than mail_max_connections clients, the master creates a new mail process pretty much like it does now, but with some new environment variables (PERSISTENT_MAIL_PROCESS and CONNECTION_ID, and ADVISE_SET[GU]ID instead of RESTRICT_SET[GU]ID) and a unix-domain socket connection back to the master. For future clients the master sends the client socket and a full dump of the environment via that connection. Persistent mail processes multiplex I/O for their connected clients and switch the whole environment and the effective user ID, group ID, and supplemental groups each time. The result is that most of dovecot's code and assumptions remain unchanged: the mail process executes imap/pop commands with the permissions of the user and with nearly the same environment so getenv() still works as expected. Some assumptions do change, for instance global state variables like last_partial and quota_set are forbidden, and must be stored per-client or per-user. Also it's no longer OK just to exit or panic on error, for that disconnects multiple users not just one. Some key entry points in the patch are create_mail_process() in mail- process.c and io_env_switch() in ioloop.c. Notes about this patch: - The base for this patch is dovecot-1.1.7 + Apple patch 9, not because this patch needs Open Directory but simply because it adds one line to the Apple attributions in COPYING. - This patch depends on Apple patch 7 (hash_table_create/destroy). - This patch introduces the following new config options for the pop3 and imap protocols: - mail_process_per_connection = yes - mail_max_connections = 20 The default value of mail_process_per_connection (yes) preserves the current secure model. Changing it to no allows multiplexing, with up to mail_max_connections simultaneous clients per process. See dovecot- example.conf for more info. - It restricts the mail_executable config option to minimize unpleasant surprises. When mail_process_per_connection = no the communication protocol between the master process and the mail processes changes as described above. A naive third-party mail_executable might break. See dovecot-example.conf for more info. - Some changes are intentionally awkward in order to minimize code deltas to simplify merges (of new dovecot releases into our source tree). These are so marked. Feel free to clean them up. - Changes tagged with "APPLE" are ours, including the whole new directory mail-common. All the untagged changes, including the entire contents of the new files mail-user.[ch], are copied straight from dovecot-1.2 (alpha3, I believe). We needed dovecot-1.2's support for multiple users in order to handle quotas correctly, and we needed it in 1.1. Specifically, we copied these changes from dovecot-1.2 (and a couple others too I think): 8082 http://hg.dovecot.org/dovecot-1.2/rev/db66611fd195 Added struct mail_user and fixed the code to support multiple users per process. 8084 http://hg.dovecot.org/dovecot-1.2/rev/f12f8c1da0bf Forgot to add mail-user.* files in previous struct mail_user commit. 8085 http://hg.dovecot.org/dovecot-1.2/rev/bf83aa9c3f4a Removed pool parameter from mail_namespaces_init*(). Use mail_user's pool intead. 8091 http://hg.dovecot.org/dovecot-1.2/rev/ceca59aaae89 quota-fs: compile fix for previous changes. 8096 http://hg.dovecot.org/dovecot-1.2/rev/f35a8a3dc06d Fixed FS quota compiling and Maildir++ quota with multiple users. 8109 http://hg.dovecot.org/dovecot-1.2/rev/e7929190cd32 fts-solr: Fixed compiling with recent struct mail_user changes. 8137 http://hg.dovecot.org/dovecot-1.2/rev/b2a258213ee0 Created mail_user_[try_]home_expand(). Used them for expanding mail directories. 8294 http://hg.dovecot.org/dovecot-1.2/rev/8aa69e3d27ef Trash plugin: Assign storage to all mailboxes at startup so errors are caught immediately. Also previous optimization change broke trash plugin when using multiple mail_users. This change fixes it to work again. If a change in the patch is not marked somehow with APPLE then it comes from dovecot-1.2. - Some other data structures also needed to be made per-client/user, for instance last_partial from imap-fetch-body.c, quota_set from quota- plugin.c, and process_primary_gid etc. from restrict-access.c. - Sending the master process SIGINFO (on platforms that support that signal) makes it print its view of all the clients connected to all the mail processes. Sending SIGINFO to an individual mail process makes it print its view. These are not necessarily always the same. For instance, on a config reload (SIGHUP), the master disconnects from the mail processes but leaves them running. The output is admittedly kinda geeky. - The logging tag for a persistent mail process is just "*" instead of the user name, since a process can serve multiple users. Some individual messages (such as connect/disconnect) identify the user. - The expire plugin uses global state variables which are incompatible with persistent mail processes. The patch detects trouble but does not fix it because we don't need it. We leave that as an exercise for you :). - The dict client interface is blocking which is unfortunate for persistent mail processes. The patch does not address this, since the delay is probably small and we don't use it anyway. - Some of the accounting is gnarly because of the need to honor mail_max_userip_connections in a world where a single user can have any number of active connections on any number of mail processes. Finally, here's the patch.

3 6

[Dovecot] assertion failed in 1.1.7 file mbox-sync.c: line 1305 (mbox_sync_handle_eof_updates)
by Diego Liziero 13 Jan '09

13 Jan '09

Dovecot 1.1.7 is running so smoothly that I gave up checking its log files daily. :) I've just had a look, and among the usual "IMAP(username): FETCH for mailbox Sent UID xx got too little data: xx vs xx" messages (that means that unfortunately sometimes some messages are still written truncated) I saw this assertion failure: file mbox-sync.c: line 1305 (mbox_sync_handle_eof_updates): assertion failed: (file_size >= sync_ctx->expunged_space + trailer_size) Regards, Diego. ---- (gdb) bt full #0 0x008a9402 in __kernel_vsyscall () No symbol table info available. #1 0x00a2fd20 in raise () from /lib/libc.so.6 No symbol table info available. #2 0x00a31631 in abort () from /lib/libc.so.6 No symbol table info available. #3 0x080f73c0 in default_fatal_finish (type=LOG_TYPE_PANIC, status=0) at failures.c:150 backtrace = 0x97df358 "/usr/libexec/dovecot/imap [0x80f739e] -> /usr/libexec/dovecot/imap [0x80f7c5f] -> /usr/libexec/dovecot/imap(i_fatal+0) [0x80f7518] -> /usr/libexec/dovecot/imap [0x80911de] -> /usr/libexec/dovecot/imap"... #4 0x080f7c5f in i_internal_fatal_handler (type=LOG_TYPE_PANIC, status=0, fmt=0x811771c "file %s: line %d (%s): assertion failed: (%s)", args=0xbfbabf14 "\020w\021\b\031\005") at failures.c:430 No locals. #5 0x080f7518 in i_panic (format=0x811771c "file %s: line %d (%s): assertion failed: (%s)") at failures.c:197 args = 0xbfbabf14 "\020w\021\b\031\005" #6 0x080911de in mbox_sync_handle_eof_updates (sync_ctx=0xbfbac0f4, mail_ctx=0xbfbac008) at mbox-sync.c:1305 st = (const struct stat *) 0x97f3420 file_size = 242 offset = 242 padding = 684287899293748052 trailer_size = 0 __PRETTY_FUNCTION__ = "mbox_sync_handle_eof_updates" #7 0x08091d77 in mbox_sync_do (sync_ctx=0xbfbac0f4, flags=0) at mbox-sync.c:1547 mail_ctx = {sync_ctx = 0xbfbac0f4, mail = {uid = 0, idx_seq = 1, keywords = {arr = {buffer = 0x0, element_size = 0}, v = 0x0, v_modifiable = 0x0}, flags = 12 '\f', uid_broken = 0, expunged = 1, pseudo = 0, from_offset = 0, body_size = 0, offset = 0, space = 243}, seq = 1, hdr_offset = 66, body_offset = 151, header_first_change = 4294967295, header_last_change = 0, header = 0x9800878, hdr_md5_sum = "�\035\214�\217\000�\004�\200\t\230��B~", content_length = 18446744073709551615, hdr_pos = {69, 12, 4294967295, 82, 32}, parsed_uid = 2, last_uid_updated_value = 0, last_uid_value_start_pos = 2, have_eoh = 1, need_rewrite = 0, seen_imapbase = 1, updated = 0, recent = 0, dirty = 0, imapbase_rewrite = 0, imapbase_updated = 0} st = (const struct stat *) 0x97f3420 i = 0 ret = 1 partial = 1 #8 0x0809277e in mbox_sync_int (mbox=0x97f0c28, flags=0) at mbox-sync.c:1806 index_sync_ctx = (struct mail_index_sync_ctx *) 0x9808b38 sync_view = (struct mail_index_view *) 0x97f1648 trans = (struct mail_index_transaction *) 0x97f3168 sync_ctx = {mbox = 0x97f0c28, flags = 0, input = 0x97f34e8, file_input = 0x97f3400, write_fd = 8, orig_mtime = 1227954336, orig_atime = 1227954336, orig_size = 242, last_stat = {st_dev = 37637, __pad1 = 0, __st_ino = 4751377, st_mode = 33152, st_nlink = 1, st_uid = 631, st_gid = 508, st_rdev = 0, __pad2 = 0, st_size = 242, st_blksize = 4096, st_blocks = 16, st_atim = { tv_sec = 1227960259, tv_nsec = 0}, st_mtim = {tv_sec = 1227954336, tv_nsec = 0}, st_ctim = {tv_sec = 1227954336, tv_nsec = 0}, st_ino = 4751377}, index_sync_ctx = 0x9808b38, sync_view = 0x97f1648, t = 0x97f3168, reset_hdr = {major_version = 0 '\0', minor_version = 0 '\0', base_header_size = 0, header_size = 0, record_size = 0, compat_flags = 0 '\0', unused = "\000\000", indexid = 0, flags = 0, uid_validity = 0, next_uid = 0, messages_count = 0, unused_old_recent_messages_count = 0, seen_messages_count = 0, deleted_messages_count = 0, first_recent_uid = 0, first_unseen_uid_lowwater = 0, first_deleted_uid_lowwater = 0, log_file_seq = 0, log_file_tail_offset = 0, log_file_head_offset = 0, sync_size = 0, sync_stamp = 0, day_stamp = 0, day_first_uid = {0, 0, 0, 0, 0, 0, 0, 0}}, hdr = 0x97f1300, header = 0x9800878, from_line = 0x9800858, base_uid_validity = 1, base_uid_last = 2, base_uid_last_offset = 0, mails = {arr = {buffer = 0x97f8c40, element_size = 52}, v = 0x97f8c40, v_modifiable = 0x97f8c40}, sync_changes = 0x97f8c60, mail_keyword_pool = 0x9800168, saved_keywords_pool = 0x97f50d8, prev_msg_uid = 2, next_uid = 3, idx_next_uid = 3, seq = 1, idx_seq = 2, need_space_seq = 0, last_nonrecent_uid = 0, expunged_space = 243, space_diff = 0, dest_first_mail = 1, first_mail_crlf_expunged = 0, delay_writes = 1, renumber_uids = 0, moved_offsets = 0, ext_modified = 0, index_reset = 0, errors = 0} sync_flags = MAIL_INDEX_SYNC_FLAG_DROP_RECENT lock_id = 23 ret = 1 changed = 1 delay_writes = true __PRETTY_FUNCTION__ = "mbox_sync_int" #9 0x08092a0a in mbox_sync (mbox=0x97f0c28, flags=0) at mbox-sync.c:1873 ret = 1 #10 0x08092ad8 in mbox_storage_sync_init (box=0x97f0c28, flags=MAILBOX_SYNC_FLAG_FIX_INCONSISTENT) at mbox-sync.c:1897 mbox = (struct mbox_mailbox *) 0x97f0c28 mbox_sync_flags = 0 ret = 0 #11 0x080b70a4 in mailbox_sync_init (box=0x97f0c28, flags=MAILBOX_SYNC_FLAG_FIX_INCONSISTENT) at mail-storage.c:516 No locals. #12 0x080b711b in mailbox_sync (box=0x97f0c28, flags=MAILBOX_SYNC_FLAG_FIX_INCONSISTENT, status_items=0, status_r=0x0) at mail-storage.c:545 ctx = (struct mailbox_sync_context *) 0x0 #13 0x0805b5b7 in cmd_close (cmd=0x97e95c8) at cmd-close.c:21 client = (struct client *) 0x97e9368 mailbox = (struct mailbox *) 0x97f0c28 storage = (struct mail_storage *) 0x97e8c80 #14 0x080625c4 in client_command_input (cmd=0x97e95c8) at client.c:580 client = (struct client *) 0x97e9368 command = (struct command *) 0x7 __PRETTY_FUNCTION__ = "client_command_input" #15 0x080627f9 in client_command_input (cmd=0x97e95c8) at client.c:629 client = (struct client *) 0x97e9368 command = (struct command *) 0x97e856c __PRETTY_FUNCTION__ = "client_command_input" #16 0x080628f7 in client_handle_next_command (client=0x97e9368, remove_io_r=0xbfbac4a5) at client.c:670 size = 11 #17 0x08062933 in client_handle_input (client=0x97e9368) at client.c:680 _data_stack_cur_id = 3 ret = 126 remove_io = false handled_commands = false #18 0x08062ac1 in client_input (client=0x97e9368) at client.c:725 cmd = (struct client_command_context *) 0xad8408 output = (struct ostream *) 0x97e94f4 bytes = 11 __PRETTY_FUNCTION__ = "client_input" #19 0x08101ecd in io_loop_handler_run (ioloop=0x97e79b0) at ioloop-epoll.c:203 ctx = (struct ioloop_handler_context *) 0x97e7aa8 events = (struct epoll_event *) 0x97e7ae8 event = (const struct epoll_event *) 0x97e7ae8 list = (struct io_list *) 0x97e9570 io = (struct io_file *) 0x9800550 tv = {tv_sec = 0, tv_usec = 354274} events_count = 4 t_id = 2 msecs = 355 ret = 1 i = 0 j = 0 call = true #20 0x08101164 in io_loop_run (ioloop=0x97e79b0) at ioloop.c:320 No locals. #21 0x0806df19 in main (argc=1, argv=0xbfbac604, envp=0xbfbac60c) at main.c:293 No locals. (gdb)

2 2

Re: [Dovecot] Can not Create Maildir using userdb
by Henry Yonathan 13 Jan '09

13 Jan '09

Thank you for your reply. So, you mean that both user_query and password_query should have the same amount of parameter and the mail parameter from user_query should have the exact parameter as userdb_mail from password_query? And also I've to turn on userdb prefetch {} Kind Regards, Henry From: Timo Sirainen [mailto:tss@iki.fi] To: Dovecot Mailing List [mailto:dovecot@dovecot.org] Subject: Re: [Dovecot] Can not Create Maildir using userdb sql > On Jan 12, 2009, at 8:38 PM, Henry Yonathan wrote: > > > password_query = SELECT email AS user, password, CONCAT('/home/vmail/ > > %d/%n/', maildir) AS userdb_home, CONCAT('maildir:/home/vmail/%d/ > > %n/', maildir) AS userdb_mail, 5000 AS userdb_uid, 5000 AS > > userdb_gid FROM view_users WHERE email = '%u'; > > Here you're trying to use prefetch userdb by returning userdb_* > fields. But they're different than here: > > > user_query = SELECT '/home/vmail/%d/%n' AS home, '/home/vmail/%d/%n/ > > Maildir' AS mail, 5000 AS uid, 5000 AS gid, > > CONCAT('maildir:storage=', ROUND(quota/1024)) AS quota FROM > > view_users WHERE email = '%u'; > > They should contain the exact same values. Here userdb_mail isn't > prefixed with "maildir:", which is your main problem. Also here you're > specifying "quota" field but in password_query you're not specifying > userdb_quota at all. > > > passdb: > > driver: sql > > args: /etc/dovecot-sql.conf > > userdb: > > driver: sql > > args: /etc/dovecot-sql.conf > > And finally you're not currently even using prefetch userdb so Dovecot > ignores all the userdb_* fields you're returning in password_query. > For enabling prefetch see http://wiki.dovecot.org/UserDatabase/Prefetch ---- Belanja buku lewat internet, GRATIS ONGKOS KIRIM ke seluruh Indonesia. Kunjungi website http://www.bookoopedia.com. Hipokuku.com - Cara Baru Isi Pulsa Lewat Internet. Kunjungi website http://www.hipokuku.com untuk informasi selengkapnya.

2 1

[Dovecot] Dovecot suggestions for new installation
by Michael Durket 13 Jan '09

13 Jan '09

I've been testing Dovecot for a while now and am preparing to replace our outdated IMAP system with it. The current system serves about 300 email users who run a mixture of clients (Eudora, Apple Mail, Thunderbird and Outlook). Can any of the many experts on this list answer a few questions: 1) What's the latest stable release of Dovecot I should be running on this new system (Dovecot will be running on a RedHat Linux AS 4 64 bit system with plenty of memory and CPU). I've been running 1.13 on my test system but I see that 1.18 just came out (although I don't know if I should switch to it immediately. 2) What configuration file options (if any) should I turn on to best handle the range of email clients listed above? 3) Are there any "sizing" options I need to worry about setting for the number of email users I have (I suspect that 300 users is considered a rather small mail system but I want to minimize its resource usage as much as possible while still giving decent performance from Dovecot). And, just out of curiosity, I'm looking into the idea of switching users from mbox format over to Maildir format on this system (it will use all locally- connected disks and have no NFS mounts). Does the Dovecot automatic migration program to migrate from mbox to Maildir work well or are there "gotchas"? Thanks.

2 1

[Dovecot] [checkpassword] I can't get a password from fd3
by Durk Strooisma 13 Jan '09

13 Jan '09

Hi all, I'm trying to implement checkpassword authentication with a simple bash script. In some way I can't get the password from file descriptor 3. The start of script looks like this: #!/bin/bash read -d '\0' -r -u 3 USERNAME="${REPLY}" read -d '\0' -r -u 3 PASSWORD="${REPLY}" if [ -z "${USERNAME}" ] || [ -z "${PASSWORD}" ] then exit 111 fi It always exits with 111, because PASSWORD is empty. In whatever way I try to retrieve data from fd3, I never happen to get more data than the username. Is this a known problem? Am I doing something silly? Replies are much appreciated! Durk FYI: GSSAPI auth works perfect and PAM worked before switching to checkpassword too. OS version : Debian 5.0 lenny amd64 Dovecot version : 1.0.15 (Debian version 1.0.15-2.3) Parts of /etc/dovecot/dovecot.conf: protocols: imap ssl_cert_file: /etc/ssl/certs/cert.pem ssl_key_file: /etc/ssl/private/cert.key ssl_cipher_list: ALL:!LOW:!SSLv2 login_dir: /var/run/dovecot/login login_executable: /usr/lib/dovecot/imap-login first_valid_uid: 998 last_valid_uid: 998 first_valid_gid: 998 last_valid_gid: 998 mail_privileged_group: mail mail_location: maildir:/srv/vmail/%Ld/%n lock_method: flock maildir_copy_with_hardlinks: yes auth default: mechanisms: gssapi plain krb5_keytab: /etc/dovecot/krb5.keytab verbose: yes debug: yes passdb: driver: checkpassword args: /usr/bin/dovecot-checkpassword userdb: driver: ldap args: /etc/dovecot/dovecot-ldap.conf

4 8