- dovecot - dovecot.org

Rebuild UIDs for mdbox
by Darac Marjal 28 Nov '16

28 Nov '16

Hello all, I think I'm hitting an issue with overly large UIDs in one of my mailboxes. I initially thought it was a problem with claws-mail hitting an infinite loop. Claws-mail /seems/ to work fine, until it gets to this one mailbox, wherein the following is logged: [20:23:04] IMAP> 86 SELECT LocalCron [20:23:04] IMAP< * OK [CLOSED] Previous mailbox closed. [20:23:04] IMAP< * FLAGS (\Answered \Flagged \Deleted \Seen \Draft Junk NonJunk) [20:23:04] IMAP< * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft Junk NonJunk \*)] Flags permitted. [20:23:04] IMAP< * 399 EXISTS [20:23:04] IMAP< * 0 RECENT [20:23:04] IMAP< * OK [UNSEEN 399] First unseen. [20:23:04] IMAP< * OK [UIDVALIDITY 1163720154] UIDs valid [20:23:04] IMAP< * OK [UIDNEXT 3069273506] Predicted next UID [20:23:04] IMAP< * OK [HIGHESTMODSEQ 120748] Highest [20:23:04] IMAP< 86 OK [READ-WRITE] Select completed (0.000 + 0.000 secs). imap-thread.c:1344:imap select run - end 0 imap-thread.c:388:generic_cb imap-thread.c:372:found imap 0x22c8f60 imap-thread.c:1431:imap select - end imap.c:3961:select: exists 399 recent 0 expunge 0 uid_validity 1163720154 can_create_flags 1 imap-thread.c:2018:imap fetch_uid - begin imap-thread.c:372:found imap 0x22c8f60 imap-thread.c:372:found imap 0x22c8f60 [20:23:04] IMAP- [fetching UIDs...] [20:23:04] IMAP> 87 UID FETCH 1:* (UID) [20:23:04] IMAP< [FETCH data - 1024 bytes] [20:23:04] IMAP< [FETCH data - 1024 bytes] [20:23:04] IMAP< [20:23:04] IMAP< [FETCH data - 1024 bytes] [20:23:04] IMAP< [FETCH data - 1024 bytes] [20:23:04] IMAP< FETCH (UID 3069273244) [20:23:04] IMAP< [FETCH data - 1024 bytes] [20:23:04] IMAP< [FETCH data - 1024 bytes] [20:23:04] IMAP< TCH (UID 3069273313) [20:23:04] IMAP< [FETCH data - 1024 bytes] [20:23:04] IMAP< [FETCH data - 1024 bytes] [20:23:04] IMAP< TCH (UID 3069273382) [20:23:04] IMAP< [FETCH data - 1024 bytes] [20:23:04] IMAP< [FETCH data - 1024 bytes] [20:23:04] IMAP< TCH (UID 3069273451) [20:23:04] IMAP< [FETCH data - 1024 bytes] [20:23:04] IMAP< [FETCH data - 551 bytes] imap-thread.c:2008:imap fetch_uid run - end 0 imap-thread.c:388:generic_cb imap-thread.c:372:found imap 0x22c8f60 imap-thread.c:2029:imap fetch_uid - end imap.c:542:unlocking session 0x22e4c00 imap.c:4663:get_num_list: got 399 msgs imap.c:4673:removing old messages from /home/darac/.claws-mail/imapcache/mail.darac.org.uk/darac(a)darac.org.uk/LocalCron imap.c:4677:get_num_list - ok - 399 folder.c:2265:Removed message 2147483647 from cache. That last line (Removed message 2147483647 from cache) is repeated ad infinitum. Now, I happened to spot that that number is 2^31 - 1 so started looking to see if claws-mail was handling UIDs incorrectly. That lead me to http://wiki.dovecot.org/Clients/NegativeUIDs. Firstly, that page states that UIDs should be 32-bit integers. If so, why do there appear to be UIDs greater than 3 billion in the mailbox? Secondly, that page gives instructions on how to fix the problem with mbox and maildir backends. However, my mail is stored in mdboxes. Is there a way to re-generate UIDs for mdbox mailboxes? Thanks in advance!

2 2

CPU for Dovecot
by Miloslav Hůla 28 Nov '16

28 Nov '16

Hi, we are planning to change hardware for our standalone Dovecot instance handling ~5800 IMAP users with 1TB mailboxes on local RAID. Is there some recommendation about CPU? We can choose from: - Intel Xeon E5-2620v4 - 2,1GHz@8,0GT 20MB cache, 8core, HT, 85W, LGA2011 - Intel Xeon E5-2623v4 - 2,6GHz@8,0GT 10MB cache, 4core, HT, 85W, LGA2011 The difference is about more cores vs. hi frequency. Thank you, Miloslav

4 4

Dovecot proxy
by Gandalf Corvotempesta 26 Nov '16

26 Nov '16

Hi to all I have some *production* pop3/inappropriate server that i would like to move under a proxy Some questions: 1. Keeping the same original hostname on the proxy (in example mail.mydomain.tld) and changing the hostname on the imap server, makes some troubles like MUA redownloading all the messages? Is dovecot (running on the imap server) happy seeing the hostname change? What about maildirs, where the hostname is wrote on the mail file? 2. Dovecot proxy will proxy the whole pop3/imap traffic or only the login auth? I don't want to expose the mailservers to internet, all imap session must be proxied through the proxy. this because I'll use local IPs on each mail server. 3. Performance for the proxy server? The same as the mailserver or higher due to the missing email computation? In example, the proxy doesn't have to access disks or emails data but has only to transmit what the mailserver osd saying 4. Like question 3, any real users for the proxy? I would like to know some info about hardware and userbase (in example: dual quad xeon 5600, 32Gb ram, 10.000 concurrent sessions) In my case I'm planning for about 100 active sessions. Can i use a small EC2 instance?

1 1

Re: (build) crash in dcrypt when "-Bsymbolic-functions" is used on linking
by Aki Tuomi 25 Nov '16

25 Nov '16

On 26.10.2016 13:48, Christian Ehrhardt wrote: > Hi, > I was wondering about a crash when building dovecot 2.2.25 on latest > Ubuntu. > I wondered as I've had the same source building on Debian just fine. > > Some debugging led me to this weird behavior: > Using this gdb command file called autoreportissue in my case: > break dcrypt_initialize > commands > p dcrypt_vfs > p &dcrypt_vfs > watch dcrypt_vfs > c > end > break dcrypt_set_vfs > commands > p dcrypt_vfs > p &dcrypt_vfs > c > end > r > > Running test-crypto on Debian and Ubuntu reported those two behaviours: > gdb -d /root/dovecot-2.2.25/src/ -x autoreportissue ./test-crypto > > Good: > Breakpoint 1, dcrypt_initialize (backend=0x555555587c02 "openssl", > set=0x0, error_r=0x0) at dcrypt.c:15 > 15 if (dcrypt_vfs != NULL) { > $1 = (struct dcrypt_vfs *) 0x0 > $2 = (struct dcrypt_vfs **) 0x555555796370 <dcrypt_vfs> > Hardware watchpoint 3: dcrypt_vfs > Breakpoint 2, dcrypt_set_vfs (vfs=0x7ffff7835020 <dcrypt_openssl_vfs>) > at dcrypt.c:56 > 56 dcrypt_vfs = vfs; > $3 = (struct dcrypt_vfs *) 0x0 > $4 = (struct dcrypt_vfs **) 0x555555796370 <dcrypt_vfs> > Hardware watchpoint 3: dcrypt_vfs > Old value = (struct dcrypt_vfs *) 0x0 > New value = (struct dcrypt_vfs *) 0x7ffff7835020 <dcrypt_openssl_vfs> > dcrypt_set_vfs (vfs=0x7ffff7835020 <dcrypt_openssl_vfs>) at dcrypt.c:57 > 57 } > > Bad: > Breakpoint 1, dcrypt_initialize (backend=0x555555589f02 "openssl", > set=0x0, error_r=0x0) at dcrypt.c:11 > 11 { > $1 = (struct dcrypt_vfs *) 0x0 > $2 = (struct dcrypt_vfs **) 0x555555798370 <dcrypt_vfs> > Hardware watchpoint 3: dcrypt_vfs > Breakpoint 2, dcrypt_set_vfs (vfs=0x7ffff780a020 <dcrypt_openssl_vfs>) > at dcrypt.c:56 > 56 dcrypt_vfs = vfs; > $3 = (struct dcrypt_vfs *) 0x0 > $4 = (struct dcrypt_vfs **) 0x7ffff780a890 <dcrypt_vfs> > Panic: file dcrypt.c: line 34 (dcrypt_initialize): assertion failed: > (dcrypt_vfs != NULL) > Error: Raw backtrace: > /root/dovecot-2.2.25/src/lib-dcrypt/test-crypto(+0x15f7c) > [0x555555569f7c] -> > /root/dovecot-2.2.25/src/lib-dcrypt/test-crypto(default_error_handler+0) > [0x55555556a030] -> > /root/dovecot-2.2.25/src/lib-dcrypt/test-crypto(i_fatal+0) > [0x55555556a2ff] -> > /root/dovecot-2.2.25/src/lib-dcrypt/test-crypto(dcrypt_initialize+0x140) > [0x55555555f030] -> > /root/dovecot-2.2.25/src/lib-dcrypt/test-crypto(main+0x23) > [0x55555556706d] -> > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) > [0x7ffff782d3f1] -> > /root/dovecot-2.2.25/src/lib-dcrypt/test-crypto(_start+0x2a) > [0x55555555edea] > Program received signal SIGABRT, Aborted. > > One can see that in the bad case the pointer of dcrypt_vfs is pointing > to something of the scope of the .libs/libdcrypt_openssl.so file and > not the dcrypt_initialize of test-crypto. > > That made me wonder even more - where would this issue of variable > scope come from. After more debugging I found that the linker flag > "-Bsymbolic-functions" is the reason. This is default on recent > Ubuntu, but not on Debian (?yet?). > > Eventually what happens is that the dcrypt_vfs becomes part of the > .libs/libdcrypt_openssl.so. So the call from there to dcrypt_set_vfs > ends up setting not the expected variable. > I was unable to come up with a reasonable fix since I'm not enough > into your sublib structure. > > For now I assume I'm gonna build the package stripping this flag in > Ubuntu. > But long term I think dovecot should fix it to work with that compiler > flag. > Therefore the report to make you aware. > > > P.S. thanks to the dovecot community for having unit tests that find > this at build time! > > -- > Christian Ehrhardt > Software Engineer, Ubuntu Server > Canonical Ltd Hi! Thank you for reporting this, we'll look into it. Aki Tuomi Dovecot oy

1 1

Could the index of dovecot be stored in database?
by eSX 25 Nov '16

25 Nov '16

Hi dovecot, I have a question about the index of dovecot. I had read the wiki, and saw that I can custom a Input streams to do read/seek/sync...by implementing a istream-*.c file. Thus maybe dovecot can read mail from a http server or any others. But I found that the index of maildir can't extend very well. There is no extern API for developer? For example, I want to put the index to database(maybe, MySQL) or any else, But I found I have no way to implement it, Unless a heavy modification? -- eSX

1 0

Patch for dovecot-antispam-plugin to support IMAP Flags indicating Junk status
by Jonas Wielicki 25 Nov '16

25 Nov '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Dear list, I made a patch which extends dovecot-antispam-plugin with support to take the addition and removal of a specific set of flags as trigger for learning. This allows for very tight integration with Thunderbird and KMail, which both set flags for Junk messages (Thunderbird sets "Junk", KMail sets "JUNK"). With this patch and according configuration, dovecot-antispam-plugin can trigger on the addition (learn as spam) and removal (learn as ham) of these flags. Combined with a sieve script which sets these flags for incoming mail which has been classified as spam, this allows for a tight and intuitive feedback loop between users and the server-side spam classification system. I tested that patch with dovecot 2.2 from Debian stretch and so far it works pretty well. I realise that dovecot-antispam-plugin has not been updated since 2013. It also has not been migrated to Github yet. If it is to be migrated there, I would be happy to submit a pull request there for discussion and merging. Otherwise, someone would have to indicate how to go forward with applying this patch to the official version, if that is desired. The patch is based on current tip and feedback is appreciated. I took some inspiration from dovecot-core/src/plugins/notify/notify-storage.c -- this requires the use of struct mail_private to be able to override the vfunc for mail_update_keywords, which I actually do not like. However, when I tried to make this patch base on the use of the notify plugin (via notify_register), I ran into the issue that the transaction is partially freed before the transaction_commit event from notify is emitted, which is unfortunate and breaks at least the mailtrain backend (which I use). This is in theory fixable, but I wanted to keep the patch as small as possible, as I have no idea whether this will eventually end up upstream or whether I’ll have to maintain that for myself. If upstream is not interested in this patch, please let me know! In that case, I would consider making an own plugin, which fewer backends (possibly only a mailtrain-like one) out of this, but frankly, I would consider that a waste of resources. In the hope that this is useful for others and hoping for feedback, Jonas Wielicki -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEG/EPV+Xzd5wEoQQIwGIDJZdiWIoFAlg4Fd0ACgkQwGIDJZdi WIpM3Q//TWVHDFx5LK14y0jOnxzE7kGthc6hakILnWqYgl7B/wlljD1HLrec//hg 9LEancjURnv3sPU/kEEpl6RRDX5kDMbXXKXMq7/1R7bNdmeMz5NEebAo7ktlCii7 ikk7MPL0F8UViY7IoW7y5NxfqMUldYseejn0GwKiVkEyqlyGLRbP7ijyzMWjpIWp eqOg/b5bNX6ARMC1O/hKhvHw4AhvSO2j73da60Xchu39gvDa2kC+MTU/kX4wb6po kPOEgFxGhpxG0+UMzUEBPLYwY3Q9x+R/PzlpUF6xEjYxgE7leVRVfCHiZLsMUGH1 itozwSzoAyDYV+Rbzk8OEBL6EGBJQHqYmNybULZsJkxhUenNeC4GRf7NcJ0jAisG rSfwr/+kD9+ErKNgtNJzfceLYGNWti44whV6kXxRxqGlIfJsJB7ZMdrj8+wSdn0p slXUoyHMUkRkN92tIPqZF5cb+5Ke+O+5hHE07Wohxw3eNoBSqtQWo81aigjaCraC ivcZX3HswTapyKtpgWMzjoWqp982crr+wtAI+AlEUKoCEqF54EnxM1kB1qoe0HcH nMgMdIkXI4VzPaUNUK206ffEtlnKNIqF3SkOkBX263L0ZJgXyCFRRj6W3z78VDdP He62tN/FPPBtmeVrmCj2ZDE+6gqtMQN52FsN4mpaq4BrCYlYB7g= =3+CU -----END PGP SIGNATURE-----

1 0

Updated my Dovecot certificate for the first time
by Steve Litt 25 Nov '16

25 Nov '16

Hi all, I've used Dovecot since February 2012, but because I kept reinstalling Linux with every major version, I never had a Dovecot self-signed certificate go bad on me before. Til now. I started using rolling release Void Linux about a year ago, and my Dovecot self-signed certificate just expired. The solution I used is contained in these documents: http://wiki2.dovecot.org/SSL/CertificateCreation http://wiki2.dovecot.org/SSL/CertificateClientImporting file:///etc/ssl/dovecot-openssl.cnf http://www.faqforge.com/linux/renew-the-dovecot-ssl-certificate-on-ubuntu-l… I basically moved my old /etc/ssl/certs/dovecot.pem and /etc/ssl/private/dovecot.pem, then edited /etc/ssl/dovecot-openssl.cnf specifically to give myself the common name of 192.168.100.2. I had earlier used my hostname, but that produced a conflict, so I just used the ip address. Then I ran dovecot-mkcert.sh to create the new self-signed cert, and finally, configured Claws-Mail to use /etc/ssl/certs/dovecot.pem as its cert. Obviously, if my Claws-Mail were on a different machine than my Dovecot, I would have had Claws-Mail point to a local copy. Alpine still gives me a bad cert warning, saying I should either fix it or disable checking. I haven't yet found a way to get Alpine to discriminate between a valid self-signed cert and a bad one. Anyway, all's good. SteveT Steve Litt November 2016 featured book: Quit Joblessness: Start Your Own Business http://www.troubleshooters.com/startbiz

7 9

Re: Updated my Dovecot certificate for the first time
by Joseph Tam 24 Nov '16

24 Nov '16

On Thu, 24 Nov 2016, Steve Litt wrote >> add the public part of the cert into your system's trusted CA store. > > Silly question, but how would you do that? You didn't say which OS you're running on (alpine runs on Windows as well), but I'll assume *nix. A previous poster showed you how to do it with a real certificate, and the steps are the same. However, the way I found out without too much fuss was to process trace my alpine process and see where it tied to load a cert $ strace -o trace.out alpine ... quit after connection $ grep -F cert traceout /1: open64("/etc/openssl/cert.pem", O_RDONLY) Err#2 ENOENT /1: stat("/etc/openssl/certs/cbf06781.0", 0xFFBF8E54) Err#2 ENOENT ... Your output will be different of course. The first load is the default pre-loaded root CAs (Thawte, etc.) supplied by OpenSSL, and the second, etc. are chained certificate lookups. You would replace the missing cert with your own self-signed public pem file. e.g. cp mypub.pem /etc/openssl/certs/cbf06781.0 For Windows, I don't know where it fetches it from. Joseph Tam <jtam.home(a)gmail.com>

1 0

Re: Updated my Dovecot certificate for the first time
by Joseph Tam 24 Nov '16

24 Nov '16

> Alpine still gives me a bad cert warning, saying I should either fix it > or disable checking. I haven't yet found a way to get Alpine to > discriminate between a valid self-signed cert and a bad one. Well, it can't discriminate since any certificate (except those in your trusted store) that asserts its own validity is suspect. You can either get alpine to not complain e.g. inbox-path={192.168.100.2:143/user=whatever/tls/novalidate-cert} or add the public part of the cert into your system's trusted CA store. Joseph Tam <jtam.home(a)gmail.com>

2 1

antispam plugin not working correctly
by Felix Zielcke 23 Nov '16

23 Nov '16

Hi, I've configured the antispam plugin like in the wiki to use the sa- learn-pipe.sh shell script. But it's not creating the log file in /tmp. Even if I add a "touch /tmp/x" in sa-learn-pipe.sh it's not created. So how I can check if it works correctly? dovecot version 2.2.26.0 /var/log/debug although contains: Nov 23 18:36:49 daedalus imap [fzielcke(a)z-51.de 80.187.96.201 UID MOVE]: antispam: mail copy: src spam: 1, dst spam: 0, src unsure: 0 Nov 23 18:36:49 daedalus imap [fzielcke(a)z-51.de 80.187.96.201 UID MOVE]: antispam: running mailtrain backend program /usr/bin/sa-learn-pipe.sh Nov 23 18:36:49 daedalus imap [fzielcke(a)z-51.de 80.187.96.201 UID MOVE]: antispam: running mailtrain backend program /usr/bin/sa-learn-pipe.sh Nov 23 18:36:49 daedalus imap [fzielcke(a)z-51.de 80.187.96.201 UID MOVE]: antispam: running mailtrain backend program parameter 1 --ham # doveconf -n |grep antispam antispam_backend = pipe antispam_debug_target = syslog antispam_pipe_program = /usr/bin/sa-learn-pipe.sh antispam_pipe_program_notspam_arg = --ham antispam_pipe_program_spam_arg = --spam antispam_pipe_tmpdir = /tmp antispam_spam_pattern_ignorecase = SPAM antispam_trash_pattern_ignorecase = trash;Deleted * antispam_verbose_debug = 1 mail_plugins = acl zlib quota imap_zlib imap_acl imap_quota antispam

1 0