- dovecot - dovecot.org

Re: mdbox: Broken virtual size for mail UID ...
by Dennis Schridde 29 Jan '15

29 Jan '15

Hello everyone! I originally asked Timo for help in this case, but I understand he is very busy these days, so I am now posting my questions to this mailinglist. On Sunday 28 Dec 2014 10:02:08 Timo Sirainen wrote: > On 28 Dec 2014, at 07:06, Dennis Schridde <devurandom(a)gmx.net> wrote: > > My harddrives crashed, and now I get "Broken virtual size for mail UID …" > > messages in the dovecot logs from some mdbox folders. I assume the logged > > emails are destroyed forever, but I would like to restore as much as > > possible of them and keep them in their corrupted / incomplete state. How > > do I do that? > > > > I guess what I need is some kind of "doveadm fsck" command, that tries its > > best to limit the damage to a minimum: Inform me about the broken emails > > and then ask me whether they should be stored in truncated form, be > > deleted, etc. Is there such a thing? > > http://dovecot.org/tools/mdbox-recover.pl Thank you! >From what I read in the Wiki and how I understand the script, I should proceed like this: #!/bin/bash broken_storage=$PWD/storage recovered_storage=$PWD/storage-recovered cd ${recovered_storage} n=1 for f in ${broken_storage}/m.* ; do mdbox-recover.pl ${f} for f in msg.* ; do mv ${f} m.${n} $((n++)) done done mv ${broken_storage} storage-backup mv ${recovered_storage} storage exit 0 Afterwards dovecot will now automatically regenerate the dovecot.map.index files from the now bare m.* files, and as the dovecot.index files in the actual mailboxes reference the GUID, and the messages in the m.* files have the GUID embedded, nothing will be lost. Is that correct? To create proper assessment of the damage, I need to know a bit more: 1. In the script on line 43, the comment says: # end of metadata block missing, finish the previous mail That seems to be wrong, since idx2!=-1 and state==STATE_BODY on that line, while I understand it should be -1 and STATE_META if we were still reading the metadata block. I guess it should be something like: # still in data block, finish the previous mail 2. Line 47 reads: # truncated / broken data? just keep writing to previous file That is the only error which could be detected in that script, right? 3. A message could also be corrupted by loosing its complete data block from \001\003 to \001\002, which cannot be detected in the script, right? 4. The only case to detect this and find other unexpected oddities would be to compare each msg.* in size and offset with the messages referenced in the dovecot.map.index file, correct? How do I do that, if the dovecot.map.index does not contain the GUID of the message? Do I need to fuzzy match the file_id, offset and size with my current values, or is there a more reliable way to do this? 5. Metadata refers to the RFC822 mail headers? Or are they also included in the body, and the mentioned metadata is dovecot-only metadata? If the latter, do they also contain the size of the body and similar information? Best regards, Dennis

1 0

Dovecot antispam
by David Mehler 29 Jan '15

29 Jan '15

Hello, This is not specifically a dovecot issue, but to it's antispam plugin, that being the case if private responses are deemed so please do so. I've got a Postfix, amavisd-new, dovecot (delivery and sieve), setup with spamassassin. It's for the most part working fine, here's the problem, false positives and false negatives. The sieve moves spamassassin-tagged messages in to the Spam folder, sometimes it misses either a false positive or negative. What I'm wanting to do and this is where the dovecot-antispam plugin comes in is automatically retrain messages, a false positive or negative needs to be moved and reclassified accordingly. I've found a lot of google results indicating dspam, when I used this the results were terrible after three months of usage I gave up on it. I've thought about using spamc with dovecot-antispam to handle retraining or crm114, but not much on getting this going. If anyone has any information on this or any other non-dspam setup that would work I'd appreciate knowing about it. Thanks. Dave.

2 3

Re: Indexing Mail faster
by Joseph Tam 29 Jan '15

29 Jan '15

Kevin writes: > Appreciate if you could help with this. I have been trying to address this > "slow search" issue for a while with very limited success(I was trying to > implement FTS also), so I will appreciate if you could support. When I'm stumped, one of the diagnostic tools I use is process tracing. Connect via IMAP, in another window/session process trace the IMAP worker process (Linux: strace -r -otrace.dump {pid})), initiate the search and quit. Then you can look through the trace and see whether it gets hung on a particular operation. Joseph Tam <jtam.home(a)gmail.com>

3 6

Re: Indexing Mail faster
by Kevin Laurie 29 Jan '15

29 Jan '15

Hi Micheal, Here is how it should look:- ~# telnet localhost imap Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. . login user(a)domain.net password . OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE SEARCH=FUZZY QUOTA] Logged in . select inbox * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Forwarded) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $Forwarded \*)] Flags permitted. * 49319 EXISTS * 12 RECENT * OK [UNSEEN 46791] First unseen. * OK [UIDVALIDITY 1414214135] UIDs valid * OK [UIDNEXT 106554] Predicted next UID * OK [NOMODSEQ] No permanent modsequences . OK [READ-WRITE] Select completed (0.128 secs). . search text hello * OK Searched 6% of the mailbox, ETA 2:34 Finally at the end you logout by . logout On Thu, Jan 29, 2015 at 4:39 PM, Kevin Laurie <superinterstellar(a)gmail.com> wrote: > Dear Michael, > You will need to telnet locally to imap. So run the following commands in > your server > > 1. telnet localhost imap > 2. . login email(a)domain.com password > 3. . select inbox > 4. . search text "passport" (This command here will search your inbox and > give you the time needed) > 5. . logout > > Notice the period(.) You need to use it . > > Hope this is clear. > Regards > Kevin > > > On Thu, Jan 29, 2015 at 4:32 PM, M. Koehler <michael.koehler(a)makomi.de> > wrote: > >> Hi Kevin, >> >> I want to test it with my dovecot but I don´t know how to measure it. >> Could you please provide my the configuration/terminal commands to get an >> reliable output? I just have mailboxes up to 4GB in VM on a proxmox server >> (Debian Wheezy at the moment). >> >> regards, >> Michael >> >> >> >> > Am 29.01.2015 um 06:29 schrieb Kevin Laurie < >> superinterstellar(a)gmail.com>: >> > >> > Dear Peter, >> > Noted. Thanks for your input. Appreciate it. >> > At this point my urgent most priority is to get FTS. Waiting 3 mins for >> a >> > body search email is bad(but of course my email box is large) >> > I need to have this sorted out by today as I have been putting this off >> for >> > too long mostly because of lack of troubleshooting documentation >> online(if >> > i do figure this out will create some for guidance) >> > Regards >> > Kevin >> > >> > On Thu, Jan 29, 2015 at 1:18 PM, Peter Hodur <petehodur(a)gmail.com> >> wrote: >> > >> >> >> >> >> >> On Thursday, January 29, 2015, Kevin Laurie < >> superinterstellar(a)gmail.com> >> >> wrote: >> >> >> >>> Dear Peter, >> >>> Oh. Sorry(didnt know you were addressing someone else) my apologies. >> But >> >>> as you can see , I am desperately trying to address this issue. >> >>> >> >> >> >> No problem ;))) i wrote because my search result is good but not so >> >> good as someone wrote here couple of days ago. >> >> >> >> >> >> >> >>> I have 1 disk with 200GB running on a VPS. >> >>> The file system is ext4. >> >>> >> >> >> >> Im not dovecot expert, but i think this is your problem. You need more >> >> iops. If you need performance, VPS may be (if connected to fast SAN) >> but >> >> basicly is not a good choice. >> >> >> >> >> >>> I think I will need to implement FTS to fix this as I will need body >> >>> searches. >> >>> >> >> >> >> FTS could be a solution. You are right. >> >> >> >> >> >>> Have you tried FTS before? >> >>> >> >> >> >> Im sorry, never :( >> >> >> >> But may be someone more skilled than me could answer the main question, >> >> how much dovecot index helps in case imap search agains body. >> >> >> >> Im not sure, but my opinion is that it helps little (may be not at all) >> >> >> >> Ps: may be you use it, but do not forget to setup delivering from mail >> >> server via LDA or LMTP instead of direct to maildirs. This is better >> >> because dovecot updates your indexes at the time of message arrival. >> Later >> >> accesses should be faster. >> >> >> >> Peter >> >> >> >> >> >> >> >>> >> >>> >> >>> Regards >> >>> Kevin >> >>> >> >> >> >>> >> >>> On Thu, Jan 29, 2015 at 12:55 PM, Peter Hodur <petehodur(a)gmail.com> >> >>> wrote: >> >>> >> >>>> Kevin, >> >>>> >> >>>> My message was not addressed to you ;) because my results are NOT so >> >>>> good as someone wrote it here. I can fulltext cca 8k messages in 7-8 >> >>>> seconds. >> >>>> >> >>>> Someone wrote that he has results cca 22k messages in 4 seconds :( >> >>>> >> >>>> Im not sure, but the answer is may be in the index and disk >> subsystem. >> >>>> >> >>>> If and only if dovecot index does not have saved in the index >> keywords >> >>>> from BODY of messages the problem is your/my disk subsystem. >> >>>> >> >>>> Generally, messages from mailing list like this are pretty small. The >> >>>> problem is that dovecot must read all messages. If you have maildir >> its >> >>>> mean open and read many files. >> >>>> >> >>>> But my test search was agains archive of my personal inbox - so not >> onoy >> >>>> small messages like this but also messages with big attachments etc. >> >>>> >> >>>> If dovecot index only headers, than it in case of full text search >> read >> >>>> whole messages - mime parts allows that plain text body can be after >> >>>> attachment etc. >> >>>> >> >>>> And here is it all about iops and throughput. >> >>>> >> >>>> How many disks you have? And what setup? >> >>>> >> >>>> Generally, the only good setup is many smaller disks in RAID 10 >> >>>> (stripped mirrors) and if your fs allows it, with added read cache - >> in >> >>>> case of zfs: l2arc on ssd. >> >>>> >> >>>> >> >>>> >> >>>> Pete >> >>>> >> >>>> >> >>>> >> >>>> On Thursday, January 29, 2015, Kevin Laurie < >> superinterstellar(a)gmail.com> >> >>>> wrote: >> >>>> >> >>>>> Hi Peter, >> >>>>> Sorry, >> >>>>> >> >>>>> I think its maildir >> >>>>> Output of my dovecot -n is listed below:- >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> # 2.2.9: /etc/dovecot/dovecot.conf >> >>>>> # OS: Linux 3.10.62-xenU-25-0e6777a-x86_64 x86_64 Ubuntu 14.04.1 LTS >> >>>>> auth_master_user_separator = * >> >>>>> auth_mechanisms = PLAIN LOGIN >> >>>>> dict { >> >>>>> acl = mysql:/etc/dovecot/dovecot- >> >>>>> share-folder.conf >> >>>>> quotadict = mysql:/etc/dovecot/dovecot-used-quota.conf >> >>>>> } >> >>>>> first_valid_uid = 2000 >> >>>>> last_valid_uid = 2000 >> >>>>> listen = * >> >>>>> log_path = /var/log/dovecot.log >> >>>>> mail_debug = yes >> >>>>> mail_gid = 2000 >> >>>>> mail_location = maildir:/%Lh/Maildir/:INDEX=/%Lh/Maildir/ >> >>>>> mail_plugins = quota fts >> >>>>> mail_uid = 2000 >> >>>>> managesieve_notify_capability = mailto >> >>>>> managesieve_sieve_capability = fileinto reject envelope >> >>>>> encoded-character vacation subaddress comparator-i;ascii-numeric >> relational >> >>>>> regex imap4flags copy include variables body enotify environment >> mailbox >> >>>>> date ihave >> >>>>> namespace { >> >>>>> inbox = yes >> >>>>> location = >> >>>>> mailbox Drafts { >> >>>>> auto = subscribe >> >>>>> special_use = \Drafts >> >>>>> } >> >>>>> mailbox Junk { >> >>>>> auto = subscribe >> >>>>> special_use = \Junk >> >>>>> } >> >>>>> mailbox Sent { >> >>>>> auto = subscribe >> >>>>> special_use = \Sent >> >>>>> } >> >>>>> mailbox "Sent Messages" { >> >>>>> auto = no >> >>>>> special_use = \Sent >> >>>>> } >> >>>>> mailbox Spam { >> >>>>> auto = no >> >>>>> special_use = \Junk >> >>>>> } >> >>>>> mailbox Trash { >> >>>>> auto = subscribe >> >>>>> special_use = \Trash >> >>>>> } >> >>>>> prefix = >> >>>>> separator = / >> >>>>> type = private >> >>>>> } >> >>>>> namespace { >> >>>>> list = children >> >>>>> location = maildir:/%%Lh/Maildir/:INDEX=/%%Lh/Maildir/Shared/%%u >> >>>>> prefix = Shared/%%u/ >> >>>>> separator = / >> >>>>> subscriptions = yes >> >>>>> type = shared >> >>>>> } >> >>>>> passdb { >> >>>>> args = /etc/dovecot/dovecot-mysql.conf >> >>>>> driver = sql >> >>>>> } >> >>>>> passdb { >> >>>>> args = /etc/dovecot/dovecot-master-users-password >> >>>>> driver = passwd-file >> >>>>> master = yes >> >>>>> } >> >>>>> plugin { >> >>>>> acl = vfile >> >>>>> acl_shared_dict = proxy::acl >> >>>>> auth_socket_path = /var/run/dovecot/auth-master >> >>>>> quota = dict:user::proxy::quotadict >> >>>>> quota_rule = *:storage=1G >> >>>>> quota_warning = storage=85%% quota-warning 85 %u >> >>>>> quota_warning2 = storage=90%% quota-warning 90 %u >> >>>>> quota_warning3 = storage=95%% quota-warning 95 %u >> >>>>> sieve = /%Lh/sieve/dovecot.sieve >> >>>>> sieve_default = /var/vmail/sieve/dovecot.sieve >> >>>>> sieve_dir = /%Lh/sieve >> >>>>> sieve_global_dir = /var/vmail/sieve >> >>>>> } >> >>>>> protocols = pop3 imap sieve lmtp >> >>>>> service auth { >> >>>>> unix_listener /var/spool/postfix/private/dovecot-auth { >> >>>>> group = postfix >> >>>>> mode = 0666 >> >>>>> user = postfix >> >>>>> } >> >>>>> unix_listener auth-master { >> >>>>> group = vmail >> >>>>> mode = 0666 >> >>>>> user = vmail >> >>>>> } >> >>>>> unix_listener auth-userdb { >> >>>>> group = vmail >> >>>>> mode = 0660 >> >>>>> user = vmail >> >>>>> } >> >>>>> } >> >>>>> service dict { >> >>>>> unix_listener dict { >> >>>>> group = vmail >> >>>>> mode = 0660 >> >>>>> user = vmail >> >>>>> } >> >>>>> } >> >>>>> service imap-login { >> >>>>> process_limit = 500 >> >>>>> service_count = 1 >> >>>>> } >> >>>>> service lmtp { >> >>>>> executable = lmtp -L >> >>>>> inet_listener lmtp { >> >>>>> port = 24 >> >>>>> } >> >>>>> process_min_avail = 5 >> >>>>> unix_listener /var/spool/postfix/private/dovecot-lmtp { >> >>>>> group = postfix >> >>>>> mode = 0600 >> >>>>> user = postfix >> >>>>> } >> >>>>> user = vmail >> >>>>> } >> >>>>> service pop3-login { >> >>>>> service_count = 1 >> >>>>> } >> >>>>> service quota-warning { >> >>>>> executable = script /usr/local/bin/dovecot-quota-warning.sh >> >>>>> unix_listener quota-warning { >> >>>>> group = vmail >> >>>>> mode = 0660 >> >>>>> user = vmail >> >>>>> } >> >>>>> } >> >>>>> ssl = required >> >>>>> ssl_cert = </etc/ssl/certs/sicl.net.crt >> >>>>> ssl_key = </etc/ssl/private/sicl.net.key >> >>>>> userdb { >> >>>>> args = /etc/dovecot/dovecot-mysql.conf >> >>>>> driver = sql >> >>>>> } >> >>>>> protocol lda { >> >>>>> auth_socket_path = /var/run/dovecot/auth-master >> >>>>> lda_mailbox_autocreate = yes >> >>>>> log_path = /var/log/dovecot-sieve.log >> >>>>> mail_plugins = quota fts sieve >> >>>>> postmaster_address = root >> >>>>> } >> >>>>> protocol lmtp { >> >>>>> info_log_path = /var/log/dovecot-lmtp.log >> >>>>> lmtp_save_to_detail_mailbox = yes >> >>>>> mail_plugins = quota sieve >> >>>>> postmaster_address = postmaster >> >>>>> recipient_delimiter = + >> >>>>> } >> >>>>> protocol imap { >> >>>>> imap_client_workarounds = tb-extra-mailbox-sep >> >>>>> mail_plugins = quota fts imap_quota >> >>>>> } >> >>>>> protocol pop3 { >> >>>>> mail_plugins = quota fts >> >>>>> pop3_client_workarounds = outlook-no-nuls oe-ns-eoh >> >>>>> pop3_uidl_format = %08Xu%08Xv >> >>>>> } >> >>>>> >> >>>>> >> >>>>> On Thu, Jan 29, 2015 at 12:37 PM, Kevin Laurie < >> >>>>> superinterstellar(a)gmail.com> wrote: >> >>>>> >> >>>>>> Dear Peter, >> >>>>>> >> >>>>>> My inbox is MDA_external >> >>>>>> Storage: 17GB of 24GB >> >>>>>> >> >>>>>> Subject / From / To is fast but FTS(Full Text Search) for body is >> >>>>>> horrible. I suppose this is where we need Apache Solr. >> >>>>>> >> >>>>>> Do you think my mail storage format is bad? Do I need to change for >> >>>>>> better performance? >> >>>>>> Please advise >> >>>>>> Kevin >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> On Thu, Jan 29, 2015 at 12:25 PM, Peter Hodur <petehodur(a)gmail.com >> > >> >>>>>> wrote: >> >>>>>> >> >>>>>>>> * Kevin Laurie <superinterstellar(a)gmail.com> 2015.01.24 19:41: >> >>>>>>>> >> >>>>>>>>> Currently the time it takes to search 25,000mails is 4mins. If >> >>>>>>> indexed >> >>>>>>>> how >> >>>>>>>>> much faster are we looking at? >> >>>>>>>> >> >>>>>>>> With a current version of Dovecot a search is pretty fast >> _without_ >> >>>>>>> using >> >>>>>>>> external indexes. I have a view defined (virtual plugin) with >> >>>>>>> around 22.000 >> >>>>>>>> messages in it, and searching the full view only takes 2.5 >> seconds: >> >>>>>>>> >> >>>>>>>> >> >>>>>>> hmmm, could you please tell me more about your setup? What storage >> >>>>>>> format >> >>>>>>> do you use? Maildir or? What is the cumulative size of your >> messages? >> >>>>>>> >> >>>>>>> My results without FTS on ZFS FS with SSD L2ARC are not so good: >> >>>>>>> >> >>>>>>> a4 select INBOX._OLD-OUTLOOK >> >>>>>>> * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Junk $NotJunk >> >>>>>>> JunkRecorded $Forwarded) >> >>>>>>> * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft >> $Junk >> >>>>>>> $NotJunk JunkRecorded $Forwarded \*)] Flags permitted. >> >>>>>>> ** 7748 EXISTS* >> >>>>>>> * 0 RECENT >> >>>>>>> * OK [UIDVALIDITY 1421198037] UIDs valid >> >>>>>>> * OK [UIDNEXT 11509] Predicted next UID >> >>>>>>> * OK [HIGHESTMODSEQ 12204] Highest >> >>>>>>> a4 OK [READ-WRITE] Select completed (0.001 secs). >> >>>>>>> >> >>>>>>> a5 search charset utf-8 body "mall" >> >>>>>>> * SEARCH 2 49 101 117 158 171 185 192 197 202 207 223 228 234 236 >> 240 >> >>>>>>> 249 >> >>>>>>> 279 280 281 287 288 289 290 297 321 327 337 344 351 360 370 373 >> 385 >> >>>>>>> 389 390 >> >>>>>>> 391 398 405 413 424 444 458 463 470 474 480 482 505 513 520 530 >> 531 >> >>>>>>> 532 533 >> >>>>>>> 543 559 560 561 562 563 566 588 593 597 625 630 639 644 656 671 >> 672 >> >>>>>>> 677 692 >> >>>>>>> 720 723 734 738 741 745 752 755 757 765 775 777 784 791 818 820 >> 821 >> >>>>>>> 833 855 >> >>>>>>> 863 864 868 881 896 910 917 922 926 928 931 991 996 997 998 1000 >> 1010 >> >>>>>>> 1011 >> >>>>>>> 1012 1014 1018 1019 1026 1047 1068 1077 1095 1101 1105 1122 1136 >> 1137 >> >>>>>>> 1140 >> >>>>>>> 1155 1160 1166 1171 1179 1180 1197 1208 1229 1239 1258 1263 1271 >> 1282 >> >>>>>>> 1286 >> >>>>>>> 1290 1298 1319 1364 1365 1370 1386 1408 1410 1429 1463 1465 1470 >> 1471 >> >>>>>>> 1494 >> >>>>>>> 1518 1522 1529 1530 1536 1541 1548 1571 1581 1585 1588 1594 1605 >> 1606 >> >>>>>>> 1611 >> >>>>>>> 1612 1619 1620 1625 1652 1666 1667 1729 1730 1731 1732 1733 1734 >> 1735 >> >>>>>>> 1781 >> >>>>>>> 1782 1817 1818 1897 1900 1921 1940 1946 1960 1972 1981 1995 1998 >> 2002 >> >>>>>>> 2006 >> >>>>>>> 2028 2049 2057 2095 2100 2157 2168 2181 2185 2192 2203 2204 2207 >> 2208 >> >>>>>>> 2210 >> >>>>>>> 2220 2225 2255 2273 2282 2283 2288 2289 2317 2320 2340 2367 2374 >> 2377 >> >>>>>>> 2378 >> >>>>>>> 2379 2384 2389 2402 2409 2436 2459 2475 2476 2488 2504 2519 2538 >> 2539 >> >>>>>>> 2551 >> >>>>>>> 2566 2572 2597 2599 2603 2617 2629 2664 2698 2716 2731 2733 2753 >> 2754 >> >>>>>>> 2780 >> >>>>>>> 2805 2808 2815 2818 2850 2861 2862 2867 2886 2896 2900 2914 2931 >> 2936 >> >>>>>>> 2938 >> >>>>>>> 2939 2950 2969 2990 3017 3019 3062 3075 3094 3101 3115 3138 3159 >> 3161 >> >>>>>>> 3178 >> >>>>>>> 3185 3190 3204 3217 3218 3248 3263 3265 3266 3273 3282 3288 3295 >> 3386 >> >>>>>>> 3428 >> >>>>>>> 3453 3476 3478 3479 3511 3548 3606 3629 3693 3694 3737 3793 3799 >> 3801 >> >>>>>>> 3808 >> >>>>>>> 3812 3814 3815 3834 3849 3860 3862 3880 3910 3917 3930 3932 3952 >> 3953 >> >>>>>>> 3954 >> >>>>>>> 3957 3959 3968 3971 3973 3978 3979 3980 4008 4022 4040 4057 4058 >> 4059 >> >>>>>>> 4063 >> >>>>>>> 4064 4066 4069 4070 4075 4096 4112 4131 4132 4133 4141 4143 4144 >> 4145 >> >>>>>>> 4146 >> >>>>>>> 4147 4167 4174 4199 4201 4202 4203 4206 4211 4217 4218 4226 4229 >> 4258 >> >>>>>>> 4259 >> >>>>>>> 4267 4287 4357 4359 4363 4364 4365 4367 4390 4391 4462 4475 4497 >> 4502 >> >>>>>>> 4538 >> >>>>>>> 4540 4552 4557 4558 4561 4563 4567 4571 4572 4573 4575 4577 4593 >> 4594 >> >>>>>>> 4604 >> >>>>>>> 4611 4619 4628 4638 4639 4662 4672 4678 4679 4692 4696 4785 4786 >> 4787 >> >>>>>>> 4788 >> >>>>>>> 4789 4792 4793 4794 4802 4817 4818 4819 4820 4836 4857 4874 4887 >> 4901 >> >>>>>>> 4905 >> >>>>>>> 4906 4907 4908 4911 4925 4928 4940 4941 4953 5060 5103 5116 5118 >> 5129 >> >>>>>>> 5131 >> >>>>>>> 5136 5158 5163 5182 5184 5203 5212 5216 5269 5270 5271 5272 5273 >> 5276 >> >>>>>>> 5277 >> >>>>>>> 5278 5286 5301 5302 5303 5304 5320 5341 5358 5371 5385 5389 5390 >> 5391 >> >>>>>>> 5392 >> >>>>>>> 5395 5409 5412 5424 5425 5437 5544 5587 5600 5602 5613 5615 5620 >> 5642 >> >>>>>>> 5647 >> >>>>>>> 5666 5668 5687 5696 5700 5736 5740 5749 5764 5783 5809 5814 5853 >> 5866 >> >>>>>>> 5867 >> >>>>>>> 5877 5888 5895 5896 5897 5898 5899 5908 5910 5911 5912 5939 5950 >> 5958 >> >>>>>>> 5990 >> >>>>>>> 6000 6059 6074 6095 6097 6112 6137 6141 6189 6193 6212 6228 6229 >> 6233 >> >>>>>>> 6271 >> >>>>>>> 6273 6275 6285 6310 6317 6335 6383 6384 6397 6427 6430 6459 6463 >> 6482 >> >>>>>>> 6492 >> >>>>>>> 6506 6565 6585 6620 6670 6673 6675 6705 6715 6716 6741 6812 6826 >> 6852 >> >>>>>>> 6859 >> >>>>>>> 6895 6896 6907 6913 6919 6935 6943 6948 6979 7023 7025 7035 7039 >> 7042 >> >>>>>>> 7108 >> >>>>>>> 7131 7145 7163 7171 7172 7194 7198 7199 7203 7256 7257 7294 7303 >> 7317 >> >>>>>>> 7322 >> >>>>>>> 7343 7344 7347 7348 7352 7386 7390 7391 7392 7393 7407 7408 7409 >> 7417 >> >>>>>>> 7418 >> >>>>>>> 7419 7420 7421 7426 7432 7437 7462 7467 7468 7473 7474 7475 7488 >> 7502 >> >>>>>>> 7503 >> >>>>>>> 7558 7588 7589 7628 7685 7695 7699 7703 7723 >> >>>>>>> *a5 OK Search completed (7.846 secs).* >> >>>>>>> >> >>>>>>> >> >>>>>>> Searching against "subject" is pretty fast, few miliseconds ... >> >>>>>>> >> >>>>>> >> >>>>>> >> >>>>> >> >>> >> >> >

1 0

Indexing Mail faster
by Kevin Laurie 29 Jan '15

29 Jan '15

Hi, I am trying to get faster search results on our webmail client(Roundcube). Besides using Lucene for FTS are there other options? Would having all mails indexed give fast results? Currently the time it takes to search 25,000mails is 4mins. If indexed how much faster are we looking at? Really appreciate if someone could advise about this. Thanks Kevin

8 28

Uniqueness of dovecot mailbox-guids
by Christian Binder Stadt Freilassing 29 Jan '15

29 Jan '15

Sorry if this is a duplicate message. I'm new in this mailing list and not sure if the first message really got posted ... Dovecot version: 2.2.9 We're using doveadm backup to migrate user mailboxes from cyrus to dovecot. I defined a master user for the old cyrus mailserver in imapc backend to get access to all the mailboxes. Now I realized that after migration every user's "Inbox" or "Trash" on the new dovecot server has the same mailbox-guid, e.g: User a: inbox dd9456... trash 5549cc... User b: inbox dd9456... trash 5549cc... Is this a wanted behaviour? If i do not migrate with doveadm backup and just manually create mailboxes, the mailbox-guids are globally unique, e.g: User a: inbox dd9456... trash 5549cc... User b: inbox c543fe... trash t49ccc... Thanks! Christian

1 0

Re: Indexing Mail faster
by Kevin Laurie 29 Jan '15

29 Jan '15

Dear Peter, Noted. Thanks for your input. Appreciate it. At this point my urgent most priority is to get FTS. Waiting 3 mins for a body search email is bad(but of course my email box is large) I need to have this sorted out by today as I have been putting this off for too long mostly because of lack of troubleshooting documentation online(if i do figure this out will create some for guidance) Regards Kevin On Thu, Jan 29, 2015 at 1:18 PM, Peter Hodur <petehodur(a)gmail.com> wrote: > > > On Thursday, January 29, 2015, Kevin Laurie <superinterstellar(a)gmail.com> > wrote: > >> Dear Peter, >> Oh. Sorry(didnt know you were addressing someone else) my apologies. But >> as you can see , I am desperately trying to address this issue. >> > > No problem ;))) i wrote because my search result is good but not so > good as someone wrote here couple of days ago. > > > >> I have 1 disk with 200GB running on a VPS. >> The file system is ext4. >> > > Im not dovecot expert, but i think this is your problem. You need more > iops. If you need performance, VPS may be (if connected to fast SAN) but > basicly is not a good choice. > > >> I think I will need to implement FTS to fix this as I will need body >> searches. >> > > FTS could be a solution. You are right. > > >> Have you tried FTS before? >> > > Im sorry, never :( > > But may be someone more skilled than me could answer the main question, > how much dovecot index helps in case imap search agains body. > > Im not sure, but my opinion is that it helps little (may be not at all) > > Ps: may be you use it, but do not forget to setup delivering from mail > server via LDA or LMTP instead of direct to maildirs. This is better > because dovecot updates your indexes at the time of message arrival. Later > accesses should be faster. > > Peter > > > >> >> >> Regards >> Kevin >> > >> >> On Thu, Jan 29, 2015 at 12:55 PM, Peter Hodur <petehodur(a)gmail.com> >> wrote: >> >>> Kevin, >>> >>> My message was not addressed to you ;) because my results are NOT so >>> good as someone wrote it here. I can fulltext cca 8k messages in 7-8 >>> seconds. >>> >>> Someone wrote that he has results cca 22k messages in 4 seconds :( >>> >>> Im not sure, but the answer is may be in the index and disk subsystem. >>> >>> If and only if dovecot index does not have saved in the index keywords >>> from BODY of messages the problem is your/my disk subsystem. >>> >>> Generally, messages from mailing list like this are pretty small. The >>> problem is that dovecot must read all messages. If you have maildir its >>> mean open and read many files. >>> >>> But my test search was agains archive of my personal inbox - so not onoy >>> small messages like this but also messages with big attachments etc. >>> >>> If dovecot index only headers, than it in case of full text search read >>> whole messages - mime parts allows that plain text body can be after >>> attachment etc. >>> >>> And here is it all about iops and throughput. >>> >>> How many disks you have? And what setup? >>> >>> Generally, the only good setup is many smaller disks in RAID 10 >>> (stripped mirrors) and if your fs allows it, with added read cache - in >>> case of zfs: l2arc on ssd. >>> >>> >>> >>> Pete >>> >>> >>> >>> On Thursday, January 29, 2015, Kevin Laurie <superinterstellar(a)gmail.com> >>> wrote: >>> >>>> Hi Peter, >>>> Sorry, >>>> >>>> I think its maildir >>>> Output of my dovecot -n is listed below:- >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> # 2.2.9: /etc/dovecot/dovecot.conf >>>> # OS: Linux 3.10.62-xenU-25-0e6777a-x86_64 x86_64 Ubuntu 14.04.1 LTS >>>> auth_master_user_separator = * >>>> auth_mechanisms = PLAIN LOGIN >>>> dict { >>>> acl = mysql:/etc/dovecot/dovecot- >>>> share-folder.conf >>>> quotadict = mysql:/etc/dovecot/dovecot-used-quota.conf >>>> } >>>> first_valid_uid = 2000 >>>> last_valid_uid = 2000 >>>> listen = * >>>> log_path = /var/log/dovecot.log >>>> mail_debug = yes >>>> mail_gid = 2000 >>>> mail_location = maildir:/%Lh/Maildir/:INDEX=/%Lh/Maildir/ >>>> mail_plugins = quota fts >>>> mail_uid = 2000 >>>> managesieve_notify_capability = mailto >>>> managesieve_sieve_capability = fileinto reject envelope >>>> encoded-character vacation subaddress comparator-i;ascii-numeric relational >>>> regex imap4flags copy include variables body enotify environment mailbox >>>> date ihave >>>> namespace { >>>> inbox = yes >>>> location = >>>> mailbox Drafts { >>>> auto = subscribe >>>> special_use = \Drafts >>>> } >>>> mailbox Junk { >>>> auto = subscribe >>>> special_use = \Junk >>>> } >>>> mailbox Sent { >>>> auto = subscribe >>>> special_use = \Sent >>>> } >>>> mailbox "Sent Messages" { >>>> auto = no >>>> special_use = \Sent >>>> } >>>> mailbox Spam { >>>> auto = no >>>> special_use = \Junk >>>> } >>>> mailbox Trash { >>>> auto = subscribe >>>> special_use = \Trash >>>> } >>>> prefix = >>>> separator = / >>>> type = private >>>> } >>>> namespace { >>>> list = children >>>> location = maildir:/%%Lh/Maildir/:INDEX=/%%Lh/Maildir/Shared/%%u >>>> prefix = Shared/%%u/ >>>> separator = / >>>> subscriptions = yes >>>> type = shared >>>> } >>>> passdb { >>>> args = /etc/dovecot/dovecot-mysql.conf >>>> driver = sql >>>> } >>>> passdb { >>>> args = /etc/dovecot/dovecot-master-users-password >>>> driver = passwd-file >>>> master = yes >>>> } >>>> plugin { >>>> acl = vfile >>>> acl_shared_dict = proxy::acl >>>> auth_socket_path = /var/run/dovecot/auth-master >>>> quota = dict:user::proxy::quotadict >>>> quota_rule = *:storage=1G >>>> quota_warning = storage=85%% quota-warning 85 %u >>>> quota_warning2 = storage=90%% quota-warning 90 %u >>>> quota_warning3 = storage=95%% quota-warning 95 %u >>>> sieve = /%Lh/sieve/dovecot.sieve >>>> sieve_default = /var/vmail/sieve/dovecot.sieve >>>> sieve_dir = /%Lh/sieve >>>> sieve_global_dir = /var/vmail/sieve >>>> } >>>> protocols = pop3 imap sieve lmtp >>>> service auth { >>>> unix_listener /var/spool/postfix/private/dovecot-auth { >>>> group = postfix >>>> mode = 0666 >>>> user = postfix >>>> } >>>> unix_listener auth-master { >>>> group = vmail >>>> mode = 0666 >>>> user = vmail >>>> } >>>> unix_listener auth-userdb { >>>> group = vmail >>>> mode = 0660 >>>> user = vmail >>>> } >>>> } >>>> service dict { >>>> unix_listener dict { >>>> group = vmail >>>> mode = 0660 >>>> user = vmail >>>> } >>>> } >>>> service imap-login { >>>> process_limit = 500 >>>> service_count = 1 >>>> } >>>> service lmtp { >>>> executable = lmtp -L >>>> inet_listener lmtp { >>>> port = 24 >>>> } >>>> process_min_avail = 5 >>>> unix_listener /var/spool/postfix/private/dovecot-lmtp { >>>> group = postfix >>>> mode = 0600 >>>> user = postfix >>>> } >>>> user = vmail >>>> } >>>> service pop3-login { >>>> service_count = 1 >>>> } >>>> service quota-warning { >>>> executable = script /usr/local/bin/dovecot-quota-warning.sh >>>> unix_listener quota-warning { >>>> group = vmail >>>> mode = 0660 >>>> user = vmail >>>> } >>>> } >>>> ssl = required >>>> ssl_cert = </etc/ssl/certs/sicl.net.crt >>>> ssl_key = </etc/ssl/private/sicl.net.key >>>> userdb { >>>> args = /etc/dovecot/dovecot-mysql.conf >>>> driver = sql >>>> } >>>> protocol lda { >>>> auth_socket_path = /var/run/dovecot/auth-master >>>> lda_mailbox_autocreate = yes >>>> log_path = /var/log/dovecot-sieve.log >>>> mail_plugins = quota fts sieve >>>> postmaster_address = root >>>> } >>>> protocol lmtp { >>>> info_log_path = /var/log/dovecot-lmtp.log >>>> lmtp_save_to_detail_mailbox = yes >>>> mail_plugins = quota sieve >>>> postmaster_address = postmaster >>>> recipient_delimiter = + >>>> } >>>> protocol imap { >>>> imap_client_workarounds = tb-extra-mailbox-sep >>>> mail_plugins = quota fts imap_quota >>>> } >>>> protocol pop3 { >>>> mail_plugins = quota fts >>>> pop3_client_workarounds = outlook-no-nuls oe-ns-eoh >>>> pop3_uidl_format = %08Xu%08Xv >>>> } >>>> >>>> >>>> On Thu, Jan 29, 2015 at 12:37 PM, Kevin Laurie < >>>> superinterstellar(a)gmail.com> wrote: >>>> >>>>> Dear Peter, >>>>> >>>>> My inbox is MDA_external >>>>> Storage: 17GB of 24GB >>>>> >>>>> Subject / From / To is fast but FTS(Full Text Search) for body is >>>>> horrible. I suppose this is where we need Apache Solr. >>>>> >>>>> Do you think my mail storage format is bad? Do I need to change for >>>>> better performance? >>>>> Please advise >>>>> Kevin >>>>> >>>>> >>>>> >>>>> On Thu, Jan 29, 2015 at 12:25 PM, Peter Hodur <petehodur(a)gmail.com> >>>>> wrote: >>>>> >>>>>> > * Kevin Laurie <superinterstellar(a)gmail.com> 2015.01.24 19:41: >>>>>> > >>>>>> > > Currently the time it takes to search 25,000mails is 4mins. If >>>>>> indexed >>>>>> > how >>>>>> > > much faster are we looking at? >>>>>> > >>>>>> > With a current version of Dovecot a search is pretty fast _without_ >>>>>> using >>>>>> > external indexes. I have a view defined (virtual plugin) with >>>>>> around 22.000 >>>>>> > messages in it, and searching the full view only takes 2.5 seconds: >>>>>> > >>>>>> > >>>>>> hmmm, could you please tell me more about your setup? What storage >>>>>> format >>>>>> do you use? Maildir or? What is the cumulative size of your messages? >>>>>> >>>>>> My results without FTS on ZFS FS with SSD L2ARC are not so good: >>>>>> >>>>>> a4 select INBOX._OLD-OUTLOOK >>>>>> * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Junk $NotJunk >>>>>> JunkRecorded $Forwarded) >>>>>> * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $Junk >>>>>> $NotJunk JunkRecorded $Forwarded \*)] Flags permitted. >>>>>> ** 7748 EXISTS* >>>>>> * 0 RECENT >>>>>> * OK [UIDVALIDITY 1421198037] UIDs valid >>>>>> * OK [UIDNEXT 11509] Predicted next UID >>>>>> * OK [HIGHESTMODSEQ 12204] Highest >>>>>> a4 OK [READ-WRITE] Select completed (0.001 secs). >>>>>> >>>>>> a5 search charset utf-8 body "mall" >>>>>> * SEARCH 2 49 101 117 158 171 185 192 197 202 207 223 228 234 236 240 >>>>>> 249 >>>>>> 279 280 281 287 288 289 290 297 321 327 337 344 351 360 370 373 385 >>>>>> 389 390 >>>>>> 391 398 405 413 424 444 458 463 470 474 480 482 505 513 520 530 531 >>>>>> 532 533 >>>>>> 543 559 560 561 562 563 566 588 593 597 625 630 639 644 656 671 672 >>>>>> 677 692 >>>>>> 720 723 734 738 741 745 752 755 757 765 775 777 784 791 818 820 821 >>>>>> 833 855 >>>>>> 863 864 868 881 896 910 917 922 926 928 931 991 996 997 998 1000 1010 >>>>>> 1011 >>>>>> 1012 1014 1018 1019 1026 1047 1068 1077 1095 1101 1105 1122 1136 1137 >>>>>> 1140 >>>>>> 1155 1160 1166 1171 1179 1180 1197 1208 1229 1239 1258 1263 1271 1282 >>>>>> 1286 >>>>>> 1290 1298 1319 1364 1365 1370 1386 1408 1410 1429 1463 1465 1470 1471 >>>>>> 1494 >>>>>> 1518 1522 1529 1530 1536 1541 1548 1571 1581 1585 1588 1594 1605 1606 >>>>>> 1611 >>>>>> 1612 1619 1620 1625 1652 1666 1667 1729 1730 1731 1732 1733 1734 1735 >>>>>> 1781 >>>>>> 1782 1817 1818 1897 1900 1921 1940 1946 1960 1972 1981 1995 1998 2002 >>>>>> 2006 >>>>>> 2028 2049 2057 2095 2100 2157 2168 2181 2185 2192 2203 2204 2207 2208 >>>>>> 2210 >>>>>> 2220 2225 2255 2273 2282 2283 2288 2289 2317 2320 2340 2367 2374 2377 >>>>>> 2378 >>>>>> 2379 2384 2389 2402 2409 2436 2459 2475 2476 2488 2504 2519 2538 2539 >>>>>> 2551 >>>>>> 2566 2572 2597 2599 2603 2617 2629 2664 2698 2716 2731 2733 2753 2754 >>>>>> 2780 >>>>>> 2805 2808 2815 2818 2850 2861 2862 2867 2886 2896 2900 2914 2931 2936 >>>>>> 2938 >>>>>> 2939 2950 2969 2990 3017 3019 3062 3075 3094 3101 3115 3138 3159 3161 >>>>>> 3178 >>>>>> 3185 3190 3204 3217 3218 3248 3263 3265 3266 3273 3282 3288 3295 3386 >>>>>> 3428 >>>>>> 3453 3476 3478 3479 3511 3548 3606 3629 3693 3694 3737 3793 3799 3801 >>>>>> 3808 >>>>>> 3812 3814 3815 3834 3849 3860 3862 3880 3910 3917 3930 3932 3952 3953 >>>>>> 3954 >>>>>> 3957 3959 3968 3971 3973 3978 3979 3980 4008 4022 4040 4057 4058 4059 >>>>>> 4063 >>>>>> 4064 4066 4069 4070 4075 4096 4112 4131 4132 4133 4141 4143 4144 4145 >>>>>> 4146 >>>>>> 4147 4167 4174 4199 4201 4202 4203 4206 4211 4217 4218 4226 4229 4258 >>>>>> 4259 >>>>>> 4267 4287 4357 4359 4363 4364 4365 4367 4390 4391 4462 4475 4497 4502 >>>>>> 4538 >>>>>> 4540 4552 4557 4558 4561 4563 4567 4571 4572 4573 4575 4577 4593 4594 >>>>>> 4604 >>>>>> 4611 4619 4628 4638 4639 4662 4672 4678 4679 4692 4696 4785 4786 4787 >>>>>> 4788 >>>>>> 4789 4792 4793 4794 4802 4817 4818 4819 4820 4836 4857 4874 4887 4901 >>>>>> 4905 >>>>>> 4906 4907 4908 4911 4925 4928 4940 4941 4953 5060 5103 5116 5118 5129 >>>>>> 5131 >>>>>> 5136 5158 5163 5182 5184 5203 5212 5216 5269 5270 5271 5272 5273 5276 >>>>>> 5277 >>>>>> 5278 5286 5301 5302 5303 5304 5320 5341 5358 5371 5385 5389 5390 5391 >>>>>> 5392 >>>>>> 5395 5409 5412 5424 5425 5437 5544 5587 5600 5602 5613 5615 5620 5642 >>>>>> 5647 >>>>>> 5666 5668 5687 5696 5700 5736 5740 5749 5764 5783 5809 5814 5853 5866 >>>>>> 5867 >>>>>> 5877 5888 5895 5896 5897 5898 5899 5908 5910 5911 5912 5939 5950 5958 >>>>>> 5990 >>>>>> 6000 6059 6074 6095 6097 6112 6137 6141 6189 6193 6212 6228 6229 6233 >>>>>> 6271 >>>>>> 6273 6275 6285 6310 6317 6335 6383 6384 6397 6427 6430 6459 6463 6482 >>>>>> 6492 >>>>>> 6506 6565 6585 6620 6670 6673 6675 6705 6715 6716 6741 6812 6826 6852 >>>>>> 6859 >>>>>> 6895 6896 6907 6913 6919 6935 6943 6948 6979 7023 7025 7035 7039 7042 >>>>>> 7108 >>>>>> 7131 7145 7163 7171 7172 7194 7198 7199 7203 7256 7257 7294 7303 7317 >>>>>> 7322 >>>>>> 7343 7344 7347 7348 7352 7386 7390 7391 7392 7393 7407 7408 7409 7417 >>>>>> 7418 >>>>>> 7419 7420 7421 7426 7432 7437 7462 7467 7468 7473 7474 7475 7488 7502 >>>>>> 7503 >>>>>> 7558 7588 7589 7628 7685 7695 7699 7703 7723 >>>>>> *a5 OK Search completed (7.846 secs).* >>>>>> >>>>>> >>>>>> Searching against "subject" is pretty fast, few miliseconds ... >>>>>> >>>>> >>>>> >>>> >>

1 0

Dovecot & gethostbyname() vulnerability
by Timo Sirainen 28 Jan '15

28 Jan '15

So just a quick note: Dovecot in general doesn't do DNS lookups, except the ones in configuration files and such. So I don't think there are any Dovecot setups which do DNS lookups for untrusted hostnames. Also, gethostbyname() is used only if IPv6 support isn't compiled into Dovecot. And IPv6 support is enabled by default if the libc is detected to support it, so pretty much all systems use it.

1 0

Uniqueness of dovecot mailbox-guids
by Christian Binder Stadt Freilassing 28 Jan '15

28 Jan '15

Dovecot version: 2.2.9 I'm using doveadm backup to migrate about hundred user mailboxes from cyrus to dovecot. I defined a master user for the old cyrus mailserver in imapc backend to get access to all the mailboxes. Now I realized that after migration every user's "Inbox" or "Trash" on the new dovecot server has the same mailbox-guid, e.g: User a: inbox dd9456... trash 5549cc... User b: inbox dd9456... trash 5549cc... Is this a wanted behaviour? If i do not migrate with doveadm backup and just manually create mailboxes, the mailbox-guids are globally unique, e.g: User a: inbox dd9456... trash 5549cc... User b: inbox c543fe... trash t49ccc... Thanks! Christian

1 0

dovecot migration
by mkawada＠redhat.com 28 Jan '15

28 Jan '15

Dear List, To migrate a dovecot server from dovecot-1.x to dovecot-2.x, then I wonder if the UIDL format "pop3_uidl_format" setting is required in the configuration file for clients not to re-download the same mails from the migrated mail server. Currently, no UIDL format is set in /etc/dovecot file of the one using dovecot-1.x server which means it is commented out as default like below. ]# grep pop3_uidl_format /etc/dovecot.conf #pop3_uidl_format = %08Xu%08Xv <---- Does this mean the "pop3_uidl_format" is in use as "%08Xu%08Xv" by default? I see the same in /etc/dovecot/conf.d/20-imap.conf of the one using dovecot-2.x, and have not been able to find any information saying it's the defualt setting. Here are the questions: 1.Can we say the "pop3_uidl_format = %08Xu%08Xv" is the enabled default setting? Or any ways to check the default? 2.Do they need to specify anything for the "pop3_uidl_format =" field for their migration in this case? Env infomation: dovecot-1.x dovecot-2.x mail_location = mbox Any information/clues would be greatly appreciated. Thanks,

3 4