- dovecot - dovecot.org

Maillog warning
by forums＠mehl-family.fr 16 May '18

16 May '18

Hi, In my maillog I see this warning message at every reception of e-mail: _STATUS=SENT (DELIVERED VIA DOVECOT SERVICE (DOVECONF: WARNING: /ETC/DOVECOT/CONF.D/90-QUOTA.CONF LINE 39: GLOBAL SETTING MAIL_PLUGINS WON'T CHAN....._ It is just a warning, the e-mail was delivered. I don't find where is the problem. Thank's. -- ########## Philippe - Forums

3 10

dovecot + cephfs - sdbox vs mdbox
by Webert de Souza Lima 16 May '18

16 May '18

I'm sending this message to both dovecot and ceph-users ML so please don't mind if something seems too obvious for you. Hi, I have a question for both dovecot and ceph lists and below I'll explain what's going on. Regarding dbox format (https://wiki2.dovecot.org/MailboxFormat/dbox) when using sdbox, a new file is stored for each email message. When using mdbox, multiple messages are appended to a single file until it reaches/passes the rotate limit. I would like to understand better how the mdbox format impacts on IO performance. I think it's generally expected that fewer larger file translate to less IO and more troughput when compared to more small files, but how does dovecot handle that with mdbox? If dovecot does flush data to storage upon each and every new email is arrived and appended to the corresponding file, would that mean that it generate the same ammount of IO as it would do with one file per message? Also, if using mdbox many messages will be appended to a said file before a new file is created. That should mean that a file descriptor is kept open for sometime by dovecot process. Using cephfs as backend, how would this impact cluster performance regarding MDS caps and inodes cached when files from thousands of users are opened and appended all over? I would like to understand this better. Why? We are a small Business Email Hosting provider with bare metal, self hosted systems, using dovecot for servicing mailboxes and cephfs for email storage. We are currently working on dovecot and storage redesign to be in production ASAP. The main objective is to serve more users with better performance, high availability and scalability. * high availability and load balancing is extremely important to us * On our current model, we're using mdbox format with dovecot, having dovecot's INDEXes stored in a replicated pool of SSDs, and messages stored in a replicated pool of HDDs (under a Cache Tier with a pool of SSDs). All using cephfs / filestore backend. Currently there are 3 clusters running dovecot 2.2.34 and ceph Jewel (10.2.9-4). - ~25K users from a few thousands of domains per cluster - ~25TB of email data per cluster - ~70GB of dovecot INDEX [meta]data per cluster - ~100MB of cephfs metadata per cluster Our goal is to build a single ceph cluster for storage that could expand in capacity, be highly available and perform well enough. I know, that's what everyone wants. Cephfs is an important choise because: - there can be multiple mountpoints, thus multiple dovecot instances on different hosts - the same storage backend is used for all dovecot instances - no need of sharding domains - dovecot is easily load balanced (with director sticking users to the same dovecot backend) On the upcoming upgrade we intent to: - upgrade ceph to 12.X (Luminous) - drop the SSD Cache Tier (because it's deprecated) - use bluestore engine I was said on freenode/#dovecot that there are many cases where SDBOX would perform better with NFS sharing. In case of cephfs, at first, I wouldn't think that would be true because more files == more generated IO, but thinking about what I said in the beginning regarding sdbox vs mdbox that could be wrong. Any thoughts will be highlt appreciated. Regards, Webert Lima DevOps Engineer at MAV Tecnologia *Belo Horizonte - Brasil* *IRC NICK - WebertRLZ*

2 2

Re: [ceph-users] dovecot + cephfs - sdbox vs mdbox
by Webert de Souza Lima 16 May '18

16 May '18

Hello Jack, yes, I imagine I'll have to do some work on tuning the block size on cephfs. Thanks for the advise. I knew that using mdbox, messages are not removed but I though that was true in sdbox too. Thanks again. We'll soon do benchmarks of sdbox vs mdbox over cephfs with bluestore backend. We'll have to do some some work on how to simulate user traffic, for writes and readings. That seems troublesome. Thanks for the plugins recommendations. I'll take the change and ask you how is the SIS status? We have used it in the past and we've had some problems with it. Regards, Webert Lima DevOps Engineer at MAV Tecnologia *Belo Horizonte - Brasil* *IRC NICK - WebertRLZ* On Wed, May 16, 2018 at 4:19 PM Jack <ceph(a)jack.fr.eu.org> wrote: > Hi, > > Many (most ?) filesystems does not store multiple files on the same block > > Thus, with sdbox, every single mail (you know, that kind of mail with 10 > lines in it) will eat an inode, and a block (4k here) > mdbox is more compact on this way > > Another difference: sdbox removes the message, mdbox does not : a single > metadata update is performed, which may be packed with others if many > files are deleted at once > > That said, I do not have experience with dovecot + cephfs, nor have made > tests for sdbox vs mdbox > > However, and this is a bit out of topic, I recommend you look at the > following dovecot's features (if not already done), as they are awesome > and will help you a lot: > - Compression (classic, https://wiki.dovecot.org/Plugins/Zlib) > - Single-Instance-Storage (aka sis, aka "attachment deduplication" : > https://www.dovecot.org/list/dovecot/2013-December/094276.html) > > Regards, > On 05/16/2018 08:37 PM, Webert de Souza Lima wrote: > > I'm sending this message to both dovecot and ceph-users ML so please > don't > > mind if something seems too obvious for you. > > > > Hi, > > > > I have a question for both dovecot and ceph lists and below I'll explain > > what's going on. > > > > Regarding dbox format (https://wiki2.dovecot.org/MailboxFormat/dbox) > when > > using sdbox, a new file is stored for each email message. > > When using mdbox, multiple messages are appended to a single file until > it > > reaches/passes the rotate limit. > > > > I would like to understand better how the mdbox format impacts on IO > > performance. > > I think it's generally expected that fewer larger file translate to less > IO > > and more troughput when compared to more small files, but how does > dovecot > > handle that with mdbox? > > If dovecot does flush data to storage upon each and every new email is > > arrived and appended to the corresponding file, would that mean that it > > generate the same ammount of IO as it would do with one file per message? > > Also, if using mdbox many messages will be appended to a said file > before a > > new file is created. That should mean that a file descriptor is kept open > > for sometime by dovecot process. > > Using cephfs as backend, how would this impact cluster performance > > regarding MDS caps and inodes cached when files from thousands of users > are > > opened and appended all over? > > > > I would like to understand this better. > > > > Why? > > We are a small Business Email Hosting provider with bare metal, self > hosted > > systems, using dovecot for servicing mailboxes and cephfs for email > storage. > > > > We are currently working on dovecot and storage redesign to be in > > production ASAP. The main objective is to serve more users with better > > performance, high availability and scalability. > > * high availability and load balancing is extremely important to us * > > > > On our current model, we're using mdbox format with dovecot, having > > dovecot's INDEXes stored in a replicated pool of SSDs, and messages > stored > > in a replicated pool of HDDs (under a Cache Tier with a pool of SSDs). > > All using cephfs / filestore backend. > > > > Currently there are 3 clusters running dovecot 2.2.34 and ceph Jewel > > (10.2.9-4). > > - ~25K users from a few thousands of domains per cluster > > - ~25TB of email data per cluster > > - ~70GB of dovecot INDEX [meta]data per cluster > > - ~100MB of cephfs metadata per cluster > > > > Our goal is to build a single ceph cluster for storage that could expand > in > > capacity, be highly available and perform well enough. I know, that's > what > > everyone wants. > > > > Cephfs is an important choise because: > > - there can be multiple mountpoints, thus multiple dovecot instances on > > different hosts > > - the same storage backend is used for all dovecot instances > > - no need of sharding domains > > - dovecot is easily load balanced (with director sticking users to the > > same dovecot backend) > > > > On the upcoming upgrade we intent to: > > - upgrade ceph to 12.X (Luminous) > > - drop the SSD Cache Tier (because it's deprecated) > > - use bluestore engine > > > > I was said on freenode/#dovecot that there are many cases where SDBOX > would > > perform better with NFS sharing. > > In case of cephfs, at first, I wouldn't think that would be true because > > more files == more generated IO, but thinking about what I said in the > > beginning regarding sdbox vs mdbox that could be wrong. > > > > Any thoughts will be highlt appreciated. > > > > Regards, > > > > Webert Lima > > DevOps Engineer at MAV Tecnologia > > *Belo Horizonte - Brasil* > > *IRC NICK - WebertRLZ* > > > > > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users(a)lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >

1 0

2.3.1 with quota and lmtp; message may be sent more than once
by Frank Ebert 16 May '18

16 May '18

Hi, I have another problem with dovecot 2.3.1 on FreeBSD (smtpd is postfix 3.3.0). When one mail is sent with different recipients and the quota limit of one user is exceeded, the message will be delivered repeatedly to all users. The queue from postfix says: (lost connection with mailserver.foo.bar[private/lmtp-dovecot] while sending end of data -- message may be sent more than once) The problem started with dovecot 2.3.1. With 2.2.3x everything was fine. Any ideas? Thanx Frank doveconf -n: auth_mechanisms = plain login auth_use_winbind = yes auth_winbind_helper_path = /usr/local/bin/ntlm_auth disable_plaintext_auth = no info_log_path = /var/log/dovecot listen = * mail_location = mdbox:~/mdbox mail_plugins = zlib quota acl mail_log notify managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacati on subaddress comparator-i;ascii-numeric relational regex imap4flags copy includ e variables body enotify environment mailbox date index ihave duplicate mime for everypart extracttext namespace { location = mdbox:/home/vmail/foo.domain/foo/mdbox prefix = Foo/ separator = / subscriptions = yes type = public } namespace { location = mdbox:/home/vmail/bar.domain/bar/mdbox prefix = Bar/ separator = / subscriptions = yes type = public } namespace { location = mdbox:/home/vmail/foobar.domain/foobar/mdbox prefix = Foobar/ separator = / subscriptions = yes type = public } namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = INBOX/ separator = / } passdb { args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { acl = vfile:/usr/local/etc/dovecot/global-acls acl_shared_dict = file:/var/lib/dovecot/shared-mailboxes mail_log_events = flag_change delete undelete expunge copy mailbox_delete mail box_rename mail_log_fields = uid box msgid size flags quota = dict:User quota::file:%h/dovecot-quota quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u sieve = ~/.dovecot.sieve sieve_dir = ~/sieve stats_command_min_time = 1 mins stats_domain_min_time = 12 hours stats_ip_min_time = 12 hours stats_memory_limit = 16 M stats_refresh = 30 stats_session_min_time = 15 mins stats_track_cmds = yes stats_user_min_time = 1 hours zlib_save = gz zlib_save_level = 6 } postmaster_address = postmaster(a)foo.bar protocols = imap lmtp sieve service auth { unix_listener /var/spool/postfix/private/auth { mode = 0666 } unix_listener auth-userdb { group = vmail user = vmail } } service lmtp { unix_listener /var/spool/postfix/private/lmtp-dovecot { group = postfix user = postfix } } service managesieve-login { inet_listener sieve { port = 4190 } process_min_avail = 0 service_count = 0 } service managesieve { process_limit = 1024 } service quota-warning { executable = script /usr/local/bin/quota-warning.sh unix_listener quota-warning { user = vmail } user = vmail } service stats { fifo_listener stats-mail { mode = 0600 user = vmail } } ssl_cert = </etc/ssl/certs/dovecot.pem ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it userdb { driver = prefetch } userdb { args = /usr/local/etc/dovecot/dovecot-ldap-userdb.conf.ext driver = ldap } protocol lmtp { mail_plugins = zlib quota acl mail_log notify sieve } protocol lda { mail_plugins = zlib quota acl mail_log notify sieve } protocol imap { mail_plugins = zlib quota acl mail_log notify imap_zlib imap_quota imap_acl }

3 6

dovecot 2.3 on Ubuntu 18.04 LTS
by Jean-Daniel Dupas 16 May '18

16 May '18

Hello, I'm running dovecot 2.3 from repo.dovecot.org <http://repo.dovecot.org/> on ubuntu 16.04 LTS, and I'm wondering if there is a scheduled date for the release of the bionic package in that repository. The Ubuntu mainstream version is based on the 2.2 branch, which prevent us to use it. Thanks.

1 0

end-to-end encryption
by Gandalf Corvotempesta 16 May '18

16 May '18

Hi to all I was looking at protonmail.com Is possible to implement and end-to-end encryption with dovecot, where server-side there is no private key to decrypt messages? If I understood properly, on protonmail the private key is encrypted with user's password, so that only an user is able to decrypt the mailbox. Anything similiar ?

5 10

Postfix/Dovecot permissions for new mailboxes
by telsch 16 May '18

16 May '18

Hello, the wiki page describe that permission should copied from root mail directory https://wiki2.dovecot.org/SharedMailboxes/Permissions The permissions for the root mail directory are set to 2770: drwxrws--- 36 root mail 4096 Apr 15 02:30 /var/spool/mail/ But newly created mailboxes get following permissions: drwx--S--- 5 newuser mail 4096 Apr 23 19:31 /var/spool/mail/newuser/ In /etc/dovecot/conf.d/10-mail.conf follow options are set: mail_gid = mail mail_privileged_group = mail mail_access_groups = mail I'm missing something to get right group permissions for new mailboxes?

2 9

sieve trace directory: error
by André Rodier 15 May '18

15 May '18

Hello, I am using a custom directory for sieve logs and trace and it apparently generate an error, in some corner case. - I have an IMAP global sieve script, that is run when appending emails into the "Archives" folder. - I have Thunderbird configured to archive emails by year When archiving an email, for instance in "Archives/2018", I often have an error message like this: Mar 31 08:03:02 homebox dovecot[14078]: imap(andre): Error: sieve: trace: creat(/home/users/andre/mails/sieve/logs//andre.Archives/2018.3.20180331-080302.14235.5.trace) failed: No such file or directory I have this configuration (amongst) for sieve logs: sieve_user_log = ~/mails/sieve/logs/ sieve_trace_dir = ~/mails/sieve/logs/ sieve_trace_level = commands sieve_trace_debug = yes sieve_trace_addresses = yes Did I a misconfigured something? Thanks for your help, André -- https://github.com/progmaticltd/homebox

3 5

dsync migration from maildir layout=fs to maildir seperator / - folders or inbox but not both
by Matt Bryant 13 May '18

13 May '18

Am attempting to bring my mailserver back to home server post house move and whilst I could use rsync have been trying to get to grips with dsync. However, I seem to have an issue - my current server specifies mail_location as maildir:/var/vmail/%d/%n/Maildir:LAYOUT=fs and my imapc config on new server is as follows ... imapc_host = old.mailserver.host imapc_user = %u imapc_master_user = master imapc_password = ********** imapc_features = $imapc_features rfc822.size fetch-headers imapc_list_prefix = INBOX imapc_port = 993 imapc_ssl = imaps imapc_ssl_verify = no the trouble is if I leave imapc_list_prefix = INBOX I get the inbox only - no folders ... If I comment this out delete maildir on target and re-run .. I get the folders but no inbox but folders are in fs layout format rather than being converted to INBOX.folder.subfolder bug perhaps ???? or am I doing something wrong ??? src dovecot version 2.0.9 tgt dovecot version 2.2.33.2 Any ideas would be appreciated ... rgds Matt -- <https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campai…>

2 3

Marks as read on mover to Archive
by ＠lbutlr 13 May '18

13 May '18

How would I setup dovecot so that when messages are moved to the Archive mailbox, they are marked as read? Would it be similar to the imap-sieve for spam tagging? imapsieve_mailbox1_name = Junk imapsieve_mailbox1_causes = COPY imapsieve_mailbox1_before = file:/usr/lib/dovecot/sieve/report-spam.sieve Or is there a simpler method? If it matters: mailbox Archive { auto = subscribe special_use = \Archive -- According to the philosopher Ly Tin Weedle, chaos is found in greatest abundance wherever order is being sought. It always defeats order, because it is better organized.

3 3