Hi John,
Thanks for taking your time looking at my issue.
On 13-Jun-24 3:42 PM, John Stoffel via dovecot wrote:
>>>>>> "George" == George Asenov via dovecot <dovecot(a)dovecot.org> writes:
>
> I don't have a real suggestion, but I do think you can clarify your problem.
>
>> No one have idea what is wrong here?
>> On 07-Jun-24 4:10 PM, George Asenov via dovecot wrote:
>>> Hello,
>>>
>>> I have very strange issue. Sieve generate copies of users messages i.e.
>>> not real copies but hardlinks for the same message. It happens to many
>>> messages but not every message and not every time, it is not a single
>>> user issue I have couple users with that issue.
>
> Are you expecting sieve to generate copies? And are the copies in the
> same folder or across folders? I.e. do you find an email in the
> INBOX, and a hardlink in the SPAM folder?
The goal here is when user move messages around imap folders to report
them as ham/spam depending on where they originate and where they go.
I think Sieve should send copy of the message to Rspamd because
otherwise client will see the message dispensary which is not intuitive.
Copies are all in the same folder. same message with many hard links
placed in the same folder. Mostly happen in Trash and Spam folders.
> What are the sizes of these emails? Are they all large? Or have
> attachements? Is there anything that's common amongst those emails?
Nothing in common as far as I can see. Sizes are verry different mostly
small.
> One idea might be to setup a test account and to just send it a bunch
> of emails to try and make the problem occur. And to also look closely
> at the rspamd logs as well.
Already done that but I'm unable to reproduce it.
> What is the size of the system memory on your dovecot server? And
> what is the size of the dovecot.index.cache file when you see this
> error? You should be able to delete the index and recreate it using
> doveadm.
>
Servers are with enough RAM the smallest one is 64GB. The memory I'm
referring is "vsz_limit=256 MB" and in my opinion it is not the issue
but a consequence of the large amount of links in the mail and from
there large index file and the result not enough memory to work with.
When I delete all the copies the "Out of memory" errors are gone.
> But from the sound of it, you have users with many thousands of emails
> in a folder or folders. Can you check to see if there's any
> relationship between users with larger numbers of hardlinks and those
> with large numbers of emails?
There are users with large number of mails but only some of them have
this issue and some don't. Also there are users with small amount that
have it which make them with large number of mails(copies).
> And maybe instead of having sieve call rspam, maybe you can put it
> into a milter and just have the
>
Rspamd is on a different server and it is set as a milter in postfix.
That is why I use curl to report spam/ham.
>>> It happens during auto reporting for spam/ham with sieve.
>>> But I'm unable to reproduce it.
>>>
>>> At some point the hardlinks copies become so many that the mailbox index
>>> files become so bug that dovecot start throwing error:
>>> ################################
>>> dovecot[3385911]: imap(redacted(a)domain.tld)<1992901><RRBL9PQX69IXfCSs>:
>>> Error: Mailbox Junk: mmap(size=520636784) failed with file
>>> /var/lib/dovecot-virtualmin/index/redacted(a)domain.tld/.Junk/dovecot.index.cache: Cannot allocate memory
>>> ################################
>>> other relevant logs are:
>>>
>>> dovecot: imap-login: Login: user=<redacted.user>, method=PLAIN,
>>> rip=YYY.YYY.YYY.YYY, lip=XXX.XXX.XXX.XXX, mpid=3393763, TLS,
>>> session=<c1Z1lPsZuPCAWqqI>
>>> dovecot: imap(redacted.user)<3393763><c1Z1lPsZuPCAWqqI>: sieve: DEBUG:
>>> learn-spam.sieve was triggered on imap.cause=COPY:
>>> msgid=<87584056G78841203D85243127W62181551P@idomziqnd>
>>> dovecot: imap(redacted.user)<3393763><c1Z1lPsZuPCAWqqI>: sieve: DEBUG:
>>> learn-spam on imap.cause=COPY: from=redacted.mail, to=redacted2.mail,
>>> subject=Asseyez-vous confortablement, n'importe oĆ¹...,
>>> msgid=<87584056G78841203D85243127W62181551P@idomziqnd>,
>>> X-Spamd-Result=default: False [4.49 / 15.00];
>>> FORGED_RECIPIENTS(2.00)[m:redacted2.mail,s:redacted.user.fr];
>>> BAYES_SPAM(1.89)[88.30%]; MID_RHS_NOT_FQDN(0.50)[];
>>> BAD_REP_POLICIES(0.10)[]; RCVD_NO_TLS_LAST(0.10)[];
>>> MIME_GOOD(-0.10)[multipart/related,multipart/alternative,text/plain];
>>> ASN(0.00)[asn:34300, ipnet:62.173.128.0/19, country:RU];
>>> RCVD_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~,4:~,5:+];
>>> RCPT_COUNT_ONE(0.00)[1]; MISSING_XM_UA(0.00)[]; ARC_NA(0.00)[];
>>> RCVD_VIA_SMTP_AUTH(0.00)[]; GREYLIST(0.00)[pass,body];
>>> R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[];
>>> R_SPF_ALLOW(0.00)[+mx]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[or.mg];
>>> NEURAL_SPAM(0.00)[0.000]
>>> dovecot: imap(redacted.user)<3393763><c1Z1lPsZuPCAWqqI>: sieve: DEBUG:
>>> learn-spam send to rspamd spam
>>> dovecot: imap(redacted.user)<3393763><c1Z1lPsZuPCAWqqI>: program
>>> exec:/var/lib/dovecot/sieve/rspamd-learn-spam.sh (3397238): Terminated
>>> with non-zero exit code 1
>>> dovecot: imap(redacted.user)<3393763><c1Z1lPsZuPCAWqqI>: Error: sieve:
>>> failed to execute to program `rspamd-learn-spam.sh': refer to server log
>>> for more information. [2024-06-03 07:36:40]
>>> dovecot: imap(redacted.user)<3393763><c1Z1lPsZuPCAWqqI>: Disconnected:
>>> Connection closed (UID FETCH finished 32.173 secs ago) in=2914 out=39237
>>> deleted=1 expunged=1 trashed=0 hdr_count=14 hdr_bytes=10705 body_count=1
>>> body_bytes=1606
>>>
>>> I know that this is because the mail which is reported is too big for
>>> curl but documentation say that
>
> Wait, how large is this email you're trying to process? So once you
> have rspamd-learn-spam.sh crash on you, then you are really having an
> rspam problem. Do you really need to scan large attachements?
The "crash" is an error from curl because the mail is too big to send it
inline as I pipe it,
> What is your rspam configuration? And have you talked to people on
> the rspamd mailing list on how to configure things?
Rspamd is on a diffrent machine and I think it is not to blame here.
>
>>> $$$$$$$$$$$$$$$$$
>>> pipe :copy :try "rspamd-learn-spam.sh";
>>> $$$$$$$$$$$$$$$$$
>>> this should ignore the error.
>>> I have tested also to change it like that:
>>> $$$$$$$$$$$$$$$$$
>>> pipe :copyĀ "rspamd-learn-spam.sh";
>>> $$$$$$$$$$$$$$$$$
>>> but the issue still persists
>
> So why are you doing a :copy here? If you're trying to say this email
> is spam, why not just move it to your spam folder, and then have
> rspamd go through your junk folder once a day instead?
Because this should be kind of invisible to users. They move mails
between folders and in the background they report as spam/ham
> Have you looked at the system logs for memory problems when these
> hardlink files are created? Are you running out of memory on general
> on this mail server?
>
No memory problems on the machine itself there is plenty of free Ram
> Instead of focusing on dovecot, back up a level and think about the
> resources allocated to the whole system, and what types of mail
> messages are showing this problem. How are they alike? What is your
> mail volume? Does the problem happen when you get lots of emails, or
> is it really random?
It is not the amount of messages coming if I comment out the config
executing spam/ham reporting and clean up the hardlinks the issue didn't
show up for months.
> Do you have old rspamd processes hanging around taking up resources?
Rspamd can't make copies on that machine because it is on a separate server.
>
> Good luck!
> John
>
>
>
>
>>> I can't confirm that the issue is that error because these errors are
>>> way less than the messages with hardlink copies.
>>> Also sometimes one mail have more than two hardlinks to it.
>>> ########################################
>>> here is a example:
>>>
>>> inode# hardlink_count filename
>>> 2430090371 23850
>>> ./Maildir/.Trash/cur/1714190559.M355157P25776.redacted.hostname,S=39259,W=40217:2,S
>>> 2430090371 23850
>>> ./Maildir/.Trash/cur/1714190562.M259778P25902.redacted.hostname,S=39259,W=40217:2,S
>>> 2430090371 23850
>>> ./Maildir/.Trash/cur/1714190565.M188090P26028.redacted.hostname,S=39259,W=40217:2,S
>>> 2430090371 23850
>>> ./Maildir/.Trash/cur/1714190568.M340582P26179.redacted.hostname,S=39259,W=40217:2,S
>>>
>>> yes this is "23850" hardlinks to the same Inode
>>> #######################################
>>> The issue is somewhere in the sieve ham/spam reporting to rspamd but
>>> cant figure out where and why.
>>>
>>> Is this a bug or my configuration is wrong?
>>>
>>>
>>> Here are all related configurations (feel free to ask for more if needed):
>>> ######################################################################
>>> # doveconf -n
>>> # 2.3.16 (7e2e900c1a): /etc/dovecot/dovecot.conf
>>> # Pigeonhole version 0.5.16 (09c29328)
>>> # OS: Linux 4.18.0-513.24.1.el8_9.x86_64 x86_64 Rocky Linux release 8.9
>>> (Green Obsidian)
>>> # Hostname: redacteddomain.tld
>>> auth_mechanisms = plain login
>>> disable_plaintext_auth = no
>>> first_valid_uid = 1000
>>> mail_location =
>>> maildir:~/Maildir:INDEX=/var/lib/dovecot-virtualmin/index/%u:CONTROL=/var/lib/dovecot-virtualmin/control/%u
>>> managesieve_notify_capability = mailto
>>> managesieve_sieve_capability = fileinto reject envelope
>>> encoded-character vacation subaddress comparator-i;ascii-numeric
>>> relational regex imap4flags copy include variables body enotify
>>> environment mailbox date index ihave duplicate mime
>>> foreverypart extracttext vacation-seconds imapsieve vnd.dovecot.imapsieve
>>> mbox_write_locks = fcntl
>>> namespace inbox {
>>> Ā inbox = yes
>>> Ā location =
>>> Ā mailbox Drafts {
>>> Ā Ā Ā auto = subscribe
>>> Ā Ā Ā special_use = \Drafts
>>> Ā }
>>> Ā mailbox Junk {
>>> Ā Ā Ā auto = create
>>> Ā Ā Ā special_use = \Junk
>>> Ā }
>>> Ā mailbox Sent {
>>> Ā Ā Ā auto = subscribe
>>> Ā Ā Ā special_use = \Sent
>>> Ā }
>>> Ā mailbox "Sent Messages" {
>>> Ā Ā Ā special_use = \Sent
>>> Ā }
>>> Ā mailbox Trash {
>>> Ā Ā Ā auto = subscribe
>>> Ā Ā Ā special_use = \Trash
>>> Ā }
>>> Ā mailbox spam {
>>> Ā Ā Ā auto = subscribe
>>> Ā Ā Ā special_use = \Junk
>>> Ā }
>>> Ā prefix =
>>> }
>>> passdb {
>>> Ā driver = pam
>>> }
>>> plugin {
>>> Ā imapsieve_mailbox1_before = file:/var/lib/dovecot/sieve/learn-spam.sieve
>>> Ā imapsieve_mailbox1_causes = COPY
>>> Ā imapsieve_mailbox1_name = spam
>>> Ā imapsieve_mailbox2_before = file:/var/lib/dovecot/sieve/learn-ham.sieve
>>> Ā imapsieve_mailbox2_causes = COPY
>>> Ā imapsieve_mailbox2_from = spam
>>> Ā imapsieve_mailbox2_name = *
>>> Ā imapsieve_mailbox3_before = file:/var/lib/dovecot/sieve/learn-spam.sieve
>>> Ā imapsieve_mailbox3_causes = COPY
>>> Ā imapsieve_mailbox3_name = Junk
>>> Ā imapsieve_mailbox4_before = file:/var/lib/dovecot/sieve/learn-ham.sieve
>>> Ā imapsieve_mailbox4_causes = COPY
>>> Ā imapsieve_mailbox4_from = Junk
>>> Ā imapsieve_mailbox4_name = *
>>> Ā quota = fs:user userquota
>>> Ā quota2 = fs:group groupquota
>>> Ā quota_grace = 100M
>>> Ā quota_warning = storage=95%% quota-warning 95 %n %d
>>> Ā quota_warning2 = storage=90%% quota-warning 90 %n %d
>>> Ā quota_warning3 = storage=80%% quota-warning 80 %n %d
>>> Ā sieve = file:~/sieve;active=~/.dovecot.sieve
>>> Ā sieve_before = /var/lib/dovecot/sieve/before-global.sieve
>>> Ā sieve_extensions = +vacation-seconds
>>> Ā sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment
>>> +vnd.dovecot.debug
>>> Ā sieve_pipe_bin_dir = /var/lib/dovecot/sieve
>>> Ā sieve_plugins = sieve_extprograms sieve_imapsieve
>>> Ā sieve_vacation_default_period = 10d
>>> Ā sieve_vacation_max_period = 30d
>>> Ā sieve_vacation_min_period = 1h
>>> }
>>> protocols = imap pop3 lmtp sieve
>>> Ā process_min_avail = 5
>>> Ā service_count = 0
>>> }
>>> service imap {
>>> Ā process_limit = 400
>>> }
>>> service lmtp {
>>> Ā unix_listener /var/spool/postfix/private/dovecot-lmtp {
>>> Ā Ā Ā group = postfix
>>> Ā Ā Ā mode = 01224
>>> Ā Ā Ā user = postfix
>>> Ā }
>>> }
>>> service pop3 {
>>> Ā process_limit = 200
>>> }
>>> service quota-warning {
>>> Ā executable = script /var/lib/dovecot/quota-warning.sh
>>> Ā service_count = 1
>>> Ā unix_listener quota-warning {
>>> Ā Ā Ā group = dovecot
>>> Ā Ā Ā mode = 0666
>>> Ā Ā Ā user = dovecot
>>> Ā }
>>> }
>>> ssl_ca = </etc/pki/dovecot/certs/ca.pem
>>> ssl_cert = </etc/pki/dovecot/certs/dovecot.pem
>>> ssl_cipher_list =
>>> ECDHE-RSA-AES256-SHA384:AES256-SHA256:AES256-SHA256:!RC4:HIGH:MEDIUM:+TLSv1:+TLSv1.1:+TLSv1.2:!MD5:!ADH:!aNULL:!eNULL:!NULL:!DH:!ADH:!EDH:!AESGCM
>>> ssl_key = # hidden, use -P to show it
>>> userdb {
>>> Ā driver = passwd
>>> }
>>> protocol lmtp {
>>> Ā mail_plugins = " sieve"
>>> Ā userdb {
>>> Ā Ā Ā args = username_format=%n /etc/passwd
>>> Ā Ā Ā driver = passwd-file
>>> Ā Ā Ā name =
>>> Ā }
>>> }
>>> protocol imap {
>>> Ā mail_plugins = " imap_quota imap_sieve quota"
>>> }
>>>
>>> ######################################################################
>>>
>>> # cat /var/lib/dovecot/sieve/learn-spam.sieve
>>> require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment",
>>> "imap4flags", "vnd.dovecot.debug", "variables"];
>>>
>>> # Logging
>>> if address :matches "from" "*" { set "FROM" "${1}"; }
>>> if address :matches "to" "*" { set "TO" "${1}"; }
>>> if header :matches "subject" "*" { set "SUBJECT" "${1}"; }
>>> if header :matches "Message-ID" "*" { set "MSGID" "${1}"; }
>>> if header :matches "X-Spamd-Result" "*" { set "XSpamdResult" "${1}"; }
>>> if environment :matches "imap.cause" "*" { set "IMAPCAUSE" "${1}"; }
>>> debug_log "learn-spam.sieve was triggered on imap.cause=${IMAPCAUSE}:
>>> msgid=${MSGID}";
>>> set "LogMsg" "learn-spam on imap.cause=${IMAPCAUSE}: from=${FROM},
>>> to=${TO}, subject=${SUBJECT}, msgid=${MSGID},
>>> X-Spamd-Result=${XSpamdResult}";
>>>
>>> # Spam-learning by sending copy with curl to rspmd
>>> if anyof (environment :is "imap.cause" "COPY", environment :is
>>> "imap.cause" "APPEND") {
>>> Ā Ā Ā debug_log "${LogMsg}";
>>> Ā Ā Ā debug_log "learn-spam send to rspamd spam";
>>> Ā Ā Ā pipe :copy :try "rspamd-learn-spam.sh";
>>> }
>>> # Catch replied or forwarded spam and send to rspamd ham
>>> elsif anyof (allof (hasflag "\\Answered", environment :contains
>>> "imap.changedflags" "\\Answered"),
>>> Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā allof (hasflag "$Forwarded", environment :contains
>>> "imap.changedflags" "$Forwarded")) {
>>> Ā Ā Ā debug_log "${LogMsg}";
>>> Ā Ā Ā debug_log "learn-spam send to rspamd ham";
>>> Ā Ā Ā pipe :copy :try "rspamd-learn-ham.sh";
>>> }
>>> ######################################################################
>>>
>>> cat /var/lib/dovecot/sieve/learn-ham.sieve
>>> require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment",
>>> "variables", "vnd.dovecot.debug"];
>>>
>>> # Exclude messages which were moved to Trash (or training mailboxes)
>>> from ham learning
>>> if environment :matches "imap.mailbox" "*" {
>>> Ā Ā Ā set "mailbox" "${1}";
>>> }
>>> if string "${mailbox}" [ "INBOX/Trash", "INBOX/Deleted Items",
>>> "INBOX/Bin", "INBOX/train_ham", "INBOX/train_prob", "INBOX/train_spam",
>>> "Trash" ] {
>>> Ā Ā Ā stop;
>>> }
>>>
>>> # Logging
>>> if address :matches "from" "*" { set "FROM" "${1}"; }
>>> if address :matches "to" "*" { set "TO" "${1}"; }
>>> if header :matches "subject" "*" { set "SUBJECT" "${1}"; }
>>> if header :matches "Message-ID" "*" { set "MSGID" "${1}"; }
>>> if header :matches "X-Spamd-Result" "*" { set "XSpamdResult" "${1}"; }
>>> if environment :matches "imap.cause" "*" { set "IMAPCAUSE" "${1}"; }
>>> debug_log "learn-ham on imap.cause=${IMAPCAUSE}: from=${FROM}, to=${TO},
>>> subject=${SUBJECT}, msgid=${MSGID}, X-Spamd-Result=${XSpamdResult}";
>>>
>>> # Ham-learning sending a copy of the message to rspamd
>>> debug_log "learn-ham send to rspamd ham";
>>> pipe :copy :try "rspamd-learn-ham.sh";
>>> ######################################################################
>>>
>>> cat /var/lib/dovecot/sieve/rspamd-learn-ham.sh
>>> #!/bin/bash
>>>
>>> function log_error() { echo `date '+%Y-%m-%d %H:%M:%S'` ERROR: $1 >&2; }
>>> function log() { echo `date '+%Y-%m-%d %H:%M:%S'` INFO: $1; }
>>>
>>> MAIL=$(tee)
>>> SERVER_LIST=("redacted.tld") #rspamd server
>>>
>>> PASSWORD="redacted"
>>>
>>>
>>> for SERVER in ${SERVER_LIST[@]}; do
>>> Ā Ā Ā log "Trying to report spam to ${SERVER}"
>>> Ā Ā Ā RETURN=$(/usr/bin/curl -s --connect-timeout 1 -H "Password:
>>> ${PASSWORD}" --data-binary --url "http://${SERVER}:11334/learnham" -d
>>> "${MAIL}")
>>> Ā Ā Ā STATUS=$?
>>> Ā Ā Ā if [ $STATUS -eq 0 ]; then
>>> Ā Ā Ā Ā Ā Ā Ā log "Spam reported to ${SERVER}: ${RETURN}"
>>> Ā Ā Ā Ā Ā Ā Ā exit 0
>>> Ā Ā Ā else
>>> Ā Ā Ā Ā Ā Ā Ā if [ $STATUS -eq 28 ]; then
>>> Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā log_error "Reporting SPAM to ${SERVER} failed: Connection
>>> timed out."
>>> Ā Ā Ā Ā Ā Ā Ā else
>>> Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā log_error "Reporting SPAM to ${SERVER} failed: CURL exit
>>> status ${STATUS}"
>>> Ā Ā Ā Ā Ā Ā Ā fi
>>> Ā Ā Ā fi
>>> done
>>>
>>> log_error "Reporting SPAM failed ${SERVERS[@]}"
>>> exit 1
>>> ######################################################################
>>>
>>> Ā cat /var/lib/dovecot/sieve/rspamd-learn-spam.sh
>>> #!/bin/bash
>>>
>>> function log_error() { echo `date '+%Y-%m-%d %H:%M:%S'` ERROR: $1 >&2; }
>>> function log() { echo `date '+%Y-%m-%d %H:%M:%S'` INFO: $1; }
>>>
>>> MAIL=$(tee)
>>> SERVER_LIST=("redacted.tld") #rspamd server
>>> PASSWORD="redacted"
>>>
>>>
>>>
>>> for SERVER in ${SERVER_LIST[@]}; do
>>> Ā Ā Ā log "Trying to report spam to ${SERVER}"
>>> Ā Ā Ā RETURN=$(/usr/bin/curl -s --connect-timeout 1 -H "Password:
>>> ${PASSWORD}" --data-binary --url "http://${SERVER}:11334/learnspam" -d
>>> "${MAIL}")
>>> Ā Ā Ā STATUS=$?
>>> Ā Ā Ā if [ $STATUS -eq 0 ]; then
>>> Ā Ā Ā Ā Ā Ā Ā log "Spam reported to ${SERVER}: ${RETURN}"
>>> Ā Ā Ā Ā Ā Ā Ā exit 0
>>> Ā Ā Ā else
>>> Ā Ā Ā Ā Ā Ā Ā if [ $STATUS -eq 28 ]; then
>>> Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā log_error "Reporting SPAM to ${SERVER} failed: Connection
>>> timed out."
>>> Ā Ā Ā Ā Ā Ā Ā else
>>> Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā Ā log_error "Reporting SPAM to ${SERVER} failed: CURL exit
>>> status ${STATUS}"
>>> Ā Ā Ā Ā Ā Ā Ā fi
>>> Ā Ā Ā fi
>>> done
>>>
>>> log_error "Reporting SPAM failed ${SERVERS[@]}"
>>> exit 1
>>> ######################################################################
>>>
>>> without all "imapsieve_mailbox.* " directives the problem is gone.
>>>
>>>
>>> Thanks to everyone in advance.
>>>
>
>> --
>> Warm regards
>> George A.
>> WPXHosting
>> _______________________________________________
>> dovecot mailing list -- dovecot(a)dovecot.org
>> To unsubscribe send an email to dovecot-leave(a)dovecot.org
> _______________________________________________
> dovecot mailing list -- dovecot(a)dovecot.org
> To unsubscribe send an email to dovecot-leave(a)dovecot.org
--
Warm regards
George A.