Virus scan + removal on a mdbox mail storage
Hi,
I need advice on how virus scan and removal can be done on a _mdbox_
mail storage?
On a maildir storage the virus scanner (e.g. clamav etc.) can detect
and remove a email that is infected, since every email and attachment
are stored in separate files.
But in mdbox the emails and attachments are compressed together in one
ore more mdbox-files ...
I am anxious to convert my mail storage for virus scanning into
maildir format, since I don't know if a virus or crypto trojan con be
activated with this converting action =:-o
Cheers Christoph.
-- Christoph Haas
On 2019-02-20 01:46, Christoph Haas via dovecot wrote:
I need advice on how virus scan and removal can be done on a _mdbox_ mail storage?
On a maildir storage the virus scanner (e.g. clamav etc.) can detect and remove a email that is infected, since every email and attachment are stored in separate files.
But in mdbox the emails and attachments are compressed together in one ore more mdbox-files ...
I am anxious to convert my mail storage for virus scanning into maildir format, since I don't know if a virus or crypto trojan con be activated with this converting action =:-o
To clarify: You want to convert your mail storage from mdbox to maildir, but you want to scan for viruses first?
You are doing things in the wrong order.
Firstly converting mail storage format is very unlikely to trigger a virus. For that to happen the virus author would need to find and write an exploit for dovecot that will trick it into treating email as executable code. While not impossible that is quite unlikely because there is no normal situation where dovecot will execute email as code. Also it is unlikely that a virus writer will target dovecot when Microsoft exchange is much more common and would be a higher value target.
Secondly, as a rule you want to scan email for viruses as it arrives and leaves, not when it is at rest in user mailboxes, again it is possible that a new virus will be discovered some time after the email arrives so a retrospective scan would find it, but that won't help you much because most users read their email and open attachments soon after the email arrives.
So my advice is to do the conversion to maildir now, then scan all the files as a one off, and going forward you should configure your email transport daemon (postfix, exim etc) to pass incoming (and possibly outgoing) email through clamav.
-- David Pottage
Hello David,
----- Nachricht von David Pottage via dovecot dovecot@dovecot.org --------- Datum: Wed, 20 Feb 2019 14:56:51 +0000 Von: David Pottage via dovecot dovecot@dovecot.org Antwort an: David Pottage david@chrestomanci.org Betreff: Re: Virus scan + removal on a mdbox mail storage An: dovecot@dovecot.org
On 2019-02-20 01:46, Christoph Haas via dovecot wrote:
I need advice on how virus scan and removal can be done on a _mdbox_ mail storage?
On a maildir storage the virus scanner (e.g. clamav etc.) can detect and remove a email that is infected, since every email and attachment are stored in separate files.
But in mdbox the emails and attachments are compressed together in one ore more mdbox-files ...
I am anxious to convert my mail storage for virus scanning into maildir format, since I don't know if a virus or crypto trojan con be activated with this converting action =:-o
To clarify: You want to convert your mail storage from mdbox to
maildir, but you want to scan for viruses first?
NO! My mail storage is mdbox. And at the moment I have no intention to
convert it to Maildir!
But I know, that virus detection and deletion is much easier with
Maildir, since every mail is represented by a file. So if there is one
mail infected, the file can easily deleted - also by external
antivirus tools. Also there are no indices with Maildir.
On the opposite in the mdbox mail storage several mails are
represented by one mdbox-file, so I'm looking for a way to detect and
if necessary remove infected mails without damaging my mdbox storage
or the indices.
One idea was to convert the mdbox storage for virus scanning on the
fly to Maildir do the antivirus stuff and then vice versa. But this
produces quite a lot of overhead ...
--> so I need a better way
You are doing things in the wrong order.
Firstly converting mail storage format is very unlikely to trigger a
virus. For that to happen the virus author would need to find and
write an exploit for dovecot that will trick it into treating email
as executable code. While not impossible that is quite unlikely
because there is no normal situation where dovecot will execute
email as code. Also it is unlikely that a virus writer will target
dovecot when Microsoft exchange is much more common and would be a
higher value target.Secondly, as a rule you want to scan email for viruses as it arrives
and leaves, not when it is at rest in user mailboxes, again it is
possible that a new virus will be discovered some time after the
email arrives so a retrospective scan would find it, but that won't
help you much because most users read their email and open
attachments soon after the email arrives.
I'm completely with you! I have of course configured my postfix with
Amavisd-new and all that stuff. But viruses evolve quite faster than
detection patterns of e.g. Clam-AV.
So it is likely, that Clam-AV didn't detect a virus when scanning the
mail-traffic on arrival and the malware now resides in the
mdbox-storage.
For this situation an afterward virus scan of the existing mail
storage on a regular basis seems to me an appropriate method to get
rid of viruses, trojans etc. that were not detected on arrival and
reside like a time bomb in my mail storage...
Btw.: what virus scanners besides Clam-AV are the people on this list
using? And how is the virus scanner implemented: via Amavisd-new or
e.g. rspamd or ...?
- I hope this question is not too offtopic for the dovecot list!
So my advice is to do the conversion to maildir now, then scan all
the files as a one off, and going forward you should configure your
email transport daemon (postfix, exim etc) to pass incoming (and
possibly outgoing) email through clamav.-- David Pottage
----- Ende der Nachricht von David Pottage via dovecot
dovecot@dovecot.org -----
Cheers Christoph.
P.S.: excuse my English - I'm no native speaker ...
-- Christoph Haas
On 2019-02-20 19:02, Christoph Haas via dovecot wrote:
On 2019-02-20 01:46, Christoph Haas via dovecot wrote:
I need advice on how virus scan and removal can be done on a _mdbox_ mail storage?
On a maildir storage the virus scanner (e.g. clamav etc.) can detect and remove a email that is infected, since every email and attachment are stored in separate files.
But in mdbox the emails and attachments are compressed together in one ore more mdbox-files ...
I am anxious to convert my mail storage for virus scanning into maildir format, since I don't know if a virus or crypto trojan con be activated with this converting action =:-o
To clarify: You want to convert your mail storage from mdbox to
maildir, but you want to scan for viruses first?NO! My mail storage is mdbox. And at the moment I have no intention to convert it to Maildir! [snip]
Could I ask why? maildir is a better storage format is almost every respect.
You are doing things in the wrong order.
Firstly converting mail storage format is very unlikely to trigger a
virus. For that to happen the virus author would need to find and
write an exploit for dovecot that will trick it into treating email
as executable code. While not impossible that is quite unlikely
because there is no normal situation where dovecot will execute email as code. Also it is unlikely that a virus writer will target dovecot when Microsoft exchange is much more common and would be a higher value target.Secondly, as a rule you want to scan email for viruses as it arrives
and leaves, not when it is at rest in user mailboxes, again it is
possible that a new virus will be discovered some time after the
email arrives so a retrospective scan would find it, but that won't
help you much because most users read their email and open
attachments soon after the email arrives.I'm completely with you! I have of course configured my postfix with Amavisd-new and all that stuff. But viruses evolve quite faster than detection patterns of e.g. Clam-AV.
So it is likely, that Clam-AV didn't detect a virus when scanning the mail-traffic on arrival and the malware now resides in the mdbox-storage.
For this situation an afterward virus scan of the existing mail storage on a regular basis seems to me an appropriate method to get rid of viruses, trojans etc. that were not detected on arrival and reside like a time bomb in my mail storage...
The thing is that users will usually open emails shortly after they arrive. Most emails are not opened again later, especially the attachments.
So if a virus laden email got through because the definitions for your anti-virus solution where not updated in time, then it is fairly likely that the user's desktop computer is now infected (the endpoint). To fix that risk, you need a traditional endpoint virus scanner. In the unlikely event that a user opens an attachment in an old email, then their endpoint security will also intervene and prevent an infection.
In other words, it all comes back to endpoint security. Without it you are very prone to a virus infection. Scanning incoming email is helpful to reduce noise and inconvenience, but it is not a substitute for endpoint security, as in any case users can be infected in plenty of other ways, such as booby trapped websites or infected USB keys that they bring into the office.
Btw.: what virus scanners besides Clam-AV are the people on this list using? And how is the virus scanner implemented: via Amavisd-new or e.g. rspamd or ...?
- I hope this question is not too offtopic for the dovecot list!
You are right, that is a little offtopic. It is realy a postfix question.
For my day job I work for Sophos (A cyber security vendor), so all this is familiar to me. If you have the budget for a commercial product, then Sophos PureMessage does have postfix support. Technical details here:
https://docs.sophos.com/msg/pmx/help/en-us/msg/pmx/tasks/GSGConfigExtPostfix...
Other AV vendors probably have similar support, but I don't know any details.
-- David Pottage
Hello David,
----- Nachricht von David Pottage via dovecot dovecot@dovecot.org --------- Datum: Thu, 21 Feb 2019 13:58:14 +0000 Von: David Pottage via dovecot dovecot@dovecot.org Antwort an: David Pottage david@chrestomanci.org Betreff: Re: Virus scan + removal on a mdbox mail storage An: dovecot@dovecot.org
[...]
NO! My mail storage is mdbox. And at the moment I have no intention to convert it to Maildir! Could I ask why? maildir is a better storage format is almost every respect.
well, I have a mailbox with about 50k emails ..., so one reason seems
to me better backup performance with mdbox, since there are much less
files to save.
Another reason - you can beat me for this - it's more freaky ;-) - no,
just kidding ...
There was some years ago an interesting lecture from Peer Heinlein
about the mdbox mail storage, I afterwards bought his "Dovecot Buch"
of OpenSource Press and sticked to mdbox.
But I'll test backup of my mail storage converted to Maildir (which
can easily be done thanks dsync)
- If there are no significant time difference, I might then change to Maildir.
[...]
The thing is that users will usually open emails shortly after they
arrive. Most emails are not opened again later, especially the
attachments.
you're right about this. And if a user has suspicions abaout a
possibly infected attachment, one can delete the whole email without
hassle.
[...]
For my day job I work for Sophos (A cyber security vendor), so all
this is familiar to me. If you have the budget for a commercial
product, then Sophos PureMessage does have postfix support.
Technical details here:https://docs.sophos.com/msg/pmx/help/en-us/msg/pmx/tasks/GSGConfigExtPostfix...
Other AV vendors probably have similar support, but I don't know any details.
-- David Pottage
I know about Sophos. Since my infrastructure is only for me and my
family, I'll use the SAV9-free package ... and will try to integrate
this with Postfix or AmaVisd.
----- Ende der Nachricht von David Pottage via dovecot
dovecot@dovecot.org -----
Christoph.
-- Christoph Haas
On 2019-02-21 22:14, Christoph Haas via dovecot wrote:
NO! My mail storage is mdbox. And at the moment I have no intention to convert it to Maildir! Could I ask why? maildir is a better storage format is almost every respect.
well, I have a mailbox with about 50k emails ..., so one reason seems to me better backup performance with mdbox, since there are much less files to save.
Assuming that you backup regularly then maildir is much better, because new emails show up as new files, while old ones are left unchanged. This means that an incremental backup only has to process new emails. With mailbox, the file for the entire folder changes every time a new email is delivered or the user edits any of them, so the whole mailbox needs to be backed up again, resulting in far more I/O and time.
The main disadvantage that I have heard is that maildir consumes a lot of inodes, but you can fix that by formatting your filesystem with more to begin with. The ext4 default is 16k per inode, but it is easy to set a different value when you format, or use an FS that does not suffer from that limitation such as XFS, (I use btrfs so I can snapshot my maildir)
[...]
For my day job I work for Sophos (A cyber security vendor), so all
this is familiar to me. If you have the budget for a commercial
product, then Sophos PureMessage does have postfix support. Technical details here:https://docs.sophos.com/msg/pmx/help/en-us/msg/pmx/tasks/GSGConfigExtPostfix...
Other AV vendors probably have similar support, but I don't know any details.
-- David Pottage
I know about Sophos. Since my infrastructure is only for me and my family, I'll use the SAV9-free package ... and will try to integrate this with Postfix or AmaVisd.
I am not certain, but I think the free versions of Sophos won't work as an email filter to avoid taking sales of the comercal product. I think you may have to stick with ClamAV, and just use Sophos on your endpoints.
-- David Pottage
On 2019-02-22, David Pottage via dovecot dovecot@dovecot.org wrote:
On 2019-02-21 22:14, Christoph Haas via dovecot wrote:
NO! My mail storage is mdbox. And at the moment I have no intention to convert it to Maildir! Could I ask why? maildir is a better storage format is almost every respect.
well, I have a mailbox with about 50k emails ..., so one reason seems to me better backup performance with mdbox, since there are much less files to save.
Assuming that you backup regularly then maildir is much better, because new emails show up as new files, while old ones are left unchanged. This means that an incremental backup only has to process new emails. With mailbox, the file for the entire folder changes every time a new email is delivered or the user edits any of them, so the whole mailbox needs to be backed up again, resulting in far more I/O and time.
It sounds like perhaps you're confusing mdbox with mbox. mdbox uses multiple files but not a single file per message. It is fairly sane for backup handling - depending on how you set things up, you can have it rotate after a fixed size, fixed time, or combination.
participants (3)
-
Christoph Haas
-
David Pottage
-
Stuart Henderson