[Dovecot] (Single instance) attachment storage
Now that v2.0.0 is only waiting for people to report bugs (and me to figure out how to fix them), I've finally had time to start doing what I actually came here (Portugal Telecom/SAPO) to do. :)
The idea is to have dbox and mdbox support saving attachments (or MIME parts in general) to separate files, which with some magic gives a possibility to do single instance attachment storage. Comments welcome.
Reading attachments
dbox metadata would contain entries like (this is a wrapped single line entry):
X1442 2742784 94/b2/01f34a9def84372a440d7a103a159ac6c9fd752b 2744378 27423 27/c8/a1dccc34d0aaa40e413b449a18810f600b4ae77b
So the format is:
"X" 1*(<offset> <byte count> <link path>)
So when reading a dbox message body, it's read as:
offset=0: <first 1442 bytes from dbox body> offset=1442: <next 2742784 bytes from external file> offset=2744226: <next 152 bytes from dbox body> offset=2744378: <next 27423 bytes from external file> offset=2744378 27423: <the rest from dbox body>
This is all done internally by creating a single istream that lazily opens the external files only when data is actually tried to be read from that part of the message.
The link paths don't have to be in any specific format. In future perhaps it can recognize different formats (even http:// urls and such).
Saving attachments separately
Message MIME structure is being parsed while message is saved. After each MIME part's headers are parsed, it's determined if this part should be stored into attachment storage. By default it only checks that the MIME part isn't multipart/* (because then its child parts would contain attachments). Plugins can also override this. For example they could try to determine if the commonly used clients/webmail always downloads and shows the MIME part when opening the mail (text/*, inline images, etc).
dbox_attachment_min_size specifies the minimum MIME part size that can be saved as an attachment. Anything smaller than that will be stored normally. While reading a potential attachment MIME part body, it's first buffered into memory until the min. size is reached. After that the attachment file is actually created and buffer flushed to it.
Each attachment filename contains a global UID part, so that no two (even identical) attachments will ever contain the same filename. But there can be multiple attachment storages in different mount points, and each one could be configured to do deduplication internally. So identical attachments should somehow be stored to same storage. This is done by taking a hash of the body and using a part of it as the path to the file. For example:
mail_location = dbox:~/dbox:ATTACHMENTS=/attachments/$/$
Each $ would be expanded to 8 bits of the hash in hex (00..ff). So the full path to an attachment could look like:
/attachments/04/f1/5ddf4d05177b3b4c7a7600008c4a11c1
Sysadmin can then create /attachment/00..ff as symlinks to different storages.
Hashing problems
Some problematic design decisions:
- Hash is taken from hardcoded first n kB vs. first
dbox_attachment_min_size bytes?
- With first n kB, dbox_attachment_min_size can be changed without
causing duplication of attachments, otherwise after the change the same attachment could get a hash to a different storage than before the change.
- If n kB is larger than dbox_attachment_min_size, it uses more memory.
- If n kB is determined to be too small to get uniform attachment
distribution to different storages, it can't be changed without recompiling.
- Hash is taken from first n bytes vs. everything?
- First n bytes are already read to memory anyway and can be hashed
efficiently. The attachment file can be created without wasting extra memory or disk I/O. If everything is hashed, the whole attachment has to be first stored to memory or to a temporary file and from there written to final storage.
- With first n bytes it's possible for an attacker to generate lots of
different large attachments that begin with the same bytes and then overflow a single storage. If everything is hashed with a secure hash function and a system-specific secret random value is added to the hash, this attack isn't possible.
I'm thinking that even though taking a hash of everything is the least efficient option, it's the safest option. It's pretty much guaranteed to give a uniform distribution across all storages, even against intentional attacks. Also the worse performance isn't probably that noticeable, especially assuming a system where local disk isn't used for storing mails, and the temporary files would be created there.
Single instance storage
All of the above assumes that if you want a single instance storage, you'll need to enable it in your storage. Now, what if you can't do that?
I've been planning on making all index/dbox code to use an abstracted out simple filesystem API rather than using POSIX directly. This work can be started by making the attachment reading/writing code use the FS API and then create a single instance storage FS plugin. The plugin would work like:
open(ha/sh/hash-guid): The destination storage is in ha/sh/ directory, so a new temp file can be created under it. The hash is part of the filename to make unlink() easier to handle.
Since the hash is already known at open() time, look up if hashes/<hash> file exists. If it does, open it.
write(): Write to the temp file. If hashes/ file is open, do a byte-by-byte comparison of the inputs. If there's a mismatch, close the hashes/ file and mark it as unusable.
finish(): a) If hashes/ file is still open and it's at EOF, link() it to our final destination filename and delete the temp file. If link() fails with ENOENT (it was just expunged), goto b. If link() fails with EMLINK (too many links), goto c. b) If hashes/ file didn't exist, link() the temp file to the hash and rename() it to the destination file. c) If the hashed file existed but wasn't the same, or if link() failed with EMLINK, link() our temp file to a second temp file and rename() it over the hashes/ file and goto a.
unlink(): If hashes/<hash> has the same inode as our file and the link count is 2, unlink() the hash file. After that unlink() our file.
One alternative to avoid using <hash> as part of the filename would be for unlink() to read the file and recalculate its hash, but that would waste disk I/O.
Another possibility would to be to not unlink() the hashes/ files immediately, but rather let some nightly cronjob to stat() through all of the files and unlink() the ones that have link count=1. This could be wastefully inefficient though.
Yet another possibility would be for the plugin to internally calculate the hash and write it somewhere. If it's at the beginning of the file, it could be read from there with some extra disk I/O. But is it worth it?..
Extra features
The attachment files begin with an extensible header. This allows a couple of extra features to reduce disk space:
The attachment could be compressed (header contains compressed-flag)
If base64 attachment is in a standardized form that can be 100%
reliably converted back to its original form, it could be stored decoded and then encoded back to original on the fly.
It would be nice if it was also possible to compress (and decompress) attachments after they were already stored. This would be possible, but it would require finding all the links to the message and recreating them to point to the new message. (Simply overwriting the file in place would require there are no readers at the same time, and that's not easy to guarantee, except if Dovecot was entirely stopped. I also considered some symlinking schemes but they seemed too complex and they'd also waste inodes and performance.)
Code status
Initial version of the attachment reading/writing code is already done and works (lacks some error handling and probably performance optimizations). The SIS plugin code is also started and should be working soon.
This code is very isolated and can't cause any destabilization unless it's enabled, so I'm thinking about just adding it to v2.0 as soon as it works, although the config file comments should indicate that it's still considered unstable.
On 7/19/2010 8:24 AM, Timo Sirainen wrote:
Now that v2.0.0 is only waiting for people to report bugs (and me to figure out how to fix them), I've finally had time to start doing what I actually came here (Portugal Telecom/SAPO) to do. :)
The idea is to have dbox and mdbox support saving attachments (or MIME parts in general) to separate files, which with some magic gives a possibility to do single instance attachment storage. Comments welcome.
YAAAY!!! Timo's gonna give us SIS!!!
Is it done yet :) ?
I'm just thinking out loud here - and I'm probably way off base. Just tell me to shut up and I'll go away and hide until you're finished.
- You've already identified that enabling this feature needs to avoid
introducing problems - including treating different-but-similar attachments as identical. In your hashing choices, you only mentioned attachment body. What about including size and date in the hash?
- You didn't explicitly define if SIS would be per-mailbox or
system-wide. Speaking for myself, and probably a few others, I'll take whatever implementation I can get - but I'd love to see it system-wide.
- Are you envisioning this as being handled totally within deliver, or
would there be a server process for consolidating the messages? I'm wondering about the impact to high-traffic sites (which mine is thankfully NOT) - if deliver needs to crunch on large messages, could this lead to time-out issues from the MTA's?
A possible alternative, have deliver write the message out as normal - but flag it for attachment processing. Then have a secondary process awakened to check for attachments and perform accordingly. So any SIS overhead becomes invisible to the MTA - other than needing available system resources for processing (and the attachment processing could be done at a lower priority).
On Mon, 2010-07-19 at 09:01 -0700, Daniel L. Miller wrote:
The idea is to have dbox and mdbox support saving attachments (or MIME parts in general) to separate files, which with some magic gives a possibility to do single instance attachment storage. Comments welcome.
YAAAY!!! Timo's gonna give us SIS!!!
Is it done yet :) ?
Well, there was a "code status" at the bottom of the mail :)
- You've already identified that enabling this feature needs to avoid
introducing problems - including treating different-but-similar attachments as identical. In your hashing choices, you only mentioned attachment body. What about including size and date in the hash?
Attachments don't have dates. Size could be included as part of the filename I guess.. Maybe it would even be a good idea..
- You didn't explicitly define if SIS would be per-mailbox or
system-wide. Speaking for myself, and probably a few others, I'll take whatever implementation I can get - but I'd love to see it system-wide.
System-wide. Of course permissions need to be properly set so all users can access them.
- Are you envisioning this as being handled totally within deliver, or
would there be a server process for consolidating the messages? I'm wondering about the impact to high-traffic sites (which mine is thankfully NOT) - if deliver needs to crunch on large messages, could this lead to time-out issues from the MTA's?
A possible alternative, have deliver write the message out as normal - but flag it for attachment processing. Then have a secondary process awakened to check for attachments and perform accordingly. So any SIS overhead becomes invisible to the MTA - other than needing available system resources for processing (and the attachment processing could be done at a lower priority).
Yeah, something like that would be possible. Or the attachment could still be stored to the attachment storage using the <hash>-<guid>[-<size>?] name and the daemon could then do the deduplication by finding any new files and seeing if they could be replaced with links to other existing files.
Timo Sirainen wrote:
Now that v2.0.0 is only waiting for people to report bugs (and me to figure out how to fix them), I've finally had time to start doing what I actually came here (Portugal Telecom/SAPO) to do. :)
The idea is to have dbox and mdbox support saving attachments (or MIME parts in general) to separate files, which with some magic gives a possibility to do single instance attachment storage. Comments welcome.
Cool.
Extra features
The attachment files begin with an extensible header. This allows a couple of extra features to reduce disk space:
- The attachment could be compressed (header contains compressed-flag)
Cool.
- If base64 attachment is in a standardized form that can be 100% reliably converted back to its original form, it could be stored decoded and then encoded back to original on the fly.
Cool.
I have thought about this issue in the past. What follows may be obvious to you already, but might as well mention rather than missing something.
Presumably you want to be able to recreate the original base64 stream exactly verbatim?
Under base64, the number of 4-byte (encoded) / 3-byte (decoded) cells per line is not fixed by the specs.
I believe the optimal value is 19 cells per line, but I have seen some systems use 18 cells per line, and I think I have seen 15 as well. Once you have three possibilities, you might as well just store the number of cells per line.
I would suggest considering the base64 format as (conceptually) having an (integer) parameter for the number of cells in each line (except for the last line).
So base64(19) would have on each line 19 cells encoding 57 (19 × 3) bytes into 76 (19 × 4) bytes.
Probably you would need to have a base64 matcher/decoder which is smarter than normal base64 decoders and checks to make sure that all lines (apart from the last) are encoded (a) canonically (e.g.. with no trailing whitespace), and (b) using the same number of cells per line.
The base64 matcher/decoder needs to return information about the cell count as well as the decoded data.
If any line is not canonical base64 or uses a different number of cells, then the base64 may still be valid but "weird" so would just be stored as the original base64 stream.
When recovering message data, obviously your base64 encoder needs to use a parameter which is the number of cells per line to encode. Then you get back your original base64 stream verbatim.
==
Some systems finish the base64 stream with a newline (which in a multipart manifests as a blank line between the base64 stream and the '--' of the MIME boundary), whereas some systems finish the base64 stream at the end of final 4-byte cell (which in a multipart manifests as the '--' of the MIME boundary appearing on the line immediately following the base64 encoded data). Your encoding allows for arbitrary data between the objects, so you would have no problem store these two cases verbatim. But something to watch out for when storing.
==
Maybe it would be a good idea to have the ability to say that an object was base64 decoded AND compressed (i.e. to recover the original stream fragment you need to decompress and base64 encode (with the relevant number of base64 cells per line)) --- as well as options for just base64 decoded or just compressed.
You could go nuts and say that it is an arbitrarily-sized filter stack, but my first guess would be that this would be too much flexibility.
It might be better to say that there can be zero or one decode/encode layers (like base64 or something else), and zero or one compression layers (like gzip or bzip2 or xz/LZMA).
Obviously whatever translations are required to recover the original stream should be encoded into the attachment file so that sysadmins can tune the storage algorithm without affecting previously stored attachments.
Bill
On Mon, 2010-07-19 at 17:29 +0100, William Blunn wrote:
- If base64 attachment is in a standardized form that can be 100%
reliably converted back to its original form, it could be stored decoded and then encoded back to original on the fly.
This is now done: http://hg.dovecot.org/dovecot-2.0-sis/rev/3ef0ac874fd7
Probably you would need to have a base64 matcher/decoder which is smarter than normal base64 decoders and checks to make sure that all lines (apart from the last) are encoded (a) canonically (e.g.. with no trailing whitespace), and (b) using the same number of cells per line.
Anything unexpected causes the attachment to be saved without decoding it.
Some systems finish the base64 stream with a newline (which in a multipart manifests as a blank line between the base64 stream and the '--' of the MIME boundary), whereas some systems finish the base64 stream at the end of final 4-byte cell (which in a multipart manifests as the '--' of the MIME boundary appearing on the line immediately following the base64 encoded data). Your encoding allows for arbitrary data between the objects, so you would have no problem store these two cases verbatim. But something to watch out for when storing.
I implemented this so that when end of base64 stream is encountered, it allows max. 1024 bytes of data after it. That data is saved in the dbox file instead of in the attachment file. So for example if the entire message body is a base64 encoded attachment but then some MTA appends a disclaimer after it, the attachment part is still saved to a separate file.
I added that "max 1024 bytes after" so that if there is some weird virus/spam/whatever attachment that claims to be base64 but then actually is mostly non-base64 data, it could take less space by saving the entire part as attachment rather than only the base64 data decoded.
Timo Sirainen wrote:
X1442 2742784 94/b2/01f34a9def84372a440d7a103a159ac6c9fd752b 2744378 27423 27/c8/a1dccc34d0aaa40e413b449a18810f600b4ae77b
So the format is:
"X" 1*(<offset> <byte count> <link path>)
...
Extra features
The attachment files begin with an extensible header. This allows a couple of extra features to reduce disk space:
The attachment could be compressed (header contains compressed-flag)
If base64 attachment is in a standardized form that can be 100%
reliably converted back to its original form, it could be stored decoded and then encoded back to original on the fly.
Consider storing the recovery filter stack in the dbox metadata rather than the attachment file.
e.g. so I put "-b64_19" after the file path to indicate that it needs to be exploded to base64 with 19 cells per line before being incorporated in the message stream.
X1442 2742784 94/b2/01f34a9def84372a440d7a103a159ac6c9fd752b -b64_19 2744378 27423 27/c8/a1dccc34d0aaa40e413b449a18810f600b4ae77b -b64_19
This means that the attachment file can be purely the attachment data.
This has a couple of upshots:
- If one person receives a message with an attachment which is encoded
with base64 at say 19 cells (76 bytes) per line, and then re-sends the same file as an attachment to someone else but their MUA encodes base64 at say 18 cells (72 bytes) per line, the attachment file can contain exactly the same data, allowing for deduplication even in this case.
- Assuming we have configured Dovecot to decode base64 but not to
compress, then the file in which we store the attachment data contains literally the exact same byte stream as if the attachment were saved out from the MUA. I don't know what practical use this might be, but it /sounds/ cool :-) Perhaps a suitable filesystem or backup-system could deduplicate both a file *and* its instance as a message attachment.
Bill
On Mon, 2010-07-19 at 18:30 +0100, William Blunn wrote:
Consider storing the recovery filter stack in the dbox metadata rather than the attachment file.
This has a couple of upshots:
- If one person receives a message with an attachment which is encoded
with base64 at say 19 cells (76 bytes) per line, and then re-sends the same file as an attachment to someone else but their MUA encodes base64 at say 18 cells (72 bytes) per line, the attachment file can contain exactly the same data, allowing for deduplication even in this case.
I thought about that also, but it would require calculating and using a hash of the decoded message (but not the compressed message). Could get complex.
- Assuming we have configured Dovecot to decode base64 but not to
compress, then the file in which we store the attachment data contains literally the exact same byte stream as if the attachment were saved out from the MUA. I don't know what practical use this might be, but it /sounds/ cool :-) Perhaps a suitable filesystem or backup-system could deduplicate both a file *and* its instance as a message attachment.
I was thinking about adding some small header to the dbox file, so they wouldn't be completely identical.
BTW. I was thinking about using "number of characters per base64 line" rather than "number of cells". I don't think it's required that line ends with a complete cell.
Timo Sirainen wrote:
BTW. I was thinking about using "number of characters per base64 line" rather than "number of cells". I don't think it's required that line ends with a complete cell.
You're right.
Looking at RFC2045, whitespace aside, the entire stream must be an integer quantity of 4 character cells. But the whitespace can appear anywhere.
So yes, number of characters per line. So my base64(19) becomes base64(76).
Bill
Timo Sirainen wrote:
On Mon, 2010-07-19 at 18:30 +0100, William Blunn wrote:
Consider storing the recovery filter stack in the dbox metadata rather than the attachment file.
This has a couple of upshots:
- If one person receives a message with an attachment which is encoded
with base64 at say 19 cells (76 bytes) per line, and then re-sends the same file as an attachment to someone else but their MUA encodes base64 at say 18 cells (72 bytes) per line, the attachment file can contain exactly the same data, allowing for deduplication even in this case.
I thought about that also, but it would require calculating and using a hash of the decoded message (but not the compressed message). Could get complex.
BTW I am not attempting to suggest a complete system for de-duplication, but rather to suggest a means by which it could be arranged that file contents became identical so that "something else" could de-duplicate them elsehow.
I would be interested to know what the hash you mention is needed for.
Also I would be interested to know why the hash of the fragment of the original message stream (regardless of base64 decodeability) would not be sufficient.
And if it isn't...
if (base64_smart_decode(&raw_data, &decoded_data, &chars_per_line) == SUCCESS) { // store decoded_data to attachment file // recovery_filter = "base64_" .concat. chars_per_line } else { // store raw_data to attachment file // recovery_filter = nothing }
// make hash of attachment file // store pointer to dbox metadata including recovery_filter
- Assuming we have configured Dovecot to decode base64 but not to
compress, then the file in which we store the attachment data contains literally the exact same byte stream as if the attachment were saved out from the MUA. I don't know what practical use this might be, but it /sounds/ cool :-) Perhaps a suitable filesystem or backup-system could deduplicate both a file *and* its instance as a message attachment.
I was thinking about adding some small header to the dbox file, so they wouldn't be completely identical.
Though that is kind of the point. If everything in the small header can go somewhere else then the small header can go away and we can store the attachment very literally.
What kind of things are you thinking to put in the small header?
Bill
On Mon, 2010-07-19 at 19:49 +0100, William Blunn wrote:
I thought about that also, but it would require calculating and using a hash of the decoded message (but not the compressed message). Could get complex.
BTW I am not attempting to suggest a complete system for de-duplication, but rather to suggest a means by which it could be arranged that file contents became identical so that "something else" could de-duplicate them elsehow.
I would be interested to know what the hash you mention is needed for.
If you rely on filesystem's deduplication, nothing. But if Dovecot does SIS internally, it needs the hash to see if the attachment is already stored.
Also I would be interested to know why the hash of the fragment of the original message stream (regardless of base64 decodeability) would not be sufficient.
If two users have the same file but with different base64-encoding, then their hashes are different and Dovecot can't do SIS.
I was thinking about adding some small header to the dbox file, so they wouldn't be completely identical.
Though that is kind of the point. If everything in the small header can go somewhere else then the small header can go away and we can store the attachment very literally.
What kind of things are you thinking to put in the small header?
I was thinking it would be nice to be able to compress attachments after they've already been delivered. Like maybe keep the attachments decoded for a few weeks and then compress them. Similar to how some people do it with Maildir. This can't work without a small header, otherwise you can't know if the attachment was originally compressed or not.
Timo Sirainen wrote:
I was thinking it would be nice to be able to compress attachments after they've already been delivered. Like maybe keep the attachments decoded for a few weeks and then compress them. Similar to how some people do it with Maildir. This can't work without a small header, otherwise you can't know if the attachment was originally compressed or not.
What about doing what people do with files and codify the compression status in the filename?
Assume the attachment file has a base name of "foo".
If you want to compress it, create a file "foo.gz" (or "foo.bz2" or "foo.xz"), and remove the original file "foo".
This assumes that if an open file is unlinked it will remain in existence until closed.
Code which wishes to read attachment files should first try to open the file using the base name. If the base name does not exist then it can try suffixes until it finds a file which opens. The suffix which worked then determines the decompression algorithm to be used.
The cost of an extra directory search (on messages which are understood to be historical in any case) should be dwarfed by the cost of decompressing the data.
This also solves the problem of how you compress a file in place. By creating the compressed version alongside, you never upset any readers.
This also helps out sysadmins because they would then be able to examine compressed attachment files straight off (zcat, bzcat, xzcat, et. al.) without having to think about any Dovecot-specific way of how the files might be laid out.
Bill
On Mon, 2010-07-19 at 16:24 +0100, Timo Sirainen wrote:
Code status
Initial version of the attachment reading/writing code is already done and works (lacks some error handling and probably performance optimizations). The SIS plugin code is also started and should be working soon.
The initial version can be tested here: http://hg.dovecot.org/dovecot-2.0-sis/
It should work with sdbox, but the settings parsing is causing mdbox to crash at startup. I need to figure out some way to fix that.
You can enable it by setting e.g.:
dbox_attachment_dir = ~/dbox/attachments
By default is does SIS, but if you want only external attachment storage, you can set:
dbox_attachment_fs = posix
(default is dbox_attachment_fs = sis posix)
TODO has:
- attachments can generate very long metadata lines. input stream reading them probably has a limit..
- save attachments base64-decoded
- if attachment stream is too small/long, log an error
- if file is completely empty, maybe show it as spaces? this could be useful for deleting viruses by truncating their files to zero bytes
- delayed deduplication daemon?
Perhaps the settings troubles could be solved simply by making the code to be common for all storage backends, even though currently only dbox would use it. Then the setting names would be mail_attachment_fs and mail_attachment_dir.
On Wed, 2010-07-21 at 21:19 +0100, Timo Sirainen wrote:
- delayed deduplication daemon?
Design:
Create a new "sis-delayed" backend for attachment fs. When creating new attachment files, it creates "hash-guid" files just like regular posix backend, but it also creates zero byte sized files with the same filenames under a different directory, e.g. creating a new attachment attachments/a8/f9/a8f91247218942-12247198278412 creates also zero sized attachments/delayed/a8f91247218942-12247198278412 file.
Deletion step works the same way as with fs-sis, so it deletes files from hashes/ directory automatically.
Everything else is a wrapper to super fs backend, just like with fs-sis.
A nightly run readdir()s through the delayed/ directory and processes each file it finds, deleting the file after processing it.
a) If the file is already gone from the attachment fs or its link count is larger than 1, it's skipped. b) If file's hash doesn't exist in hashes/ directory, link() the file to hashes/hash. Handle EMLINK the same as d) c) If byte-by-byte comparison finds that the file is the same as in hashes/ directory, replace file with the existing one: link() it to a temp file, make sure that the temp file's inode is still the same as the file we used for comparing, and then rename() it over the attachment file. d) Otherwise (hash collision/too many links), replace the hashes/ file with the new file: link() attachment to hashes/temp and rename() it to hashes/hash.
Simple, quite efficient, NFS safe. :) Some other thoughts:
It would be possible to have each server run this at the same time without doing duplicate work by before processing a file, rename() it under delayed/hostnames/<hostname>/ directory and later delete it from there. Also at startup go through any such files in the directory in case the previous run crashed. Perhaps also all hostname directories should be stat()ed once in a while and if their mtimes are too old, someone else could go look inside if there are any files and process them. The hostname/ directories could always be rmdired when the last file in them is gone.
So what binary/process should be doing this? "doveadm sis" command maybe?.. But would there be any other useful commands except for this deduplication? Maybe:
- doveadm sis deduplicate
- doveadm sis cleanup (for going through and deleting any files from
hashes/ directories that have link count=1)
- anything else?..
Hi
The idea is to have dbox and mdbox support saving attachments (or MIME parts in general) to separate files, which with some magic gives a possibility to do single instance attachment storage. Comments welcome.
This is a really interesting idea. I have previously given it some thought. My 2p
- Being able to ask "the server" if it has an attachment matching a
specific hash would be useful for a bunch of other reasons. This result needs to be (crytographically) unique and hence the hash needs to be a good hash (MD5/SHA or better) of the complete attachment, ideally after decoding
- It might be useful to be able to find attachments with a specific
hash regardless of whether the attachment has been spat out separately (think of a use case where we want to be able to spot a 2KB footer gif which on it's own isn't worth worrying about, but some offline scan later discovers 90% of emails contain this gif and we wish to split it off as a policy decision).
- Storing attachments by hash may be interesting for use with
specialist filesystems, eg an interesting direction that dbox could take might be to store the headers and message text in some (compressed?) format with high linear read rates and most attachments in a some key/value storage system?
- Many modern IMAP clients are starting to download attachments on
demand. Need to be able to supply only parts of the email efficiently without needing to pull in the blobs. Stated another way, it's desirable not to peek inside the blobs to be able to fetch arbitrary mime parts
It's going to be easy to break signed emails... Need to be careful
In many cases this isn't a performance win... It's still a *great*
feature, but two disk seeks outweigh a lot of linear read speed.
- When something gets corrupted... It's worth pondering about how we
can audit and find unreferenced "blobs" later?
Some of the use cases I have for these features (just in case you care...). We have a feature which is a bit like the opposite of one of these services for sending big attachments. When users email arrives we remove all attachments that meet our criteria and replace them with links to the files. This requires being able to give users a coded link which can later be decoded to refer to a specific attachment. If this change offered us additional ways to find attachments by hash or whatever then it would be extremely useful
Another feature we offer is a client application which compresses and reduces bandwidth when sending/receiving emails. We currently don't try and hash bits of email, but it's an idea I have been mulling over for IMAP users where we typically see the data sent via SMTP, then uploaded to the imap "sent items", then often downloaded again when the client polls the sent items for new messages (durr). Being able to see if we have binary content which matches a specific hash could be extremely interesting
I'm not sure if with your current proposal I can do 100% of the above?
For example it's not clear if 4) is still possible? Also without a
"guaranteed" hash we can't use the hash as a lookup key in a key/value
storage system (which implies another mapping of keys to keys is
required). Can we do an (efficient) offline scan of messages looking for
duplicated hash keys (ie can the server calculate hashes for all
attachment parts ahead of time)
Sounds extremely interesting. Look forward to seeing this develop!
Cheers
Ed W
On Tue, 2010-08-24 at 13:42 +0100, Ed W wrote:
Hi
The idea is to have dbox and mdbox support saving attachments (or MIME parts in general) to separate files, which with some magic gives a possibility to do single instance attachment storage. Comments welcome.
This is a really interesting idea. I have previously given it some thought. My 2p
- Being able to ask "the server" if it has an attachment matching a
specific hash would be useful for a bunch of other reasons.
If you have a hash 351641b73feb7cf7e87e5a8c3ca9a37d7b21e525, you can see if it exists with:
ls /attachments/35/16/hashes/351641b73feb7cf7e87e5a8c3ca9a37d7b21e525
This result needs to be (crytographically) unique and hence the hash needs to be a good hash (MD5/SHA or better) of the complete attachment,
Currently it uses SHA1, but this can be changed anytime. I didn't bother to make it configurable. The hash's security isn't a huge issue since it does byte-by-byte comparison anyway.
ideally after decoding
The hash is after decoding base64, if attachment is saved decoded, and that happens if it can be re-encoded exactly as it was.
- It might be useful to be able to find attachments with a specific
hash regardless of whether the attachment has been spat out separately (think of a use case where we want to be able to spot a 2KB footer gif which on it's own isn't worth worrying about, but some offline scan later discovers 90% of emails contain this gif and we wish to split it off as a policy decision).
I guess that would be possible, but it would require reading and parsing all of the mail files. That could take a while. The finding part wouldn't be all that much work, but separating attachments out of already saved mails is kind of annoying.
- Storing attachments by hash may be interesting for use with
specialist filesystems, eg an interesting direction that dbox could take might be to store the headers and message text in some (compressed?) format with high linear read rates and most attachments in a some key/value storage system?
The attachment I/O is done via filesystem API, so this would be possible easily by just writing FS API backend for a key-value database.
- Many modern IMAP clients are starting to download attachments on
demand. Need to be able to supply only parts of the email efficiently without needing to pull in the blobs. Stated another way, it's desirable not to peek inside the blobs to be able to fetch arbitrary mime parts
This is already done .. in theory anyway. I'm not sure yet if some prefetching code causes the attachments to be read unnecessarily. Should test it.
- It's going to be easy to break signed emails... Need to be careful
Yeah, I wasn't planning on breaking them.
- In many cases this isn't a performance win... It's still a *great*
feature, but two disk seeks outweigh a lot of linear read speed.
Sure, not a performance win. But that's not what it was meant for. :) But if only >1MB (or so) attachments were stored separately that should get rid of the worst offenders without impacting performance much.
- When something gets corrupted... It's worth pondering about how we
can audit and find unreferenced "blobs" later?
Dovecot logs an error when it finds something unexpected. But there's not a whole lot it can do then. And finding such broken attachments .. well, I guess this'll already do it:
doveadm fetch -A body all > /dev/null
Some of the use cases I have for these features (just in case you care...). We have a feature which is a bit like the opposite of one of these services for sending big attachments. When users email arrives we remove all attachments that meet our criteria and replace them with links to the files. This requires being able to give users a coded link which can later be decoded to refer to a specific attachment. If this change offered us additional ways to find attachments by hash or whatever then it would be extremely useful
I'm not sure if this change will help much. If the attachment changes (especially in size) there will be problems..
Another feature we offer is a client application which compresses and reduces bandwidth when sending/receiving emails. We currently don't try and hash bits of email, but it's an idea I have been mulling over for IMAP users where we typically see the data sent via SMTP, then uploaded to the imap "sent items", then often downloaded again when the client polls the sent items for new messages (durr). Being able to see if we have binary content which matches a specific hash could be extremely interesting
Related to that, I've been thinking of a transparent caching Dovecot proxy.
I'm not sure if with your current proposal I can do 100% of the above?
For example it's not clear if 4) is still possible? Also without a "guaranteed" hash we can't use the hash as a lookup key in a key/value storage system (which implies another mapping of keys to keys is required).
Yeah, attachment-instance-key -> attachment-key -> attachment data lookup would be the only safe way to do this.
Can we do an (efficient) offline scan of messages looking for duplicated hash keys (ie can the server calculate hashes for all attachment parts ahead of time)
Well .. the way it works is that you have files:
hash-guid hash2-guid2 hashes/hash hashes/hash2
If two attachments have the same hash but different content, you'll end up with:
hash-guid1 hash-guid2 hashes/hash
Where hash-guid1 and hash-guid2 are different files, and only one of them is hard linked to hashes/hash. To find duplicates, you can stat() all files and find which have identical hash but different inode.
Hi, thanks for responding
If you have a hash 351641b73feb7cf7e87e5a8c3ca9a37d7b21e525, you can see if it exists with:
ls /attachments/35/16/hashes/351641b73feb7cf7e87e5a8c3ca9a37d7b21e525
This would be great for a bunch of uses, if the hash were unique and determinable from just some other content. eg if I could take a random file from my filesystem, compute a hash from it, and then checking for the existence of /attachments/*/*/hashes/my_hash determines whether there are any messages referencing that attachment?
format with high linear read rates and most attachments in a some key/value storage system?
The attachment I/O is done via filesystem API, so this would be possible easily by just writing FS API backend for a key-value database.
Cool
- When something gets corrupted... It's worth pondering about how we
can audit and find unreferenced "blobs" later?
Dovecot logs an error when it finds something unexpected. But there's not a whole lot it can do then. And finding such broken attachments .. well, I guess this'll already do it:
I was actually pondering the next step where there is some kind of single instance storage, say hardlinks, and you want to avoid a final dangling reference where there are no emails referencing that attachment? Such an issue depends on certain implementations of the single instance storage which finally develops, but it seems like a common problem to several of the obvious ways to do it?
A related (but probably fairly rare) question would be (efficiently)
finding all the messages which reference an attachment with a given
unique hash. I can contrive a few reasons to ask this question, but
perhaps someone else will tell me they are dying to known this stuff?
(Find all emails which reference the company 2010 accounts pdf... Find
all emails from employees with an attachment that matches our shadow
password file...). Seems like we can do this just fine, only it will
involve a lot of stats right now?
links to the files. This requires being able to give users a coded link which can later be decoded to refer to a specific attachment. If this change offered us additional ways to find attachments by hash or whatever then it would be extremely useful
I'm not sure if this change will help much. If the attachment changes (especially in size) there will be problems..
A unique hash would allow me to give out very simple URL links to customers, eg http://mysite/attachments/SHA_Hash
At present I have a fairly convoluted scheme which uses message ids and this has a whole bunch of issues...
Related to that, I've been thinking of a transparent caching Dovecot proxy.
I could be interested in sponsoring such work if it got it higher up the ToDo list?! Please contact me with a proposal?
I'm not sure if with your current proposal I can do 100% of the above? For example it's not clear if 4) is still possible? Also without a "guaranteed" hash we can't use the hash as a lookup key in a key/value storage system (which implies another mapping of keys to keys is required).
Yeah, attachment-instance-key -> attachment-key -> attachment data lookup would be the only safe way to do this.
However, if the hash were a hash of the full message then we could completely avoid the double indirection? Within the limits of sensible probability, hashes can be considered unique and so there is "no" possibility of collision.
If you were to use highly unique hashes as your keys then you can dispense with certain levels of indirection here and hashes would be "guaranteed" unique if the content is unique. This allows you to do a straight compare of all hashes anywhere and two hashes the same == same content
(I'm caveating all "unique" claims in case someone points out that it's theoretically possible to get collisions)Well .. the way it works is that you have files:
hash-guid hash2-guid2 hashes/hash hashes/hash2
If two attachments have the same hash but different content, you'll end up with:
But if our hash were unique and based on the entire message then we should never get duplicated hash values? (obviously at the cost of more CPU and IO)
I sense I have misunderstood something, so please be gentle..?
Great feature anyway
Cheers
Ed W
On Tue, 2010-08-24 at 15:48 +0100, Ed W wrote:
Hi, thanks for responding
If you have a hash 351641b73feb7cf7e87e5a8c3ca9a37d7b21e525, you can see if it exists with:
ls /attachments/35/16/hashes/351641b73feb7cf7e87e5a8c3ca9a37d7b21e525
This would be great for a bunch of uses, if the hash were unique and determinable from just some other content. eg if I could take a random file from my filesystem, compute a hash from it, and then checking for the existence of /attachments/*/*/hashes/my_hash determines whether there are any messages referencing that attachment?
Yeah, that would work. And the */* part can also be optimized by just using the first 4 chars of the hash.
- When something gets corrupted... It's worth pondering about how we
can audit and find unreferenced "blobs" later?
Dovecot logs an error when it finds something unexpected. But there's not a whole lot it can do then. And finding such broken attachments .. well, I guess this'll already do it:
I was actually pondering the next step where there is some kind of single instance storage, say hardlinks, and you want to avoid a final dangling reference where there are no emails referencing that attachment? Such an issue depends on certain implementations of the single instance storage which finally develops, but it seems like a common problem to several of the obvious ways to do it?
Current implementation checks how many hard links are left for the hash while deleting it. If it's deleting the last reference then the final hashes/hash file is also deleted. It's of course possible that if it crashes between these two deletes then there are some dangling hashes left. I was thinking about maybe creating a tool to find and delete those. It's easily done by just deleting everything from /attachments/*/*/hashes/ directories that have a link count of 1.
A bigger problem is if a user's dbox directory is deleted/corrupted so that unused files are left in /attachments/*/*/ directory. Finding and deleting those pretty much requires reading through all dbox files for all users..
A related (but probably fairly rare) question would be (efficiently) finding all the messages which reference an attachment with a given unique hash.
Yeah. Not something I was planning on supporting.
I can contrive a few reasons to ask this question, but perhaps someone else will tell me they are dying to known this stuff?
(Find all emails which reference the company 2010 accounts pdf... Find all emails from employees with an attachment that matches our shadow password file...). Seems like we can do this just fine, only it will involve a lot of stats right now?
The mail -> attachment references are stored only inside dbox files in the metadata fields.
links to the files. This requires being able to give users a coded link which can later be decoded to refer to a specific attachment. If this change offered us additional ways to find attachments by hash or whatever then it would be extremely useful
I'm not sure if this change will help much. If the attachment changes (especially in size) there will be problems..
A unique hash would allow me to give out very simple URL links to customers, eg http://mysite/attachments/SHA_Hash
Ah. Yeah, that would work. You could just give the hash-guid reference so that the link will no longer work after the message gets deleted. Although the attachment hash-guid isn't available in any easy way, so you'd have to add some extra code.
Related to that, I've been thinking of a transparent caching Dovecot proxy.
I could be interested in sponsoring such work if it got it higher up the ToDo list?! Please contact me with a proposal?
The related parts are:
- Writing IMAP client lib-storage backend (possibly supporting, but not
requiring some Dovecot-specific extensions). But this can be annoyingly slow, because lib-storage API is synchronous. So it needs:
- Change lib-storage API to allow backends to be asynchronous. This is
wanted also for other high-latency backends, like key-value databases. It could also improve everyone elses' performance by adding async disk I/O support.
- Add a caching proxy lib-storage backend that supports transparent
caching messages and/or indexes for other lib-storage backends.
I'm interested in 2. anyway. Also 1 is probably a nice and easy way to test that it works, and 1 is also going to be nice for using dsync to migrate to Dovecot from other random IMAP servers :) Then 3 probably won't be all that difficult to implement. Anyway, I'm full time employed until the end of November, although I'm not exactly sure what I'll be working on soon..
I'm not sure if with your current proposal I can do 100% of the above? For example it's not clear if 4) is still possible? Also without a "guaranteed" hash we can't use the hash as a lookup key in a key/value storage system (which implies another mapping of keys to keys is required).
Yeah, attachment-instance-key -> attachment-key -> attachment data lookup would be the only safe way to do this.
However, if the hash were a hash of the full message then we could completely avoid the double indirection? Within the limits of sensible probability, hashes can be considered unique and so there is "no" possibility of collision.
The hash is already a full hash of the message. I don't really like the idea of trusting that a hash is unique. Especially because this could be attacked against. Someone could read another user's attachment if they only knew its hash and then were able to create another file with the same hash and send it to themselves in the same system. (Sure, this would require someone breaking SHA1. But the attachments with their SHA1 hashes could exist for many more years.)
I might make Dovecot trust the hash optionally anyway, but not unconditionally.
On Tue, 2010-08-24 at 11:36 -0500, Mike Abbott wrote:
Current implementation checks how many hard links are left for the hash
I haven't followed all the details and speculation, but this talk of hard-linking makes me wonder how SIS will work with mail stores spread across multiple file systems.
It'll work very nicely. The files are:
attachments/ha/sh/hash-guid, which is a hard link to: attachments/ha/sh/hashes/hash
So it's always a hard link under a child hashes/ directory. The attachments/* or attachments/*/* directories can point to any number of different mount points. Currently it's hard coded to two directory levels, allowing max. 65536 different dirs/mount points. I don't know if I should bother making this configurable.
On Tue, 2010-08-24 at 17:43 +0100, Timo Sirainen wrote:
On Tue, 2010-08-24 at 11:36 -0500, Mike Abbott wrote:
Current implementation checks how many hard links are left for the hash
I haven't followed all the details and speculation, but this talk of hard-linking makes me wonder how SIS will work with mail stores spread across multiple file systems.
It'll work very nicely.
Also if you don't like the SIS implementation, it's simple to change it. There are two separate implementations already (immediate and delayed). And they don't take much code:
dovecot-2.0-sis/src/lib-fs% wc -l fs-sis*[ch] 431 fs-sis.c 58 fs-sis-common.c 14 fs-sis-common.h 322 fs-sis-queue.c
On 24.8.2010, at 22.53, Mike Abbott wrote:
attachments/ha/sh/hash-guid, which is a hard link to: attachments/ha/sh/hashes/hash
If I store, say, all students' mail directories on one file system, and all teachers' mail on another, will your SIS store one copy of each attachment, or one copy per file system?
It depends on your configuration.. The attachment directory is a setting. I was thinking that it it would typically be the same for all users, so if you have two filesystems, you'd need to decide which one will have the /attachments directory. Or you could put half the attachments to fs1 and the rest to fs2 (/attachments/00 .. 7f to fs1, /attachments/80 .. ff to fs2). Or something like that.
If you really want to separate students' and teachers' attachments to separate filesystems, you'll then have userdb return a separate attachment directory for either students or teachers. Then they won't share anything and Dovecot doesn't even know the existence of the other attachment storage.
On 8/24/2010 4:19 PM, Timo Sirainen wrote:
It depends on your configuration.. The attachment directory is a setting. I was thinking that it it would typically be the same for all users, so if you have two filesystems, you'd need to decide which one will have the /attachments directory.
Dunno if I can come up with a use case immediately, but I'll bet someone will. Would making the attachments folder a userdb option be a pain?
On Fri, 2010-08-27 at 09:41 -0700, Daniel L. Miller wrote:
On 8/24/2010 4:19 PM, Timo Sirainen wrote:
It depends on your configuration.. The attachment directory is a setting. I was thinking that it it would typically be the same for all users, so if you have two filesystems, you'd need to decide which one will have the /attachments directory.
Dunno if I can come up with a use case immediately, but I'll bet someone will. Would making the attachments folder a userdb option be a pain?
You can already override any setting from userdb, including mail_attachment_dir.
On 24/08/2010 16:48, Timo Sirainen wrote:
Current implementation checks how many hard links are left for the hash while deleting it. If it's deleting the last reference then the final hashes/hash file is also deleted.
I sense an interesting race possibility here?
The hash is already a full hash of the message. I don't really like the idea of trusting that a hash is unique.
If SHA-1 becomes breakable in sensible time then you have a whole host of other attack vectors right now. I believe your mercurial repo is using SHA-1 hashes to detect tampering for example? (Also SSL, TLS, PGP, SSH and a bunch of other rarely used applications...)
At the moment SHA-256 is considered "good enough for the US government". SHA-3 should be out in a couple of years
Especially because this could be attacked against. Someone could read another user's attachment if they only knew its hash and then were able to create another file with the same hash and send it to themselves in the same system.
I can't argue that unknown security issues won't be found, because you can only talk about the known ones by definition...
That said I don't see that you can ever solve the de-duplicating problem if you don't trust your hash algorithm? At some point you are going to bite the bullet and say that attachment A and B have the same hash so lets hard link them together? At that point you are vulnerable to someone pulling off some way to disrupt your system if they can figure out how to generate attachments with arbitrary hashes?
At the moment I would claim that you are just automatically generating a very complicated filename. If you never trust your hash then you might as well instead simply use one of the existing GUID algorithms, if you trust your hash then you use that. I don't really see the point of a halfway house really?
I might make Dovecot trust the hash optionally anyway, but not unconditionally.
I don't really see how you can get around trusting the hashes at some point if you are de-duping?
SHA-1 will become breakable at some point for certain. I don't think that makes trusting SHA-1 hashes useless though. Various programming techniques can still be used to push out the life of this technique quite a bit further. For example:
- Compute relatively cheap secondary hash, eg even CRC32. Causing a
collision in two hashes is likely to be more difficult than a single hash
- Check attachment length. Likely this will make it harder to generate
a collision
- You already commented that it's reasonably hard to access the hash in
the first place (caveat idiots like me...)
- Use SHA-2 or some other hash, as of right now there are no attacks
against SHA-2, likely it has a few years life..?
Just a thought?
Cheers
Ed W
On 24.8.2010, at 23.16, Ed W wrote:
On 24/08/2010 16:48, Timo Sirainen wrote:
Current implementation checks how many hard links are left for the hash while deleting it. If it's deleting the last reference then the final hashes/hash file is also deleted.
I sense an interesting race possibility here?
Yes, but it doesn't matter much. The worst that can happen is that the file gets duplicated about once (because hashes/file gets deleted too early).
The hash is already a full hash of the message. I don't really like the idea of trusting that a hash is unique.
If SHA-1 becomes breakable in sensible time then you have a whole host of other attack vectors right now.
But not Dovecot itself.
I believe your mercurial repo is using SHA-1 hashes to detect tampering for example? (Also SSL, TLS, PGP, SSH and a bunch of other rarely used applications...)
Yeah, but those aren't Dovecot itself. :)
I can't argue that unknown security issues won't be found, because you can only talk about the known ones by definition...
That said I don't see that you can ever solve the de-duplicating problem if you don't trust your hash algorithm? At some point you are going to bite the bullet and say that attachment A and B have the same hash so lets hard link them together? At that point you are vulnerable to someone pulling off some way to disrupt your system if they can figure out how to generate attachments with arbitrary hashes?
By default Dovecot will do byte-by-byte comparison of the data before hard linking them together. It's not trusting the hash, it's only using it to quickly find potential files for deduplication.
BTW. http://valerieaurora.org/review/hash.html talks about this same thing.
On 24.8.2010, at 23.16, Ed W wrote:
At the moment I would claim that you are just automatically generating a very complicated filename. If you never trust your hash then you might as well instead simply use one of the existing GUID algorithms, if you trust your hash then you use that. I don't really see the point of a halfway house really?
Oh and this current scheme of hash-guid + hashes/hash hard linking is required in any case to keep track of reference counting. Unconditionally trusting the hash wouldn't make it any simpler. With key-value databases you'd have to figure out some other way to keep track of how many references there are to the attachment.
On 8/24/2010 4:35 PM, Timo Sirainen wrote:
On 24.8.2010, at 23.16, Ed W wrote:
At the moment I would claim that you are just automatically generating a very complicated filename. If you never trust your hash then you might as well instead simply use one of the existing GUID algorithms, if you trust your hash then you use that. I don't really see the point of a halfway house really?
Oh and this current scheme of hash-guid + hashes/hash hard linking is required in any case to keep track of reference counting. Unconditionally trusting the hash wouldn't make it any simpler. With key-value databases you'd have to figure out some other way to keep track of how many references there are to the attachment.
Can you append some "trivial" information from the data file to the hash in generating the file name to help ensure uniqueness? Like filesize, mimetype, and/or date?
On Fri, 2010-08-27 at 09:34 -0700, Daniel L. Miller wrote:
On 8/24/2010 4:35 PM, Timo Sirainen wrote:
On 24.8.2010, at 23.16, Ed W wrote:
At the moment I would claim that you are just automatically generating a very complicated filename. If you never trust your hash then you might as well instead simply use one of the existing GUID algorithms, if you trust your hash then you use that. I don't really see the point of a halfway house really?
Oh and this current scheme of hash-guid + hashes/hash hard linking is required in any case to keep track of reference counting. Unconditionally trusting the hash wouldn't make it any simpler. With key-value databases you'd have to figure out some other way to keep track of how many references there are to the attachment.
Can you append some "trivial" information from the data file to the hash in generating the file name to help ensure uniqueness? Like filesize,
I guess size could be there at least optionally, I'm not sure about as default.
mimetype,
I think different clients could use different MIME types sometimes, causing unnecessary duplicates.
and/or date?
I don't think attachments ever have dates? But if they did, again the problem of causing unnecessary duplicates.
On Tue, 2010-08-31 at 20:16 +0100, Timo Sirainen wrote:
Can you append some "trivial" information from the data file to the hash in generating the file name to help ensure uniqueness? Like filesize,
I guess size could be there at least optionally, I'm not sure about as default.
Now there's a mail_attachment_hash setting, where default is %{sha1} and you can use variables:
- %{md4}, %{md5}, %{sha1}, %{sha256}, %{sha512}
- %{size} = 64bit size as hex
- %X{size} = size with leading zeroes dropped
- %{method:n} = include only first n bits of the hash (e.g.
%{sha256:128}
- %B{method} = base64 encode instead of hex encode, don't include
trailing '=' chars
So you could have for example:
mail_attachment_size = %{sha512:128}-%{md5}-%X{size}
I think the SIS code is now just about done. Maybe after some more testing I'll copy it to dovecot-2.0 branch.
On 9/16/2010 11:20 AM, Timo Sirainen wrote:
On Tue, 2010-08-31 at 20:16 +0100, Timo Sirainen wrote:
Can you append some "trivial" information from the data file to the hash in generating the file name to help ensure uniqueness? Like filesize,
I guess size could be there at least optionally, I'm not sure about as default.
I think the SIS code is now just about done. Maybe after some more testing I'll copy it to dovecot-2.0 branch.
Something I wasn't quite clear about previously - does the SIS provide single-instance only within each account, or is it server wide? The examples given for mail_attachment_dir sometimes left me thinking it's a subdirectory off each user's home folder, while other examples seem global.
On 17.9.2010, at 2.10, Daniel L. Miller wrote:
Something I wasn't quite clear about previously - does the SIS provide single-instance only within each account, or is it server wide?
Server-wide. It would be pretty pointless otherwise.
The examples given for mail_attachment_dir sometimes left me thinking it's a subdirectory off each user's home folder, while other examples seem global.
I can't remember. Maybe I gave some bad examples. But yeah, it's still possible to make it per-user rather than global if someone really wants to..
participants (5)
-
Daniel L. Miller
-
Ed W
-
Mike Abbott
-
Timo Sirainen
-
William Blunn