[Dovecot] handling filename of compressed messages
i'm running dovecot 1.2.9 and, among other things, i have the zlib plugin enabled. A shell script runs once a day to compresses all not still compressed files on the users maildirs.
so far so good, it works fine, there's no problem on that.
what i think is not happening 100% correctly is the following:
1) user receives a message
2) BEFORE the compress routine runs, filename is:new/1263436052.V6814I43300b4M146002.correio.domain.com.br,S=238207
3) AFTER the compress routine, filename is now:new/1263436052.V6814I43300b4M146002.correio.domain.com.br,S=238207.gz
4) user check his mail .... and now the filename iscur/1263436052.V6814I43300b4M146002.correio.domain.com.br,S=238207.gz:2,S
5) user replies that mail .... and now the filename iscur/1263436052.V6814I43300b4M146002.correio.domain.com.br,S=238207.gz:2,RS
i'm aware that dovecot doesnt need the '.gz' extension on the file to recognize that as a gzipped file. Even without the .gz extension, dovecot is correctly reading the file. the '.gz' is not even necessary on the filename and still dovecot can handle it fine.
anyway, i would like to suggest dovecot to handle the filename to keep the .gz extension, if that exists. This could help sysadmins to easily identify gzipped files on the maildirs and make anything with them, if that's desired.
i would like to see dovecot changing filenames properly and keeping the .gz extension always, for example
cur/1263436052.V6814I43300b4M146002.correio.domain.com.br,S=238207:2,S.gz instead of cur/1263436052.V6814I43300b4M146002.correio.domain.com.br,S=238207.gz:2,S
cur/1263436052.V6814I43300b4M146002.correio.domain.com.br,S=238207:2,RS.gz instead of cur/1263436052.V6814I43300b4M146002.correio.domain.com.br,S=238207.gz:2,RS
--
Atenciosamente / Sincerily,
Leonardo Rodrigues
Solutti Tecnologia
http://www.solutti.com.br
Minha armadilha de SPAM, NÃO mandem email
gertrudes@solutti.com.br
My SPAMTRAP, do not email itOn 01/14/2010 03:33 AM Leonardo Rodrigues wrote:
i'm running dovecot 1.2.9 and, among other things, i have the zlibplugin enabled. A shell script runs once a day to compresses all not still compressed files on the users maildirs.
so far so good, it works fine, there's no problem on that. what i think is not happening 100% correctly is the following: 1) user receives a message 2) BEFORE the compress routine runs, filename is:new/1263436052.V6814I43300b4M146002.correio.domain.com.br,S=238207
3) AFTER the compress routine, filename is now:new/1263436052.V6814I43300b4M146002.correio.domain.com.br,S=238207.gz
*PLING* And here is the problem. :-) I've found it. May I keep it?
… i'm aware that dovecot doesnt need the '.gz' extension on the file to recognize that as a gzipped file. Even without the .gz extension, dovecot is correctly reading the file. the '.gz' is not even necessary on the filename and still dovecot can handle it fine.
The .gz is only wrong in the filename
anyway, i would like to suggest dovecot to handle the filename tokeep the .gz extension, if that exists. This could help sysadmins to easily identify gzipped files on the maildirs and make anything with them, if that's desired.
i would like to see dovecot changing filenames properly and keepingthe .gz extension always, for example
cur/1263436052.V6814I43300b4M146002.correio.domain.com.br,S=238207:2,S.gz instead of cur/1263436052.V6814I43300b4M146002.correio.domain.com.br,S=238207.gz:2,S
cur/1263436052.V6814I43300b4M146002.correio.domain.com.br,S=238207:2,RS.gz instead of cur/1263436052.V6814I43300b4M146002.correio.domain.com.br,S=238207.gz:2,RS
No, you don't wont to have such file names.
http://wiki.dovecot.org/Plugins/Zlib: … Dovecot can now read the file, but to avoid compressing it again … include e.g. a "Z" flag in the file name … (e.g. 1223212411.M907959P17184.host,S=3271:2,SZ)
The article links also to: http://cr.yp.to/proto/maildir.html: … What can I put in info? When you move a file from new to cur, you have to change its name from uniq to uniq:info … info starting with "2,": Each character after the comma is an independent flag. …
In short: You want to rename 1263436052.V6814I43300b4M146002.correio.domain.com.br,S=238207 to 1263436052.V6814I43300b4M146002.correio.domain.com.br,S=238207:2,Z
Regards, Pascal
The trapper recommends today: beeffeed.1001404@localdomain.org
Pascal Volk escreveu:
No, you don't wont to have such file names.
http://wiki.dovecot.org/Plugins/Zlib: … Dovecot can now read the file, but to avoid compressing it again … include e.g. a "Z" flag in the file name … (e.g. 1223212411.M907959P17184.host,S=3271:2,SZ)
The article links also to: http://cr.yp.to/proto/maildir.html: … What can I put in info? When you move a file from new to cur, you have to change its name from uniq to uniq:info … info starting with "2,": Each character after the comma is an independent flag. …
In short: You want to rename 1263436052.V6814I43300b4M146002.correio.domain.com.br,S=238207 to 1263436052.V6814I43300b4M146002.correio.domain.com.br,S=238207:2,Z
got it .... so, basically, i shouldnt be using .gz at all, is that ok ?
i understood that, when moving from new to cur, filename should be appended of ':2,'. But filenames on new, as i understood, should never be changed. As soon as they are read there (on new), they should be moved to cur.
what would be the correct way of compressing messages on the new directory and adding the Z flag ? Would be any problem if, when compressing on new, add the ':2,' suffix ? Indeed it would be ':2,Z' suffix one
i tried it ...gzipped the file new/1263462832.V6814I43300c8M993425.correio.domain.com.br,S=10137 and got the gzipped file to be named new/1263462832.V6814I43300c8M993425.correio.domain.com.br,S=10137:2,Z and excluded the original file
when checked mail, file was correctly moved to cur when replied to that mail, file was correctly renamed to cur/1263462832.V6814I43300c8M993425.correio.domain.com.br,S=10137:2,RSZ
is that OK to have filenames on new with the ':2,Z' suffix ?--
Atenciosamente / Sincerily,
Leonardo Rodrigues
Solutti Tecnologia
http://www.solutti.com.br
Minha armadilha de SPAM, NÃO mandem email
gertrudes@solutti.com.br
My SPAMTRAP, do not email itOn Qui, 14 Jan 2010, Leonardo Rodrigues wrote:
got it .... so, basically, i shouldnt be using .gz at all, is that ok ?
Yes.
i understood that, when moving from new to cur, filename should
be appended of ':2,'. But filenames on new, as i understood, should
never be changed. As soon as they are read there (on new), they
should be moved to cur.what would be the correct way of compressing messages on the new
directory and adding the Z flag ? Would be any problem if, when
compressing on new, add the ':2,' suffix ? Indeed it would be ':2,Z'
suffix one
How about not compressing mails still in new/? They shouldn't remain
there for very long anyway, you could compress only the ones in cur/.
-- I have a very strange feeling about this... -- Luke Skywalker
Eduardo M KALINOWSKI eduardo@kalinowski.com.br
Quoting Eduardo M KALINOWSKI eduardo@kalinowski.com.br:
On Qui, 14 Jan 2010, Leonardo Rodrigues wrote:
got it .... so, basically, i shouldnt be using .gz at all, is that ok ?
Yes.
i understood that, when moving from new to cur, filename should
be appended of ':2,'. But filenames on new, as i understood, should
never be changed. As soon as they are read there (on new), they
should be moved to cur.what would be the correct way of compressing messages on the new
directory and adding the Z flag ? Would be any problem if, when
compressing on new, add the ':2,' suffix ? Indeed it would be
':2,Z' suffix oneHow about not compressing mails still in new/? They shouldn't remain
there for very long anyway, you could compress only the ones in cur/.
If people read their emails atleast daily, it would work ok that way.
But with several people that only read emails monthly or even not at
all, it helps to compress the new folder also.
Patrick Domack escreveu:
If people read their emails atleast daily, it would work ok that way. But with several people that only read emails monthly or even not at all, it helps to compress the new folder also.
let's not forget those 'created and forgot' mailboxes, who stands there on the server with hundreds of Mbs of messages and nobody checks that. They are all on new/ folder and, probably, will never got to cur/ because they will never be checked.
OK .... i know that shouldnt happen ..... but in real life this happens quite commonly.
--
Atenciosamente / Sincerily,
Leonardo Rodrigues
Solutti Tecnologia
http://www.solutti.com.br
Minha armadilha de SPAM, NÃO mandem email
gertrudes@solutti.com.br
My SPAMTRAP, do not email itOn 14.1.2010, at 11.58, Leonardo Rodrigues wrote:
In short: You want to rename 1263436052.V6814I43300b4M146002.correio.domain.com.br,S=238207 to 1263436052.V6814I43300b4M146002.correio.domain.com.br,S=238207:2,Z
got it .... so, basically, i shouldnt be using .gz at all, is that ok ?
Right. Don't change the base filename.
i understood that, when moving from new to cur, filename should be appended of ':2,'. But filenames on new, as i understood, should never be changed. As soon as they are read there (on new), they should be moved to cur.
Maildir spec says files in new/ shouldn't have the :2, part, but Dovecot is fine with it.
Timo Sirainen escreveu:
Maildir spec says files in new/ shouldn't have the :2, part, but Dovecot is fine with it.
i'm trying to get the sample script for compressing maildirs from http://wiki.dovecot.org/Plugins/Zlib (right on the end of the page) but link seems to be not accessible.
is this script available somewhere else ?--
Atenciosamente / Sincerily,
Leonardo Rodrigues
Solutti Tecnologia
http://www.solutti.com.br
Minha armadilha de SPAM, NÃO mandem email
gertrudes@solutti.com.br
My SPAMTRAP, do not email itOn 01/14/10 20:58, Leonardo Rodrigues wrote:
what would be the correct way of compressing messages on the new directory and adding the Z flag ? Would be any problem if, when compressing on new, add the ':2,' suffix ? Indeed it would be ':2,Z' suffix one
OK... since I've been wanting compressed mail for sometime.. I'm going to throw my hat into the ring here...
Timo -- any tips on helping dovecot deliver compress mails on delivery?
It would solve a lot of problems here, from what I can see, to have a deliver plugin that compresses on delivery and sets the Z flag in the filename.
-- Curtis
Curtis Maloney escreveu:
Timo -- any tips on helping dovecot deliver compress mails on delivery?
It would solve a lot of problems here, from what I can see, to have a deliver plugin that compresses on delivery and sets the Z flag in the filename.
that would be indeed lovely !!!! when this feature gets implemented on dovecot delivery agent, i'll definitely stop using postfix virtual delivery agent.
--
Atenciosamente / Sincerily,
Leonardo Rodrigues
Solutti Tecnologia
http://www.solutti.com.br
Minha armadilha de SPAM, NÃO mandem email
gertrudes@solutti.com.br
My SPAMTRAP, do not email itLeonardo Rodrigues escreveu:
Curtis Maloney escreveu:
Timo -- any tips on helping dovecot deliver compress mails on delivery?
It would solve a lot of problems here, from what I can see, to have a deliver plugin that compresses on delivery and sets the Z flag in the filename.
that would be indeed lovely !!!! when this feature gets implemented on dovecot delivery agent, i'll definitely stop using postfix virtual delivery agent.
again on this subject of compressing files ..... i understand perfectly that i really dont want the '.gz' or '.bz2' filename. I dont want anything more than adding the Z flag to the compressed file.
The 'script compress idea' from the Wiki page sais:Dovecot can now read the file, but to avoid compressing it again on the next run, you'll probably want to rename it again to include e.g. a "Z" flag in the file name to mark that it was compressed (e.g. 1223212411.M907959P17184.host,S=3271:2,SZ).
If i rename the file and add the Z flag, i will be braking the dovecot-uidlist file, which contain the filenames without the Z flag. Users would have to download the compressed files again because the filename was changed. That on big mailboxes and slow connections (specially after the very 1st compression routine run) can be a BIG problem.
keeping the filename intact, not even adding the Z flag, would be a great idea to avoid clients from redownloading messages ??? Compression routines would probably have to use file command to identify already compressed files, which would badly impact on the performance of its run ... not being able to identify already compressed files based on its filename would be very bad ....
--
Atenciosamente / Sincerily,
Leonardo Rodrigues
Solutti Tecnologia
http://www.solutti.com.br
Minha armadilha de SPAM, NÃO mandem email
gertrudes@solutti.com.br
My SPAMTRAP, do not email itOn 01/17/2010 12:06 PM, Leonardo Rodrigues wrote:
The 'script compress idea' from the Wiki page sais: Dovecot can now read the file, but to avoid compressing it again on the next run, you'll probably want to rename it again to include e.g. a "Z" flag in the file name to mark that it was compressed (e.g. 1223212411.M907959P17184.host,S=3271:2,SZ).
If i rename the file and add the Z flag, i will be braking thedovecot-uidlist file, which contain the filenames without the Z flag. Users would have to download the compressed files again because the filename was changed. That on big mailboxes and slow connections (specially after the very 1st compression routine run) can be a BIG problem.
keeping the filename intact, not even adding the Z flag, would be agreat idea to avoid clients from redownloading messages ??? Compression routines would probably have to use file command to identify already compressed files, which would badly impact on the performance of its run ... not being able to identify already compressed files based on its filename would be very bad ....
I think that file name comparison is based only on the part before the ":", which identifies the message. What comes after are flags that might be set/unset independently of dovecot if another program also accesses the mail directory.
Note that it is not necessary to include the "Z" flag to prevent double compression. If you really want, you can keep the same file, just be sure in your compression script to check if files are compressed by looking at the first bytes: both gzip and bzip2 files always start with some fixed bytes.
-- The nation that controls magnetism controls the universe. -- Chester Gould/Dick Tracy
Eduardo M KALINOWSKI eduardo@kalinowski.com.br
participants (6)
-
Curtis Maloney
-
Eduardo M KALINOWSKI
-
Leonardo Rodrigues
-
Pascal Volk
-
Patrick Domack
-
Timo Sirainen