- dovecot - dovecot.org

RE: [Dovecot] Using MySQL to store email?
by Joshua Goodall 08 Jun '06

08 Jun '06

Marc Perkel scribbled: > Repeat after me - think outside the box. Did you miss the part where I said we've already tried this? Not only have we thought outside the box, but we've smashed it up and built several alternatives to see which one sucked the least. Turns out, SQL backends suck the most. Please try to keep up :) Marc Perkel also scribbled: > To do this with MySQL would be trivial. To do it without a database > isn't possible. Actually you could do that with a tiny Sieve extension. I don't think you really understand what you are even proposing. Your statement would be better phrased "to do it without an index would be slow". If you wanted to create a database server for mail, with a mail-specific query language, then that database might be called "Dovecot", and the query language might be called "IMAP". Innovation is not an excuse for lousy engineering. As usual, it turns out to be bad design practice to say "let's deploy shinytoy for feature X!" rather than say "we need capability X! What's a creative way to achieve that?". Which is why necessity is the mother of invention. /JG

1 0

RE: [Dovecot] Using MySQL to store email?
by Joshua Goodall 07 Jun '06

07 Jun '06

Marc Perkel: > Suppose I want to move all message from a specific host to another folder. > UPDATE table SET folder=new-folder WHERE host=value; > > With databases you can do things you would have to write complex programs > to do with files. Things that you would consider to be impossible are > trivial with databases. > > This outside the box. People who optimise their systems for shiny cute features rather than the common case get all the hacks they deserve. Repeat after me: design for the common case. Also your proposed sql above and implications (arbitrary operations based on mail metadata i.e. headers) seems to involve indexing every mail header on delivery. Good luck with that one large scale mail storage. Did you want a mail database, or an OLAP system? >-) /jg

3 2

RE: [Dovecot] Using MySQL to store email?
by Joshua Goodall 07 Jun '06

07 Jun '06

Marc Perkel trolled: > So Timo, > > Have you considered the idea of storing all the email in a MySQL > database? Seems to me that MySQL could somplify all the backend stuff > that everyone struggles with and with replication one could create very > massive and reliable systems. What would it take to use MySQL that way? Maybe we could recode Dovecot in PHP, too. Sorry, that was nasty. However, having recently experimented with a very large mail storage system using mysql, I strongly advise against this avenue. In particular, the two main backends have serious deficiencies when you get up to the multi-gigabyte ranges: * MyISAM doesn't restart in constant time. * InnoDB's next-key locking kills performance on large blob inserts & deletes. Anyway, what do you mean, everyone struggles with backends? Dovecot's backends are fast, modular and reliable. Regarding replication; all good logical volume managers can do this. Regarding massive; all good filesystems can do this. Regarding reliable; I guarantee that mysql reliability (when properly managed) is still lower than that of a filesystem (when properly managed). In fact, the whole argument is specious. Filesystems already are databases, optimised for storing arbitrary-sized data objects using a hierarchical key, which is practically the definition of mail storage. /jg

2 1

[Dovecot] Dspam Plugin
by Timothy White 07 Jun '06

07 Jun '06

Johannes, do you have any plans (regarding your dspam plugin) to: a) Integrate it into libdspam rather than calling the dspam binary? b) Any ideas for scaling it better, and implementing it scaled better? I notice a slight discussion about scaling, and you have chosen not to pursue it. Is that because it's too hard, or just because you don't need to? Thanks Tim p.s. If I knew how to use the libdspam stuff, I'd make the changes myself... I might have to learn ;-) -- Linux Counter user #273956

2 1

[Dovecot] Dovecot configure script for 1.0b8 not including SQL drivers?
by Will Buik 06 Jun '06

06 Jun '06

On 6/5/06, grant beattie <grant(a)grunta.com> wrote: > On Mon, Jun 05, 2006 at 05:57:36PM -0400, Will Buik wrote: > > > I am having a problem with the configure script for Dovecot 1.0b8. It > > doesn't seem to be compiling in sql engine driver support. I have > > specified --with-sql and --with-pgsql but when the script is done it > > says "Building with SQL drivers ..............:" but lists no drivers > > afterwards. I tried compiling it anyway but it logs the error " > > auth(default): Unknown database driver 'pgsql' " so the it defiantly > > didn't get compiled in. My pgsql includes and libs are in a non > > standard location but I specified that in the CPPFLAGS and LDFLAGS > > before the ./configure invocation. What have I done wrong here, or is > > there a bug in the script? > > > > The whole invocation is: > > CPPFLAGS='-I/usr/local/openssl-0.9.8b/include > > -I/usr/local/pgsql-8.1.4/include' > > LDFLAGS='-L/usr/local/openssl-0.9.8b/lib -L/usr/local/pgsql-8.1.4/lib' > > ./configure --prefix=/usr/local/dovecot-1.0b8 > > --sysconfdir=/etc/dovecot --localstatedir=/var --with-ioloop=eooll > > --with-notify=dnotify --without-pop3d --with-storages=maildir > > --with-sql --with-pgsql --with-gnu-ld > > what does config.log say? and does it report any problems with > finding openssl? > > the problem may be that the shared libraries can't be found at > runtime, in which case you will need to set the rpath in the binaries > that are built. you didn't mention which platform/compiler you're > using, but this is done using the -Wl,-R<directory> argument for gcc > and Sun Studio compiler/Solaris ld, at least (-Wl passes the next > argument to the linker, -R tells the linker to include this path in the > binary's rpath). > > grant. I looked at the config.log file and it doesn't look like it's having any problems with openssl, but when I looked for pgsql I got what looked like some errors. configure:29296: checking for PQconnectdb in -lpq | char PQconnectdb (); | PQconnectdb (); ac_cv_lib_pq_PQconnectdb=no /usr/local/pgsql-8.1.4/lib/libpq.so: undefined reference to `dlerror' /usr/local/pgsql-8.1.4/lib/libpq.so: undefined reference to `dlclose' /usr/local/pgsql-8.1.4/lib/libpq.so: undefined reference to `dlopen' /usr/local/pgsql-8.1.4/lib/libpq.so: undefined reference to `dlsym'

2 1

Re: [Dovecot] Using MySQL to store email?
by nodata 06 Jun '06

06 Jun '06

> On 06-06-2006 15:48:20 +0200, nodata wrote: >> Putting everything in a database would provide one benefit: >> * Less storage space needed due to duplicated e-mail > > ... assuming that the mails are completely identical (i.e. have no slightly different headers) and that the database backend does expensive duplicate elimination on its string heaps. Yes, like Exchange does (unless you did something "clever" with the headers). > So the real benefit you get I think is: > * full text searching capabilities based on the IR-like indices > provided/maintained by the database backend, such as for instance: http://dev.mysql.com/doc/refman/5.0/en/fulltext-query-expansion.html If the original poster if struggling with a filesystem version of dovecot, and wants to create "massive and reliable systems" using MySQL, then he's going to have trouble. Adding MySQL won't do that.

1 0

[Dovecot] vpopmail
by C J Kenneth Tan -- OptimaNumerics 06 Jun '06

06 Jun '06

I was running 1.0.beta3 code with no problems. I recently switched to 1.0.beta8 and I noticed that vpopmail's pop-before-smtp functionality stopped working. I am using vpopmail authentication. Has anyone else encountered this problem? Thank you. Kenneth Tan ----------------------------------------------------------------------- C J Kenneth Tan, PhD OptimaNumerics Ltd Telephone: +44 798 941 7838 E-mail: cjtan(a)OptimaNumerics.com Telephone: +44 207 099 4428 Web: http://www.OptimaNumerics.com Facsimile: +44 207 100 4572 -----------------------------------------------------------------------

8 24

[Dovecot] Procmail and dovecot interactions
by Jeff Turner 06 Jun '06

06 Jun '06

Hi, I'm using procmail to deliver to Maildir directories: DEFAULT=${HOME}/Maildir/ I notice dovecot keeps a 'dovecot-uidlist' plus cache and index files, in each maildir. How is it that dovecot keeps these up to date when the contents of cur/ keeps changing? Is dovecot scanning cur/ every so often checking for changes? Isn't this slow for large folders (100k+)? I note on the Dovecot LDA page[1], that it keeps the "Dovecot index base up to date". Does this mean Dovecot isn't expected to work properly with a delivery agent which doesn't know about its indexes? Cheers, Jeff [1] http://wiki.dovecot.org/LDA

2 2

[Dovecot] Dovecot configure script for 1.0b8 not including SQL drivers?
by Will Buik 06 Jun '06

06 Jun '06

I am having a problem with the configure script for Dovecot 1.0b8. It doesn't seem to be compiling in sql engine driver support. I have specified --with-sql and --with-pgsql but when the script is done it says "Building with SQL drivers ..............:" but lists no drivers afterwards. I tried compiling it anyway but it logs the error " auth(default): Unknown database driver 'pgsql' " so the it defiantly didn't get compiled in. My pgsql includes and libs are in a non standard location but I specified that in the CPPFLAGS and LDFLAGS before the ./configure invocation. What have I done wrong here, or is there a bug in the script? The whole invocation is: CPPFLAGS='-I/usr/local/openssl-0.9.8b/include -I/usr/local/pgsql-8.1.4/include' LDFLAGS='-L/usr/local/openssl-0.9.8b/lib -L/usr/local/pgsql-8.1.4/lib' ./configure --prefix=/usr/local/dovecot-1.0b8 --sysconfdir=/etc/dovecot --localstatedir=/var --with-ioloop=eooll --with-notify=dnotify --without-pop3d --with-storages=maildir --with-sql --with-pgsql --with-gnu-ld Thanks in advance.

2 1

[Dovecot] Maildir, procmail, and locks
by Mark Nienberg 05 Jun '06

05 Jun '06

I'm using procmail as the local delivery agent for Maildir style mailboxes. I discovered that I was able to send mail to other users but was unable to send mail to myself. Examples from verbose procmail log: **** Successful message to another user **** procmail: [24589] Fri Jun 2 15:03:34 2006 procmail: Assigning "PATH=/home/david/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/X11R6/bin" procmail: Assigning "LASTFOLDER=/home/david/Maildir/new/msg.4AAdC" procmail: Opening "/home/david/Maildir/new/msg.4AAdC" procmail: Acquiring kernel-lock procmail: Notified comsat: "david@0:/home/david/Maildir/new/msg.4AAdC" From mark.nienberg(a)tippingmar.com Fri Jun 2 15:03:34 2006 Subject: test to other user Folder: /home/david/Maildir/new/msg.4AAdC **** Failed message to myself **** procmail: [24596] Fri Jun 2 15:05:13 2006 procmail: Assigning "PATH=/home/mark/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/X11R6/bin" procmail: Couldn't create or rename temp file "/home/mark/Maildir/new/msg.A" procmail: Assigning "LASTFOLDER=/var/mail/mark" procmail: Opening "/var/mail/mark" procmail: Acquiring kernel-lock procmail: Notified comsat: "mark@686:/var/mail/mark" From mark.nienberg(a)tippingmar.com Fri Jun 2 15:05:13 2006 Subject: test to self Folder: /var/mail/mark Anyway, I changed dovecot's lock_method from the default fcntl to flock and the problem disappeared. I'll add this to the wiki unless someone points out an error in my method. Mark Nienberg

3 8