[Dovecot] Dovecot shared library to replace libc-client
Hi all Dovecotters,
Lots of programs (e.g, PHP) link to a library called c-client, which is a derivative of the original IMAP implementation, UW IMAP.
(For those new to UW IMAP, read here: http://www.washington.edu/imap/ )
UW IMAP is not in as much of active development as it used to be, so I am curious:
I see there is a dovecot shared library. I haven't looked into the details, but here are things I'm interested in:
- Replacing libc-client's use as a client library
In Debian, for example, you can search for packages that depend on this library as a client. It looks like this:
$ ~ apt-cache rdepends libc-client2007e
libc-client2007e Reverse Depends: uw-mailutils uw-imapd libc-client2007e-dev ipopd prayer php5-imap mailsync libmail-cclient-perl asterisk aolserver4-nsimap uw-mailutils libc-client2007e-dev prayer php5-imap mailsync libmail-cclient-perl asterisk-voicemail-imapstorage
Let's say, hypothetically, we wanted to switch php5-imap to using Dovecot where it currently uses libc-client.
I envision creating libdovecot-c-client-alike that is a set of headers and a library that is API-compatible with (at least a subset of) c-client. You can call that a "compatibility shim." Then e.g. php5-imap could be given the path to those headers and the corresponding libdovecot-c-client-alike library, and when it thinks it is linking to c-client, it could instead link to the libdovecot-c-client-alike.
This might be convenient if you want to limit how much of a public API is presented by the current "dovecot.so" that gets installed in e.g. /usr/lib/dovecot/. The compatiblity shim could have a small API, and if you don't want provide ABI guarantees within dovecot.so, the shim could dlopen() dovecot.so rather than link to it.
- Use of Dovecot shared library within alpine, embedding the imapd
Right now, the mail client "alpine" embeds a copy of the UW IMAP source. It uses this when accessing local mail spools, for example.
If Dovecot's IMAPd were available as a shared library, perhaps with a c-client-like API, (although not necessarily -- it would be feasible to upgrade alpine to a different API), then alpine could use Dovecot's mail drivers directly.
So, those two are the dream. Timo and others, what are your thoughts?
Thanks for reading this far!
-- Asheesh.
P.S. The radical request #2 would be an excellent, decisive way to end a sad thread in the Debian bug tracker about Alpine + Maildir: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=405762 .
On 3.7.2012, at 8.53, Asheesh Laroia wrote:
I see there is a dovecot shared library. I haven't looked into the details, but here are things I'm interested in:
- Replacing libc-client's use as a client library .. I envision creating libdovecot-c-client-alike that is a set of headers and a library that is API-compatible with (at least a subset of) c-client. You can call that a "compatibility shim." Then e.g. php5-imap could be given the path to those headers and the corresponding libdovecot-c-client-alike library, and when it thinks it is linking to c-client, it could instead link to the libdovecot-c-client-alike.
Yeah, that's a possibility. Although Dovecot's libraries are still more about the server side stuff than client side stuff, so it's possible that there are many important missing things. Also libc-client is commonly used to do IMAP access and Dovecot's imapc backend is still lacking quite a lot of that functionality.
This might be convenient if you want to limit how much of a public API is presented by the current "dovecot.so" that gets installed in e.g. /usr/lib/dovecot/. The compatiblity shim could have a small API, and if you don't want provide ABI guarantees within dovecot.so, the shim could dlopen() dovecot.so rather than link to it.
I'm still not ready to give ABI or even API guarantees to libdovecot.. There are still several important large changes to do and I don't really want to keep a ton of ugly backwards compatibility stuff just for external users of the library. Also another potential problem is that libdovecot.so doesn't use a global namespace prefix for all of its functions, so linking it with php could cause symbol name conflicts (especially md5_*, sha1_* and such could cause trouble, like they already have caused with libmysql).
Once Dovecot becomes more "finished" (a few years?) I could consider API/ABI guarantees.. Of course nothing prevents anyone else from distributing a (patched) libdovecot already that actually does give some ABI guarantees. I just don't want to spend time on it. And v2.1 -> v2.2 -> v2.3 etc. transitions are going to be large changes.
- Use of Dovecot shared library within alpine, embedding the imapd
Right now, the mail client "alpine" embeds a copy of the UW IMAP source. It uses this when accessing local mail spools, for example.
If Dovecot's IMAPd were available as a shared library, perhaps with a c-client-like API, (although not necessarily -- it would be feasible to upgrade alpine to a different API), then alpine could use Dovecot's mail drivers directly.
I wonder if it would make any sense to for Alpine not use libdovecot API directly but rather talk IMAP protocol to Dovecot code (maybe running in a separate process)? The Dovecot configuration could be passed pretty easily from Alpine code without requiring any extra config files.
On Tue, 3 Jul 2012, Timo Sirainen wrote:
On 3.7.2012, at 8.53, Asheesh Laroia wrote:
I see there is a dovecot shared library. I haven't looked into the details, but here are things I'm interested in:
- Replacing libc-client's use as a client library .. I envision creating libdovecot-c-client-alike that is a set of headers and a library that is API-compatible with (at least a subset of) c-client. You can call that a "compatibility shim." Then e.g. php5-imap could be given the path to those headers and the corresponding libdovecot-c-client-alike library, and when it thinks it is linking to c-client, it could instead link to the libdovecot-c-client-alike.
Yeah, that's a possibility. Although Dovecot's libraries are still more about the server side stuff than client side stuff, so it's possible that there are many important missing things. Also libc-client is commonly used to do IMAP access and Dovecot's imapc backend is still lacking quite a lot of that functionality.
Timo, I appreciate the super speedy response!
This might be convenient if you want to limit how much of a public API is presented by the current "dovecot.so" that gets installed in e.g. /usr/lib/dovecot/. The compatiblity shim could have a small API, and if you don't want provide ABI guarantees within dovecot.so, the shim could dlopen() dovecot.so rather than link to it.
I'm still not ready to give ABI or even API guarantees to libdovecot.. There are still several important large changes to do and I don't really want to keep a ton of ugly backwards compatibility stuff just for external users of the library. Also another potential problem is that libdovecot.so doesn't use a global namespace prefix for all of its functions, so linking it with php could cause symbol name conflicts (especially md5_*, sha1_* and such could cause trouble, like they already have caused with libmysql).
Yeah, I totally understand your desire to not make backwards compatiblity a goal of the project.
Interesting point about the global namespace prefix. Is this something you'd be willing to reconsider, and start using a global namespace prefix?
Once Dovecot becomes more "finished" (a few years?) I could consider API/ABI guarantees.. Of course nothing prevents anyone else from distributing a (patched) libdovecot already that actually does give some ABI guarantees. I just don't want to spend time on it. And v2.1 -> v2.2 -> v2.3 etc. transitions are going to be large changes.
Yeah -- what I think is the most sensible, at the moment, is to distribute a small shim that has reasonably-tight dependencies to dovecot itself, and so when you upgrade dovecot, you probably have to upgrade the shim. So it proxies away the instability in dovecot, and provides a small, stable API/ABI.
That's something that it seems you might not be interested in, but I wonder if I can convince you otherwise.
If not, I might try convincing others to write it, but I'm hoping you might since you are so great! (-:
- Use of Dovecot shared library within alpine, embedding the imapd
Right now, the mail client "alpine" embeds a copy of the UW IMAP source. It uses this when accessing local mail spools, for example.
If Dovecot's IMAPd were available as a shared library, perhaps with a c-client-like API, (although not necessarily -- it would be feasible to upgrade alpine to a different API), then alpine could use Dovecot's mail drivers directly.
I wonder if it would make any sense to for Alpine not use libdovecot API directly but rather talk IMAP protocol to Dovecot code (maybe running in a separate process)? The Dovecot configuration could be passed pretty easily from Alpine code without requiring any extra config files.
That's my fallback plan at the moment, yeah. It seems like more work, though, but it has some serious tidiness possibly going for it.
-- Asheesh.
On 3.7.2012, at 9.18, Asheesh Laroia wrote:
I'm still not ready to give ABI or even API guarantees to libdovecot.. There are still several important large changes to do and I don't really want to keep a ton of ugly backwards compatibility stuff just for external users of the library. Also another potential problem is that libdovecot.so doesn't use a global namespace prefix for all of its functions, so linking it with php could cause symbol name conflicts (especially md5_*, sha1_* and such could cause trouble, like they already have caused with libmysql).
Yeah, I totally understand your desire to not make backwards compatiblity a goal of the project.
Interesting point about the global namespace prefix. Is this something you'd be willing to reconsider, and start using a global namespace prefix?
Dovecot in any case will have many different prefixes (I don't want to change ALL functions to begin with the same one), but there is one that is somewhat commonly used already: "i_" which originally indicated "irssi library" :)
dovecot-2.1/src/lib% grep '\bi_' *.h|wc -l 175
Much of the rest of the functions in lib/ could be prefixed with "i_" I guess. But there are several annoying problems, such as t_ prefix is also commonly used in many places and I don't want to remove them (but they probably won't be a real problem either). Then there's i_stream_*() for input streams vs. o_stream_*() for output streams. But for some of the clearly unproblematic ones the i_ prefix could be added, like for md5/sha/etc small and potentially conflicting functions.
Also I think GNU ld supports some way of hiding all but the explicitly wanted symbols from libraries, which would allow libdovecot to use all of its functions internally without the danger of them conflicting with outside users. But I'm not entirely sure how that works, maybe it needs to be done at linking stage which won't be good for Dovecot binaries that link with libdovecot.
Once Dovecot becomes more "finished" (a few years?) I could consider API/ABI guarantees.. Of course nothing prevents anyone else from distributing a (patched) libdovecot already that actually does give some ABI guarantees. I just don't want to spend time on it. And v2.1 -> v2.2 -> v2.3 etc. transitions are going to be large changes.
Yeah -- what I think is the most sensible, at the moment, is to distribute a small shim that has reasonably-tight dependencies to dovecot itself, and so when you upgrade dovecot, you probably have to upgrade the shim. So it proxies away the instability in dovecot, and provides a small, stable API/ABI.
That's something that it seems you might not be interested in, but I wonder if I can convince you otherwise.
If not, I might try convincing others to write it, but I'm hoping you might since you are so great! (-:
The best I can do is that at some point in future (might still be 1-2 years) I want to provide scripting language bindings for Dovecot APIs, maybe even allow creating plugins with non-C languages. Those APIs would then pretty much have to be frozen so that they don't break. Before that I need to do a few more larger API breakages to support some missing things.
I wonder if it would make any sense to for Alpine not use libdovecot API directly but rather talk IMAP protocol to Dovecot code (maybe running in a separate process)? The Dovecot configuration could be passed pretty easily from Alpine code without requiring any extra config files.
That's my fallback plan at the moment, yeah. It seems like more work, though, but it has some serious tidiness possibly going for it.
You could have Alpine keep the libc-client code and simply use it to connect to local Dovecot via IMAP protocol. All of the mailbox access would then go via Dovecot. I don't think it would be much work, mainly you'd need to create "libc-client mail storage configuration" -> "Dovecot settings in key=value format", fork a new process, put the settings to environment, exec imap and that's about it.
participants (2)
-
Asheesh Laroia
-
Timo Sirainen