[Dovecot] Dovecot 2.0.rc3 doesn't compile with custom OpenSSL

[Andreas M. Kirchwitz]

Hi folks,

when building Dovecot 2.0.rc3 (never used any other 2.0.X before,
so this issue might not be new to that specific release), then
compilation fails on anything that is linked against the new
"libdovecot-login.so" library.

For example, "imap-login" fails:

/bin/sh ../../libtool --tag=CC   --mode=link gcc  -std=gnu99 -g -O2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -Wstrict-aliasing=2 -I/usr/local/ssl/include    -o imap-login client.o client-authenticate.o imap-login-settings.o imap-proxy.o ../../src/login-common/libdovecot-login.la ../../src/lib-dovecot/libdovecot.la -lrt
libtool: link: gcc -std=gnu99 -g -O2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -Wstrict-aliasing=2 -I/usr/local/ssl/include -o .libs/imap-login client.o client-authenticate.o imap-login-settings.o imap-proxy.o  ../../src/login-common/.libs/libdovecot-login.so ../../src/lib-dovecot/.libs/libdovecot.so -lrt -Wl,-rpath -Wl,/usr/local/Dovecot-2.0.rc3/lib/dovecot
/usr/bin/ld: warning: libcrypto.so.1.0.0, needed by ../../src/login-common/.libs/libdovecot-login.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libssl.so.1.0.0, needed by ../../src/login-common/.libs/libdovecot-login.so, not found (try using -rpath or -rpath-link)
../../src/login-common/.libs/libdovecot-login.so: undefined reference to `OBJ_txt2nid'
[...]
collect2: ld returned 1 exit status
make[3]: *** [imap-login] Error 1
make[3]: Leaving directory `/usr/local/src/dovecot-2.0.rc3/src/imap-login'

I'm not using system OpenSSL but a custom installation in /usr/local/ssl.
According to the documentation, Dovecot is smart enough to get everything
it needs from the /usr/local/ssl/lib/pkgconfig/*.pc files. Dovecot 1.2.X
just compiles fine if configured like this:

	./configure --prefix=/usr/local/${DOVECOTVER} --with-ssl=openssl

The problem with Dovecot 2.0.rc3 is, that libdovecot-login.so isn't given
the correct pathes for the OpenSSL libraries:

	$ ldd src/login-common/.libs/libdovecot-login.so
	linux-gate.so.1 =>  (0x00b6e000)
	libssl.so.1.0.0 => not found
	libcrypto.so.1.0.0 => not found
	libdl.so.2 => /lib/libdl.so.2 (0x00b2a000)
	librt.so.1 => /lib/librt.so.1 (0x004d4000)
	libc.so.6 => /lib/libc.so.6 (0x00110000)
	/lib/ld-linux.so.2 (0x006e7000)
	libpthread.so.0 => /lib/libpthread.so.0 (0x009c6000)

So everything depending on libdovecot-login.so will fail.
Setting "SSL_CFLAGS" and "SSL_LIBS" manually doesn't help because
Dovecot already sets these perfectly fine via pkg-config:

	env SSL_CFLAGS="-I/usr/local/ssl/include" SSL_LIBS="-L/usr/local/ssl/lib -Wl,-R/usr/local/ssl/lib -lcrypto -lssl" ./configure --prefix=/usr/local/${DOVECOTVER} --with-ssl=openssl

Only the hard way works:

	env CPPFLAGS="-I/usr/local/ssl/include" LDFLAGS="-L/usr/local/ssl/lib -Wl,-R/usr/local/ssl/lib -lcrypto -lssl" ./configure --prefix=/usr/local/${DOVECOTVER} --with-ssl=openssl

However, setting CPPFLAGS and LDFLAGS shouldn't be necessary because
SSL_CFLAGS and SSL_LIBS already contain the information that is needed
to build a proper library. They are just not used when linking
libdovecot-login.so ;-)

	Greetings, Andreas