Building failures with 2.4.0 and 2.4.1 on MacOS: var_expand_crypt
I'm trying to build dovecot, both at tags 2.4.0 and 2.4.1, both failing with the same undefined-symbols link error.
I've pulled the source from GitHub onto an Apple M2 Ultra Mac Studio running Sonoma 14.6.1.
FWIW, I'm picking up various dependencies as packages from Homebrew via the CFLAGS and LDFLAGS args below.
The prep steps before "make": $ ./autogen.sh $ CFLAGS="-I/opt/homebrew/include" LDFLAGS="-L/opt/homebrew/lib" ./configure --prefix=/opt/homebrew --with-bzlib --with-ldap --with-mysql --with-pam --with-sqlite --without-icu $ PATH="/opt/homebrew/opt/bison/bin:$PATH" make ...
N.B.: The PATH tweak is solely intended to pick up a usable version (3.8.2) of bison, since the Xcode-supplied binary is too old (v2.3), and doesn't recognize certain cmdline flags.
Also, getting even this far with 2.4.0 requires the "touch build-aux/config.rpath" workaround, and the 2-line patch to src/lib-http/http-server.h suggested on this thread: https://dovecot.org/mailman3/archives/list/dovecot@dovecot.org/thread/MFHSI5....
Anyway, here's an excerpt of the build stdout/stderr.
$ PATH="/opt/homebrew/opt/bison/bin:$PATH" make ... CCLD var_expand_crypt.la Undefined symbols for architecture arm64: "_binary_to_hex_append", referenced from: _var_expand_encrypt in var-expand-crypt.o _var_expand_encrypt in var-expand-crypt.o "_buffer_append", referenced from: _var_expand_crypt_settings in var-expand-crypt.o _var_expand_crypt_settings in var-expand-crypt.o "_buffer_append_c", referenced from: _str_append_c in var-expand-crypt.o ... ld: symbol(s) not found for architecture arm64 clang: error: linker command failed with exit code 1 (use -v to see invocation) make[3]: *** [var_expand_crypt.la] Error 1 make[2]: *** [all-recursive] Error 1 make[1]: *** [all-recursive] Error 1 make: *** [all] Error 2 $
Hopefully this is enough info to go on, but LMK if not.
I've gotten no replies as yet, but I diagnosed a bit further.
One of the symbols that the link step complained about: binary_to_hex_append, referenced in src/lib-var-expand-crypt/var-expand-crypt.c, declared in the header file src/lib/hex-binary.h.
$ cd src/lib $ nm .libs/liblib.a ... hex-binary.o: 0000000000000000 T _binary_to_hex 0000000000000308 T _binary_to_hex_append $
IOW, that function has been successfully compiled and appears in both hex-binary.o and the object archive "src/lib/.libs/liblib.a".
Meanwhile, here's the actual compile command for lib-var-expand-crypt:
$ cd src/lib-var-expand-crypt $ make -n echo " CCLD " var_expand_crypt.la;/bin/sh ../../libtool --silent --tag=CC --mode=link gcc -std=gnu11 -I/opt/homebrew/include -fstack-protector-strong -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -Wno-duplicate-decl-specifier -Wstrict-aliasing=2 -export-dynamic -avoid-version -module -no-undefined -L/opt/homebrew/lib -o var_expand_crypt.la -rpath /opt/homebrew/lib/dovecot var-expand-crypt.lo ... $
Nothing in that link step seems to reference "liblib.a" (or "liblib.la", or anything like that).
Why doesn't this link step seem to reference the right library?
Does 2.4.1 build for anyone on MacOS??
It's been a *long* while since I dealt with building source with the GNU toolchain (libtool and friends), so I'm a bit rusty here.
Help, please?
On Mon, Jul 14, 2025 at 5:57 AM rmfuhrer--- via dovecot <dovecot@dovecot.org> wrote:
I've gotten no replies as yet, but I diagnosed a bit further.
One of the symbols that the link step complained about: binary_to_hex_append, referenced in src/lib-var-expand-crypt/var-expand-crypt.c, declared in the header file src/lib/hex-binary.h.
$ cd src/lib $ nm .libs/liblib.a ... hex-binary.o: 0000000000000000 T _binary_to_hex 0000000000000308 T _binary_to_hex_append $
IOW, that function has been successfully compiled and appears in both hex-binary.o and the object archive "src/lib/.libs/liblib.a".
Meanwhile, here's the actual compile command for lib-var-expand-crypt:
$ cd src/lib-var-expand-crypt $ make -n echo " CCLD " var_expand_crypt.la;/bin/sh ../../libtool --silent --tag=CC --mode=link gcc -std=gnu11 -I/opt/homebrew/include -fstack-protector-strong -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -Wno-duplicate-decl-specifier -Wstrict-aliasing=2 -export-dynamic -avoid-version -module -no-undefined -L/opt/homebrew/lib -o var_expand_crypt.la -rpath /opt/homebrew/lib/dovecot var-expand-crypt.lo ... $
Nothing in that link step seems to reference "liblib.a" (or "liblib.la", or anything like that).
Why doesn't this link step seem to reference the right library?
Does 2.4.1 build for anyone on MacOS??
It's been a *long* while since I dealt with building source with the GNU toolchain (libtool and friends), so I'm a bit rusty here.
Help, please?
Hello there,
Sometime around March this year, someone mentioned on this list that they installed Dovecot 2.4 using Homebrew. I am not an expert with MacOS, but I would like to know if you could go that route instead of a manual compile.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 In an Internet failure case, the #1 suspect is a constant: DNS. "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
On Mon, Jul 14, 2025 at 5:57AM rmfuhrer--- via dovecot <[1]dovecot@dovecot.org> wrote:
I've gotten no replies as yet, but I diagnosed a bit further.
One of the symbols that the link step complained about:
binary_to_hex_append, referenced in
src/lib-var-expand-crypt/var-expand-crypt.c, declared in the header file
src/lib/hex-binary.h.
$ cd src/lib
$ nm .libs/liblib.a
...
hex-binary.o:
0000000000000000 T _binary_to_hex
0000000000000308 T _binary_to_hex_append
$
IOW, that function has been successfully compiled and appears in both
hex-binary.o and the object archive "src/lib/.libs/liblib.a".
Meanwhile, here's the actual compile command for lib-var-expand-crypt:
$ cd src/lib-var-expand-crypt
$ make -n
echo " CCLD " [2]var_expand_crypt.la;/bin/sh ../../libtool --silent
--tag=CC --mode=link gcc
-std=gnu11 -I/opt/homebrew/include -fstack-protector-strong
-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2
-Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith
-Wchar-subscripts
-Wformat=2 -Wbad-function-cast -Wno-duplicate-decl-specifier
-Wstrict-aliasing=2
-export-dynamic -avoid-version -module -no-undefined
-L/opt/homebrew/lib -o [3]var_expand_crypt.la
-rpath /opt/homebrew/lib/dovecot var-expand-crypt.lo
...
$
Nothing in that link step seems to reference "liblib.a" (or
"[4]liblib.la", or anything like that).
Why doesn't this link step seem to reference the right library?
Does 2.4.1 build for anyone on MacOS??
It's been a *long* while since I dealt with building source with the GNU
toolchain (libtool and friends), so I'm a bit rusty here.
Help, please?Hello there, Sometime around March this year, someone mentioned on this list that they installed Dovecot 2.4 using Homebrew. I am not an expert with MacOS, but I would like to know if you could go that route instead of a manual compile.
Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 In an Internet failure case, the #1 suspect is a constant: DNS. "Oh, the cruft.", egrep -v '^$|^.*#' -\_(Tu)_/- :-) [How to ask smart questions: [5]http://www.catb.org/~esr/faqs/smart-questions.html]
References
Visible links
Thanks for the reply, but I've already been getting dovecot from Homebrew for many years now, with a local formula tweak to compile in MySQL support for auth tables.
I'm exploring the possibly of switching to building from source, to partially or totally decouple dovecot from semi-automatic Homebrew upgrades that often lead to breakage.
In particular, breakage often occurs when I upgrade MacOS, which triggers a flurry of more-or-less mandatory Homebrew package upgrades, which may break dovecot (or MySQL underneath).
I've also been considering containerizing dovecot, and running either under Docker or perhaps in the cloud, but that requires running within a Linux image (not actually a problem, but has a different set of configs from what I'm now using under MacOS), and it requires an approach for mailbox storage that interacts well with backups (e.g., that efficiently backs up small deltas for modest Maildir changes, rather than backing up an entire Linux container or even the entire Maildir directory structure each time).
I made some more headway. AFAICT, lib-var-expand-crypt is missing dependencies from Makefile.am, which should have been declared via LIBADD. The following allows the build to progress *much* further; way past the link issues with lib-var-expand-crypt: $ cd src $ git diff lib-var-expand-crypt/Makefile.am diff --git a/src/lib-var-expand-crypt/Makefile.am b/src/lib-var-expand-crypt/Makefile.am index 8f829bcc02..ff5b83f22c 100644 --- a/src/lib-var-expand-crypt/Makefile.am +++ b/src/lib-var-expand-crypt/Makefile.am @@ -9,6 +9,8 @@ AM_CPPFLAGS = \ var_expand_crypt_la_SOURCES = \ var-expand-crypt.c +# rmfuhrer 07/14/2025: Trying to satisfy link-time references to binary_to_hex_append and friends... +var_expand_crypt_la_LIBADD = ../lib/liblib.la ../lib-dcrypt/libdcrypt.la ../lib-var-expand/libvar_expand.la ../lib-json/libjson.la var_expand_crypt_la_LDFLAGS = $(MODULE_LIBS) -avoid-version -module pkginc_libdir=$(pkgincludedir) $ Should I file a bug or pull request? I'm somewhat confused as to how 2.4.0 built for anyone (on any platform) without it... It's now dying while trying to compile ldap-utils.c with an undefined symbol: $ cd src $ PATH="/opt/homebrew/opt/bison/bin:$PATH" make ... ldap-utils.c:77:27: error: use of undeclared identifier 'LDAP_OPT_X_TLS_ECNAME' 77 | if (ldap_set_opt_str(ld, LDAP_OPT_X_TLS_ECNAME, | ^ 3 warnings and 1 error generated. make[3]: *** [ldap-utils.lo] Error 1 make[2]: *** [all-recursive] Error 1 make[1]: *** [all-recursive] Error 1 make: *** [all] Error 2 $ The ldap.h that gets pulled in is from the MacOS SDK There are lots of deprecation warnings from the compiler about LDAP-related stuff defined in the SDK like this: ldap-utils.c:14:8: warning: 'ldap_set_option' is deprecated: first deprecated in macOS 10.11 - use OpenDirectory Framework [-Wdeprecated-declarations] 14 | ret = ldap_set_option(ld, opt, value); | ^ /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/ldap.h:984:1: note: 'ldap_set_option' has been explicitly marked deprecated here That said, sure enough, the SDK's ldap.h doesn't define LDAP_OPT_X_TLS_ECNAME, on which ldap-utils.c depends. The SDK ldap.h defines LDAP_API_VERSION as 3001, but the SDK's ldap_features.h defines LDAP_VENDOR_VERSION as 2.4.28 (aka "20428"). More oddly, I have OpenLDAP v2.6.9 installed via Homebrew, in /opt/homebrew/opt/openldap (where the compiler isn't looking), and yet it also defines LDAP_API_VERSION to be 3001. OTOH, its LDAP_VENDOR_VERSION is 2.6.9 (aka "20609"), and it actually does define LDAP_OPT_X_TLS_ECNAME. Why didn't configure "realize" that the SDK version of OpenLDAP was too old? I guess configure has an insufficient test for the OpenLDAP version required? Should I file a bug against dovecot (on GitHub) for this?
After adding /opt/homebrew/opt/openldap/{include,lib} to CFLAGS and LDFLAGS in my configure invocation, the build finally succeeded. :-) For posterity, here are the final build invocations that worked for me on MacOS 14.6.1 with Xcode 16.0: $ ./autogen.sh $ CFLAGS="-I/opt/homebrew/include -I/opt/homebrew/opt/openldap/include" LDFLAGS="-L/opt/homebrew/lib -L/opt/homebrew/opt/openldap/lib" ./configure --prefix=/opt/homebrew --with-bzlib --with-ldap --with-mysql --with-pam --with-sqlite --without-icu $ PATH="/opt/homebrew/opt/bison/bin:$PATH" make $ along with the following diffs: $ git diff src diff --git a/src/lib-http/http-server.h b/src/lib-http/http-server.h index 91412b570b..c27170777a 100644 --- a/src/lib-http/http-server.h +++ b/src/lib-http/http-server.h @@ -49,8 +49,8 @@ struct http_server_settings { /* The kernel send/receive buffer sizes used for the connection sockets. Configuring this is mainly useful for the test suite. The kernel defaults are used when these settings are 0. */ - size_t socket_send_buffer_size; - size_t socket_recv_buffer_size; + uoff_t socket_send_buffer_size; + uoff_t socket_recv_buffer_size; }; diff --git a/src/lib-var-expand-crypt/Makefile.am b/src/lib-var-expand-crypt/Makefile.am index 8f829bcc02..ff5b83f22c 100644 --- a/src/lib-var-expand-crypt/Makefile.am +++ b/src/lib-var-expand-crypt/Makefile.am @@ -9,6 +9,8 @@ AM_CPPFLAGS = \ var_expand_crypt_la_SOURCES = \ var-expand-crypt.c +# Add in missing link-time library dependency references. +var_expand_crypt_la_LIBADD = ../lib/liblib.la ../lib-dcrypt/libdcrypt.la ../lib-var-expand/libvar_expand.la ../lib-json/libjson.la var_expand_crypt_la_LDFLAGS = $(MODULE_LIBS) -avoid-version -module pkginc_libdir=$(pkgincludedir) $ Note that there were *three* library dependencies missing from src/lib-var-expand-crypt (see the above diff): - dcrypt - var-expand - json
On 15. Jul 2025, at 4.40, rmfuhrer--- via dovecot <dovecot@dovecot.org> wrote:
diff --git a/src/lib-var-expand-crypt/Makefile.am b/src/lib-var-expand-crypt/Makefile.am index 8f829bcc02..ff5b83f22c 100644 --- a/src/lib-var-expand-crypt/Makefile.am +++ b/src/lib-var-expand-crypt/Makefile.am @@ -9,6 +9,8 @@ AM_CPPFLAGS = \ var_expand_crypt_la_SOURCES = \ var-expand-crypt.c
+# rmfuhrer 07/14/2025: Trying to satisfy link-time references to binary_to_hex_append and friends... +var_expand_crypt_la_LIBADD = ../lib/liblib.la <http://liblib.la/> ../lib-dcrypt/libdcrypt.la <http://libdcrypt.la/> ../lib-var-expand/libvar_expand.la <http://libvar_expand.la/> ../lib-json/libjson.la <http://libjson.la/> var_expand_crypt_la_LDFLAGS = $(MODULE_LIBS) -avoid-version -module
pkginc_libdir=$(pkgincludedir) $
Here's a different fix: https://github.com/dovecot/core/commit/16c850b8d7fa781099315b6a7cc68e6fb5cde... On 15. Jul 2025, at 4.40, rmfuhrer--- via dovecot <dovecot@dovecot.org> wrote: diff --git a/src/lib-var-expand-crypt/Makefile.am b/src/lib-var-expand-crypt/Makefile.am index 8f829bcc02..ff5b83f22c 100644 --- a/src/lib-var-expand-crypt/Makefile.am +++ b/src/lib-var-expand-crypt/Makefile.am @@ -9,6 +9,8 @@ AM_CPPFLAGS = \ var_expand_crypt_la_SOURCES = \ var-expand-crypt.c +# rmfuhrer 07/14/2025: Trying to satisfy link-time references to binary_to_hex_append and friends... +var_expand_crypt_la_LIBADD = ../lib/[1]liblib.la ../lib-dcrypt/[2]libdcrypt.la ../lib-var-expand/[3]libvar_expand.la ../lib-json/[4]libjson.la var_expand_crypt_la_LDFLAGS = $(MODULE_LIBS) -avoid-version -module pkginc_libdir=$(pkgincludedir) $ Here's a different fix: [5]https://github.com/dovecot/core/commit/16c850b8d7fa781099315b6a7cc68e6fb5cde... References Visible links 1. http://liblib.la/ 2. http://libdcrypt.la/ 3. http://libvar_expand.la/ 4. http://libjson.la/ 5. https://github.com/dovecot/core/commit/16c850b8d7fa781099315b6a7cc68e6fb5cde...
participants (3)
-
Odhiambo Washington
-
rmfuhrer＠optonline.net
-
Timo Sirainen