February 2016 - dovecot

Redundant and Geobalancing setup
by Cedric Malitte 15 Feb '16

15 Feb '16

Hi, I use dovecot for a long time now, but only as a single isolated server each time. I joined a company a few years back. We had trouble with compagnies hosting our mail, supposedly full redundant and so on. The company is small, but we have many dealers around the world, and it's growing. Mail became the fist choice for clients to contact the dealers. No mail, and we loose sales. For now we have a single server ( with a backup ) on east coast. And sometimes peoples from EU complain about speed.... ah users :) What I'd like to implement is a redundant system with 2 servers, one in NA, one in EU. And I'd also like to be able to add another server if needed on the west coast. Idea is, that if a server goes down, the users will be able to still receive and send mails, and never loose mails. For geobalacing and failover, I read that I can do it with DNS ( I'm with easydns ). I'm at the first stage where I collect informations that I try to understand and foresee a solution. First idea is to set up servers with a mysql master, slaves and a glusterfs in replica mode on the servers. I tried glusterfs on FreeBSD and OMG, it's slow as hell ! ( well maybe it's a trouble on the VMs nics ) On centos it's way better. But I read there might be trouble/index corruption for the mail storage on "shared" space using maildir. I also had a look at dsync, but I wonder if it can be used on more than 2 servers. I found many pages on dovecot clusters using shared storage NFS mounted, but I feel it's not really what I need as the servers will be in different datacenters. So any guide, clue hint would be really appreciated for me to do my homework ! Regards. Cedric

3 5

Re: Question on setting up dovecot
by Jeffery Small 15 Feb '16

15 Feb '16

The problem is solved! A great thanks to everyone for all the valuable information and suggestions you offered. I learned quite a bit, especially about talking to the dovecot server using the telnet/openssl suggestions and the available POP3 commands. Many of you were correct about your diagnosis, but I had some things happening that were creating confusion. Here is a brief summary of what I discovered. There were a few initial dovecot configuration issues which I eventually ironed out and, as I was reporting at the end of last week, the diagnostic logs seemed to indicate that the connection was successful. Yes, it turned out that Outlook was indeed set to leave a copy of the emails on the server for two weeks -- a setting hidden away on a sub-menu. Now, why this did not seem to be the case was because every time Outlook connected to the Solaris system to retrieve mail, all the messages were immediately deleted (despite this setting), so I was expecting the same behavior on the Linux machine and wasn't looking for the setting. There is something different about how the pop3 handler is working on the two systems. The second problem, which some of you guessed, was that the emails left on the server were not being downloaded because when I wasn't looking, Outlook had already downloaded and tagged them. So, even when I thought there was new mail (new to me anyway) in the inbox, Outlook had beaten me to them so that when I looked, I could see no new mail flowing over to the Windows machine. Disabling the retention setting removed the ambiguity and now I see any new mail in the inbox moving to the other system. Thanks again to everyone! -- Jeff

1 0

ANNOTATE plugin? Squirrel uses it for EXPIRATION information
by Heiko Schlittermann 14 Feb '16

14 Feb '16

Hi, it seems that Squirrel mail uses Mailbox annotations for storing Expire times on the Server. It's an Cyrus server currently. (I've no clue how cyrexpire is able to read it's information from the annotiations) Does dovecot support some similiar mechanism? Or - does anybody know anything about how it's supposed to work on Cyrus and how it can be emulated/simulated with dovecot? Thanks in advance, Best regards from Dresden/Germany Viele Grüße aus Dresden Heiko Schlittermann -- SCHLITTERMANN.de ---------------------------- internet & unix support - Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} - gnupg encrypted messages are welcome --------------- key ID: F69376CE - ! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -

2 2

Segmentation fault on doveadm search -A with a huge user base
by Heiko Schlittermann 13 Feb '16

13 Feb '16

Hi, I'm using dovecot 2.2.9 with a director/backend setup. The user base is about 4711 users currently. If I start at one of the directors doveadm search -A all savedbefore 5000d it terminates with doveadm(1rrissma): Error: doveadm server disconnected before handshake: EOF doveadm(1rrissma): Error: 2001:638:913:f33::5:ff:24245: Command search failed for 1phaaman: EOF doveadm(1rrissma): Error: doveadm server failure doveadm: Error: Failed to iterate through some users The backend logs about that ... Feb 12 23:09:41 backend1 dovecot: doveadm(2001:638:913:f33::5:1,1mstrei2): Error: doveadm search: Client sent unknown parameter: ? Feb 12 23:09:41 backend1 dovecot: doveadm: Error: search: invalid option -- '5' Feb 12 23:09:41 backend1 dovecot: doveadm(2001:638:913:f33::5:1,1mstrei2): Error: doveadm search: Client sent unknown parameter: ? Feb 12 23:09:41 backend1 dovecot: doveadm: Error: search: invalid option -- '5' Feb 12 23:09:41 backend1 dovecot: doveadm(2001:638:913:f33::5:1,1mstrei2): Error: doveadm search: Client sent unknown parameter: ? Feb 12 23:09:41 backend1 dovecot: doveadm: Error: search: invalid option -- '0' Feb 12 23:09:41 backend1 dovecot: doveadm(2001:638:913:f33::5:1,1mstrei2): Error: doveadm search: Client sent unknown parameter: ? Feb 12 23:09:41 backend1 dovecot: doveadm: Error: search: invalid option -- '0' Feb 12 23:09:41 backend1 dovecot: doveadm(2001:638:913:f33::5:1,1mstrei2): Error: doveadm search: Client sent unknown parameter: ? Feb 12 23:09:41 backend1 dovecot: doveadm: Error: search: invalid option -- '0' Feb 12 23:09:41 backend1 dovecot: doveadm(2001:638:913:f33::5:1,1mstrei2): Error: doveadm search: Client sent unknown parameter: ? Feb 12 23:09:41 backend1 dovecot: doveadm: Error: search: invalid option -- 'd' Feb 12 23:09:41 backend1 dovecot: doveadm(2001:638:913:f33::5:1,1mstrei2): Error: doveadm search: Client sent unknown parameter: ? Feb 12 23:09:41 backend1 dovecot: doveadm(2001:638:913:f33::5:1,1mstrei2): Fatal: master: service(doveadm): child 11235 killed with signal 11 (core dumped) Is this a known behaviour and fixed already, or do I need to do more investigation? (PS: Running the same command on one of the backends works w/o failure) Best regards from Dresden/Germany Viele Grüße aus Dresden Heiko Schlittermann -- SCHLITTERMANN.de ---------------------------- internet & unix support - Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} - gnupg encrypted messages are welcome --------------- key ID: F69376CE - ! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -

2 2

Lots of zero-byte hard link files in cur (and new/tmp), cannot see messages in folder
by Charles Marcus 12 Feb '16

12 Feb '16

Hello, I have an el-cheapo shared hosting account on Dreamhost, and have had it for a very long time. For the most part everything usually works fairly well, considering I do keep a lot of folders, and mail, on some of my accounts. They are running dovecot, but still don't have a response as to the version, or doveconf -n output yet. My problem is, one of my most used folders, which was working fine up until a week or so ago, stopped loading the messages, and after some frustrating troubleshooting via email with people who don't listen very well, I finally got a tarball of this folder, and they are using maildir. There are about 24,000 messages in there (non-zero-byte files). This number sounds about right. All other folders (including INBOX, Sent, etc) are still working fine. The problem, though, is there are over 815,000 zero-byte-files in the cur directory, all showing as hardlinks (looks like maybe a whole bunch of duplicates for each of the real message files in the cur directory). There are also 43 non-zero-byte message files in the new directory, and 1,515 of these zero-byte hardlinks (to message files in the new directory). There are also no non-zero-byte message files in the tmp directory, but there are 52 of the hardlinks, linked to something in the new directory. I've never seen any of these kinds of zero-byte files before on the one server I managed for a long time (not shared, just used for a single domain). Anyone ever seen this before? Would running: doveadm index -u myuser * or doveadm force-resync -u myuser * be appropriate commands to try to repair the damage (whatever it is)? Any other commands I could suggest running? Thanks. I know I haven't given much to go on. Charles /* */

2 2

Multiple quota rules from LDAP userdb?
by Heiko Schlittermann 12 Feb '16

12 Feb '16

Hello, I'm just asking myself, how I get multiple quota rules into the LDAP userdb. Dovecot is 2.2.9 userdb_attrs = … fooQuotaRule=quota_rule=%$ which allows me to have an user db entry as fooQuotaRule: *:storage=1000:messages=50 fooQuotaRule: Trash:storage=500 But, if I'd like to have another quota rule for a submailbox of that user? doveadm user returns the first field only field value uid 500 gid 500 home /volumes/dovecot/home/4b/heiko mail maildir:~:INBOX=/volumes/dovecot/inbox/4b/heiko:INDEX=/volumes/dovecot/cache/4b/heiko quota_rule *:storage=1000:messages=50 Inventing additional LDAP attributs fooQuotaRule2, ... doesn't scale well. Best regards from Dresden/Germany Viele Grüße aus Dresden Heiko Schlittermann -- SCHLITTERMANN.de ---------------------------- internet & unix support - Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} - gnupg encrypted messages are welcome --------------- key ID: F69376CE - ! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -

2 2

Autoexpunge on INBOX? And disabling autoexpunge for a mailbox via userdb_import?
by Tom Johnson 12 Feb '16

12 Feb '16

I have autoexpunge working for various special_use mailboxes, but I'd like it to optionally be used for users for their inbox as well. I'd also like to be able to disable autoexpunge for some users for certain directories. Is it possible to specify an autoexpunge period for the main inbox? And can autoexpunge be turned off via userdb_import? Here's some detail behind my questions: I have the following in 15-mailboxes.conf, and all is working great for the autoexpunge feature for the Junk and Trash mailboxes: namespace inbox { separator = . mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk autoexpunge = 15d } mailbox Trash { auto = subscribe special_use = \Trash autoexpunge = 15d } mailbox Sent { auto = subscribe special_use = \Sent } } And I can change the autoexpunge field for those mailboxes via userdb_import in a SQL query - that's working great. But I would like for some users to expire the main inbox and the Sent mailbox, and for others for there to be no expiration. Imagine a scenario where there are users who get a maximum of 15 days of storage - everything older than that should expire out. Other users are standard email users - their inboxes, personal folders, Sent mail should never expire. Adding autoexpunge to the main mailbox gives an error: namespace inbox { separator = . autoexpunge = 999d ... ... } doveconf: Fatal: Error in configuration file /etc/dovecot/conf.d/15-mailboxes.conf line 48: Unknown setting: autoexpunge So is there a way to add an autoexpunge setting to the main inbox? And can I set autoexpunge to 0 via userdb_import to disable autoexpunge for a particular mailbox? That doesn't give any error, but does it properly disable autoexpunge for that mailbox? Thanks- Tom

2 1

Re: Question on setting up dovecot
by Jeffery Small 12 Feb '16

12 Feb '16

Many thanks to Joseph Tam and Steffen Kaiser for pointing me to the logging information that I should have been using in the first place. Now that I have separate log files and debugging information, I'm still at a loss to see why the mail is not being pulled over from the Linux system to the Windows 7 machine. To recap, here is the output from doveconf -n: ---------------------------------------------------------------------- # 2.2.18: /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.8 (0c4ae064f307+) # OS: Linux 4.2.0-27-generic x86_64 Ubuntu 15.10 auth_debug = yes auth_verbose = yes disable_plaintext_auth = no info_log_path = /var/log/dovecot.info log_path = /var/log/dovecot mail_debug = yes mail_location = mbox:~/lib/Mail:INBOX=/var/mail/%u passdb { driver = pam } protocols = imap pop3 ssl_cert = </etc/dovecot/dovecot.pem ssl_key = </etc/dovecot/private/dovecot.pem userdb { driver = passwd } userdb { driver = passwd } verbose_ssl = yes ---------------------------------------------------------------------- Here is the configuration logging/debug info: ---------------------------------------------------------------------- log_path = /var/log/dovecot info_log_path = /var/log/dovecot.info auth_verbose = yes auth_debug = yes mail_debug = yes verbose_ssl = yes ---------------------------------------------------------------------- And here is the complete set of Debug/Info information logged when Outlook on the Win7 system attempts a Send/Receive operation. Long lines broken for readability: ---------------------------------------------------------------------- Feb 11 08:47:49 auth: Debug: Loading modules from directory: \ /usr/lib/dovecot/modules/auth Feb 11 08:47:49 auth: Debug: Read auth token secret from \ /var/run/dovecot/auth-token-secret.dat Feb 11 08:47:49 auth: Debug: auth client connected (pid=7861) Feb 11 08:47:49 auth: Debug: client in: AUTH 1 PLAIN \ service=pop3 session=dF2ST4ErtQDMa1sC \ lip=<server_IP> rip=<client_IP> \ lport=110 rport=54453 resp=<hidden> Feb 11 08:47:49 auth-worker(7864): Debug: Loading modules from directory: \ /usr/lib/dovecot/modules/auth Feb 11 08:47:49 auth-worker(7864): Debug: pam(sherril,<client_IP>): \ lookup service=dovecot Feb 11 08:47:49 auth-worker(7864): Debug: pam(sherril,<client_IP>): \ #1/1 style=1 msg=Password: Feb 11 08:47:49 auth: Debug: client passdb out: OK 1 user=sherril Feb 11 08:47:49 auth: Debug: master in: REQUEST 1975517185 7861 1 \ b48910c85f39498e2fe83277d7e94028 session_pid=7866 Feb 11 08:47:49 auth-worker(7864): Debug: pam(sherril,<client_IP>): lookup Feb 11 08:47:49 auth: Debug: master userdb out: USER 1975517185 \ sherril system_groups_user=sherril uid=1001 gid=1001 \ home=/u/sherril Feb 11 08:47:49 pop3-login: Info: Login: user=<sherril>, method=PLAIN, \ rip=<client_IP>, lip=<server_IP>, mpid=7866, \ session=<dF2ST4ErtQDMa1sC> Feb 11 08:47:49 pop3(sherril): Debug: Effective uid=1001, gid=1001, \ home=/u/sherril Feb 11 08:47:49 pop3(sherril): Debug: fs: root=/u/sherril/lib/Mail, \ index=, indexpvt=, control=, inbox=/var/mail/sherril, alt= Feb 11 08:47:49 pop3(sherril): Info: Disconnected: Logged out top=0/0, \ retr=0/0, del=0/3, size=36928 ---------------------------------------------------------------------- There are no errors. We have a pop3 connection. The user's name, uid, gid, password, home directory are all correct. The inbox is listed as /var/mail/sherril which is correct, and this file has the correct ownership and permissions: # ls -l /var/mail/sherril -rw-rw---- 1 sherril mail 194900 Feb 11 09:00 /var/mail/sherril Yet, no mail is pulled over. Are there any clues here? Thanks, -- Jeff

5 6

Re: RFC: HTTP based storage API
by David Niklas 12 Feb '16

12 Feb '16

On 08 Feb 2016, at 11:01, Nagy, Attila <bra(a)fsn.hu> wrote: > Nearly every popular programming language has an LMTP/POP/IMAP > implementation, most of them suck in many different ways. > I don't know any server or library which provides a well-established, > compatible protocol frontend with an open backend API, which could be > used to easily make a custom storage backend for the LMTP/POP/IMAP > frontend in any language, in any programming paradigm, irregardless > of the frontend implementation. Sorry to come in late, but as a curl lover I'd like to know what is wrong with their implementation of pop3 and imap (project homepage curl.haxx.se). Thanks, David

2 1

[BUG] Dovecot 2.2.21 crashing with assertion failed: (!have_wait_unfinished || unfinished_count > 0)
by Gary Stanley 12 Feb '16

12 Feb '16

Hello Dovecot's IMAP process appears to be crashing at random times for a few users, but one is crashing at least 3 to 5 times an hour. I applied these patches per http://dovecot.org/list/dovecot/2016-January/102933.html /https://github.com/dovecot/core/commit/1ddf959a750f3860feff4ab3f0e908f32740978f.diff //https://github.com/dovecot/core/commit/c8e9fa2ffa2566e75f0500808b1bc9bf5d9db024.diff //https://github.com/dovecot/core/commit/15307c2c91854e766bd9fb095d611a29b3f7572e.diff //https://github.com/dovecot/core/commit/c7801f830c7d2e7d340065cdd5a5c795b1726223.diff //https://github.com/dovecot/core/commit/f136b0050b3125b466af73984177250b7ed1a3be.diff / But it's still crashing. # 2.2.21 (5345f22): /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-604.30.3.lve1.3.63.el6.x86_64 x86_64 CloudLinux Server release 6.7 (Aleksei Gubarev) auth_cache_size = 1 k auth_mechanisms = plain login auth_username_chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789!#$-=?^_{}~./@+%" disable_plaintext_auth = no first_valid_uid = 201 mail_plugins = " zlib" maildir_very_dirty_syncs = yes namespace { inbox = yes location = mailbox Archive { auto = create special_use = \Archive } mailbox Archives { auto = no special_use = \Archive } mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = create special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { auto = no special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = INBOX. type = private } passdb { args = /usr/local/cpanel/bin/dovecot-wrap driver = checkpassword } plugin { acl = vfile quota = maildir quota_rule = INBOX.Trash:ignore } protocols = imap pop3 service auth { unix_listener auth-client { mode = 0666 } } service imap-login { client_limit = 500 inet_listener imap { address = * } inet_listener imaps { address = * } process_limit = 50 process_min_avail = 2 service_count = 0 vsz_limit = 64 M } service imap { process_limit = 512 vsz_limit = 256 M } service managesieve-login { client_limit = 500 process_limit = 50 process_min_avail = 2 service_count = 0 vsz_limit = 64 M } service managesieve { process_limit = 512 vsz_limit = 256 M } service pop3-login { client_limit = 500 inet_listener pop3 { address = * } inet_listener pop3s { address = * } process_limit = 50 process_min_avail = 2 service_count = 0 vsz_limit = 64 M } service pop3 { process_limit = 512 vsz_limit = 256 M } ssl_cert = </etc/dovecot/ssl/dovecot.crt ssl_cipher_list = ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA ssl_key = </etc/dovecot/ssl/dovecot.key ssl_protocols = !SSLv2 !SSLv3 userdb { driver = prefetch } userdb { args = /usr/local/cpanel/bin/dovecot-wrap driver = checkpassword } protocol imap { imap_capability = +NAMESPACE imap_idle_notify_interval = 24 mins imap_logout_format = in=%i, out=%o, bytes=%i/%o mail_max_userip_connections = 20 mail_plugins = acl quota imap_quota imap_zlib } protocol pop3 { mail_max_userip_connections = 3 mail_plugins = quota pop3_logout_format = top=%t/%p, retr=%r/%b, del=%d/%m, size=%s, bytes=%i/%o pop3_uidl_format = UID%u-%v } protocol lda { postmaster_address = postmaster(a)example.com } 1 18:23:54 gw-ud3 dovecot: imap(xxxx(a)xxxx.com): Panic: file imap-client.c: line 849 (client_check_command_hangs): assertion failed: (!have_wait_unfinished || unfinished_count > 0) Feb 11 18:23:54 gw-ud3 dovecot: imap(xxxx(a)xxxx.com): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0x8096a) [0x7f996c47e96a] -> /usr/lib64/dovecot/libdovecot.so.0(+0x809d6) [0x7f996c47e9d6]$ Feb 11 18:23:55 gw-ud3 dovecot: imap(xxxx(a)xxxx.com): Fatal: master: service(imap): child 329519 killed with signal 6 (core dumped) Here's the backtrace: (gdb) bt full #0 0x00007f046eb2e625 in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 resultvar = 0 pid = 98454 selftid = 98454 #1 0x00007f046eb2fe05 in abort () at abort.c:92 save_stage = 2 act = {__sigaction_handler = {sa_handler = 0x2, sa_sigaction = 0x2}, sa_mask = {__val = {139657018134932, 1, 18488448, 0, 139657017886537, 139657017552035, 139657017530872, 206158430224, 140725557583888, 140725557583680, 4362456, 140725557583788, 18440536, 0, 0, 6717455502603727992}}, sa_flags = 1635021663, sa_restorer = 0x7ffd38dee060} sigs = {__val = {32, 0 <repeats 15 times>}} #2 0x00007f046ef10a60 in default_fatal_finish (type=<value optimized out>, status=0) at failures.c:201 backtrace = 0x1196158 "/usr/lib64/dovecot/libdovecot.so.0(+0x80a6a) [0x7f046ef10a6a] -> /usr/lib64/dovecot/libdovecot.so.0(+0x80ad6) [0x7f046ef10ad6] -> /usr/lib64/dovecot/libdovecot.so.0(i_error+0) [0x7f046eeb994f] -> dove"... #3 0x00007f046ef10ad6 in i_internal_fatal_handler (ctx=0x7ffd38dee080, format=<value optimized out>, args=<value optimized out>) at failures.c:670 status = 0 #4 0x00007f046eeb994f in i_panic (format=0x18096 <Address 0x18096 out of bounds>) at failures.c:275 ctx = {type = LOG_TYPE_PANIC, exit_status = 0, timestamp = 0x0, timestamp_usecs = 0} args = {{gp_offset = 40, fp_offset = 48, overflow_arg_area = 0x7ffd38dee160, reg_save_area = 0x7ffd38dee0a0}} #5 0x0000000000418247 in client_continue_pending_input () No symbol table info available. #6 0x00007f046ef22fdb in io_loop_call_io (io=0x11c5860) at ioloop.c:559 ioloop = 0x119e740 t_id = 2 __FUNCTION__ = "io_loop_call_io" #7 0x00007f046ef24577 in io_loop_handler_run_internal (ioloop=<value optimized out>) at ioloop-epoll.c:220 ctx = 0x11a0250 events = <value optimized out> event = 0x11a10c0 list = 0x11a1c80 io = <value optimized out> tv = {tv_sec = 29, tv_usec = 999480} events_count = <value optimized out> msecs = <value optimized out> ret = 1 i = <value optimized out> call = <value optimized out> ---Type <return> to continue, or q <return> to quit--- __FUNCTION__ = "io_loop_handler_run_internal" #8 0x00007f046ef23085 in io_loop_handler_run (ioloop=0x119e740) at ioloop.c:607 No locals. #9 0x00007f046ef23228 in io_loop_run (ioloop=0x119e740) at ioloop.c:583 __FUNCTION__ = "io_loop_run" #10 0x00007f046eebeca3 in master_service_run (service=0x119e5e0, callback=<value optimized out>) at master-service.c:640 No locals. #11 0x0000000000424c5c in main () No symbol table info available.

2 2