dovecot-lda crash after upgrade to 2.2.28
Hi!
dovecot: lda(my@email): Panic: file mail-namespace.c: line 709 (mail_namespace_find): assertion failed: (ns != NULL) My Dovecot config stills the same from previous 2.2.27 installation to I believe that problem in new Dovecot code related namespace. System: # uname -v FreeBSD 11.0-RELEASE-p8 #0: Wed Feb 22 06:12:04 UTC 2017
root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC Options: root@beta:/usr/ports/mail/dovecot2 # make showconfig | grep =on DOCS=on: Build and/or install documentation EXAMPLES=on: Build and/or install examples KQUEUE=on: kqueue(2) support LIBWRAP=on: TCP wrapper support LZ4=on: LZ4 compression support MYSQL=on: MySQL database support GSSAPI_BASE=on: Use GSSAPI from base Config: # doveconf -n # 2.2.27 (c0f36b0): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.4.16 (fed8554) # OS: FreeBSD 11.0-RELEASE-p8 amd64 ufs auth_mechanisms = plain login default_login_user = dovecot dict { sqluserquota = mysql:/usr/local/etc/dovecot/dovecot-dict-sql-user.conf } disable_plaintext_auth = no first_valid_gid = 0 first_valid_uid = 25 lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes login_access_sockets = tcpwrap login_greeting = Dovecot ready! login_log_format_elements = user=<%u> method=%m rip=%r lip=%l %c mail_access_groups = mail mail_gid = 6 mail_location = maildir:/var/mail/%d/%n mail_plugins = acl quota trash mail_privileged_group = mail mail_uid = 26 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace { location = maildir:/var/mail/%d/.public:INDEXPVT=/var/mail/%d/%n/public
I posted this problem few days ago in FreeBSD bugtracker https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=217364 So, the problem is in crash of dovecot-lda client while local mailbox delivery with this message in log prefix = public. separator = . subscriptions = no type = public } namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = separator = . type = private } passdb { args = /usr/local/etc/dovecot/dovecot-sql.conf driver = sql } plugin { acl = vfile::cache_secs=300 antispam_backend = mailtrain antispam_mail_notspam = %u-revoke antispam_mail_sendmail = /usr/local/etc/dovecot/move-cmd.sh antispam_mail_sendmail_args = antispam_mail_spam = %u-report antispam_spam = Junk antispam_trash = Trash quota = dict:user_quota::proxy::sqluserquota quota_grace = 10%% quota_rule2 = Trash:storage=+10%% quota_rule3 = Junk:storage=+10%% quota_warning = storage=100%% quota-exceeded 100 %u quota_warning2 = storage=95%% quota-warning 95 %u quota_warning3 = storage=90%% quota-warning 90 %u quota_warning4 = storage=75%% quota-warning 75 %u sieve = ~/dovecot.sieve sieve_before = /usr/local/etc/dovecot/default.sieve sieve_dir = ~/sieve sieve_global_path = /usr/local/etc/dovecot/default.sieve trash = /usr/local/etc/dovecot/dovecot-trash.conf } protocols = imap pop3 lmtp sieve service auth { unix_listener auth-client { mode = 0660 user = mailnull } unix_listener auth-master { mode = 0660 user = mailnull } user = root } service dict { unix_listener dict { mode = 0660 user = mailnull } } service imap-login { chroot = login client_limit = 64 executable = /usr/local/libexec/dovecot/imap-login inet_listener imap { port = 143 } inet_listener imaps { port = 993 } process_limit = 32 process_min_avail = 8 service_count = 1 user = $default_login_user vsz_limit = 64 M } service imap { executable = /usr/local/libexec/dovecot/imap } service managesieve-login { inet_listener sieve { port = 4190 } process_min_avail = 1 service_count = 1 user = $default_login_user vsz_limit = 64 M } service managesieve { process_limit = 10 } service pop3-login { chroot = login client_limit = 64 executable = /usr/local/libexec/dovecot/pop3-login inet_listener pop3 { port = 110 } inet_listener pop3s { port = 995 } process_limit = 32 process_min_avail = 8 service_count = 1 user = $default_login_user vsz_limit = 64 M } service pop3 { executable = /usr/local/libexec/dovecot/pop3 } service quota-warning { executable = script /usr/local/etc/dovecot/quota_warning.sh unix_listener quota-warning { mode = 0660 user = mailnull } } service tcpwrap { unix_listener login/tcpwrap { mode = 0600 user = $default_login_user } } ssl_ca =
-- With best regards, Max Kostikov
BBM: 24CA5DF8 | W: https://kostikov.co
On 28.02.2017 13:16, Max Kostikov wrote:
Hi!
I posted this problem few days ago in FreeBSD bugtracker https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=217364 So, the problem is in crash of dovecot-lda client while local mailbox delivery with this message in log
dovecot: lda(my@email): Panic: file mail-namespace.c: line 709 (mail_namespace_find): assertion failed: (ns != NULL)
Core backtrace would help alot!
Aki Tuomi Dovecot oy
I just recreated configuration of Dovecot 2.2.28 and got the same error but in imap service (it was there in logs too as for dovecot-lda but I don't saw it at time).
Feb 27 20:09:41 beta dovecot: imap(postmaster@peek.ru): Panic: file mail-namespace.c: line 709 (mail_namespace_find): assertion failed: (ns != NULL) Feb 27 20:09:47 beta dovecot: lda(my@domain.ru): Panic: file mail-namespace.c: line 709 (mail_namespace_find): assertion failed: (ns != NULL)
So I show you only imap backtrace below.
# gdb /usr/local/libexec/dovecot/imap 1 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols found)... Attaching to program: /usr/local/libexec/dovecot/imap, process 1 ptrace: Invalid argument. Core was generated by `imap'. Program terminated with signal 6, Aborted. Reading symbols from /usr/local/lib/dovecot/libdovecot-lda.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/dovecot/libdovecot-lda.so.0 Reading symbols from /usr/local/lib/dovecot/libdovecot-storage.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/dovecot/libdovecot-storage.so.0 Reading symbols from /usr/local/lib/dovecot/libdovecot.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/dovecot/libdovecot.so.0 Reading symbols from /lib/libc.so.7...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /usr/lib/libkrb5.so.11...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libkrb5.so.11 Reading symbols from /usr/lib/libgssapi.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgssapi.so.10 Reading symbols from /usr/lib/libgssapi_krb5.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgssapi_krb5.so.10 Reading symbols from /usr/lib/libasn1.so.11...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libasn1.so.11 Reading symbols from /usr/lib/libcom_err.so.5...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libcom_err.so.5 Reading symbols from /lib/libcrypt.so.5...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypt.so.5 Reading symbols from /lib/libcrypto.so.8...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypto.so.8 Reading symbols from /usr/lib/libhx509.so.11...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libhx509.so.11 Reading symbols from /usr/lib/libroken.so.11...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libroken.so.11 Reading symbols from /usr/lib/libwind.so.11...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libwind.so.11 Reading symbols from /usr/lib/libheimbase.so.11...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libheimbase.so.11 Reading symbols from /usr/lib/libprivateheimipcc.so.11...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libprivateheimipcc.so.11 Reading symbols from /lib/libthr.so.3...(no debugging symbols found)...done. Loaded symbols for /lib/libthr.so.3 Reading symbols from /usr/local/lib/dovecot/lib01_acl_plugin.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/dovecot/lib01_acl_plugin.so Reading symbols from /usr/local/lib/dovecot/lib02_imap_acl_plugin.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/dovecot/lib02_imap_acl_plugin.so Reading symbols from /usr/local/lib/dovecot/lib10_quota_plugin.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/dovecot/lib10_quota_plugin.so Reading symbols from /usr/local/lib/dovecot/lib11_imap_quota_plugin.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/dovecot/lib11_imap_quota_plugin.so Reading symbols from /usr/local/lib/dovecot/lib11_trash_plugin.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/dovecot/lib11_trash_plugin.so Reading symbols from /usr/local/lib/dovecot/lib90_antispam_plugin.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/dovecot/lib90_antispam_plugin.so Reading symbols from /usr/local/lib/compat/pkg/nss_winbind.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/compat/pkg/nss_winbind.so.1 Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols found)...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x000000001117655a in thr_kill () from /lib/libc.so.7 [New Thread 13616000 (LWP 100455/<unknown>)] (gdb) quit
Aki Tuomi писал 2017-02-28 13:20:
On 28.02.2017 13:16, Max Kostikov wrote:
Hi!
I posted this problem few days ago in FreeBSD bugtracker https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=217364 So, the problem is in crash of dovecot-lda client while local mailbox delivery with this message in log
dovecot: lda(my@email): Panic: file mail-namespace.c: line 709 (mail_namespace_find): assertion failed: (ns != NULL)
Core backtrace would help alot!
-- With best regards, Max Kostikov
BBM: 24CA5DF8 | W: https://kostikov.co
On February 28, 2017 at 4:32 PM Max Kostikov max@kostikov.co wrote:
I just recreated configuration of Dovecot 2.2.28 and got the same error but in imap service (it was there in logs too as for dovecot-lda but I don't saw it at time).
Feb 27 20:09:41 beta dovecot: imap(postmaster@peek.ru): Panic: file mail-namespace.c: line 709 (mail_namespace_find): assertion failed: (ns != NULL) Feb 27 20:09:47 beta dovecot: lda(my@domain.ru): Panic: file mail-namespace.c: line 709 (mail_namespace_find): assertion failed: (ns != NULL)
So I show you only imap backtrace below.
Can you please issue
bt full
in gdb and post the output here?
Aki
We (FreeBSD) had a user report similar.
On 2/28/17, 9:42 AM, "dovecot on behalf of Aki Tuomi"
> On February 28, 2017 at 4:32 PM Max Kostikov <max@kostikov.co> wrote:
>
>
> I just recreated configuration of Dovecot 2.2.28 and got the same error
> but in imap service (it was there in logs too as for dovecot-lda but I
> don't saw it at time).
> > Feb 27 20:09:41 beta dovecot: imap(postmaster@peek.ru): Panic: file
> > mail-namespace.c: line 709 (mail_namespace_find): assertion failed: (ns
> > != NULL)
> > Feb 27 20:09:47 beta dovecot: lda(my@domain.ru): Panic: file
> > mail-namespace.c: line 709 (mail_namespace_find): assertion failed: (ns
> > != NULL)
>
> So I show you only imap backtrace below.
Can you please issue
bt full
in gdb and post the output here?
Aki
Probably I need to compile 2.2.28 from sources with debug flags. Can you tell me wich flags I need to set for more informative backtrace in gdb?
Aki Tuomi писал 2017-02-28 17:42:
Can you please issue bt full in gdb and post the output here?
-- With best regards, Max Kostikov
BBM: 24CA5DF8 | W: https://kostikov.co
With best regards, Max Kostikov
BBM: 24CA5DF8 | W: https://kostikov.co
Your backtrace would've been informative already if you had issued bt full. Just opening it with gdb is not sufficient.
Aki
On February 28, 2017 at 5:55 PM Max Kostikov max@kostikov.co wrote:
Probably I need to compile 2.2.28 from sources with debug flags. Can you tell me wich flags I need to set for more informative backtrace in gdb?
Aki Tuomi писал 2017-02-28 17:42:
Can you please issue bt full in gdb and post the output here?
-- With best regards, Max Kostikov
BBM: 24CA5DF8 | W: https://kostikov.co
With best regards, Max Kostikov
BBM: 24CA5DF8 | W: https://kostikov.co
Got it. Here is full backtrace output.
# gdb /usr/local/libexec/dovecot/imap 1 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols found)... Attaching to program: /usr/local/libexec/dovecot/imap, process 1 ptrace: Invalid argument. Core was generated by `imap'. Program terminated with signal 6, Aborted. Reading symbols from /usr/local/lib/dovecot/libdovecot-lda.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/dovecot/libdovecot-lda.so.0 Reading symbols from /usr/local/lib/dovecot/libdovecot-storage.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/dovecot/libdovecot-storage.so.0 Reading symbols from /usr/local/lib/dovecot/libdovecot.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/dovecot/libdovecot.so.0 Reading symbols from /lib/libc.so.7...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /usr/lib/libkrb5.so.11...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libkrb5.so.11 Reading symbols from /usr/lib/libgssapi.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgssapi.so.10 Reading symbols from /usr/lib/libgssapi_krb5.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgssapi_krb5.so.10 Reading symbols from /usr/lib/libasn1.so.11...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libasn1.so.11 Reading symbols from /usr/lib/libcom_err.so.5...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libcom_err.so.5 Reading symbols from /lib/libcrypt.so.5...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypt.so.5 Reading symbols from /lib/libcrypto.so.8...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypto.so.8 Reading symbols from /usr/lib/libhx509.so.11...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libhx509.so.11 Reading symbols from /usr/lib/libroken.so.11...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libroken.so.11 Reading symbols from /usr/lib/libwind.so.11...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libwind.so.11 Reading symbols from /usr/lib/libheimbase.so.11...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libheimbase.so.11 Reading symbols from /usr/lib/libprivateheimipcc.so.11...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libprivateheimipcc.so.11 Reading symbols from /lib/libthr.so.3...(no debugging symbols found)...done. Loaded symbols for /lib/libthr.so.3 Reading symbols from /usr/local/lib/dovecot/lib01_acl_plugin.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/dovecot/lib01_acl_plugin.so Reading symbols from /usr/local/lib/dovecot/lib02_imap_acl_plugin.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/dovecot/lib02_imap_acl_plugin.so Reading symbols from /usr/local/lib/dovecot/lib10_quota_plugin.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/dovecot/lib10_quota_plugin.so Reading symbols from /usr/local/lib/dovecot/lib11_imap_quota_plugin.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/dovecot/lib11_imap_quota_plugin.so Reading symbols from /usr/local/lib/dovecot/lib11_trash_plugin.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/dovecot/lib11_trash_plugin.so Reading symbols from /usr/local/lib/dovecot/lib90_antispam_plugin.so...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/dovecot/lib90_antispam_plugin.so Reading symbols from /usr/local/lib/compat/pkg/nss_winbind.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/local/lib/compat/pkg/nss_winbind.so.1 Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols found)...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x000000001117655a in thr_kill () from /lib/libc.so.7 [New Thread 13616000 (LWP 100216/<unknown>)] (gdb) bt #0 0x000000001117655a in thr_kill () from /lib/libc.so.7 #1 0x000000001117652b in raise () from /lib/libc.so.7 #2 0x0000000011176499 in abort () from /lib/libc.so.7 #3 0x0000000010e2af54 in default_error_handler () from /usr/local/lib/dovecot/libdovecot.so.0 #4 0x0000000010e2bc5b in i_set_failure_ignore_errors () from /usr/local/lib/dovecot/libdovecot.so.0 #5 0x0000000010e2b1f9 in i_fatal () from /usr/local/lib/dovecot/libdovecot.so.0 #6 0x0000000010aa190a in mail_namespace_find () from /usr/local/lib/dovecot/libdovecot-storage.so.0 #7 0x000000001402e7e2 in trash_plugin_deinit () from /usr/local/lib/dovecot/lib11_trash_plugin.so #8 0x0000000010aaf20b in hook_mail_user_created () from /usr/local/lib/dovecot/libdovecot-storage.so.0 #9 0x0000000010ab374f in mail_user_init () from /usr/local/lib/dovecot/libdovecot-storage.so.0 #10 0x0000000010ab1c52 in mail_storage_service_next () from /usr/local/lib/dovecot/libdovecot-storage.so.0 #11 0x0000000010ab2435 in mail_storage_service_lookup_next () from /usr/local/lib/dovecot/libdovecot-storage.so.0 #12 0x0000000000425896 in client_create_from_input () #13 0x0000000000425fb2 in main () (gdb) bt full #0 0x000000001117655a in thr_kill () from /lib/libc.so.7 No symbol table info available. #1 0x000000001117652b in raise () from /lib/libc.so.7 No symbol table info available. #2 0x0000000011176499 in abort () from /lib/libc.so.7 No symbol table info available. #3 0x0000000010e2af54 in default_error_handler () from /usr/local/lib/dovecot/libdovecot.so.0 No symbol table info available. #4 0x0000000010e2bc5b in i_set_failure_ignore_errors () from /usr/local/lib/dovecot/libdovecot.so.0 No symbol table info available. #5 0x0000000010e2b1f9 in i_fatal () from /usr/local/lib/dovecot/libdovecot.so.0 No symbol table info available. #6 0x0000000010aa190a in mail_namespace_find () from /usr/local/lib/dovecot/libdovecot-storage.so.0 No symbol table info available. #7 0x000000001402e7e2 in trash_plugin_deinit () from /usr/local/lib/dovecot/lib11_trash_plugin.so No symbol table info available. #8 0x0000000010aaf20b in hook_mail_user_created () from /usr/local/lib/dovecot/libdovecot-storage.so.0 No symbol table info available. #9 0x0000000010ab374f in mail_user_init () from /usr/local/lib/dovecot/libdovecot-storage.so.0 No symbol table info available. #10 0x0000000010ab1c52 in mail_storage_service_next () from /usr/local/lib/dovecot/libdovecot-storage.so.0 No symbol table info available. #11 0x0000000010ab2435 in mail_storage_service_lookup_next () from /usr/local/lib/dovecot/libdovecot-storage.so.0 No symbol table info available. #12 0x0000000000425896 in client_create_from_input () No symbol table info available. #13 0x0000000000425fb2 in main () No symbol table info available. (gdb) quit
Aki Tuomi писал 2017-02-28 18:34:
Your backtrace would've been informative already if you had issued bt full. Just opening it with gdb is not sufficient.
-- With best regards, Max Kostikov
BBM: 24CA5DF8 | W: https://kostikov.co
On February 28, 2017 at 7:43 PM Max Kostikov max@kostikov.co wrote:
Got it. Here is full backtrace output.
Would appear the bug is in 'Trash' plugin. We'll open an issue about this, thank you for reporting this.
Aki
Thank you. Will be waiting for Dovecot update.
Aki Tuomi писал 2017-02-28 20:34:
On February 28, 2017 at 7:43 PM Max Kostikov max@kostikov.co wrote:
Got it. Here is full backtrace output.
Would appear the bug is in 'Trash' plugin. We'll open an issue about this, thank you for reporting this.
Aki
-- With best regards, Max Kostikov
BBM: 24CA5DF8 | W: https://kostikov.co
On 28.02.2017 20:40, Max Kostikov wrote:
Thank you. Will be waiting for Dovecot update.
Aki Tuomi писал 2017-02-28 20:34:
On February 28, 2017 at 7:43 PM Max Kostikov max@kostikov.co wrote:
Got it. Here is full backtrace output.
Would appear the bug is in 'Trash' plugin. We'll open an issue about this, thank you for reporting this.
Aki
Hi!
We believe this issue is now fixed with
https://github.com/dovecot/core/commit/326fb016a23480e4ff8dcc03dc80e76812859...
Aki Tuomi Dovecot oy
Thank you. This patch is fully solves the problem.
Aki Tuomi писал 2017-03-01 14:02:
On 28.02.2017 20:40, Max Kostikov wrote:
Thank you. Will be waiting for Dovecot update.
Aki Tuomi писал 2017-02-28 20:34:
On February 28, 2017 at 7:43 PM Max Kostikov max@kostikov.co wrote:
Got it. Here is full backtrace output.
Would appear the bug is in 'Trash' plugin. We'll open an issue about this, thank you for reporting this.
Aki
Hi!
We believe this issue is now fixed with
https://github.com/dovecot/core/commit/326fb016a23480e4ff8dcc03dc80e76812859...
Aki Tuomi Dovecot oy
-- With best regards, Max Kostikov
BBM: 24CA5DF8 | W: https://kostikov.co
participants (3)
-
Aki Tuomi
-
Larry Rosenman
-
Max Kostikov