[Dovecot] Temporary Failure that's Permanent!
Hi;
I continue to be plagued with this error:
Oct 25 10:49:18 myserver postfix/pipe[3712]: 0423257901AB: to=foo@bar.com, relay=dovecot, delay=109318, delays=109318/0.14/0/0.1, dsn=4.3.0, status=deferred (temporary failure
It's been a week that I've been working day and night to get this fixed.
I forgot to add this information:
[root]# doveconf -n auth_mechanisms = plain login first_valid_gid = 12 first_valid_uid = 86 mail_location = maildir:/var/vmail/%d/%u passdb { args = /usr/local/etc/dovecot/sql.conf driver = sql } plugin { quota = maildir:storage=10240:messages=1000 trash = /usr/local/etc/dovecot/trash.conf } protocols = imap pop3 service auth { unix_listener auth-userdb { group = postfix mode = 0660 user = postfix } user = root } ssl_cert =
[root] # vi main.cf queue_directory = /var/spool/postfix myorigin = $mydomain command_directory = /usr/sbin daemon_directory = /usr/libexec/postfix mail_owner = postfix inet_interfaces = all unknown_local_recipient_reject_code = 550 debug_peer_list = sendmail_path = /usr/sbin/sendmail.postfix newaliases_path = /usr/bin/newaliases mailq_path = /usr/bin/mailq setgid_group = postdrop html_directory = no manpage_directory = /usr/local/man sample_directory = /etc/postfix readme_directory = no mydomain = myserver.com mydestination = $mydomain, $myhostname, localhost.$mydomain mail_spool_directory = /var/spool/mail home_mailbox = Mailbox disable_vrfy_command = yes show_user_unknown_table_name = no
data_directory = /var/lib/postfix # --------------- local settings ------------------ myhostname = myserver.com inet_interfaces = localhost, $myhostname mynetworks = $config_directory/mynetworks #mydestination = localhost.$mydomain, localhost, $myhostname #uncomment if you need relay_domains... do not list domains in both relay and virtual relay_domains = proxy:mysql:$config_directory/mysql_relay_domains_maps.cf # ---------------------- VIRTUAL DOMAINS START ---------------------- virtual_mailbox_base = /var/vmail virtual_mailbox_domains = proxy:mysql:$config_directory/mysql_virtual_domains_maps.cf virtual_mailbox_maps = proxy:mysql:$config_directory/mysql_virtual_mailbox_maps.cf virtual_alias_maps = proxy:mysql:$config_directory/mysql_virtual_alias_maps.cf virtual_mailbox_limit_maps = proxy:mysql:/etc/postfix/mysql_virtual_mailbox_limit_maps.cf virtual_minimum_uid = 89 virtual_uid_maps = static:89 virtual_gid_maps = static:89 virtual_transport = dovecot dovecot_destination_recipient_limit = 1 # ---------------------- VIRTUAL DOMAINS END ---------------------- # ---------------------- SASL PART START ---------------------- smtpd_sasl_auth_enable = yes smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination smtpd_sasl_security_options = noanonymous broken_sasl_auth_clients = yes smtpd_sasl_type = dovecot smtpd_sasl_path = /var/spool/postfix/private/auth smtpd_sasl_application_name = smtpd # ---------------------- SASL PART END ---------------------- smtpd_soft_error_limit = 10 smtpd_hard_error_limit = 20 smtpd_helo_required = yes disable_vrfy_command = yes non_fqdn_reject_code = 504 invalid_hostname_reject_code = 450 maps_rbl_reject_code = 554 alias_maps = hash:/etc/aliases reject_unknown_client = false reject_unknown_hostname = false mailbox_command = /usr/local/libexec/dovecot/deliver -f "$SENDER" -a "$RECIPIENT"
[root] # vi master.cf smtp inet n - n - - smtpd submission inet n - - - - smtpd -D -o smtpd_tls_security_level=encrypt -o smtpd_sasl_auth_enable=yes -o smtpd_sasl_type=dovecot -o smtpd_sasl_path=private/auth -o smtpd_sasl_security_options=noanonymous -o smtpd_sasl_local_domain=$myhostname -o smtpd_client_restrictions=permit_sasl_authenticated,reject -o smtpd_sender_login_maps=hash:/etc/postfix/virtual -o smtpd_sender_restrictions=reject_sender_login_mismatch -o smtpd_recipient_restrictions=reject_unknown_recipient_domain,reject_non_fqdn_recipient,permit_sasl_authenticated,reject amavisfeed unix - - n - 2 lmtp -o lmtp_data_done_timeout=1200 -o lmtp_send_xforward_command=yes -o disable_dns_lookups=yes -o max_use=20 587 inet n - n - - smtpd pickup fifo n - n 60 1 pickup cleanup unix n - n - 0 cleanup qmgr fifo n - n 300 1 qmgr tlsmgr unix - - n 1000? 1 tlsmgr rewrite unix - - n - - trivial-rewrite bounce unix - - n - 0 bounce defer unix - - n - 0 bounce trace unix - - n - 0 bounce verify unix - - n - 1 verify flush unix n - n 1000? 0 flush proxymap unix - - n - - proxymap smtp unix - - n - - smtp relay unix - - n - - smtp -o fallback_relay= showq unix n - n - - showq error unix - - n - - error discard unix - - n - - discard local unix - n n - - local virtual unix - n n - - virtual lmtp unix - - n - - lmtp anvil unix - - n - 1 anvil scache unix - - n - 1 scache maildrop unix - n n - - pipe flags=DRhu user=vmail argv=/usr/local/bin/maildrop -d ${recipient} uucp unix - n n - - pipe flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient) ifmail unix - n n - - pipe flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient) bsmtp unix - n n - - pipe flags=Fq. user=foo argv=/usr/local/sbin/bsmtp -f $sender $nexthop $recipient retry unix - - n - - error proxywrite unix - - n - 1 proxymap dovecot unix - n n - - pipe flags=DRhu user=dovecot argv=/usr/local/libexec/dovecot/dovecot-lda -f ${sender} ${recipient} spamfilter unix - n n - - pipe flags=Rq user=spamfilter argv=/usr/local/bin/spamfilter -f ${sender} -- ${recipient}
TIA, Jack
Jack,
A word of advice... you need to take a breath, and START OVER.
Postfix+Dovecot works very well together... you are doing something wrong, and need to figure out what it is.
You are chasing your tail, and will never figure out what you are doing wrong the way you are trying to troubleshoot this right now.
Posting the same question over and over here and on the postfix list will not get you anywhere.
Charles
On 2011-10-25 2:33 PM, Jack Fredrikson jackfredrikson@yahoo.com wrote:
Hi;
I continue to be plagued with this error:
Oct 25 10:49:18 myserver postfix/pipe[3712]: 0423257901AB: to=foo@bar.com, relay=dovecot, delay=109318, delays=109318/0.14/0/0.1, dsn=4.3.0, status=deferred (temporary failure
It's been a week that I've been working day and night to get this fixed.
I forgot to add this information:
[root]# doveconf -n auth_mechanisms = plain login first_valid_gid = 12 first_valid_uid = 86 mail_location = maildir:/var/vmail/%d/%u passdb { args = /usr/local/etc/dovecot/sql.conf driver = sql } plugin { quota = maildir:storage=10240:messages=1000 trash = /usr/local/etc/dovecot/trash.conf } protocols = imap pop3 service auth { unix_listener auth-userdb { group = postfix mode = 0660 user = postfix } user = root } ssl_cert =
[root] # vi main.cf queue_directory = /var/spool/postfix myorigin = $mydomain command_directory = /usr/sbin daemon_directory = /usr/libexec/postfix mail_owner = postfix inet_interfaces = all unknown_local_recipient_reject_code = 550 debug_peer_list = sendmail_path = /usr/sbin/sendmail.postfix newaliases_path = /usr/bin/newaliases mailq_path = /usr/bin/mailq setgid_group = postdrop html_directory = no manpage_directory = /usr/local/man sample_directory = /etc/postfix readme_directory = no mydomain = myserver.com mydestination = $mydomain, $myhostname, localhost.$mydomain mail_spool_directory = /var/spool/mail home_mailbox = Mailbox disable_vrfy_command = yes show_user_unknown_table_name = no
data_directory = /var/lib/postfix # --------------- local settings ------------------ myhostname = myserver.com inet_interfaces = localhost, $myhostname mynetworks = $config_directory/mynetworks #mydestination = localhost.$mydomain, localhost, $myhostname #uncomment if you need relay_domains... do not list domains in both relay and virtual relay_domains = proxy:mysql:$config_directory/mysql_relay_domains_maps.cf # ---------------------- VIRTUAL DOMAINS START ---------------------- virtual_mailbox_base = /var/vmail virtual_mailbox_domains = proxy:mysql:$config_directory/mysql_virtual_domains_maps.cf virtual_mailbox_maps = proxy:mysql:$config_directory/mysql_virtual_mailbox_maps.cf virtual_alias_maps = proxy:mysql:$config_directory/mysql_virtual_alias_maps.cf virtual_mailbox_limit_maps = proxy:mysql:/etc/postfix/mysql_virtual_mailbox_limit_maps.cf virtual_minimum_uid = 89 virtual_uid_maps = static:89 virtual_gid_maps = static:89 virtual_transport = dovecot dovecot_destination_recipient_limit = 1 # ---------------------- VIRTUAL DOMAINS END ---------------------- # ---------------------- SASL PART START ---------------------- smtpd_sasl_auth_enable = yes smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination smtpd_sasl_security_options = noanonymous broken_sasl_auth_clients = yes smtpd_sasl_type = dovecot smtpd_sasl_path = /var/spool/postfix/private/auth smtpd_sasl_application_name = smtpd # ---------------------- SASL PART END ---------------------- smtpd_soft_error_limit = 10 smtpd_hard_error_limit = 20 smtpd_helo_required = yes disable_vrfy_command = yes non_fqdn_reject_code = 504 invalid_hostname_reject_code = 450 maps_rbl_reject_code = 554 alias_maps = hash:/etc/aliases reject_unknown_client = false reject_unknown_hostname = false mailbox_command = /usr/local/libexec/dovecot/deliver -f "$SENDER" -a "$RECIPIENT"
[root] # vi master.cf smtp inet n - n - - smtpd submission inet n - - - - smtpd -D -o smtpd_tls_security_level=encrypt -o smtpd_sasl_auth_enable=yes -o smtpd_sasl_type=dovecot -o smtpd_sasl_path=private/auth -o smtpd_sasl_security_options=noanonymous -o smtpd_sasl_local_domain=$myhostname -o smtpd_client_restrictions=permit_sasl_authenticated,reject -o smtpd_sender_login_maps=hash:/etc/postfix/virtual -o smtpd_sender_restrictions=reject_sender_login_mismatch -o smtpd_recipient_restrictions=reject_unknown_recipient_domain,reject_non_fqdn_recipient,permit_sasl_authenticated,reject amavisfeed unix - - n - 2 lmtp -o lmtp_data_done_timeout=1200 -o lmtp_send_xforward_command=yes -o disable_dns_lookups=yes -o max_use=20 587 inet n - n - - smtpd pickup fifo n - n 60 1 pickup cleanup unix n - n - 0 cleanup qmgr fifo n - n 300 1 qmgr tlsmgr unix - - n 1000? 1 tlsmgr rewrite unix - - n - - trivial-rewrite bounce unix - - n - 0 bounce defer unix - - n - 0 bounce trace unix - - n - 0 bounce verify unix - - n - 1 verify flush unix n - n 1000? 0 flush proxymap unix - - n - - proxymap smtp unix - - n - - smtp relay unix - - n - - smtp -o fallback_relay= showq unix n - n - - showq error unix - - n - - error discard unix - - n - - discard local unix - n n - - local virtual unix - n n - - virtual lmtp unix - - n - - lmtp anvil unix - - n - 1 anvil scache unix - - n - 1 scache maildrop unix - n n - - pipe flags=DRhu user=vmail argv=/usr/local/bin/maildrop -d ${recipient} uucp unix - n n - - pipe flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient) ifmail unix - n n - - pipe flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient) bsmtp unix - n n - - pipe flags=Fq. user=foo argv=/usr/local/sbin/bsmtp -f $sender $nexthop $recipient retry unix - - n - - error proxywrite unix - - n - 1 proxymap dovecot unix - n n - - pipe flags=DRhu user=dovecot argv=/usr/local/libexec/dovecot/dovecot-lda -f ${sender} ${recipient} spamfilter unix - n n - - pipe flags=Rq user=spamfilter argv=/usr/local/bin/spamfilter -f ${sender} -- ${recipient}
TIA, Jack
--
Best regards,
Charles Marcus I.T. Director Media Brokers International, Inc. 678.514.6200 x224 | 678.514.6299 fax
On 10/25/2011 2:34 PM, Charles Marcus wrote:
Jack,
A word of advice... you need to take a breath, and START OVER.
Yep. Start by ditching that old as dirt POS distro known as CentOS. And don't roll-your-own-Dovecot from source. Simply install Debian 6, or get your VPS provider to swap your image for it, and:
~$ aptitude install dovecot-imapd dovecot-common
You'll get version 1.2.15, which is probably fine as you don't need the advanced features of 2.x anyway. Then the only thing to worry about now is getting dovecot.conf setup correctly, which is a lot easier than digging yourself out of your current hole.
-- Stan
*sigh*
As usual Stanley, you do not contribute anything constructive, only trollish jibberish Charles' advice however would be my next step, but at this point I think it's a bit drastic.
On Wed, 2011-10-26 at 02:33 -0500, Stan Hoeppner wrote:
On 10/25/2011 2:34 PM, Charles Marcus wrote:
Jack,
A word of advice... you need to take a breath, and START OVER.
Yep. Start by ditching that old as dirt POS distro known as CentOS. And don't roll-your-own-Dovecot from source. Simply install Debian 6, or get your VPS provider to swap your image for it, and:
~$ aptitude install dovecot-imapd dovecot-common
You'll get version 1.2.15, which is probably fine as you don't need the advanced features of 2.x anyway. Then the only thing to worry about now is getting dovecot.conf setup correctly, which is a lot easier than digging yourself out of your current hole.
On Wed, 26 Oct 2011, Noel Butler wrote:
*sigh*
As usual Stanley, you do not contribute anything constructive, only trollish jibberish
I don't know where both of you live, but can't you just grab a coffee and find out what it is that makes a mockery of any discussion between the two of you? Or why you can't seem to get along? Make sure the coffee isn't too hot so you can at least throw it at one another without getting any lawyer involved :-P
How hard would it be to just not feed the trolls, if you're so convinced Stan is one. He's not even trying to get on your nerves, and yet you take it personal.
I could have gone and *sigh* about Hadmuts not-so-polite last reply a few days ago, but I figured that wouldn't have helped anyone. Probably not even my ego.
Back on topic: there sems to be an issue that doesn't handle temporary failures properly (i.e., it should at least record the details of the failure in some logs). This may very well be related to a locally built Dovecot, or being unfamiliar with Postfix/Dovecot configuration, or both. You could try something like 'su' or 'sudo' to break the transactions into separate parts, to see what's generating the tempfail and why. Maybe we're missing some log entries right above the quoted postfix tempfail, that shows some non-null exit code.
-- Maarten
On 10/26/2011 3:36 AM, Maarten Bezemer wrote:
and find out what it is that makes a mockery of any discussion between the two of you?
Discussion??
I *thoroughly* blocked Noel Butler about a year ago. Should I define *thoroughly*? That means direct to MX, list mail, smoke signals, and yodels from a mountain top.
You're unfairly associating me with this nitwit. I extricated myself from his mud pit long ago.
-- Stan
On Wed, Oct 26, 2011 at 05:39:53AM -0500, Stan Hoeppner wrote:
[..]
Stan, we all know your 'affection' for Noel Butler, as well as we all know him.
But please keep this off list. Like Maarten pointed out, it's not worth the trouble and only bothers everybody.
Dennis
P.S. I know this should be send off list, but you have a habit to block very savagely :)
On Mit, 2011-10-26 at 02:33 -0500, Stan Hoeppner wrote:
On 10/25/2011 2:34 PM, Charles Marcus wrote: [....]
A word of advice... you need to take a breath, and START OVER.
Yep. Start by ditching that old as dirt POS distro known as CentOS.
Dovecot works fine on my CentOS-5.* and -6.
And don't roll-your-own-Dovecot from source. [....]
And I used it self-compiled (because there were at times no current packages) and nowadays an RPM from some repo. Granted with sendmail as MTA but that shouldn't make a big difference.
Bernd-- Bernd Petrovitsch Email : bernd@petrovitsch.priv.at LUGA : http://www.luga.at
On 10/26/2011 8:48 AM, Bernd Petrovitsch wrote:
On Mit, 2011-10-26 at 02:33 -0500, Stan Hoeppner wrote:
On 10/25/2011 2:34 PM, Charles Marcus wrote: [....]
A word of advice... you need to take a breath, and START OVER.
Yep. Start by ditching that old as dirt POS distro known as CentOS.
Dovecot works fine on my CentOS-5.* and -6.
Yes, because you know what you're doing.
And don't roll-your-own-Dovecot from source. [....]
And I used it self-compiled (because there were at times no current packages) and nowadays an RPM from some repo. Granted with sendmail as MTA but that shouldn't make a big difference.
See above comment.
The OP is having trouble with a roll-your-own combo Dovecot and Postfix install. He went this route because the CentOS 5.x packages are old as dirt. Now he's having severe problems getting the RYO stuff working properly.
If he has the option of using a distro with far more current packages, it would be much easier for him to get up and running. Thus my suggestion to give Debian a try. Any distro he's comfortable with and has access to, that has relatively up to date Dovecot and Postfix versions, would be just as suitable.
The bulk of his problems stem from issues revolving around installing from source. Moving to recent distro packages will very likely solve most of his problems.
-- Stan
please do not feed the troll
On Thu, Oct 27, 2011 at 12:54 PM, Dan Swartzendruber dswartz@druber.comwrote:
Ubuntu has a nice dovecot-postfix combo package.
Why do you troll You do not help OP in any way saying, use another distribution, this is a clear sign of TROLL Too many on this list, if you not going to help OP piss off and dont click that reply button and ignore tham, how about we tell you to use another distribution that is current like gentoo, freebsd, slackware, or even a mac everytime you have a problem.
Why Timo let you get away with this so long I cant believe. and if you for one minute think debian is a current you are bigger dickhead than you come across as.
do not reply to me either I killfile trolls and I cant believe I did not fuck you off a long time ago, all you do is go into 15 page gooblygook hoping to dazzle people with your _lack_ of knowledge.
What do they say a little knowledge in the hands of some is .... You remind me of that ninkenpoop Karl from the ubuntu list, maybe you are him in disguise
On Thu, Oct 27, 2011 at 12:51 PM, Stan Hoeppner stan@hardwarefreak.comwrote:
On 10/26/2011 8:48 AM, Bernd Petrovitsch wrote:
On Mit, 2011-10-26 at 02:33 -0500, Stan Hoeppner wrote:
On 10/25/2011 2:34 PM, Charles Marcus wrote: [....]
A word of advice... you need to take a breath, and START OVER.
Yep. Start by ditching that old as dirt POS distro known as CentOS.
Dovecot works fine on my CentOS-5.* and -6.
Yes, because you know what you're doing.
And don't roll-your-own-Dovecot from source. [....]
And I used it self-compiled (because there were at times no current packages) and nowadays an RPM from some repo. Granted with sendmail as MTA but that shouldn't make a big difference.
See above comment.
The OP is having trouble with a roll-your-own combo Dovecot and Postfix install. He went this route because the CentOS 5.x packages are old as dirt. Now he's having severe problems getting the RYO stuff working properly.
If he has the option of using a distro with far more current packages, it would be much easier for him to get up and running. Thus my suggestion to give Debian a try. Any distro he's comfortable with and has access to, that has relatively up to date Dovecot and Postfix versions, would be just as suitable.
The bulk of his problems stem from issues revolving around installing from source. Moving to recent distro packages will very likely solve most of his problems.
-- Stan
On Mit, 2011-10-26 at 21:51 -0500, Stan Hoeppner wrote:
On 10/26/2011 8:48 AM, Bernd Petrovitsch wrote:
On Mit, 2011-10-26 at 02:33 -0500, Stan Hoeppner wrote:
On 10/25/2011 2:34 PM, Charles Marcus wrote: [....]
A word of advice... you need to take a breath, and START OVER. [....] Dovecot works fine on my CentOS-5.* and -6.
Yes, because you know what you're doing.
Every root/sysadmin/administrator is supposed to know what s/he is doing
- even if one runs Ubuntu;-)
[....]
And don't roll-your-own-Dovecot from source.
BTW compiling dovecot is the usual `configure; make; make install' IIRC.
For the configuration: if it doesn't work out of the box, you have to dive into it - even with .rpm/.deb based installs. And if it works out of the box, it may have a too "open" default configuration. SCNR ....
[....]
And I used it self-compiled (because there were at times no current packages) and nowadays an RPM from some repo. Granted with sendmail as MTA but that shouldn't make a big difference. [...] The OP is having trouble with a roll-your-own combo Dovecot and Postfix install. He went this route because the CentOS 5.x packages are old as dirt. Now he's having severe problems getting the RYO stuff working
Yes, because CentOS-5.* is old as dirt as it is basically an extremely stable enterprise distribution.
properly. [...] If he has the option of using a distro with far more current packages, it would be much easier for him to get up and running. Thus my
Does he has that option? Running an old-and-stable distribution may have other reasons than "too lazy to upgrade".
suggestion to give Debian a try. Any distro he's comfortable with and has access to, that has relatively up to date Dovecot and Postfix versions, would be just as suitable.
Debian/stable is (or at least was) usually similar old as SuSE-enterprise and RHEL/CentOS.
The bulk of his problems stem from issues revolving around installing
If you know that exactly where his problems are, perhaps you should help him to solve it and just tell him to start somewhere else from scratch.
from source. Moving to recent distro packages will very likely solve most of his problems.
The most simple usual RHEL/CentOS (and Debian/Stale BTW while we're at it) solution is to try rebuilt the src.rpm (or src.deb, respectively) from the current (or a newer) Fedora. That might or not work without problems (like dependencies on quite newer version of some libraries) but the try is a matter on an hour or so.
Kind regards, Bernd
Bernd Petrovitsch Email : bernd@petrovitsch.priv.at LUGA : http://www.luga.at
On 10/27/2011 7:51 AM, Bernd Petrovitsch wrote:
Debian/stable is (or at least was) usually similar old as SuSE-enterprise and RHEL/CentOS.
This is simply not true. Debian stable has consistently newer packages across the board compared to RHEL or CentOS. Debian Backports offers even newer packages. The Red Hat ecosystem has no equivalent back port project TTBOMK.
-- Stan
Hi,
I've been away for a while, and the previous threads I stopped reading because it makes reading War and Peace look more enticing.
Have you run in debug mode?
On Tue, 2011-10-25 at 11:33 -0700, Jack Fredrikson wrote:
Hi;
I continue to be plagued with this error:
Oct 25 10:49:18 myserver postfix/pipe[3712]: 0423257901AB: to=foo@bar.com, relay=dovecot, delay=109318, delays=109318/0.14/0/0.1, dsn=4.3.0, status=deferred (temporary failure
So, it looks like an LDA issue
It's been a week that I've been working day and night to get this fixed.
I only have v1.2 config files to compare at present, we aborted use of 2.0 for internal reasons, I know (sadly) many things changed in the config format so I wont begin to assume you have correct components in the right blocks because I'd for sure end up being wrong.
What you could try, is looking at the exceed quota examples, and try injecting a message for a user that way using dovecot's deliver and see if it stores the message correctly.
Does pop3 work for the users?
On Tue, 2011-10-25 at 11:33 -0700, Jack Fredrikson wrote:
mailbox_command = /usr/local/libexec/dovecot/deliver -f "$SENDER" -a "$RECIPIENT"
get rid of this ^^^^^^^^^^^^^ in postifx main.cf
dovecot unix - n n - - pipe flags=DRhu user=dovecot argv=/usr/local/libexec/dovecot/dovecot-lda -f ${sender} ${recipient}
in master, try
dovecot unix - n n - - pipe flags=DRhu user=dovecot argv=/usr/local/libexec/dovecot/dovecot-lda -f ${sender} -e -d ${user}@${nexthop}
Are you really using user dovecot?
Did you build this from source, if so show us the options you used
participants (9)
-
Bernd Petrovitsch
-
Charles Marcus
-
Dan Swartzendruber
-
Dennis Guhl
-
Jack Fredrikson
-
Maarten Bezemer
-
Nick Edwards
-
Noel Butler
-
Stan Hoeppner