Hi,

I've just started to use postfix, so if I missed something obvious please
don't shoot me

I've setup postfix to accept mail for a couple of domains. I use a mysql
backend to store the information for those virtual domains. As a test I
shut down the mysql backend while postfix was running. I then started a
SMTP session using telnet and to my surprise the session hung after
the MAIL FROM command. I'd expect a 4xx error after RCPT TO, but I'm not
even getting at that stage.

The logs are complaining about trivial-rewrite not being able to talk to
mysqld, which is understandable. Here's a piece of the log, I've reformatted
it so it's a bit more readable and doesn't exceed 80 columns

------//------
Apr 13 21:51:40 [postfix/trivial-rewrite] warning: connect to mysql server
unix:/var/run/mysqld/mysqld.sock: Can't connect to local MySQL server
through socket '/var/run/mysqld/mysqld.sock' (2)
Apr 13 21:51:40 [postfix/trivial-rewrite] fatal:
mysql:/etc/postfix/mysql/virtual_alias_maps.cf(0,100): table lookup problem
Apr 13 21:51:41 [postfix/smtpd] warning: premature end-of-input on
private/rewrite socket while reading input attribute name
Apr 13 21:51:41 [postfix/smtpd] warning: problem talking to service rewrite:
Success
Apr 13 21:51:41 [postfix/master] warning: process
/usr/lib/postfix/trivial-rewrite pid 30844 exit status 1
Apr 13 21:51:41 [postfix/master] warning: /usr/lib/postfix/trivial-rewrite:
bad command startup -- throttling
------//------

Is there any way to get postfix to accept the MAIL FROM and then, after the
RCPT TO, return a 4xx code?

Thanx in advance,

--
Jeffrey Overman

Everybody needs to believe in something, I believe I'll have another beer...

Jeffrey Overman wrote:
> Hi,
>
> I've just started to use postfix, so if I missed something obvious please
> don't shoot me
>
> I've setup postfix to accept mail for a couple of domains. I use a mysql
> backend to store the information for those virtual domains. As a test I
> shut down the mysql backend while postfix was running. I then started a
> SMTP session using telnet and to my surprise the session hung after
> the MAIL FROM command. I'd expect a 4xx error after RCPT TO, but I'm not
> even getting at that stage.
>
> The logs are complaining about trivial-rewrite not being able to talk to
> mysqld, which is understandable. Here's a piece of the log, I've reformatted
> it so it's a bit more readable and doesn't exceed 80 columns
>
> ------//------
> Apr 13 21:51:40 [postfix/trivial-rewrite] warning: connect to mysql server
> unix:/var/run/mysqld/mysqld.sock: Can't connect to local MySQL server
> through socket '/var/run/mysqld/mysqld.sock' (2)
> Apr 13 21:51:40 [postfix/trivial-rewrite] fatal:
> mysql:/etc/postfix/mysql/virtual_alias_maps.cf(0,100): table lookup problem
> Apr 13 21:51:41 [postfix/smtpd] warning: premature end-of-input on
> private/rewrite socket while reading input attribute name
> Apr 13 21:51:41 [postfix/smtpd] warning: problem talking to service rewrite:
> Success
> Apr 13 21:51:41 [postfix/master] warning: process
> /usr/lib/postfix/trivial-rewrite pid 30844 exit status 1
> Apr 13 21:51:41 [postfix/master] warning: /usr/lib/postfix/trivial-rewrite:
> bad command startup -- throttling
> ------//------
>
> Is there any way to get postfix to accept the MAIL FROM and then, after the
> RCPT TO, return a 4xx code?
>
> Thanx in advance,
>

The MYSQL.readme says:

The MySQL configuration interface setup allows for multiple mysql databases:
you can use one for a virtual table, one for an access table, and one for an
aliases table if you want.

Since sites that have a need for multiple mail exchangers may enjoy the
convenience of using a networked mailer database, but do not want to introduce
a single point of failure to their system, we've included the ability to have
Postfix reference multiple hosts for access to a single mysql map. This will
work if sites set up mirrored mysql databases on two or more hosts. Whenever
queries fail with an error at one host, the rest of the hosts will be tried in
random order. If no mysql server hosts are reachable, then mail will be
deferred until at least one of those hosts is reachable.

It sounds like it's supposed to time out after some period (5 min?) and then issue a 4xx code.

--
Greg

On Fri, 14 Apr 2006 16:36:12 +0000, Greg Hackney wrote:

> Jeffrey Overman wrote:
>> Hi,
>>
>> I've just started to use postfix, so if I missed something obvious please
>> don't shoot me
>>
>> I've setup postfix to accept mail for a couple of domains. I use a mysql
>> backend to store the information for those virtual domains. As a test I
>> shut down the mysql backend while postfix was running. I then started a
>> SMTP session using telnet and to my surprise the session hung after
>> the MAIL FROM command. I'd expect a 4xx error after RCPT TO, but I'm not
>> even getting at that stage.
>>
>> The logs are complaining about trivial-rewrite not being able to talk to
>> mysqld, which is understandable. Here's a piece of the log, I've reformatted
>> it so it's a bit more readable and doesn't exceed 80 columns
>>
>> ------//------
>> Apr 13 21:51:40 [postfix/trivial-rewrite] warning: connect to mysql server
>> unix:/var/run/mysqld/mysqld.sock: Can't connect to local MySQL server
>> through socket '/var/run/mysqld/mysqld.sock' (2)
>> Apr 13 21:51:40 [postfix/trivial-rewrite] fatal:
>> mysql:/etc/postfix/mysql/virtual_alias_maps.cf(0,100): table lookup problem
>> Apr 13 21:51:41 [postfix/smtpd] warning: premature end-of-input on
>> private/rewrite socket while reading input attribute name
>> Apr 13 21:51:41 [postfix/smtpd] warning: problem talking to service rewrite:
>> Success
>> Apr 13 21:51:41 [postfix/master] warning: process
>> /usr/lib/postfix/trivial-rewrite pid 30844 exit status 1
>> Apr 13 21:51:41 [postfix/master] warning: /usr/lib/postfix/trivial-rewrite:
>> bad command startup -- throttling
>> ------//------
>>
>> Is there any way to get postfix to accept the MAIL FROM and then, after the
>> RCPT TO, return a 4xx code?
>>
>> Thanx in advance,
>>
>
> The MYSQL.readme says:
>
> The MySQL configuration interface setup allows for multiple mysql databases:
> you can use one for a virtual table, one for an access table, and one for an
> aliases table if you want.
>
> Since sites that have a need for multiple mail exchangers may enjoy the
> convenience of using a networked mailer database, but do not want to introduce
> a single point of failure to their system, we've included the ability to have
> Postfix reference multiple hosts for access to a single mysql map. This will
> work if sites set up mirrored mysql databases on two or more hosts. Whenever
> queries fail with an error at one host, the rest of the hosts will be tried in
> random order. If no mysql server hosts are reachable, then mail will be
> deferred until at least one of those hosts is reachable.
>
> It sounds like it's supposed to time out after some period (5 min?) and then issue a 4xx code.

One would indeed think it'd time out. However, I've got a session open
for more than 45 minutes and it still didn't time out The only timeout
options I can find with a value higher than 45 minutes are:

daemon_timeout = 18000s
smtp_tls_session_cache_timeout = 3600s
smtpd_tls_session_cache_timeout = 3600s

The only relevant option seems to be the daemon_timeout one. However, I'm
a bit reluctant to set that one too low. Maybe 5 hours is a bit too much,
but I'd like to see a 4xx response within a minute if my mysql backend is
down and I don't want to set the daemon_timeout to 1 minute as virus/spam
scanning might take longer than that with large emails.

Another thing I still don't understand is why the session hangs after MAIL
FROM as I don't have any access restrictions which depend on the mysql
backend. It'd seem logical to time out after RCPT TO as most recipient
information is stored in the database.

Any ideas?

--
Jeffrey Overman

Everybody needs to believe in something, I believe I'll have another beer...

On Sat, 15 Apr 2006 12:32:26 +0200, Jeffrey Overman wrote:

[...]

> daemon_timeout = 18000s
> smtp_tls_session_cache_timeout = 3600s
> smtpd_tls_session_cache_timeout = 3600s
>
> The only relevant option seems to be the daemon_timeout one. However, I'm
> a bit reluctant to set that one too low. Maybe 5 hours is a bit too much,
> but I'd like to see a 4xx response within a minute if my mysql backend is
> down and I don't want to set the daemon_timeout to 1 minute as virus/spam
> scanning might take longer than that with large emails.

As a test I set the daemon_timeout to 1 minute. That doesn't much as
the connection is dropped without issuing a 4xx code.

Greetz

--
Jeffrey Overman

Everybody needs to believe in something, I believe I'll have another beer...

Jeffrey Overman wrote:
> On Sat, 15 Apr 2006 12:32:26 +0200, Jeffrey Overman wrote:
>
> [...]
>
>
>> daemon_timeout = 18000s
>> smtp_tls_session_cache_timeout = 3600s
>> smtpd_tls_session_cache_timeout = 3600s
>>
>>The only relevant option seems to be the daemon_timeout one. However, I'm
>>a bit reluctant to set that one too low. Maybe 5 hours is a bit too much,
>>but I'd like to see a 4xx response within a minute if my mysql backend is
>>down and I don't want to set the daemon_timeout to 1 minute as virus/spam
>>scanning might take longer than that with large emails.
>
>
> As a test I set the daemon_timeout to 1 minute. That doesn't much as
> the connection is dropped without issuing a 4xx code.
>
> Greetz

I'm running the Snapshot version of Postfix. On mine, for testing, I configured
in a mysql query to check recipient validity, but purposely don't have the mysql deamon running.

Right after the RCPT TO, the SMTP connection just drops immediately, with no return code sent.


Apr 16 11:28:57 oldhippie postfix/smtpd[29279]: warning: connect to mysql server 127.0.0.1: Can't connect to MySQL server on '127.0.0.1' (111)
Apr 16 11:28:57 oldhippie postfix/smtpd[29279]: fatal: mysql:/etc/postfix/recipients.cf: table lookup problem
Apr 16 11:28:58 oldhippie postfix/master[29216]: warning: process /usr/libexec/postfix/smtpd pid 29279 exit status 1
Apr 16 11:28:58 oldhippie postfix/master[29216]: warning: /usr/libexec/postfix/smtpd: bad command startup -- throttling

--
Greg

On Sun, 16 Apr 2006 16:39:18 +0000, Greg Hackney wrote:

> Jeffrey Overman wrote:
>> On Sat, 15 Apr 2006 12:32:26 +0200, Jeffrey Overman wrote:
>>
>> [...]
>>
>>
>>> daemon_timeout = 18000s
>>> smtp_tls_session_cache_timeout = 3600s
>>> smtpd_tls_session_cache_timeout = 3600s
>>>
>>>The only relevant option seems to be the daemon_timeout one. However, I'm
>>>a bit reluctant to set that one too low. Maybe 5 hours is a bit too much,
>>>but I'd like to see a 4xx response within a minute if my mysql backend is
>>>down and I don't want to set the daemon_timeout to 1 minute as virus/spam
>>>scanning might take longer than that with large emails.
>>
>>
>> As a test I set the daemon_timeout to 1 minute. That doesn't much as
>> the connection is dropped without issuing a 4xx code.
>>
>> Greetz
>
> I'm running the Snapshot version of Postfix. On mine, for testing, I configured
> in a mysql query to check recipient validity, but purposely don't have the mysql deamon running.
>
> Right after the RCPT TO, the SMTP connection just drops immediately,
> with no return code sent.

Hmm, in my opinion this violates RFC 2821 section 4.2.

------//------
4.2 SMTP Replies

Replies to SMTP commands serve to ensure the synchronization of
requests and actions in the process of mail transfer and to guarantee
that the SMTP client always knows the state of the SMTP server. Every
command MUST generate exactly one reply.
------//------

Just dropping the connection doesn't seem right to me. It does the job,
but it's not a very elegant solution.

I just ran a similar test on another MTA of mine, which runs exim 4.61,
and it returns a 451 code immediately when the mysql backend is down.
That's what I'd expect it to do.

Do you think it's worth mentioning this issue to the postfix developers?
Or is this behaviour intentional? In which case I'll probably stop using
postfix, although I like a lot of its features.

Greetz,

--
Jeffrey Overman

Everybody needs to believe in something, I believe I'll have another beer...

> Do you think it's worth mentioning this issue to the postfix developers?

Yes, I just sent them a query.
--
Greg

> Do you think it's worth mentioning this issue to the postfix developers?

The first reply came from Wietse Venema (the author of Postfix) within a
couple of hours:

That code pre-dates the use of SQL for SMTP server access maps;
when a DB or DBM lookup fails the system is so sick that continuing
makes no sense.

It should not be difficult to replace this by code that pretends
that the table returns "450 4.3.5 Server configuration error"
and then logs a nasty warning to the maillog file.


> Or is this behaviour intentional? In which case I'll probably stop using
> postfix, although I like a lot of its features.

Then a few minutes later, he created the following patch for smtpd_check.c
--
Greg



*** ./smtpd_check.c- Fri Mar 24 15:45:43 2006
--- ./smtpd_check.c Mon Apr 17 10:39:17 2006
***************
*** 2258,2265 ****
CHK_ACCESS_RETURN(check_table_result(state, table, value, name,
reply_name, reply_class,
def_acl), FOUND);
! if (dict_errno != 0)
! msg_fatal("%s: table lookup problem", table);
}
CHK_ACCESS_RETURN(SMTPD_CHECK_DUNNO, MISSED);
}
--- 2258,2270 ----
CHK_ACCESS_RETURN(check_table_result(state, table, value, name,
reply_name, reply_class,
def_acl), FOUND);
! if (dict_errno != 0) {
! msg_warn("%s: table lookup problem", table);
! value = "450 4.3.0 Server configuration error";
! CHK_ACCESS_RETURN(check_table_result(state, table, value, name,
! reply_name, reply_class,
! def_acl), FOUND);
! }
}
CHK_ACCESS_RETURN(SMTPD_CHECK_DUNNO, MISSED);
}
***************
*** 2299,2306 ****
CHK_DOMAIN_RETURN(check_table_result(state, table, value,
domain, reply_name, reply_class,
def_acl), FOUND);
! if (dict_errno != 0)
! msg_fatal("%s: table lookup problem", table);
}
/* Don't apply subdomain magic to numerical hostnames. */
if (maybe_numerical
--- 2304,2316 ----
CHK_DOMAIN_RETURN(check_table_result(state, table, value,
domain, reply_name, reply_class,
def_acl), FOUND);
! if (dict_errno != 0) {
! msg_warn("%s: table lookup problem", table);
! value = "450 4.3.0 Server configuration error";
! CHK_DOMAIN_RETURN(check_table_result(state, table, value,
! domain, reply_name, reply_class,
! def_acl), FOUND);
! }
}
/* Don't apply subdomain magic to numerical hostnames. */
if (maybe_numerical
***************
*** 2353,2360 ****
CHK_ADDR_RETURN(check_table_result(state, table, value, address,
reply_name, reply_class,
def_acl), FOUND);
! if (dict_errno != 0)
! msg_fatal("%s: table lookup problem", table);
}
flags = PARTIAL;
} while (split_at_right(addr, delim));
--- 2363,2375 ----
CHK_ADDR_RETURN(check_table_result(state, table, value, address,
reply_name, reply_class,
def_acl), FOUND);
! if (dict_errno != 0) {
! msg_warn("%s: table lookup problem", table);
! value = "450 4.3.0 Server configuration error";
! CHK_ADDR_RETURN(check_table_result(state, table, value, address,
! reply_name, reply_class,
! def_acl), FOUND);
! }
}
flags = PARTIAL;
} while (split_at_right(addr, delim));

On Mon, 17 Apr 2006 16:26:20 +0000, Greg Hackney wrote:

>> Do you think it's worth mentioning this issue to the postfix developers?
>
> The first reply came from Wietse Venema (the author of Postfix) within a
> couple of hours:
>
> That code pre-dates the use of SQL for SMTP server access maps;
> when a DB or DBM lookup fails the system is so sick that continuing
> makes no sense.

Ah, that explains it

> It should not be difficult to replace this by code that pretends
> that the table returns "450 4.3.5 Server configuration error"
> and then logs a nasty warning to the maillog file.

That'd be nice.

>> Or is this behaviour intentional? In which case I'll probably stop using
>> postfix, although I like a lot of its features.
>
> Then a few minutes later, he created the following patch for smtpd_check.c

Thanx.

However, after patching smtpd_check.c, recompiling and reinstalling postfix
the session still hung after MAIL FROM Because I was under some pressure
to get the MTA up and running I went for exim, which handles the situation
perfect.

I might give postfix another go in the future, for now I'm going with exim.

Thanx for your and Wietse's effort!

--
Jeffrey Overman

Everybody needs to believe in something, I believe I'll have another beer...

> after patching smtpd_check.c, recompiling and reinstalling postfix
> the session still hung after MAIL FROM

If you have the time to explain, what was your particular testing scenario?

The way I tested was to configure the use of a mysql recipient map for RCPT TO,
and then defined a hostname that wasn't running a MySQL server process.

--
Greg

On Tue, 25 Apr 2006 18:19:44 +0000, Greg Hackney wrote:

>> after patching smtpd_check.c, recompiling and reinstalling postfix
>> the session still hung after MAIL FROM
>
> If you have the time to explain, what was your particular testing scenario?

Sure, no problem. If the problem gets solved, I'll definitely take
another look at postfix.

> The way I tested was to configure the use of a mysql recipient map for RCPT TO,
> and then defined a hostname that wasn't running a MySQL server process.

Below you'll find the main.cf I used, with some details obscured.

------//------

# /etc/postfix/main.cf

################################################## ######################
# Server settings #
################################################## ######################
queue_directory = /var/spool/postfix
command_directory = /usr/sbin
daemon_directory = /usr/lib/postfix
html_directory = /usr/share/doc/postfix-2.2.10/html
manpage_directory = /usr/share/man
sample_directory = /etc/postfix
readme_directory = /usr/share/doc/postfix-2.2.10/readme
mail_spool_directory = /var/spool/mail

sendmail_path = /usr/sbin/sendmail
newaliases_path = /usr/bin/newaliases
mailq_path = /usr/bin/mailq

setgid_group = postdrop
mail_owner = postfix

smtpd_banner = $myhostname ESMTP $mail_name

local_destination_concurrency_limit = 2
default_destination_concurrency_limit = 20

myhostname = mail.domain.tld
mydomain = domain.tld
myorigin = $mydomain

inet_interfaces = $myhostname localhost
smtp_bind_address = xxx.xxx.xxx.xxx

mynetworks_style = host

################################################## ######################
# Mail delivery #
################################################## ######################
mydestination = $myhostname localhost.$mydomain localhost

home_mailbox = .maildir/

local_transport = error:local delivery is disabled
virtual_transport = virtual

transport_maps = mysql:/etc/postfix/mysql/transport_maps.cf

################################################## ######################
# Authentication #
################################################## ######################
smtpd_sasl_auth_enable = yes
smtpd_sasl2_auth_enable = yes
smtpd_sasl_security_options = noanonymous
broken_sasl_auth_clients = yes
smtpd_sasl_local_domain =

smtpd_recipient_restrictions = permit_sasl_authenticated,
permit_mynetworks,
reject_unauth_destination

smtp_use_tls = yes
smtp_tls_note_starttls_offer = yes
smtpd_use_tls = yes
smtpd_tls_key_file = /etc/postfix/tls/mail.domain.tld.key.pem
smtpd_tls_cert_file = /etc/postfix/tls/mail.domain.tld.cert.pem
smtpd_tls_CAfile = /etc/postfix/tls/cacert.pem
smtpd_tls_loglevel = 3
smtpd_tls_received_header = yes
smtpd_tls_session_cache_timeout = 3600s
tls_random_source = dev:/dev/urandom

################################################## ######################
# Virtual user setup #
################################################## ######################
virtual_mailbox_domains = mysql:/etc/postfix/mysql/virtual_domains.cf

virtual_minimum_uid = 1000
virtual_uid_maps = static:60004
virtual_gid_maps = static:60004
# the bogus_mailbox_maps.cf is required because apparently postfix
# first looks for a valid mailbox before looking at the transport_maps
virtual_mailbox_maps = mysql:/etc/postfix/mysql/virtual_mailbox_maps.cf
mysql:/etc/postfix/mysql/bogus_mailbox_maps.cf
virtual_alias_maps = mysql:/etc/postfix/mysql/virtual_alias_maps.cf
virtual_mailbox_base = /

------//------

When I use above config, and shutdown mysqld, any SMTP session seems to
hang after MAIL FROM. I've also tried to add virtual_mailbox_domains
with fixed values for my 3 domains, it didn't make a difference
unfortunately.

If you need more information, let me know.

Greetz,

--
Jeffrey Overman

Everybody needs to believe in something, I believe I'll have another beer...

Jeffrey Overman wrote:

> When I use above config, and shutdown mysqld, any SMTP session seems to
> hang after MAIL FROM. I've also tried to add virtual_mailbox_domains
> with fixed values for my 3 domains, it didn't make a difference
> unfortunately.
>
> If you need more information, let me know.


Here's what I've got so far. The problem occurs in the area of using MySQL tables for
virtual_mailbox_domains, virtual_mailbox_maps, virtual_alias_maps, virtual_alias_domains.

One of the code developers, Victor Duchovni, just wrote:

When this uses SQL, the trivial rewrite service (and indirectly the
queue manager) are subject to SQL latency, availability limitations, ...

DO NOT (on critical systems) use SQL or LDAP in this context. Postfix
has no support for an unreliable trivial-rewrite service (it is supposed
to be trivial and correspondingly reliable).

He's also adamant about not using SQL and LDAP for transport maps.

Nonetheless, he had me try something: I changed virtual_mailbox_domains and virtual_alias_domains
to a list of names within the main.cf file, rather than pointing to MySQL tables. After that,
virtual_mailbox_maps and virtual_alias_maps produced an immediate 4XX code when the MySQL server
was unavailable.

More later, maybe.
--
Greg