It's already well established that sendmail's built in LDAP support has
severe performance problems when used for the access_db.

This post spells out some modifications to allow a more efficient approach in
sendmail's LDAP support, instead of relying on external solutions (like a
milter or socketmap daemon). I've previously posted detailed explanations of
what's wrong with the current data layout and what a better layout would be.
This post re-casts those details into the context of the sendmail LDAP map code.

The key goal is to move the search iteration process from sendmail to LDAP,
and avoid having sendmail generate the hundreds of unnecessary queries that
it currently uses. Accomplishing this goal means the map data must be
organized hierarchically in the LDAP server, so anyone wishing to use this
approach will need to dump all of their current LDAP maps back to flat text
format and reload it using the new format. (That step is pretty simple; the
ldapmap tool that I wrote for my LDAP milter can be re-used for this task.)

One thing to note - this new approach will have the biggest benefit for the
access map. It may also benefit the mailertable and genericstable since they
also do iterated lookups on hierarchically structured keys, but since they do
far fewer queries, the difference will be minimal. It wouldn't affect the
alias map at all since that's just a flat namespace to begin with.

Here's an example set of LDAP entries using the current approach:

dn: sendmailMTAKey=Connect:example.com, sendmailMTAMapName=access,
dc=sendmail, dc=org
objectClass: sendmailMTA
objectClass: sendmailMTAMap
objectClass: sendmailMTAMapObject
sendmailMTAMapName: access
sendmailMTACluster: Servers
sendmailMTAKey: Connect:example.com
sendmailMTAMapValue: OK

dn: sendmailMTAKey=GreetPause:example.com, sendmailMTAMapName=access,
dc=sendmail, dc=org
objectClass: sendmailMTA
objectClass: sendmailMTAMap
objectClass: sendmailMTAMapObject
sendmailMTAMapName: access
sendmailMTACluster: Servers
sendmailMTAKey: GreetPause:example.com
sendmailMTAMapValue: 100

Here's how the LDAP data would look in the new approach:

dn: dc=com, sendmailMTAMapName=access, dc=sendmail, dc=org
objectClass: sendmailMTA
objectClass: sendmailMTAMap
objectClass: sendmailMTAMapObject
dc: com
sendmailMTACluster: Servers

dn: dc=example, dc=com, sendmailMTAMapName=access, dc=sendmail, dc=org
objectClass: sendmailMTA
objectClass: sendmailMTAMap
objectClass: sendmailMTAMapObject
dc: example
sendmailMTACluster: Servers
sendmailMTAMapValue: Connect: OK
sendmailMTAMapValue: GreetPause: 100

(Default values could be provided in the dc=com entry if they were actually
needed.)

The difference is in how the flat map entries are stored in LDAP. Generally a
map entry is just a key-value pair and the keys are unique. A key has the
form [tag:]name where the optional tag can further qualify the key. Instead
of leaving the key as a flat namespace we split it so that the hierarchical
name can be used directly, and store the map value as [tag:]<space>value.

There would need to be a few changes to the sendmail.cf file to take
advantage of the new approach.

I proposed that we add an optional `iterated_map' argument to the relevant
FEATUREs that indicates that the map will handle iteration itself. We would
also add a "-I" option to the map definitions to indicate that the map must
perform iteration.

So to enable this feature you might use:

FEATURE(`access_db', `ldap:-I -v sendmailMTAMapValue' `iterated_map')

To make things explicit I would introduce a "%k" item to represent the
expanded key name. So the map definition would use
ldap -I -b %k,sendmailMTAMapName=access,dc=sendmail,dc=org
-v sendmailMTAMapValue

The expansion of %k for various inputs would be:
example.com dc=example,dc=com
127.0.0.1 dc=1,dc=0,dc=0,dc=127,dc=in-addr
joe.blow@example.com uid=joe.blow,dc=example,dc=com
[fedc:ba98:7654:3210:fedc:ba98:7654:3210]
dc=3210,dc=7654,dc=fedc,dc=3210,dc=7654,dc=ba98,dc =fedc,dc=in6-addr

If the map is configured for use with subdomains, you might have a map entry
like
Connect:.example.com OK

This would become
sendmailMTAMapValue: .Connect: OK
in the above dc=example,dc=com,<base> entry.

Since the changes I've proposed are not backward-compatible, Claus Assman has
asked for strong justification to move ahead with it. Apparently the
miserable performance of the current approach isn't a strong enough
justification by itself. The proposed alternative (just have a cron job
extract the LDAP map into a local map DB) is frankly a gross hack, a bandaid
to make up for a fundamentally broken LDAP client, and it neutralizes one of
the main advantages of using LDAP in the first place - centralized rule
management with distributed access and realtime control.

Personally, as a software engineer, I believe efficiency is part of
correctness - code that produces the right answer, but takes too long to
deliver the answer, is still wrong. It's obvious that not everybody defines
correctness this way.

Speaking as an OpenLDAP developer, I want this fixed because the misuse of
LDAP here gives LDAP a bad name through no fault of ours. Speaking as an
owner of Symas I will fix it anyway, because my customers want it. The
question for the sendmail community is whether you want to have the fix too.
As it stands, Claus Assman is unconvinced that the changes are worth taking
in, even though they will make life easier for everyone and eventually put an
end to the "Sendmail LDAP Performance" questions. You folks in the sendmail
community who are using LDAP - chime in here if it matters to you.
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc
OpenLDAP Core Team http://www.openldap.org/project/

On Jan 28, 10:16 pm, Howard Chu <XYZ....@highlandsun.com> wrote:
> It's already well established that sendmail's built in LDAP support has
> severe performance problems when used for the access_db.

Make a patch that adds either a new map type such as ldapcache or
parameter to ldap which handles converting iterative queries
internally into cacheable data -- theory being that if it is self
encapsulating it is no worse than all the other _FFR features which
can be used by those who know how and can be picked up by sendmail.org
without too much of a fuss.

Butting heads with Claus and sendmail.org s no one. (Disclaimer:
They havent taken any of my feature patches either, so my advice may
be worth diddly.)

Perhaps you could even have it fall back to old style query in certain
circumstances.

On Jan 28, 9:39 pm, "jmai...@ttec.com" <jmai...@ttec.com> wrote:
> On Jan 28, 10:16 pm, Howard Chu <XYZ....@highlandsun.com> wrote:
>
> > It's already well established that sendmail's built in LDAP support has
> > severe performance problems when used for the access_db.

> Make a patch that adds either a new map type such as ldapcache or
> parameter to ldap which handles converting iterative queries
> internally into cacheable data -- theory being that if it is self
> encapsulating it is no worse than all the other _FFR features which
> can be used by those who know how and can be picked up by sendmail.org
> without too much of a fuss.

Thanks for responding.

I already proposed a caching solution, that proposal didn't fly
either. I agree with Claus' reasons there too - the goal is to cut
down the number of map lookup invocations. Caching is only half a
solution, and it adds a lot of unnecessary complexity to the LDAP map
code.

> Butting heads with Claus and sendmail.org s no one. (Disclaimer:
> They havent taken any of my feature patches either, so my advice may
> be worth diddly.)

We may be butting heads, but I posted this thread on his
recommendation.
1) to lay out more details of the proposed changes
2) to get feedback from the newsgroup

> Perhaps you could even have it fall back to old style query in certain
> circumstances.

I think if the map is defined with the -I option there's no reason to
use the old style query. And if the map is not defined with the -I
option, then it will only use the old style query.

On Jan 28, 7:16 pm, Howard Chu <XYZ....@highlandsun.com> wrote:
> If the map is configured for use with subdomains, you might have a map entry
> like
> Connect:.example.com OK
>
> This would become
> sendmailMTAMapValue: .Connect: OK
> in the above dc=example,dc=com,<base> entry.

I received a private email asking about the odd looking notation here.
The reason for this choice is simply that the dot is semantically of
higher precedence than the Connect: keyword.

The existing sendmail maps are pretty generic and don't take advantage
of the semantics of what they're mapping. I suppose generality is nice
for multi-purpose tools, but when you're working with an app with a
well-defined, highly focused purpose like an MTA, I think you should
make use of the application context.

For example, the key layout is irrelevant for a flat file or a hash
structure, but if you used a Btree you could take advantage of the
inherent locality of reference in the keys and get a significant
reduction in DB I/O. As such, the key munging that I've proposed isn't
only beneficial for LDAP queries, it would also improve performance
for local DB lookups.

Take the set of keys

Connect:foo.villain.org REJECT
Connect:bar.friend.com RELAY
To:gone@bar.friend.com REJECT
To:bar.friend.com OK
GreetPause:villain.org 1000

If you actually mapped them according to semantic precedence, like so:
org.villain:GreetPause 1000
org.villain.foo:Connect REJECT
com.friend.bar:Connect: RELAY
com.friend.bar:To OK
com.friend.bar@gone:To REJECT

then for any given connection the majority of your DB lookups are
pretty much guaranteed to hit in the cache, with almost all of the
lookups for a given email address hitting on the same DB page. With
the current naive mapping, it's pretty much guaranteed that all of the
lookups for a given address will hit separate DB pages, and DB caching
will generally be ineffective.

Of course it's not necessary to parse the keys to take advantage of
locality of reference. For sendmail it would be sufficient to blindly
reverse every input keystring before passing it to the DB, and you'd
get all of the cache benefits.

> -- Howard Chu
> Chief Architect, Symas Corp. http://www.symas.com
> Director, Highland Sun http://highlandsun.com/hyc
> OpenLDAP Core Team http://www.openldap.org/project/