Re: Er.... for Sendmail users......

First of all I deal with small mail servers. My current client only has
9 users. That includes me. Most of what you are talking about doesn't
apply to small mail servers.

Matthew Schumacher wrote:
>
> Interesting, a sendmail debate on the mailing list and I somehow missed
> it :)
>
> Damien, I mean no disrespect, but the solution you offer above has some
> pretty major issues when it comes to serving any real amount of load.
> While it may be simple for you to set up, it is nowhere close to as
> efficient as a sendmail/mimedefang/cyrus system.
>
> Here are the major problems with it:
>
> 1. Procmail. Procmail doesn't get called until the smtp processing
> engine is done with the smtp transaction. This means that delivery
> issues like quota expired are not handled until after the smtp
> transaction is complete. When you do your spam and virus filtering at
> this level then you are pretty much forced to generate a new bounce
> message and send it back to the reply-to address or skip notifications
> altogether. Either way, your dealing with double bounces or all of the
> messages you reject go into the bit bucket.
>
I went with the quick and dirty solution for this. If I had a large mail
server with lots of users I would look for something better.
> 2. Performance. By starting and stopping procmail (and probably
> formail) on every message you are waisting a lot of resources on
> something that really should stay in memory. This is why clamd and
> spamd are run as a daemon. They are always running ready to process
> mail. Now I'm sure you can argue that the difference is small, but on a
> busy mail system every little bit adds up to a lot.
>
Again, I went with the quick and dirty solution. There may be a way to
connect Postfix, SpamAssassin and ClamAV directly. However, most of the
Postfix examples use Procmail or Amavisd. In my case the overhead is
nothing. I only have 9 users.
> 3. IMAP performance. Most IMAP/POP3 servers store messages in the mbox
> or maildir format. Both have pretty big performance issues. The mbox
> format is just a large file with the messages all concatenated together,
> while maildir has a directory with a file for each message. Mbox has a
> very slow startup time because the mail system has to open this huge
> text file and start parsing, but once the parsing is done it can stream
> the headers pretty fast. Maildir has the opposite problem. If you open
> one message it can open a single text file and stream it really fast,
> but if you ask it for all of your headers, maildir has to open, parse,
> and close every single file in the dir. Either way, a mailbox with a
> couple of thousand messages starts getting real slow. Since your using
> procmail I'll assume your running mbox and either have a small mailbox
> or wait for it to open.
>
> Cyrus has the best of both worlds. It stores its messages like maildir,
> but then keeps the headers and other information indexed in BDB or
> skiplist format. So opening a folder, streaming headers, and opening a
> file are all very fast. It also can update a search index nightly so
> searching your mailbox is very fast. My IMAP server on a very modest
> box is able to search the BODY of my 4525 message INBOX in about 3
> seconds. I doubt Courier-IMAP could even open my mailbox that fast.
>
I'm using IMAP over SSL and I don't notice any problems. Email messages
open as if they were local files. On a large mail server this might be a
problem. However, if that happens I would start farming things out. That
way you spread the load over multiple servers.
> 4. Efficiency. Sendmail uses milters to filter mail which means that
> sendmail can do it's filtering in real time (during the smtp
> transaction) against 1 or more milters. This means that you can accept
> or reject the message in SMTP and never again send out a bounce message
> that will almost certainly double bounce back to you. My mail systems
> will either except the message and write it to disk, or they reject the
> message and tell you why. Since these milters run in memory as a daemon
> (and can use tcp sockets so they don't even need to be on the same host)
> they perform very well. Mimedefang is a perl based milter that has a
> multiplexor that keeps a number of perl filters running ready to go all
> of the time. Not only is the performance very good, but since you are
> washing everything though perl there really isn't anything you can't do.
> One of my mail systems takes winmail.dat files strips out the
> attachments and reattaches them to the message. It's about 20 lines of
> perl and only took a few hours to write. Another point of efficiency
> with cyrus is that it has a daemon to accept the messages to be written
> to the mailstore. This daemon listens on a unix socket and is ready to
> go when sendmail connects and feeds it a message. Having all of these
> things running saves a ton of IO on the system. My mail system doesn't
> touch the disk until it's actually writing the message to disk.
>
>
I don't know much about "milters" but Postfix has that option. I
followed some instructions off the net once. After I was done I had
"milter" support and some settings. While they looked cool it just added
a lot of management overhead. There may be a use for them but I can't
think of any. Again, I'm dealing with small mail servers.
> 5. Flexibility. What do you do when a spammer starts hitting you
> really hard? Well if you have a mail system that can filter at the smtp
> level you have some options. My mail system stores the spamassassin
> score in a database and on the 5th message in a 24 hour period it looks
> up the score and starts rejecting inbound messages from systems known to
> send spam. This takes place after the mail from: and rcpt to: so users
> that want to whitelist still can, but in the mean time a spammer that
> wants to send me 50000000 messages only gets 5 per day. Any attempts
> maid after the first 5 cost me one database and one ldap lookup which is
> nothing compared to procmail/formail/spamassassin/etc. If your mail
> system doesn't filter during smtp then you pretty much only have 2
> options, live with the abuse, or firewall the spammer and live with the
> false positives.
>
>
I use Postgrey which does greylisting. I don't know how new this
technology is but it's new to me. It blocks new email for a period of
time. The idea here is that stammers only send email once. To get past
greylisting you need to send the message more then once. The email
message continues to be blocked until the time is up. In my clients case
it's 10 minutes. I also have the option of black listing and white listing.

This stops spam at the SMTP level. It also cuts down on false positives
because less spam reaches SpamAssassin and ClamAV.

I think greylisitng is a gift from the Gods. At first I didn't believe
it would would work. After checking the logs and talking with my client
I was hooked.
> As far as ease of use, yea, initial setup is easier, but after you have
> a couple of complex mail problems to solve sendmail/mimedefang is much
> easier in the end. It's kinda like windows, setting it up is trivial,
> but everything else (like backups) is a major pain.
>
> I should also point out that m4 config files are really a good idea.
> How many times have you upgraded a software package to find out the old
> config file doesn't work with the updated version? Your left to go
> figure out why it doesn't like the old config file. The problem is even
> worse in a system like sendmail because much of how it routes mail isn't
> in the binary, it's in the config file. The solution: m4 macros. With
> M4 you state that you want some feature like a RBL with one line of
> config and it automatically includes everything needed to make it work.
> If a new version of sendmail comes out, you just remake your config
> file from the mc macro file and you get the latest config file and
> features. Here is an example to illustrate my point. Suppose you want
> to make sendmail only except messages for email addresses that live in
> your Active Directory LDAP server. You write the config to make
> sendmail lookup each address against LDAP, then put it in your mc file,
> then on every future version of sendmail you simply `m4 sendmail.mc >
> sendmail.cf` and your done. This is *WAY* simpler than the qmail way of
> doing things which is to patch the source (because there is no way to
> add this stuff to the config) then make sure your patch works with all
> of the other patches you run. I would much rather run the m4 command
> and have a very flexible config file than keep up with a zillion patches
> to make the binary work the way you need it to.
>
> Hopefully this wasn't such a long boring read (hopefully it even made
> sense), but I did want to point out that procmail/mbox is really not the
> best way to go when dealing with any significant amount of email load.
>
> later,
> schu
>
You can backup the Postfix config directory. If something goes wrong
just put it back and restart postfix. I do a backup of /etc any way.
That way if I mess something up I can put it back.

Sendmail has a lot going for it. However, Postfix has the same options
including "milters", it's secure and management can be as simple or
complex as you need.
---------
To unsubscribe, send email to <aklug-request@aklug.org>
with 'unsubscribe' in the message body.
Received on Thu Aug 31 17:11:22 2006