milter-ahead/1.13
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| text!/path/map.txt | R/O text file, memory hash |
| /path/map.db | Berkeley DB hash format |
| db!/path/map.db | Berkeley DB hash format |
| db!btree!/path/map.db | Berkeley DB btree format |
| sql!/path/database | An SQLite3 database |
| socketmap!host:port | Sendmail style socket-map |
| socketmap!/path/local/socket | Sendmail style socket-map |
| socketmap!123.45.67.89:port | Sendmail style socket-map |
| socketmap![2001:0DB8::1234]:port | Sendmail style socket-map |
If :port is omitted, the default is 7953.
The access-db contains key-value pairs. Lookups are performed from most to least specific, stopping on the first entry found. Keys are case-insensitive.
There are essentially three types of keys used in the access-db. Many of the tags available will use one or more of these lookup sequences.
- IP Address Lookups
-
An IP address lookup is typically applied to the connected SMTP client.
It will start with a complete IPv4 or IPv6 address and break it down
on delimiter boundaries from right to left.
IPv6 Lookup IPv4 Lookup tag:2001:0DB8:0:0:0:0:1234:5678 tag:192.0.2.9 tag:2001:0DB8:0:0:0:0:1234 tag:192.0.2 tag:2001:0DB8:0:0:0:0 tag:192.0 tag:2001:0DB8:0:0:0 tag:192 tag:2001:0DB8:0:0 tag:2001:0DB8:0 tag:2001:0DB8 tag:2001 Note that the compact form of an IPv6 address, "
2001:0DB8::1234:5678", cannot be used. Only the full IPv6 address format, with all intervening zeros, is currently supported. - Domain Name Lookups
-
A domain lookup may be applied to either the connected SMTP client,
where the client's host name found through a DNS PTR record is searched for, or
using the domain portion of an mail address (see below). A domain lookup will try the IP-domain literal
if applicable, then continue with the FQDN, breaking it down one label
at a time from left to right.
tag:[ipv6:2001:0DB8::1234:5678] tag:[192.0.2.9] tag:sub.domain.tld tag:domain.tld tag:tld tag: Note that the bare tag is often used to specify system wide defaults.
- Mail Address Lookups
-
A mail address lookup is similar to a domain lookup, but the search
first starts with a complete mail address, before trying the address's
domain, and finally only the local part of the address.
tag:account@sub.domain.tld tag:sub.domain.tld tag:domain.tld tag:tld tag:account@ tag: Note that the bare tag is often used to specify system wide defaults.
Tags
The following list outlines the available tags and their supported key lookups:
- milter-ahead-Auth:auth_authen
- milter-ahead-Auth:
-
Used to black or white list by authenticated user (SMTP AUTH) as given
by the macro
{auth_authen}. The string searched by the pattern list will be the sender's mail address. The empty form of milter-ahead-Auth: allows for a milter specific default only when{auth_authen}is defined. - milter-ahead-Connect:ip:From:mail
- milter-ahead-Connect:domain:From:mail
- This set of combination tags are used to black or white list sender addresses when sent from a given SMTP client. The sender address can be easily forged and using the From: tag by itself could allow spam with a forged address. By adding the sender's SMTP client as an extra constraint, it is possible to limit such abuse. Note that the lookup variants with blank IP, domain, or mail are not supported.
- milter-ahead-Connect:ip
- milter-ahead-Connect:domain
- Connect:ip
- Connect:domain
- Used to black or white list an SMTP client. Note that Sendmail also has special semantics for Connect:.
- milter-ahead-Connect:ip:To:mail
- milter-ahead-Connect:domain:To:mail
- This set of combination tags are used to black or white list recipient addresses that a given SMTP client may contact. This allows for finer granularity of control in place of the To: tag. Note that the lookup variants with blank IP, domain, or mail are not supported.
- milter-ahead-From:mail:To:mail
- This set of combination tags are used to black or white list a pair of sender and recipient addresses. This allows for finer granularity of control in place of the To: tag. Note that the lookup variants with blank mail elements are not supported.
- milter-ahead-From:mail
- From:mail
- Used to black or white list a sender's mail address. Black listing using this tag is fine, but white listing is not recommended as it is too easy for someone to fake the sender address or domain. In the case of a +detailed mail address, the left hand side of the +detail is used for the sender@ lookup. Note that Sendmail also has special semantics for From:.
- milter-ahead-To:mail
- Spam:mail
- To:mail
- Used to black or white list a recipient's mail address. In the case of a +detailed mail address, the left hand side of the +detail is used for the sender@ lookup. Note that Sendmail also has special semantics for To: and Spam:.
Action
When a key lookup matches, then the value returned is a pattern list, which in
its simplest and most common form is either an action word like
OK, REJECT, etc.;
a numerical value; or some arbitrary string depending on the tag involved. For example:
Connect:192.168.0 OK Rate-Connect:fsl.com 17 Msg-Limit-From:hotmail.com 150/30m
The action words supported by milter-ahead are:
OK White list, by-pass one or more tests. REJECT Black list, reject connection, sender, recipient, etc. SKIP Stop lookup and return no result ie. continue testing. DUNNO Same as SKIP, commonly used by postfix.
Pattern Lists
In most instances, the above forms of key lookup and values are sufficient. However, there may be times when finer granularity of control is required, in which case pattern lists can be used. A pattern list is a white space separated list of pattern-action pairs followed by an optional default action. The supported types are:
[network/cidr]action Classless Inter-Domain Routing
(only with IP address lookups)!pattern!action Simple fast text matching. /regex/action POSIX Extended Regular Expressions.
The simple pattern matching, !pattern!, uses an asterisk (*) for a wildcard, scanning over zero or more characters; a question-mark (?) matches any single character; a backslash followed by any character treats that character as a literal. This method always tries to match the beginning and end of string. For example:
!abc! exact match for 'abc' !abc*! match 'abc' at start of string !*abc! match 'abc' at the end of string !abc*def! match 'abc' at the start and match 'def' at the end, maybe with stuff in between. !*abc*def*! find 'abc', then find 'def'
The following is an example using a simple pattern to reject client connections that originate from a range of IP addresses of an ISP assigned to ADSL customers. Using a pattern like this allows you to drop connects from the ISP's ADSL, while still accepting connections from mail and web servers.
Connect:hananet.net !adsl-*-*.usr.hananet.net!REJECT
If you know that an ISP's mail and web servers follow a standard naming convention, you might prefer to only accept mail from those instead. We include web servers here to handle the case where a web server might have to send a mail response based on a form being filled in.
Connect:hananet.net !smtp*.hananet.net!OK !www*.hananet.net!OK REJECT
Note that SPF was designed to help mail servers identify originators of mail, so creating patterns as shown in the above two examples is not normally required. However, SPF is still considered experimental and not as widely deployed as one might hope.
The next example, /regex/, uses Extended Regular Expressions to validate the
format of the local-part of an AOL address, which must be between 3 and 16
characters long, can contain dots and RFC 2822 ``atext'' characters except % and /.
The SKIP word allows the one regular expression to validate the format of the address and continue if the
pattern matches; otherwise if the regular expression failed to match, REJECT the suspect aol.com address.
From:grandma@aol.com OK
From:aol.com /^[a-zA-Z0-9!#$&'*+=?^_`{|}~.-]{3,16}@aol.com$/SKIP REJECT
The discussion of Extended Regular Expressions is vast and complex, well beyond the scope of this document. There are many on-line tutorials and references available and the book Mastering Regular Expressions, 3e from O'Reilly covers the topic in depth.
If you need to use a pattern list, then try and follow these suggestions:
- A pattern cannot be used as the key in an access-db lookup. Key-value tables work with constants for the keys using a predefined lookup order as outlined above.
- Use the key lookup as a selector to find a pattern list.
- Use simple !pattern! matching where possible, as it will be faster than Extended Regular Expressions, /regex/.
- Avoid using pattern lists with bare tag variants that specify a global default. It will more often than not cause a lot of unnecessary attempts to match a pattern.
- Keep your pattern lists short & simple.
The milter-ahead-Connect:, milter-ahead-From:, and milter-ahead-To: tags provide a milter specific means to override the Sendmail and Postfix variants. For example, you normally white list your local network through any and all milters, but on the odd occasion you might want to actually scan mail from inside going out, without removing the Connect: tag that allows Sendmail or Postfix to relay for your network or white listing for other milters. So for example if you have tags like:
To:mx.example.com RELAY
You might have to add milter specific overrides in order to make sure the mail still gets filtered:
To:mx.example.com RELAY milter-ahead-To:mx.example.com SKIP
Some additional examples:
milter-ahead-Connect:80.94 [80.94.96.0/20]OK REJECT Accept connections from the netblock 80.94.96.0/20 (80.94.96.0 through to 80.94.111.255) and rejecting anything else in 80.94.0.0/16. milter-ahead-Connect:192.0.2 /^192\.0\.2\.8[0-9]/OK REJECT Accept connections from 192.0.2.80 through to 192.0.2.89, reject everything else in 192.0.2.0/24. milter-ahead-From:example.com /^john@.+/OK /^fred\+.*@.*/OK REJECT Accept mail from <john@example.com> and <fred@example.com> when fred's address contains a plus-detail in the address. Reject everything else from example.com. milter-ahead-To:example.net !*+*@*!REJECT !*.smith@*!REJECT /^[0-9].*/REJECT Reject mail to example.net using a plus-detail address or to any user who's last name is "smith" or addresses starting with a digit. No default given, so it is assumed to be SKIP..
bdb,
flatfile, hash.
text!/path/map.txt R/O text file, memory hash /path/map.db Berkeley DB hash format db!/path/map.db Berkeley DB hash format db!btree!/path/map.db Berkeley DB btree format sql!/path/database An SQLite3 database socketmap!host:port Sendmail style socket-map socketmap!/path/local/socket Sendmail style socket-map socketmap!123.45.67.89:port Sendmail style socket-map socketmap![2001:0DB8::1234]:port Sendmail style socket-map
The recipient's domain is first used as the lookup key followed by each parent domain component that makes up the domain name. If no key is found, then the value of {rcpt_host} will be used.
When a key is found, then the value returned must be the name or IP
of the mail server to consult surrounded by square brackets. An optional :port
specifier may follow the closing square bracket. To consult the MX of another
domain specify the other domain name to lookup. This only works when
mx-lookup is enabled. Note that :port
cannot be used in this case, because using an MX implies SMTP port 25.
Below is an example of what the database file might look like with using IPv4, IPv6, hostname, and MX of domain values:
snert.biz [192.0.2.7] snert.com [192.0.2.7]:1234 snert.eu [2001:0DB8::beef] snert.fr [2001:0DB8::beef]:1234 snert.info [pop.snert.example] mx.snert.net [smtp.snert.example]:1234 snert.net snert.org
all all reserved IP address 0 accept all. benchmark 198.18.0.0/15 RFC 2544 link-local 169.254.0.0/16, FE80::/10 RFC 3330, 3513 localhost 127.0.0.1, ::1 RFC 3330, 3513 loopback 127.0.0.0/8 excluding 127.0.0.1 multicast 224.0.0.0/4, FF00::/8 RFC 3330, 3513 private-a 10.0.0.0/8 RFC 3330 private-b 172.16.0.0/12 RFC 3330 private-c 192.168.0.0/16 RFC 3330 reserved IPv6 unassigned prefixes RFC 3513 site-local FEC0::/10 RFC 3513 test-net 192.0.2.0/24, 2001:DB8::/32 RFC 3513, 3849 this-net 0.0.0.0/8, ::0 RFC 3330, 3513
milter-ahead.
Typically a unix named socket or a host:port. This value must match the value specified for
the INPUT_MAIL_FILTER() macro in the sendmail.mc file. The accepted syntax is:
{unix|local}:/path/to/file- A named pipe. (default)
inet:port@{hostname|ip-address}- An IPV4 socket.
inet6:port@{hostname|ip-address}- An IPV6 socket.
Spammers often attempt to by-pass spam filters by sending email directly to secondary MX machines, which often have weaker requirements. This option essentially demands that a client only deliver to the primary MX when it is available.
kill -QUIT `cat /var/run/milter/milter-ahead.pid`
§ all All messages § 0 Log nothing. § info General info messages. (default) § trace Trace progress through the milter. § parse Details from parsing addresses or special strings. debug Lots of debug messages. § dialog I/O from Communications dialog state State transitions of message body scanner. § dns Trace & debug of DNS operations § cache Cache get/put/gc operations. § database Sendmail database lookups. § socket-fd Socket open & close calls § socket-all All socket operations & I/O § libmilter libmilter engine diagnostics
SMTP Responses
This is the list of possible SMTP responses generated by milter-ahead.
- 553 5.1.0 imbalanced angle brackets in path
- The path given for a MAIL or RCPT command is missing a closing angle bracket
- 553 5.1.0 address does not conform to RFC 2821 syntax
- The address is missing the angle brackets, < and >, as required by the RFC grammar.
- 553 5.1.0 local-part too long
- The stuff before the @ is too long.
- 553 5.1.[37] invalid local part
- The stuff before the @ sign contains unacceptable characters.
- 553 5.1.0 domain name too long
- The stuff after the @ is too long.
- 553 5.1.7 address incomplete
- Expecting a domain.tld after the @ sign and found none.
- 553 5.1.[37] invalid domain name
- The domain after the @ sign contains unacceptable characters.
- 553 5.1.0 cannot deliver to null address
- Sender said RCPT TO:<> and Sendmail didn't catch it.
- 550 5.7.1 routed address relaying denied
- The recipient address tried to use the %-hack for routed address relaying. See reject-percent-relay option.
- xyz x.7.1 server [^ ]+ for <[^>]+> communication error
- An internal error occured. See the mail log for details.
- xyz x.7.1 server [^ ]+ for <[^>]+> not answering
- The server given by the {rcpt_host} macro or call-ahead-host option could not be reached.
- xyz x.7.1 server [^ ]+ for <[^>]+> responded with a busy signal
- The server given by the {rcpt_host} macro or call-ahead-host option return a 421 busy signal.
- xyz x.7.1 server [^ ]+ for <[^>]+> provides no SMTP service
- The server given by the {rcpt_host} macro or call-ahead-host option return a 554 no service.
- xyz x.7.1 server [^ ]+ for <[^>]+> responded with \".*\"
- The server given by the {rcpt_host} macro or call-ahead-host option return an unexpected welcome response.
- xyz x.7.1 server [^ ]+ for <[^>]+> did not accept HELO
- Received something other than a 250 return code for the HELO command.
- xyz x.7.1 server [^ ]+ for <[^>]+> does not accept <> address as required by RFC 821, 1123, 2505, and 2821
- The MAIL FROM:<> command was reject contrary to RFC requirements. See the relay-mail-from option.
- xyz x.7.1 server [^ ]+ for <[^>]+> rejected sender address saying \".*\"
- The MAIL FROM:<sender> was rejected. See the relay-mail-from option.
- xyz x.7.1 server [^ ]+ for <[^>]+> rejected address saying \".*\"
- The RCPT TO:<recipient> command was rejected.
- 450 4.7.1 primary MX [^ ]+ online and accepting mail
- See the primary-up-reject option.
Installation
Download:
milter-ahead/1.13 md5sum Change Log LibSnert md5sum Change Log Sendmail 8.14 http://www.sendmail.org/ Berkeley DB http://www.sleepycat.com/ -
If you have never built a milter for Sendmail, then please make sure that you build and install
libmilter, which is not built by default when you build Sendmail. Please read thelibmilterdocumentation. Briefly, it should be something like this:cd (path to)/sendmail-8.14.0/libmilter sh Build -c install
-
The build process for
libsnertandmilter-aheadis pretty straight forward once you havelibmilterinstalled:cd (path to)/com/snert/src/lib ./configure make build cd ../milter-ahead ./configure make build make install
Both configuration scripts have some options that allow you to override defaults. Those options are listed with:
./configure --help
-
An example
/usr/local/share/examples/milter-ahead/milter-ahead.mcis supplied. This file should be reviewed and the necessary elements inserted into your Sendmail.mcfile andsendmail.cfrebuilt. Please note the comments on the general milter flags. -
Once installed and configured, start
milter-aheadand then restart Sendmail. An example startup script is provided in/usr/local/share/examples/milter-ahead/milter-ahead.sh. The default options can be altered by specifying them on the command-line or within a/etc/mail/milter-ahead.cf. Themilter-ahead.cfis parsed first followed by the command-line options.
Notes
Currently tested platforms:
Cobalt Qube 1 with Linux RH 5.1 (mips 2.0.34 kernel);Linux RH 5.1 (Intel x386 2.2.25 kernel);FreeBSD 4.8, 4.9 (Intel x386);OpenBSD 3.6 (Intel x386)-
The minimum desired file ownership and permissions are as follows for a typical Linux system. For FreeBSD, NetBSD, and OpenBSD the binary and cache locations may differ, but have the same permissions.
Process user ``milter'' is primary member of group ``milter'' and secondary member of group ``smmsp''. Note that the milter should be started as root, so that it can create a .pid file and .socket file in /var/run; after which it will switch process ownership to milter:milter before starting the accept socket thread.
/etc/mail/ root:smmsp 0750 drwxr-x--- /etc/mail/access.db root:smmsp 0640 -rw-r----- /etc/mail/sendmail.cf root:smmsp 0640 -rw-r----- /etc/mail/milter-ahead.cf root:root 0644 -rw-r--r-- /var/run/milter/milter-ahead.pid milter:milter 0644 -rw-r--r-- /var/run/milter/milter-ahead.socket milter:milter 0644 srw-r--r-- /var/db/milter-ahead milter:milter 0644 -rw-r--r-- (*BSD) /var/cache/milter-ahead milter:milter 0644 -rw-r--r-- (linux) /usr/local/libexec/milter-ahead root:milter 0550 -r-xr-x--- We would like to express our thanks to Derek Balling for his support at http://www.milter.org/ and to April Lorenzen for her poking and proding to create this milter.
User Feeback
- April Lorenzen
- We have a sendmail gateway in front of an IMail pop/webmail server. milter-ahead makes it effortless for the gateway to know both users and aliases on the pop server, without messy alias/user gathering scripts and cron jobs. milter-ahead allows us to reject mail to unknown users immediately, before accepting the message content. We are using it to reject thousands of dictionary attack spammers, viruses, and addresses that got on "millions of addresses" spammer cds from customers who used to have catch all accounts.
License Agreement 1.8
SNERTSOFT & CO. ARE WILLING TO LICENSE THE SOFTWARE IDENTIFIED ABOVE TO YOU ONLY UPON THE CONDITION THAT YOU ACCEPT ALL OF THE TERMS CONTAINED IN THIS LICENSE AGREEMENT. PLEASE READ THE AGREEMENT CAREFULLY. BY DOWNLOADING OR INSTALLING THIS SOFTWARE, YOU ACCEPT THE TERMS OF THE AGREEMENT.
Definitions
``Package'' means the identified above in source and/or binary form, any other machine readable materials provided (including, but not limited to documentation, sample files, data files), any updates or error corrections, and its derivative works.
``Organisation'' means a legal entity or an individual.
``You'' (or ``Your'') means an Organisation exercising rights under, and complying with all of the terms of, this License or a future version of this License issued under Section 6.1. For legal entities, ``You'' includes any entity which controls, is controlled by, or is under common control with You. For purposes of this definition,``control'' means (a) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (b) ownership of more than fifty percent (50%) of the outstanding shares or beneficial ownership of such entity.
``SnertSoft'' means Anthony C. Howe situated in Cannes, France (SIRET #489 259 937 00014).
``SnertSoft & Co.'' means SnertSoft and all authorised & licensed partners, such as value-added resellers or appliance manufacturers.
Statement of Original Work
The Package is an original work written by SnertSoft, with exception of following third party code:
MD5 routines written by L. Peter Deutsch, based on RFC 1321;
SQLite3 package is written by D. Richard Hipp of Hwaci and is in the public domain.
strnatcmp.c is written by Martin Pool of sourcefrog.net and has a BSD style license.
Protothreads headers originally written by Adam Dunkels and has a BSD style license.
License To Use
You may install and use this Package, without modifications, exclusively on machines for which You have purchased a license, provided You retain this notice, SnertSoft's copyright notice, any and all license control methods (see below), and any links within the Package back to the most current online versions of this License and Disclaimer.
You may copy, share, distribute, modify, and create derivative works from the user manuals and any related documentation solely for Your internal business purposes, such as in-house documentation, training manuals, or reference material.
Restrictions
Redistribution, including but not limited to books, CDROMS, download mirrors, floppy diskettes, hard disks, hardcopy print outs, online archives, solid state disks, streaming tapes, or other current or future forms of storage or communication media of the Package, with or without modifications, including any and all derivative works such as source patches, binaries, binary patches, or similar is expressly forbidden without prior written permission in hardcopy (ie. letter or fax) signed and dated by SnertSoft.
It is expressly forbidden for You to use the Package, in whole or in part, in any other software or appliance without prior written permission in hardcopy (ie. letter or fax) signed and dated by SnertSoft.
It is expressly forbidden for You to use the Package to develop any software or other technology having the same primary function as the Package, including but not limited to using the Package in any development or test procedure that seeks to develop like software or other technology, or determine if such software or other technology performs in a similar manner as the Package.
You may not sell, rent, lease, or transfer the Package to third parties without prior written permission in hardcopy (ie. letter or fax) signed and dated by SnertSoft.
Termination
This Agreement is effective until terminated. You may terminate this Agreement at any time by destroying all copies of the Package.
This Agreement will terminate immediately without notice from SnertSoft if You fail to comply with any provision of this Agreement.
Either party may terminate this Agreement immediately should any portion of the Package become, or in either party's opinion be likely to become, the subject of a claim of infringement of any intellectual property right. Upon Termination, You must destroy all copies of the Package.
Versions Of The License
New Versions. SnertSoft may publish revised and/or new versions of the License from time to time. Each version will be given a distinguishing version number.
Effect of New Versions. Once a version of the Package has been published under a particular version of the License, You may always continue to use it under the terms of that License version. You may also choose to use such Package under the terms of the most current version of the License published by SnertSoft.
No one other than SnertSoft has the right to modify the terms applicable to the Package created under this License.
Disclaimer
THIS SOFTWARE IS PROVIDED "AS IS" AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO WAY SHALL SNERTSOFT OR LICENSEE BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
License Control
The Package may use one or more license control methods including, but not limited to, license key activation, periodic reporting of Package details and IP address of installation to SnertSoft & Co., or remote license verification by SnertSoft & Co.. Any information reported to or gathered by SnertSoft & Co. shall remain strictly confidential and the private property of SnertSoft & Co.. Under no circumstances will SnertSoft & Co. resell or release this information to third parties, unless demanded by court order.
Support
Support is provided for one year from date of purchase and only for SnertSoft's original Package that was purchased directly from SnertSoft. Additional support beyond the first year can be obtained from SnertSoft on time & materials basis or from one of SnertSoft's authorised partners.
Support for the Package obtained from authorised partners, such as value-added resllers or appliance manufacturers, will be supplied by those partners. SnertSoft will not support the Package without proof of purchase from SnertSoft, such as an Order N° or Invoice N°.
Package enhancements requests and product suggestions are always welcome. A community mailing list is available; please refer to SnertSoft web site Support area for details.
Gifts
Gifts from the author's Amazon US or Amazon UK wishlist (search by mail address <achowe at snert dot com>) are welcomed for the continued encouragement, moral support, and ego pumping needed to work in foreign non-english speaking lands.
pretzels since 24 January 2004