Mailcorral Documentation

0. Preface

Eric Wilde, ewilde@bsmdevelopment.com

V2.2.0, 2020 Mar 20

0.1 Copyright

Copyright (c) 2002, 2003, 2004, 2009, 2012, 2013, 2020 Eric Wilde

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts. A copy of the license is included in Appendix A, entitled "GNU Free Documentation License".

0.2 Distribution

This document is also available in PDF format at:

MailCorral Documentation

0.3 Contributions

BSM Development is a small company of software developers (perhaps, like yourself) and we would appreciate, if you so choose, a small contribution to pay for this software. We hope you perceive it to be of value and find it useful to you. As a suggestion, we would like to think that it is worth at least US $100.00. However, you may send whatever amount that you feel this product is worth. If you'd like to make a contribution, you can do so directly via the contributions page of the BSM Development site (www.bsmdevelopment.com/Products/Contribute.html).

Alternately, you can send a check via snail mail to:

BSM Development
44 Whitewood Circle
Norwood, MA 02062

Please make your check payable to "BSM Development", thanks.

0.4 Change Log

Here are the version numbers, dates and a brief description of each program change, in reverse chronological order:

2.2.0

• 2020 Mar 19 - Add options to distrust users (with respect to certain local options) and reject entities with warnings.
• 2020 Mar 18 - Do a better job of classifying inline MIME types and produce more informative errors and warnings.
• 2020 Mar 17 - Add additional well-known file and MIME types.

2.1.2

• 2013 Jul 21 - Fix possible "sed: command not found" error in configure script on some systems.
• 2013 Jan 27 - Add the ability to process the SpamAssassin updates tree, to find versioned whitelists.
• 2013 Jan 25 - Fix compile warnings for the free function in configuration processing. Fix reentrancy problem with user configuration files.

2.1.1

• 2012 Jun 8 - Fix compile warnings for string/stdio functions in expression evaluator and other modules. Add image/png MIME type and .png extension to the list of acceptable mail attachements. Fix bogus definition of REMAILHDRBYPASS. Fix longstanding problem with mail from local addresses being erroneously filtered when the CheckExternal flag is not set.

2.1.0

• 2009 Nov 21 - Add .wmv extension to the file types list as an OK file.
• 2009 Nov 15 - Fix problem in with WeakRequire when module not found in MailRelease and SpamNotify.

2.0.1

• 2009 Aug 7 - Fix quote handling by NormalizeName.
• 2009 Aug 3 - Fix problem writing empty message body to spam file when spam messages also contain viruses and the virus scanner deletes the entire message.

2.0.0

• 2009 Feb 4 - Add encoding and decoding of uuencoded entities.
• 2009 Jan 19 - Add support for ClamAV via clamd.
• 2009 Jan 16 - Add support for the spamd 1.4 protocol and the HEADERS command.
• 2009 Jan 8 - Add archiving of messages to disk or to a third party archiver via email forwarding. Add bounceback suppression for bounced messages from non-functioning third party archivers. Use header from name for blacklist and whitelist checks, along with envelope from name. Have name normalization strip "<>" and "()" from addresses.
• 2004 Apr 19 - Add support for evaluating results from user supplied arbitrons, plus internal arbitrons, via logical expressions.
• 2004 Jan 15 - Allow users to programatically supply arbitrons.

1.3.0

• 2007 Aug 13 - Fix problem with multiply defined bool under sendmail 8.14.1.

1.2.0

• 2003 Aug 21 - Fix problem with HTML tag parameters that are a single quote crashing filter (filter goes defunct).
• 2003 Jun 13 - Try "htdocs" as well as "html" for the template file location in ConfigEdit.cgi.
• 2003 Jun 8 - Add exit message if errors found in command line or options.
• 2003 May 29 - Treat "text" as "text/plain". Fix crash when internal spam arbitron is used.
• 2003 May 15 - Allow all_spam_to for non-local users.
• 2003 May 12 - Fix problem with messages containing empty text entities.
• 2003 May 5 - Remove HTML tags from names to release in MailRelease.
• 2003 May 2 - Tweak MailRelease error responses and add a couple more messages. Fix apparent Perl bug in MailRelease whereby push of lc($1) pushes garbage. This affected domain lookup.

1.1.3

• 2003 May 1 - Conditionally load Zlib and DB_File, for real this time, in MailRelease and SpamNotify.
• 2003 Apr 26 - Have SpamNotify unzip corralled messages, if necessary (just in case someone zips all the spam in the corral, before we have a chance to send out the notifications). Fix problem with '>' and '<' in the HTML component of MailRelease notifications.

1.1.2

• 2003 Apr 24 - Add an option to keep all messages in the corral, regardless of whether they were altered or not. Add the ability to zip older corralled messages, if desired.
• 2003 Apr 17 - Include the latest SpamNotify in the distro.
• 2003 Apr 14 - Add global ignore options.
• 2003 Apr 13 - Allow source RPM to build on any i386 version of RedHat. Template redhat.mc is missing on RH 8.x. The RPM now picks something else.
• 2003 Apr 11 - Update filterclean to get rid of empty partition directories.
• 2003 Apr 10 - Fix compiler windge about ctime pointer.

1.1.1

• 2003 Apr 8 - Don't reencode second HTML entity if not changed. Allow buffer space for badly formatted Quoted-printable entities.
• 2003 Apr 3 - Use tag delimiters to end mismatched quotes.
• 2003 Apr 2 - Handle mangling of short MIME types properly. Ignore "Content-type: text" in headers.
• 2003 Mar 31 - Fix problem with addresses containing domain names before the '@'.
• 2003 Mar 27 - Check messages from everybody, if paranoid.

1.1.0

• 2003 Mar 22 - Fixed a problem in HTML tag scanning when unmatched quotes are encountered.
• 2003 Mar 19 - Rebuild messages with attachments only to allow virus and spam warning text to be inserted, permit deletion of the attachment, etc.
• 2003 Mar 17 - Handle Apple resource forks better.
• 2003 Mar 16 - Allow attachments containing viruses to be deleted.
• 2003 Mar 12 - Partition spam as well.
• 2003 Mar 11 - Properly handle MIME type messages that have empty bodies.
• 2003 Mar 10 - Fix improper use of HAVE_DBM ifdef.

1.0.18

• 2003 Mar 9 - Implement ommitted cases in non-delivery test. Test for Base64 encoded text/HTML MIME entities in internal, statistical arbitron. Add text/plain and text/html as special cases for "UnknownDisposition MIME". Remove junk after plussed user names.
• 2003 Mar 8 - Allow user to specify timeout for arbitron connection.
• 2003 Mar 7 - Use VerifyUser instead of getpwnam to look up home directories, thereby avoiding a crash due to a threads bug.

1.0.17

• 2003 Mar 4 - Lowercase external addresses used to create corral files.
• 2003 Mar 3 - Fix up the "mailto" link in the remail message so that it is really a link that works.
• 2003 Mar 1 - Allow IP addresses to be supplied in the local domain list. Accept full reports from SpamAssassin versions greater than or equal to 2.50. Make user name check optional in MailRelease. Allow MailRelease to release corralled mail without path and partition name.
• 2003 Feb 28 - Add options to include SMTP AUTH and other special classes of users in the local domain (for purposes of internal -> internal/external determination). Remove the background parameter from the HTML body tag.
• 2003 Feb 27 - Add options to filter mail transiting the system. Change the handling of unknown file types so they are disposed of according to the user's preference.
• 2003 Feb 26 - Allow statistical filter to be tuned through config file options.

1.0.16

• 2003 Feb 19 - Remove certain HTML escape sequences.
• 2003 Feb 17 - Fix base 64 re-encoding problem for templates with no line feeds (was causing occasional crash on badly-formed encoded messages). Fix problem with HTML tag quoting algorithm.
• 2003 Feb 16 - Allow enough buffer space for badly formatted Base 64 encoded messages to be modified.
• 2003 Feb 13 - Create a man page for sendmailfilter.
• 2003 Feb 12 - Split message and deliver altered/unaltered copies to those who request virus filtering and those who don't.
• 2003 Feb 9 - Allow filtered viruses to be partitioned in the corral. Delete recipients from the envelope, if they request non-delivery of spam.
• 2003 Feb 7 - Cache /etc/passwd entries to avoid crash due to threads bug.
• 2003 Feb 6 - Add statistical spam fast path checks.
• 2003 Feb 4 - Remove HTML comments embedded in the middle of words.
• 2003 Jan 30 - Use proxy userid for DBM lookup if no options found under regular lookup.
• 2003 Jan 20 - Include sample sendmail.cf file in distro.
• 2003 Jan 19 - Check for extra options on command line. Make spam arbitron error reportage more comprehensive.
• 2003 Jan 16 - Save PID and initial command string in PID file at startup.

1.0.15

• 2003 Jan 12 - Allow virus/spam tags to be defined in smfopts.h. Add options for ignoring spam and/or virus filtering.
• 2003 Jan 11 - Don't corral spam for nonexistant users.
• 2003 Jan 10 - Use the virtual user table to determine local users. Only process spam if the virtual users are local users.
• 2003 Jan 7 - Convert the spam arbitron IP address ahead of time to avoid a crash in gethostbyname on BSD.
• 2002 Dec 24 - Bail out if invalid options are detected on the command line or in the global config file.

1.0.14

• 2002 Dec 15 - Fix crash introduced by language support. Fixed loop caused by messages with bogus initial MIME separators.
• 2002 Dec 13 - Disable possible exploit through the "name" parameter of the "Content-Type" header. Fixed bug that caused a memory overrun when mangling certain file names.

1.0.13

• 2002 Dec 7 - Use flock simulator on systems with no flock.
• 2002 Dec 6 - Fixed problem with overly-greedy regular expression not updating repeating fields greater than nine in ConfigEdit.cgi.
• 2002 Nov 26 - Sum user options if requested to do so.
• 2002 Nov 24 - Accept the ".csv" attachment type.
• 2002 Nov 23 - Use the "X-Accept-Language" tag to set the language used for message inserts.
• 2002 Nov 22 - Handle blanks, etc. in to/from names.
• 2002 Nov 18 - Find attachments in malformed MIME entities.
• 2002 Nov 14 - Improved message insertion for encoded mail.
• 2002 Nov 13 - Added support for HTML tag filtering by parameter. Allow innocuous parameters for the <body> and <meta> tags.
• 2002 Nov 12 - Changes to support Solaris properly during configuration. Fixed problem with percent signs in addresses causing syslog to crash the filter.
• 2002 Nov 8 - Fixed possible exploit through attachments defined directly in message headers.
• 2002 Nov 6 - Allow the <html> tag to have parameters (for Outlook's sake).
• 2002 Nov 5 - Strip pesky <Ctrl-M> from user names in MailRelease. Handle long file names wrapped by mail readers. Clear BASH_ENV in case running suidperl.
• 2002 Oct 21 - Fixed problem with comments being replaced in poorly formed messages.

1.0.12

• 2002 Oct 18 - Fix local AutoRemail option having no effect.
• 2002 Oct 18 - Include all SpamAssassin response headers in reports, not just X-Spam-Status and X-Spam-Checker-Version.
• 2002 Oct 16 - Add missing move of X-Tag options to local.
• 2002 Oct 13 - Use fully qualified names for config database lookup if "-q" or "QualifyNames" is used. Add the Language, LanguageDirectory and "-L" options.

1.0.11

• 2002 Oct 9 - Accept ISO encodings in MIME file name parameters.
• 2002 Oct 7 - Added generic config file editor, supporting flat files and DBM files for storing user parameters.
• 2002 Oct 5 - Allow config lookup via DBM database.

1.0.10

• 2002 Sep 27 - Remove leading space from domain names, since people appear not to be able to read the documentation. Fix problem with fputs returning different return code under BSD. Updated list of HTML tags to allow most innocuous ones. Download is now available in RPM form for RedHat Linux.
• 2002 Sep 26 - Fix errno being reported incorrectly as zero.
• 2002 Sep 24 - Change the corral to /var/spool/MailCorral. Create the corral directory if it doesn't exist.
• 2002 Sep 23 - Use the first local recipient for local options processing.

1.0.9

• 2002 Sep 21 - Force add of "localhost" to domain list, if not already present.
• 2002 Sep 18 - Fix problem with "-r" and "-rm" introduced by the adding of options as keywords.
• 2002 Sep 17 - Removed a possible exploit that relies on HTML tags being split across two pieces of an attachment.
• 2002 Sep 14 - Switch to wholly percentage based spam determination because Spam Assassin returns the wrong flag value when SQL preferences are used.

1.0.8

• 2002 Sep 10 - Added option to turn off spam fast path. Fixed problem with HTML reports, sent to postmaster, not having MIME headers. Fix handling of headers in responses from spamd 1.3 and up.
• 2002 Sep 9 - Allow all local options to be supplied on a per-user basis from a local config file. Allow local options to turn off global options. Add a protocol type for the internal fast path spam arbitron.
• 2002 Sep 8 - Add message options to allow tailoring of text inserted into filtered mail. Allow options to be continued on multiple lines.
• 2002 Sep 7 - Allow all command line options to be specified as keywords in config files. Allow all options to be supplied in a global config file.
• 2002 Sep 6 - Add flockfile/funlockfile around writes to debug file to reduce overhead caused by locking in low-level I/O routines. Fix random crashes when writing to logfile under heavy threading.
• 2002 Sep 4 - Add queue ID to log entries to assist in debugging when multi-tasking.

1.0.7

• 2002 Sep 2 - Added MailRelease to allow filtered messages to be remailed.
• 2002 Sep 1 - Fix crash when processing straight message/rfc822 types (i.e. not MIME). Allow filtered messages to be automatically remailed.

1.0.6

• 2002 Aug 27 - Added the domain names option and domain names file support.
• 2002 Aug 26 - Add support for spamd 1.3 and the INFO command.

1.0.5

• 2002 Aug 22 - Update list of filtered file extensions. Add flag for checking external deliveries too. Add flag for spam reporting always. Add flags for envelope and version info in the headers.
• 2002 Aug 21 - Remove parameters from <body> tag when laundering HTML. All good/bad HTML tags are now configurable. Allow more innocuous HTML tags to pass through unscathed. Only open save and work files if really necessary to process message, instead of on spec.
• 2002 Aug 19 - Added bad file type ".mhtml". Added good file types ".p7c", ".p7m" and ".p7s". Added good MIME types "application/pkcs7-mime" and "application/pkcs7-signature". Fixed problem with named text being marked as an attachment when certain mail readers treat it as part of the message. Fixed a problem with cleanup during abort, crashing if no recipient seen.
• 2002 Aug 17 - Use received header date for spam tag instead of date header (which can be omitted). Format sendmail socket address for debugging.
• 2002 Aug 15 - Fixed problem with unterminated HTML tags not being laundered properly.

1.0.4

• 2002 Aug 14 - Eliminated erroneous spam check when plain old mail messages are remailed. Added fast path spam lookup for white and black lists.
• 2002 Aug 12 - Fix missing "[SPAM]" tag, if message delivery is bypassed for reasons other than spam. Fix problems with malformed MIME entities that have no trailing separator.
• 2002 Aug 10 - Added support for Spam Assassin option files.
• 2002 Aug 9 - Added debug levels.

1.0.3

• 2002 Aug 6 - Added options for spam delivery modes ("-sc", "-sd", "-st") and reporting types ("-s1", "-s2", "-s3"). Added support for spam reporting levels two and three for Spam Assassin. Fixed problem with inserted text when all text/plain and text/html components of a message are encoded base64.
• 2002 Aug 5 - Added <iframe> (the favorite tag of virus professionals everywhere) to the list of HTML tags that are always removed, regardless of the "-h" flag setting. Remove <cr><lf>s from message subjects, per bug in Outlook that allows virus launcers in subjects.
• 2002 Aug 4 - Move spam processing to spamfilter.c. Fixed memory leak when spam checking plain old mail messages. Improved spam checking performance for multipart MIME types.

1.0.2

• 2002 Aug 3 - Fix problem with corralled spam only being held for the last recipient in a list of recipients, when multiple recipients passed to SMTP for a single messge. Problem occurred when envelope was addressed to more than one user. Spam would be detected, processed and corralled but only one notification was sent to a single recipient (last one on the envelope), not all of the recipients.
• 2002 Aug 2 - Allow multiple domains in local domain list for "-i" option, etc.
• 2002 Aug 1 - Add "-d" (debugging option) for dynamic debugging.

1.0.1

• 2002 Jul 29 - Add the "-q" (qualify recipient names in the spam corral) and "-u" (supply a proxy userid for filter criteria lookup) options.

1.0

• 2002 Jul 12 - Add spam filtering.
• 2002 Jun 20 - Add decode of MIME components for HTML check.
• 2002 Feb 25 - Initial coding.