Mailcorral Documentation

1. Installation

The steps to installing the MailCorral filter on a Linux or Unix system, running sendmail, are:

  1. Check that your system meets the requirements.
  2. Rebuild sendmail to use milter, if you haven't already done so.
  3. Compile and install the sendmail filter program.
  4. Configure sendmail to invoke milter.
  5. Hack the sendmail startup script to start the filter.
  6. Install the optional message remailer, if you wish to use it (you will need to create a message handler robot and configure sendmail to allow forwarding to work).
  7. Install the optional spam notifier, if you wish to use it.
  8. Set up a cron job to periodically clean up the filter files, etc.
  9. Configure any local options in the system and/or user configuration files.
  10. If you are installing the RPM for your system, you can skip most of these steps and go straight to the RPM install.

1.1 Requirements

Before beginning with the installation of MailCorral, you might want to check that your system meets the requirements listed below:

You should note that MailCorral was developed on a Linux system. If your system is something other than Linux or Unix, you might want to investigate whether the sendmail/milter configuration is similar and whether standard Unix-type software will run on your machine. The sendmail filter does nothing particularly special so it has a good chance of working on other systems, but you never know.

1.1.1 Install the Optional Prerequisites

If you will be using any of the optional arbitrons or will be sending/receiving encrypted email messages, install those optional components noted below, according to their directions, and make sure that they are working (e.g. that spamd and/or clamd are up and running) before proceeding with the steps outlined herein. Once again, the optional components are:

1.2 Rebuild Sendmail

Before you can proceed with filtering mail messages, you need a version of sendmail that supports milter (version 8.12.0 and above). You can get sendmail from www.sendmail.org.

If you haven't already done so, compile sendmail with the flag "-DMILTER" set. If it was already compiled without it, add the following line to ./sendmail/devtools/Site/site.config.m4:

APPENDDEF(`conf_sendmail_ENVDEF', `-DMILTER')dnl

If you are starting from scratch, make sure that this line appears in that file before you start.

If necessary, delete the entire sendmail object directory named for your operating system (e.g. ./sendmail/obj.Linux.2.2.16-22.i586/sendmail) and rebuild using "sh Build". Don't forget to install the new sendmail into your system by issuing a "sh Build install" from the top sendmail directory.

Switch to the libmilter directory and build "libmilter.a". This is required before the sendmail filter can be built. "sh Build" from that directory should do the trick.

1.3 Compile the Filter

1.3.1 Untar the Distribution

You should untar the distribution files to a separate directory where the filter can be built on its own. By default, the tar ball has all of the distribution files in a subdirectory called "MailCorral-x.x.x". We suggest that you put the filter in a subdirectory off of the main sendmail directory. For example, if you have a main sendmail directory that looks something like this:


  SendMail
  -rw-r--r--  1  421781 Apr  9 18:42 MailCorral-1.1.1.tar.gz
  -rwxr--r--  1  290418 Sep 19  2001 sendmail-8.11.6-1.7.1.i386.rpm
  drwxr-xr-x 22    4096 Sep 19  2001 sendmail-8.12.0
  -rwxr--r--  1 1783911 Sep 14  2001 sendmail.8.12.0.tar.gz

You could cd to this directory and unpack the tar ball with one of the following commands:

tar -xvzf MailCorral-1.1.1.tar.gz
gunzip -c MailCorral-1.1.1.tar.gz | tar -xv

If you do so, your directory structure will now look something like this:


  SendMail
  drwxrwxr-x  2    4096 Apr 19 17:49 MailCorral-1.1.1
  -rw-r--r--  1  421781 Apr  9 18:42 MailCorral-1.1.1.tar.gz
  -rwxr--r--  1  290418 Sep 19  2001 sendmail-8.11.6-1.7.1.i386.rpm
  drwxr-xr-x 22    4096 Sep 19  2001 sendmail-8.12.0
  -rwxr--r--  1 1783911 Sep 14  2001 sendmail.8.12.0.tar.gz

1.3.2 Run "configure"

The distribution is meant to configure itself using a shell script produced by the GNU autoconf program. To configure the makefile which will build the filter, switch to the installed directory and type "./configure". If you're using "csh" on an old version of System V, you might need to type "sh ./configure" instead to prevent "csh" from trying to execute "configure" itself.

The "configure" shell script attempts to guess correct values for various system-dependent variables used during compilation. If you followed the notes above about where to install the filter, running "configure" is all you need do. If you have more than one version of sendmail or you put the filter in a directory not in the sendmail tree, you'll need to supply the name of the sendmail source directory. To do this, type:

./configure --with-sendmail=/home/joeblow/sendmail/sendmail-8.12.3

and replace the sample path name with the actual pathname for the sendmail source tree.

Running "configure" takes awhile. While running, it prints some messages telling which features it is checking for.

The "configure" shell script creates a shell script "config.status" that you can run in the future to recreate the current configuration, and a file "config.log" containing compiler output (useful mainly for debugging "configure"). It can also use an optional file (typically called "config.cache" and enabled with "--cache-file=config.cache" or simply "-C") that saves the results of its tests to speed up reconfiguring.

1.3.3 Specifying the System Type

There may be some features "configure" cannot figure out automatically but needs to determine for the type of machine the sendmail filter will run on. Usually, assuming the filter is being built to run on the same architecture as the build is done on, "configure" can figure that out but if it prints a message saying it cannot guess the machine type, give it the "--build=type" option. type can either be a short name for the system type, such as "sun4", or a canonical name which has the form:

cpu-company-system

where system can have one of these forms:

os or kernel-os

See the file "config.sub" for the possible values of each field.

If you want to use a cross compiler, that generates code for a platform different from the build platform, you should specify the "host" platform (i.e. the platform on which the generated programs will eventually be run) with "--host=type".

1.3.4 Sharing Defaults

If you want to set default values for "configure" scripts to share, you can create a site shell script called "config.site" that gives default values for variables like "CC", "cache_file", and "prefix". "configure" looks for "PREFIX/share/config.site" if it exists, then "PREFIX/etc/config.site" if it exists. Or, you can set the "CONFIG_SITE" environment variable to the location of the site script. A warning: although this "configure" script does so, not all "configure" scripts look for a site script.

1.3.5 Defining Variables

Variables not defined in a site shell script can be set in the environment passed to "configure". However, some packages may run "configure" again during the build, and the customized values of these variables may be lost. In order to avoid this problem, you should set them in the "configure" command line, using "VAR=value". For example:

./configure CC=/usr/local2/bin/gcc

will cause the specified gcc to be used as the C compiler (unless it is overridden in the site shell script).

To find out what options "configure" recognizes and how to control it, type "./configure --help" for more details.

1.3.6 Altering the Makefile

If you need to hack the Makefile that is generated by "configure", here is a description of its variables:

SRCDIR The source directory containing the sendmail filter source and all of its related files. The filter will be built in this directory.
SENDMAILDIR The directory where the source for the version of sendmail that is being used is found. Usually, this is relative to the current directory (above it), but an absolute path may also be given.
OS The name of the directory, off of the sendmail directory, where the OS-specific version of sendmail is built (look around, you'll find it). It is this directory that you delete when you need to rebuild sendmail, by the way.

If you are cross-compiling the filter, there may be more than one directory for the different operating systems and hardware platforms that are targeted. Configuration attempts to pick the right one, based on the target.

BINDIR The binary directory where sendmail and the filter are installed into the system.
SMINIT The sendmail initialization script that is used to stop and start sendmail.
CONFIGDIR The directory where the sendmail configuration files are found (e.g. /etc/mail).
MANDIR The directory where the man pages are to be installed (e.g. /usr/share/man).

1.3.7 Altering Fixed Constants and Variables

The sendmail filter uses a number of more or less fixed constants that are set in a header file that is included in the filter modules when they are compiled. All of these constants will probably only need to be changed once, if at all, to reflect your installation preferences. The header file is called "smfopts.h". It contains system specific information, the texts of all error messages generated, lists of all MIME types and attachments filtered, etc.

The first time you run configure to install the sendmail filter, "smfopts.h" is created in your source directory. You are free to hack it to your heart's content. When you install subsequent releases of the sendmail filter into the same source directory, "smfopts.h" will not be overwritten. Rather, a separate file will be created called "smfopts.h.new" that will contain the new header file. If any changes have been made to the header file, you will be issued a warning at compile time and instructed to compare the two files. This can be done with diff, for example:

diff smfopts.h.new smfopts.h

Diff will show you what changes you need to make to the your "smfopts.h" to bring it up to date with the latest version. Be advised, however, that you need not update "smfopts.h" to the latest version. You can safely ignore the warning message, since "smfopts.h" has a default file ("smfopts_def.h") which contains all of the latest constants, etc. Anything missing from "smfopts.h" will be supplied by "smfopts_def.h" so that the compile will still proceed. However, you will probably want to keep you copy of "smfopts.h" up to date to prevent the chance that the mail filter's behavior will not be quite up to snuff.

When you are hacking "smfopts.h", note that we use a setting of five for tabs so the source will look weird unless you set your tabs to five also). Here is a list of the constants and variables that you might want to change:

TECHSUPPORT The name of the person whom users should contact if they need assistance with filtered messages.
REMAILROBOT The username of the message remailer robot. This name will be used in the instructions to recipients of modified messages, telling them how to get the unmodified message mailed to them. Essentially, they send a message to the username given here.
POSTMASTER The name of the person to whom warning messages should be emailed if any viruses or unknown filter types are found. Don't define this, if you don't want these messages.
DOMAIN The name or names of the local domain. Used for detecting whether a message is being mailed locally so that virus filtering may be bypassed or not (see the "-e" and "-i" options). May be a single domain name or a list of domain names, separated by commas. It may also include file names. Any name that begins with a '/', '~' or '.' is assumed to be a file name.

If a file name is given, the file should contain a list of local domain names, one per line. Blank lines and comments beginning with '#' are ignored. Typically, this feature would be used to point to the file "/etc/mail/local-host-names" or the same place where sendmail gets local domain name information from.

The domain name "localhost" is forced into the list at the end, if it isn't specified. This name is always assumed to be a local domain name by sendmail.

The "-D" command line option or the "DomainList" config file parameter may also be used to specify domain information. If either is used, it overrides the value compiled in by this variable.

SUBJTAGVIRUS A tag string that will be added to the front of the subject for any messages that might possibly contain a virus. Used to identify that message for special processing by any mailer receiving it. The default is "[VIRUS] ".
SUBJTAGSPAM A tag string that will be added to the front of the subject for any messages that contain spam. Used to identify that message for special processing by any mailer receiving it. The default is "[SPAM] ".
SPAMKEY A key string used to generate spam bypass tags. Pick a string (up to eight characters). Whoever has this key string may be able to pass spam through your system without checks so don't let it be known what it is.
SPAMHDRBYPASS A mail header used to pass spam bypass tags. You can change this name to make it harder for a spammer, should they crack your key string (above), to pass spam. This header is temporarily added to messages sidelined because of spam, to allow them to be remailed without filtering at a later date.
REMAILHDRBYPASS A mail header used to pass remail bypass tags. You can change this name to make it harder for an outside user, should they crack your key string (above), to bypass message filtering. This header is temporarily added to filtered messages, stored in the corral because they were altered to remove a virus. It allows them to be remailed without filtering at a later date.
SPAMHDRSTATS A mail header used to hold spam statistics for those messages that contain spam. All remailed spam will have this header, which gives the sendmail filter's reasons for tagging the message as spam. The use of third party spam arbitration packages may cause other headers to be added.
SPAMSTATSTPL A template that defines how the spam statistics, in the spam statistics header, are formatted for display.
VIRUSHDRSTATS A mail header used to hold virus statistics for those messages that contain viruses. All remailed messages with viruses will have this header, which gives the sendmail filter's reasons for tagging the message as containing a virus. The use of third party virus arbitration packages may cause other headers to be added.
VIRUSSTATSTPL A template that defines how the virus statistics, in the virus statistics header, are formatted for display.
MSGEVALERR A message that is used to inform the user of any evaluation errors that occur in the logical expressions that they supply for evaluating the results of spam and virus checks.
ARBITRONTIMEOUT The default number of seconds that the filter should wait for any external arbitrons (both third party and programmable) to return a decision about whether the portion of the message that it was asked to scan contains a virus or is spam or not.
XENVFROMHDR A mail header that is added by the "-Xenv" option to give the name of the sender, as passed to sendmail on the envelope.
XENVTOHDR A mail header that is added by the "-Xenv" option to give the names of all of the recipients, as passed to sendmail on the envelope.
XVERSIONHDR A mail header that is added by the "-Xver" option to show the name of the sendmail filter and its version number.
MAXADDRESSLENGTH The maximum length of a mail address that will be handled by the filter. The current value of 255 should be sufficient.
MAXSUBJECTLENGTH The maximum length of a message subject that will be handled by the filter. The current value of 255 should be sufficient.
MAXDATELENGTH The maximum length of message date stamps that will be handled by the filter. The current value of 32 should be sufficient.
CORRAL_DIR The directory where all of the corralled messages and spam will be placed (per the templates below). You can change the path name but be careful that the path name and substituted text in the templates does not come to more than 255 bytes.
RECV_NAME_TEMPLATE The template used to determine where a saved copy of the actual message received, for any filtered messages, should be placed and how it should be named. You can change the template but be careful that the path name for the corral directory (above) and substituted text in the sprintf part does not come to more than 255 bytes. Also, be certain to allow for three components to be substituted into the name (i.e. leave the "%s" and "%X" components there in some form).
SPAM_NAME_TEMPLATE The template used to determine where saved copies of any spam messages received should be kept and how they are named. Same rules as for RECV_NAME_TEMPLATE.
REPL_NAME_TEMPLATE A template used to make up lock file names for replies to spammers. Same rules as for RECV_NAME_TEMPLATE.
DEBUG_FILE_NAME The name of the file where debugging information is written. Defining this name causes debugging information to always be written to this file. For transient debugging, the "-d" command line option or the "DebugFile" config file parameter is available.
DEBUG_SENDMAIL A flag that can be defined to debug sendmail itself, as well as the filter.
PROGNAME The name of the sendmail filter, used for error reporting.
PIDFILE The name of the PID file that is used to track the filter's running status.
GLOBALOPTIONS The name or names of the global sendmail filter configuration file. The names are searched in descending order and the search stops at the first one that is found.
LOCALOPTIONS The name or names of the local sendmail filter configuration file. The names are searched in descending order and the search stops at the first one that is found. The character '~' signifies the home directory of the recipient of the message.
chMsgxxx All of the variables that begin "chMsg" contain the text of the various messages that the filter inserts into filtered mail messages. You can change these messages to read however you'd like but please be aware of any substitutions into them and retain the same number of substitutions in your replacement message text.
AcceptableTypes This variable defines the table of acceptable MIME types that can appear inline in a message. All others are considered unacceptable (by virtue of the fact that they might conceivably contain a virus).
FilteredFiles This variable defines the table of file extensions which are examined for viruses and the actions to be taken if they are found in a message.
RejectedHTMLs This variable defines the table of dangerous HTML tags that will always be laundered out of a message, regardless of where they are found. Note that many mail readers will interpret these tags, despite their being found in non-HTML messages, hence the reason for their removal.
AcceptableHTMLs This variable defines the table of harmless and therefore acceptable HTML tags that will never be laundered out of a message. The implication here is that, if a tag isn't found on this list, it will be laundered out. This list only applies inside <html> and </html> tag pairs.

1.3.8 Compiling and Installing

After making any changes to the makefile and "smfopts.h", run make to compile the filter. You can do this as "JoeUser" because there is nothing special taking place at this step. Simply type "make" in the install directory.

To install the filter, become super-duper user and type "make install". This will also restart sendmail which will have no effect until you hack its startup script (in Section 1.5). However, if you make future changes to the filter, "make install" will install it and restart it.

Once you have installed the filter, you can remove the program binaries and object files from the source code directory, if you wish, by typing "make clean".

1.4 Configure Sendmail

Hack the mc file for "sendmail.cf" (this is way better than hacking "sendmail.cf" directly, which is definitely not for the faint of heart). Start by copying the canned configuration supplied with the OS, if you haven't already done so. For example:

cp /usr/lib/sendmail-cf/cf/redhat.mc /usr/lib/sendmail-cf/cf/mysys.mc

Hack /usr/lib/sendmail-cf/cf/mysys.mc, and make sure the following lines are present or are added before the "MAILER" definitions:

define(`_FFR_MILTER', `1')dnl
INPUT_MAIL_FILTER(`filter1',`S=inet:2526@localhost, F=R')dnl
define(`confINPUT_MAIL_FILTERS',`filter1')dnl

The port number (2526) is the port that sendmail will use to talk to the filter on. You can change it to whatever you like, so long as you also change the port number used to start the filter (see Section 1.5).

Build a new "sendmail.cf" by running the M4 macro processor over the ".mc" file. You can use the following commands.

m4 mysys.mc >mysys.cf
cp mysys.cf /etc/sendmail.cf

Note that the default values for timeouts between sendmail and the filter are sometimes too small (i.e. 10 seconds). If you experience problems with timeouts (usually when the spam arbitron takes too long to identify a message), you may want to use:

INPUT_MAIL_FILTER(`filter1',`S=inet:2526@localhost, F=R, T=S:10s;R:60s;E:5m')dnl

A full description of the sendmail parameters that apply to the filter can be found at www.sendmail.org/~gshapiro/8.10.Training/milterconfig.html.

1.5 Hack the Startup Script

Hack the sendmail startup script (e.g. for Linux, "/etc/rc.d/init.d/sendmail") and add something to this effect:


      # Start the mail filter
      echo -n $"Starting sendmailfilter: "
      /usr/sbin/sendmailfilter \
           -h -i -p inet:2526@localhost >/dev/nul 2>&1 &
      FILTVAL=$?
      [ $FILTVAL -eq 0 ] && success "sendmailfilter startup" \
           || failure "sendmailfilter startup"
      echo
           .
           .
           .
      echo -n $"Shutting down sendmailfilter: "
      killproc sendmailfilter
      echo

You want to start the filter after sendmail is started and stop it before sendmail is stopped. The port number (2526) is the same one that was defined in the sendmail configuration file (Section 1.4, above). A sample startup file for Linux is supplied in "sendmail.cf", included with this package. A sample startup script for BSD is supplied in "rc.sendmail", also included with this package.

You may want to set the optional flag "-D" on the startup line for the sendmail filter. I run with '-D "/etc/mail/local-host-names"' to set the list of local domain names. Also, if you're going to be using a spam arbitron daemon (e.g. SpamAssassin), set the "-s" parameter to use the port that spamd is listening on (e.g. "-s spamd:2527@localhost"). And, if you're going to be using a virus arbitron daemon (e.g. ClamAV), set the "-v" parameter to use the port that clamd is listening on (e.g. "-v clamd:2528@localhost").

If you are an ISP and you have mail users that don't really exist (i.e. have no home directory) or are delivering mail to other machines, you might want to look at the "-u" and "-q" parameters respectively. Setting the "-u" parameter is especially important if you have users with no home directories and you are going to be using Spam Assassin to scan mail messages for spam.

Restart sendmail so that it reads the new config file and starts the filter. For example, on Linux:

/etc/rc.d/init.d/sendmail restart

Alternately, on Linux, you can copy the "sendmailfilter.rc" file to "/etc/rc.d/init.d/sendmailfilter" and run "chkconfig --add sendmailfilter". This will start/stop the filter separately from sendmail, hopefully in the correct order.

1.6 Install the Message Remailer

MailCorral ships with exact copies of the essential spam notification and message handling programs of SpamCorral, its sister spam handling product. SpamNotify is described in section 1.7 (below), while this section describes MailRelease.

If you wish to allow the recipients of filtered messages to remail them unattended from the corral to themselves, you will need to install MailRelease, the optional message remailer. You will also need to enable this feature by including the "-rm" option on the filter command line or using "AutoRemail" in the config file.

To set up message remailing, you first need to create a message handler robot and configure sendmail to allow forwarding to work, as well as configuring the message remailing utility to run the way you want.

1.6.1 Configure the Message Remailer

Hack the source file MailRelease.pl and change the appropriate variables therein (an easier way is described in the SpamCorral documentation. which shows how to alter this file as well as set up foreign language support). Be sure to make any changes to this file, before you run "make install" so that they will be copied from the build directory to the appropriate install locations. Whichever method you use, you can change these variables:

$CORRAL The path name of the directory where the original copies of filtered messages are saved until they are released or deleted. This should match the directory used in the CORRAL_DIR manifest constant set in smfopts.h (see section 1.3.7).
$LOG_FILE The name of the log file where information about all messages that are remailed, etc. is written. You will need to define this variable to and set it to a valid file name to turn on logging (which is off by default).
$REMAIL_HDR_BYPASS The remail header string that is used to indicate that a message can be remailed. This string must match the tag placed in the message by MailCorral (defined in smfopts.h).
$MAXHEADERLEN The maximum number of message header bytes to read while looking for remail tags in compressed, corralled messages. If the remail tag isn't found after reading this many bytes, the message cannot be released for remailing. Note that this only applies to compressed messages.

There is a tradeoff between the speed of releasing messages and the number of bytes read while looking for the remail tag. If the users are releasing large numbers of messages, you might to reduce this variable from its default setting of 16384 to a smaller value. However, be advised that smaller values (e.g. 4096) might cause some messages with long delivery lists (common with some types of spam) to be unreleasable when compressed.

$IGNORE_DOMAINS This variable should be defined to any value, if you wish MailRelease to ignore the user's domain name when processing release requests. Normally, the domain name of the requestor is checked against a list of valid domain names, before they are allowed to release corralled mail. This feature is normally turned on, since it isn't troubling to most users and turning it off is a moderate security hole.
$CHECK_USER_NAMES This variable should be defined to any value if you wish MailRelease to check that a user's name matches the recipient of the message before it can be remailed. This feature defaults to off, since it proved to be more trouble than it was worth to most users. However, it is a big security hole.
$DELETE_ON_RELEASE This variable should be defined to any value if you wish MailRelease to delete messages from the corral, once they are released for remailing. This keeps the corral cleaner but it can cause problems if the message is not delivered for some reason. Consequently, this feature defaults to off and released messages are kept in the corral until the regularly scheduled cleanup program finds and deletes them.
@LOCAL_DOMAINS The list of domain names that are considered local to this site. Messages will only be remailed to users who send mail from one of the approved domains. This prevents outside users from remailing messages containing viruses, etc.

The list may also include file names, which can contain lists of local domain names, one per line. Typically, this feature would be used to point to the file "/etc/mail/local-host-names" or the same place where sendmail gets local domain name information from.

$ERRORXXXX A series of error messages that can be sent back to the user, if release fails for some reason or another.

Test out any changes you make to this program by running it standalone. In production, it will be running from a ".forward" file so it is likely to fail silently. Note that this test assumes that you've installed the filter and gotten it working and that it has corralled some messages in the corral, as you will be trying to release one of those messages during the test.

To test MailRelease.pl, make up a test email message in a file and pass it through the script. Your test email message must include a from header and a line that matches a filtered message file in the mail corral (the message must have been filtered with the "-rm" option turned on). Note that it is of the utmost importance that there be a blank line between the "From:" line and the "File:" line in the test message, just as there would be in a real mail message. For example:

From: joe@localhost

File: /var/spool/MailCorral/recv_from_mrvirus@virusesrus.com_3F9AE27C

If your test file was called "testrel", as root, try the following command:

./MailRelease.pl <testrel

Note that you should pick a filtered message that you don't really care about, for this test, since it will be deleted once it is remailed. If all is working, "joe" should see the remailed message.

Warning: if the filtered message contains a virus, you should be extremely careful, since you will be remailing the virus through the system without any filtering. It will arrive unscathed and fully functional in Joe's mailbox.

1.6.2 Create a Message Handler Robot

The message handler robot is actually a separate userid that is created for the sole purpose of releasing filtered messages to the general public whenever users ask it to do so.

A copy of each filtered message is added, untouched, to the message corral by the sendmail filter when a received message is altered to protect against viruses, etc. The altered message sent to the recipient indicates how the user can have the unaltered copy of the message remailed to them by sending a message to a particular user (the message handler robot).

The user sends a message to the userid that you create in this section. The userid has a .forward file which directs all incoming mail to the message handler ("MailRelease.pl"). This script scans the mail that it receives, from the other users, and releases the appropriate unfiltered message to them for delivery.

First, you must set up a new userid with a name of your choice. If you altered the REMAILROBOT variable in Altering Fixed Constants and Variables step, you should pick a userid that matches what you chose there. Otherwise, the default name is "mailrobot". As an example, to set up a new userid under Linux:

useradd -c "Message handling robot" -m mailrobot

Note that it is very important that the userid which you create have a valid shell named in its /etc/passwd entry. This shell need not be one that actually works (if you wish to keep people from logging in to the robot's userid) but it must be one that is named in /etc/shells. The default shell, assigned by new user creation, should work but be aware of this fact if you assign any specialty shells to this userid. Sendmail will not forward mail to MailRelease.pl if this isn't the case.

Next, we are going to copy "MailRelease.pl" to either the mail robot's home directory or a subdirectory under it (henceforth, we assume the home directory). But, before we do this, make certain that whatever directory you choose is not writable, except to the owner (no, not even the owner's group can have write permissions). If the directory is writable by anyone but the owner, sendmail will not forward mail to MailRelease.pl

Now, copy "MailRelease.pl" to the mail robot's home directory. Give it the right kind of permissions, etc. to allow sendmail to execute it when it reads the ".forward" file. Generally, sendmail runs as the recipient of a message so this script should have permissions that look like "-rwsrwxr-x" and it should have ownership of root/mailrobot. Note that this program is running as setuid root to allow it to manipulate the files that it needs to mess with. Be careful if you make any modifications to this program. Also, on SuSE there is a permission system which will reset the setuid bit each time SuseConfig is run so you will have to change the permissions in /etc/permissions.easy to 4775 for MailRelease as well, otherwise they will have to be reset manually each time SuseConfig is run.

Pay particular attention to the operation of the setuid bit. Some operating systems (e.g. Linux) ignore the setuid bit on shell scripts, for security reasons. If yours is one of these operating systems, Perl may attempt to simulate the action of setuid. If it does, MailRelease.pl will run correctly when invoked under a non-root userid. If Perl doesn't simulate setuid, you will need to run the setuid version of Perl. Normally, the "configure" script determines this fact and makes the appropriate changes to MailRelease.pl. However, if it doesn't you will have to hack the first line of MailRelease.pl and point it at "suidperl" (probably the same path as is already there but with the name of "suidperl" instead [if you can't find suidperl, it may not be installed, in which case you will need to get it and install it -- under RedHat Linux, suidperl comes as a separate RPM, called perl-suidperl]). You should check that MailRelease.pl can release messages from the corral when run from the userid mailrobot to make sure setuid is working (run the same test as above but do "su mailrobot" first).

On some systems, suidperl will not work either (e.g. on RedHat 7, with Perl 8.5.0, I could not get it to work under any circumstances). In this case, create the following C wrapper ("MailRelease.c"):

#define REAL_PATH "/home/mailrobot/MailRelease.pl"
main(ac, av)
    char **av;
{
execv(REAL_PATH, av);
}

Compile it with your favorite C compiler (e.g. "cc -o MailRelease MailRelease.c"). Give this program the setuid root permissions instead of MailRelease.pl.

Set up a ".forward" file under this userid that reads:

"|MailRelease.pl"

Note that in some cases, sendmail requires that the full path name of the executable file be used in the ".forward" file and that the name used match exactly the name that is pointed to by the symlink in the "/usr/adm/sm.bin" or "/etc/smrsh" file (see the next section). In that case, the ".forward" file should look like this:

"|/home/mailrobot/MailRelease.pl"

Or, if you had to resort to compiling a C wrapper for the shell script, change the ".forward" file to read:

"|MailRelease"

Make sure that the permissions are "-rw-------" and the owner/group is set to the userid (e.g. "mailrobot") that you created above or forwarding won't work. Sendmail checks the ownership and permissions, for security reasons, before it will do any forwarding.

1.6.3 Additional Sendmail Configuration

Sendmail now has additional security that prevents ".forward" files from executing programs directly. It uses a special shell ("smrsh") that performs extra security checking before allowing a program to be run from a ".forward" file. This shell is very restrictive, almost to the point of being ridiculous, resulting in a good many reasons why it should prevent MailRelease.pl from working (many of which fail silently). You should first read "man smrsh" for more information about "smrsh" and then follow these instructions carefully to get it working.

Note that, if you don't follow these instructions to the letter, there is a very good chance that sendmail will simply drop the message to your mail robot or, just as bad, deliver it to the mail robot's local mailbox (this is a good thing to check ["mail -u mailrobot"], by the way, if the sendmail log says your messages are being delivered but MailRelease.pl is, none the less, failing to release anything). You will get no error message or warning as to why this happens. Sendmail's attitude is that you made a mistake so its not going to do what you wanted and it isn't going to give you any clues why, either. If you end up reading its code to see why its not forwarding messages to your mail robot, you'll see that it even has a flag that it sets (DONTBLAMESENDMAIL) to indicate its not its fault. Too funny! Hopefully, if you follow these instructions scrupulously, it won't come to that.

When you built sendmail, smrsh should have been built along with it. If not, build it now. If the sendmail install did not do so already, copy smrsh to a convenient location (e.g. "/usr/sbin/smrsh"), probably the same directory as sendmail resides in. Apply execute permissions so that the world can execute smrsh, if they aren't already there.

For forwarding of any kind to work, the shell chosen for the userid that is forwarding must be a "valid" shell. This means that the shell named in the userid's entry in /etc/passwd must be listed in the file /etc/shells. Verify this fact. The shell chosen need not be one that actually works (if you wish to keep people from logging in to the robot's userid) but it must be one that is named in /etc/shells. The default shell, assigned by new user creation, should work but be aware of this fact if you assign any specialty shells to this userid.

To get smrsh to execute programs, you need to create a directory named "/usr/adm/sm.bin". You must give it ownership and permissions of root/root/755 and put into it soft links to any programs which will be executed by a ".forward" file. Be very careful what programs you put in this directory because programs therein can be executed by mail messages if a hapless user decides to add them to their ".forward" files.

In our case, we need to put in a soft link to the message handling robot. For example:

ln -s /home/mailrobot/MailRelease.pl /usr/adm/sm.bin/MailRelease.pl

Or, if the C wrapper was used to get MailRelease to run setuid root, try something like this:

ln -s /home/mailrobot/MailRelease /usr/adm/sm.bin/MailRelease

Alternatively, some versions of sendmail use "/etc/smrsh" to implement this feature. You can look at the source code for sendmail (specifically, see the code in smrsh.c) or try both of them until you get it to work.

The sendmail configuration file must be altered to enable message forwarding. This is done by adding some lines to the ".mc" file for "sendmail.cf" (this is way better than hacking "sendmail.cf" directly, which is definitely not for the faint of heart) and running it through M4. Start by copying the canned configuration supplied with the OS, if you haven't already done so and add the following:

FEATURE(`smrsh', `/usr/sbin/smrsh')dnl

Note that, in the above line, you should change "/usr/sbin/smrsh" to point to wherever you actually installed smrsh.

Also note that the default "cost" for mail passed to the program mailer that invokes smrsh is "expensive". This has the unwanted effect of placing mail forwarded to programs into the mail queue where it can sit for hours before being delivered. I cannot even speculate why this choice was made for the default but you may not wish it to be so.

If you would like your mailrobot to respond quickly to requests, you will need to change the expense of the local mailer. First, you should check your sendmail.cf file to see if the expensive flag is, indeed, one of the flags being passed to the program mailer. If you see the letter 'e' in the flags, defined by the parameter "F=", for the "prog" mailer, you'll need to remove it. For example, if your config file reads something like:

Mprog,     P=/usr/sbin/smrsh, F=lsDFMoqu9e,
     S=EnvFromL/HdrFromL, R=EnvToL/HdrToL, D=$z:/,
     T=X-Unix/X-Unix/X-Unix, A=smrsh -c $u

You will need to change the expense of this mailer. To do this, add the following line to the ".mc" file:

define(`LOCAL_SHELL_FLAGS', `u9')dnl

Be sure that this line occurs before the line that says "MAILER(`local')", since that macro uses the values set by it. And, if there is no definition for a local mailer, be sure to add it.

Compile the configuration and install it, per the directions in the "Configure Sendmail" section, above.

If you prefer to hack the "sendmail.cf" file directly, you should add something that looks like this to the end of the file:

Mprog,     P=/usr/sbin/smrsh, F=lsDFMoqu9,
     S=EnvFromL/HdrFromL, R=EnvToL/HdrToL, D=$z:/,
     T=X-Unix/X-Unix/X-Unix, A=smrsh -c $u

Its time to try smrsh to see if will run MailRelease.pl. To do this, logon as JoeUser (i.e. not root) and run MailRelease through smrsh, using the following command (or whatever is appropriate for where you installed smrsh):

/usr/sbin/smrsh -c /home/mailrobot/MailRelease.pl

Due to the fact that smrsh looks up all of the programs it executes in /usr/adm/sm.bin, it might seem to you that the actual path used to execute MailRelease.pl is unimportant. One might think that only the program name matters, and you might even be correct for some of the earlier versions of smrsh. However, it would now appear that smrsh is checking to see that the path that you use to execute the program matches exactly the path pointed to by the symlink in /usr/adm/sm.bin. So, you should use the correct path wherever you reference MailRelease.pl (e.g. in the command above and in the .forward file).

If you can't get smrsh to execute MailRelease.pl, check all of the permissions, etc. and keep trying until it works. If it does execute, type a couple of characters (it reads from stdin) and give it an eof. It should do nothing but terminate. Or, if you like, you can feed it the test message you made up earlier and see if it really works, although here we're basically just testing to see that smrsh will load the program.

For the next step, you'll probably want to turn on logging by aiming $LOG_FILE at a file (e.g. "/var/log/mailrobot"). Once this is done, send a message to mailrobot that has the name of a corralled piece of mail as its subject line. For example:

mail -s /var/spool/MailCorral/recv_from_ew@jg.sp_3E2278A4 mailrobot

You should see a log entry for the request and, if the message is a real one, it should get released. If no log entry is created, you may need to run MailRelease with suidperl (see the "Create a Message Handler Robot" section, above). You should also check to see that the message is not in the queue, waiting to be delivered (see above for sendmail configuration changes needed to circumvent this), that the permissions on MailRelease are "-rwsrwxr-x" and that it has ownership of root/mailrobot.

If MailRelease doesn't work, its usually not its fault. There are a lot of things that must be right before sendmail will condescend to execute the program. Please check them all before blaming MailRelease.

1.6.4 Alternative to Message Handler Robot

Kai Schaetzl has suggested an alternative to setting up a separate userid for the message handler robot. Although my personal preference is the separate userid approach, it is somewhat difficult to set up and get working. Kai's alternative is easier and works fine on SuSE (we haven't tried it elsewhere). Basically, here's what he had to do:

To the aliases file (/etc/aliases or /etc/mail/aliases), add this alias (or something like it):

mailrobot: |/usr/sbin/MailRelease.pl

Set the setuid bit for MailRelease. If you'd like to restrict access to this program, use the following permissions:


  -rwsr-x---  1  root daemon  14831 Mar 11 18:08 MailRelease.pl

Make sure that /usr/sbin/suidperl has the setuid bit on, as well. With this arrangement, mail is piped through MailRelease by sendmail as "daemon:daemon". That's the reason for setting MailRelease to group daemon. If the permissions are not set correctly, you'll get an unknown sendmail error 162. Sendmail itself doesn't seem to be running as "daemon:daemon", maybe as "mail:mail", so the shell for piping may be different from the user sendmail runs as.

Note that the same warning about the SuSE permission system applies here. This system will reset the setuid bit each time SuseConfig is run so you will have to change the permissions in /etc/permissions.easy to 4755 for MailRelease, otherwise they will have to be reset manually each time SuseConfig is run.

1.7 Install the Optional Spam Notifier

MailCorral ships with exact copies of the essential spam notification and message handling programs of SpamCorral, its sister spam handling product. MailRelease is described in section 1.6 (above). This section describes the optional spam notifier SpamNotify.

If you are corralling spam and you wish to receive regular notifications of the corralled spam, you should install SpamNotify. If you don't corral spam or don't care about notifications, you can skip this section. For more complete documentation on this program, see the SpamCorral documentation.

1.7.1 Configure the Spam Notifier

To configure the spam notifier, you can hack the source file SpamNotify.pl and change the appropriate variables therein (an easier way is described in the SpamCorral documentation. which shows how to alter this file as well as set up foreign language support). Be sure to make any changes to this file, before you run "make install" so that they will be copied from the build directory to the appropriate install locations. Whichever method you use, you can change these variables:

$CORRAL The path name of the directory where suspended spam is saved until it is released or deleted. If you are using MailCorral, this should match the directory used in the SPAM_NAME_TEMPLATE manifest constant set in smfopts.h.
$STAMPFILE The name of a file to use for holding the timestamp of the last spam notification message. You may pick any name you like, provided it doesn't collide with the names used by the mail fitler (e.g. none of the names in the message file name templates in MailCorral's smfopts.h).
$SPAMROBOT The userid of the spam handler robot (that will be created in Section 1.6.2, above). However, you probably won't have to change this variable if you are using a version of sendmail that forwards mail to programs via smrsh. The reason is because SpamNotify automatically detects which userid is being used for the spam handling robot, at runtime. The value set by this variable is only used as a fallback, default. Note that, if a value is set in the config file, it is taken as the absolute name and the results of automatic detection are ignored.
$MAXHEADERLEN The maximum number of message header bytes to be read while processing messages for notification. If the end of the headers isn't found after reading this many bytes, the message is skipped and no notification is sent.

There is a tradeoff between the speed of processing messages and the number of bytes read while looking for headers. If there is a large amount of corralled spam, you might want to reduce this variable from its default setting of 16384 to a smaller value. However, be advised that smaller values (e.g. 4096) might cause some messages with long delivery lists (common with some types of spam) to be skipped.

$DATEFMT The date/time format string that is to be passed to sprintf whenever date stamps are formatted for display. This value consists of the format pattern string, followed by the variables to be substituted into it. Note that the double quotes around the format string and the dollar signs in the variable names must be escaped, since this value is evaluated multiple times. If you don't escape these items, your date stamps will come out broken or all zeros.

Within this parameter, you can use the variables: "$Year" (the full, four-digit year number); "$Mon" (the two-digit month number); "$MDay" (the two-digit day of the month); "$Hour" (the twenty-four hour clock); "$Min" (the minutes in the hour); "$Sec" (the seconds in the minute). The default value is:

\"%04d-%02d-%02d %02d:%02d:%02d\", \$Year, \$Mon, \$MDay, \$Hour, \$Min, \$Sec

which gives a date stamp that looks like "2003-04-18 03:27:05". If you live in Europe, you might want to try:

\"%02d.%02d.%04d %02d:%02d\", \$MDay, \$Mon, \$Year, \$Hour, \$Min

which gives a date stamp that looks like "18.04.2003 03:27".

$NOTIFYSUBJ The subject line to be used on notification messages sent to the users about spam held for them.
$NOTIFYMSG The text of the message that is sent to the users, describing the spam that is waiting for them and what to do about it. This is a multi-line message that begins the body of the notification message.
$XXXXTAG The various tags that are used in the body of the notification message to identify the components of each piece of spam. Things like the from, subject and date headers.

1.7.2 Set up Periodic Notifications

Spam that is corralled by the filter is placed in the corral, pending release for remailing by the recipient or, ultimately, deletion by the cleanup prcoess (after a set period of time has elapsed). In order for the recipients of the spam to know what they have been sent, so that they can decide whether they want it remailed to them or not, the spam notifier should be run at regular intervals.

The spam notifier is run by an entry in your cron table, probably once or twice a day (depending on how often your users want to see what spam they've received). I like to do these kinds of things at the end of the day, when the load on my systems is low. Here's the crontab entry that I use:

# Once a day at 01:00, notify the users of all of the spam received in
# the last 24 hours.
00 1 * * * root /home/mailrobot/SpamNotify.pl

Alternately, if you are expecting huge amounts of spam, you may wish to partition the corral (this puts less load on the file system and avoids "out of resources" type crashes when processing the corralled spam). Here is a crontab entry that requests 24 hour partitioning (the default):

# Once a day at 01:00, notify the users of all of the spam received in
# the last 24 hours. Partition the corral at the same time.
00 1 * * * root /home/mailrobot/SpamNotify.pl --Partition

Note that the filter also has a partition setting which applies to all corralled messages (viruses as well as spam). If you use this setting (and, if you want partitioning, you should), you needn't worry about SpamNotify doing any partitioning, although it doesn't hurt to do it both places. SpamNotify will partition any corralled messages that didn't get partitioned by the filter (e.g. accidentally), otherwise it will do nothing. See the "Partition" or "-pn" options elsewhere in this document.

1.8 Set up Periodic Cleanup Jobs

The sendmail filter generates an unaltered copy of every email message that is not delivered asis. You should add something to your cron table to clean these files up after a reasonable length of time.

The script "filterclean" cleans up all of the older filtered messages, retaining those that are less than thirty days old (note that the directory name must match the template name in smfopts.h):

# Once a day at 00:30, remove all of the filtered mail messages that are
# over 30 days old.
30 0 * * * root /etc/mail/filterclean

Optionally, you may hack this script to enable compression of all messages older than 7 days old. To do this, uncomment (by removing the '#' from the beginning of the line) the last line in the script that reads:

#/usr/bin/find /var/spool/MailCorral/ -not -name \*.gz -depth -type f -mtime +7 -exec gzip \{\} \;

If you create debugging files (enabled by default), you should clean them up weekly using logrotate. Here's the lines you need to add to the logrotate config file (/etc/logrotate.conf):

/var/log/sendmailfilter {
     missingok
     notifempty
}

1.9 Configuring Local Options

The operation of the sendmail filter can be altered by configuration parameters in two configuration files. One is a global configuration file that applies to all of the messages processed by the filter. The other is a local configuration file that only applies to the messages for an individual recipient. Typically, there is one such file for each recipient, in their home directory.

Unless the GLOBALOPTIONS and/or LOCALOPTIONS variables in smfopts.h are changed or the "-C" flag is used on the command line, the global configuration file will be "/etc/mail/sendmailfilter.cf" or "/etc/sendmailfilter.cf" and the local configuration file will be ".sendmailfilter", in the recipient's home directory.

Either of these files can contain filtering options, spam processing options, message inserts text, white lists, black lists and spam delivery options. In addition, the globals options file can contain other startup options, including those that can be specified on the command line.

You will probably want to update the global options files with spam processing information specific to your installation. For example, it is suggested that, if you constantly receive spam from a particular domain, you should add that domain to the global black list. See the Configuration section of the documentation for more information on which options can be set in these files.

The configuration files can be edited at any time. The sendmail filter must be restarted for new global options to take effect. Changes to the local configuration file take effect with the receipt of the next message destined for the user in question.

1.10 Installing the RPM on RedHat Linux

MailCorral is available for installation on RedHat Linux in RPM form, which reduces the complexity of the installation considerably. The RPM is built on RedHat 7.0 and 8.0 and requires sendmail 8.12.0 and above. There should be no reason why it won't work on later versions of RedHat Linux, as there aren't any real OS dependencies in the filter. Similarly, it should work on later versions of sendmail (such as 8.12.9).

To install the filter, download the appropriate RPM for your system and place it in a suitable directory on your machine. As root or some other equally omnipotent user, type the following command:

rpm -U /xxx/MailCorral-vvv.i386.rpm

where "xxx" is the directory where you saved the RPM and "vvv" is the version information for the RPM you downloaded (e.g. "1.1.2-1.8.0"). If you prefer, you can install the RPM using a visual package manger such as GnoRPM.

The RPM will set the following filter parameters to the values noted. Most of them can be overridden in the configuration file (see the Configuration section of the documentation for more information on which options can be set). The settings are:

TECHSUPPORT The name of the person whom users should contact if they need assistance with filtered messages. Set to "postmaster".
REMAILROBOT The username of the message remailer robot. Set to "mailrobot". A user account for "mailrobot" is also created by the RPM.
POSTMASTER The name of the person to whom warning messages should be emailed if any viruses or unknown filter types are found. Left undefined so that no warning messages are sent.
DOMAIN The name or names of the local domain. Set to a bogus domain name of "yourdomain.com". You will need to override this with the "-D" command line option or the "DomainList" config file parameter.
CORRAL_DIR The directory where all of the corralled messages and spam will be placed. Set to "/var/spool/MailCorral".
DEBUG_FILE_NAME The name of the file where debugging information is written. This name is left undefined. If you want to create a debugging file, the "-d" command line option or "DebugFile" config file parameter are available.
GLOBALOPTIONS The names of the global sendmail filter configuration files. Set to "/etc/mail/sendmailfilter.cf;/etc/sendmailfilter.cf".
LOCALOPTIONS The name of the local sendmail filter configuration file. Set to "~/.sendmailfilter".

The RPM installs the following files into the directories noted:

/etc/mail/filterclean A shell script that is run by the daily cron job to clean up any corralled filtered messages and spam that are older than 30 days. You may alter this script to reduce or increase the number of days that corralled messages are kept.
/etc/rc.d/init.d/sendmailfilter A startup script that is run, at boot time or when the runlevel is changed, to start/stop the filter. You may want edit this script to change the options used to start the filter.
/usr/sbin/MailRelease.pl A Perl script that is used by the mailrobot to automatically release mail that is corralled because it contains viruses, etc.
/usr/sbin/SpamNotify.pl A Perl script that is run at regular intervals (daily) to send out notification messages of all spam that is corralled since the last time it was run.
/usr/sbin/sendmailfilter The sendmail filter itself.
/usr/share/doc/MailCorral-xxx The documentation directory. The string "xxx" is replaced with the version of MailCorral contained in the RPM.

In addition to installing files, the RPM carries out most of the work necessary to configure the filter. It configures the following items:

The RPM copies "/usr/lib/sendmail-cf/cf/redhat.mc", the RedHat supplied sendmail macro configuration file, to "/usr/lib/sendmail-cf/cf/MailCorral.mc" and hacks it so that the filter will be invoked by sendmail on port 2526. It then compiles the macro file to yield "/usr/lib/sendmail-cf/cf/MailCorral.cf".

The RPM copies adds the filter startup script "/etc/rc.d/init.d/sendmailfilter" into the appropriate start/stop directories at the right runlevels so that the filter will be started/stopped at boot time and when runlevels are changed.

A userid of "mailrobot" is created and set up, if it doesn't exist. This userid will be configured to receive mail messages sent to it requesting that corralled messages be released and to carry out these requests.

The SpamNotify script will be copied to the home directory of "mailrobot" and set up to run at daily intervals through a link placed in /etc/cron.daily.

The corral cleanup script will be set up to run daily (whenever files in "/etc/cron.daily" are run). This will cause corralled messages over 30 days old to be deleted.

All that user need do, after the RPM is installed, is complete the hacking of "MailCorral.mc". Compare it to your sendmail config file and make whatever changes are necessary to configure sendmail as you currently have it set up. If you prefer, add the three lines needed to start the filter to your existing configuration file. The "Configure Sendmail" section (above) talks about how to make the changes and then compile and install the configuration file in the sendmail directory.