EMAIL FILTER VALIDATION SUITE
Most email viruses employ similar propagation techniques and
are, consequently, very similar in external appearance to one another. That
being the case, it is possible to design an email virus filter that removes
viruses based on outward appearance. This is highly desirable, since it
will ensure that, even when a new virus comes along, your filter will be
able to screen for it.
On the other hand, the penalty for failing to detect a virus
in an email message is high. How do you know if you've covered all the
bases by detecting all of the important propagation techniques? This page
will allow you to download a email filter validation suite that consists of
generic email messages which can be passed through your filter to test all
of the popular methods of virus propagation. If your filter detects each
message and handles it correctly, it is probably ready for prime time.
Although not nearly as malicious as virus writers, spammers
are sometimes equally sneaky. Since many installations have systems in
place to detect and eliminate spam (such as BSM Development's
this puts a crimp in the spammer's style. In order for spam to work, it
must be delivered to its target. Figuring out new ways to slip spam by
email filters is one of the things that spammers do periodically. This
validation suite also presents tests for some of the latest spam
Here is a list of the messages included in the test suite and
the filter criteria they are intended to test. The name of each test is
given as well.
The messages beginning with "baddoc_" generally test the
limits of mail filters and/or their ability to handle misbehaved or
- To begin with, a simple test message that is plain text. It doesn't
contain any MIME entities, although it is tagged with a "Content-Type" of
"text". Your filter should process this message as a plain text message
with no alterations (baddoc_text_plain).
- A message with a number of small attachments, all of them supposedly
innocuous. The filter should ignore them and pass them untouched
- A second message with several small attachments, that could be
harmful and are at least suspect. The filter should give a warning about
each of them but otherwise pass them untouched
- A third message with a number of small attachments, that are assumed
to be harmful. The filter should do something to render them harmless
(e.g. rename them so that they have extensions which are not
automatically opened) and give a severe warning about each of them
- A message with a single, large attached document of dubious nature.
Tests the ability of the filter to handle large attachments properly
(since milter spans messages in chunks) and to detect possibly harmful
- A message may contain a single attachment with no message text at
all. This happens when someone mails only the attachment with no
explanatory text. A virus filter must be able to handle this test case by
at least examining the attachment and rejecting malicious ones, whether it
actually generates a warning message or not. Note that, in our test case,
we also played a few games with the MIME description of the attachment (as
a virus might) to give your filter an extra test for thoroughness
- Alternately, a message may consist of a single attachment that
contains message text. Some mailers treat such attachments as simple
messages, despite their being marked as attachments. For example,
"text/html" and "text/plain" entities that are given attachment names in
the "Content-Type" header of a message are often treated this way. A virus
filter should examine these kind of attachments and alter or reject the
malicious ones (baddoc_attachment_only_html).
- The attachments in this message contain Apple resource forks which are
sent, along with the attachment they describe, in pairs. Your filter
should deal with them as a single entity and not produce double warnings,
etc., about the resource forks themselves. Only the actual attachements
should be processed (baddoc_apple_double).
- All of the attachments in this message have no file type. If your
filter is based on file extensions only, it should warn you about them
unless it can examine the MIME type as well as the file type. If that
is the case, the harmless MIME types should be ignored and the others
should be generate a warning (baddoc_attachment_untyped).
- Some mailers may accidentally send MIME-encoded messages that have no
message body. A sample of one such message is included to test the
robustness of your filter (baddoc_body_empty).
- Mail messages that are forwarded from one user to the next often
include multiple levels of nested mail delivery information, mixed HTML
and plain text components. A test message that demonstrates such nesting
of delivery information and mixed content is provided to stress test a
filter's ability to handle multiple levels and apparently confusing
- Many email packages immediately interpret HTML in an effort to
properly render formatted messages. Good news for viruses, because this
indiscriminate rendering of HTML makes for a fine propagation tool. HTML
can appear as the only component of a MIME-encoded message, as in the
sample supplied, in which case it should be suitably laundered to remove
all harmful tags. Note that this example contains some tags broken in
typical ways used by virus writers to exercise your filter's HTML tag
- Another place where HTML can appear is in a multipart, MIME-encoded
message. A second message represents this case, which should be suitably
laundered to remove all harmful tags (baddoc_html_mime).
- Unfortunately, some email readers are prepared to interpret embedded
HTML, even when it occurs in a plain text message (i.e. not MIME-encoded).
This could have disasterous effects, if your filter does not look for HTML
tags in plain text messages. Another message presents a sample of this
kind of embedded HTML. Although it is less likely to be harmful, it
should still be processed (baddoc_html_embedded).
- Not all HTML in mail messages is particularly well-behaved. Much of
it is malformed. A test case is presented, containing mangled HTML that
should, none-the-less, be processed and rendered harmless
- On the other hand, some HTML is harmless, although it appears not to
be because tags can be included inside comments. Various mailers such as
Word under Outlook use the technique of embedding formatting information
into comments in HTML messages. An email filter should be able to handle
this kind of embedded formatting so a test case is supplied
- HTML is often encoded to obscure its true nature from simple message
filters. This message contains poorly-behaved HTML (similar to
baddoc_html_malformed) that is also encoded. It should, be processed and
rendered harmless, and then reencoded, thereby testing a filter's
ability to modify encoded messages (baddoc_html_encoded_malformed).
- Although uuencoding has been largely surpassed by Base64 encoding,
messages using this encoding are occasionally received. This message has
an HTML-only, MIME-encoded body that contains HTML tags that should be
removed by a filter. It tests the ability of the filter to decode
uuencoded HTML, launder it, insert a message into it and then re-encode
- Outlook is a ubiquitous mailer that is quite capable of producing
some real hum-dingers when it comes to HTML encoded messages. If Word is
set as Outlook's text editor, much of the messages generated by Outlook in
HTML mode are word and HTML formatting tags. One such message is included
as a typical example of what is possible. A good virus filter should
reject little or none of this message (baddoc_html_outlook_typical).
- Finally, it is possible for HTML to be sent as an attachment to a
message. Such attached HTML cannot be altered, since we can only surmise
its purpose, but a warning should be issued about it. A test for this
case is included (baddoc_html_attachment).
- An email message which includes all of the acceptable MIME types as
inline entities (i.e. not attachments). Since some email programs might
interpret these types directly and not treat them as an attachment, they
must be filtered separately from the attachments. The filter should allow
all of these types through (baddoc_inline_mime_bad).
- An email message which includes a sampling of known unacceptable MIME
types as inline entities (i.e. not attachments). Since some email
programs might interpret these types directly and not treat them as an
attachment, they must be filtered separately from the attachments. There
are many MIME types and it isn't possible to know what they all are, in
advance. This being the case, the filter should only allow those MIME
types, that it specifically knows about, through. The ones in this
message should be rejected as unknown/bad (baddoc_inline_mime_ok).
- A message may contain a single inline MIME (i.e. not an attachment)
with no message text at all. This happens when someone mails only the
MIME entity with no explanatory text. Since some email programs might
interpret this type directly and not treat it as an attachment, it must be
filtered separately from attachments. A virus filter must be able to
handle this test case by at least examining the MIME type and rejecting
malicious ones, whether it actually generates a warning message or not
- Some mail handlers may generate messages with MIME components that
don't conform to the standard. Regardless of this, many mail readers
will still handle them. In this case, a multipart/alternative section
appears to be misplaced because it is used solely for the purpose of
enclosing a virus scanner's certification. By the way, this message is
known to crash one version of the Perl module Email::StripMIME which is
frequently used in email filters, hence the reason for this test case
- As a test of whether your email filter can insert warnings into badly
formed messages, this example claims to be multipart/alternative but the
text portion of the alternative message is missing. Your filter should
still be able to insert a message into the HTML portion
- In order to detect spam/viruses in a message, an email filter may
need to decode encoded messages. This message contains an HTML component
that has been encoded to obscure it. The encoding has been done in
violation of the Base 64 rules, thereby producing an excessively long
encoded value, probably to cause filters to blow up and miss the spam
inside the message. The resultant message, while technically invalid is
handled by many mail readers and so must also be handled by your filter
- This message contains an HTML component that has a single line that
is very long in order to test if an email filter can handle very long lines
without blowing up (baddoc_html_overrun_straight).
Mail archivers are often implemented as a mail handling robot
to which regular email messages are simply sent by an MTA. The MTA simply
duplicates every mail message it sees and forwards the copy, through regular
channels to the archiver. Should the archiver crash for any reason, the
messages sent to it may be bounced by the MTA. This often leads to the
bounces being sent back to the original sender, thereby causing much
confusion, since they never sent any message to the archiver.
To support mail archiving, your filter may wish to detect
bounce messages and trash those resulting from delivery failures to your
mail archive robot's address. The messages beginning with "bounce_" are
meant to test this ability in particular, as well as a filter's ability to
handle bounce messages in general.
- A bounce message from a CommuniGate MTA
- A bounce message from Google's Gmail (bounce_gmail).
- A failure to deliver bounce message from Postfix
- A message delayed message from Postfix (bounce_postfix_warning).
- A failure to deliver bounce message from Sendmail
- A failure to deliver bounce message from Sendmail, caused by the
message being relayed through too many hosts. (bounce_sendmail_hops).
- A message delayed message from Sendmail
- A piece of spam that is disguised as a bounce message. This is a
pretty common techinque used by spammers. The spam content is usually
in the bogus delivery report. (bounce_spam).
The messages beginning with "spam_" are designed to test the
spam classifier in your mail filter.
- A simple spam delivery check message is supplied with an obvious spam
sender name. When you are developing your filter, you can simply check
for this name to identify that the spam is delivered (or not) properly
- Other than analyzing a message's headers, the most important portions
of a message to examine for spam are its plain text and HTML components.
Sample messages that include pretty typical spam are supplied in both
plain text and HTML forms (spam_text, spam_html).
- Spam may be sent in HTML format but as the text of a simple message. A
mail scanner should be able to handle this by looking for HTML in the text
part of the message. This test message (spam_text_html_paypal)
represents a typical phishing message from a well-known auction site that
should certainly be detected as spam. If your virus scanner is on the ball,
it may also detect the fake customer service (renewal) URL as a virus.
- As with virus writers, the spammers are constantly trying to
"improve" their delivery methods so that spam filters will not detect and
reject their messages. Encoding the message is one way of doing that.
Spam test cases in both encoded plain text and encoded HTML form are
provided (spam_text_encoded, spam_html_encoded).
- One powerful technique for spam filtering is the concept of the black
and whitelist, which involves comparing the sender's address against a list
of acceptable addresses. Spammers may try to bypass a scanner's list checks
by obscuring the address using quotes, comments, etc. This test case
(spam_text_addr_obscured) contains a spam, text message that will
probably pass through most spam scanners undetected. One would hope that the
black/whitelist catches the message but its "From" address has been obscured
to test the limits of your scanner's address parser.
- Some spammers may try to use long, encoded HTML to cause a mail
filter to give up or crash, thereby causing their message to be bypassed
and delivered to the recipient. This message simulates such a case,
where the HTML is spanned over many lines but, when decoded, turns into a
single, excessively long line (spam_html_long).
- The spammers are a bunch of guys who are really grasping at straws,
sometimes. A sample message is supplied that is multipart/alternative
with two components (text and HTML). Both are encoded Base64 to hide the
fact that they are spam. Anybody who encodes a text/plain segment is
obviously up to no good so it must be an act of desperation to make such
an obvious move (spam_mixed_encoded).
- With the advent of Bayesian classifiers that look for particular
words in messages to see if they contain spam, the spammers have taken to
obfuscating words by sending the message as HTML and inserting comments
into the middle of the words. HTML renders the text correctly but a
scanner may miss the key words. A good filter will remove such comments
as demonstrated by this test message (spam_html_masked_comment).
- Another technique used by spammers to mask words from Bayesian
classifiers is to represent the characters in the words as HTML escape
sequences. This message tests a filter's ability to convert HTML escapes
into regular characters so that the classifier can see the key words
- Since some email readers will interpret HTML in the body of the
message (even though its supposed to be text), this spam message uses
HTML in the body and obscures the content from the Bayesian classifier
by inserting comments into key words. This tests a filter's ability to
remove obscuring information from HTML in the message body
- HTML spam can appear as the only component of a MIME-encoded message,
as in the sample supplied. The spam filter should be able to handle this
- In a couple of variations on a theme, text and HTML spam can appear
as the only component of a MIME-encoded message but also be encoded Base64
so that the spam filter won't see that it is spam. The spam filter should
be able to detect the spam in both cases supplied
- This message contains some rather obvious spam of the penile
enhancement variety that is found in the HTML portion of a
multipart/alternative message. It is the only component, incidentally,
bypassing the rule about multipart/alternative messages containing at
least a text portion (spam_html_viagra).
- Whether through omission or comission, quite a bit of spam arrives
with broken MIME components. The most common error is the omission of the
last MIME segment or segments terminator for all nested MIME levels. Most
mail readers blithly interpret this faux pas so a good spam filter must
accept it too (spam_mime_malformed).
- Another common error is the omission of any valid MIME segments in a
message marked as MIME. Many mail readers will simply interpret this kind
of broken message as text, showing the recipient the Spam. This being the
case a good spam filter must accept such broken messages as text too. In
our example, the message appears to be valid but, in reality, the MIME
separators are broken and the message has no MIME entities at all
- Detection of spam is not always easy. The test suite includes a pair
of test cases that stretch the limits of traditional spam detectors. The
content marks them as spam but you never know (spam_text_notfilt1,
- If your mail filter employs a statistical check to find spam quickly
by counting the number of "bad" HTML tags in a message, this test should
exercise it with a multi-part message that contains many suspicious tags
in the HTML component (spam_stats_mixed).
The messages beginning with "virus_" are meant to test the
virus detection component of your mail filter.
- A very common email virus is one that includes a virus component as
an attachment and an automatic launcher, in the form of an HTML message,
that opens the virus. When your typical email package encounters it, it
interprets the HTML and launches the virus before the user even has a
chance to do anything about it. A sample of this kind of virus, using a
harmless virus component (in case it slips through), is provided
- Lately, we've been seeing a virus that looks like a mail system return
receipt for a message that was not delivered. I figure this is probably
some clever virus writer's attempt to sneak a virus by if the filters have
some special case checking in place for delivery-status messages and/or a
sender of postmaster. Either that or they are hoping to lull the user
into a false sense of security and convince them to open the message
because it is from a legit source. A sample virus that employs this
technique plus the usual HTML launcher is given
- This message is an example of a virus launcher in the HTML portion
of the message that is obscured by tricks with comments, etc. Perhaps not
all mail readers will be promiscuous enough to accept some of these tricks
but, if they do, it spells big trouble because the virus is launched when
the user opens the message (i.e. before they've had a chance to read it)
- Some mail readers may accept multiple HTML MIME components and
assemble them into a single message when displaying them. If this is the
case, a possible virus exploit would be to split a virus launcher across
two HTML components. A sample of this kind of virus launcher is given
- One method of delivering a virus launcher, that is applicable to
Outlook only, is to include HTML tags in the subject line of a message,
separated by a carriage return. Apparently, Outlook sees the carriage
return and starts interpreting the HTML in the subject as part of the
body. This works to the virus' advantage, since many virus filters don't
examine the subject for malicious HTML tags. Because it is perfectly
feasable to include a launcher in the subject that launches a viral
attachment, an example of this kind of virus is given
- Since spam is becoming ubiquitous, I guess some clever virus writer
figured to take advantage of this fact. I recently received a piece of
pornographic spam that featured an executable attachment that purported to
show pornographic photos. Click on the attachment and, "You've got
disease!", to paraphrase AOL. You've got to admit, its a brilliant
scheme. Masquerade as an obnoxious message to hide the fact that you're
really a malicious one. Also, legitimate spam filtering must look for
viruses as well so this tests that case too. You'll love this example
- To test whether your email filter or the virus scanner that it
invokes can find an actual virus, this test message includes the sample
virus distributed with Clam AV as several attachments. Your scanner
should find and delete both of the attachments to this message, or at the
very least render them harmless. (virus_clam_attach).
- Another variation on the theme of the HTML virus launcher plus
attached viral payload is the encoded launcher. Although nearly all
"text/html" MIME type entities are sent as "7Bit" or some other unecoded
form, this is not a requirement. Encoding the launcher as "Base64"
so that the virus filter will not see it is a pretty typical trick. This
message supplies an example of this kind of encoded virus, using the HTML
version of the Clam AV sample virus. Your scanner should remove the link
and/or delete the message. (virus_clam_encoded).
- To test whether your email filter can detect an actual virus, the
HTML version of the Clam AV sample virus is embedded in the text of this
message as a clickable link. Your scanner should remove the link and/or
delete the message. (virus_clam_html).
How It Works:
We provide you with the suite of test messages and a shell
script that can be used to submit some or all of the messages to your mail
delivery program (e.g. the "mail" command).
Once you have your filter working, send the messages from the
validation suite through your mail system and observe the results. Each
message includes text that describes the test it is meant to perform and the
results that should be expected. Anything that doesn't conform to the
expectd results should be investigated.
You can also use the validation suite as a regression test
after filter code changes have been made.
Specially crafted email: the validation suite uses
hand-made email messages that contain generic varieties of the virus and spam
delivery techniques most commonly used. The messages are designed to exercise
all paths through a typical email filter program.
Safe to use: denatured viruses that cannot hurt
your system, even if they get through, are used in the validation suite.
They are much safer than using real, live viruses (but, if you'd like to
live dangerously, go ahead -- you certainly can use live viruses to test
your filter, just don't complain when you find out that we told you so).