FutureQuest.net Services Support Our Data Center Community About Contact
Support Center » Knowledgebase » Custom Filters: Advanced Email Scripting
 Custom Filters: Advanced Email Scripting
Article This tutorial on Advanced Custom Filters covers filtering topics for Site Owners who want to highly customize their email filtering by running their own scripts and programs on incoming email. You may also want to review some of FutureQuest's related tutorials on email filtering, including the use of FutureQuest's easy-to-use Built-In filters, by visiting the main Email Filter tutorial.

Notice: This tutorial presents advanced topics. Prerequisite knowledge of server configuration, how to work with scripts, and file permissions is assumed. The examples presented here are presented with NO WARRANTY and without direct support. USE AT YOUR OWN RISK. For further assistance you may visit the Community Forums.

With a custom filter, a program can be invoked every time an email is received. Any program or script that can be run from the command line, and can read from Standard Input (STDIN), can be used as a filter program. The raw email is made available to the program on Standard Input. Any error code returned by the script determines whether a message will be delivered or not. If the program is set up as a Simple Filter, anything printed to Standard Output (STDOUT) is considered an error message. If the program is set up as a Processor filter, then the processor will attempt to deliver whatever is printed to Standard Output as the email message. Below we provide some documentation, links and simple examples to help get you started.

Environment Variables

Qmail does make certain environment variables available to email scripts. You may wish to consult the Qmail documentation on this topic. The environment variables available to scripts are explained in the documentation for the qmail-command man page. Exit codes that may be important for your scripts are also explained on the same page.

Simple Filter

A Simple Filter reads the incoming email message from Standard Input. Anything optionally printed to Standard Output is considered the text of an error message. The fate of the message, as to whether it will be delivered normally, bounced, or blackholed (deleted) is governed by the exit code of the script. The basic exit codes are as follows:
  • 0 - Process the message normally.
  • 99 - Do not process further delivery instructions.
  • 100 - Bounce the email.
  • 111 - Temporary, non-fatal error. Message delivery will be re-attempted later.
More details on exit codes are available at the qmail-command man page.

Processor

A Processor also reads the incoming email message from Standard Input. In contrast to the Simple Filters, however, anything printed to Standard Output is considered the new email to be delivered, replacing the original, incoming email. Processor scripts allow you to modify or change the email in some way, such as inserting new header fields, adding message footers, removing attachments, and the like. (Note, however, the destination address cannot be modified even if the headers of the message are changed by the processor.) The same exit codes apply as above. Therefore, if you run a Processor script that exits 99, no email will be delivered, regardless whether any output was printed or not. On the other hand, if the Processor script exits 0, Qmail will deliver a message. If there has been no output from the script, then it will deliver a blank message.

It is best if the script checks to make sure the email that it outputs is properly formatted, so that no errors occur when Qmail tries to deliver the message. It is up to the author of the script to do this, as Qmail will not. For further information on proper email formatting, please consult the appropriate RFCs.

General Installation Instructions

To install a script for a particular email account you will need to visit the Email Manager section of your CNC. Note that email filters run under the same group ID (GID) as the xdomain user, which is in the same group as your shell login userID. Therefore, you will need to place the scripts in a directory which allows group permissions to read and execute the script. Usually the /big/dom/xdomain directory is a safe place. Do not place email scripts within your $HOME directory (/big/dom/xdomain/username), as the mail system does not have sufficient permissions to access files in that directory, and such scripts would be unable to execute.

As an example, suppose you wished to install a script file /big/dom/xdomain/myscript.pl for the email address user@domain.tld. Then, in your Email Manager you would click on the account name for the email address user. This will take you to the Mailbox Properties screen for that account. On the row for "Custom Filters", click on the "edit" link to the right. This takes you to the Custom Filters page for that email account. It may be useful to review the "Helpful Hints" at the bottom of that page.

To set up the filter, you will need to select either the Simple Filter or Processor option from the drop-down menu. Which of these you should choose depends on how your script works, as explained above. Then in the adjacent text box you need to enter the command which invokes your script. If the file has been made executable (i.e. has 755 permissions) with the correct interpreter path at the top of the file, then in the box you would simply enter /big/dom/xdomain/myscript.pl (changing the xdomain and script name accordingly) and save the filter settings.

Make sure that if you upload the script to your web space, that you upload in ASCII (text) mode, and set the permissions on the file to 755.

Testing Your Script

FutureQuest® always recommends thoroughly testing your scripts before deploying them on a production server, such as one of the FutureQuest® Community Servers. If a faulty script causes email on one of your accounts to be lost, there is no way for FutureQuest® to recover the lost email. Moreover, if a faulty script were to cause delays and other problems with the mail delivery system, this would be a violation of FutureQuest's Terms of Service.

Recommended practice is to:
  1. Test the script at the command line on your own computer.
  2. Upload the script to the FutureQuest® servers and test it at the command line.
  3. Install the script on a test email account and test there.
  4. If your script comes through a thorough battery of tests in each of the preceding steps, then you may feel comfortable with installing it on a live email account.
For testing on your own computer, you will need to install Perl, Python, C, PHP or whatever interpreter or compiler is needed to run your script. For Linux, Windows and MacOS X, you can test your scripts at the command line using input redirection. To do this, save a test email in a text file. For example, you could save the following in the file message.txt: 

Date: 22 Oct 2002 05:10:39 -0000
Message-ID: <20021022175100@example.com>
From: user@example.com

Subject: Sales question       
This is the message body.
This is the second line.
This is the last line.


It is important to have at least one blank line separating the email headers from the message body. For further email specifications, please refer to the email RFCs.

Then, at a command prompt, enter a command such as:

$ perl /big/dom/xdomain/myscript.pl < message.txt

What this does, is send the contents of the file message.txt to the script on Standard Input, which is the same way the script receives an actual incoming email when running as an email filter in the mail delivery system. Please note that if your script uses environment variables, you may need to set them before running your script at the command line, or have your script handle missing environment variables appropriately. On a Linux server, such as the FutureQuest® Community Servers, you can set your environment variables for testing purposes via the env command. For more details consult the man page for env.

For command line testing on a Windows system, use a DOS Window or Command Prompt, which will look something like this:

C:\> perl C:\scriptdir\myscript.pl < message.txt

For Macintosh Classic OS, Perl and Python offer a simulated command prompt environment for testing programs. Another option is the Macintosh Programmer's Workshop tool from Apple.

After your script is performing successfully at a command line prompt on your computer and also at the command line prompt on a FutureQuest® server, you should install on a test email account as described in the previous section, General Installation Instructions. Send emails to the test account to test the script. If the script is not performing as desired and you are having trouble diagnosing the problem, you may wish to do either of the following:

  1. Have your script write debugging statements to a log file.
  2. Review the error messages in the file /big/dom/xdomain/logs_email/filter.log
When you are satisfied that the script is working properly on the test account, you may consider it safe to try on a "real" email account that is receiving email that is important to you. Always be careful, however. The assurance that your script is safe to use on a real email account is only as good as the thoroughness of the testing process that you put it through. In practice, it sometimes happens that script authors overlook certain conditions that should have been tested, but in actual use they rear their ugly heads! Unfortunately, such oversights can not only lead to lost email, but also to excessive resource usage and abuse, which can ultimately lead to the consequences mentioned in the Terms of Service.

In addition to making sure your script functions correctly, also make sure to put sensible limits in place to guard against excessive resource usage and abuse. For example, make sure to close opened files and sockets as soon as possible. Set reasonable timeout limits on sockets. Note that email scripts have a maximum of 15 seconds of allowed execution time. If your script does not finish execution within that allowed time limit, the script will be killed and email will remain in the queue for attempted re-delivery at a later time. If the script is unable to finish execution within the allowed time limit, then after many delivery retries, the system will bounce the email back to the sender as undeliverable.

Combining Commands

As discussed above, for a script file with executable permissions, simply entering the full path to a script into the Custom Filter editing page will invoke the script when incoming email is received.

It is possible, though, to form more complicated commands for entering into the CNC Custom Filter page. Commands can be combined, just as one would do at the command line, using pipes and the other usual conventions of the bash shell, such as using the output of one command as an argument of another.

Some examples (note that each of these would be entered as a single line in the CNC):

bouncesaying "An Error Message" /big/dom/xdomain/myscript.pl

/big/dom/xdomain/myscript.pl | othercommand

if [ <some test condition> ]; then /big/dom/xdomain/myscript.pl; exit 99; fi

You can see additional examples that use pipes, and a number of available programs such as bouncesaying, condredirect and others, in the Recipe CookBook.

Another useful command in some situations is the Qmail preline command which inserts the standard UUCP From_ line and then feeds the message to a program. You may need to use preline for any programs that may require this From_ line be present in order to work correctly.

Order of Filter Execution

Please note that Custom Filters are executed after SpamAssassin and Built-in Filters, if such filters are also enabled on the same account. For more details about the order of filter execution, please see the on Filter Order.

Example Scripts

Here we present some scripts, to provide some concrete examples of the information presented above. These are all rather simple scripts, for the purposes of illustration and learning only. They do little, if any, error checking or error trapping, and pretty much assume that everything will work without problems.

addmailtag.pl - Processor - Perl Script
This script appends a "footer" or "tag" to each incoming email. It is a Processor script, as it reads the incoming email from Standard Input, and then prints out a slightly different email to Standard Output. In this example, the difference is that it appends a small footer to the message from a text file. The modified message, as printed to Standard Output, is the message that is delivered to the mailbox. 

#!/usr/local/bin/perl
# name of file where text for footer
# tag is stored
$filename = "tag.txt";
# read the original email message from 
# STDIN one line at a time and
# print each line to STDOUT
while (<STDIN>) { print; }
# open the file with the contents
# for the tag
open(TAGFILE, "<$filename");
# read in the contents of the tag
# file a line at a time and
# print each line to STDOUT
while (<TAGFILE>) { print; }
close(TAGFILE);

If the file tag.txt is in the same directory as the addmailtag.pl file, and the contents of tag.txt are: 


-- 
Here is my footer tag.
Ta-da!

Then, this small footer would be appended to each incoming email before it is delivered to its destination mailbox.

To install this script, you could save it in the directory /big/dom/xdomain as addmailtag.pl and also save the file tag.txt in the same directory. Make sure that permissions on addmailtag.pl are set to 755. Then, in the CNC's custom filter page, select "Processor" for the type of filter and enter /big/dom/xdomain/addmailtag.pl as the filter command.


mailtofile.py - Simple Filter - Python Script
This script is a simple filter. It does not change the email in any way, and the email is delivered as usual. What this script does, is read the incoming email on Standard Input, and append it to the file ./outmessage.txt. 

#!/usr/bin/python

import sys, os

# open the file to which data will be written
outfile = open("./outmessage.txt", "a")

while 1:
    # read in the message 1000 bytes at a time
    message_data = sys.stdin.read(1000)

    # if no data to read, exit loop
    if not message_data:
        break

    # if data, write it to the file
    outfile.write(message_data)
outfile.close()


The remaining scripts are linked in text files, due to formatting issues. You can view them by clicking on the link, or right click to save them to a file.

mailtest.pl - Simple Filter - Perl Script
This Perl script is a Simple Filter, that reads the incoming email from standard input, and then prints a number of the environment variables, plus the original message, to a file called proc.test. Running this script and reviewing the contents of the proc.test file will give insight into regarding permissions, user and group IDs and environment variables on the FutureQuest® mail system. Note that this script overwrites the file proc.test each time it is executed.

To view or save the script please click on the link below:
mailtest.pl.txt

addheader.py - Processor - Python Script
This Processor script adds a custom header to incoming email before it is delivered. The incoming message is read from Standard Input. Then it is modified, and the new message is printed to Standard Output.

To view or save the script please click on the link below:
addheader.py.txt

date_check.py - Simple Filter - Python Script
Checks whether incoming email contains a date that is older than a year, or more than two days in the future. If so, it deletes the email. If the email has no Date header, it is also deleted. The idea is that email with dates that are very old, in the future, or simply not there, are usually spam. Deleting the message is accomplished by returning a 99 exit code.

To view or save the script please click on the link below:
date_check.py.txt

Article Details
Article ID: 98

 Back
 Log in [Lost Password] 
E-mail:
Password:
Remember Me:
 
 Search
 Article Options
 Print Article
 PDF Version
 E-mail Article
 Add to Favorites
Home | Register | Submit a Ticket | Knowledgebase | News | Downloads
Language:

Help Desk Software by Kayako eSupport v3.70.01