Getting SpamAssassin to work on a Windows 2003 Server with Exchange 2003

Our mailserver was acting up and in the hopes of calming it down, we upgraded to Exchange 2003 and Windows 2003 Server. We also upgraded the hardware that it resided on. We had a few issues with the upgrade and migration to new hardware - so it wasn't top priority for me to get the SpamAssassin installation working on it again.

But the amount of spam that we are getting is getting out of control and the RBL features in Exchange 2003 aren't enough (several of our clients were getting blocked and were angry about it, so we had to disable those anyway).
I also got a call from one of the high-ups today and he was rather insistent we get the spam filter back on - not sure if it was because of large amounts of spam (which he gets), or more because of this.

So today I am going to be going step-by-step through the install process and describing it here as follows...

Side note: I am using RDC to get to the machine so that I don't have to leave my desk. I am doing all of this as admin on this server with full rights.
Also note that I highly recommend doing this when your server volume is very low and users are not on the system (weekend? late at night?) since you will have to reboot.
I ran into minor problems (I have added bits of text in here so that you won't have to also have the same problems I did - mostly due to network issues and having to reboot - so nothing major) and it is easier to worry about just a few messages than it is to worry about missing the peak flow messages.

These are all following directions from a few sites. Largely Michael Bell's effort at Open Hand Home and Christopher Lewis's Exchange SpamAssassin Sink which is a far superior evolution of my own Exchange Sink (I won't bother linking to it since there is not much sense in it anymore).

Much of this text is similar or even the same in some cases as their descriptions - I am just trying to consolidate the process and leave any notes here that I may have.
If you would rather go directly through their processes separately, that works too.

Note that this type of installation works well on small business systems, but will likely have more problems under high load systems. Our network has ~25 user accounts and gets/sends about 1000 e-mails (2000 total) a day. That can vary widely, but this setup has worked well for us (or variations on it).
This configuration forces SpamAssassin to work in serial (one message at a time) and therefore if you have large/heavy loads, it feasibly might not be the best idea for your system. That said, there are arguments against having it spawn many processes to deal with lots of mail as well - depends on how you view that issue.
The e-mail message comes in, triggers the sink, gets run through Perl/SpamAssassin, and is dealt with... after all of that, then the next message is dealt with. All of this is only on incoming mail.

If you have a large organization with multiple domains and many users, I would question whether or not this is ideal.

Our particular server setup is Exchange 2003 on Windows Server 20003 on an Intel P4 3Ghz+ with HT, 2GB of RAM, and ~400GB of disk space (although we only really use/access about 80-120GB of that for mail use - the rest is backup and caching).
We also had great success with the same system (but with my old version of the sink) on a PIII 1Ghz with 512MB RAM and a 30GB disk running Windows 2000 and Exchange 2000.

Step 1: Install 5.6.1.x of Active State Perl. Downloading this will require you filling out your name, e-mail, and company name to get to the download screen (it is a 8.47MB file). I just pretty much hit Next/OK through all of the defaults in the installation (I did uncheck any IIS associations for Perl at this point since we don't do anything with our IIS beyond webmail and I don't want to have to worry about it).

Step 2: Install NMake for Windows v1.5. this just means you extract the file and then put both name.exe and nmake.err in "C:\Perl\bin" (or wherever you installed Perl). This is a very fast download.

Step 3: While those had been downloading, I also got Chris's Exchange Sink and Net::DNS .12 which were both fast downloads and we will need them later.

Step 4: I fired up a command window (Start->Run->"cmd") and then in there started PPM to get some new packages. (Out of curiosity I did a "search spam" while in there to see if there was a SpamAssassin package - there wasn't but there was a "spam cannibal" which after hitting Google leads to this - I will have to read up on that later) Now in PPM we type "search Time::HiRes". I always do a search first if I am reading someone else's instructions - that way in case they typed it wrong I can find what they likely meant. From there type "install Time-HiRes" - after that does its thing, then "install DB_File" again just letting it do its thing.

Step 5: Now we need to install Net::DNS, but apparently there is some issue with it from PPM (or there was at one point and I'm not adventurous enough now to be the one to test to see if the newer version has resolved the issue) - so we are going to install and use the Net::DNS v.12 that we downloaded earlier. Unzip the file and take note of where it put its contents. Again go to a command window (Start->Run->"cmd") and then navigate to wherever that file that you unzipped is (the uncompressed one - it will look like "Net-DNS-0.12") via the command window. Once in there, then type "Perl Makefile.pl" and that takes less than a second. Then type "nmake". That too blazes by. Then type "nmake install" and that rips by and is done in a blink. You can then delete that folder and move on.

Step 6: Download the newest SpamAssassin (I am currently in love with the bleeding edge 3.00cvs found at the bottom of the downloads page so that is what I will be using).

Step 7: Unzip the SpamAssassin zip into the C: drive (you can put it wherever you want, but you are going to have to keep track of that on your own). Mine now looks like "C:\Mail-SpamAssassin-3.000000".

Step 8: Open a command prompt (Start->Run->"cmd") and then navigate to your SpamAssassin directory (again, mine is "C:\Mail-SpamAssassin-3.000000") via the command prompt and then type "Perl makefile.pl" and then let it do its thing. (man am I glad that Win2k3 has tab completion now in the command window) Fill out the questions how they apply to you. The first one is usually the admin or head IT person at your place (if the head IT person leaves, then that isn't much help any more). Yes, run the network tests. Then we are done - but don't close that command window yet!

Step 9: Still in that command prompt we now type "nmake". Mine dies with "NMAKE : fatal error U1077: 'C:\Windows\system32\cmd.exe' : return code..." and then some more stuff. That seems to be right around the spamc stuff which I don't know that we need/want - so I am going to continue on and test SA and see if it is working - if not, then we need to either 1) try to get this to build and ignore the spamc, or 2) download 2.6x and get that to work since we know that is good for this (although not as great as 3.00 is for beating spam from my own observations). I tried "nmake install" and it still complains about fatal errors relating to spamc - again, I will still try tests to see if it is totally dead, or just that part.
Looking in C:\Perl\bin - there is no "spam..." anything, so I am starting to get the feeling this failed - which means I then need to try to get this to not bother with spamc (since that doesn't work on Windows anyway).

**remember that spamc doesn't work under the Win32 install of spamassassin - so the issues we are seeing here are likely related to that.

Hmm, I went into the Makefile in the base spamassassin directory and started deleting references to the spamc install and eventually deleted a whole chunk of code. After doing that, I could finally nmake and make again.
I am going to try to do a diff on the original makefile and my new one to see which lines it was that I changed (not sure all changes I did were needed).
Will get to that later though - first I want to make sure this works....

(To make these edits you might want a text editor that can show line numbers easily, such as TextPad)

Line 308 that starts with "CONFIGURE" and ends with "spamc/configure.pl" should be commented out (add with no quotes a "#" to the very start of that line).

On line 544 you will want to remove the text that says "spamc/spamc$(EXE_EXT)"

On line 549 you will want to remove the text that says "$(INST_SCRIPT)\spamc$(EXE_EXT)"

On line 553 you want to remove the text that says "$(INST_SCRIPT)\spamc$(EXE_EXT)"

And then finally you will want to comment out (by putting, without quotes, a "#" at the start of each of the lines) 561-565 - that will prevent that from running.

After that, it will then work (so type "nmake" and when that is successful, then "nmake install").

Step 10: Change C:\Perl\bin\spamassassin.bat so that it is no longer read-only (right click, properties, uncheck read only). Then open that and on the first line after "@ECHO OFF" add (no quotes) "SET RES_NAMESERVERS=ipaddress" where ipaddress is the ip of your DNS server (this is needed for RBL checks which you may have turned off) - to add multiple IP addresses (which is what I have to do) - just put a space between them on that line.
For example: "SET RES_NAMESERVERS=192.168.0.10 192.168.0.11"
Those aren't really my DNS servers, but you get the point.

Then do the same sort of change to C:\Perl\bin\sa-learn.bat (don't forget to take off the read only).

Step 11: Open C:\Perl\share\spamassassin\local.cf and add the lines (if you don't have one, then create a file with that name at that location and put these in there):

use_razor2 0
use_dcc 0
use_pyzor 0

These are placed there to make sure that these aren't used under Windows - they break and cause issues if they are attempted to be used.

I am skipping the spamassassin documentation since it is all online and I want to get this up faster. There is a lot to go through that I encourage you to explore on your own - it is beyond the scope of this particular walkthrough.

Step 12: Now we are going to test the test spam in that spamassassin directory. In your command window (you didn't close that did you?)
We want to type "spamassassin -D < sample-spam.txt" - that has the debug flag there, so a TON of stuff will scroll by - but if it all looks good and it marks it as spam, then you are good to go - which we are! (*whew*)

Just to make sure - create a test directory at C:\test and in there put the "sample-spam.txt" and "sample-nonspam.txt" files in there.
Then we run (from the command line in the C:\test directory) "spamassassin -D -t < sample-nonspam.txt > nospam.txt" and then also "spamassassin -D -t < sample-spam.txt > spam.txt".
Then look in the spam/nospam text files and see that everything is okay - and it is.

Step 12: I had an existing "local.cf" file in my C:\Perl\share\spamassassin folder on our old mailserver that has a lot of our custom rules and whitelist data in it, so I copied that over to the new mail server in the same path.
If you are doing a new install with no custom local.cf - that file doesn't seem to be in the install by default - so I am curious if they have taken it out of the 3.00 install - will find out soon enough I suppose (if that is the case, then it would simply be a matter of moving the data from local.cf over to whatever the current global preferences file is).

Step 13: We are now pretty much done with SpamAssassin and now want to install the Exchange event sink from Chris. Event sinks are trigger based code that will get run on various events and are a feature of Exchange 2000 and 2003. Essentially what is going to happen is that for each mail coming in, that will trigger a "I'm here!" event to the server which will then run any code associated with that - and our SpamAssassin plugin is going to be that code.

First up we want to reboot our machine - this will setup the system path that we established before - if you look you will see it is there, but in order for all programs/services to be aware of it, you will need to reboot.

Once the server has rebooted, double-click on the "ESA-setup..." file that we downloaded earlier from Chris's site.

The installer will open up a config file and we can then edit that. It defaults to pointing to C:\perl\bin\spamassassin.bat and that is what I have, so I am good to go - if I had installed that elsewhere, then I would need to change that to the appropriate place.

I am not adding any extra flags to spamassassin since it works fine for me the way it is.

I want to debug during the initial testing process to see that this is working. After, when it is all okay, I am not debugging, so then turn that to false.

The "ReallyBadSpam" feature is great - it basically is a value that you can set where if the spam is really blatant, then it doesn't even bother sending that to the client folder in Exchange, it just let's it go and die. This is excellent - especially good if you Exchange server is outside of your main office network connection... unfortunately not the case for where I am at now - so we still have to use up our bandwidth to get that (but it is still a great feature to limit what my end users have to see).
I left that as a default of 25 - if I get any complaints I will adjust it one way or the other. In my own experiences on Unix systems that I have, SpamAssassin 3.0 seems to give slightly higher values to things, so it is feasible that this number is just about right for now.

I am leaving the save spam feature on and that will save out the ham/spam (good/bad mail) and I can then later go through and sort it out as to being right/wrong and then run sa-learn on it. That will make the Bayes system start to work (it needs to learn some number of messages before it is good enough and therefore is turned off).
Once I feel that it has learned enough, then I can go in and edit this to FALSE for saving the mail out so as to limit the storage and extra processing.

I made the "ClearReturnReceipt" false because I don't want this to interfere with anything my users and their clients do and then have them come complaining to me.

**note that depending on where these are saved out and the server access privileges that it is feasible that someone could see someone else's mail - also, when you review these for ham/spam (ham meaning "not spam") you are going to see people's mail - if you are an admin, that shouldn't be that big a deal anyway, but it is worth noting.

I am all done with that, so I then did a File->Save on that INI file that was opened up and then clicked "Next" in the install process.

On the last screen of the install I left the "Run Install_ESA_Sink.bat" checked and then clicked "Finish".

It popped up a command window, did its thing, and then went away.

Step 14: So now we need to see if it is working. The easiest way is to go to C:\ESA\Ham (or spam) and then watch for incoming mail. I went to an outside account and sent mail to my account here just to make sure something would be coming in (mainly necessary since I tested this outside of normal hours - otherwise there would be plenty of traffic to generate it).

Sure enough, it shows up in the Logs and the "Ham" files. I wait a bit and see that spam is showing up in the "Spam" directory too. All is working well.

I can now go into my Outlook and open the message and look at its headers to see if SpamAssassin is looking like it is behaving. Sure enough, I can open the message and then go to View->Options and in there see the headers and with that the SpamAssassin additions.

All is well.

**If there is something wrong and you want to turn the sink off, then run C:\ESA\Remove_ESA_Sink.bat assuming that is where you installed it. To install it again, then of course just run C:\ESA\Install_ESA_Sink.bat

Step 15: Then just send out a note to users on how to configure rules to look for the proper thing in the headers to pull out spam and put it into another folder, or mark it, or delete it - whatever you choose to do.

Final Notes

Once you get it confirmed to be working, it is good to turn the debugging to "false" since you don't need the complicated log entries.

You will want to train the SA database to learn on the ham/spam directories. This will require you to manually sift through those files (I use TextPad) and look at the content and then move them over to other folders if they don't belong there.
After about 2000-3000 messages of training, you will have done plenty and then you can turn the setting in the INI file so that it doesn't save out the messages.
("sa-learn --spam C:\ESA\spam" and "sa-learn --ham C"\ESA\ham" are the ways you teach it to learn. Just to save space you will probably want to delete the ".out" mail files after you have had sa-learn run on them. Technically it keeps track of messages that it has learned on so that it won't count them twice, so if you never empty the directory - which I wouldn't recommend - it will skip the messages it has seen before... so they say.)

On other Exchange sinks I have noticed that there are rare occasions (and I have no clue what causes these occasions) that Event Sinks will just stop working without any warning.
You will then need to "reinstall" the sink by running C:\ESA\Install_ESA_Sink.bat (and any other sinks that you had running). In the year that I have been using Event Sinks in Exchange, I have only had to do this - reinstalling the sinks - once (under Exchange 2000).

Hopefully that is enough to get you on your way. There are obviously other programs you can use, and other implementations (a physical hardware gateway that perhaps runs another OS and filters the mail, or a commercial program, or some combination), but this walkthrough was really only intended for this specific case using available free resources and some time.

Feel free to discuss any of the above, and or to ask questions about any of it.

Posted by Eric at May 4, 2004 09:42 AM | TrackBack