Steven Engelhardt

Today I ran across an invaluable resource for Windows Forms development: George Shepherd’s Windows Forms FAQ. The DataGrid section is especially relevant to my recent work.

While the development SQL server is being repaired, I wanted to discuss a class I wrote to assist with writing threaded code in .NET: EventGeneratingThread.

At work I had to create a Windows Forms application which, upon loading, populates a DataGrid using data from a SQL server. Rather than load the data synchronously during the Form.Load event, which would delay the display of the form, I decided to load the data asynchronously using a background thread. However, as I soon found out, if a child thread’s ThreadStart method fails by throwing an Exception, the Exception will be silently lost. This problem manifested itself when the development SQL server recently crashed but no UI error message was displayed.

The simplest solution is to modify your ThreadStart method to catch exceptions and act accordingly:

Thread t = new Thread(new ThreadStart(Process));
t.Start();

void Process()
{
    try
    {
        ReallyProcess();
    }
    catch (Exception ex)
    {
        ProcessFailed(ex);
    }
}

void ReallyProcess()
{
    // Do the actual processing work
}

void ProcessFailed(Exception ex)
{
    // Handle a processing failure (display an error dialog box?)
}

The above pattern would then be repeated every time a child thread is created. Therefore, I decided to separate this common code into a class called EventGeneratingThread. EventGeneratingThread strives to duplicate the interface of Thread (unfortunately, it can’t inherit from Thread because Thread is sealed) and adds events to signal whether the provided ThreadStart method completed or generated an exception. In addition to this, I decided to support the ability for a user of EventGeneratingThread to provide an object that implements ISynchronizeInvoke (such as a System.Windows.Forms.Form object), on which the events will be subsequently fired.

Internally, EventGeneratingThread works very similarly to the above code. It takes a ThreadStart method as a parameter, but rather than provide that to the Thread object it creates, it stores it as a member variable and provides its own method, InternalStart, to the created Thread object. InternalStart wraps the call to the stored ThreadStart method in a try/catch block and fires the appropriate event.

While the full Thread interface isn’t implemented yet, here’s EventGeneratingThread as it currently stands:

/// <summary>
/// A thread which generates events which correspond to important
/// events.
/// </summary>
public sealed class EventGeneratingThread
{
    private Thread m_t;
    private ThreadStart m_realStart;
    private ISynchronizeInvoke m_syncInvoke = null;

    public EventGeneratingThread(ThreadStart start)
    {
        m_t = new Thread(new ThreadStart(InternalStart));
        m_realStart = start;
    }

    public event EventHandler Completed;
    public event ThreadExceptionEventHandler ThreadException;

    public string Name
    {
        set { m_t.Name = value; }
    }

    public ThreadPriority Priority
    {
        set { m_t.Priority = value; }
    }

    public ISynchronizeInvoke SynchronizeInvoke
    {
        set { m_syncInvoke = value; }
    }

    public void Start()
    {
        m_t.Start();
    }

    private void InternalStart()
    {
        try
        {
            m_realStart();
            FireCompleted();
        }
        catch (Exception ex)
        {
            FireThreadException(ex);
        }
    }

    private void FireCompleted()
    {
        Fire(Completed, this, new EventArgs());
    }

    private void FireThreadException(Exception ex)
    {
        Fire(ThreadException, this, new ThreadExceptionEventArgs(ex));
    }

    private void Fire
        (
        Delegate dlg,
        params object[] args
        )
    {
        if (dlg != null)
        {
            if (m_syncInvoke != null)
            {
                m_syncInvoke.Invoke(dlg, args);
            }
            else
            {
                dlg.DynamicInvoke(args);
            }
        }
    }
}

And here’s an example of its use:

private void MainForm_Load(object sender, System.EventArgs e)
{
    this.Cursor = Cursors.WaitCursor;

    EventGeneratingThread t = new EventGeneratingThread(new ThreadStart(FillDataSetInBackground));
    t.SynchronizeInvoke = this;
    t.Completed += new EventHandler(FillDataSetCompleted);
    t.ThreadException += new ThreadExceptionEventHandler(FillDataSetFailed);
    t.Name = "Populate data grid";
    t.Priority = ThreadPriority.BelowNormal;
    t.Start();
}

private void FillDataSetInBackground()
{
    dsProductList = ... // Fill data set
}

private void FillDataSetCompleted(object sender, EventArgs e)
{
    Debug.Assert(!this.InvokeRequired);

    dgProductList.DataSource = ... // Set the appropriate data source
    ResetCursor();
}

private void FillDataSetFailed(object sender, ThreadExceptionEventArgs t)
{
    Debug.Assert(!this.InvokeRequired);

    ResetCursor();

    string errorMessage = string.Format("Error: Loading data failed.\\n\\nError message:\\n{0}\\n\\nStack trace:\\n{1}", t.Exception.Message, t.Exception.StackTrace);
    MessageBox.Show(this, errorMessage, "Error: Loading Failed", MessageBoxButtons.OK, MessageBoxIcon.Error);
    Application.Exit();
}

URL “purity” is important to me, so I wanted to note the following evolt.org article: Making “clean” URLs with Apache and PHP.

To quickly summarize, my email setup is a combination of fetchmail, procmail, postfix, bogofilter, and mutt (with my libESMTP patch) using mbox files on a colocated Sun Fire V100 running OpenBSD/sparc64. For the most part I very much like this setup. In the past I have tinkered with SpamAssassin, maildir, and IMAP; I may end up setting up IMAP again in the future. BTW, I have an old entry describing how to properly integrate bogofilter and mutt.

The purpose of this post is to describe how I archive a copy of every piece of incoming email using these tools. I find these archives extremely valuable, whether for the occasional historical search or in that it enables me to be highly aggressive in keeping my mailboxes clean by deleting messages. Recently I have been contemplating performing statistical analysis of received spam based on these archives.

The first step to take when setting up a historical email archive is to consider how the email will be stored. For my regular email, I use the system-default mailbox for my inbox (/var/mail/sengelha) and store all my sorted email in mbox files in ~/.mail/mailboxes/. For backups, I keep mail in monthly archives in the mbox file ~/.mail/backups/YYYY/YYYY-MM, where YYYY is the current year and MM is the two-digit month (01 = January and so forth). For all past months I compress the mail using bzip2 -9 to save hard drive space. I used to keep daily archives but per the suggestion of Dan Sachs I recently moved to monthly to achieve better compression ratios. To give you an idea of the sizes that are involved, my October 2004 email archive is 21,947,942 bytes (7,734,053 bytes compressed). Undoubtedly much of this is spam.

My setup has two major parts:

1. Save a copy of every incoming email to the ~/.mail/backups/YYYY/YYYY-MM file.
2. Compress all past monthly archives using bzip2 -9.

To save a copy, I must first ensure that the directory ~/.mail/backups/YYYY/ already exists. In fact, I lost some email in January one year because the directory didn’t exist so the email backups were never saved. I have the following rule to my .procmailrc to ensure the directory always exists:

:0wc
* ! ? test -d $HOME/.mail/backups/`date +%Y`
| mkdir -p $HOME/.mail/backups/`date +%Y`

I then use the following rule to save a copy of every incoming email:

# Make backups of all mail received in format YYYY/YYYY-MM
:0c
$HOME/.mail/backups/`date +%Y`/`date +%Y-%m`

I wrote the scripts bz2compressdir and compress-mail-backups (see below) to compress all past monthly email archives. I execute compress-mail-backups nightly using cron. (Monthly would suffice but there is effectively no penalty for running compress-mail-backups too frequently.)

bz2compressdir:

#!/usr/bin/perl -w
#
# bz2compressdir: Compress all files in a directory with bzip2
#
# $Id: bz2compressdir,v 1.2 2004/02/19 05:42:35 sengelha Exp $

use strict;
use File::Find;
use Getopt::Std;

my $usage = <<EOF;
Usage: $0 [options] dir1 [dir2 dir3 ...]

-e <regexp>  exclude <regexp> from files to compress
-h           print usage and exit
EOF
my %opts = ();

sub handleFile {
    if (-f && !/.bz2$/ && (!$opts{'e'} || !/$opts{'e'}/)) {
        `bzip2 -9 $_`;
    }
}

getopts('he:', %opts) or die $usage;

die $usage if ($opts{'h'});
die $usage if ($#ARGV == -1);

foreach my $dirname (@ARGV) {
    die "$dirname: Not a directory" if (! -d $dirname);

    find(&handleFile, $dirname);
}

compress-mail-backups:

#!/bin/sh
#
# compress-mail-backups: Go through every mail backup directory,
# compressing with bz2compressdir
#
# $Id: compress-mail-backups,v 1.2 2004/02/19 05:42:35 sengelha Exp $

exec nice bz2compressdir -e `date +%Y-%m` $HOME/.mail/backups/200?