Threading Hell

So, when I designed Caterpillar
2.0 I wrote in a fairly simple threading model. Didn't need anything too
fancy as things tended to all finish fairly quickly.

Then Caterpillar 3.0 came along... which I'm still holding off on
releasing because I have yet to go a week without tweaking something.
I'm trying really hard not to. It was bug free.. well, all my
code was bug free, there are still some minor bugs in the WYSIWYG editor
that most people would probably never see. Anyway, I'm not really bug
fixing so much as doing usability testing and fixing those issues. A big
one came up a couple days ago. New feeds were just taking way to long to
add. At least a minute, during which time the dialog was still up and
you couldn't do anything else. Your average user wouldn't have hit that
issue for a while but I'm subscribed to over 70 feeds, and right now
there are 4306 undeleted entries in my database and another 1000 still
hanging out for comparison against new entries to prevent duplicates
(they'll eventually go away on their own). There's another 15000+ rows
in one of the other db tables and 1.8 Mb of search indexes. Each new
entry gets shoved into or compared against all that, which,
unsurprisingly takes a while.

So, I had to move some of the processing to background threads so that
people could get on with the business of reading their feeds.
Unfortunately I hadn't originally designed Caterpillar to be so
exceptionally multithreaded. So all these things that used to happen
sequentially were all happening at once. One process would be updating a
Vector while another was trying to iterate through it. Things were
getting saved in the wrong order. Chaos! Chaos I tell you!

So, now, two days later I am happy to report that order has been
restored. Feeds add themselves in only slightly longer than it takes to
download the feed, and entries show up as they finish processing. I
leave the adding dialog up until the feed is finished being downloaded
in case there are any problems and it needs to let you know about
anything.

While I was at it I added in some functionality. I know I know... can't
leave well enough alone. Anyway, in the off chance that something
happens to your search indexes you can now rebuild them. It now has a
little number telling you how many entries you have or how many search
results there were. I think there was something else but I forget it
now...

Damn, I just found another bug. Grr. Maybe I can fix that one tonight...
shouldn't be too bad.

My goal in all this is to have Caterpillar 3.0 be an absolutely kick ass
steady product that any non-geek can easily use and count on. A

Oh, and Caterpillar 3.0 will be shareware which is part of the reason
I'm working so hard on it for you all. It's not right to expect people
to pay for half done buggy software.