New Caterpillar Manual on the way.

I've been burning the midnight oil getting Caterpillar 3.0's documentation ready. I'm up to 50 pages now thanks in no small part to VodooPad. I'll need to run a custom perl script on the HTML it produces to improve some of the linking but even with having to write that it's saved me hours, and hours. This is a great app.

Who would have thought there could be 50 pages of documentation? Given, many of the pages are only a few paragraphs long but still. I've been working hard to make it easy to navigate and find the information you want, and not have to deal with anything else. Kind of like Caterpillar itself.

Eating my own dog food

The ultimate test of any application is when the author(s) put it all on the line and start using their own stuff. Well, I've been using Caterpillar since the start, but there was this one new feature that I was hesitant to use.. Import.

You see, to really test import I had to export what I had, delete everything, and import what I exported, then hope against hope that it was all still there because I didn't want to loose some of those entries I'd saved. But I made a promise when I released Caterpillar 2.0. I said, "You can save any entry permanently." And I don't think it really counts if to have permanent you have to never upgrade. So, I backed it all up, exported, wiped out the db and imported it all again, only to find an exact duplicate of what I exported. Yaaay.

Export and Import are now ready to go for Caterpillar 3.0. Upgrading is pretty painless, although if you have 120 feeds and 4500+ entries like I did, you may have to wait a few minutes for everything to get reindexed.

There are only a couple minor bugs remaining (many many squished bugs and improvements) and absolutely no more new features to add in before the release. Sorry it's been taking so long folks.

Threading Hell

So, when I designed Caterpillar
2.0 I wrote in a fairly simple threading model. Didn't need anything too
fancy as things tended to all finish fairly quickly.

Then Caterpillar 3.0 came along... which I'm still holding off on
releasing because I have yet to go a week without tweaking something.
I'm trying really hard not to. It was bug free.. well, all my
code was bug free, there are still some minor bugs in the WYSIWYG editor
that most people would probably never see. Anyway, I'm not really bug
fixing so much as doing usability testing and fixing those issues. A big
one came up a couple days ago. New feeds were just taking way to long to
add. At least a minute, during which time the dialog was still up and
you couldn't do anything else. Your average user wouldn't have hit that
issue for a while but I'm subscribed to over 70 feeds, and right now
there are 4306 undeleted entries in my database and another 1000 still
hanging out for comparison against new entries to prevent duplicates
(they'll eventually go away on their own). There's another 15000+ rows
in one of the other db tables and 1.8 Mb of search indexes. Each new
entry gets shoved into or compared against all that, which,
unsurprisingly takes a while.

So, I had to move some of the processing to background threads so that
people could get on with the business of reading their feeds.
Unfortunately I hadn't originally designed Caterpillar to be so
exceptionally multithreaded. So all these things that used to happen
sequentially were all happening at once. One process would be updating a
Vector while another was trying to iterate through it. Things were
getting saved in the wrong order. Chaos! Chaos I tell you!

So, now, two days later I am happy to report that order has been
restored. Feeds add themselves in only slightly longer than it takes to
download the feed, and entries show up as they finish processing. I
leave the adding dialog up until the feed is finished being downloaded
in case there are any problems and it needs to let you know about
anything.

While I was at it I added in some functionality. I know I know... can't
leave well enough alone. Anyway, in the off chance that something
happens to your search indexes you can now rebuild them. It now has a
little number telling you how many entries you have or how many search
results there were. I think there was something else but I forget it
now...

Damn, I just found another bug. Grr. Maybe I can fix that one tonight...
shouldn't be too bad.

My goal in all this is to have Caterpillar 3.0 be an absolutely kick ass
steady product that any non-geek can easily use and count on. A

Oh, and Caterpillar 3.0 will be shareware which is part of the reason
I'm working so hard on it for you all. It's not right to expect people
to pay for half done buggy software.

Caterpillar Status Report

Still just trying to let Caterpillar 3.0 bake before release. The blogging features are working well ...everything's working well but a few small bugs have cropped, and been quickly squashed, by letting it just run 24/7. I just need more interesting feeds to subscribe to.. Know of any?

On a related note. I'm doing all my blogging from Caterpillar now. It's just that much easier to use it than a web page and I always have it running. I need to add in multiple blog support. I've got the structure in place to support it I just haven't implemented it yet. I think I am going to hold off and get 3.0 out the door then just make 3.1 support multiple blogs. I figure most people only have one so it's not that big of a deal.

Caterpillar gets Conditional Gets

Caterpillar Now supports HTTP Conditional GET. This means less load on the servers you download from. Faster feed loading for you, and less of a resource when it's checking for new entries. Just a few lines of code an everyone benefits. This feature will be available in the next release.

The next release will hopefully be available in about a week. I'm fine tuning the big new feature still. ;)

P.S. If you haven't filled out the poll yet please do so. Getting more input on that feature will really help as it's a lot of work and I don't want to do it if nobody is interested

Feed Icons for Caterpillar

I had a brainstorm last night. A solution to the problem of feed icons. favicons! This will be available in the next release of Caterpillar (2.5). I"m amazed at how useful they are. I thought it'd be a nice thing to have but it makes picking the items you want to read that much easier.

For the geekily curious the problem was that while you can specify an icon in an RSS feed the icons people use are generally huge and of such variad sizes that automatically resizing them to something useful that still looked good and was readable just wasn't realistic. But then I thought of favicons. Lots of sites have them and they're perfectly sized and made to be viewed at a ridiculously small size.

Do you want a web interface for Caterpillar?

I've been considering offering a web interface to Caterpillar. This would be in addition the the current interface. The emphasis would again be on making things simple and fast although I'd have a professional designer create the actual look and feel. A number of users have noted interest because they either like web interfaces or because the web interface allows them to run their aggregator on their home computer but access it from anywhere. Please take a second and fill out this poll.

P.S. I've got a bunch of enhancements, new features, and bug fixes ready to go for the next release. A web version would come out later.

Read smarter

Imagine what would happen if your News Reader knew what you liked and let you know before you started weeding through all the new entries yourself.

It will.

Caterpillar 2.0 is released

Caterpillar 2.0 is now available! Checkout the features and screenshots.

I'd like to give special thanks to the many open source projects upon which Caterpillar is built, to Kevin A Burton for his his Java implementation of Mark Pilgrim's feed parsing library, to Knarphie and John for their feedback on the beta versions, and to Mizzy for putting up with my blabbing about more software.

I'd also like to thank anyone who takes the time to download it. If you like it please consider making a contribution so that I can afford the time to make it better for all of us.