ShoeChicken

Shoechicken Goal Changes

2007-02-16T23:34:00.000-06:00

Some fundamental assumptions as to how Shoechicken will be used have changed. We've decided to move from a personal computer application to a more service oriented approach. In changing these fundamental assumptions, much code will have to change. New code is being developed in Ruby as both a learning exercise and an attempt to speed up development.

Rerating unread but unrated items

2006-10-31T00:25:00.000-06:00

Well, been a long time since I last posted. But let's just pick up where we left off. A good deal of work has continued on shoechicken. We have possibly gained another develop who's currently looking into integrating hibernate into shoechicken to abstract the database more than our current "helper" O.R. Mapping classes.

As noted by this posts title, here's a current question that needs answering. If items have already been rated for a user, but he or she has not yet read them yet, should those items be rerated when the next batch rating occurs? Here's a mini brain dump on the issue.

To start this off, a few basic assumptions can be made; the reclocation of an item should really represent the item's place within that set of items. It would seem reasonable to presume GUIs are going to have a difficult time presenting every single item in a potentially large set of items; thus, we can assume that an item for which shoechicken has received no kind of event has not been viewed by a user. Based on these assumption, every time a user requests a set of rated items shoechicken should do its best to reclocate each article, thereby taking into account relationships among the items.

Thinking over these terms leads to the thought that not rerating previously rated items would placee previously rated items having ratings not relevant to the current data set. Whereas, if we do rerate them then all unread articles are reclocated relative to one another.

LaTex and Genetic Algorithms

2006-01-30T00:51:00.000-06:00

Spent a lot of time this week getting the paper ready for the conference. I decided to use LaTeX as the conference put an outline file out as .tex. I've only had brief foray in using LaTex, so formatting the paper had a steep learning curve. I do have to say however that it was quite fruitful. I now appreciate the power of LaTeX and will probably use it again for further papers.

Anyway, moving past the paper, I started working on the shochicken. Much of the work was re-familiarizing myself with the research I performed last year. I am really quite excited about this part of the project! Since a lot of the work has been more laying out the foundations, there isn't really much to report on progress with the shoechicken.

I did have what could turn out to be quite a good idea though. We'd assumed a fairly traditional machine learning approach for shoechicken. The system would employ some kind of reinforcement learning to build a user model. However, it may be a good idea to look into genetic algorithmic techniques. We could let the Shochickens breed!!! Sounds a little crazy, but there could be a great deal of merit to this approach. Only just thought about it, but will research the topic with some zest over the next week or so and let you all know how it goes!

Completed the FeedSubscriptionService, Bufunkalo, and Shredder

2006-01-21T21:53:00.000-06:00

The FeedSubscriptionService, Bufunkalo, and Shredder are all completed and tested! The next items on the hit list are the Coordinator (previously known as UIAgent) and the Shoechicken itself. The UIAgent was renamed because its role now is more of a centralized module that ties all other pieces together.

Michael and I have begun laying out the internals of both the Coordinator and the Shoechicken. The Coordinator is going to involve working out some messy synchronization issues between the modules, but we think we've got the main operations mapped out.

I've started writing code for the Shoechicken. I will probably start by creating a test case for the Shoechicken and build the internal event chain gradually. The first step will be to pull the data from the database and build the Article/Keyword matrix. I am going to have to create some fake user models with which to compute that ratings, so that will probably be next. Then, after combining the new data with the user model, we'll start working on using LSI to row reduce the data.

My wife suggested that we look to possibly adapting Shoechicken to perform a similar task for video footage. Shoechicken could recommend new movie trailers, news footage, and other kinds of video footage. After talking it over for a while it really seems like a possible direction for Shoechicken in the future. It would of course require delving into video image processing, which sounds like a good challenge. Let the fun begin!!

Shoechicken Paper Accepted!

2006-01-01T11:12:00.000-06:00

Here's the first update of 2006; Happy New Year everyone!

Well the first piece of news is that the paper Michael and I submitted to CATA-2006: ISCA 21st International Conference on Computers and their Applications was accepted!!! Go Shoechicken! Now the battle to come up with funding to get us to Seattle begins ...

As far as the project itself goes, over Christmas I've implemented but have not written test cases for the Bufunkalo module. That leaves test cases for the Shredder and Bufunkalo modules as the next two goals. After that we can move onto the User Interface Agent (which could possibly be renamed now that its function has changed), and the Shoechicken itself.

Points that came up while implementing the Bufunkalo were the fact that in many cases I either needed to listen for FeedEvents or FeedErrorEvents exclusively. I worked around this by just leaving the unnecessary method unimplemented; however, Michael and I have spoken about the possibility of separating the FeedListener into a FeedListener and another interface FeedErrorListener. We will discuss this in more detail after Christmas.

State of the Chicken

2005-11-15T13:54:00.000-06:00

Wow, it's been a long time since I posted. A lot has happened with the Shoechicken project in the past few weeks.

James and I have completed another draft of the paper. Hopefully, we'll have it up on the website soon. This draft will be our submission to the 21st International Conference on Computers and their Applications.

I have been tinkering with the design (well, not design per se) of the Shoechicken database lately. Most of what I changed was just nitpicking, but since I'm the one working with it at the moment I figured I would go ahead and make the changes. For instance, the table names have been changed to the singular (i.e. "Feeds" is now "Feed"). From a programming standpoint it just didn't feel right working with it in the plural when dealing with individual records. Second, I changed the auto-numbered *Id fields of the tables to simply "id." It was just redundant to type "Feed.feedId," so the new "Feed.id" is much better. Foreign keys are still referenced using the old *Id names, since they refer to the "id" fields in different tables. Lastly, I added a field for the title of a feed to the Feed table. The database design website has been updated to reflect the changes. The scripts for initiating the changes have been comitted to the repository, along with a copy of the empty shoechicken database.

I have given up on the Wiki for writing class specifications. Half of this comes from the fact that it didn't like me saving changes, so it was wasting my time without getting anything published. The other half comes from the fact that I can write code a lot faster than I can specify what it should do. All in all, Shoechicken just won't be completed in any reasonable time if we switch to this approach now. It was a beautiful dream though...

Well, all of the database changes weren't made for nothing. A lot of my time has been spent working on the parts of the Shredder that deal with the database. I have been developing the ArticleDatabaseHelper class, which will be used to add/retrieve/remove data related to the article tables in the database (Author, Context, Feed, Item, ItemToKeyword, and Keyword). The class is nearly completed, with only the methods related to the ItemToKeyword table outstanding. The class already weighs in as a beast of nearly 1,700 lines of code, which I'm sure it will break before it is completed. James and I still need to discuss the approach the ArticleDatabaseHelper will take when dealing with the inability to obtain Connection objects from the ConnectionManager. Additionally, I have added its related exception, the DatabaseProcessingException, to the repository. At this time, no test cases exist for either of these classes. I will work on test cases as soon as I have access to the completed ConnectionManager class.

In addition to the above mentioned classes, I have implemented the Shredder classes which use the ArticleDatabaseHelper to add their appropriate "shreds" to the database. I have commited the ContextEntryCreator, ItemEntryCreator, KeywordEntryCreation, and KeywordCountEntryCreator classes to the repository. These classes have not been tested either, since they are dependent on the completed ArticleDatabaseHelper class. Additionally, James and I need to discuss how these classes will handle exceptions, and that functionally will need to be added to them.

Having Fun with Testing

2005-10-12T00:05:00.000-05:00

Well, I just finished writing my first junit test case. It took quite a bit of effort, a bit more than I'd expected to be honest. However, I really do think it's worth the effort in the end. There is no way that I would perform all the test cases by hand otherwise, particularly not on a regular basis. I tried to follow the philosophy of writing the unit test before writing any implementation of the code. I wrote the skeleton of ConnectionManager beforehand so I could use auto-completion while writing the test.

On another completely different note, I'm writing this blog entry using a toolbar applet for Gnome. It allowed me to enter the blog info. for the shoechicken blog and then bring up a mini blog entry window with a single click. So, if this entry comes out all garbled up, you know why.

I also realized today that java 1.5 added a thread pool implementation in java.util.concurrent. This could be very useful when implementing the Bufunkalo. I'm also wondering, way ahead of when I need to be, whether java.nio might be very useful for the rating part of the Shoechicken; I'll have to look further into that, just wanted to have it out there to remind us.

I spent some time looking into writing an ant build script for Shoechicken this week. I help maintain ant build files at work, but haven't ever written one from scratch. Luckily one for Shoechicken would be fairly rudimentary, but may be worth the time. Currently Michael and I are both using Eclipse to build the project, but a standardized build file from which to create the project may be a good idea. I'm thinking that the build file would only take a few hours to make too. Michael and I can talk about it when we meet tomorrow.

I've been checking the search engines and we're still only up on Yahoo, Altavista, and AllTheWeb. I make sure to click on our links on each search engine; not sure that it does anything but it would seem logical for the search engines to keep track of what people actually click on?

That's all for now, but should be plenty more to post after Michael and I meet tomorrow.

Wiki Wiki Wiki

2005-10-10T16:38:00.000-05:00

James and I have been working on designing the classes related to handling database connections and performing database queries and updates. Since we originally designed the ConnectionManager class, the design has had to be retooled because not everything was thought through thoroughly enough. I think that part of these issues fall from the fact that there is no formal specification for the modules or classes. Being the Software Engineering major, and apparently insane, I feel that there is a need to start building at least semi-formal specifications of the modules and classes. This will lend some traceability to our design decisions and, if we do it properly, should privide us a better understanding of the system and reduce the number of design mistakes we make. So, to that end, I have created a Shoechicken Wiki that we may post specifications on.

It is my suggestion that Wiki pages be created for each module/class that we design, before it is actually "designed." These pages should outline the purpose of the class, how it will interact with other classes, and most importantly its behaviors and error conditions. The Wiki will allow us to easily update the specification as the need arises, which in the beginning will likely be very common but should hopefully become infrequent over time. We should probably develop a "template" for the specification of a class or module, so that all entries will contain the same information and present it in a common format.

The Shoechicken Website received a few small updates. The Documentation section saw the addition of a design page for the database classes. It was the creation of this page that spawned the idea of the Wiki, which could hold formal specifications of these designs. The site now uses a feature of Apache called Server Side Includes (SSI) to include the header on each page. This will make page maintenance much easier. Other, smaller, changes have been made across the site.

The class which spawned this mad tirade of mine is the ConnectionManager class. Originally, James and I had designed the class with a public constructor, so that any class could create one. Then, probably because it was getting late at night, decided (at my erroneous encouragement) that the class should be a Singleton which will load the settings for a ConnectionManager from a properties file. Later, James pointed out the likelyhood that eventually the item and user model databases may split, and that this approach would limit the ability to connect to multiple databases. James and I have started dicussing a Factory approach that would create one ConnectionManager per database, but would allow for multiple databases. This approach is still under discussion.

The Sky is Falling

2005-10-03T10:56:00.000-05:00

Well, Shoechicken is encountering a few setbacks at the moment. It turns out that by leaving the rating agent as a black box, we did not appropriately design the rest of the system to support the Rating Agent. The key issue is that items (news articles) should not be delivered individually, but instead in groups, to the Rating Agent. The benefits of this (according to James) are less computation by the Rating Agent by not having to rebuild a large matrix for each article and more accurate ratings by comparing related articles to each other at the time of rating.

The architecture of the system is in a state of flux. Over the weekend James and I went back to the drawing board (quite literally a whiteboard) to resolve some of the above issues. Much of the functionality has been moved to the UI Agent, which is responsible for managing data from the Feed Subscription Service and the Shoechickens. It has been decided that there will be a 1-to-1 ratio between users and Shoechickens, meaning that for each user in the system there will be a Shoechicken (read Rating Agent + Feedback Interpreter) assigned to that user. The new design still supports the ideas of services mentioned by James, and that of a multi-user environment. The approach will add some additional modules and work, but should benefit the project in the long run.

The UI Agent is now responsibly for managing requests for updates, and ensuring that Shochickens are notified of these updates. The UI Agent will keep track of all active requests from all users, and when requests for article downloads are fulfilled the UI Agent will post events to the appropriate Shoechickens for the users issuing the requests.

A module accidentally left out in the previous design is now called the Bufunkalo, in honor of Hooper P. Bufunkalo (supposed discoverer of the Shoechicken). This module is responsible from capturing feeds and breaking them up into individual items before sending them to the Shredder. The module also weeds out articles that the Shredder has already seen, so that they are not processed again. Before sending articles to the Shredder, it downloads the content of the article if it is available so that the Shredder may break up the content into keywords. Another key aspect of the Bufunkalo is that it will contain a thread pooling system, which will allow multiple articles to be processed simultaneously. This will also prevent any other services which have called the Bufunkalo from having the wait for the Bufunkalo to finish processing (such as the Feed Subscription Service).

Additionally, we determined the need for a UserToFeed table in the database. This table maps a username to a list of feeds that user is subscribed to. This will allow the UI Agent to recreate subscriptions when a user reconnects. This will also provide the Rating Agent with the ability to determine which feeds it should be checking for new articles in the database. This table has been added to the SQL script used to create the database and the database itself in CVS.

The package names have been changed for most of the project. They were getting ridiculously long. For instance, the longest package name was com.shoechicken.shoechicken.bufunkalo.shredder.keywordcounter.event. Now, the same package falls under com.shoechicken.sc.bfnklo.shred.kwcntr.event. The new, more compact, package names should save some typing.

Thoughts about the rating agent

2005-09-26T00:56:00.000-05:00

As Michael had already posted, I spent some time this weekend working on the database, only to discover Michael was working on it too. We'll have to work on that whole planning thing. We've got the empty shell of the database in place now though.

I've also been thinking about the internals of the rating agent and some of the conceptual functionality of shoechicken. One thing I was thinking about is whether the rating agent could be thought of as a system service. This would allow news to be aggregated even when the GUI (Frying Pan) was not active. Since the RSS/Atom feeds cycle through articles, having the service running in the background would allow the retrieval and reclocation of articles that could otherwise be lost. At first this seemed to be an alright idea, but after talking with Rachel she pointed out how that may put some people off the application. Also, if someone doesn't check the news very often they're probably not too interested in what transpired 2 weeks ago.

Part of what got me thinking about this was the possibility of re-reclocating articles that had not yet been read by the user. This doesn't seem to make sense if the article has already been posted in the Frying Pan as articles could jump around in the list too much. However, if the article had not yet been posted to the GUI, it would seem plausible to reclocate that article again to find its most appropriate ordered location in the unread article set. The problem is that this situation is not likely to occur if Shoechicken only runs when the GUI is active.

So, my final thoughts (thanks largely to Rachel) are that the user could be given the option to have shoechicken collect news in the background, or just when the GUI is open. I know this is thinking a little way down the road, but my goal is to not prevent the former from happening through design choices.

Onto the Shoechicken rating agent internals.

1. Shoechicken should receive FeedItemEvents from the Bufunkalo. He can then build a vector representation of this article with each component being a normalized TF-IDF weight. The IDF part will be calculated from the set of all unread articles.

2. The TF-IDF weight vector can then be passed onto an LSI component that can put the new vector into the item vector space and reduce the dimensionality. Although calculating this can be expensive, I'm not sure that it will be that bad for our data set, it will be easiest to just recompute the SVD each time instead of folding the new vector in. This can be changed in future implementations but isn't necessary for us right now.

3. The reduced new vector should end up at the reclocating component which can compute the item's rating based on the user model. The rating will then be stored in the database and then UI Agent can be notified.

The feedback/AI component isn't addressed here as I think this is best left to a later date. Laying out the AI shouldn't alter the design of this part too much. It's more the case that the AI will tweak each component to try and improve their effectiveness'.

This is what I've thought up so far. Michael and I are going to meet on Tuesday and will probably talk over the ideas in more detail. Hopefully I can start on some coding mid next week!

Shoechicken Database Completed

2005-09-25T23:03:00.000-05:00

Busy day with Shoechicken.

The Shoechicken Database has been committed to the CVS repository, along with the scripts used to create it. The database the Shoechicken project is using is HSQLDB Database. The database in the repository is a skeleton of a complete database. It contains no data, only definitions of the tables. Sad thing is that both James and I constructed the database at the same time today, so there was a bit of overlapping work and some of it was not used (sorry James). The good thing is both James and I got some experience using HSQLDB, which will pay off in the coming weeks of development.

The Shoechicken website has been updated with new documentation on the database. Additionally, some dead links on the site were fixed. The most important update was the addition of the Database Design page under the documentation section. The page should serve as a reference for any designers who need to check the table designs, data types, and contraints used in the database.

The Blog Creation

2005-09-20T18:56:00.000-05:00

This is the first blog entry for ShoeChicken, the Bearded AI. The blog is a means of communication between the developers and anyone else who wants to comment on ShoeChicken.