Anders Conbere

Those of you who don't know I've been working for Fidgt for the last couple of months. Mostly consulting on how best to integrate with the rest of the open social networking world, as well as how we can be the best participants possible. As part of this I've had the joy of being able to work with a number of the available python libraries for things like Microformats, FaceBook, and Flickr.

Sadly, very few of the available tools met our needs going forward, as such much of my time lately has been spent figuring out how to make a set of generic tools that will provide a basis for building up access to each of these services and their web service api's.

One of the first problems we had was that many of them either presumed a single user type solution (a class instantiated with a username and password), this is sub obtimal for our high load use case, and a bit difficult to work with when wanting to be able to generically query various social networks. The result is that the first thing I did was write an Auth layer. The auth layer is very simply a database access layer in sqlalchemy and some other associated functions to ease use. It stores the relation between a username that we use, a username that the remote service uses and a token or password for that user at that network. Since most of the authorization patterns use a very similar workflow for authorizing access from users this has let me abstract much of the background storage and retrieval of tokens.

So where are we on this? We currently have working libraries for FlickrAuth, and FacebookAuth and we're moving towards an OAuth solution.

Writing these libraries is when I began to really understand the beauty of oAuth. Ahhh... imagine only having to ever write one of these libraries and have it "just work" across multiple networks. And not only does oAuth solidify how we go about authorizing a user, but also defines the way we make signed requests to the service, so that i no longer have to write 5 different parameter serialization functions one for each service. It's a fantastic achievement, and my only beef is that it's flexibility lends itself to a much more complex set of tools to use it.

My last post mentioned that I was disappointed in the state of current python microformats parsers, and that I had some plans to fix that. A week or so later, and many iterations I've arrived at something I feel pretty good about. So I'd like to release to the world the Python Microformat Parser. It's currently capable of parsing xfn, rel="tag", hCard and hCal. There is "support" for hAtom but the xls transform I'm using isn't playing nice.

So I would love for people to checkout the code, find bugs, make changes, etc. I think that in all microformats are a great way of adding data to html, and a good way to get web developers started in creating semantic xml data on the web, but a lack of quality parsers seems to plague at least python, hopefully we can get started on a dialog to resolve that.

I've just finished most of the work on a new python microformat parser. For the most part what it does is quite simple. It uses lxml and their html parser to pull an html block into memory, transforming it into valid xml, then uses brian suda's great xsl transforms to convert that into valid vCard and vCalendar items which are parsed by vObject.

Surprisingly this all works pretty well, and is actually quite fast. That being said it suffers from a few problems. The vCard implementation doesn't seem to support multiple typed data well (the interface to get that back out is cludgy), and the hAtom xsl document I'm using seems to return empty strings.

So right now I'm claiming really quite good support for vCard and cCalendar, near support for hAtom, as well as including a parser for xfn, and rel="tag".

The main draw back to this glue type method is that I'm not providing a consistent object interface, though in time I imagine I could clean that up. Overall it's support for the full range of vCard and vCalendar attributes is much better than that of the previous python parser, and it's chances of being easily extendable are high (all we would need to do is write new xsl documents).

Well, I finally sat down and did a serious overhaul of the old blog. I've moved all my django hosting onto slicehost which has been fantastic. I've got most things back up and running, did a big visual overhaul, and I broke my resume application.

Well, I "fixed" it, but there was data loss involved in that, so it will be coming back together slowly here.

I've been up to my ears in projects lately. I've been putting together some of the pieces for a new travel guide site crafty traveler, mostly dealing with how to do the GIS data parts of that. I've been having some problems getting all my GIS tool compiled on that box which is hosted at Joyent and runs solaris, but the joyent support has actually been really good and hopefully with a little bit more tweaking that will come together.

I've also been working on a golf social networking site, primarily building widgets running off of ruby on rails, that's been a struggle, but after a bunch of initial road blocks I've figured out how to do most things in rails. It's an incredibly complex system, and I'm not to sure why it has all that overhead, but it's good for me. The javascript widgets have totally thrown me for a loop, but I'm sure I can work that out.

anyway... busy :-D

Last Saturday we did another 6 hour startup, this time the goal was to create a game trading site that used the xmpp as the backend for storing data about who you're friends are. This was probably a little too ambitious and the result is at gamewrangle.com. We had a distinct lack of designers present and it shows. That being said we actually accomplished a good deal of work, I think everyone was pretty pumped about the project and it "works".

The basic idea is pretty simple.

1. you're a gamer, you have games you don't play anymore, there are games you haven't played and want to.
2. you have friends who are gamers as well in the same situation.
3. these friends are in your messenger buddy list
4. by alerting you to the existence of who has and wants what, we can facilitate trading games between you and your friends.

The mechanism is simple, just add a list of the games you are willing to trade, and a list of games you want, and we go and find the "matches".

The business plan involves using the rich data from game entry to display targeted advertising or amazon referral links.

I learned a couple of key things, mostly regarding the use of my own jabber libraries, and how important it will be to get the jabber devs to understand how powerful an authorization workflow would be.

Last Saturday I took part in an event called SaturdayHouse where a group of people get together at someone's house to collaborate, share ideas, think, read and in general share in the company of others. This Saturday a friend of mine Brian Dorsey of NoonHat fame, suggested that we attempt to take that time to go from idea to production on a single idea, limiting ourselves to 6 hours.

The idea was simple, create a website that let people put in a username, have the app go crawl various sites looking for them and check what the tags are and if it has a tag "remind:YYYY-MM-DD" then remind them about that item when the date comes up. The Technology choices were made by Brian, we would be using Django (HURAH) the python web framework and Python. The result of this six hour code jam is TagMindr, "Put any bookmark in a time capsule and we'll send it to your future self.".

The way it works right now is it crawls your feeds on del.icio.us and magnolia based on the tag "tagmindr" and then does a second search for any items with the tag "remind:YYYY-MM-DD". And it works! Also the design we came up with has proven to be quite flexible and adding in Magnolia and Flickr support only took a few minutes (so if you think of any other sites that allow tagging like this tell me!).

In all the event was fantastic, I met a ton of great people, and actually got to work with them on a project that is live right now. Discussion has continued and people are still committing changes to the svn repository. So I have high hopes for what can come of this, more events like it, more collaboration, more ideas brought to life.

As most people around me know, I've been working pretty hard on disseminating the idea of using jabber as a backbone for a truly open social network, as well as attempting to build a set of libraries exposing a simple web api to provide for this.

The two web frameworks I'm most comfortable with are Ruby on Rails and Django (and PHP but I'm not ready to back down that road just yet). So I've been moving forward building some simple web applications in these frameworks, and in general I'm calling the project xmpp-psn short for Extensible Messaging and Presence Protocol - Portable Social Network (I know that's a terrible marketing name, and a mouthful, but I'm focussed on building the tools more than thinking up good names, if you have suggestions... email me!).

Well, Friday talking to Blaine Cook, he brought up a point I had yet to consider and lends itself to a rather tricky problem. I've been writing these tools as if they were there only entry point to the app, of course this means that in order to scale the app you have to put it on better and better hardware, in particular it means given the way that most web servers work (spawning a new instance of the ruby/python interpreter through fastcgi or mod_*) that it can't run on any multiprocess or threaded web server.

So what's the solution?

I think for now I'm going to ignore it. There are solutions that exist drb or dynamic ruby offers a messaging type solution (much like how erlang deals with concurrency) a solution for messaging through apache sessions that I've forgotten the link to and more. The problem with all of these is that the would require a) a lot of restructuring of code I already have written, and b) a lot of time spent thinking about how to do it. So, considering how small my target audience is at the moment, and that I'm more concerned with showing people what can be done, I'm just going to let it slide. Hopefully someone will come along and help me work out that kink when activity picks up.

Today I finished the first version of a Django app that lets you build a social networking site on top of Jabber (XMPP). You can find out more about the basics of the project here, but basically it's an http layer on top of a common python XMPP library that lets you do most of the roster management available through XMPP by way of a simple restful api.

The workflow for using it is simple, just install the app into an existing django project, and that will expose the api. From there you can post your jid and password to the /login/ url which returns a uuid and a roster in JSON format for your web app to digest. (note: this is horrible security practice, don't do this in the wild) Now that you have a uuid you have access to the rest of the api, which links to a thread on the sever that keep track of your jabber session.

the following is a list of the supported roster functions:

• disconnect
• roster
• subscribe
• unsubscribe
• authorize
• unauthorize
• remove
• login