Anders Conbere

I feel like this will probably get lost in the huge torrent of comments here. But I think the sentiment of this is important. I've been talking a lot about "how to make creating social networks easier" for the last year. Part of this discussion has involved the people associated with Social Network Portability and some of it has Involved the people associated with Data Portability.

For the last couple of years a number of people have been fighting a battle waged in the world of publishing. "How do we get closed networks to open more of their data". During that time I've been working on problems like "How can we use xmpp's roster as a distributed social network". I feel like the problem of publishing data has largely been solved, this was a marketing problem and the space is now familiar with many of the contemporary companies and the idea has reached a tipping point. What remains is a problem of digestion and one that has gone relatively unheralded. That is, how do we make this data that is now being exposed useful and relevant.

One of the interesting things that I find happens when you reframe the problem like this, is that "importing" users data becomes minimally interesting. As a social network built around providing a set of features being able to import a starting set of social data is only minimally helpful. My big problems still exist. I need to build tools to help me manage those relationships and I have to market my product such that the user base is large enough to make my application useful. If we extend that into other types of data the same kinds of problems emerge.

Basically, we've tackled a problem that exists for consumers, "I want my data back", and done a pretty good job. But forgotten that these tools we're providing need users themselves, and that the users of these tools are not consumers of social applications, but rather the creators of new social networks.

New social networks need a user base, but the creators also want to reduce the complexity of creating their applications. This means that tools that we provide these creators need to offer something. They need to not only help the creator build a user base quickly, but also provide tools for managing complexity. In the case of importing relationship data, the tools should include abstractions for adding and removing friends, and storing the state of that data off site. The tools might want to bring with them features that are painful to create in and of themselves, like messaging, and presence. The should make the life of the creators of social networks easier, and the fact of the matter is, is that right now getting social data doesn't appear to be the problem that social networks creators are facing.

So how do we approach this? I think the first thing to do is to start using the tools available and imaging ways to bring things to the table. If we look at OpenID and oAuth, and open social, those technologies have seen fairly serious attention from large sites. And if you look all of those technologies help reduce the complexity for the people making social networks. They either help reduce the overhead of exposing api's, logging in users, or creating applications on top of your tools. When we look at the other technologies (XFN, FOAF, Microformats, etc.) that Data portability is championing I think we see a very different pattern. I know I've suggested that xfn offers almost nothing back to creators, but there are solutions I've found for FOAF (pinback + oauth + sparql), and certainly XMPP brings a lot to the table. That being said, these tools don't exist yet, and they'll take time to build. But until our tools are tailored for the creators, I don't think we'll see significant adoption.

When I first started playing in the Social Network Portability space I was working on some very light weight social network applications and wanted a way to interact with a global set of users quickly and easily. Right away I discovered foaf and xfn. I was immediately disenchanted with both technologies for what at the time I termed "statelessness", and in some late night discussion I thought about how my IM buddy list captured all the details I wanted from my friends networks, but allowed me to dynamically update it. From that came xmpp-psn, and numerous other projects that have all since failed horribly.

The state of SNP is not the rosy one that is given by the web community. Yes we have XFN, yes we have FOAF and XMPP, and DISO and SIOC. But most all of these technologies have massive flaws. That being said I think there's real hope for both XMPP and FOAF (more on why I'm leaving XFN off this list later).

So getting back to "Statelessness", XFN and FOAF both suffer from being static representations of a social network, both have the ability to describe complex graphs, be pulled real time, and as such aggregated. But the problems arise when a social network decides to make use of this data (say through google's Social Graph API), at this point the data leaves the control of the user and since it's stateless it becomes a "copy" of that data on the social networks network store, and in FOAF and XFN these technologies do not provide the means to communicate new changes back to the original source.

The current method of thinking seems to suggest that each time I join a new social network I'll point them to some canonical representation of my identity (a foaf file, an openID etc.) and the social network will then process the FOAF or XFN data, store the nodes and arcs in their database and connect me via that data to other users in their network. Then they in turn will publish that data. So the current model starts to fall apart at this point, as networks are then forced to aggregate that social data from each other in oder to stay relevant. This results in either having to be smart about caching and networking, or exponentially increasing number of polling connections (presuming we're using http). XFN further exasperates the difference between what's available for the Social Network provider and the canonical definition of the graph (by way of hrefs) as publishers really have almost no way of changing or removing content from that representation once it's been indexed, spread across multiple pages and distributed across the net.

Contrast this now with a technology like XMPP where the roster is stored on a particular server, queryable from any client in the network of federated XMPP clients, and can be adjusted realtime. This means that in order for social networks to stay up to date with XMPP they make a simple call to the users XMPP server and parse the results, and if they have changes they want to make, they can them communicate those changes back to the XMPP server. Looking at the traffic we now are back at a linearly increasing traffic graph, and the complexity of the operations has been reduced significantly. This being said, XMPP has it's own host of issues, not the least of which is it's ability to interact with the web over http, and the complexity of programing api's to communicate with it as it stands today.

It should be clear that the current state of FOAF and XFN aren't really going to cut it for todays social networks. They are great static publish side technologies, but that's not the world we live in with social networks. Social networks are wildly dynamic and they shift with context and need real time awareness. But but FOAF and XFN are doing something great, they are actually getting that data published and out there. Now we need to turn around and look at how those technologies can be made better for the creators of social networks.

So what do we do about it? Surprisingly completely on it's own and unrelated the rdf community has created some technology that I think might be extremely relevant here. It's called SPARQL and you can imagine it as being a simple SQL like query language for RDF stores. This it seems would make an incredible tool for defining the communication methods between hosts providing FOAF data. Now this problem isn't even close to being solved either, but the steps to solve it aren't particularly hard. There have been some interesting extensions to SPARQL that allow for inserts and updates. If we standardize on that as a means of communication then the supplier of FOAF data now needs to provide a means of digesting those SPARQL queries and announcing that they support updates to their FOAF file. Couple this with the work that Dan Brickley's been doing on oAuth bindings for FOAF and you have a powerful tool for not only representing your relationship data, but maintaining the state of those relationships close to realtime.

As for XFN? I don't think XFN is up for the job I don't see any way to repair the problems I've mentioned above with the state of XFN. (If you have one please tell me!)

Those of you who don't know I've been working for Fidgt for the last couple of months. Mostly consulting on how best to integrate with the rest of the open social networking world, as well as how we can be the best participants possible. As part of this I've had the joy of being able to work with a number of the available python libraries for things like Microformats, FaceBook, and Flickr.

Sadly, very few of the available tools met our needs going forward, as such much of my time lately has been spent figuring out how to make a set of generic tools that will provide a basis for building up access to each of these services and their web service api's.

One of the first problems we had was that many of them either presumed a single user type solution (a class instantiated with a username and password), this is sub obtimal for our high load use case, and a bit difficult to work with when wanting to be able to generically query various social networks. The result is that the first thing I did was write an Auth layer. The auth layer is very simply a database access layer in sqlalchemy and some other associated functions to ease use. It stores the relation between a username that we use, a username that the remote service uses and a token or password for that user at that network. Since most of the authorization patterns use a very similar workflow for authorizing access from users this has let me abstract much of the background storage and retrieval of tokens.

So where are we on this? We currently have working libraries for FlickrAuth, and FacebookAuth and we're moving towards an OAuth solution.

Writing these libraries is when I began to really understand the beauty of oAuth. Ahhh... imagine only having to ever write one of these libraries and have it "just work" across multiple networks. And not only does oAuth solidify how we go about authorizing a user, but also defines the way we make signed requests to the service, so that i no longer have to write 5 different parameter serialization functions one for each service. It's a fantastic achievement, and my only beef is that it's flexibility lends itself to a much more complex set of tools to use it.