Anders Conbere

I think that we've seen some fairly convincing examples of how labeling or tagging can reduce the complexity of grouping sets of data, in particular when it might be difficult to assign the data items into only on individual group. Some big uses of tagging as the primary form of grouping includes gmail, delicious, and flickr. However in XMPP our roster grouping are still relegated to binning or boxing (an item in a group exists in one and only one group). But roster items aren't simple data types, they represent our relationships with people! and people often don't belong to just one group. Rather the people in our lives often belong to many different intersecting groups (my good friend caleb, is both part of the programmers in my life, and my close friends, and my child hood friends, and the people I play soccer with, and climbing). There doesn't seem to be an technical limitation in creating tags or labels for XMPP roster items, there are some questions to be answered about how to store the relations, and what semantics to use when querying them, but these aren't insurmountable.

as an initial reference for XEP's that look / act similar there is

• MetaContacts
• Annotations

My only worry is that both of these use the Storage protocol, and I question how easy it would be to form queries like 'retrieve me all the users with tags "X" and "Y"' Which might be out of band for this XEP but I suspect that queries like that might be powerful.

We are beyond the point where social networks should be valued on the number of locked in customers they posses. This was the wrong way to place value on social networks to begin with, but it's almost silly now. When looking at social network valuation we need to know a couple of things

1) what are the quality of the users who are participating? * that is, how much money do they spend on the products being sold (ads or otherwise)

A social network that engages people with high disposable incomes, and are likely to purchase products online is much more likely to sell products on that network via advertising. If your network is a particular niche, the strength of that relationship might also come into play, as the quality of the products being sold (advertising) might as a result also increase.

2) how active are those users on the network? * that is, how much time are they spending engaging with the network, and how is that resulting in higher sales.

A social network that engages people to spend more time engaging with people, the product and advertising is worth more than a social networks that fails to do those things. Further the correlation between that activity and sales of products or ads interacted with are extremely important metrics when attempting to understand how valuable the time spent there is.

3) What is the quality of the service your application is providing? * How likely is it that your service will continue to engage users in participation.

If you have a crappy photo editing social network, it's less likely to continue to draw engagement from your users over time than a high quality product in the same niche.

So what does this do to the way we perceive the value of social networks today? If you don't care where your users come from, only that they are active on your site and engaged, what benefits do you have to lockin? I would argue none! I think this idea is terrifying to the folks at Myspace and Facebook, but this kind of valuation has huge ramifications to little sites like dopplr or tripit. These sites have no perceived value in building a social network. They are selling a service, and if they could build their product off of a pre-existing network that was large enough to continue to flow traffic through them at an acceptable rate, I would argue that economically that makes way more sense than attempting to entice people into creating new networks there.

This is what Facebook is selling with their app platform, what google wants to see happen with open social, and where a lot of the conversation about Social Network Portability should be happening. This is a fundamental change in the way that the majority of people view the valuation of social applications, and a difficult one to describe. But this is the power of tools like XMPP, FOAF and XFN. They begin to bring about a decentralized social network that the creators of social networks can use to dramatically increase the valuation of their application (by bringing more people through their app, and allowing them to focus on the quality of the services they provide), which for most social network creators, is how they make their money.

Facebook and MySpace might not be rushing out to join this game, but all the small services should be looking around and saying. "Why are we busy building social networks when we could be using a network that already exists to drive more traffic through our site and thus generate more revenue?" And that's the question really, "how do we generate more revenue?"

I feel like this will probably get lost in the huge torrent of comments here. But I think the sentiment of this is important. I've been talking a lot about "how to make creating social networks easier" for the last year. Part of this discussion has involved the people associated with Social Network Portability and some of it has Involved the people associated with Data Portability.

For the last couple of years a number of people have been fighting a battle waged in the world of publishing. "How do we get closed networks to open more of their data". During that time I've been working on problems like "How can we use xmpp's roster as a distributed social network". I feel like the problem of publishing data has largely been solved, this was a marketing problem and the space is now familiar with many of the contemporary companies and the idea has reached a tipping point. What remains is a problem of digestion and one that has gone relatively unheralded. That is, how do we make this data that is now being exposed useful and relevant.

One of the interesting things that I find happens when you reframe the problem like this, is that "importing" users data becomes minimally interesting. As a social network built around providing a set of features being able to import a starting set of social data is only minimally helpful. My big problems still exist. I need to build tools to help me manage those relationships and I have to market my product such that the user base is large enough to make my application useful. If we extend that into other types of data the same kinds of problems emerge.

Basically, we've tackled a problem that exists for consumers, "I want my data back", and done a pretty good job. But forgotten that these tools we're providing need users themselves, and that the users of these tools are not consumers of social applications, but rather the creators of new social networks.

New social networks need a user base, but the creators also want to reduce the complexity of creating their applications. This means that tools that we provide these creators need to offer something. They need to not only help the creator build a user base quickly, but also provide tools for managing complexity. In the case of importing relationship data, the tools should include abstractions for adding and removing friends, and storing the state of that data off site. The tools might want to bring with them features that are painful to create in and of themselves, like messaging, and presence. The should make the life of the creators of social networks easier, and the fact of the matter is, is that right now getting social data doesn't appear to be the problem that social networks creators are facing.

So how do we approach this? I think the first thing to do is to start using the tools available and imaging ways to bring things to the table. If we look at OpenID and oAuth, and open social, those technologies have seen fairly serious attention from large sites. And if you look all of those technologies help reduce the complexity for the people making social networks. They either help reduce the overhead of exposing api's, logging in users, or creating applications on top of your tools. When we look at the other technologies (XFN, FOAF, Microformats, etc.) that Data portability is championing I think we see a very different pattern. I know I've suggested that xfn offers almost nothing back to creators, but there are solutions I've found for FOAF (pinback + oauth + sparql), and certainly XMPP brings a lot to the table. That being said, these tools don't exist yet, and they'll take time to build. But until our tools are tailored for the creators, I don't think we'll see significant adoption.

When I first started playing in the Social Network Portability space I was working on some very light weight social network applications and wanted a way to interact with a global set of users quickly and easily. Right away I discovered foaf and xfn. I was immediately disenchanted with both technologies for what at the time I termed "statelessness", and in some late night discussion I thought about how my IM buddy list captured all the details I wanted from my friends networks, but allowed me to dynamically update it. From that came xmpp-psn, and numerous other projects that have all since failed horribly.

The state of SNP is not the rosy one that is given by the web community. Yes we have XFN, yes we have FOAF and XMPP, and DISO and SIOC. But most all of these technologies have massive flaws. That being said I think there's real hope for both XMPP and FOAF (more on why I'm leaving XFN off this list later).

So getting back to "Statelessness", XFN and FOAF both suffer from being static representations of a social network, both have the ability to describe complex graphs, be pulled real time, and as such aggregated. But the problems arise when a social network decides to make use of this data (say through google's Social Graph API), at this point the data leaves the control of the user and since it's stateless it becomes a "copy" of that data on the social networks network store, and in FOAF and XFN these technologies do not provide the means to communicate new changes back to the original source.

The current method of thinking seems to suggest that each time I join a new social network I'll point them to some canonical representation of my identity (a foaf file, an openID etc.) and the social network will then process the FOAF or XFN data, store the nodes and arcs in their database and connect me via that data to other users in their network. Then they in turn will publish that data. So the current model starts to fall apart at this point, as networks are then forced to aggregate that social data from each other in oder to stay relevant. This results in either having to be smart about caching and networking, or exponentially increasing number of polling connections (presuming we're using http). XFN further exasperates the difference between what's available for the Social Network provider and the canonical definition of the graph (by way of hrefs) as publishers really have almost no way of changing or removing content from that representation once it's been indexed, spread across multiple pages and distributed across the net.

Contrast this now with a technology like XMPP where the roster is stored on a particular server, queryable from any client in the network of federated XMPP clients, and can be adjusted realtime. This means that in order for social networks to stay up to date with XMPP they make a simple call to the users XMPP server and parse the results, and if they have changes they want to make, they can them communicate those changes back to the XMPP server. Looking at the traffic we now are back at a linearly increasing traffic graph, and the complexity of the operations has been reduced significantly. This being said, XMPP has it's own host of issues, not the least of which is it's ability to interact with the web over http, and the complexity of programing api's to communicate with it as it stands today.

It should be clear that the current state of FOAF and XFN aren't really going to cut it for todays social networks. They are great static publish side technologies, but that's not the world we live in with social networks. Social networks are wildly dynamic and they shift with context and need real time awareness. But but FOAF and XFN are doing something great, they are actually getting that data published and out there. Now we need to turn around and look at how those technologies can be made better for the creators of social networks.

So what do we do about it? Surprisingly completely on it's own and unrelated the rdf community has created some technology that I think might be extremely relevant here. It's called SPARQL and you can imagine it as being a simple SQL like query language for RDF stores. This it seems would make an incredible tool for defining the communication methods between hosts providing FOAF data. Now this problem isn't even close to being solved either, but the steps to solve it aren't particularly hard. There have been some interesting extensions to SPARQL that allow for inserts and updates. If we standardize on that as a means of communication then the supplier of FOAF data now needs to provide a means of digesting those SPARQL queries and announcing that they support updates to their FOAF file. Couple this with the work that Dan Brickley's been doing on oAuth bindings for FOAF and you have a powerful tool for not only representing your relationship data, but maintaining the state of those relationships close to realtime.

As for XFN? I don't think XFN is up for the job I don't see any way to repair the problems I've mentioned above with the state of XFN. (If you have one please tell me!)

I Gave a talk on XMPP two days ago as part of the XNP webcamp in Cork, I attempted to make my Presentation in the style of Lessig. The result is that these notes aren't particularly interesting, but I should have a link to the video up soon. The talk went very well and was quite exciting to give. I thought it had gone well after finishing it, but hadn't realized the impact it had until we had adjourned for lunch and a small group slowly grew around me to discuss the implications of real time messaging on their various ventures.

Overall a simply fantastic experience. Now to see if I can reduce my 20 minute talk to 5 and give it at ignite seattle :-D

So, talk about storm in a teapot.

As I hinted at in my update there was a pretty significant amount of miscommunication that went on here. Starting with a poorly named service, a terribly worded change log when that poorly named service changed and then a series of not so helpful posts to the newsgroup. The real story is that XMPP isn't meeting all of google's mobile needs (it wasn't designed to), they had broken compatibility to make use of only a few of XMPP's features, and changed the name to match that. These changes have no relevance to the rest of Gtalk, and the team encourages people to use some of the freely available java implementations on android.

But that's somewhat less exciting than where we're at now. After some frantic IM'ing and emailing we've got some really great communication started between the Google android team and the XMPP community. The community has been explosively discussing ways to improve mobile efficiency and a flurry of tests have popped up (and talks of an XEP).

My only regret is that the Android team didn't feel like it could start that way. I'm not sure if this is a result of the pressure of working on android and having to ship, or if it's a failing on the part of the XMPP community to reach out to developers. But this is exactly the sort of thing that Open Source projects try to prevent, the fact is that the xmpp community cares what the implementers think, and the problems they run into. And we want to make xmpp better for them.

Anyway I want to give a big thanks to Dan Morrill of Google's Android team for working with me in making this happen, and his fantastic post to the XMPP mailing list.

A post floated by the XMPP standards mailing list today about a change to the android spec involving their use of XMPP. The post is a scary statement from a company that has shown such willingness to work with the XMPP community to help build their Gtalk service.

The com.google.android.xmppService package has been replaced by the com.google.android.gtalkservice package. This is driven by the fact that the GTalk API is not XMPP compliant, and will be less so going forward. The reason is that XMPP is too verbose and inefficient for mobile network connection, and the GTalk API will be moving to a binary encoding for the protocol between the client and the server.

This distresses me. Not because Google wants to do their own thing with their chat service, but because they feel like these are issues that either can't be remedied, or can't be brought to the xmpp community. The result of course is that the standards list has started a discussion about ways that the problem of network efficiency (on the mobile scale. For many standard uses XMPP is fantastically efficient) can be tackled.

Peter Saint-Andre compiled a list of discussion points that could help with this.

1. Recommendations regarding when to use the TCP binding and when to use the HTTP binding (BOSH).

2. Compression via TLS or XEP-0138 (use it!). Also binary XML as a compression mechanism.

3. Fast reconnect to avoid TLS+SASL+resource-binding packets.

4. ETags for roster-get (see XEP-0150, let's resurrect that).

5. Advisability of presence-only connections (no roster-get, just send presence and whatever you receive is nice).

Anyway I hope that Google recognizes the XMPP communities willingness to work with them to provide standard ways for them to accomplish these goals. Or that they should at least be having this discussion in the XMPP space so that other implementations of the protocol can interface or learn from them.

Update:

There's been an update on the developer google group for android clarifying part of what's going on here. It appears that Google had a particular subset of the functionality of a full messaging stack in mind when adding xmpp into android, and that to optimize that subset of functions they're employing a proprietary protocol.

So this might just be a case of over bad wording on their changes page, which has since percolated into the rest of the XMPP community. That being said there's something silly about implementing a non-standard solution for messaging on the device without consulting with the xmpp community and foundation on ways that you could make that standards compliant. And I think that if you look at what the "right thing" to do is (I'll quote Mark Atwood here)

fix gtalk, not break android, and XMPP compression will beat their proprietary binary protocol

That being said, I think this is good wake up call to the XMPP community that this is a growing problem space, one that we need to think about and possibly one we should employee resources to tackle. And I think it would be very noble of google to work with the community in understanding the kinds of limitations their facing, the tools they need, and the ways that xmpp can help meet those needs.

Last Saturday we did another 6 hour startup, this time the goal was to create a game trading site that used the xmpp as the backend for storing data about who you're friends are. This was probably a little too ambitious and the result is at gamewrangle.com. We had a distinct lack of designers present and it shows. That being said we actually accomplished a good deal of work, I think everyone was pretty pumped about the project and it "works".

The basic idea is pretty simple.

1. you're a gamer, you have games you don't play anymore, there are games you haven't played and want to.
2. you have friends who are gamers as well in the same situation.
3. these friends are in your messenger buddy list
4. by alerting you to the existence of who has and wants what, we can facilitate trading games between you and your friends.

The mechanism is simple, just add a list of the games you are willing to trade, and a list of games you want, and we go and find the "matches".

The business plan involves using the rich data from game entry to display targeted advertising or amazon referral links.

I learned a couple of key things, mostly regarding the use of my own jabber libraries, and how important it will be to get the jabber devs to understand how powerful an authorization workflow would be.

Today I finished the first working xmpp-psn client, it's a small ruby module, and will let you talk http to a running xmpp-psn server anywhere to do xmpp/jabber. I don't have it in subversion yet, but it should be there soon as I clean up a few things and finish some testing. In all the performance looks to be pretty good, and it works :-D

A python client should be coming soon!

As most people around me know, I've been working pretty hard on disseminating the idea of using jabber as a backbone for a truly open social network, as well as attempting to build a set of libraries exposing a simple web api to provide for this.

The two web frameworks I'm most comfortable with are Ruby on Rails and Django (and PHP but I'm not ready to back down that road just yet). So I've been moving forward building some simple web applications in these frameworks, and in general I'm calling the project xmpp-psn short for Extensible Messaging and Presence Protocol - Portable Social Network (I know that's a terrible marketing name, and a mouthful, but I'm focussed on building the tools more than thinking up good names, if you have suggestions... email me!).

Well, Friday talking to Blaine Cook, he brought up a point I had yet to consider and lends itself to a rather tricky problem. I've been writing these tools as if they were there only entry point to the app, of course this means that in order to scale the app you have to put it on better and better hardware, in particular it means given the way that most web servers work (spawning a new instance of the ruby/python interpreter through fastcgi or mod_*) that it can't run on any multiprocess or threaded web server.

So what's the solution?

I think for now I'm going to ignore it. There are solutions that exist drb or dynamic ruby offers a messaging type solution (much like how erlang deals with concurrency) a solution for messaging through apache sessions that I've forgotten the link to and more. The problem with all of these is that the would require a) a lot of restructuring of code I already have written, and b) a lot of time spent thinking about how to do it. So, considering how small my target audience is at the moment, and that I'm more concerned with showing people what can be done, I'm just going to let it slide. Hopefully someone will come along and help me work out that kink when activity picks up.

Today I finished the first version of a Django app that lets you build a social networking site on top of Jabber (XMPP). You can find out more about the basics of the project here, but basically it's an http layer on top of a common python XMPP library that lets you do most of the roster management available through XMPP by way of a simple restful api.

The workflow for using it is simple, just install the app into an existing django project, and that will expose the api. From there you can post your jid and password to the /login/ url which returns a uuid and a roster in JSON format for your web app to digest. (note: this is horrible security practice, don't do this in the wild) Now that you have a uuid you have access to the rest of the api, which links to a thread on the sever that keep track of your jabber session.

the following is a list of the supported roster functions:

• disconnect
• roster
• subscribe
• unsubscribe
• authorize
• unauthorize
• remove
• login

I've been spending sometime thinking about one of my primary concerns with social networking tools today, the non-portability of the data you put into them. That is, you spend a great deal of time on any given social networking service discovering, and managing your social network, yet all the work that's done there is in general trapped on that service.

There are some services that have taken a more socially conscious attitude and allow you to export that data. From the look of Leah Culvers blog pownce (and by association probably digg) will be providing a service to manage your social graphs, Geni has a feature that allows you to export the family tree you've created and so on.

The solutions I've seen thus far are varied and there appears to be a lot of work going on in this space to come up with a solutions that works for service providers (facebook, myspace, pownce) as well as users of these services (my friends). Most of what I'm seeing is a kind of mashup of XFN, microformats, openID and scraping of sites. (The pownce solutions is actually a lot better than this in that it relies on a two way communication between the given services providing an actual api for the service to tie into). Other solutions have involved XML exports, or FOAF, but in the end all of these services rely on a static technology(XML) and a web service to manage the source of that XML document that services such as myspace and facebook could tie into.

I feel like these types of solutions are only stop gap measures for a type of problem that begs for a decentralized dynamic approach.

The approach I've been taking is too take a look at what some of the other protocols available on the web already provide for us. In particular I think that most of what people are looking for in terms of a tool to collect and provide social graph data is cleverly handled by XMPP (jabber).

To better understand where I'm coming from here I want to lay out some of the basic problems we're trying to tackle with portable social networking. As I see them they are:

1. a unique identification system: you have to have some way of uniquely identifying members of your social graph. This could be via openID or another uri type structure, or how email/xmpp does it with unique names per domain.

2. an standardized format for defining relationships: XFN, FOAF, or just a list of friends will do. But people need to know what kind of data they're getting back. Looking at how most providers deal with friend data, the most important datum to know is who you recognize as being a "friend"

3. a simple safe web api for requesting this data for use on other networks.

After one tackles those three primary problems the secondary problems are things like:

1. The system should be distributed: This will allow for the easy use and scaling of the system.

2. The system should be defined in a clear open standard: We don't want people to be tied to a particular language or implementation.

given that problem definition XMPP provides for nearly all of those and more. So how does it work?

We use XMPP's existing network, Jabber, to provide a decentralized set of servers for us to use. Each user on the jabber network has a unique id provided of the form xxx@domain.tld, authentication is over SASL and takes place in direct communication to the users jabber server from the client.

The xmpp term for a social graph is a "roster" or buddy list. The roster provides a list of people with whom you have a trust relationship. By requesting a roster from an xmpp server you receive an xml file with a list of trusted users.

So, we as a social networking provider, ask for jabber authentication information at signup, those details are used to request a roster from their xmpp server, which we then use to form a starting social network on our service. Add a friend links or remove a friend links make a call to the xmpp server to remove or add the unique id's to and from their roster, this can either make direct calls to the webservice api or that can be done on the backend. In this way services are easily able to deploy their own xmpp gateway and yet any work done on their network is directly translated into changes in the jabber roster.

What excites me about this solution is that it actually gives something back to the network providers that they aren't getting now, for free. It gives them chat and presence! not only that but it allows users to manage their friend networks in the ways that they are comfortable with now, either through their current network providers (facebook, myspace), or through their im client, etc.