These are all click-throughs from tweets. They are all referred by Twitter’s URL shortener (t.co). Unfortunately this data is fairly useless. I’d like to know who the influential tweeters are and what the tweets said. I could search Twitter for the full URLs, but Twitter search is – quite frankly – rubbish for this.

My initial reaction to this vague and seemingly obfuscated data was annoyance, but after doing some research I now see that this data is actually a bonus. That doesn’t mean it can’t be improved though.

How referrer tracking normally works

If the request to t.co contains a ‘Referer’ (sic) header, Twitter responds with a regular HTTP redirect. This type of redirect keeps the original referrer intact and so that is what goes into your web stats. This is what we’d like ideally like to see – to be able to click through to the actual tweet.

URL shorteners have their critics, but shorteners don’t in themselves prevent us seeing where our traffic originally came from.

BUT

There are circumstances under which your browser won’t send the original referrer to t.co when you click a link -

1. If you’re not clicking from an actual Web page. (e.g. from an app or Twitter client)
2. If you’re clicking from a secure web page (e.g. https://twitter.com/)

You’ve guessed it; this accounts for the huge majority of clicks from Tweets. Possibly all of them.

So why are we seeing t.co at all?

Without the referrer header, your stats would show these clicks as ‘direct traffic’. The reason t.co shows up at all is because Twitter do something different to other shorteners such as bit.ly.

If there is no referrer in the request to t.co and the user agent is a normal browser (i.e. you’re a human), Twitter responds with an HTML document containing both a JavaScript redirect and a meta refresh fallback. The upshot of this is that an interim web page is inserted in between the referring source and your target page. In this case t.co ends up in your web stats instead of the source.

A purist might argue that this breaks the web, but from an analytics perspective this is more useful than the clicks showing up as ‘direct traffic’. At least we know they’re from tweets.

Room for improvement

Obviously there are privacy issue with tracking users’ clicks, (hence the missing referrer in clicks over SSL) But for people with a public Twitter timeline it would be very handy if Twitter could provide richer data in t.co links beyond just where they link to. This could simply be X-Twitter-User-Id and/or X-Twitter-Tweet-Id headers in the response showing who originally shortened the URL and/or who is passing it on.

An API for this would be ideal, but of course Twitter have their own Analytics system, so I don’t expect they’ll want to make t.co any better for the non-paying customers.

Campaign tracking on the cheap

If you use Google Analytics and don’t pay for any fancy social media management tools, it’s very easy to embed Google Analytics campaign tracking tokens into your original URL.

This is the URL I use in my Twitblock tweet button:
http://twitblock.org/?utm_source=tco&utm_medium=tweet&utm_campaign=share

All traffic clicking on the shortened t.co links will show up in the Campaign section of your traffic sources. If you forget the names of the parameters you can use this handy link builder.

I wasn’t looking forward to trying them all out to decide which library to use, but as it turns out I didn’t have to. When I discovered wkhtmltopdf, my decision was made. (I know that sounds like a cheesy marketing testimonial, bare with me)

I only needed to convert HTML documents to PDF. i.e. I didn’t need to programmatically draw vectors, or anything like that. That’s a pretty important prerequisite. When you consider how good HTML and CSS are for laying out a document, it makes you wonder why you’d do it any other way. The catch is that your PDF generator needs to render your HTML as well as a browser does.

wkhtmltopdf is built on WebKit which means it does exactly that – it renders the PDF as well as a browser, because it basically is a browser. It even runs the JavaScript. The fact that other libraries have their own [possibly bespoke] rendering engines with huge limitations seems pretty crazy once you’ve seen wkhtmltopdf in action.

Anyway, I thought I’d share a few things that I encountered while getting this integrated to my PHP application.

PHP Bindings

Despite my initial excitement upon discovering  php-wkhtmltox (PHP extension for wkhtmltopdf and wkhtmltoimage), I ended up not using this extension. My main reasons were as follows:

• Rather opaque configuration options
• Clumsy when using with stdin and stdout

I decided to execute the wkhtmltopdf binary via the shell and use PHPs procopen and related functions to pipe in my HTML and grab my PDF without faffing around with temporary files. I’m unsure of the comparative overheads in using the shell versus using a PHP extension, but it works very well regardless.

Compiling from source

Don’t.

I’m not an expert sysadmin but I’ve compiled plenty of software on the Linux and Mac command ines and this did not go well. It requires first compiling the open source version of Quicktime, which is almost 1GB of source code and was building for over an hour before I hit Ctrl-C on my Mac. It failed first time on my Linux machines… basically I gave up.

Static binaries are available for both Linux and Mac, and they worked fine for me. I recommend saving yourself the pain and just using them, even if they’re not available for the latest version of wkhtmltopdf. (The Mac binary is a couple of versions old).

Fonts

On my Linux servers, after installing a whole bunch of X11 stuff I probably didn’t need and definitely didn’t understand, I discovered via the comments here that I needed to install the urw-fonts package. If you don’t have the fonts installed correctly you get black squares instead of glyphs.

JavaScript errors

wkhtmltopdf is WebKit, which means it executes JavaScript too. I was using the same HTML template to render PDFs as our ‘print’ page. This page calls window.print() to invoke the printer dialogue. As it turns out (and it took me some head-scratching) wkhtmlpdf dies when you call this function, and it does so without any decent explanation of the error.

Rather than muck around with my PHP template, I just passed --disable-javascript to wkhtmltopdf, and everything was fine.

It’s actually the 8th meetup, not the 10th which is a shame. But what the hell, it sounds good.

In line with the micro-sponsor model we ran for FF5, we’re inviting companies to sponsor a pint (or 10). Starting at £4, we’re placing their logo on our wall of sponsors. The bigger the pledge, the bigger the logo.

You can even embed the sponsor widget on your own blog, as I’m doing here. (Get the embed code).

So, if you’ve gone the extra mile for your clients this year (and not billed them for every minute of every phone call like a solicitor), then embed the widget, send them the link, and ask them kindly to buy you a pint.

Oh, and half the money raised will go to charity.

These are the companies who have sponsored so far.

His follow-up post describes Facebook’s response, but the ‘tracking’ cookie to which I was referring has not been removed. According to Nik’s post, Facebook admit this will remain after logout to track the browser, but for ‘safety and spam purposes’.

According to this WSJ article, ‘not all of the data is logged’. That’s good.

The bottom line for me is that Facebook are so powerful that they need to be as answerable to their populous as a government. That means a certain level of transparency and being clear about their intentions. If they go back on their word, who holds them accountable? Are our laws even adequate? Should Facebook be audited, or should we just trust them?

I don’t expect I’d be too happy about having my servers audited, but I’m not Facebook. When nearly half a billion people log into your site each day to give you their data, you have a serious amount of responsibility on your shoulders.

Appendix

As it happens, I couldn’t replicate Nik’s findings. He found that the user ID cookie was not deleted at log out and continued to be sent to Facebook. I can’t explain that; but regardless, my issue was with an anonymous tracking cookie that remains today.

Here’s a quick technical explanation of how this tracking would be possible.

The cookie I refer to is an anonymous identifier with the name datr. This is set when you visit facebook.com, regardless of logging in. Once you do log in, its value does not change. Crucially, when you log out the value does not change either. This means that subsequent Like button impressions could be associated with your account despite your user ID no longer being sent along with it. If the full dataset was stored it would be trivial to associate this anonymous browsing data with your account.

Wow.

This adds a “Health and Wellness” Life Event to your Timeline.
(Timeline is new Zuckspeak for Wall, and Wellness is American for, erm.. Health)

Why would you tell a company that sells data that you were ill? … seriously, why?

I’ve already written my theories about what markets Facebook could enter with this kind of data at their disposal. I wrote about it on my own blog, and later a much shorter piece for NMA.

Previously I wrote about the problem of extrapolating this data from ‘noise’ and how the technology to do that doesn’t seem to have arrived yet. A nearer-term solution would be to get people to voluntarily participate in medical history form-filling in order to to structure that data. The Timeline seems like the perfect vehicle.

I’m staggered to see this so quickly. It seems like a very bold addition and I’m wondering whether the appearance of Google+ has caused Facebook to act more hastily than usual. They’ve been nudging us for years, but they seem to have suddenly given us a rather big push. Some will revolt, but will it put a dent in their 800 million? Unlikely, but never say ‘never’.

“If you don’t like it, why don’t you quit?”

This morning I read Chris Applegate’s post about quitting Facebook. As much as I sympathise, I’m not quitting. Here’s why.

It’s ‘normal‘

I don’t want to be absent from Facebook any more than I want to be without a telephone. Quitting Facebook is – to me – opting out of a societal norm. There are serious limits to the practicality of this.

For all my nerdy ramblings about privacy and such, I want be normal as much as anyone. If the vast majority of society are on Facebook, then that’s where you’ll find me. If that makes me a sheep then fine – throw one at me.

In terms of the day-to-day, I don’t want to miss out on things such as event invitations, or holiday photos. When meeting new people, saying “look me up Facebook” has become almost as common as swapping phone numbers.

Internet ID

You can’t get a job without a fixed address and these days it’s almost impossible to sign up for anything without an email address. What’s next? What if not having a Facebook account was suspicious? If Facebook ends up becoming some sort of Internet ID (I’m not joking) then being absent could be problematic.

Facebook seems to be very keen on your real identity to the point that it’s against their terms of use to have multiple, personal accounts. Eric Schmidt has referred to Google+ as an identity service, but Facebook are many miles ahead.

You might be against such a concept, but if 800 million other people don’t agree with you, then how feasible is your opting out going to be?

Move to Google+?

The only purpose of Google+ at the moment appears to be keeping Facebook on their toes. We don’t really know what role Google+ will play in future – whether it will house Facebook refugees, or whether the two will coexist with differing, even complementary roles. All I know now is that it’s not a satisfactory alternative if I were to quit Facebook.

I’m in control

At least I assume I am.

If I don’t want to upload my baby photos to my Timeline, I don’t have to (and I won’t). Of course I don’t want to see ex girlfriends in my Timeline, but I won’t; I’ve already removed them. I don’t have to read the Guardian inside a frictionless Facebook app so everyone can see what I’m looking at, I’ll just go to their site (for now).

While I still have these choices I’m relatively happy and will simply manage my account as I see fit. If these actions somehow become inescapable, then perhaps I will think again, but for now I see no immediate reason to delete my account.

Another F8, another set of powerful new features, and yet I’m not surprised by any of them. I’m not saying I predicted them (I didn’t) but the announcements are typical of what I see as a clear pattern in Facebook’s evolution.

Without discussing specific features, the pattern of Facebook’s ‘progress’ typically adheres to these traits:

1. More connections;
2. Less friction;
3. Deeper integration.

That’s Facebook marketing language. I interpret these traits follows:

1. More data;
2. Less opportunity for users to think;
   
3. Placing Facebook at the centre of the Web.

Yes, I’m a cynic. Did you not know?

1. More connections, more data

You are the product. Increasing connections increases Facebook’s yield. The more demographic and behavioural data the company can collect, the more valuable and powerful they become. This is their business model. We know this.

I’m comfortable with the arrangement, because I feel as though I’m in control. However, the data from others is just getting too noisy for my liking. The new Graph connections (compounded by the following point) could result in realtime information overload. Is there any limit to what can be connected? “Tim W opened his fridge door”.

2. Less friction, less thinking

This is where Facebook excel. Their subtle UI changes over the years are quite ingenious. Adding your face next to a comment form below every item encourages us to contribute more. Increasing our engagement while increasing their yield. I bet it was a very successful tweak.

I noted today that the new Guardian app doesn’t have the standard permissions dialogue explaining that the app will post to your wall. Instead the Connect button says “Log in and add to Timeline“. The user will give less thought to what data they are publishing.

This could be considered good UX (and it is) but it’s also another nudge towards absolute complacency.

3. Facebook at the centre of the Web

Of course this is a figurative centre – What I mean is that Facebook are increasingly a platform on which everything else is merely a ‘social app’ (in Zuckspeak). Facebook define themselves as THE social layer of the Web, much as the Web is THE media layer of the Internet.

So, if everything on the web were social, then Facebook would be everything, right? Zuckerberg said at F8 that some things, such as Healthcare and Finance, would not be social “for a while”. That man has goals.

Prophesying aside, the fact that today major brands advertise their Facebook page on television rather than try to drive traffic to their own URLs demonstrates how effectively Facebook have manoeuvred themselves into this position over the past few years.

This is a powerful position to be in. Fortunately there is competition. Facebook will have to fight Google for the centre of the web. That’s a topic for another post. It’s going to be an interesting couple of years ahead.

I firmly believe that if any company can dethrone Facebook, that company is Google. But from what I’ve seen of Google+ so far, I can’t quite imagine a MySpace-style emigration happening just yet.

I’ve recently been asking Google+ fans to give me a good reason to use it. By far the most popular answer to my question is that it provides better privacy and filtering options.

Circles is great. It’s about as simple to use as it can be, but I don’t believe it’s anything like a game-changer.

Facebook already has friend lists

In terms of who you share with, Facebook has had ‘friend lists’ for some time. Not only does Facebook have these, but it also allows ad hoc exclusions for individual posts. I can share something to my ‘Family’ list, but exclude Mum if I choose to. As far as I can see, G+ doesn’t support this. Of course Google can add it in future, but my point is that the Circles concept is not a USP; it’s just centre-stage and has a better UI.

Nobody knows about friend lists anyway

Facebook friend lists appear to be a little-known, or at least little-used feature. This is possibly due to being somewhat hidden, and this in turn is possibly because Facebook don’t really like you to be particularly private.

Facebook have already reacted to G+ by surfacing these features through an improved UI. If they think they’re likely to lose users to G+ on account of Circles then they can bring Friend Lists into greater prominence. They’re in a strong position to do so; they already have the infrastructure and I doubt that most of their audience even know about G+ yet.

It’s complicated

I’ve used lists a lot. I have lists of people who live near me, lists of my closest friends, list of people I have professional connections with; even lists of people I follow on Twitter. Managing these lists is not just time consuming, but it’s much more complicated that it sounds. As soon as you opt to conduct yourself in this way you start seeing how difficult it really is.

Life is more complicated than putting people into neat little pigeon holes with labels on them. Have you ever tried to throw a party and only invite people from a certain corner of your life? There’s always a guest you have to invite for ‘political’ reasons, or a guest who won’t come unless some other person is there; or isn’t there. You probably like some of your work colleagues while thoroughly disliking others. It’s never as straight-forward as you’d hoped.

And then there’s the great ‘offline’ loophole. Maybe I want to share something with just the boys, but my brother is checking Facebook while sat next to his wife. All the privacy settings in the world can’t get around that.

So, I gave up. I only maintain a single list nowadays.

If something really needs to be hidden from people then you probably shouldn’t be writing it on the Internet anyway. (not that I take my own advice).

But Circles filters your Stream too.

This is a good feature. I prefer it to Facebook’s obscure Edgerank algorithm, or whatever mysterious methods they’re currently using to filter your News Feed. References to ‘top stories’  (or ‘highlights’ as they seemed to be called last week), or News Feed vs Most Recent feeds – it’s all rather confusing and I think Facebook need to sort it out.

The point is that they can sort it out. If Facebook users start adopting friend lists in greater numbers, then perhaps Facebook will support lists in the News Feed settings. Perhaps they’ll make these settings more prominent too, as they’re currently hidden at the bottom of an infinitely scrolling page, such that you have to hit the ‘End’ key to reach it.

All of this is solvable by Facebook, and what’s more they can learn from what the early adopters say about G+ before the majority of their user base are even aware that it exists.

Do people even want privacy?

This is the ultimate question in my mind. Social networks offer privacy features because enough of us demand them, but I’m not convinced that privacy is so important to the average Facebook user that they’d jump ship to improve it. If Facebook friend list adoption doesn’t increase after some UI changes, then perhaps it’ll be because people aren’t that interested in the feature, or have found it as complex as I have. Even if people say they want this functionality, will they actually use it?

I’d like to eat my own words on this front, but I see little evidence that orthodox privacy concerns are of much importance to ordinary users. Perhaps Google would do well to observe how teenagers appropriate their own privacy techniques regardless of the tools designed for them by adults.

Circles isn’t a killer feature

It’s not like Google aren’t innovative. In addition to their wide range of indispensable services, they will eventually have a the browser, the mobile and maybe even the desktop sewn up. But a social hub that ties all of this together is going to need a killer USP to start a Facebook exodus, and I don’t think we’ve seen it yet.

Whether Circles is better than Facebook’s privacy model or not, I think Google need to get better at coming up with social features we actually want, even before we know it ourselves. This is one of Facebook’s great strengths and I’m not sure Google are too great at it.

Well, no. Nobody seems particularly outraged; possibly because the site I’m alluding to is for sharing photographs of men without their consent. Tubecrush.net.

I don’t feel I need to list everything that’s hideous about this service. I hope that any intelligent person can see why the concept is morally broken, and legally questionable.

[Update]

Upon reflection, it isn’t the photographs which I find most alarming. As my commenters point out, the photographs are intended to be complimentary, and as they point out themselves: you have no right to privacy in public places. Plus, there are exceptions in data protection where photography is for artistic purposes.  (So that just leaves morals then).

I think the time and location is possibly more creepy as it implies you might be able to find this person again if you wished to do so. This is essentially surveillance data. Their legal disclaimers make no mention of time and location data. They do however indemnify themselves from any ‘damage’ arising from someone featured on the site being ‘communicated’ with. A clear acknowledgement that posting this kind of data is potentially dangerous.

Our goal with Brandfeed is to connect brands and the media more effectively. There are similar efforts on both sides of this fence, but our approach is to provide a centralised database of pressrooms with a journalists’ dashboard all under one roof. The idea as that our pressroom format will become familiar to journalists, so they always know where to go when the they need a media contact, logo asset, or hi-res photo of a CEO. Additionally we offer discovery and notification features so journalists can ‘follow’ the brands they’re interested in, receive content and suggestions in a low-noise environment that they control.

With Dan’s experience in brand communication strategy and my background in tech and creative, we think this is going to be pretty special.

We’re not ready to launch yet, but yesterday we went public. It’s great to be out; I no longer have to say “I’m working on this thing I can’t tell you about yet“. I’m a great believer in transparency and not keeping ideas to yourself, but that’s another blog post

We’ll be giving private previews of Brandfeed to industry friends very soon. After we iterate from that round of feedback we’ll be running a more public beta programme. If you’re a journalist, brand representative or PR you can register for the beta now.

Update – 06 Aug
We are now in a public beta phase.