One comment usually brings the issue home quite nicely though. I ask:

“How often to you mention being drunk, or being hungover on your Facebook wall?”

- the answer is invariably “often”

“What if in five years you can’t get life insurance because you’ve been profiled as a high risk for alcohol-related illness?”

Should I be paranoid?

Long version coming up

I got a Debenhams store card the other week. I was asked the usual credit-check questions, but after the first dozen I got suspicious. I asked the lady if these questions were ‘optional’; she confirmed that they were, so I opted out.

We’ve become more-or-less comfortable with the concept of credit checks – they’re a necessity of our financial lives. But these optional questions were pushing the boundary a bit, they seemed like more general demographic profiling questions. I suspect that my card was approved as soon as they had my name and address.

Most people are aware (I hope) of how our purchasing habits (through things like Nectar) can be used to build up a profile of us as consumers. It all seems pretty harmless when you get a voucher for tiramisu through your door, but what if marketing data affected more important things in your life than free cake?

Insurance profiling

A recent article in Wired led me to an experiment by Deloitte Consulting. The experiment was a predictive modelling approach to the usual methods of Aviva’s life insurance underwriting. 37% of the model’s predictive ability came from consumer-marketing data. Think about that for a moment — not just facts in your medical history, but what you like to do at weekends. What we’re talking about here is extrapolation of the probability of your death.

Not only does this concept scare the crap out of me, but I immediately think about all the data I’ve pumped into Facebook and Twitter over the years. Extracting value from this kind of noisy data is clearly a hot area for startups too. Some firms (like Google-acquired Fflick) are using machine learning to turn status updates into discernible data. Once this technology is more reliable, the face of consumer profiling will take a serious leap.

The direct marketing division of Equifax was acquired last year by Alliance Data Systems for $117 million. (that’s 0.2% of Facebook’s valuation). We already know the value Facebook can bring to targeted advertising. Just imagine if Facebook entered the profiling market at this level. The ubiquitous Like button even gives Facebook the potential ability to know what other sites you visit. You may not even need to tell them you prefer the FT to the Daily Mail – they may already know.

An obvious caveat to Facebook’s profiling ability is that your Facebook isn’t necessarily tied directly to your legal identity, or exact postal address. I am merely “Tim W” and they don’t have my mobile number. However, I imagine I’m in the minority here – Facebook are very aggressive in farming this data. They even use an irritating Captcha to coerce you into ‘verifying’ your account.

How long before you can enter a person’s name and address into a system and get back a quantified likelihood of that person crashing a car, getting arrested, dying of liver failure, or skiing off a cliff?

Datasift (a new product by Tweetmeme, currently in rather exclusive alpha) offers sentiment analysis as part of their streaming filters for Twitter.

Valley-based Fflick are developing their own sentiment engine via machine learning algorithms. The current manifestation of this is a movie review site, but they will be pursuing other verticals – no doubt once the tech has improved and they’ve got some $$s.

Qwiki, which I wrote about yesterday, appears to be on the artificial intelligence trail too. The task of establishing whether content is relevant/important/canonical is an incredibly daunting task to automate.

Finally [prompting this post] this morning I see a product launched by Lewis PR: Chatterscope monitors brand mentions and performs sentiment analysis – A free alternative to Radian6 and Alterian, perhaps? Monitoring and alert functionality is obviously useful, but sentiment analysis – that’s the marketing holy grail, and I’ve always been sceptical.

It’s complicated

Chatterscope’s sentiment engine appears to be little more than keyword proximity matched against a list.  I’m not going to berate them, because if a PR agency came to me with this brief and a few thousand pounds, they’d end up with something very similar. It’s a very hard problem to solve – personally I’m not sure I’d attempt to solve it.

Last Christmas I wrote a simple app (called the XmasFactor) that identified who [on Twitter] was in a Christmassy mood. It was very obvious, very quickly, that it was going to be hugely flawed.  Just mentioning “jingle bells” does not imply sentiment – “Jesus, I wish they would stop singing jingle bells”. It was only a bit of fun, but that won’t stop people calling you out when something doesn’t work.

The complexity of language is such that words alone do not always indicate sentiment accurately. In the first instance, you need to parse a sentence to infer meaning into the mere words. We’re talking about a pretty big margin of error here. Consider -

• Grammar: “Far from the best phone on the market”
• Sarcasm: “No reception in here. great, thanks Orange”
• Context: “Great big scratch on my iPhone”
• Slang: “The new iPhone is bad ass”

It’s clear that these problems require human-like thinking – the kind of thinking that comes from learning. Continual learning, because human culture itself is a moving target. Fflick have people with PhDs in artificial intelligence developing learning machines. I consider myself a good developer, but this kind of stuff is so far out of my league, it makes my eyes water.

Intelligence

There are numerous products and software libraries that provide the natural language processing required for this kind of analysis, so it’s not entirely beyond the reach of the rest of us. However, the fact that a company like Fflick is developing its own technology, suggests these tools have a long way to go, and there’s a lot up for grabs.

Companies tend to be shady about what third party software they use, because [I'd imagine] this knowledge may expose weaknesses in their products, as well as making life easier for their competitors. It appears common for firms to say “we’ve developed our own algorithm” — The allure of the ‘trade secret’.

Nick Halstead of Tweetmeme [when asked at DevNest]  wouldn’t say what sentiment engine was employed in Datasift, but alluded to some third party software being used. (anecdotal, sorry, but I was there).

[Update: Datasift tweeted that they use Lexalytics]

I am told that Alterian developed their own engine, and employ some human-power, i.e. some manual analysis. Radian6′s algorithm was apparently built in-house, whatever that means. And their dashboard seems to provide some sentiment tuning using a human-powered thumbs up/down approach. (I’ve not used either product).

Fflick also use some human-power via Amazon’s Mechanical Turk, although this is only to teach the machines, not to process the live data. See this video for some great insights into Fflick’s technology.

Good luck to anyone working in this field. Personally, I’m more interested in harnessing people power. Social networks give us this in abundance, and until artificial intelligence is truly accessible to me as a developer, I’m steering clear of it. Moreover, I’m not writing any cheques that my code can’t cash.

Last night on Twitter I referred to the fact I was drinking wine. Not interesting, but true. During the following few hours I had four new followers that all appeared to be in some way concerned with wine; whether they were selling wine, writing about wine .. whatever. This is nothing new, but it’s definitely on the increase. If you look critically at your Twitter followers you’ll probably realise that a hell of a lot of them are in some way following you in order to (please excuse my non-marketing background) sell you stuff. Whether they are humans or ‘bots’, they are profiling you.

I use Twitter to promote myself, (many people do), but in general it’s a mutual exchange between like-minded people. Having hundreds of companies literally monitoring you to see if you mention things that could lead to a sale is pretty horrific in my opinion. The culture of Twitter is different to that of Facebook et al, because ‘friends’ don’t have to be mutual, and following people you don’t know is quite acceptable. However, I think Twitter, or the applications through which we use Twitter are going to have to change, because this is only going to get worse. (Spam 2.0?)

Is blocking actually effective? Your tweets are still public, indexable content. You enter into that by signing up to a service like Twitter. I concede by tweeting, or blogging that my words are more-or-less public property. However, the Twitter API makes the marketeers’ jobs much easier, they can neatly compartmentalize you. If it makes you as uncomfortable as it does me, block them! #dmblock

There’s too much ambiguity and room for misconception in a single headline. Online-reading habits being what they are, if you’re not interested in the headline, you won’t read what’s under it. I suppose this is why people use rather abstract strap-lines like jQuery’s “Write less, do more” and “jQuery is a new kind of JavaScript library“.

I don’t think JASPA can get way with statements this abstract, so I’m currently running with:

“Write JavaScript with the expressive power of ActionScript”

Not catchy, I know; needs work. I’ve also put a real-time compiler demo on the JASPA home page to try and get the message across. I may regret this as the compiler uses a lot of juice. Good job it’s not popular