‘new’ retweets

Twitter re-invented the community-bred retweet function over a year ago. The reasons for this are debatable and people tend to have personal preferences on the issue. Personally I think they did it for two reasons: To improve data integrity, and to lower the barriers for participation for new users. Twitter are making bold moves to conquer the ‘mainstream’ and making it easier to contribute content is important – Look no further than Facebook’s like button, to understand that.

Regardless of your personal preference, the fact is that Twitter don’t have control over every desktop and mobile client out there. (well, not yet). The problem for Twitter is how to ensure that data is put into their system the way they want it.

Personally, I’m old school

As a Twitter user (not as a developer) I prefer the old style RT. Why? Because I am selfish and vain. I want to add my own comment to the content I am forwarding on. I’m not sharing it with you entirely for your own good (selfish) – I want you to know what I think about it too. I also want to be notified when I’m retweeted (vain), and I want to know what you think about it too.

I also think the old style RT is an important part of the Twitter subculture, something that I feel will die out if Twitter truly conquer the mainstream.

‘new’ twitter

The new twitter.com website has no UI elements to support the old style RT. (For the record, the old site doesn’t either). Performing an old style RT is made hard – or at least deliberately omitted as a feature.

The official iPhone client has a “Quote tweet” button, possibly a throw back to its independence as Tweetie 2. I’m interested to see whether Twitter add a “Quote tweet” button to twitter.com, or whether they remove it from the iPhone client.

I see no reason why users’ preferences can’t be facilitated, as well has Twitter’s own requirements. Perhaps the retweet model will shift again.

‘RT’ button

So, if you’re using the new Twitter and finding yourself annoyed that the UI is telling you how to use the service, then try out my button.

It is a JavaScript bookmarklet. This means it runs JavaScript code on the page you are viewing when you click it in your toolbar. It loads a script from an external source, so I can update that without you having to reinstall the button.

Be wary of installing JavaScript bookmarklets from untrusted sources. If you want to examine the uncompressed source code, it appears below -

–> Dictionary.com <– drag to toolbar

Dictionary.com provide an ‘official’ bookmarklet, here:
http://dictionary.reference.com/tools/bookmarklets.html

I’ve improved it a bit

• Selected text uses window.getSelection(), not document.getSelection()
• Trims junk out of current selection
• Opens a new window so you don’t lose the current page
• Uses encodeURIComponent(), not escape()

Here’s the uncompressed source on Github

I’ve been having fun playing with node.js over the past year, but have had little, or no excuse to use it in any production work, so I thought I’d set myself a challenge and build a module. That challenge was firstly to create a simple AMF gateway for Flash remoting, and secondarily to see if an RTMP socket server was achievable in node.

If you don’t know about “node” – It’s a JavaScript runtime that allows you to write socket servers. I like it a lot – it brings asynchronous, event-driven programming to the server side and provides a truly global variable scope across all connections. I’ll blog about it in more detail later, perhaps.

At Public we do a lot of Flash work, and regularly implement Flash remoting using a PHP AMF gateway. I wasn’t necessarily looking to replace this stock approach with node, but node offers proper socket connections that PHP can’t, so I was imagining the possibilities of using node as a free, and more flexible alternative to Flash Media Server. Not for streaming media, but for real-time messaging, for example in multi-player games. If I’m honest though, I did this mostly for fun, an academic exercise and as an excuse to work with node.

node-amf

node-amf is on GitHub as a public repository: http://github.com/timwhitlock/node-amf

The first step was to write a pure JavaScript AMF implementation. i.e. a library for serializing and deserializing AMF packets. If you’re not familiar with AMF (Action Message Format) – it’s a binary serialization and messaging format invented by Adobe for passing data back and forth between Flash and a remote server. I am not the only person to implement this in JavaScript: mid-way through my project I noticed amf.js appear – just a little too late for me. This library is probably much better than my own, but mine is designed specifically for node, so I shall stick with it for now.

With a working AMF library all that remained was to create a HTTP gateway. There are many examples of AMF gateways in other languages, including PHP. The general approach is that a single request/response exchange carries one or more messages, each one calling a web service on the server and returning the result. AMF also has the ability to invoke methods in the Flash client on response – an underused feature.

Node makes it incredibly easy to implement an HTTP server, all that I had to do was decide how to expose the web services to the gateway. I decided to do this by passing a user-defined JavaScript object when initializing the server – each property of the object is a function callable by name – example AMF gateway here. This suitably sandboxes the method calls, and ensures the client cannot execute arbitrary JavaScript functions – that would be bad.

This part of the project is largely complete and working. It has not been used in production yet, so if you’re brave enough to use it, please let me know how it’s going!

node-rtmp

node-rtmp is currently under the same project as node-amf because it’s dependant upon the AMF library. This incomplete, highly unstable, and experimental work is under the node-rtmp subdirectory with examples here.

If you’re not familiar with RTMP (Real Time Messaging Protocol) it provides bi-directional streaming media and messaging over persistent socket connections. It’s the messaging I’m interested in. Flash talks to Flash Media Server over RTMP sockets and can securely call methods in both directions.

As it turns out the RTMP specification is very badly written. (The AMF specs were quite easy to read, although possibly much simpler). If I was a cynic, I may even suggest that Adobe has purposefully written them badly to avoid third party developers being able to adhere to the usage license which insists in adequate conformance to the spec. However, it doesn’t seem to have stopped numerous projects such as librtmp from gluing the pieces together.

This is the point where I realised I might be out of my depth. I am currently in the process of glueing the pieces together myself. Armed with the dodgy specification, Wireshark, and the fragmented information dotted around the Internet, I am making some slow progress. At the time of writing I have the RTMP handshake working correctly, and have just about deciphered the command messaging packets.

I may give up before this is at all useful.

librtmp and node add-ons

As you may have noticed, I’m talking about implementing all of this in pure JavaScript, and you may be thinking that’s nuts. You are probably right. As libraries, these don’t really need to be written in JavaScript. In addition to pure JavaScript modules, node supports add-ons – compiled libraries which can be built against C and C++ system libraries. A good example of this is node-memcache.

C and C++ is unfortunately outside my skill set, but I imagine it may be possible to build an RTMP node add-on using librtmp. Anyone fancy a crack at that?

If you want to zoom a map to fit a particular distance as tightly as possible, you need to know the scale of each of the 19 levels. In metres per pixel, I worked them out to be as follows:

zoom m/px
  19  0.19
  18  0.37
  17  0.74
  16  1.48
  15  3
  14  6
  13  12
  12  24
  11  48
  10  95
   9  190
   8  378
   7  752
   6  1,485
   5  2,909
   4  5,540
   3  10,064
   2  16,355
   1  21,282

So if you needed to calculate the zoom level to fit an area of a given radius you could use a function like this:

function radiusToZoom( r ){
    var w = myMapInstance.getSize().width;
    var d = r * 2;
    var zooms = [,21282,16355,10064,5540,2909,1485,752,378,190,95,48,24,12,6,3,1.48,0.74,0.37,0.19];
    var z = 20, m;
    while( zooms[--z] ){
        m = zooms[z] * w;
        if( d < m ){
            break;
        }
    }
    return z;
}

• Download jParser 1.0.0 (recommended)
  
• Download jParser devel package (Full source and build scripts)
• See the library examples running at apps.timwhitlock.info/jparser

The library has been split in two:

1. jTokenizer – A JavaScript tokenizer designed to mimic the PHP tokenizer.
2. jParser - The fully blown JavaScript syntactical parser which generates a parse tree.

The reason for the split is that for most purposes where you think you need a parser, you in fact just need a tokenizer. The tokenizer library is about 15KB, whereas the parser is over 700KB (minified), so you can see why you might not want to include it unnecessarily.

The library files jparser.php and jtokenizer.php are self-contained, minified files for production use. If you wish to inspect or modify the code you will need to download the devel package. This package provides a build script which collates the libraries into their distributable files.

jTokenizer

Possible uses for the tokenizer include code highlighting and simple manipulation of JavaScript source code.

The main function you will want to use is j_token_get_all which behaves the same as the PHP token_get_all function with the addition of a column number as well as a line number. Additionally there is the j_token_name as per the PHP token_name function.

jParser

This is a full, syntactical parser. On its own it simply generates a parse tree which can be traversed and manipulated. There is no proper documentation on this yet, but take a look at the node classes in the devel package if you are serious about doing something useful with this parser.

Some other notes in no particular order

The full parser uses a lot of juice. I recommend giving PHP loads of memory, and be careful what you throw at it if you’re going to run it on a production server.

A parser is not an interpreter or a JavaScript engine. If you want to develop such a thing in PHP you might be insane, but it could be done with this parser as a base.

The JParser parse tree is purposefully not a full tree, it collapses redundant nodes to save memory. If you want to see a full tree then take a look at the JParserRaw class. (devel package required)

Splitting the parsing process into two parts (tokenize/parse) is probably not the most efficient and probably uses more memory than it would another way. However, I figured it would be neat to mimic the PHP tokenizer functionality so that parsers could be built that take a stream of PHP tokens.

> http://www.webdevgeekly.com/r/www.webdevgeekly.com/mp3/geekly6.mp3

I should clarify a few things for anyone listening:

1. I did get a rude email from Douglas Crockford, but not nasty or in any way aggressive. When I very politely asked for his opinion on the project he responded: “I don’t care for ActionScript“; No “hello”, no signature… Call me old fashioned, but I think that’s unnecessarily rude.

2. “I wonder how they’re doing parts of this” (Harmony and ECMA4 gets mentioned)
As any JS evangelist will tell you, Adobe went off and did pretty much their own thing with AS2/3; JAS syntax follows that path as opposed to the current path of ECMAScript. The ‘compiler’ is implemented in pure PHP. You can test the JAS Parser on its own here, or test the full compiler at jaspa.org.uk. Rather than a JavaScript engine, or interpreter like SpiderMonkey that might run in a browser, the JASPA compiler is really a converter which allows you to deploy regular JavaScript to browsers that have no idea how to interpret JAS, or any other super-set of ECMAScript for that matter.

3. Providing libraries that ActionScript developers are used to – e.g. jaspa.net.URLVariables
I do attempt to follow AS conventions where possible. The URLVariables class is a good example, because it is API agnostic. That is to say that it is a rather abstract utility that could be used in DOM scripting, but equally in any other API, such as server side JavaScript for Flash Media Server for example. It is important to note that the JASPA compiler and JAS syntax are not in any may married to these libaries and APIs. It is platform for which anyone is free to develop their own such utilities.

4. Finally to anyone who wants to try out JASPA, please bear in mind that it is a non-commercial project with a single developer who has a full-time job. The “they” is really a “he”. The compiler itself is a beta, and some of the API classes I’ve been putting together are somewhat alpha.

↓ JavaScript grammar file for jParser

This file is in a (probably non-standard) variant of BNF notation. I’m not generating the tables with a tool like ANTLR primarily because I don’t write C. I should also point out that I don’t come from a formalized computer science background, so don’t expect this to be 100% conventional.

I’ve developed a native PHP parse table generator that in turn uses a parser (based on itself) to parse this BNF grammar into a table. If you understand grammar files like this you’ll notice something a bit odd – The terminal symbols don’t go right down to individual characters, the grammar expects the source code to already have been tokenized into significant chunks, such as J_NUMBER representing an already identified numeric token. This was done deliberately to be compatibile with PHP’s own Tokenizer functions. The underlying parser framework was designed such that PHP token based grammars could also be developed.

I’ve had to disappoint so far, because it’s not currently available for download. My web analytics also tells me that a lot of people are finding this blog by Googling “php javascript parser“. There’s clearly a need out there, so I’d better do something about it.

I know why I want a JavaScript parser, but what kind of things might you need it for? What API features/functions would you like to see? Please post a comment and let me know. Watch this space for a release, and in the mean time here’s some food for thought …

Do you really need a full parser?

Perhaps all you really need is a tokenizer. Suppose you want to highlight JavaScript code – you don’t need a full parser; all you need to do is break the source code into meaningful chunks (strings, numbers etc..) . How these chunks work together to form syntax is not an issue. I envisage releasing a lighter-weight library containing just the tokenizer (jTokenizer I guess). This will work exactly the same as the PHP tokenizer functions.

Minifying, or obfuscating JavaScript

If you’re after a minifier, there are lighter-weight PHP options available. However, I’ve released a command line version of my JavaScript minifier and obfuscator which is quite handy. It’s more powerful than most JavaScript minifiers, because it supports automatic semicolon insertion. This means it can reduce any code to a single line.

Sadly the people I feel need to appreciate his point the most probably won’t get past the second paragraph. That point, or at least the one I took away from it, is best summarized by this quote:

“[...] as we have higher and higher level programming tools with better and better abstractions, becoming a proficient programmer is getting harder and harder.”

Bang on. Except, maybe not harder, but just less likely. In the [invariably rapid] world of creative web development this is known as “getting shit done”. Young developers in my business don’t learn JavaScript; they learn jQuery. They don’t learn PHP, they learn [ahem] WordPress. (as if PHP wasn’t already a high level abstraction). Now don’t get me wrong! I’m not pleased about this, but there is a happy medium to be found somewhere, and I don’t believe we’ve all settled on it yet.

Joel is pretty hard on the idea that abstraction saves us time, and seems to argue that on balance the time saved working is still spent learning. Maybe this is true of people who do the learning, but many do not. And sometimes it just isn’t practical. The ActionScript world seems to be dominated at the moment by PaperVision 3D. Obviously you should have buttoned up your AS3 before diving into a library like this, but as for a solid understanding of the 3D engine; what are you going to do? Pop out for a quick doctorate in 3D mathematics in your lunch break? There is not doubt that this huge abstraction empowers many developers with some real might, with some very real time saving.

I have always maintained that the more you understand about what lies beneath the abstraction layer in which you are working the better you will be at your job. Would you train someone on Dreamweaver, before they proved they could write HTML? However, there is a balance to be found that works for you and your business. Web development would be impossible without the many layers that exist between the code in an HTML page and the OS of the server dishing it up. As the web evolves more layers tend to get added, and nowadays popular code libraries are adding layers within layers.

At the bare minimum you have this: (please excuse rudimentary linear format, and refrain from posting flowcharts)
HTML < Browser < HTTP < Server

Then we add a wizzy AJAX front end and some back end architecture:
JavaScript < HTML < Browser < HTTP < Server < PHP < MySQL

Then we use some cool 3rd party libraries to save us some work:
jQuery plugins < jQuery < JavaScript < HTML < Browser < HTTP < Server < Drupal modules < Drupal < PHP < MySQL

The more layers, the more points for possible failure. If you only have knowledge (i.e. control) of one or two of these points you are in a weaker position when things go wrong. If you have knowledge of all these layers you are in a much stronger position to harness the value of the extra layers, rather than suffer from them. This kind of philosophy is rarely evaluated when people make business decisions off the back of a Drupal press-release, but it’s one that I ensure I keep an eye on, especially when recruiting.

This area is of particular interest to me at the moment, so expect more posts on the topic.

I just read this blog article by John Resig, the author of jQuery. The article in general is a criticism of various projects like my very own “JASPA” – A criticism of frameworks that attempt to bring JavaScript into the 21st century by abstracting it into a different language.

I knew I wasn’t alone with JASPA – There is GWT and Jangaroo for Java, Pyjamas for Python, and Cappuccino with its own syntax called Objective-J. These projects are all slightly different takes on one core principal – that to truly make JavaScript “better” you can’t just write JavaScript, you have to introduce a more advanced syntax to leverage more advanced features.

Understandably JavaScript evangelists like John don’t all like this idea, and I do understand why. John makes some good points; most of which I completely agree with, but also some that I feel apply equally to his own jQuery and libraries like it.

Native JavaScript vs foreign language syntax

Pure-JavaScript libraries like jQuery, Prototype, Base2, (the list is endless) all attempt to address the short-comings of JavaScript from the inside, so to speak. That is to say that they are implemented in JavaScript and can never escape JavaScript. John argues that this is preferable, but I argue that the way in which these kind of libraries attempt to bend the syntax of JavaScript make them guilty of most of the same charges. Furthermore I argue that these libraries stand in the way of someone new to JavaScript gaining any real understanding of it.

My main defence of JASPA is of course that the language syntax is that of ActionScript (AS3), which regardless of its rocky relationship with ECMAScript 4, is still an evolution of JavaScript, and not a total departure from it, which is what John Resig seems to be most offended by.

So naturally I posted a comment about my feelings..

.. except not only did I manage to cock up the posting of the comment, but for all I know it will be taken down to punish me for my shameless self-promotion. So because of these reasons, and because I feel I made some good points, I thought I’d re-post my comment here with a tiny bit of editing.

--8<-------
I see your point when it comes to non-ECMAScript languages, but what about ActionScript? I chose AS3 to implement my “abstraction” called JASPA – jaspa.org.uk

I know many JavaScript evangelists reject ActionScript, (especially due to its recent history with ES4), but it’s like writing an extended version of JavaScript rather than a whole different language.

The OOP enhancements in ActionScript are intuitive and the syntax for procedural code remains largely the same. I would in fact argue that the OOP enhancements in ActionScript are much more intuitive than constructs like this:

$(document).ready(function(){
   // ...
});

- or sugaring approaches like this:

Class.method('example', function(){
   // ...
});

I appreciate what these kind of constructs achieve, but I cannot agree that they are particularly intuitive or respectful to the natural form of JavaScript. I would go further and suggest that people who aren’t JavaScript experts may not see how these formations really relate to JavaScript syntax, particularly due to the use of whitespace.

There some libraries that convert functions to strings and overwrite them with altered versions – surely this approach is even less intuitive and has even less respect for JavaScript than an extended form which provides real OOP enhancements such as classes, packages, imports, and super; not to mention strict typing.

Anyhow, I would ask that anyone looking at JASPA realises that what I am doing is purely out of respect for what JavaScript is and what it isn’t.