Thoughts on the Netflix API Closure

A year ago Netflix announced that they were shuttering their public API: no new API keys or affiliates and no more support. Earlier this week they announced that the entire public API will be shutdown by November 2014.

This is interesting news and its been covered in various places already, including this good overview at Programmable Web. I find it  interesting because its the first time that I can recall an public API being so visibly switched out for a closed, private alternative. Netflix will still offer an API but only for a limited set of eight existing affiliates and (of course) their own applications. Private APIs have always existed and will continue to do so, but the trend to date has been about these being made public, rather than a move in the opposite direction.

It’s reasonable to consider if this might be the first of a new trend, or whether its just an outlier. Netflix have been reasonably forthcoming about their API design decisions so I expect many others will be reflecting on their decision and whether it would make sense for them.

But does it make sense at all?

If you read this article by Daniel Jacobson (Director of Engineering for the Netflix API) you can get more detail on the decision and some insight into their thought process. By closing the public API and focusing on a few affiliates Jacobson suggests that they are able to optimise the API to fit the needs of those specific consumers. The article suggests that a fine-grained resource-oriented API is excellent for supporting largely un-mediated use by a wide range of different consumers with a range of different use cases. In contrast an API that is optimised for fewer use cases and types of query may be able to offer better performance. An API with a smaller surface area will have lower maintenance overheads. Support overheads will also be lower because there’s few interactions to consider and a smaller user base making them.

That rationale is hard to argue with from either a technical or business perspective. If you have a small number of users driving most of your revenue and a long tail of users generating little or no revenue but with a high support code, it mostly makes sense to follow the revenue. I don’t buy all of the technical rationale though. It would be possible to support a mixture of resource types in the API, as well as a mixture of support and service level agreements. So I suspect the business drivers are the main rationale here. APIs have generally meant businesses giving up control, if Netflix are able to make this work then I would be surprised if more business don’t do the same eventually, as a means to regain that control.

But by withdrawing from any kind of public API Netflix are essentially admitting that they don’t see any further innovation happening around their API: what they’ve seen so far is everything they’re going to see. They’re not expecting a sudden new type of usage to drive revenue and users to the service. Or at least not enough to warrant maintaining a more generic API. If they felt that the community was growing, or building new and interesting applications that benefited their business, they’d keep the API open. By restricting it they’re admitting that closer integration with a small number of applications is a better investment. It’s a standard vertical integration move that gives them greater control over all user experience with their platform. It wouldn’t surprise me if they acquired some of these applications in the future.

However it all feels a bit short-sighted to me as they’re essentially withdrawing from the Web. They’re no longer going to be able to benefit from any of the network effects of having their API be a part of the wider web and remixable (within their Terms of Service) with other services and datasets. Innovation will be limited to just those companies they’re choosing to work with through an “experience” driven API. That feels like a bottleneck in the making.

It’s always possible to optimise a business and an API to support a limited set of interactions, but that type of close coupling inevitably results in less flexibility. Personally I’d be backing the Web.

RDF Data Access Options, or Isn’t HTTP already the API?

This is a follow-up to my blog post from yesterday about RDF and JSON. Ed Summers tweeted to say:

…your blog post suggests that an API for linked data is needed; isn’t http already the API?

I couldn’t answer that in 140 characters, so am writing this post to elaborate a little on the last section of my post in which I suggested that “there’s a big data access gulf between de-referencing URIs and performing SPARQL queries”. What exactly do I mean there? And why do I think that the Linked Data API helps?

Is Your Website Your API?

Most Linked Data presentations that discuss the publishing of data to the web typically run through the Linked Data principles. At point three we reach the recommend that:

“When someone looks up a URI, provide useful information, using the standards (RDF, SPARQL)

This has encourages us to create sites that consist of a mesh of interconnected resources described using RDF. We can “follow our nose” through those relationships to find more information.

This gives us two fundamental two data access options:

  • Resource Lookups: by dereferencing APIs we can obtain a (typically) complete description of a resource
  • Graph Traversal: following relationships and recursively de-referencing URIs to retrieve descriptions of related entities; this is (typically, not not necessarily) reconstituted into a graph on the client

However, if we take the “Your Website Is Your API” idea seriously, then we should be able to reflect all of the different points of interaction of that website as RDF, not just resource lookups (viewing a page) and graph traversal (clicking around).

As Tom Coates noted back in 2006 in “Native to a Web of Data“, good data-driven websites will have “list views and batch manipulation interfaces”. So we should be able to provide RDF views of those areas of functionality too. This gives us another kind of access option:

  • Listing: ability to retrieve lists/collections of things; navigation through those lists, e.g. by paging; and list manipulation, e.g. by filtering or sorting.

It’s possible to handle much of that by building some additional structure into your dataset, e.g. creating RDF Lists (or similar) of useful collections of resources. But if you bake this into your data then those views will potentially need to be re-evaluated every time the data changes. And even then there is still no way for a user to manipulate the views, e.g. to page or sort them.

So to achieve the most flexibility you need a more dynamic way of extracting and ordering portions of the underlying data. This is the role that SPARQL often fulfills, it provides some really useful ways to manipulate RDF graphs, and you can achieve far more with it than just extracting and manipulating lists of things.

SPARQL also supports another kind of access option that would otherwise require traversing some or all of the remote graph.

One example would be: “does this graph contain any foaf:name predicates?” or “does anything in this graph relate to”. These kinds of existence checks, as well as more complex graph pattern matching, also tend to be the domain of SPARQL queries. It’s more expressive and potentially more efficient to just use a query language for that kind of question. So this gives us a fourth option:

  • Existence Checks: ability to determine whether a particular structure is present in a graph

Interestingly though they are not often the kinds of questions that you can “ask” of a website. There’s no real correlation with typical web browsing features although searching comes close for simple existence check queries.

Where the Linked Data API fits in

So there are at least four kinds of data access option. I doubt whether its exhaustive, but its a useful starting point for discussion.

SPARQL can handle all of these options and more. The graph pattern matching features, and provision of four query types lets us perform any of these kinds of interaction. For example A common way of implementing Resource Lookups over a triple store is to use a DESCRIBE or a CONSTRUCT query.

However the problem, as I see it, is that when we resort to writing SPARQL graph patterns in order to request, say, a list of people, then we’ve kind of stepped around HTTP. We’re no longer specifying and refining our query by interacting with web resources via parameterised URLs, we’re tunnelling the request for what we want in a SPARQL query sent to an endpoint.

From a hypermedia perspective it would be much better if there were a way to be able to handle the “Listing” access option using something that was better integrated with HTTP. It also happens that this might actually be easier for the majority of web developers to get to grips with, because they no longer have to learn SPARQL.

This is what I meant by a “RESTful API” in yesterday’s blog post. In my mind, “Listing things” sits in between Resource Lookups and Existence Checks or complex pattern matching in terms of access options.

It’s precisely this role that the Linked Data API is intended to fulfil. It defines a way to dynamically generate lists of resources from an underlying RDF graph, along with ways to manipulate those collections of resources, e.g. by sorting and filtering. It’s possible to use it to define a number of useful list views for an RDF dataset that nicely complements the relationships present in the data. It’s actually defined in terms of executing SPARQL queries over that graph, but this isn’t obvious to the end user.

These features are supplemented with the definition of simple XML and JSON formats, to supplement the RDF serializations that it supports. This is really intended to encourage adoption by making it easier to process the data using non RDF tools.

So, Isn’t HTTP the API?

Which brings me to the answer to Ed’s question: isn’t HTTP the API we need? The answer is yes, but we need more than just HTTP, we also need well defined media-types.

Mike Amundsen has created a nice categorisation of media types and a description of different types of factors they contain: H Factor.

Section of Fielding’s dissertation explains that:

Control data defines the purpose of a message between components, such as the action being requested or the meaning of a response. It is also used to parameterize requests and override the default behavior of some connecting elements.

As it stands today neither RDF nor the Linked Data API specification ticks all of the the HFactor boxes. What we’ve really done so far is define how to parameterise some requests, e.g. to filter or sort based on a property value, but we’ve not yet defined that in a standard media type; the API configuration captures a lot of the requisite information but isn’t quite there.

That’s a long rambly blog post for a Friday night! Hopefully I’ve clarified what I was referring to yesterday. I absolutely don’t want to see anyone define an API for RDF that steps around HTTP. We need something that is much more closely aligned with the web. And hopefully I’ve also answered Ed’s question.

Explaining REST and Hypertext: Spam-E the Spam Cleaning Robot

I’m going to add to Sam Ruby’s amusement and throw in my attempt to explicate some of Roy Fielding’s recent discussion of what makes an API RESTful. If you’ve not read the post and all the comments then I encourage you to do so: there’s some great tidbits in there that have certainly given me pause for thought.

The following attempts to illustrate my understanding of REST. Perhaps bizarrely, I’ve chosen to focus more on the client than on the design of the server, e.g. what resources it exposes, etc. This is because I don’t think enough focus has been placed on the client, particularly when it comes to the hypermedia constraint. And I think that often, when we focus on how to design an “API”, we’re glossing over some important aspects of the REST architecture which includes after all, other types of actors, including both clients and intermediaries.

I’ve also deliberately chosen not to draw much on existing specifications, again its too easy to muddy the waters with irrelevant details.

Anyway, I’m well prepared to stand corrected on any or all of the below. Will be interested to hear if anyone has any comments.

Lets imagine there are two mime types.

The first is called application/x-wiki-description. It define a JSON format that describes the basic structure of a Wiki website. The format includes a mixture of simple data items, URIs and URI templates that collectively describe:

  • the name of the wiki
  • the email address of the administrator
  • a link to the Recent Changes resource
  • a link to the Main page
  • a link to the license statement
  • a link to the search page (as a URI template, that may include a search term)
  • a link to parameterized RSS feed (as a URI template that may include a date)

Another mime type is application/x-wiki-page-versions. This is another JSON based format that describes the version history of a wiki page. The format is an ordered collection of links. Each resource in that list is a prior version of the wiki page; the most recent page is first in the list.
Spam-E is a little web robot that has been programmed with the smarts to understand several mime types:

  • application/x-wiki-description
  • application/x-wiki-page-versions
  • RSS and Atom

Spam-E also understands a profile of XHTML that defines two elements: one that points to a resource capable of serving wiki descriptions, another that points to a resource that can return wiki page version descriptions..

Spam-E has internal logic that has been designed to detect SPAM in XHTML pages. It also has a fully functioning HTTP client. And it also has been programmed with logic appropriate to processing those specific media types.

Initially, when starting Spam-E does nothing. It waits to receive a link, e.g. via a simple user interface. Its in a steady state waiting for input.

Spam-E then receives a link. The robot immediates dereferences the link. It does so by submitting a GET request to the URL, and includes an Accept header:

Accept: x-wiki/description;q=1.0, x-wiki/page-versions;q=0.9, application/xhtml+xml;q=0.8, application/atom+xml;q=0.5, application/rss+xml;q=0.4

This clearly states Spam-E’s preference to receive specific mime-types.

In this instance is receives an XHTML document in return. Not ideal, but Spam-E knows how to handle it. After parsing it, it turns out that this is not a specific profile of XHTML that Spam-E understands, so it simply extract all the anchor elements from the file and uses it to widen its search for wiki spam. Another way to say this is that Spam-E has changed its status to one of searching. This state transition has been triggered by following a link, receiving and processing a specific mimetype. This is “hypermedia as the engine of application state” in action.

Spam-E performs this deference-parse-traverse operation several times before finding an XHTML document that conforms to the profile it understands. The document contains a link to a resource that should be capable of serving a wiki description representation.

Spam-E is now in discovery mode. Spam-E uses an Accept header of application/x-wiki-description when following the link and is returned a matching representation. Spam-E parses the JSON and now has additional information at its disposal: it knows how to search the wiki, how to find the RSS feed, how to contact the wiki administrator, etc.

Spam-E now enters Spam Detection mode. It requests, with a suitable Accept header, the recent changes resource, stating a preference for Atom documents. It instead gets an RSS feed, but thats fine because Spam-E still knows how to process that. For each entry in the feed, Spam-E requests the wiki page, using an Accept header of application/xhtml+xml.

Spam-E now tries to find if there is spam on the page by applying its local spam detection logic. In this instance Spam-E discovers some spam on the page. It checks the XHTML document it was returned and discovers that it conforms to a known profile and that embedded in a link element is a reference to the “versions” resource. Spam-E dereferences this link using an Accept header of application/x-wiki-page-versions.

Spam-E, who is now in Spam Cleaning mode, fetches each version in turn and performs spam detection on it. If spam is found, then Spam-E performs a DELETE request on the URI. This will remove that version of the wiki page from the wiki. Someone browsing the original URI of the page will now see an earlier, spam free version.

Once it has finished its cycle of spam detection and cleaning, Spam-E reverts to search mode until it runs out of new URIs.

There are several important points to underline here:

Firstly, at no point did the authors of Spam-E have to have any prior knowledge about the URL structure of any site that the robot might visit. All that Spam-E was programmed with was logic relating to some defined media types (or extension points of a media type in the case of the XHTML profiles) and the basic semantics of HTTP.

Secondly, no one had to publish any service description documents, or define any API end points. No one had to define what operations could be carried out on specific resources, or what response codes would be returned. All information was found by traversing links and by following the semantics of HTTP.

Thirdly, the Spam-E application basically went through a series of state transitions triggered by what media types it received when requesting certain URIs. The application is basically a simple state machine.

Anyway, hopefully that is a useful example. Again, I’m very happy to take feedback. Comments are disabled on this blog, but feel free to drop me a mail (see the Feedback link).

XML Hypertext: Not Dead, Merely Resting?

“The dreams of XML hypertext are dead, or at least thoroughly dormant”

Simon St Laurent’s article on XQuery is an interesting read. But I think the above statement is worth discussing. Is XML hypertext really dead? Or, if its dormant, is it going to remain so?
Firstly what is XML hypertext? I presume from the context of the quote that Simon is referring to client side use of XML on the web. To me this incorporates several use cases including both the use of XML for presentation (XHTML, SVG, etc) and for data publishing (RSS, Atom, XML based web services). There is an obvious need for linking in both of these use cases.
Where I’d agree with St. Laurent is that most of the existing work here is dormant or duplicated. For example while SVG makes use of XLink, its not used in RSS and Atom, and was deemed not flexible enough for use in XHTML due to issues with attribute naming. However the basic model, labelled links with activation indicators (onLoad, onClick, etc) seems to be shared across vocabularies. But still, XLink has been a Recommendation since 2001 and has yet to set the world on fire.
However where I’d disagree with Simon is that XLink or XML hypertext is thoroughly dormant. Much as I hate to make predictions, I think we’re only just gaining any appreciation of the power of producing good hypertext, because we’re only now seeing the large scale publishing of machine-processable, interrelated data that makes linking worthwhile.
I think growing appreciation of the REST architecture is driving a greater understanding of the benefits of highly linked resources. Sure, we all know its good practice to avoid making web pages that are “dead ends”, but not everyone is publishing data to the same guidelines. The principle of “Hypermedia as the engine of application state” is still not widely understood; it’s a piece of REST Zen that benefits from practical implementation.
Hypertext simplifies client-side development as it avoids spreading the requirement that the client must know how to construct URIs: this reduces coupling. It also simplifies client logic as “the navigation options” (i.e. state transfers) can be presented by the server as the result of previous interactions; the client can simply select from amongst the labelled options. For example if the client needs to access some specific data, e.g. a list of my recently published photos, it can select the appropriate link to retrieve (assuming its available).
That link may be to an entirely different service.
In an XTech 2005 paper I tried to argue (I suspect not very clearly) that linking offers the best route to integration of data from multiple web services. Linking as a means to easier mashing.
If the current data publishing trends continue then I suspect there’s going to be a growing understanding of the benefits of hypertext and this inevitably drive some renewed interest in XLink or a related technology.
What I personally like about RDF in this regard is the “closure” it offers: every resource has a URI, every schema has a URI, every Property and Class has a URI so the data, metadata and schemas can be linked together, and this offers some very powerful capabilities.

Benefits of Refactoring to REST

Edd sent me a pointer to a nice article from Scott Raymond called “Refactoring to REST” in which he outlines how his application code was improved and simplified by adopting a more RESTful design. The application here was built on Rails and used the Simply Restful plugin to nudge Rails into a more RESTful aspect.
I’ve noticed a similar reduction in complexity when moving to RESTful application design. I’ve tended to describe this as reducing the “surface area” of the application: the smaller the surface area, the less code is required. It also follows that the smaller the surface area, the less URL types are required. You end up with fewer fairly standard URL patterns which identify resources, rather than RPC-style “method” oriented URLs.
This has some nice properties. For the client a given server application becomes more easily substitutable, as there’s less coupling. And on the server side it clarifies the “points of contact” of the application with the web (of data). It also makes it easier to maintain permanent links because as the URLs are simpler and more identifiable they’re easier to rewrite/redirect as an application evolves or changes architecture.

Microformats and REST

Just noticed Danny’s posting about the new microformat-rest mailing list. I was going to start analysing this but see that Joe Gregorio has already done a good job.
I don’t think that microformats have much to add to REST as an architectural pattern. It certainly doesn’t merit subsetting its use with HTTP; that definitely is overreaching. Championing microformats, and vocabulary re-use in general, is a good thing though, as I’ve talked about before. I think there’s more mileage to be had in pursuing that angle, as well as hypermedia (the other neglected aspect of REST), than there is in subsetting the pattern.
If that doesn’t seem to be a useful approach, then one plea I’d make to the microformats communicate is that they come up with a name for this new architectural style, and actually relate it to REST in a more formal way. Just like Rohit Kare did with ARRESTED.
But I don’t think thats the actual goal. It feels more about trying to define some best practices for deploying RESTful web services. And that’s something I do agree with.