Improving the global open data index

The 2015 edition of the Global Open Data Index was published this week. From what I’ve seen it’s the result of an enormous volunteer effort and there’s a lot to celebrate. For example, the high ranking of Rwanda resulting from ongoing efforts to improve their open data publication. Owen Boswarva has also highlighted the need for the UK and other highly ranked countries to not get complacent.

Unfortunately, I have a few issues with the index as it stands which I wanted to document and which I hope may be useful input to the revisions for the 2016 review noted at the end of this article. My aim is to provide some constructive feedback rather than to disparage any of the volunteer work or to attempt to discredit any of the results.

My examples below draw on the UK scores simply because its the country with which I’m most familiar. I was also involved in a few recent email discussions relating to the compilation of the final scores and some last minute revisions to the dataset definitions.

Disclaimers aside, here are the problems that I think are worth identifying.

Lack of comparability

Firstly, it should be highlighted that the 2015 index is based on a different set of criteria than previous years. A consultation earlier this year lead to some revisions to the index. These revisions included both the addition of new datasets and revisions to the assessment criteria for some of the existing datasets.

The use of a different set of criteria mean its not really possible to compare rankings of countries between years. You can make comparisons between countries on the 2015 rankings, but you can’t really compare the rank of a single country between years as they are being assessed on different information. The website doesn’t make this clear at all on the ranking pages.

Even worse, the information as presented, is highly misleading. If you look at the UK results for election data in 2015 and then look at the results for 2014 you’ll see that the 2014 page is displaying the scores for 2014 but the assessment criteria for 2015. It should be showing the assessment criteria for 2014 instead. This makes is seem like the UK has gone backwards from 100% open to 0% open for the same criteria, rather than being assessed in a completely different way.

If you look at the Wayback Machine entry for the 2014 results, you’ll see the original criteria.

Recommended fixes:

• Remove comparisons between years, or at least provide a clear warning indicator about interpretation
• Ensure that historical assessments include the original criteria used (I can only assume this is a bug)

Lack of progress indicators

Related to the above, another failing of the index is that it doesn’t measure progress towards a desired outcome.

Sticking with the election data example, the changes in the assessment criteria included requirements to report additional vote counts and, most significantly, that data should be reported at the level of the polling station rather than by constituency.

My understanding is the the Electoral Commission have always published the additional counts, e.g. invalid and spoiled ballots. Its only the change in the level of aggregation of results that is different. The UK doesn’t report to polling station level (more on that below). But the data that is available – the same data that previously was 100% open – is still available. But its continued availability has been completely ignored.

This raises important questions about how viable it is for a country to use the index as a means to measure or chart its progress: changes to the criteria can completely discount all previous efforts that went into improving how data is published, and the ongoing publication of valuable data.

Recommended fixes:

• Include some additional context in the report to acknowledge what data does exist in the domain being measured, and how it aligns with criteria
• Where reporting to a particular level of detail is required, include some intermediary stages, e.g. “national”, “regional”, “local”, that can help identify where data is available but only at a more aggregated level

Inconsistent and debatable reporting requirements

The reporting requirements in different areas of the index are also somewhat inconsistent. In some areas a very high level of detail is expected, whereas in others less detail is acceptable. This is particularly obvious with datasets that have a geographical component. For example:

• Location data should be available to postcode (or similar) level
• Election data must now be reported to polling station, rather than constituency as previously indexed
• The weather data criteria doesn’t indicate any geographic reporting requirements
• Similarly the national statistics criteria simply ask for an annual population count, but its unclear at what level is acceptable. E.g. is a single count for the whole country permissible?

While we might naturally expect different criteria for different domains, this belies the fact that:

• Address data is hugely valuable, widely identified as a critical dataset for lots of applications, but its availability is not measured in the index. The UK gets full marks whereas it should really be marked down because of its closed address database. The UK is getting a free ride where it should be held up as being behind the curve. Unless the criteria are revised again, Australia’s move to open its national address database will be ignored.
• Changing the level of reporting of election data in the UK will require a change in the way that votes are counted. It may be possible to report turnout at a station level, but we would need to change aspects of the electoral system in order to reach the level required by the index. Is this really warranted?

The general issue is that the rationale behind the definition of specific criteria is not clear.

Recommended fixes:

• Develop a more tightly defined set of assessment criteria and supplement these with a clear rationale, e.g. what user needs are being met or unmet if data is published to a specific level?
• Ensure that a wide representation of data consumers have included in drafting any new criteria
• Review all of the current criteria, with domain experts and current consumers of these datasets, to understand whether their needs are being met

Inconsistent scoring against criteria

I also think that there’s some inconsistent scoring across the index. One area where there seems to be varied interpretation is around the question: “Available in bulk?”. The definition of this question states:

“Data is available in bulk if the whole dataset can be downloaded easily. It is considered non-bulk if the citizens are limited to getting parts of the dataset through an online interface. For example, if restricted to querying a web form and retrieving a few results at a time from a very large database.”

Looking at the UK scores we have:

1. Location data is marked as available. However there are several separate datasets which cover this area, e.g. list of postcodes are in one dataset, an administrative boundaries are in another. The user has to fill in a form to get access to a separate download URL for each product. The registration step is noted but ignored in the assessment. And it seems that the reviewer is happy with downloading separate files for each dataset
2. Weather data is marked as available. But the details say that you have to sign-up to use an API on the Azure Marketplace to retrieve the data, which you then harvest using their API. Presumably having first written the custom code required to retrieve it, is this not limiting access to parts of the dataset? And specifically, limiting its access to specific types of users? It’s not what I’d consider to be a bulk download. The reviewer has also missed the download service on data.gov.uk which does actually provide the information in bulk
3. Air quality data is not available in bulk. However you can access all the historical data as Atom files, or have all the historical data for all monitoring sites emailed to you, or access the data as RData files via the openair package. Is this presenting a problem for users of air quality data in the UK?

Personally, as a data consumer I’m happy with the packaging all of these datasets. I definitely agree that the means of access could be improved, e.g. provision of direct download links without registration forms, provision of manifest files to enable easier mirroring. And I have a preference for the use of simpler formats (CSV vs Atom), etc. But that’s not what is being assessed here.

Again, its really a user needs issue. Is it important to have bulk access to all of the UK’s air quality data in a single zip file, or all of the historical data for a particular location in a readily accessible form? The appropriate level of packaging will depend on the use case of the consumer.

Recommended fixes:

• Provide additional guidance for reviewers
• Ensure that there are multiple reviewers for each assessment, including those familiar with not just the domain but also the local publishing infrastructure
• Engage with the data publishers throughout the review process and enable them to contribute to the assessments

Hopefully this feedback is taken in the spirit it’s offered: as constructive input into improving a useful part of the open data landscape.