24 different tabular formats for half-hourly energy data

A couple of months ago I wrote a post that provided some background on the data we use in Energy Sparks. The largest data source comes from gas and electricity meters (consumption) and solar panels (generation). While we're integrating with APIs that allow us to access data from smart meters, for the foreseeable future most … Continue reading 24 different tabular formats for half-hourly energy data

12 ways to improve the GDS guidance on reference data publishing

GDS have published some guidance about publishing reference data for reuse across government. I've had a read and it contains a good set of recommendations. But some of them could be clearer. And I feel like some important areas aren't covered. So I thought I'd write this post to capture my feedback. Like the original … Continue reading 12 ways to improve the GDS guidance on reference data publishing

Brief review of revisions and corrections policies for official statistics

In my earlier post on the importance of tracking updates to datasets I noted that the UK Statistics Authority Code of Practice includes a requirement that publishers of official statistics must publish a policy that describes their approach to revisions and corrections. See 3.9 in┬áT3: Orderly Release, which states: "Scheduled revisions or unscheduled corrections to … Continue reading Brief review of revisions and corrections policies for official statistics

The importance of tracking dataset retractions and updates

There are lots of recent examples of researchers collecting and releasing datasets which end up raising serious ethical and legal concerns. The IBM facial recognition dataset being just one example that springs to mind. I read an interesting post exploring how facial recognition datasets are being widely used despite being taken down due to ethical … Continue reading The importance of tracking dataset retractions and updates

Increasing inclusion around open standards for data

I read an interesting article this week by Ana Brandusescu, Michael Canares and Silvana Fumega. Called "Open data standards design behind closed doors?" it explores issues of inclusion and equity around the development of "open data standards" (which I'm reading as "open standards for data"). Ana, Michael and Silvana rightly highlight that standards development is … Continue reading Increasing inclusion around open standards for data

What kinds of data is it useful to include in a register?

Registers are useful lists of information. A register might be a list of countries, companies, or registered doctors. Or addresses. At the ODI we did a whole report on registers. It looks at different types of registers and how they're governed. And GDS built a whole infrastructure to support them being published and used across … Continue reading What kinds of data is it useful to include in a register?