A couple of months ago I wrote a post that provided some background on the data we use in Energy Sparks. The largest data source comes from gas and electricity meters (consumption) and solar panels (generation). While we're integrating with APIs that allow us to access data from smart meters, for the foreseeable future most … Continue reading 24 different tabular formats for half-hourly energy data
Category: Data
Building data validators
This is a post about building tools to validate data. I wanted to share a few reflections based on helping to design and build a few different public and private tools, as well as my experience as a user. I like using data validators to check my homework. I've been using a few different recently … Continue reading Building data validators
12 ways to improve the GDS guidance on reference data publishing
GDS have published some guidance about publishing reference data for reuse across government. I've had a read and it contains a good set of recommendations. But some of them could be clearer. And I feel like some important areas aren't covered. So I thought I'd write this post to capture my feedback. Like the original … Continue reading 12 ways to improve the GDS guidance on reference data publishing
Brief review of revisions and corrections policies for official statistics
In my earlier post on the importance of tracking updates to datasets I noted that the UK Statistics Authority Code of Practice includes a requirement that publishers of official statistics must publish a policy that describes their approach to revisions and corrections. See 3.9 in T3: Orderly Release, which states: "Scheduled revisions or unscheduled corrections to … Continue reading Brief review of revisions and corrections policies for official statistics
The importance of tracking dataset retractions and updates
There are lots of recent examples of researchers collecting and releasing datasets which end up raising serious ethical and legal concerns. The IBM facial recognition dataset being just one example that springs to mind. I read an interesting post exploring how facial recognition datasets are being widely used despite being taken down due to ethical … Continue reading The importance of tracking dataset retractions and updates
A letter from the future about numbers
It's an odd now looking at early 21st century content in the Internet Archive. So little nuance. It feels a little like watching those old black and white movies. All that colour which was just right there. But now lost. Easy to imagine that life was just monochrome. Harder to imagine the richer colours. Or … Continue reading A letter from the future about numbers
Four types of innovation around data
Vaughn Tan's The Uncertainty Mindset is one of the most fascinating books I've read this year. It's an exploration of how to build R&D teams drawing on lessons learned in high-end kitchens around the world. I love cooking and I'm interested in creative R&D and what makes high-performing teams work well. I'd strongly recommend it … Continue reading Four types of innovation around data
Increasing inclusion around open standards for data
I read an interesting article this week by Ana Brandusescu, Michael Canares and Silvana Fumega. Called "Open data standards design behind closed doors?" it explores issues of inclusion and equity around the development of "open data standards" (which I'm reading as "open standards for data"). Ana, Michael and Silvana rightly highlight that standards development is … Continue reading Increasing inclusion around open standards for data
FAIR, fairer, fairest?
"FAIR" (or "FAIR data") is an term that I've been bumping into more and more frequently. For example, its included in the UK's recently published Geospatial Strategy. FAIR is an acronym that stands for Findable, Accessible, Interoperable and Reusable. It defines a set of principles that highlight some important aspects of publishing machine-readable data well. … Continue reading FAIR, fairer, fairest?
What kinds of data is it useful to include in a register?
Registers are useful lists of information. A register might be a list of countries, companies, or registered doctors. Or addresses. At the ODI we did a whole report on registers. It looks at different types of registers and how they're governed. And GDS built a whole infrastructure to support them being published and used across … Continue reading What kinds of data is it useful to include in a register?
Lost Boy 