Now that I work in the energy sector I’m trying to pay closer attention to how the data infrastructure in that area is evolving.
Here’s a round up of some current and recent projects that I’ve been keeping an eye on. Along with some thoughts on their scope, overlaps and potential outcomes.
Ofgem review of data best practices
In 2021 Ofgem published a set of principles, called the “Data Best Practice” guidance. They are a set of eleven principles intended to guide organisations in the energy sector towards publishing more open data.
It encourages a “presumed open” approach and recommends the types of general best practice that you can see in other principles, e.g. FAIR.
The principles are binding for a small number of organisations (those operating the UK’s energy networks) but are otherwise voluntary.
Ofgem recently asked for feedback on the principles, with comments closing at the end of October. The feedback is phrased as a kind of retrospective: what’s going well, what could be improved, what would encourage organisations to adopt them, etc. There’s one specific question about whether providing more concrete guidance on data formats would be helpful.
It will be interesting to see what kind of feedback Ofgem have received. My hope is that it will prompt:
- An expansion of which organisations should be following the principles. I see no reason why they shouldn’t apply to every licensed organisation in the sector
- Specific requirements for organisations to publish specific datasets, in well-defined formats, rather than encouraging a “presumed open” approach and then leaving them to work through the details on what can be published and how
- A proactive regulatory approach to assessing whether organisations are actually applying the principles as expected, e.g. what data should be now expect to see under shared and open licences?
I’ve done quite a bit of work in the last few years supporting organisations in adopting the FAIR data principles. What I’ve learned is that while principles can provide a good basis for building a shared vision, they always need to be supported by specific actionable guidance.
You cannot assume that everyone knows how to put those principles into practice.
Without detailed, sector and dataset specific guidance — use these formats, this metadata and this data should be open, while this data should be shared — people are just left with working through all of the details for themselves. This creates friction. And that friction results in not being published well, and plenty of room for excuses and uncertainty that leaves data not being published at all.
Digital spine feasibility study
In January 2022 the Energy Digitalisation Taskforce published a report that provided a set of recommendations aimed at creating a digitalised Net Zero energy system.
The report includes a number of recommendations aimed at improving the data infrastructure in the sector. Including creating a “data sharing fabric”, an energy asset register, a data catalogue, improving data standards and creating a “digital spine”.
Unfortunately the report doesn’t clearly define what it means by a “fabric” or a “spine”, just that the former is intended to support data sharing and the latter to improve interoperability. In practice there’s a lot of different ways this technical infrastructure might be delivered.
The spine appears like it’s intended to be middleware that sits between individual organisations and the broader energy network making it easier to expose data in standard formats.
This looks to be different to, for example, the NHS spine which fulfils a similar role but includes some centrally coordinated services to ensure connectivity and interoperability across the sector. I’ve not been able to examples of “spines” in other sectors.
BEIS have commissioned a study to determine “the needs case, benefits, scope and costs of an energy system ‘digital spine’”. That procurement completed at the end of November, but I don’t believe the winners have been announced yet.
Unfortunately, from the outside, it feels a bit like a broad solution has been proposed (open-source middleware) which the procurement is then focused on scopin. Rather than starting from a broader user research question of “what is required to improve interoperability in the energy sector?” then identifying the most useful interventions.
For example, I’ve previously written about some low-hanging fruit that would increase interoperability of half-hourly meter data.
An open source middleware layer might be a useful intervention, but there’s risks that other necessary work is overlooked. For example, if an open source middleware is going to convert data into standardised forms, then what do those look like? Do they need to be developed first?
Does the UK energy sector even have a good track record of creating and adopting open source infrastructure? Or is there ground work required to build that acceptance and capacity first?
Maybe all this was covered in the research behind the Taskforce report that recommended the spine, but its not clear at this point. It will be interesting to see the results of the study.
Smart Meter Energy Data Repository Programme
The Smart Meter Energy Data Repository (SEDR) programme is intended to “determine the technical and commercial feasibility of a smart meter energy data repository, quantify the benefits and costs of such a smart meter energy data repository, and simulate how it could work”.
The procurement for this piece of work closed in July, but again I don’t think the winner has been announced. I’ve been interviewed by them as part of their user research, so I know the project is up and running!
Update: 3rd January 2023 the three funded projects have been announced.
The current smart meter network has at most 2 years of data distributed across every smart meter in the UK. Access to data requires querying individual meters by sending them messages.
Would it be useful to have a single repository of data, making it easier to query and work with? What types of applications would benefit from that data infrastructure? What are the privacy implications? What would the technical infrastructure look like? These are all questions that will be considered in this project.
I’ve written before about why the UK energy sector should learn from open banking and just develop standardised APIs to provide a high-level interface to the smart meter network.
I’ve also written about the problems of trying to access half-hourly data for the non-domestic market.
I won’t repeat that here, but will note a few things that I raised in the user research:
- People are very nervous about the use of smart meter data. For example, see the recent row over the government plans to harvest this data for the Energy Price Guarantee scheme. The current data infrastructure was designed not to have a single large dataset. There are some big trust, privacy and security issues to navigate here
- The focus should be on improving access to half-hourly data, not just smart meter data. The smart meter roll-out is not finished and smart meters are not mandated (and in some cases not suitable) for non-domestic customers. There’s nothing specific about the data from smart meters that makes it different to process, store or analyse than other sources of half-hourly data.
- Non-domestic use cases need to be considered too
- Would running this repository be part of the DCC’s remit, or do we need another data institution with responsibility for managing that infrastructure?
- Ofgem previously consulted around the benefits of providing a repository of half-hourly data, citing lack of innovation and lack of support for data sharing as a rationale. This highlighted some potential benefits, some resistance in the market but creating it was ultimately kicked down the line. There are things to learn from that project.
- Providing access to a repository of data could be useful to support prototyping and development of new services. That type of repository might hold synthetic or anonymised data, rather than live, recent data. But there needs to be a pathway to allow developers to move from this type of infrastructure to integrating with the live system. At the moment there aren’t many “DCC Other Providers” offering higher level APIs and these aren’t standardised
- If the repository did provide access to live data, there’s still a decision to be made about how much is stored. E.g. is data only stored for a limited time, or will it be an historical archive too? These enable different use cases and have different scalability, sustainability and security issues
- When it comes to analysing smart meter data, you need more than just the half-hourly readings. You need a range of other sources: (historic) tariff information, descriptions of alternative tariffs (to support switching use cases), links between a customer/property and the meters, geospatial data, weather data, etc. Not all of this exists in shared or open formats in the existing infrastructure. How much of this would be part of a planned repository?
This is potentially a critical piece of infrastructure, so it needs some careful planning and execution.
Smart Meter System based Internet of Things applications programme
Another current BEIS programme is the Smart Meter System based Internet of Things applications programme.
This one is looking at whether it is possible to use the existing Smart Meter communications network and infrastructure run by the DCC to support other IoT applications. For example, to support monitoring of “smart buildings” or other parts of the energy data system.
Update: 3rd January 2023 the three funded projects have been announced.
This seems like a sensible approach, as we don’t necessarily need separate infrastructure for what might be very similar requirements.
But the Smart Meter Energy Data Repository project shows that the current infrastructure is not meeting existing needs, so it’s reasonable to assume that there will be additional requirements for these IoT use cases too. Hopefully considerations of these other use cases are at least on the radar of the SEDR review as they might offer additional insights.
As you can see there’s a lot happening around the UK’s energy data infrastructure. I’ve not even touched on the work of Icebreaker One or the SmartDCC “Data for Good” Project.
Lost Boy 