Elinor Ostrom and data infrastructure

One of the topics that most interests me at the moment is how we design systems and organisations that contribute to the creation and maintenance of the open data commons.

This is more than a purely academic interest. If we can understand the characteristics of successful open data projects like Open Street Map or Musicbrainz then we could replicate them in other areas. My hope is that we may be able to define a useful tool-kit of organisational and technical design patterns that make it more likely for other similar projects to proceed. These patterns might also give us a way to evaluate and improve other existing systems.

A lot of the current discussion around this topic is going on under the “data infrastructure” heading. Also related is the idea of open data as a public good.

While I believe that open data is a public good, I do wonder whether particular styles of data infrastructure and licensing arrangements mean that data might sometimes be a club good. But lots more thinking and reading to be done there. Economics isn’t my area of expertise.

That said, if you’re interested in data infrastructure then I’d recommend looking at the work of Elinor Ostrom. She received a Nobel Prize for her research exploring how communities self-organise to managing the commons. Her work was instrumental in debunking the idea of the “tragedy of the commons”.

A key outcome of Ostrom’s work was the definition of 8 principles for designing organisations that manage common-pool resources. While her focus was on common-pool resources rather than public goods, the principles define a framework that can be applied more generally. And, as this article on the influences of Ostrom’s work notes, “any group whose members must work together to achieve a common goal is vulnerable to self-serving behaviors and should benefit from the same principles“.

The Open Data Institute have defined some high-level principles for strengthening data infrastructure. These include working in the open, designing collaborative models, building with the web, and balancing stakeholder interests.

I think you can usefully read Ostrom’s principles as more detailed guidance for how to create digital communities that collaborate to create and maintain open data. In fact if you’ve been part of any online community or taken part in community-building activities, I think those principles should resonate pretty strongly.

As an illustration, here are each of the principles and some suggested questions that are relevant to digital communities and open data. Ostrom highlights that communities will have:

  1. Clearly defined boundaries (clear definition of the contents of the common pool resource and effective exclusion of external un-entitled parties)
    • what is the purpose of the data infrastructure?
    • what community does it serve, and how are they identified?
    • what are the key data assets that the infrastructure will produce?
    • when will it’s mission be complete?
  2. Rules regarding the appropriation and provision of common resources that are adapted to local conditions;
    • how are the data assets and guidance provided by the community licensed?
    • what are the forms of attribution and other social norms that apply to use of the resources?
    • what are the guidelines that apply to contributions from the community?
    • how are new contributors guided towards becoming productive members of the community?
    • what are the means by which people can access and reuse the data?
  3. Collective-choice arrangements that allow most resource appropriators to participate in the decision-making process;
    • how does the community share ideas about how the infrastructure should evolve?
    • what are the decision making processes and the tools used to support them?
    • if poor quality data is added, how is this discussed, highlighted and improved?
    • how are differences of opinion, or innovative ideas relating to e.g. data modelling or organisation issues, discussed within the community?
  4. Effective monitoring by monitors who are part of or accountable to the appropriators;
    • how are contributions to the data assets managed or reviewed by moderators?
    • how does the community measure its progress and activity?
    • how are moderators identified and promoted? how might their privileges be removed?
    • how are good uses of the infrastructure showcased?
    • what metrics are available to measure data quality, coverage, etc?
  5. A scale of graduated sanctions for resource appropriators who violate community rules;
    • how is spam and other wilful misuse identified and dealt with?
    • how is abusive behaviour dealt with?
    • how does the community document and share its norms?
    • what are the means by which contributors gain or lose privileges?
  6. Mechanisms of conflict resolution that are cheap and of easy access;
    • what process are used to resolve debates and make decisions?
    • how can data quality issues be flagged and address?
    • what are the mechanisms by which community members can share their opinions, or have their voice heard?
    • how are the results of debate and key decisions recorded?
  7. Self-determination of the community recognized by higher-level authorities;
    • what type of organisation is used to manage the community resources?
    • what is the process by which other organisations engage with the community and/or its representatives?
  8. In the case of larger common-pool resources, organization in the form of multiple layers of nested enterprises, with small local CPRs at the base level
    • how does the community interact with other similar initiatives, e.g. in a sector or broader community?

Most importantly implementing a viable digital community for managing the data commons means that we must build with the web, which is another of the ODI’s principles.

As ever, if you have thoughts then let me know!

“Open”

For the purposes of having something to point to in future, here’s a list of different meanings of “open” that I’ve encountered.

XYZ is “open” because:

  • It’s on the web
  • It’s free to use
  • It’s published under an open licence
  • It’s published under a custom licence, which limits some types of use (usually commercial, often everything except personal)
  • It’s published under an open licence, but we’ve not checked too deeply in whether we can do that
  • It’s free to use, so long as you do so within our app or application
  • There’s a restricted/limited access free version
  • There’s documentation on how it works
  • It was (or is) being made in public, with equal participation by anyone
  • It was (or is) being made in public, lead by a consortium or group that has limitation on membership (even if just fees)
  • It was (or is) being made privately, but the results are then being made available publicly for you to use

I gather that at IODC “open washing” was a frequently referenced topic. It’s not surprising given the variety of ways in which the word “open” is used. Many of which are not open at all. And the list I’ve given above is hardly comprehensive. This is why the Open Definition is such an important reference. Even if it may have it’s faults.

Depending on your needs, any or all of those definitions might be fine. But “open” for you, may not be “open” for everyone. So let’s not lose sight of the goal and keep checking that we’re using that word correctly.

And, importantly, if we’re really making things open to make them better, then we might need to more open to collaboration. Open isn’t entirely about licensing either.

 

Current gaps in the open data standards framework

In this post I want to highlight what I think are some fairly large gaps in the standards we have for publishing and consuming data on the web. My purpose for writing these down is to try and fill in gaps in my own knowledge, so leave a comment if you think I’m missing something (there’s probably loads!)

To define the scope of those standards, lets try and answer two questions.

Question 1: What are the various activities that we might want to carry out around an open dataset?

  • A. Discover the metadata and documentation about a dataset
  • B. Download or otherwise extract the contents of a dataset
  • C. Manage a dataset within a platform, e.g. create and publish it, update or delete it
  • D. Monitor a dataset for updates
  • E. Extract metrics about a dataset, e.g. a description of its contents or quality metrics
  • F. Mirror a dataset to another location, e.g. exporting its metadata and contents
  • G. Link or reconcile some data against a dataset or register

Question 2: What are the various activities that we might want to carry out around an open data catalogue?

  • V. Find whether a dataset exists, e.g. via a search or similar interface
  • X. List the contents of the platform, e.g. its datasets or other published assets
  • Y. Manage user accounts, e.g. to create accounts, or grant or remove rights from specific accounts
  • Z. Extract usage statistics, e.g. metrics on use of the platform and the datasets it contains

Now, based on that quick review: which of these areas of functionality are covered by existing standards?

  • DCAT and its extensions gives us a way to describe a dataset (A) and can be used to find download links which addresses part of (B). But it doesn’t say how the metadata is to be discovered by clients.
  • The draft Data Catalog Vocabulary starts to address parts of (E) but also doesn’t address discovery of published metrics
  • ODATA provides a means for querying and manipulating data via a RESTful interface (B, C). Although I don’t think it recognises a dataset as such, just resources exposed over the web
  • SPARQL (query, update, etc) also provides a means for similar operations (B, C), but on RDF data.
  • The Linked Data Platform specification also offers a similar set of functionality (B, C)
  • If a platform exposes its catalogue using DCAT then a client could use that to list its contents (X)
  • The draft Linked Data Notifications specification covers monitoring and synchronising of data (D)
  • Data Packages provide a means for packaging metadata and contents of dataset for download and mirroring (B, F)

I think there’s a number of obvious gaps around discovery and platform (portal) functionality. API and metadata discovery is also something could usefully be addressed.

If you’re managing and publishing data as RDF and Linked Data then you’re slightly better covered at least in terms of standards, if not in actual platform and tool support. The majority of current portals don’t manage data as RDF or Linked Data. They’re focused on either tabular or maybe geographic datasets.

This means that portability among the current crop of portals is actually pretty low. Moving between a platform means moving between different entirely different sets of APIs and workflows. I’m not sure that’s ideal. I don’t feel like we’ve yet created a very coherent set of standards.

What do you think? What am I missing?

Why are bulk downloads of open data important?

I was really pleased to see that at the GODAN Summit last week the USDA announced the launch of its Branded Food Product Database, providing nutritional information on over 80,000 food products. Product reference data is an area that has been long under-represented in the open data commons, so its great to see data of this type being made available. Nutritional data is also an area in which I’m developing a personal interest.

The database is in the public domain, so anyone can use it for any purpose. It’s also been made available via a (rate-limited) API that allows it to be easily searched. For many people the liberal licence and machine-readability will be enough to place a tick in the “open data” box. And I’m inclined to agree.

In the balance

However, as Owen Boswarva observed, the lack of a bulk download option means that the dataset technically doesn’t meet the open definition. The latest version of the definition states that data “must be provided as a whole…and should be downloadable via the Internet”. This softens the language used in the previous version which required data to be “available in bulk”.

The question is, does this matter? Is the open definition taking an overly pedantic view or is it enough, as many people would argue for the data to be openly licensed?

I think having a clear definition of what makes data open, as opposed to closed or shared, is essential as it helps us focus discussion around what we want to achieve: the ability for anyone to access, use and share data, for any purpose.

It’s important to understand how licensing or accessibility restrictions might stop us from achieving those goals. Because then we can make informed decisions, with an understanding of the impacts.

I’m less interested in using the definition as a means of beating up on data publishers. It’s a tool we can use to understand how a dataset has been published.

That said, I’m never going to stop getting cross about people talking about “open data” that doesn’t have an open licence. That’s the line I won’t cross!

Bulking up

I think its an unequivocally good thing that the USDA have made this public domain data available. So lets focus on what the impacts of their decision to not publish a bulk download. Something which I suspect would be very easy for them to do. It’s a large spreadsheet, not “big data”.

The API help page notes that the service is “intended primarily to assist application developers wishing to incorporate nutrient data into their applications or websites”. And I think it achieves that for the most part. But there are some uses of data that are harder when the machine-readable version is only available via an API.

Here’s a quick, and non-exhaustive list of the ways a dataset could be used:

  • A developer may want to create a new interface to the dataset, to improve on the USDA’s own website
  • A developer may want to query it to add some extra features to an existing website
  • A developer may want to use the data in a mobile application
  • A developer may want to use the data in desktop application
  • A developer may want to enrich the dataset with additional information and re-publish it
  • A data scientist might want to use the data as part of an analysis
  • An archivist might want to package the dataset and place a copy in the Internet Archive to preserve it
  • A scientist might want to use the data as part of their experimental analysis
  • An organisation might want to provide a mirror of the USDA data (and perhaps service) to help it scale
  • A developer might want to use the data inside services like Amazon or Google public datasets, or Kaggle, or data.world
  • A data journalist might want to analyse the data as part of a story
  • ….etc.

Basically there are a lot of different use cases, which vary based on:

  • the technical expertise of the user
  • the technical infrastructure in which the data is being used
  • whether all or only part of the dataset is required
  • whether custom processing or analysis is required
  • whether the results are being distributed

What’s important to highlight is that all of these use cases can be supported by a bulk download. But many of them are easier if there is an API available.

How much easier depends on the design of the API. And the trade-off by making data easier to use is that it increases the cost and effort to publish the data. The USDA are obviously aware of that cost, because they’ve added some rate-limits to the API.

Many of the use cases are harder if the publisher is only providing an API. Again, it will depend on the design of the API how much harder.

Personally I always advocate having bulk downloads by default and APIs available on a best effort basis. This is because it supports the broadest possible set of use cases. In particular it helps make data portable so that it can be used in a variety of platforms. And as there are no well-adopted standard APIs for managing and querying open datasets, bulk downloads offer the most portability across platforms.

Of course there are some datasets, those that are particularly large or rapidly changing, where it is harder to provide a useful, regularly updated data dump. In those cases provision via an API or other infrastructure is a reasonable compromise.

Balancing the scales

Returning to the USDA’s specific goals they are definitely assisting developers in incorporating nutrient data into their applications. But they’re not necessarily making it easy for all application developers, or even all types of user.

Presumably they’ve made a conscious decision to focus on querying the data over use cases involving bulk analysis, archiving and mirroring. This might not be an ideal trade-off for some. And if you feel disadvantaged then you should take time to engage with the USDA to explain what you’d like to achieve.

But the fact that the data is openly licensed and in a machine-readable form means that it’s possible for a third-party intermediary or aggregator to collect and republish the data as a bulk download. It’ll just be less efficient for them to do it than the USDA. The gap can be filled by someone else.

Which is why I think its so important to focus on licensing. It’s the real enabler behind making something open. Without an open licence you can’t get any real (legal) value-add from your community.

And if you don’t want to enable that, then why are you sharing data in the first place?

This post is part of a series called “basic questions about data“.

People like you are in this dataset

One of the recent projects we’ve done at Bath: Hacked is to explore a sample of the Strava Metro data covering the city of Bath. I’m not going to cover all of the project details in this post, but if you’re interested then I suggest you read this introductory post and then look at some of the different ways we presented and analysed the data.

From the start of the project we decided that we wanted to show the local (cycling) community what insights we might be able to draw from the dataset and illustrate some of the ways it might be used.

Our first step was to describe the dataset and how it was collected. We then outlined some questions we might ask of the data. And we tried to assess how representative the dataset was of the local cycling community by comparing it with data from the last census.

The reactions were really interesting. I spent a great deal of time on social media patiently answering questions and objections. I wanted to help answer those questions and understand what issues and concerns people might have in using this type of data.

I found that there were broadly two different types of feedback.

Visible participation

The first, more positive response, was from existing or previous Strava users surprised or delighted that their data might contribute towards this type of analysis. Some people shared the fact that they only logged some types of rides, while others explained that they already logged all of their activity including commutes and recreational riding. I saw one comment from a user who was now determined to do this more diligently, just so they could contribute to the Metro dataset.

A lesson here is that even users who understand that their data is being collected can still be surprised in the ways that the data might be re-purposed.  This is a data literacy issue: how can we help non-specialists understand the incredible malleability of data?

I think the reaction also reinforces the point that people will often contribute more if they think their data can be used for social good. Or just that people like them are also contributing.

This is important if we want to  encourage more participation in the maintenance of data infrastructure. Commercial organisations would do well to think about how open data and data philanthropy might drive more use of their platforms rather than threaten them.

Even if the Strava data were completely open there are still challenges in its use and interpretation. This creates the space for value-added services. (btw, if anyone wants help with using the Strava Metro data then I’m happy to discuss how Bath: Hacked could help out!)

Two tribes

The second, more negative response, was from people who didn’t use Strava and often had strong opinions about the service. I’ll step lightly over the details here. But, while I want to avoid being critical (because I’m genuinely not), I want to share a variety of the responses I saw:

  • I don’t use this dataset, so it can’t tell you anything about how I cycle
  • I don’t understand why people might use the service, so I’m suspicious of what the data might include
  • I think only a certain type of people use the service so its only representative of them, not me
  • I think people only use this service in a specific way, e.g. not for regular commutes, and so the data has limited use
  • I’m suspicious about the reliability of the data, so distrust it.

I’d think I’d sum all of that up as: “people like me don’t use this service, so any data you have isn’t representative of me or my community“.

This is exactly the issue we tried to shed some light on in our first two blog posts. So clearly we failed at that! Something to improve on in future.

The real lesson for me here is that people need to see themselves in a dataset.

If  we don’t help someone understand whether a dataset is representative of them, then it’s use will be viewed with suspicion and doubt. It doesn’t matter how rigorous the data collection and analysis process might be behind the scenes, it’s important to help find ways for people to see that for themselves. This isn’t a data literacy issue: it’s a problem with how we effectively communicate and build trust in data.

If we increasingly want to use data as a mirror of society, then people need to be able to see themselves in its reflection.

If they can see how they might be a valuable part of a dataset, then they may be more willing to contribute. If they can see whether they (or people like them) are represented in a dataset, then they may be more willing to accept insights drawn from that data.

Story telling is likely to be a useful tool here, but I wonder whether there are other complementary ways to approach these issues?

Help me use your data

I’ve been interviewed a couple of times recently by people interested in understanding how best to publish data to make it useful for others.  Once by a startup and a couple of times by researchers. The core of the discussion has essentially been the same question: “how do you know if a dataset will be useful to you?”

I’ve given essentially the same answer each time. When I’m sifting through dataset descriptions, either in a portal or via a web search, my first stage of filtering involves looking for:

  1. A brief summary of the dataset: e.g. a title and a description
  2. The licence
  3. Some idea of its coverage, e.g. geographic coverage, scope of time series, level of aggregation, etc
  4. Whether it’s in a usable format

Beyond, that there’s a lot more that I’m interested in: the provenance of the data, its timeliness and a variety of quality indicators. But those pieces of information are what I’m looking for right at the start. I’ll happily jump through hoops to massage some data into a better format. But if the licence or coverage isn’t right then its useless to me.

We can frame these as questions:

  1. What is it? (Description)
  2. Can I use it? (Licence)
  3. Will it help answer my question? (in whole, or  part)
  4. How difficult will it be to use? (format, technical characteristics)

It’s frustrating how often these essentials aren’t readily available.

Here’s an example of why this is important.

A weather data example

I’m currently working on a project that needs access to local weather observations. I want openly licensed temperature readings for my local area.

My initial port of call was the Met Office Hourly Site Specific Observations. The product description is a useful overview and the terms of use make the licensing clear. Questions 1 & 2 answered.

However I couldn’t find a list of sites to answer Question 3. Eventually I found the API documentation for the service that would generate me a list of sites. But I can only access that with an API key. So I’ve signed up, obtained a key, made the API call, downloaded the JSON, converted it into CSV, uploaded it to Carto and then made a map.

And now I can answer Question 3. The closest site is in Bristol and so the service isn’t useful to me at all. Time wasted, but hopefully not all the effort because now you can just look at the map. But the Met Office could simply have published a map. There is one of the whole network, but they don’t all contribute to the open dataset.

So I started to look at the OpenWeatherMap API. They also have an API endpoint that exposes weather data for a specific station. Or stations within a geographic area. But again, they’ve not actually published a map that would let me see if there are any local to me. I might have missed something so I’ve asked them.

In both cases I’m having to get into invest time and some technical effort in answering questions which should be part of the documentation. They could even use their own APIs to create an interactive map for people to use!

As a result I’m going to end up using wunderground. By browsing the user facing part of their site I’ve been able to confirm there are several local weather stations. And hopefully these will be exposed via the API. (But I’m going to have to dig a bit to check on the terms of use. Sigh.)

If you really want me to use your data then you need to help me to use it. Think about my user experience. Help me understand what your dataset contains before I have to actually poke around inside it.

Reputation data portability

Yesterday I went to the ODI lunchtime lecture on portability of reputation data. It was an interesting discussion which triggered a few thoughts which I thought I’d share here.

The debate was prompted by a call for evidence from the Department formally known as BIS around consumer data and account switching:

“The government would like to understand whether the reputation data earned by a user on a particular platform could be used to help them win business or prove their trustworthiness in other contexts. We would also be interested in views on the technical and other challenges that would be associated with making this reputation data portable”

The consultation includes this question:

“What new opportunities or risks for businesses, workers and consumers would be created if they were able to port their reputation and feedback data between platforms?”

It also asks about the barriers that might hinder this type of portability.

One useful way to answer these questions is to break them down into smaller pieces:

  1. Should consumers be able to access data they’ve contributed in a platform?
  2. Should businesses be able to access data about them in a platform, e.g. reputation data such as reviews
  3. Should businesses and consumers be able to move this this data between platforms?
  4. Should it be permitted for that data to be reused by others, e.g. in competing platforms?

The first two questions are about exporting data.

The third and fourth questions are really about portability and data licensing.

I would say that broadly the answers to all these questions is: Yes.

I think consumers and businesses should be able to access this data and, further, that it should be machine-readable open data. They should also be able to access any of their personal data held in the platform, but this isn’t really an area of debate. The new EU GDPR regulations requires platforms to provide you with your data if you request it, although it doesn’t (to my knowledge) require it to be in a machine-readable reusable form.

I think this also answers the last question, the data should be reusable. However I expect there to be resistance from platforms as where this type of data is currently made available it is done so under non-open terms. For example, via API agreements that prohibit some forms of reuse, such as use in a competing service.

The question on portability is trickier though. While I think  that portability is something to aspire to, in practice it is going to be difficult to achieve.

Portability requires more than just creating a data standard to enable export and import of data, or APIs that enable more dynamic synchronisation. I think that’s the easy part.

Portability would also require platforms to agree or converge around how reputation data is collected and calculated. It’s no good moving data from one system to another if they have incompatible definitions. There are many ways in which platforms might differ:

  • They can use a different rating scheme, e.g. 5 stars, 10 stars, or just “likes”
  • They might allow, or require, a text review in addition to a rating
  • They can allow anonymous reviews or require users to make themselves known
  • They can allow anyone to review any service or business (e.g. TripAdvisor, Amazon product reviews), or they can enforce that reviews are only made when there has been evidence of a transaction (e.g. rating a supplier on EBay or Amazon)
  • Related, they might allow both forms of review, but distinguish those that are based on a transaction
  • Or they may not allow explicit reviews at all and measure reputation in some other way (e.g. completing transactions within an expected time period, or number of sales made)
  • …etc, etc

And this is without even getting into the weeds of what users think they are reviewing. For example are you reviewing the restaurant, the service you received on a specific visit, the menu choice with respect to your preferences, or perhaps even a specific person that delivered that service. I think we’ve all seen examples of all of those variations even within single platforms.

XKCD has nicely summarised a variety of issues with rating systems in these three cartoons. And we shouldn’t forget the creative ways in which review systems get repurposed.

It’s important to highlight here that this type of variation doesn’t really occur with data like banking transactions, utility bills, etc. I tend to think portability there is much easier to achieve. These is variation, but this is typically around charging models not in the meaning and method of collection of the data.

Is all the variation in rating and review schemes warranted? Perhaps not. Some convergence might actually be useful. But these variations are also likely to be key parts of the user experience and functionality of the platform. So I’m personally very wary about restricting product developers in innovating in this area.

In my view rather than focusing on portability, we should be asking for this data to be published as open data. This will then open the possibility for the data to be aggregated and presented across platforms.

Enabling the creation of aggregated reference points for reputation data may be more practical that requiring true portability across platforms. In fact we have models for this already: price comparison sites and credit reference agencies. In fact if these data becomes more open it seems likely that credit agencies will be the first to benefit from it.