Loomio
Fri 15 Jun 2018

How would you measure OSM UK?

JN
Jez Nicholson Public Seen by 330

I'm collecting ideas on how to measure how successful OSM is in the UK. How would you, personally, measure 'success'?

A number of measures already exist, what are they?

The whole thing can be flawed. 'Success' is a subjective term. No single measure tells the whole story, and many numbers can be misinterpreted. But let's see if it possible to find some figures that can guide us.

Suggest some and i'll collate them into a dot vote or similar.

Criticism of a measure is welcome, but please be gentle.

JN

Jez Nicholson Fri 15 Jun 2018

Brand Recognition: How many people have heard of OSM, split by different groups. I would expect more Software Developers to have heard of OSM than members of the general public have.

To test it out I did a straw poll in a Freelancers Facebook group: 107 saw the poll, 11 knew OSM, 15 different...81 abstained.

RW

Robert Whittaker Fri 15 Jun 2018

Ideally, I'd measure success by the number of instances of people using the map (and map data) in the wild. This would include maps on websites displaying stuff, printed and web "find us"maps, and people emailing map links to each other. It's not obvious to me how to actually measure this though. Globally the number of search engine results for "OpenStreetMap Contributors" might be a useful measure. It would be a bit of work, but perhaps we could annually check all the FTSE 100 companies, central government departments, and top-level local councils, and see how many include maps on their websites, and how many of those are OSM based.

GM(

In the directors meeting Jez also asked how to measure active mappers around the UK (e.g. is Birmingham more/less successful than Southampton). The website that first srpung to mind was Overview of OSM Contributors which I think might take mapper locations from hdyc the main editing in last 6 months. This might not be ideal for identifying people that edit in multiple places, and I think even if a mapper mainly edits abroad from the UK then it is a success(of one metric) for OSM in the UK.

Got point from Robert on how much people use OSM. There are 2 measures within that too, as increasingly people will use OSM (Pokemon Go, Snapchat, Apple Maps, and website no longer using Google) without knowing about OSM. Even if it has a visible acknowledgement, people might not read that. The map style is big for branding (and previously caused resistance to not using Google), but we cannot rely on that. This isn't helping find a metric!

When I give talks I start with asking for a show of hands: heard of OSM, knowingly used OSM(such as visiting the website), edited OSM. I should've recorded the results over the years, but I've only used it for identifying/confirming the audience I'm about to speak to.

JN

Jez Nicholson Mon 18 Jun 2018

  • Frequency of city/area meetups
  • Number of members of OSMUK
  • Postbox/post-office/FHRS/etc. completion rates
N

Neil Tue 19 Jun 2018

Try routing from Lands End to John O'Groats?

Foot is odd: https://www.openstreetmap.org/directions?engine=graphhopper_foot&route=50.06523%2C-5.71368%3B58.64401%2C-3.07003#map=6/54.038/-4.731
Car ones don't work, bicycle does :-)

What I was really interested was how distances compared with other mapping services -- a test of the overall "system".

BP

Brian Prangle Wed 11 Jul 2018

Depends whether you want to measure the map or the community. One that might do both is based on Notes - insertion rate, geographic spread, clean up rate

JN

Jez Nicholson Thu 12 Jul 2018

I'm still vague on whether i'm measuring the health of the map, or the health of the community. Maybe the two go hand-in-hand. I'm sure that there won't be one single measure that tells the story.

There was the ITO World https://wiki.openstreetmap.org/wiki/OSM_and_OSL_differences_analysis

@brianprangle just reminded me about Pascal Nice's work on an OSM 'Rank' and other measures http://neis-one.org/

Also, in my past work with marketing people they have a number of measures like RFM (Recency, Frequency, Monetary-value) with variations like RFE (Recency, Frequency, Engagement) that might be applicable - i.e. how long is it since you last mapped, how frequently do you map, and how long do you edit for https://en.wikipedia.org/wiki/RFM_(customer_value)

I intend to bring this discussion to a close soon and pick some to start with.

TS

Tony Shield Fri 13 Jul 2018

I would be looking at four areas
* People who collect data and update the database
* users of the database for whatever purpose
* accuracy and relevance and completeness of the database
* leadership, management and administration of the OSM UK community and database

Some of these areas can be measured objectively with tools available, others can be subjective; most are a combination of both. Combining all of these together could be difficult with a degree of subjectivity. A fundamental question is "Is OSM UK growing"

My view about the contributing people is to look at the history of UK based users - where are they contributing, frequency of contribution, if someone signs up are they active or dormant - are there any patterns of their contributions.

Assuming that contributors are individuals are the consumers individuals or organisations or a mixture. How can we measure their satisfaction with the database?, relevance of the data offered, completeness and accuracy? The routing example given previously illustrates the point well.

Accuracy and Relevance and Completeness of the database - questions such as is the data consistent? - recent footpath analysis and questions lead the way here. Assessment of the Post Boxes challenge tells us about the completeness of the data and the health and distribution of active contributors. Analysing previous challenges is also relevant.
Leadership, management and administration - always so difficult to measure or assess, much depends on the assessment tools available and used. Are there targets paths and plans? are they being followed? Are they reasonable and are they dependant on resources? Can resources be increased or organised?

An agreement of methods of data collection and how to analyse them is essential - questionnaires need to be tested before they are issued and the range of possible answers analysed to test if the questions add value.
Assessments such as these are of limited value in isolation, several iterations will give greater insights, size of assessment report is not relevant but the quality is and the report in itself is an indicator of the health of OSM UK.

Do hope this is helpful.

JN

Jez Nicholson Mon 16 Jul 2018

The new https://mym.rtijn.org/ tool pulls 'mappers near you' based on an OSM relation id.

JN

Jez Nicholson Tue 16 Oct 2018

I've been pointed at Osmand http://osmand.net/osm_live as well, for lists of active users

R

RobJN Tue 16 Oct 2018

I started a chat about this with other local groups via their telegram/IRC/riot supergroup. A few other suggestions but not all covered UK.

R

RobJN Thu 25 Oct 2018

So I think we should be looking for a monthly report and it it probably easier to start with stats about the community of mappers. Some ideas:

  • Number of people mapping within each month. -- Sub-level splits by experience level (e.g. new mapper, infrequent, junior, senior, gold); or -- Sub-level splits by activity (e.g. how many of the total number of people mapping each month did just 1 edit, 2-10 edits, 10+ edits).

It might also be worth splitting this by region (England, Scotland, Wales, NI, Isle of Man, CI).

We may also want to pull out a stat specifically about new mappers and/or experienced mappers who have no contributed recently.

Of the tools identified so far, Neis-One has the user categories (new mapper, infrequent, junior, senior, gold). It also has the OSMStats website which shows number of mappers per day. It also has stats about the top 500 mappers in the UK in the last 2 months. The OSMAnd Live tool also has good user stats by month.

The following are less useful: Meet Your Mapper seems too complex as it only runs on small areas and gives you everything back to the start of time. It does include last edit from each user for the selected geographic region though. Ohsome seems to be just Germany and Nepal for now. It feels like this will be better when it comes to measuring the data completeness rather than the community. For MissingMaps the stats are ok but it must be filtering somehow as I cannot find myself in it. Finally, osm-estratti.wmflabs.org is Italy only.

How to continue? I suggest mocking up a dashboard so that we have a visual idea of what we are aiming for. We can then see if we can do this on our own or if it is worth asking Pascal Neis and/or OSMAnd for some help.

Once we have this done, we may want other metrics about community. For example, number of users subscribed to our social media.

After that it becomes harder and I feel we should engage with others to try to figure out stats of map accuracy / completeness. For now Robert Whittakers work is great for UK theme specific projects.

NA

Nick Ananin Wed 26 Aug

Looking at the articles - one of the objects is to "promote and facilitate the use of OpenStreetMap data by individuals and organisations in the United Kingdom". Perhaps a measure of this is to have a record (list) of organisations using OSM maps in the UK. Perhaps as a starting point why not publish on our web site all the sites that we know of that are using OSM maps relevant to the UK i.e filter/review the list shown at https://wiki.openstreetmap.org/wiki/They_are_using_OpenStreetMap but also add to it e.g. http://marine.gov.scot/maps/1112, http://www.britainfromabove.org.uk/ etc. That would provide a measure of usage and might be of interest to other users/mappers