Hoodie Community Dashboard

[gr2m]

If you asked me:

How many active contributors does Hoodie have today?

I could not answer it. Nor could any other maintainer from any other Open Source project that I asked so far. And this is a problem, because Open Source Burnout is real and yet we don’t measure the underlying problems in ways we measure code quality.

What we don’t measure, we cannot improve.

The question about active contributors is only one aspect. What I am really interested in how well balanced the community is between active users, contributors and maintainers.

Goal

The goal for the Hoodie Community Dashboard is to be able to answer this question at all times, and make the underlying measurements transparent to everyone.

Out of scope:
In future, I would also measure the success / impact of the Hoodie community which would include things like the reach we have, number of first-time open source contributors, diversity numbers etc.

Measurements

1. Work load

Measuring amount of users is hard for Open Source project, for good reason. But while it would be nice to know how many active users we have to measure the success of Hoodie, we are only interested in how much work load people produce that the Hoodie community has to take care of. Things that we can measure are

• number of open issues
• number of open pull request
• avg. time until response
• ... what else?

2. Active contributors

At Hoodie, we think contributions go beyond code and documentation. Equally important is the work from our editorial and design team, people helping answer questions in slack or on GitHub. In opposite to Load of users, we are not interested in amount of contributions, but in amount of different people who do the contributions, as we are not interested to have a few people do huge amounts of work, but in having a big group to balance the work load.

We can experiment with the details, but for a start I would define active as "contributed within the past month"

A contribution can be one of the following (from people who are part of the Contributors Team on GitHub)

• comment
• commit to pull request (all commits should go through pull requests)
• reaction
• donation on Open Collective
• number of new contributors
• number of contributors that became inactive
• number of contributors that became active again
• ... what else?

3. Active maintainers

Traditionally maintainers are seen as gate keepers in Open Source projects, often times referred to as "committers". At Hoodie, we see maintainers being in charge to maintain and grow the space in which people enjoy becoming and staying an active contributor. Just like with contributors, we are less interested in the total amount of work by maintainers, and more interested in the total amount of active contributors.

Activities by maintainers are

• Pull Request reviews
• number of camp issues created (https://github.com/hoodiehq/camp/issues)
• number of contributors / maintainers added to the respective Hoodie Team
• ... what else?

Visualisations

To be done.

Basically I would love to see different charts, the main one showing the "community climate" indicator (or however we want to call it) over time.

I would like to add these visualisations to hoodie.camp (it currently is a simple prototype only showing open issues).

Besides having a website, I would like to be able to send out weekly and monthly reports via email

Feedback

We are actively discussing all aspects of the Hoodie dashboard and are very interested in your thoughts, questions and insights into existing tools or our experiences with other Open Source communities

[hzoo]

https://libraries.io/ has a lot of awesome stuff by @andrew which you probably know about.

1. Load by users

Stats like # issues/PRs should be easy to track via https://www.githubarchive.org/ or just the github API since it's pretty straightforward. Also big query, 2, and more.

Actually just looking at the winners/entries for the data challenge might give some more inspiration in general - https://github.com/blog/1864-third-annual-github-data-challenge. Not sure why there wasn't one last/this year though.

I know there was http://issuestats.com/ (might be down) that tracked avg time to close an issue/merge a PR + badges and a graph (won the github data challege previously).

You can also track activity of 3rd party stuff (tweets with @hoodiehq), stackoverflow questions, slack messages (probably hard to measure if we aren't paying).

Probably not very many data points but other things like # of blog posts related, # of meetups/conferences/videos, # of talks.

2. Active contributors

Comments are really good - also active conversations via slack/twitter or other ways we do community engagement.

[jasonLaster]

Thanks @hzoo, I think it would be nice to share some Big Query APIs

We're also working on gathering contributor information in a google sheet as well. The sheet helps us keep track of the names and backgrounds of our contributors so that we can answer several maintainer questions:

• do we have a contributor who has a skill that can help (designer, right-to-left language speaker, linux user)
• how many contributors do we have who have mastered some skills, have domain knowledge in a component, what do they enjoy doing
• when was the last time we talked. what's going on in their life. burn out risk. opportunity to be mentored, grow...

We are also working on the right process for checking in on the community:

• discussing it in weekly meetings
  • go over active contributors
  • discuss possible churn, opportunities for mentorship, promotion

[lasomethingsomething]

@gr2m Saw a tweet directing readers to this issue -- very exciting. I work in Berlin at Zalando (huge, publicly traded company) as open source evangelist. Would love to chat, but in the meantime wanted to share these links to some dashboards (apologies if they're already known to you);

• Zalando's dashboard; code here; we currently rank projects using a formula that rewards recent activity, stars, # of contributors and maintainers
• Bitergia's Grimoire Labs; cc @dicortazar

I'm aware of other initiatives falling along these lines; happy to talk more.

[dicortazar]

Hi there, willing to help with metrics :)

[andrew]

Let me know if I can help pulling any statistics out of @librariesio for you

[dicortazar]

BTW, I run a small analysis of ten projects of HoodieHQ, you can have a look at http://cauldron.io/dashboards/hoodiehq . This is based on the tech we have at grimoirelab as referenced by @LappleApple .

You can see aggregated info per data source (Git, GitHub Issues and GitHub Pull Requests) and each panel provides several charts and tables. You can drill down, filter or even export the data and use your own viz.

Indeed you have info you already mention such as the number of open pull requests and issues, people involved in them, time to close those pull requests and issues, time zone distribution of commits and developers and others.

Hope this is useful!

[nayafia]

So happy you're doing this! A few more ideas:

avg. time until response

Maybe average time til an issue/PR is closed, as well? Basically tracking how long it takes to resolve. I think support departments often track this.

number of open issues
number of open pull request

Also the number of opened issues/PRs, i.e. the rate at which they are being opened each week/month/whatever (which is more about growth).

number of new contributors

This is probably implied, but I'd also track the number of repeat contributors, the ratio of first-time to repeat, and how that changes over time.

Visualisations

If you haven't seen it, icecrime's vossibility project might be useful here.

[lasomethingsomething]

+1 to @nayafia's ideas here. Nadia, are you saying that vossibility addresses your point? From the README I can't tell, exactly.

@icecrime, do you have other docs? (if not, let me know if you need help on that; I tend to follow this README template I created for Zalando.

[nolanlawson]

# of StackOverflow questions and average time until a StackOverflow response are also good things to track.

Dunno about Slack, but for IRC there's https://botbot.me/ which tracks logs and can be used to calculate messages per day (probably need to average it over number of users, though, because of bots).

[gr2m]

Cate mentioned foss-heartbeat by @sarahsharp

[nayafia]

@LappleApple I meant that vossibility might be a useful tool for creating dashboards/visualizations of any GitHub data collected. vossibility-collector has a bit more info in its README.

[icecrime]

Thanks for the ping!

Vossibility is a tool I created to help me manage the 🐳 Docker open source project. I do wish it was easier to consume or use in other projects, it's mostly a matter of documentation 😞

TL;DR: vossibility takes GitHub data, transforms it to extract the information you want and to enrich where necessary, sends all this to Elastic Search, and then you can use the wonderful Kibana as a frontend.

These are some examples of how I'm using vossibility today:

• Visualize time to process pull requests:

• Visualize how the activity is spread among the different repositories, and how this evolves over time:

• Understand where contributions are coming from (Docker employed maintainers, external maintainers, or the broader community), and the respective merge ratio (merged/total ratio) for each:

• Visualize what kind of issues are opened through pie charts that break down our different area/*, kind/* and version/* GitHub labels:

• Build a "live changelog" by listing all merged pull request that bear a special impact/changelog label.

• Measuring activity in order to identify potential maintainers. For this, we're using the number of unique GitHub items (issues and pull requests combined) a user has interacted with (meaning: opened, commented on, reviewed, etc.). The goal is to identify people who care about more than their very own issues or contributions: opening a pull request counts just as much as commenting on someone else's, and commenting 20 times on your own issue won't contribute more than opening it in the first place.

• Frankly useless, but fun idea of generating an automated "weekly digest" of what happened on the project.

Happy to discuss more if this kind of information is helpful 👍

[dicortazar]

Hey just to mention (do not want to spam! ^^) that grimoirelab supports the following data sources: askbot, bugzilla, confluence, discourse, gerrit, git, github issues, github pull requests, mbox, jenkins, jira, mediawiki, meetup, phabricator, pipermail, redmine, rss, stackexchange (stackoverflow), supybot, telegram, kitsune and remo. There's some extra info of Perceval, the retrieval tool at: https://github.com/GrimoireLab/perceval

That means, that having all of that information in a database (ElasticSearch mainly), we all can go for the metrics that you're mentioning such as people and evolution of contributors in all of those data sources, activity for all of those data sources, etc... And then on top of that, build more advanced analysis, such as the demographics of the community or some others.

btw, is any of you attending FOSDEM? that could be a great place to meet and discuss about metrics. We're also having this workshop to talk about metrics and show how to use the grimoire toolchain, just in case you're interested [http://grimoirelab.github.io/con/] and we also have this collaborative book https://jgbarah.gitbooks.io/grimoirelab-training/content/ where that's also detailed.