The core team -- a bottleneck of the diaspora* development
First of all, I appreciate the job that the core team does for the diaspora* project. They are appreciably high qualified professionals who dedicate their personal time to the project.
On the one hand, diaspora* project has a non-trivial history, high media coverage in the past and still sometimes in the present. Diaspora* has a comparatively large community. There are people who like diaspora* and need diaspora and want diaspora* to develop and thrive. Diaspora* has good conditions for thriving, again, comparatively to many other free as in freedom software projects. Of course one can't know for sure. World is changing fast. Technologies change even faster. Maybe community's interest for federated social networking will go down as has gone down interest for the desktop GNU/Linux as the smart mobile devices market emerged. We may be on a peak now or may be not, but if we are, then we don't have too much opportunities left to retain as a tool that makes the world a better place. I don't believe that the world is constantly changing to the better. I think that some events may turn it to worse. So what I say is not about trying to jump in the train of progress. It rather about giving people an instrument for a change before it's too late. Of course I don't like to present the thing as a Mission, but that is what I believe diasporians want -- a tool that will make their lives and the world better. And if diaspora* fail, then they may at the end decide, "fine, I like the federation idea, but I can't replace facebook with it, so whatever". So, basically, diaspora* has a rare (nearly unique) opportunity to make an important improvement in the technological part of the social development.
On the other hand, diaspora*'s growth rate is limited mainly by amount of the core team participation. In the past I thought that decent free software projects stagnate nevertheless because there are not enough coders who want to participate. At least for diaspora* it doesn't seem to be true any more for me. People who maintain the project -- the coreteam -- these are who are responsible for planning, QA, code review, releases, they are the bottleneck (they can do coding also, but that's just a special case of a random contribution :)). The number of opened pull requests is always around 25-30. So there are enough people who write code. But it gets painfully long to get the code reviewed and merged.
I did some job for diaspora*, and it still hasn't received sufficient feedback. I did my job full-time, and this means that having one full-time coder at the time is too much for diaspora*. I know I'm not that brilliant as a programmer and I could have done my job better, so that it took less time of the maintainers to explain their points to me and to spend less time testing my changes. However I am capable of 1) improving and learning; 2) getting things done. I gave a resource to diaspora* that many other free software projects don't have. And it was spent inefficiently. That formally is my fault, but I'm also very surprised with the fact the core team members never helped me to avoid this issue, like it isn't in the interests of the project.
Of course there is nothing wrong in that people from the core team want to share only some limited time as volunteers. They have jobs, families and all the different things people normally have. And they participate to the project for fun mostly, not for some high idealistic goals like that which diaspora* began with. So we can't demand more participation from them.
It's a complex issue and I don't now how to solve it. The issue is that you can't catch a programmer a make him/her do a code review that she is unfamiliar with. You need to know the code base to do the job that is the bottleneck currently. Or you doesn't? I don't know.
So what I think of we can do to solve this issue as a brainstorming:
• Be more risky when introducing the changes to the code and care less if something fails. Value development speed (to a reasonable extent) more than stability.
• Create an "experimental" branch where we merge stuff more quickly with a declared low-responsibility level.
• Find some risky podmins who are ready to test highly experimental features.
• Create a target group of podmins who will install highly experimental versions and give a verbose feedback.
• Improve tests automation so that less time is spent on the changes testing.
• Automate code review to a possible extent.
• Create a QA team who will do the job of making releases and testing and take these responsibilities away from the core team
• Call for more volunteers and give them as much job as they can do without the deep knowledge of the code base.
• Take someone and train him/her to make code reviews of the proposed changes.
• Collect some money and spend them on something of the above or something else.
Opinions?
David Morley Sat 12 Nov 2016 2:04PM
Great discussion. I think you have many "lurkers" like myself around this project, we are interested and love the project, we have no idea how to be more apart of it. I would suggest if the core team has some items that are non-core they could list some of us might be able to grab them and help the core team focus on things their time would be best used for.
Theatre-X Sat 12 Nov 2016 2:34PM
I can't do much in the way of technical abilities even though I've done some translation stuff in the past. I would be interested in doing full-time management. However I would need some time to wrap my head around the current workings and happenings of the production. For what it's worth.
Dennis Schubert Sun 13 Nov 2016 12:35AM
@fabianrbz for example did some nice PRs and has been invited to join the team pretty quickly
He has been inactive for a long time and is no longer part of any team, see here.
I highly doubt we're able to "just find someone" or even "pay someone" to do project management. First of all, good managers are already working in this position and get a lot of money for that. I highly doubt we're able to collect $100k+ yearly to pay a manager. Second, people who want to manage this project need to understand the project, know its goals, know its community and related. You can't just pick a random person and say "here you are, good luck!" or we'll be dead.
but this is a bit of an over-generalization
Eh, nope, that's been said after looking at the user statistics.
Comrade Senya Sun 13 Nov 2016 7:17PM
We will not change the world. Sorry.
(disclaimer: this paragraph is just a long rant on the "philosophical" question, if your time is limited, you may want to skip it).
Yes, we will. Actually, I never stated anything about diaspora* getting "big" or making more people care about their online privacy. That's not what I fight for. However, what I'm talking about, is that we can take care of people who care about online privacy today, and who want to have a good tool while they are fine with some shortcomings like no address book synchronization (which can be viewed as a virtue). If diaspora gets better, they'll stay, if not, then, these people can turn to "I can't hide so why care?". Even if they still care about online privacy, diaspora can make lives of these people better. People do change the world by their everyday decisions. Diaspora community is a part of the world, and by changing Diaspora to better or to worse you change the world. That's not a goal or a dream or a utopia, that's reality and we're responsible for every decision we make in that respect. Dennis, being a maintainer of a projects with large user base, you make a change more considerable, than you would want to. You can state it like "I'm just playing around here, nothing serious". By this you just ignore all the people who are starving to see "alternative" technologies being developed. And that changes the world as well. You deny that "alternative" technologies is capable of being something more important than just a playground and state that one shouldn't fight it. And that's a legit point of view. I disagree with it, I think "alternative" technologies is something that can grow to become a serious tool for a number of people affecting their lives. But that's not the point. Just don't say that we don't change the world. We do, the world changes everyday by decisions of people.
I remember @denschub committed to prioritising your work after the release of 0.6.0.0, so hopefully things will speed up now. Perhaps he can give you a progress report.
Honestly? I can't.
I took your comment at 23:31:45 of our last meeting – 'now that 0.6.0.0 is done, I will focus on senyas pr' – to mean you'd be prioritising his work on account migration.
@goob is right, @denschub, you, said that. And afterwards the only thing you did, was only a single message on #908. In any case I can't force you, so if you just ignore my work I can't do anything about it. But we need a way to introduce new features and this is a separate issue then, so I created a separate loomio thread for this.
If someone feels like doing management stuff full time, I'd never block that - but so far that has yet to happen.
Actually we can try doing project management collectively using online tools like Loomio.
We could maybe start with a wiki page of things to check when doing a review, which could be useful for both reviewers and developers who want to check their work themselves .
That's what I miss! Having a nice guide on code review may help both contributors and new reviewers of the project. May we request such guide from the members of the core team? Their point is important here since currently these are they who is define what a good review is in scope of the diaspora* project. If you can't write a new guide, maybe we can adopt guidelines from some different project/source?
Improve tests automation so that less time is spent on the changes testing.
Automate code review to a possible extent.
We're already doing that. We have styleguide checks and automated tests that run on all tests for every PR. What do you think we could improve here?
Well, I can't say anything on improving the code review automation. That's a rather research topic. However, at least I can say about improving testing by introducing more tests within a test infrastructure, like I proposed many times earlier. Automatic test pods deployment + tests using diaspora client interface. That what can give us more confidence and I need to hear opinions on the progress I've made in this respect (e.g. this PR).
Create a QA team who will do the job of making releases and testing and take these responsibilities away from the core team
Sure. Who do you feel like is capable of doing that and are they willing to take that responsibility? Would be nice.
It depends on what it consists in. My impression is that making releases is rather a routine operation and can be done without an extremely deep background, isn't it? I mean that if we find a volunteer who takes it responsible enough, it's not hard to teach him/her a procedure and take the load from you. It doesn't require deep knowledge of the diaspora* design.
Also, yes, as Steffen said: small PRs are much easier to review. Large PRs, for example your stuff (still talking to Senya here), is hellish compelx and I don't even know some parts of the sources you touched, making it even harder to review.
The last release was a few days late since I had to deal with other stuff and at the moment, I am fully loaded with paid stuff, basically being responsible for two large projects is completely eating away my brain. If I'm able to do anything after work, that's basically reviewing simple PRs and trying to kick rails5 arse, but reviewing large PRs is impossible.
Basically, with a present situation we end up being completely unable to introduce any big changes to diaspora. Even if there is a contributor who writes something, the "maintenance" part of a pull-request is completely hopeless. So without taking any measures diaspora* will be nearly stagnating.
Everybody can do reviews on github (they even have a nice new review-feature since a few weeks :) ), and with this they get to know the code-base better and better and can do more code-reviews. Maybe this works, but we need people for this.
So basically we don't need code reviewers to know diaspora source too well, but rather we need them to have good Ruby and RoR understanding and experience. It simplifies the issue in the respect, that we can find someone who is unfamiliar with the code and ask them doing the reviews. However good professionals are rare, and even commercial companies who pay good money have hard times hiring a decent engineer.
So maybe we can keep an aspect in the diaspora HQ account with those podmins, and when a PR feels ready but needs to be tested before being merged in develop, we can ping them and suggest them to test it
I like this idea. Maybe we can even do it in public, so we don't need an aspect, but, maybe a separate account/hashtag/mailing list/etc.
As we discussed in IRC with @SuperTux88, some PRs are fine with getting tested in a local dev environment and it is possible that we don't need to disturb podmins with it, but just find some people who are ready to help and do tests locally. To make decent tests one should possess the skill of "breaking things". So a random person doesn't fit, but possibly we can still find someone.
I have another idea that just came into my mind. First I thought of making some publicly available test-deployment of diaspora. So that it is easy to perform tests. For example I made PR6818, and I deploy a pod with this version so that people can play with it, try it and try to break it. But then I thought, what if we create a whole separate federation for testing? Starting with a few pods. We may explicitly state it as something unstable and unreliable that doesn't fit to everyday use. And test PRs freely in it. I'm pretty sure there is a number of people who will happily want to use it and explore. This way we can have pods with upstream vanilla versions deployed there and pods with PRs deployed there and allow developers to deploy their own test pods for any purposes. We can then freely test new things there and don't be afraid of losing people's data. Of course, we need some means of separation this from the main diaspora federation, because interaction of the test and the real federation can cause troubles. For example, we can create this test federation within some public VPN network with some local VPN-wide domain names. Thus you can also avoid using a real domain names for testing purposes, which is good, because some kind of damage in data can't be fixed: if we lost a private key of a user he is lost to the federation.
So to sum up my post.
These are what kind of jobs we have that could help us to avoid stagnation (or if you don't like it, to speed up diaspora* development):
• a web-engineer with good Ruby and RoR background to do reviews;
• people who are good at breaking things (finding issues) to do decent tests of PRs;
• people who can continuously contribute their time and energy to take the release making procedure away from the core team;
And the other ideas I like are:
• Do the project management collectively using online tools as much as we can;
• Introduce a code review guide;
• Develop test infrastructure (a separate test federation?)
As we all known, volunteers can do a great job. And at the same time, even if we have money to pay someone for some job, it doesn't guarantee that we manage to find a person who will want to do it properly and responsibly. Also, diaspora* has the image of a project that constantly asks for money and doesn't deliver what was promised. Unfortunately, I made this situation even worse. (And though it is all my fault, but I still keep wondering why no one from the experienced diaspora* members ever foresaw what it can get to and at least warn me). Having money to pay surely increases our chances to persist and develop, but I don't feel like it's okay to ask the community to donate money once more. What about my personal participation, the fact is, that I have to earn money for living. And if nothing changes I'll have to get a "normal" job after a few months (I already searched for something part-time, but software companies normally don't like this type of employment). Since I promised that I'll finish the migration project whatever it stands, I will keep contributing to diaspora in my free time just like the core team does, but that will definitely make it slow to deliver and doesn't save the project from stagnation (I mean what I said before about the fact we can't release any big feature).
Sorry, if I missed addressing any other stated points, it just gets complicated enough already.
Flaburgan Thu 29 Dec 2016 9:55PM
I'm doing a "up" here because unfortunately, nothing really evolved and I thought that as most of the core team is at the #33c3 actually maybe this topic can be discussed more easily.
@comradesenya you're talking about testing. I feel the current state is good enough. PRs are tested locally, eventually using #7142 before being merged in develop. The code then sit in the develop branch for a while, with several pods with hundred users testing it. Until now, we never had a big regression breaking everything. So, testing is not the problem.
@denschub you're talking about a project manager, I guess by that you mean a product owner aka someone defining priorities, the goal, where the project should go, resources management etc. In the case of the migration feature, the users clearly showed their interest. No-one in the core stood against it, the feature interested everyone. The developer was found and paid, no resource management here. So the project management wasn't the problem.
To me, the problem is clear: we're missing people to do the code review. Because the lack of time of @supertux88 @jhass and @denschub and maybe @florianstaudacher and because the lack of the competence from other members like @steffenvanbergerem and myself on this part of the code.
We need to rise the capacity of the core-team if we want to save diaspora* from stagnation.
We can:
- Improve the usage of the time the core already spend on diaspora = delegate the simple tasks to non-core members. They can help by doing pre-reviews, by helping during the release time, by helping to manage translations (we recently made progress on that thanks to @denschub), by answering questions on IRC from podmins etc... (please complete that list). To be really efficient here, it would be nice to know which tasks are the most time consuming for core-members so we can focus on them to help. Maybe the core-members can note how many time they spend on each tasks?
We can also attack the problem the other way, by looking at what we can do and checking there if we can help. @davidmorley said above he could maybe help. What does he know? He is a skilled sysadmin. So he can maybe help maintaining diasporafoundation.org server? (Just raising the idea) Hey, that is a task done by a core-membere we didn't think about until now... (Gonna catch'em all!)
- Allow the core to have more time to work on diaspora, this decision mainly has to be taken by the members themselves, we're not going to force anyone here of course, managing his time is something really personal. To take a day-off once a month (or a week) to work on d* could be a solution. d* could even maybe pay this day. This has been suggested before but most of the core members answered they were not interested and this is absolutely understandable. A more fun way to do that could be to do it in a hackathon, we pick a date and all take 3-4 days off and work full time on diaspora* during this time, maybe not IRL as it can be complicated, time consuming and expensive but all together linked by IRC / video chat / whatever. This can be a fun way to give a big boost to diaspora.
- Add new members to the core team, as said taking the example of @fabianrbz even if he is not part of the core anymore, the actual core team doesn't look closed to new people joining (please confirm). So, if some non-core developers are doing some nice pre-reviews giving useful hints and doing good testing, they could join the core. The problem now becomes "how to find new contributors".
To succeed, we need to work on these 3 points.
About point 1, the solution could be as simple as core-team raising on IRC "I'm going to do X, this is not a complicated task, if someone wants to do it for me we can do it together for the first time and then he'll be able to do it alone".
About point 2, a lot of things are possible, it's up to you my friends, what you want to do.
About point 3, there are many ideas and most of them can be done without being part of the core: create a code review guide, be more active with the diaspora HQ account and on twitter, search for contributors in the rails community or elsewhere... I can definitely be more active on this topic, though I don't want to work hard to bring new contributors and then see their PRs being never reviewed... :P
@augier and @comradesenya you can also try to review each other PRs ;)
Wow, that was a way bigger text than I thought I was going to write. But I feel it was worth writing :) I'm happy to be part of this awesome project with all of you mates!
Comrade Senya Thu 16 Feb 2017 10:55PM
@flaburgan, so basically we can improve the situation by taking tasks away from the core team (IRC responses, diasporafoundation.org management, loomio discussions, github management, any type of typical request from the community, etc) and by attracting new contributors, which can possibly be done systematically.
Bad news are that there seem to be no one willing to do this currently. Good news are that it doesn't have to be someone who knows how to code for Rails and it's not necessary to have knowledge of the code base to do that. So it is more likely to find someone, because the requirements are less strict than for a code reviewer. Maybe we need someone like a "community manager". I know diaspora* had one before who left, but I don't know why he wasn't replaced by somebody new then.
So one of the simplest things we can do is to start searching for possible "community management" contributors by making posts and promoting this as an "open position". It is not necessary to have one single person for it, but if we find at least one it would be a good thing already.
We already have a page in the "how to help" style, like most OSS projects do. This page isn't that useful as it could be because it doesn't explain what we need more and what are more important contributions for the project. Maybe we should describe the "community management" job in details and integrate it to that page, so that people who come there could realise that this could actually help project, if they come to IRC, Loomio and GitHub and help the team in communication with the community (e.g. by answering typical requests).
Also we as the long term contributors know what diaspora project miss more and what less. For example we don't need new feature requests too much, because it is unlikely they will be implemented on our own with our present resources. Opening a pod also is not something that will help diaspora development itself, we have a decent number of pods already. So maybe we should write down some priorities on what we need more and less to the Other ways to contribute.
So to sum up, we have to define what we need for our development, what kind of contributions and then promote it in hope that we can reach someone. Besides of changing the wiki, making a blog post on this topic can be worthwhile.
Also we should write a blog post about "why the federated web is something worth developing and participating" so that the newcomers can find some motivation there.
Flaburgan · Fri 11 Nov 2016 11:27PM
About you suggestion:
The diaspora* code was (and some parts still are) a real mess. We don't want to go back to that, especially because to be backward compatible most of the time.
I'm not a fan of that either. It means more code managing work and as raised above, can create fear of breaking stuff, without knowing what the cause is. The suggestion to keep the develop branch but test specific PRs on some specific pod looks easier and more useful. Which bring us to
I do that already. I think that all podmins who are running the develop branch are doing that already. So maybe we can keep an aspect in the diaspora HQ account with those podmins, and when a PR feels ready but needs to be tested before being merged in develop, we can ping them and suggest them to test it? Of course, they have to understand that things can break and be ready to rollback.
If you have suggestions about how to improve what we currently do, please share :)
Again, the problem is more about finding those persons than to manage that.
We're already trying to do that with the newcomer label, and we're calling for help in each release announcement. I would love to see more contributors coming, not sure how to do so.
We could maybe start with a wiki page of things to check when doing a review, which could be useful for both reviewers and developers who want to check themselves their work.
We actually do have a few hundreds dollars we could use, but we don't know how to spend them efficiently actually.
Again, thanks for rising this discussion :)