Improving Federation

The first step to all of this is still open: https://www.loomio.org/discussions/612?proposal=463

I think this is a really good starting point. We could set something up on the wiki that specifically dives into the problems we have, and I think we could probably get a much better map of how we want to address things.

Of course, I think that it's also important to consider existing and developing standards for federation. There is a larger problem than just our own platform here, and it involves every decentralized social network's ability to interoperate. Everyone has a difference piece to the puzzle,

Ideally, it would be wonderful to have an open standard that all social platforms could communicate through, but OStatus is slow in development and not really a one-size-fits-all solution, Tent is still in its infancy, and Zot2 is still largely only used in the experimental development of Friendica Red. There are also some platforms out there that use XMPP purely for broadcasting federation, and probably several more unused standards that no one's even heard about yet.

So that leaves us with the following general questions:

1. How exactly do we want to "fix" federation?
2. We have a lot of people on pods currently. How can we make sure federation scales out to support a large network of people, both on shared pods and their own lone ones?
3. What standards can we build on that will provide us proper structure for the features we want? (Example: If we want Groups to work with federation, what's the easiest, sanest way to implement that?)
4. What are the most awful problems currently plaguing how our existing system works?

The first step is to separate the federation to the rest of the code. After that, we can try different implementation for federation (our own one we currently have, Tent, XMPP...) make some stress-tests, do what we want, and choose the best solution at the end.

We can probably work with the Tent team when the federation will be separate. But we need to do it first. @florianstaudacher @jonnehass who are knowing enough the base code to do that? Is it possible to organize a meeting? I'm sure that 3/4 persons working together can do the big part of the job in one week-end.

What are the biggest problems in the federation? Could we start with listing them? Anyone want to start with a few? :)

Are you talking about bug that are caused by the federation? Just check github with the federation tag

The most annoying are:
* Public posts not visible between pods (go on a profile of someone from your pod, and then by entering hispod.tld/u/hishandle and you'll see that there is often more posts there)
* Delete post / comment are not federated
* Images are not federated (but this is maybe fixed by #3940)
* Profile is not always up to date (but this is maybe fixed by #3976)

Not a really a bug, but you will not find a post containing a tag by searching this tag if it is not already in your pod. Solving that would be awesome.

For us newbs, could one of the more experienced devs give at least a high level explanation on how federation is currently co-mingled into the app? Was there ever a plan made for putting federation into it's own layer or did that only get as far as deciding it should be done?

OK thanks @flaburgan.

For individual problems that are not really problems due to the big design there is an easy answer = they need to be fixed just like any other bug. I don't think we need much discussion on how to fix these and they certainly don't need to wait for federation being put in to a layer.

But larger changes should really be talked about carefully before implementing anything. We must have a plan for federation breaking changes and preferably do those in stages with a clear roadmap.

Personally I would like the small issues to be fixed as far as they can and then start working with separating the federation.

• Maybe a public post on DiasporaHQ and the website with simple explanations for users about what federation is, the problems and a small roadmap will be good for people to follow what is going oh here…

I believe indeed that pod communications (ie federation), which is decentralization, and account migration from one pod to another are what make D* special and really needed. This all about the first fundraising at kickstarter and its success. As long as they are not implemented and working, D* goals are not achieved and people will not see the point of such a social network among so many others.

This why communication toward users and newby devs should be more clear about these two points (decentralization and migration).

Public posts not visible between pods (go on a profile of
someone from your pod, and then by entering
hispod.tld/u/hishandle and you'll see that there is often
more posts there)

This is an interesting one. We should definitely fix this problem. Is there an issue for this already @flaburgan or a test case I can observe?

Delete post / comment are not federated

We're kind-of discussing that API in https://www.loomio.org/discussions/4093. I think we should see how that discussion and hopefully proposals turn out before making any decisions on deleting content through federation. I do feel like that is a necessary feature. Everybody makes mistakes. :)

The only major, horrible problem I can really see with federation is that it's very difficult for people who have no idea how Diaspora works to learn about how we federate content and build upon that. What little documentation exists is mostly concerned with how we securely authenticate content, which does matter but could really be encapsulated in a library (leaving that documentation unnecessary for most hackers).

@tomscott I'm sure there is an issue about that, I'll take a look.

But about the federation, there is two possible reasons that the post was not federated : a problem in the protocol or a problem in the processing of the queue by Resque (now Sidekiq). Sometimes it failed, and if it's not retried, the post will never be federated. So I don't know what caused this problem, we have to make test with small pods to precisely monitor what happens.