Loomio

Improving Federation

L3MNcakes
L3MNcakes Public Seen by 101

It seems like we are throwing an awful lot of feature proposals on the back-burner due to a need to improve the Federation protocol. In contrast, I have seen little to no discussion on actually improving Federation. I started this discussion hoping to change that.

Perhaps we could start by:

  1. Identifying all the problems with Federation that are currently holding us back

  2. Combine these into an easy-to-read document/wiki article so new developers can get a better a sense of what is going on (and probably be more able to contribute! ;)) and more experienced developers can have a single document to reference while brainstorming. (If there's not somebody on the "Documentation Editors" team that wants to do this, I will happily volunteer.)

  3. Establish an open team of developers who would like to collaborate and solve these complex problems. (Count me in!)

Please chime in with your thoughts. The above course of action is merely a suggestion, so if you have a better idea on how to approach this, please share!

Jonne Haß

Jonne Haß May 1st, 2013 20:59

The first step to all of this is still open: https://www.loomio.org/discussions/612?proposal=463

Sean Tilley

Sean Tilley May 1st, 2013 21:02

I think this is a really good starting point. We could set something up on the wiki that specifically dives into the problems we have, and I think we could probably get a much better map of how we want to address things.

Of course, I think that it's also important to consider existing and developing standards for federation. There is a larger problem than just our own platform here, and it involves every decentralized social network's ability to interoperate. Everyone has a difference piece to the puzzle,

Ideally, it would be wonderful to have an open standard that all social platforms could communicate through, but OStatus is slow in development and not really a one-size-fits-all solution, Tent is still in its infancy, and Zot2 is still largely only used in the experimental development of Friendica Red. There are also some platforms out there that use XMPP purely for broadcasting federation, and probably several more unused standards that no one's even heard about yet.

So that leaves us with the following general questions:

  1. How exactly do we want to "fix" federation?
  2. We have a lot of people on pods currently. How can we make sure federation scales out to support a large network of people, both on shared pods and their own lone ones?
  3. What standards can we build on that will provide us proper structure for the features we want? (Example: If we want Groups to work with federation, what's the easiest, sanest way to implement that?)
  4. What are the most awful problems currently plaguing how our existing system works?
Flaburgan

Flaburgan May 2nd, 2013 08:46

The first step is to separate the federation to the rest of the code. After that, we can try different implementation for federation (our own one we currently have, Tent, XMPP...) make some stress-tests, do what we want, and choose the best solution at the end.

We can probably work with the Tent team when the federation will be separate. But we need to do it first. @florianstaudacher @jonnehass who are knowing enough the base code to do that? Is it possible to organize a meeting? I'm sure that 3/4 persons working together can do the big part of the job in one week-end.

Jason Robinson

Jason Robinson May 2nd, 2013 11:55

What are the biggest problems in the federation? Could we start with listing them? Anyone want to start with a few? :)

Flaburgan

Flaburgan May 2nd, 2013 12:17

Are you talking about bug that are caused by the federation? Just check github with the federation tag

The most annoying are:
* Public posts not visible between pods (go on a profile of someone from your pod, and then by entering hispod.tld/u/hishandle and you'll see that there is often more posts there)
* Delete post / comment are not federated
* Images are not federated (but this is maybe fixed by #3940)
* Profile is not always up to date (but this is maybe fixed by #3976)

Not a really a bug, but you will not find a post containing a tag by searching this tag if it is not already in your pod. Solving that would be awesome.

L3MNcakes

L3MNcakes May 2nd, 2013 16:19

For us newbs, could one of the more experienced devs give at least a high level explanation on how federation is currently co-mingled into the app? Was there ever a plan made for putting federation into it's own layer or did that only get as far as deciding it should be done?

Jason Robinson

Jason Robinson May 3rd, 2013 07:12

OK thanks @flaburgan.

For individual problems that are not really problems due to the big design there is an easy answer = they need to be fixed just like any other bug. I don't think we need much discussion on how to fix these and they certainly don't need to wait for federation being put in to a layer.

But larger changes should really be talked about carefully before implementing anything. We must have a plan for federation breaking changes and preferably do those in stages with a clear roadmap.

Personally I would like the small issues to be fixed as far as they can and then start working with separating the federation.

Elm

Elm May 7th, 2013 09:46

  • Maybe a public post on DiasporaHQ and the website with simple explanations for users about what federation is, the problems and a small roadmap will be good for people to follow what is going oh here…

I believe indeed that pod communications (ie federation), which is decentralization, and account migration from one pod to another are what make D* special and really needed. This all about the first fundraising at kickstarter and its success. As long as they are not implemented and working, D* goals are not achieved and people will not see the point of such a social network among so many others.

This why communication toward users and newby devs should be more clear about these two points (decentralization and migration).

Tom Scott

Tom Scott June 2nd, 2013 17:36

Public posts not visible between pods (go on a profile of
someone from your pod, and then by entering
hispod.tld/u/hishandle and you'll see that there is often
more posts there)

This is an interesting one. We should definitely fix this problem. Is there an issue for this already @flaburgan or a test case I can observe?

Delete post / comment are not federated

We're kind-of discussing that API in https://www.loomio.org/discussions/4093. I think we should see how that discussion and hopefully proposals turn out before making any decisions on deleting content through federation. I do feel like that is a necessary feature. Everybody makes mistakes. :)

The only major, horrible problem I can really see with federation is that it's very difficult for people who have no idea how Diaspora works to learn about how we federate content and build upon that. What little documentation exists is mostly concerned with how we securely authenticate content, which does matter but could really be encapsulated in a library (leaving that documentation unnecessary for most hackers).

Flaburgan

Flaburgan June 3rd, 2013 08:00

@tomscott I'm sure there is an issue about that, I'll take a look.

But about the federation, there is two possible reasons that the post was not federated : a problem in the protocol or a problem in the processing of the queue by Resque (now Sidekiq). Sometimes it failed, and if it's not retried, the post will never be federated. So I don't know what caused this problem, we have to make test with small pods to precisely monitor what happens.

Nick

Nick June 16th, 2013 21:24

@flaburgan - as a non-techy, what would separating the federation from the rest of the code involve?

Nick

Nick June 16th, 2013 21:26

ok, sorry just saw that separate discussion and it seems to be something that has been agreed on pretty unanimously. Ignore comment!

Flaburgan

Flaburgan June 17th, 2013 09:28

@nickdowson no problem ;)

L3MNcakes

L3MNcakes started a proposal June 25th, 2013 17:57

First Step : Separate Federation Layer Closed 6:00pm - Tuesday 9 Jul 2013

I would like to propose that we move forward in improving federation with a first step of separating federation into it's own layer. This would involve :

1) Forming a working group of developers/collaborators who would like to volunteer their time to this problem. I think it's important to have a defined (though very much open) group dedicated to this problem, because simply leaving it open for an individual to pick up and run with leaves me with the feeling that it's simply going to never get done.

2) That group would come up with detailed plan for separating out federation and any potential problems that might arise. Communication can be done through IRC Meeting/Email Chain/Loomio/whatever works best for the people involved. The goal here is to produce an organized document that can be easily understood by newcomers who want to jump in and help out as well as a reference for those involved to ensure everybody is on the same page with what exactly needs to be done.

3) Plan could be verified/approved by other developers/collaborators through Loomio. (Maybe this step isn't necessary, but I think it's always a good idea to get outside opinions before implementing a plan.)

4) Plan is broken down into manageable tasks and put into Github where developers can pick them up and begin making code changes.

Results
Agree - 13
Abstain - 13
Disagree - 13
Block - 13
13 people have voted (8%)
L3MNcakes

L3MNcakes
Agree
June 25th, 2013 18:03

goob

goob
Agree
June 25th, 2013 18:36

We already have agreement to separate fed into a layer - https://www.loomio.org/discussions/612?proposal=463 - and I agree your practical steps would be an excellent way to go about it.

Elm

Elm
Agree
June 25th, 2013 18:47

Good way to go !

matl

matl
Agree
June 25th, 2013 18:47

theradialactive

theradialactive
Agree
June 25th, 2013 18:48

Faldrian

Faldrian
Agree
June 25th, 2013 19:19

Nick

Nick
Agree
June 25th, 2013 20:52

great plan!

Rasmus Fuhse

Rasmus Fuhse
Agree
June 26th, 2013 09:53

Yeah, that's important!

goob

goob June 26th, 2013 11:09

I see the link in my vote didn't catch. Here it is properly:

Put federation into a layer

Just to show there is already an agreement to do this, so L3MNcakes's practical proposal of forming a working group builds on this previous agreement.

Sean Tilley

Sean Tilley
Agree
June 26th, 2013 19:24

This is great, but I think the bigger issue here is not a matter of want, but rather a matter of developer muscle and resources. Federation is pretty complex, and so our federation system needs to be fully documented and studied.

L3MNcakes

L3MNcakes June 27th, 2013 16:12

@seantilleycommunit :: Good points, but not necessarily show stoppers. If we could even get 2-3 devs working on this to start out, I think we could take it a long way. I'm one! =) It would be nice if we had a developer who knows the code well to help out, but in the event that doesn't happen, I'll happily be the one to tear through the code until I understand what's going on. I would love to see some of the newer devs jump in to help figure it out too! I know the task is a daunting one, but it has to be done by somebody if we want our app to succede. This is one of the core pieces of our network and it's really holding us back on the features we can implement. I really hate telling users, "We can't do that because the distributed part of our distributed network sucks." I'd much rather start moving forward on this with limited manpower than continue waiting for a magical influx of dev power.

Sean Tilley

Sean Tilley June 27th, 2013 19:03

@l3mncakes If you want to take a look at our federation system, by all means take a look.

Here's some wiki entries that may be useful to anyone interested:

Seth Martin

Seth Martin
Agree
June 27th, 2013 20:01

I can't think of anything more important for this project.

L3MNcakes

L3MNcakes June 27th, 2013 20:56

@seantilleycommunit Sweet! Thanks for listing those resources here. Should be very helpful.

M

mathis
Agree
June 28th, 2013 06:52

Tom Scott

Tom Scott June 28th, 2013 20:17

Come on guys, do we really need a proposal to make a plan? :)

goob

goob June 28th, 2013 21:15

Absolutely, Tom. It's a means of agreeing a plan and getting something done.

L3MNcakes

L3MNcakes June 29th, 2013 16:35

@tomscott haha, I thought it would be best way to get the ball rolling on this and since it purposes doing things a little differently than I've seen them done before, I wanted to make sure the community had a say in the decision. C=

Dennis Schubert

Dennis Schubert
Agree
June 29th, 2013 22:10

BG

Bruno Guerra
Agree
June 29th, 2013 22:23

Florian Staudacher

Florian Staudacher June 30th, 2013 15:58

Florian Staudacher

Florian Staudacher June 30th, 2013 16:23

I hope nobody feels left out by me starting ahead on a clean-room implementation for federation... but it really is not such a complicated or huge task for it to require elaborated documents or planning.

My plan of stuff to implement for this gem is:
- DO NOT BUILD ON RAILS (bonus: lightning fast specs) [done]

  • get xml de/serialization on database-unrelated entities to work in both directions [done]

  • build a structure resembling what is currently known as 'to_diaspora_xml' [done]

  • put payload in salmon envelope and sign/encrypt that as it is currently done in D* [pending]

  • simulate sending/receiving inside a proof-of-concept test-app [pending]

Sean Tilley

Sean Tilley July 1st, 2013 02:04

@florianstaudacher Really cool stuff! Watching, will keep an eye on this. Do you think the availability of federation in a gem might, in theory, allow a developer the ability to incorporate Diaspora federation into just about any Rails app? :)

L3MNcakes

L3MNcakes July 1st, 2013 02:11

@florianstaudacher Cool! Thanks for jumping on that. I don't think planning or documentation is ever a bad thing, and it also gives an opportunity for others to learn new stuff and help out if they want, but I appreciate that you're taking the time to get stuff done.

goob

goob July 1st, 2013 09:56

Fantastic work, Florian.

Flaburgan

Flaburgan July 1st, 2013 13:31

@florianstaudacher thank you for dealing with that!

Florian Staudacher

Florian Staudacher July 1st, 2013 15:54

@seantilleycommunit in theory, yes, you could re-use the gem in other ruby projects. but keep in mind that the gem just represents the "business logic". you'd have to handle the http routes and the database persistence in the app - that's really out of scope for the gem.

Florian Staudacher

Florian Staudacher July 1st, 2013 21:29

  • put payload in salmon envelope and sign/encrypt that as it is currently done in D* [done]

;)

L3MNcakes

L3MNcakes July 1st, 2013 21:43

Started trying to put together some better documentation around this. Currently working off the articles Sean posted here and trying to make them more readable and useful. Not done by any means yet, but just a heads up on what I'm doing.

https://docs.google.com/document/d/1QEhIx10zFnBgAtf_GL6AMT9GvWjxfPE28QFb2rPIbHU/edit?usp=sharing

Florian Staudacher

Florian Staudacher July 5th, 2013 22:52

alright, the code in my github repo should now have every entity we use for federation, also the ones with nested entities inside (e.g. StatusMessage -> Photo, Location).
they can be de-/serialized from/to XML and are meant to be put in a Salmon envelope that can be de-/encrypted.
I still want to implement some rudimentary validation classes, so the received objects can be sanity-checked before they are being handled by an actual app, but apart from that I think we could start to think about how to test this on its own and with another instance of D* running the current federation code.

Flaburgan

Flaburgan July 11th, 2013 07:39

@florianstaudacher please ping me when you think the code is stable enough to be pulled on my pod.

Florian Staudacher

Florian Staudacher July 12th, 2013 14:00

... it will need a lot more work before we can actually run a pod with it.
not only do we have to give it a decent test period, also a good part of the diaspora code will have to be adapted just to use the new gem.

F

fabianrbz July 12th, 2013 15:49

@florianstaudacher I'll be more than happy to help you with this. I'll try to give a look at it this weekend, just let me know what needs to be done

Florian Staudacher

Florian Staudacher July 15th, 2013 14:37

Well, to anyone who would like to help, I can suggest to review my code and the docs I write with it.

I may be an intermediate-advanced programmer but that certainly doesn't mean everything I write is pure gold. I want this to be as good as it can get, so we have a rock-solid library we can use for Diaspora* and possibly to have something other projects could use to integrate federation into their software.

Anyway, this will become the new reference implementation of our protocol, so this has to be impeccable, readable and understandable.

Sean Tilley

Sean Tilley July 15th, 2013 19:25

@florianstaudacher Would it be possible to move these updated docs into the federation section of our wiki at some point? It'd be really great to brush up what little information we have on how the protocol works, and update it with all the information you've put together. :)

Nick

Nick July 16th, 2013 14:47

Hey, without understanding the technical stuff, it seems like there's been a lot of progress with this.
Seems to me like an agreed standard for decentralised social network federation should be a major outcome that the diaspora project should be working towards - and that this needs to be something that can be used by other social networks. I think it would be really good to work with a few other distributed social networks (movim and gnustatus come to mind as to my mind projects with potential) from an early stage to develop a protocol that can be flexibly and easily extended and some kind of process/foundation that becomes the agreed way of agreeing and improving this.
Now might be a good time to start thinking about this could be made happen? Working with groups such as the free network foundation (thefnf.org), the free software foundation, the electronic frontier foundation and others might be a way forward?

Nick

Nick July 16th, 2013 14:51

On a totally different note, any thoughts on how diaspora should deal with linking to profiles? On facebook etc. someone can give the http address to their profile which anyone can click on, look at and then press to subscribe/friend/etc. On diaspora this is more complicated if the profile is hosted on a different pod - plus being logged in or out and consistency of how sites look becomes a bit of an issue. Plugins or third party cookies might be a way of solving this but bring up obvious privacy issues... (sorry if I've not explained this issue very well...)

goob

goob July 16th, 2013 15:53

Nick, you can use http://dia.so/ to get a URL for your profile which will work on any pod.

Nick

Nick July 17th, 2013 08:52

ah, thanks @goob I didn't realise this, will check that out!

Jason Robinson

Jason Robinson July 18th, 2013 10:23

It's still a third-party tool ;) Something like this should be in the future integrated to the Foundation(tm) and officially supported in the code.

Also I'm fantasizing of a link in posts to take the user to "the post on my pod". For example when someone gives me a link to a post on another pod, I click it, I cannot reply or reshare unless I find it on my pod. Just needs some ajax query, and of course post must exist on the other pod.

DU

[deactivated account] July 21st, 2013 10:13

Has this descision https://www.loomio.org/discussions/612?proposal=463 been worked into GitHub issues? Or how will it be moved into actionable items?

goob

goob July 21st, 2013 10:36

@wilhelm , Florian's code, linked on this thread, is the start of the solution to the proposal you link to. See https://github.com/Raven24/diaspora-federation

Florian Staudacher

Florian Staudacher August 1st, 2013 14:04

alright, I'm quasi-finished with what I had planned to include in the gem.
Only one more design/architectural issue I'd like to bring up and solve with the wisdom of the crowd:

We have a lot of XML in the protocols around D*, so I have chosen the 'ox' gem to handle XML parsing and generation. It promises to be fast and has all the features we need...
Except for one occasion, where we use hCard - which is HTML and uses classes to label its data fields inside standard HTML elements. Ox doesn't know CSS or XPath selectors, so I can't select the elements based on the used class names directly. At that point I brought in libxml-ruby, which has support for XPath, which is enough to solve the problem in this case.

Personally I feel it's unclean to include another lib just for that one use case, and now I'm torn between leaving it like that, or just switching to libxml or possibly nokogiri entirely...

Ideas?

Jonne Haß

Jonne Haß August 1st, 2013 15:52

I think we should switch to Nokogiri. While it may not be the fastest library, it's the most feature complete and battle tested one, so using it should be future proof.

L3MNcakes

L3MNcakes August 1st, 2013 15:58

@florianstaudacher -- I don't really know much about the various libs themselves, but I agree that it feels dirty to include a separate lib for just that case. My first instinct is to modify ox so it could support that use case and then we only have to use one lib. Would this be feasible?

goob

goob August 1st, 2013 16:25

If you feel that ox performs better for the other cases, it might be worth considering whether Diaspora's hCard could be parsed using XML. I've had a look online and found suggestions that this is possible with hCard. This may not be a good idea in Diaspora's case, or may not be a clean solution in itself, but if you'd prefer to use ox, it might be worth considering whether it could work.

Apologies if I'm speaking out of turn, not really understanding the technical issues. It just struck me as a potential alternative approach to switching libraries, if you really like that particular gem.

Florian Staudacher

Florian Staudacher August 1st, 2013 17:00

Well, as far as performance goes, our XML documents are at most a few KB in size, so any speed variances should be almost undetectable.

Modifying Ox to support advanced selectors is way out of scope for this task and I don't think it's in the spirit of the lib - being mostly fast and lightweight.

I thought about parsing the hCard as XML, but I decided against it, since the idea behind XML is structure and hCard as such has no structure.

I guess I'll be moving to Nokogiri, it seems the least controversial... ;)

Florian Staudacher

Florian Staudacher August 4th, 2013 23:28

(in case anybody missed my post on d*)

okay, porting to nokogiri is done. now it's time to give the code a thorough review. so if you have any ruby knowledge, look it through and give me feedback on anything that might jump in your eyes.

documentation and readme will follow later

goob

goob August 5th, 2013 12:48

I've just started a new discussion with some ideas for adding the possibility of pulls to Diaspora's push model for federation. I didn't want to start a new discussion on this thread, which might confuse things.

Hope it's useful.

Nick

Nick August 17th, 2013 15:19

I created a new discussion for what protocol we use & working with other social networks:

https://www.loomio.org/discussions/6233

Florian Staudacher

Florian Staudacher August 21st, 2013 17:55

another update: the Readme contains more infos, and I think the docs should be mostly finished.
So if you get a chance, you can have a look at the docs (http://rdoc.info/github/Raven24/diaspora-federation/master/frames) and tell me if everything is understandable and usable for implementing the gem.