Loomio

Improving Federation

L
L3MNcakes Public Seen by 101

It seems like we are throwing an awful lot of feature proposals on the back-burner due to a need to improve the Federation protocol. In contrast, I have seen little to no discussion on actually improving Federation. I started this discussion hoping to change that.

Perhaps we could start by:

  1. Identifying all the problems with Federation that are currently holding us back

  2. Combine these into an easy-to-read document/wiki article so new developers can get a better a sense of what is going on (and probably be more able to contribute! ;)) and more experienced developers can have a single document to reference while brainstorming. (If there's not somebody on the "Documentation Editors" team that wants to do this, I will happily volunteer.)

  3. Establish an open team of developers who would like to collaborate and solve these complex problems. (Count me in!)

Please chime in with your thoughts. The above course of action is merely a suggestion, so if you have a better idea on how to approach this, please share!

JH

Jonne Haß Wed 1 May 2013

The first step to all of this is still open: https://www.loomio.org/discussions/612?proposal=463

ST

Sean Tilley Wed 1 May 2013

I think this is a really good starting point. We could set something up on the wiki that specifically dives into the problems we have, and I think we could probably get a much better map of how we want to address things.

Of course, I think that it's also important to consider existing and developing standards for federation. There is a larger problem than just our own platform here, and it involves every decentralized social network's ability to interoperate. Everyone has a difference piece to the puzzle,

Ideally, it would be wonderful to have an open standard that all social platforms could communicate through, but OStatus is slow in development and not really a one-size-fits-all solution, Tent is still in its infancy, and Zot2 is still largely only used in the experimental development of Friendica Red. There are also some platforms out there that use XMPP purely for broadcasting federation, and probably several more unused standards that no one's even heard about yet.

So that leaves us with the following general questions:

  1. How exactly do we want to "fix" federation?
  2. We have a lot of people on pods currently. How can we make sure federation scales out to support a large network of people, both on shared pods and their own lone ones?
  3. What standards can we build on that will provide us proper structure for the features we want? (Example: If we want Groups to work with federation, what's the easiest, sanest way to implement that?)
  4. What are the most awful problems currently plaguing how our existing system works?
F

Flaburgan Thu 2 May 2013

The first step is to separate the federation to the rest of the code. After that, we can try different implementation for federation (our own one we currently have, Tent, XMPP...) make some stress-tests, do what we want, and choose the best solution at the end.

We can probably work with the Tent team when the federation will be separate. But we need to do it first. @florianstaudacher @jonnehass who are knowing enough the base code to do that? Is it possible to organize a meeting? I'm sure that 3/4 persons working together can do the big part of the job in one week-end.

JR

Jason Robinson Thu 2 May 2013

What are the biggest problems in the federation? Could we start with listing them? Anyone want to start with a few? :)

F

Flaburgan Thu 2 May 2013

Are you talking about bug that are caused by the federation? Just check github with the federation tag

The most annoying are:
* Public posts not visible between pods (go on a profile of someone from your pod, and then by entering hispod.tld/u/hishandle and you'll see that there is often more posts there)
* Delete post / comment are not federated
* Images are not federated (but this is maybe fixed by #3940)
* Profile is not always up to date (but this is maybe fixed by #3976)

Not a really a bug, but you will not find a post containing a tag by searching this tag if it is not already in your pod. Solving that would be awesome.

L

L3MNcakes Thu 2 May 2013

For us newbs, could one of the more experienced devs give at least a high level explanation on how federation is currently co-mingled into the app? Was there ever a plan made for putting federation into it's own layer or did that only get as far as deciding it should be done?

JR

Jason Robinson Fri 3 May 2013

OK thanks @flaburgan.

For individual problems that are not really problems due to the big design there is an easy answer = they need to be fixed just like any other bug. I don't think we need much discussion on how to fix these and they certainly don't need to wait for federation being put in to a layer.

But larger changes should really be talked about carefully before implementing anything. We must have a plan for federation breaking changes and preferably do those in stages with a clear roadmap.

Personally I would like the small issues to be fixed as far as they can and then start working with separating the federation.

E

Elm Tue 7 May 2013

  • Maybe a public post on DiasporaHQ and the website with simple explanations for users about what federation is, the problems and a small roadmap will be good for people to follow what is going oh here…

I believe indeed that pod communications (ie federation), which is decentralization, and account migration from one pod to another are what make D* special and really needed. This all about the first fundraising at kickstarter and its success. As long as they are not implemented and working, D* goals are not achieved and people will not see the point of such a social network among so many others.

This why communication toward users and newby devs should be more clear about these two points (decentralization and migration).

TS

Tom Scott Sun 2 Jun 2013

Public posts not visible between pods (go on a profile of
someone from your pod, and then by entering
hispod.tld/u/hishandle and you'll see that there is often
more posts there)

This is an interesting one. We should definitely fix this problem. Is there an issue for this already @flaburgan or a test case I can observe?

Delete post / comment are not federated

We're kind-of discussing that API in https://www.loomio.org/discussions/4093. I think we should see how that discussion and hopefully proposals turn out before making any decisions on deleting content through federation. I do feel like that is a necessary feature. Everybody makes mistakes. :)

The only major, horrible problem I can really see with federation is that it's very difficult for people who have no idea how Diaspora works to learn about how we federate content and build upon that. What little documentation exists is mostly concerned with how we securely authenticate content, which does matter but could really be encapsulated in a library (leaving that documentation unnecessary for most hackers).

F

Flaburgan Mon 3 Jun 2013

@tomscott I'm sure there is an issue about that, I'll take a look.

But about the federation, there is two possible reasons that the post was not federated : a problem in the protocol or a problem in the processing of the queue by Resque (now Sidekiq). Sometimes it failed, and if it's not retried, the post will never be federated. So I don't know what caused this problem, we have to make test with small pods to precisely monitor what happens.

N

Nick Sun 16 Jun 2013

@flaburgan - as a non-techy, what would separating the federation from the rest of the code involve?

N

Nick Sun 16 Jun 2013

ok, sorry just saw that separate discussion and it seems to be something that has been agreed on pretty unanimously. Ignore comment!

F

Flaburgan Mon 17 Jun 2013

@nickdowson no problem ;)

L

L3MNcakes started a proposal Tue 25 Jun 2013

First Step : Separate Federation Layer Closed Tue 9 Jul 2013

I would like to propose that we move forward in improving federation with a first step of separating federation into it's own layer. This would involve :

1) Forming a working group of developers/collaborators who would like to volunteer their time to this problem. I think it's important to have a defined (though very much open) group dedicated to this problem, because simply leaving it open for an individual to pick up and run with leaves me with the feeling that it's simply going to never get done.

2) That group would come up with detailed plan for separating out federation and any potential problems that might arise. Communication can be done through IRC Meeting/Email Chain/Loomio/whatever works best for the people involved. The goal here is to produce an organized document that can be easily understood by newcomers who want to jump in and help out as well as a reference for those involved to ensure everybody is on the same page with what exactly needs to be done.

3) Plan could be verified/approved by other developers/collaborators through Loomio. (Maybe this step isn't necessary, but I think it's always a good idea to get outside opinions before implementing a plan.)

4) Plan is broken down into manageable tasks and put into Github where developers can pick them up and begin making code changes.

Agree - 13
Abstain - 13
Disagree - 13
Block - 13
13 people have voted (8%)
L

L3MNcakes
Agree
Tue 25 Jun 2013

G

goob
Agree
Tue 25 Jun 2013

We already have agreement to separate fed into a layer - https://www.loomio.org/discussions/612?proposal=463 - and I agree your practical steps would be an excellent way to go about it.

E

Elm
Agree
Tue 25 Jun 2013

Good way to go !

M

matl
Agree
Tue 25 Jun 2013

T

theradialactive
Agree
Tue 25 Jun 2013

F

Faldrian
Agree
Tue 25 Jun 2013

N

Nick
Agree
Tue 25 Jun 2013

great plan!

RF

Rasmus Fuhse
Agree
Wed 26 Jun 2013

Yeah, that's important!

ST

Sean Tilley
Agree
Wed 26 Jun 2013

This is great, but I think the bigger issue here is not a matter of want, but rather a matter of developer muscle and resources. Federation is pretty complex, and so our federation system needs to be fully documented and studied.

SM

Seth Martin
Agree
Thu 27 Jun 2013

I can't think of anything more important for this project.

M

mathis
Agree
Fri 28 Jun 2013

DS

Dennis Schubert
Agree
Sat 29 Jun 2013

BG

Bruno Guerra
Agree
Sat 29 Jun 2013

G

goob Wed 26 Jun 2013

I see the link in my vote didn't catch. Here it is properly:

Put federation into a layer

Just to show there is already an agreement to do this, so L3MNcakes's practical proposal of forming a working group builds on this previous agreement.

L

L3MNcakes Thu 27 Jun 2013

@seantilleycommunit :: Good points, but not necessarily show stoppers. If we could even get 2-3 devs working on this to start out, I think we could take it a long way. I'm one! =) It would be nice if we had a developer who knows the code well to help out, but in the event that doesn't happen, I'll happily be the one to tear through the code until I understand what's going on. I would love to see some of the newer devs jump in to help figure it out too! I know the task is a daunting one, but it has to be done by somebody if we want our app to succede. This is one of the core pieces of our network and it's really holding us back on the features we can implement. I really hate telling users, "We can't do that because the distributed part of our distributed network sucks." I'd much rather start moving forward on this with limited manpower than continue waiting for a magical influx of dev power.

ST

Sean Tilley Thu 27 Jun 2013

@l3mncakes If you want to take a look at our federation system, by all means take a look.

Here's some wiki entries that may be useful to anyone interested:

L

L3MNcakes Thu 27 Jun 2013

@seantilleycommunit Sweet! Thanks for listing those resources here. Should be very helpful.

TS

Tom Scott Fri 28 Jun 2013

Come on guys, do we really need a proposal to make a plan? :)

G

goob Fri 28 Jun 2013

Absolutely, Tom. It's a means of agreeing a plan and getting something done.

L

L3MNcakes Sat 29 Jun 2013

@tomscott haha, I thought it would be best way to get the ball rolling on this and since it purposes doing things a little differently than I've seen them done before, I wanted to make sure the community had a say in the decision. C=

FS

Florian Staudacher Sun 30 Jun 2013

FS

Florian Staudacher Sun 30 Jun 2013

I hope nobody feels left out by me starting ahead on a clean-room implementation for federation... but it really is not such a complicated or huge task for it to require elaborated documents or planning.

My plan of stuff to implement for this gem is:
- DO NOT BUILD ON RAILS (bonus: lightning fast specs) [done]

  • get xml de/serialization on database-unrelated entities to work in both directions [done]

  • build a structure resembling what is currently known as 'to_diaspora_xml' [done]

  • put payload in salmon envelope and sign/encrypt that as it is currently done in D* [pending]

  • simulate sending/receiving inside a proof-of-concept test-app [pending]

ST

Sean Tilley Mon 1 Jul 2013

@florianstaudacher Really cool stuff! Watching, will keep an eye on this. Do you think the availability of federation in a gem might, in theory, allow a developer the ability to incorporate Diaspora federation into just about any Rails app? :)

L

L3MNcakes Mon 1 Jul 2013

@florianstaudacher Cool! Thanks for jumping on that. I don't think planning or documentation is ever a bad thing, and it also gives an opportunity for others to learn new stuff and help out if they want, but I appreciate that you're taking the time to get stuff done.

G

goob Mon 1 Jul 2013

Fantastic work, Florian.

F

Flaburgan Mon 1 Jul 2013

@florianstaudacher thank you for dealing with that!

FS

Florian Staudacher Mon 1 Jul 2013

@seantilleycommunit in theory, yes, you could re-use the gem in other ruby projects. but keep in mind that the gem just represents the "business logic". you'd have to handle the http routes and the database persistence in the app - that's really out of scope for the gem.

FS

Florian Staudacher Mon 1 Jul 2013

  • put payload in salmon envelope and sign/encrypt that as it is currently done in D* [done]

;)

L

L3MNcakes Mon 1 Jul 2013

Started trying to put together some better documentation around this. Currently working off the articles Sean posted here and trying to make them more readable and useful. Not done by any means yet, but just a heads up on what I'm doing.

https://docs.google.com/document/d/1QEhIx10zFnBgAtf_GL6AMT9GvWjxfPE28QFb2rPIbHU/edit?usp=sharing

FS

Florian Staudacher Fri 5 Jul 2013

alright, the code in my github repo should now have every entity we use for federation, also the ones with nested entities inside (e.g. StatusMessage -> Photo, Location).
they can be de-/serialized from/to XML and are meant to be put in a Salmon envelope that can be de-/encrypted.
I still want to implement some rudimentary validation classes, so the received objects can be sanity-checked before they are being handled by an actual app, but apart from that I think we could start to think about how to test this on its own and with another instance of D* running the current federation code.

F

Flaburgan Thu 11 Jul 2013

@florianstaudacher please ping me when you think the code is stable enough to be pulled on my pod.

FS

Florian Staudacher Fri 12 Jul 2013

... it will need a lot more work before we can actually run a pod with it.
not only do we have to give it a decent test period, also a good part of the diaspora code will have to be adapted just to use the new gem.

F

fabianrbz Fri 12 Jul 2013

@florianstaudacher I'll be more than happy to help you with this. I'll try to give a look at it this weekend, just let me know what needs to be done

FS

Florian Staudacher Mon 15 Jul 2013

Well, to anyone who would like to help, I can suggest to review my code and the docs I write with it.

I may be an intermediate-advanced programmer but that certainly doesn't mean everything I write is pure gold. I want this to be as good as it can get, so we have a rock-solid library we can use for Diaspora* and possibly to have something other projects could use to integrate federation into their software.

Anyway, this will become the new reference implementation of our protocol, so this has to be impeccable, readable and understandable.

ST

Sean Tilley Mon 15 Jul 2013

@florianstaudacher Would it be possible to move these updated docs into the federation section of our wiki at some point? It'd be really great to brush up what little information we have on how the protocol works, and update it with all the information you've put together. :)

N

Nick Tue 16 Jul 2013

Hey, without understanding the technical stuff, it seems like there's been a lot of progress with this.
Seems to me like an agreed standard for decentralised social network federation should be a major outcome that the diaspora project should be working towards - and that this needs to be something that can be used by other social networks. I think it would be really good to work with a few other distributed social networks (movim and gnustatus come to mind as to my mind projects with potential) from an early stage to develop a protocol that can be flexibly and easily extended and some kind of process/foundation that becomes the agreed way of agreeing and improving this.
Now might be a good time to start thinking about this could be made happen? Working with groups such as the free network foundation (thefnf.org), the free software foundation, the electronic frontier foundation and others might be a way forward?

N

Nick Tue 16 Jul 2013

On a totally different note, any thoughts on how diaspora should deal with linking to profiles? On facebook etc. someone can give the http address to their profile which anyone can click on, look at and then press to subscribe/friend/etc. On diaspora this is more complicated if the profile is hosted on a different pod - plus being logged in or out and consistency of how sites look becomes a bit of an issue. Plugins or third party cookies might be a way of solving this but bring up obvious privacy issues... (sorry if I've not explained this issue very well...)

G

goob Tue 16 Jul 2013

Nick, you can use http://dia.so/ to get a URL for your profile which will work on any pod.

N

Nick Wed 17 Jul 2013

ah, thanks @goob I didn't realise this, will check that out!

JR

Jason Robinson Thu 18 Jul 2013

It's still a third-party tool ;) Something like this should be in the future integrated to the Foundation(tm) and officially supported in the code.

Also I'm fantasizing of a link in posts to take the user to "the post on my pod". For example when someone gives me a link to a post on another pod, I click it, I cannot reply or reshare unless I find it on my pod. Just needs some ajax query, and of course post must exist on the other pod.

DU

[deactivated account] Sun 21 Jul 2013

Has this descision https://www.loomio.org/discussions/612?proposal=463 been worked into GitHub issues? Or how will it be moved into actionable items?

G

goob Sun 21 Jul 2013

@wilhelm , Florian's code, linked on this thread, is the start of the solution to the proposal you link to. See https://github.com/Raven24/diaspora-federation

FS

Florian Staudacher Thu 1 Aug 2013

alright, I'm quasi-finished with what I had planned to include in the gem.
Only one more design/architectural issue I'd like to bring up and solve with the wisdom of the crowd:

We have a lot of XML in the protocols around D*, so I have chosen the 'ox' gem to handle XML parsing and generation. It promises to be fast and has all the features we need...
Except for one occasion, where we use hCard - which is HTML and uses classes to label its data fields inside standard HTML elements. Ox doesn't know CSS or XPath selectors, so I can't select the elements based on the used class names directly. At that point I brought in libxml-ruby, which has support for XPath, which is enough to solve the problem in this case.

Personally I feel it's unclean to include another lib just for that one use case, and now I'm torn between leaving it like that, or just switching to libxml or possibly nokogiri entirely...

Ideas?

JH

Jonne Haß Thu 1 Aug 2013

I think we should switch to Nokogiri. While it may not be the fastest library, it's the most feature complete and battle tested one, so using it should be future proof.

L

L3MNcakes Thu 1 Aug 2013

@florianstaudacher -- I don't really know much about the various libs themselves, but I agree that it feels dirty to include a separate lib for just that case. My first instinct is to modify ox so it could support that use case and then we only have to use one lib. Would this be feasible?

G

goob Thu 1 Aug 2013

If you feel that ox performs better for the other cases, it might be worth considering whether Diaspora's hCard could be parsed using XML. I've had a look online and found suggestions that this is possible with hCard. This may not be a good idea in Diaspora's case, or may not be a clean solution in itself, but if you'd prefer to use ox, it might be worth considering whether it could work.

Apologies if I'm speaking out of turn, not really understanding the technical issues. It just struck me as a potential alternative approach to switching libraries, if you really like that particular gem.

FS

Florian Staudacher Thu 1 Aug 2013

Well, as far as performance goes, our XML documents are at most a few KB in size, so any speed variances should be almost undetectable.

Modifying Ox to support advanced selectors is way out of scope for this task and I don't think it's in the spirit of the lib - being mostly fast and lightweight.

I thought about parsing the hCard as XML, but I decided against it, since the idea behind XML is structure and hCard as such has no structure.

I guess I'll be moving to Nokogiri, it seems the least controversial... ;)

FS

Florian Staudacher Sun 4 Aug 2013

(in case anybody missed my post on d*)

okay, porting to nokogiri is done. now it's time to give the code a thorough review. so if you have any ruby knowledge, look it through and give me feedback on anything that might jump in your eyes.

documentation and readme will follow later

G

goob Mon 5 Aug 2013

I've just started a new discussion with some ideas for adding the possibility of pulls to Diaspora's push model for federation. I didn't want to start a new discussion on this thread, which might confuse things.

Hope it's useful.

N

Nick Sat 17 Aug 2013

I created a new discussion for what protocol we use & working with other social networks:

https://www.loomio.org/discussions/6233

FS

Florian Staudacher Wed 21 Aug 2013

another update: the Readme contains more infos, and I think the docs should be mostly finished.
So if you get a chance, you can have a look at the docs (http://rdoc.info/github/Raven24/diaspora-federation/master/frames) and tell me if everything is understandable and usable for implementing the gem.