Loomio
April 30th, 2013 11:42

Implement realtime chat, possibly using XMPP

diasp_eu
diasp_eu Public Seen by 574

follow up
https://github.com/diaspora/diaspora/issues/4076

I've searched all issues for "chat" and all which suggest implementing chat are closed.
There seems to be no explanation of why they are closed, so I am opening a new one.

This issue shall be about basic chat functionality:

seeing who is online
chat between two users
For multiuser chat, a separate issue is suitable.

I think it is critical for the success of Diaspora to have chat, so I am requesting that you add this to a milestone.Thank you.

Flaburgan

Flaburgan April 30th, 2013 11:51

I think WebRTC is THE solution because really powerful (look at this demonstration).

As @dennisschubert said, the technology is not mature yet, but will be available for every Firefox and Chrome users this summer. I think that 80% of our users use one of these two browsers, as we do not currently support IE and Safari+Opera have small % of users on desktop, for me, a release with WebRTC during the summer is something acceptable. But we are here for debate :)

goob

goob April 30th, 2013 13:09

I think that chat is one of the features which has been put on hold until the Diaspora API has been finalised and made stable, but one of our more technical members will be able to confirm whether or not that is the case.

But it may well be a good idea to discuss now what will be the best way to implement chat, and which protocol(s) will be most appropriate, now so that those decisions have already been made once it's time to start coding the chat facility.

Shmerl

Shmerl April 30th, 2013 14:36

WebRTC is an JavaScript API. It doesn't mandate any protocol which you can implement through it. For chat there is no need to build server functionality into Diaspora itself - it would be crude engineering and would be too restrictive. It's better to take ready and robust solutions (for example ejabberd), and to run the XMPP server alongside Diaspora pod (let's say on the same host). Diaspora just needs a UI to communicate through it. And if user wants to use standalone client - it will work as well, since XMPP is a standard protocol.

For media, just use XMPP/Jingle as usual. If you want to work on Jingle over WebRTC - be my guest, but I don't think there are any ready solutions, and this will require a lot of work potentially. Also WebRTC will require support from XMPP servers, since you can't simply route Jingle over WebRTC relying on clients alone.

M

mathis April 30th, 2013 19:05

ther was an old work of a diaspora video-chat made by vittorio cuculo: https://github.com/vcuculo/diaspora/wiki/Setup-chat-and-videochat i can contact him if he want to contibute again , it's an italian user.

Ruxton

Ruxton May 1st, 2013 02:26

while i'd love to see this i think it's something that should be focused on at a later date when other things like federation are all sorted. The back end to things like federation are likely to provide a better interface for something like this.

PG

Paul Greindl May 1st, 2013 19:02

@ruxton Yes of course, we can always delay everything to a later point of time, but then we'll never come anywhere...?

Tom Scott

Tom Scott May 1st, 2013 19:53

@shmerl first of all, you're wrong about your assumptions on WebRTC. it is in fact a peer-to-peer chat API, which is why it's so interesting to this project because it's decentralized, e.g., your chat messages never touch the actual web app server. in fact, the diaspora app is just to facilitate logging in and using the WebRTC service, if that's how you're instant messaging.

I like the idea of XMPP. I feel like it may be a neat experiment to see what other kind of data we can federate. The problem with XMPP is that it requires another server to be set up and maintained. It requires people who are already having trouble getting a Rails app installed to have to install/configure ejabberd or an equivalent server. Additionally, podmins hosting on Heroku are out of luck, because you can't run processes on any port other than :80, which is the one we need for the web app. So Heroku users would have to pay for another VPS just to house their XMPP server, or find space on an existing VPS for such a task. While I agree with you here that XMPP might be a more feature-rich and "trusted" solution to our problem, there's no question that using XMPP would increase the complexity and reduce understandability for many new developers. Which is why we've been so nervous about implementing it. The last thing Diaspora needs is its podmins feeling alienated.

I personally think that if our goal is chat, we should focus on some sort of decentralized solution, which is looking every day more like WebRTC. If our focus is on changing the way we federate data and getting chat "for free" along with it, then XMPP seems to be the way to go.

I would also wager that since we aren't responsible for any IM traffic over the Diaspora server(s), that WebRTC is "cheaper" solution than XMPP, simply because we don't need a central server to facilitate communication between two people.

Shmerl

Shmerl May 1st, 2013 20:12

@Tom Scott: No, I'm not wrong about WebRTC. It's not a chat API. It's a JavaScript framework for RTC media streams. See https://en.wikipedia.org/wiki/WebRTC
It doesn't dictate the signalling protocol (which can be Jingle or SIP for example). If you use XMPP, Jingle is the natural path for signaling protocol (which should be routed through WebRTC if you want a JavaScript browser solution).

Using XMPP should be a solid must, otherwise you'll be proliferating non interoperable IM networks. No thanks - there are tons of these out there which can't talk to each other. We don't need another one, just to put it into Diaspora. Better even, if someone is so desperate to have an IM, then take a normal standalone client, connect to any of many existing XMPP servers - and that's it. No reinventing the wheel needed.

I understand the issues of running an extra server. But it only makes sense to do so, since it clearly separates Diaspora from IM service. Yes, Heroku is not an option. But Heroku limitations and idiosyncrasies should not dictate Diaspora design and limit your choices. Just don't use Heroku of you need an accompanying IM service.

Tom Scott

Tom Scott May 1st, 2013 20:58

@shmerl "Just don't use Heroku"? There are a lot of pods already on Heroku (disclaimer: mine included). This is an unsustainable suggestion, and it's kind of a big "fuck you" to all of the podmins who are hosting free/personal pods on Heroku. I'm not going to put others through that just because I want to use Diaspora as my IM provider.

"WebRTC (Web Real-Time Communication) is an API definition being drafted by the World Wide Web Consortium (W3C) to enable browser to browser applications for voice calling, video chat and P2P file sharing without plugins."

To me, that indicates that WebRTC does not require a server. Perhaps it isn't necessarily a protocol for chat, but I'm pretty sure that WebRTC is a peer-to-peer technology which means its data is not routed through a central server. This is a big win for us, because we are always looking for better solutions to the centralization problem. It ensures that the only 2 people who see the data are the 2 people having the conversation.

I like that a lot for personal chat communications, as long as this discussion is focused on that particular problem. What I don't think we have in this discussion is a grip on what the problem is that we're trying to solve. Are we just trying to give our users a basic chat service? Or do we want to use the XMPP protocol for something a bit beyond chat services, like perhaps as a means to re-invent the Federation Protocol?

Tom Scott

Tom Scott May 1st, 2013 20:59

Another disclaimer: I can easily run an ejabberd on the VPS that I already own, so that particular problem isn't something I have to solve for my own use case, but I can see how it would be an issue for others...

goob

goob May 1st, 2013 21:01

@paulgreindl the problem is if features are developed before federation has been resolved (and stripped into its own code layer) and before an API has been finalised, those features might suddenly become broken when the code base changes at a later date. This has already happened with a chat feature and many other Diaspora features about a year ago, which put development back a long way when many developers became disillusioned at this happening. I think it's important to make sure this doesn't happen again, and so to focus on making the underlying code base stable enough so that features then added won't get broken when other parts of the code base change.

Shmerl

Shmerl May 1st, 2013 21:12

@Tom Scott:

"Just don't use Heroku"? There are a lot
of pods already on Heroku. This is an
unsustainable suggestion

It means don't use Heroku for XMPP server. If you need one - find another hosting to dedicate to IM part of your pod. If you can't - don't use XMPP server. Heroku should not limit Diaspora design - no one forces anyone to use it and Diaspora should not mandate any limitations based on Heroku preference.

that indicates that WebRTC does not require
a server. Perhaps it isn't necessarily a protocol for chat

Exactly, it's not a protocol for chat, nether it is a signaling protocol which is needed for sensible P2P media communications (most commonly used signalling protocols are SIP and Jingle). It's up to developers do decide what to route through WebRTC. See http://www.webrtc.org/faq#TOC-Why-should-I-use-WebRTC-
As I said, it makes sense not to reinvent the wheel, and to use existing protocols. Otherwise you'll end up with another walled service which won't be interoperable with anything outside itself.

Shmerl

Shmerl May 1st, 2013 21:20

Again, this discussion has nothing to do with Diaspora federation. It's about chat (IM/media) functionality. Federating the IM part should not be restricted to federating D*. I.e. if you use XMPP for IM part, it's automatically federated to the whole XMPP network. It doesn't relate to where D* itself is federated (as a social network).

goob

goob May 1st, 2013 21:33

Shmerl, if that's a response to what I said, my mentioning federation is because that is one of the biggest things to be solved, and until the federation code has been put into its own layer, any future changes to the federation code (of which there will by necessity by many, possibly almost totally changing the federation code) risk breaking features such as chat, which has already happened once before. Therefore it makes sense to either fix and make stable the fundamental stuff such as federation and the API, or to put them in their own layers so that future changes to them don't affect other things, before spending too much time creating 'window-dressing' features.

I think Diaspora would struggle to survive another catasrophe in which loads of features, carefully created by developers over much time, suddenly broke because of a necessary change in the fundamental network code. We'd do well to avoid this happening again.

Shmerl

Shmerl May 1st, 2013 22:12

@Goob: Sure, fixing D* federation should be a priority I think. But I was answering to @Tom Scott, who for some reason tried to tie together D* federation and the choice of XMPP for IM component. I was saying that these are separate subjects, and should not be put together in this thread (except as you said, putting D* federation as a first priority).

PG

Paul Greindl May 2nd, 2013 05:09

@goob This is absolutely valid for federation dependent features but not for a XMPP based chat...

Rasmus Fuhse

Rasmus Fuhse May 2nd, 2013 07:40

My opinion is that an xmpp-chat for diaspora-federation is only a matter of a user-flag that says "this user has the following xmpp-chat-account". The rest (having a web-client for the chat, having a jabberd-server on the pod or an external server/account, or using WebRTC) is not affecting the diaspora-federation in any way. So using XMPP seems quite nice to me and should be able to be done even today, when federation is not put into it's own layer yet.

Flaburgan

Flaburgan May 2nd, 2013 08:33

Okay so this debate is more or less a choice between XMPP and WebRTC. As a Mozilla contributor, I'll contact some WebRTC guys asking if they can come here because I think we do not have enough knowledge to take a decision.

Anyway, here are the arguments for the moment:

Maturity:
XMPP is far more mature and supported than WebRTC, no doubt. WebRTC works on only few browsers for the moment, and this is a problem. Moreover, specifications can change because the techno was not released in stable for the moment.

Interoperability
XMPP is used by a lot of network (Jabber, GTalk, Facebook...). Unfortunately, Facebook is completely closed and Google started to close gtalk too (you cannot add a gtalk contact from another xmpp account for few weeks now). Anyway, there are a lot a websites / services (jabber, movim, libertree, etc) which used XMPP, far more than WebRTC and there are desktop clients for XMPP too. So for the moment, XMPP wins. In the other hand, it's possible that WebRTC becomes the replacement of XMPP in the future...

Complexity
About development, we do not know enough WebRTC to compare to XMPP implementation.
About pod administration, has said, XMPP is more complex than WebRTC. It needs another server, more sysadmin stuff, more server bandwidth / power, special configuration (ports other than 80)... and is not completely decentralized as it needs to use a server.
Remember, in a perfect Diaspora world, everybody has his own pod under his bed, in a raspberry. I see the server part becomes more and more complex with the time, and I don't like that. WebRTC totally solves this problem by processing between browsers directly.

Features
Here is the killer point in favor of WebRTC. Chatting, audio, video, file sharing, conf call... (remember, this is "the" killer feature of g+, officially...). This is AWESOME.

Moreover, we are in a moment where companies are trying to put social in them. If we add such features, Diaspora can be interesting for them. And companies can put money / developers in the projects which interested them. This is a really important point, we can maybe find a business model here (and we need one!!).

Jason Robinson

Jason Robinson May 2nd, 2013 12:09

A big huge +1 for XMPP chat support.

In addition to an actual chat UI, we only need to add a field in the user settings about the users XMPP handle. If the pod provides one, the user can opt to use that one. If the user does not like it or has an existing one, they will probably use that one.

I would probably use my Facebook XMPP handle since that is where I do most of my chatting with my friends.

We have a gazillion of things to do - let's not start building a platform for chatting as we can leverage on XMPP for that and just offer a simple client in the UI AND the possibility for podmins to run a server tied to their pod. If they want, Heroku podmins don't need to do if they don't want. Or they can run one on their VPS.

Jason Robinson

Jason Robinson May 2nd, 2013 12:10

Personally I don't need an additional chat handle - allowing Diaspora* users to contact me easily via my existing XMPP handles would be cool though.

Flaburgan

Flaburgan May 2nd, 2013 12:33

I would probably use my Facebook XMPP handle since that is where I do most of my chatting with my friends.

But by doing that, you will not be able to talk to me who have a Jabber account, aren't you?

Btw, integrate in Diaspora an existing account could be a good idea, but how do you add contacts? When you start sharing with someone, it automatically send a request to his xmpp account? And how the other user answer? The asymmetric relationship of Diaspora is really different of the symmetric relationship in xmpp..

Jason Robinson

Jason Robinson May 2nd, 2013 12:59

OK maybe Facebook XMPP based chat engine was a bad example. But anyway, something existent which is widely used, like @shmerl has been saying

Shmerl

Shmerl May 2nd, 2013 16:30

@Flaburgan:

Okay so this debate is more or less a choice between
XMPP and WebRTC.

I think you are missing a point. XMPP and WebRTC are incomparable - they are different level frameworks. XMPP is a complete IM protocol, with signalling extensions like Jingle, which use P2P RTC for media chat (video/audio).

WebRTC is not a protocol, it's a low level JavaScript API which allows using RTC from the browser. I.e. it doesn't dictate any protocol whatsoever.

Therefore it's not a question of XMPP vs WebRTC. It's a question of what protocol do you want to use for chat? And if you use WebRTC for conferencing part of that chat (which makes sense for browser based client), what protocol do you want to implement through it? Remember again,WebRTC is not a protocol. See it as a tool, through which you implement your media protocol in JavaScript.

The way I see it - full blown and feature rich chat should look like this:

  1. IM component should use XMPP protocol (for example using strophe.js preferably with WebSockets, instead of BOSH).
  2. Conferencing component (video/audio chat) should use XMPP/Jingle signalling protocol, utilizing WebRTC framework to route it from the browser.

I hope this clarifies things a bit.

Shmerl

Shmerl May 2nd, 2013 16:35

Anyway, my advice to anyone who proposes to use WebRTC in any way - read first about what it is, and what it isn't.

Shmerl

Shmerl May 2nd, 2013 16:47

@Jason Robinson:

I would probably use my Facebook XMPP
handle since that is where I do most of my chatting
with my friends.

I agree that users should be able to set their XMPP handle, which can be from another server, not necessarily from the server shipped with the pod. However it has to be a federated server naturally. Facebook XMPP server is not federated, so we can forget about it right away. The fact that it's XMPP becomes irrelevant, since it's cut off from the XMPP network.

Google Talk is federated. Their invite blocking problems were temporary and they already resolved that issue.

C

Christophe May 2nd, 2013 19:51

I'm running a XMPP server and a Diaspora pod side by side at wk3.org, so the users already have matching handles.

Since nobody came up with SIP yet, I'd like to suggest that. There's already a HTML5 client: sipML5 And we totally need to consider CU-RTC-WEB too.

Shmerl

Shmerl May 2nd, 2013 23:39

SIP is fine for media, except that it doesn't integrate with XMPP, instead it integrates with its own IM protocol - SIMPLE. However Jingle is a natural extension of XMPP, and if you are already using XMPP for IM/chat, it makes sense to use Jingle rather than SIP.

Rasmus Fuhse

Rasmus Fuhse May 3rd, 2013 05:14

Like Shmerl said, WebRTC is just a channel that can be used for any kind of content. Yet it will be easiest to implement video-chat and phone-calls since it has the best developed libraries.
My question would be (and I'm not sure what the answer is): is it possible to fully mimic an XMPP with webRTC? There are some asynchronous components in XMPP that would need to be done by the Diaspora-server like "does the user exists", "give me metadata of the user" and "send a friendship request", probably more. I am quite sure that this can't be done entirely just by using WebRTC.

But it would be definitely a good architecture, if the user can opt if he/she wants to use a) an existing XMPP-account, b) a provided XMPP-account by the pod (like Christophe's solution) c) a WebRTC-client for XMPP or d) no XMPP account at all.

If we want to focus not only on chatting, but also on video-calls and this stuff, we should consider this probably now, because this can be hardly be done by XMPP only. I have no idea what would be the way for that.

Jason Robinson

Jason Robinson May 3rd, 2013 07:19

I see this discussion finally moving in a good direction :)

Shmerl

Shmerl May 3rd, 2013 07:59

@Rasmus Fuhse: For text messages you don't need WebRTC. It's a clear overkill. JavaScript offers you WebSockets already. WebRTC is targeted for media streams (i.e. audio/video), that's why it's dealing with codecs and the like. WebSocktes is a generic API for duplex communication through JavaScript. It fits perfectly for text part of XMPP. Some support for that is required on the backed (before WebSockets, the only method from the browser was BOSH). Now servers like ejabberd are catching up with WebSockets support. For media, WebRTC is an obvious API, but as I said - the protocol still needs to be defined, and it makes sense to use Jingle there:

#1. Text XMPP can work like this:
JS client (browser 1) <-> XMPP over WebSockets <-> XMPP Server <-> XMPP over WebSockets <-> JS client (browser 2)
#2. Media XMPP (Jingle) can work like this:
JS client (browser 1) <-> Jingle over WebRTC <-> JS client (browser 2).

Except that in #2 you still need some support of such routing from the servers, since there are security limitations. See some discussion in the comments here:

https://hacks.mozilla.org/2012/09/full-webrtc-support-is-soon-coming-to-a-web-browser-near-you/

Shmerl

Shmerl May 3rd, 2013 08:02

The benefit of using XMPP and Jingle is that instead of JS client you can put any client (Pidgin, Jitsi etc.), and it would still work, since it will talk a standard protocol.

Tom Scott

Tom Scott May 3rd, 2013 14:37

@shmerl If we are speaking of adding XMPP as an IM service only, then we may have a reasoning behind this. We can use XMPP for a lot more than just IM, if we were so inclined. I believe both Facebook and Twitter are/were using XMPP for realtime communication between their client side and server side code.

If we do this, I'd like to give the user the freedom to run XMPP or not. If they don't want it, they can disable its use. If they do want it, it is their responsibility to provide a server and XMPP protocol support on the backend. Diaspora is essentially just providing a nice web-based client to use with an XMPP server.

For a first version, I feel like it'd be easier to just focus on text chat, and then move on to voice/video implementation later. Are there free software implementations of this on the client side I can take a look at? Also, what is everyone's consensus on using a Flash component to handle video chat for regression into older browser clients?

Shmerl

Shmerl May 3rd, 2013 15:17

@Tom Scott: Yes, we can use XMPP for Diaspora's own protocol, but as I said, this is out of scope for this particular discussion.

Back to IM: XMPP surely can be optional. Podmin can simply not associate the pod with XMPP server (and not install that server). There is no need to make it mandatory.

Regarding clients - Text/IM libraries are already around. Example:
http://strophe.im/strophejs/
https://gist.github.com/processone/739147

Audio/video conferencing and Jignle support through JavaScript is still catching up and you'll have to do the research. But that can be put to phase 2, with text IM being the first phase.

C

Christophe May 5th, 2013 14:13

Who is the person requested to implement this feature?

Jason Robinson

Jason Robinson May 5th, 2013 18:32

@christophe - chat has been one of the most wanted features since the beginning, requested by many people

Flaburgan

Flaburgan May 6th, 2013 07:58

@tomscott

Are there free software implementations of this on the client side I can take a look at?

Look at Jappix Mini ;)

what is everyone's consensus on using a Flash component to handle video chat for regression into older browser clients?

I'll block anything using Flash. This technology does not exist anymore :p

Tom Scott

Tom Scott May 6th, 2013 12:21

@flaburgan uhh, i think you misunderstood my question. but it was a stupid question anyway, because i'm pretty sure we can get websockets to work all the way down to IE 6 without using flash.

@shmerl i do know about Strophe. should be a decent choice for communicating with the XMPP server. let's focus on text chat right now. would Jappix Mini or the Facebook IM design work for our purposes?

Tom Scott

Tom Scott May 6th, 2013 12:21

and by "Jappix Mini" i mean the way it looks, not the product itself. we can't really use Jappix as a product because we don't control its code.

DU

[deactivated account] May 7th, 2013 07:16

How would you link Diaspora accounts with XMPP accounts?

Do you intend to allow users to provide an existing XMPP account or should they register a new account (on the connected XMPP server) through Diaspora? (see XEP 0077)

Tom Scott

Tom Scott May 7th, 2013 21:16

@rekado when i was playing around with ejabberd, i found the easiest solution would be to simply create an XMPP user for each diaspora user on the pod. and when new users are added/removed, we create/destroy their corresponding jabber user. seems pretty easy since both of them are using a relational DB.

DU

[deactivated account] May 8th, 2013 04:06

@Tom: I would advise against modifying the DB directly as you'd couple Diaspora too tightly with a particular XMPP server implementation (and maybe even version). XEP 0077 is the standard way to register accounts through XMPP. (My apologies if this is what you meant.)

Christian Giménez

Christian Giménez May 9th, 2013 03:18

Hi to all!
This is my first time I use loomio so please have patience!

I don't know how much voice and votation I have, but I give my personal opinion about this:

I met lot of people that wants to have a chat feature, not a voice call neither video calls. Also I met people that just said "ah! that doesn't have a chat: it's boring!" so I cannot convince him to use D*. In other words: people got acustom to chatting instead of talking.

One problem about voice and video is not the implementation, is the Internet "speed" and bandwith. If you're going to add voice or video, please be sure to make it clearly optional.
If you ask my opinion about this, we should concentrate on chat. Video and calls are too much and has its own consequences(bandwith, services, privacy, etc.). At least, we should have a chat!

About XMPP, it do support creating counts through its protocol(I remember Pidgin asking if I want to create the account or use an already created). Then, the problem is if ejabberd or jabber.org will may be overpopulated and if they can stand so much people registered.
This makes clear why should be a separate service and not hosted in Diaspora servers, maybe some pods cannot stand such load. Diaspora can give an easy and beautiful
web interface, but the rest is up to jabber.org(and we support Jabber!).

I haven't take a look into WebRTC, but for now I vote in favor of XMPP: is mature enough for give more robustness to D*.
Cheers!

Jason Robinson

Jason Robinson May 9th, 2013 06:13

@christiangimenez - everybody has a voice on Diaspora* :) Welcome!

Tom Scott

Tom Scott May 9th, 2013 15:58

@rekado at this point, i think just supporting a single XMPP server would be sufficient for our needs, but regardless if we can use a standard to make sure that our code is "future-proofed" for any other standards-compliant XMPP server that gets invented in the future, that seems like the "right way" to go.

Tom Scott

Tom Scott May 9th, 2013 16:00

@rekado to answer your question, i think XEP-0077 (http://xmpp.org/extensions/xep-0077.html) is a fine spec to go by when registering accounts on the XMPP server. there is an XMPP lib for Ruby, so we can send messages to the configured server like when a user is registered through the web interface.

DU

[deactivated account] May 10th, 2013 01:08

Actually, there's more than one XMPP lib for Ruby. I recommend using blather (rather than XMPP4R), because of its cleaner design thanks to running inside an EM loop.

Florian Staudacher

Florian Staudacher May 10th, 2013 13:58

so, this is the direction I see this discussion going:

D* associates a Jabber ID with a D* user. That could be a user-specified, existing JID - or - D* offers to create a Jabber account on a user-specified server via XMPP protocol (could have a nice auto-complete for a few known servers).
(the password would need to be stored in the DB unhashed, but we could at least encrypt it with the users private key)

The UI offers a 'chat' feature, that's more or less simply a Jabber client in the browser, connected to whichever account is specified in the D* users profile.
The exact implementation details are still to be decided...

Steffen van Bergerem

Steffen van Bergerem May 10th, 2013 17:07

@florianstaudacher I just signed up because I wanted to write exactly what you already said ;-)

Just adding this: When user A starts sharing with user B the JID of A could be send to B. When B also starts sharing with A (so both know each others JID) we could add the JIDs to the jabber contactlist

Seth Martin

Seth Martin May 10th, 2013 18:55

It appears that many existing XMPP users will have to either get new accounts or install BOSH on their server to use such a feature.

Christian Giménez

Christian Giménez May 10th, 2013 20:38

@steffenvanbergerem do you mean that user A adds B or just display it?
If adding, D* should ask in which group to add B.
This is useful when we're talking that D* server forward messages to Jabber servers.

@sethmartin what @florianstaudacher mean is that D* will only associate an XMPP account to the user, and then will only act as a client(maybe using Javascript or other).
If you have an existing account you associate it with your D* user.

I suppose that there's no need to alter the server functionality, the problem reduces to find a good free jabber client to embed into the webpage. Or, if I forgot something about securities issues, the server just act as a passarel, fowarding messages to the Jabber server.
Am I right? :-S

Christian Giménez

Christian Giménez May 10th, 2013 20:47

I found lots of free(as in freedom) Jabber clients and libraries written in JavaScript: JSJaC, Candy. Take a look at the Jappix Mini feature looks interesting...

Hope this helps a bit more...

Jonne Haß

Jonne Haß May 10th, 2013 21:59

@christiangimenez all of these require BOSH because that's the way to write XMPP clients for the browser....

Christian Giménez

Christian Giménez May 10th, 2013 22:37

You're correct, it use BOSH. I learn something new! :-P

DU

[deactivated account] May 11th, 2013 00:12

If you were fine with the idea that users had to create a new XMPP account through Diaspora, then you would not need to use BOSH and the JS side of things could be rather simple (= no need for a full XMPP client in JS). Instead of distributing a full client to the users, you could push messages through a websocket in either direction; the websocket server process then takes over the role of an XMPP client.

The benefit here is: no need for BOSH, smaller JS footprint, easier to customise the chat interface, you can use Ruby + blather to interface with the XMPP server in an event loop rather than using the chopped up XMPP-via-HTTP thing that is BOSH.

Florian Staudacher

Florian Staudacher May 11th, 2013 10:56

yeah sure, that way we could achieve much tighter integration with our application, but it would also increase the work required to do this feature.

Also, I'm no expert on Rails but the googling I have done lately indicated that Rails is not particularly good at doing anything else apart from HTTP request-response cycles. Websockets and all the other nice tricks require a separate event loop that's outside the Rails app, meaning additional ports would be needed to be opened for D*. (Please correct me, if I am wrong.)
So what I'm saying: a standalone chat server in ruby appears to be no problem, but integrating one in D* - even if it's just to relay XMPP messages to the client browsers - is a completely different story.

DU

[deactivated account] May 12th, 2013 00:12

Websockets and all the other nice tricks require a separate event loop that's outside the Rails app, meaning additional ports would be needed to be opened for D*

To support websockets you only need to run a websocket server. This is a simple, separate process (e.g. written in Ruby) that dispatches on messages that come in via websocket. This process can (and should) operate independently from the Rails application. Thanks to library support for websockets, it's very simple to make this work. The separate process could be spawned during initialisation of the Rails app, so it doesn't even need to be started separately.

One additional port is needed to be open, but I don't consider this a problem --- it's certainly much less of a problem than installing and configuring an XMPP server.

Jonne Haß

Jonne Haß started a proposal May 12th, 2013 07:44

Find a solution for chat that works on (free) Heroku Closed 9:14am - Wednesday 22 May 2013

Outcome
by Jonne Haß April 25th, 2017 05:54

We don't need a solution that works with a free Heroku account alone but can mandate an external service. But it needs to able to communicate with pods installed on Heroku.

Yes: Yes, free should work too.
Abstain: I tend to want it but free is not mandatory.
No: Lets screw our Heroku users.

Given that our biggest pod runs on Heroku and that the discussion all goes into the direction of separate ports/servers/processes lets decide this once and for all to ease the discussion.

Results
Agree - 8
Abstain - 10
Disagree - 5
Block - 0
23 people have voted (0%)
Jonne Haß

Jonne Haß
Agree
May 12th, 2013 07:44

Steven Hancock

Steven Hancock
Agree
May 12th, 2013 07:47

Sean Tilley

Sean Tilley
Agree
May 12th, 2013 07:49

This needs to happen. We cannot create a feature disparity between our pods just based on what host a podmin is on; that creates more problems than it solves. :/

Shmerl

Shmerl May 12th, 2013 07:52

BOSH is not the only way, and not a preferable one. Better one is Websockets.

As for Heroku - that will require to come up with XMPP, I'm not sure that Heroku is at all geared towards that. It seems to be targeted only to web applications, not to other type of servers. So pushing XMPP server in Heroku would seem like using a wrong tool for it. While it's good if that could be possible, but any kind of approach with it would be not straightforward. Much cleaner solution for those who rely on Heroku is simply to find another host where to run their XMPP server.

Shmerl

Shmerl
Agree
May 12th, 2013 07:54

If such solution is possible with XMPP - why not. But it seems rather non trivial. Better evaluate if this effort is really required, or there are easier workarounds.

diasp_eu

diasp_eu
Agree
May 12th, 2013 08:45

Steffen van Bergerem

Steffen van Bergerem
Abstain
May 12th, 2013 09:20

I think we could let the podmin configure if he/she wants xmpp support. In that case heroku users will get a working installation of diaspora - just without chat.

Florian Staudacher

Florian Staudacher
Agree
May 12th, 2013 11:00

Jonne Haß

Jonne Haß May 12th, 2013 13:41

Just to clarify what agreeing to this proposal means: "The WebSockets protocol is not yet supported on the Cedar stack." https://devcenter.heroku.com/articles/using-socket-io-with-node-js-on-heroku

It's also impossible to open arbitrary ports, so running an in process XMPP server is also impossible.

Thirdly I wouldn't consider requiring a third party XMPP server as "working on Heroku".

Florian Staudacher

Florian Staudacher May 12th, 2013 18:26

long-polling/comet! ;)

Jonne Haß

Jonne Haß May 12th, 2013 18:28

I don't think our issue is S2C. Our problem is S2S.

Shmerl

Shmerl May 12th, 2013 18:50

Jonne Haß: Then clarify please what kind of options for XMPP server are possible on Heroku? From what you said, it sounds like they are non existent. So what is the poll about then?

Seth Martin

Seth Martin
Disagree
May 12th, 2013 19:37

I'm not interested in any other solution than XMPP because I want to take my chats with me when leaving the browser. I hardly consider it screwing Heroku users by requiring they use a third party XMPP server if they want to chat.

Seth Martin

Seth Martin May 12th, 2013 19:49

Although it's not ideal to trust yet another party with your data, third party BOSH can also be used for the few XMPP users that don't have BOSH. IMO the ideal solution here would be a simple Heroku compatible Jappix mini type plugin like Friendica uses.

Jonne Haß

Jonne Haß May 12th, 2013 20:27

That's what I want to clarify. I've not come over a decent solution that allows using XMPP on Heroku without addtional pain/costs for either the users or the maintainer. From my current knowledge it's impossible. But then a large percent of our userbase is on Heroku, one can only guess the numbers, but I'd be surprised if it's below 30%. And we can't say these 30% are on a well maintained and committed pod currently, sadly. While I didn't read every word in this discussion I saw a clear tendency to use XMPP, possibly over Websocket or BOSH. Again this is not possible on Heroku.

So we need to decide if we want to further seek for a solution that works on Heroku or allow one that doesn't work well there. Or we won't move forward in this discussion. That's what this poll is about, support chat on Heroku or not.

(That "possibly using XMPP is in the discussions title is probably a bit confusing and should probably be changed. Nonetheless it's an option and not decided yet.)

Shmerl

Shmerl May 12th, 2013 21:57

If supporting Heroku means not using XMPP for the chat - then I'm voting No. My Yes vote was only for the effort to find how to use XMPP on Heroku.

Shmerl

Shmerl May 12th, 2013 21:59

I'd propose to create a poll for using XMPP as a solid base for chat protocol then. And see what people say.

Shmerl

Shmerl May 12th, 2013 22:08

I agree with what Seth Martin said:

I'm not interested in any other solution than
XMPP because I want to take my chats with me
when leaving the browser. I hardly consider it
screwing Heroku users by requiring they use a third
party XMPP server if they want to chat.

Christian Giménez

Christian Giménez
Agree
May 12th, 2013 22:19

I want support for XMPP. If Heroku can give support for XMPP, for me is OK.

Nevertheless, I agree with Seth: We cannot trust or data on third parties people.
I turn into "yes" because Chat feature is really needed.

Christian Giménez

Christian Giménez
Agree
May 12th, 2013 22:20

I want support for XMPP. If Heroku can give support for XMPP, for me is fine.

Nevertheless, I agree with Seth: We cannot trust or data on third parties people.
I turn into "yes" because Chat feature is really needed.

Jonne Haß

Jonne Haß May 12th, 2013 22:31

@shmerl Please edit your vote then.

Shmerl

Shmerl
Disagree
May 12th, 2013 22:33

If such solution is possible with XMPP is possible on Heroku - why not. But if not, better evaluate if this effort is really required, or there are easier workarounds, like simply using an external XMPP server on the other hosting.

Shmerl

Shmerl
Disagree
May 12th, 2013 22:34

If such solution with XMPP is possible on Heroku - why not. But if not, better evaluate if this effort is really required, or there are easier workarounds, like simply using an external XMPP server on the other hosting.

DU

[deactivated account]
Abstain
May 12th, 2013 23:21

"Abstain" really means "abstain" here. If it was my project I wouldn't want to be limited by the feature set offered by one hosting provider. (Embedding a full XMPP client in Diaspora kind of defeats the purpose, in my opinion. Embedded < dedicated).

Tom Scott

Tom Scott
Agree
May 13th, 2013 00:29

Tom Scott

Tom Scott
Abstain
May 13th, 2013 00:33

You should be able to run a pod, for free, on Heroku. Without having to set up ejabberd or some other XMPP server. If more advanced users want more advanced things, more power to them.

I don't use the CloudFront features, but I'm glad they exist.

Tom Scott

Tom Scott May 13th, 2013 00:40

@rekado blather looks interesting...honestly i had no idea anything other than xmpp4r existed as a library. wasn't too excited about using something that hasn't been updated in 3 years. ;-)

I imagine this whole XMPP feature would look something like the setup for an email server. In which you tell it what server to connect to, and what username/password to use. You can also perhaps enable or disable the "sync accounts" feature which would basically create an account on the XMPP server according to XEP-0077. This would enable users to use their existing XMPP server on Diaspora IM.

I think what most people here really can agree on is that they all want a web-based chat UI inside Diaspora. For some, this may mean a secure XMPP server for sending messages that they control. For others, it might mean just hooking up their jabber.org account and simply using the UI. We should make each option available.

Diego*

Diego*
Agree
May 13th, 2013 03:27

Mikhail Shirkov

Mikhail Shirkov
Abstain
May 13th, 2013 06:19

Jason Robinson

Jason Robinson May 13th, 2013 06:54

Loomio isn't taking my vote. Though to me the options are broken. XMPP will be split in no and abstain votes, while non-xmpp gets all the yes votes. I think we should count no/abstain against yes, the options are not well thought out.

Anyway, seriously, are we going to plan the feature set of Diaspora* (free, open source, privacy aware etc) on the whims of one proprietary company? If there is a better option than XMPP then let's hear it - but this HDD (Heroku Driven Design) talk is just mad.

Jason Robinson

Jason Robinson May 13th, 2013 06:55

Of course we should allow pods to run on Heroku as easily as possible but it should not control core functionality design.

Jonne Haß

Jonne Haß May 13th, 2013 08:54

No problem in counting the majority as absolute one, i.e. not reached if not > 50% of all votes are yes. In fact that was my original thinking.

I don't want to fight for Heroku. I want to know whether to support it or not. I see the discussion stuck otherwise.

Jason Robinson

Jason Robinson
Abstain
May 13th, 2013 10:38

Heroku should not dictate our plan for chat. As long as we don't break existing functionality in Heroku pods.

diasp_eu

diasp_eu May 13th, 2013 11:26

Free at herokus means a starter database with upto 10k rows? Because of the current federation system and duplicate posts you reach 10k very soon, hence you need to pay for running diaspora on heroku. Who is running diaspora on heroku for free?

I know joindiaspora.com runs on heroku but why not using an external xmpp server on another machine?

We should find a solution for chat, not a solution for heroku!

diasp_eu

diasp_eu
Abstain
May 13th, 2013 11:26

Rasmus Fuhse

Rasmus Fuhse May 13th, 2013 12:30

Even if heroku is not allowing XMPP, our decision is not: XMPP or not XMPP. We could also implement a fully working XMPP-chat with a per pod option to disable XMPP-chat for the pod/it's users. Users of the pod wouldn't be able to chat. But that is due to limitations of the crappy Heroku settings. Users of better servers will be able to chat with each other.

We shouldn't let Heroku dictate us how we are using our network. But on the other side we should create our software in a way that it runs even on low-profile machines. We don't need to implement an evil XMPP-hack for Heroku, but we need the option to run a non-XMPP-diaspora-account.

Rasmus Fuhse

Rasmus Fuhse
Disagree
May 13th, 2013 12:31

Let there be XMPP and give the chat-module the possibility that singe diaspora-accounts might not be linked to an XMPP-account.

Christian Giménez

Christian Giménez May 13th, 2013 16:18

Then, I don't get very well the idea of the Yes/Abstain/No options. What does these means then?

If yes means implement it on Heroku, then I have to disagree or abstain... Heroku is no important mather as long as XMPP works. Unless we agree that this is just a start until we decide another better/free/descentralized alternative.

We have to consider that we shouldn't limite the people runing pods to use Heroku. I mean, if I want to start up a pod, I can do it in my own work/home without any Heroku's connections.

I can see that we all agree to use XMPP. Is a semi-descentralized protocol and that is atractive! That is cool! :-)

Sean Tilley

Sean Tilley May 13th, 2013 18:58

@jasonrobinson I don't think it's so much about bending to the "whims" of Heroku, so much as it is the reality that many Diaspora pods are already hosted there. Crippling a featureset for users on a specific host is just as bad as bending to the whims of that host in regard to development.

I think going forward, we need to consider alternatives to Heroku, if we really want to encourage podmins to move over to be able to benefit from chat. I have heard that OpenShift has some promise, and @jonnehass did some work a while ago making diaspora work with OpenShift.

Basically, if we want to develop XMPP-powered chat and not worry about Heroku, we should come up with a good alternative to it that can better suit podmin needs in regards to these kinds of features we want to do. Otherwise, we'll need to find something else that works for everybody, regardless of what host they're on.

goob

goob
Abstain
May 13th, 2013 21:17

One the one hand, good to find a solution which works with current pod hosts. On the other hand, might be best to find best chat solution for the Diaspora network as a whole, and podmins can then find a host which can provide this. I can't decide.

Jason Robinson

Jason Robinson May 14th, 2013 06:42

@seantilleycommunit implementing a chat solution that is optional but doesn't work on Heroku will not cripple the existing user base. They just will not get the new feature.

I really do believe we should not base any design on a single hosting solution. Be it Heroku, Openshift or any other. Let's find the best solution out there, make sure it works on as many platforms as possible and then decide on that. Be it XMPP or something else.

matl

matl
Abstain
May 15th, 2013 08:46

Tom Scott

Tom Scott May 16th, 2013 17:10

@diaspeu you're misunderstanding how that works. the database doesn't lock, it merely begins deleting old records. perfectly fine for a social network that is more about what's happening right now than what happened 5 years ago. for example, what i wrote on Facebook 5 years ago is not only meaningless but terribly uninteresting to me today.

As I've said before, and I will say again, if we do this I volunteer to either code it or head up the team that's tasked to make this all work.