Loomio
January 8th, 2016 13:18

Backup and restore

Pavithran S
Pavithran S Public Seen by 688

The last I heard about backup and restore hasnt been given much interest. Any reason for that?
https://github.com/diaspora/diaspora/issues/5343 has some progress.

CS

Comrade Senya January 8th, 2016 14:27

I hope I can participate in it in a while, after some present jobs are done.

Jason Robinson

Jason Robinson January 8th, 2016 21:20

The latest automatic back up / restore spec proposal includes the import parts in this issue. Would be fancy to get more comments on it, split it into smaller chunks and dump a lot of bounty money on the issues - maybe get something to happen :)

Anyone wanting to give their insight, please check the proposal and comment in issue #908 . If no comments are received after some time I could just create a proposal to accept the proposal as a guideline for development.

Please don't start working on any import thingy before at least considering that proposal ;)

Pavithran S

Pavithran S January 9th, 2016 13:50

Its a very detailed proposal. Isn't there a simple way like mysql dump to csv/json and import csv/json ? Pod to pod communication sounds very complex.

Jason Robinson

Jason Robinson January 9th, 2016 18:45

Dumping database would not work for ordinary users very well - and certainly would not work in the situation where the pod just disappears. And, importing raw SQL dumps into other pods would probably not be a very nice thing to do :)

Regarding JSON, yes we already allow exporting data. The proposal details how this data could be to not only restore the data to a new pod (profile, etc), but also migrate the identity, and thus take control of previously shared posts etc.

Pavithran S

Pavithran S January 9th, 2016 20:59

Regarding JSON, yes we already allow exporting data.

Yes now we need an import function but your proposal looks too big and lot of stuff needs to be implemented. Agreed its the optimal way to go forward.

but also migrate the identity, and thus take control of previously shared posts etc.

Can't we have something just for exporting all the posts and its comments which are imported back under new account. " Taking Control" sounds complex.

Jason Robinson

Jason Robinson January 9th, 2016 21:02

Can’t we have something just for exporting all the posts and its comments which are imported back under new account. “ Taking Control” sounds complex.

That would create duplicates and there would be no interaction possibility. No comments to the old ones would arrive to your new identity, for example.

Sure, you can have (almost) anything (sane) - just find a coder to do it :) The issue has been there for a while waiting for someone to grab it. This proposal aims to solve the disappearing pod problem and clean migration. That doesn't stop anyone doing partial or full imports before a "proper" migrate solution is done.

Steffen van Bergerem

Steffen van Bergerem January 9th, 2016 21:04

Can’t we have something just for exporting all the posts and its comments which are imported back under new account. “ Taking Control” sounds complex.

So you would like to share all your previous posts again from your new profile?

CS

Comrade Senya January 22nd, 2016 14:33

So we have User and Person models. On migration we must create a new user but do we preserve the same Person model which was known for that person? I suppose we should.

CS

Comrade Senya January 22nd, 2016 14:39

Profile data export package contains also posts and comments which can be ignored during the restore process.

I don't think posts and comments can be ignored. It is possible that the new pod doesn't contain some posts (especially private). I believe it must not be dropped and we should fetch them on restore also.

CS

Comrade Senya January 22nd, 2016 14:42

https://wiki.diasporafoundation.org/Account_Backup_And_Restore#Backup_delivery_message

"backup" is signed and encrypted using the user private key

Where do we store signature? Don't we have to have a separate field in the schema for that?

CS

Comrade Senya January 22nd, 2016 15:08

Pods should schedule backups of user data once a week to the backup pods.

I think it is not frequent enough. Some people post frequently, and one week would be a huge loss for them. How about making that adjustable according to person's activity level? Or simply backup every day if there were changes. If there weren't, don't backup at all.

CS

Comrade Senya January 22nd, 2016 15:19

I also beleive we must support some kind of limitation for pods backup capacity. For example a pod wants to store backups, but no more than for 100 people. Then it shows that it doesn't receive backups anymore, but previous still work.

Jason Robinson

Jason Robinson January 22nd, 2016 19:30

So we have User and Person models. On migration we must create a new user but do we preserve the same Person model which was known for that person? I suppose we should.

Yes, we should keep the same Person as it is the same identity, just moved local instead of remote.

I don’t think posts and comments can be ignored. It is possible that the new pod doesn’t contain some posts (especially private). I believe it must not be dropped and we should fetch them on restore also.

Well, it's not feasible to import posts or comments imho. What purpose would that even solve? Your contacts would not see any of the uploaded private posts unless you also push them out to them - which would be bad because they already have them from before.

The main point is to save the identity, not every crumb of data related to it.

Where do we store signature? Don’t we have to have a separate field in the schema for that?

This would be the same signing method we use for delivering content - ie the receiver only accepts it after verifying the signature in the payload against the public key of the person. Whether this should live in the diaspora federation gem where the code is readily available is up to question. I'd say no, but of course it would be easier to implement that way. I don't consider this part part of the protocol as such, but then that is not in any way under specification anyway.

I think it is not frequent enough. Some people post frequently, and one week would be a huge loss for them. How about making that adjustable according to person’s activity level? Or simply backup every day if there were changes. If there weren’t, don’t backup at all.

I believe this can be left as detail when the thing actually works. If we leave out posts and comments, which I think is the only real way to do it, then there is absolutely no sense in sending backups daily.

I also beleive we must support some kind of limitation for pods backup capacity. For example a pod wants to store backups, but no more than for 100 people. Then it shows that it doesn’t receive backups anymore, but previous still work.

Again, if posts and comments are not part of the archive, it will always be very small. Additionally, backup pods would be randomized, guaranteeing some balancing of load between pods. Of course we can introduce a setting like this, but personally it doesn't sound super useful, considering how much data pods store normally compared to how much this would add.

One interesting case is what happens if the pod has turned off sign-ups in the event that someone wants to restore? I think we should bypass the sign-up not being on and let the user restore. This would speak towards the setting you mention, so that pods don't gather too many possible identities that could activate suddenly.

Great comments, thanks!

CS

Comrade Senya January 24th, 2016 19:30

Well, private posts may contain some valueable information for the user herself. I don't have extra copies of texts I posted, and I don't like to lose these texts. Moreover it would be definitely nice to preserve a possibility to continue conversation on some private post that was going on before the move. I don't see why we should push them again. A guid of the post is preserved, so all new comments will be federated well to the right place.

Not to say about public posts which might be wanted to be shared with some new contacts.

So I think posts, the content is extremely important part of the network. That's why people are in it. And the restore feature without posts restore would look unfinished.

CS

Comrade Senya January 24th, 2016 19:38

Maybe if we push posts to the restore pod as if there were some subscriber on it could make restore easier, since we won't need frequent backup then and any special posts restore feature. At least we could do that for public posts, since for private posts that would imply trust for restore server even before a password was entered. That is not really acceptable.

CS

Comrade Senya January 24th, 2016 19:46

https://wiki.diasporafoundation.org/Account_Backup_And_Restore#Backup_delivery_message

“backup” is signed and encrypted using the user private key. Once opened, it should contain the following schema...

It's not very clear the structure of the backup field before we "open" it. It must be some data array containing signature and encrypted data, right?

CS

Comrade Senya January 25th, 2016 11:59

BTW, does any other software in the federation (friendica, redmatrix) do some sort of backup restore?

CS

Comrade Senya January 26th, 2016 18:33

So here are my changes to the spec according to what we've discussed

https://wiki.diasporafoundation.org/index.php?title=Account_Backup_And_Restore&diff=4454&oldid=4418

Jason Robinson

Jason Robinson January 26th, 2016 20:26

@comradesenya unfortunately I'll unlikely have much time for going through these before friday - I'll reply then. Until then, it would be nice if the wiki could be left alone before accepting changes here in Loomio. I don't agree (still) to some things like including posts and comments in the backup archive and I'm not prepared to change my opinion unless some others join in and support that idea and also give valid technical ways to do it sanely which at the moment is missing.

Anyway, lets continue discussion on that, for a few days I'll have to pass on comments. I'll clean up the wiki page to reflect those items that have been mutually agreed upon then.

Once we have a clear understanding we can do a proposal.

CS

Comrade Senya January 26th, 2016 21:10

TBH, I don't see any technical problems on posts restore. Everything seems to me pretty straightforward. We just add the posts to the database of the backup pod if they aren't there yet. Maybe I miss something?

On contrary, not restoring posts would lead to weird situations, like when after you've moved some of your contacts does comment on an old post of you. Comment gets federated to your new pod, but parent post is not there! Because we haven't merged it.

Jason Robinson

Jason Robinson February 7th, 2016 20:57

So, I've now cleaned and published the spec for final discussion, cleanup and voting.

@comradesenya I left in your good idea to retry moved messages (which a small change to play with status codes instead of sending more messages around). But I removed the pointer regarding posts and comments and filed it as an issue instead.

There are also quite a few TODO's in the spec (will log issues out of these) - and I'm pretty sure there are in general things that need fixing.

All in all, no one has given strong opposition to the idea itself, so hoping we can lock down a spec 1.0, approve it for implementing in diaspora* and then it can be implemented (hopefully by @comradesenya ;)).

So basically;

  • The spec is written as markdown and should be worked on via preferably issues and pull requests
  • I chose Gitlab because not everybody likes github and because it is nice and because TheFederation was not available in Github ;) You can create an account using your Github login easily. For those who want to participate but don't like Gitlab, please feel free to use other ways to communicate or even send patch files if you want.
  • The spec is "owned" by The Federation, or more like namespaced. I would like to keep editorship until 1.0 at least though.
  • The spec was generalized into diaspora* like federated social networks, not just diaspora*. I like the idea of being able to move across platforms too. Platform specific stuff doesn't belong to the spec but should be left to implementation.

The spec is live as a working draft version 0.1.0 here: https://the-federation.info/specs/backup-restore/

The git repository is here: https://gitlab.com/TheFederation/backup-restore

Sorry this took a little while. Ping also @jhass as you've commented quite a bit on this before in github.

CS

Comrade Senya February 8th, 2016 16:54

There is a thing we haven't discussed before.

If the pod from which a person has moved to some new place is still alive, shouldn't we block registration for that name on an old pod at least for a while?

  1. If a user has moved, and some new user registers on the old pod with the same name as the previous user had, then it could be possible, that somebody will try to discover the old person by the handle or even send a message

  2. It is theoretically possible, that somewhere exists a pod, which has discovered a person, but the new pod didn't know about that pod, so "I've moved" message wasn't sent to it. In this case discovery with webfinger could return 302 code and "I've moved" message as a body so that the pod could process this message instantly.

  3. At first after the feature is introduced in the diaspora* source code, it will be usual that there are many pods that are not up-to-date and they won't get "I've moved" message in time. So, probably, it would be nice to block registrations with the same handles as some moved users had, to lower inconsistencies produced on the network.

  4. At first I thought about making kinda blocking period for a handle before it could be available again. However, I think, that we could consider user handles as an inexhaustible resource, so probably we don't need to unblock them authomatically after a period someone has moved. Maybe we could introduce an action at the podmin page to free the account that was previously occupied by someone who had moved with statistics of the discovery requests for this old account shown. Then, the podmin could free this account on request by user like "Hi! Could you please make the handle of a previously moved person available for registration again?".

CS

Comrade Senya February 8th, 2016 17:00

TODO: To avoid an extra endpoint, should we use a version of NodeInfo instead?

@jhass had strong objection against it, so probably his opinion is to be considered.

CS

Comrade Senya February 9th, 2016 04:57

TODO: Define full schema of content or place content keys in the first level.

I guess it's fine if I base the content schema on this?

Jason Robinson

Jason Robinson February 9th, 2016 20:39

If a user has moved, and some new user registers on the old pod with the same name as the previous user had, then it could be possible, that somebody will try to discover the old person by the handle or even send a message

We could add a recommendation to the spec that old local usernames are reserved permanently - that would make sense since that is what happens when an account is deleted in diaspora afaik (though would have to check to make sure). I've made an issue.

But really, the keys will have been regenerated so there isn't really a possibility of hijacking an identity like this.

It is theoretically possible, that somewhere exists a pod, which has discovered a person, but the new pod didn’t know about that pod, so “I’ve moved” message wasn’t sent to it. In this case discovery with webfinger could return 302 code and “I’ve moved” message as a body so that the pod could process this message instantly.

You mean store the whole "I've moved" message payload on the old pod and respond to webfinger queries made towards the old closed identity. That sounds good otherwise but to make it work webfinger would have to pick up the message as a response. I think that might be a bit outside spec?

3 and 4 relate to 1 - where I think we should just recommend reserving forever.

TODO: To avoid an extra endpoint, should we use a version of NodeInfo instead?
@jhass had strong objection against it, so probably his opinion is to be considered.

Well, I didn't mean the NodeInfo but a NodeInfo :P But, I think it is out of scope of this spec anyway, better keep it simple with a dedicated clean endpoint.

I guess it’s fine if I base the content schema on this?

Yes, but generalized for the spec, so we shouldn't include all the keys but a generic set. diaspora* can of course implement a larger set containing the full amount of data used in our profiles.

Dennis Schubert

Dennis Schubert February 10th, 2016 09:08

Yes, but generalized for the spec, so we shouldn't include all the keys but a generic set.

If you truely want to support "all" networks, you have to define a minimal set of keys. Otherwise, diaspora may use username where other networks might use user and everything becomes a complete mess.

CS

Comrade Senya February 10th, 2016 19:27

I believe we may define a minimal set of keys as required and everything else as optional, so other networks pick what they would like to support.

Jason Robinson

Jason Robinson February 10th, 2016 19:36

Defining a single all-around schema with strict keys is hard, especially as we ourselves use pretty unique names like "aspects". Making our export not use our own key names just to support this would be odd.

What about aliases? The spec defines the basic expected keys using the most common key names that would be expected in a generic spec, and then we define a set of aliases, for example aspects could map to something that might be more generic like contact_groups.

These aliases can be expanded as seen fit in future spec versions.

CS

Comrade Senya February 10th, 2016 19:44

In the PR I introduced format by generating a schema basing on our exported archive json document using http://jsonschema.net/

Then I manually edited it to fix some issues and I removed most of the keys from the "required" section, leaving
"name",
"private_key",
"profile",
"contacts".

The latter two - "profile" and "contacts" beign objects themselves doesn't have required fields though. So we can either make them optional as well, or make some of their contents required.

Dennis Schubert

Dennis Schubert February 10th, 2016 19:44

Defining a single all-around schema with strict keys is hard, especially as we ourselves use pretty unique names like "aspects".

Ah c'mon. What do you want to achieve? Do you want something that looks fancy but nobody is able to use between implementations or do you want to have a spec that defines a standard way to exchange account data?

If you want a spec, then define keys. Define username, contacts, groups, birthday and stuff. No alias. One key has exactly one descriptor, no exceptions. How implementations call these things internally is completely irrelevant to your spec, since you should define a data format, not a set of rules on how social networks should behave internally. Diaspora may put diaspora_id in the username field, Friendca my call the same database column lookhowfancytheuseriscalled. The spec does not care. And the spec should not care.

Jason Robinson

Jason Robinson February 10th, 2016 19:48

@dennisschubert so how do we use our JSON exports to restore then as they would be incompatible - or are you saying the diaspora JSON exports would be renamed?

Dennis Schubert

Dennis Schubert February 10th, 2016 19:50

If you write a spec that is well-written and suitable for diasporas need (that is, we can somehow export all fields), I'm sure it's an easy task adapting the spec...

Dennis Schubert

Dennis Schubert February 10th, 2016 19:53

Think about the other side, the people implementing specs. What would you do if you see a spec that literally says "username contains the users name. Be aware that this field also may be called profile_name, name or fancyfancy". How are you supposed to implement that? Rolling dices while implementing specs is surely not a good thing.

CS

Comrade Senya February 10th, 2016 20:20

Do we follow the federation protocol for the "Delivery package" and for the "I've moved" messages? If so, we must use XML instead of JSON.

CS

Comrade Senya February 10th, 2016 20:25

Also, if we reuse federation protocol for these messages, we don't have to define signing methods in our spec, since signing is already implemented on the salmon level.

CS

Comrade Senya February 10th, 2016 20:30

But there is a problem, if we want to rely on the federation protocol in the spec, we must refer to the federation protocol spec to stay formal, but AFAIK there is no formal specification document for the federation protocol.

CS

Comrade Senya February 10th, 2016 20:48

And in this case "Backup server receive route" is the usual public receive route for the server.

Jason Robinson

Jason Robinson February 10th, 2016 20:50

I don't think we should mix this and diaspora federation together. At least the spec can't be written like that.

More comments tomorrow, zzzz...

CS

Comrade Senya February 10th, 2016 21:02

Well, I don't know, that is exactly how we can reuse endpoints and signing code also. I feel that this is optimal to reuse the federation protocol, but not to invent one more way to exchange messages. To write the spec we'll need some formal definition of the federation protocol then. At least some short version.

@dennisschubert, what do you think?

Dennis Schubert

Dennis Schubert February 10th, 2016 21:05

but AFAIK there is no formal specification document for the federation protocol.

True, and at this point, there is no point in writing one. After the federation-gem efforts are done, I intent to formalize what we have at that point so we have a reference documentation for further usage and other networks as well.

I'd like to add something for Jason here. You are trying to write a spec, which is much appreciated. "Spec" is short for specification, but I get the feeling that you are actually trying to be as vague as somehow possible. I see why you're trying to do this, but if you want to come out with something that people might actually use, you have to do the opposite. Be as specific as possible. Otherwise, people will never be able to adopt the spec, everyone is going to maintain their own set interpretation errors and nothing will ever be compatible.


Example: Defining terminology.

Bad:

username contains a string.

Good:

username contains an identifier that uniquely identifies an user across the network.


Example: Defining formats.

Bad:

groups contains an array of contact groups.

Good:

groups contains an array of objects specifying groups. Group objects should contain a title attribute as well as a members array. title should be the human readable identifier. members should be an array of usernames.


Do you understand what I'm trying to say?

Jason Robinson

Jason Robinson February 11th, 2016 20:09

Regarding diaspora protocol. Yes, the initial version in the wiki that I wrote in October had the idea that the whole backup/restore would be implemented with minimal changes by hooking up to the current federation endpoints, signing, etc. This is definitely the easy route, I and to be honest I wouldn't of imagined any other way.

However, it is clear that there is heavy work going into the protocol itself and I'm not at all sure it makes sense to smash a concept like this that has absolutely nothing to do with social content in. This would weaken not only the protocol but also make the backup/restore feature unstable due to breaking changes in the protocol itself.

As such, I began looking at it from outside of diaspora. As a separate feature, not conflicting or requiring the diaspora, or any other, protocol, it would become much more stable to implement. The downsides are of course clear; more time spent on defining the spec (or specification, thanks Dennis) and more time spent on development (signing methods, endpoints, etc). The end result however would be cleaner and more stable not just for the backup/restore feature, but also the diaspora federation protocol.

I know it is hard for people here to see the federated web outside the context of diaspora, but that is exactly what I would like to do here.

So, there are two ways forward:

1) Implement using the diaspora protocol. Some of the code would then have to be implemented in the diaspora federation gem, making it a permanent feature there.
2) Continue writing a more generic specification and implement using that, with no changes at all to the diaspora federation gem.

I'm not too interested in 1) to be honest, but if that is chosen, then imho implementation can start whenever a basic flow is agreed upon, and it's already there written waiting for comments. There is no need to write fancy specifications because implementing it would require understanding something that has no specification and that needs reverse engineering. If anyone is able to do that they should be able to follow this half done spec already.

I'm really interested in 2) and that doesn't really even depend on diaspora implementing it. For that, there is much work to do and many comments needed, from outside of diaspora developers. I'll do what I can to get some comments.

Btw, regarding 2), I'd like to make use of as much of existing or upcoming standards as possible.

Using ActivityStreams2 for the JSON messages for example would enable using existing and upcoming parser libraries - the spec is currently a W3C working draft and is moving on nicely. Backup/restore would use extensions to it where needed as per AS2 spec.

For signatures, there are a few that have been mentioned within the W3C SocialWG group. The most important thing would be to use something that has support on implementation or standardization level. It is possible signatures will be a part of ActivityPub, the federation spec from the SocialWG, but this is not guaranteed unfortunately.

So, how should it be for diaspora - backup/restore on top of the diaspora protocol or backup/restore implemented as a feature nothing to do with the diaspora protocol?

Should I create a proposal now, seems a good thing to decide before moving on with specifics?

Dennis Schubert

Dennis Schubert February 11th, 2016 20:12

I still don't see the connection between a spec describing how data should be exported and a spec describing how servers should communicate with each other.

Jason Robinson

Jason Robinson February 11th, 2016 20:16

I still don’t see the connection between a spec describing how data should be exported and a spec describing how servers should communicate with each other.

You're looking at only the export/import part - which is manual. I only added that by request from @jhass . The main idea of the spec is to make backups automatic, using the whole network as a way to protect user identities (= one pod goes down, only some backups are lost temporarily).

To make this kind of automated system, servers need to understand each other. I'm not sure how that can be done without defining how servers talk to each other in the spec.

Dennis Schubert

Dennis Schubert February 11th, 2016 20:27

I see. As usual, I like well-defined standards more than self-hacked stuff. If you feel like AS is your way of exchanging stuff, sure, go ahead.

But how does ActivityStreams solve your issue? Sure, you got a way to exchange json objects, but ActivityStreams is rather incomplete at defining the fields (which, obviously, is done on purpose to keep AS extendable). You still have to very carefully design the actual fields to export? Isn't that the main issue?

Jason Robinson

Jason Robinson February 11th, 2016 20:51

You still have to very carefully design the actual fields to export? Isn’t that the main issue?

Yes that is one large issue. It's certainly not the main issue imho, I'd say the signing is very critical.

Anyway, assuming diaspora as a project would want a backup/restore feature, and let's assume for the sake of this discussion that the answer is yes, a decision needs to be made I think whether to implement it on top of the current federation layer or implement it as a separate feature not related to federation layer.

And if you want to discuss content further, I'd say implementing on top of the current diaspora protocol we should just go with the current fields as per export schema (more basic set for the automated backup archives, but from the same set). A clearly defined schema is needed however for 2).

Btw, it's nice you didn't even read the document before commenting on it ;)

Dennis Schubert

Dennis Schubert February 11th, 2016 20:53

Oh, don't worry, I have read it. I stopped at the point you tried to specify how the user has to sign in and what buttons he has to click as well as the point where you started to assume rules about how implementations should store their stuff in the database, which is why I wrote the initial comment. Nice try, though. Maybe it'll work next time.

Feel free to open a proposal.

Jason Robinson

Jason Robinson February 11th, 2016 20:56

Thanks Dennis, as polite to other people as always :)

Jason Robinson

Jason Robinson started a proposal February 11th, 2016 21:30

If diaspora* implements the backup/restore specification, should it base it on the diaspora federation protocol? Closed 10:07pm - Sunday 28 Feb 2016

Assuming diaspora* might want to implement a backup/restore process (which enables migrating across pods), should it build this support into the diaspora protocol or base it on a separate specification, not modifying the diaspora protocol to support this feature?

Note! The backup/restore spec COULD be one like https://the-federation.info/specs/backup-restore/, but this proposal is NOT about whether to implement this specification or not.

Votes:
* YES - any backup/export type specification should be built on top of the diaspora federation protocol
* NO - any backup/export type specification should not be implemented on top of the diaspora federation protocol.

For clarifications, see for example comment https://www.loomio.org/d/qsVJ2K1t/backup-and-restore#comment-923083

Results
Agree - 2
Abstain - 2
Disagree - 2
Block - 2
5 people have voted (0%)
CS

Comrade Senya
Abstain
February 11th, 2016 22:26

Well, I like both ideas. Can't decide yet. See my comment.

CS

Comrade Senya February 11th, 2016 22:40

Well, I like both ideas. The first option I like for its simplicity of implementation. The second is good as it is a completely separate piece of functionality, and the generalization would make it possible to use the spec even in some applications where we don't have social media at all. Like integrating backup/restore in some completely different platform, like, for example, Loomio. It is a big, complex and interesting task. The main question is whether it makes much sense to implement this spec somewhere outside the diaspora* federation world? If one could name a few decent applications of the spec outside the world of the software which supports the diaspora* federation protocol, I would probably support the latter proposition.

I believe that the issues with diaspora* federation protocol instability is not a big deal for the topic, because there will be transition periods which will guarantee backward compatibility for sensible periods of time, so it'll be completely transparent for the feature because we operate on the upper level and the transport level is provided to us.

RZ

Renato Zippert
Disagree
February 12th, 2016 00:20

Looks safer to me to avoid allowing user information traveling on the network in case of any kind of vulnerability that could trigger the process, entirely or partially, without user consent, to a malicious destination or with a man-in-the-middle.

Dennis Schubert

Dennis Schubert February 12th, 2016 00:23

@renatozippert "disagree" does not mean the data will not be sent over the network. Apparently, that's not even a point open for discussion here. The question is if the spec should be based on diasporas protocol or something different like ActivityStreams.

RZ

Renato Zippert February 12th, 2016 00:32

Thanks @dennisschubert... I'll abstain for now then and read more about it. I need to understand this better.

RZ

Renato Zippert
Abstain
February 12th, 2016 00:33

Looks safer to me to avoid allowing user information traveling on the network in case of any kind of vulnerability that could trigger the process, entirely or partially, without user consent, to a malicious destination or with a man-in-the-middle.

Jason Robinson

Jason Robinson February 12th, 2016 18:00

If it helps, we can first vote whether to work on this feature at all - if that is required to continue. I feel it would be better to decide how to do it first and then vote to approve it. It would not be fair to vote on approving something that isn't finished as a plan.

@renatozippert the whole backup/restore idea here is about automatic encrypted backups over the network. If you are wary of this please vote disagree when there is a vote to approve the feature for implementation - as said the current proposal is not approving the feature for implementation.

@dennisschubert - ActivityStreams2 is not a protocol, it is a content serialization specification. I doubt it can be used to federate anything...

RZ

Renato Zippert
Abstain
February 12th, 2016 20:48

Can't see clearly the benefits or issues of both options.

RZ

Renato Zippert February 12th, 2016 20:54

The problem of basing this on the protocol would be having to change the protocol that is related to other projects?
Right now I tend to agree with basing on the protocol, as creating the backups and moving profiles would be much more elegantly implemented with protocol support, not as something "external".

Florian Staudacher

Florian Staudacher February 13th, 2016 12:19

Ultimately I would love to see our federation protocol getting used for all kinds of stuff, but right now I think it wouldn't make sense to take it much further than "social media" content.
For me, an account backup-replication protocol is out of scope of the original federation. I'd imagine in a far away future you could sync your account backup to some not-even-related-to-diaspora service (e.g. owncloud?) and that shouldn't depend on implementing anything else other than the backup endpoint, imho.

CS

Comrade Senya February 14th, 2016 23:30

Ok, how about the following idea.

We agree on the backup/restore message set and implement it with both federation and non-federation based transport protocol? This is redundant work, but not that big, and it will make happy everyone. After a while, we'll see, which version got integrated to the third party services (that's why we standartize it at all) and drop the one, that is redundant. I read that this is how decisions were made in Linux kernel sometimes - by including two competing technologies.

So, as for the specification, it must then be made agnostic to the transport protocol used.

P.S. Does XMPP have anything on the topic?

Elm

Elm February 19th, 2016 11:43

Hi ! I do not understand what is technically going on here, but it is great to hear that this fundamental feature is on its way. Thanks !

Bady

Bady
Abstain
February 23rd, 2016 04:47

It'll be great if someone can explain the difference between two (backup/restore based on and not based on federation protocol) in layman terms so that people like me can take a better decision about it!

Bady

Bady
Abstain
February 23rd, 2016 04:48

I see lot of comments here, but it'll be great if someone can explain the difference between two (backup/restore based on and not based on federation protocol) in layman terms so that people like me can take a better decision about it!

Bady

Bady
Abstain
February 23rd, 2016 04:49

I can see a lot of comments here, but it'll be great if someone can explain the difference between two (backup/restore based on and not based on federation protocol) in layman terms so that people like me can take a better decision about it!

CS

Comrade Senya
Agree
February 23rd, 2016 11:10

I believe, reusability is valuable and the acc. b/r spec may be written in a neutral way (no links to federation, so may be used outside) while staying compatible with the federation protocol.

goob

goob February 23rd, 2016 14:24

I don't consider myself competent to vote on this, so will leave it up to those of you with the technical knowledge required.

hewiak

hewiak
Agree
February 24th, 2016 20:38

usability trumps just about all other arguments, of which security remains among the most important

C

CSammy
Abstain
February 25th, 2016 01:12

I'm against voting exclusively for one or the other solution. To me, it is not clear whether this vote is on single accounts or entire pods. I also don't want to vote over anything in regards to federation before the new federation gem is finished.

CS

Comrade Senya started a proposal February 29th, 2016 14:50

Shall we encypt the backup archive when a user exports it manually? Closed 3:07am - Tuesday 8 Mar 2016

Currently we have the archive export feature and it is being exported unencrypted. It is convenient since a user can browse the contents of the archive himself. On the other hand, if the user by his mistake (or some malicious software) passes the archive to a third party, it would lead to bad consequences. We might encrypt the archive with a password like it was proposed for the automatic backup feature. (Here is a possible rough implementation of the archive encyprtion presented.) This way it gonna be more safe, but unreadable without some extra tools (Here is a snippet for decrypting archives produced by the modified export feature).

Results
Agree - 2
Abstain - 2
Disagree - 2
Block - 2
13 people have voted (0%)
Brad Koehn

Brad Koehn February 29th, 2016 15:07

I would recommend not encrypting the data; it adds additional complexity to the system that the developer and user ultimately needs to keep track of, introduces additional failure points, while not providing much in the way of actual benefit to the user.

Any user can encrypt the data him/herself using a variety of tools once its downloaded.

Dennis Schubert

Dennis Schubert
Disagree
February 29th, 2016 19:46

If the user is not able to keep the manual export safe, he is also not able to keep the key/passphrase used for decryption save, so there is no loss and no win.

DU

[deactivated account] February 29th, 2016 21:48

I also would recommend not to encrypt the data; however it would be good to tell users, that the data is unencrypted and should be kept in a safe place before downloading.

Brad Koehn

Brad Koehn
Block
February 29th, 2016 22:48

goob

goob
Disagree
March 1st, 2016 18:51

If a user wants to encrypt their backup they can do so once they have downloaded it. I think we should have 'encryption' when restoring a backup to a new account, by using a private key produced by the originating pod, or something like that.

Dennis Schubert

Dennis Schubert March 1st, 2016 18:52

@bradkoehn Blocking proposals is only valid if the proposal itself is invalid, which it is not. Please don't block, but disagree.

Brad Koehn

Brad Koehn
Disagree
March 1st, 2016 19:22

Chris

Chris March 1st, 2016 19:37

I'm going to abstain from voting because I've had next to zero participation in the whole backup/restore conversation. And I feel like there are people here who know better than me what should happen. But I have this to say:

Isn't this why we use https? So that files are transferred over encrypted lines? I think it would just introduce another layer of complexity, for not much benefit.

And correct me if I'm wrong, but if you were going to send an encrypted archive, it would have to be encrypted on the server side...using keys generated....somewhere; either locally or on the server and they'd have to be sent over the same (again, already encrypted) lines we're worried about someone tapping. I'd rather just do all of it locally.

I've just repeated what other people have already said, and probably poorly. So, my current recommendation is no, but I'm not an expert so I'm not voting.

Maybe encryption should be optional?

Chris

Chris
Abstain
March 1st, 2016 19:38

Isn’t this why we use https? So that files are transferred over encrypted lines? I think it would just introduce another layer of complexity, for not much benefit.

goob

goob March 3rd, 2016 20:01

Blocking proposals is only valid if the proposal itself is invalid, which it is not. Please don’t block, but disagree.

When was that decided, Dennis? My memory is that the decision was that a 'block' was considered a 'strong disagreement' and no more. At the moment the only proposal regarding this that I can find is this one. I'd be grateful if you'd point me to the proposal which changed this so that I know the current situation. Thanks.

G

Globulle
Disagree
March 4th, 2016 06:41

It is useful to strongly warn the user to keep the archive in a safe (and potentially encrypted) place, but this is not diaspora*'s responsibility to do it.

Dennis Schubert

Dennis Schubert March 4th, 2016 08:16

When was that decided, Dennis?

Actually, good question. That's how we handled in the past. Do you feel like we should write an official documentation?

CS

Comrade Senya March 4th, 2016 08:45

This says:

Block does not prevent the proposal from passing; it is simply a strong message that more discussion might be needed.

Dennis Schubert

Dennis Schubert March 4th, 2016 08:45

So I shall be wrong! Sorry. :)

goob

goob March 4th, 2016 15:46

Oh, thanks for finding that, Senya. I knew I'd read it somewhere!

Pavithran S

Pavithran S
Disagree
March 5th, 2016 13:21

Please don't overdo things and increase complexity. Its users duty to encrypt or keep his data save. Since it was mentioned by Chris that we anyways use https for transfers shouldn't that be enough? :open_mouth:

Ravenbird

Ravenbird
Abstain
March 7th, 2016 07:35

Mathijs de Bruin

Mathijs de Bruin
Agree
March 7th, 2016 08:33

Voting behaviour of community members should be private at all times. When using OpenSSL as in the snippet decryption is simple enough. When in doubt, make it opt-out.

Elm

Elm
Agree
March 7th, 2016 12:59

I’m OK for encryption if this feature can be coded afterwards (since it is not first on the priority list) and is made “opt-outable” (for people who does not need such privacy security).

Frode Lindeijer

Frode Lindeijer
Abstain
March 7th, 2016 16:19

How about making it optional through an "Encrypt exported data" checkbox? :monkey:

Steffen van Bergerem

Steffen van Bergerem
Disagree
March 7th, 2016 16:58

C

CSammy
Disagree
March 7th, 2016 21:12

It adds unnecessary complexity for a feature that's ultimately the user's responsibility. Export should not be possible without using https though. If I'd want my backup to be encrypted, I'd decrypt the pre-encryp. backup and encrypt it myself again.

SuperTux88

SuperTux88
Disagree
March 8th, 2016 00:47

This doesn't add security, because the password/key would be sent over the same connection as the backup-archive. It only adds complexity on the server side and on the user-side, if the user wants to open the archive.

CS

Comrade Senya started a proposal May 10th, 2016 22:20

Include comments of other people to the user's data archive Closed 1:02am - Monday 23 May 2016

Outcome
by Comrade Senya April 25th, 2017 05:44

Proposal passes, comments will be included in the user's archive.

We have the feature request for this. The user's data archive which we export now includes only user's own comments. But if we want to import posts with comments from the archive, the comments won't look consistent unless we also include other people's comments for that posts to the archive. However it maybe viewed as data safety violation. Whether we should add other people's comments to the user data archive?

Results
Agree - 12
Abstain - 12
Disagree - 12
Block - 12
18 people have voted (0%)
CS

Comrade Senya
Agree
May 10th, 2016 22:26

The user who exports his data already has access to the other people's comments on his posts. So the export feature will just represent the data which is already available in a different manner, it won't expose anything new, so it isn't unsafe.

Dennis Schubert

Dennis Schubert
Disagree
May 10th, 2016 22:29

When a user comments under a limited post, the commenter expects that the comment will not leave the scope of given comment. During exporting, data can be lost (for example by people not caring about their stuff, uploading them to public sources...)

SuperTux88

SuperTux88
Agree
May 11th, 2016 01:10

We can't control it anyway (everybody can copy/paste). Also, if the post-author moves to another pod, the new pod needs to know all comment, so that comment-authors still can delete their comments (which is relayed over the post-author)

G

Globulle
Agree
May 11th, 2016 07:20

When you post on the profile of someone else, you know that you are not completely in control. And as long as a limited post remains limited, I assume the risk of undesirable leak is very weak.

Flaburgan

Flaburgan May 11th, 2016 08:08

To comment on a limited post means you trust your friend and his podmin. With this proposition, a user can change his pod, and you will be forced to trust the new podmin for the old data you posted. On another hand, you haven't access to your friend contacts anyway so you don't know with which pod the comment is shared initially, and the whole model of diaspora* is based on the assumption that you trust your friend, and your friend trust his podmin, so the situation seems not worse here than before. The only case left is how the user will store the archive. I'm wondering what Facebook does? Does it include friends comments in the export?

Elm

Elm
Agree
May 11th, 2016 08:17

G

Globulle May 11th, 2016 08:19

I'm wondering what Facebook does? Does it include friends comments in the export?

It may have changed since I did it (more than 1 year ago) but it didn't saved any comment (either your comments on other's messages nor friends comments on your messages).
Note that Facebook export was not intended to import the profile elsewhere later.

Erwan Guyader

Erwan Guyader
Agree
May 11th, 2016 10:48