Export group data

@richarddbartlett c'mon paid system for back-up our data?

It must be somehow an automated system that downloads your group data and send it to an email as zip file.

@christaklis yep, I don't like the sound of a paid system either.

I'm just saying we have an easy solution for manual per-group backup, that wouldn't translate easily to an automated system.

So manual backup is easy to do for free, but I don't know what the solution is for automated per-group backup without incurring some costs.

My hunch is that it would require some redevelopment of our database structure. @jonlemmon is this something your brother might be able to advise on?

Just to clarify, manual means User initiated. At any time, anyone will be able to request the data. The computer will do the work and email you when it is ready, no humans involved. This is suitable for anyone wanting to move from loomio.org to a custom loomio installation with a few hours notice.

Automatically providing the backups, daily, right now would be a wasteful use of computation so it's by request from the interface.

If it did what Chris describes i.e. It must be somehow an automated system that downloads your group data and send it to an email as zip file. I would be very happy.

If it was available as an optional service for those that want/need it hopefully it wouldn't be a wasteful use of computation time.

Surely a script could be written to automate the manual pull request and then to download the data dump that you talk about @robertguthrie ....or is it more difficult than what I am imagining?

The problem is that extracting a single group from the database is computationally hard. It takes a few hours. It would only take a few people who want a daily dump to overload our ability to provide.

We are doing a thorough job of regularly backing up the whole site's data keeping your groups information safe, along with the other groups. At this stage it's a case of trust us to backup the data and make your data available by request, or move to your own hosting solution.

We really can't offer anything else at this stage sorry.

@robertguthrie Thanks. I really appreciate the explanation. I now understand your difficulties in doing something that I had presumed was very easy!

Thanks Neil. Currently it's just a reality of the system and available options.

Typically, the day Loomio has a problem and goes down for a long time, all active groups will ask for their backup data at the same time and so the problem of extraction time for the data of a single group will occur. So the time for the last group who asked it to get their backup data will be high. So the "solution" of just trusting that backups are done and not implementing anything is not a real solution in my opinion.

If many groups ask at once we can scale immediately to handle it. Each backup request will run in it's own dyno.

@neilmorris I also underestimated the complexity of this process. It turns our that a group can easily have in excess of a million objects associated with it.

for example, each discussion will need:
- a list of who comments, [and for each of these, a list of who liked the comment, all the attachments associated with that comment a list of any events (like at-mentions) associated with the comment, a list of any emails which were triggered by those notifications]
- a list of who has seen the discussion and when they last saw it
- a list of changes made the the context, with a list of all former versions
- a list of each motion ever raised in discussion [ with similar tree of items to comments above]

the full picture is cycling through all levels of the tree below, making individual calls to the database for each of a million plus items.

(apologies first tree didn't copy well)

group:
|-memberships:
|---events:
|-----notifications:
|-membership_requests:
|-pending_membership_requests:
|-admin_memberships:
|---events:
|-----notifications:
|-pending_invitations:
|-discussions:
|---motions:
|-----votes:
|-------events:
|---------notifications:
|-----did_not_votes:
|-----events:
|-------notifications:
|-----motion_readers:
|---comments:
|-----comment_votes:
|-------events:
|---------notifications:
|-----events:
|-------notifications:
|-----attachments:
|---events:
|-----notifications:
|---discussion_readers:
|-subgroups:
|---memberships:
|-----events:
|-------notifications:
|---membership_requests:
|---pending_membership_requests:
|---admin_memberships:
|-----events:
|-------notifications:
|---pending_invitations:
|---discussions:
|-----motions:
|-------votes:
|---------events:
|-----------notifications:
|-------did_not_votes:
|-------events:
|---------notifications:
|-------motion_readers:
|-----comments:
|-------comment_votes:
|---------events:
|-----------notifications:
|-------events:
|---------notifications:
|-------attachments:
|-----events:
|-------notifications:
|-----discussion_readers: