Loomio

Export group data

QG
Quentin Grimaud Public Seen by 645

It would be nice if we could export group data (members, conversations, decisions), so that we could import it then in a local loomio instance, or at least have a backup in case of a problem of access with the official loomio instance. We could also process decisions, generate statistics, or browse the history of all decisions more easily.

CT

Chris Taklis Tue 14 Jan 2014

i like the idea!!!

RDB

Richard D. Bartlett Tue 14 Jan 2014

Yep, I couldn't agree more!

We've got a first version of this planned and @mixirving has been working on some of the mechanics behind it. There would be a button on your group settings page for 'export group data', which would then run a background process on our servers and email you with a download link when it is ready. For a first stage, the export would be delivered as an SQL dump that will be really easy to import into another Loomio installation.

DU

[deactivated account] Wed 15 Jan 2014

Would it be possible to have an rsync type service so we didn't have to do it manually?

This will be an important service for groups/organisations that depend on Loomio for their operation.

RDB

Richard D. Bartlett Thu 16 Jan 2014

@neilmorris I'm not sure, perhaps @robertguthrie can weigh in.

As I understand it, the approach I've described is cheap in development and expensive in computation, so it might be inefficient to run it regularly. At the moment "your data" is not a concept that the database really understands.

Moving to a more computationally efficient process is going to be expensive in terms of development time, and will constrain us as we make changes to the data structure in future.

The whole database is backed up regularly though, so there is not a significant risk to organisations that depend on the service.

DU

[deactivated account] Fri 17 Jan 2014

@richarddbartlett :-) I suspect you probably underestimate the risk involved while I probably overestimate the risk. I suspect we'll find a middle way before too long.

I'll be interested in @robertguthrie's comments.

RG

Rob Guthrie Fri 17 Jan 2014

What risk are you talking about Neil? We store our backups on S3, which is a very reliable place to store them. Happy to consider another layer of backup too.

This is backing up the entire Loomio database, which is quite quick and easy to do.

To unwind one group and their users from the rest of the database takes a few hours. You would only do it if you want to move from loomio.org to another loomio installation - not to provide extra redundancy to your group's data.

DU

[deactivated account] Sat 18 Jan 2014

@robertguthrie @richarddbartlett It's the fact that only 8% of start ups, such as Loomio, actually succeed in the long term. I'd hate to be in situation where Loomio ran out of money/energy, closed it doors, administrators appointed and we users of it didn't have access to our data, especially in terms of our organisational records if things have all been recorded on Loomio.

While within the Loomio team this is probably an unthinkable scenario, there is a small risk this may happen and therefore I would prefer to have a regular backup of my groups data which I could access, and then restore in my own instance of Loomio.

RDB

Richard D. Bartlett Sun 19 Jan 2014

@neilmorris there is 0% risk that Loomio just switches off one day without warning. In the worst case scenario, we'll be sure to give everyone adequate time to extract their data, using the tool as described above. And of course, if that's unsatisfactory, anyone can always run their own instance.

DU

[deactivated account] Sun 19 Jan 2014

If there is a way of manually doing an ‘export of group data’ then there must be a relatively easy way to automate the process of downloading it and exporting it to another server.

RDB

Richard D. Bartlett Tue 21 Jan 2014

@neilmorris yes it could be automated but as I said it is too computationally expensive to be the default for all groups, so I think we could only offer that as a paid service. From a marketing perspective though, I'd hate to advertise that service, because it creates the impression that you're at risk if you don't pay for backups.

CT

Chris Taklis Wed 22 Jan 2014

@richarddbartlett c'mon paid system for back-up our data?

It must be somehow an automated system that downloads your group data and send it to an email as zip file.

RDB

Richard D. Bartlett Wed 22 Jan 2014

@christaklis yep, I don't like the sound of a paid system either.

I'm just saying we have an easy solution for manual per-group backup, that wouldn't translate easily to an automated system.

So manual backup is easy to do for free, but I don't know what the solution is for automated per-group backup without incurring some costs.

My hunch is that it would require some redevelopment of our database structure. @jonlemmon is this something your brother might be able to advise on?

RG

Rob Guthrie Wed 22 Jan 2014

Just to clarify, manual means User initiated. At any time, anyone will be able to request the data. The computer will do the work and email you when it is ready, no humans involved. This is suitable for anyone wanting to move from loomio.org to a custom loomio installation with a few hours notice.

Automatically providing the backups, daily, right now would be a wasteful use of computation so it's by request from the interface.

DU

[deactivated account] Tue 28 Jan 2014

If it did what Chris describes i.e. It must be somehow an automated system that downloads your group data and send it to an email as zip file. I would be very happy.

If it was available as an optional service for those that want/need it hopefully it wouldn't be a wasteful use of computation time.

Surely a script could be written to automate the manual pull request and then to download the data dump that you talk about @robertguthrie ....or is it more difficult than what I am imagining?

RG

Rob Guthrie Tue 28 Jan 2014

The problem is that extracting a single group from the database is computationally hard. It takes a few hours. It would only take a few people who want a daily dump to overload our ability to provide.

We are doing a thorough job of regularly backing up the whole site's data keeping your groups information safe, along with the other groups. At this stage it's a case of trust us to backup the data and make your data available by request, or move to your own hosting solution.

We really can't offer anything else at this stage sorry.

DU

[deactivated account] Tue 28 Jan 2014

@robertguthrie Thanks. I really appreciate the explanation. I now understand your difficulties in doing something that I had presumed was very easy!

RG

Rob Guthrie Wed 29 Jan 2014

Thanks Neil. Currently it's just a reality of the system and available options.

QG

Quentin Grimaud Thu 30 Jan 2014

Typically, the day Loomio has a problem and goes down for a long time, all active groups will ask for their backup data at the same time and so the problem of extraction time for the data of a single group will occur. So the time for the last group who asked it to get their backup data will be high. So the "solution" of just trusting that backups are done and not implementing anything is not a real solution in my opinion.

RG

Rob Guthrie Thu 30 Jan 2014

If many groups ask at once we can scale immediately to handle it. Each backup request will run in it's own dyno.

MI

mix irving Mon 3 Feb 2014

@neilmorris I also underestimated the complexity of this process. It turns our that a group can easily have in excess of a million objects associated with it.

for example, each discussion will need:
- a list of who comments, [and for each of these, a list of who liked the comment, all the attachments associated with that comment a list of any events (like at-mentions) associated with the comment, a list of any emails which were triggered by those notifications]
- a list of who has seen the discussion and when they last saw it
- a list of changes made the the context, with a list of all former versions
- a list of each motion ever raised in discussion [ with similar tree of items to comments above]

the full picture is cycling through all levels of the tree below, making individual calls to the database for each of a million plus items.

(apologies first tree didn't copy well)

group:
|-memberships:
|---events:
|-----notifications:
|-membership_requests:
|-pending_membership_requests:
|-admin_memberships:
|---events:
|-----notifications:
|-pending_invitations:
|-discussions:
|---motions:
|-----votes:
|-------events:
|---------notifications:
|-----did_not_votes:
|-----events:
|-------notifications:
|-----motion_readers:
|---comments:
|-----comment_votes:
|-------events:
|---------notifications:
|-----events:
|-------notifications:
|-----attachments:
|---events:
|-----notifications:
|---discussion_readers:
|-subgroups:
|---memberships:
|-----events:
|-------notifications:
|---membership_requests:
|---pending_membership_requests:
|---admin_memberships:
|-----events:
|-------notifications:
|---pending_invitations:
|---discussions:
|-----motions:
|-------votes:
|---------events:
|-----------notifications:
|-------did_not_votes:
|-------events:
|---------notifications:
|-------motion_readers:
|-----comments:
|-------comment_votes:
|---------events:
|-----------notifications:
|-------events:
|---------notifications:
|-------attachments:
|-----events:
|-------notifications:
|-----discussion_readers:

MI

mix irving Mon 3 Feb 2014

we're getting there, but optimising the solution will take a little time. At the moment, as @robertguthrie has indicated, we can export, though the process does have some turn around time.
We are absolutely committed to improving that process, and we look forward to rolling out an iteration #agile!

RT

rory tb Fri 7 Feb 2014

Would it also be possible to export more data such as contact info and geocoding ala Nationbuilder style? that would be super cool!

RDB

Richard D. Bartlett Fri 7 Feb 2014

Nice idea @rorytb. In my spare time I've been playing with ideas for what kind of data could be useful to export, e.g. as a group coordinator, I'd like to track member participation over time. Seems like the perfect kind of job for a volunteer contributor to tinker with.

CLC

Captain Lyre Calliope Mon 17 Mar 2014

I'm currently talking about piloting Loomio in Code for Boston, and this functionality would make it much easier for us to start a group here on the Loomio website and then migrate to our own deployment later on. Is there a sense yet as to timeline for landing this export/import functionality?

AI

Alanna Irving Mon 17 Mar 2014

@mixirving @robertguthrie or @richarddbartlett could probably give an update...

RG

Rob Guthrie Mon 17 Mar 2014

Shortly after crowdfunding we'll be back to feature development. In the meantime we can manually extract your data for you to move to a separate loomio instance if you need.

G

GasparI Sat 20 Aug 2016

Hi! Has there been any development in this area? It would be an important step towards data freedom. I wouldn't even mind paying a few dollars for it via PayPal.
It could be called as data migration, not backup, so it would be also a positive marketing message.

MB

Matthew Bartlett Wed 24 Aug 2016

Hi Gasparl! Thanks for resuscitating this discussion. We can now provide a full group data export for groups on our paid plans — in either HTML or spreadsheet format.

DS

DOSSE SOSSOUGA Sat 20 Aug 2016

ok

AI

Alanna Irving Wed 24 Aug 2016

@matthewbartlett Does this relate to the data analytics and CSV download features?

DS

DOSSE SOSSOUGA Tue 9 May 2017

OK