Loomio

Loomio not running after update

MM
Michael McKinsey Public Seen by 44

Hi,

I have been running a loomio install for quite some time (years). It has been a long time since I did an update.

I ran the following:

cd loomio-deploy/

docker-compose down

docker-compose pull

docker-compose run app rake db:migrate

ERROR: No such service: app

I upgraded docker, and did a general update on my system, which is Ubuntu 14.04. I had to do a kernel update to get docker running after the update, but that seems to be working fine.

When I use:

docker-compose up -d

I end up with a:

502 Bad Gateway


nginx/1.17.5

When looking at:

docker-compose logs -f

I am getting a massive amount of scrolling text, it is difficult to pull out what might be important, it just keeps cycling through and shooting out massive lists of text.

I will include some snipits from the log in the hopes it may help. Any advice as to how to proceed would be great.

Thanks!

RG

Rob Guthrie Thu 21 Nov 2019

Hi @Michael McKinsey. try a git pull origin master to begin with.

MM

Michael McKinsey Thu 21 Nov 2019

@Rob Guthrie Thanks for the quick response - that seemed to get me further along. Now the update command exists and it runs, however seems to fail ending with:

docker-compose run app rake db:migrate

7f6f03435000-7f6f03436000 rw-p 00003000 00:3d 77 /usr/local/lib/ruby/2.6.0/x86_64-linux/enc/encdb.so

7f6f03436000-7f6f03461000 r--p 00000000 00:3d 50 /usr/local/lib/libruby.so.2.6.5

7f6f03461000-7f6f03695000 r-xp 0002b000 00:3d 50 /usr/local/lib/libruby.so.2.6.5

7f6f03695000-7f6f0377b000 r--p 0025f000 00:3d 50 /usr/local/lib/libruby.so.2.6.5

7f6f0377b000-7f6f03781000 r--p 00344000 00:3d 50 /usr/local/lib/libruby.so.2.6.5

7f6f03781000-7f6f03784000 rw-p 0034a000 00:3d 50 /usr/local/lib/libruby.so.2.6.5

7f6f03784000-7f6f03797000 rw-p 00000000 00:00 0

7f6f03797000-7f6f03798000 r--p 00000000 00:3d 40 /lib/x86_64-linux-gnu/ld-2.28.so

7f6f03798000-7f6f037b6000 r-xp 00001000 00:3d 40 /lib/x86_64-linux-gnu/ld-2.28.so

7f6f037b6000-7f6f037be000 r--p 0001f000 00:3d 40 /lib/x86_64-linux-gnu/ld-2.28.so

7f6f037be000-7f6f037bf000 r--p 00026000 00:3d 40 /lib/x86_64-linux-gnu/ld-2.28.so

7f6f037bf000-7f6f037c0000 rw-p 00027000 00:3d 40 /lib/x86_64-linux-gnu/ld-2.28.so

7f6f037c0000-7f6f037c1000 rw-p 00000000 00:00 0

7ffd31c1e000-7ffd3241d000 rw-p 00000000 00:00 0 [stack]

7ffd32559000-7ffd3255c000 r--p 00000000 00:00 0 [vvar]

7ffd3255c000-7ffd3255e000 r-xp 00000000 00:00 0 [vdso]

ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

[NOTE]

You may have encountered a bug in the Ruby interpreter or extension libraries.

Bug reports are welcome.

For details: https://www.ruby-lang.org/bugreport.html

SEGV received in ILL handler

RG

Rob Guthrie Thu 21 Nov 2019

You may want to backup your env file then run the generate config stuff from the readme again. I think that will change your default Docker images

MM

Michael McKinsey Thu 21 Nov 2019

@Rob Guthrie

./scripts/create_env loomio.fr33.me mmc1800@gmail.com

Changed my SMTP settings back to the way they are in my backed up env

root@loomio:~/loomio-deploy# docker-compose run app rake db:migrate

WARNING: Found orphan containers (loomiodeployloomio1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.

Creating loomio-redis

Creating loomio-db

/usr/local/bundle/gems/ffi-1.11.1/lib/ffi/library.rb:112: [BUG] Illegal instruction at 0x00007f8d467197a2

ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]

-- Control frame information -----------------------------------------------

c:0084 p:---- s:0462 e:000461 CFUNC :open

c:0083 p:0022 s:0456 e:000455 BLOCK /usr/local/bundle/gems/ffi-1.11.1/lib/ffi/library.rb:112 [FINISH]

c:0082 p:---- s:0447 e:000446 CFUNC :each

c:0081 p:0113 s:0443 e:000442 BLOCK /usr/local/bundle/gems/ffi-1.11.1/lib/ffi/library.rb:109 [FINISH]

c:0080 p:---- s:0436 e:000435 CFUNC :map

c:0079 p:0069 s:0432 e:000431 METHOD /usr/local/bundle/gems/ffi-1.11.1/lib/ffi/library.rb:99

c:0078 p:0079 s:0425 e:000424 CLASS /usr/local/bundle/gems/sassc-2.2.1/lib/sassc/native.rb:11

c:0077 p:0007 s:0421 e:000420 CLASS /usr/local/bundle/gems/sassc-2.2.1/lib/sassc/native.rb:6

c:0076 p:0014 s:0418 e:000417 TOP /usr/local/bundle/gems/sassc-2.2.1/lib/sassc/native.rb:5 [FINISH]

c:0075 p:---- s:0415 e:000414 CFUNC

...

7ffd2172e000-7ffd21731000 r--p 00000000 00:00 0 [vvar]

7ffd21731000-7ffd21733000 r-xp 00000000 00:00 0 [vdso]

ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

[NOTE]

You may have encountered a bug in the Ruby interpreter or extension libraries.

Bug reports are welcome.

For details: https://www.ruby-lang.org/bugreport.html

SEGV received in ILL handler

MM

Michael McKinsey Thu 21 Nov 2019

@Rob Guthrie I have backups of the database and info etc. If I need to wipe this server and install on a newer distro or something and I can import the databases that would be fine. I realize 14.04 is getting pretty old. I really hope I will not lose the loomio groups on this server, as they are active, but I wanted to upgrade to get the newer features.

RG

Rob Guthrie Thu 21 Nov 2019

Illegal instruction may mean that is required. If you can run this command (which I took from the backup script)

docker exec loomio-db su - postgres -c 'pg_dump -O -Fc loomio_production -f /pgdumps/loomio_production.dump'


and keep a copy of that loomio_production.dump and your uploads folder then we can run the migrations over the database from a fresh install.

MM

Michael McKinsey Thu 21 Nov 2019

okay I am up to try that getting this error:

root@loomio:~/loomio-deploy# docker exec loomio-db su - postgres -c 'pgdump -O -Fc loomioproduction -f /pgdumps/loomio_production.dump'

pgdump: [custom archiver] could not open output file "/pgdumps/loomioproduction.dump": Permission denied

RG

Rob Guthrie Thu 21 Nov 2019

can you run this first

docker-compose up db -d 

that might help it run, mapping that directory it cannot access

MM

Michael McKinsey Thu 21 Nov 2019

root@loomio:~/loomio-deploy# docker-compose up db -d

WARNING: Found orphan containers (loomiodeployloomio1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.

ERROR: No such service: -d

RG

Rob Guthrie Thu 21 Nov 2019

try shuffling the param around. maybe docker-compose up -d db

RG

Rob Guthrie Thu 21 Nov 2019

and remove your orphans. that's safe

MM

Michael McKinsey Thu 21 Nov 2019

root@loomio:~/loomio-deploy# docker-compose up -d db

Creating loomio-db

root@loomio:~/loomio-deploy# docker exec loomio-db su - postgres -c 'pgdump -O -Fc loomioproduction -f /pgdumps/loomio_production.dump'

pgdump: [custom archiver] could not open output file "/pgdumps/loomioproduction.dump": Permission denied

root@loomio:~/loomio-deploy#

RG

Rob Guthrie Thu 21 Nov 2019


docker exec -ti loomio-db su - postgres -c 'pg_dump loomio_production' > loomio_production.sql

How about that one? do you know anything about pg_dump and what flag to use?

MM

Michael McKinsey Thu 21 Nov 2019

@Rob Guthrie Looks like that worked:

root@loomio:~/loomio-deploy# ls -lah|grep -i loom

-rw-r--r-- 1 root root 14M Nov 21 12:28 loomio_production.sql

so I need my uploads folder and that file to bring to a new install - I will grab that and spin up another VM on the same IP - how far should I go with the install before checking back in how to get this database imported?

RG

Rob Guthrie Thu 21 Nov 2019

I would also take a copy of your pgdata folder, you may not need to import the sql if that comes along too. the sql just means you can easily move the data between postgresql versions if the pgdata dir is incompatible with it's new host

MM

Michael McKinsey Thu 21 Nov 2019

@Rob Guthrie I am getting the same error on an 18.04 fresh install. Not sure what is causing this, but I have not copied anything over from the other machine. Following the install instructions at https://github.com/loomio/loomio-deploy - the only thing I did differently is I installed docker directly following instructions at https://docs.docker.com/install/linux/docker-ce/ubuntu/ because I was getting some errors about not finding the docker binary using the snap install. This is all fresh and clean install.

root@loomio:~/loomio-deploy# docker-compose run app rake db:setup

..... lots of scrolling text ...

7f586c967000-7f586c968000 rw-p 00000000 00:00 0

7fff70f43000-7fff71742000 rw-p 00000000 00:00 0 [stack]

7fff717e1000-7fff717e4000 r--p 00000000 00:00 0 [vvar]

7fff717e4000-7fff717e6000 r-xp 00000000 00:00 0 [vdso]

ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

[NOTE]

You may have encountered a bug in the Ruby interpreter or extension libraries.

Bug reports are welcome.

For details: https://www.ruby-lang.org/bugreport.html

SEGV received in ILL handler

RG

Rob Guthrie Thu 21 Nov 2019

what cpu architecture is your server? I wonder if this is 32bit or 64bit issue

MM

Michael McKinsey Thu 21 Nov 2019

root@loomio:~/loomio-deploy# uname -a

Linux loomio 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12 10:36:11 UTC 2019 x8664 x8664 x86_64 GNU/Linux

RG

Rob Guthrie Fri 22 Nov 2019

Sorry, I'm stumped right now. I'll keep thinking. Your .env file uses "latest" for the loomio image tag right? These images are running well on my boxes with the sameish uname results.

 4.18.0-25-generic #26-Ubuntu SMP Mon Jun 24 09:32:08 UTC 2019 x8664 x8664 x86_64 GNU/Linux

MM

Michael McKinsey Fri 22 Nov 2019

Nothing in the env file says anything about image tag or "latest".

RG

Rob Guthrie Fri 22 Nov 2019

no, the "dot env" file ie: .env

MM

Michael McKinsey Fri 22 Nov 2019

root@loomio:~/loomio-deploy# more .env
LOOMIO_CONTAINER_IMAGE=loomio/loomio
LOOMIO_CONTAINER_TAG=latest
MAILIN_CONTAINER_IMAGE=loomio/mailin-docker
MAILIN_CONTAINER_TAG=latest
POSTGRES_CONTAINER_TAG=9.6
RG

Rob Guthrie Fri 22 Nov 2019

I have seen this issue recently with someone else. So It's a thing. I'll see if I can find some energy to look properly over the next few days.

MM

Michael McKinsey Fri 22 Nov 2019

I am poking around in the dark a bit, could it be the ruby version? Whatever it is the update broke it in the same way on the old box as the new fresh 18.04 install.

RG

Rob Guthrie Fri 22 Nov 2019

is it a cloud VPS?

MM

Michael McKinsey Fri 22 Nov 2019

It is a KVM virtual machine running on a bare metal server I have in a DC

RG

Rob Guthrie Fri 22 Nov 2019

Oh I know! Maybe you can just build it yourself!

put a copy of the loomio repo into your loomio-deploy folder:

git clone git@github.com:loomio/loomio.git (or maybe the https url?)

and modify docker-compose.yml to look a bit like this (I've added build and commented image)

MM

Michael McKinsey Sat 23 Nov 2019

hmmm I guess I can go down that road, but then updates will be a hassle it would seem. Although this update has brought the server down and not sure I can recover it so hard to be any worse I guess.

RG

Rob Guthrie Sat 23 Nov 2019

It’s super easy to update the source. Hit pull origin master. And you can always switch back to an image later if they start working for you.

MM

Michael McKinsey Sat 23 Nov 2019

okay I will give it a shot - sorry to take your time and I appreciate your availability. I was an original contributor to the project in the first crowd funding and I love Loomio. I am always trying to get orgs I work with to use it, and I hope I can get this back up in time to not have people think it is a bad idea to use it.

RG

Rob Guthrie Sat 23 Nov 2019

Keep those backup files safe and we can do it. The migrations might need hand holding but I know it’s possible.

MM

Michael McKinsey Sat 23 Nov 2019

@Rob Guthrie

So here is an issue - the command that is failing is:

docker-compose run app rake db:setup

I may not be following the logic completely, but I do not think this is even spinning up (sorry I do not have a lot of experience with what terms to use for docker components) worker, it spins up loomio-redis and loomio-db and then it gets an illegal instruction at

root@loomio:~/loomio-deploy# docker-compose run app rake db:setup

Creating loomio-redis ... done

Creating loomio-db ... done

/usr/local/bundle/gems/ffi-1.11.1/lib/ffi/library.rb:112: [BUG] Illegal instruction at 0x00007fd0f83b97a2

ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]

-- Control frame information -----------------------------------------------

c:0084 p:---- s:0462 e:000461 CFUNC :open

c:0083 p:0022 s:0456 e:000455 BLOCK /usr/local/bundle/gems/ffi-1.11.1/lib/ffi/library.rb:112 [FINISH]

If I am understanding what you asked me to do above, I am replacing the container of loomio-worker with some local build command, but I am erroring out before we ever run worker. Were you expecting me to change all the containers with local build commands? Apologies this is all a bit new to me. I realize I may be missing something simple.

Am I following? And does it make sense that the above change made no difference? I get the same thing when I try to build the DB.

It leaves loomio-redis and loomio-db up, I need to close them down.

[NOTE]

You may have encountered a bug in the Ruby interpreter or extension libraries.

Bug reports are welcome.

For details: https://www.ruby-lang.org/bugreport.html

SEGV received in ILL handler

root@loomio:~/loomio-deploy# docker-compose down

Stopping loomio-deployapprun_1 ... done

Stopping loomio-db ... done

Stopping loomio-redis ... done

Removing loomio-deployapprun_1 ... done

Removing loomio-db ... done

Removing loomio-redis ... done

RG

Rob Guthrie Sat 23 Nov 2019

Yes. Swap both Loomio containers to build and leave others as is. If you’re getting the same error as the images that’s interesting but sad

MM

Michael McKinsey Sat 23 Nov 2019

@Rob Guthrie

Okay, so I am making progress. I changed app and worker to build, and that seemed to be enough to get everything to start up. I tried with just app, but worker was crashing, so I changed that back to build. I can now get a login screen on the app when I start it.

Prior to starting it I tared up the pgdata folder and moved it over to the new server and replaced the pgdata folder, and then ran the rake migrate command which seemed to work (had never worked before) and then copied the env file over from the old server. I tarred up and replaced the upload folder.

I was getting errors and would not start with the new env file, and I looked at it and decided I should copy the DeviseSecret and SecretCookieToken entries to the old env file from the new one and that worked.

Now I get a login screen, which allows me to use google to login as I could on the old server, but after I login I get a "Something went wrong" loomio screen.

Feel like I am getting close.

On closer examination I do not believe the app rake migrate command is actually working. I deleted pgdata and reverted back to the old database several times now. When I run the commmand:

docker-compose run app rake db:migrate

it starts to do the migration:

Migrating to AddAnnouncements (20171114224912)
== 20171114224912 AddAnnouncements: migrating =================================
-- create_table(:announcements, {:id=>:integer})
   -> 2.5373s
== 20171114224912 AddAnnouncements: migrated (2.5375s) ========================

Migrating to AddAuthorToAnnouncement (20171115232230)
== 20171115232230 AddAuthorToAnnouncement: migrating ==========================
-- add_column(:announcements, :author_id, :integer, {:default=>false, :index=>true})
   -> 2.3210s
== 20171115232230 AddAuthorToAnnouncement: migrated (2.3212s) =================

Then it errors out:

Migrating to UpdateMeetingScores (20180416223444)

== 20180416223444 UpdateMeetingScores: migrating ==============================

rake aborted!

StandardError: An error has occurred, this and all later migrations canceled:

unknown attribute 'score_counts' for PollOption.

/usr/local/bundle/gems/activemodel-5.2.3/lib/activemodel/attributeassignment.rb:53:in `assignattribute'

When I just spin up the server after this partial migration, the server will run, and it lets me login with google (the email login button doesn't work) and I get the "something went wrong" screen.

A couple of times going through various slightly different order of operations I have gotten it to log me in and then it thinks I am a new user (so obviously the database is not getting imported).

Sorry for the mess, but I have certainly gotten much closer now, though we still seem to be a bit off.

RG

Rob Guthrie Sat 23 Nov 2019

Ok great!! Congratulations!! Almost there. I’ll pick this up tomorrow

MM

Michael McKinsey Sat 23 Nov 2019

@Rob Guthrie

Thanks. I have been trying various things to get this to work, might need to start fresh on the new install but I believe I have been following the logic of what is happening so it should be okay though I have made many modifications to env files and deleted and restored pgdata many times and have been running it in various states trying to figure out what is going on. Often I end up in a state where it looks like the database server is either not starting or it cannot be reached by the other containers.

I have daily backups of the database and they are even sent off the old server, so having the data from before any of this happened is no problem (I have a cron script running pgdump every day and backing up the uploads folder and mailing that all off of the server as well as copying it to a remote FTP).

MM

Michael McKinsey Sat 23 Nov 2019

@Rob Guthrie

I renamed the loomio-deploy on the new server and started fresh - I can get everything to run according to the install (seems to all be working - could be some failing going on but I don't see anything that jumps out to me in the logs) by replacing the containers for app and worker with a local build.

How to import the database from the old server and get it to do a migration etc., and the exact way to bring in what I need from the other env file (primarily the google authentication and my email settings) - are left to do. Also I assume I need to restore my uploads folder.

I have it set up and ready to run as a fresh install, will no modifications to anything, and it boots and runs as a clean install (asks me for email login and assumes I am a new user). Letsencrypt sets up the cert, everything seems good. I just have it sitting spun down as I do not want my users logging into it.

I have my env file from the old server, a tarred up version of the pgdata folder. I am afraid that database folder may have something wrong with it from trying to get the old server to do a migration and get it to work over there, but since app would never run from the containers I am not sure if the migration ever did anything (I do not believe it did). I also have a pgdump from prior to any issues when the old server was running fine (November 20th). I also have a tar of the uploads folder sitting on the new server ready to go.

I have the old server available and running on a different IP so I can grab anything from there that might be needed (and I could possibly try to local builds over there and see if I can get it to run, but at this point I thought it would be better to get the 18.04 server running).

I thought this would be the best state to start from to try to restore the old site. Let me know how you think it is best to proceed.

Thanks again for your help.

RG

Rob Guthrie Sat 23 Nov 2019

Right, the issue I'm thinking we need to focus in on is the migrations, right? You're saying your system works up until: 20180416223444 UpdateMeetingScores

That migration is only to adjust the values on timepolls, which we can do later if necessary anyway.

Reading on https://stackoverflow.com/questions/8810732/how-do-you-skip-failed-migrations-rake-dbmigrate I see that you can insert a record into the migrations table to say that it's already been run, essentially skipping the migration.

So are you able to use the open psql console command from the readme to run

insert into schema_migrations (version) values ('20180416223444');

and see if that does the trick?

MM

Michael McKinsey Sun 24 Nov 2019

I will give that a shot. The problem for me is that while I have quite a bit of experience at using databases from the CLI in a normal environment, I am often confused by the process of spinning up the docker containers and using commands from inside them to take action and when and why one chooses to do that in some situations.

Can you tell me the method you would use to get into the console? Do we do this just from the CLI of the host machine OS or is there some way I am spinning up a docker container and accessing the psql command line from there? I do not even have the psql tools installed on the host so I would need to do that if we are talking about manipulating the database from there, or else can you give me at least a starting point to build the command that would open a psql terminal interactively to my CLI with the docker container?

RG

Rob Guthrie Sun 24 Nov 2019

this uses psq in the container. from the loomip-deploy readme:

docker exec -ti loomio-db su - postgres -c 'psql loomio_production'

RG

Rob Guthrie Mon 25 Nov 2019

Hi, just wondering how you got on. You're so close now!

RG

Rob Guthrie Mon 25 Nov 2019

@Michael McKinsey I've just found a likely reason for the illegal instruction bug. The next version of the docker image should work for you.

MM

Michael McKinsey Mon 25 Nov 2019

@Rob Guthrie

Okay, this was quite an adventure. It looks like I have it all working now. The manual insert into the migrate was needed in order to finish the migration, and I ran app and worker as build instead of in the images. I ended up using the actual pgdata tar of the old database data rather than importing the backups.

It looks like everything is working as it should, I was in the middle of a rather time sensitive proposal with a group of people, so we dealt with emails and I am trying to get them to use loomio now, we shall see shortly if it is actually working correctly, but so far so good.

Thank you for all the help getting this problem solved, not sure why my database was broken so that it could not finish the migrate. I imagine that would have stopped me even if the new images had not been crashing.

I will try to keep my install more current, I certainly have learned more about how Loomio works on the internals than I ever wanted to know. The version 2 is different enough that at first I thought something was broken, but it seems to be working and looking at the loomio site, it is just a different layout in the new version. Looks good and the new features I was trying to get (change the invitation link, and remove a member from a group) seem to be working as hoped.

Your support and help with this has really been phenomenal. I don't know enough about the structure of the loomio team to know who you are, or why you were helping me, but whatever made that happen I am really grateful for the help and there is little/no chance I would have been able to save my install and keep using loomio without you helping me through this problem.

RG

Rob Guthrie Mon 25 Nov 2019

I'm the lead developer and designer on Loomio. There is one other part time developer who helps keep me sane. It's a 4 employee company.

There are not a lot of people running self hosted Loomio's, and I want to be sure to support those brave people who do attempt it, where I can.

We only managed to release Loomio 2 because of our user community reporting bugs and giving feedback during a long public beta period.

Thanks for using Loomio, and reporting your journey, you've improved things for the next person.