Loomio
Thu 21 Nov 2019 4:56PM

Loomio not running after update

MM Michael McKinsey Public Seen by 47

Hi,

I have been running a loomio install for quite some time (years). It has been a long time since I did an update.

I ran the following:

cd loomio-deploy/

docker-compose down

docker-compose pull

docker-compose run app rake db:migrate

ERROR: No such service: app

I upgraded docker, and did a general update on my system, which is Ubuntu 14.04. I had to do a kernel update to get docker running after the update, but that seems to be working fine.

When I use:

docker-compose up -d

I end up with a:

502 Bad Gateway


nginx/1.17.5

When looking at:

docker-compose logs -f

I am getting a massive amount of scrolling text, it is difficult to pull out what might be important, it just keeps cycling through and shooting out massive lists of text.

I will include some snipits from the log in the hopes it may help. Any advice as to how to proceed would be great.

Thanks!

RG

Robert Guthrie Sat 23 Nov 2019 5:00AM

Keep those backup files safe and we can do it. The migrations might need hand holding but I know it’s possible.

MM

Michael McKinsey Sat 23 Nov 2019 5:21AM

@Rob Guthrie

So here is an issue - the command that is failing is:

docker-compose run app rake db:setup

I may not be following the logic completely, but I do not think this is even spinning up (sorry I do not have a lot of experience with what terms to use for docker components) worker, it spins up loomio-redis and loomio-db and then it gets an illegal instruction at

root@loomio:~/loomio-deploy# docker-compose run app rake db:setup

Creating loomio-redis ... done

Creating loomio-db ... done

/usr/local/bundle/gems/ffi-1.11.1/lib/ffi/library.rb:112: [BUG] Illegal instruction at 0x00007fd0f83b97a2

ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]

-- Control frame information -----------------------------------------------

c:0084 p:---- s:0462 e:000461 CFUNC :open

c:0083 p:0022 s:0456 e:000455 BLOCK /usr/local/bundle/gems/ffi-1.11.1/lib/ffi/library.rb:112 [FINISH]

If I am understanding what you asked me to do above, I am replacing the container of loomio-worker with some local build command, but I am erroring out before we ever run worker. Were you expecting me to change all the containers with local build commands? Apologies this is all a bit new to me. I realize I may be missing something simple.

Am I following? And does it make sense that the above change made no difference? I get the same thing when I try to build the DB.

It leaves loomio-redis and loomio-db up, I need to close them down.

[NOTE]

You may have encountered a bug in the Ruby interpreter or extension libraries.

Bug reports are welcome.

For details: https://www.ruby-lang.org/bugreport.html

SEGV received in ILL handler

root@loomio:~/loomio-deploy# docker-compose down

Stopping loomio-deployapprun_1 ... done

Stopping loomio-db ... done

Stopping loomio-redis ... done

Removing loomio-deployapprun_1 ... done

Removing loomio-db ... done

Removing loomio-redis ... done

RG

Robert Guthrie Sat 23 Nov 2019 5:46AM

Yes. Swap both Loomio containers to build and leave others as is. If you’re getting the same error as the images that’s interesting but sad

MM

Michael McKinsey Sat 23 Nov 2019 7:30AM

@Rob Guthrie

Okay, so I am making progress. I changed app and worker to build, and that seemed to be enough to get everything to start up. I tried with just app, but worker was crashing, so I changed that back to build. I can now get a login screen on the app when I start it.

Prior to starting it I tared up the pgdata folder and moved it over to the new server and replaced the pgdata folder, and then ran the rake migrate command which seemed to work (had never worked before) and then copied the env file over from the old server. I tarred up and replaced the upload folder.

I was getting errors and would not start with the new env file, and I looked at it and decided I should copy the DeviseSecret and SecretCookieToken entries to the old env file from the new one and that worked.

Now I get a login screen, which allows me to use google to login as I could on the old server, but after I login I get a "Something went wrong" loomio screen.

Feel like I am getting close.

On closer examination I do not believe the app rake migrate command is actually working. I deleted pgdata and reverted back to the old database several times now. When I run the commmand:

docker-compose run app rake db:migrate

it starts to do the migration:

Migrating to AddAnnouncements (20171114224912)
== 20171114224912 AddAnnouncements: migrating =================================
-- create_table(:announcements, {:id=>:integer})
   -> 2.5373s
== 20171114224912 AddAnnouncements: migrated (2.5375s) ========================

Migrating to AddAuthorToAnnouncement (20171115232230)
== 20171115232230 AddAuthorToAnnouncement: migrating ==========================
-- add_column(:announcements, :author_id, :integer, {:default=>false, :index=>true})
   -> 2.3210s
== 20171115232230 AddAuthorToAnnouncement: migrated (2.3212s) =================

Then it errors out:

Migrating to UpdateMeetingScores (20180416223444)

== 20180416223444 UpdateMeetingScores: migrating ==============================

rake aborted!

StandardError: An error has occurred, this and all later migrations canceled:

unknown attribute 'score_counts' for PollOption.

/usr/local/bundle/gems/activemodel-5.2.3/lib/activemodel/attributeassignment.rb:53:in `assignattribute'

When I just spin up the server after this partial migration, the server will run, and it lets me login with google (the email login button doesn't work) and I get the "something went wrong" screen.

A couple of times going through various slightly different order of operations I have gotten it to log me in and then it thinks I am a new user (so obviously the database is not getting imported).

Sorry for the mess, but I have certainly gotten much closer now, though we still seem to be a bit off.

RG

Robert Guthrie Sat 23 Nov 2019 7:56AM

Ok great!! Congratulations!! Almost there. I’ll pick this up tomorrow

MM

Michael McKinsey Sat 23 Nov 2019 2:59PM

@Rob Guthrie

Thanks. I have been trying various things to get this to work, might need to start fresh on the new install but I believe I have been following the logic of what is happening so it should be okay though I have made many modifications to env files and deleted and restored pgdata many times and have been running it in various states trying to figure out what is going on. Often I end up in a state where it looks like the database server is either not starting or it cannot be reached by the other containers.

I have daily backups of the database and they are even sent off the old server, so having the data from before any of this happened is no problem (I have a cron script running pgdump every day and backing up the uploads folder and mailing that all off of the server as well as copying it to a remote FTP).

MM

Michael McKinsey Sat 23 Nov 2019 3:31PM

@Rob Guthrie

I renamed the loomio-deploy on the new server and started fresh - I can get everything to run according to the install (seems to all be working - could be some failing going on but I don't see anything that jumps out to me in the logs) by replacing the containers for app and worker with a local build.

How to import the database from the old server and get it to do a migration etc., and the exact way to bring in what I need from the other env file (primarily the google authentication and my email settings) - are left to do. Also I assume I need to restore my uploads folder.

I have it set up and ready to run as a fresh install, will no modifications to anything, and it boots and runs as a clean install (asks me for email login and assumes I am a new user). Letsencrypt sets up the cert, everything seems good. I just have it sitting spun down as I do not want my users logging into it.

I have my env file from the old server, a tarred up version of the pgdata folder. I am afraid that database folder may have something wrong with it from trying to get the old server to do a migration and get it to work over there, but since app would never run from the containers I am not sure if the migration ever did anything (I do not believe it did). I also have a pgdump from prior to any issues when the old server was running fine (November 20th). I also have a tar of the uploads folder sitting on the new server ready to go.

I have the old server available and running on a different IP so I can grab anything from there that might be needed (and I could possibly try to local builds over there and see if I can get it to run, but at this point I thought it would be better to get the 18.04 server running).

I thought this would be the best state to start from to try to restore the old site. Let me know how you think it is best to proceed.

Thanks again for your help.

RG

Robert Guthrie Sat 23 Nov 2019 9:40PM

Right, the issue I'm thinking we need to focus in on is the migrations, right? You're saying your system works up until: 20180416223444 UpdateMeetingScores

That migration is only to adjust the values on timepolls, which we can do later if necessary anyway.

Reading on https://stackoverflow.com/questions/8810732/how-do-you-skip-failed-migrations-rake-dbmigrate I see that you can insert a record into the migrations table to say that it's already been run, essentially skipping the migration.

So are you able to use the open psql console command from the readme to run

insert into schema_migrations (version) values ('20180416223444');

and see if that does the trick?

MM

Michael McKinsey Sun 24 Nov 2019 2:45PM

I will give that a shot. The problem for me is that while I have quite a bit of experience at using databases from the CLI in a normal environment, I am often confused by the process of spinning up the docker containers and using commands from inside them to take action and when and why one chooses to do that in some situations.

Can you tell me the method you would use to get into the console? Do we do this just from the CLI of the host machine OS or is there some way I am spinning up a docker container and accessing the psql command line from there? I do not even have the psql tools installed on the host so I would need to do that if we are talking about manipulating the database from there, or else can you give me at least a starting point to build the command that would open a psql terminal interactively to my CLI with the docker container?

RG

Robert Guthrie Sun 24 Nov 2019 5:44PM

this uses psq in the container. from the loomip-deploy readme:

docker exec -ti loomio-db su - postgres -c 'psql loomio_production'

RG

Robert Guthrie Mon 25 Nov 2019 3:24AM

Hi, just wondering how you got on. You're so close now!

RG

Robert Guthrie Mon 25 Nov 2019 7:32AM

@Michael McKinsey I've just found a likely reason for the illegal instruction bug. The next version of the docker image should work for you.

MM

Michael McKinsey Mon 25 Nov 2019 2:39PM

@Rob Guthrie

Okay, this was quite an adventure. It looks like I have it all working now. The manual insert into the migrate was needed in order to finish the migration, and I ran app and worker as build instead of in the images. I ended up using the actual pgdata tar of the old database data rather than importing the backups.

It looks like everything is working as it should, I was in the middle of a rather time sensitive proposal with a group of people, so we dealt with emails and I am trying to get them to use loomio now, we shall see shortly if it is actually working correctly, but so far so good.

Thank you for all the help getting this problem solved, not sure why my database was broken so that it could not finish the migrate. I imagine that would have stopped me even if the new images had not been crashing.

I will try to keep my install more current, I certainly have learned more about how Loomio works on the internals than I ever wanted to know. The version 2 is different enough that at first I thought something was broken, but it seems to be working and looking at the loomio site, it is just a different layout in the new version. Looks good and the new features I was trying to get (change the invitation link, and remove a member from a group) seem to be working as hoped.

Your support and help with this has really been phenomenal. I don't know enough about the structure of the loomio team to know who you are, or why you were helping me, but whatever made that happen I am really grateful for the help and there is little/no chance I would have been able to save my install and keep using loomio without you helping me through this problem.

RG

Robert Guthrie Mon 25 Nov 2019 6:33PM

I'm the lead developer and designer on Loomio. There is one other part time developer who helps keep me sane. It's a 4 employee company.

There are not a lot of people running self hosted Loomio's, and I want to be sure to support those brave people who do attempt it, where I can.

We only managed to release Loomio 2 because of our user community reporting bugs and giving feedback during a long public beta period.

Thanks for using Loomio, and reporting your journey, you've improved things for the next person.