Struggling with Docker, is there another way?
I have been trying on and off for the last week or so to install Loomio using the docker method, I've put in quite a few hours, and I am finding it frustrating.
I'm an experienced developer and web server admin but I have never used Docker before. When things go wrong I usually know how to start debugging them, but with Docker there is a whole layer in between which I don't understand, and I find it quite hard to accept that I will have to learn about it just to install Loomio. I know it's a useful thing to learn, but I was hoping to get Loomio up and running quickly so as to start using it! A whole array of docker containers linked together is no mean feat to understand.
I get the impression that @robertguthrie is the only person who knows the loomio docker system inside out and is therefore swamped with requests for help, but that you are a busy chap, like the rest of us, and cannot be expected to be there at the beck and call of anyone who attempts installation.
If there was a way of installing it that did not require this extra layer, perhaps people would be able to help each other more, and get help from Stack Overflow and the like.
It's also difficult to make configuration changes - I tried to run it on a different port, but failed. Which means I have to use a whole VPS for this, rather than running it on the same server as another bunch of websites. In fact being forced to use nginx makes this difficult anyway; I have a bunch of apache vhosts and was hoping this would just be another one.
Also, the docker container uses a lot of RAM, which is what drives up the cost of a VPS. So it is costing a lot to run this.
I'm sorry, I realise this probably comes across as a rant, but I think it is important for free software projects to be relatively easy to install, and not to rely on one beleagured person as the font of all knowledge ...

Robert Guthrie Fri 12 Apr 2019 8:26PM
Hi @naomir , totally appreciate your post, thanks for writing it.
I'll just add my reasoning first, then address your questions.
I've chosen the Docker path because it gives you a secure, full featured loomio install in the least number of steps. That is: You get SSL/HTTPS, inbound email, simple DNS settings, background jobs and crontab, redis and postgresql, and even a simple backup system.
Our crowdfunding campaign in 2014? promised a "one-click" install process - which is actually kind of impossible, but this is the closest I've been able to come to providing a simple to setup processs for hosting your own Loomio install.
Personally I find that docker keeps the level of complexity down.
If you're trying to run this alongside other services, or in a custom way, then it will get much less straightforward - that's just how it is. If you don't use Docker, then you lose the ability to easily update your system if I change a requirement. (like when I added redis to the stack 4 months ago, or when we moved from FAYE to actioncable last year).
I try to spend only a small amount of my time on the devops side of Loomio, becuase I want to spend as much time as possible writing good features and UX.
I think that running your own instance is always going to be something that requires a high level of server/linux/database experience, if you want to do it really well.. I've done my best to make it easy to get started, and also keep loomio.org running so people don't have to do this at all.
That said, I really like it when people such as yourself come along and actually use this work I've done. Sometimes I wonder if anyone is using it - because most people don't communicate with me... and we very rarely get any contributions, which I understand - because it is hard to contribute to a complex project.
I'd be happy to add/contribute to documentation about running your own production server. Have you seen the guide on setting up your own development environment? That covers quite a lot of this. (and may also need updating)

Klaus Kobald Wed 22 May 2019 7:01PM
The docker guide does not work for a new blank server, because the command docker exec -ti loomio-db su - postgres -c 'createdb loomio_production' fails with Error: No such container: loomio-db
This is obvious, because no container is running at that time. I tried with the docker-compose up, but it leads to a complete mess I think. At the end I only get "500 Server Error" with no useful log information.

Robert Guthrie Wed 22 May 2019 7:14PM
Ok, I'll run through a setup from blank again and update the guide. If I may ask, where do you plan to use Loomio?
Brendan Butler Wed 22 May 2019 7:20PM
I am in the same situation. I appreciate you taking the time out of your day to do this :) Personally, I plan to use Loomio for a project I am helping organize development for named Music Stash.

Klaus Kobald Wed 22 May 2019 7:24PM
We would like to use it for the austrian burningman community. :heart:

Robert Guthrie Wed 22 May 2019 7:57PM
Ok @klauskobald and @brendanbutler please try this:
docker-compose up -d db
docker exec -ti loomio-db su - postgres -c 'createdb loomio_production'
docker-compose run app rake db:setup
Brendan Butler Wed 22 May 2019 8:32PM
@robertguthrie
I've tested with these new commands in replace of database integration; it worked flawlessly up until the docker-compose up -d after the Cron tab is edited. No errors bringing it up, until the very last container (let's encrypt):
And I get 502 bad gateway errors when browsing to the domain

Robert Guthrie Wed 22 May 2019 8:34PM
It can take upto 5 minutes for the app to boot. You will see bad gateway until then. I'll add that to the readme. You can see it boot with the logs command.
Brendan Butler Wed 22 May 2019 8:35PM
Alright thanks so much, I'll update you in a few minutes then :)Do I need to worry about that letsencrypt error?

Robert Guthrie Wed 22 May 2019 8:35PM
Also you need to be sure that port 80 is open and your dns is correct for letsencrypt.

Robert Guthrie Wed 22 May 2019 8:36PM
Boarding a plane. I'll check back later today
Brendan Butler Wed 22 May 2019 8:37PM
Port 80 and 443 are both open; using a Google VPS and I had selected both to be open when creating it. DNS is also set up (other than MX, I was going to set that up when I had it all working; is that maybe the problem?)
Safe travels!!!
Brendan Butler Wed 22 May 2019 9:00PM
Alright, still getting the 503 error and checked the logs and saw a bunch of various errors. Most are regarding the let's encrypt, but I saw some that weren't related. I am on the way home now, once I get home I will try copy pasting the entire error log into a Pastebin
Brendan Butler Wed 22 May 2019 10:03PM
Got home. The logs are VERY large,
I am unsure what in here is relating to my problem; though I am assuming its not all those letsencrypt errors as that would only deal with the SSL (I should still be able to connect to http, where Chrome would mention its insecure).
The weird part I find is that LetsEncrypt did seem to work; the page has the secure lock and it says valid from today on:

Klaus Kobald Wed 22 May 2019 10:50PM
YES!
And it helps watching the log, because it takes some while for building!
So I´m getting the login page and will continue tomorrow! Greetings from Austria! :clap:

Robert Guthrie Wed 22 May 2019 11:38PM
restarting with docker-compose down; docker-compose up -d
can sort out let's encrypt issues sometimes. Also docker-compose logs letsencrypt
will show just logs from that container.
Brendan Butler Thu 23 May 2019 1:30AM
Restarting the docker-compose seemed to fix the letsencrypt; and it all got up with no errors. I have been watching the logs and I am seeing no errors, but it's been roughly 2 hours and I am getting 503 when trying to connect.
loomio-worker | [Worker(host:4c6775a2dec8 pid:7)] 6 jobs processed at 0.0526 j/s, 0 failed
loomio-worker | 2019-05-23T01:11:32+0000: [Worker(host:4c6775a2dec8 pid:7)] 6 jobs processed at 0.0526 j/s, 0 failed
loomio-nginx | nginx.1 | 2019/05/23 01:28:50 [error] 84#84: *20 connect() failed (111: Connection refused) while connecting to upstream, client: 72.137.176.178, server: test......xyz, request: "GET / HTTP/2.0", upstream: "http://172.17.0.5
:3000/", host: "test......xyz"
loomio-nginx | nginx.1 | test.....xyz 72.137.176.178 - - [23/May/2019:01:28:50 +0000] "GET / HTTP/2.0" 502 575 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36"
loomio-nginx | nginx.1 | test......xyz 72.137.176.178 - - [23/May/2019:01:28:50 +0000] "GET /favicon.ico HTTP/2.0" 502 575 "https://test.....xyz/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/
74.0.3729.157 Safari/537.36"
loomio-nginx | nginx.1 | 2019/05/23 01:28:50 [warn] 84#84: *20 upstream server temporarily disabled while connecting to upstream, client: 72.137.176.178, server: test....xyz, request: "GET / HTTP/2.0", upstream: "http://172.17.0.5:3000/"
, host: "test......xyz"

Robert Guthrie Thu 23 May 2019 8:48AM
It would be a good idea to run docker-compose logs app
to see just the rails logs - that's likely where the issue is.
Brendan Butler Thu 23 May 2019 11:00AM
After waking up and checking the website (I left the serverto do it's thing) I found this:https://i.godhub.xyz/05-23-2019-rLNLLRwiv1.png
Maybe this has to do with my cpu, it was near its limit when I started to install this. Maybe when I get my new server into my datacenter in the up coming days I can try again on a more powerful machine.

Robert Guthrie Thu 23 May 2019 8:05PM
Interesting.
CPU power is unlikely to be the issue - RAM possibly. the CSS compilation is a first request only issue too, so after it's compiled it will be fine.
You can set the ENV variable RACK_TIMEOUT_SERVICE_TIMEOUT=30 or more to give the server more time to do work before killing the request.
Yesterday we merged the new client into the master branch, and over the next couple of weeks we'll fine tune it so it's ready for a public beta then replacing the current client. One of the many advantages of the new client are that it can be compiled at docker image build time (before you docker-compose pull the image), so startup time should be only a few seconds.

Robert Guthrie Fri 12 Apr 2019 8:32PM
These days I never run any server software without docker. It's the most important server skill I have.

Robert Guthrie Fri 12 Apr 2019 8:53PM
Thanks for taking the time to work on Loomio and tell me about it, I really do want to support you however I can - which is often a little slowly... sorry.
Naomi Fri 12 Apr 2019 9:07PM
Thanks @robertguthrie, I appreciate you taking the time to read my post and give it such a considered answer. I think in a way it comes down to different mindsets. Like you say, it keeps the level of complexity down, but only if everything goes to plan - and then if it doesn't, it suddenly becomes very complicated, and the usual Linux knowledge isn't enough to fix it. I find it's quite unusual for things to "just work" for me - I often end up delving into them for one reason or another - so the inability to (easily) do that is stressful. It's the same way I feel if I ever go near an IPad - I keep hearing how easy they are and how they just work, but they make no sense to me, I'm like "just give me a shell and no-one gets hurt". I guess it's a sense of not being in control.
However, I am already gaining knowledge of Docker from this. which is indeed useful and becoming more and more necessary as it gains popularity.
I will have a look at the development site docs you mention, and may have stuff to add, although it might be another few days before I get a chance to work on it

Robert Guthrie Fri 12 Apr 2019 10:56PM
I would recommend that you try putting the shared server requirement aside, and try setting up loomio on a blank VPS as described, it should only take 30 minutes to an hour.
I totally respect the approach of taking a different path though, that's how I learned everything I know - including the value of just sticking to the path :)
Another really kind and enthusiastic person is wanting to create and manage a set of Loomio debian packages. Is that something that you think would be helpful to you? https://github.com/loomio/loomio/issues/5283
Naomi · Fri 12 Apr 2019 12:43PM
P.S. I worked out how to run it on a different port - I turned off apache, which was competing for port 80 of course, and ran
docker-compose up
which caused the SSL certificate to be generated (for which it must be running on port 80). Then I went intodocker-compose.yml
and "80:80" to "81:80" and "443:443" to "444:443" , did "docker-compose down" and then "up -d" again and lo and behold I could access loomio on port 444 and (after restarting apache) could access my other sites on 443! Happy days. I'm having an issue with email now though, which I'll start a new thread for. I attempting to debug now with the help of a more knowledgeable person, and will post any info that might help others in that thread.@robertguthrie if I get through this I would like to compile what I've learnt into a troubleshooting guide - would you be interested in that being added to your repo?