Installation issue - blank page - nginx upstream error
Hi!
I've installed the latest version of loomio-deploy on a fresh Digital Ocean Ubuntu 17.04 x64 droplet, following all steps in the guide.
The installation completes, but I can't access the page. Firefox renders a blank page, with no content, and chrome gives me an http "421 824245dfe7f9 You talk too soon" message only.
My log for a single request:
loomio_1 | Started GET "/" for x.x.x.x at 2017-10-30 16:01:22 +0000
loomio_1 | Processing by RootController#index as HTML
loomio_1 | Redirected to https://loomio.appning.org/dashboard
loomio_1 | Completed 302 Found in 293ms (ActiveRecord: 0.0ms)
nginx_1 | nginx.1 | loomio.appning.org x.x.x.x - - [30/Oct/2017:16:01:23 +0000] "GET / HTTP/2.0" 302 112 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:56.0) Gecko/20100101 Firefox/56.0"
nginx_1 | nginx.1 | 2017/10/30 16:01:23 [error] 52#52: *23 connect() failed (111: Connection refused) while connecting to upstream, client: x.x.x.x, server: loomio.appning.org, request: "GET /dashboard HTTP/2.0", upstream: "http://172.17.0.5:3000/dashboard", host: "loomio.appning.org"
nginx_1 | nginx.1 | 2017/10/30 16:01:23 [warn] 52#52: *23 upstream server temporarily disabled while connecting to upstream, client: x.x.x.x, server: loomio.appning.org, request: "GET /dashboard HTTP/2.0", upstream: "http://172.17.0.5:3000/dashboard", host: "loomio.appning.org"
nginx_1 | nginx.1 | 2017/10/30 16:01:23 [error] 52#52: *23 upstream sent no valid HTTP/1.0 header while reading response header from upstream, client: x.x.x.x, server: loomio.appning.org, request: "GET /dashboard HTTP/2.0", upstream: "http://172.17.0.6:25/dashboard", host: "loomio.appning.org"
nginx_1 | nginx.1 | loomio.appning.org x.x.x.x - - [30/Oct/2017:16:01:23 +0000] "GET /dashboard HTTP/2.0" 009 36 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:56.0) Gecko/20100101 Firefox/56.0"
This is the second time i deploy loomio on this instance - the first time I thought I had screwed up, so I rebuilt the droplet and started again.
Any help? I have some urgency in solving this issue and no ideas on what to do. Thanks!
Bump!
I setup an install the other day and it worked well. I don't see what the problem could be.
but this seems wrong:
http://172.17.0.6:25/dashboard
why is it referencing port 25? it should be loomio:3000
it seems like you've edited your docker-compose.yml a bit heavily. can you share that?
I get the exact same error on a fresh install of Ubuntu 14.04. I followed the instructions for loomio-deploy exactly, except for setting up SMTP, which I assume shouldn't be an issue. I watched the resource usage, and it's thrashing the server, so I believe the issue is caused by a timeout somewhere.
Think a server upgrade would fix things?
Can you share your env and docker-compose files please @williamharader and @josesimoes?
Sorry for the latency. Sure, here they are:
docker-compose.yml: https://pastebin.com/gahneye8
env: https://pastebin.com/18S5j6Cx
I've switched sensitive data with XXX. Also, I've only edited env, not docker-compose.
Thank you for helping!
just to check: Have you run
docker-compose down; docker-compose pull; docker-compose up -d
to ensure you're running latest images?
The whole problem started when I updated Loomio. From there, I've cleaned all docker images and containers and re-installed (again). Same problem. Then I re-built the droplet, and re-installed Loomio. No good. So the problem is very probably in my env file, but... Where?
do you have javascript console errors?
Here's my env file: https://pastebin.com/CttuG7dX
And here's docker-compose.yml: https://pastebin.com/QGyzkrW3
Here's the error I get through docker when trying to access the site:
nginx_1 | nginx.1 | 2017/11/08 01:50:44 [error] 52#52: *1 upstream sent no valid HTTP/1.0 header while reading response header from upstream, client: {my ip}, server: server.website, request: "GET / HTTP/1.1", upstream: "http://172.17.0.6:25/", host: "server.website"
And on the browser's side it displays "421 a69af862b7c0 You talk too soon" after a little over a minute of waiting.
Hrrmm.. Sorry to keep saying this, but are you sure you've pulled a new copy, today even? I did see that bug for a day while I was working with SSL, but I've since updated things.
And do a git pull origin master too I've made a minor change to the docker-compose.yml
I'll try again today or tomorrow, at most. I'll let you know the results. Thanks!
All working! I did exactly what you asked, but this time it worked. Maybe the images I downloaded when I had the problem had the "421 You talk too soon" bug?
I noticed that I had overwritten my smtp setting when I updated loomio, but even lacking those all went well. I've restored the smtp settings after bringing loomio up and everything is running smoothly.
I still have the upstream errors on the nginx container, but apparently they have no effect. Related to this github issue, I think: https://github.com/loomio/loomio-deploy/issues/13
Thanks @robertguthrie , I don't what I'd do without your continued support. Big cheers!
I reinstalled everything with the latest version and it's working now! Thanks @robertguthrie
I see these messages from Docker repeated every 10 seconds, but they don't see to be affecting the site:
db_1 | ERROR: database "loomio_production" already exists
db_1 | STATEMENT: CREATE DATABASE "loomio_production" ENCODING = 'utf8'
db_1 | WARNING: hash indexes are not WAL-logged and their use is discouraged
letsencrypt_1 | 2017/11/08 19:58:50 Received event die for container 0266b70b5a22
nginx_1 | dockergen.1 | 2017/11/08 19:58:50 Received event die for container 0266b70b5a22
nginx_1 | dockergen.1 | 2017/11/08 19:58:50 Generated '/etc/nginx/conf.d/default.conf' from 8 containers
nginx_1 | dockergen.1 | 2017/11/08 19:58:50 Running 'nginx -s reload'
nginx_1 | dockergen.1 | 2017/11/08 19:58:50 Error running notify command: nginx -s reload, exit status 1
letsencrypt_1 | 2017/11/08 19:58:51 Received event start for container 0266b70b5a22
nginx_1 | dockergen.1 | 2017/11/08 19:58:51 Received event start for container 0266b70b5a22
nginx_1 | dockergen.1 | 2017/11/08 19:58:51 Generated '/etc/nginx/conf.d/default.conf' from 8 containers
nginx_1 | dockergen.1 | 2017/11/08 19:58:51 Running 'nginx -s reload'
Sorry about all this hassle. Glad we're all sorted now.
Hey, I'm also getting this error now and then. I'm not sure what it's related to, but it keeps appearing in the logs:
nginx_1 | nginx.1 | 2018/01/21 11:26:15 [error] 74#74: *113 upstream prematurely closed connection while reading response header from upstream, client: [MY LOCAL IP], server: faye.talk.theborderland.se, request: "POST /faye HTTP/2.0", upstream: "http://[MY SERVER IP]:3000/faye", host: "faye.talk.theborderland.se", referrer: "https://talk.theborderland.se/dashboard"
I'm using the latest Stable release. Any ideas?
@hugi does this cause problems for users? I have not seen this error before, but it's HTTP/2.0 which seems a bit modern for our stack, I wonder if it's just a new client trying http2 to see if it's supported.
I’m experiencing a few different issues that might perhaps be related to problems with Faye.
- Some users have reported issues with not being able to load certain threads while logged in, that load if they read them while not logged in
- Sometimes text that has been posted by a user most recently in a comment or post stays behind and appears in the input box when trying to write a new comment or post.
- Some users have reported not being able to get rid of Unread markers for certain threads.
A previous request rendered the following (different) output - note the upstream error:
loomio_1 | Started GET "/" for x.x.x.x at 2017-10-30 16:00:47 +0000
loomio_1 | Processing by RootController#index as HTML
loomio_1 | Redirected to https://loomio.appning.org/dashboard
loomio_1 | Completed 302 Found in 56ms (ActiveRecord: 0.0ms)
nginx_1 | nginx.1 | loomio.appning.org x.x.x.x - - [30/Oct/2017:16:00:47 +0000] "GET / HTTP/2.0" 302 112 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:56.0) Gecko/20100101 Firefox/56.0"
nginx_1 | nginx.1 | 2017/10/30 16:00:47 [error] 52#52: *21 upstream sent no valid HTTP/1.0 header while reading response header from upstream, client: x.x.x.x, server: loomio.appning.org, request: "GET /dashboard HTTP/2.0", upstream: "http://172.17.0.6:25/dashboard", host: "loomio.appning.org"
nginx_1 | nginx.1 | loomio.appning.org x.x.x.x - - [30/Oct/2017:16:00:47 +0000] "GET /dashboard HTTP/2.0" 009 36 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:56.0) Gecko/20100101 Firefox/56.0"