Loomio
Tue 24 Jan 2023 9:16PM

Correct configuration for non-docker nginx setup

YG Yochai Gal Public Seen by 343

Hello! I am trying to integrate loomio in an already complex setup (ubuntu 20.04 server). I am running an nginx server (not in a container) and would simply like to point nginx to the local port that loomio uses, as a reverse proxy.

My understanding is that I should only need to remove the nginx and lets-encrypt bits, forward the port (I use 3000 for something else so I forward to 4000) and then setup the reverse proxy. I also need to disable FORCE_SSL in the .env file. Correct?

Even without nginx, I should be able to access the service directly at http://myurl:4000. When I do this, the logs tell me: "Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma." Meanwhile when I got to the IP+ raw URL I get: ERR_SSL_PROTOCOL_ERROR; this happens even with my nginx server disabled.

I'm not super familiar with ruby however, so that's as far as I've gotten. I'd appreciate any help you might give!

This is my docker-compose.yml:

version: '3'

services:
  app:
    container_name: loomio-app  
    image: ${LOOMIO_CONTAINER_IMAGE}:${LOOMIO_CONTAINER_TAG}
    restart: unless-stopped
    ports:
      - 4000:3000
    expose:
      - 3000
    env_file: ./.env
    environment:
      - VIRTUAL_HOST=${CANONICAL_HOST}
      - LETSENCRYPT_HOST=${CANONICAL_HOST}
    volumes:
      - ./uploads:/loomio/public/system
      - ./storage:/loomio/storage
      - ./files:/loomio/public/files
      - ./plugins:/loomio/plugins/docker
      - ./import:/import
      - ./tmp:/loomio/tmp
    networks:
      - main
    depends_on:
      - db
      - redis

  worker:
    container_name: loomio-worker
    image: ${LOOMIO_CONTAINER_IMAGE}:${LOOMIO_CONTAINER_TAG}
    restart: always
    networks:
      - main
    env_file: ./.env
    environment:
      - TASK=worker
    volumes:
      - ./uploads:/loomio/public/system
      - ./storage:/loomio/storage
      - ./files:/loomio/public/files
      - ./plugins:/loomio/plugins/docker
      - ./tmp:/loomio/tmp

  db:
    container_name: loomio-db
    image: postgres:${POSTGRES_CONTAINER_TAG}
    restart: unless-stopped
    networks:
      - main
    env_file: ./.env
    volumes:
      - ./pgdata:/pgdata
      - ./pgdumps:/pgdumps
    environment:
     - PGDATA=/pgdata

  redis:
    container_name: loomio-redis
    image: redis:5.0
    restart: unless-stopped
    networks:
      - main

  mailin:
    container_name: loomio-mailin
    image: ${MAILIN_CONTAINER_IMAGE}:${MAILIN_CONTAINER_TAG}
    restart: unless-stopped
    networks:
      - main
    ports:
      - "25:25"
    environment:
      - WEBHOOK_URL=http://app:3000/email_processor/

  channels:
    container_name: loomio-channels
    image: loomio/loomio_channel_server
    restart: unless-stopped
    networks:
      - main
    env_file: ./.env
    depends_on:
      - redis
    environment:
      - VIRTUAL_HOST=channels.${CANONICAL_HOST}
      - LETSENCRYPT_HOST=channels.${CANONICAL_HOST}

  pgbackups:
    image: prodrigestivill/postgres-backup-local
    restart: always
    user: postgres:postgres
    volumes:
      - ./pgdumps:/backups
    links:
      - db
    depends_on:
      - db
    environment:
      - POSTGRES_HOST=db
      - POSTGRES_DB=loomio_production
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=password
      - POSTGRES_EXTRA_OPTS=-Z6 --schema=public --blobs
      - SCHEDULE=@daily
      - BACKUP_KEEP_DAYS=7
      - BACKUP_KEEP_WEEKS=1
      - BACKUP_KEEP_MONTHS=1
      - HEALTHCHECK_PORT=8080
networks:
  main:

volumes:
  certs:
  vhost:
  html:
  acme:

My .env file:

LOOMIO_CONTAINER_IMAGE=loomio/loomio
LOOMIO_CONTAINER_TAG=stable
MAILIN_CONTAINER_IMAGE=loomio/mailin-docker
MAILIN_CONTAINER_TAG=latest
POSTGRES_CONTAINER_TAG=12.2

# this is the hostname of your app eg: loomio.org
CANONICAL_HOST=loomio.mydomain.com

# the human name of the app (Default Loomio)
SITE_NAME=loomio.mydomain.com

# reply-to in email notifications
REPLY_HOSTNAME=loomio.mydomain.com


# channels
CHANNELS_URI=wss://channels.loomio.mydomain.com

# uncomment this if you want a default subdomain of www (eg: www.loomio.org)
# DEFAULT_SUBDOMAIN=www

# smtp settings
SUPPORT_EMAIL=[my_email]

SMTP_AUTH=plain
SMTP_DOMAIN=my.domain.com
SMTP_SERVER=smtp.server
SMTP_PORT=465
SMTP_USERNAME=[my_email]
SMTP_PASSWORD=[mypass]
SMTP_USE_SSL=1
# to disable SSL comment out line rather than changing to 0

# helper bot is the account which welcomes people to their groups.
[email protected]
RAILS_ENV=production

# Number of webserver processes and threads
# threads are per worker. See https://github.com/puma/puma
PUMA_WORKERS=2
MIN_THREADS=12
MAX_THREADS=12

# Force all connections to be https
# FORCE_SSL=1

# Enable rate limiting on group creation, other POST actions
USE_RACK_ATTACK=1
RACK_ATTACK_RATE_MULTPLIER=5
RACK_ATTACK_TIME_MULTPLIER=1

# Postgres
POSTGRES_PASSWORD=password
POSTGRES_DB=loomio_production
DATABASE_URL=postgresql://postgres:password@db/loomio_production

# Redis URL
REDIS_URL=redis://redis:6379/0

# attachment storage service
# local will keep attachments on the server's disk under ./storage
# for cloud storage (recommended) try amazon, digitalocean or s3_compatible

ACTIVE_STORAGE_SERVICE=local 

# stoage.yml for reference
# amazon:
#   service: S3
#   access_key_id: <%= ENV['AWS_ACCESS_KEY_ID'] %>
#   secret_access_key: <%= ENV['AWS_SECRET_ACCESS_KEY'] %>
#   bucket: <%= ENV['AWS_BUCKET'] %>
#   region: <%= ENV['AWS_REGION'] %>
#
# digitalocean:
#   service: S3
#   endpoint: <%= ENV['DO_ENDPOINT'] %>
#   access_key_id: <%= ENV['DO_ACCESS_KEY_ID'] %>
#   secret_access_key: <%= ENV['DO_SECRET_ACCESS_KEY'] %>
#   bucket: <%= ENV['DO_BUCKET'] %>
#   region: ignored
# 
# s3_compatible:
#   service: S3
#   endpoint: <%= ENV.fetch('STORAGE_ENDPOINT', '') %>
#   access_key_id: <%= ENV.fetch('STORAGE_ACCESS_KEY_ID', '') %>
#   secret_access_key: <%= ENV.fetch('STORAGE_SECRET_ACCESS_KEY', '') %>
#   region: <%= ENV.fetch('STORAGE_REGION', '') %>
#   bucket: <%= ENV.fetch('STORAGE_BUCKET_NAME', '') %>
#   force_path_style: <%= ENV.fetch('STORAGE_FORCE_PATH_STYLE', false) %>

# Send catch up email (missed yesterday) weekly
# EMAIL_CATCH_UP_WEEKLY=1

# subscribe on participation default for new users
# uncomment this to change "subscribe on participation" to be false for new users
# EMAIL_ON_PARTICIPATION_DEFAULT_FALSE=1

# Uncomment these to disable features
# FEATURES_DISABLE_CREATE_USER=1     # users must be invited
# FEATURES_DISABLE_CREATE_GROUP=1    # users cannot create groups
# FEATURES_DISABLE_PUBLIC_GROUPS=1   # disable /explore
# FEATURES_DISABLE_HELP_LINK=1       # disable the help link
# MAX_PENDING_INVITATIONS=100        # maximum unaccepted invitations a group have have
# FEATURES_VOTE_REACTIONS=1          # allow reactions to votes

# Enable search engines to index public content
# ALLOW_ROBOTS=1

# SAML SSO
# SAML_APP_KEY=1 # just a flag, keep value as 1
# SAML_IDP_METADATA_URL=https://saml-metadata-url-provided-by-your-SSO-provider.com/12356

# Sentry DSN
# SENTRY_PUBLIC_DSN=https://[email protected]/123

# monitoring with Posthog
# POSTHOG_HOST=https://posthog.example.com
# POSTHOG_KEY=phc_1234567890

# Disable login via email (usually when you have enabled SSO of some kind)
# FEATURES_DISABLE_EMAIL_LOGIN=1

# oauth providers, to let your users login using external accounts
# FACEBOOK_APP_KEY=REPLACE
# FACEBOOK_APP_SECRET=REPLACE
# TWITTER_APP_KEY=REPLACE
# TWITTER_APP_SECRET=REPLACE
# GOOGLE_APP_KEY=REPLACE
# GOOGLE_APP_SECRET=REPLACE

# Theme images
# images should be a multiple of 32px tall.
# THEME_ICON_SRC=/files/icon.png
# THEME_APP_LOGO_SRC=/files/logo.svg
# THEME_EMAIL_HEADER_LOGO_SRC=/files/logo_128h.png
# THEME_EMAIL_FOOTER_LOGO_SRC=/files/logo_64h.png

# used in emails. use rgb or hsl values, not hex
# THEME_PRIMARY_COLOR=rgb(255,167,38)
# THEME_ACCENT_COLOR=rgb(0,188,212)
# THEME_TEXT_ON_PRIMARY_COLOR=rgb(255,255,255)
# THEME_TEXT_ON_ACCENT_COLOR=rgb(255,255,255)

# tell clients to reload when the server is upgraded
LOOMIO_SYSTEM_RELOAD=1
DEVISE_SECRET=22iafSfaxx75rkTdXaAO3JziFzWmmq6dheqyP1CUVs2k728cHjUADmGqG+AJocyv
SECRET_COOKIE_TOKEN=GFA6BeB9ppKANbfliQqsfp7C8wUAQToMD7oRjQmqpb3AsVnyiQPnn3R3HZc6Ehkt

My docker logs:

loomio@service:~/loomio-deploy$ docker-compose logs -f
Attaching to loomio-app, loomio-channels, loomiodeploy_pgbackups_1, loomio-db, loomio-worker, loomio-mailin, loomio-redis
loomio-channels | 
loomio-channels | > [email protected] start /app
loomio-channels | > node --harmony index.js
loomio-channels | 
loomio-channels | Allowed origin https://loomio.mydomain.com
loomio-channels | booting bots!
loomio-channels | bot redis connected
loomio-app   | [7] Puma starting in cluster mode...
loomio-app   | [7] * Puma version: 6.0.0 (ruby 2.7.6-p219) ("Sunflower")
loomio-app   | [7] *  Min threads: 12
loomio-app   | [7] *  Max threads: 12
loomio-app   | [7] *  Environment: production
loomio-app   | [7] *   Master PID: 7
loomio-app   | [7] *      Workers: 2
loomio-app   | [7] *     Restarts: (✔) hot (✖) phased
loomio-app   | [7] * Preloading application
loomio-app   | /usr/local/lib/ruby/2.7.0/net/protocol.rb:66: warning: already initialized constant Net::ProtocRetryError
loomio-app   | /usr/local/bundle/gems/net-protocol-0.1.3/lib/net/protocol.rb:68: warning: previous definition of ProtocRetryError was here
loomio-app   | /usr/local/lib/ruby/2.7.0/net/protocol.rb:206: warning: already initialized constant Net::BufferedIO::BUFSIZE
loomio-app   | /usr/local/bundle/gems/net-protocol-0.1.3/lib/net/protocol.rb:208: warning: previous definition of BUFSIZE was here
loomio-app   | /usr/local/lib/ruby/2.7.0/net/protocol.rb:503: warning: already initialized constant Net::NetPrivate::Socket
loomio-app   | /usr/local/bundle/gems/net-protocol-0.1.3/lib/net/protocol.rb:504: warning: previous definition of Socket was here
loomio-app   | /loomio/config/initializers/sidekiq.rb:2: warning: Sidekiq's Delayed Extensions will be removed in Sidekiq 7.0
loomio-app   | `Redis#exists(key)` will return an Integer by default in redis-rb 4.3. The option to explicitly disable this behaviour via `Redis.exists_returns_integer` will be removed in 5.0. You should use `exists?` instead.
loomio-app   | [7] * Listening on http://0.0.0.0:3000
loomio-app   | [7] Use Ctrl-C to stop
loomio-app   | [7] - Worker 0 (PID: 10) booted in 0.01s, phase: 0
loomio-app   | [7] - Worker 1 (PID: 24) booted in 0.0s, phase: 0
pgbackups_1  | 2023/01/24 21:10:32 Running version: v0.0.10
pgbackups_1  | 2023/01/24 21:10:33 new cron: @daily
loomio-worker | /usr/local/lib/ruby/2.7.0/net/protocol.rb:66: warning: already initialized constant Net::ProtocRetryError
loomio-mailin | (node:8) Warning: Accessing non-existent property 'padLevels' of module exports inside circular dependency
loomio-mailin | (Use `node --trace-warnings ...` to show where the warning was created)
pgbackups_1  | 2023/01/24 21:10:33 Opening port 8080 for health checking
loomio-worker | /usr/local/bundle/gems/net-protocol-0.1.3/lib/net/protocol.rb:68: warning: previous definition of ProtocRetryError was here
loomio-mailin | info: Mailin v3.0.4
loomio-mailin | info: Webhook url: http://app:3000/email_processor/
loomio-mailin | info: Log file: /var/log/mailin.log
loomio-mailin | info: Spam score computation is disabled
loomio-mailin | (node:15) Warning: Accessing non-existent property 'padLevels' of module exports inside circular dependency
loomio-mailin | (Use `node --trace-warnings ...` to show where the warning was created)
loomio-worker | /usr/local/lib/ruby/2.7.0/net/protocol.rb:206: warning: already initialized constant Net::BufferedIO::BUFSIZE
loomio-mailin | info: Mailin Smtp server listening on port 25
loomio-mailin | warn: Webhook http://app:3000/email_processor/ seems invalid or down. You may want to double check the webhook url.
loomio-worker | /usr/local/bundle/gems/net-protocol-0.1.3/lib/net/protocol.rb:208: warning: previous definition of BUFSIZE was here
loomio-worker | /usr/local/lib/ruby/2.7.0/net/protocol.rb:503: warning: already initialized constant Net::NetPrivate::Socket
loomio-worker | /usr/local/bundle/gems/net-protocol-0.1.3/lib/net/protocol.rb:504: warning: previous definition of Socket was here
loomio-worker | /loomio/config/initializers/sidekiq.rb:2: warning: Sidekiq's Delayed Extensions will be removed in Sidekiq 7.0
loomio-worker | `Redis#exists(key)` will return an Integer by default in redis-rb 4.3. The option to explicitly disable this behaviour via `Redis.exists_returns_integer` will be removed in 5.0. You should use `exists?` instead.
loomio-worker | 2023-01-24T21:10:38.398Z pid=7 tid=8c7 INFO: Booted Rails 6.1.7 application in production environment
loomio-worker | 2023-01-24T21:10:38.398Z pid=7 tid=8c7 INFO: Running in ruby 2.7.6p219 (2022-04-12 revision c9c2245c0a) [x86_64-linux]
loomio-worker | 2023-01-24T21:10:38.398Z pid=7 tid=8c7 INFO: See LICENSE and the LGPL-3.0 for licensing details.
loomio-worker | 2023-01-24T21:10:38.398Z pid=7 tid=8c7 INFO: Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org
loomio-db    | 
loomio-db    | PostgreSQL Database directory appears to contain a database; Skipping initialization
loomio-redis | 1:C 24 Jan 2023 21:10:34.317 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
loomio-redis | 1:C 24 Jan 2023 21:10:34.317 # Redis version=5.0.14, bits=64, commit=00000000, modified=0, pid=1, just started
loomio-redis | 1:C 24 Jan 2023 21:10:34.317 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
loomio-redis | 1:M 24 Jan 2023 21:10:34.318 * Running mode=standalone, port=6379.
loomio-redis | 1:M 24 Jan 2023 21:10:34.318 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
loomio-redis | 1:M 24 Jan 2023 21:10:34.318 # Server initialized
loomio-redis | 1:M 24 Jan 2023 21:10:34.318 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
loomio-redis | 1:M 24 Jan 2023 21:10:34.318 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
loomio-redis | 1:M 24 Jan 2023 21:10:34.318 * Ready to accept connections
loomio-db    | 
loomio-db    | 2023-01-24 21:10:32.227 UTC [1] LOG:  starting PostgreSQL 12.2 (Debian 12.2-2.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
loomio-db    | 2023-01-24 21:10:32.227 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
loomio-db    | 2023-01-24 21:10:32.227 UTC [1] LOG:  listening on IPv6 address "::", port 5432
loomio-db    | 2023-01-24 21:10:32.231 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
loomio-db    | 2023-01-24 21:10:32.253 UTC [26] LOG:  database system was shut down at 2023-01-24 19:37:28 UTC
loomio-db    | 2023-01-24 21:10:32.262 UTC [1] LOG:  database system is ready to accept connections
loomio-app   | 2023-01-24 21:11:32 +0000 HTTP parse error, malformed request: #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?>
loomio-app   | 2023-01-24 21:11:32 +0000 HTTP parse error, malformed request: #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?>

And my nginx (though I don't think it should matter):

server {
        listen 80;
        listen [::]:80;
        server_name loomio.mydomain.com;
        return 301 https://$server_name$request_uri;
}

server {
        listen 443 ssl http2;
        listen [::]:443 ssl http2;
	ssl_certificate /etc/letsencrypt/live/loomio.mydomain.com/fullchain.pem; # managed by Certbot
	ssl_certificate_key /etc/letsencrypt/live/loomio.mydomain.com/privkey.pem; # managed by Certbot
	include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
	ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
	server_name loomio.mydomain.com;
        
	location ~ /.well-known {
                allow all;
        }
        location / {
	proxy_pass http://127.0.0.1:4000;
        proxy_http_version  1.1;
        proxy_cache_bypass  $http_upgrade;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
	client_max_body_size 200m;
}
}

Thanks!

RG

Robert Guthrie Wed 25 Jan 2023 8:15PM

Hi @Yochai Gal it looks like you're doing it right. The message is saying that you must use https to connect to Loomio.

However application.rb:37 is not taking notice of FORCE_SSL, so I've updated it in the master branch, so you should be able to disable https by removing/commenting the FORCE_SSL line from your config.

YG

Yochai Gal Wed 25 Jan 2023 8:36PM

Thanks! But shouldn't it work through my https/nginx config? I get the same error either way (nginx doesn't report any errors).

Re: FORCE_SSL, I was wondering about that (and wow, 0 = true in ruby!).

RG

Robert Guthrie Thu 26 Jan 2023 1:34AM

Ya, it should work. I don't know what's wrong, sorry.

If you disable SSL, can you connect directly without your nginx frontend?

YG

Yochai Gal Thu 26 Jan 2023 4:35PM

I updated to latest image using these instructions.

I have disabled HTTPS by remarking FORCE_SSL=1 in the .env file.

However, it does not allow me to connect directly, with or without nginx. Nor can I connect directly (e.g. http://myurl.com:4000) I'm also seeing this error in the logs:

warn: Webhook http://app:3000/email_processor/ seems invalid or down. You may want to double check the webhook url.

And when I go to the raw URL (no nginx):

HTTP parse error, malformed request: #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?>

Notably, nginx doesn't see any errors.

RG

Robert Guthrie Fri 27 Jan 2023 10:43PM

Looks like rails is still expecting a https request. Can you try using https?

RG

Robert Guthrie Fri 27 Jan 2023 10:44PM

The http://app:3000/email_processor/ warning is noting to worry about, it's normal on startup.

YG

Yochai Gal Fri 27 Jan 2023 10:48PM

https://myurl.com:4000 gets the same error. I turned off my non-docker nginx for the test.

YG

Yochai Gal Fri 27 Jan 2023 10:49PM

Doing

docker exec -it loomio-app bash

and

echo $FORCE_SSL

Correctly shows a blank line.

YG

Yochai Gal Fri 27 Jan 2023 11:05PM

I got it!

You basically cannot access it without a reverse proxy (even on the raw URL). I followed these instructions on the Mastodon github (???) and borrowed the nginx conf (mostly). My final nginx config was:

        }
        location / {
        proxy_pass http://127.0.0.1:4000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_pass_header Server;
        proxy_buffering off;
        proxy_redirect off;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
  }
RG

Robert Guthrie Sun 29 Jan 2023 10:30PM

Wow! Well done, and thanks for posting your solution! Nice work.