Loomio

Infrastructure upgrades/Improvements to technical operations

Victor Matekole PublicSeen by 22

TL;DR: We need to increase monthly spend on servers and Victor will be supporting @mayel in tech. ops. Additionally, we need someone outside of euro zone to be available for emergency response.

@mayel and I had a meeting last wk, where we discussed:

  1. Improvements to the infrastructure, to support scaling and improved reliability
  2. 24/7 follow the sun support
  3. Improved monitoring
  4. My supporting role to @mayel

An additional server is required to increase performance, improve reliability, and increase disk capacity. Total monthly costs will increase to $25 for the following spec:

Virtual server for front-end (Nginx serving about page, maintenance page, membership PHP code, and proxying Mastodon)
$5 per month
2 X86 64bit Cores
2GB memory
200Mbit/s Unmetered
50GB SSD storage

Dedicated server for back-end (running Mastodon stack, serving static content)
$16 per month
4 Dedicated X86 64bit Cores
8GB RAM
300Mbit/s Unmetered
50+50GB SSD storage

We currently have 30G of media/attachments, we'll need to add additional SSD storage as needed, in slices of 150GB at $4 per month. Additionally, monitoring is to be improved to the underlying services by using Datadog(free) and Pingdom(paid for by Seedbloom).

To improve operations and take some load off @mayel — I will be supporting him. However, we need at least one additional person within the U.S. timezone to respond to outages, since @mayel and I are in eurozone.

I'll be putting the increased monthly budget to vote later this wk.

Michele Kipiel

Thank you for keeping us up to date, you guys are doing an awesome job!
Looking forward to the vote.

Matthew Cropp

Thanks for your work on this! For the US based tech team person, perhaps we should post an ask to the instance hashtagged #SocialCoop if a volunteer is not immediately apparent on Loomio?

Greg

Did the budget increase vote happen? Did you find another US-ish admin volunteer? Anything left to do here? :)

Mayel de Borniol

@greg52

@victormatekole and I are getting ready to switch over to an improved setup with Docker swarm across 2 servers. We definitely need more tech hands, especially in non-Europe-ish timezones. We'll do our best to document everything and set up monitoring tools, etc.

Michele Kipiel

Thanks for all the great work guys! Let us know what are the new costs you are incurring into, if any, so we can help!

cloudwater

Is there a way to integrate images hosted off site into the Mastodon UI? Encouraging users to host their images elsewhere (e.g. on their own servers) might be a good way to limit cost, but I don't see that happening unless you can actually view the image in Mastodon.

Mayel de Borniol

@lehighcommunalist that's a good idea, which I would pursue as a feature request to Mastodon devs.

FYI Mastodon downloads media attachments uploaded by users of other instances that we federate with onto our server anyway. (We can run a script to regularly delete older attachments from other instances)

Victor Matekole

@lehighcommunalist Mastodon is a Ruby Rails stack and supports the use of S3-compatible image distribution (via paperclip). In the new environment (yet to switchover) we have moved all user-generated content to our DreamObjects CDN. This is sponsored by Seedbloom until the finances of the coop mature. The cost is quite minimal for now, considering our usage. We should expect faster image downloads globally and considerably less disk usage. Unfortunately, we won't be able to support fast uploads, at least for now — uploads still need to go via server first, resized and then pushed to CDN. The vote for increased budget happened here — https://www.loomio.org/d/Ad4tPJK5/expense-server-upgrades.

Victor Matekole

In addition @lehighcommunalist I saw your proposal — https://www.loomio.org/p/ch4G0jyB. Unfortunately, can't comment on proposals, so stating here... Could you elaborate a little on how you would see this being implemented? Do you mean user-generated comment being evenly distributed across Mastodon instances similar to emerging technologies like Storj or Swarm but in a federated fashion as opposed to a pure decentralised model? Would love to know your thoughts.