Proposal: increase our Tech Working Group monthly infrastructure spend to ~€150
Update (2022-11-25): this proposal is superseded by https://www.loomio.com/p/PrfjuJAv/proposal-assign-a-250-monthly-discretionary-budget-to-the-tech-working-group, which takes into account feedback by the community.
Hi there!
We are discussing scalability and reliability strategy in the Tech Working Group this week, after the great work by @protean which brought us to the latest Mastodon version safely.
We have located two servers in Hetzner under auction that would let us augment and replace Runko, our current server with hardware from ~2013. To lease them would signify an extra expense of <€100 monthly:
Once they are set up, we could likely retire Runko (with their aging hardware) and save those ~40€/mo if we want that -- or replace it with newer hardware at a similar price, keeping our monthly server spend in the €150 range.
@Nathan Schneider, we discussed having a discretionary budget elsewhere -- do you think this proposal needs more details and a consultation with the community or do you feel comfortable approving it? Moving fast would enable us to start working on balancing the current load better, and prepare us to grow the instance further if we want to do that and the opportunity arises. But I'm unsure what the right way forward it.
@Ian Smith @Bo Jeanes among others from the TWG reviewed so far, review of the rest of the working groups and the wider community is of course very welcome :)
Bo Jeanes Wed 23 Nov 2022 11:38PM
Yes, though it is an open question as to how these servers are used (e.g. putting data stores on one, application on the other). So whether or not it would provide redundancy in that sense is unclear.
We could consider having the DB run on both with one being a replica and that would give us some failover capacity. Interesting to consider...
Geraldo Fernandez (@yerald@social.coop Wed 23 Nov 2022 11:37PM
I'm very much in favor of this increase and lease since trusted members of TWG feel it is a good use of funds. If we do make this purchase, I think it'd be really helpful if someone could write up a very brief explanation of the intended present & aspirational future uses of the three servers we'd then have for members (like me) with limited understandings of all the complexities involved in running our coop's technical backend.
I think we should also take whatever steps are necessary to move forward with approving a discretionary budget for the TWG. Am I correct in understanding that currently there's not a functioning separate TWG ops team like there is for community, so the core of TWG people making decisions is a bit amorphous? If this is the case, and assembling a firm ops team isn't something that we can/should move forward with right this moment, maybe we could set down a precedent that either the community OPs team or the on-call person would just be notified/consulted when that discretionary budget is used, so there's some degree of process?
Flancian Wed 23 Nov 2022 11:44PM
Thank you, Geraldo!
I think of Community Working Group as a single group currently; in one recent meeting we actually discussed restructuring IIUC (https://www.loomio.com/d/yh0aFRbU/community-working-group-ops-team-minutes has minutes). Is there a different group that ever meets as the non-Ops Community Group? :) I'm relatively new so I'm unsure, but I think not. @Matt Noyes @emi do @Mica Fisher @Sam Whited probably know more.
I think the TWG should have an oncall rotation like the CWG does; but I'm unsure what introducing an ops/non-ops split would buy us. I understand an approval process for expenses makes sense, and I agree having the oncall from some other Working Group approve might be ideal.
FWIW I'm oncall for the CWG this week so technically I approved this proposal which was initiated by @Ian Smith :)
Matt Noyes Wed 23 Nov 2022 11:47PM
There used to be a meaningful distinction between the CWG and the CWG Ops Team, but the CWG has atrophied so CWG ops team proposals just go straight to the SC membership.
Sam Whited Wed 23 Nov 2022 11:41PM
I'm generally in favor of this; I'm not sure if it's the new traffic load or the 4.0 upgrade, but everything seems to be showing a loading bar now (meaning no disrespect to the tech team, it's not unusable slow or anything and the 4.0 upgrade was definitely worth it, thank you for doing the work to get it running for us all!), so some beefier hardware seems like a great idea to handle the increased load.
Matt Noyes Wed 23 Nov 2022 11:44PM
Would it make sense to talk with our friends at MayFirst.Coop and/or Meet.Coop to see if they would be interested in sharing hosting resources? (Assuming that is relevant/possible -- not a tech person, myself ;-))? Always on the lookout for possibilities for inter-cooperation. @Jamie Gaehring @mike_hales
Flancian Wed 23 Nov 2022 11:45PM
Yes, this sounds lovely!
I think @Calix from Coop Cloud who recently joined our coop individually as well as the TWG might also have interesting pointers/opinions here.
Matt Noyes Wed 23 Nov 2022 11:48PM
Good idea -- there may be other co-ops interested, too.
Joseph Andriano Wed 23 Nov 2022 11:59PM
I was wondering the same thing about May First. I looked into them quite a bit when the proposal was going around about social.coop joining, and it got me wondering about some of my own hosting needs. They seem to have a good service
My vote here is to go along with what the TWG recommends, but I’m curious how they might be able to play a part in all this.
Eliot Lash · Wed 23 Nov 2022 11:24PM
I'm in favor of us preparing to handle more traffic and get off of aging hardware as it sounds as if there's a good amount of wiggle room in our budget. The mastodon server seems to be what people have joined social co-op for in the first place so it's important to make sure we have adequate resources to meet increasing demand. Additional redundancy is important too so we can continue to serve traffic if one of the servers goes down.