Proposal: increase our Tech Working Group monthly infrastructure spend to ~€150
Update (2022-11-25): this proposal is superseded by https://www.loomio.com/p/PrfjuJAv/proposal-assign-a-250-monthly-discretionary-budget-to-the-tech-working-group, which takes into account feedback by the community.
Hi there!
We are discussing scalability and reliability strategy in the Tech Working Group this week, after the great work by @protean which brought us to the latest Mastodon version safely.
We have located two servers in Hetzner under auction that would let us augment and replace Runko, our current server with hardware from ~2013. To lease them would signify an extra expense of <€100 monthly:
Once they are set up, we could likely retire Runko (with their aging hardware) and save those ~40€/mo if we want that -- or replace it with newer hardware at a similar price, keeping our monthly server spend in the €150 range.
@Nathan Schneider, we discussed having a discretionary budget elsewhere -- do you think this proposal needs more details and a consultation with the community or do you feel comfortable approving it? Moving fast would enable us to start working on balancing the current load better, and prepare us to grow the instance further if we want to do that and the opportunity arises. But I'm unsure what the right way forward it.
@Ian Smith @Bo Jeanes among others from the TWG reviewed so far, review of the rest of the working groups and the wider community is of course very welcome :)
Yes, though it is an open question as to how these servers are used (e.g. putting data stores on one, application on the other). So whether or not it would provide redundancy in that sense is unclear.
We could consider having the DB run on both with one being a replica and that would give us some failover capacity. Interesting to consider...
I'm very much in favor of this increase and lease since trusted members of TWG feel it is a good use of funds. If we do make this purchase, I think it'd be really helpful if someone could write up a very brief explanation of the intended present & aspirational future uses of the three servers we'd then have for members (like me) with limited understandings of all the complexities involved in running our coop's technical backend.
I think we should also take whatever steps are necessary to move forward with approving a discretionary budget for the TWG. Am I correct in understanding that currently there's not a functioning separate TWG ops team like there is for community, so the core of TWG people making decisions is a bit amorphous? If this is the case, and assembling a firm ops team isn't something that we can/should move forward with right this moment, maybe we could set down a precedent that either the community OPs team or the on-call person would just be notified/consulted when that discretionary budget is used, so there's some degree of process?
Thank you, Geraldo!
I think of Community Working Group as a single group currently; in one recent meeting we actually discussed restructuring IIUC (https://www.loomio.com/d/yh0aFRbU/community-working-group-ops-team-minutes has minutes). Is there a different group that ever meets as the non-Ops Community Group? :) I'm relatively new so I'm unsure, but I think not. @Matt Noyes @emi do @Mica Fisher @Sam Whited probably know more.
I think the TWG should have an oncall rotation like the CWG does; but I'm unsure what introducing an ops/non-ops split would buy us. I understand an approval process for expenses makes sense, and I agree having the oncall from some other Working Group approve might be ideal.
FWIW I'm oncall for the CWG this week so technically I approved this proposal which was initiated by @Ian Smith :)
There used to be a meaningful distinction between the CWG and the CWG Ops Team, but the CWG has atrophied so CWG ops team proposals just go straight to the SC membership.
I'm generally in favor of this; I'm not sure if it's the new traffic load or the 4.0 upgrade, but everything seems to be showing a loading bar now (meaning no disrespect to the tech team, it's not unusable slow or anything and the 4.0 upgrade was definitely worth it, thank you for doing the work to get it running for us all!), so some beefier hardware seems like a great idea to handle the increased load.
Would it make sense to talk with our friends at MayFirst.Coop and/or Meet.Coop to see if they would be interested in sharing hosting resources? (Assuming that is relevant/possible -- not a tech person, myself ;-))? Always on the lookout for possibilities for inter-cooperation. @Jamie Gaehring @mike_hales
Yes, this sounds lovely!
I think @Calix from Coop Cloud who recently joined our coop individually as well as the TWG might also have interesting pointers/opinions here.
Good idea -- there may be other co-ops interested, too.
I was wondering the same thing about May First. I looked into them quite a bit when the proposal was going around about social.coop joining, and it got me wondering about some of my own hosting needs. They seem to have a good service
My vote here is to go along with what the TWG recommends, but I’m curious how they might be able to play a part in all this.
Finally catching up on all this, but sounds good! I'd be interested in knowing more how May First Coop works in general, as much out of personal curiosity as anything else.
It looks like we have space for this in our budget. I'm all in favor! (cf. https://opencollective.com/socialcoop#category-BUDGET and https://www.loomio.com/d/jSvw79qE/we-have-room-in-our-budget-to-do-more-what-should-we-do-)
This sounds good. Certainly in favor of supporting growth and sustainability of hosting solutions. If there are promising links to be made with other coops and May First on this front also favor that as makes best sense in terms of cost and flexibility.
Sounds like a good idea to me! We're all here to use this server, so IMO this is our first priority for funds.
Proposal: increase our Tech Working Group monthly infrastructure spend to ~€150
proposal by Darren Closed Fri 25 Nov 2022 1:07AM
Poll closed to be replaced by a new reformulated proposal
This poll was labelled urgent. It no longer is.
It was labeled URGENT as it appeared the funds were required quickly (read below) and so that it could comply with our [bylaws](https://wiki.social.coop/rules-and-bylaws/Bylaws-for-social.coop.html)
Can see more details about the expenses in the [thread](https://www.loomio.com/d/Oakf6qgh/proposal-increase-our-tech-working-group-monthly-infrastructure-spend-to-150)
People whove already indicated they support the proposal can be manually added to the poll totals after it closes
Results
| Results | Option | Votes | % of votes cast | % of eligible voters | |
|---|---|---|---|---|---|
| Agree | 26 | 57 | 10 |            | |
| Abstain | 12 | 26 | 5 |        | |
| Disagree | 8 | 17 | 3 |     | |
| Not voted | 219 | 83 |                   |
Members of the tech working group have been super active applying upgrades to our Mastodon server over the last few days. Very happy for the necessary resources to be made available in a timely manner to smooth their work and make our instance more robust
I am, feeling very conflicted, voting "no" because:
I don't know anything that would justify this being an "URGENT" vote; what is the time pressure to decide this within 24 hours instead of a week?
There is still disagreement between folks in the "Tech Working Group" Matrix channel about this change; I'm not sure if folks were just referring to the TWG "Ops" members, but I don't think it's accurate to say that the TWG has agreed to ask for this.
Sounds great! Is the plan replicate data and/or application deployments across the two servers?
In general, I suggest having some buffer for any working group to make temporary decisions to use a modest amount of extra funds for any urgent need without getting a full-membership vote. Such decisions should be done in ways that are easy to reverse if possible. Then, the WG should explain why the extra funds were needed and to propose to the membership any long-term changes.
Long-term changes should not be made urgently. Urgent temporary changes should not have excessive obstacles.
Don't see why this is urgent? Am I missing something?
Also, I had hope we would explore doing this jointly with Meet.Coop and/or MayFirst.Coop
150/month is entirely reasonable for a service of our size and represents 15% of our current open collective income
I very much support the proposal, but it seems like it might not be clear that members of the TWG all feel this is the right way to go yet. May be preferable to take more time to consider, refine the proposal/process & possibly address the discretionary fund matter and/or potential partnerships with MayFirst first. Would support the proposal if these issues are worked out to make sure it actually represents the desired course of action from the TWG at this time, or a refined proposal is made.
Voting "no" because I don't understand why this proposal is marked as "urgent", and there doesn't seem to be clarity that the TWG agrees on this direction.
We should make sure there is a working buffer that might be even higher than this amount. We’re onboarding more people and seeing more traffic, it’s good to have this in place so that Service Döner degrade for members new and old.
This all sounds very reasonable to me, but I don't know enough to actually vote on this, so I'm abstaining. It also seems like there may be some disagreement among the people who this most affects, which seems not ideal.
I'm in favor of increasing the budget promptly, to give the TWG flexibility to make decisions that it agrees to make to manage increased demand.
Seems like the extra spend for the huge increase in new users is warrented
Changing my vote to disagree, considering this proposal doesn't seem to have had consensus of the Tech Working Group, and that "auctions" are ongoing. It might be best to bring forward a whole new proposal once the TWG has reached consensus.
Having read the discussion I agree enough in principle with the idea to support it, but I share the reservations about making this an 'Urgent' all-member vote. There ought to be a better way of handling these kind of decisions.
The market for computer hardware is a Dutch Auction. Work out what you want to do, choose the appropriate specs for the machine you need for the tasks, and then wait till the price falls to the level that it's within your budget. Buying on impulse means that you're not getting a bargain.
Am in favour but see an amendment and call for more discussion/clarification, so abstaining.
Sound reasoning. I agree with the proposal.
While I agree that such changes should ideally be discussed in a slower fashion, I trust the tech working group to have thought the decision through for long enough and only needs quick validation from us all.
Let's strike while the iron is hot
I think the proposal has merit, but I disagree with it being market as urgent during the a national holiday in the USA. Those interested may not have the time to actually review it. To be clear, I would be voting yes but for this process issue, which is why I chose the middle-of-the-road abstain position.
Voting disagree
I think this is a good plan (after all I suggested it) but I think it requires more discussion in the TWG before moving forward. There are also some unresolved issues:
First, the payment method for the Hetzner account needs to be addressed as it is currently @Nick S Paypal, and I believe he has not had a chance to review this idea at all. This is a blocker for me.
Second, the DNS credentials need to be recovered. This is also waiting on @Nick S and is a technical blocker.
To make the vote I would like to have:
A summary with all relevant context: I.e. Increase the Hosting and Infrastructure budget from $1,200 to $1,800 per year. This brings Hosting and Infrastructure from 10% to 15% of our gross annual income; and reduces our "no-income" runway from 24 months to 23 months)
Clarity on which members of the TWG prefer/accept/object to the proposal.
If the TWG is in favor, I would prefer they held this directly (W/Finance Group) rather than a member vote.
Conflicted. I want to make sure they have the funds to do the work. I see some good reasons to delay until a more solid plan is shared.
Not 100% sure what the vote is asking for at this point, so abstaining. But I support the TWG Ops Team having ample budget, and discretion over the spend of that budget.
I'm changing my vote, as I persuaded by the discourse that's gone on.
I'd unconditionally support getting a single server to replace Runko and think it's within the purview of TWG's discretionary spending. We can continue discussing and exploring architectural changes we would make to support scale can carry on non-urgently and in parallel to wider group discussions about the future of the coop.
@Darren can you clarify why is this "urgent" and needs to be decided by tomorrow? (Is this because of a time limit to get that extra server? If so, I didn't see that stated anywhere explicitly.)
The folks whove been busy working on our servers the last few days have expressed a desire to get the new servers ASAP/tomorrow. I suspect this wont be a controversial vote and as I understand it theres no minimum contract on the servers and no set up fee so we can back out at any time if thats later felt to be necessary.
If you are interested [this](https://matrix.to/#/!akkaZImONyQWKswVdt:matrix.org/$166924586016282GGxJN:matrix.org?via=matrix.org&via=autonomic.zone&via=beeper.com) (and much more) has all been discussed in the tech working group matrix room
lol, apparently more controversial than i imagined. Possibly i misunderstood the urgency. Interested in opinions and happy to alter (extend?) or cancel the poll if the urgency isnt really there and folks feel we need longer to consider or time to redraft the proposal
As I hinted in my vote, if a proposal is uncontroversial and urgent/timely, it shouldn't go out to a general member vote, it should just be decided by the WG. I think members have an intuition that if something is urgent and has to go to an all-member vote, it must be because the decision is non-trivial, has serious ramifications or something.
I imagine there's less controversy over the proposal than there is discomfort with all members being asked to rubber-stamp an urgent decision.
this reflects my feelings: I either feel like the TWG should have discretion to keep the shit online in the case this is time critical manner and we can sort out details later, or I feel like we should have more time to consider. personally I am like "hell yeah go get the dang servers" since I've seen several financial check-ins up thread. but yeah more a hesitant vote on use of process.
Suggested amendment: make this a normal (not urgent) proposal and reset the end date to Nov 30.
I agree, this needs more time.
To quote @Flancian
I posted in the open chat room about the server auction. my default is to buy them tomorrow -- I wanted to do it tonight but unless you think there's a risk we lose them it seems letting the community participate in this decision is a good idea.
I'm inclined to let it run as is till they can chime in. Particularly because we are running on an old server which was struggling a fair bit before the mastodon upgrade, we have plenty of funds, this isnt an irreversible decision.
Personally Im happy the setvers are getting attention. Stuff had got neglected.
To give some more clarity on TWG discussions the only disagreement I saw in the tech group chat was an uncertainty we needed all three servers, the person that raised that was happy to step aside as they werent doing the work and they were happy to let those that are move things forward. @Calix is, i think, also in the TWG
AFAIK, the urgency is because this price is part of an "auction".
Right now, the people involved in the TWG discussion appear (to me) to be split on how we architecturally use these resources.
However, it's clear that our current server is older and not particularly cost-effective. Another less-contentious proposal that I think the TWG would more unanimously back is simply getting one server to replace the current one which, other than the overlapping month(s), would be pretty equivalent cost-wise to the current setup.
Then, non-urgently, the TWG could continue its exploration into whether separating the database from the application (or similar proposals) is merited.
The "Auction" is ongoing, in my experience the offers there are fairly consistent.
I'm in favor of us preparing to handle more traffic and get off of aging hardware as it sounds as if there's a good amount of wiggle room in our budget. The mastodon server seems to be what people have joined social co-op for in the first place so it's important to make sure we have adequate resources to meet increasing demand. Additional redundancy is important too so we can continue to serve traffic if one of the servers goes down.