I'm in favor of us preparing to handle more traffic and get off of aging hardware as it sounds as if there's a good amount of wiggle room in our budget. The mastodon server seems to be what people have joined social co-op for in the first place so it's important to make sure we have adequate resources to meet increasing demand. Additional redundancy is important too so we can continue to serve traffic if one of the servers goes down.

Yes, though it is an open question as to how these servers are used (e.g. putting data stores on one, application on the other). So whether or not it would provide redundancy in that sense is unclear.

We could consider having the DB run on both with one being a replica and that would give us some failover capacity. Interesting to consider...

I'm very much in favor of this increase and lease since trusted members of TWG feel it is a good use of funds. If we do make this purchase, I think it'd be really helpful if someone could write up a very brief explanation of the intended present & aspirational future uses of the three servers we'd then have for members (like me) with limited understandings of all the complexities involved in running our coop's technical backend.

I think we should also take whatever steps are necessary to move forward with approving a discretionary budget for the TWG. Am I correct in understanding that currently there's not a functioning separate TWG ops team like there is for community, so the core of TWG people making decisions is a bit amorphous? If this is the case, and assembling a firm ops team isn't something that we can/should move forward with right this moment, maybe we could set down a precedent that either the community OPs team or the on-call person would just be notified/consulted when that discretionary budget is used, so there's some degree of process?

Thank you, Geraldo!

I think of Community Working Group as a single group currently; in one recent meeting we actually discussed restructuring IIUC (https://www.loomio.com/d/yh0aFRbU/community-working-group-ops-team-minutes has minutes). Is there a different group that ever meets as the non-Ops Community Group? :) I'm relatively new so I'm unsure, but I think not. @Matt Noyes @emi do @Mica Fisher @Sam Whited probably know more.

I think the TWG should have an oncall rotation like the CWG does; but I'm unsure what introducing an ops/non-ops split would buy us. I understand an approval process for expenses makes sense, and I agree having the oncall from some other Working Group approve might be ideal.

FWIW I'm oncall for the CWG this week so technically I approved this proposal which was initiated by @Ian Smith :)

There used to be a meaningful distinction between the CWG and the CWG Ops Team, but the CWG has atrophied so CWG ops team proposals just go straight to the SC membership.

I'm generally in favor of this; I'm not sure if it's the new traffic load or the 4.0 upgrade, but everything seems to be showing a loading bar now (meaning no disrespect to the tech team, it's not unusable slow or anything and the 4.0 upgrade was definitely worth it, thank you for doing the work to get it running for us all!), so some beefier hardware seems like a great idea to handle the increased load.

Would it make sense to talk with our friends at MayFirst.Coop and/or Meet.Coop to see if they would be interested in sharing hosting resources? (Assuming that is relevant/possible -- not a tech person, myself ;-))? Always on the lookout for possibilities for inter-cooperation. @Jamie Gaehring @mike_hales

Yes, this sounds lovely!

I think @Calix from Coop Cloud who recently joined our coop individually as well as the TWG might also have interesting pointers/opinions here.

Good idea -- there may be other co-ops interested, too.

I was wondering the same thing about May First. I looked into them quite a bit when the proposal was going around about social.coop joining, and it got me wondering about some of my own hosting needs. They seem to have a good service

My vote here is to go along with what the TWG recommends, but I’m curious how they might be able to play a part in all this.

Finally catching up on all this, but sounds good! I'd be interested in knowing more how May First Coop works in general, as much out of personal curiosity as anything else.

It looks like we have space for this in our budget. I'm all in favor! (cf. https://opencollective.com/socialcoop#category-BUDGET and https://www.loomio.com/d/jSvw79qE/we-have-room-in-our-budget-to-do-more-what-should-we-do-)

This sounds good. Certainly in favor of supporting growth and sustainability of hosting solutions. If there are promising links to be made with other coops and May First on this front also favor that as makes best sense in terms of cost and flexibility.

Sounds like a good idea to me! We're all here to use this server, so IMO this is our first priority for funds.

@Darren can you clarify why is this "urgent" and needs to be decided by tomorrow? (Is this because of a time limit to get that extra server? If so, I didn't see that stated anywhere explicitly.)

The folks whove been busy working on our servers the last few days have expressed a desire to get the new servers ASAP/tomorrow. I suspect this wont be a controversial vote and as I understand it theres no minimum contract on the servers and no set up fee so we can back out at any time if thats later felt to be necessary.

If you are interested [this](https://matrix.to/#/!akkaZImONyQWKswVdt:matrix.org/$166924586016282GGxJN:matrix.org?via=matrix.org&via=autonomic.zone&via=beeper.com) (and much more) has all been discussed in the tech working group matrix room

lol, apparently more controversial than i imagined. Possibly i misunderstood the urgency. Interested in opinions and happy to alter (extend?) or cancel the poll if the urgency isnt really there and folks feel we need longer to consider or time to redraft the proposal

As I hinted in my vote, if a proposal is uncontroversial and urgent/timely, it shouldn't go out to a general member vote, it should just be decided by the WG. I think members have an intuition that if something is urgent and has to go to an all-member vote, it must be because the decision is non-trivial, has serious ramifications or something.

I imagine there's less controversy over the proposal than there is discomfort with all members being asked to rubber-stamp an urgent decision.

this reflects my feelings: I either feel like the TWG should have discretion to keep the shit online in the case this is time critical manner and we can sort out details later, or I feel like we should have more time to consider. personally I am like "hell yeah go get the dang servers" since I've seen several financial check-ins up thread. but yeah more a hesitant vote on use of process.

Suggested amendment: make this a normal (not urgent) proposal and reset the end date to Nov 30.

I agree, this needs more time.

To quote @Flancian

I posted in the open chat room about the server auction. my default is to buy them tomorrow -- I wanted to do it tonight but unless you think there's a risk we lose them it seems letting the community participate in this decision is a good idea.

I'm inclined to let it run as is till they can chime in. Particularly because we are running on an old server which was struggling a fair bit before the mastodon upgrade, we have plenty of funds, this isnt an irreversible decision.

Personally Im happy the setvers are getting attention. Stuff had got neglected.

To give some more clarity on TWG discussions the only disagreement I saw in the tech group chat was an uncertainty we needed all three servers, the person that raised that was happy to step aside as they werent doing the work and they were happy to let those that are move things forward. @Calix is, i think, also in the TWG

AFAIK, the urgency is because this price is part of an "auction".

Right now, the people involved in the TWG discussion appear (to me) to be split on how we architecturally use these resources.

However, it's clear that our current server is older and not particularly cost-effective. Another less-contentious proposal that I think the TWG would more unanimously back is simply getting one server to replace the current one which, other than the overlapping month(s), would be pretty equivalent cost-wise to the current setup.

Then, non-urgently, the TWG could continue its exploration into whether separating the database from the application (or similar proposals) is merited.

The "Auction" is ongoing, in my experience the offers there are fairly consistent.

And I would personally support replacing the server 1:1; my concerns are about adding architectural complexity (specifically I saw Wireguard, and Docker Swarm, mentioned) which will reduce the pool of potential folks who could help with TWG work – which seems especially dangerous given that the TWG has been down to 0-to-1 people before.

That was just my speculation about why it was considered urgent, but I think @Flancian has since further explained why below.

Yep very fair.

FWIW: Docker Swarm is not a significant departure from our current strategy, as it still is essentially a docker-compose.yml-based workflow. I don't think Wireguard is necessary and I too would be concerned about adding that into the mix for the reasons you've expressed.

Personally, I only want us to separate DB and app onto two machines, even if we changed nothing else.

I'm pretty familiar with Docker Swarm – I helped implement a project entirely based on it 😅 – and I'd disagree that it's "not a significant departure": using it requires learning about the "stack" and "service" Docker concepts (and related management commands), potentially adjusting to a different mental model because of the potential decoupling of where the compose file lives vs where the stack is deployed, and the UI is very gory (`docker-compose restart` vs `docker service scale foo_bar=0 && docker service scale foo_bar=1`, `docker-compose stop` vs `docker stack rm`).

I think swarm is better, for sure, but I think it's worth asking who in TWG knows it, or is willing to learn, before introducing it.

idk how to vote after reading the whole thread! gave some context above!

• I support the discretion of our working groups!

• if there is some time-critical auction that nets us some needed hardware in a time of budgetary surplus I'm in favor of them making the buy and briefing afterwards with invitation to clarify spending process!

• I do think we should revise the urgent vote process to need to include some description about why the vote needs to be urgent, because I think process concerns were the major ones reading the above thread!

• again thank you tech ops/tech working group for bringing this item and being on the lookout for good hardware! appreciate your work !!!!!

In addition to the comments I made in my vote, there are some other things making me think this should get better consensus before moving forward.

I think there is a larger discussion to be had around how the co-op should scale. I do not personally that mega coops or mega instances are good for democratic governance. Scaling servers now would moot that discussion as there would no longer be an external pressure (server capacity) to force it.

Since the latest update, we are seeing more headroom on the server due to bug fixes and improvements in the underlying libraries, so this is not as urgent as it would have been a week ago.

Important:

There is a caveat here: The instance has a lot of inactive accounts. It's like a 2:1 ratio of inactive to active. If a large number of those accounts suddenly became active again this would change the urgency.

I think these are important considerations, but not relevant at this time. The current server is ageing, and redundancy is always a good thing. What happens if good ol' Runko hits the sack?

I believe my last comment did not send, my apologies if for some reason the comment is just not visible on my screen.

I think this discussion is important to have, but I don't think it's relevant to this proposal. Runko is ageing, and I fear for what would happen if they suddenly hit the sack. New servers and redundancy is always good, regardless of intended scale.

As theres more TWG folks now chiming in saying they feel the process needs more time Im going to give it the full week and remove the URGENT🚨

Is it possible to repropose this with more info as a formal recommendation voted out of the TWG? I’m seeing conflicting discussion about whether the TWG thinks this is a good idea. I’d like it if we could have a proposal with the various options/costs and vote on that. I’ll admit I’m pretty confused at this point.

Hi all! Thank you all for your comments. Thank you particularly Darren for starting the poll!

Although I agree that making it not urgent seems like a reasonable thing here based on community feedback, I am (full disclosure) personally a bit disappointed by the fact that this will make it so that we won't have the server(s) available until November 30 and thus we will not be able to start working on them until December. The TWG is volunteer-run and free time to dedicate to Social.coop comes spottily and in batches for most of us; for example I have time to work on server setup this weekend but I'm unsure about the next one. Then again, the fact that some TWG members voted against the proposal shows that I misinterpreted our level of alignment on the plan going forward.

If I wrote the original post again, I would probably make it only about getting one new server to start replacing old Runko as suggested by @Bo Jeanes above.

I would love to see this proposal coupled with a more comprehensive rebudgeting, such as what the Finance Working Group has been asking for weeks. This way, also, the TWG could make calls like this one its own, within its internal budget.

For instance, here I suggested we move toward something like £250 each for TWG and CWG to be distributed as each group decides. (The downside of making both equal is that TWG would spans most on payments for services, as opposed to labor, but TWG currently does not pay for labor at all.)

Could we reframe this proposal more in terms of a unified TWG monthly budget?

Sure, that sounds good -- that was my intention by having the primary framing be about monthly infrastructure spend regardless of the new servers use case, but the example at hand and the long term policy got mixed up a bit in the proposal.

I'm a bit wary of 'coupling', though -- while we do all this, will we be waiting for convergence and continuing to rely on old Runko? My hunch is that I should go ahead and get one server now and just pay for one year of hosting myself, honestly. If that doesn't make sense, I am happy to commit to paying for any costs until we turn it back down if the community decides against getting it, or against setting a TWG budget so we can pay for it.

If it feels urgent, I think go ahead. But I feel confident we can establish an appropriate budget to ensure you get paid back, and that this sort of thing isn't necessary in the future.

Yeah, you're right. I think the urgency isn't here (or, if it is, it is less present than two weeks ago before @Ian Smith upgraded the instance and got rid of a lot of tech debt). I think I was being a bit irrational because I was frustrated by what I perceived as a roadblock to the work I was hoping to do this weekend :)

I'll start that other proposal.

Thank you for your feedback! Done :) The new proposal is:

https://www.loomio.com/p/PrfjuJAv/proposal-assign-a-250-monthly-discretionary-budget-to-the-tech-working-group

Pretty Irrelevant, but what will become of Runko should this proposal pass?

This was answered in the original post actually:

Once they are set up, we could likely retire Runko (with their aging hardware) and save those ~40€/mo if we want that -- or replace it with newer hardware at a similar price, keeping our monthly server spend in the €150 range.

I'm unfamiliar with the hardware in use, but I imagine it is still relatively powerful. What does "retirement" entail? I would love to see it donated to a university, TWG, charity, or even community member instead of turning it into e-waste

We don’t own the server. It’s rented.

Ah, that's the clarification I was missing :)

It is great to see so much thought and work being put into making social.coop better and more sustainable. Because we link tech and finance to community and try to cultivate a democratic culture, it's important that the process of deliberation and decision-making be democratic, even when that seems (or is) inefficient. My suggestion is that the TWG take all the feedback so far and make a new proposal (or more than one, if it seems better to separate issues) that we can then discuss and vote on.

One proposal could be to dedicate a specific budget to each WG for the members of those teams to use as they see fit. (That assumes that WGs are elected by the members, even if they remain open to other members to participate.)

Another proposal could be to make a one-off purchase of a new server(s).

A third one could be to task a couple of people to talk with Meet.Coop, MayFirst.coop, Cloud.coop... about possible shared hosting and other tech collaboration.

If we want tech to be democratized, we have to step away from the move fast and break things approach, and integrate deliberation and discussion -- which is what we are doing here. If you are in a hurry, take the long way, as they say in Japan.

I think holding hard assets like a server is probably not best for Social.coop. I would advocate a combination of proposals 1 and 2 here: Set a fixed and generous tech budget, and encourage the TWG to seek resources through partnering with tech co-ops.

I am a MayFirst.org board member and would be happy to relay any info or bring questions to the MFMT group. We are working to set up some skill share sessions delving into the free software tools we offer to our members and the public - Jitsi with interpretation features, NextCloud etc.

Deliberation is a good approach. Agaric and some friends are in active discussions on a democratic voting platform - http://visionsunite.org join us in these formative talks every week on Thursdays at 3pm ET on CommunityBridge in the show and tell room. https://communitybridge.com/show-and-tell the visitor access code: elephant9

I think this proposal, even though it got superseded, renewed much needed conversation about the goals of social.coop and how our system architecture can support them. Thank you!