Loomio

Proposal: increase our Tech Working Group monthly infrastructure spend to ~€150

Flancian PublicSeen by 159

Update (2022-11-25): this proposal is superseded by https://www.loomio.com/p/PrfjuJAv/proposal-assign-a-250-monthly-discretionary-budget-to-the-tech-working-group, which takes into account feedback by the community.

Hi there!

We are discussing scalability and reliability strategy in the Tech Working Group this week, after the great work by @protean which brought us to the latest Mastodon version safely.

We have located two servers in Hetzner under auction that would let us augment and replace Runko, our current server with hardware from ~2013. To lease them would signify an extra expense of <€100 monthly:

Once they are set up, we could likely retire Runko (with their aging hardware) and save those ~40€/mo if we want that -- or replace it with newer hardware at a similar price, keeping our monthly server spend in the €150 range.

@Nathan Schneider, we discussed having a discretionary budget elsewhere -- do you think this proposal needs more details and a consultation with the community or do you feel comfortable approving it? Moving fast would enable us to start working on balancing the current load better, and prepare us to grow the instance further if we want to do that and the opportunity arises. But I'm unsure what the right way forward it.

@Ian Smith @Bo Jeanes among others from the TWG reviewed so far, review of the rest of the working groups and the wider community is of course very welcome :)

Calix

And I would personally support replacing the server 1:1; my concerns are about adding architectural complexity (specifically I saw Wireguard, and Docker Swarm, mentioned) which will reduce the pool of potential folks who could help with TWG work – which seems especially dangerous given that the TWG has been down to 0-to-1 people before.

Bo Jeanes

That was just my speculation about why it was considered urgent, but I think @Flancian has since further explained why below.

Bo Jeanes

Yep very fair.

FWIW: Docker Swarm is not a significant departure from our current strategy, as it still is essentially a docker-compose.yml-based workflow. I don't think Wireguard is necessary and I too would be concerned about adding that into the mix for the reasons you've expressed.

Personally, I only want us to separate DB and app onto two machines, even if we changed nothing else.

Calix

I'm pretty familiar with Docker Swarm – I helped implement a project entirely based on it 😅 – and I'd disagree that it's "not a significant departure": using it requires learning about the "stack" and "service" Docker concepts (and related management commands), potentially adjusting to a different mental model because of the potential decoupling of where the compose file lives vs where the stack is deployed, and the UI is very gory (`docker-compose restart` vs `docker service scale foo_bar=0 && docker service scale foo_bar=1`, `docker-compose stop` vs `docker stack rm`).

I think swarm is better, for sure, but I think it's worth asking who in TWG knows it, or is willing to learn, before introducing it.

jonny

idk how to vote after reading the whole thread! gave some context above!

  • I support the discretion of our working groups!

  • if there is some time-critical auction that nets us some needed hardware in a time of budgetary surplus I'm in favor of them making the buy and briefing afterwards with invitation to clarify spending process!

  • I do think we should revise the urgent vote process to need to include some description about why the vote needs to be urgent, because I think process concerns were the major ones reading the above thread!

  • again thank you tech ops/tech working group for bringing this item and being on the lookout for good hardware! appreciate your work !!!!!

Ian Smith

In addition to the comments I made in my vote, there are some other things making me think this should get better consensus before moving forward.

I think there is a larger discussion to be had around how the co-op should scale. I do not personally that mega coops or mega instances are good for democratic governance. Scaling servers now would moot that discussion as there would no longer be an external pressure (server capacity) to force it.

Since the latest update, we are seeing more headroom on the server due to bug fixes and improvements in the underlying libraries, so this is not as urgent as it would have been a week ago.

Important:

There is a caveat here: The instance has a lot of inactive accounts. It's like a 2:1 ratio of inactive to active. If a large number of those accounts suddenly became active again this would change the urgency.

Evan Boehs

I think these are important considerations, but not relevant at this time. The current server is ageing, and redundancy is always a good thing. What happens if good ol' Runko hits the sack?

Evan Boehs

I believe my last comment did not send, my apologies if for some reason the comment is just not visible on my screen.

I think this discussion is important to have, but I don't think it's relevant to this proposal. Runko is ageing, and I fear for what would happen if they suddenly hit the sack. New servers and redundancy is always good, regardless of intended scale.

Darren

As theres more TWG folks now chiming in saying they feel the process needs more time Im going to give it the full week and remove the URGENT🚨

Joseph Andriano

Is it possible to repropose this with more info as a formal recommendation voted out of the TWG? I’m seeing conflicting discussion about whether the TWG thinks this is a good idea. I’d like it if we could have a proposal with the various options/costs and vote on that. I’ll admit I’m pretty confused at this point.

Flancian

Hi all! Thank you all for your comments. Thank you particularly Darren for starting the poll!

Although I agree that making it not urgent seems like a reasonable thing here based on community feedback, I am (full disclosure) personally a bit disappointed by the fact that this will make it so that we won't have the server(s) available until November 30 and thus we will not be able to start working on them until December. The TWG is volunteer-run and free time to dedicate to Social.coop comes spottily and in batches for most of us; for example I have time to work on server setup this weekend but I'm unsure about the next one. Then again, the fact that some TWG members voted against the proposal shows that I misinterpreted our level of alignment on the plan going forward.

If I wrote the original post again, I would probably make it only about getting one new server to start replacing old Runko as suggested by @Bo Jeanes above.

Nathan Schneider

I would love to see this proposal coupled with a more comprehensive rebudgeting, such as what the Finance Working Group has been asking for weeks. This way, also, the TWG could make calls like this one its own, within its internal budget.

For instance, here I suggested we move toward something like £250 each for TWG and CWG to be distributed as each group decides. (The downside of making both equal is that TWG would spans most on payments for services, as opposed to labor, but TWG currently does not pay for labor at all.)

Could we reframe this proposal more in terms of a unified TWG monthly budget?

Flancian

Sure, that sounds good -- that was my intention by having the primary framing be about monthly infrastructure spend regardless of the new servers use case, but the example at hand and the long term policy got mixed up a bit in the proposal.

I'm a bit wary of 'coupling', though -- while we do all this, will we be waiting for convergence and continuing to rely on old Runko? My hunch is that I should go ahead and get one server now and just pay for one year of hosting myself, honestly. If that doesn't make sense, I am happy to commit to paying for any costs until we turn it back down if the community decides against getting it, or against setting a TWG budget so we can pay for it.

Nathan Schneider

If it feels urgent, I think go ahead. But I feel confident we can establish an appropriate budget to ensure you get paid back, and that this sort of thing isn't necessary in the future.

Flancian

Yeah, you're right. I think the urgency isn't here (or, if it is, it is less present than two weeks ago before @Ian Smith upgraded the instance and got rid of a lot of tech debt). I think I was being a bit irrational because I was frustrated by what I perceived as a roadblock to the work I was hoping to do this weekend :)

I'll start that other proposal.

Evan Boehs

Pretty Irrelevant, but what will become of Runko should this proposal pass?

Bo Jeanes

This was answered in the original post actually:

Once they are set up, we could likely retire Runko (with their aging hardware) and save those ~40€/mo if we want that -- or replace it with newer hardware at a similar price, keeping our monthly server spend in the €150 range.

Evan Boehs

I'm unfamiliar with the hardware in use, but I imagine it is still relatively powerful. What does "retirement" entail? I would love to see it donated to a university, TWG, charity, or even community member instead of turning it into e-waste

Bo Jeanes

We don’t own the server. It’s rented.

Evan Boehs

Ah, that's the clarification I was missing :)

Matt Noyes

It is great to see so much thought and work being put into making social.coop better and more sustainable. Because we link tech and finance to community and try to cultivate a democratic culture, it's important that the process of deliberation and decision-making be democratic, even when that seems (or is) inefficient. My suggestion is that the TWG take all the feedback so far and make a new proposal (or more than one, if it seems better to separate issues) that we can then discuss and vote on.

One proposal could be to dedicate a specific budget to each WG for the members of those teams to use as they see fit. (That assumes that WGs are elected by the members, even if they remain open to other members to participate.)

Another proposal could be to make a one-off purchase of a new server(s).

A third one could be to task a couple of people to talk with Meet.Coop, MayFirst.coop, Cloud.coop... about possible shared hosting and other tech collaboration.

If we want tech to be democratized, we have to step away from the move fast and break things approach, and integrate deliberation and discussion -- which is what we are doing here. If you are in a hurry, take the long way, as they say in Japan.

Nathan Schneider

I think holding hard assets like a server is probably not best for Social.coop. I would advocate a combination of proposals 1 and 2 here: Set a fixed and generous tech budget, and encourage the TWG to seek resources through partnering with tech co-ops.

freescholar

I am a MayFirst.org board member and would be happy to relay any info or bring questions to the MFMT group. We are working to set up some skill share sessions delving into the free software tools we offer to our members and the public - Jitsi with interpretation features, NextCloud etc.

Deliberation is a good approach. Agaric and some friends are in active discussions on a democratic voting platform - http://visionsunite.org join us in these formative talks every week on Thursdays at 3pm ET on CommunityBridge in the show and tell room. https://communitybridge.com/show-and-tell the visitor access code: elephant9

Ed Summers @edsu

I think this proposal, even though it got superseded, renewed much needed conversation about the goals of social.coop and how our system architecture can support them. Thank you!