Fri 7 Nov 2014 4:06AM

OpenAppJS Technical Update

ST Simon Tegg Public Seen by 174

The biggest news in technical land is that Paul Frazee and others have hacked up phoenix a distributed social network built on top of Dominic Tarr’s secure scuttlebutt. A usable version is now ready! This is important for OpenApp because:

  • This is the same system we planned to use for federating instances of openApp groups. We will likely just use phoenix/secure scuttlebutt as a ‘base layer’ for supplying federation functionality to OpenApp.
  • These developers will continue to work on some of the harder technical problems (like local cacheing with distributed databases) that were raised in earlier technical discussions here.
  • They’re also using mercury for the frontend UI like the OpenAppJS team which raises the possibility of closer collaboration between these teams.
  • This is dovetailing nicely with a strategy of ‘open source at scale’ - part of this is to just build on top of the best of the ‘emerging web’, rather than try and reinvent all the things.

To recap, secure scuttlebutt implements a ‘secure gossip protocol’ - peers (in OpenApp a large group like Enspiral) share secure data, i.e. ‘gossip’ with other instances they trust, who in turn ‘gossip’ with others. Data propagates through system without a centralised server. The idea is that OpenApp groups would have access to the public data of other groups without all of the security problems and economics of being centrally hosted.

I’ll be setting up my own local instance this weekend to try it out.

For OpenAppJS team specifically:
Mikey has been working cobudget which is now being tested internally in Enspiral for this month’s budget round!
I’ve been hacking on Person-UI the main component behind craftodex. Initially, I found this a bit frustrating as I spent a lot of time getting used to mercury. Perhaps it was because I would hack on this in the evenings when I have low brain function, or because mercury restricts you a bit to make it easier and more performant in the long run. I only started to ‘get’ mercury a month ago and now I’ve come round to it.

The API is starting to stabilise and I’m looking forward to doing more radical things like adding other entities, and viewing groups ‘as a network’ or ‘as holons’. I’ll comment on some of the more interesting technical details that are going on, in the comments here or in the forthcoming design update.


Bob Haugen Fri 7 Nov 2014 12:05PM

This is exciting progress! I will be grateful for your suffering when it's my turn to learn mercury. Loved the routing#8 conversation!

I can see gossip as a great protocol for value streams.

  • Published by a value network.
  • Like Activity Streams, but about value network activities.
  • Needs, ideas, offers, requests, plans, new recipes, creations, work to do, stuff we need, notices about income and income distribution, etc.

I can't see it much for contract-formation between agents, which is a back-and-forth protocol that needs stronger consistency, and is nobody else's business until the principals agree on something. But maybe I am missing something?


Josef Davies-Coates Fri 7 Nov 2014 3:04PM

Many thanks for another great update @simontegg - keep 'em coming :)

I do wish @tav and @tomsalfield would take the time to engage here too because I know they've done useful relevant thinking and work related to all this stuff.

One thing @tomsalfield mentioned to me the other day when I mentioned the plans for 'people' and 'groups' apps was that 'groups' possibly isn't the way forward and that perhaps 'people' and 'spaces' would be more useful. We didn't actually chat any more about it though so I'm not really sure what he meant/ what the difference is. Hopefully he'll chime in. If not I'll try to remember to ask him and then feed back :)


Bob Haugen Fri 7 Nov 2014 4:10PM

@josefdaviescoates - @tav said something about Ethereum in another Loomio discussion that I can't find now that is what got me thinking about the minuses of blockchains...always hoped for a re-appearance to learn more.

And yeah, would like to understand what @tomsalfield meant by spaces.


Josef Davies-Coates Fri 7 Nov 2014 4:20PM

Hey @bobhaugen it was actually me relaying something @tav had written to me in an e-mail.

Here it is:


Mikey Fri 7 Nov 2014 10:33PM

i'm currently 88c9aa687a076aa530cc18d7921cec1f90f7779ad25915204d272a695072f58a on phoenix, we hang out in #stackvm (node.js mad science), #scuttlebutt (phoenix and secure-scuttlebutt), and #virtualdom (mercury) on freenode irc.


Mikey Fri 7 Nov 2014 10:41PM

another good description of the "secure gossip protocol" is in the secure-scuttlebutt README:

SecureScuttlebutt uses a signed block-chain per identity. Each block points to the previous block, the signing key, and contains a short message and a signature. Every identity has their own block-chain.

Each block-chain is an append-only data structure that can be written to exclusively by the keys' owner. Since the chains are append only, replication is simple, request the chain for that id, since the latest item you know about.

Now, each node is the master of their own feed, so scuttlebutt replication is just like doing many simultaneous master-slave replications.

Since messages are signed, and replication is eventually consistent, it does not matter if A receives B's messages via C, they can verify B's messages offline (i.e, when A is out of contact with B)

@bobhaugen: you could send encrypted messages back and forth through the network until you reach an agreement, then publish the agreement to a wider audience. gossip is how the network distributes messages, it doesn't mean everyone needs to see your messages.


Bob Haugen Sat 8 Nov 2014 12:21AM

@ahdinosaur - I understand that gossip messages can be private. I was thinking more about an elaborate protocol with eventual consistency for a simpler situation that would like consistency right now: binary agreement. So I am ready to believe that I am missing something, since lots of people seem to think that blockchains are the way to go.

This is the human-level protocol I want: http://conversationsforaction.com/cfa-playground

And I think I would like all of the messages in that conversation to be added to the same append-only document, signed every time. Unbroken record of the whole economic interaction.

If that can fit within gossip and not confuse the participants, it's all ok with me.


Mikey Sat 8 Nov 2014 1:04AM

@bobhaugen cool, yeah as far as i know, the Conversations for Action protocol would work great within the per-agent message feeds being described. i'm quite excited to make this happen, it will probably involve various message types relating to the various stages of the conversation. would it be possible for you to document the CfA protocol through descriptions of message-passing (e.g. "at this stage of the conversation, here are the possible messages to send and the respective next stages")?


Bob Haugen Sat 8 Nov 2014 1:16AM

@ahdinosaur I'll give it a shot next week some time.


Bob Haugen Sun 9 Nov 2014 1:45PM

I started a CfA discussion here. More to come after breakfast.


Bob Haugen Mon 8 Jun 2015 2:04PM

@ahdinosaur - just coming back to this conversation after 7 months (last message above). We got somebody interested in some NRP UI experiments, who has some JS experience but is also eager to learn.

The state of the JS client-side frameworks apparently continues to warp. If I understand correctly, you fell in love with mercury, and then fell out of love and went back to React. Is that correct? Can you say more about why?

(I was just reading this, which made sense to me, but then I don't actually do this kind of work yet:
It's in the mercury camp.)


Mikey Tue 9 Jun 2015 2:39AM

@bobhaugen sweet. you're correct in that there are no clear winners in client-side frameworks, we're only using React because it's easy to use, well-supported, and has an active ecosystem of re-usable components, mercury is still awesome but requires more work to use, neither is perfect. however i reckon there are clear winners in the patterns that are being adopted, of most interest to me are "functional reactive programming" and "kappa architecture". only time will tell what actually happens, but maybe this helps share where i'm at.


Rob Guthrie Tue 9 Jun 2015 3:01AM

I'm in awe of Omniscient right now.. I think it looks really great without being too different (ie: written in ClojureScript)


Bob Haugen Tue 9 Jun 2015 10:55AM

@robertguthrie -if I understand this correctly, Omniscient rides on React?

So, does that give you access to those re-usable components that @ahdinosaur talked about? Plus immutable data? Or is it off in its own universe?

And Mikey, what are a couple of those re-usable components that you get the most benefit from?

I'm interested in where this stuff is going. Seems like each of the hot js frameworks evolves in a succession, where they fix what they thought their predecessor did wrong. The problem is the succession may have no end...so you pick something, and now here comes something better (maybe)...


Mikey Wed 10 Jun 2015 2:58AM

regarding Omniscient, yes it lets you use re-usable React components, plus it uses Facebook's immutable data structures (including the respective cursors library), so is in the same universe.

what's nice about active ecosystems is having access to everything you can imagine, to use or learn from, similar to the Node.js module ecosystem in general. i can't single out individual components that provide me with sheer awesome value, it's the ecosystem in general that does.

i agree that hot frameworks will continue to succeed and die, that's why i'm much more interested in learning the underlying patterns that won't become obsolete in a few months. also, i prefer using libraries that implement a focused scope, as they have a longer lifespan, and are much easier to swap in and out as conditions change.

sorry about the urls, markdown seems to work differently in Loomio's new angular interface.


Bob Haugen Wed 10 Jun 2015 10:38AM

"markdown seems to work differently in Loomio's new angular interface."

Sure does. Now my plain urls that worked in the previous interface don't work in this one, and yours that did not work in the previous IF do work in this one.