Reputation
We had a meeting with Linc yesterday about some work he is doing on a reputation engine. I thought I'd document the conversation and leave the notes here for reference.
Background
Linc does a million things but this year he is consolidating all his projects into one theme: collaborative communities. Now he and Samson are working on RepShare: an open source context-dependent reputation engine.
Today's meeting was to figure out how RepShare could work with Loomio, or if not, at least help Loomio think about reputation.
People respond strongly to completion, recognition, mastery. Reputations can be applied to product/thing (review); a person (accruing karma), or trolls (one or two bad karma points = troll).
RepShare seems to be focused more on rating people as opposed actions; Loomio has been thinking more about actions, e.g. 'this comment is useful'.
c.f. Reddit's rating system selects for the most viral content. Slashdot's rating system is nearly perfect in some regards (it does a good job of incentivising good content) but at the expense of being fairly inaccessible due to complexity.
How does Loomio incentivise constructive behaviour? How/why does Loomio want go down the reputation path? With the right reputation engine, Loomio could become known for having particularly wise users/groups, who could take that reputation and reuse it in other contexts.
What behaviour do we want to incentivise/disincentivise?
Incentivise
- Contribution: informative, insightful, useful, helpful
- Wisdom: respectful, diverse perspective, bullshit detector (encourages constructive dissent),
- Distributed facilitation: this user consider the needs of the group over their own interests; users reward other users' individual acts of facilitation;
- Administration: putting discussions in the right place, or cleaning up tags, updating the discussion wiki
- Participation: voting, commenting, # of decisions made; (User was involved in a decision that reached completion.)
- Mastery: people are excited about becoming expert collaborators; the group is satisfied with the outcome of a decision;
- Efficiency, effectiveness, motivation, timeliness (e.g. simply display length of time a discussion has been open)
- Celebration/praise: e.g. Yammer praises seem to always be genuine - provides a system for individuals to reward each other (let the humans do it)
- Adaptive: generic enough to be universal and flexible enough to adapt to different contexts (let the humans interpret)
- Break down real/virtual barrier (Loomio translates online communication into real world action)
- Action: ‘tell’ ratings matched by ‘show’ ratings: talk is good, action is better.
Disincentivise
- bad behaviour (hide/flag for moderation): unhelpful, inflammatory, biased, off-topic contributions
- false scarcity and unproductive competition (doing well in a at the expense of others)
- heroics, ego (think, what is the opposite of a leaderboard?)
- people gaming the system for the sake of getting points i.e. confusing the rating for the thing it is trying to measure
Other thoughts
- Bring privilege to light in a non-confrontational manner (data can be used to highlight structural inequity e.g. consider the difference between '80% of members agree with this proposal' vs. '100% of members who are not young males disagree with this proposal')
- Distinguish between good and bad inactivity. i.e. lack of participation can indicate trust in the group and successful delegation, or alienation/disenfranchisement/etc. One option is to ask the non-active group members how satisfied they are with the outcomes the group is coming to on their behalf.
- Could be used to inform decisions about delegation of authority, e.g. users with X karma points are eligible to represent their group in inter-group discussions.
There's a big opportunity to balance individual and group ratings in a unique way, e.g.:
group AND/OR individual completes a task
group AND/OR individual are recognised and praised
group AND/OR individual develop mastery
Side effects
What are some behaviours that may be inadvertently incentivised by a reputation engine?
- Cult of personality where celebrities have the most power; reputation centralises decision-making influence in a way that is unlikely to be purely meritocratic.
- Reputation score becomes more important than the thing it is trying to measure.
- Reputation becomes so ubiquitous as to force everyone to use it if they want to participate in any online service (destroys anonymity)
How to prevent that?
- Randomisation + fuzziness: filter out bullshit, leaving high-value content then mix it up a bit: tells people that this is a group of things that people have decided are good, vs. as opposed to this is the highest value thing.
- Don’t quantify too strictly - social phenomenon are fuzzy, humans are way better at them than computers or ratings engines. Don't try to over-specify anything that humans do easily.
- Keep some scores privately viewable, i.e. recognize progress without encouraging ego.
- Contextualise the ratings of a given person’s wisdom, i.e. we should trust the experienced engineer in engineering questions, but that doesn't mean we should trust them in education questions.
- Give humans scope to to interpret the ratings as they would like
Would make a good plugin?
This could also make a really nice technical challenge to bundle up and pitch to MOBIE for matched funding
Richard I cannot praise your note-taking and reporting enough!