Scoring system

From Metagovernment - Government of, by, and for all the people

Jump to: navigation, search

The scoring system is the logic used by Metascore software to evaluate proposals and perhaps also a users reputation system.

This is a preliminary description of how the scoring system will work. As Metascore develops, this may change substantially.

Contents

Proposal scoring

Proposals and debate

Anyone can make a proposal within a community, and for each proposal there is a debate. Each debate consists of posts by users, either in response directly to a proposal, or in response to other posts. Each post may be rated by any user as they please. Presentation of posts can be filtered by different factors, among them the rating they are given by the community. There is no limit on the number of proposals for any particular issue, nor on the complexity or simplicity of a proposal.

How proposals become policies

Each proposal is scored by users, if they wish to do so. Users may assign positive or negative scores to proposals, and may assign synthesis scores (see below). A proposal becomes and remains remains policy for as long as it can maintain a consensus score and is not contradicted by a law from a higher government (as decided by the courts of that higher government). In order to prevent overly-frequent transitions in policies, once a policy attains a consensus score, the threshold for removing the consensus score is raised somewhat (Schmitt trigger).

Synthesis scoring

In addition to scoring proposals, users may also provide a synthesis score in relation to two or more other proposals. The synthesis score reflects how effectively (in their opinion) the proposal synthesizes the other proposals, providing a proposal which they feel addresses the conflicting concerns of the proponents of the other proposals. In the consensus scoring, below, a synthesizing proposal effectively "steals" points from the proposals that it synthesizes.

Consensus scoring

A consensus score is a formula consisting of the positive and negative scores assigned to a proposal, and any synthesis scores applied to or against that proposal. Each of these scores is weighted by each user casting the scores. The formula for determining the consensus score will be crafted such that a proposal which does not have significant support from the community will not become a policy. To become a policy, a proposal must have a significantly higher score for it than against it, and must not be significantly synthesized by another proposal.

User scoring

Note that there has been significant dispute over the utility of user scoring. We will accommodate user scoring in early versions of Metascore and see what sort of effect it has when used in real-world systems.

User score

Each user has an overall sum score, which starts with a base score and is modified by ratings of one's posts, ratings of one's self, one's ratings of others, and various weighting factors. A user who registers as an individual person (as opposed to an anonymous e-mail address) is granted a very large multiplier bonus. Validated registration will not be possible until guiding nonprofit organizations are created and funded.

Post score

Each message one posts is scored by readers of the post who wish to score it. Those scores are then weighted by the reader's own overall score, and the reader's score in that topic and region.

Community factor

Each user has a score factor for each community. Being a registered member of a community gives one a large bonus in that community. Each post one makes in a community gives one a small bonus in that community, weighted by ratings of one's posts in that community.

General features of scores

User rating is essentially the quantification of trust. Similarly, item rating is essentially the quantification of support. Interest (in contrast to support) can be calculated by summing the number of users watching the debate. Ratings may have the following properties:

Ratings are static

Whenever anything is scored, all variables are calculated by their present values. A later change in a value does not affect that score. For example, if a reader with a low score rates a post, then later that reader accrues a much higher score; their rating of that post does not change with their own score. They can, however, re-rate a post, which would then override their previous rating.

Time deprecation

Each score degrades over time such that one tenth of the original score is subtracted each year, and after ten years, the score is removed entirely. The math for this subtraction will be constructed such that a diminishing score does not negatively impact a rating; only that it counts less and less as time progresses. Users may re-cast any score, which resets this deprecation.




Discussion

Procedure for discussions.

Resolution: None.

Deadline: None.

Issue

Title: Metascore's scoring system

Author: Metagovernment

(Explanation of the problem.)

Comments

  • This very issue exposes a weakness in all of the current proposals; regardless of whether it's weighted or not, each proposes a simple yes or no. However, not all issues (notably including this very one) can simply be relegated to yes or no. In such an example (which may happen more often than not), we might instead allow voters to rank the 9 proposals, or their favorite three. Or, in another hypothetical, they might answer "Of the available US$50000, how much should we allocate to purchasing the new playground equipment?" and vote with their dollars. --Aaronwinborn 02:04, 7 May 2009 (UTC)
  • The problem with ranking available proposals is that they have to first be organized, ie: there needs to be some logical grouping of proposals within which the proposals are comparable. That kind of structure lends itself very easily to this kind of issue (what type of scoring system) but not very well to a more general question (should we raise taxes). It's possible to organize proposals by tag, and we could even have an 'issue' or 'problem' tag that organizes proposals by the problem meant to be solved, but, as with many things, each proposal will likely be an attempt to solve several problems at once (perhaps to their detriment). And, if a proposal has a very high rating with respect to one problem but a low rating with respect to another, it may be hard to tease out which components of a proposal are positive and which can be thrown. Leading to the possibility of someone taking a proposal apart into pieces (all related to the same question) in which a user may rate variations of the same proposal very similarly. It might be useful to impose some kind of structure on the proposals -- like, perhaps, have proposal-like objects that serve no function other than to identify a problem or issue. Users can vote on these 'issues' as to their relative importance. This would have the benefit of soliciting proposals with a purpose, organizing content, and allowing people to brainstorm collectively before a good solution is known. It would also facilitate a greater variety of scoring mechanisms, like that Aaron mentions above. At the same time, not every proposal would lend itself to this kind of structure, and so the structure may become limiting. Thoughts? --Jacki Buros 20:04, 11 June 2009 (UTC)

Proposal 1

Title: Nuanced, recursive scoring

Author: Ed Pastore

Users and proposals would each have a score in relation to each tag assigned to them. The score is an amalgam of the scores given to the proposal or user by other users. This is the original mechanism proposed on the Metagovernment website.

Users can assign a score of anywhere from -10 to 10 (where 0 is equivalent to a vote of "present" for quorum purposes).

Possible advantages and disadvantages of this system are presented below.

Builds consensus, scales well

In a system where there can be many, many competing proposals all attempting to address a single issue, users need a way to give nuanced support for proposals. If users are restricted to only voting for or against each proposal, then they either have to only vote for the proposals they feel are the very best, or they have to vote for many proposals, including ones they do not really like. Both scenarios create barriers to consensus.

User scoring allows increased participation

If users themselves do not have scores, then again people are restricted to on/off states. People either are fully allowed to participate in a community, or may not participate at all.

If instead users have nuanced user scores, then everyone can be allowed to participate in every discussion, and the user scores can act as a mechanism for mediating participation. Otherwise, how would a convicted criminal be allowed to participate in a community? Do they have to be completely excluded, or can they just be given a low score? Likewise, how can small children be allowed to participate in their community? Do we have to use an arbitrary cutoff like the age of 18? With a nuanced user score, absolutely everyone can participate, and the scoring system provides a natural balance to that participating.

Comments

  • Though a problem with a -10/+10 score would be does a final talley of 1 pass a measure where 10,000,000 voted -1 and 5,000,000 voted +2 with the tie breaker of +1? --Aaronwinborn 01:47, 7 May 2009 (UTC)
This issue is less problematic as the trust network becomes interconnected/robust and where voting is mapped to level of personal commitment (see logarithmic suggestion below). But at some level, with whatever voting system, there is no getting around the wildly swinging balance at the center of a deeply polarized issue. (Such behavior should be an automatic flag to send the proposal to a synthesis area for further review.) Marcos 02:11, 20 October 2009 (UTC)
  • Suggest logarithmic voting is more natural (and scales better) than linear (although it should be noted that many voting systems often describe and accompany their linear vote number with a more-or-less logarithmic text description (ex. 5) The best! 4) I like it 3) unsure 2) don't like 1) Yuck!)). Use of base 4 maps well with the body. 5 degrees (rather than 10) is likely sufficient for most everything (4^0=1, 4^1=4, 4^2=16, 4^3=64, 4^4=256) along with their negative counterparts.
    • Logarithmic is attuned to body. From the thumb (+1); x4=to the hand, x4=limbs, x4=to body, x4=to family.
    • Easy iconography. (need pics:) Thumbs-up, Hand, Limbs (icon with shovel?), Body, Family
    • Log votes can translate easily into amount of commitment:
      • Thumbs up: simple assent (seconds of commitment)
      • Hand up: Willing to contribute, edit (minutes)
      • Limbs: Willing to pitch in and work at it, make initial drafts, etc (hours)
      • Body: Important enough to put life on hold for it (days)
      • Family: Puttings one's life and value system on the line.
    • Easier accountability: mapping to body gives users a better sense of what is (or will be) expected.
    • Natural for social network: Logarithmic voting maps well to how we view our social trust networks (met once, friendly, ..., committed partnership)
    • Better view of the landscape: Logarithmic voting makes it easy to search for important issues (how many have put their life on the line for this proposal?)
    • Scales well.
  • Suggest change title to: Nuanced, recursive, logarithmic scoring (Redundancy intentional)
  • Note: the vast majority of issues will be handled sufficiently with a simple thumbs-up and thumbs-down vote. The rest are really for editors and generally committed persons or a deeply personal issue to/for an otherwise general user. (So you see this too (i.e. participation) is naturally logarithmic).

Do you support this proposal?

  • Yes Marcos 02:11, 20 October 2009 (UTC)

Proposal 2

Title: +/- or (yes/no) scoring

Author: Jacki Buros

Users can assign a + or - to a proposal.

The problem with having too wide a range is that every user defines her/his threshold of acceptability—my 3 may mean I approve, someone else's 3 may mean no, but better than the rest. Even if we define what '3' means, as you suggest below, users are prone to ignore these recommendations. I say we keep it simple: yes, no, not sure. Anything more nuanced warrants a comment, or too much complexity (like ranking among yes and no responses).

In market research, where I have some experience, surveys are typically given a finer gradation of possible response choices where we want their response to be more heavily influenced by the user's gut response. The fewer choices given, the more we force them to come to a decision, i.e. think about their response and thus the response becomes more heavily influenced by rational thought. In this case, contrary to what was posted by others on the list, I think having a less finely graded system (such as yes/no) will be more likely to foster consensus than having a -10/+10 or -1/+1 nearly-continuous grading system where users can pick a number more or less at random, thus facilitating impulsive decisions. Having a less finely graded scoring system will also force users to voice their more nuanced opinions to qualify their 'yes' or 'no' instead of giving a score of 5.7 which they may feel expresses their opinion without their having to expend the effort to comment specifically.

Comments

  • I think that may be true in limited scenarios, but we are talking about something that many people would be participating in frequently and regularly. After regularly using a nuanced scoring system for even a few days, people would start to make nuanced scores. Systems like IMDB seem to do very well with a 10-point scale, because their regular users are familiar with 10-point rating. Example: http://www.imdb.com/title/tt0411061/ratings — Ed Pastore
  • Yes/No is basically the same as the apache system, with anything between +1 and -1 not being counted, but just to express sentiment. We could just count "yes" and "no", that's probably the least confusing. — Manuel Barkhau
  • I very much like the three-option system here (+, -, no vote). People can still express fractions without it leading away from consensus-building, and on the other hand it's very similar to the for/against votes used in gov't today (less culture shock). I'm willing to + it with the caveat that we should NOT consider Resolution 0 sacred in any way, even if it passes. Some of the stuff ("active in the mailing list in the past month") will change as our system of interaction changes, and some other stuff (72 hours, 90%) are ballparks that will probably bear refinement. Humphrey 01:43, 19 June 2008 (UTC)

Do you support this proposal?

Proposal 3

Title: 1-5 scoring

Author: Paul Codd MacDonald

Users can assign a score from 1 to 5, such as in a 5-star system.

I propose a slightly less geeky scale than the Apache project (which we discussed in a previous thread): that people vote by expressing an integer score from -10 to 10.

Much less geeky would be a common 5 star rating. When do we really need to know more than *no way, **has flaws, ***acceptable, ****worthwhile, *****awesome - or words to that effect. We can retain the maths you propose but attach -100 to *, +10 to ***** etc.

I propose that the system be flexible enough to allow a bunch of geeks to over-ride the 5 star system to be able to access more subtle levels of control but my advice is by default - KEEP IT SIMPLE.

And that unless there is compelling reason to do otherwise, scores are accepted for 72 hours. At the end of 72 hours, the scores are simply added together, and a positive score means approval.

I propose that communities are given the choice to allow or disallow time limited resolutions. In my own view they can sometimes be very dangerous and would need accompanied by safeguards to make sure decisions are quorate. There are a number of important US laws in force today that were passed by VERY small numbers of elected members while the rest of the house was not there! By default I would prefer if decisions were made once "critical mass" is reached. What critical mass actually is can be decided by each community and might be different for different kinds of resolutions.

To reproduce Apache's veto, I also propose that a vote of -10 be tallied as -100.

In some cases we may need some other more flexible and scaleable maths, but this will get us started.

People can express a score with an explanation, or simply a score with nothing else. If anyone votes -10, then the resolution is stalled until that person can be convinced to change their mind (probably by the group changing the resolution in some way). A change to the resolution resets the 72-hour clock. Unless they say otherwise in their proposal, let us automatically assume that an author of a resolution gives it a +10.

I also agree that a veto needs to have an explanation, even a holding explanation like "My research results are coming in against this, I'll collate them and post my ideas tomorrow."

Comments

  • I am not a mathematician and am not sure which would be preferable: a scale that ranges from negative to positive or a scale that ranges from a low number to a high one. But by making adjusters for the stars, I think what you are proposing comes out to something like -100, -5, 0, 5, 10. But ***=0 doesn't seem to make sense, does it? — Ed Pastore
Despite the natural selection of simply ignoring uninteresting proposals and letting them die via "starvation", negative scores are desirable in instances where there is willful vandalism or trolling--such edits don't simply have "no value" they have anti-value. It's like having a parasite within a living cell--one shouldn't simply ignore it, expecting it to go away when it is actively seeking to destroy; however, the preceding must be viewed in the context that at some point there can be no rejection simply because there there is a point where there is no such thing as "I do not exist", at such point a higher level of authority must be called and the individual must be addressed. cf. Wittgenstein (namely "All that exists is true" -- the very act of rejection strengthens the existence of Other), and (separately) conflict resolution.
One perhaps should not be able to negative vote more than the number of points they have at their disposal and the points that one gains with the negative score should be taken from the vandal's balance. While this might seem to promote vindictive retaliation, ultimately the vandal will get voted downward in the trust network (scaled or factored away) so that his/her votes will have little effect (the suppression system).
  • This system is confusing to me, and not at all intuitive. Considering that in the end, a motion must pass or fail, it's too fuzzy to determine pass or fail if it has an average of, say, 2.7 stars. Would that fail because it's less than 3 (the middle star), or pass because it's greater than 2.5 (half of 5)? I think most people would understand a simple yes/no, or a positive or negative score. Ratings are great for deciding where to eat for dinner; probably not so good for legislation. --Aaronwinborn 01:46, 7 May 2009 (UTC)

Do you support this proposal?

Proposal 4

Title: Regular voting

Author: Ben Ochoa

Once a proposal is approved for a vote, everyone in the community who is registered to vote can vote for or against the proposal within a time limit. The proposal should only be allowed to become law if there is a quorum.

I think the basic process for passing a law should follow the system set up in Robert's Rules of Order:

  1. The law is introduced.
  2. A citizen moves for a vote on the proposed law.
  3. A different citizen seconds the law.
  4. The community has a limited amount of time to vote on the law. Most laws will fail because of a lack of quorum.

This system will generate a lot of noise. So, citizens will need to be able to list proposed laws according to when they expire, total votes, total views, etc. There could also be committees to vet laws before they are put to a vote by the full board. The system will require very active participation from the citizens to encourage people to vote for laws.

I think anything more complicated than basic voting will be very difficult to explain to the public and even more difficult to implement.

Comments

(Comments to the proposal here.)

Do you support this proposal?

Proposal 5

Title: Binary consensus-based rating

Author: AurSaraf

  • A resolution candidate is proposed in the wiki and can be rated positively (+) or negatively (-) by any member, by signing (wiki syntax ~~~~) in the appropriate subsection.
  • It is allowed to move your rating but not to rate the same candidate twice at the same time.
  • A - rating must be accompanied by an explanation in the discussion page.

If, at any point in time,

  • 72 hours have already passed since a resolution was posted,
  • 90% of the users active in the mailing list in the past month have rated that resolution +,
  • no user active in the mailing list in the past month had rated it -,
  • an email is sent to the mailing list with the subject "[resolution] <a title for the resolution>" and the resolution quoted in the message body,
  • no other resolution in the same thread was accepted,

then the resolution is accepted.

For convenience, proposed resolutions must be complete—they cannot reference other resolutions by name but must instead quote anything that they wish to incorporate.

Comments

(Comments to the proposal here.)

Do you support this proposal?

Proposal 6

Title: ...

Author: Quickfitter

My proposal is philosophical rather than technical. The scoring system should apply in the first instance to people, based on the relationship between them. A similar structure may then be carried over to the rating of their proposals.

Since all of our knowledge and beliefs arise anyhow from a consideration of how much we trust our sources of information, the scoring system should aim to make the process of assessing trust explicit and to provide it with a numerical structure.

My suggestion is that each person who interacts in any way with the site be asked to rate each person they interact with on a 15 point scale. The choice of the number 15 allows for the possibility of a quick (shallow), a normal (neutral) or a deep (complete) rating to be given. If the user chooses to give a quick rating, they simply check either 'negative' (band 1-5) 'positive' (band 6-10) or 'extremely positive' (band 11-15), giving a score out of 3. If a normal rating is chosen, it is given on a scale of 1 to 5 (each corresponding to a 3 point band on the 15 point scale), and a full rating is on a scale of 1 to 15.

Each user enters with a rating of 5 out of 15. A threshold is set for moving a user up or down one point. Once many highly rated individuals have recently rated a person lower than their current setting, they will move down one point. Once many highly rated individuals have recently rated a person higher than their current setting, they will move up one point. Gaining or losing a point takes a reasonably long time and a considerable body of consensus.

Everyone's comments and ratings are weighted, so that the opinion of someone with a rating of 1-3 will be almost completely irrelevant, while people with a rating of 9-10 will be so influential they will effectively be leaders (higher aggregate personal ratings will probably not come out as possible in practice). Likewise, more weight is given to ratings out of 15 (on the assumption that people who take the trouble to give them care more about what they say with their rating) than to ratings out of 5, and less still to ratings out of 3.

Any person can introduce a person personally known to them: if the introducer's rating is higher than 5, and they rate the introducee very highly, it might be possible for the introducee to start with a higher rating.

This mechanism allows each individual to keep and update a personal rating score towards each of the other people they care about, and to aggregate those scores for each individual without any changes being too sudden or radical. The scores represent the amount of respect people are prepared to accord one another, and correspond roughly to these considerations:

  1. Will not kill
  2. Will allow air, water and food
  3. Will allow shelter, clothes, bed
  4. Will not physically abuse or deliberately harm
  5. Will treat neutrally, and allow freedom of movement and work
  6. Will respect beliefs and allow freedom of expression
  7. Will respect property and not attempt to compete directly for it
  8. Constructive and basically friendly relationship: enough trust to trade fairly
  9. Strong goodwill: may enter partnership or alliance
  10. Solid friendship and complete trust
  11. Fondness and a desire to share
  12. Will sacrifice own interest to a considerable extent to contribute to this person
  13. This person could be a lover or life partner, and inspires devotion and faithfulness
  14. Desire to let this person into every aspect of private and mental life: would die at their request
  15. Fervent insistence that this person is greatly superior to others and all should defer to them

This scale articulates a lot of detail at the extremes. Very few people, obviously, will ever have an aggregate rating above about 10. Should this actually become a widespread system of government, it will automatically mean that anyone with an aggregate rating below 4 is likely to be badly treated - this could be the basis for developing this as a system of justice as well as a system of government: the apparent dangers of mob rule should be taken care of by the weighting towards the views of those who are most respected and admired. It should never be forgotten, though, that Hitler was greatly respected and admired by most of the German community in the 1930s. Once any detailed system is set up it should be tested with a simulation of 1930s Germany to see how much influence a Hitler could have.

Ratings on this five-point scale will correspond to points 2, 5, 8, 11 & 14 above, so they should be labelled '1 dislike', '2 neutral', '3 friend', '4 close friend' '5 worship'.

Individual proposals can be treated analogously. All proposals start on 5. Proposals introduced by people with very high ratings might be allowed to come in higher. Anyone who wants to comment on a proposal can contribute to the discussion - in order to do so they must first rate it as it stands (given all the discussion to date). People can also just rate without commenting. They can offer a rating out of 3, 5 or 15, as above.

Just as it takes a fair amount of comment from influential (and therefore admired) people to move any individual person's rating off 5 points, most proposals will be little commented on, will remain on a neutral 5, and will therefore be ignored.

As a proposal's rating moves up it is brought more to the attention of wider subsets of the community, and is taken more seriously. Influential people's voices can always move it back down. A threshold is set - 12, maybe - above which any proposal is enacted (no matter how long it may have taken to get there) because its having that rating means that pretty much everyone who counts thinks that it's a really good idea.

Comments

  • My preference would be that each proposal would be rated based solely on its own merit. The moment you start directly rating people, you introduce a whole set of politcs and probably a class system. If you make good proposals, they should speak for themselves and not have to rely on your social position. Even further, this enables abuse and corruption, since you now only have to influence very few people in order to further your objectives. Today we call this lobbying and I hope we can reduce it with a system where people give scores without taking into account the standing of the submitter, be it positive or negative.--Mbarkhau 20:37, 29 March 2009 (UTC)
  • We have talked about several different forms of user scoring, but for the moment we are focusing on building the basic system, then seeing what people do with it. Personally I do think that user scoring will be necessary, but the programmers are unlikely to build it until other parts of the framework are more mature. By that time, we hope to have a better idea of which kind(s) of user scoring to implement. — Ed Pastore 04:36, 30 March 2009 (UTC)
  • My proposal doesn't tie suggestions to the people who make them. It could be implemented only for suggestions without setting up the rating system for people at all: although I personally think that the rating system for people is prior, and ultimately impossible to avoid. The point is to create a mechanism that allows two central things to happen: 1) ideas can be prioritised gradually as a discussion progresses and more people rate them more highly; 2) 'voters' can decide to give a very quick and shallow immediate reaction, if they don't care much, or a more precise analysis if the topic is important to them. — User:Quickfitter
  • I also think user-rating is inevitable (as in the creation of a reputation economy). But from the programmers' perspective they have to start somewhere, and the underlying structure (a distributed version control system) means that they need to focus on the basics first. I would love to see user-scoring implemented from the outset, but the programmers aren't focusing there and… we simply haven't built a consensus toward it yet. Separately, note that we are also talking about a broad mesh of communities. It seems likely that user-scores should vary from community to community. This reflects reality in that someone who is greatly respected in one community may have no influence whatsoever in another. The example I gave on the list is that someone who is broadly revered in the community of Britney Spears fans should not necessarily be considered a top thinker in the community of nuclear disarmament activists. — Ed Pastore 15:09, 31 March 2009 (UTC)
  • I have no interest in participating in any sysytem that supports a cult of personality of any sort. We should not be encouraging folks to compound the Fundamental Attribution Error (FAE). Instead, we should be fostering understanding that expertise is, in the words of James Surowiecki, "spectacularly narrow." Toward that end, it is the ideas or, more specifically, the documentation (i.e., the records) that should be assessed (scored). Moreover, good ideas are a dime a dozen. What is rare and, therefore, more valuable are good implementations. If it is results about which we care, we should focus on goals and objectives that have been explicitly documented in terms against which progress can be measured and reported to stakeholders. Of course, to the degree that the product is the person (e.g., a stand-up commedian), subjective ratings by members of the audience would be appropriate. However, if that is the sort of thing on which this group wishes to focus, its charter should be amended and I will soon be signing off (since I have little interest in such an endeavor). - Owen Ambur
  • As I noted, there is no consensus for user scoring. See for example Talk:Basic principles. Thus in accordance with our third basic principle, there is no policy to that effect. I do feel that we will eventually come to an accord in regards to user scoring, but only one that is highly nuanced. The general agreement we have had to date is that at some point we should allow for some sort of user scoring mechanism within Metascore, but not give it any significance. Then when the software is released onto small communities, hopefully some will choose to give some significance user scoring, while many others will not. From that testbed, we should then gather feedback and see if any sort of user scoring makes sense. (P.S. Note to sign your name with a timestamp, use four tildes ~ in a row.) — Ed Pastore 16:58, 31 March 2009 (UTC)

Do you support this proposal?

Proposal 7

Title: Apache-like rating

Author: Ed Pastore

I propose a slightly less geeky scale than the Apache project: that people vote by expressing an integer score from -10 to 10. And that unless there is compelling reason to do otherwise, scores are accepted for 72 hours. At the end of 72 hours, the scores are simply added together, and a positive score means approval. To reproduce Apache's veto, I also propose that a vote of -10 be tallied as -100. People can express a score with an explanation, or simply a score with nothing else. If anyone votes -10, then the resolution is stalled until that person can be convinced to change their mind (probably by the group changing the resolution in some way). A change to the resolution resets the 72-hour clock. Unless they say otherwise in their proposal, let us automatically assume that an author of a resolution gives it a +10.

Comments

A veto MUST be accompanied by an explanation. The veto is void, if none is given. --Manel Barkhau

Wow, so decimals are geeky? I personally liked the apache system, because you can express your opinion without any commitment, by staying between +1 and -1, which if I understand correctly, isn't possible with this system. Or is a score with this system just like apache * 10, and only 10 or -10 really count, with the rest only expressing sentiment.

I did not investigate Apache's system thoroughly, but I think the only way in which mine differs is that all scores are tallied at the end (and that yes, it is multiplied by ten). So a score of 5 mathematically expresses half-support. The reason I propose nuanced scoring like this is that people can show different levels of support for different competing propositions. If one has no opinion but wants to comment, one can vote zero. — Ed Pastore

"a vote of -10 be tallied as -100. People can express a score with an"

Is this to be understood as it takes 10 people to override one veto? I suggested that as a convenience for this group, since we have 11 members. If everyone participated in a vote and ten people gave it total support 10*10=100, one person could still veto with a -10(*10)=-100. — Ed Pastore

I'd like the following modification. A veto MUST be accompanied by an explanation. The veto is void, if none is given.

+8 — Ed Pastore

— Manuel Barkhau

I don't like it. I don't feel there's any merit to scores that say "I half support this decision". For such a small group, even for beta-testing, I'd support a system where you need X% of the members to vote "+" and 0% of the members to vote "-", with no time limit (it is decided the moment X% voted "+" and all "-" votes, if any, were already removed, provided that there were at least 72 hours to comment on the latest revision).

But

  • You were the first to offer a system,
  • It achieves the purpose of consensus based decision making
  • There's always a third suggestion that's better than the second, and if we open this we will never decide on how to decide
  • A system similar to my own is already in testing in my design document's writing process
  • This system doesn't protect itself, so we can try other systems after we get a feel for how this one works

So I feel it makes lots of sense to agree to your suggestion.

I propose a change, and that is that ALL negative votes are weighted x10 (otherwise it is a majority rule decision process with veto rights for everyone, not a consensus decision process).

However, with or without the amendment, you have my +10

I am fully behind you on this.

It will be interesting to see how this works out.

— Aur Saraf

It would only be a majority rule if everyone in favor voted +10. But people should feel free to vote less for anything they are not totally excited about. For example, given your commentary on my resolution, I would suggest you change your score to something more like +2.
Nope. If five users vote +3 and four users vote -3, the majority wins, even though it is far from consensus. — Aur Saraf
Those four who scored it a -3 were not opposed enough to the resolution to give it a more negative score; so it passes. If they were more against it, they could have each given it a -4, and the resolution would have failed, even though they were in the minority. — Ed Pastore
If all negatives are scored *10, it means that any expression of negative attitude is effectively a veto unless there is extreme support from all other members. So, I give your above amendment: -7
No, it only means that consensus is defined as "~90% agreement". — Aur Saraf
(In your system, that would be close to a veto unless everyone else was wildly in favor. So in that case I would change my vote to a -3.) —Ed Pastore
The idea of consensus is that everyone should mildly agree, or at least almost everybody wildly agree. Isn't it? — Aur Saraf
But multiplying negatives by 10 means that any slight disagreement is a very strong vote against the bill. I could, however, see multiplying negatives by X, and then let X be tweaked until a good definition of consensus is found in the wild. — Ed Pastore
As a non-developer, I'm a fan of the --1, -1, 0, +1, ++1 system I saw earlier.
This easily translated into "i'm strongly opposed," "i don't care," or "I strongly agree" to me.
All the extra numbers seem add an unnecessary level of complexity.
If you're keen on complexity, the --/++1 system included decimal points for those wishing a finer level of specificity, WITHOUT requiring everyone else to do so.
— Christopher Ritter

-2 (-20%, -.2, etc). I don't like "72 hours", especially not so soon. People should be able to discuss until they are ready to get behind something. For instance, I like the three-option system below better, but this system would have passed had it followed its own rules (it's been over three days since the amendment was added). Humphrey 03:02, 20 June 2008 (UTC)

Do you support this proposal?

Proposal 8

Title: Free-form Access and Registration Tallying

Author: WhiteHorse

I recommend an integer rating from 0-100 or 0-256 (8-bits) for computational simplicity and then allowing the users to "skin" their voting apparatus to suit themselves. Perhaps it could just be a simple text-file or cookie on the user's computer which shows whatever they want for each value. You could throw out a couple of examples and let people choose which interface is best for them. Offer some guidelines on how to make skins and subject them to peer review to prevent people from submitting "spoof" skins which list 0=good, 100=bad... etc. If you <wisely> choose the 0-100 system, you'll be able to apply all of the statistical methodologies of the last 100+ years including: Pass/Fail, mean, median, mode, normal distribution, standard deviation, etc. You could even make proposals pass based on "within x standard deviations of the mean". It would give you a more complete picture of how much people agree/disagree with a proposal and how many people will be adversely affected.

Comments

(Comments to the proposal here.)

Do you support this proposal?

Proposal 9

Title: Five options voting system with conservative scoring

Author: Gustavo Menezes (Menezesgc 17:29, 2 April 2009 (UTC))

My proposal is kind of a synthesis of previous ideas, with new features. In my proposal Metascore would have five voting options, each one scored as described below:

1- As it is, I do not care whether this proposal passes or not (0 points)

2- I do not want this proposal to pass at all (-100 points)

3- I do not want this proposal to pass, but might accept it with major amendments (-10 points)

4- I want this proposal to pass, albeit with minor amendments (+1 point)

5- I want this proposal to pass as it is (+10 points)

If the final score is positive the proposal passes. If the final score is 0 or less, it does not.

Advantages over other voting options

The reason for 5 voting options is that I agree with those who think that too many options (such as form -10 to +10, or 21 options!) is too many to make any sense. On the other hand, a yes/no kind of vote is way too simple. And, as stated before, simple 5 star systems do not represent an important aspect of voting, the “I don´t care” vote (of course, while still counting the vote). My guess is those five I proposed should pretty much represent the thinking of any voter on a proposal.

Moreover, calculations are easy, can be easily translated into percentages and analyzed with simple statistics. The system is very intuitive for the user, and a simple explanation followed by examples can show a voter the reasoning behind the scores. I do not think the scores should be variables chosen by the community, since this could lead to malicious deviations of the consensus principle.

It is also possible to join options 3 and 4 into one such as: I do not want this proposal to pass, but might accept it with amendments. However, I guess passing an almost perfect proposal is better than not passing at all. You can always change it in the future if it is not working. Besides, in the scoring system proposed the value of option 4 is really low anyway. The true importance of it at this moment is for the analysis of tests, where one can observe how communities tend to vote and if more or less nuances of voting options are necessary.

Amendments

Those who want amendments should insert them in the proposal page (maybe a whole lot of amendments should be voted together with the main proposal) or vote for already proposed amendments. This might work in small communities but in complex ones it will be a mess. We will have to think of more operative ways to implement this.

Vetos

I do not think vetos must have any comments on it, simply because this kind of scoring system allows for some vetos while a proposal is approved. I am rather doubtful that a complete consensus can be made on any subject matter. In small communities sometimes compromises can be reached, but in complex ones I am convinced they cannot; sometimes compromise is just not possible, but someone will have to give for the good of the community (even unwillingly).

Advantages of scoring system

As for the points counted for each type of vote, I have tested different score types, some representing already proposed scores, and I found interesting stuff (see tables and discussion below). The above set of scores was the best option amongst those I tested, taking as a premise the “consensus” idea.

Time range of voting

I think we should open a new page to discuss this matter, as it is just not important when it comes to scoring system. I think we should give at least a longer period of vote for each proposal (instead of 72 hours, as proposed, something around 1 week), in order to avoid time-directed bias (such as people who only vote on Saturdays, losing votes starting on Sundays). However, my best proposal would not include short time limits for voting, but rather long ones (like 1 month) with the proposal only being valid for approval once a minimum percentage of voters do vote on it (such as 75%). If we set small time limits without minimum amount of voters, proposals with very little acceptance on the population overall will be passing. And, worst, by small, unrepresentative, but very active, members of the communities.

User rating

I do not like ratings of users by other users. If any user rating should be implemented, then it should follow some rule based on the approval of his proposals, geography, and/or other kinds of objective, computer calculated ratings. In my point of view, subjective rating will only bring back many pernicious issues of the democracy we have today.

Scoring system test

I run a very simple test on an excel worksheet, which I could not import in here since metagovernment prohibited uploading. I sent the file through the Start Comittee list and if anyone can attach it here, please feel free to do it. In anyway, if you read the following paragraphs without the file you may be able grasp what is really pertinent to the whole testing thing:

Situations 1 and 2: If the score of the vote of those who do not want to pass a proposal has the same weight as those who want it very strongly, then proposals will pass very easily. If, on the other hand, the weight is heavier for those who do not want it to pass, then the voting becomes very conservative. Solution: I think passing proposals should be conservative, to agree with the principle of consensus. Therefore, the score of negative votes should be way higher than those of positive votes.

Situations 3, 4 and 5: Unless you give more weight to negative votes (overall) than to positive votes, even if the majority wants changes in a proposal, it will pass as it is. It also shows that score option 3 (similar to that proposed by Ed in the Apache-like score system) will allow non-consensus proposals to be approved. Solution: (a) give a lot of weight to negative votes in comparison with positive votes (like in score option 5); and/or, (b) set a minimum limit of type 5 votes for a proposal to pass (such as >50%, or the majority of voters).

Situations 4 and 6: If the majority of voters do not care for a proposal to pass, then the weight of the votes determines if the proposal passes or not. For instance, if the type of vote 2 weights 10 times that of type of vote 5, then you will only need >10 times more positive voters than negative voters for a proposal to pass. This creates crazy approvals, such as in situation 6, even in very conservative score methods. Solution: (a) set a maximum limit of type 1 votes for a proposal to pass (such as <50%, or the minority of voters); (b) set a minimum limit of type 5 votes for a proposal to pass (such as >50%, or the majority of voters); and/or, set an option in metascore for the community to choose whether they want these kind of votes to be valid.

Situations 7 and 8: Pretty clear. Unless you give more weight to negative votes (overall) than to positive votes, simple majority will be enough to pass proposals. Solution: As noted before, the score of negative votes should be way higher than those of positive votes.

Situations 9 through 12: the heavier the weight of negative votes (such as in score options 4 or 5), the more you will need to have consensus-like votes for a proposal to pass. Solution: negative votes should have higher scores (in modulus) than positive ones, to observe consensus-like approvals.

Situations 3, 10 and 12 through 15: if the weight of voters who want changes in the proposal (types 3 and 4) is equal to those who want it or not at all, even if the majority wants changes in a proposal, it will pass as it is. This means that the lighter the weight of kinds of votes 3 and 4, the more difficult it is for a proposal to pass (compare situations 10 and 12). Solution: voting types 3 and 4 should have scores way lower than types 2 and 5, to make it improbable for disputed proposals to pass.

Voting options that Metascore should allow users to decide on

Optional: If (X)% of voters do not care whether the proposal passes or not, then the proposal do not pass.

Optional: If (X)% of voters do not want the proposal to pass at all, then the proposal do not pass. (In the scoring system suggested X is by default 9,090909…091%, since even if all other voter chooses option 5, the score will still be negative).

Optional: If (X)% of voters do not want this proposal to pass, but might accept it with major amendments, then the proposal do not pass.

Optional: If (X)% of voters want the proposal to pass as it is, then the proposal passes. (In the scoring system suggested X is by default 90,909090…91%, since even if all other voter chooses option 2, the score will still be positive)

Comments

(Comments to the proposal here.)

Do you support this proposal?

Proposal 10

Title: ...

Author: Jacki Buros 20:23, 11 June 2009 (UTC)

It might be useful to have there be some cost to voting -- well, to any vote beyond a simple yes/no. Votes indicating strength of approval/disapproval should have value in order to be meaningful. Perhaps people are given an allowance of 'extra votes' per issue/topic or per time period, which they can then allocate to various proposals the way one allocates money to goods/services (these are tallied separately from yes/no votes). If there's something I feel really, *really* positive or negative about, there should be a way for me to demonstrate this degree of caring so that it has *real* meaning -- ie, it comes at some cost to me that other people can relate to.

Let me be clear, the cost should *not* be financial. But, this is in some ways an application of econometrics to political/legislative frameworks.

As a way to incorporate user ratings, one could potentially 'earn' extra votes by gaining positive ratings from other users. This might encourage useful participation in the system and promote consensus.

The idea is in its early stages, haven't thought it through exactly, but thought I'd put it out there.

Comments

(Comments to the proposal here.)

Do you support this proposal?

Proposal 11

Title: -3...+3 scoring

Author: Trinar

In my proposal I use the range -3 to +3. The scores have this meaning:

-3 I am strongly against the proposal

-2 I am against the proposal, but with greater editing I could vote for it

-1 I am against the proposal, but with a little editing I could vote for it

0 I don't care

+1 I am for the proposal, but with greater editing it can be better

+2 I am for the proposal, but with little editing it can be better

+3 I am strongly for the proposal

In the beginning of the voting, all ratings are set to 0 and only the people that want to vote do it. The other will have 0 rating and don't influence the decision.

The people can rate the proposal and in the end of the voting time (for example 72 hours), if all of the ratings are not negative and there is minimal one positive, the proposal is approved.

If there are some negative, the proposal can be revised an put for voting again.

For every rating there must be an explanation.

trinar 18:03, 24 July 2009 (UTC)

Comments

(Comments to the proposal here.)

Proposal X

I would suggest that wiki based methods that favor collaboration (wiki editing) rather than submitting and voting is the only process that scales. You plant a flag saying "the winning proposal will be here", stand back and let the fur fly. Allowing multiple proposals to grow is secondary and not essential, but helps if you have more than a dozen active editors. Idealpragmatist 20:07, 16 August 2009 (UTC)

in a high traffic situation, the ability to rank in order to "float" favored position to the top is desirable. High traffic political wikis would be a nice problem to have.

[1] [2]

Personal tools