New Karma System (Theory)

[APTX]

I don't really know how this starten (didn't read the whole thread) but if you'd ask me I would start with defining karma.
What I mean is we have to know what this thing is for (what we expect) and a few baisic things like if the value depends on time (when time passes the value tends to become 0) or rather static, if it is describing a single post/all posts or the user himself. Is that anywhere in this thread? I would like to know what the devs expected from karma.

[vph]

Defining 'karma' isn't an easy thing. My understanding is, in Buddhism karma is about cause and effect. What you get now is the effect of what you've done. Its vague definition in a mysterious way suits well for the name of a rating system.

It looks like in this theory here, every user has a karma weight. I think the karma weight of a user should be affected in TWO ways:

(1) When he creates a thread and gets feedbacks from others, other users will rate his topic. These scores -- after being normalized (perhaps more weight given to those with larger karmas) with potential outliers being filtered out (for example, dropping max & min) -- collectively will be affecting the thread starter's karma weight in some way. This means that if someone is "doing good" by starting a good topic, his karma weight should increase. And vice versa.

(2) When he rates someone's post. His karma should be based on how much his score agrees with the average score. Remember that the "average score" mentioned in (1) is presumed to be the best approximation to the quality of the thread. This means if someone is "doing good" by making good judgment his karma weight should increase. And vice versa.


Taken (1) and (2) together, I feel then the Karma system stays closer to its meaning. A person is responsible for his action.

That doesn't mean the system can't be cheated. If all responders to a thread form a coalition against (or for) the topic starter, and if the system can not detect such a scenario, then the rating of the post is not what it should be. But there is no way to detect such a thing automatically without human judgment, as far as I know. In this case, then it is best to put the matter in the hand of the moderators.

I think these ideas need to be straighten out first before defining a set of mathematical rules.

[MKruer]

Hi there,

I'm a newbie to the forum here. I think a rating system is an interesting feature for a forum to have, but I can understand why it's difficult to implement it properly, as designed and no side effect. So I understand why people are not much interested in having the feature right now.

Nonetheless, the theory of rating system is interesting in its own right. If we could come up with one worth implementing, I feel it could take phpbb to a newer dimension.

Others interested in continuing this discussion?

There are a few things I'm still not sure with the Karma Rating theory described here.

The gist of the reason why the original Karma Rating system was removed was because it was abused, and not lightly, but to the point where the entire system failed, and no amount of ban-aid fixing would have solved the problem. Because Karma Rating was relatively low on the list, it was decided to be dropped then spending the time to fix a system that was fundamentally flawed. The “NEW” Karma Rating is trying to fix all the flaws at the most fundamental of levels, and not have any of the weakness of the original. For now its just theory because I would rather spend the time try to think how to break the system before its built then build it, break it and have to rebuild the system over and over until it work correctly.

[MKruer]

I like the whole idea, but why not do something simple like slashdot has? Users get points by reviewing reviews, and then get to use those points to review peoples posts.

Example:
I have one point from reviewing peoples reviews.
I go to a new thread and like what User A said, I then spend my one point to rate him one a scale of 1-5 based on that post.

User B logs in and is asked to review a few random reviews.
User B gets to see the thread I was looking at and the user I rated, but does not get to see my user name.

User B then rates my review as Fair or Unfair, which then gets calculated into how often i get points to rate people with.

---

I hope that kinda made sense... Its just my initial thoughts on the idea before I head to my next class.

I know what you are saying but I do not know if it’s practical for a bulletin board system. The user need to be identified for bulletin board to work properly

[MKruer]

Defining 'karma' isn't an easy thing. My understanding is, in Buddhism karma is about cause and effect. What you get now is the effect of what you've done. Its vague definition in a mysterious way suits well for the name of a rating system.

It looks like in this theory here, every user has a karma weight. I think the karma weight of a user should be affected in TWO ways:

(1) When he creates a thread and gets feedbacks from others, other users will rate his topic. These scores -- after being normalized (perhaps more weight given to those with larger karmas) with potential outliers being filtered out (for example, dropping max & min) -- collectively will be affecting the thread starter's karma weight in some way. This means that if someone is "doing good" by starting a good topic, his karma weight should increase. And vice versa.

(2) When he rates someone's post. His karma should be based on how much his score agrees with the average score. Remember that the "average score" mentioned in (1) is presumed to be the best approximation to the quality of the thread. This means if someone is "doing good" by making good judgment his karma weight should increase. And vice versa.


Taken (1) and (2) together, I feel then the Karma system stays closer to its meaning. A person is responsible for his action.

That doesn't mean the system can't be cheated. If all responders to a thread form a coalition against (or for) the topic starter, and if the system can not detect such a scenario, then the rating of the post is not what it should be. But there is no way to detect such a thing automatically without human judgment, as far as I know. In this case, then it is best to put the matter in the hand of the moderators.

I think these ideas need to be straighten out first before defining a set of mathematical rules.

Excellent sum up.

Right now (we) are thinking a way to make it so groups of people can not target one person. The best solution so far (which would work) would be to keep an individual recorded for every user on user action. The problem with this method is that data would be incredibly large to store.

[vph]

It is not clear if keeping a record will help. How can you tell if a group of people unfairly form a coalition to work against someone, or artificially jack up karma of someone they like? It may be the case that the person under scrutiny deserves that negative/positive rating.

Would it be better just to let the system take care of itself naturally? Let's say someone is being heavily biased against (or for) in a closed community, and consequently his karma is low (or high). The question is: does his karma rating precisely capture his karma? In a philosophical way, I'd tend to say yes (independently of MY or YOUR judgement of his topics). The community is a closed world. If they like someone his/her karma is up. If they don't his/her karma is down. Ideally, this should be related to the quality of his/her threads. And in some way, it does. The point that I want to make is who are we as outsiders to think we can judge the karma of this person better than those inside his/her community. In other worlds, if his/her karma is low or high within that community, it may very well a proper reflection of his/her importance in that community.

I don't know the answer. So I'm just throwing out ideas.

Being the devil's advocate, someone already reported having bad experiences with rating systems. So obviously, it may not be that simple to yield to natural selection (in this case, being let the system flow without detecting coalition).

I think one important thing to consider is not giving too much importance to the karma weight itself, for example when computing the score of a thread; or when awarding or penalizing scores deviating from the average score of a thread. The reason is that if you put much significance to the karma weight, AND if the system is abused it will crash more quickly, because if there's an unwanted way to increase or decrease karma weights quickly, there's a tendency that the unwanted phenomenon speeds up at a very fast rate (something usual and expected in chaos theory).

On the other hand, if we let the system evolve *slowly*, it may be the case that unwanted/cheated phenomenon will be fuzzied out by other actions. If they don't, then maybe that phenomenon is actually what it should be; in other words, there's no abnormalty in the system (as argued above). I tend to believe that this is the case.

In other words, maybe ... just maybe, the right karma system is a simple system to address point (1) and (2) above, BUT with the right (slow) rate of evolution.

[DragonlordP]

Some thoughts on ratings systems. I'm a little busy right now so it will be short and maybe silly said, oh well.
Most of us have met politicians. We all know how disgusting and colorless people they are, just because they try to get ppl to like them. You know how the bigger scums go up in a society. In a much smaller scale, but in a similar scale whatsoever, that's what happens when you put a ratings system to a community. Some people, not all but definetely some, will try to get their karma up and that often means become colorless, always agreeing with most, trying to express the "average opinion" etc. In other words, disgusting in the same way politicians are.
That is a personal view and I know many won't understand it, but I for one will surely not put a ratings system on my board.

[vph]

DragonP, I'm not sure precisely what your points are. It's clear that an effective rating is difficult to implement. It maybe the case that it's impossible to prevent abuse from such a system. But such a system is undoubtedly useful for many of us. Many online stores have implemented them and have experienced abuse too. In a recent incident when amazon.com accidentally revealed the information about reviewers people saw pretty funny cases where competitors put down one another. That said, the system is still useful for many amazon users.

I think the theory in itself is interesting. It's not surprising that the realizations thus far have been ineffective

[warmweer]

I think the theory in itself is interesting. It's not surprising that the realizations thus far have been ineffective

Ineffective? I wouldn't call it that. I've seen a couple of karma systems and all work well when the system uses objective parameters. The problem is that marking a post is never objective. The described system in this topic would unavoidably attach greater weight to the opinions of the more proficient evaluators whether they are "good" or "bad". Also, marks which are way above or below average would have less influence (read value) than "average" marks, a belief which is inherently biased.

Whichever final karma is reached, it will be correct according to the algorithm used; use another algorithm to calculate the karma and the result will be a different karma level, which is equally correct.

Just as we will all rate posts differently, we will all have a different feeling about the correctness of the calculated karma, yet none of us will be able to prove that another value is the correct one.

Even a simple rating system as to whether a post is on topic, off topic or "don't know" is prone to misinterpretation because we all have a different view on what is on topic and what not.

This in no way implies that I think this topic is worthless, on the contrary, it's very interesting and will eventually be usefull in trying to set up a karma system for some purpose. But It does express my opinion that a karma system should not be a standard feature (unless it can be turned off or at least have a couple of changeable settings so that it can be finetuned to the admin's liking - consider it potential for a mod .

edit: unfortunately my quote might give the impression that I disagree with the posts of vph (and DragonlordP). That is not so and I would rate the posts highly - but then others could disagree and lower my rating - can it get lower even?

[R. U. Serious]

In a Karma System what exactly should be rated:
- The like/dislike of a user? (personality-based?)
- The agreements/disagreement-factor? (meaning of the content of his posts)
- The quality of the discussion that his posts create?

There is no clear cut answer for this. On support forums it probably should be the second one, where the meaning/content will be the most interesting (is it correct? is it complete?). And on forums that try to be a a high quality platform for discussions, you want people to rate according to the third: Did the user not use personal attacks or unfair rhetorics and did he give a well reasoned explanation and arguments for his opinion? On forums that are mainly oriented to build communities of people and where talking offtopic and killing time is the main attraction, the ratings will tend to be personality-based.

The problem is, that this is often not a clear-cut case, and the even bigger problem is, that users will even be a lot less likely to give any thought to it. What you will get, is a lot of users rating for a lot of different reasons. And what is the meaning of the average of that? I'd say there isn't any. Some pointed out that "karma" fits so well, because it is so vague, IMHO that's the reason why it will not work very well.

Dhn (I think) proposed a system that is not based in points, but on specific attributes (funny, helpful, etc.; much like slashdot) which could help guide the way people use a rating-system. Another helpful thing IMO is (and I know a lot of people will disagree), that only a closed group of people are allowed to rate. "Forum-Membership" alone is to loose a thing to define who can vote who can't, because it is too easy to create sockpuppets and the like. With a closed group that gets appointed e.g. by the the forums' admin/mod-team, you can more or less be sure that if you only choose trusted and mature members, that there will be little gaming int the system. A side-effect that some people will not like (but which I find is a positive feature) will be that that appointed group will in a way set the tone of the forum - it will "choose" what is exemplary and/or accepted behaviour and what is not.