New Karma System (Theory)

[MKruer]

New Karma System (Theory)
As many of you know, the Karma System that was suppose to go into the next releases phpBB was removed. The reason why, was not lack of programming skills, but a way to effectively, and fairly judge every post by a user. The purpose of this thread is to create, at least in theory, a rating system will work even under the intense scrutiny.

I have been working on this (Karma Mod Theory) after it was dropped from phpBB 2.2.x/3.0.x The two major theories implement in this version is a way to recentralize a users rating over time to greatly diminish past bad behavior, and a way to dynamically adjust the person who is rating a post if they are consistently to high or tow low. Other improvements are rewarding people for posting “relevant” posts, and give people that do post more “relevant” posts the ability for their rating to be worth more. Check it out and see if you like it. I have wrapped my mind around this, and have not found any major flaws in this system.

The only reason why I am posting this here instead of the main site is because there are certain assumption in this mod (if you want to call it that) that are not easily possible with the current version of phpBB. At the very least you my find it an interesting read.

Ground Rules
* It has to be as idiot proof as possible (simple interface)
* It has to be anti abusive as possible (not allowing someone/some group to consistently screw other people)
* It should be able to be used as a metric to find posts that are more “relevant”.

“Anti-Abuse” Policy (Theory)
The easiest way I can think to implement an “Anti-Abuse” policy is quite simple, IMO.
Each user carries Karma Weight; Karma Weight is the sum total of all the Karma Rating from that person, divided by the number of Karma Rating posts by that person. This is used to set where the “middle ground” is for any given user. If someone constantly marks someone else low, then the “middle ground” for the person rating the post would also low.

Example:
If Joe Blow has and User Karma Weight 3, and marks a post with a 2, the real Karma Rating would be -1
If Sally Something has a User Karma Weight 7, and marks the same post a 3, the real Karma Rating would be -4

(Note, the formula may need to changed to reflect the 5 standard deviations based upon of the middle ground, i.e. Joe Blow has a User Karma Weight 3, so 3 is the new 5 for him, and this would mean that real Karma Rating of 2 would be ((5/3)*2) for 3.333, which is what the post would have been marked if Joe Blow had an average User Karma Weight of 5 )

Weight of Authority (Theory)
The theory of Weight of Authority on this is that everyone carries some type of weight when marking Karma. It works two ways, first, the more you post, the more weight you carry, and depending on your lot on the forum special Special Ranks, Moderators, Administrators, you may carry additional weight.

Normalized Rank
Normalized Rank is an administrator definable field that is used determine how productive a user has been. The Normalized Rank is not based upon total number of posts, but rather the number of replies that user gets to one of their posts. The idea behind this is to reward people for continuing a thread, and also to prevent spamming issues to artificially increase the Normalized Rank. If a user replies to themselves, the total number of posts do not increase, only a reply form a different person will accomplish this.

Normalized Rank Cont.
Starting user carries a weight of 1. As the user posts more and more, and graduates from a “new poster” to a “novice” user, the weight of their Karma Rating also rises. The rise should be subtitle, as to be over whelming. For this reason I would recommend that each rise in Normalized Rank from a “new poster” to “novice” to “standard” to “elite” etc… be a one tenth of a point (0.1) this may not seem much but over time, this will start to stack up.

Special Ranks.
These are Normalized Rank that can only be assigned by Moderators and/or Administrators. These work on a multiplier system where the Special Ranks is multiplied into the Normalized Rank to create the final Weight of Authority

Moderators
Moderator is a class of Special Ranks that are allowed some administrated ability and should “always” out weights any special rank. Again, this works on a multiplier system where the Moderators weight is multiplied into the normal Rank to create the final Weight of Authority. Moderators have the authority to override all users but not Administrators

Administrators
Administrator carries/can carry the ultimate Weight of Authority. Administrators[/b should always outrank every other class. Unlike the Moderator or Special Rank, and Normalized Ranks. Administrators have the authority to override all users including Moderators.


Base Lining (Centering) Theory
If a post is replied to, but not rated, the original post will receive an automatic 5 (5 being the default value, neither good nor bad) points. By definition if a post is replied to, but not rated, neither good nor bad, then it should be assumed that the post was neutral in nature. This means that if Joe Blow posts a message and Sally Something replies to it but does not rate Joe Blows post then, Joe Blow gets credit for the post (which increases his authority) and because the default is 5, the 5 is calculated into Joe Blows Karma Rating there by centering it.

Karma Weight & Karma Rating
Karma Weight tells you how valid the Karma Rating is. i.e. If the weight is heavy, the rating is more accurate then if it was light. With this system, you can isolate individual posts as being most relevant within a thread. (IMNSHO this system is worth it for that alone.)

Equations, Equations, Equations!

A = User (1) Karma Rating
B = User (2) Karma Rating

J = Post Karma Rating
K =Post Karma Weight

O = Thread Karma Rating
P = Thread Karma Weight

X = User (1) Karma Weight
Y = User (2) Karma Weight

To calculate a single post a person can rate the relevance of the post from 0-10 (5 is assumed default if no rating is giving upon reply)

(X*A) + (J*K) = J*K (J/K gives you the current rating for the post)
(J*K) + (Y*B) = Y*B (Y/B givers to the new rating for the post. The replied post gives more weight to the original poster, as well as adjusts original posters rating)
(J*K) + (J+K) = O*P (ratings of multiple posts within a thread can will establish both the Thread Karma Weight and Thread Karma Rating)

That’s it.

Ground Rules Check up!
* It has to be as idiot proof as possible (simple interface)
Yes. Regardless of whether a person marks the post, the simple act of posting a reply gives credit to the original poster.
* It has to be anti abusive as possible (not allowing someone/some group to consistently screw other people)
Because the system gives an automatic default value, as well as adjust for individual consistently marking low scores, the system well recentralize all users over time.
* It should be able to be used as a metric to find posts that are more “relevant”.
Yes, Posts that are replied to carry more weight, and the Weight plus the Rating give more “relevance” to the post or thread

Last minute additions
The one possible flaw in the above mentioned scheme is that over time, the Users Karma Weight can become so great that any marking for or against the user might not impact the users base rating. The simple solution to this is to track only the last 50-100 posts. This will allow for the base rate to fluctuate more as to provide more up to date users base rating based upon the users more immediate posting habits.

In order to increase the accuracy for the Karma Weight during posting, you my want to do some preprocessing to make sure that the post was replied to has something worth increasing the weight for. i.e. If Joe Blow posts a one page message, and Sally Something says “Cool Idea!” and then Joe blow replies to Sally Something, because Sally Something reply was so shore, it is not appropriate for her weight to be increased. A simple metric such as length can be used to increase the appropriateness of the weight, and there could be an administrator/moderator check box override if the post is under the minimum length but should be included in Sally Something weight or Visa versa.

One of the questions that was brought up was how to “reward” people for staying active. The simplest solution I think would be to use the Users Karma Weight and include a time factor. The time factor would be something to the affect the more a user posts the more up to date he or she will be on the forums, and hence should be considered more “relevant”.

At minimum you would like users to post once a day. This information is easily attained because of the Users Karma Weight. Users Karma Weight * the number of posts replied using the Users Karma Weight will give you the number of “relevant” posts per day. This in turn could be multiplied back into the Users Karma Weight on the submission of Karma Rating. i.e. If Joe Blow has a Weight of Authority (Theory) of 3 and posts only 0.5 posts in the last 30-50 days then his Karma Weight would be only 1.5 (3*0.5). so because he has not posted frequently in the recent past, his Weight of Authority will be diminished. (Anything over 1 post per day would be calculated as 1 as not to over power the Weight of Authority by unnatural means.)

[Natan]

I'll be honest, I didn't read this entire thing, but all I can say is that if it takes this much explanation, it's probably going to be too big a deal to bother with.

[Eelke]

I did read all of it, but didn't understand everything. It's hard to understand something like this from just equations; the textual explanation didn't have enough clues to understand the meaning of the terms you are using (i.e. Karma Weight, Karma Rating). Also... your equations aren't equations... how can this be true, unless X*A=0?

(X*A) + (J*K) = J*K

However, one thing I did understand enough to challenge, which was the anti-abuse system. You say that someone that consistently gives low scores will not suceed because the system will automatically compensate. Although this would take care of the total trolls that just came to a board to bash anyone and everyone (an also just the chronically synic), it wouldn't help in case someone just is out to get a single person and otherwise gives fair scores.

Having said that, that is only one observation, from only understanding the whole thing partially. If you really want to explain this, I would start out very simple, explaining the basic algorithm and only then start to explain the different compensations you came up with for the problems the basic algorithm would have.

[MKruer]

Let me try to clarify

Everything is calculated form the above.

I did read all of it, but didn't understand everything. It's hard to understand something like this from just equations; the textual explanation didn't have enough clues to understand the meaning of the terms you are using (i.e. Karma Weight, Karma Rating).

Karma Weight is how much authority a user has when rating a post. The more authority, the more their rating counts.
Karma weight takes into account how many posts, if you are a moderator, what you average on posting, and how frequently you post.

Karma Rating is the average rating they have over the last x amounts of posts. (this is the only thing the user would see)

Also... your equations aren't equations... how can this be true, unless X*A=0? (X*A) + (J*K) = J*K

The Equation is a self referencing equation, each iteration (post) changes the base value.

This is the trickiest part, and I was having problems expressing it. Let me see if I can

For the time being I will assume that you understand to some degree what User Weight i.e. Weight of Authority is.

To find the Rating of the post it is ALL the users weight times normalized value devided by sum of all the weight.

Joe Blow (Weight of Authority = 3) (Rating of the post = 6)
Sally Something (Weight of Authority = 1) (Rating of the post = 10)
Jane Doe (Weight of Authority = 2) (Rating of the post = 4)

Real rating of the post is (3*6 + 1*10 + 2*4)/(3 + 1 + 2) = 6 this is what I meant by (X*A) + (J*K) = J*K (J/K gives you the current rating for the post) hope that clarify that.

However, one thing I did understand enough to challenge, which was the anti-abuse system. You say that someone that consistently gives low scores will not suceed because the system will automatically compensate. Although this would take care of the total trolls that just came to a board to bash anyone and everyone (an also just the chronically synic), it wouldn't help in case someone just is out to get a single person and otherwise gives fair scores.

When I first though of the system I was concerned that there would be too much data to process, but after thinking it through, the maximum amount of data would be Number of posts * Number of users. So if you wanted to get down to it, if there was a list of all rating by people fore each an every post, then yes, you could make it so the anti abuse is so tight that it could be on a person by person bases. The information stored in this table would only have to be the Post ID, the rating of the post (After the initial processing) and the Posters Karma Rating at the time of the post. Everything else exists.

Having said that, that is only one observation, from only understanding the whole thing partially. If you really want to explain this, I would start out very simple, explaining the basic algorithm and only then start to explain the different compensations you came up with for the problems the basic algorithm would have.

The is just theory at the moment so discussing the process is not going to hurt anything.

[Rocketeer]

This makes a lot of sense, but I don't think it quite addresses the abuse issue.

Let's take the average board... there's going to be some good posts, some worthless ones... probably a fairly even split between ones that deserve positive karma and those that deserve negative karma.

So if Harry (a "good" person) rates every post, he's going to end up with a User Karma Weight of about 5 (or whatever the midpoint is).
However, if Sid (a "bad" person) decides to rate all his friends' posts with 10 karma, and all his enemies' posts with 0, he'll end up with the same User Karma rating as Harry.
This means that Harry and Sid have the same "importance" when rating posts, even though Harry is being good and helpful, and Sid is not.

I think, ideally, you also need to give weight based on how well the user agrees with other users of the board. i.e. if 10 different users give a post a similar rating, then their karma weighting should go up - even if they are giving a low rating to a post. In this way you reward people who go along with the general attitude of the forum.
Unfortunately, the code required for this would be a real nightmare...

[the_dan]

That looks like a really interesting idea - Keep going with it

[MKruer]

This makes a lot of sense, but I don't think it quite addresses the abuse issue.

Let's take the average board... there's going to be some good posts, some worthless ones... probably a fairly even split between ones that deserve positive karma and those that deserve negative karma.

So if Harry (a "good" person) rates every post, he's going to end up with a User Karma Weight of about 5 (or whatever the midpoint is).
However, if Sid (a "bad" person) decides to rate all his friends' posts with 10 karma, and all his enemies' posts with 0, he'll end up with the same User Karma rating as Harry.
This means that Harry and Sid have the same "importance" when rating posts, even though Harry is being good and helpful, and Sid is not.

You are correct, this is the major possible fatal flaw in the design, however this is not insurmountable. If there is a history being kept for each user that you vote on (say 20+ posts) then you can calculate the weight of that person, specifically. It would be the combined weight that would result in the real rating.

I think, ideally, you also need to give weight based on how well the user agrees with other users of the board. i.e. if 10 different users give a post a similar rating, then their karma weighting should go up - even if they are giving a low rating to a post. In this way you reward people who go along with the general attitude of the forum.
Unfortunately, the code required for this would be a real nightmare...

So something like for ever positive response give a add a multiplier ratio value?

Joe Blow (Weight of Authority = 3) (Rating of the post = 6)
Sally Something (Weight of Authority = 1) (Rating of the post = 10)
Jane Doe (Weight of Authority = 2) (Rating of the post = 4)

Real rating of the post is
((3*6 + 1*10)*(1+0.666) + (2*4)*(1+.333)) / (3 + 1 + 2) * (0 .666/0.333) => (46.6 + 10.6) / (6)* (0 .666/0.333) = 6.4
Another test
((3*6)*(1+0.333) + (1*1 + 2*4)*(1+.666)) / (3 + 1 + 2) * (0 .666/0.333) => (24 + 15) / (6)* (0 .666/0.333) = 4.3

I would have to play around with this some more, but it looks like it might work. The big trick is to make it impossible to go above 10 or below 0.

[MrTufty]

You could just cap it at the extents. But either way a good idea, although I don't have my maths head on! Maybe once Olympus is out you should try coding this in as a mod?

[MKruer]

Unfortunately although god gifted me with a mind that can see the larger picture, my programming skills suck.

[Rocketeer]

*Expletive Deleted* I just typed a really clever, complicated method that solves some of the issues... then accidently hit the "Back" button on my keyboard (cunning located next to the up arrow) and lost the whole thing... Darn. Hopefully, I'll get it right again this time...

If there is a history being kept for each user that you vote on (say 20+ posts) then you can calculate the weight of that person, specifically. It would be the combined weight that would result in the real rating.

That's a pretty good idea... I think it would work, and get around the "nice to friends, nasty to enemies" issue, but I don't know if there would be too much data for it to be successfully implemented. I don't know enough about MySQL performance to comment too much, but it could quickly grow out of control - anyone know better (say a 3 column table, with User1, User2 and Average Karma)?

So something like for ever positive response give a add a multiplier ratio value?

I was more thinking along these lines...

• User A posts a message - no karma change
• User B rates it 7 - no karma change, he could be being nice or nasty, no way of knowing
• User C rates it 3 - still no karma change, because there's no way of knowing which of B and C was being accurate.
• User D rates it 8 - now B and D should be rewarded, and/or C punished, because the consensus is clearly around the 7/8 mark, not 3.

One way of doing this would be to record the Min, Max and Average karma rating for each post. The average should ignore the minimum and maximum rating - so you'd need at least 3 ratings before the post had an average. If user X rates a post above the current Max or below the current Min, he should get no karma weight change. If he rates it within (say) 2 points of the average, he should get a positive karma weight change.
This way, if one person has an extreme opinion on the post, it will get ignored, but if there's an even split through the community, the average will reflect that.

The big trick is to make it impossible to go above 10 or below 0.

True... but you can cheat by limiting the values in code, so that anything above 10 is reduced to 10, and anything below 0 is increased to 0... or you could just have an infinite scale...

Hmmm... you've got me thinking a lot about this... which is silly, because I would probably never implement a karma system on any of my forums (my users would try to abuse it, fail, then get bored and not bother with it at all).