New Karma System (Theory)

Discuss features as they are added to the new version. Give us your feedback. Don't post bug reports, feature requests, support questions or suggestions here.
Forum rules
Discuss features as they are added to the new version. Give us your feedback. Don't post bug reports, feature requests, support questions or suggestions here. Feature requests are closed.
Locked
vph
Registered User
Posts: 31
Joined: Mon Apr 11, 2005 6:37 pm

Re: New Karma System (Theory)

Post by vph »

Guys,

I am attempting to nail down the updating rules, intially proposed by the author of this thread. Mathematically, I think it needs to be put in the right form, and it needs revision too.

In case you are not familiar with Bayes's theorem please read up on it. Something like this is a good source:
http://www.ericdigests.org/2001-3/old.htm" target="_blank

thanks. I'll be back later tonight.

User avatar
dhn
Registered User
Posts: 1518
Joined: Wed Jul 04, 2001 8:10 am
Location: Around the corner
Contact:

Re: New Karma System (Theory)

Post by dhn »

dhn wrote: Anyway, my problem with Karma is that whenever you allow people to down-vote other posts, it will be abused. If users don't like a certain post, not voting for them is enough. If you like it, give it a thumbs up or a funny or informative rating. There is no need for a thumbs down. A post is either informative, or it isn't.
I feel ignored. :(
Image

Magnotta
Registered User
Posts: 80
Joined: Wed Feb 09, 2005 12:49 am

Re: New Karma System (Theory)

Post by Magnotta »

dhn wrote:
dhn wrote: Anyway, my problem with Karma is that whenever you allow people to down-vote other posts, it will be abused. If users don't like a certain post, not voting for them is enough. If you like it, give it a thumbs up or a funny or informative rating. There is no need for a thumbs down. A post is either informative, or it isn't.
I feel ignored. :(
Same here. I thought I came up with a rather good idea for the karma system on how to do it, in a way which I believe would help cut back on some of the "cheating the system" that would normally happen. Only response was "sounds complicated".

warmweer
Registered User
Posts: 118
Joined: Wed Jul 09, 2003 5:27 pm
Location: Belgium

Re: New Karma System (Theory)

Post by warmweer »

not same here though.

Whichever system is used to calculate karma, it won't suit my "needs". Fact is I probably have a totally different idea on what karma is/should be.
But I like the brainstorming as I always like to look at matters from different viewpoints.

to vph : Bayes' theorem is nothing more than calculating probabilities after deciding on the probabilities of certain events. Once those probabilities are "identified", 99% of the problem is solved. But what's the probability of a correct answer to a post? What is a correct answer to a post? Bayes' theorem deals with objective parameters while karma rating is still subjective (as far as I'm concerned based upon the posts in this topic).

The closest I've seen to a " correct" definition of karma is : the way other people feel about somebody('s posts). Still leaves quite a lot of room for subjectivity (even prejudice).

Just to amplify on the point of prejudice: if you know somebody's post (history) and generally agree with that person, you will be inclined to rate the post higher, similarly a pain in the ass will get a lower than normal score even for a perfectly correct and informative post. It's human nature. It's easy to say that these will cancel each other out but I'ld like to see statistical proof of that (BTW you can't prove anything with statistics except that assumptions were wrong - you can never prove them right).

Again, my position is that karma is too dependent on the usergroup for it to be a standard feature in a board, but I re-emphasise: this topic can lead to a couple of mods.
Procrastination is my hobby, but I keep on postponing it.

vph
Registered User
Posts: 31
Joined: Mon Apr 11, 2005 6:37 pm

Re: New Karma System (Theory)

Post by vph »

Hi everyone,
Let me attempt to formulate this Karma system mathematically. Please shoot me down -- gently -- if I'm doing something stupid. :)

Let's start with Bayes' Theorem, which gives the technical tool of updating information [Ref: http://en.wikipedia.org/wiki/Bayes'_theorem ]

(I)
Let A, B be events. Then P(A | B) = P(A) * P(B | A)/P(B), where
P(A | B) is the probability of A happens given that B happens; and P(A) is the probability of event A happens. Same for P(B | A) and P(B).

This mechanism allows the probability of A to be updated , P(A|B), as the result of the occurence of event B.


(II)
In an alternative form, Bayes' theorem can be stated as:

P(A) = P(A|B1)*P(B1) + P(A|B2)*P(B2) + .... + P(A|Bn)

where P(B1) + P(B2) + .... + P(Bn) = 1.

In this form, the probability of event A is computed as the result of the occurence of event B1, B2, .... , Bn.


----

Here's the overview: form (I) allows the karma of a user to be updated from the karma rating of his post; and form (II) allows the karma rating of his post to be computed.

(to be continued)

vph
Registered User
Posts: 31
Joined: Mon Apr 11, 2005 6:37 pm

Re: New Karma System (Theory)

Post by vph »

How to update karma/rating after a thread is rated

Let R be the event that a user has rating P(R), which is a number between 0 and 1.

Let T be a thread started by the same user. Suppose T has the cumulative rating of P(T). The question is given T, what should the user's updated rating be?

View these quantities as probabilities, Bayes' theorem implies:

P(R | T) = P(R) * P(T | R) / P(T)

P(R | T) is the updated rating (karma) of the user as the result of the rated thread T.

P(T) is computed from the ratings of repliers to the threads. The specifics will be specified later.

P(R) is the user's current rating (karma).

P(T | R) is the probability that the user with rating P(R) posts a thread which receives a rating P(T). This is tricky to compute. I have no formula for this yet, but it can be done.

I claim that this way of updating karma prevents some of the abuses.


(to be continued)
[edit to correct the formula]

Now how do we compute P(T|R)?

There are 2 cases: (1) P(T) >= P(R), and (2) P(T) < P(R). Right now, I think the most sensible, and conservative way is to maintain that the updated karma P(R|T) should be "in between" the two ratings. This means that the karma is affected, but conservatively.

In case (1), we have P(T) >= P(R|T) > P(R). If you work out the math it boils down to, P(T) < P(T|R) < P(T)*P(T)/P(R). So we just assign P(T|R) to be something in between these 2 values, for example 0.5 * (P(T) + P(T)*P(T))/P(R).

In case (2), we have conversely P(T)*P(T)/P(R) < P(T|R) < P(T).


Conclusion, this mechanism gives a "by-the-book" way of updating karma. After, a thread T is rated with P(T) and based on the current karma P(R), the new karma is P(R|T).



What is *nice* what this method is that you can adjust the degree of reliability of the rating P(T). Let's say we conservative assign P(T|R) to be something "between the 2 values" as above; for example right in the middle, P(T|R) = 0.5 * (P(T)+ P(T)*P(T))/P(R).

In either cases, you can work it out with a little algebra that P(R|T) the upgraded karma is exactly (P(T) + P(R)) * 0.5

Now you don't have to pick right in the middle of the two values. You can be biased toward the old rating P(R), or the new old P(T). And the result will be according to your biased. This means that we can have a little meter available to Admins, who can then subjectively control the rate of update of their system. If they feel comfortable with the way their users rate, then they can bias toward new ratings, P(T). Else, they can be conservative, biasing toward the old rating P(R).
Last edited by vph on Thu Apr 28, 2005 4:38 am, edited 7 times in total.

profpete
Registered User
Posts: 140
Joined: Wed Dec 08, 2004 10:49 pm
Location: Wales, UK

Re: New Karma System (Theory)

Post by profpete »

Having just written a university dissertation based around filtering of spam, of which bayesian was the choice to implement, can I suggest you take a look at chi-squared algorithms. I didn't go into that part too deeply, leaving it to the evaluation for future research, but it may be worth looking into for this...

vph
Registered User
Posts: 31
Joined: Mon Apr 11, 2005 6:37 pm

Re: New Karma System (Theory)

Post by vph »

How to compute the karma of a thread P(T)


Suppose that the thread is rated by n users with respective ratings R1, R2, ..., Rn. Then, Bayes' theorem tells us that

P(T) = P(T|R1)*P(R1) + P(T|R2)*P(R2) + .... + P(T|Rn)*P(Rn)

First, the term P(T|R1)*P(R1) is exactly what the original author of this thread wanted. The rating of person with karma P(R1) should be based on his karma, P(R1). This rating is P(T|R1)*P(R1). What is P(T|R1) ? It is precisely the rating that the user with rating/karma P(R1) gives to the author of the thread. In other words, P(T|R1)*P(R1) is the effective rating of the person with karma P(R1).

Second, for this to work as Bayes' theorem, it must be the case that P(R1) + ... P(Rn) = 1. This is the normalization that we have been talking about. So the correct formula would be

P(T) = P(T|R1)*P(R1)' + P(T|R2)*P(R2)' + .... + P(T|Rn)*P(Rn)'

where, P(R1)' = P(R1)/Z, where Z = P(R1) + ... + P(Rn).

Same for P(R2)' , ..., P(Rn)'.

So there we go.....

vph
Registered User
Posts: 31
Joined: Mon Apr 11, 2005 6:37 pm

Re: New Karma System (Theory)

Post by vph »

These formulations bear high resemblance to MKruer's original formulation (on 1st page). One of the differences is that i my formulation, the karma of a thread is computed as:

P(T) = P(T|R1)*P(R1)' + P(T|R2)*P(R2)' + .... + P(T|Rn)*P(Rn)'
where, P(R1)' = P(R1)/Z, where Z = P(R1) + ... + P(Rn).
Same for P(R2)' , ..., P(Rn)'.

whereas in MKruer's formulation, it is computed as (I think)

P(T) = P(T|R1)*P(R1) + P(T|R2)*P(R2) + .... + P(T|Rn)*P(Rn)

which is the un-normalized version of the above, which doesn't follow strictly Bayes' theorem, becaues P(R1) + .... + P(Rn) could be greater than 1.



Note: because of this normalization every time a rating is received, the karma of the thread, P(T), needs to be recomputed. The number of steps taken for the update each time will be linearly proportional to the number of ratings.

MKruer
Registered User
Posts: 156
Joined: Sun Jul 20, 2003 9:01 pm

Re: New Karma System (Theory)

Post by MKruer »

I think they are the same, but I am going between 0 and 10 where Bayes' theorem is between 0 and 1. If I explained it properly it should be imposable for the rating to go above or below the min/max amounts. And for each standard deviation, something like 83% will fit between 4-6, and 14% will fall between 3-4, and 6-7, <%2 will fall in between 2-3, 7-8 ratings, etc…

I also added the idea of weight to give more athority to some users unlike Bayes' Theroy.

I could be totaly worng, though.

Locked