Rules
Terms of Use

Topic Options
#254689 - Fri Jan 21 2005 04:42 PM Another question about quiz rankings
d2407 Offline
Participant

Registered: Mon Jan 03 2005
Posts: 31
Loc: Rochester New York USA    
I just noticed that a number of my "sunglasses" quizzes just plummeted in the rankings. I once within the past four weeks had four "sunglass" quizzes, the top one being something like #270. Now I've got three sunglass quizzes, the top one is 1158, and the average user rating for all my quizzes is down by (I think) more than half a point from where it had been.

I'm not really concerned with my quiz rankings; if I enjoy making them - and why bother to make them if I don't enjoy doing so? - and if they get accepted, and if I get heartwarming compliments from quiz takers, hey, all's cool with that.

But I do have some questions about the ranking process:

1. Are rankings based on EVERYONE who has ever rated a quiz? Or do individual ratings of a quiz expire after some time (and if so, how long)? How about if someone rates a quiz, then their membership terminates? Is their rating pulled from the database?

2. I think I read a while back that some people - I'm not sure who... editors? gold members? - have their ratings for a quiz somehow count more than others. How does this work?

3. Does an individual only get one chance to rate a quiz, just as his/her score only counts the first time he/she takes a quiz?

4. (Roughly) what percentage of quiz takers rate quizzes? In all honesty, I doubt I've rated more than 1% of the quizzes I've taken.

Like I said, I'm not griping about how my quizzes rank, but I'm curious about how this all works. Even though I bet the rankings are based on a fairly small number of ratings, for a quiz to drop more than 700 places in the rankings in a month, or to lose roughly half a point in its rating, strikes me as odd, and I'd like to know more about how the system works.

Top
#254690 - Sat Jan 22 2005 09:32 AM Re: Another question about quiz rankings
Bruyere Offline
Star Poster

Registered: Sat Feb 10 2001
Posts: 18791
Loc: California USA
Glad to see you're enjoying it regardless of the ebb and flow of ratings. As I've said frequently, the mathematical reasons don't concern me and that's not my domain but I'll take a stab at one or two or these.


1. Are rankings based on EVERYONE who has ever rated a quiz? Or do individual ratings of a quiz expire after some time (and if so, how long)? How about if someone rates a quiz, then their membership terminates? Is their rating pulled from the database?

I'm pretty sure the first 'Sub' answer is no, but I doubt the latter is true. Not sure though.


2. I think I read a while back that some people - I'm not sure who... editors? gold members? - have their ratings for a quiz somehow count more than others. How does this work?
I'm almost certain this isn't true, but, most people who do become gold members care about the site enough to do better work. That's my guess. Editors are chosen for their quiz-writing abilities amongst other things, so even if they weren't editors, they'd probably rank higher.

3. Does an individual only get one chance to rate a quiz, just as his/her score only counts the first time he/she takes a quiz?
Oh, I think this is a YES.

4. (Roughly) what percentage of quiz takers rate quizzes? In all honesty, I doubt I've rated more than 1% of the quizzes I've taken.

Now, this intrigues me in someone as curious about ratings as you seem to be, which I understand. I don't remember the stats, once again, (annoying, isn't it) but, I'd say I rank about half of the ones I play if I feel they're excellent. I rarely take a quiz I feel is poor as I can guess what sort of thing is going to be a good quiz and, as I edit many different types of quizzes, I disqualify myself for about a year! I might not take one of Uglybird's until next year on a complicated subject.

So if you do appreciate the quiz, I'd rate it if I were you. Or Thumper's method, if you can't think of something good to say.. I tend to go by that one.

I've seen the rankings fluctuate myself and was happy that one of mine was finally rising a few thousand in the ranks as I didn't think it was that much of a dud, so when it was near the bottom of the pile for a few weeks, I wondered why.

Now, the trick is to make your quizzes user friendly, not in any way "dumb them down", but, make them attainable by a larger group of people. I have a tough quiz or two that rank pretty high. You just have to make it pleasant to read.

Oh, yesterday I had a daily mixture that was the epitome of why I don't take many quizzes unless I know the author's style or the subject matter interests me. I had about five out of my ten that had the nitpicking year differences for some date, or a star's birthday, that just fell flat for me.
If I did see someone with five of those on a quiz, I might rate it as not so good. One perhaps, but not three or four on the same event.
_________________________
I was born under a wandering star.

Top
#254691 - Sat Jan 22 2005 10:50 AM Re: Another question about quiz rankings
agony Offline

Administrator

Registered: Sat Mar 29 2003
Posts: 14743
Loc: Western Canada
I'm pretty sure there are some old threads on this matter that explain the mathematics of the ranking methods in as much detail as Terry is willing to go into.
The question of some people's rankings counting more than others - there is something built into the sytem to prevent a member from taking out a whole bunch of memberships in different names (forbidden in any case) or rounding up all their acquaintances to take out memberships in order to either build up or knock down the ranking of an individual quiz. That is, if your best friend has written a quiz, and you get everyone you know to take out a membership solely in order to rank his quiz highly, it won't really work (So don't try it!). Exactly HOW it won't work, though, is totally beyond me.
When I first started writing quizzes, ranking meant a good deal to me, and I would eagerly await the updated ranks every week. Now, though, months will go by without my checking on ranks, and I am sometimes surprised to see which quizzes have risen, and which sank. I go much more on the notes that I get from players (In fact, I'll tell you a secret - I keep copies of all my compliments, and have even been known to gloat over them when I need a pick me up)

Top
#254692 - Sat Jan 22 2005 08:22 PM Re: Another question about quiz rankings
MotherGoose Offline
Multiloquent

Registered: Mon Apr 22 2002
Posts: 4799
Loc: Western Australia
I keep a casual eye on my quiz rankings, not because I really care about them, but because it fascinates me to see how they move up and down. I know Terry puts a lot of hard work into working out an algorithm that is as fair as possible, taking lots of factors into account. After he fine-tuned it the last time, one of my quizzes which had been really popular plummeted in the rankings, but since then others have risen to take its place.

I have written a few quiz series. At the moment, one of my quizzes is number 3 in its category. What I can't understand is that this quiz is the second in the series and the first one in the series ranks far below it. Yet, in my humble opinion, the first quiz is better than the second quiz. I usually use all my best trivia in the first quiz. Oh well, I guess it just proves that Agatha Christie was probably right: "Authors are always said to be no judge of their own books" (or in this case, quizzes).

As far as rating other people's quizzes goes, I ALWAYS rate a quiz after I've played it.
_________________________
Don't say "I can't" ... say " I haven't learned how, yet." (Reg Bolton)

Top
#254693 - Sat Jan 22 2005 10:04 PM Re: Another question about quiz rankings
d2407 Offline
Participant

Registered: Mon Jan 03 2005
Posts: 31
Loc: Rochester New York USA    
I have played a bunch of quizzes tonight, and doing my utmost to be a good sport, have rated each one.

This has, unfortunately, prompted more questions:

1. On the algorithm, is someone weighted by the number of quizzes he/she has taken, quizzes graded, or something else? I know the formula favors those who are more active.

2. I unknowingly played several quizzes I'd already played long ago - I learned I'd played them when my score information told me that I had. But I was still given an opportunity to grade the quiz. So what happens? If I play it again, will I not be given the chance to grade it? Did my grading of the quiz today not count because I'd already played it once? Or does the database track that I've played once, and that I've graded once, separately?

Top
#254694 - Mon Jan 24 2005 04:48 AM Re: Another question about quiz rankings
Taesma Offline
Prolific

Registered: Fri Jun 20 2003
Posts: 1179
Loc: Bay Area California USA      
Quote:

As far as rating other people's quizzes goes, I ALWAYS rate a quiz after I've played it.




Me too. I can't really see a reason not to. It takes all of two or three seconds, less time than answering a question, really.
I feel that if I liked a quiz, I should reward the author for entertaining me. And if I thought it wasn't very well done, I feel that maybe rating it not so well will give a more deserving quiz a chance to rise in the ranks.
I don't care about the rankings so much myself, but it is kind of fun to see how many times your quiz has been played. And the ranking does let you know if you've written a quiz worth playing, I guess. And maybe be able to improve your work next time.
_________________________
"A bookstore is one of the only pieces of evidence we have that people are still thinking." ~ Jerry Seinfeld

Top
#254695 - Mon Jan 24 2005 05:13 AM Re: Another question about quiz rankings
Shrivats Offline
Enthusiast

Registered: Wed Jun 30 2004
Posts: 463
Loc: Dubai, UAE
Sometimes if I forget to rate a Quiz, I go back a little later, play it again (it won't count of course), and then rate it, so I think you can rate a quiz even if you have played it before. As for keeping an eye on the ratings, well, I've written my first series over the past two months, so I'm keeping a very close eye on them, but then that might be because I'm a relatively new quiz maker.
_________________________
Life is like Pi, natural, irrational, infinite, and very important.

Top
#254696 - Mon Jan 24 2005 09:05 AM Re: Another question about quiz rankings
d2407 Offline
Participant

Registered: Mon Jan 03 2005
Posts: 31
Loc: Rochester New York USA    
Quote:

Sometimes if I forget to rate a Quiz, I go back a little later, play it again (it won't count of course), and then rate it, so I think you can rate a quiz even if you have played it before.




I just tested this. Played an excellent quiz, rated it, went back, played it again - and the system allowed me to rate the quiz again.

Although I am sure that behind the scenes, something noted that I'd already rated it, maybe that would be a "nice to have" improvement, not allowing a user to rate a quiz more than once (the system already reminds people who've PLAYED a quiz more than once).

Top
#254697 - Mon Jan 24 2005 11:13 AM Re: Another question about quiz rankings
Bruyere Offline
Star Poster

Registered: Sat Feb 10 2001
Posts: 18791
Loc: California USA
I'm going to assume you read Terry's last post about the ratings change? The one on the 'What's New?' link on your member page?
Have a look at that one.

I think I'll paste it in here for your convenience as it looks as though we're having the 'leap' Terry mentioned:

Quiz Ranking Changes

I have just built a new quiz ranking algorithm to better handle the thousands of quizzes we have nowadays. The original one wasn't designed for nearly that many and has caused problems recently.

Just to let you know that it may not be that accurate just yet -- it will take a little bit for me to test it properly.

One change is that *ALL* online quizzes will now get a rank, whether it's been played 20 times or not.

Here are the rules:

- new quizzes do not get a rank. A new quiz will be any quiz that has been online for less than 20 days. New quizzes will float at the top of their categories for as long as they are "new". The timer for newness starts when it is FIRST placed online.

- all other quizzes older than 20 days, regardless of # of times played, gets ranked. If the system is unable to find enough data about a quiz (i.e. it has been played too few times), the system will determine what the average rating of all quizzes is and insert your quiz in the middle. In other words, quizzes that have been played only a few times will gain a "middle rank". As soon as it is played enough times more it will be ranked based on its data, and may leap up or down in the rankings.

There, end of quote.

I'd almost bank on the fact that your rating will only count once and that it's obviously the first time you rate a quiz that counts, whether you rated it the first time or not.

So in my scheme of things, we've just had the shuffling happen so that the ratings your quiz had accumulated just were entered in.

I had one quiz go up a couple ten thousand in the ranks the other day as I kept on wondering if it was the ultimate dud quiz or what. Whew, what a relief, it isn't the worst dud in the world apparently.

As I've seen some amazing things happen with that ratings system over the years I've been here, I'm just tranquil with the fact that it's really one of the fairest ways.

How does it concern the average quiz writer? Well, I think that from my standpoint as an editor and quizwriter, I know what sorts of quizzes reach a larger audience and what sorts will reach a 'chosen few'. If I want to take a risk of writing a different sort of quiz in a popular category, then it might plummet in the ratings for those people expecting a typical type of quiz and it might rise eventually when people who know my work take it.

I bet one of the things that affects your overall ratings is developing an 'audience' of fellow quizwriters who have you on their 'notify me if this person writes a new quiz' list etc. This isn't the same thing that Agony described but a true sense of community amongst writers.

When Terry gets back from his grand tour, he'll undoubtedly tweak the system even more, don't worry about that, but, essentially, the ratings system really has improved to include more fairness and exclude the possiblity of cheating.
_________________________
I was born under a wandering star.

Top
#254698 - Mon Jan 24 2005 11:25 PM Re: Another question about quiz rankings
Nannanut Offline
Multiloquent

Registered: Sat Jan 10 2004
Posts: 2470
Loc: Wollongong NSW Australia      
If your quiz has not been rated by players the arbitrary number of times over the first 20 days of its life - the system automatically sits your quiz at around 3.8. The 3.8 ranking will sit you well below the half way mark - probably close to 38,000th in quiz position. There were over 56,000 quizzes onsite at last week's rankings.

It will sit there until 20 rankings have been recorded and then it will be sent to its new home. One player I talked to recently had had a quiz sitting on 3.8 for weeks and had virtually forgotten about it.

Suddenly it was in the top 100 which meant it had been given its true ranking but the jump was definitely very substantial.
_________________________
Nothing matters half so much as you think it does.

Editor: Movies, Celebrities and General

Top

Moderator:  agony