A quiz rank is now based on two components:

- how users playing the quiz have rated the quiz at the end
- how good, in general, submissions from the quiz author are. If a user constantly submits high quality quizzes that require few corrections from editors, have a lot of interesting information, and are generally well written (a la thejazzkickazz), the user gets a bonus to the rank of *ALL* his/her quizzes. Likewise, an author who constantly submits junk, ignores guidelines, etc gets a penalty to ALL of his/her quizzes. The system handles all of this automatically at rank time.

The thinking is as follows: A ranking should reflect the overall quality and popularity of a quiz. Popularity is gauged by users, quality is gauged by editors.

Terry

Hmm. This strikes me as strange. Am I to understand that, as of now, one of only two criteria by which individual quizzes will be ranked will be the subjective, collective impressions of the editors as to the overall quality of other quizzes?

I don't want to come off as catty, but... It seems to me that logical category boundaries are being crossed. Have you considered, as an alternative, having a little icon (like the sunglasses or thumbs-ups) that goes next to the name of the quizmaker as opposed to the name of the individual quiz?

I would think that by this expedient you could convey the message you want to convey (namely, "here is a quizmaker whose quizzes tend to be of high quality") without running the systematic risk of also conveying an incorrect message (namely, "this particular quiz is extra-fine" when in fact it is not (even Homer nods, after all).)

(Another, possibly lesser pair of concerns this raises in my mind is as follows. First, it seems to me that there is a real risk of obscure but quality quizmakers falling through the cracks and not receiving their "due" credit under the system. This would result in what Lani Guinier characterizes as "double-dipping" by the majority; popular quizzes on popular topics get the benefit of both criteria, but unpopular quizzes on unpopular topics can fail to receive even the kick they should get under the "overall quality" criterion. Second, the quizmaker who manages to produce one true chef d'oeuvre but who the rest of the time produces silly fluff gets a downward rank for the one really great quiz, so no one knows to expect it to be good. Thus diamonds in the rough and one-hit-wonders get inaccurately downranked, a disservice to all the quiztakers.)

Maybe (probably) you thought all this through already, but I thought I would throw in my $.02 as a noninsider quizmaker.

Terry, a player that consistently submits junk should be sent a personal message to tell them that they are. Players may think they're submitting good quizzes, but if editors don't agree, then how are quiz makers supposed to know that people really do hate their quiz or quizzes?

I hope that the new system isn't biased towards quizzes based on the USA, because I feel that some of my Australian-based quizzes have really copped it, in terms of being poorly ranked.

If you feel that I have submitted junk, please let me know, and I'll do anything I can to raise the level of the quiz up to a satisfactory standard.

The only real change that I see here is the addition of the editor component to the ranking. I think that can only improve the rating of well-thought-out but obscure quizzes. I think the editors have a pretty fair grasp of what's going on in the quizmaking arena. Unfortunately, average quiztakers tend to rate a quiz based on how well they do on it, not how good they think the quiz is.

I like the idea of having an editorial component, but I'm not quite so fond of having the quality of one quiz effect the rank of another. The quizzes I wrote have been written already. If I submit a series of truly amazing quizzes now, it doesn't affect the quality of my previous quizzes and it doesn't seem like it should have any effect on their ranking.

Nmck...I can assure you that players who submit 'junk' are notified in a very thorough manner. Most quizzes in that substrata are not allowed online. Of course, junk can be a subjective term...I may find that any quiz on a sit-com or soap opera is 'junk', no matter how thrilling the quiz may be. As for there being an American bias, unfortunately I do think such a thing creeps into the place to a degree...most of the players on this site are American and hence they will tend to rank the quizzes on topics familiar with them more highly. However, this does not always have to be the case. A majority of my quizzes are not based on U.S. based topics, but many of them seem to be ranked very highly nonetheless.

I have seen one of your quizzes thus far, and I certainly wouldn't categorize it as junk. However, that subjective element comes into play for every quiz in terms of how the players will rank them. I think the player ratings are still the key component of the quiz rankings.

Kevin...I don't think there need be a concern for the phenomenon you described. If you consistently submit great quizzes your rankings will reflect that. Terry's new ratings system works with a multiple of factors, not just player ratings and editor ratings. In the case where you find a quizmaker that consistently submits marginally good quizzes but in one case submits a true gem, it will be recognized by the players and therefore will rise above the rest of that particular players oeuvre.

What I am wondering is how the algorithm truly works. Do the players get some points off every time they receive a correction message from an editor, or does any correction counts, or there's someone out there just looking through your history and deciding.
There's also the notion that the system might not be very fair in a sense that people are better informed on some subjects than other. For example, I may have very well made quizzes on one subject and make poor, junky quizzes when I try to make something on a completely different subject.
It just sounds like the new system is overly generalized and may be subjective.

Lanire, everyone, including editors, receive corrections every now and then. It's a daily part of the FT life. Editors receive them when they submit quizzes to another category.

If you submit a quiz and you get a rejection, then you fix that correction. That stands for something. Your ranking will not be affected.

However, if you refuse time and time and time again to correct the errors, and it seems like the editors have to pull teeth to get your quiz online, then it can affect your quizzes.

When editors look at quizzes and quizmakers, we look to see if the quizmakers promptly correct the errors, are not rude or vulgar, and if the quizmaker makes quizzes which are basically error free.

Editors are not biased against anyone, regardless of where they live.

I like the idea of the editors having some input into a quiz ranking. I'm not sure it's a good idea to have the rankings of individual quizes be partly based on the quality of all of a person's quizzes. But, I could be wrong, so what the heck, let's try it . I do like the idea of giving quiz makers who consistently make good quizzes some kind of symbol by their name.

Clarifications:

- the editor component is based on the quality of quizzes submitted by authors. It is, in general, blind to the type or category or obscurity of the quizzes. Popularity has *nothing* to do with the editor component.

- how are the editor components set? Here's how. Every FunTrivia author has a background "quality rating". As users submit quizzes, FunTrivia editors can slightly change this rating. If you traditionally submitted poor quizzes but now submit brilliantly written quizzes, your rating will slowly rise over time. Likewise, if you constantly submit poorly written quizzes that ignore the category guidelines, your quality rating will drop over time. We think it's only fair that quizzes created by people who obviously spend a great deal of time researching, writing, proofreading and abiding by the category guidelines get a ranking boost over other quizzes. Players will be rewarded for the time they put into their work and penalized for spamming the quiz system with drivel.

Terry

I guess that will work, though I still have the same concern as before (that in the situation you described the poor quality quizzes of an author will also slowly rise higher and higher as the author submits good quizes, and, conversely, a good quiz might see its ranking plummet as the author submits a few (unrelated) horrible quizzes and sees his/her rank drop).

I guess what I would prefer to see would be separate rankings for the quizzes and the authors. Though the link's broken now, there used to be a spot on the site ranking the most popular authors. This would seem to be the ideal spot to include data on the overall quality of an author's quizzes (as measured by editors). As suggested above, the highest quality authors could then get some sort of recognition/symbol by their name, which would still be recognizable to peple browsing through the categories and looking for quizzes by reliably excellent authors.

nmck, regarding the Aussie quizzes, I want to reassure you that they will not be down-rated because of the quiz content. I had three quizzes rated in the top 200, and all three were proudly and unashamedly Australian.

As a matter of fact I see that this may be a 'plus'- we seem to be the 'flavour of the month' with our American friends.

I edit in a couple of categories (one of which is over 80% American content), and I like to feel that I show no bias regardless of nationality.

I've got to agree with Kev (which should come as no surprise since he was agreeing with me); rather than have general quizmaker reliability act an an imperfect (and inevitably, some of the time, misleading) proxy for actual, particular quiz quality, it makes more sense to rate individual quizzes and quizmakers independently from one another.

I suspect the reason this wasn't adopted in the first place is because there is another concern as well: that the player ratings are a poor reflection of intrinsic quality, and that editor rankings would be more accurate... and yet no one wants to assign the editors the herculean task of individually rating every single quiz.

So perhaps the thing to do is have: (i) the player ratings; (ii) the quizmaker icon o' excellence; and (iii) an expanded "editor's choice" icon, something like where the editors identify roughly the top 20% in their categories for a "bronze", the top 5% for a "silver" and the top 1% for a "gold" icon.

Xaosdog...I'm not certain from your comments, but it appears to me that you would prefer to have a return to the previous quiz ranking system, in which quiz ratings were entirely dependent on the vagaries of the quiztakers. Surely the new algorhythm Terry has put into place is no worse than the previous 5 pt. system wherein a fantastically written quiz could be hammered by quiztakers simply on account of the fact that they scored poorly on the quiz. I would much rather have some sort of weighted ratings system than one which is completely reliant on public opinion.

I'm not suggesting that the new ratings system is perfect, and I imagine that Terry will tweak it here and there for the next month or so to make it as equitable as possible, but I find that it makes much more sense to rely on multiple indicators in determining whether a quiz is of high quality or not than to rely entirely on a democratic vote of the people.

Well, I certainly see your point, Jazz -- like I said, the player ratings are at best a poor reflection of intrinsic quiz quality. Reform was called for.

I'll go further -- the thumb icon means significantly more than the sunglass icon -- and would mean significantly more even if as many thumbs were awarded as sunglasses.

I'll go still further -- I once took a frolic through the top 100 quizzes as rated under the old system, and found that there was a tiny percentage of diamonds in the rough, but that they were just as likely as lower ranked quizzes to be without organizing principle, ill-written, and lacking in pedagogical value.

Yet further (as a means of showing that I am not biased against the new reform for selfish reasons): as far as I can tell my own position in the ranks went unchanged with the new system, and if anything I would expect to do "better" than before under the new system, since about 15% of my quizzes are "editor's choice" quizzes.

Nevertheless, the purist in me sees problems with the just-implemented reform. Rather than tweak a fundamentally misleading measure (player ratings), I would prefer to leave player ratings alone, and interpret them with a shaker of salt, and have in addition a quizmaker rating, based entirely on the better-informed, more objective views of the editors.

I would probably rely on the quizmaker rating to the exclusion of the player ratings; other players, with biases closer to those of the "norm" than mine, would likely rely entirely upon the player ratings; many players would probably benefit from both ratings.

But being able to evaluate independent measures -- effective at different levels of operation -- independently and separately from one another would be more valuable (I have to assume) to everyone than the new system, which involves calculating both factors but then hiding them, rendering each less informative.

Consider: the factors are not two different measures of the same thing, but rather reflections of different informational aspects of two different levels of the same thing. To "average" the numbers, we are (a) creating a mathematical construct which lacks a basis in reality and (b) turning the different things which each factor measures into statistical noise, rather than simply reporting the very information each factor was conceived to measure.

So: why generate the interesting and valuable data that a quizmaker rating represents, only to throw it largely away? Show us the data!

Terry,

I started out writing this as a private message (b/c of the first part) but decided to cut and paste it to here (b/c of the second part).

I don't want to say this in the forum [although I have done so now], but: since two of my quizzes are now in there, I HAVE to assume that the new system REALLY DOES reflect quality better! ; )

No, but seriously, the "glory" is nice, but my not-precisely-humble opinion remains that we'd get better info if you posted the top 30 player-ranked quizzes in one place, and the editors' top 30 quizmakers in another.

Still, I've put in my $.02, and I'll stop hounding you about it. There's no denying that it works better than the old way.

And, in the final analysis, it is probably correct to say that the new system will work very well on the right-hand side of the distribution; that is, the Top 30 that results will really be a best of the quizzes by the best of the authors. And that is, frankly, just about all I care about, since I am not really interested in seeking out the stinkers-by-good-authors. My ONLY concern is losing the gems-by-bad-authors -- but it is entirely possible that the latter is a mythical beast.

This strikes me as basically unfair. If a poor author shapes up (perhaps by being exposed to good quizzes by Jazz et al.) and writes a good quiz, his/her rankings are penalized by earlier indiscretions. In my opinion, a quiz ranking should be based solely on the qualities of that particular quiz. If you want to tweak the ranking system, I think a fairer way would be to add bonus points if all the "Additional Info" was filled in (you would need to develop an algorithm to weed out spurious Additional Info).

No root, if an author shapes up and submits a good quiz, then his author rating will rise. Author qualtiy ratings are far biased in the direction of recent submissions over old stuff.

Terry

I have discovered that this new ranking system, where editors get to have a say whether a quiz is good or bad, plays into the hands of editor's quizzes. You'll find that most quizzes with sunglasses were created by editors, but prior to the new ranking rule, quizzes made by non-editors were more common in the top 1500.

For example, the "Fawlty Towers" category has seen dramatic changes since editors were allowed to control game rankings. Ladymacb29 now has two "Fawlty Towers" quizzes in the top 1500 (one quiz at 556) and I have none. Whereas, prior to the new rule, I had 2 or 3 in the top 1500 (one quiz at about 300) and those quizzes were ahead of Ladymacb29's quizzes.

I rather suspect that editors are pumping up the average quiz ratings of their own quizzes, and this is not fair. Editors, enough is enough!

I did say at first that USA-based quizzes would be favoured, but this hasn't turned out to be the case. That's one of the rare positive comments that I can make about the new system!

In reply to:

---

I rather suspect that editors are pumping up the average quiz ratings of their own quizzes, and this is not fair. Editors, enough is enough!

---

Editors may not manipulate their own ratings. On the 1-10 scale, Terry has assigned all editors rankings of 8. Yes, editors are assigned an above-average rating, but that does make sense- quizmakers have to show excellent quizmaking skills in order to become editors.

I agree with crisw. The editors' skills at making quizzes are not the same as the general populace. Editors know what should go into a good quiz and know what sorts of things make quizzes rated higher.

Also, editors are only rated an 8 on a 10 point scale. What does this mean? People can be rated higher than the editors.

I forgot to point out something: Before Terry modified the ranking this last time, I had a quiz that was in 12th place. Where is that quiz now? 36th. So my quizzes actually are now rated LOWER than they had been.

Nmck,

I just did a rough analysis to see if your claims could be supported. My analysis included examining the top 50 ranked quizzes from each category. I found that out of a total of 950 quizzes among the top ranked in our 19 categories, 33% (or 321) were created by current Funtrivia editors, while 67% (or 629) were created by others. This clearly belies your claim that 'most' of the top 1500 quizzes were created by Funtrivia editors.

Now, while it is true that most of the top ranked quizzes were not created by editors, it perhaps still seems strange that 33% of them were. I think there are a couple of very plausible explanations for this phenomenon. First, as has been suggested by my co-editors, the editors at Funtrivia are selected from among the very best quizmakers on this site. It only makes sense that, as a result of this, you will see a preponderance of quizzes created by FT editors among the top ranked, and I would guess that prior to the changes in the ratings system recently affected by Terry, FT editors were still very healthily represented among the top ranked quizzes at this website. Second, the ability of editors to create a ranking of quizmakers is rather new. What does this mean? It means that the work of rating quizmakers is not complete. Over the next month or two the editors will attempt to rank a majority of the active quizmakers on this website. This will certainly lead to further dwindling of editor representation among the top 50 lists. Please allow the editors to do the work (which, might I add, they all do *without pay*) of 'balancing' the playing field here at Funtrivia over the next few months.

As Crisw said, each editor has been given an automatic '8' rating. The '8' rating corresponds to a person who creates 'consistently good quizzes'. A ranking of '9' would suggest that the quizmaker is among the 'top 1%' while a 10 is reserved for the 'top .1%'. Since there have been around 6,000 total quizmakers at this site, a ranking of 9 could conceivable be given to 60 quizmakers. I would give high consideration for ALL of our editors to be among the top 60 quizmakers on this website, and will debate with anyone who suggests otherwise. However, the editors have willingly accepted the blanket rating of 8, in order that their be some fairness amongst the quizmaking populace here at Funtrivia.

Now, Nmck, if you still feel that the system is unfair, even after having read the above. Please list either the names of the editors you feel are undeserving of representation amongst the highest ranking quizzes, or the offending quizzes, and we can have a look at the quality therein. I think that, if accusations are made, there should be some factual data to support them...please present it here.

Jazz

[Edited for faulty math...08/14/02]