The criteria for setting New Questions actually says:

Ask yourself whether English speakers around the world of varied backgrounds would have a reasonable chance (say, 80%) of getting the question correct. If the answer is no, then the question is too difficult for this form. We encourage you to submit obscure/specific questions in quizzes.

This would mean on average you should get an average of 8/10.

Now I'm not having a moan as the questions are the same for everyone, but we were being pushed to play and rate New Questions, if everyone is getting a low score are they still going to play this game.

I don't know about anyone else but an 8/10 is an achievement lately and not the norm, so what is happening with the New Questions being submitted?

80% is a goal to aim for, but I know that I personally would not reject a well-written question on the grounds that only 50-60% of players would probably get it right.

A picture is worth a thousand words. So, here's a snapshot. As of 4PM FTT (FT Time) the following figures were taken from the results pages of the 15 running sets of the New Question Game. I noted not the expected 8/10 but a more realistic 5 out of 10 level (highest ranking of a 5 correct player). Remember we aren't talking the 80% expected by the rules for each question. This is a measure of how many weren't able to answer at least half the questions presented and I think that speaks more to pmarney's posting.

Set#--high 5 rank out of total--total =/less than that 5 rank--% less than that level
1--29/39--11--28
2--14/31--18--58
3--12/40--29--72
4--15/30--16--53
5--17/29--13--45
6--9/32---24--75
7--3/33---31--94
8--14/39--26--26
9--19/46--28--61
10-22/30--9---30
11-20/39--20--51
12-8/28---21--75
13-30/48--19--40
14-8/44---37--84
15-12/38--27--71
How to read example using set #1: out of 39 total listed in the results table position 29 held a 5 correct out of the 10 questions. There were no five correct above that position in ranking. Some below that position may have had 6 or 7 correct but time penalties gave them lower scores than that high 5. There were 11 such players listed below that first 5. Eleven represents 28% of the total players who played set #1.

% of total players with =/less than the highest ranking 5 correct
329/546 = 60%

This means that more than half the players couldn't get half the answers correct to the New Questions. Does that mean they are too difficult? Should some be not accepted because of difficulty (whether it's obscurity or poorly posed questions)?
Not my call. I just took the picture.
I would recommend that someone who can see the numbers by Category check the figures that way to evaluate if there is a trend by topic.

The entire point of the New Question game is to figure out the difficulty and likability of questions to be used for other games in the future.

If you find a question is difficult, but you enjoyed the question, rate it well. If you did not like it, don't rate it as highly. If a question is far too obscure and nobody is getting it right, something will be done.

But, as I said, the game is designed for this very thing. So, yes, they are difficult questions a lot of the time, as they should be.

Also, these are all questions not yet used in hourly games. In a lot of games, high scores are a result of memory and repetition.

An average of 5 right in the new question game is to be expected.

80% of people getting a single question correct is a totally different stat to people getting 80% of questions in a set correct.

Just because the average people are getting 5/10, could possibly mean that there were a few questions in that set that were easy and a few that were hard. You can only expect people to know what they know and to know what they find out. That's part of the fun of writing questions - to figure out what people may or may not know. It's also part of the fun of playing these games - to find out what I know and what I don't know (and also what I can learn for next time).

I would also add that it can be hard to judge the difficulty of a question at the acceptance stage. The editors can certainly weed out the truly obscure questions, but one man's obscurity could be another man's piece of cake. The editors don't have a crystal ball we can use to tell if a question is going to be easy or rated well by players, and as salami has pointed out, that's really the function of the new question game itself.

As an author, I know I have often been surprised when some of my own questions turned out with a much lower percentage correct than expected, and vice versa. I'm sure every regular contributor to the NQG has similar stories!

Besides which, the 80% target was set because that is the level of difficulty appropriate for a number of games, which need to have new data fed in to keep them fresh. Questions that are popular (rated well) will be added to the appropriate game - I have seen some in Who's the Smartest, where you would not expect 80% correct except from those who have successfully memorised the old database. Even harder ones could end up in Obscurity. It wouldn't make sense for editors to refuse new and interesting questions just because they are harder to answer than their intended level of difficulty. Editors are not omniscient, and cannot always predict what players will or will not know. (I personally have a lot of trouble writing a question that is easy enough to get 80% correct, even though they may seem quite simple as I write them.) They can recognize insanely obscure and difficult questions, or vastly simplistic ones, but anything in-between is best categorised for difficulty by having them played. If it's too hard or too easy, it won't count towards the relevant badge, but it can still be used.

(This is based on having seen the numbers at the time I did the 5/10 but I didn't write them down. In terms of photo images, you might call it "Persistence of Vision", sorry Dali)
If I'd noted those who'd had 8, 9 or 10 correct responses. I'd only have counted about 5% of the players, the other 35% going to 6, 7 and slow 8s. I think that can definiely be equated to questions being nowhere near that "80% ought to get them right" rule.

But the reason for skewed numbers, as I said in recommendation, might be due to just a couple of the categories posing the hardship to players. I am not saying it for sure, but just that someone with access to the actual results should do a measure. The addition of a variable that is never guessed right in everyone's results would definitely skew the results (meaning 8/10 is the best you'd get right from the get go). A set that carried two battleship Qs and three stringed instrument music score reading Qs might kill a whole set for most people. Many of the sets do seem to be presenting only 5-6 categories and not the 10 you'd expect.

It has been weeks since I've seen a 10 Q game that didn't have two Qs each from a couple of authors in them (but they are often in different cats for each individual but sometimes the same Cat) and sometimes they aren't written clearly enough to get on a quick read through. They tend to be the ones that are complicated questions (could that be a contributing factor: Question size?). How often do editors suggest that an author save a difficult seeming Q to put into a quiz instead of putting it in Single Qs? Could there be a Miscellaneous: Tough as a Single Q Category in Quizzes established that authors could submit tough Qs (and at the five Q size?)?

That would be treating quizzes as a dumping ground for questions that have nowhere else to go, instead of expecting authors to craft a quiz with ten careful, usually related, questions.

There is no reason to expect ten different categories in a 10-question quiz, anyway. I have no idea how they are selected, but I know that the number of questions submitted to different categories is highly variable. Some categories will simply have more available questions, so will be overrepresented no matter what algorithm is used to set up the batches.

I understand that it is a different stat, but if you put 10 questions with 80% correct in a set and asked players to answer them, the average player would score 8/10. So if most players are scoring less, there is at least 1 question which doesn't have the correct difficulty. Of course, like with any average, it depends on who plays it, but the more people you test, the more accurate the figure is going to get.

Thanks mehaul, I was trying to put a follow up post yesterday on the lines of your first post as when I looked at the New Questions scores for all yesterday after 12 hours only 25 to 30% have scored 8 or more.

As I said originally it does not worry me at all because the questions are the same for everyone, but why do people stop playing certain games, in my view it's either because there are no more badges to win and low scores all the time, and as this was a Quiz the powers that be wanted us all to play and rate questions I just thought it was worth mentioning. Again I have enjoyed the responses and as I have said in the past it's good to talk

Today after 100 players only 10% have scored 8 or more

I used the individual set results because that 'All' results table stops showing results for the 100th ranking. There may be many more past 100. The best that can be said statistically from only seeing the top 100 is that at most the figure for 8 or more correct is 10% (though I sense you watched the 'All' page as it reached 100). My data was taken with 1/3 left to the day, a time by which a good percentage of the plays have been made. But a sample size of 500 is going to give a measure with an extremely fair confidence in values determined from a total population of 700. Here just a few minutes after you I saw 12 at 8 or above out of the 100 in 'All'. with set totals of 8.4.6.9,17,6,7,3,8,11,9,9,5,7 and 9 totalling 118 played and during my writing those numbers, the number of 8s on the 'All' page had increased by 2 to 14 (just shy of 12%).
Edit: I just found that the 'Winners' link on the NQG home page (Why have I never looked there before?) gives each sets winner along with the total players for that day in each set. Yesterday's total players of the NQG came out to be 785. My sample of 546 representted 70% of the population.

Now, do you want gloss or matte finish on that snapshot?

80% is a difficult target to achieve. The chances of finding 80% of people who play across the entire world is extremely difficult. A lot of questions are biased towards a particular country. Naturally, the percentage will be considerably lower because, while it may be a simple question in the original country, it's near impossible elsewhere.

I have come across several questions that are decidedly "easy", if I lived in the UK. But I do not, and therefore, get most of those type questions wrong.

So, while the TARGET might be 80%, realize that this game does not know which is truly 80% or not... They've never been played. The game is designed to find those 80% questions, not use only them.

I stand by what I said before. 8 is an extremely difficult target to achieve in this game. We do not know which questions will be rated high or low, or which will be considered easy or difficult. One of my, what I thought was, most difficult questions, ended up with 85% getting it right and has a rating of over .45. That's very good. I never would have thought it would get that. Alternately, I wrote a seemingly easy question, and it only has 20% or so getting it right, and it's rated pretty poorly.

So this game is designed to test those questions, and see where they fit. To say that one should easily get 8 right every time is not an accurate statement. These questions do not all fit the criteria; the game is testing to see if they do or not.

Please know that I mean no disrespect, so I apologize if it comes off that way.



Mehaul, you lost me when you started using numbers and I got confused. :P

The system uses the forwarding actions developed for quizzes to present Single Questions in the New Question Game, that is one editor determines worthiness (I don't want to say they don't confer but that that is not called for and that there is no 80% rule to consider in quizzes and that that system works there fine and dandy) So, the situation is as described (seemingly too many Qs which are harder than the requirement rule get placed in the NQG), how the Q will fair is unknown and mostly unknowable. I might recommend a system that could overcome that difficulty. Once an editor has accepted a Q and feels all the typoes and inaccuracies are ferreted out, submit the Q to the entire panel of editors to take! If 75% of them can't answer it, how would they expect the rest of FT to? Those failing to meet 75% there go back to the author for rework. Those passing the try-out go on to the NQG. I said 75% to cover the variation of the population from the desired 80%.

Okay and not upsetting me at all Swami, if the numbers aren't upsetting you. The discussion can't hurt the situation. Since you have mentioned one Cat in your example, I'd like to add another that seems to be accepting sub standards (ie, below the rule level and not meaning poorly constructed). But first, you mentioned a Geographic topic. True many may know UK answers, many may not, but for sure 99% don't know where Phanar, Pharaway is in the world so that Q would never rise to the required rule level. Yet Qs such as that are getting posted. The same goes for the Sports Cat in that, as you say, there is a niche crowd that will never miss the question and the rest of us will never do better than the 25% correct guess rate. Come on, how many know that a silver goal rule even exists in the favorite sport for 90% of the world? And in baseball there was a question about the infield fly rule which even the umpires have to confer on in many cases that topic being so obscure. Both were good questions but should have been seen as pushing it to get 80% to answer correctly.

Remember that questions determined to be tough can be made easier with hints (they are allowed after all) and answer sets can be set up to point toward the right answer. Those two atrategies aren't being employed in some cats I sense because some editors don't favor their use (from quiz experience?). I have had Qs rejected at first because the wrong answers were considered easily spotted as wrong. Well, dang it, there's that rule about MAKING the Q&As fit an expected 80% correct level.

BTW: Phanar is a Greek Orthodox College in Istanbul (I looked it up).

The problem is, mehaul, we don't KNOW what percent knows where Phanar is until someone writes such a question and it is put in the game. We cannot simply reject every question that looks too difficult. The ultimate goal is to reach 80%, true. However, it is simply not possible to have every question in the game reach that goal. I am starting to sound like a broken record when I say this, but this game is designed to figure out which ones DO reach that goal... It isn't comprised of only those that do.

Alright, I won't post anymore. I don't want to turn this thing into a huge argument (or, at least, I don't want it to escalate)

What?



Because we don't know everything.



This sounds exhausting. It's difficult enough to get working questions for the New Questions game that make the grade for the QQ badges and now we're discussing a series of editing sessions with multiple editors?



How can one know this, the amount of people who know it I mean.

The editors do their best to avoid placing extremely obscure questions online but sometimes we try to make the exception for well-written questions which may garner good response, even if they are a bit difficult.

This game will *always* be subjective in the sense that we may never know which questions will be well-received from a subject-matter standpoint (editors DON'T see the ratings after they go online, nor do they see the percentages) and it will always be a fair bit difficult. They are all *new* questions; we try to avoid duplicates. You will not have seen them. That alone should bring a sense of difficulty.

If someone asked 'In which city, straddling the border between two continents, can a religious scholar visit the Phanar Greek Orthodox College?', I'd find it a fairly smart question to place online; it doesn't require specific knowledge of the school to be answerable and, should the player find it interesting enough to read the info, they'll get something out of it.

But that doesn't matter-- I'm only one rating. Other players may hate the question and other players may love it. Does that make it a bad question? Does it only make it bad if 70% of editors get it right on the first go? Should it be rewritten? How many times?

Maybe I am in the minority, but I don't mind difficult questions in the NQG. I know that none of the ones that are too difficult will end up in the hourlies (except maybe in the Obscurity game or the GMM, if they ever need an infusion of new questions), but I like seeing them when they are well written, and I learn something from them. Those questions may not end up being fodder for hourlies, but they do make it interesting to play the NQG. I like the game in and of itself, and I will really like the badgelet when I am finally able to get it! In the meantime, enough good questions will come through at the ever-elusive 80% rating to make their way into the appropriate hourly game. It all seems to work well, in my view.

Sue

edited to correct a typo

''In which city, straddling the border between two continents, can a religious scholar visit the Phanar Greek Orthodox College?'

I love that question.
The ones I always fail to get are about American sportspersons, but I usually score around 70%. If I score lower it's because of the sports questions.
NQ gives a welcome variety of questions that are more global than many others.
That's not a whinge about the level of questions requiring an in-depth knowledge of American culture at all, this is largely and American site after all.
I'd be sorry to see the NQG lose its individuality as a quiz.

Hi shuehorn, we can be a minority together!

Please also note that while questions with around 80% correct are the most widely usable on the site, we are not ONLY looking for this level of difficulty. It's just a good target to aim for because authors tend to underestimate difficulty and there is a huge variance. The 80% questions are just very suitable for mixed games because those games are most enjoyable if the average scores are around 8/10 or 12/15.

Question Quest content has so far almost exclusively been used in mixed games and the system-generated bot quizzes. However the more the question base grows, the more we will also be able to use it in GC, Smartest and Piece of Cake, all of which require different correctness percentages. Any question that beats 55% and is generally regarded a fun one to play is a question that will sooner or later show up in a major game. This is why all of those qualify for the Question Quest badge.

(Given some time, even the more difficult ones will see the light of day. Can you imagine GC Impossible with those 30-45% ones? I can and I'm looking forward to it.)

Thanks for this insight, Wes. I was hoping that some of the delicious, yet difficult, questions that I have rated highly in the NQG might receive a place of their own in the future. I agree with you and am looking forward to it too.

Sue

Well, gosh darn. here is me in a position of enjoying the game, agreeing with most, no, all of the points made. I merely came up with a measure to back pmarney's assessment (and, yes, some opinions on the reasons for such). And everyone else has agreed that there are indeed many tough Qs in the NQG (I feel that job is done). It's that "guideline" in the Single Question submission criteria that's the problem. Take out the reference there to the 80% and it won't be a problem. I'm sure the editors are capable of weeding the 5% of the questions no one would get. I don't think even Impossible wants those in its bag of tricks.
Any editor that knows how a Q will fair in the correct answer arena, is the one I want to take to the race track.

This could just be me, but I still don't see the problem.

As Wes has said, this is a good target to aim for because authors tend to underestimate difficulty and there is a huge variance. Also, the game is looking for, and finds a home for, a wider range of questions beyond those that can be answered by 80% of all players.

Now, if the statement you were objecting to was something along the lines of "We expressly and unconditionally guarantee that 80% of players playing the New Question Game will be able to answer the questions correctly", I could see a problem, and a justification for removing such a line, but that's not the case, so again, I don't see the problem.

A lot of great points made here. I have no problem with the difficulty of the questions and I think using a game as a vetting tool for questions is a great idea. However, I do not think it should be a team game. With the variance in difficulty levels between sets and the more than occasional incorrect answers that slip through (I believe I had three or four just last month), it seems to make an uneven playing field for team game purposes. If a team score is brought down by an incorrect question, that seems off, to me. In fact, I am not sure there is any other team game where the possibility even exists. I think there is a pretty good reason why games like Obscure and Impossible aren't team games, and often NQ seems to fall closer to those than any other team game. Again, I think the game itself is great, and a perfect way to figure out where questions will fit best, but I don't think a team score based on that process fits that idea.

Edited to add: When I say incorrect answer, I mean a problem with the actual question and answers provided, not just getting a question wrong (I certainly had more than four wrong last month, lol)