Speaking for myself, I don't often look at this. If I am editing a really horrendous quiz, I may go back and look at the times "Yep, 23 minutes from start to finish", but mostly it doesn't figure in.
A lot of our best authors write their quizzes in something like WordPerfect, in order to access the spellcheck. (Anyone reading this who has spelling issues, please note).
So, no, don't worry about this. We usually only look at the times in order to confirm our own worst fears.