The meta post advertizing site evaluation seems to give very vague guidelines on how exactly we should judge a give Q&A when voting. It merely says:
Let's try to look at this site through the eyes of someone who's never seen it before, and see how we stack up against the rest of the 'Net.
Run a few Google searches to see how easy they are to find and compare the answers we have with the information available on other sites.
This seems pretty general. However, from some discussions on podcasts, I have formed an impression - perhaps an incorrect one, that the desired methodology for evaluating a specific question should be:
- Search the internet for answers to that question
- Compare non-SE answers to the (presumably, the best) SE answer.
- If SE answer(s) provide information materially better than the rest of internet, then vote "Excellent".
- If SE info is about the same, vote "Satisfactory".
- If the answer is worse than what can be found on internet (for example pure opinion, no sources, wrong), vote "Needs Improvement".
However, based on some patterns in the last Politics.SE evaluation, this is clearly not how everyone sees what the voting rules should be.
So, the high-level question is "What are the rules on how to decide how to vote on a specific question?".
Specific sub-questions are:
If the Q&A contains a poor question but a great (better than the rest of Internet) answer, should the quality of the question itself affect the vote (e.g., assuming that the same answer to an improved question rates "Excellent", would the vote be expected to change to something lower for the same answer to a poorer question?)
If the Q&A contains a great answer (better than the rest of Internet) alongside one or more poorer answers, should the quality of the worse answers affect the evaluation vote (e.g., assuming that the same answer if it was the sole answer rates "Excellent", would the vote be expected to change to something lower for the same Q&A that also have 2 awful answers?)
- Sub-sub question: would the answer to this depend on the relative votes of a great answer vs. poor answers? E.g. rate "Needs improvement" if the great answer is at +1 and poor one at +5; and rate "Excellent" if vice versa?
If the Q&A contains the same quanity of information as can be found - cumulitevely - on the internet, but carefully compiled from more than one internet source, does that qualify as "stacking up excellently against the rest of the Net"?