Home
 
 
11 March 2006 @ 05:52 pm
In which a mere four star rating indicates a problem.  
The science fair was very cool indeed. I hope I didn't judge too harshly - I didn't check off anything higher than "average" on my judging form unless the project actually seemed better than most in the category in question. That's what I was told to do, so it should be OK as long as most of the other judges also followed the directions properly.

I got email recently from a company I had bought a book from through amazon.com. The company specializes in selling education-related books to teachers. They complained that I had given them a four star rating, which is "a killer on Amazon. It keeps customers away." They wanted to know what they'd done wrong to get such a low rating.

I replied telling them that a 4 is not a low rating, it is the second highest rating I could give. I bought one book from them once, and had no problem. If I gave my highest rating for that, I'd have no way to recognize sellers who really distinguished themselves. If buyers are giving 5 star ratings willy-nilly to everyone they do business with who doesn't give them a problem, I'd say it's they who are doing something wrong. What's the point of having five different possible ratings if you treat them as "bad," "bad," "bad," "bad," and "acceptable?" If that was the aim, the rating system should only have two possible ratings. I told the booksellers that what they were doing "wrong" was catering to educators, who care about and give thought to the sensible use of grading systems.

I've had many eBay sellers call me a "Terrific eBayer A+++++++++" simply because I paid my bill, so thinking that high praise should be difficult to earn apparently puts me outside the mainstream. At least eBay has the sense not to have five different official ratings, or they'd probably have the same problem Amazon has. It wouldn't be acceptable to rate someone merely "good." Only a rating of "outstanding in their excellence" would be considered positive.
 
 
( 10 comments — Post a new comment )
R. S. Buchanan | יהשוע: wee bastard[info]yehoshua on March 11th, 2006 11:20 pm (UTC)
I had a similar experience recently in a restaurant that had little "how was our service" cards on the table. Everything was good, nothing was miraculous, so I checked off all 4s and left a healthy tip.

I was accosted on the way out by the waitress, who was furious that I was trying to get her in trouble by giving her a 4 in all categories. It seems that management only wants to retain staff who can get 5s at all times. Gosh, I just sort of figured that the tip (22% because I didn't feel like messing with fiddly small change) was enough praise and generosity on my part. That will learn me, and BTW see if you ever get a nickle from me again, kid.

Stupid humans.
The Frozen Capybara: cthulu eats chicken[info]frozencapybara on March 11th, 2006 11:33 pm (UTC)
That reminds me of the time a car salesman urged me to give his dealership the highest rating because "anything less gets us in trouble with the company - we lose 90% of points for that one rating." This is notable because this was the car dealership which had ordered the wrong car for me, blamed me for their error, failed to get me a car for another month before they finally deigned to give me someone else's cast-off car (which did meet my requirements, but still), and then made me drive up to the (hour's-drive-away) dealership twice to get the car before finally getting their act together and having the car ready to drive off the lot.

If that's what passes for five-star service in the car dealership world, I feel a little less weird about giving a company five stars because they don't screw up.
Andrew: dubious[info]pawo on March 11th, 2006 11:35 pm (UTC)
Dear God, you didn't give them the five-star rating they wanted, did you?
Z-Gryphon: angry Scotsman[info]z_gryphon on March 12th, 2006 12:52 am (UTC)
"anything less gets us in trouble with the company - we lose 90% of points for that one rating."

"Well, maybe you should have thought of that before you sucked."
The Frozen Capybara: cthulu eats chicken[info]frozencapybara on March 12th, 2006 01:15 am (UTC)
I would have told the guy that, if I hadn't been rendered utterly speechless by the sheer audacity of the guy for asking for it. As it was, I gave him a funny look and left.

Unfortunately, I'm not sure I ever got the survey - I was planning on giving them the lowest rating pretty much across the board, but didn't get the chance.
MegaZone[info]zonereyrie on March 12th, 2006 12:16 am (UTC)
I've noticed this too, but I stick to my guns. When I do surveys with a 1-5 scale I treat 3 as average - 4 is above average, 5 is excellent. I will not give out 5's just because service was *good*. Maybe 4's, but if service was just acceptable - no screw ups, nothing frustrating, then it is probably just average and gets a 3.

If I did a survey for my dinner last night at 99, the food itself would be a 3 or 4, but service was a 2 or 3. The waitress screwed up my order serveral times - I order the combo appetizer with nachos, no sour cream or olives, and it came with a HUGE dollop of sour cream, OK, I was able to spoon that off, but annoying. I ordered the salad with no tomatoes - it came with two large tomato slices. My drink was low when she brought my dinner - and ran out right after, and she didn't come by again for quite a while, and then I had to flag her down to get a refill. And the place wasn't packed, it was almost empty, and I could see her hanging out at the station chatting with other staff, joking around, so it wasn't being busy.

And I bet if I left a poor review she'd be angry too.

3 is a 'meets expecations', 4 is 'exceeds', and 5 is 'outstanding' - if you want a high rating, you have to earn it.
Gower[info]gower on March 12th, 2006 01:00 am (UTC)
This is naturally on the mind of anyone who gives students grades all the time. When I taught at Texas, my chair told me that using "C" as the average grade was unacceptable.

I am part of the problem. I do not use "C" as an average grade. I suppose I ought to. On the other hand, a grade is not the same thing as a rating, is it?
The Darker[info]mindways on March 12th, 2006 06:40 pm (UTC)
That's the difficulty with grades - what they "mean" isn't defined anywhere except by their usage. I've seen some schools publish little "what the grades mean" booklets - but unless nearly the entire faculty adheres to those standards, the reality of what grades are given will outweigh a little pamphlet.

[random rambling]

Both with grades and with ratings - inflation so that 'highest' is the expected norm presents the difficulty that you lose all way to distinguish excellence. There are arenas where this is less damaging than others - on eBay, for instance, while knowing that a seller will go above and beyond the call of duty for me is *nice*, it's nowhere near as vital as knowing that they're not a schmuck who'll stiff me, or someone impossible to contact. Learning (!negative) is more important than (positive).

I think one reason that ratings of '4' on Amazon or EBay or whatnot are viewed as unfortunate may be that companies tend to have a rating composed of mostly 5s, some 1s, and a few statistical anomalies in the middle. A "4.XXXXX" rating can be viewed as a shorthand of "ratio of 5s (OK experiences) to 1s (bad experiences)", in which case that '4' comes across as '1/4 of a 1' to all but those customers who bother to click through to the detailed reviews.

It would be neat to see a site that used a more information-rich system - for instance, instead of an average rating, a small bar-graph indicating proportion of each of the five ratings - but making such a thing unobtrusive, information-dense, reasonably browser-generic, and not excessively bandwidth-consumptive would take a bit of effort.
[info]krzzl on March 13th, 2006 03:13 am (UTC)
I hate rating systems...
But I think lots of other people, especially corporate, manager, marketing etc. types like them, because they can get "quantitative justification" for things that tend to be qualitative.

Some examples:
Part of our annual performance reviews use a rating scale that is usually 1-5. I have had some managers who have given me lots of high ratings, and others who feel that a 5 is impossble, and absolute perfection. If it's impossible, why put it on the scale?

An unnamed department at an unnamed academic institution puts a lot of weight for tenure on getting 95% or higher of agree and strongly agree on course evaluations.

I prefer the rubric form of grading/rating. You still end up with some kind of letter or numerical score, but it gives you some criteria to work with, and seems a little better.

generic mike: crimbo deathmatch[info]mpgalvin on March 13th, 2006 09:14 am (UTC)
a GEORGE ZIMMER approacheth...
ah, you got your internets in thier commmerce. or perhaps they got thier commerce in your internets. anyways, commerce is made of ones and fives and stupids. on teh bizzernets, everything is divided into BEST EVAR and SUCKS, and nuanced opinions are a legendary rare loot... at best.

i mean, just look at game / movie / pc hardware reviews... OH WAIT DONT.