Have Ratings/Tasting Notes Gone Too Far?

I don't think it is a secret that I have never been a fan of published whisky ratings. Perhaps it's the aura of objectivity concealing the raw subjectivity that raises my ire.

Ratings along any scale: 1-10; 1-100; A-F; 1 - 5 stars; it doesn't matter- all carry with them the appearance of some application of "science" behind their bestowment. If, in fact, there were any way to truly assign a score to a whisky (or anything else for that matter) truly based on scientific discovery and a true absolute scale (think temperature), I'd have no issue with it.

The problem is that the scores are granted not on an absolute scale, but on a relative scale - and not a consistent relativity at that - even when scored by the same reviewer.

Ask any renowned whisky maven (or drinking buddy) his or her favorite dram, and you are likely to be given (if you're lucky) a short list, along with a litany of exclusions and limitations, about how "favorites" depends on the time of day, time of year, food accompaniment, mood or present company. And all that is quite fair.

How then can the same person give, what to all intents and purposes is a score based on an apparently absolute scale? I believe they simply cannot.

As a result all ratings are relative - a whisky is scored relative to the mood you are in, the other whiskies you might be trying, where you are sitting as you try them, what you had for dinner, the time of day, the argument you had with your girlfriend the other day, if you know anyone at the distillery, what you perceive others think of the expression, etc.

That being said, I don't like ratings, don't agree with the concept, and hereby call for a general ban on the use of any rating system in the whisky industry (Yeah, right. I'll keep dreaming). In the meantime, I shall continue to take ratings (as should you) with a grain of salt - even from the most respected reviewer.

"But, Kevin" you say "How will I know what to buy?"

I'm not calling for a ban on tasting notes, or for recommendations from "experts" to be abandoned - far from it. But tell the truth - as much as you love Jim, do you agree with every rating he's ever given? Of course not - but if, for the most part you trust his tasting notes, understand his style, and generally trust his opinion, then by all means, continue to do so. (By the way, replace the name Jim with Michael, Martine, Dave, Tom, Dick, Harry or whomever you like - I like Dave and Martine).

So, we've established that I dislike the ratings, but I'm fine with tasting notes and recommendations - as long as they contain actionable, understandable information and advice.

Such was not the case in a recent issue of a well-known magazine - a magazine that I read, respect, and look forward to receiving.

The magazine shall remain nameless, as shall the reviewer (who is "new") - but the comments contained in his tasting notes shall not be so lucky.

One of the reviews in question:

Edradour Burgundy Finish

Nose: Butyric, peach air freshener and a queasy cakiness. Like some showdown between two gateaux: as if a Battenburg attempted to out-nancy a French fancy.
Palate:  Incredibly sweet, sugary and then nothing of merit. Better than the nose.
Finish: Pass!
Comments: Not to my taste. Only useful as a weapon in hand-to-hand combat.
Score: 5 1/2

Let me translate the fancy words:

Butyric:

1. Relating to, containing, or derived from butter.
2. Relating to or derived from butyric acid.

Why not just use "Butter" or "Buttery"? Because "butyric" describes a very specific fault in a whisky, which is not detectable by everyone, but has been described as being similar to "baby sick". If you can detect it, it'll put you off. If you can't detect it, it won't bother you at all.

Gateaux:

• A cake or pastry, especially a light one filled with custard, fruit, or nuts.

If you are anything like me, you love you some gateaux. Morning, noon, and night, it's gateaux. O.K. I'm lying, call me uncouth but I wouldn't know a gateaux if I were allowed to eat one off Salma Hayek's stomach.

Battenburg:

• A fancy tea cake with marzipan.

Is that anything like a Pop-tart?

And the comments:

"Not to my taste."

That's a fair statement. Not everything is for everyone. I don't like beans with my breakfast.

"Only useful as a weapon in hand-to-hand combat."

O.K. Stop right there. How does this statement help me (a reader) decide if this whisky is for me? This isn't a tasting note. It's an insult.

And finally the score: a 5 1/2 out of a possible 10. This reviewer's co-reviewer gave the same expression a 7 3/4. As a reader, I want to know - why the disparity in scoring. Which one of them is right? To whom do I listen, since I don't know either of the reviewers...

Would it be helpful to know that one of the reviewers is a contributor to the Malt Whisky Yearbook  and buyer for Royal Mile, while the other is Michael Jackson's research assistant? Would that give the reviewers some street cred in your book?

But wait, there's more

Other terms used to describe the whiskies reviewed by the reviewer in question:

"Germolene" (An antiseptic ointment sold in the UK)

"Savlon" (Another antiseptic from the UK)

"Tropical fruit flavoured Germolene" (Variation on a theme)

"Pink Germolene" (Are we seeing a pattern here?)

"Silagey" (Silage is fermented, high-moisture forage to be fed to ruminants -  cud-chewing animals like cattle and sheep.)

"Wine Gums" (Wine gums are chewy, firm sweets similar to gumdrops, except they are not sugar-coated. They are popular in the United Kingdom and  do not contain wine.)

"Yacht Varnish" (What does this smell like? You see, I use an odor-free varnish on MY Yacht!)

"Squirt of cat pee" (Lovely)

It's quite clear that this guy has a hell of a nose. But what is he trying to do? Impress people with his ability to detect "water cigarette ash and horses" (another descriptor used) or give readers some information they can use?

Obviously it's the former, because there's little usable information for the average reader/buyer. AND if there's little usable information, is this a review or mental masturbation? Obviously it's the latter.

Once more on the scores

Surprisingly, this reviewer gave a 7 3/4 to Mackmyra. No, I am NOT making any judgment here about Mackmyra - I quite liked it (and it's an upcoming story on The Scotch Blog) - I'm just pointing out the utter futility in using an scale of 1-10 to define the nuances of any two (or ten) whiskies.

Since the reviewer obviously hated the Edradour, then a score of 5 1/2 must be a pretty low score on the scale - Does this make the advance in score from 5 to 6 to 7 to 8 to 9 to 10 exponential? If so, is the Mackmyra that much better than the Edradour? Is the Bunnahabhain 25 year old (which scored an 8) that much better than the Mackmyra?

The ratings lose all meaningfulness when viewed in a comparative way. Yet by their very nature, they must be viewed relative to other scores (and the scores of other reviewers). So even as an absolute scale - they are relative.

Either way - they are unreliable and must be done away with.

By the way, the reviewer in question's co-reviewer gave the Single Malts of Scotland Clynelish 1972 (available from the Whisky Exchange) a 6 3/4. I had the opportunity to try this the night before Whisky Live London, and I must say, a room full of people disagree with this reviewer.

One industry guy, who shall remain nameless, suggested a new scale whereby each whisky is rated relative to itself, and considers costs - for example "Good, but not worth the price" and "You should definitely go out of your way to get yourself a bottle of this stuff!"

Hmm. Simple, easy to understand, AND if someone trusts your palate, it's actionable information. This will require more thought...

Comments

[Philip Schoolcraft]

I agree with your assessment of whisky ratings. And to make matters worse, the unnamed reviewer you cite stole the "hand to hand combat" line from an old Monty Python routine about Australian table wines.

[P. Taborsky]

I couldn't agree more; there is too much fascination with ratings in too many publications. It is about time someone has said so!

As well, the problem is not only that ratings are relative and subjective, but they are linear. That is, ratings force upon one the perception that any two whiskies can be compared and one determined to be 'better' than the other. I.e., that all whiskies can be some how arranged on a linear scale from best to worst. Yet we are constantly told that one of the industry's strengths is its diversity (which presumably would mean that there is in fact a lot of incomparability).

[morgan]

Actually I found myself smiling at some of the snarky adjectives used by that first reviewer. The guy does this for a living and writing with a little character, for better or worse. Perhaps including footnotes, in the overly verbose style of David Foster Wallace, for us layman.

The "buy/don't buy" review seems overly simplistic for something as complex scotch. Instead of a raw (and subjective) score I would rather read about how the bottle compares with other similar scotch in its class/region/etc. Give me some information about the distillery, some history of the brand. I am more likely to make a purchase if any of that piques my interest.

[sku]

Here, here. Great post. I still blame Robert Parker for the preeminence of ratings. Ratings offer a shortcut for the lazy taster...I don't have to decide if this is good enough because an expert already told me.

I actually sense a hint of mockery in the tasting notes you excerpted. I happen to think extensive tasting notes, for both wine and whisky reviews, are mostly BS...often devolving into free association: "Nose of raspberry, chocolate, burnt cigar butt, followed by Normandy Butter, sea salt, pinecones, etc..." I have never found these type of notes helpful tasting or and enjoying a beverage.

[Jody]

Excellent post, Kevin. Coincidently, I think the latest issue of Malt Advocate has an article by Jim Murray about how to describe one's whisky tasting experience; an informative read. I never could remember tasting lingo. I usually stick to "tastes like dirt" (that's GOOD thing), "grassy", "sweet", "smokey"... common terms I think most of us can relate to.

[Harry]

I like tasting notes though like you I tend to use simple terms when it will suffice though when I really do smell ripe bosch pears or kumquats I'll use those terms rather than just 'fruity' or 'citrousy'. I won't use brand names though.

I've stopped using scores for a long time on any sort of scale but I think that industry guy has the right idea. I rate whiskies in four categories: 1) buy at any cost; 2) buy at a reasonable cost; 3) drink if offered; 4) pour in spitoon. I find that works fairly well to show amazing whiskies (1), good whiskies for the price (2), drinkable whiskies that I wouldn't buy (3), and dreadful drams.

[Serge]

Hi Kevin and distinguished readers of the Scotch Blog,

Well well well, as somebody who rates his malts daily (but does that as a hobby), I must say that I... agree with most of your comments.

Ratings are all individual, abbreviated comments. They should be seen as opinions, and readers should try to know more about the guy or girl who gave them before they rush out and buy a bottle. His or her tastes, his or her biases (professional or not), his or her experiences etc. Otherwise I agree it's totally useless.

People who don't share Robert Parker's tastes don't like Robert Parker's ratings, basta.

Now, I believe there are different ways of coming up with 'better' ratings (i.e. just a bit more reliable ones):
- Calculating average ratings (3, 6, 12 tasters)
- Blind tasting
- Tasting malts in short series, gathering similar whiskies
- And, above all, always using benchmark whiskies (Ardbeg 10, Macallan 12 etc.) to calibrate nose and palate before any tasting session. It's the best way of being more or less sure that '80 points today' mean the same as 80 points yesterday or 80 points tomorrow.

Now, as for scales, I agree with Kevin, they all mean the same. What's important is what's happening around the borders, and the small pinches that will let a whisky pass from 'good' to 'very good' or from '89' to '90' are exactly the same. People don't bother with 'intermediate' ratings, scores, marks, corks, thumbs up, whatever.

Just my tuppence, as a modest and humble self-styled whisky connoisseur (as some friends would say).

Santé

Serge

[Bamber]

Ratings are pretty meaningless, but people want them. People feel safe with absolutes. They want a label for their pain, and a score for their Scotch.

What's the best colour ?

[Serge]

Bamber, the best colour is the one you prefer. Same with whisky.

[Arthur]

Just spotted this old post and no need to be shy Kevin - I'm happy to name and shame myself...that was me.

That's my drinking style, I love AND hate whiskies and feel quite strongly about them. Sorry for all the rather specific descriptors, that's just what I smelt.

I decided to stop doing the Whisky Mag tastings a while back as I:
- was a bit busy with the day job and a new baby.
- suddenly realised I thin-skinned and opionated (the worst combination of all for a reviewer!) I realise my writing style was a bit like Marmite (a vegetable extract spread popular in the UK that was advertised on the basis of loving it or hating it...Laphroaig say they stole that line from them)

Leaving my own tasting notes aside, as buyer for SMWS previously and now Royal Mile Whiskies I have noticed a few things:
- The same whisky put to the same people blind on different occasions gets very different scores (tested on SMWS panel more than once and experienced RMW staff too)
- Punters suddenly get very excited sometimes about average bottlings. This especially happens seems to happen a lot with specialist customers/collectors who respond to internet buzz. Often the source of this can be Whisky Festivals, and we've seen it happen at our own Whisky Fringe. There's always a couple of spirits that the crowd suddenly decide is The Best within the first hour of opening our doors; the word spreads like wildfire, and the stampede starts. Often this is very unpredictable, and one year the clear winner at WHISKY Fringe was a 23 Year Old Guatemalan Rum. Limburg turns out some unusual buzz whiskies too.

The way to bypass all this is to taste whiskies blind, but very, very few people seem to do this.

At RMW staff tastings and when trying new things at the end of the day at our desks I tend to dish out glasses to staff blind. After they've said if they like them or not and given a rough score we tend to ask 'How much is that whisky worth?' Always interesting what staff say! (although they no-one ever seems to give a price over £50, the cheapskates)

I'm rambling now, I'll stop.