Standardisation Of Brew Comps

[rough60]

Hi all,
This is going to open up a massive bag of worms, and it is by no means directed at any judges or comp organisers, they all do a fantastic job and allow us all to have a bit of fun and get some feedback on what we make.
What I was thinking was, are we really getting the right feedback/results that we deserve, for example, my Kolsch won the NSW Hybrid class and my Blonde Ale didn't place, that Kolsch beat Barry Cranstons Kolsch (congrates Barry on your recent top results and I am just using your placings to clarify my point.) in the NSW comp, and his came 2nd in the ANAWBS mash paddle, (pressuming it was the same Kolsch). Now how can my Blonde ale beat my Kolsch in the Twisted comp after the Kolsch had performed so well in "bigger" comps?
This is just a question that may make you think, I know I prefered my Kolsch over my blonde ale, so I would have judged it higher on that basis, so do judges subconsiously judge beers higher if they prefer them instead of comparing them directly to the guidelines?
Again, I take my results with a grain of salt, and do enjoy the feedback I receive, but I don't beleive that they should dictate how you brew, or how you evaluate you beer.
This isn't a dig at any judge, comp, or organisation that runs them, but just something to think about, especially for the new brewer.
Cheers.

Edit to clarify what comps I'm talking about.

 

[Stuster]

I think that's a really good point you raise there, Luke. I've experienced similarly baffling differences in scores for the same beer in different competitions.

I was initially a bit confused by what you said until I re-read the results from the Top Twister comp and saw your blond and kolsch there. I guess it does come down to the fact that judging is subjective. It's really how those three people tasted your beer on that particular day (and all the other beers the judges may have). There are others factors that could come into it as well of course. Transport and storage of the bottles can make a difference for one. As well as how good the beer is in and of itself, there's also how those judges evaluate your beer in terms of the styles. So judges will have different ideas of what makes that style, as well as how much importance they put on the being to style.

It's an important thing to remember if you get bad (or good) feedback from comps, and the brewer should also trust their own judgment on that beer. They're the one who's probably drunk the most of it after all. :chug:

 

[rough60]

Yeah Stuster, things like transportation or even serving temp would change the profile of the beer, but when the the beers (again I'm talking about my Kolsch V's my Blonde ale) are shipped togother, presumably stored together, etc. I would expect simiar placings for each beer.
I haven't been lucky enough to attend a comp yet, but hopfully will be able to make the AABS and help out with cleanig classes or whatever, just to see how they run (and to taste a few brews).
Cheers.

 

[rough60]

One other point, I think that feedback on a "rough/out of style/off" beer is extremely useful, I don't think that many of us really get to try good quality examples of all the styles we make, the guidelines can be pretty loose, and it definately helps to improve our understanding of what ingredient or other variables help make the certain type of beer we are after.
Cheers.

 

[Kai]

Hi all,
This is going to open up a massive bag of worms, and it is by no means directed at any judges or comp organisers, they all do a fantastic job and allow us all to have a bit of fun and get some feedback on what we make.
What I was thinking was, are we really getting the right feedback/results that we deserve, for example, my Kolsch won the NSW Hybrid class and my Blonde Ale didn't place, that Kolsch beat Barry Cranstons Kolsch (congrates Barry on your recent top results and I am just using your placings to clarify my point.) in the NSW comp, and his came 2nd in the ANAWBS mash paddle, (pressuming it was the same Kolsch). Now how can my Blonde ale beat my Kolsch after the Kolsch had performed so well in "bigger" comps?

I'm confused, when did your blonde beat your koelsch?

 

[Kai]

.
I was initially a bit confused by what you said until I re-read the results from the Top Twister comp and saw your blond and kolsch there

Ahah

 

[rough60]

Yeah when I reread me first post it wasn't that clear to me either!
My Blonde never placed in the ANAWBS or the NSW comp, my Kolsch wasn't entered in the ANAWBS but got 1st in the NSW comp, then the Blonde beat the Kolsch in the Twisted comp.
Hope this makes it a bit clearer.
Cheers.

 

[chris.taylor.98]

Hi Luke,

Having judged a few events here in Vic, here are some comments and observations in regards to issues raised.

i) From what I have observed it is extremely difficult to objectively judge beers. People not only have different preferences, but also have different abilities in what they can and can't taste. So with any given judging panel you will expect differences in results regardless of any other factors. I guess this is pretty tough when entries are only split by a point or two.

ii) Style guide-lines are only that, a guide-line. To really be able to discern a style correctly you also need experience. This can only be achieved if we have a good pool of judges to call upon that regularly that have experience in both tasting these styles on a regular basis, and in judging them in comps. Unfortunately here in Victoria this is not always the case, but we are working on it.

iii) Sometimes it is possible to get a bad bottle while the rest of the batch is fine. I had a beer that came 2nd in Vicbrew, 2nd in AABC, then second last in the Beerfest a couple of months latter. Again didn't make much sense to me at the time, but could have been a contaminated bottle that I entered.

iv) It sometimes incredibly difficult to split the top couple of beers in a flight, and in some regards there is an element of luck involved in how the places are determined. By the sounds of it you are doing very well, and I don't think that not placing in another competition should take anything away from placing at another. Rest assured that not many undeserving entries make it into the top 4 places.

Here in Victoria we are having a big push to try and get more people certified under the BJCP. This goes a long way to help improving the quality of the judges, and hopefully make judging consistent across the various competitions. Note it is not a necessity, but it really does teach a lot especially to less experienced judges like myself.

If you have questions about particular results, and are unhappy about the feedback given, or have questions about it, then try and get in touch with the competition organisers. Usually they judges names are published, so it is not too hard to pass feedback to them. The orgainisations should be responsive to competitors if they really want to improve the judging process, and the appeal of their competition. I know that I personally would welcome feedback from people who's beer I had judged.

If you really want to know about what goes on comps, then get involved. There is always a place for one more, and there is usually comps will be more than happy to get you into the judging process. Also you will learn a lot about brewing and the styles by doing this.

Cheers

Chris

 

[Stuster]

I think going along to a comp and doing some stewarding (at least ) is a great idea. I'm sure Ray would appreciate it.

I guess if you look at the ratings of commercial beers on say ratebeer, with beers that will often be more similar in terms of condition, there's a huge variation in people's appreciation of different beers. Vive la difference! :lol:

I know that I personally would welcome feedback from people whose beer I had judged.

+1.

 

[chris.taylor.98]

I think going along to a comp and doing some stewarding (at least ) is a great idea. I'm sure Ray would appreciate it.

I guess if you look at the ratings of commercial beers on say ratebeer, with beers that will often be more similar in terms of condition, there's a huge variation in people's appreciation of different beers. Vive la difference! :lol:

You can also request to be a 4th non-participating judge in a flight ( ie your scoring and judging does not get counted ).

To be honest, you really aren't going to get an appreciation of it till you try it. You don't have to be a great brewer to be a great judge, and its not life or death (still haven't received any death threats from the first comp I judged, god knows I probably deserved some ).

So get in there and give it a go, you will be glad you did.

 

[rough60]

Chris, thats exactly what I'm getting at, beer is a personal preference. I think I've mentioned on here before that I make beer that I like or would like to try, if it fits a comp catagory I'll enter it. I was making the point that, if you make an APA for example that you love and it doesn't do well in a comp, don't be discouraged. Even the judges in a comp have their own ideas as to what fits (and how well) into certain classes, maybe it's the range in the style and the judges experience within that style that dictates how a certain beer fits in.
For example, a judge may know an APA must be 10-28 EBC and 30-50 IBU (roughly), if 9 out of 10 beers are 20+EBC and 40+IBU and the judge normally drinks APAs in this range, a beer in this range would possibly be judged higher, because of personal experience, than an APA that was 11EBC and 32 IBU for example.
Also alot of beer styles overlap, which throws another variable in.
Hopefully will get a chance to attend the AABC and help out.
Cheers

 

[capretta]

yeah i just stewarded for the first time at the nsw comp and it was interesting for sure. one thing i will say is that best of show is not nessesarily representative of the best beer as there were different judging panels, and over the weekend different judging combinations. some judges consistantly marked higher and some marked lower. you also find if you have several crap beers in a row then a good one, that may score higher than if that beer followed several excellent ones.

rest assured though that there was not a bad beer that placed, as it was quite easy to seperate poor and middle beers from the outstanding ones, but the difference between the outstanding ones was small.

as to which beer expresses the style best? well apart from blatant errors that is probably the must subjective bit. have you read the guidelines for fruit specialty? something like " whatever best displays the perfect marriage between whatever beer style and whatever fruit the brewer uses.." pretty open ended if you ask me..

 

[newguy]

Like others have already said, how well a beer does depends on the judges, the time of day, what they had to eat prior to judging, and the other beers in the flight. There are probably other factors, but these have been the most obvious to me over the past 10 years of judging.

• Judges: we all have our preferences. Part of the process of becoming a good judge is learning what your preferences are, and properly compensating for them. You like malty beers? Try not to mark malty beers higher than they should be. You like hoppy beers? Try not to punish the maltier ones. Two vastly different commercial calibration beers prior to the start of judging (one malty, one hoppy) really help to clue judges in to these types of preferences and therefore leads to more consistent judging. All judges sit down and judge these two beers, the results are quickly tabulated, and the average, high, and low score for each is discussed. Again, the purpose isn't to embarrass anyone, merely try to align everyone's tastes. Once the calibration beers are out of the way, judging begins.
• Time of day: like it or not, judges taste things differently depending on the time of day. We all do. Sometimes a beer's placement depends on whether it was judged in the morning, afternoon, or evening.
• What the judge had to eat prior to judging: quite self-explanatory. Most judges will try to avoid heavily spiced food, but every now and again you'll find one that just had to have a curry for lunch when they know they are scheduled to judge cream & blonde ales in the afternoon. Related to this is the order of judging. Most competition organizers will try to organize the schedule so that the flights get progressively more full flavoured & bitter, instead of the other way around. My worst judging experience was trying to judge the light hybrids (cream ale, Kolsch, blonde ale) after doing the strong Belgian class. I couldn't taste anything!
• Other beers in the flight: self explanatory. Our club's most experienced judge & brewer always sums it up by comparing a brewing competition to a karate competition. Your chances of medaling depends not only on your skill, but on who else shows up. Don't read too much into relative placements. Same thing goes for comments on judging sheets. However, do pay attention to trends - if you enter the same beer into 3 competitions, and 2/3 agreed on some aspect, then you can safely believe those comments. If you get really weird/contradictory comments from one competition, don't necessarily believe them. Maybe they mixed up your entry with someone else's (it does happen, not often mind you), or perhaps that one bottle was bad/infected. Whenever I run across a really bad beer, I always call for the 2nd bottle just in case. If the 2nd is better, grade that one, not the first. But do note on the judging sheet that the first was bad so that the entrant can address their bottle washing/sanitation procedures. When I was in charge of coordinating the judging at my club's last comp, I made this a policy. We did run across quite a few entries where the 1st bottle was off, but the 2nd was okay.

Finally, the most important thing that will guarantee consistency is to have experienced judges. To get experienced judges, you need more competitions, and more importantly, more entries. So by supporting homebrew competitions, homebrewers are helping to ensure consistent judging.

One last thing: for the hell of it, I sent one of my beers to a competition in Australia (from Canada). The comments & scores I received were virtual mirror images of comments & scores I have received from local and US competitions. From this admittedly small sample, I'd say that the judges you have there are pretty damn good.

 

[Thirsty Boy]

I really appreciate the comments of the experienced judges here, its a great insight.

Some of the things that newguy said really struck a chord about what good judging should be like. But, and not having a go at anyone because I cant do it myself; Judging is going to be like everything else in the world... the majority of people who do it are going to be competent at a fairly average level, there will be a few really good judges who do things like self calibrate for their own preferences, and there will be a few really bad ones who might have passed their driving test, but who still speed, run red lights and cant reverse park to save themselves.. If you happen to get two or three of the latter style judges on your panel... you results are going to be rubbish, and visa versa.

A lot of it is going to come down to the experience of not only the judges, but the organisers of the comps. Do they know the judges, do they have a feeling for their relative strengths and weaknesses, are they aware that Joe Blow over there is fine judging porters and stouts, but cant taste Diacetyl or DMS to save his life, and we shouldn't put him on the light lager panel... Of course, if there is a struggle to get enough judges to just get everything tasted, then the poor buggers haven't got a lot of choice have they? and the quality of the results/feedback is bound to suffer.

Which I think means that the rest of us slackers, need to get off our bums, get ourselves to a comp, and stick up our hands to help out; judging, stewarding, lugging boxes of entries... whatever. What Chris said about being involved as a non-participating judge sounded like a fantastic way to really get up your levels of knowledge and confidence.

I also love the notion of the two commercial calibration beers. Is this a common practice at Aussie comps?? if not, it should be. I can really see it helping to iron out the highs and lows and also as a tool for the organisers to get a really solid feel for some of the various merits and faults of different judges.

Time to start whipping the palate into shape and put my money where my mouth is I s'pose.

Thirsty

 

[bconnery]

The crux of the issue, as has been mentioned here a few times in different ways, is that a competition attempts to apply standardisation and objectiveness to a subjective process.

I've had the same experience recently with a stout of mine. Gold medal score in the BABBS club comp, 1st place in the QLD qualifying comp, didn't place, didn't medal and didn't score that well in the other QLD state comp.

There could be all sorts of factors. Judging by the score mine was related to a bad bottle but it is hard to say. At the QLD qualifying comp I had a 10pt score difference between two of the judges and the third and the comments made it sound like they were tasting a different beer...

Without wanting to come out in favour or against the BJCP and open that can of worms, I would argue that the more standardisation you can have in place the better as it reduces the impact of the subjective factors.
This does mean that some brewers, myself most definitely included, Australian comps don't always have a place for sour orange or christmas beer , might find themselves out of style, but it doesn't really matter. You enter what you can if you choose to, and you see how you go.

I also helped out at our club comp recently, and it is a reasonably big one, and let's remember, what we ask of the judges is for them to sit down starting at around 9am, drink beer all morning with nothing but plain crackers or similar for sustenance, rate each one, take notes, and still remain objective right up until they are judging the strong ales...
I think they do a pretty good job if you ask me!

 

[newguy]

Which I think means that the rest of us slackers, need to get off our bums, get ourselves to a comp, and stick up our hands to help out; judging, stewarding, lugging boxes of entries... whatever. What Chris said about being involved as a non-participating judge sounded like a fantastic way to really get up your levels of knowledge and confidence.

By all means, do help out. I've organized the judging at our comp before, and I can tell you what a stressful experience it is. Especially when several judges who promised to be there don't show. Even if you steward, you still get to sample the entries once the judges are done. It's a fantastic way to hone your tasting skills without the pressure of judging. You will have a significant advantage once you do decide to take the judging plunge.

While I'm at it, I think the most valuable and most overlooked aspect of the judging score sheets happens to be feedback (I also think of it as justification). Feedback involves telling the entrant what to change or what was objectionable so that the next time they brew the beer, they can earn a higher score. I can't tell you how many times I've received score sheets that say nothing but positive things, but the score is 28/50. Or 33/50. C'mon! If the beer scores a perfect 50, then and only then does the judge not have to offer feedback. Anything less than 50 requires that the judge somehow justify their score.

For example, here are the comments on a Scottish Heavy 70/ that I entered in a somewhat local competition a while back. If you can tell me what the judge didn't like about it, please let me know. I can't figure it out for the life of me.

Aroma: 6/12. Malt is present in aroma.
Appearance: 3/3. Nice amber colour, good clarity, good head retention.
Flavour: 10/20. Good flavour with lots of malt but with enough hops for balance.
Mouthfeel: 2.5/5. Creamy mouthfeel, very good clean tasting with little diacetyl.
Overall Impression: 6.5/10. A good beer that I would be happy to drink.
Total: 28/50

This is perhaps one of the worst score sheets that I've ever seen, and I've seen a lot. The major question here is "what do I have to do differently next time to score higher?" From these comments, I'll never know. I didn't mean for this post to be a personal vent. I just want to stress what I feel is an important point for any judges reading this: justify your score.

 

[Fatgodzilla]

By all means, do help out. I've organized the judging at our comp before, and I can tell you what a stressful experience it is. Especially when several judges who promised to be there don't show. Even if you steward, you still get to sample the entries once the judges are done. It's a fantastic way to hone your tasting skills without the pressure of judging. You will have a significant advantage once you do decide to take the judging plunge.

While I'm at it, I think the most valuable and most overlooked aspect of the judging score sheets happens to be feedback (I also think of it as justification). Feedback involves telling the entrant what to change or what was objectionable so that the next time they brew the beer, they can earn a higher score. I can't tell you how many times I've received score sheets that say nothing but positive things, but the score is 28/50. Or 33/50. C'mon! If the beer scores a perfect 50, then and only then does the judge not have to offer feedback. Anything less than 50 requires that the judge somehow justify their score.

For example, here are the comments on a Scottish Heavy 70/ that I entered in a somewhat local competition a while back. If you can tell me what the judge didn't like about it, please let me know. I can't figure it out for the life of me.

Aroma: 6/12. Malt is present in aroma.
Appearance: 3/3. Nice amber colour, good clarity, good head retention.
Flavour: 10/20. Good flavour with lots of malt but with enough hops for balance.
Mouthfeel: 2.5/5. Creamy mouthfeel, very good clean tasting with little diacetyl.
Overall Impression: 6.5/10. A good beer that I would be happy to drink.
Total: 28/50

This is perhaps one of the worst score sheets that I've ever seen, and I've seen a lot. The major question here is "what do I have to do differently next time to score higher?" From these comments, I'll never know. I didn't mean for this post to be a personal vent. I just want to stress what I feel is an important point for any judges reading this: justify your score.

Can see why you are confused. Was this the only judging sheet you got (was there only one judge). In amateur judging though, you get what you pay for.

But from my limited experience, there is a difference between brewing a good beer that you wish to show off and brewing a beer that meets the style requirements. State titles such as recently held should, IMO, be about brews made specifically for the championships to predefined style criteria. It is a judging of the BREWER and his/her ability to brew to a strict style guideline, not the brew itself. If you make a fantastic beer - good for you. Enjoy it, share it with your mates. Enter it in club competition and agricultural shows. Have fun with it.

It has certainly been suggested before, but I think state & national championships should be restricted to a small number of styles that changes every year. It should be about the brewer having to make an entry specifically for the competition. In that way, we shall see just how good you really are as a brewer. It will also make it easier to judge as there isn't a wide range of brews that need to be judged. At the same time, judges get to concentrate on a narrow band of brews at each sitting and have a better chance to improve their competancy and accuracy.

That's my two bobs worth.

See you at the Nationals where once again I volunteer to take home the garbage !!

 

[Andyd]

For example, here are the comments on a Scottish Heavy 70/ that I entered in a somewhat local competition a while back. If you can tell me what the judge didn't like about it, please let me know. I can't figure it out for the life of me.

Looks like a green judge to me...

Personally I haven't judged any comps yet (stewarded one), but will, and am doing the BJCP course at the moment. However I have done a lot of umpiring/refereeing over the last 20 years, and one of the things that stands out is the difference between "understanding the rules" and "applying the rules".

In the context of beer judging, what does that mean? Basically when you start off as a beer judge you get exposed to the idea of evaluating one beer. Identifying tastes, aromas, etc. As a judge you start to develop a vocabularly that serves to allow you to describe what you're experiencing while you drink a beer. Experience improves this immensly of course, but if you have a basic idea of what you are looking for, you'll be able to (hopefully accurately) produce comments like those you've seen above. This is a skill that can be mastered by exposing yourself to many many examples of style individually.

The harder part is being able to establish a scale to work on, so that you can not just evaluate a beer, but also place it in comparrison to others you have tasted in a flight. This is much more difficult as a judge starting out. You might be able to detect maltiness in a beer, but how much? Is the level you've detected more appropriate to the style than the last beer you tasted, or the one before that? This part takes competition experience, rather than simply exposure to different styles, so it's much harder to gain, especially if you don't travel from state to state to judge (and I don't know of anyone who does that!). I've made some suggestions to VicBrew about how we could improve the situation with this respect, so we'll see if anything comes of that over coming months.

So if the judge who wrote that sheet is a new judge, it would not suprise me.

Rest assured though, that the panel needs to arrive at a point where all judges are within a defined point margin of each other (it was 7 points at beerfest last year I believe - I don't know about other comps, esp. in other states). So even if this judge was new, his score was almost certainly consistent (to an extent) with the other judges.

In short - if you do get feedback like this, consider the other judging sheets too and see if you can't derive some insight from those.

But whatever you do, please be appreciative of the new judges - without them volunteering their time there would be no competitions.

(Oh, and while you're at it, support your local brew club too - the administrative overhead alone in running the comps is massive and comes almost exclusively from clubs at the moment.)

 

[newguy]

Unfortunately the judge who produced that evaluation is not a novice. I know for a fact that he's been a judge for over 10 years, and he holds the Certified rank. The other judge who evaluated that beer had much better comments - malt was perhaps a tad low, body a bit on the thin side. Perfect! Just the sort of thing that will help me to improve that recipe. And get this - the 2nd judge was an apprentice (hadn't written the BJCP exam).

What's really odd is that the standard procedure is to pair novice judges with experienced ones so the novices won't write substandard sheets. They're also paired that way so that the novice can ask questions of the other judge in order to better recognise not only individual flavours and aromas, but also to help recognise the subteties of a style. In this particular case, the experienced judge could stand to learn a thing or two from the novice!

I'd also like to offer my $.02 worth regarding the issue of brewing to style and setting particular accepted styles for a given comp. In all honesty, to be an award winning brewer, you can't be proud. Time & again, I hear stories from winning brewers regarding how they totally missed what they were originally trying to brew. In the end, they objectively evaluated the result, perhaps asked other brewers, and entered it as something else. And won. In a sense, trying to hit a particular style is indeed difficult. Paring down the list of acceptable styles for a given competition would indeed distinguish the really good brewers from everyone else. However, that hurts your judges. We all learn through experience, and being exposed to as many different styles as possible helps us to learn. Having a restricted subset of styles in a competition may make organization easier, but it places restrictions on your judges. Again, just my $.02.

 

[Fatgodzilla]

I'd also like to offer my $.02 worth regarding the issue of brewing to style and setting particular accepted styles for a given comp. In all honesty, to be an award winning brewer, you can't be proud. Time & again, I hear stories from winning brewers regarding how they totally missed what they were originally trying to brew. In the end, they objectively evaluated the result, perhaps asked other brewers, and entered it as something else. And won. In a sense, trying to hit a particular style is indeed difficult. Paring down the list of acceptable styles for a given competition would indeed distinguish the really good brewers from everyone else. However, that hurts your judges. We all learn through experience, and being exposed to as many different styles as possible helps us to learn. Having a restricted subset of styles in a competition may make organization easier, but it places restrictions on your judges. Again, just my $.02.

I'll see your 2 cents and raise you two.
1. For a state / national competition, set the styles by making sure you really can't balls up one brew and enter it in another catagory. You'll know in advance what the styles are to be judged, so if you fail first time, you have time to try again (and again). Each year the style changes so you can't enter the same brew each year (such as with a barleywine / super IPA etc).
2. Judges should be honing their skills at local levels and not at a state level comp. That said, at last months NSW championships. judges sat in on two days of morning and afternoon sessions with three tables of judges. While there was some overlapping on smaller classes, at most the judges only got to judge one third of the entries. Some sat out a session or went home early, so realistically only got to judge about one fifth of the classes judged. So really there is nothing in my restricted classes theory that will dampen the learning restrictions on judges.

But I can see where you're coming from.