Archived

This topic is now archived and is closed to further replies.

Irise

A New Raiding Metric

32 posts in this topic

Anyone familiar with NBA knows that a player cannot just be measured by the main stats of Points, Rebounds, Steals, Assists and Blocked Shots. There are also efficiency stats which are even more useful for discerning the difference between players who come off the bench.

Similarly I was wondering if anyone out there would like to develop a tool for aggregating WWS reports and spitting out efficiency stats. For example, for a healer a stat would be the average number of deaths on Illidan while he or she was in the raid. One could refine this even further by only counting a death if the dead person was at least 20% of the focus for that particular healer. Once you get a few months worth of data under your belt you could see the difference between those healers who spam short heals to top the meters vs the players who prevent deaths.

Another good example of this would be the Robots before Illidari Council. It is far better for a priest to wait for the robot to nuke someone rather than spamming flash or CoH on the raid. However, WWS and other tools would not be able to pick up on this.

I think this idea could be expanded even further for DPS and Tanks as well. For example, when comparing two enhancement shamans it is sometimes hard to tell whether one does a better job of totem twisting. However, an efficiency stat which takes increase in raid dps into account could solve this problem.

Share this post


Link to post
Share on other sites

Right off the bat, it seems you'd have major issues with confounding variables. No two boss pulls are going to be exactly the same. The number of deaths could increase when Priest A is swapped for Priest B because B sucks at healing, the tank had a brain fart and forgot to hit Shield Block, someone pulled aggro, etc.

It could be mildly interesting to look at data like this, but there's so many different variables in even the simplest encounter that it's probably hopeless to actually isolate any one factor.

Share this post


Link to post
Share on other sites
Right off the bat, it seems you'd have major issues with confounding variables. No two boss pulls are going to be exactly the same. The number of deaths could increase when Priest A is swapped for Priest B because B sucks at healing, the tank had a brain fart and forgot to hit Shield Block, someone pulled aggro, etc.

It could be mildly interesting to look at data like this, but there's so many different variables in even the simplest encounter that it's probably hopeless to actually isolate any one factor.

The same problem is posed in basketball as well where each player's stats depends not only who was on the floor as well but how they played. There is no perfect way to calculate this stat but there are methodologies.

For example, suppose you want to know how much healing does a particular healer do on Illidan? First you bring up 10 different Illidan kills and normalize them for total healing done. Then what we are interested in is what fraction of healing is done by that particular healer while we take other healers into account. So assuming that the total healing is done is a function of the healers present we can do a linear approximation of it via a regression. The resulting data would tell us what fraction of the healing is done via each healer on average.

Another idea would to simply look at the number of deaths per Illidari Council kill. In this case we are looking at a very straightforward regression over 5-9 months of data.

I realize that this may involve too much work to actually be implemented (certainly beyond the amount of free time that I have). However, I thought it would be a good idea to toss around and see if anyone is interested. What I am suggesting certainly isn't anything new to those who have been following the development of PER in basketball and I hoping someone who has more experience than me will pick up this project.

PS: There was a thread a while back where Matlab was used to analyze combat logs. This would seem like an ideal project it.

Share this post


Link to post
Share on other sites

I guess my question would be, are the tools currently available to analyze raid performance really inadequate? I would think that just through WWS and general observation it'd be pretty apparent who's doing their job and who isn't. Also barriers to change aren't nearly as large in WoW raiding as in your example, the NBA.

I'm sure it'd be nice to have more precise tools, but I don't think the additional information you'd glean would be nearly worth the amount of effort it would take to model something like this.

Share this post


Link to post
Share on other sites
I guess my question would be, are the tools currently available to analyze raid performance really inadequate? I would think that just through WWS and general observation it'd be pretty apparent who's doing their job and who isn't.

I agree that instinct is more important than stats in assigning raid slots. (Even though I did read Moneyball. And my opinion doesn't matter, because I'm not a raid leader.)

However, no raid analysis tools that I know of make it easy to compare performance over time, which (I think) is at the core of what the original poster is talking about. As a simple example, for my own personal betterment purposes, I would love to see a chart of my DPS on the same benchmark fight from week to week. If I want to build something like that now, I have to manually construct a spreadsheet from my guild's WWS archives.

Share this post


Link to post
Share on other sites
The same problem is posed in basketball as well where each player's stats depends not only who was on the floor as well but how they played. There is no perfect way to calculate this stat but there are methodologies.

Personally the problem I see with any analogy to Real Life sports teams is that the number of datapoints are harder to controll and the variables are greater in an online situation. In your basketball analogy the coach probably has an order of magnitude more time per player for analysis than raid leaders do. Also the coach can run drills in specific situations that you just can't do in WoW. Also spec makes a much bigger difference to WoW, a mediocre rebounder can't walk up to a trainer and respec into a midcourt intercept specialist by handing over some cash. So how would your proposed metric deal with respecs? You can't compare my performance on Illidan this week as a COH priest with my performance next week where I have had to respec to be the guild DS bitch. I'm going to be performing two totally different functions and responsible for totally different targets. (Tank vs Raid healing)

Finally the tools we have, WWS etc, are notorious for not recording certain features well, especially things like Prayer of Mending or lifebloom. Analysing a healer is already more gut feeling and instinct than it is for DPS and I can't see a metric like you are proposing solving this problem or being a valuable tool.

To be effective you need a full timestamped combat log for the actions of every single actor in the fight or you just can't say if someone is playing well after the fact. Your original example of the robots leading to council is a classic example. Apart from one priest who is on reactive shield duty to stop 1 shots what can the other priests do other than try to keep the raid at full health? How would you know that the shield priest failed and that the target of the 1 shot nuke wasn't already at 1/2 health because the raid healers were slacking? With the currently available tools you can't, you have to have been there and paying close attention to everyone's lifebars.

Share this post


Link to post
Share on other sites

On the general topic of interesting metrics that you'd like to see, I'd like a mod or WWS filter for prevented deaths - this would show 5-10 seconds of relevant combat log after a spike in incoming DPS on that player. This would be interesting both to see which healers are consistently landing heals in that window as well as the steps that the damaged player takes or doesn't take to save himself.

Share this post


Link to post
Share on other sites

I don't like those fancy statistical metrics that are all the rage in sports these days. So there is the first flaw I see off the bat. I'd have to be convinced that they could be more useful with WWS than evaluating sports before I'd even consider tackling the number of variables you'd have to account for in a raid setting.

Just as an example from basketball that clearly shows what kind of ridiculous conclusions those metrics can lead to. For a few years looking at those efficiency numbers would have told you that Brad Miller was a better Center than Shaquill O'Neal. (This was when shaq was still in LA and his first year in Miami if I recall correctly), and that's just absurd.

We're getting saturated with stats now and while min maxing is great and all... People are starting to get lost in them IMO. Way back when my guild was still working it's way through SSC, we were on FLK our tank on the shaman died and one of our fury warriors had enough awareness and quick enough reflexes to equip his shield and tanking weapon... switch to defensive and taunt the shaman before we lost anyone and continued tanking him until the shaman died in his dps gear. I'll take that fury warrior in my raid any day over the guy next to him who generally out dpsed him around that time but didn't even blink to break his dps rotation when the tank went down.

Share this post


Link to post
Share on other sites
We're getting saturated with stats now and while min maxing is great and all... People are starting to get lost in them IMO.

An alternate explanation is that most "things" in WoW are old by now. People look for meaningful new things to do or think about. Some of them end up being meaningful, some of them don't.

Share this post


Link to post
Share on other sites

A major problem I see with deaths is that in many boss fights, one death can cascade down to many deaths, both avoidable and unavoidable. Can WWS accurately determine sequence of events such that proper "blame" can be attributed?

Share this post


Link to post
Share on other sites

Most of those newfangled stats are invented by analysts, so they can have something to talk about on days when there aren't games.

The number of permutations and combinations are just staggering, and the use for the data is relatively small. In order to get a useful set of data you probably need 100ish data points - by the time you get to that amount of data you should be well past the bosses you have data for and onto new ones.

Share this post


Link to post
Share on other sites
A major problem I see with deaths is that in many boss fights, one death can cascade down to many deaths, both avoidable and unavoidable. Can WWS accurately determine sequence of events such that proper "blame" can be attributed?

A person's presence % can be used to figure out when someone died. The lower the %, the sooner they died. IIRC instant rezzes (soulstone, etc) aren't dealt with very well by WWS, so you'd need to watch for that. If you take good notes on who had what assignments, you can pretty easily figure out how errors propagate. e.g. "The warlocks were dead before the third wave of adds came, so we didn't have enough AOE and they overrun the raid".

Share this post


Link to post
Share on other sites
... an example from basketball that clearly shows what kind of ridiculous conclusions those metrics can lead to...

It goes to show that it is very easy to feed some numbers into a devised formula and come up with some metric, but whether the metric is truly meaningful is a different matter. If you don't devise the formula in any theoretically rigorous way, then there's not much value in the output either.

I do like malthrin's 'prevented deaths' idea, it is something that currently no meters pick up on but is still useful to know about. Commonly, we'll look after a wipe and check death logs to see where things went wrong, but in general there are still no tools for looking after a successful attempt and seeing where you could easily have wiped but didn't. It would also help illustrate flaws in the strategy that get obscured by the fact that you won (which cause you to come back and wipe repeatedly the next week because you aren't quite so lucky and don't get those clutch saves).

Share this post


Link to post
Share on other sites

Gruul the Dragonslayer does shatter, sending everyone flying around the room. 2 players at are low health, and don't notice each other. They end up next to each other, both get maximum amount of dmg from shatter and they both die. Was it healers fault for not topping them up to 100%, or players fault for not noticing? Which healers fault was that? Gary was assigned to raid heal people, but he didn't. All he did was heal tanks. It was Harry who healed those 2 players all the time, even when he was not supposed to. Was it Gary or Harry who failed the healing of those 2 players?

Hydross the Unstable is about to cross the line and turn into frost phase. However, one of the priests has cast a prayer of mending on a warlock who has Vile Sludge ticking. On the second Hydross changes phase, sludge ticks, prayer of mending heals, causes threath on the warlock, Hydross runs straight over the line to warlock, spawns extra pack of adds and wipes the raid. Without prayer of mending tank could have gained enough threath to hold aggro from the warlock. In reality it was the priests prayer of mending that wiped the raid. Which priests fault was that?

Mount Hyjal, trash waves. Infernals rain down from the sky, but one of the tanks drags infernals and several other mobs far away from raid out of range of heals. Only 1 healer notices this and goes to heal the tank. Healers that remained gained huge amounts of healing done. How do you produce statistics of "efficiency" from that situation?

Thanks to how Jupiter's moons are stationed that night a couple of melee run accidently into a Illidan's demon phase shadowbolts getting 2-shot. How in the world can ANY healers statistics be affected by a thing like this?

I don't think there are any valid ways to deduce how well a healer keeps people alive from a combatlog.

How do you decide how well tank performed? When tank died, was it tank that did a mistake with timing shield block wrong or was it the healers fault who couldn't heal the crushing blows? When dps pulls aggro and dies, was it tank who did poor threath or was it dps who did too much?

And how exacly do you look deeper into DPS' performance except pure dmg done?

For example, when comparing two enhancement shamans it is sometimes hard to tell whether one does a better job of totem twisting. However, an efficiency stat which takes increase in raid dps into account could solve this problem.

So the ench shaman who always gets in melee dps group did a better job at totem twisting than ench shaman who doesn't get into melee groups thanks to bad leadership?

There are way too many factors and way too many assumptions you have to do in order to make these stats. Even in huge sample like 100 fights.

Share this post


Link to post
Share on other sites
There are way too many factors and way too many assumptions you have to do in order to make these stats. Even in huge sample like 100 fights.

While it may be useless and/or futile to turn this stuff into a raid benchmark per say, I'd say it's actually useful. Of course there will be assumptions going on, but an over-time outlook on raid performance would not be useless. If you just want to look at a single number and go oh ok, DPS1 is better than DPS2, then of course it's useless just like a WWS parse would be.

Having the data summarized is definitely useful to have. You just don't need to use it like some sort of bible.

If Gruul sends two players flying next to each other, both shatter each other and die, fine. You have one sample point. Over the course of many fights, if you have one player dying 50% of the time, then it is something that is worth looking at. You don't necessarily have to boot him because he's awful, he might just be Wi-flagged and just has to choose between killing 1 person and 10. Or maybe he really is terrible.

If PoM caused aggro during a Hydross transition, fine, you have a sample point. If this happens consistently, and you can't put the blame on anyone, you can say ok, hey hunters, we're having trouble during this transition, can you try to misdirect onto the tanks?

Mount Hyjal, one healer runs off to heal a lone tank and his healing output sucks as a result, and this is something that happens consistently. Ok, next Hyjal raid, you might see two dots off in the corner so you can dismiss the information.

Having data that analyzes death prevention/nonprevention and whatever else... it's a very useful thing to have. I'd liken it to the be.imba site if that's still around. Instead of analyzing your character and making comments on the gear and talents, it'll just say hey, over the course of 100 gruul fights, this character died to shatter 50 times, these characters produced the biggest shatter damage over those fights, during this parry/crush damage spike, these healers squeezed heals in and prevented a possible wipe.

You can dismiss some of the info to bad luck, but like you can also see oh ok, HealerA and HealerB are great at healing the tank, they save the tank from a lot of deaths. We'll make sure HealerA and B are assigned to healing the main tank when possible, and leave the raid healing to other guys.

Yeah, it's probably impossible to make a single number to say ok, this healer is better than that healer, but you can definitely get some trends going which can help you optimize your raids.

Share this post


Link to post
Share on other sites

I've been tossing around a few ideas for this sort of thing recently, since my professional life has so far focused on the research and analysis fields. I definitely think WWS needs some sort of analysis tool that can generate reports rather than just numbers.

A few ideas I've had include:

- a consumables report. Who consistently uses potions/healthstones versus who doesn't. Who dies while their pots/healthstones aren't on cooldown (ie they don't use them). Who uses flasks/elixirs/oils/food buffs etc.

- a "retard" report -- people who consistently take avoidable damage (eg Supremus volcanos, Doomfire, etc).

- a decurse/dispell report. Dispell counts by themselves are fairly meaningless; I want to see who is taking damage from dispellable abilities, organised by who eventually dispelled them. This shows you how quick/attentive your dispellers are (or aren't).

Yes, I'm aware that most of this information can be extracted from WWS with time and patience, but having tools that could automatically do this would be a massive help.

I'd put something like this together myself, but I don't have a lot of programming experience :(

Interested to hear other people's thoughts.

Share this post


Link to post
Share on other sites

Webmeister, your ideas sound useful and I could certainly see a value to reports such as these.

That said the kind of cross raid ranking over time reports that the Original Poster is discussing seem to be of very little value and would not be of any use across raids as each raid does things differently so unlike a raw dps # or tanking rotation, things like death numbers / healer are not usable if you don't know the way that a given raid approaches an encounter.

Saraya, I think you may be overcomplicating things. Your idea would require a large number of reliable datapoints before you could draw anything valuable out on an encounter by encounter basis. To get a realistic sample you would need at least 20 minimum valid reports (I.E. Kills or extended attempts) and I cannot think of many bosses that aren't totally on farm with alts subbing in where you would get 20 datapoints before you move on. Additionally most of the things you could draw out from such a report are much more easily addressed if you have reasonably observant players in your raid.

If PoM caused aggro during a Hydross transition, fine, you have a sample point. If this happens consistently, and you can't put the blame on anyone, you can say ok, hey hunters, we're having trouble during this transition, can you try to misdirect onto the tanks?

The easy solution is to find the person who was on the wrong side of the line, which any observant player using an aggro monitor such as grid will know, and yell at them for being on the wrong side of the god damned line at the transition. No complex analyses needed and if they keep doing it they get benched or kicked.

Share this post


Link to post
Share on other sites

The idea of making a RDPS meter has amused me for a while. The sad thing is not so much that the logic is convoluted (although it is indeed) but that the results would be... well, not conducive to harmony.

Share this post


Link to post
Share on other sites

I was mostly trying to respond to the specific examples described in the post above mine. Obviously a phase transition like Hydross is easy as hell to spot, but for more complex fights, its much harder to be able to keep an eye on everyone. And if you had 10 attempts on a boss, each taking even as little as 30 seconds each, I think its still enough to draw data from it. Maybe you could just take the median death time from all the raid and calculate that as the wipe time, and see what's causing all the wipes to happen.

I might also be looking for too much information because in the past I would raid with people who don't admit to doing anything wrong(we're talking dying from spout and denying it even though their corpse is 100 yards away), and its really hard to fix a mistake if you don't know what's going wrong. Seeing subtle stuff like prevented deaths or the ideas webmeister tossed out would help greatly.

Regarding the consumable scan, that'd probably be only suited for short term consumables instead of flasks and elixirs. A elixir/flask scanner would be better served as part of some Addon that scans the raid for missing buffs(maybe it already exists).

If you really wanted to make it intricate, you could also check for trinket/talent cooldowns. Like is this person clicking trinkets during heroism? Did a tank die even though no one had used Nature's Swiftness? This may only be possible if it was drawn from a lengthy raid combat log, although it might point out bad things if people went out to respec between bosses.

Share this post


Link to post
Share on other sites
I've been tossing around a few ideas for this sort of thing recently, since my professional life has so far focused on the research and analysis fields. I definitely think WWS needs some sort of analysis tool that can generate reports rather than just numbers.

A few ideas I've had include:

- a consumables report. Who consistently uses potions/healthstones versus who doesn't. Who dies while their pots/healthstones aren't on cooldown (ie they don't use them). Who uses flasks/elixirs/oils/food buffs etc.

- a "retard" report -- people who consistently take avoidable damage (eg Supremus volcanos, Doomfire, etc).

- a decurse/dispell report. Dispell counts by themselves are fairly meaningless; I want to see who is taking damage from dispellable abilities, organised by who eventually dispelled them. This shows you how quick/attentive your dispellers are (or aren't).

Yes, I'm aware that most of this information can be extracted from WWS with time and patience, but having tools that could automatically do this would be a massive help.

I'd put something like this together myself, but I don't have a lot of programming experience :(

Interested to hear other people's thoughts.

I realy like this ideas. I just remember how many times i just fail to track down whats go wrong or i need realy much time to scroll every single Combat log. Some time before i do it for Archimonde, to track death causes, but it was a long way to track the events. And now we have some problems with the new combat log (not showing death for example).

Most of it i like the -retard report- idea. And it should be trackable. Many Bosses has a lot of possible checks:

MShazzrah - Fatal Attraction tics

RoS - wrong / missed interrupts

Bloodboil - 2 Bloodboil debuffs - or someone forget to move

Teron Gorefiend - last 2 ghosts when Construkts doing damage to Raidmember

and so one

Share this post


Link to post
Share on other sites
I guess my question would be, are the tools currently available to analyze raid performance really inadequate? I would think that just through WWS and general observation it'd be pretty apparent who's doing their job and who isn't. Also barriers to change aren't nearly as large in WoW raiding as in your example, the NBA.

I'm sure it'd be nice to have more precise tools, but I don't think the additional information you'd glean would be nearly worth the amount of effort it would take to model something like this.

There *is* information missing though, I miss proper time slicing, modeling of specific traits over time (how did the raids dps look over the course of the fight, how did the different bloodlust groups do during bloodlust and so on) and of course, yes, I'd like to compare this data to previous data to see how people are progressing. I'd also like to know exactly how much that feral druids aura contributed to the group he was in. I'd also like to see what selected sets of people were doing when the tank hit 42 health but lived through it. When did the last heal from each of his designated healers land? And then I'd want the possibility to move to their timeline and see where their time was spent, maybe I'll find that a stupid warlock did something that made him scream out for healing and drew the healers attention.

Finding this information isn't hard given the proper tools. The tools aren't even *that* hard to write. They just require an interface that doesn't think of itself as a glorified damage meter.

Yes, I've been working on something down this path, but no, I don't really have anything to show people unless you wish to see perl stubs. I could use a few proper combat logs though, and, well, I'd also like to see chat logs for the same raids at some point.

Share this post


Link to post
Share on other sites

I am actually willing to tackle a few of these ideas. Specifically the Retard Report, Deaths Averted, and Consumables.

It should be fairly easy to accomplish, especially due to the new combat log data, and especially if more than 1 person is running it, and it is syncronizing through the mod channel. If it is also saved from week to week, you can easily show a 'vs last week' comparison, and '4 week average' or something, and tie it into DBM or BW boss engage and boss death alerts for easy distinction. A bit of finess and you could easily make it similar to WWS, being able to look at the whole night,or individual boss kills/attempts.

Retard Report:

It would require information from each boss of the specific abilities, then watch for damage from them, and start tallying up the damage against players as it occurs. Even abilities that people will take a bit of damage from them, and the retard check is how fast you move away, is simply looking at how much someone took. Even random ones, like Mother. If someone gets it every single time, they will definately be high on the damage, but looking at the average per application is easy to see if the average is acceptably low. Granted, this can also be inferred with awareness and gut-reaction, but other bosses not so much. For example, Archi's doomfires. Damage taken from doomfire isn't quite accurate, but you could 1) uses of pvp trinket during fear (buff gained between fear gained and fear faded), 2) doomfire gained during fear, 3) doomfire gained without fear. Here the mod would look for the application through the high damage first hit (given a range, to account for talent reduction)

Deaths Avoided:

Simple enough. A few mods already do the same math with Ardent Defender saves. +1 to a healer when A) the size of their heal is greater than the amount of health remaining on the target and B) the next incoming hit is larger than the health value the tank was at, and C) that incoming hit was within a set period of time after the saving heal.

Can also show the differences between tanks. A tank with a really high # of saved counts vs a tank without puts into numbers healer gut reaction that "tank A seems to take more damage or is harder to heal". Only should be given to raid leaders, since you don't want healers competing to top this meter.

Consumables:

Really only useful on DPS burn fights like Brutallus, but I know my guild leadership would find use for it, since they already look to WWS the next day to check consumable usage counts. That plus the constant "The cauldron is still up!!! Make it disappear!!! Who doesn't have one yet!!!" Would be able to be quickly solved. (Similar to how Blizzard fixed the age old Raid Warning: "LOOT THE DOG!")

Are any of these as needed as a simple meter or WWS? Probably not. Would they still find use to raid leaders? I think so.

Share this post


Link to post
Share on other sites
That plus the constant "The cauldron is still up!!! Make it disappear!!! Who doesn't have one yet!!!" Would be able to be quickly solved.

Well, assuming the entire raid is using oRA2 or CTRA, you can use their item check commands to see who's missing it.

Share this post


Link to post
Share on other sites

I assume you are looking for a way to capitalize on the objective nature of WWS to indicate who is "good" and who is "bad" in your raid. There is only one situation where I could see such information being useful: for a raider to gain leverage when applying to a new guild.

I believe that the only meaningful way to judge a player is in fact subjective, and any statistical approach to quantifying a raider's performance is bound to be flawed. Anyone willing to pay attention can quickly develop a sense for each raider's strengths and weaknesses - no need for an advanced WWS report for that. As a result, I am a firm believer that character references are the best raiding metric. Unfortunately, you will not find those in a WWS parse.

Share this post


Link to post
Share on other sites
Well, assuming the entire raid is using oRA2 or CTRA, you can use their item check commands to see who's missing it.

Unless, like a large amount of people, they have item checks disabled.

Share this post


Link to post
Share on other sites

  • Recently Browsing   0 members

    No registered users viewing this page.