Jump to content


Photo

A New Raiding Metric


  • Please log in to reply
31 replies to this topic

#1 Irise

Irise

    Von Kaiser

  • Members
  • 72 posts

Posted 07 May 2008 - 04:39 PM

Anyone familiar with NBA knows that a player cannot just be measured by the main stats of Points, Rebounds, Steals, Assists and Blocked Shots. There are also efficiency stats which are even more useful for discerning the difference between players who come off the bench.

Similarly I was wondering if anyone out there would like to develop a tool for aggregating WWS reports and spitting out efficiency stats. For example, for a healer a stat would be the average number of deaths on Illidan while he or she was in the raid. One could refine this even further by only counting a death if the dead person was at least 20% of the focus for that particular healer. Once you get a few months worth of data under your belt you could see the difference between those healers who spam short heals to top the meters vs the players who prevent deaths.

Another good example of this would be the Robots before Illidari Council. It is far better for a priest to wait for the robot to nuke someone rather than spamming flash or CoH on the raid. However, WWS and other tools would not be able to pick up on this.

I think this idea could be expanded even further for DPS and Tanks as well. For example, when comparing two enhancement shamans it is sometimes hard to tell whether one does a better job of totem twisting. However, an efficiency stat which takes increase in raid dps into account could solve this problem.

#2 Docjowles

Docjowles

    Mr. Sandman

  • Members
  • 5620 posts

Posted 07 May 2008 - 07:13 PM

Right off the bat, it seems you'd have major issues with confounding variables. No two boss pulls are going to be exactly the same. The number of deaths could increase when Priest A is swapped for Priest B because B sucks at healing, the tank had a brain fart and forgot to hit Shield Block, someone pulled aggro, etc.

It could be mildly interesting to look at data like this, but there's so many different variables in even the simplest encounter that it's probably hopeless to actually isolate any one factor.

#3 Irise

Irise

    Von Kaiser

  • Members
  • 72 posts

Posted 07 May 2008 - 07:55 PM

Right off the bat, it seems you'd have major issues with confounding variables. No two boss pulls are going to be exactly the same. The number of deaths could increase when Priest A is swapped for Priest B because B sucks at healing, the tank had a brain fart and forgot to hit Shield Block, someone pulled aggro, etc.

It could be mildly interesting to look at data like this, but there's so many different variables in even the simplest encounter that it's probably hopeless to actually isolate any one factor.


The same problem is posed in basketball as well where each player's stats depends not only who was on the floor as well but how they played. There is no perfect way to calculate this stat but there are methodologies.

For example, suppose you want to know how much healing does a particular healer do on Illidan? First you bring up 10 different Illidan kills and normalize them for total healing done. Then what we are interested in is what fraction of healing is done by that particular healer while we take other healers into account. So assuming that the total healing is done is a function of the healers present we can do a linear approximation of it via a regression. The resulting data would tell us what fraction of the healing is done via each healer on average.

Another idea would to simply look at the number of deaths per Illidari Council kill. In this case we are looking at a very straightforward regression over 5-9 months of data.

I realize that this may involve too much work to actually be implemented (certainly beyond the amount of free time that I have). However, I thought it would be a good idea to toss around and see if anyone is interested. What I am suggesting certainly isn't anything new to those who have been following the development of PER in basketball and I hoping someone who has more experience than me will pick up this project.

PS: There was a thread a while back where Matlab was used to analyze combat logs. This would seem like an ideal project it.

#4 Pixen

Pixen

    Piston Honda

  • Members
  • 171 posts

Posted 07 May 2008 - 08:02 PM

I guess my question would be, are the tools currently available to analyze raid performance really inadequate? I would think that just through WWS and general observation it'd be pretty apparent who's doing their job and who isn't. Also barriers to change aren't nearly as large in WoW raiding as in your example, the NBA.

I'm sure it'd be nice to have more precise tools, but I don't think the additional information you'd glean would be nearly worth the amount of effort it would take to model something like this.

"They were bad, stop trying to figure out why bad players do bad things."


#5 Taiowa

Taiowa

    Glass Joe

  • Members
  • 23 posts

Posted 07 May 2008 - 08:16 PM

I guess my question would be, are the tools currently available to analyze raid performance really inadequate? I would think that just through WWS and general observation it'd be pretty apparent who's doing their job and who isn't.


I agree that instinct is more important than stats in assigning raid slots. (Even though I did read Moneyball. And my opinion doesn't matter, because I'm not a raid leader.)

However, no raid analysis tools that I know of make it easy to compare performance over time, which (I think) is at the core of what the original poster is talking about. As a simple example, for my own personal betterment purposes, I would love to see a chart of my DPS on the same benchmark fight from week to week. If I want to build something like that now, I have to manually construct a spreadsheet from my guild's WWS archives.

#6 Ellyh

Ellyh

    Don Flamenco

  • Members
  • 324 posts

Posted 07 May 2008 - 08:44 PM

The same problem is posed in basketball as well where each player's stats depends not only who was on the floor as well but how they played. There is no perfect way to calculate this stat but there are methodologies.


Personally the problem I see with any analogy to Real Life sports teams is that the number of datapoints are harder to controll and the variables are greater in an online situation. In your basketball analogy the coach probably has an order of magnitude more time per player for analysis than raid leaders do. Also the coach can run drills in specific situations that you just can't do in WoW. Also spec makes a much bigger difference to WoW, a mediocre rebounder can't walk up to a trainer and respec into a midcourt intercept specialist by handing over some cash. So how would your proposed metric deal with respecs? You can't compare my performance on Illidan this week as a COH priest with my performance next week where I have had to respec to be the guild DS bitch. I'm going to be performing two totally different functions and responsible for totally different targets. (Tank vs Raid healing)

Finally the tools we have, WWS etc, are notorious for not recording certain features well, especially things like Prayer of Mending or lifebloom. Analysing a healer is already more gut feeling and instinct than it is for DPS and I can't see a metric like you are proposing solving this problem or being a valuable tool.

To be effective you need a full timestamped combat log for the actions of every single actor in the fight or you just can't say if someone is playing well after the fact. Your original example of the robots leading to council is a classic example. Apart from one priest who is on reactive shield duty to stop 1 shots what can the other priests do other than try to keep the raid at full health? How would you know that the shield priest failed and that the target of the 1 shot nuke wasn't already at 1/2 health because the raid healers were slacking? With the currently available tools you can't, you have to have been there and paying close attention to everyone's lifebars.

#7 malthrin

malthrin

    stalemate associate

  •  Benefactors
  • 9107 posts

Posted 07 May 2008 - 08:45 PM

On the general topic of interesting metrics that you'd like to see, I'd like a mod or WWS filter for prevented deaths - this would show 5-10 seconds of relevant combat log after a spike in incoming DPS on that player. This would be interesting both to see which healers are consistently landing heals in that window as well as the steps that the damaged player takes or doesn't take to save himself.

#8 deadlights

deadlights

    Banned

  • Banned
  • 93 posts

Posted 07 May 2008 - 08:57 PM

I don't like those fancy statistical metrics that are all the rage in sports these days. So there is the first flaw I see off the bat. I'd have to be convinced that they could be more useful with WWS than evaluating sports before I'd even consider tackling the number of variables you'd have to account for in a raid setting.

Just as an example from basketball that clearly shows what kind of ridiculous conclusions those metrics can lead to. For a few years looking at those efficiency numbers would have told you that Brad Miller was a better Center than Shaquill O'Neal. (This was when shaq was still in LA and his first year in Miami if I recall correctly), and that's just absurd.

We're getting saturated with stats now and while min maxing is great and all... People are starting to get lost in them IMO. Way back when my guild was still working it's way through SSC, we were on FLK our tank on the shaman died and one of our fury warriors had enough awareness and quick enough reflexes to equip his shield and tanking weapon... switch to defensive and taunt the shaman before we lost anyone and continued tanking him until the shaman died in his dps gear. I'll take that fury warrior in my raid any day over the guy next to him who generally out dpsed him around that time but didn't even blink to break his dps rotation when the tank went down.

#9 Disquette

Disquette

    doop doop de doooo

  •  Patrons
  • 5519 posts

Posted 07 May 2008 - 09:30 PM

We're getting saturated with stats now and while min maxing is great and all... People are starting to get lost in them IMO.


An alternate explanation is that most "things" in WoW are old by now. People look for meaningful new things to do or think about. Some of them end up being meaningful, some of them don't.
http://us.battle.net...66766?page=3#41
Let me map a priority list out for you so that you can refer to it in the future:
1. Money 2. Money 3. PvE 4. Mages 5. Companion pets 6. PvP

#10 crimsonsentinel

crimsonsentinel

    Bald Bull

  • Members
  • 1725 posts

Posted 07 May 2008 - 10:35 PM

A major problem I see with deaths is that in many boss fights, one death can cascade down to many deaths, both avoidable and unavoidable. Can WWS accurately determine sequence of events such that proper "blame" can be attributed?

#11 oldmandennis

oldmandennis

    Don Flamenco

  • Members
  • 375 posts

Posted 07 May 2008 - 11:09 PM

Most of those newfangled stats are invented by analysts, so they can have something to talk about on days when there aren't games.

The number of permutations and combinations are just staggering, and the use for the data is relatively small. In order to get a useful set of data you probably need 100ish data points - by the time you get to that amount of data you should be well past the bosses you have data for and onto new ones.

#12 Mode

Mode

    Piston Honda

  • Members
  • 238 posts

Posted 07 May 2008 - 11:10 PM

A major problem I see with deaths is that in many boss fights, one death can cascade down to many deaths, both avoidable and unavoidable. Can WWS accurately determine sequence of events such that proper "blame" can be attributed?

A person's presence % can be used to figure out when someone died. The lower the %, the sooner they died. IIRC instant rezzes (soulstone, etc) aren't dealt with very well by WWS, so you'd need to watch for that. If you take good notes on who had what assignments, you can pretty easily figure out how errors propagate. e.g. "The warlocks were dead before the third wave of adds came, so we didn't have enough AOE and they overrun the raid".

#13 Morwen

Morwen

    Piston Honda

  • Members
  • 112 posts

Posted 07 May 2008 - 11:36 PM

... an example from basketball that clearly shows what kind of ridiculous conclusions those metrics can lead to...


It goes to show that it is very easy to feed some numbers into a devised formula and come up with some metric, but whether the metric is truly meaningful is a different matter. If you don't devise the formula in any theoretically rigorous way, then there's not much value in the output either.

I do like malthrin's 'prevented deaths' idea, it is something that currently no meters pick up on but is still useful to know about. Commonly, we'll look after a wipe and check death logs to see where things went wrong, but in general there are still no tools for looking after a successful attempt and seeing where you could easily have wiped but didn't. It would also help illustrate flaws in the strategy that get obscured by the fact that you won (which cause you to come back and wipe repeatedly the next week because you aren't quite so lucky and don't get those clutch saves).

#14 Vihermaali

Vihermaali

    Piston Honda

  • Members
  • 194 posts

Posted 08 May 2008 - 12:26 AM

Gruul the Dragonslayer does shatter, sending everyone flying around the room. 2 players at are low health, and don't notice each other. They end up next to each other, both get maximum amount of dmg from shatter and they both die. Was it healers fault for not topping them up to 100%, or players fault for not noticing? Which healers fault was that? Gary was assigned to raid heal people, but he didn't. All he did was heal tanks. It was Harry who healed those 2 players all the time, even when he was not supposed to. Was it Gary or Harry who failed the healing of those 2 players?

Hydross the Unstable is about to cross the line and turn into frost phase. However, one of the priests has cast a prayer of mending on a warlock who has Vile Sludge ticking. On the second Hydross changes phase, sludge ticks, prayer of mending heals, causes threath on the warlock, Hydross runs straight over the line to warlock, spawns extra pack of adds and wipes the raid. Without prayer of mending tank could have gained enough threath to hold aggro from the warlock. In reality it was the priests prayer of mending that wiped the raid. Which priests fault was that?

Mount Hyjal, trash waves. Infernals rain down from the sky, but one of the tanks drags infernals and several other mobs far away from raid out of range of heals. Only 1 healer notices this and goes to heal the tank. Healers that remained gained huge amounts of healing done. How do you produce statistics of "efficiency" from that situation?

Thanks to how Jupiter's moons are stationed that night a couple of melee run accidently into a Illidan's demon phase shadowbolts getting 2-shot. How in the world can ANY healers statistics be affected by a thing like this?

I don't think there are any valid ways to deduce how well a healer keeps people alive from a combatlog.

How do you decide how well tank performed? When tank died, was it tank that did a mistake with timing shield block wrong or was it the healers fault who couldn't heal the crushing blows? When dps pulls aggro and dies, was it tank who did poor threath or was it dps who did too much?

And how exacly do you look deeper into DPS' performance except pure dmg done?

For example, when comparing two enhancement shamans it is sometimes hard to tell whether one does a better job of totem twisting. However, an efficiency stat which takes increase in raid dps into account could solve this problem.

So the ench shaman who always gets in melee dps group did a better job at totem twisting than ench shaman who doesn't get into melee groups thanks to bad leadership?

There are way too many factors and way too many assumptions you have to do in order to make these stats. Even in huge sample like 100 fights.

#15 Saraya

Saraya

    Piston Honda

  • Members
  • 159 posts

Posted 08 May 2008 - 03:49 AM

There are way too many factors and way too many assumptions you have to do in order to make these stats. Even in huge sample like 100 fights.

While it may be useless and/or futile to turn this stuff into a raid benchmark per say, I'd say it's actually useful. Of course there will be assumptions going on, but an over-time outlook on raid performance would not be useless. If you just want to look at a single number and go oh ok, DPS1 is better than DPS2, then of course it's useless just like a WWS parse would be.

Having the data summarized is definitely useful to have. You just don't need to use it like some sort of bible.

If Gruul sends two players flying next to each other, both shatter each other and die, fine. You have one sample point. Over the course of many fights, if you have one player dying 50% of the time, then it is something that is worth looking at. You don't necessarily have to boot him because he's awful, he might just be Wi-flagged and just has to choose between killing 1 person and 10. Or maybe he really is terrible.

If PoM caused aggro during a Hydross transition, fine, you have a sample point. If this happens consistently, and you can't put the blame on anyone, you can say ok, hey hunters, we're having trouble during this transition, can you try to misdirect onto the tanks?

Mount Hyjal, one healer runs off to heal a lone tank and his healing output sucks as a result, and this is something that happens consistently. Ok, next Hyjal raid, you might see two dots off in the corner so you can dismiss the information.

Having data that analyzes death prevention/nonprevention and whatever else... it's a very useful thing to have. I'd liken it to the be.imba site if that's still around. Instead of analyzing your character and making comments on the gear and talents, it'll just say hey, over the course of 100 gruul fights, this character died to shatter 50 times, these characters produced the biggest shatter damage over those fights, during this parry/crush damage spike, these healers squeezed heals in and prevented a possible wipe.

You can dismiss some of the info to bad luck, but like you can also see oh ok, HealerA and HealerB are great at healing the tank, they save the tank from a lot of deaths. We'll make sure HealerA and B are assigned to healing the main tank when possible, and leave the raid healing to other guys.

Yeah, it's probably impossible to make a single number to say ok, this healer is better than that healer, but you can definitely get some trends going which can help you optimize your raids.

#16 webmeister

webmeister

    Von Kaiser

  • Members
  • 33 posts

Posted 08 May 2008 - 06:05 AM

I've been tossing around a few ideas for this sort of thing recently, since my professional life has so far focused on the research and analysis fields. I definitely think WWS needs some sort of analysis tool that can generate reports rather than just numbers.

A few ideas I've had include:
- a consumables report. Who consistently uses potions/healthstones versus who doesn't. Who dies while their pots/healthstones aren't on cooldown (ie they don't use them). Who uses flasks/elixirs/oils/food buffs etc.
- a "retard" report -- people who consistently take avoidable damage (eg Supremus volcanos, Doomfire, etc).
- a decurse/dispell report. Dispell counts by themselves are fairly meaningless; I want to see who is taking damage from dispellable abilities, organised by who eventually dispelled them. This shows you how quick/attentive your dispellers are (or aren't).

Yes, I'm aware that most of this information can be extracted from WWS with time and patience, but having tools that could automatically do this would be a massive help.

I'd put something like this together myself, but I don't have a lot of programming experience :(

Interested to hear other people's thoughts.

#17 Ellyh

Ellyh

    Don Flamenco

  • Members
  • 324 posts

Posted 08 May 2008 - 06:23 AM

Webmeister, your ideas sound useful and I could certainly see a value to reports such as these.

That said the kind of cross raid ranking over time reports that the Original Poster is discussing seem to be of very little value and would not be of any use across raids as each raid does things differently so unlike a raw dps # or tanking rotation, things like death numbers / healer are not usable if you don't know the way that a given raid approaches an encounter.

Saraya, I think you may be overcomplicating things. Your idea would require a large number of reliable datapoints before you could draw anything valuable out on an encounter by encounter basis. To get a realistic sample you would need at least 20 minimum valid reports (I.E. Kills or extended attempts) and I cannot think of many bosses that aren't totally on farm with alts subbing in where you would get 20 datapoints before you move on. Additionally most of the things you could draw out from such a report are much more easily addressed if you have reasonably observant players in your raid.

If PoM caused aggro during a Hydross transition, fine, you have a sample point. If this happens consistently, and you can't put the blame on anyone, you can say ok, hey hunters, we're having trouble during this transition, can you try to misdirect onto the tanks?



The easy solution is to find the person who was on the wrong side of the line, which any observant player using an aggro monitor such as grid will know, and yell at them for being on the wrong side of the god damned line at the transition. No complex analyses needed and if they keep doing it they get benched or kicked.

#18 Northerner

Northerner

    Great Tiger

  • Members
  • 825 posts

Posted 08 May 2008 - 07:10 AM

The idea of making a RDPS meter has amused me for a while. The sad thing is not so much that the logic is convoluted (although it is indeed) but that the results would be... well, not conducive to harmony.

#19 Saraya

Saraya

    Piston Honda

  • Members
  • 159 posts

Posted 08 May 2008 - 07:11 AM

I was mostly trying to respond to the specific examples described in the post above mine. Obviously a phase transition like Hydross is easy as hell to spot, but for more complex fights, its much harder to be able to keep an eye on everyone. And if you had 10 attempts on a boss, each taking even as little as 30 seconds each, I think its still enough to draw data from it. Maybe you could just take the median death time from all the raid and calculate that as the wipe time, and see what's causing all the wipes to happen.

I might also be looking for too much information because in the past I would raid with people who don't admit to doing anything wrong(we're talking dying from spout and denying it even though their corpse is 100 yards away), and its really hard to fix a mistake if you don't know what's going wrong. Seeing subtle stuff like prevented deaths or the ideas webmeister tossed out would help greatly.

Regarding the consumable scan, that'd probably be only suited for short term consumables instead of flasks and elixirs. A elixir/flask scanner would be better served as part of some Addon that scans the raid for missing buffs(maybe it already exists).

If you really wanted to make it intricate, you could also check for trinket/talent cooldowns. Like is this person clicking trinkets during heroism? Did a tank die even though no one had used Nature's Swiftness? This may only be possible if it was drawn from a lengthy raid combat log, although it might point out bad things if people went out to respec between bosses.

#20 Medusa

Medusa

    Von Kaiser

  • Members
  • 34 posts

Posted 08 May 2008 - 07:44 AM

I've been tossing around a few ideas for this sort of thing recently, since my professional life has so far focused on the research and analysis fields. I definitely think WWS needs some sort of analysis tool that can generate reports rather than just numbers.

A few ideas I've had include:
- a consumables report. Who consistently uses potions/healthstones versus who doesn't. Who dies while their pots/healthstones aren't on cooldown (ie they don't use them). Who uses flasks/elixirs/oils/food buffs etc.
- a "retard" report -- people who consistently take avoidable damage (eg Supremus volcanos, Doomfire, etc).
- a decurse/dispell report. Dispell counts by themselves are fairly meaningless; I want to see who is taking damage from dispellable abilities, organised by who eventually dispelled them. This shows you how quick/attentive your dispellers are (or aren't).

Yes, I'm aware that most of this information can be extracted from WWS with time and patience, but having tools that could automatically do this would be a massive help.

I'd put something like this together myself, but I don't have a lot of programming experience :(

Interested to hear other people's thoughts.


I realy like this ideas. I just remember how many times i just fail to track down whats go wrong or i need realy much time to scroll every single Combat log. Some time before i do it for Archimonde, to track death causes, but it was a long way to track the events. And now we have some problems with the new combat log (not showing death for example).

Most of it i like the -retard report- idea. And it should be trackable. Many Bosses has a lot of possible checks:
MShazzrah - Fatal Attraction tics
RoS - wrong / missed interrupts
Bloodboil - 2 Bloodboil debuffs - or someone forget to move
Teron Gorefiend - last 2 ghosts when Construkts doing damage to Raidmember
and so one




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users