Jump to content


Photo

DPS Simulator


  • This topic is locked This topic is locked
133 replies to this topic

#21 Nite_Moogle

Nite_Moogle

    I prefer the term treasure hunting

  • Allied Members
  • 11,288 posts

Posted 30 July 2008 - 05:53 PM

On the other hand, if you're interested in EP values, even a really good simulator doesn't tend to do as good a job as a more deterministic method (spreadsheets, etc.), for the simple reason that the variance in damage requires one to run the simulator for an extremely long time in order to get accurate values out.

Flurry is the main thing that makes this extremely difficult to represent accurately because it is a positive feedback loop. You can estimate it but it's not going to be accurate. Unpredictable rage gain and accurate modeling of rage income and expenditure make eyeballing warrior damage quite a bit more complex than most other classes that have much more static resources to lean on for determining damage cycles. It isn't to say this can't be done, but it's nearly as complex as just writing a far more flexible simulator.

What I would like to see:
For a display window, have 3 seperate columns streaming events with timestamps. Column 1 would have auto attacks and heroic strikes, Column 2 would have special attack damage, and Column 3 would have buffs gaining and fading.

This is so impractical it isn't even funny. The simulator I've written thus far takes under 30 seconds to simulate 45 hours of combat.

Eh, my nostalgia goggles aren't as good as they used to be.


#22 dedmonwakeen

dedmonwakeen

    Bald Bull

  • Members
  • 1,302 posts

Posted 30 July 2008 - 06:04 PM

For instance, lets assume for the moment that the standard deviation of damage for a real 6-minute fight is 50 DPS, and one wants EP that are accurate to within 1%..


Not to rehash entire discussions we've had in other threads on this subject, but.....

I'd like to change your sentence slightly: We would want simulation-generated EPs with greater accuracy than those generated using closed-form approximations (aka formulation). A calculator can deliver a deterministic EP value with virtually arbitrary precision. But "precision" is not the same thing as "accuracy".

In my honest opinion: If a calculator can deliver EP values with an accuracy of 1%, there really is no need to use a simulator. It might be fun to write one..... but the usability of a calculator is far superior.

If the calculator cannot deliver a requisite level of accuracy, then the question becomes: "Can simulation deliver markedly better accuracy in a reasonable amount of time?"

It may not be possible to deliver EP values accurate to 1% with simulation...... but it may be possible to deliver values more accurate than those of formulation for some class/party/raid setups.

Grim: Please note that Aldriana still makes a very, very important point. Do whatever you can to reduce variation where possible because this will significantly reduce your runtime.

#23 Aldriana

Aldriana

    Mike Tyson

  • Moderators
  • 13,510 posts

Posted 30 July 2008 - 06:41 PM

Agreed, there is an important distinction to be made between precision and accuracy. However, I think there's also something to be said for always getting the same answer - having the relative value of items slide relative to one another depending on luck in the simulation strikes me as a bad thing, so I would argue that it's better to have infinite precision values accurate to 5% than infinite accuracy values precise to 5%. Which basically means that if your calculator is even remotely close, it's going to do a better job on EP than most simulators.

That said: I confess I don't actually know how good the warrior spreadsheets are at the moment. I'm certainly willing to believe there are complications that make it challenging, but, at the risk of sounding cynical: it's been my impression that everyone thinks the modeling of their class is harder than everyone else's. Which is not to say that modeling warriors isn't hard - I'm sure it is. But a lot of other classes also have hard problems of similar sorts and have managed to concoct highly accurate models, so it would surprise me a bit if warriors couldn't do the same. Which, again, is not to say that you should or shouldn't do that, or this - I'm just saying, you should be clear on what information you want, and how well a simulator is going to do towards that end. If your ultimate goal is good EP values, it might make more sense to invest time in a really good calculator rather than a really good simulator. If you're more interested in just a good damage estimator, a good simulator is certainly a better investiture of time. So I simply recommend that you be clear about what you're going for.

#24 dedmonwakeen

dedmonwakeen

    Bald Bull

  • Members
  • 1,302 posts

Posted 30 July 2008 - 07:03 PM

Grim, even if the simulator does not become the "go-to" tool, it would still be a very critical aid for confirming the accuracy of the formulation used by spreadsheets and calculators.

#25 landsoul

landsoul

    Bald Bull

  • Members
  • 1,025 posts

Posted 30 July 2008 - 09:32 PM

Grim, even if the simulator does not become the "go-to" tool, it would still be a very critical aid for confirming the accuracy of the formulation used by spreadsheets and calculators.


This is like comparing apples to oranges to try and determine if the orange is right.

This is so impractical it isn't even funny. The simulator I've written thus far takes under 30 seconds to simulate 45 hours of combat.


What if you wanted to personally check the results of your work for 1 minute of fight data or would you Save a log file of 45 hours of fighting? I guess you could do a log file instead of a display, but a 3 column system would make it easier on the human to verify his work.

#26 Gruntle

Gruntle

    King Hippo

  • Members
  • 509 posts

Posted 30 July 2008 - 10:29 PM

Agreed, there is an important distinction to be made between precision and accuracy. However, I think there's also something to be said for always getting the same answer - having the relative value of items slide relative to one another depending on luck in the simulation strikes me as a bad thing, so I would argue that it's better to have infinite precision values accurate to 5% than infinite accuracy values precise to 5%. Which basically means that if your calculator is even remotely close, it's going to do a better job on EP than most simulators.

That said: I confess I don't actually know how good the warrior spreadsheets are at the moment. I'm certainly willing to believe there are complications that make it challenging, but, at the risk of sounding cynical: it's been my impression that everyone thinks the modeling of their class is harder than everyone else's. Which is not to say that modeling warriors isn't hard - I'm sure it is. But a lot of other classes also have hard problems of similar sorts and have managed to concoct highly accurate models, so it would surprise me a bit if warriors couldn't do the same. Which, again, is not to say that you should or shouldn't do that, or this - I'm just saying, you should be clear on what information you want, and how well a simulator is going to do towards that end. If your ultimate goal is good EP values, it might make more sense to invest time in a really good calculator rather than a really good simulator. If you're more interested in just a good damage estimator, a good simulator is certainly a better investiture of time. So I simply recommend that you be clear about what you're going for.


The current warrior spreadsheets are good but have problems with a number of things due to not being able to simulate the spikyness of rage generation. The main issue in my opinion is weapon speeds. The current spreadsheets cannot simulate one of the main strengths of fast weapons, i.e. the smooth rage generation. A simulator could also be used to study at what rage level it is "safe" to use Heroic strike. Finally, I'm not sure the flurry modeling is really doing things as accurate as you could get in a simulator. Mid-swing haste is a real effect (it has been measured by drAllcom in one of the warrior threads) and will affect different weapon speed setups differently.

For Wotlk it's probable that we will not know initially how to optimize dps. A simulator could be used to figure out which abilities to use and when. Based on the current abilities and talents it's very likely that the set cycle we've been using for tbc might not be good at all. Bloodsurge procs and slamming will make a mess of the very ordered cycle that's used in the sheets. I think a simulator will be a better tool to use (at least initially, maybe a spreadsheet/calculator using analytical models can be constructed later based on the findings of the simulator).

I've read elsewhere that people trying to make swing simulators run into problems with convergence/variance. It might be the rounding problem that landsoul is discussing that messes things up, but it feels like that type of issues should be solvable. In your simulator, Nite_Moogle, do you reach convergence after that 30 second run (45 minute combat time)? Is the variance very high between different 30 sec runs?

edit: a thong is not the same as a thing...

#27 Aldriana

Aldriana

    Mike Tyson

  • Moderators
  • 13,510 posts

Posted 30 July 2008 - 10:40 PM

So, I can see where paying attention to variance (aka spikiness) makes things a lot harder. On the other hand, that is, for instance, almost exactly the sort of problem I'm working on for my rogue calculator right now. Figuring out how often droughts in Combat Potency procs, Ruthlessness procs, and Relentless Strikes procs conjoin to cause SnD to drop is a challenging problem. Figuring out how often unlucky streaks will occur causing Deadly Poison to drop is a challenging problem. And, honestly, we don't have perfect answer to these sorts of questions. But I think we do understand them well enough to get within 1% if that's the goal. Does the same apply for warriors? I have no idea. But I wouldn't be willing to bet much against it. Hard problems are made to be solved.

#28 dedmonwakeen

dedmonwakeen

    Bald Bull

  • Members
  • 1,302 posts

Posted 30 July 2008 - 11:22 PM

This is like comparing apples to oranges to try and determine if the orange is right.


I have to disagree pretty strongly.

If we have to use fruit analogies....
I think it is more like comparing two different ways to count apples.......

Confirming the behavior model of simulation is considerably more straight-forward than proving a reduced approximation accurately models complex game mechanics........ which is why simulation can be helpful to confirm formulation.

#29 Nite_Moogle

Nite_Moogle

    I prefer the term treasure hunting

  • Allied Members
  • 11,288 posts

Posted 31 July 2008 - 02:43 AM

I've read elsewhere that people trying to make swing simulators run into problems with convergence/variance. It might be the rounding problem that landsoul is discussing that messes things up, but it feels like that type of issues should be solvable. In your simulator, Nite_Moogle, do you reach convergence after that 30 second run (45 minute combat time)? Is the variance very high between different 30 sec runs?

You don't start to see very consistent results until you have several days of combat time per run. The strategy I took is to repeat a fight of X duration a large number of times (which makes handling things like trinkets and death wish more comparable to real fights) and use the compiled statistics from all runs to determine the averages. You can have extremely high variations between consecutive fights depending on RNG, especially where you are not capped on hit or expertise. Part of the goal was to create an efficient simulator that would be able to do very large numbers of short fights repeatedly in order to make it possible to re-run these simulations with different stats to determine stat weighting. The latter part lacks implementation which is why I've not released it; it's not terribly useful in its current form.

Eh, my nostalgia goggles aren't as good as they used to be.


#30 dedmonwakeen

dedmonwakeen

    Bald Bull

  • Members
  • 1,302 posts

Posted 31 July 2008 - 10:17 AM

Examples of variance that are "fair" to model "incorrectly":
Base weapon damage: Just take the average of min/max
Direct-dmg spells: just take the average of min/max
Partial resists: Do not use a randomized bucket sytem centered around the default mitigation..... just use the default mitigation pct every time.

#31 dr_AllCOM3

dr_AllCOM3

    Great Tiger

  • Members
  • 985 posts

Posted 31 July 2008 - 11:25 AM

I have an almost finished paperdoll character builder and the basics of a simulator. You can even add other classes. All that in lua and as an addon. I just didn't finished it, because SWP gear was clearly better and right now I don't even play.
Posted Image
It's basically like Rawr, but the way I like it. Easier to use and more customizable. Item comparison is done with a spreadsheet, can't be done via simulation because of rounding errors (even after 1000000 fights).
The addon has some bugs (gems..) and I couldn't force myself to finish the build-in spreadsheet. The simulator is event driven, no need to check every x milliseconds (it is very accurate btw). You can insert anything there. I even figured out stuff like Flurry or procs. It can also be slowed down, so your game doesn't lock up and you can do other stuff while waiting. You should be able to do anything with the data you get, sky is the limit. I made a thread in the german forum once, but I think I lost it. They also aren't helpful at all, noone there has a clue about deeper warrior mechanics;).


This is so impractical it isn't even funny. The simulator I've written thus far takes under 30 seconds to simulate 45 hours of combat.

Mine is ten to twenty times faster, if I recall correctly. Depends on how much stuff you cram into it.

For instance, lets assume for the moment that the standard deviation of damage for a real 6-minute fight is 50 DPS, and one wants EP that are accurate to within 1%.

Standard deviation is way higher, so high you can't use the numbers for item comparison.

If the calculator cannot deliver a requisite level of accuracy, then the question becomes: "Can simulation deliver markedly better accuracy in a reasonable amount of time?"
[...]
Grim: Please note that Aldriana still makes a very, very important point. Do whatever you can to reduce variation where possible because this will significantly reduce your runtime.

Some things can't be simplified, mostly because of the rage machanic. It will at least be very interesting.
Random numbers are not what slows a simulator down and if you do one you want it to be as realistic as you can.

#32 dedmonwakeen

dedmonwakeen

    Bald Bull

  • Members
  • 1,302 posts

Posted 31 July 2008 - 02:12 PM

Random numbers are not what slows a simulator down and if you do one you want it to be as realistic as you can.


RNG does not exactly slow down the simulator, but.....

Unnecessary randomness increases variance, which in turn increases the number of iterations required for reasonable convergence.

#33 Gruntle

Gruntle

    King Hippo

  • Members
  • 509 posts

Posted 01 August 2008 - 07:18 AM

There is one thing I'm not quite getting. We're simulating the dps output and getting extremely high variance, how does this variance compare to the true variance for "real" wow fights? My dps normally varies by about 10-20% up and down for a specific boss fight (and out of those my guess is that more than half is due to human errors). Is the simulation variance worse than this? In that case something is very wrong with the simulators.

Still, even a 5% variance would make it impossible to use for item comparisons. Would it be possible to instead use a set list of random numbers? That is, you use exactly the same random numbers for all swings but only change gear stats. Some things would have to be rerandomized (e.g. increased rage income makes it possible to do an extra Bloodsurge slam or whatever), but in principle the variance should go down a lot just by having a set random number list for all white swings.

#34 lightstrike

lightstrike

    Von Kaiser

  • Members
  • 58 posts

Posted 01 August 2008 - 09:55 AM

I was thinking if a bunch of "specialized" RNGs would work. Basically, a rng-state would be kept per type of result required: white swings, yellow attacks, on-hit procs, on-crit procs, etc...
These specialized RNGs would be adaptive, basically after running for a while, they would tend to "correct" themselves, and try to yield results that converge to what would be expected of them, or within some set threshold around it... The randomness will be kept somewhat, as for different runs of the simulator, the auto correcting would happen at different times, and never during the initial first few results, ensuring a negligible chance of repeating sequences... This would be statistically incorrect, but correct enough, and it would drastically cut on the amount of simulations required for truly convergent results...

Also, when every modern CPU has at least two physical cores and/or hyper-threading, it's a waste to not make use of this parallelism, so this would again greatly reduce the amount of run-time needed, in addition to making the whole thing scalable...

If we want to get fancy, network-distributed simulations would also be something to consider at the expense of a lot of work, but would make those million-hour simulations possible in a matter of seconds, especially if we're talking about simulating 25-men raids a few hundred-thousand times. A "1-coordinator with n-slaves" system would be somewhat easy to implement given enough time... I believe there are open source frameworks that deal with most of the synchronization and network communication work so it might not even be a lot of work... Might be somewhat unrealistic to think of this as a one-man project, though, especially for a game...


There's a morning un-caffeinated brain-fart...

#35 dedmonwakeen

dedmonwakeen

    Bald Bull

  • Members
  • 1,302 posts

Posted 01 August 2008 - 03:21 PM

There is one thing I'm not quite getting. We're simulating the dps output and getting extremely high variance, how does this variance compare to the true variance for "real" wow fights? My dps normally varies by about 10-20% up and down for a specific boss fight (and out of those my guess is that more than half is due to human errors). Is the simulation variance worse than this? In that case something is very wrong with the simulators.


When running a multi-player sim for just 10 cpu seconds a bunch of times I saw a max variance of only 0.5% (it was the Enhancement Shaman).

Unfortunately, 0.5% variance is still too high for EP generation, but that was only a 10sec run of not-yet-optimized code......

In order to generate valid EP values, the dps_increase from increasing a stat must be far greater than observed dps_variance.

For stats that scale linearly, this is easy: Just increase the stat of interest very high.

For stats that do NOT scale linearly (which is many of them) it is not so easy. If your bracket around the baseline is too large (base-stat_delta, base+stat_delta) then the calculated slope across that range may not represent the actual slope at the baseline point.

But as you narrow your bracket to reduce error from non-linear scaling, you fall prey to error introduced from variance.

When I was maintaining scaling tables for casters over a broad range of gear I found that some stats (and classes) were easy to calculate whereas other stats (and classes) were much harder. Where "easy" and "hard" represent the amount of runtime required to get convergence.

I have not yet modeled Warriors, so I cannot speak for that class in particular........ but so far I have yet to run into a class/spec that could not be analyzed via simulation in a "reasonable" amount of time.

#36 landsoul

landsoul

    Bald Bull

  • Members
  • 1,025 posts

Posted 01 August 2008 - 04:08 PM

Wouldnt you figure if you set a specific seed (set RNG values) to not change then wouldnt you be blatantly accepting a possible incorrect set of numbers/calculations? If the variance is high, the sim wouldn't accomplish anything. What was the purpose of this project in the first place? The purpose was to try and see if a sim could be more accurate, reliable, and be desired more useful than a spreadsheet program, and possible be able to determine rage spending choices when multiple attacks are available during execute phase. I would be a little qualmsy about using something that took shortcuts.

#37 Grim13

Grim13

    Piston Honda

  • Members
  • 105 posts

Posted 01 August 2008 - 10:14 PM

I think this is a great idea, if done correctly. If not apporached with the right frame of mind this cuold turn into a huge nightmare, Grim. The strengths of this idea is so huge and I wish it could be easy but I fear that there are an overwhelming amount of variables to be able to build enough loops for. Is this a brute-force project or a finesse project? A combination of both?

Potential Roadblocks
An event-driven system I fear could propogate into inaccuracy inflation. For example, if the calculated swing time was 2.93 seconds but in reality was 2.9348732 via calculation, that .0048732 could add up over time to be 3 seconds over a long fight. Multiply that by 10 different mechanics, and the automatic decision making engine will start making incorrect decisions even if one mechanic is slightly off.

Computer processors are pretty fast these days. Even with a 1ms (.001s) resolution we just don't know how fast it will run. It might just be that what you think would be slow having too many instances going through too many loops, might be too fast for you to even be able to follow in a display.


Haste mechanics I fear will completely run you around. What happens when you crit on your off-hand whirlwind, proccing DST and a new flurry when your main hand is in mid-swing? You would have to take the remainder of your main hand swing's time and add the haste application to only that section.

The problem of procs is that they are not static. How could you update dynamic changes to the warrior's stat table to your calculation loops while the loops are running?

Suggestions
Give it a shot, but make the code available for the community to see, test, and comment on. This project is going to be a big deal, with lots of little small parts forming into bigger and bigger parts that will finally come together. You could probably start working on it now and have untalented mechanics working befre the final WOTLK changes are made. Build the input, and an auto attack loop and let's see how it looks!

Make it resolution based, because at least that way you can make a variable that defines time left on swing that can be modified on the fly. Procced haste while midswing? then you do "Set Swing_Left =Swing_Left/(1+Haste_Proc_Percent).

Ask for a lot of help. You sound like you may not know a lot of neat programming tricks, which is what will be needed here. I don't know very much myself, but there are plenty of people I'm sure that do who would also have interest in helping.

What I would like to see:
For a display window, have 3 seperate columns streaming events with timestamps. Column 1 would have auto attacks and heroic strikes, Column 2 would have special attack damage, and Column 3 would have buffs gaining and fading.




Dude, you're pretty much right on the exact same page as me. Flurry and similar proc-based haste mechanics make event-based a bad idea, imo. Have to be able to handle those mid-swing procs. Really, that is my entire motivation for taking on this project: To get a more accurate picture of warrior DPS. From the spreadsheet work I've done, I can see where there are some serious issues. People mentioned ones like mid-swing haste, flurry up-time, rage spikes/starvation, etc... Another HUGE deal though is execute when dual-wielding with different speed weapons. There is simply NO good way to model this in a spreadsheet. I don't have a lot of confidence in the model I currently use for execute speed in the spreadsheet. It's a lot better than a lot of the other methods that have been used, but at the end of the day it's still a stop-gap.

I'm wanting to do this because I am interested in the results, and because it looks like a fun challenge for me. Language-wise, I am most likely going to use C++, since it's the only language I really know. All this new shit like Java and Python and C# wasn't invented yet when I learned how to program. I do think I'm gonna say fuck the GUI for now and use data file input.

Ohhh, another idea that popped into my head....maybe do support for two different configs and have them run side-by-side. Like, if you want to use different rage spending parameters or different specs or something.

Also, I don't know what my availability for working on this is going to be for the near future. There is very likely going to be a strike starting on Sunday. I am a contractor, so not only will I still be working, but I'll be working 12 hours per day, 6 days per week. Now, if it's really dead here that means MORE time to work on stuff, but with so many less people, I may end up getting swamped. No idea how long the strike will last either. We'll see. I mainly say this as a reminder to people to be patient. I intend to follow this project through. It's just gonna be a little slow getting off the ground.

#38 dedmonwakeen

dedmonwakeen

    Bald Bull

  • Members
  • 1,302 posts

Posted 01 August 2008 - 11:35 PM

Dude, you're pretty much right on the exact same page as me. Flurry and similar proc-based haste mechanics make event-based a bad idea, imo. Have to be able to handle those mid-swing procs.


Both systems are "event-based"...... The difference is merely how those events are stored.

A classic "timing-wheel" simulator uses a specific granularity..... which become the buckets on the wheel.

A simple ordered link list is another alternative.

In both cases, rescheduling a swing event represents the same amount of complexity and runtime.

A timing wheel system makes for faster event inserts into the "future"....... and the cost of potentially many empty slots in the wheel that you waste your time checking for "next event".

My recommendation to you is to abstract the event storage system from the rest of your mechanics so you can tweak it at a later date.

For instance, when I'm only simulating one player, the simple ordered event list is fastest..... When I pile 10+ players into the simulator, the timing wheel starts to look good.

#39 Machinator

Machinator

    Don Flamenco

  • Members
  • 498 posts

Posted 02 August 2008 - 07:51 PM

You don't start to see very consistent results until you have several days of combat time per run. The strategy I took is to repeat a fight of X duration a large number of times (which makes handling things like trinkets and death wish more comparable to real fights)


I know it wouldnt be able to distinguish between using DW 2 or 3 times a fight, but would just running one long sim and stopping when the average dps change is less than a really small number give a good baseline? Something is nagging in my head saying no but I dont know what.


For the sim a feature I would like to see is record wasted rage, such as rage >100. I think that would be very important for things like Titan's Grip or sword spec.

Also how does WoW handle rounding? It seems like it keeps the remainder for the next hit but Ive never seen it tested.
"Information is ammunition."

#40 Grim13

Grim13

    Piston Honda

  • Members
  • 105 posts

Posted 07 August 2008 - 10:27 PM

Thanks for all the replies everyone. I appreciate the input very much. I just wanted to check in and mention that I am nearing completion of the spreadsheet project, which means I'll be able to start working on the simulator project soon. I've put out a new beta of the sheet for testing. I know it's pretty buggy right now, but that's why I'm calling it a beta. Much easier to get a bunch of folks to help me find the bugs.

http://elitistjerks....p42/#post845392




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users