Jump to content


Photo

Spreadsheets: Deep Space Nine


  • Please log in to reply
102 replies to this topic

#21 Mechakisc

Mechakisc

    Glass Joe

  • Members
  • 15 posts
  • LocationFallon NV

Posted 11 April 2012 - 11:14 PM

Our 10 man raid had two hunters and no rogue through most of Firelands, up until the announcement of Dragon Soul and the legendaries. I was the one best positioned to switch to rogue so our guild could pick up the daggers. I'll never go back to hunter - as much as I enjoyed the class, I learned that I vastly prefer melee - and rogue in particular.

Shadowcraft made a big difference in my ability to ramp up quickly, and I'm very grateful for it. I very much prefer solutions that are online to spreadsheets or addons (I run too many addons as it is).

I have an IT/web hosting/sysadmin background with limited coding exposure (HTML, PERL, some Python awhile back), including some MS SQL management. I dunno if I can actually be helpful, but I will contribute any way that I can. I do not have access to any spare physical servers at this time, though I have an old google app account with 100 email accounts available - the new ones seem to only have 50. I don't know if that makes a difference to traffics and such, but - again - if I can be helpful, I'd like to be.

#22 fierydemise

fierydemise

    Von Kaiser

  • Members
  • 52 posts

Posted 12 April 2012 - 12:28 AM

The biggest thing I would like to see in a theory crafting tool is a CD planner. We've generally had some basic rules about CD usage to work with a tool to actually experiment with would be incredibly useful.

Additionally, since I think this is the right place, I'm throwing my hat in the ring so to speak on helping with Shadowcraft or a similar community developed tool in MOP.

#23 Pathal

Pathal

    Don Flamenco

  • • Guide Author
  • 345 posts

Posted 12 April 2012 - 02:32 AM

I carried this discussion over the official US forums, to try and get a few more opinions on the matter.

Overall everything seemed rather predictable. The biggest concerns seemed to be about stability, keeping up with changes/known bugs (without having to go to this forum), and customizing the way the reforger works (custom EPs and custom caps). A couple people were excited about the idea of a mobile version, which may be my fault, but I can't see it ever being a serious goal and I'm just mentioning it again for the sake of being complete.

I still think we can accomplish more end user control without being overbearing, or confusing those less interested. Things like parryable, bonus haste effects (Sinestra), and setting the target's level (& armor, potentially) shouldn't be terribly difficult to add to a gui.

Another thought of mine. What about a set series of files, hosted publicly and away from the engine that has a list of the current known bugs, and the currently accepted BiS for each spec. Apps could then pull that info and use it however they want without having to hardcode this info into the engine or client/server somewhere. The issue is that it probably has to be a static URI, and there would probably have to be some sort of shared access. The alternative would be parsing the first post for each thread, but that seems messy.

#24 Seliathan

Seliathan

    Von Kaiser

  • Members
  • 69 posts

Posted 12 April 2012 - 11:25 AM

I still think we can accomplish more end user control without being overbearing, or confusing those less interested. Things like parryable, bonus haste effects (Sinestra), and setting the target's level (& armor, potentially) shouldn't be terribly difficult to add to a gui.


I totally second that, seeing how several EP values might change considering that a few fights in Cataclysm had either a haste buff (Sinestra, Madness, Alysrazor) or increased/decreased the value of soft-capping expertise, which then might lead to a complete reforge. Unfortunately, for haste buffs being evaluated properly, one would have to somehow take energy capping into consideration in such a tool, which as far as I can guess is still pretty much impossible?

#25 Docrev

Docrev

    Piston Honda

  • Members
  • 216 posts

Posted 12 April 2012 - 11:51 AM

Energy capping being modeled really depends on how it approaches it. It should be entirely feasible if you do a full simulation, but if you use averaged values, you probably can't.

#26 Shibs

Shibs

    Glass Joe

  • Members
  • 2 posts

Posted 12 April 2012 - 04:56 PM

I think the biggest shortcoming of ShadowCraft is in terms of gear recommendations. I think the spreadsheets of Wrath were much better at presenting upgrade recommendations than ShadowCraft is. As it turns out making good gear recommendations is surprisingly complicated, and something that you really need to put time and effort into to get it right; and in Cata, we didn't. And I kind of wish we had.

I honestly don't think this is as relevant as one would assume from face value. We have the means to generate a BiS list, albeit reliant on a bit of trial and error; it is a one time process. From there, regardless of how transitory gearing may appear to be, we are all working with a predefined shopping list. I feel like knowing when X may be better than Y in my current set even though Y is BiS for a full BiS set is a fringe scenario that isn't as practically helpful as knowing the full BiS set to begin with.

While, in the sense of having a feature rich tool, this is something to consider, I would maybe take the design approach of "these are the ULTIMATE upgrade recommendations" - rather than worry about how to construct an interface showing every single sidegrade along the way.

#27 Aldriana

Aldriana

    Mike Tyson

  • Moderators
  • 13510 posts

Posted 12 April 2012 - 06:16 PM

One thing I've learned while working on spreadsheets over the years is that to a surprisingly large extent, knowing what's best isn't actually enough. For a variety of reasons, people want to know the viability of alternatives. When you're progressing through Dragon Soul, knowing that the Legendaries are BIS doesn't help you much, as you can't necessarily get them until you've already done a fair portion of the instance. And knowing that Heroic No'kaled is the best alternative doesn't really help either, as if you're still in progression mode you're clearly not killing Heroic Madness. So there's value in knowing how good each of the various alternatives is.

Moreover, some rogues don't ever clear Heroics at all. Some don't even clear normal mode. Some guilds use DKP so its important to know how large an upgrade things are and not just what's ultimately best - the journey is as important as the destination.

And this applies to other aspects of such tools as well. In my TBC and Wrath spreadsheets, I did a lot of work to allow the spreadsheet to figure out the optimal cycle for people, so all you had to do was enter gear to get recommendations. And there's value in that. But the ability to say "how big a DPS loss is it to drop rupture from my rotation" or "how large is the gap between high-rupture and low-rupture" is also valuable.

Hence, while ShadowCraft in its current form is certainly adequate for some use cases, I think think their would be value in having something similar to what spreadsheets had - a column where you can see exactly what your available upgrades are, and how much benefit they provide, and have the ability to filter that list by what content is available to you. Is it as important as having good underlying modeling? Maybe not. But it is a feature that I miss in ShadowCraft.

#28 Shibs

Shibs

    Glass Joe

  • Members
  • 2 posts

Posted 12 April 2012 - 07:33 PM

I can definitely see the merits in all of what you're saying. In hindsight, I guess I was actually pretty reliant on the spreadsheet that sp00n had been maintaining with all of the gear options clearly laid out, then manually pulling EP's from ShadowCraft to plug back into the spreadsheet. An integrated list like that would have been quite convenient.

At this point into (or even post) progression, the spreadsheet doesn't really matter anymore, since now we're just fishing for specific drops to fill in BiS gaps. That doesn't mean it doesn't have its use in the big picture. I'll concede, I was being a bit shortsighted. Gear recommendations and filtering options would be a pretty neat addition.

That being said, one thing I'd personally like to see more of is flexibility with cycles (like your example of high-rupture vs low-rupture). I realize that this has been discussed, that most users likely lack the technical aptitude to mess around with the back end (and that it's not likely going to prove useful anyway). What if a scripting language was developed within the engine to do the modeling, give us the basic options (rupture? when to use rvs? etc) hard coded through the main interface, but then give us a sandbox to enter our own behaviors into? Not sure how useful it would be, but it would certainly be fun to play with.

#29 Rfeann

Rfeann

    Von Kaiser

  • Members
  • 99 posts

Posted 12 April 2012 - 08:22 PM

But the ability to say "how big a DPS loss is it to drop rupture from my rotation" or "how large is the gap between high-rupture and low-rupture" is also valuable.


This would particularly be the case in the context of fights with unique mechanics that could potentially be toggled or manually accounted for within the UI, such as Seliathan's suggestion to somehow account for energy-capping for X% of the fight, or to account for receiving a Y% damage buff for Z% of the encounter (as with Mr. Ping-Pong), or to account for time spent in front of the boss.

To riff off this, for purposes of self-analysis, it might be pretty cool to see a breakdown of our component damage. If I look over on the right side of the page and see that my optimal perfect-rotation-on-Patchwerk DPS is 48675.60, I could see value in knowing what percentage of that "should" come from white damage, what percent from SS, Deadly Poison, RvS-enhanced Eviscerates, Rupture, etc.

These are things that may be intuitive to folks with a greater brain capacity than mine, but when I'm going back over logs after a fight, that kind of breakdown could help me more quickly and accurately pinpoint the weak spots in my execution -- particularly if we could "toggle" for unique fight mechanics at the same time.

#30 Aldriana

Aldriana

    Mike Tyson

  • Moderators
  • 13510 posts

Posted 12 April 2012 - 08:47 PM

Its interesting to note that some of these things are actually supported in the ShadowCraft backend (and to some extent in the frontend as well). For instance, there are some cycle options - a limited number of them to be sure, but they do exist. And its sort of in the nature of a calculator that supporting arbitrary cycles is hard - while Anticipation might make it a little bit easier, I seriously doubt we're going to get to a point where you can specify your own cycle rules and expect ShadowCraft to sort out the implications.

Similarly, damage breakdowns are already done by the backend; all that's really missing on that front is UI to display it.

Ultimately, though, I think the next big step forward for calculators like ShadowCraft will be to weaken the restrictions on the sort of fight they model. As noted, ShadowCraft, like most tools that have ever been written, focuses on the Patchwerk case; but how many Patchwerk type fights do we really have these days? Ultraxion is the only fight in Dragon Soul that even comes particularly close, and even there the interruptions do have an effect on your cycle. Real fights have adds to Blade Flurry or FoK and interruptions and target switches, and while modeling such things is significantly harder than the Patchwerk case, I think its becoming increasingly important and would like to see some work done on supporting such fights. And it probably is more feasible than arbitrary cycle modeling.

#31 Furtim

Furtim

    Piston Honda

  • Members
  • 127 posts

Posted 12 April 2012 - 08:59 PM

However, Antiarc indicated that he had free hosting he was willing to use, hence why we're running on his server. Given that he seems less willing/able to pay attention to it these days, we probably do want to find another hosting solution for Mists.


When it came to updates, he had said previously it was best for one of the devs to send a Pull Request on GitHub because he was on that constantly. To get those latest fixes to the backend implemented required me to contact him directly since I don't have ability with GitHub, but once I did he was very quick about updating it after he got my request.

Hey folks, I've been out of town for the week. I'm going to see if I can get things patched up in the next couple of days.

@Aldriana, if convenient, send me a pull request on GitHub - that shows up in my email and makes it easier to do the merge. Not a huge deal, but reduces the chances I'll miss it.



#32 fierydemise

fierydemise

    Von Kaiser

  • Members
  • 52 posts

Posted 12 April 2012 - 09:17 PM

A thought for encounter modeling.

Right now shadowcraft effectively works in 1 time step, it models based on averages over the entire fight. Instead you could model in ideally arbitrary length time steps. Each time step would internally use the same calculations as we have now, average energy regen, average cp gen, etc. which would vary based on character and fight parameters. Obviously this would take some restructuring but in principle it should be able to use many of the same calculations we have now, just calculated on a time step rather then fight basis.

#33 laod

laod

    Glass Joe

  • Members
  • 8 posts

Posted 13 April 2012 - 03:15 AM

It might be worth spending some time thinking about how to make modeling more accessible. I get the sense that the perceived complexity is what puts a lot of people off. It would be interesting to know what you guys spent the bulk of your time on in terms of modeling after you got done with the initial implementation. I'm guessing that it was mostly tweaks and bugfixes or implementing new skills or procs.

I can imagine an interface that helps a user gather the data needed to model a skill for example. Maybe you start by picking a comparable skill that's already been modeled, then the interface could provide instructions for producing and uploading a log that demonstrates the skill so the coefficients could be converged upon, etc. It would be tons of work for skilled programmers but possibly less work than those same programmers doing all of the modeling.

As for specific implementation details: what about js or coffeescript (pythonesque js - gets turned in to js and run on the browser or node just like it)? That would give you the expressiveness of an high level language with the accessibility and offline characteristics of a spreadsheet.
Posted Image

#34 Aldriana

Aldriana

    Mike Tyson

  • Moderators
  • 13510 posts

Posted 13 April 2012 - 05:07 AM

I can't speak for all people who contributed to ShadowCraft, but from an engine perspective, my sense is that the leading sources of ongoing work were a) set bonuses, B) procs whose mechanics were sufficiently unusual that the basic proc modeling built in from the outset did not apply, and c) improving modelings in places where we'd half-assed it in the first place in the interest of getting things out for launch.

That said: I don't think the model is any more complicated than it needs to be. The fact that its big and complicated and somewhat nasty to modify is precisely because a good model of what's going on is actually pretty complicated. Its just not going to be possible to write a general model that you can just plug new abilities into with a reasonable expectation of accuracy, because every ability is used differently and has slightly different mechanics, and they interact in subtle ways. It will get somewhat easier in Mists - the simpler talent system means you need fewer options to support various talents, and Anticipation simplifies a lot of cycle calculation logic... but sooner or later it will be useful to have non-Anticipation cycles, and there will still be complicated procs and set bonuses, so it will remain a somewhat complicated problem to solve. And I just don't see any way around that.

#35 Thaela

Thaela

    Von Kaiser

  • Members
  • 81 posts

Posted 13 April 2012 - 05:43 AM

I can imagine an interface that helps a user gather the data needed to model a skill for example. Maybe you start by picking a comparable skill that's already been modeled, then the interface could provide instructions for producing and uploading a log that demonstrates the skill so the coefficients could be converged upon, etc. It would be tons of work for skilled programmers but possibly less work than those same programmers doing all of the modeling.


A thought for encounter modeling.

Right now shadowcraft effectively works in 1 time step, it models based on averages over the entire fight. Instead you could model in ideally arbitrary length time steps. Each time step would internally use the same calculations as we have now, average energy regen, average cp gen, etc. which would vary based on character and fight parameters. Obviously this would take some restructuring but in principle it should be able to use many of the same calculations we have now, just calculated on a time step rather then fight basis.


Most of the complexity is more around cycles and how skills interact with each other within the scope of the cycle, rather than just implementing the basic effects of a skill. Exposing base coefficients for skills seems simple enough (and is often all that needed changing for a patch), but having the ability to allow for the addition of a new skill is, except in the simplest of cases, beyond a reasonable complexity to attempt to implement.

While I think allowing the modelling of a fight to be split into sections to be modelled separately is probably a positive step forward, thought needs to be placed on how to cope with edge cases; skills with long cooldowns which don't apply to each step of the fight, or conversely, coping with skills/effects which need to traverse fight sections (which strikes me a significantly difficult problem to solve). Easier implemented via a step by step simulation. It also requires significantly more input from the user, but I think that is an acceptable situation.

I found the UI side of Shadowcraft a significantly positive step forward. I didn't find it particularly difficult to see what my nearest upgrades where (and I definitely fit into the lower end raider that isn't going to see the BIS stuff as Aldriana mentions). I don't see a large benefit as a user in being able to make significant cycle changes, but I do miss seeing the intermediate calculation steps and being able to play with the base coefficients (in the basic sense).

#36 fierydemise

fierydemise

    Von Kaiser

  • Members
  • 52 posts

Posted 13 April 2012 - 03:31 PM

While I think allowing the modelling of a fight to be split into sections to be modelled separately is probably a positive step forward, thought needs to be placed on how to cope with edge cases; skills with long cooldowns which don't apply to each step of the fight, or conversely, coping with skills/effects which need to traverse fight sections (which strikes me a significantly difficult problem to solve). Easier implemented via a step by step simulation. It also requires significantly more input from the user, but I think that is an acceptable situation.

Thinking completely programmaticly about this dealing with skills/effects that cross steps doesn't seem that difficult for most things. You could use the previous time step as an argument to the next time step, from the derived information of the previous time step you could know about the next one. If AR had 33% uptime in a 30 second timestep then you know that AR has 5+(if GoAR)5+(if 4p_t13)+ seconds remaining and similar logic could be used for just about any buff with a known duration and cd, buffs like landslide would be a bit more complex in this approach. You could also just explicitly pass values into each time step instead of calculating them from previous time steps. Dealing with when to use long cd abilities would probably involve passing in a use parameter in each time step. There are a number of ways to structure this but the problems you outline don't seem too difficult.

How this approach compares to a simulation is a question I really don't have an answer to. I was thinking about how to model encounters and CD usage without changing too much of the underlying design. All the calculations you need to do per time step all ready exist so while the program flow would need to change the calculations would not have to significantly.

#37 Aldriana

Aldriana

    Mike Tyson

  • Moderators
  • 13510 posts

Posted 13 April 2012 - 04:05 PM

I think there's two main challenges in doing more encounter-based modeling.

First, there's the simple challenge of defining a syntax to describe the various patterns a fight can take. Some mechanics - particularly procs - are very sensitive to the exact pattern of uptime and downtime, so you can't just take the naive approach of looking at % time on target. There are differences between a fight segment that lasts for a particular amount of time, and one that lasts for a particular amount of health. There are differences between a target that dies or goes immune causing you to switch off it, and one where DoTs can keep ticking as you switch to a higher priority target. And so on. Coming up with a model that can handle all these different special cases turns out to be pretty hard.

Second, the hard part of DPS modeling is usually figuring out how many moves of each type you perform in a given period of time - your cycle, basically. And usual approaches for figuring this out rely on the fact that you're generally doing the same sequence of moves many times in a row, so you can pull time-averaging tricks to simplify some calculations. But as the fight segments get shorter, the edge effects get larger, and it gets harder and harder to break the circular dependencies and account for the large variance of cycle elements.

Hence, while a general approach to fight segments would certainly be useful, I don't know how feasible it will be. At the very least, it seems ambitious to attempt initially. If it were me, I might start not with fight length and interruption patterns, but with modeling for AoE situations - knowing how Blade Flurry, FoK, and Crimson Tempest influence your stat priorities on various numbers of targets seems valuable.

To get modeling of different fight lengths and interruption patterns, I might try a simplified model based on breaking fights into segments - i.e., rather than modeling a "6 minute fight", you model it as a 20 second rampup, and 330 seconds of "sustained DPS", and a 10 second "conclusion". This would significantly increase your ability to model shorter fights, although there's still a question of how you deal with temporary buffs and cooldowns and the like.

Or perhaps someone out there has a clever idea for getting around these problems, and in that case I look forward to seeing it; all I know is that I haven't yet been able to come up with a satisfactory way of modeling all the different complicated fight patterns in detail.

#38 Shadefoot

Shadefoot

    Von Kaiser

  • Members
  • 36 posts

Posted 13 April 2012 - 04:18 PM

Ever since I discovered them, I used the spreadsheets and, more recently, Shadowcraft. I'm the sort of person that can understand and explain to someone why something works better than something else once someone else has proved the maths, but the maths goes way over my head to the point I wouldn't have a clue where to start with working things out from scratch. Therefore I have found these tools invaluable to me and heavily rely on them. The work undertaken by all the people that have worked on them over the years is incredible and thank you for it.

Spreadsheets - It was quite fun to be able to poke around with stat weightings and cycles but gear recommendations were difficult to gauge without selecting and recalculating after writing previous results down. I did like that, if an item had an error, or was not in the database for whatever reason, you could quite easily add it manually.

Shadowcraft - I do like the interface - I'm a visual person and it helps me see at a glance what I have and whether it looks right to me. I also like the fact that the items list is much much longer than the spreadsheet's ones ever were - I have runs of dire luck where upgrade items just will not drop and it's nice to be able to still get a reasonable model with the odd older item still in various sockets. Visually again I do like the numbered bar-graph of how much of an improvement an item is - is it worth spending tokens/dkp etc... to get an item for slot A now, or would it be better to save up for an item in slot B because the difference is that much greater.

As many people have already said there are elements of each system that were great and others that didn't work as fluidly. I like the visual aspects of Shadowcraft and the "tweakability" of the spreadsheets. I like the fact that Shadowcraft is online as I can access it from pretty much anywhere without worrying about whether it has the latest data on what my character has, or whether the code is the most recent. I liked how you could choose where your upgrades were coming from in the spreadsheets (deselecting heroic modes if your guild wasn't there yet).

It would probably be way too much of a nightmare to implement but an interesting future feature would be if you could tick-box what classes are actually attending a particular raid and make suggestions as to spec or rotation alterations that might help out. There are certainly fights we've done with such a bad group comp (but without the option to change due to availability of raiders/characters) where we've got through on almost luck alone and I've wondered whether I could do something to improve our chances without knowing quite what. I guess a lot of that is down to experience though.

I think anyone that takes up the gauntlet of developing the next generation of rogue tools has a pretty solid position to work from from the point of what people want or expect to see. I think the only difficulty is the "frills" added outside of the necessary parts and the tools used to create such a tool.

#39 Belegûr

Belegûr

    Piston Honda

  • Members
  • 148 posts

Posted 14 April 2012 - 01:23 PM

One huge difficulty I see with simulating short fights, in relation to what Aldriana mentionned, is the RNG factor. On a 6-minute fight, most RNG tends to cancel itself out. On a 20-second burst phase, your DPS can be, all other things equal, radically different depending on things such as trinket procs: whether strong trinkets are on or off ICD is going to have a huge effect, especially if it's a 15-20s proc (the case with most trinkets).

And that's not even taking into account the problem of CD management. Bosses with repetitive phases can very much change depending on your comp. Typically, any fight with HP%-based burst phases, as opposed to timer-based burst phases, will mean that you will reach a certain point where your CDs are mostly up on burst phases, and then you'll be back to a point where they aren't. I remember this being considered problematic even on patchwerk-style bosses - Deathbringer Saurfang to be precise - where the spreadsheet gave mut rogues above combat rogues, but where combat rogues got out above because the duration of the fight gave them just enough time to slip in an extra AR.


All in all, I'm not that sure that actual, precise burst-phase simulating is possible. Simulating fights based on, as Aldriana put it, a simple Ramp-up | Cruise speed | Execution phase model, might be more interesting, especially in allowing you to time your fights, and make spec decisions based on knowing if you'll be able to use more of a certain CD to greater effect.


Finally, on an unrelated note: modeling AE would be awesome. For fights such as Yor'sahj, Blackhorn, Ragnaros or even Halfus, being able to know how AE is going to come into play would be fascinating indeed.
Posted Image

#40 Pancake3

Pancake3

    Glass Joe

  • Members
  • 10 posts

Posted 17 April 2012 - 10:15 PM

Personally, while it seems like a really great idea to be able to model specific fight mechanics, I really think that's a bit too ambitious. Not just for some of the very good reasons already pointed out, but also because overall raid strategy is not always set in stone. For example do you make the Yor'sahj model keeping the rogue on the boss the whole time or switching to the ooze? Sure it's often times (especially anymore with the raid debuff) to keep the rogue on the boss, but that's not always the case, some groups may still need the rogue to switch. Sometimes what's ideal for the rogue isn't ideal for the raidgroup as a whole. Most often these types of strategy changes deal with when you're struggling/progressing with a fight (or just one aspect of it), and as Aldriana pointed out that's just as important (arguably more-so) as optimizing your dps when you're already in best-in-slot gear.

It just seems that trying to model each fight specifically would be trying to do too much. I much rather agree with what I think (or thought, I might have read a bit into it) Alriana suggested, in instead looking to model different types of situations (you only mentioned AoE situations, but I think there are situations that could be modeled as well). Currently Patchwerk is the only option, but obviously there are others. AoE would be worth modeling, being in front of the boss could be modeled, maybe a situation with frequent target swaps. Maybe add in an option for modeling against non-raid bosses (not everyone raids on every toon, but they may still like to be able to compare upgrades for 5mans, and it shouldn't be too difficult, it'd be the same thing but different hit/exp caps). Modeling short term burst situations, and probably other situations that I'm not thinking of.

This wouldn't necessarily tell us the exact "best" setup for a particular fight, but it would give us more information than shadowcraft currently does, and we could use that information to better adapt ourselves to the needs of individual fights. It may not tell us "for this fight do this", but it would tell us "this is best for this situation, that is best for that situation" and then we can take that information and say "on this fight I'm going to be in this situation a lot, but not in that situation much, so I should go with this setup". We could take that information and apply it to different fights based on what we'll do in those fights.

It'd be the same type of critical thinking that we already need to use, we'd just have some more numbers to help base our decisions off of. And new fights or fights with variable strategies wouldn't require additional work, we'd just have to reconsider which numbers to worry about. The only "new" work that would have to be done in this regard is if Blizzard surprised us by tossing a fight our way that involved some new situation that we haven't been in before.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users