Archived

This topic is now archived and is closed to further replies.

Aldriana

Spreadsheets: Deep Space Nine

103 posts in this topic

I think the biggest shortcoming of ShadowCraft is in terms of gear recommendations. I think the spreadsheets of Wrath were much better at presenting upgrade recommendations than ShadowCraft is. As it turns out making good gear recommendations is surprisingly complicated, and something that you really need to put time and effort into to get it right; and in Cata, we didn't. And I kind of wish we had.

I honestly don't think this is as relevant as one would assume from face value. We have the means to generate a BiS list, albeit reliant on a bit of trial and error; it is a one time process. From there, regardless of how transitory gearing may appear to be, we are all working with a predefined shopping list. I feel like knowing when X may be better than Y in my current set even though Y is BiS for a full BiS set is a fringe scenario that isn't as practically helpful as knowing the full BiS set to begin with.

While, in the sense of having a feature rich tool, this is something to consider, I would maybe take the design approach of "these are the ULTIMATE upgrade recommendations" - rather than worry about how to construct an interface showing every single sidegrade along the way.

Share this post


Link to post
Share on other sites

One thing I've learned while working on spreadsheets over the years is that to a surprisingly large extent, knowing what's best isn't actually enough. For a variety of reasons, people want to know the viability of alternatives. When you're progressing through Dragon Soul, knowing that the Legendaries are BIS doesn't help you much, as you can't necessarily get them until you've already done a fair portion of the instance. And knowing that Heroic No'kaled is the best alternative doesn't really help either, as if you're still in progression mode you're clearly not killing Heroic Madness. So there's value in knowing how good each of the various alternatives is.

Moreover, some rogues don't ever clear Heroics at all. Some don't even clear normal mode. Some guilds use DKP so its important to know how large an upgrade things are and not just what's ultimately best - the journey is as important as the destination.

And this applies to other aspects of such tools as well. In my TBC and Wrath spreadsheets, I did a lot of work to allow the spreadsheet to figure out the optimal cycle for people, so all you had to do was enter gear to get recommendations. And there's value in that. But the ability to say "how big a DPS loss is it to drop rupture from my rotation" or "how large is the gap between high-rupture and low-rupture" is also valuable.

Hence, while ShadowCraft in its current form is certainly adequate for some use cases, I think think their would be value in having something similar to what spreadsheets had - a column where you can see exactly what your available upgrades are, and how much benefit they provide, and have the ability to filter that list by what content is available to you. Is it as important as having good underlying modeling? Maybe not. But it is a feature that I miss in ShadowCraft.

Share this post


Link to post
Share on other sites

I can definitely see the merits in all of what you're saying. In hindsight, I guess I was actually pretty reliant on the spreadsheet that sp00n had been maintaining with all of the gear options clearly laid out, then manually pulling EP's from ShadowCraft to plug back into the spreadsheet. An integrated list like that would have been quite convenient.

At this point into (or even post) progression, the spreadsheet doesn't really matter anymore, since now we're just fishing for specific drops to fill in BiS gaps. That doesn't mean it doesn't have its use in the big picture. I'll concede, I was being a bit shortsighted. Gear recommendations and filtering options would be a pretty neat addition.

That being said, one thing I'd personally like to see more of is flexibility with cycles (like your example of high-rupture vs low-rupture). I realize that this has been discussed, that most users likely lack the technical aptitude to mess around with the back end (and that it's not likely going to prove useful anyway). What if a scripting language was developed within the engine to do the modeling, give us the basic options (rupture? when to use rvs? etc) hard coded through the main interface, but then give us a sandbox to enter our own behaviors into? Not sure how useful it would be, but it would certainly be fun to play with.

Share this post


Link to post
Share on other sites
But the ability to say "how big a DPS loss is it to drop rupture from my rotation" or "how large is the gap between high-rupture and low-rupture" is also valuable.

This would particularly be the case in the context of fights with unique mechanics that could potentially be toggled or manually accounted for within the UI, such as Seliathan's suggestion to somehow account for energy-capping for X% of the fight, or to account for receiving a Y% damage buff for Z% of the encounter (as with Mr. Ping-Pong), or to account for time spent in front of the boss.

To riff off this, for purposes of self-analysis, it might be pretty cool to see a breakdown of our component damage. If I look over on the right side of the page and see that my optimal perfect-rotation-on-Patchwerk DPS is 48675.60, I could see value in knowing what percentage of that "should" come from white damage, what percent from SS, Deadly Poison, RvS-enhanced Eviscerates, Rupture, etc.

These are things that may be intuitive to folks with a greater brain capacity than mine, but when I'm going back over logs after a fight, that kind of breakdown could help me more quickly and accurately pinpoint the weak spots in my execution -- particularly if we could "toggle" for unique fight mechanics at the same time.

Share this post


Link to post
Share on other sites

Its interesting to note that some of these things are actually supported in the ShadowCraft backend (and to some extent in the frontend as well). For instance, there are some cycle options - a limited number of them to be sure, but they do exist. And its sort of in the nature of a calculator that supporting arbitrary cycles is hard - while Anticipation might make it a little bit easier, I seriously doubt we're going to get to a point where you can specify your own cycle rules and expect ShadowCraft to sort out the implications.

Similarly, damage breakdowns are already done by the backend; all that's really missing on that front is UI to display it.

Ultimately, though, I think the next big step forward for calculators like ShadowCraft will be to weaken the restrictions on the sort of fight they model. As noted, ShadowCraft, like most tools that have ever been written, focuses on the Patchwerk case; but how many Patchwerk type fights do we really have these days? Ultraxion is the only fight in Dragon Soul that even comes particularly close, and even there the interruptions do have an effect on your cycle. Real fights have adds to Blade Flurry or FoK and interruptions and target switches, and while modeling such things is significantly harder than the Patchwerk case, I think its becoming increasingly important and would like to see some work done on supporting such fights. And it probably is more feasible than arbitrary cycle modeling.

Share this post


Link to post
Share on other sites
However, Antiarc indicated that he had free hosting he was willing to use, hence why we're running on his server. Given that he seems less willing/able to pay attention to it these days, we probably do want to find another hosting solution for Mists.

When it came to updates, he had said previously it was best for one of the devs to send a Pull Request on GitHub because he was on that constantly. To get those latest fixes to the backend implemented required me to contact him directly since I don't have ability with GitHub, but once I did he was very quick about updating it after he got my request.

Hey folks, I've been out of town for the week. I'm going to see if I can get things patched up in the next couple of days.

@Aldriana, if convenient, send me a pull request on GitHub - that shows up in my email and makes it easier to do the merge. Not a huge deal, but reduces the chances I'll miss it.

Share this post


Link to post
Share on other sites

A thought for encounter modeling.

Right now shadowcraft effectively works in 1 time step, it models based on averages over the entire fight. Instead you could model in ideally arbitrary length time steps. Each time step would internally use the same calculations as we have now, average energy regen, average cp gen, etc. which would vary based on character and fight parameters. Obviously this would take some restructuring but in principle it should be able to use many of the same calculations we have now, just calculated on a time step rather then fight basis.

Share this post


Link to post
Share on other sites

It might be worth spending some time thinking about how to make modeling more accessible. I get the sense that the perceived complexity is what puts a lot of people off. It would be interesting to know what you guys spent the bulk of your time on in terms of modeling after you got done with the initial implementation. I'm guessing that it was mostly tweaks and bugfixes or implementing new skills or procs.

I can imagine an interface that helps a user gather the data needed to model a skill for example. Maybe you start by picking a comparable skill that's already been modeled, then the interface could provide instructions for producing and uploading a log that demonstrates the skill so the coefficients could be converged upon, etc. It would be tons of work for skilled programmers but possibly less work than those same programmers doing all of the modeling.

As for specific implementation details: what about js or coffeescript (pythonesque js - gets turned in to js and run on the browser or node just like it)? That would give you the expressiveness of an high level language with the accessibility and offline characteristics of a spreadsheet.

Share this post


Link to post
Share on other sites

I can't speak for all people who contributed to ShadowCraft, but from an engine perspective, my sense is that the leading sources of ongoing work were a) set bonuses, b) procs whose mechanics were sufficiently unusual that the basic proc modeling built in from the outset did not apply, and c) improving modelings in places where we'd half-assed it in the first place in the interest of getting things out for launch.

That said: I don't think the model is any more complicated than it needs to be. The fact that its big and complicated and somewhat nasty to modify is precisely because a good model of what's going on is actually pretty complicated. Its just not going to be possible to write a general model that you can just plug new abilities into with a reasonable expectation of accuracy, because every ability is used differently and has slightly different mechanics, and they interact in subtle ways. It will get somewhat easier in Mists - the simpler talent system means you need fewer options to support various talents, and Anticipation simplifies a lot of cycle calculation logic... but sooner or later it will be useful to have non-Anticipation cycles, and there will still be complicated procs and set bonuses, so it will remain a somewhat complicated problem to solve. And I just don't see any way around that.

Share this post


Link to post
Share on other sites
I can imagine an interface that helps a user gather the data needed to model a skill for example. Maybe you start by picking a comparable skill that's already been modeled, then the interface could provide instructions for producing and uploading a log that demonstrates the skill so the coefficients could be converged upon, etc. It would be tons of work for skilled programmers but possibly less work than those same programmers doing all of the modeling.

A thought for encounter modeling.

Right now shadowcraft effectively works in 1 time step, it models based on averages over the entire fight. Instead you could model in ideally arbitrary length time steps. Each time step would internally use the same calculations as we have now, average energy regen, average cp gen, etc. which would vary based on character and fight parameters. Obviously this would take some restructuring but in principle it should be able to use many of the same calculations we have now, just calculated on a time step rather then fight basis.

Most of the complexity is more around cycles and how skills interact with each other within the scope of the cycle, rather than just implementing the basic effects of a skill. Exposing base coefficients for skills seems simple enough (and is often all that needed changing for a patch), but having the ability to allow for the addition of a new skill is, except in the simplest of cases, beyond a reasonable complexity to attempt to implement.

While I think allowing the modelling of a fight to be split into sections to be modelled separately is probably a positive step forward, thought needs to be placed on how to cope with edge cases; skills with long cooldowns which don't apply to each step of the fight, or conversely, coping with skills/effects which need to traverse fight sections (which strikes me a significantly difficult problem to solve). Easier implemented via a step by step simulation. It also requires significantly more input from the user, but I think that is an acceptable situation.

I found the UI side of Shadowcraft a significantly positive step forward. I didn't find it particularly difficult to see what my nearest upgrades where (and I definitely fit into the lower end raider that isn't going to see the BIS stuff as Aldriana mentions). I don't see a large benefit as a user in being able to make significant cycle changes, but I do miss seeing the intermediate calculation steps and being able to play with the base coefficients (in the basic sense).

Share this post


Link to post
Share on other sites
While I think allowing the modelling of a fight to be split into sections to be modelled separately is probably a positive step forward, thought needs to be placed on how to cope with edge cases; skills with long cooldowns which don't apply to each step of the fight, or conversely, coping with skills/effects which need to traverse fight sections (which strikes me a significantly difficult problem to solve). Easier implemented via a step by step simulation. It also requires significantly more input from the user, but I think that is an acceptable situation.

Thinking completely programmaticly about this dealing with skills/effects that cross steps doesn't seem that difficult for most things. You could use the previous time step as an argument to the next time step, from the derived information of the previous time step you could know about the next one. If AR had 33% uptime in a 30 second timestep then you know that AR has 5+(if GoAR)5+(if 4p_t13)+ seconds remaining and similar logic could be used for just about any buff with a known duration and cd, buffs like landslide would be a bit more complex in this approach. You could also just explicitly pass values into each time step instead of calculating them from previous time steps. Dealing with when to use long cd abilities would probably involve passing in a use parameter in each time step. There are a number of ways to structure this but the problems you outline don't seem too difficult.

How this approach compares to a simulation is a question I really don't have an answer to. I was thinking about how to model encounters and CD usage without changing too much of the underlying design. All the calculations you need to do per time step all ready exist so while the program flow would need to change the calculations would not have to significantly.

Share this post


Link to post
Share on other sites

I think there's two main challenges in doing more encounter-based modeling.

First, there's the simple challenge of defining a syntax to describe the various patterns a fight can take. Some mechanics - particularly procs - are very sensitive to the exact pattern of uptime and downtime, so you can't just take the naive approach of looking at % time on target. There are differences between a fight segment that lasts for a particular amount of time, and one that lasts for a particular amount of health. There are differences between a target that dies or goes immune causing you to switch off it, and one where DoTs can keep ticking as you switch to a higher priority target. And so on. Coming up with a model that can handle all these different special cases turns out to be pretty hard.

Second, the hard part of DPS modeling is usually figuring out how many moves of each type you perform in a given period of time - your cycle, basically. And usual approaches for figuring this out rely on the fact that you're generally doing the same sequence of moves many times in a row, so you can pull time-averaging tricks to simplify some calculations. But as the fight segments get shorter, the edge effects get larger, and it gets harder and harder to break the circular dependencies and account for the large variance of cycle elements.

Hence, while a general approach to fight segments would certainly be useful, I don't know how feasible it will be. At the very least, it seems ambitious to attempt initially. If it were me, I might start not with fight length and interruption patterns, but with modeling for AoE situations - knowing how Blade Flurry, FoK, and Crimson Tempest influence your stat priorities on various numbers of targets seems valuable.

To get modeling of different fight lengths and interruption patterns, I might try a simplified model based on breaking fights into segments - i.e., rather than modeling a "6 minute fight", you model it as a 20 second rampup, and 330 seconds of "sustained DPS", and a 10 second "conclusion". This would significantly increase your ability to model shorter fights, although there's still a question of how you deal with temporary buffs and cooldowns and the like.

Or perhaps someone out there has a clever idea for getting around these problems, and in that case I look forward to seeing it; all I know is that I haven't yet been able to come up with a satisfactory way of modeling all the different complicated fight patterns in detail.

Share this post


Link to post
Share on other sites

Ever since I discovered them, I used the spreadsheets and, more recently, Shadowcraft. I'm the sort of person that can understand and explain to someone why something works better than something else once someone else has proved the maths, but the maths goes way over my head to the point I wouldn't have a clue where to start with working things out from scratch. Therefore I have found these tools invaluable to me and heavily rely on them. The work undertaken by all the people that have worked on them over the years is incredible and thank you for it.

Spreadsheets - It was quite fun to be able to poke around with stat weightings and cycles but gear recommendations were difficult to gauge without selecting and recalculating after writing previous results down. I did like that, if an item had an error, or was not in the database for whatever reason, you could quite easily add it manually.

Shadowcraft - I do like the interface - I'm a visual person and it helps me see at a glance what I have and whether it looks right to me. I also like the fact that the items list is much much longer than the spreadsheet's ones ever were - I have runs of dire luck where upgrade items just will not drop and it's nice to be able to still get a reasonable model with the odd older item still in various sockets. Visually again I do like the numbered bar-graph of how much of an improvement an item is - is it worth spending tokens/dkp etc... to get an item for slot A now, or would it be better to save up for an item in slot B because the difference is that much greater.

As many people have already said there are elements of each system that were great and others that didn't work as fluidly. I like the visual aspects of Shadowcraft and the "tweakability" of the spreadsheets. I like the fact that Shadowcraft is online as I can access it from pretty much anywhere without worrying about whether it has the latest data on what my character has, or whether the code is the most recent. I liked how you could choose where your upgrades were coming from in the spreadsheets (deselecting heroic modes if your guild wasn't there yet).

It would probably be way too much of a nightmare to implement but an interesting future feature would be if you could tick-box what classes are actually attending a particular raid and make suggestions as to spec or rotation alterations that might help out. There are certainly fights we've done with such a bad group comp (but without the option to change due to availability of raiders/characters) where we've got through on almost luck alone and I've wondered whether I could do something to improve our chances without knowing quite what. I guess a lot of that is down to experience though.

I think anyone that takes up the gauntlet of developing the next generation of rogue tools has a pretty solid position to work from from the point of what people want or expect to see. I think the only difficulty is the "frills" added outside of the necessary parts and the tools used to create such a tool.

Share this post


Link to post
Share on other sites

One huge difficulty I see with simulating short fights, in relation to what Aldriana mentionned, is the RNG factor. On a 6-minute fight, most RNG tends to cancel itself out. On a 20-second burst phase, your DPS can be, all other things equal, radically different depending on things such as trinket procs: whether strong trinkets are on or off ICD is going to have a huge effect, especially if it's a 15-20s proc (the case with most trinkets).

And that's not even taking into account the problem of CD management. Bosses with repetitive phases can very much change depending on your comp. Typically, any fight with HP%-based burst phases, as opposed to timer-based burst phases, will mean that you will reach a certain point where your CDs are mostly up on burst phases, and then you'll be back to a point where they aren't. I remember this being considered problematic even on patchwerk-style bosses - Deathbringer Saurfang to be precise - where the spreadsheet gave mut rogues above combat rogues, but where combat rogues got out above because the duration of the fight gave them just enough time to slip in an extra AR.

All in all, I'm not that sure that actual, precise burst-phase simulating is possible. Simulating fights based on, as Aldriana put it, a simple Ramp-up | Cruise speed | Execution phase model, might be more interesting, especially in allowing you to time your fights, and make spec decisions based on knowing if you'll be able to use more of a certain CD to greater effect.

Finally, on an unrelated note: modeling AE would be awesome. For fights such as Yor'sahj, Blackhorn, Ragnaros or even Halfus, being able to know how AE is going to come into play would be fascinating indeed.

Share this post


Link to post
Share on other sites

Personally, while it seems like a really great idea to be able to model specific fight mechanics, I really think that's a bit too ambitious. Not just for some of the very good reasons already pointed out, but also because overall raid strategy is not always set in stone. For example do you make the Yor'sahj model keeping the rogue on the boss the whole time or switching to the ooze? Sure it's often times (especially anymore with the raid debuff) to keep the rogue on the boss, but that's not always the case, some groups may still need the rogue to switch. Sometimes what's ideal for the rogue isn't ideal for the raidgroup as a whole. Most often these types of strategy changes deal with when you're struggling/progressing with a fight (or just one aspect of it), and as Aldriana pointed out that's just as important (arguably more-so) as optimizing your dps when you're already in best-in-slot gear.

It just seems that trying to model each fight specifically would be trying to do too much. I much rather agree with what I think (or thought, I might have read a bit into it) Alriana suggested, in instead looking to model different types of situations (you only mentioned AoE situations, but I think there are situations that could be modeled as well). Currently Patchwerk is the only option, but obviously there are others. AoE would be worth modeling, being in front of the boss could be modeled, maybe a situation with frequent target swaps. Maybe add in an option for modeling against non-raid bosses (not everyone raids on every toon, but they may still like to be able to compare upgrades for 5mans, and it shouldn't be too difficult, it'd be the same thing but different hit/exp caps). Modeling short term burst situations, and probably other situations that I'm not thinking of.

This wouldn't necessarily tell us the exact "best" setup for a particular fight, but it would give us more information than shadowcraft currently does, and we could use that information to better adapt ourselves to the needs of individual fights. It may not tell us "for this fight do this", but it would tell us "this is best for this situation, that is best for that situation" and then we can take that information and say "on this fight I'm going to be in this situation a lot, but not in that situation much, so I should go with this setup". We could take that information and apply it to different fights based on what we'll do in those fights.

It'd be the same type of critical thinking that we already need to use, we'd just have some more numbers to help base our decisions off of. And new fights or fights with variable strategies wouldn't require additional work, we'd just have to reconsider which numbers to worry about. The only "new" work that would have to be done in this regard is if Blizzard surprised us by tossing a fight our way that involved some new situation that we haven't been in before.

Share this post


Link to post
Share on other sites
Text..

Well, if there was a scheduling application coupled with it. You could pick a boss fight and add in certain blocks that you could change position and length of. Such as "Out of Melee (In throw range): 10 sec" "New target" "Boss specific mechanic (2 stacks)" etc.

Then if you manage to code the base to take these blocks into account, it would seem it would be quite easy to simply add new blocks with different code in them.

This might prove to be very helpful, but having a background in programming; getting the foundation to work would be quite a pain.

Share this post


Link to post
Share on other sites

I am afraid that those time-blocks might be simply too short to provide any accurate results with current modelling that is based on averaging everything (average buff uptimes, average combo points per second, average finishers per time and so on).

Share this post


Link to post
Share on other sites

I found using the Shadowcraft tool more fun to use then the excel spreadsheets of the past. So I would say keep with that approach.

As for the code part I would love to help. Even though I am a .Net MVC guy, I am capable in Java but have very little experience with Python and none in Ruby. Either way I am sure I could be of assistance. Just let me know what you need of me.

Share this post


Link to post
Share on other sites

Personally i think the ability to lock things such as reforges and gems before pressing optimise would be really handy.

Share this post


Link to post
Share on other sites

I previously only used the Shadowcraft spreadsheet, however I was using AskMrRobot for all of my other toons so i switched to using that on my rogue.

I feel that that site has a little less customization, but gives a gear upgrade path. The only difference I found is the difference in reforges, they don't match at all but I think it's because the EP values are different. Would it be worth talking to the devs of the site rather than trying to completely re-invent the wheel? Seems like it would be simpler...

Share this post


Link to post
Share on other sites

I actually enjoyed and currently miss the spreadsheets. Although Shadowcraft is wonderful tool and big props to those that put it together, I personally would go back to speadsheets in an AR heartbeat. I think its because I get to see all of the numbers and make changes if desired.

Share this post


Link to post
Share on other sites

As a Mac User the Spreadsheets were tricky, never really knew whether they were working right, as not all of the front end functionality imported as intended. Shadowcraft is a huge boon to gearing and forging my character.

I love you guys for publishing it and love the detail and ability it gives to tweak individual reforges, gear and buffs.

Keep up the excellent and hugely appreciated work.

Share this post


Link to post
Share on other sites

Did we/anyone ever decide how to move forwards with the Mists iteration of the engine? Seems like sooner rather than later would be best, I'm at a point where I will need to know the interface.

Additionally, I have a followup to earlier thoughts. I really think there is a middle ground between exposing all the settings of the engine, and keeping it simple for those who don't need to know it. A single textfield can store commands and settings, and the engine can parse and handle them. This solution actual solves other issues like trouble we had with Hemorrhage weaving with the current Shadowcraft, since the command "hemo_interval: 25;" could be passed to the engine, and the engine could deal with it as needed. This means the UI doesn't have to be fixed or changed to handle the option, and it would just need to pull a new engine.

Share this post


Link to post
Share on other sites

I've been meaning to update you guys on this, but haven't found the time to do so. I fully intend to work on the update and shape out the new models; but really, work has kept me busy so it's very possible that I won't be able to do so before release. If no other develpment has been made elsewhere, I guess we can simply update the python library and start producing some stat weights.

As for the inteface, I crossed a couple emails with Antiarc and he's interested in hosting and updating the site, so we can count on that. Note, however, that he isn't playing wow now, so any feedback needs to be directed to him either to his email or through the issues in the ShadowCraft-ui project in github.

About the hemorrhage thing: it could have defaulted to something else; I chose to default use_hemorrhage='never' in the settings module because, iirc, at the time it was written (namely before DS actual release) it wasn't a clear winner. In any case, I would rather have everything exposed, yes.

Share this post


Link to post
Share on other sites

Thought I'd go ahead and post even though wheels look like they've already been set in motion. I'm all for the Shadowcraft approach and think that the only key flaw was not enough support from the people that could help. What Antiarc and everyone that worked on the code did was amazing and for the amount I can guess the site was used even more so.

Not only was Shadowcraft a great quick and easy tool for me to use but it allowed anyone that came to myself/other guildies chance to help them use a site that answered pretty much any of their questions.

I'm at a point myself where I can only offer so much help on the coding front but if it ever comes to a point where Antiarc can't host or doesn’t have the tools to be able to make the hosting easy, I'm sure I'll be able to work something out with Curse/MMOC/Myself to give you guys and gals that can code and spend the time to the stuff you need.

Share this post


Link to post
Share on other sites

  • Recently Browsing   0 members

    No registered users viewing this page.