Jump to content


Photo

How to get XML with PHP from armory


  • Please log in to reply
122 replies to this topic

#21 Nadiar

Nadiar

    Von Kaiser

  • Members
  • 91 posts

Posted 23 July 2008 - 01:17 AM

I'm using Google AppEngine for my Armory-crawling site. If anyone else is using it, AppEngine's URL fetching ability doesn't let one set the User-agent so only the HTML responses are available. If anyone knows a workaround I'd love to hear it.

Here's my tool, Gear Wishlist that generates a list of potential gear upgrades. It doesn't support power-user tweaking, but it a simple way to eyeball what's out there.


This isn't really a great thread for it, but it would be cool to get source code for it when you have something working.

Also, right now "Silver Hand" doesn't seem to work at all (says "Error: Server Error"), but "Silver+Hand" pops up asking me if I want Feral DPS or Tanking, and a couple times it has shown my current gear beyond the popup asking for which feral spec I am. After clicking a spec it then continues to "Error: Server Error"

#22 bonkers

bonkers

    Glass Joe

  • Members
  • 2 posts

Posted 23 July 2008 - 06:31 AM

This isn't really a great thread for it, but it would be cool to get source code for it when you have something working.

Also, right now "Silver Hand" doesn't seem to work at all (says "Error: Server Error"), but "Silver+Hand" pops up asking me if I want Feral DPS or Tanking, and a couple times it has shown my current gear beyond the popup asking for which feral spec I am. After clicking a spec it then continues to "Error: Server Error"


Interesting, I hadn't tested with realm names containing spaces. I'll look into it, thanks for the heads-up!

(The server logs show a lot of errors trying to fetch character sheets from the armory.)

EDIT: I made a trivial change to replace spaces with '+' for the armory url generation and it seems to be working for you now.

#23 Furizaa

Furizaa

    Somewhat usefull

  • Members
  • 44 posts

Posted 23 July 2008 - 09:16 AM

After i realized that I'm to dump to recompile PHP with cURL, i come up with this to get the Armory XMLs


$ctx = stream_context_create(Array("http" => Array("user_agent" => "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1")));
$charXml = (@file_get_contents(ARMORY_URL.urlencode($name),0,$ctx));


Works great until now on Ultionis Sanguinis for Member-Pages and Item-Tooltips... if you don't use IE, but thats a style problem.

#24 inktomi

inktomi

    Von Kaiser

  • Members
  • 25 posts

Posted 11 February 2009 - 07:28 AM

I did something very similar, but in Java. Now I get "Server is busy. Please try again later." for all my requests. Seems like I got my IP banned.

Just a friendly warning to remember to put a delay between your requests. Don't want to see others getting banned.
<irL> - looking for members today! Inktomi#1227

#25 Trouble

Trouble

    Bald Bull

  • Members
  • 1,448 posts

Posted 11 February 2009 - 12:59 PM

You have to throttle your requests. If you exceed 45 requests in like a minute (or one per 1.5 seconds over a sustained period) you'll start pulling 503's. If you keep hammering after you're pulling 503's then it'll ban you for a day or so. It only lasts for a day or so, but it's your warning.

#26 NinJOu

NinJOu

    Von Kaiser

  • Members
  • 55 posts

Posted 11 February 2009 - 02:39 PM

Hey guys,

I'm looking for an armory parser that can compare raid achievement for all my guildies. We have up to 50 active members and it would be easier if raid achievement tracking could be automated =)

Does anyone have heard of this type of parser ? Or should i just begin to write my own ?

Thx in advance=)

#27 Reliknom

Reliknom

    Piston Honda

  • Members
  • 114 posts

Posted 12 February 2009 - 09:31 AM

If there's a tool like this, or if someone could make one, I would be very much interested as well. :)
I've already asked around in my guild if any of the professional IT guys could put together the code, but there have been no volunteers yet.
"Morituri Nolumus Mori!"

#28 Furizaa

Furizaa

    Somewhat usefull

  • Members
  • 44 posts

Posted 12 February 2009 - 10:49 AM

This is a very interesting idea. I will try to setup a raw prototype when i get out of the office this evening. It's always jaw-dropping what you can develop with the use of raw Armory data. See wowprogress.com or similar pages.

#29 Tanoh

Tanoh

    Piston Honda

  • Members
  • 200 posts

Posted 13 February 2009 - 11:08 AM

I'm looking for an armory parser that can compare raid achievement for all my guildies. We have up to 50 active members and it would be easier if raid achievement tracking could be automated =)

Does anyone have heard of this type of parser ? Or should i just begin to write my own ?


Your post inspired me to write one for my guild. It's just done for my personal enjoyment and not going to be made into a general tool. It's written in perl and uses a common package I've developed for all my armory digging.

The result can be seen here: Insomnia achievements

#30 NinJOu

NinJOu

    Von Kaiser

  • Members
  • 55 posts

Posted 13 February 2009 - 12:58 PM

Really good, simple & very fast to read.

Maybe you can share your perl script ? I assume the only variables are your server and guild name ?

#31 Tanoh

Tanoh

    Piston Honda

  • Members
  • 200 posts

Posted 16 February 2009 - 10:30 AM

Cleaned up the code and made a standalone version. Without many of the features my lib has, for example there's no caching at all in this one. But it works.

Source: APtracker.pl

Copy these two images, or draw your own (preferebly 16x16 or you have to change the source to make the tables not completly broken)

http://www.insomnia-...org/data/ok.png
http://www.insomnia-...ta/not_done.png

It should work out of the box with a normal perl installation (at least it does on mine.)

Chances are you're not on "Earthen Ring" (EU) so edit the source and change the three variables $realm, $guild and $region to your realm, guild name and region.

The file "AP_out.html" will be created. If you move it to some other place, make sure the two .png files are in the same directory.

#32 Furizaa

Furizaa

    Somewhat usefull

  • Members
  • 44 posts

Posted 16 February 2009 - 12:57 PM

I got hooked up to this parser stuff (again) and wrote a new parser from scratch in PHP. It highly depends on OOP, the SPL and SimpleXml - so don't try it < PHP 5.2. It's designed to fit in a ZendFramework environment but also runs completely standalone. You are able to plugin various classes for ArmoryResolvers ( currently supported: StreamContext and untested cUrl ), CacheLayers ( Mysqli ), and ArmoryPages ( Character, CharacterAchievements ) or write your own against the provided interfaces. There are also demo clients included which should demonstrate the usage of the parser.

Source for the simple client w/o using cache or errorhandling
#
<?php
//Get all needed includes
require_once dirname(__FILE__) . DIRECTORY_SEPARATOR . 'ArmoryStream' . DIRECTORY_SEPARATOR . 'ArmoryStream.php';
require_once dirname(__FILE__) . DIRECTORY_SEPARATOR . 'ArmoryStream' . DIRECTORY_SEPARATOR . 'Page' . DIRECTORY_SEPARATOR . 'Achievements.php';

//Setup the StreamObject and tell it to use the european Armory data.
$oClient = new ArmoryStream();
$oClient->setLocation( ArmoryStream::LOCATION_EUROPE );

//Setup the PageObject. Use the Achievement page and get FeastsOfStrength of a character
$oCharacterPage = new ArmoryStream_Page_Achievements();
$oCharacterPage->setCharacter('Nnoitra')
->setRealm("Gul'dan")
->setCategory( ArmoryStream_Page_Achievements::CATEGORY_FEASTSOFSTRENGTH );

//Stream data from Armory into the PageObject
$oClient->streamIntoPage( $oCharacterPage );

//Write the FeastsOfStrength into the HTTP-Response
echo "Nnoitra of Gul'dan<br />";
foreach ( $oCharacterPage->Xml->category->achievement as $oAchievement ) {
echo "
<img width='18px' height='18px' src='http://eu.wowarmory.com/wow-icons/_images/51x51/".$oAchievement['icon'].".jpg' />
".$oAchievement['title']."<br />
";
}


Theres also a demo for a client with errorhandling and one with enabled mysqli cache in the download package.

Client in action (shows Feasts of Strength for a character):
http://armorytest.en...ient_cached.php <-- Sorry, not working anymore after server crash. :(

Directory:
/ArmoryParser
  ArmoryParser.php
  Exception.php
  /Resolver
    Interface.php
    Exception.php
    StreamContext.php
    CUrl.php
  /Page
    Interface.php
    Exception.php
    Base.php
    Character.php
    Achievements.php
  /Cache
    Interface.php
    Exception.php
    Base.php
    MySqli.php

  • ArmoryParser.php The base class.
  • Resolver/StreamContext.php Classes under the Resolver directory are the real Armory parser which get the XML stream off the armory. StreamContext uses a simple but working file_get_contents(). You are able to provide your own Resolver as long it's implementing the ResolverInterface.
  • Resolver/CUrl.php This uses the code from page 1 of this thread to get the XML and is not tested since I am to dump/lazy to recompile my PHP with cUrl.
  • Page/Character.php Use an instance of this class to get characterdata from the Armory. The objects holds the request information (charname, realm, ...) and handles the result. Implement your own Armory pages by creating classes that extends Pages_Base and implements Pages_Interface.
  • Page/Achievements.php Same as above. Gets the achievementpage from the Armory.
  • cache/MySqli.php A cacheadapter that uses PHP's MySqli extension. Provide a databaseconnection and it will create it's tables itself if needed. Same here - write your own wich extends Cache_Base and implements Cache_Interface ( like MySql or Zend_Db_Table_Abstract ).

It should be really stable due the exceptionhandling (see client.php), but it's a lot of work to do - especially implement more page classes to cover the whole Armory.

I wrote this stuff for my own projects that involve the Armory, but I decided the share the code with the community. As long as there are people wo are interessted in the parser, I can provite updates. The code will elvolve during my current 'in mind' project. If someone wants to contribute, I can offer access to the SVN-Repo (at last i hope so... new to svn). I'm going to implement a public page to compare achievements for guilds with that parser and the ZendFramework in my sparetime.

Here's the source:
http://furizaa.end-g...rser.legacy.tar

**Update 03.10.2010**
Since this code is now over 1 year old and I'm not playing WoW anymore I can't give support on this. But in general it should still work thou.
**End Update**

Rev19: Fixed US-Armory support - for real
Rev18: Fixed US-Armory support
Rev10: First release

If you find this piece of code usefull let me know. If even its just to see what is PHP capable of with a grain of OOP.

#33 NinJOu

NinJOu

    Von Kaiser

  • Members
  • 55 posts

Posted 16 February 2009 - 02:29 PM

Cleaned up the code and made a standalone version. Without many of the features my lib has, for example there's no caching at all in this one. But it works.



Thanks for your work.

I also spent some of my week end on a PHP script based on what's been posted on the first page. I'm gonna clean it (too many vardump) and paste if here if anybody don't like perl :P

#34 arison

arison

    King Hippo

  •  Patrons
  • 578 posts

Posted 17 February 2009 - 09:38 AM

Hey guys,

I'm looking for an armory parser that can compare raid achievement for all my guildies. We have up to 50 active members and it would be easier if raid achievement tracking could be automated =)

Does anyone have heard of this type of parser ? Or should i just begin to write my own ?

Thx in advance=)


I've taken a python script I use for my guild and put it up on Google Code.

wow-scripts - Google Code

It produces nice reports like

http://perl.pattern.net/glory.txt.utf8

(First is a list by player, then at the bottom, a list by achievement).

Not flashy, but it works, and is reasonably extensible.
<Temerity> - Always recruiting. 12 hrs PST schedule - Valen#1972

#35 Starfox

Starfox

    King Hippo

  • Members
  • 578 posts

Posted 17 February 2009 - 11:40 PM

I've taken a python script I use for my guild and put it up on Google Code.

wow-scripts - Google Code

It produces nice reports like

http://perl.pattern.net/glory.txt.utf8

(First is a list by player, then at the bottom, a list by achievement).

Not flashy, but it works, and is reasonably extensible.


Oh nice, just had to edit the armory url to use the EU server and after some tries i realised that the options parser does not like 's and i have to use "s, german windows perhaps.
Thanks for that nonflashy approach :)
Hello.
Light the fuse.
For all my homies.
Do not run, we are your friends.
SimulationCraft Druid Guy

#36 Furizaa

Furizaa

    Somewhat usefull

  • Members
  • 44 posts

Posted 23 February 2009 - 11:22 PM

In the last few days, i try'd hard to get a service running where you enter a guild, a server and a MetaAchievement and the service provides a nicely sorted table where you are able to compare those. It works in theory: The "CompareRequest" is saved as a job in a database and a deamon in the background fetches the achievements per character from the Armory. A little progress-page with Ajax let's you know wich queue position you got with a nicely animated progressbar.

The average fetchtime of a guild is like 10-15 seconds - and here start the problems. If more than ... like 5 guilds are in the queue, the wait time will be very long. I can bypass this by providing a short link to the user ( http://www.awesome.tld/gt66sr ) and tell him, he can come back later with the provided URL. BUT: There is also the EU Armory. The average time to fetch a guild with the EU Armory is 10 minutes due the mass amount of timeouts it produces. I currently have no solution for that.

If i request a page on the EU Armory via standart Browser, i get the page in no time :awesome:

Posted Image

#37 winkiller

winkiller

    Von Kaiser

  • Members
  • 43 posts

Posted 24 February 2009 - 01:47 AM

In the last few days, i try'd hard to get a service running where you enter a guild, a server and a MetaAchievement and the service provides a nicely sorted table where you are able to compare those. It works in theory: The "CompareRequest" is saved as a job in a database and a deamon in the background fetches the achievements per character from the Armory. A little progress-page with Ajax let's you know wich queue position you got with a nicely animated progressbar.

The average fetchtime of a guild is like 10-15 seconds - and here start the problems. If more than ... like 5 guilds are in the queue, the wait time will be very long. I can bypass this by providing a short link to the user ( http://www.awesome.tld/gt66sr ) and tell him, he can come back later with the provided URL. BUT: There is also the EU Armory. The average time to fetch a guild with the EU Armory is 10 minutes due the mass amount of timeouts it produces. I currently have no solution for that.

If i request a page on the EU Armory via standart Browser, i get the page in no time :awesome:


Well, if the sourcecode was available, it would be no problem if people could run it on their own servers ;) (It looks awesome, I'd love to have this)

As far as I know you should have 1.5s between each request, that's what I was told to avoid a 24h IP ban - I couldn't reproduce it yet though...

To your problem, I've written a little tutorial a while ago (Automated Armory Access Tutorial) where there's some info how, for example, let cURL send a "real browser" User Agent, that sometimes helps with the Armory (plain cURL often failed me).

And yes, EU armory is horrible, I can run our "Who in guild killed which boss how often" tool only at 5-7 in the morning without errors, whole day it's slooow.
Live fast. Die young. Spirit rez. Take revenge.

#38 Furizaa

Furizaa

    Somewhat usefull

  • Members
  • 44 posts

Posted 24 February 2009 - 09:21 AM

This would be a pain in the ass to install on differend servers ( or at last for me to document how to do this ;) ) with the background deamon and framework stuff. I will try to solve the Armory problems. If I fail i can make the source avialable anytime.

For the request i use the Zend_Http_Client library wich uses ( i guess ) fsockopen. With FireBug I was able to copy the whole Http request headers for use in the daemon, but still i get timeouts with the script where i get a clean response with my browser. It's not like the timeouts begin with the later requests - they appear from the very beginning of the fetching process.

I guess i have to prefetch the data like WoWProgress does it for a LIST ( caps indended ) of guilds.

#39 Xeres

Xeres

    Glass Joe

  • Members
  • 20 posts

Posted 27 February 2009 - 04:06 PM

Nice work guys. I'm currently trying to get the Armory Tool from mmo-champ running for parsing the armory to check the Hodir Reputation or any reputation for my guild. Modifying this looks pretty easy but I end up with tons of mysql errors which I can't explain as everythings looks setup correct in the configs for the parser as well as in XAMP. So my question is, is there any parser already out there, that can parse the XMLs for individual reputation tabs or single reputations? As I lack knowledge on the PHP part and couldn't find anything on google and various forums I end up with this posting here.

#40 Furizaa

Furizaa

    Somewhat usefull

  • Members
  • 44 posts

Posted 28 February 2009 - 11:32 AM

There are plenty of options listed in this thread. Even a full featured tutorial by Winkiller (Automated Armory Access Tutorial - codeschmie.de) wich i think is very good. You can boil down all the parsers to a few lines of code which manage to retrieve the XML from the Armory... and there are plenty of ways to do it - i ended up of using the Zend_Http_Client object. One of the simplest ways is to rape file_get_contents:

$pageString = "http://eu.wowarmory.com/character-reputation.xml?r=Gul'dan&n=Nnoitra";

$getRequest = stream_context_create( array(
'http' => array(
'user_agent' => "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1"
)
) );

$xmlString = file_get_contents( urlencode( $pageString ) , 0, $getRequest );


Now do whatever you want with the xmlString. Note: Using file_get_contents the xmlString appears to be empty if you try to write it to the response. Use a Php XML library or extension to load the string and process it further - or use cURL.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users