Jump to content


Armory Scraping


  • Please log in to reply
51 replies to this topic

#41 Slackie

Slackie

    Bald Bull

  •  Patrons
  • 2,003 posts

Posted 10 February 2011 - 07:10 AM

The pageSize key in the URL defines the maximum number of auctions you want returned for a single query. The maximum you can set this value to is 200. Well, you can set it higher, but the maximum number of auctions you will get back from a single query is 200 even if you set the number higher.

That being said, if you want more than 200 results, you can actually change the start and end values to do your own pagination. If you queried for start=0&end=200 to get the first 200 results for "Cloth" and then used start=200&end=400 for the same query, you should get 200 more auctions that are different from the ones in your first result set.

#42 Slackie

Slackie

    Bald Bull

  •  Patrons
  • 2,003 posts

Posted 11 February 2011 - 12:43 AM

Has anyone found a way to get the guild tabard form the guild summary page? It seems to be built using javascript and I can't seem to find a way to actually get it from the html I'm trying to parse.


It looks like Blizzard is using <canvas> tags with JavaScript to generate the tabards. If you take a look at the data from the guild page as well as [1] that will probably get you started. You would just parse the data from the JavaScript on the guild page and use [1] to figure out how to craft a URL that will point to your particular guild tabard.

On your guild page Thuggernaut @ Mal'Ganis - Game - World of Warcraft look for the following code:

<canvas id="guild-tabard" width="240" height="240" style="display: inline;">
	<div class="guild-tabard-default tabard-horde"></div>
</canvas>

<script type="text/javascript">
	//<![CDATA[
			$(document).ready(function() {
				var tabard = new GuildTabard('guild-tabard', {
					'ring': 'horde',
					'bg': [ 0, 41 ],
					'border': [ 5, 14 ],
					'emblem': [ 92, 14 ]
				});
			});
	//]]>
	</script>

This results in the JS function retrieving the following URLs:

http://us.battle.net.../ring-horde.png
http://us.battle.net...bards/bg_00.png
http://us.battle.net...s/border_05.png
http://us.battle.net...s/emblem_92.png

You can find the little "hooks" that are on the top here: http://us.battle.net...bards/hooks.png

There are also some color maps for the background color and shadows but I didn't dig much into how that all works. Looking at [1] will help in that respect.

[1] http://us.battle.net...ld-tabard.js?v7

The undermine journal came back up today, does that mean that XML data feeds are back up too? I have always parsed HTML but I am interested in using XML for obvious reasons!


The pre-Cataclysm XML/JSON feeds never went away (well, maybe for a couple days right after the new Armory launched). They are still in the same location. Blizzard mentioned in a forum post after the new Armory launched that they were working on a new API for the Armory. Until they are finished with the new API most folks are just using the XML/JSON data from pre-migration or just parsing HTML.

#43 Ramides

Ramides

    Glass Joe

  • Members
  • 5 posts

Posted 23 February 2011 - 04:28 PM

Hello,

I'm not sure if you can help me, but I try. I want to write an application (C#) to search in the auction house.

With "FireBug" I looked into the Battle.net login process and tried to reproduce it with C#. Unfortunately, this has not worked.

Perhaps it would help if I first show you the source code: the Battle.Net login (not perfect, but enough for a start).
public string Login(string AccountName, string Password)
{
    string URL = "https://eu.battle.net/login/de/login.frag?secureOrigin=true";
    string Post = "accountName=" + AccountName + "&password=" + Password;

    string ServerResponse = SendPOSTRequest(URL, Post);

    if (ServerResponse.IndexOf("success") > 0 && ServerResponse.IndexOf("loginTicket") > 0)
    {
        int Start = ServerResponse.IndexOf("loginTicket")+16;
        ServerResponse = ServerResponse.Substring(Start);

        int End = ServerResponse.IndexOf("\"}")-1;
        ServerResponse = ServerResponse.Substring(0,End);

        LoginKey = ServerResponse;

        return ServerResponse;
    }

    return String.Empty;
}

private string SendPOSTRequest(string URL, string Post)
{
    HttpWebRequest Request = (HttpWebRequest)WebRequest.Create(URL);
    Request.Method = "POST";
    Request.ContentType = "application/x-www-form-urlencoded";
    Request.CookieContainer = new CookieContainer();

    Byte[] byteArray = Encoding.UTF8.GetBytes(Post);
    Request.ContentLength = byteArray.Length;

    Stream DataStream = Request.GetRequestStream();
    DataStream.Write(byteArray, 0, byteArray.Length);
    DataStream.Close();

    HttpWebResponse Response = (HttpWebResponse)Request.GetResponse();
    DataStream = Response.GetResponseStream();
    cookies = Response.Cookies;
    StreamReader reader = new StreamReader(DataStream);
    string ServerResponse = reader.ReadToEnd();
    reader.Close();
    DataStream.Close();
    Response.Close();

    return ServerResponse;
}

After the execution I have the "Login Key" in JSON format (picked out via substring) and 4 cookies (JSESSIONID, BA Tassadar, login.key, cl) in a cookie collection.

After Battle.Net Login I want to retrieve data from the auction house. This is my code:
public string Test()
{
    if (!LoginOK) return String.Empty;
    string URL = "https://eu.battle.net/wow/de/vault/character/auction/alliance/?ST=" + LoginKey;
            
    string ServerResponse = SendGETRequest(URL);

    return ServerResponse;
}
private string SendGETRequest(string URL)
{
    HttpWebRequest Request = (HttpWebRequest)WebRequest.Create(URL);
    Request.Method = "GET";
    Request.CookieContainer = new CookieContainer();
    Request.CookieContainer.Add(cookies);

    HttpWebResponse Response = (HttpWebResponse)Request.GetResponse();
    Stream DataStream = Response.GetResponseStream();
    cookies = Response.Cookies;
    StreamReader reader = new StreamReader(DataStream);
    string ServerResponse = reader.ReadToEnd();
    reader.Close();
    DataStream.Close();
    Response.Close();

    return ServerResponse;
}

But every time I try this code, I get a 404 error at this line:
HttpWebResponse Response = (HttpWebResponse)Request.GetResponse();

Does anyone have any idea what I'm doing wrong?

Thank you in advance for help.

#44 Slackie

Slackie

    Bald Bull

  •  Patrons
  • 2,003 posts

Posted 23 February 2011 - 11:06 PM

Anyone know the json url for returning active auctions, ala https://us.battle.ne.../horde/auctions ?

(I can scrap that by parsing the xhtml, but it's always logged into the same character -- whatever I choose on battle.net, not what I logged in as -- i.e. see http://elitistjerks....p2/#post1828240 above for the issue of switching characters with the new armory login)


There is no JSON URL. You must parse XML or HTML to get active auctions.

#45 Riddance

Riddance

    Glass Joe

  • Members
  • 2 posts

Posted 01 March 2011 - 08:31 AM

Hello,

...
Does anyone have any idea what I'm doing wrong?

Thank you in advance for help.


Seems you are also working on a C# implementation ;)

I had the same problem and the problem was the non existing User-Agent Setting in the Get-Request.

So just set
Request.UserAgent = "Dummievalue";

after
Request.Method = "GET";

and you are getting your result. Back to programming class 1 ;)

#46 Vajel

Vajel

    Glass Joe

  • Members
  • 2 posts

Posted 09 March 2011 - 09:57 AM

Any idea how to get from Battle.net character's achievement date and time insteed date only?

#47 ccKep

ccKep

    Glass Joe

  • Members
  • 6 posts

Posted 24 March 2011 - 02:22 PM

Any idea how to get from Battle.net character's achievement date and time insteed date only?


You can still read the old armory pages for that, just append the rhtml var to the variable.

For example:
http://eu.wowarmory.com/character-achievements.xml?r=Blackhand&cn=shuzzle[B]&rhtml[/B]

Your browser will still show the "New armory, this way ->" page, but the document contains the XML data you're looking for, just take a look at the source.

#48 luggebanane

luggebanane

    Glass Joe

  • Members
  • 1 posts

Posted 17 April 2011 - 02:25 PM

Hi,
here you can find my examples to Access the act. Armory with PHP or JS (JSON).

PHP Projects - Armory API

@Vajel
look in source/class/php/Class.Armory.php for function GetCharAchivement()
that could maybe solve your Problem

AchDate = $xpath->query('//span[@class=date"]');

#49 tashin

tashin

    Glass Joe

  • Members
  • 1 posts

Posted 19 April 2011 - 08:15 AM

Hey Will, looks like we visit the same forums... ^^


@ Topic:

I'm currently coding an open source project in PHP on code.google.com that plugs into the new API.

> wow-data-access - World of Warcraft Data Access Library - Google Project Hosting


Short introduction:

Currently, due to the lack of released API functions on Blizzards behalf, I'm only providing realm status and icons.

The module is designed around a core object that uses one or more "data channels" to load and save (aka cache) data. The most typical setup would be a primary MySQL channel and the BattleNet as a secondary channel.

This means that a call like this one:
$icon = WowDataAccess::obj()->openIcon(array('icon' => 'inv_chest_leather_raiddruid_i_01'));

... will fetch the the icon from the Battle.net website if it is not cached in the MySQL database and save it into the database.

You could use this Object then to output the image:
echo(
'<img '.
'src="data:image/jpeg;base64,'.base64_encode($icon->get('icon')).'" '.
'alt="'.$icon->get('key').'" '.
'title="'.$icon->get('key').'" '.
'/>'
);


#50 freddk

freddk

    Glass Joe

  • Members
  • 1 posts

Posted 09 August 2011 - 04:37 PM

This should get you started:

browse: GET https://us.battle.net/wow/en/vault/character/auction/horde/
bid_auction: POST https://us.battle.net/wow/en/vault/character/auction/horde/bid
cancel_auction: POST https://us.battle.net/wow/en/vault/character/auction/horde/cancel
create_auction: POST https://us.battle.net/wow/en/vault/character/auction/horde/createAuction
deposit (ticket request): POST https://us.battle.net/wow/en/vault/character/auction/horde/deposit
money: POST https://us.battle.net/wow/en/vault/character/auction/horde/money
my_auctions: GET https://us.battle.net/wow/en/vault/character/auction/horde/auctions
my_bids: GET https://us.battle.net/wow/en/vault/character/auction/horde/bids

Keep in mind that the "horde" portion of the URL is dynamic and can be "horde", "alliance", or "neutral".

So far I've found that the bid, cancel, create, deposit (ticket request) and money requests all return native JSON.

I still haven't found a way to get the browse, "my auctions", or "my bids" requests to return anything but HTML.

Also, upon looking at this I got the idea that since all of the POST requests returned JSON that maybe any POST request would, but sending POST instead of GET to the browse URL still results in HTML.


Which the recently changes using the old wowarmory urls is no longer possible. Has anyone had any luck posting,buying or cancelling an auction? I have managed to login but get status 500 (internal server error) when trying to create an auction using the link above. I have the xtoken and xstoken cookies but havent created a ticket.

#51 dlmac

dlmac

    Glass Joe

  • Members
  • 1 posts

Posted 14 August 2011 - 12:59 PM

I think if you are getting a status 500 error then you are setting the parameters incorrectly. I don't use those commands but I am having issues switching characters. It seems bliz has hidden this mid of functionality very well. All command I send for switching a characters just return with status 500 or 404.

#52 Vajel

Vajel

    Glass Joe

  • Members
  • 2 posts

Posted 22 October 2011 - 09:36 PM

Ok, now i'm ready to show you worldofranks.com

Enjoy :)




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users