Jump to content


Armory Scraping


  • Please log in to reply
51 replies to this topic

#1 Guest_Abbi_*

Guest_Abbi_*
  • Guests

Posted 23 October 2008 - 02:37 PM

Has anyone figured out how to scrape the XML for Armory calendars yet? Using Firefox as the user-agent and going through the login process works fine, but once I get to the actual calendar page (https://www.wowarmor...hen Ring&n=Abbi, e.g.) I get a very barebones XML page with only my basic character information on it.

Examining the XSLT stylesheet leads me to think that http://www.wowarmory.../js/calendar.js has the meat of the calendar stuff, but my JavaScript is not good enough to figure out how to get event data. Given event data, I'd be writing perl to generate .ICS files for iCal.

#2 xergio

xergio

    Glass Joe

  • Members
  • 2 posts

Posted 23 October 2008 - 03:29 PM

The calendar use JSON instead XML to retrieve the information. You can use Firebug and inspect the Net tab.

#3 codeguy

codeguy

    Glass Joe

  • Members
  • 1 posts

Posted 14 March 2009 - 12:26 AM

I'm wondering if you're still working on a project of this sort? I have the same desire, to pull armory data and store locally. Over the past couple days I have been working on something to do this, and it's near complete. The only problem with the armory calendar data is that you need to login. This is not a problem to program, however it will deter many users as they will need to supply their login information to the script. I imagine the only users that would be comfortable in doing so would be those that can execute this locally and have complete control over that information, as to automate it the script needs a way to retrieve the login information without it being supplied each execution. This works very well for me as I host my own Solaris 10 box, I know the login information isn't going to be seen by anyone but me, and if it is- well then I have much larger problems than a compromised wow account.

To make a long post short, I wanted to share some useful information regarding access this data via the armory mechanically.

Rather than create some fancy perl code to connect to the armory to download the calendar, I choose to use wget. An initial wget call logs into the armory and stores a cookie on my server, the subsequent calls pass the stored cookie. The actual calendar call is as simple as:

/usr/sfw/bin/wget --keep-session-cookies --user-agent "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1" --load-cookies="/vdsk/projects/wow/armory/tmp/calcookie.tmp" --output-document "/dsk/projects/wow/armory/tmp/calholidayWeekly.tmp" "http://www.wowarmory.com/vault/calendar/month-world.json?type=holidayWeekly&month=3&callback=calendar&year=2009&loc=1&loginType=com" > /dev/null 2>&1


As an example of the calendar data returned from the armory, below is the Darkmoon schedule for March 2009.

calendar({"now":1236989274033,"month":3,"events":[{"summary":"Darkmoon Faire","calendarType":"darkmoon","start":1236596400033,"calendarTypeId":6,"end":1237201200033,"icon":"Calendar_DarkmoonFaireElwynn","description":"The Darkmoon Faire is here, this time in idyllic Elwynn Forest.\r\n\r\nMeet Silas Darkmoon and his troupe, play games that test mind and nerve, and behold exotic sights from the four corners of Azeroth... and beyond!","priority":1}],"calendarType":"darkmoon","year":2009,"tz":-25200000});

Obviously the script would need to parse through this information to produce something meaningful, but again that's not a problem at all.

You can also craft messages to the armory site to get the detail list from the high level calendar entry, and even accept or decline the invitation. I, however, don't see a lot of use in accepting or declining mechanically, some user decision would need to be made, and personally I make the decision in game. My script will simply be used to download calendar activity and send email notifications, to me, when an event as been added, removed or is about to occur.

#4 Stauf

Stauf

    Glass Joe

  • Members
  • 1 posts

Posted 23 April 2009 - 04:14 AM

I've been able to authenticate to Battle.Net and scrape calendar data (turning it into iCalendar data for iCal / Google Cal) -- Armory Calendar to ICS

The issue is, at the moment to do it, I need to collect Battle.Net login info. The code is at http://aftermath.cx/...ls/Calendar.zip if you want to try it yourself (requires PHP with JSON support, that cookies.tmp be writable, and the HTTP auth may not work on your server (hack to make it work with php-cgi))

#5 Slackie

Slackie

    Bald Bull

  •  Patrons
  • 2,003 posts

Posted 20 May 2010 - 12:07 PM

(thread necro)

Blizzard has released an API that allows access to the auction house. Here is some technical data to aid in writing things to make use of this functionality.

Get JSON results: http://www.wowarmory...use/search.json
Get XML results: http://www.wowarmory...earch/index.xml (be sure to include rhtml=false or you will get HTML data back!)

note: There is a maximum of 200 results returned per query.

Search parameters you can use:

[table=head]Key|Description|Values
filterId|maps to ah categories|(see below)
maxLvl|maximum level required|any valid character level
minLvl|minimum level required|any valid character level
qual|item quality|0=poor, 1=common, 2=uncommon, 3=rare, 4=epic
reverse|reverse the sort output|true, false
sort|column to sort by|rarity, quantity, level, time, buyout

Filter categories:

[table=head]Id|Description
0|Weapon
1|Armor
2|Container
3|Consumable
4|Glyph
5|Trade Goods
6|Projectile
7|Quiver
8|Recipe
9|Gem
10|Miscellaneous

note: You can filter into sub-categories and sub-sub-categories by separating the id's with a comma. filterId=0,78 would show one-handed axes. I haven't figured out how to gather up what all the sub-category id's are short of searching by hand, if you have any ideas, send me a PM.

Data that is returned:

[table=head]Key|Description
auc|Auction ID
bid|Bid price
buy|Buyout price
charges|Charges
date|Date posted
icon|Icon
id|Item ID
ilvl|Item Level
n|Item name
nbid|Next minimum bid
ppuBid|Price per unit (bid)
ppuBuy|Price per unit (buyout)
qual|Item quality
quan|Quantity
req|Required level
seller|Seller
time|Time left (1=short,2=medium,3=long,4=very long)

#6 malthrin

malthrin

    stalemate associate

  • Moderators
  • 9,107 posts

Posted 21 May 2010 - 04:52 PM

All done! If anyone cares to fill in the missing categories, that'd be nice.

[TABLE]FilterId result
0 Weapon
1 Armor
2 Container
3 Consumable
4 Glyph
5 Trade Goods
6 Projectile
7 Quiver (all)
8 Recipe
9 Gem
10 Miscellaneous
11 Quest
12
13 Two-Handed Maces
14 Cooking Recipe
15 Herb Bag
16 Dagger
17 Orange Gem
18 One-Handed Maces
19 Bullets
20 Enchanting Bag
21 Arrows
22 Thrown Weapon
23 One-Handed Swords
24
25 Purple Gem
26 Soul Shard Bag
27 Meta Gem
28 Warlock Glyph
29 Leatherworking Bag
30 Mage Glyph
31 Reagent
32 Trade Goods (Elemental)
33 Fishing Poles
34 Armor (Misc)
35 Fist Weapons
36
37 Inscription Bag
38 Holiday
39 Junk
40
41 Bow
42 Wand
43 Armor (Cloth)
44 Leatherworking Recipe
45 Alchemy Recipe
46 Food & Drink
47 Two-Handed Swords
48
49 Crossbows
50 Blue Gem
51 Jewelcrafting Recipe
52 Yellow Gem
53 Totem
54 Enchanting Recipe
55
56 Druid Glyph
57 Recipe (Book)
58 Container (Bag)
59 First Aid Recipe
60 Idol
61 Staff
62 Misc (Other)
63 Polearm
64 Gun
65 Rogue Glyph
66 Warrior Glyph
67 Shield
68 Ammo Pouch
69 Mount
70 Blacksmithing Recipe
71 Death Knight Glyph
72
73 Red Gem
74 Libram
75 Tailoring Recipe
76 Quiver
77 Armor (Leather)
78 One-Handed Axes
79
80 Paladin Glyph
81 Quest
82 Two-Handed Axes
83 Armor (Mail)
84 Priest Glyph
85
86
87 Armor (Plate)
88 Hunter Glyph
89 Prismatic Gem
90 Trade Goods (Enchanting)
91 Pet
92 Misc. Weapons
93 Shaman Glyph
94 Engineering Recipe
95 Mining Bag
96 Green Gem
97 Jewelcrafting Bag
98 Simple Gem
99 Cloth
100 Potion
101 Leather
102 Elixir
103 Metal & Stone
104 Flask
105 Meat
106 Bandage
107 Herb
108 Item Enhancement
109 Scroll
110 Consumable (Other)
111 Jewelcrafting
112 Inscription
113 Devices
114 Explosives
115 Trade Goods (Materials)
116 Trade Goods (Other)
117 Armor Enchantment
118 Weapon Enchantment
119 Armor (Misc Head)
120 Armor (Neck)
121 Armor (Shirt)
122 Armor (Ring)
123 Armor (Trinket)
124 Armor (Offhand)
125 Armor (Cloth Head)
126 Armor (Cloth Shoulder)
127 Armor (Cloth Chest)
128 Armor (Cloth Waist)
129 Armor (Cloth Legs)
130 Armor (Cloth Feet)
131 Armor (Cloth Wrists)
132 Armor (Cloth Hands)
133 Armor (Cloak)
134 Armor (Leather Head)
135 Armor (Leather Shoulder)
136 Armor (Leather Chest)
137 Armor (Leather Waist)
138 Armor (Leather Legs)
139 Armor (Leather Feet)
140 Armor (Leather Wrists)
141 Armor (Leather Hands)
142 Armor (Mail Head)
143 Armor (Mail Shoulder)
144 Armor (Mail Chest)
145 Armor (Mail Waist)
146 Armor (Mail Legs)
147 Armor (Mail Feet)
148 Armor (Mail Wrists)
149 Armor (Mail Hands)
150 Armor (Plate Head)
151 Armor (Plate Shoulder)
152 Armor (Plate Chest)
153 Armor (Plate Waist)
154 Armor (Plate Legs)
155 Armor (Plate Feet)
156 Armor (Plate Wrists)
157 Armor (Plate Hands)[/TABLE]
Lampkin in EJBSG 28 | Anders in EJBSG 24 | Cavil in EJBSG 20
Boomer in EJBSG 19 | Roslin in EJBSG 17 | Roslin in EJBSG 13 | Roslin in EJBSG 8
MTG Online draft viewer

#7 Antiarc

Antiarc

    Still alive

  • Members
  • 1,444 posts

Posted 21 May 2010 - 05:37 PM

I've discovered that by using a Keep-Alive header on my connections, I can pull data at approximately one request per 0.55 sec, rather than waiting that 0.55 sec + 1 sec per HTTP connection. If you can reuse connections in conjunction with keepalives, it'll speed you up a lot.
Need a Mumble server? I run MMO-Mumble for all your voice chat needs. | My rogue planning tool: Shadowcraft

#8 thefool808

thefool808

    Von Kaiser

  • Members
  • 40 posts

Posted 21 May 2010 - 08:00 PM

Here's a quick and dirty ruby implementation to get people started. You can get the cookie string straight from your cookies, or use firebug to look at the headers of a normal request after logging into the Armory.

#971617 - Pastie

#9 Antiarc

Antiarc

    Still alive

  • Members
  • 1,444 posts

Posted 21 May 2010 - 08:37 PM

I've got a Ruby project up and running already, actually. It uses Mechanize with keepalives to optimize throughput, and stores data in MongoDB for fast analysis and reporting.

http://github.com/cheald/AuctionBoss/
Need a Mumble server? I run MMO-Mumble for all your voice chat needs. | My rogue planning tool: Shadowcraft

#10 malthrin

malthrin

    stalemate associate

  • Moderators
  • 9,107 posts

Posted 22 May 2010 - 01:18 AM

Another search parameter, f, specifies which auction house to query.

Value|Description
0|Alliance AH
1|Horde AH
2|Neutral AH


There are other valid values - probably corresponding to the various in-game auction house NPCs? But this is all you need.


edit:
Tested these sorting parameters from the javascript: 'sort' and 'reverse'. Reverse is a bool for ascending/descending. Sort can have the following values:
Value|Description
rarity|epics first
name|alphabetical
level|required level, highest first
ilvl|item level, highest first
bid|stack bid, lowest first
buyout|stack buyout, lowest first
unitbid|price per unit, lowest first
unitbuyout|price per unit, lowest first
time|time remaining, shortest first

Lampkin in EJBSG 28 | Anders in EJBSG 24 | Cavil in EJBSG 20
Boomer in EJBSG 19 | Roslin in EJBSG 17 | Roslin in EJBSG 13 | Roslin in EJBSG 8
MTG Online draft viewer

#11 malthrin

malthrin

    stalemate associate

  • Moderators
  • 9,107 posts

Posted 22 May 2010 - 04:43 PM

Bids and buyouts are done via bid.json. The parameters are:
Parameter|Description
auc|AuctionId
money|Amount to bid
f|AH to buy from

If your bid amount is less than the buyout, you place a bid on the item. If it is equal to the buyout, you buy out the item. If it's more than the buyout... someone else can test what happens then!

Here's the relevant javascript:
http://www.wowarmory...tion/auction.js
Lampkin in EJBSG 28 | Anders in EJBSG 24 | Cavil in EJBSG 20
Boomer in EJBSG 19 | Roslin in EJBSG 17 | Roslin in EJBSG 13 | Roslin in EJBSG 8
MTG Online draft viewer

#12 malthrin

malthrin

    stalemate associate

  • Moderators
  • 9,107 posts

Posted 22 May 2010 - 05:32 PM

Some other functions:
http://www.wowarmory.../inventory.json
http://www.wowarmory...house/mail.json
http://www.wowarmory...ouse/money.json
http://www.wowarmory...se/faction.json This one can get the value for f. Guess f stands for faction!
Lampkin in EJBSG 28 | Anders in EJBSG 24 | Cavil in EJBSG 20
Boomer in EJBSG 19 | Roslin in EJBSG 17 | Roslin in EJBSG 13 | Roslin in EJBSG 8
MTG Online draft viewer

#13 luvmachine

luvmachine

    Glass Joe

  • Members
  • 1 posts

Posted 24 May 2010 - 01:54 AM

Using bid.json and bidding for more than the buyout results in a buyout of the item for the listed price, no extra money is spent.

mail.json is used to list the mail in your box while takeMail.json is used to move it from your mailbox to your inventory.

Any help on getting the search results in XML, I can only seem to find them in JSON.

EDIT: Also, changing 'pageSize' in the search parameters will give you more results (up to 50), and changing the 'start' parameter can be used to get further down the list after what the initial results show.

#14 Slackie

Slackie

    Bald Bull

  •  Patrons
  • 2,003 posts

Posted 24 May 2010 - 04:16 AM

Any help on getting the search results in XML, I can only seem to find them in JSON.


Check a few posts up for another one by me. It has the XML URL at the top.

How to post a new auction:

Step one: You need a "ticket", you get this by sending a GET request to http://www.wowarmory...se/deposit.json

Options:

[table=head]Key|Description
id|Item ID
duration|Length of auction (0=12h, 1=24h, 2=48h)
quan|Number of items
stacks|Number of stacks

You will get a response that contains a few variables:

[table=head]Key|Description
ticket|Ticket
deposit|Cost to create this auction (per item)
totalDeposit|Cost to create this auction (overall)
suggestedPrice|Unknown

Step two: You use the ticket you have from step one and send a GET request to http://www.wowarmory...use/create.json

Options:

[table=head]Key|Description
bid|Bid price for this item
buyout|Buyout price for this item
id|Item ID
duration|Length of auction (0=12h, 1=24h, 2=48h)
quan|Number of items
stacks|Number of stacks
ticket|Ticket you got from deposit.json
guid|This will be blank for a single-stack auction

note: If you are creating a multi-stack auction, create.json will return data in the guid field. Use the data from the guid field to populate the ticket field of your next request.

#15 Slackie

Slackie

    Bald Bull

  •  Patrons
  • 2,003 posts

Posted 02 June 2010 - 04:34 AM

Anyone seeing a large amount of timeouts via the new AH API over the last 36 hours or so? I tried increasing my polling delay but still get errors which leads me to believe this is an actual problem not just polling too quickly.

#16 malthrin

malthrin

    stalemate associate

  • Moderators
  • 9,107 posts

Posted 02 June 2010 - 04:32 PM

It's not the same message as the query throttle. You also run into the same error while browsing the web interface, so I'd wager it's a problem on their end.

edit: looks like it's mostly fixed this afternoon
Lampkin in EJBSG 28 | Anders in EJBSG 24 | Cavil in EJBSG 20
Boomer in EJBSG 19 | Roslin in EJBSG 17 | Roslin in EJBSG 13 | Roslin in EJBSG 8
MTG Online draft viewer

#17 Stack

Stack

    Glass Joe

  • Members
  • 5 posts

Posted 06 June 2010 - 02:30 AM

I was able to get the main summary page http://www.wowarmory...dex.xml#summary into my program as XML but has anyone found the JSON equivalent link, it would be a lot simpler if I could route all the responses from the armory thru JSON.

EDIT: Found it http://www.wowarmory...use/status.json

{"auctionStatus":{"factionId":0,"mailInfo":{"current":2,"max":50,"total":2,"fullPercent":4},"activeHighBids":0,"activeOutBids":0,"auctionsActive":0,"auctionsCancelled":2,"auctionsExpired":0,"auctionsSold":0,"bidsCancelled":0,"bidsWon":0,"goldEarned":0},"command":{"f":0,"cn":"charactor","r":"realm"},"pra":{"status":"active"}}

Another http://www.wowarmory...use/cancel.json

{"success":true,"command":{"auc":901314668,"f":0,"cn":"charactor","r":"realm"},"pra":{"status":"active"}}

Another important one http://www.wowarmory...ect-submit.json

Takes a CN=charactor&r=realm as post data which returns.

{"success":true}


#18 Slackie

Slackie

    Bald Bull

  •  Patrons
  • 2,003 posts

Posted 06 June 2010 - 06:38 AM

Thanks for finding status.json, I hadn't seen that before. Does anyone know of a way to find current transactions remaining without actually posting a new auction or bidding on something? That would be useful.

Here are all the various URLs I know about, just so everyone is on the same page (I didn't add status.json from above yet)

root: "http://www.wowarmory.com/auctionhouse/"
status: "http://www.wowarmory.com/login-status.xml"

# Miscellaneous
money: "http://www.wowarmory.com/auctionhouse/money.json?%s"
alerts: "http://www.wowarmory.com/vault/calendar/alerts-user.json?%s"
inventory: "http://www.wowarmory.com/auctionhouse/inventory.json?%s"

# Calendar
calendar: "http://www.wowarmory.com/vault/calendar/month-user.json?%s"
calendar_world: "http://www.wowarmory.com/vault/calendar/month-world.json?%s"
calendar_details: "http://www.wowarmory.com/vault/calendar/detail.json?%s"

# Mailbox
mailbox: "http://www.wowarmory.com/auctionhouse/mail.json?%s"
mailbox_move: "http://www.wowarmory.com/auctionhouse/takeMail.json?%s"

# Auctionhouse related
auction_post: "http://www.wowarmory.com/auctionhouse/create.json?%s"
auction_browse: "http://www.wowarmory.com/auctionhouse/search.json?pageSize=50&rhtml=false&%s"
auction_active: "http://www.wowarmory.com/auctionhouse/auctions/index.xml?rhtml=false&%s"
auction_cancel: "http://www.wowarmory.com/auctionhouse/cancel.json?%s"
auction_deposit: "http://www.wowarmory.com/auctionhouse/deposit.json?%s"
auction_buy_get: "http://www.wowarmory.com/auctionhouse/bid.json?%s"
auction_buy_post: "http://www.wowarmory.com/auctionhouse/bid.json"


#19 Stack

Stack

    Glass Joe

  • Members
  • 5 posts

Posted 07 June 2010 - 12:59 AM

I've been working with the takeMail.json all day. Based on what I've watched while using the wowamory.com site, ive concluded that you can only claim money remotely and can only see items in your mail box that you could possibly relist.

what is odd is that wowarmory doesn't supply a auc id when it does its call -9223372036854775808 is the value I get when I omit it in my post data, including it doesn't get you mail that is not just money however.

sample rejection
{
  "error": {
    "code": 112,
    "error": true,
    "message": "Mail not found"
  },
  "command": {
    "auc": -9223372036854775808,
    "mailIds": "2119124985",
    "f": 0,
    "cn": "Nullberri",
    "r": "Bonechewer"
  }
}

sample success
{
	"mailIds":[2120517468],
	"money":950000,
	"command":
		{
			"auc":-9223372036854775808,
			"mailIds":"2120517468",
			"f":0,
			"cn":"Nullberri",
			"r":"Bonechewer"
		},
	"pra":{"status":"active"}
}


#20 Psy7th

Psy7th

    Glass Joe

  • Members
  • 11 posts

Posted 14 June 2010 - 12:27 PM

For what it's worth: -9223372036854775808 is really -(2^63) or signed long 0x8000 0000 0000 0000.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users