wowhead to npc_vendor and npc_trainer perl script for TDB 4.3.4

I keep finding vendors that have no entry in the npc_vendor table, but their creature_template has the flag for vendor. There have even been trainers that seemed to have incomplete lists of what they train. This script will scan the wowhead page (or range of pages) source and generate queries for the data found. In the case of trainers, wowhead didn’t seem to track the level required to learn spells, but all the other fields are there. An NPC that is vendor and trainer will have both queries built in a single pass of the HTML. A comment header is created for each query. A blank line is also added at the end of the values.

Granted, this script is something that I consider a work-in-progress project, but I hadn’t been able to find a similar tool that was already out there. I chose perl as the language to use for portability. Plus, someone with a bit more experience in perl than I have might be able to expand on this to make it better.

Usage information is included as comments in the perl script.

http://pastebin.com/D6fd9bky

Sample output:

-- ------------- --
-- Rohok (16583) --
-- ------------- --

INSERT INTO `npc_vendor` (`entry`, `slot`, `item`, `maxcount`, `incrtime`, `ExtendedCost`, `type`) VALUES
        (16583, 1, 5956, 0, 0, 0, 1), -- Blacksmith Hammer (ilvl 1)
        (16583, 2, 2901, 0, 0, 0, 1), -- Mining Pick (ilvl 4)
        (16583, 3, 2880, 0, 0, 0, 1), -- Weak Flux (ilvl 5)
        (16583, 4, 3466, 0, 0, 0, 1), -- Strong Flux (ilvl 25)
        (16583, 5, 3857, 0, 0, 0, 1), -- Coal (ilvl 30)
        (16583, 6, 18567, 0, 0, 0, 1), -- Elemental Flux (ilvl 60)
        (16583, 7, 23638, 1, 20, 0, 1); -- Plans: Lesser Ward of Shielding (ilvl 68)

One problem is wowhead is not considered a primary source, so, this would only be useful to server admins who don’t care about correct data, another problem is wowhead is 2 patches advanced from master, and 1 patch from the 4.3.4 branch, so…

Ok, so what’s the alternative, then? Are you going to personally log into a retail 4.3.4 server and sniff the “correct” data on every vendor/trainer in the game? Unless you are actually a Blizz employee, I don’t see you being able to get the “correct” data, either. In the meantime, apps and scripts to pull data from “other” sources will be developed and used. There are loads of “fun” repacks that have content that was never in the retail game. While this script may not be useful to you, it very well could be useful to many other people.

Sorry to break it to you but that truly is how it’s been done since TrinityCore started. Pretty much all data is sniffed. A few devs just sniff every time they login so even walking past a mob will sniff the vendor entry and all these sniffs are uploaded in which we can search by creature/gameobject/etc/etc.

Sniffing a MoP server is no more accurate than wowhead. Even with Cata, many vendors contents were changed. An example of that is every vendor that used to sell ammo. If sniffed data is all that was being used, then the TDB for 4.3.4 is absolute proof that sniffing is not absolutely perfect, either.

Don’t get me wrong. I’m not putting the project down in any way. I’m simply saying that denying data based on flawed reasoning is a bad choice. It simply would take a Blizzard employee with the ability to access the genuine server data for the appropriate game version to get 100% accurate data. That is just a fact that this entire community needs to accept.

Besides, based on your own signature, any sort of data is perfectly acceptable for merely testing the project to see if it is working as expected. Since this is not a project for public server use, and it’s merely an academic execise, being critical of data sources is pretty much calling yourself a liar.

we have tons of sniffed data going back to at least TBC… But, I have wished for a long time that the main dev team would concentrate on the framework, and leave the specific data to others.

Additional “sniffed data” fact: merely walking past NPCs has obviously not provided the gossip, vendor, and trainer data. Without adding any 3rd party data, TDB 4.3.4 contains many NPCs with missing gossip, vendor, or trainer data. Too lazy to parse the collected sniffs completely? Or were there no sniffs of the appropriate data available?

There’s an extremely valid reason for utilities such as my script (including a Windows app that I can’t use on a Mac). Until such time as “correct” data is committed, mostly accurate data is still far better than missing and/or incomplete data. People running a server, for whatever reason, can’t see if their “academic toy” is working properly. Besides, I find it hilarious to see vendor trash being sold BY a vendor (ammo, deprecated vials, etc), not to mention the obsolete (item-based) currency that is still handed out. It may not be game breaking in nature, but it’s no more accurate than pulling data from places like wowhead.

As I said previosuly, I’m not intending to put down the project. I would abolutely love to see a pre-MoP server that was exactly like the retail game before the 5.0 patch was pushed. In fact, I’d love to see the last patch of each version (1.x, 2.x, 3.x, and 4.x) done perfectly. In the meantime, slightly wrong data beats no data.

Isn’t it just funny to see all this negativity on this simple script, but a Windows app that does much more (with data from the exact same source that I use) doesn’t have a single comment like I’m seeing? Check out http://www.trinitycore.org/f/topic/6241-wowhead-data-parser/ to see what I mean.

Let me see if I got this straight. A windows app that pulls wowhead data for adding to TDB is posted about in this forum, and you guys say nothing. I post a perl script that does less manipulation of data, but my perl script has 2 of you jumping on it to put it down (i.e. wowhead isn’t a proper source). In that put down, you both make claims of using sniffs, which is the exact same process that wowhead uses for it’s collected data. Exactly where is the difference between your sniffs and wowhead’s sniffed data? Then, I happen to notice that mining obsidium, elementium, and pyrite was giving me WotLK gems during the looting of the ore deposit. That really wasn’t something that was built from sniffs.

Is it just me, or can anyone else see the problem here?

First off, that other one is from a year ago, and I don’t participate in zombie threads unless absolutely necessary, secondly, wowhead doesn’t sniff one goddamn thing, all of their data is collected by addons for the client, addons that make no distinction between official or some crappy fun server, and that can be manipulated by the uploader before uploading it.

Kylroi perhaps you should stop taking this so very personal. We have nothing against you, it’s just the method this project achieves obtaining its data and it has always been this way.

we’re also not trying to stop you from working on this tool, or, really to stop anyone else from using it, I’m just saying, don’t expect this tool to be used to make official changes to data.

Well, thank you for finally being polite with your comments, but let me point out that I never asked for it to be used for official commits. Exactly where did I make even a hint at that, let alone a demand for immediate acceptance? I apologize for my defensiveness, but purely negative comments seemed like an attack on myself and/or my project.

Considering that I primarily use a Mac system to run my client and work with the database, all of the Windows utilities that I keep seeing are effectively useless to me. Based on what you’ve been saying, those programs would have been no better than what I wrote in perl, anyway. At least with perl, my project is easily ported to Windows and Linux (and even other operating systems, if desired).

Oh, and I agree that the core functionality should be the priority of the developers. Forget content and get the actual functionality going. Implement every opcode, spell, or whatever that can be. While I think the “encryption” (was there more than 1 opcode effected?) could be brute force cracked, I suspect that hasn’t been done to avoid legal action. It would be nice to have a patched x64 client to use, but I haven’t even found documented source that patches the x86 client.