4.3.4 extractors - I figured out why they're "broken"

I will link to this post in my bug report since the presentation here is nicer than Github…

As you’ve probably guessed from the “HowTo Patch” threads that the extractors seem to give mixed results, specifically where Map.dbc is sometimes zero bytes.

After debugging mapextractor.exe I noticed that it walks up the patch chain and extracts the “most recent” version of a DBC. What it doesn’t take into account for is that some patches are incremental patches.

Short Version:

Some DBC files are incremental with a header “PTCH” causing them to fail extraction and overwrite the valid ones with a 0-byte file.

vmapextractor.exe fails to load a crap-ton of “ADT” files because of “err=2” which is “File not found” and using Ladik’s MPQ editor you see they actually don’t exist.

Detailed Version:

[SPOILER]mapextractor.exe

If you search for Map.dbc in Ladik’s MPQ editor, you find this informaion:

[ul][li]locale-enUS.mpq[/li]
[li]wow-update-enUS-13914.mpq[/li]
Attributes: C-S-P

[li]Properties: Compressed, Incremental, Single Unit[/li]
[/ul]
[li]wow-update-enUS-14333.mpq[/li]
[ul][li]Attributes: C-S-P[/li]
[li]Properties: Compressed, Incremental, Single Unit[/li]
[/ul]
[li]wow-update-enUS-14946.mpq[/li]
[ul][li]Attributes: C-S-P[/li]
[li]Properties: Compressed, Incremental, Single Unit[/li]
[/ul]

Attributes: C----
Properties: Compressed
So in this case, the valid Map.dbc is located in locale-enUS.mpq however the extractor uses the one in wow-update-enUS-14946.mpq because it’s newer but since it’s an incremental file the extraction fails:

[ul][li]The valid DBC has the header “WDBC”[/li]
[li]The incremental DBC has a header “PTCH”[/li]
[/ul]

vmapextractor.exe

This fails to load a crap-ton of “ADT” files because of “err=2” which is “File not found”.

One example is: “World\Maps\Kalimdor\Kalimdor_33_59_obj0.adt” which actually doesn’t exist, that particular sequence only goes up to “Kalimdor_33_55”. Where does the extractor get the list of files to open? Is it read from each WMO in the “buildings” folder? If so, why is it looking for so many things that don’t exist? Couple possibilities:

[ul][li]It doesn’t matter, we can still use the WMOs[/li]
[li]The missing ADT files are part of the streaming service that Blizzard implemented[/li]

[/ul]
zone data is streamed to your client when you enter a new zone (so we can’t ever get them now)

[/SPOILER]

Fix (partial):

When extracting DBC files, the extractors should only extract a higher version if it doesn’t have a “PTCH” header. If it does, it needs to fall back until it finds the one with the “WDBC” header.

Function ReadMpqFileSingleUnit() has this comment:


		 // If the file is an incremental patch, the size of compressed data

		 // is determined as pFileEntry->dwCmpSize - sizeof(TPatchInfo)

but the logic doesn’t seem to work properly. If I remove all the incremental patches and run the extractor I get a vaild DBC. If I put them back and run it I get a 0-byte DBC.

In order to get proper extracted DBCs and maps on a 17 GB or 23 GB client, I had to remove the following:

[SPOILER][ul][li]\Data[/li]
[li]\Data\enUS[/li]
wow-update-enUS-13914.MPQ

[li]wow-update-enUS-14007.MPQ[/li]
[li]wow-update-enUS-14333.MPQ[/li]
[li]wow-update-enUS-14480.MPQ[/li]
[li]wow-update-enUS-14545.MPQ[/li]
[li]wow-update-enUS-14946.MPQ[/li]
[li]wow-update-enUS-15005.MPQ[/li]
[li]wow-update-enUS-15050.MPQ[/li]
[/ul]

wow-update-13164.MPQ
wow-update-13205.MPQ
wow-update-13287.MPQ
wow-update-13329.MPQ
wow-update-13596.MPQ
wow-update-13623.MPQ
wow-update-base-13914.MPQ
wow-update-base-14007.MPQ
wow-update-base-14333.MPQ
wow-update-base-14480.MPQ
wow-update-base-14545.MPQ
wow-update-base-14946.MPQ
wow-update-base-15005.MPQ
wow-update-base-15050.MPQ

[/SPOILER]

Summary

Apparently the map extractor doesn’t properly handle incremental DBC files with a “PTCH” header and vmapextractor is using a list of stuff to extract that doesn’t exist…

Both extractors handle incremental patches just fine thanks to StormLib (SFileOpenPatchArchive)

About MPQ files with numbers lower than 15211 - correct, they need to be removed; patch 4.3.2 did that

And finally err=2 - wow maps are by default squares of 64x64 grids where not all grids exist. There is no way of knowing that without attempting to extract them (the extractor runs 2x for (i to 64) instead of reading file list because its simpler). That is why you can ignore those “errors”

@Shauren

I think incremental MPQs is different than incremental DBCs. The stormlib can open an incremental MPQ but the extractors can’t extract a DBC marked as incremental, they’re two different things.

When I extract “Map.dbc” (which is labeled as “incremental” with a header of “PTCH”) from an incremental MPQ using Ladik’s MPQ editor, I get a file that’s a few MB in size (albeit useless because of the bad header). When the extractors do it, the file is zero bytes. They might be able to open an incremental MPQ but clearly they can’t extract the DBCs properly.

Thanks for explaining the “err=2” bit though. I was concerned because I knew Blizzard had moved to a streaming setup so I didn’t know if I had incomplete files.

Not quite. The api I posted (http://www.zezula.net/en/mpq/stormlib/sfileopenpatcharchive.html) opens the archive in patch mode - that means it expects files with PTCH header inside. Now, those files can have two patching modes: BSD0 and COPY. First is the binary diff patch type (incremental) second instructs that the file should just be extracted overriding what was done before. I’m saying this because MPQ files not opened with SFileOpenPatchArchive but containing PTCH files will use the old method which is always COPY.

@Shauren

Hmm, well I understand the explanation but I really can’t wrap my head around why the resulting file is 0 bytes. Even before it is extracted (valid header or not), the viewer shows it has a size > 0 bytes.

Anyway, I saw you updated the tools so this should be moot now. Thanks for the commit.