So, I’ve run a comparison of the CEP core haks using the toolset and have found that there are 100s (if not thousands) of duplicated files. Without a way to dump the readout to a TXT file, the only thing I can do is dump all the core haks into a single directory working from core7 up to core0 and then rebuilding the core haks with the content redistributed among the core haks.
In the end NO files would be missing - only the duplicates would be deleted, BUT the files would be in different places in the core haks. I’d probably sort them the same way that CEP1 did. The HAK name would NOT change.
That won’t be a problem, but there is other way to do it.
extrack all core haks into individual folders
make backups of all of them
start copying stufff from folders by last core hak in order to first using this method:
selecct all files in folder, cut, past into folder of core hak above this one
if the windows reports a duplicates, select ignore so the duplicated files remain in the original folder
once this is done and the folders core2 to 8 or how many are there will only contain duplicates, start cmparing the folders with the backups you created and then select matching files and delete them on both sides
after this is done, remake the core haks from the backup folders which now won’t contain duplicates
Same filenames , different core haks. The files in the upper core haks (those higher up in the module hak list) are overwriting the files in the lower haks - so versioning isn’t really an issue.
There are actually 780 duplicate files.
@Shadooow - I think reorganization is the way to go as its a mess trying to find anything in the core haks as there really isn’t any reason to how they are laid out.
While I’m at it - any reason to keep TGA versions of textures that have a DDS version? I think we’re all beyond the point where GPUs can’t handle DDS as well as TGA (if that was ever a thing).
Reorganization won’t be possible as sorting them by type into the core haks wouldn’t be compatible with 1.69’s 16K limit. If I can’t put all male or female racial parts, creatures, and placeables into there own sorted haks there isn’t a point.
I recently started deleting them from my haks as well. And converting the rest to DDS.
In past android wasn’t able to display DDS but that was fixed. Since EE cannot be run on older OS like XP and older HW I think nobody should ever run into this issue anymore.
Lastly, even though not everyone plays EE, half the CC already offers only DDS so such users would be unable to play already and let us know this is happening. But nobody reported invisible models in a decade so I think there is no harm removing TGAs.
However, I ran into crash issue with iOS version of NWN and DDS. Turns out it can’t handle very small textures converted to DDS, so if you plan to convert TGA to DDS to trim the CEP2 size, only do it with textures larger than 32x32. Beamdog doesn’t care about this one, I wouldn’t expect them to fix it.
The benefit to players is probably minor compared to the risk.
However, if it has to be done for some reason, working from core7 to core0 would indeed ensure that the files actually used by the game remain the same.
I’m finding lots of instances of stuff that should be fixed:
models overwriting identical models lower down in the hak tree
binary models being overwritten by ascii models
textures included in 1.69 from DLA were put into CEP (DLA horses - textures are identical)
neck model 7 (DLA quiver overwritten by neck cloak). I renamed the DLA quiver to 17
TAD included multiple copies of his textures from reforged in multiple places in the core haks
etc.
I could go on, but I think you see what I’m getting at…some stuff, the way the core haks currently are, IS broken and is easily FIXED
The neck model is the only model rename and I’m fairly certain that 17 has never been used by anything. Any PW that may be using that slot will doubtless have their own tophak anyway.
This is probably heresy but… I’d remove the compiled models unless a pw has speed problems, the difference in speed will be negligible. Also for smaller models, you could actually be saving space because the compiled models may well be bigger. Also, depending on what they were compiled with, it is possible that the act of compiling them could have introduced low level bugs into the models. I seem to remember that one compiler introduced a spelling mistake into a certain keyword (can’t remember the keyword involved though).
To consider: DDS format uses “lossy compression” which damages details. It is not suitable as an image editing format. It should load faster, though.
Keeping both in the CEP core takes extra space of course, so to save it, you could ship only DDS but keep their matching TGA in a separate “master” archive.
I had a look into this, there are indeed around 800 duplicates. And there are often differences (different file-sizes), which points out, that there are different versions.
As Proleric states “The benefit to players is probably minor compared to the risk.”
Btw. You assume, that the higher coreX has always the latest version. From what I’ve seen, builders tend to attach the haks has in natural order (Core0 first and Core7 last) which means that the older file has priority.
If I were you, I would just delete the unused, surplus stuff (if you must …), and leave the structure as it is. This way it will be (probably) compatible to existing modules.
The point (for me, at least) is that loading 7 >> 0 should ensure that the version actually used by the game now is preserved (provided builders have the specified hak order).
Talk of using the latest version, un / compiled version etc is scary, because that might change how modules behave, bearing in mind that current practice is to go live without beta testing.
A few assumptions being made here. Hopefully this will clear that up and alleviate any concerns.
[Edited for Clarity] The HAK order for the core haks is in descending order core0 > core1 > core2…etc. The models in upper level core haks (core0 being the highest, then core1, etc.) ARE what is actually being used by CEP.
I’m not just willy-nilly overwriting files here, I’m comparing versions to see which is indeed the “newer” version. So far - at halfway through - it is clear that the files lower in the hak tree are older versions.
Regarding binary vs. ascii models - in all cases I have encountered, the binary model IS the one used by the game.
2.67 will be thoroughly beta tested before going live. 2.66 was released with minor testing because the changes were small and 99% occurred solely in 2da files. The hotfixes were released to address minor issues. None of these issues were “game breaking”.
