Hi all, I am working on something akin to a package manager for NWN CC, which would allow builders to easily import individual bits of CC (think e.g. single placeable) without having to manually manage their haks or using bloated compilations like CEP.
The full proposal is listed at nwccc/proposal.md at master · mtijanic/nwccc · GitHub and the initial prototype is at GitHub - mtijanic/nwccc: Neverwinter Custom Content Compiler
I’d very much appreciate any feedback you might have. Thanks!
From my experience while working on the “bloated Compilation” CEP 
I can say, that there is no unified format of the publications of CCC in any way.
I guess it would be a great help for many builders (like me, heh …
) to have a tool to import tidbits of this Compilation (or Project Q) into own haks. CEP as source should be a bit easier than CCC.
Which is one of the reasons for not doing this proposed tool. So how many copies of the most popular models/textures will end up on peoples hard drives? Anyone remember back in the pre-EE days when loads of persistent worlds all had the CEP in their haks?
Surely a better solution to this perceived problem would be for someone to take on the task of converting NIT to other operating systems (currently windows only). It works and works well without the need for new tool.
TR
I like the idea and believe that it has a lot of positive application for NWN. To address @Tarot_Redhand concern about multiple copies of content:
Wonder if its possible to make a new manifest file type that points to all the content stored in a repository in the user install (nwsync folder?) and builds the HAK when the module is loaded, deleting the HAK when the module is closed.
(sorry for the late replies, unexpected lack of free time)
If I understand @Tarot_Redhand’s concern, it is that using this tool would result in multiple copies of the same files (in different haks) on players’ machines. This is actually already a solved problem with nwsync. In the PW world, no one worries about this anymore as the files are deduplicated regardless of how they’re packaged by a particular PW. You only ever keep one copy of a file in your local nwsync database, even if you play on 100 PWs that use it as part of 100 different haks.
The SP world seems to still be stuck in the HAK distribution approach, but surely the better solution here is to just stop doing that and distribute new SP modules through nwsync too? We certainly have the technology. The game already offers some of those as part of the curated modules screen, and nwvault has the infra to host additional community-curated mods that would also be available for download from within the game: https://sync.neverwintervault.org/
Surely a better solution to this perceived problem would be for someone to take on the task of converting NIT to other operating systems (currently windows only). It works and works well without the need for new tool.
I think there’s a bit of confusion about who the tools are meant to serve. NIT is an awesome tool for players to download and play NWN modules. No player would ever touch NWCCC though, it is meant primarily for builders. They do completely different things.
NWCCC offers the builder an easy way to integrate CC into their module, without having to merge 2DA files or relying on someone else’s generic compilation with all its quirks and faults. A player would not see any difference if the module was built with NWCCC or manually.
Wonder if its possible to make a new manifest file type that points to all the content stored in a repository in the user install (nwsync folder?) and builds the HAK when the module is loaded, deleting the HAK when the module is closed.
Now, this sounds like a good feature for NIT to have, regardless of NWCCC. Even today, players might have all of the content in nwsync from playing on a PW, but if a module uses HAKs they’d need to download a second (or third, tenth) copy just to play it. NIT (or another such tool) could build these haks just-in-time, all the while keeping the primary source of resource files in nwsync.
Bear with me here. I’m a poor player - and possibly just as bad at building 
I’m torn between thinking this sounds really good - and not understanding the work flow. Am I right that :
NIT automatically downloads all the files you need to play a module ( I know this works already as I’ve used it on asset heavy modules) As a player solely I have no need of this new system?
NWCC allows builders to select individual files to add to their modules rather than relying on bundled up haks? If I, as a builder want to add something that requires an updated 2da it somehow happens automatically? (I’m thinking that’s unlikely?)
How does your program avoid the “generic compilation quirks and faults” you talk about?
Under NWCC would the creator of a hak still get credit for their work in their number of downloads on the Vault?
I just can’t visualise the workflow here. How about if we take a simple example with a module I am building that currently requires an extra hak containing 5/6 files plus a 2da update. The required files are on a project page on the Vault. Currently, I would download hak X and add it through custom content. I may not need all the the files in the hak but hey that’s okay as the one I need will work. However, I would have to manually add a line to a 2da quite often.
Where would the NWCC program reside, at what point would I open it and say I want to use hak X from the Vault page? Does it fetch it and add it automatically to Custom content in the toolset or does it just download it for me?
Or am I completely confused here? A step by step would be handy for those of us afflicted with an ability to visualise new systems and an inability to understand programming !
Hey, great questions!
- NIT automatically downloads all the files you need to play a module ( I know this works already as I’ve used it on asset heavy modules) As a player solely I have no need of this new system?
Correct! If you never open the toolset, you never need to know that NWCCC exists. Just use NIT, or the ingame “Curated Content” tab, or play on PWs as usual.
- NWCC allows builders to select individual files to add to their modules rather than relying on bundled up haks? If I, as a builder want to add something that requires an updated 2da it somehow happens automatically? (I’m thinking that’s unlikely?)
Correct! If you’re a builder, and you’re using CEP, but you really want this one particular statue that is not in CEP, you’d need to download the second hak with the statue, then manually merge placeables.2da and put them in your third mymod_top.hak file. With this tool, you’d just double-click on cool_statue.nwc file and it would automatically integrate it into your module and update your 2DA.
- How does your program avoid the “generic compilation quirks and faults” you talk about?
The idea is that with NWCCC you wouldn’t need to use e.g. CEP/projectQ if all you want is just some placeable and creature models. Adding CEP/pQ to your module has a lot of side effects (IIRC: It adds new player races, it adds more heads/armor parts that might not fit your lore, it replaces some models with different ones); it comes with a lot of bells and whistles which you may not want, but the process to disable them is not always simple.
The main reasons people opted to use these compilations are:
(A) Many players already have it downloaded, so it was simpler for them to play your mod
(B) It was easier to deal with a bit of bloat than to compile your own compilation from scratch
What I’m saying is that with NWCCC, (B) is no longer a factor, because the tool can automate the hard parts for you. All you need to do is browse and pick the bits you want. And most importantly, you can do it iteratively - build with what you have, then when you need a cool statue, add it quickly, then continue building.
- Under NWCC would the creator of a hak still get credit for their work in their number of downloads on the Vault?
No, because the files are not downloaded from the vault as such. Instead, NWCCC produces a credits.txt file that lists exactly which assets are used in a module, who authored them and under which license they are available. This file would be included in the module download, and the module builder would pick a good way to display this. Some options would be:
To be clear, the goal is not to solve the problem of someone stealing credit. The goal is to make it easy for the well-behaved builder that wants to display credits and honor the license term to do it in a simple, straightforward manner.
Now, for the step-by-step workflow: You are building a fresh module, no haks yet. In the first area you are building, you realize a placeable that pstemarie made would fit really well.
psm_cool_statue.nwc file somewhere; it will likely be available from several places:statues.zip archive on the vault that includes it.nwc file in the NWCCC tools. This will:placeables.2da to also add the line needed to use the statuecredits.txt file with author/license info for the new statue.You can now immediately use your statue in the toolset. Bit later, you realize you could really use a corpse model someone published in one of the CCCs. You repeat steps (2) and (3) to import it. You might find that TR maintains a project page containing .nwc files corresponding to every CCC ever, organized by month. You browse that to find the .nwc file you need, double-click it, and it automatically imports the corpse plc into your module.
The important thing to note is that the .nwc file simply describes the content, doesn’t actually contain it. It is perfectly okay - even desirable - for others to copy and re-publish the .nwc files, as ‘curators’. You would then have GUI tools that know about these curated lists, and you would browse individual placeables through those (looking at screenshots), importing the ones you cared about. I know the command line of nwn_ccc.exe from the proposal looks scary, but it is a protocol and a building block. The actual tool you’d use would be more like File Explorer browsing a bunch of images, and each image has an “add to hak” button you can click.
Well, this was a wall of text, but I hope it clarified some points. Please ask more directed questions if not 
Wow! Thanks for the speedy and fulsome reply!
This sounds pretty good . Disclaimer here: I probably won’t be able to use it as I use a Mac with Crossover for the toolset while playing the game itself natively. Although, if all it needs is changing of paths to files, who knows? Crossover is getting better all the time.
A couple of clarifications if you don’t mind.
I’m assuming it will be set to use the first line which is both empty and not allocated to some other upcoming development? This is a biggie if I never have to go near another 2da !
So, the assets still go into the hak folder rather than , say, override which is good. I create a hak to contain all extra assets then enter its name and the program automatically puts it inside that hak, so in essence each built module would only have a single hak? Seems much easier.
Would the program do this for a hak on my desktop which is then added to the module or from within Edit/Custom content inside the toolset?
Thanks again for the detail. Colour me intrigued . . . ! 
as I use a Mac
FYI, the low level nwn_ccc.exe runs natively on Mac and Linux too (in fact, it’s developed exclusively on Mac and Linux, still completely untested on Windows!). I just use the .exe suffix in the documentation to make it clear it’s an executable program, since that’s what most people are familiar with. The GUI integrations are a different story, and I can’t make any promises there just yet.
I’m assuming it will be set to use the first line which is both empty and not allocated to some other upcoming development? This is a biggie if I never have to go near another 2da !
It uses the first completely empty 2DA, i.e. one where LABEL column is ****. Which means it skips all those XXX_RESERVED lines many 2DAs have.
So, the assets still go into the hak folder rather than , say, override which is good. I create a hak to contain all extra assets then enter its name and the program automatically puts it inside that hak, so in essence each built module would only have a single hak? Seems much easier.
Yes, although if you wanted to, you could break it into multiple haks.
Would the program do this for a hak on my desktop which is then added to the module or from within Edit/Custom content inside the toolset?
In the current design, you first take an empty hak and assign it to the module via Edit/Custom Content, and then the program populates the hak. It’s actually super easy to also assign the hak to the mod outside of the toolset, so it could do that too, if I can figure out a good way to let users control such behavior.
No,putting empty hak inside the toolset is super easy - no need for desktop.
Yay !!! If you need anyone to test anything on a Mac (using Crossover for toolset) just let me know.
You need to make that clear in your proposal on github. As it is now, it sounds like a competitor to NIT.
Bear with this next one. I never play any mp/pw so have no experience of nwsync. Where is the original hak kept, on the player’s machine or as part of the module’s project page or elsewhere?
TR
I would love to try something like this, as I am working on a large-ish SP module and I am concerned about hak bloat. I always worry about screwing up the 2DAs too, so this might make my life easier. I am a Windows user (or will be once I finish moving) and completely new to nwsync, so if you need a LCD tester, I might be it.
The PW/nwsync workflow looks like this currently:
{filename, hash} pairs that are needed to play on the server.When a you connect to a server, you download the manifest to see the exact list of files you need. Then, the game checks for which files you are missing and only downloads those; files you already have are not downloaded. As a player, you never see how the PW organized stuff in their haks, because you are really only dealing with individual files in nwsync.
There are advanced tools to reconstruct PW’s haks from nwsync, but a player never needs those. You only need the haks if you are building the module, and the PWs usually distribute them to their builders directly.
In case of an SP module published by nwsync (e.g. stuff in Curated Content game screen), it works very similarly. Only the original builder actually deals with hak files. And if another builder wants to edit someone else’s module, they’d need to use one of those ‘advanced tools’ to unpack it out of nwsync into .mod+.hak, so they can open it in the toolset, because the toolset does not understand nwsync.
As an illustration I use CEP in the next question as it is almost universally on peoples machines (the major exceptions would be newbies machines). So what happens if I use your program to create a hak and it grabs stuff from CEP? Does your system check on the users machine to see if they already have that content because they already have CEP? What about less used haks such as those from the CTP or More Doors?
TR
NWCCC does not run on end user machines, it’s run by builders. If builders already have CEP installed and want to pull some files out of it, by default nwn_ccc.exe will only scan stuff downloaded via nwsync and won’t scan all your haks, so the builder might need to re-download. There’s alternatively an option to scan locally, but maybe not simple to use.
When this builder then makes a new hak that contains bits of CEP, what the players do depends entirely on how the mod is published. If published through nwsync, players only download the bits they were missing, from ingame.
If published as mod+hak on the vault, players need to re-download the whole hak.
Does this clear things up a bit?