A simple vault installer proposal

@Surazal Just some general random remarks about NIT. In NIT when it says Extended Edition are you referring to the Enhanced Edition? Seen a few people using that term. Also have you considering bundling 7z with NIT to make installation of NIT easier? 7z is distributed under the LGPL I believe that allows it to be bundled? Currently there is no way of getting out of that dialog box looking for the 7z.exe. The close button just popups the same dialog again. There should be a way to cancel, abort or postpone this. The same seems to happen for managing saved games. When NIT can’t determine which module a saved game belongs to it pops up a dialog box that doesn’t allow canceling or skipping.

To me NIT is more of a Module Manager than just a Installation Tool. This is what I am thinking for a simplify Installation only front- end based from the discussions above:

  1. When the user launches this simplified front-end (SNIT?) there will be only 2 buttons. a) Install from URL as discussed above. b) Install from file. There is nothing else.
  2. When installation is done. Just present the player with a play button. That is it. :slight_smile:

The policy will always be minimum user intervention so we will just make use of all existing rules that NIT currently have for placing files and only bother user if truly needed and always provide a Cancel button for the user to opt out.

What do you think?

I strongly doubt NIT could help. PW players want to just click on a server name and play without having to actually download anything themselves, let alone have the downloads installed.

@pscythe
I don’t know how I landed up using Extended instead of Enhanced Edition. I am due to upload a stability release in the coming week and will change all the references to Enhanced before releasing.

If you follow Bloodright’s NWAlleys link, you will see there are three versions that could be downloaded. So there needs to be a way of determining which is the latest version. The same issue may arise when specifying a project url that has multiple versions available for download.

Presumably, the Vault has some type of database containing information that would be more amenable to programmatic access rather than scraping web pages and having to deal with multiple formats (eg new lines after related projects or spaces or commas).

I could package the 7-Zip DLLs that NIT uses, but have avoided doing this so far because I think 7-Zip is worth having installed when a user is going to start using Mods.

I would be open to the simplified front end you mentioned, but how would we define “perfect” projects so that other links would not be accepted. Also, the way NIT works does require the current interface to be visible. Changing this aspect, may require significant refactoring - I would need to spend some time determining the impact on the existing code.

The “do not allow Cancel” for 7-Zip and the module linking is to prevent problems that would cause crashes or lose play time information. The latter can be avoided if I could extend the Game Save Name to Module database that is included with NIT or figure out how NWN’s save name is determined (I think I asked some questions on this topic and did not find some rule I could follow or accessing the .mod file records was beyond me at the time - could certainly be revisited to try and remove the prompt).

Bottom line is anything is possible, but may need significant changes to achieve. I am sure there are some aspects that I will need help in understanding so that I can increase automation (eg how to navigate a Mod’s structure so I can find the name NWN uses for the .SAV file name, establishing what to download from a Project link, etc).

1 Like

Yes the simplest approach would be to present the list of files to the user let him decide, that of course breaks my approach of wanting minimum user intervention but it can’t be helped sometimes. We can also do some guessing. NWAlleyways have 3 files with similar names, the only difference are limited to the last 3 characters and they’re all numbers, so maybe sort them in descending order and take the top one? We might also have to program in domain knownledge for somethings too like what you have done with the file extension maps and such. eg: From an required project URL we can surmise that it is requires CEP and then see if the user already have CEP installed…etc.

Yes that would be much more preferable. But I am not sure how available is @niv is to help with such things. Heard he is real busy.

We can always recommend that to the user rather than requiring them to install it. There are other archivers like winrar that does a fair job with vault downloads.

I don’t really know until we’ve scrapped enough project pages to find out. I suppose projects with just single files or multiple files with no conflicts or name clashes would be perfect?

Okay I see. That is non-trivial then. Would it work if you minimise NIT and pop up this new simplified window?

I suspect DB restrictions might be the case. Could you create a different “unknown” or “defer” category? Can NIT just skip over and forget about them? Or does that break data integrity?

Yes anything could be automated if we understand it well enough and that is always the biggest hurdle. :smile:

Question is why do we need to list previous versions assuming new version is interchangeable without issue which in case of NWAlleys is true…

@Shadooow
Some authors like to maintain the number of downloads count.

This example is only one of many variations. However, parsing for latest version is possible. Knowing the date last modified would be even better.

Point taken. When we start to make the simplification changes, I will bundle the required DLL and EXEs (or automatically install 7-Zip) - will need to review usage to determine the best approach.

Any thoughts on how to scrape the pages?

Not really, but I could try and hide the NIT interface by making it transparent and see if that works. Another option is to provide a “Wizard” button that provides the main interface for users that want to use the simplified features.

Would break the way NIT is designed to work. In other words, would require significant changes. From what I recall, the name NWN uses is held in the Mod’s record information. I could not figure out how to work out the location of the record so I could read the information. Perhaps there is someone who could help me understand how to determine the location within the file, which would allow me automate this. It was something I planned to revisit at some point, because I don’t like the prompt that is currently required.

Final issue is the demand for such a facility. NIT’s user base for the current version is low. It does seem to increase over a period of time (when I am not releasing updates). Your original thought of initiating the install from the project page is still the option that would generate the most usage (but I have no idea how that would work).

Things I could do now…

  1. Create a self-extracting EXE to install NIT (and 7-Zip).
  2. Work with someone to automate NWN’s .sav file to the Mod (ie required record location within Mod).

Other thoughts
Use the Wizard approach to simplify download and install. The Wizard could be started by default and remain until the user dismisses it or exits NIT. So an Install, Play, Dismiss and Exit button.

Changing the premise that NIT uses to install and manage Mods will require extensive work and may take quite some time, depending on how far we want to go.

1 Like

Snit’s revenge!

I understand. Was just deleting old beta versions of my unofficial patch 1.72 and was bit sad to lose 2k downloads that way but I think that transparency is better than having high download counts

Yes that would certainly help (but sometimes they can be unreliable). Not sure if we can get the modified date without downloading the file first. Will have to check.

I will do some prototyping with grep & sed filter chains then port them over to .NET

So there is no simple way to disable management of Saved Games entirely? I thought we shouldn’t bother beginners with it at all and let it be an option for advanced players only.

Using a hex editor, could you find what you need as a text string in the .sav file? Could we then use the offset or surrounding byte patterns to locate the text string?

That sounds good!

NIT is a powerful tool. If we can simplify the work flow we should attract more users.

1 Like

Looks like the project pages are well formed enough for easy scrapping:

# extracting attachments    
$ grep file-icon /f/tmp/Bloodright\ -\ The\ Blood\ Royal\ _\ The\ Neverwinter\ Vault.html  | tr = '\n' | tr \& '\n' | grep http
https://neverwintervault.org/sites/neverwintervault.org/files/project/1986/files/bloodright_-_the_blood_royal.rar

# extract required projects
$ grep required-projects /f/tmp/Bloodright\ -\ The\ Blood\ Royal\ _\ The\ Neverwinter\ Vault.html  | tr = '\n' | tr \" '\n' | grep http
http://neverwintervault.org/project/nwn1/hakpak/tileset/nwalleys
http://neverwintervault.org/project/nwn1/audio/music/bloodright-music-hak
1 Like

You need to look at the structure of the Mod file. The required string will vary based on things like the length of the description. There is information on this record structure, it was just beyond me to figure how to determine the offset. So a hex editor would not help, what would help is understanding how to navigate the information (in the same way that NWN Explorer does).

This is something I can give some thought to.

Thanks for this. I am not familiar with grep - can it be called by VB.NET and what does it return?

1 Like

You got any documentations or source code describing the file format? If you tell me which fields we’re looking for maybe I can help.

Grep is a command for filtering text files, it prints out lines matching the pattern you give it. It can be called by VB.NET like any .exe but I thought we could do it all in .NET anyway. I am actually trying to rewrite what I did with the grep filters in VB.NET learning VB.NET and Visual Studio as I go. LOL

A proper VB module will take a while.

1 Like

The mod file is an erf, essentially. It is basically a binary archive format. Within that are a number of other files, mostly GFF, text (uncompiled scripts) and compiled scripts. GFF file format is documented on the vault somewhere. As Surazal says you can’t just find an offset and use that from one mod to another. Or even from the same mod after editing it. You’d need to find the relevant file in the archive, then read that data as a GFF file to find the field you are looking for. There are tools/libraries to do this but based on extracted files, not in place in the archive file.

grep is a beautiful thing.

1 Like

Thanks for that.

So what is the name of this GFF file and the fields we need?

Certainly is! What I did using grep under 2 mins took me 3 hours and 80 lines of VB.NET to do. LOL

Scrapper is almost done. Just need to get the API sorted.

F:\Visual Studio Apps\Scraper\Scraper\bin\Debug> .\Scraper.exe https://neverwintervault.org/project/nwn1/module/snow-hunt
    Attachments
    https://neverwintervault.org/sites/neverwintervault.org/files/project/9188/files/snow_hunt.zip
    https://neverwintervault.org/sites/neverwintervault.org/files/project/9188/files/snow_hunt_readme.txt
    Required Projects
    http://neverwintervault.org/project/nwn1/hakpak/ccc-2011-dec-winter
    http://neverwintervault.org/project/nwn1/hakpak/tileset/senemenelas-interior-tilesets
    http://neverwintervault.org/project/nwn1/hakpak/tileset/wild-woods-v07-wild-lands-tilesets

I don’t know. It depends on what information you are looking for. I was not reading the whole discussion. Module information like the name of the module is in module.ifo. But that’s the name of the module as defined in the toolset not the name of the file. I think Surazal already hit that issue at one point in his work.

1 Like

You can download the NWN Omnibus, which has all the information I could find.

I will need to remind myself what the field was so will need some time to research.

Look forward to seeing what it does and how we can use it.

Sure did :slight_smile: Just have to remember what I found out. Hope NWN Explorer will remind me.

1 Like

Was easier than I thought. Using NWN Explorer it is the module.ifo file and the field is Mod_Name. This is what NWN uses to determine the .sav file name.

I could pre-process all the mods so I have a lookup table to avoid having to ask the user.

The problem I encountered (probably ignorance coupled with stupidity) is that could not figure out how to locate the field. I was able to do it for the Mod_Description, but logic defied me for the Name field.

Any help will be welcome.

1 Like

Once you have downloaded and extracted the Omnibus, the information I am after is described in the NWN Omnibus\data\BioWare For Developers\IFO Format folder.

1 Like