nwn2_CritterStatsEditor Gff/Utc file(s)

I got a bit too fedup with the vagaries of creature-stats (blueprint) editing in the Electron toolset.

i also wanted to get a grip on the GFF specification.

So am presently sinking my time into [nwn2_CritterStatsEditor] GeneralGFF (github c#). It won’t will be a generic GFF editor, although it’s primary purpose is to edit UTC blueprints. i intend it to provide hints etc during editing, so we/you/me doesn’t have to go hunting through 2da files for what the assorted values translate to …

It will support Nwn2’s 32-char CResRef field. In fact, the GFF-spec itself doesn’t limit the length of CResRef fields* … so i’ll probably end up issuing a warning instead of an error if the length of the string exceeds 16/32 chars. Ie, will try to keep the code adaptable to the general GFF spec.

 
* read: if a GFF-editor truncates nwn2-style (32-char) resrefs to nwn1-style (16-char), it’s because the programmer of the editor designed it so; said truncation is not a characteristic of the GFF spec itself. [The fundamental limitation is rather based on the maximum size of a file on your hardrive 255 bytes.]

/will see how it goes, where it leads

2 Likes

Looks promising, and it seems to works well on Linux too :slight_smile:

One feature I’d like to see is that you editor keeps the gff file structure as generated by the toolset (i.e. gff field order, string order, …) so that I can quickly see what fields have been changed between two versions of the UTC file, like this:
image
Though I don’t think many people use diff tools (this one is git diff) on their projects, so probably you’d better ignore this if it takes too much work.

Also I think you should provide at least minimal support for editing other types of blueprints (basic field editing should be enough for most cases), so people don’t have to switch between different tools.

1 Like

sweet. yeah i’ve done enough c#/.net programming to have a grip on basically keeping things compatible w/ Linux and Mono

hm. That might be a bit much for me at present. The GFF stucture was totally new to me a week ago and just getting it to parse out was a bit … frustrating … so, will keep this in mind but, practically, if and when i get a file to write out correctly I’ll just want to relax and cook a hotdog.

the idea is always pulling at me ofc. My impulse at present is to just get a creature-editor working the way I want, then either expand it or start from scratch and re-implement its functionality : the endgoal of CSE (critter stats editor) isn’t simply to edit UTCs – it’s to display accurate lists of what the values mean while editing UTCs … (similar to Yata Paths for crafting/spells.2da files, but for UTC files). This is inevitably going to bloat code-complexity to the point that, atm, I feel a need to stay away from trying to create a generic editor and focus on the mission.

that is, my goal is to edit creatures with a 3rd party editor that displays at least the basic level of information that can be seen in the toolset when editing a creature blueprint, which could then mean invoking a UTI editor etc

1 Like

3.4. Fields
The Field Array contains all the Fields in the GFF file except for the Top-Level Struct.

Figure 3.4: Field Array

  • Field 0 (Top-Level Struct)

:cookiemonster:

EDIT: TopLevelStructs are not included in the FieldsArray. /tested

a freshly added LeftHand item

even the toolset output doesn’t get it right all the time …

System.NullReferenceException: Object reference not set to an instance of an object.
   at OEIShared.IO.GFF.GFFStructFieldDictionary.Add(GFFField cField)
   at NWN2Toolset.NWN2.Data.NWN2ScriptVarTable.SaveToGFFList()
   at NWN2Toolset.NWN2.Data.Templates.NWN2ObjectTemplateData.SaveIntoGFFStruct(GFFStruct cStruct, Boolean bIsBlueprint, String sNameLabel)
   at NWN2Toolset.NWN2.Data.Templates.NWN2CreatureTemplate.SaveEverythingIntoGFFStruct(GFFStruct cStruct, Boolean bIsBlueprint)
   at NWN2Toolset.NWN2.Data.Blueprints.NWN2CreatureBlueprint.SaveEverythingIntoGFFStruct(GFFStruct cStruct, Boolean bIsBlueprint)
   at NWN2Toolset.NWN2.Data.Blueprints.NWN2CreatureBlueprint.OEISerialize(Stream cStream)
   at NWN2Toolset.NWN2.Data.Blueprints.NWN2CreatureBlueprint.OEISerialize(String sFilename)
   at NWN2Toolset.NWN2.Views.NWN2BlueprintView.ᐌ(Object , EventArgs )
   at System.Windows.Forms.MenuItem.OnClick(EventArgs e)

not a chance  :|

I finally got some input and output to match:

input:

StructOffset= 56
StructCount= 210
FieldOffset= 2576
FieldCount= 735
LabelOffset= 11396
LabelCount= 166
FieldDataOffset= 14052
FieldDataCount= 145
FieldIndicesOffset= 14197
FieldIndicesCount= 2712
ListIndicesOffset= 16909
ListIndicesCount= 244

output:

start_Structs= 56
records_Structs= 210
start_Fields= 2576
records_Fields= 735
start_Labels= 11396
records_Labels= 166
offset_Data= 14052
bytes_Data= 145
start_FieldIds= 14197
bytes_FieldIds= 2712
start_ListIds= 16909
bytes_ListIds= 244

The values are the same. The design-spec of GFF doesn’t make it easy. Some things are ids, others are offsets, some things are ids unless they’re offsets and vice versa. The idea behind GFF is great, but if i were on the team that wrote the spec I would have refactored the cr*p out of it.

still have a ways to go before the data in the output is accurate/readable tho

 

VIVALDI - Four Seasons - Alexandra Conunova - Orchestre International de Genève

2 Likes

Write/reload success!

The catch was that, in the algorithm i wrote to output the data, the TopLevelStruct was being written to the end of the Structs array. So i just slid it to the front of the array, tweaked a few other lines … needs more testing ofc

the input and output routines are generic – i believe they could be used for a generic editor quite easily*

 
* with attendant pitfalls: the engine is going to expect things on a stricter basis than a truly generic GFF editor would could or should handle …

 
-> GeneralGFF (editor)


^ Encounter GFF file

to test my input/output with more complicated files, like .are’s .gic’s and .git’s … tests have been stable and accurate so far

/relief

2 Likes

my confidence suffers when the engine outputs stuff like this

reatureDice_TlkEdit2

1 Like

Hi KevL,

Still cheering you on to success though! :ok_man:

And even though I have no clue as to what you are doing, all I can say is that that issue looks like a combination of loop, variable and code placement setting … if its anything like problems I experience in the NWN toolset. :wink: Or of course … a script needs recompiling since a change. :woozy_face:

Cheers, Lance.

thanks Lance

to be clear, the output shown above is output by NwN2 when saving a game.

I came across it while searching around my Nwn2 files to make a list of what filetypes are GFFs. This is what i have so far, btw

    /// <summary>
    /// The types of gff-files available.
    /// </summary>
    enum GffType : byte
    {
        generic,    //  0
        ARE,        //  1
        BIC,        //  2
        FAC,        //  3
        GIC,        //  4
        GIT,        //  5
        IFO,        //  6
        JRL,        //  7
        ROS,        //  8
        ULT,        //  9
        UPE,        // 10
        UTC,        // 11
        UTD,        // 12
        UTE,        // 13
        UTI,        // 14
        UTM,        // 15
        UTP,        // 16
        UTS,        // 17
        UTT,        // 18
        UTW         // 19
    }

So … while arbitrarily testing my in/out routines, I opened a savedgame’s playerlist.IFO file and it threw an exception. I was mildly surprised but started debugging … CExoLocString, which is a GFF field-type, can store strings in various localized languages:

    /// <summary>
    /// The types of languages available.
    /// @note The values will be used as integers as well.
    /// </summary>
    enum Languages : uint
    {
        English            =   0,
        French             =   1,
        German             =   2,
        Italian            =   3,
        Spanish            =   4,
        Polish             =   5,
        Russian            =   6, // <- nwn2 add (not in the nwn1 doc)
        Korean             = 128,
        ChineseTraditional = 129,
        ChineseSimplified  = 130,
        Japanese           = 131,

        // TlkEdit2 defines another Locale: "GffToken".
        // It can appear as either the
        //   "LocalizedName" (eg. "Battleaxe +2")
        // or
        //   "LastName" (eg. "DoneOnce7=1")
        // within an item's Struct under either
        //   "Equip_ItemList"
        // or
        //   "ItemList"
        // .
        GffToken = 4294967294 // 0xFFFFFFFE
    }

And with the help of TlkEdit2, noticed that a ‘valid’ language-type can be the so-called “GffToken” – it’s not described in the docs, however. I fixed that and took a closer look at savedgames’ IFO and BIC files … the player.BIC data is also stored in playerlist.IFO (the latter holds data about all current player characters, and seems to be the file/data that’s reloaded instead of player.BIC). And started noticing funnythings in the BIC/IFO gffs. Things like that field getting truncated above.

And things like this:

SaveThrow_TlkEdit2

but uh, integers shouldn’t be getting stored as Chars (i mean, that’s just silly). Chars are essentially letters that make up a string … (and just to be funner, they’re defined as 1-byte in C++ but as 2-bytes in C# – the designers of the GFF spec seem to have used the C++ 1-byte type, but that’s ASCII, rather ANSI, since ASCII is only 7-bits and it was ANSI that extended it to all 8-bits of a byte … but that’s not necessarily big enough to hold a character in UTF8 … and some strings in NwN2 are UTF8 …). So i have to sort that out.

Because here’s an example of what a “char” usually looks like:

SaveThrow_GGFF_will

And when I continued perusing BIC/IFO files in both TlkEdit2 and my GeneralGFF app for comparison, I began to notice how a bunch of stuff wasn’t really making strict sense in our character data. Eventually i saw the truncated labels and chose to watch random Utube videos for the rest of the day.

The other good part is, TlkEdit2’s output and my output appear to be the same (apart from the interpretation of Chars)

 
edit for clarity (sic)

2 Likes

It looks like your pointers are offset by one byte. A good way to fix that is to dump the pointer offsets at different stages of parsing/serialization, open the file in a hex editor (I like wxHexEditor), and compare them.

CExoLocString stores both a StrRef and a set of translated strings.

  • If the StrRef isn’t 0xFF_FF_FF_FF, then the CExoLocString should be translated using the TLK entry with this value
  • If the StrRef is 0xFF_FF_FF_FF, the CExoLocString should be translated using the language -> string map, with the user’s current language, or default to English language

I’ve never seen the “GffToken” language anywhere. Skywing’s nwn2datalib is a good reference for nwn2 files, and it does not show it: https://github.com/SkywingvL/nwn2dev-public/blob/b3bd8afd9a9f6c5e29b62654ff72257701240d14/NWN2DataLib/TlkFileReader.h#L73

The type char in C is simply a byte, which can be signed (-127 -> 128) or unsigned (0 -> 255), and is sometimes represented as an ASCII letter, but it is primarily an integer. In GFF specs, char is a signed byte, and byte is an unsigned byte.

Good luck with your GFF implementation, this thing is really messy and it takes some time to get it right :confused:

hey Crom, thanks for the input

to make sure we’re on the same page, I’ve changed repos: nwn2_CritterStatsEditor is currently defunct (will transfer whatever ideas i want from it later)

The current repo is GeneralGFF

okay, to biz

yep. point #2 is mostly handled by the engine; that is, the GFF simply stores the language id + masculine/feminine for any strings that may be held under the CExoLocString strref, but the default->fallback fetching mechanism is handled by the game.

- from Bioware_Aurora_LocalizedStrings_Format

but my app isn’t fetching strings, it’s only setting localized strings (if there are any…) with an exact langid+M/F value.

(thanks for the link to Skywing’s repo)

But have a look at this

    // BWLanguages.cs
    public enum BWLanguage
    {
        English,
        French,
        German,
        Italian,
        Spanish,
        Polish,
        Russian,
        Korean = 128,
        ChineseTraditional,
        ChineseSimplified,
        Japanese
    }

it adds Russian … and doesn’t have LangLastTlkLanguage. go figure eh

 

well, they’re output by the Nwn2 savegame mechanism … have a look at these 2 screenshots of GFFEditor:

the first shows correct local-string info in the right panel, but the second does not recognize what TlkEdit2 calls “GffToken” (see third screenshot)



not sure what you mean. Offset by one byte from toolset output? (i use HxD and now know enough about the GFF spec to write them by hand btw – i’ve had to write a few of them to narrow in on some past issues). But i’m not aware of any off by one errors at present …

this just in

System.OverflowException: Arithmetic operation resulted in an overflow.
   at generalgff.GffWriter.AddComplexDataField(TreeNode node) in c:\GIT\GeneralGFF\GffWriter.cs:line 599
   at generalgff.GffWriter.AddField(TreeNode node) in c:\GIT\GeneralGFF\GffWriter.cs:line 388
   at generalgff.GffWriter.AddStruct(TreeNode node, Boolean tls) in c:\GIT\GeneralGFF\GffWriter.cs:line 317
   at generalgff.GffWriter.AddComplexListField(TreeNode node) in c:\GIT\GeneralGFF\GffWriter.cs:line 670
   at generalgff.GffWriter.AddField(TreeNode node) in c:\GIT\GeneralGFF\GffWriter.cs:line 393
   at generalgff.GffWriter.AddStruct(TreeNode node, Boolean tls) in c:\GIT\GeneralGFF\GffWriter.cs:line 317
   at generalgff.GffWriter.AddComplexListField(TreeNode node) in c:\GIT\GeneralGFF\GffWriter.cs:line 670
   at generalgff.GffWriter.AddField(TreeNode node) in c:\GIT\GeneralGFF\GffWriter.cs:line 393
   at generalgff.GffWriter.AddStruct(TreeNode node, Boolean tls) in c:\GIT\GeneralGFF\GffWriter.cs:line 317
   at generalgff.GffWriter.WriteGFFfile(String pfe, TreeView tl, String ver) in c:\GIT\GeneralGFF\GffWriter.cs:line 65
   at generalgff.GeneralGFF.fileclick_Save(Object sender, EventArgs e) in c:\GIT\GeneralGFF\GeneralGFF.cs:line 311
   at System.Windows.Forms.MenuItem.OnClick(EventArgs e)

will look into it tomorrow, but if you mean something else, pls be specific w/ steps to repro  :)

 

okay i’ll implement CHAR as signed byte.

gfL

case FieldTypes.CHAR:
    field.CHAR = (sbyte)buffer[0]; // arithmetic overflow
case FieldTypes.CHAR:
    field.CHAR = (sbyte)(object)buffer[0]; // invalid cast
case FieldTypes.CHAR:
    var t = (sbyte[])(object)new[]{ buffer[0] };
    field.CHAR = t[0];

thanks to Jon Skeet for the insight

in other words, i actually came across a “-1” value of CHAR …

Since in 2s compliment of a byte -1 is represented by 255 (hex FF) you might be able to test for that instead. Twos compliment represents negative numbers by inverting the bits of the positive version of the number -

-1 becomes 1 becomes 254 (00000001 becomes 11111110)

and then adds 1 to it (11111110 + 00000001 = 11111111 = hex FF = 255 = -1)

TR

hey Tarot, i thought briefly about checking

if (val > 127)

or as you say i could do the twos complement bit-twiddle …

 
but i’d rather just shove the responsibility off on the compiler  ;)

(eg, what if it’s not a twos complement machine?) unlikely, but … (albeit even less likely than a big-endian machine…)

/anyway, did some fixes to the write mechanic and all seems well atm. (apart from a half dozen niggledy TODOs)

@CromFr
am still wondering what you mean by this

if it’s about achieving consistency with your codebase i’m willing to look into it (esp. if it’s just “off by one”)

The language IDs are different for every other games that shares the same engine. Xoreos has a list of different language mappings: https://github.com/xoreos/xoreos-tools#tlk-language-ids-and-encodings
LangLastTlkLanguage is probably here only for code maintenance purpose, i.e. not a real/useable language code.

Nevermind, I thought you had issues with the first character of strings being skipped

1 Like

perhaps i should explain where i got “Russian=6” from…

I have a project w/ Tani’s nwn2packer source, it uses OEIShared.dll

If i Ctrl+click on the OEIShared functions in the IDE, it brings up ‘source’ for OEIShared. Ctrl+clicks in that source allows browsing through what appears to be around 16 classes in the OEIShared namespace (or similar/related). One of the classes is BWLanguages, which contains a/the BWLanguage enum, and that enumerates “Russian=6”

will leave it as is for now and see how it goes … i think @Aqvilinus is Russian … maybe someone with a Russian installation of NwN2 (if it exists) can test it someday

 
UPDATE (19.11.8): “Russian” in the toolset

ts_CExoLocString_Russian

1 Like

here’s an interesting 16 min video on the history of 2’s complement
Binary: Plusses & Minuses (Why We Use Two’s Complement) - Computerphile

 
 
GeneralGFF is basically ready for beta release… (if anyone wants to compile the c#/.net 3.5 source just to try it out feel free). Both the toolset and Nwn2 recognized my output and displayed no issues (so far). i still want to play around with it for a while to smooth rough edges … (like figure out what to do about backups – it currently renames a file (that’s about to be overrwitten) with a .GGB backup extension but this feels like clutter, idk)

 
the CrittersStatsEditor (which is what I really want this for) has yet to be implemented

ps Even better explanation of 2s comp
Twos complement: Negative numbers in binary

1 Like

still pluggin away …

TODO: code a bunch of validity checks when loading a file