Reading Mod file information

I am trying to use a VB.NET program to retrieve the Mod_Name field from a Mod file.

I believe my initial issue is that I am unable to determine how to locate the start position of the IFO record within the file.

I have been using the NWN Omnibus to understand how these file are structured, but am too stupid to work out how to derive the start position of the IFI portion of the file.

Any help in understanding how I could achieve this will be very welcome and much appreciated.

Think I know what the problem is. You’re reading in old chars (which was 1 byte back in the day) like new chars (which are 2 bytes nowadays).

@pscythe
I may well be missing your something, but I am not sure that explains the problems. Accessing the Mod Description works perfectly. Also, the information as defined in the NWN Omnibus, specifies 1 byte rather than 2.

Have you been able to verify this?

Remember that NwN was built before the days of the various utf encodings. Internally NwN is still in ANSI/ASCII i.e. byte sized characters. I had a problem with editing 2da files recently, the fix for which was to tell Notepad++ to use ANSI encoding as the default as in one update or another they had switched to one the utf encodings.

Trent has said in the streams that they are looking into this but it is my guess that if they do change it, that it will be quite some time before it happens. It will involve quite a lot of work.

TR

@Tarot_Redhand That is exactly my point.

But somehow @Surazal manages to retrieve the ERF file description using BinaryReader.readChars which reads in 2-byte Unicode chars instead of the 1-byte ANSI char that NWN was using. Does VB’s Option Compare Text changes that to 1-byte chars?? Anyway I am wring a mod reader from scratch strictly using readBytes so there is no ambiguity.

I haven’t tried running your code but that was the first thing I spotted when studying it.

The other problem was that the code was trying to read the ERF resources list without going through the Keys list first to locate module.ifo. You had to search the keys list for resref module.ifo then that gives you a index into the resources list which would give you a offset into the actual IFO.

That’s good info - I will take another look at the Omnibus and see what I can work out. Many thanks.

According to my copy of a VB programmers reference we are dealing with streams which I never use. Reading about the ReadChars method it appears that it reads a single character and advances the reader’s position according to the streams encoding and the character. So it would seem that stream classes somehow know the encoding of the text being read in.

TR

The default encoding is actually UTF-8 which can be 1 to 6 bytes depending on the character itself. So this explains why it works for @Surazal as the module descriptions are in ASCII which is 1-byte chars.

anyway this is what I do to read 4-chars as NWN sees chars using the BinaryReader.

System.Text.Encoding.ASCII.GetString(reader.ReadBytes(4))

Okay, it is done.

F:\ModReader> ModReader.exe "To Heir is Human Custom.mod"
Module Description=
"To Heir is Human"
by BioWare

Set in the heart of Cormanthor, near Myth Drannor, a band of rangers, searching
for a kidnapped child, has been defeated by a drow army.  The few remaining surv
ivors have given up hope of rescuing the heir to a barony. Who dares face the mi
ght of the drow to save an innocent child?

Recommended Levels: 5 - 15
Number of Players: 1 - 4

Mod_Name=To Heir is Human

The code is a bit messy but it was mostly a hack

Option Compare Text
Option Explicit On
'Option Strict On

Imports System.IO

Public Class ErfReader
    
    Private Const BufferSize As Integer = 4096 * 2
    Public reader As BinaryReader
    Private fStream As FileStream

    ' For localised descriptions of the module
    Private locStringCount As Int32     ' LanguageCount
    Private locStringSize As Int32      ' LocalizedStringSize
    Private locStringOffset As Int32

    Private keysCount As Int32
    Private keysOffset As Int32
    Private resourcesOffset As Int32

    ' ERF keys have a fixed size
    Private Const keySize = 16 + 4 + 2 + 2

    Sub debug(str As String)
        Console.WriteLine("Got: |" + str + "|")
    End Sub

    Function checkType(TypeStr As String)
        Return (TypeStr = "MOD " Or TypeStr = "ERF " Or TypeStr = "SAV ")
    End Function

    Function checkVersion(VersionStr As String)
        Return (VersionStr = "V1.0")
    End Function


    Public Function open(fname As String)
        fStream = New FileStream(fname, FileMode.Open, FileAccess.Read, FileShare.Read, BufferSize, FileOptions.RandomAccess)
        reader = New BinaryReader(fStream)
        Dim type As String = System.Text.Encoding.ASCII.GetString(reader.ReadBytes(4))
        Dim version As String = System.Text.Encoding.ASCII.GetString(reader.ReadBytes(4))

        ' Not sure if these are signed?
        locStringCount = reader.ReadInt32
        locStringSize = reader.ReadInt32
        keysCount = reader.ReadInt32

        locStringOffset = reader.ReadInt32
        keysOffset = reader.ReadInt32
        resourcesOffset = reader.ReadInt32

        ' What is this??
        'Dim descOffset As Int32 = reader.ReadInt32

        Return checkType(type) And checkVersion(version)
    End Function

    Public Sub close()
        reader.Close()
        fStream.Close()
    End Sub

    Public Function GetDescription()
        reader.BaseStream.Position = locStringOffset

        Dim attempts As Int32 = 0

        Do While attempts < locStringCount
            Dim languageId As Int32 = reader.ReadInt32
            Dim size As Int32 = reader.ReadInt32
            Dim description As String = System.Text.Encoding.ASCII.GetString(reader.ReadBytes(size))

            If (languageId = 0) Then   ' English
                Return description
            End If
            attempts = attempts + 1
        Loop

        Return "***No English Description found for Module.***"
    End Function

    ' Returns the ResId from KeyList for the given resref and type
    Public Function FindFile(name0 As String, type As Integer)
        Dim name As String = name0.ToLower

        reader.BaseStream.Position = keysOffset
        Dim seen As Int32 = 0

        Do While seen < keysCount
            Dim resref As String = System.Text.Encoding.ASCII.GetString(reader.ReadBytes(16))
            Dim resid As Int32 = reader.ReadInt32
            Dim resType As Int16 = reader.ReadInt16
            reader.ReadBytes(2)  ' discard unused

            If (resref = name And type = resType) Then
                'Console.WriteLine(name + " of Type=" + type.ToString + " Found at " + seen.ToString + " with resId=" + resid.ToString)
                Return resid
            End If
            'Console.WriteLine(resref + resType.ToString)
            seen = seen + 1
        Loop
        Return -1
    End Function



    Private Const resourceElemSize = 4 + 4

    ' Returns the FileOffset for the given ResId
    Public Function GetResourcePosition(ResId As Int32)
        Dim resource = New ResourcePosition
        reader.BaseStream.Position = resourcesOffset + ResId * resourceElemSize
        resource.offset = reader.ReadInt32
        resource.size = reader.ReadInt32
        Return resource
    End Function
End Class



Public Class ResourcePosition
    Public offset As Int32   ' Offset from file
    Public size As Int32
End Class



Public Class IfoReader
    Private reader As BinaryReader
    Private position As ResourcePosition

    Private structOffset As UInt32
    Private structCount As UInt32
    Private fieldOffset As UInt32
    Private fieldCount As UInt32
    Private labelOffset As UInt32
    Private labelCount As UInt32
    Private fieldDataOffset As UInt32
    Private fieldDataCount As UInt32
    Private fieldIndicesOffset As UInt32
    Private fieldIndicesCount As UInt32
    Private listIndicesOffst As UInt32
    Private listIndicesCount As UInt32


    Public Sub New(r As BinaryReader, resPos As ResourcePosition)
        reader = r
        position = resPos
    End Sub


    ' Res is pointing to the module.IFO within the ERF
    Public Function ReadHeader()

        reader.BaseStream.Position = position.offset
        Dim type As String = System.Text.Encoding.ASCII.GetString(reader.ReadBytes(4))
        Dim version As String = System.Text.Encoding.ASCII.GetString(reader.ReadBytes(4))

        structOffset = reader.ReadUInt32
        structCount = reader.ReadUInt32
        fieldOffset = reader.ReadUInt32     ' field array
        fieldCount = reader.ReadUInt32
        labelOffset = reader.ReadUInt32     ' label array
        labelCount = reader.ReadUInt32
        fieldDataOffset = reader.ReadUInt32
        fieldDataCount = reader.ReadUInt32
        fieldIndicesOffset = reader.ReadUInt32
        fieldIndicesCount = reader.ReadUInt32
        listIndicesOffst = reader.ReadUInt32
        listIndicesCount = reader.ReadUInt32

        Return checkIfoHeader(type, version)
    End Function

    Public Function GetModName() As Object
        If Not ReadHeader() Then
            Console.WriteLine("Not IFO file")
        End If

        'Goto the very first struct
        reader.BaseStream.Position = position.offset + structOffset

        Dim structType As Int32 = reader.ReadInt32  ' structType is actually signed
        Dim structDataOffset As UInt32 = reader.ReadUInt32  ' index into field indices array
        Dim structFieldCount As UInt32 = reader.ReadUInt32

        If Not structType = &HFFFFFFFF Then
            Console.WriteLine("**** Failed to find top struct !!!")
        End If

        'Console.WriteLine("struct offset=" + structOffset.ToString)
        'Console.WriteLine("struct type=" + structType.ToString)
        'Console.WriteLine("struct count=" + structCount.ToString)
        'Console.WriteLine("struct field count=" + structFieldCount.ToString)
        'Console.WriteLine("struct data offset=" + structDataOffset.ToString)

        ' Looking into the field indices array 
        reader.BaseStream.Position = position.offset + fieldIndicesOffset

        ' retrieve all the indices first
        Dim seen As Int32 = 0
        Dim indices As List(Of UInt32) = New List(Of UInt32)
        Do While seen < structFieldCount
            Dim index As UInt32 = reader.ReadUInt32
            indices.Add(index)
            'Console.WriteLine("field index=" + index.ToString)
            seen = seen + 1
        Loop

        ' now inspect the fields array        
        Dim currPosition As UInt32 = position.offset + fieldOffset
        For Each index In indices
            reader.BaseStream.Position = currPosition
            Dim fieldType As Int32 = reader.ReadInt32  ' signed
            Dim fieldLabelIndex As UInt32 = reader.ReadUInt32    ' index into label array
            Dim fieldDataOffset As UInt32 = reader.ReadUInt32    ' byte offset into Field Data Block
            currPosition = reader.BaseStream.Position

            Dim label = GetLabel(fieldLabelIndex)
            'Console.WriteLine("Label = " + label)
            If (label = "Mod_Name") Then
                Return GetFieldDataCExoLocString(fieldDataOffset)
            End If
        Next
        Return "*** Mod_Name field not found ***"
    End Function

    Private Function GetFieldDataCExoLocString(offset As UInteger) As String
        reader.BaseStream.Position = position.offset + fieldDataOffset + offset
        Dim totalSize As UInt32 = reader.ReadUInt32    ' total size not including totalSize itself
        Dim strref As UInt32 = reader.ReadUInt32   ' into dialog.tlk
        Dim count As UInt32 = reader.ReadUInt32    ' Number of substrings here

        'Console.WriteLine(count.ToString + " substrings")
        Dim seen As UInt32 = 0
        Do While seen < count
            Dim stringId = reader.ReadUInt32   ' unsigned?
            Dim stringLen = reader.ReadUInt32   ' unsigned ?
            Dim contents = System.Text.Encoding.ASCII.GetString(reader.ReadBytes(stringLen))
            'Console.WriteLine("stringId=" + stringId.ToString + "  |" + contents + "|")
            If (stringId = 0) Then
                Return contents
            End If
            seen = seen + 1
        Loop
        Return "*** Mod_Name value not found ***"
    End Function

    Private Const labelSize = 16
    Private Function GetLabel(offset As UInt32)
        reader.BaseStream.Position = position.offset + labelOffset + offset * labelSize
        Return System.Text.Encoding.ASCII.GetString(reader.ReadBytes(labelSize))
    End Function



    Function checkIfoHeader(type As String, version As String)
        Return (type = "IFO " And version = "V3.2")
    End Function
End Class

Public Module ModReader
    Public Sub main()
        Dim arguments As String() = Environment.GetCommandLineArgs()
        Dim reader As ErfReader = New ErfReader

        If (Not reader.open(arguments(1))) Then
            Console.WriteLine("wrong right file")
            Return
        End If

        Console.WriteLine("Module Description=")
        Console.WriteLine(reader.GetDescription)
        Console.WriteLine()

        Dim resId = reader.FindFile("module", 2014)  ' 2014 for IFOs
        Dim resPos = reader.GetResourcePosition(resId)

        Dim ifo = New IfoReader(reader.reader, resPos)
        Console.WriteLine("Mod_Name=" + ifo.GetModName)

    End Sub
End Module

The Bioware docs are fairly reliable as far as this task is concern. The only mistake was in the GFF docs where struct-type is actually a signed 32bit int as it is used rather than the unisgned one it says in places.

@pscythe
Thanks for this and all the effort you have made to help me - very much appreciated. Will take me some time to work through the code so I understand it. I had managed to compute the IFO location before you submitted your code, but was stuck on processing the IFO information - so getting your solution is very welcome.

Once I understand your code, I will integrate into NIT and then start changing how the .sav name is mapped to the NIT mod. Will also have to code some version migration code to tidy up what is no longer needed.

I will release the automated .sav to Mod name change before starting to think about a Basic user interface option.

1 Like

Let me if you got any questions. The GFF is a whole lot of indirection. LOL