Xj Conversion Notes

[kion]

I've been looking into xj models and comparing them with nj models to see if there is a way to export xj models into a more readable format. I'm going to write what I've found so far here to clear my mind and hopefully someone else can provide incite. My priority right now is mostly on documenting the formats and looking for parallels between the two.

NJ Filetype

So far the simplest model in the game i've found is the item box that drops on the ground. I will be using that for this description. Here's a screen cap of what the basic file type looks like:



In red at the top, NJCM chucks seem to be a tree structure of nodes starting with a first node that appears at the top of the file. The structure for Nodes is:

typedef struct {
EvalFlags : DWord 0x17
Model : DWord 0x170
Position : Single[3] {0,0,0}
Angle : Sint32[3] {0,0,0}
Scale : Sint32[3] {1,1,1}
Child : DWord NULL
Sibling : DWord NULL
} NJS_NODE

Blue at the bottom is the pointer from the Node to the Model structure.

typedef struct {
Vertex_List : DWord 0xA4
Mesh_List : DWord 0x34
Center : Single[3] {0,0,0}
Radius : Single {-4.098}
} NJS_MODEL

The Vertex List is the part outlined in green. Not too sure about the first three entries. They look like 16 bit entries, I'm just not entirely sure what they're for. 0xFF000000 seems to mark the end of the vertex array.

typedef struct {
Type : Word 0x29
Unknown : Word 0x31
Start : Word 0x00
Num_Verts : 0x08
Array : [ Pos Single[3], Normal Single[3] ]
} NJS_VERTEXT_LIST

As for the Mesh_List in purple, I'm guessing that's what that is. I haven't found enough information to really give an accurate guess as to what that is. I think it has a word for type, word for number of entries, TU TV and then a list of something (Color? Textures?) ending with 0xFF000000.

 

[kion]

XJ File type

So that's everything I have so far for the .nj filetype. Aside from the missing mesh information, assuming that's what it is, it's mostly self contained at pretty simple. That's for a single node, but even multiple nodes simply follow the same structure and it's just a matter of adding more pointers. XJ files on the other hand start to get a lot weirder, the base file with the NJTL and NJCM tags removed is showed in the image below.


I ran out of colors when trying to label everything and probably used too many colors when grouping bytes. As such, rather than referering to sections by their colors, I labeled each group on the side from A to H. And Will be referring to each one by 'Section <Letter>'.

Starting off at the top of the file with Section A, fortunately the file header is exactly the same as given in the NJ file format.

typedef struct {
EvalFlags : DWord 0x17
Model : DWord 0x38
Position : Single[3] {0,0,0}
Angle : Sint32[3] {0,0,0}
Scale : Sint32[3] {1,1,1}
Child : DWord NULL
Sibling : DWord NULL
} NJS_NODE

So we have the model pointer at 0x38 which is Section B following immediately after the header, and already things have started to get pretty different. This is my best guess for what the structure of this section is.

typedef struct{
Unkown1 : DWord 0x00
Vertex_List : DWord 0x260
Template : DWord 0x01
Mesh_List : DWord : 0x280
Template : DWord 0x01
Unkown2 : DWord 0x00
Unkown3 : DWord 0x00
Center : Single[3]
Radius : Single
} NJS_XJ_MODEL

Unkown1 i really have no clue about. I've only looked at a few models, but so far all of them seem to have 0x00 here. Unkown2 and Unkown3 could likely be another pointer that simply isn't used. I haven't looked into that too deeply yet. But at least for now, we have two pointers to look at so let's start with the Vertex pointer.

Down in Green at Section D is where the Vertex_List pointer points to, but I'm not exactly sure what to make of it.

typedef struct{
Type : DWord 0x07
List : DWord 0x80
Unkown4 : DWord 0x24
Unkown5 : DWord 0x0D
} NJS_XJ_VERTEX_LABEL

Since there are no actual vertexes listed in this struct, I'm labeling it as the "Vertex Label". The first DWord could likely be evalflags as 0x07 seems to be a common number which pops up associated with that. List seems to be an obvious pointer, 0x80 is marked with Section C which contains all of the Float types for Vertices. Not sure what significance 0x24 could have for Unkown4, it's not a pointer since that would overlap with the file header, so it could be another flag or type. And lastly Unkown5 0x0D or 13 in decimal is likely the number of vertices, though I'm not sure of this.

Section E below this entry could be allocated for more pointers, or simply padding since nothing seems to point to this location.

Looking up at Section C we have The vertex List. Comparing it to the NJ model at the top, we have the same Values ["A272FEBF", "443A0240", "6CE0C5B6", "31CD13BF", F205513F", "F0A79D33"] followed by three new values before continuing on with values found in the NJ model at the top. My Current guess for this struct is:

typedef struct{
Pos : Single[3],
Normal : Single[3],
TU : Single
TV : Single
Color : Single(?)
} XJ_VERTEX_ENTRY

Which makes Num_Vertexes a possible value for Unkown5 in NJS_XJ_VERTEX_LABEL.

Goint all of the way back to NJS_XJ_MODEL, let's take a look at the Mesh_List pointer down at 0x2E0, labeled Section H, we have another entry that doesn't entirely make a lot of sense.

typedef struct{
Pointer_1 : DWord 0x280
Value_1 : DWord 0x04
Pointer_2 : DWord 0x2C0
Value_2 : DWord 0x0E
} MYSTERY_POINTERS

Pointer_1 points to Section F, which seems to just be a list of either words or dwords, [0x02, 0x04, 0x05, 0x00, 0x03, 0x06, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x06, 0x00, 0x00]. So I really have no idea is there is any significance for Value_1 being 0x04 when pointing to this. Also I wrote this array as if the entries were 32 bit dwords, but looking at the entry below, it would make more sense if there were also 16 bits.

Pointer_2 points to Section G which seems to clearly have a list of Words: [0x02, 0x03, 0x06, 0x07, 0x05, 0x03, 0x01, 0x08, 0x00, 0x09, 0x04, 0x0A, 0x0B, 0x0C, 0x00, 0x00]. The interesting part is that this list goes up to the value 0x0D - 1 meaning that again Unkown 5 is likely the number of vertexes. Though I'm not sure what this array is for. Value_2 0x0E is 15, and given there are 16 entries, one of the zero's on the end could have been added onto the end for padding.

 

[kion]

Exported from xj file.

 

[Andy]

Ooohh lala. Very interesting work you've done so far!
Can't wait to see where this goes!
Wish I could help with things like this :roll:

 

[kion]

NJTL headers are pretty simple.



Header is NJTL followed by a pointer to the end of the section. This offsets the pointers in the entry by 8 bytes. Header in blue is:

typedef struct {
NJS_TEXNAME *textures; /* Texture name list */
Uint16 nbTex; /* Number of textures */
} NJS_TEXLIST;

So 3 textures and it points down to 0x08 (+0x08 for offset) right below the header. The NJS_TEXNAME entry is right below that, in green.

typedef struct {
void *filename; /* Texture file name */
Uint32 attr; /* Texture attributes */
void *texaddr; /* Texture attributes */
} NJS_TEXNAME;

So pointer to the file name, and then two dwords for attributes (if it has them). In this case it doesn't and we have the names for our three textures, at 0x2C, 0x48 and 0x64 given by the purple outline right below it.

And that seems to be it. One of the few times the Ninja guide pdf actually has accurate data on the game's data structure.

 

[kion]

Haven't looked at textures yet, but stuff is starting to fall into place. Sword exported from psobb's Item_Model.afs file.

 

[kion]

Some of these are turning out better than others. I think I just need to figure out how to handle rotation and scale, and we should be in business.

 

[tofuman]

Nice work. Thanks for sharing!

 

[kion]

Nice work. Thanks for sharing!

I definitely want to leave a paper trail for anyone else that might be interested in stuff like this. Also it helps me a lot to write everything out and be able to refer back to it without having to remember everything. Also Tofu, I might have to bother you about PRC decompression sometime down the line. I'd like to write a program than decompressed the PlyMotionData.pr2 files and exports all of the .njm files in it. I haven't been able to find anything about pr2 or prc anywhere on the internet, just your PRC tool which seems to be made of pixie dust and fairy juice as far as I can tell. I can always call it from the command line, but something integrated would be ideal.

I'm looking into fixing rotation on the models right now. Does anyone know which weapon this is and have a picture of what it's supposed to look like, so I can make sure I'm on the right track?

 

[tofuman]

Sure thing. It's best to add me on Skype. I'm generally available most days.

 

[Soly]

I'm looking into fixing rotation on the models right now. Does anyone know which weapon this is and have a picture of what it's supposed to look like, so I can make sure I'm on the right track?

I'm sure that's a dark meteor... okno
That seems to be a PWand http://i.imgur.com/YMYu0jV.jpg

 

[Aleron Ives]

BlueCrab has been working on PRSD (PR2/PR3) compression for inclusion in his PSO toolset, but I don't think he's finished it yet. Fun fact: his PRS compressor is even more efficient than Sega's, so if you have a file that must be within a certain size (e.g. the Episode III login packet) that crashes the game if it's too big, I think he has the only PRS compressor that can match (let alone exceed) Sega's compression efficiency.

 

[kion]

For PRS i found this python implementation that works pretty well. Not exactly perfect, but I was able to port it to Node and extracted each file from a bml, decompress it and write it to the file system, though I haven't done any extensive testing. Is PRC a variation on the PRS decompression? Google doesn't turn up anything useful for "Sega pr2/prc/prsd decompression".

Also played around with rotation and eval flags yesterday. No luck in getting everything to line up and working just yet. Without textures, it's hard to tell exactly what's what, so it looks like my next detour is going to be textures.

 

[Aleron Ives]

The PR2 file contains the data payload, and the PR3 file contains some instructions on how to adjust the pointers in the PR2 file. PRSD is apparently a slightly more secure form of PRS. If you want to change the PR2 file, you have to update the PR3 file, too.

 

[kion]

If you want to change the PR2 file, you have to update the PR3 file, too.

Generally the focus is on extraction, not really interested in putting things back.

 

[Aleron Ives]

Why would you care about extracting things if you can't put them back? I thought the whole point of figuring out PSO's formats was to be able to change things. :?

 

[kion]

To play around with stuff and things in Unity and WebGL. Seems like a huge amount of work to try and write tools and proprietary formats into a client whose code base can't be changed.

 

[Aleron Ives]

People reskin items all the time, and being able to convert standard OBJ models into PSO formats would be the most important step towards adding entirely new items (as adding reskinned items using existing models is already possible).

 

[Andy]

On a side note,
what program are you using to view the models in those pictures above?

 

[kion]

Program name is in the upper left hand corner of the picture . 3D Builder, i think it's default in Windows 8, 10.