___________________________________________ - PSSG TOOLs - And Modding Tips [GreedFall] ___________________________________________ May 22, 2020 Version: 2.04 Written by: Dheu Email: Dheuster@gmail.com Use subject: PSSG TOOLs 2.04 QUICK SUMMARY ------------------------------------------------------------------------------- A collection of PSSG tools I created while modding GreedFall. Internet searches for PSSG don't provide many hits, so I thought it best to put this out there. This distribution also includes modding tips and advice/examples for developers looking to mod GreedFall. It is by no means comprehensive. Just some organized notes I made while working on my mods. ------------------------------------------------------------------------------- I. > > > > PSSG Overview ------------------------------------------------------------------------------- What is PSSG? PSSG is a container format for game resources. Basically it is a custom Binary XML representation. Imagine you have an OGG audio file. In the Silk Engine, it would save the resource in a .pssg file. It "wraps" the ogg file with binary data that the game needs to use the data within the game. These wrappers have a header that indicate the contents to the engine and then a body with the data itself. Sometimes a single PSSG wraps multiple resources. If you know the PSSG header format, you can figure out where the ogg data is and extract it. Some PSSG files wrap other binary data in whole. For example, the audio data is a fully independent ogg audio file. So if you extract the HEX from the PSSG and throw it into its own file, you can play it. Other types such as DDS images will have the raw image data, but will excluded the DDS headers since the PSSG container already has the information provided by a DDS header. In these cases, you may have to add a header to the binary content in order to view in a regular viewer. History and Compatibility: -------------------------- From what I can tell, PSSG was a format invented for the Phyre Engine: A Sony PlayStation engine released in 2008 to promote creation of games for the PS3. The engine and dev environment were made free because Sony wanted games for their platforms. As far as I can tell, support for the engine ended in 2016 when the PS4 came out. However, it was free and cross-platform. So a number of development shops took the engine and expanded it into their own customized versions (updating it in-house to meet the demands of newer Direct X standards). The Ego Engine and more recently the Silk Engine are byproducts of the Phyre Engine. GreedFall uses the Silk Engine. So why the history? These dev shops didn't customize everything. Only the parts that they needed. So an app made for an ego engine game will often times work with silk engine pssg files. EGO PSSG Editor --------------- One such program is the PSSG Editor created by Ryder25 mentioned in the Standard Tools section of the GreedFall modding tips. The latest release of EGO PSSG Editor was 2017. The EGO PSSG EDITOR can edit most PSSG files and can be used to update many resources, however PSSG has eveolved. Silk engine in particulat made changes to the spec that PSSG EDITOR can't handle. Sometimes you try to open a PSSG file and the editor just barfs. More concerning is that I have found situations where it silently fails (doesn't inform you of any issues) and corrupts the output. An example is the music/audio files in greedfall. If you convert a PSSG file that wraps audio to XML using EGO and then converted it back to PSSG with no changes, sometimes the PSSG is a different size. And if you place that pssg in the overrides directory (as a pgz), the associated audio in game may stop working. PSSG CONVERTER -------------- To help with some of the issues mentioned above, I made pssgConverter.py. It took about 6 weeks of poking and prodding to figure out the Silk PSSG format and then another 6 weeks to get a stable script. When I repeat the steps that broke the audio using the EGO PSSG EDITOR with pssgConverter, everything works as expected. The key to the script is that it is not schema driven. It reads the file in and makes a best guess at the schema for that specific file. It then saves the schema to a .pschema file and writes out the data to XML. Users can then update/edit the XML file and convert the XML back to PSSG. They can also update the .pschema file to render some of the binary chunks in different ways to make editing the XML easier. PSSG CONVERTER is a command line script, so it isn't as nice or user friendly as EGO PSSG Editor, but as a command line script, you can also easily run it on many files at once. My GreedFallModding tips make use of this ability to extract all the VOCAL files to XML so that the spoken line (part of the LIP data) can then be extracted from the XML using a script to create a list of all lines spoken by any given NPC. With thousands of lines, doing something like that with EGO PSSG Editor would be impracticable. ------------------------------------------------------------------------------- I. > > > > PSSG CONVERTER ------------------------------------------------------------------------------- A. Overview: pssgConverter.py understands PSSG. It converts between PSSG formats and XML. The XML can then be edited and converted back to the original resource for use in game as an override. B. Prerequisites: You will need 7-zip to extract the .spk files to a directory outside greedfall and expose the games resources. You will need to install Python. pssgConverter works with either Python 2.x or 3.x, 3.x is about twice as fast, but 2.x uses about half as much memory. If you have less than 16 GB of RAM, you will want to stick with 2.x. If you have more than 16GB of RAM, you should be fine with 3.x. If you have exactly 16GB of RAM, you may want to install both. My Setup: I have 16GB of RAM and I installed the latest 3.x for Windows but then I also installed CYGWIN which comes with Python 2.x by default. For pretty much everything, I use 3.x, but there are 3 large texture files that I can't extract with 3.x because my machine runs out of memory. For those 3 files, I can extract them using CYGWIN with python2 C. Usage: pssgConverter.py [options] infile [outfile] [pschemaFile] Flags: -h : detailed usage/help -i# : indent XML # spaces (ie: -i4) -a# : set analysis level to # (0 = default) -s : safe mode (use if you see warnings) -f : forcing conversion (even if no change detected) -o : force create/overwrite pschema (even if it exists) -p : show progress (if script takes more than 5 sec) -d : print debug feedback -t : print trace feedback -toXML : force conversion to xml (Assume file is NOT XML) -fromXML : force conversion to pssg (Assume file is XML) Example: Convert PGZ to XML $ pssgConverter.py foo.pgz Extracting [foo.pgz] -> [foo.pgz.pssg] Converting [foo.pgz.pssg] -> [foo.pgz.xml] Example: Convert XML to PGZ $ pssgConverter.py foo.pgz.xml Backing Up [foo.pgz] -> [foo.pgz.bak] Converting [foo.pgz.xml] -> [foo.pgz.pssg] using schemaSource [foo.pgz.pschema] Updating [foo.pgz.pssg] -> [foo.pgz] You can also specify your own filename and/or schema: Example: Convert PGZ to XML using Analysis Level 1 and Safe Mode with custom XML and schema: $ pssgConverter.py -a1 -s foo.pgz custom.xml custom.pschema [INFO] : Analysis Level [1] [INFO] : SafeMode Enabled Extracting [hea_huf_hybride_03.pgz] -> [hea_huf_hybride_03.pgz.pssg] Converting [hea_huf_hybride_03.pgz.pssg] -> [custom.xml] Example: Convert XML from previous example to PGZ: $ pssgConverter.py custom.xml newfile.pgz Converting [custom.xml] -> [newfile.pgz.pssg] using schemaSource [custom.pschema] Updating [newfile.pgz.pssg] -> [newfile.pgz] D. Details: Reads in PSSG and attempts to make an educated guess at the files schema. Stores schema to file as .pschema. Finally, converts the binary pssg file to XML based on the pschema. If a .pschema already exists, the script will use it as-is. This allows users to make adjustments on how chunks of binary data are displayed. The following types are supported by pschema files: TYPE Supported Aliases Size ----------- ------------------ ----- BYTE byte 1 Byte BOOLEAN boolean 1 Byte INT32 int 4 Bytes UINT32 uint 4 Bytes FLOAT float 4 Bytes SHORT short 2 Bytes USHORT ushort 2 Bytes STRING string, str, char[] N Bytes VECTOR vector 12 Bytes BYTEARRAY byte[] N Bytes INT32ARRAY int[], int32[] N*4 Bytes UINT32ARRAY uint[], uint32[] N*4 Bytes FLOATARRAY float[] N*4 Bytes SHORTARRAY short[] N*2 Bytes USHORTARRAY ushort[] N*2 Bytes Within the pschema file, if you assign a type, you can use a single string such as: "_Type" : "INT", You can also provide a type definition with an array of types: "_Type" : ["FLOAT","FLOAT",,"FLOAT","INT"], We could also define the Type above like so: "_Type" : ["FLOAT[3]","INT"], Bound Arrays: A bound array is an array with a finite size. With the exception of STRING, all types support bound arrays. Above, Float[3] is a bound array. It means 3 Floats. Unbound Arrays: All types except STRING, VECTOR and BOOLEAN support unbound arrays. For example: "byte[]". <- All (remaining) content would be displayed as bytes (hex). Unbound arrays can be used independantly or appear at the end of a type definition. If any other types follow an unbound array in a type definition, those types are ignored. STRING is considered an unbound array. You will need the .pschema used to decode the original PSSG to xml in order to convert the XML back to PSSG. This script will look for it in the same directory as the XML file. While you can always generate the pschema, any custom tweaks would be lost so make sure to backup any custom edits. Analysis Level: Option to attempt to guess at the structure of binary values that normally render as hex. How much time is spent guessing is determined by the level. Max (implemented) level is 1. SafeMode: If you see warnings when converting from XML back to PSSG then you need to re-extract the offending pssg to xml with this flag. It adds "_" to the names of all elements and attribtues to eliminate any issues when converting from XML back to PSSG For Even more detail on PSSG format, view the python file pssgConverter.py A breakdown of The schema is provided within. E. Batch Processing: Batch processing refers to processing an entire directory instead of a single file. This distro includes some WINDOWS BATCH files that help users who aren't comfortable with command line tools. Basically you throw pssgConverter.py into a directory and then also copy in one or both of the Batch Files: 1) pssgConvertToXmlBatch.cmd 2) pssgConvertFromXMLBatch.cmd You can then Double-Click the .cmd file in explorer and it will open a console and kick off, applying the pssgConverter.py script to any compatible files in the directory. The "toXML" version looks for a specific list of well known file extensions that are either compressed PSSG or uncompressed PSSG. The "fromXML" version simply looks for any and all XML files, backs up the original file and then re-writes the original. The fromXML version also performs a basic timestamp check and skips any files that look like they haven't been modified since creation. C:\path\to\datalocal>pssgConvertToXmlBatch_r.cmd If you happen to have console open, you can also run the batch files with flags that will get applied to all the files. So if you want Analysis Level 1 applied, xml file indention to be 2 and you prefer safeMode for all XML, you can run it like so: C:\path\to\datalocal\dir>pssgConvertToXmlBatch.cmd -i2 -a1 -s Also remember that you can run scripts easily from CYGWIN without needing the batch scripts. For example: $ for f in *.xml ; do ./pssgConverter.py $f; done There are also RECURSIVE versions of the batch files under the directory batch\recursive. These process the current directory AND subdirectories. So if you just wanted to extract EVERYTHING (keep in mind, this will more than double the amount of space things take on your computer), you could run the recurisive version from the root directory: And about... 8 hours later everything would be extracted (I'm not kidding) If you do decide to extract everything, I recommend installing Python 3 into CYGWIN, updating your python symbolic link to use it (See GreedFall Modding Tips) and run it from cygwin. I find the cygwin python 3 is about 1.5 times faster than the windows version and ~3X faster than python 2. F. Drag N Drop For users who do no like console, you can copy pssgConverterDragNDrop.cmd to the same directory as pssgConverter.py. If you drag any file onto pssgConverterDragNDrop.cmd, it will open a console and pass the filename on to pssgConverter.py. It also pauses once the command is complete so you can read any feedback incase there was an issue. G. Notes: - If you are using Python 32 bit, you may run into memory errors on some files (texture and level files can get big) - If you have CYGWIN, but you use the batch .cmd file, CYGWIN will kick off a windows CMD shell to run the file, which will use the windows version of Python you have installed. So if your windows version of python is 32 bit and your CYGWIN version is 64 bit, using the .cmd file may limit the memory available to the script. H. Compatibility with EGO Editor The XML is not compatible with EGO, but the PSSG is (assuming EGO can handle the PSSG). PSSG Converter labels all XML output with the version 2.3.0 header. EGO labels with 1.0.0. PSSG converter thus checks the version when you work with XML and will reject EGO exported XML files. ------------------------------------------------------------------------------- II. > > > > PSSG Extract DDS ------------------------------------------------------------------------------- A. Overview pssgExtractDDS.py decompresses .pgz/stxu/stxh files to .pssg, then extracts image data and slaps a .dds header on them so that other apps can open the file. .pgz : Low Res Texture (bad extension as it is used by other things) .stxh : High Res Texture .stxu : Ultra Res Texture B. Prerequisites: You will need to install Python 2.x or 3.x and use 7-zip to extract the .spk files to a directory (outside greedfall) to expose the games resources. Within you will find directories named "textures". Unlike pssgConverter, pssgExtractDDS is optimized to work with large files without memory issues (since I knew when I made it that I would be processing large texture files) C. Usage: pssgExtractDDS.py .pzg pssgExtractDDS.py .stxh pssgExtractDDS.py .stxu Flags: -h : detailed help + notes -d : show debug feedback -t : show trace feedback Example: Convert PGZ to .dds: pssgExtractDDS.py somefile_diff.pgz Output: ..x..dds') D. Details: The safest bet is to unpack all .spk files found under GreedFall/packs to a common directory outside the game folder. (You don't want to unpack within the game folder, or it may try to load everything as overrides a nd make the game unplayable). The most note-worthy package for texture developers is: datapchighres.spk As the name implies, it contains most of the "ultra" quality textures. They can also be found elsewhere, but a large number are stored here in a single location. I suspect these are the textures that never get loaded unless the user has selected Ultra-Settings in their graphics setup. Most users will want to edit the stxu.dds versions and then scale down and clobber the lower res versions for older hardware. E. Batch processing: Batch processing refers to processing an entire directory instead of a single file. This distro includes some WINDOWS BATCH files that help users who aren't comfortable with command line tools. Using Explorer, copy these two files to a textures folder within Greedfall pssgExtractDDS.py pssgExtractDDSBatch.cmd Then double click on pssgExtractDDSBatch.cmd This will convert all textures to DDS. The extracted filenames will look like: somefile.stxu.x..dds CYGWIN: You can also use the .cmd file under CYGWIN, however it will use the windows version of your python installation. If you wish to use the CYGWIN version of PYTHON in your batch command, you will need to do it the cygwin way: Copy pssgExtractDDS.py to the textures folder, then: $ cd /cygdrive/c/path/to/textures/') $ for f in * ; do pssgExtractDDS.py $f; done') F. Drag n Drop: For users who do no like console, you can copy pssgExtractDDSDragNDrop.cmd to the same directory as pssgExtractDDS.py. If you drag any file onto pssgExtractDDSDragNDrop.cmd, it will open a console and pass the filename on to pssgExtractDDS.py. It also pauses once the command is complete so you can read any feedback incase there was an issue. G. Notes: Latest Gimp (2.10.18) does not handle most the DDS files used by GreedFall however Blender3D does. You can also download NVIDIAs texture tools exporter here: https://developer.nvidia.com/nvidia-texture-tools-exporter Use pssgMergeDDS.py to convert back to PSSG ------------------------------------------------------------------------------- III. > > > > PSSG Merge DDS ------------------------------------------------------------------------------- A. Overview: pssgMergeDDS.py is a convenience script that injects DDS data into an existing PSSG file and then recompresses the file using the files original extension. The script assumes the input file STARTS WITH the name of the original file. So if the original file was: somefile.stxu Then the script would expect the DDS file to begin with those characters. If there is extra info between the .stxu and the .dds ... it is just ignored. B. Prerequisites: This script is designed to be used in conjuction with pssgExtractDDS.py C. Usage: pssgMergeDDS.py originalfile.pgz.dds') pssgMergeDDS.py original.stxh.dds') pssgMergeDDS.py original.stxu.dds') Flags: -h : detailed help + notes') -f : force update (even if dds file hasnt changed) -d : show debug feedback') -t : show trace feedback') Example: Convert somefile.stxu.1024x1024.dds back to somefile.stxu: pssgMergeDDS.py somefile.stxu.1024x1024.dds D. Details: See PSSG Extract DDS above. This is meant to be used in conjuction with that script. E. Batch processing: Batch processing refers to processing an entire directory instead of a single file. This distro includes some WINDOWS BATCH files that help users who aren't comfortable with command line tools. Using Explorer, copy these two files to a textures folder within Greedfall pssgMergeDDS.py pssgMergeDDSBatch.cmd Then double click on pssgMergeDDSBatch.cmd This will merge any changed textures back into their PSSG files. The original PSSG file will be backed up with a .bak extension if it does not exist. CYGWIN: You can also use the .cmd file under CYGWIN, however it will use the windows version of your python installation. If you wish to use the CYGWIN version of PYTHON in your batch command, you will need to do it the cygwin way: Copy pssgMergeDDS.py to the textures folder, then: $ cd /cygdrive/c/path/to/textures/ $ for f in *.dds ; do pssgMergeDDS.py $f; done F. Drag n Drop: For users who do no like console, you can copy pssgMergeDDSDragNDrop.cmd to the same directory as pssgMergeDDS.py. If you drag any file onto pssgMergeDDSDragNDrop.cmd, it will open a console and pass the filename on to pssgMergeDDS.py. It also pauses once the command is complete so you can read any feedback incase there was an issue. G. Notes: - This script INJECTS dds data back into the file it was extracted from. To do that it needs to know the source filename. Therefore, this script expects the filename to begin with the name of the source file and that the original file resides in the same directory. Changing compression types (BC1 to BC7 for example) has not been tested and may not work. To be safe, save DDS files with their original BCn (UNORM) settings. - Latest Gimp (2.10.18) does not handle the DDS files used by ') GreedFall. However Blender3D does. You can also download the') NVIDIAs texture tools exporter here: ') https://developer.nvidia.com/nvidia-texture-tools-exporter') - Use pssgExtractDDS.py to convert PSSG to .dds files') ------------------------------------------------------------------------------- IV. > > > > RESTORE BAK ------------------------------------------------------------------------------- In the Batch directory, there is a windows batch script called restoreBak.cmd This script was designed to scan the current directory for any .bak files and clobber their sources. foo.bak => clobbers foo foo.pgz.bak => clobbers foo.pgz foo.pgz.pssg.bak => clobbers foo.pgz.pssg I needed a script like this when developing the python scripts because I would often make mistakes and break lots of files. However, most users shouldn't need this (unless they are messing with the python code). None the less, here it is incase you need it. There is also a recursive version under batch/recursive/restoreBak_r.cmd that applies to the current directory AND drills into subdirectories. Use with caution as you are more likely to accidently revert something you didn't mean to with that one. ------------------------------------------------------------------------------- V. > > > > Version History ------------------------------------------------------------------------------- 1.00: Mixed Initial release. 1.01: Mixed Updated script to treat Colors as Floats instead of UINTS. Updated script so it doesn't open files in read/write mode (to avoid permission issues). Fixed bug where script excluded unused attributes when writing the pschema. This sometimes caused header errors when exporting XML back to PSSG. Fixed bug where DEBUG messages (-v) would show the wrong values for VECTORS and COLORS. Updated Audio Section of Modding Tips Added Misc -> Levels and Maps Section to Modding Tips. 1.02: Mixed Updated script to support c-style hex in binary XML tags, mostly to make it easier to paste exported binary data from HxD. Added section to Modding Tips to addresses Audio (music). How to change and add new music content. 2.00: All Script Complete re-write of the script so that users can define composite types in pschema files. Updated the code during the re-write to support both Python 2.x and 3.x. Added guessing logic to deduce likely structure of binary value chunks. 2.01: All Script Renamed the script from pssgExtracter to pssgConverter to clarify what the script was capable of Renamed the Mod on Nexus Mods to "PSSG Tools and GreedFall Modding Tips". PSSG is used by many games. Added python environment headers to script so it could be ran stand-alone. Updated documentation and examples to reflect the various name/usage changes. Added this readme to track changes. 2.02: All Script Added pssgExportTextures.py 2.03: Mixed Renamed pssgExportTextures.py to pssgExtractDDS.py Added pssgMergeDDS.py Updated Textures section of Modding Tips with walk through on editing textures. 2.04: Mixed pssgConverter: New custom XML parser (wont barf on bad PSSG xml attribute values) pssgConverter: Now supports compressed and uncompressed PSSG (no 7-zip necessary) pssgConverter: Encoding Speed enhancements (about 100x faster) pssgConverter: Better type guessing/analysis (fixed several edge case bugs). pssgConverter: New safe mode if you see type warnings. pssgConverter: Use '-p' to show progress on larger files (default with batch commands) pssgMergeDDS: Fixed write permission issues pssgMergeDDS: Encoding Speed is faster pssgOggExtract: Updated to be Python 3.x compatible moddingtips: Updated instructions to reflect new features. moddingtips: Added tip for searching the games .txt files (TymmeRMortis) moddingtips: Added some content to the items section added: restorebak script added: batch processing scripts added: dragNdrop scripts ------------------------------------------------------------------------------- VI. > > > > Contributing Authors ------------------------------------------------------------------------------- Dheu TymmeRMortis