---------------------------------------------------------------- ESP Scalpel - Nov 2013 Aaron Logue http://www.cryogenius.com/ ESP Scalpel is a fast and powerful tool for performing surgery on ESP file elements. It does this by loading an ESP file into memory as a linked list of groups. Command line switches can be used to select a specific group for expansion into subchains of records and subrecords. Records and subrecords that have been expanded can then be operated on using generic primitives such as extract and write to file, delete, insert, replace using previously extracted records, overwrite with string or binary literals, and so on. Changed block sizes are recalculated, compressed sections are recompressed, and new files can be written to (or old ones overwritten if one is confident or foolhardy or both). ESP Scalpel has very little knowledge about flags, fields or the values of specific records. It manipulates blocks of data at the record or subrecord level while handling decompression, recompression, and resizing of records and subrecords. ESP Scalpel and the portions of its source code written by me (see below for a clear description of the portions not written by me) is provided freely for use by others along with my sincere thanks for all of the time and effort that others have put into mods, documentation, tutorials, and tools that I have enjoyed so much. ESP Scalpel uses code from zlib 1.2.8 by Mark Adler and Jean-loup Gailly and available from http://zlib.net/ to handle ESP file compression. Source files from zlib128.zip used, without alteration and included in the scalpel_source directory purely for convenience by those wishing to recompile scalpel.exe are: crc32.h deflate.h gzguts.h inffast.h inffixed.h inflate.h inftrees.h trees.h zconf.h zlib.h zutil.h adler32.c crc32.c deflate.c infback.c inffast.c inflate.c inftrees.c trees.c zutil.c See the license in zlib.h and http://zlib.net/ for further information. ESP Scalpel comes without warranty or guarantee of any kind. Under no circumstances will the author be liable for any problems that may arise from its use. It is a sharp instrument, so if you are using the -w switch to write, keep your stuff backed up and know what you've told it to do before you hit Enter. ---------------------------------------------------------------- INSTALLATION Unzip the contents of the data directory - scalpel.exe, scalpel.dat, and if you'd like to check out the demonstration, ethel.bat - to your skyrim\data directory. Unzip the text files wherever you would like to read them from. ESP Scalpel is a command-line tool that you run repeatedly from a C:\...> command prompt, so you may also want to make a shortcut to a command prompt for convenience. To do that, right-click on your desktop, select New, then Shortcut, enter or browse to C:\windows\system32\cmd.exe, then re-edit the properties of your cmd.exe icon and change "Start in:" to C:\games\steam\steamapps\common\skyrim\data or whatever the correct path on your machine is. Click on the cmd.exe shortcut that you created, and you should get a command window with a C:\yaddayaddayadda\skyrim\data> prompt. ESP Scalpel does not need to be in skyrim\data. If you'd like to play around with it separate from your Skyrim install, what you can do instead is copy scalpel.exe, scalpel.dat, ethel.bat, and the text files into the directory of your choice and copy skyrim.esm from your skyrim\data directory to there as well. If using Windows Explorer, be sure to use ctrl-drag or copy and paste so that you copy it rather than move it. scalpel.exe does not care what its own name is, so you could copy it to s.exe if.. lol, that wasn't intentional, I swear.. you were using it and wanted to type less. UNINSTALLATION ESP Scalpel doesn't create registry entries or write to My Documents or anything like that, so all you need to do is delete whatever you unzipped. That was most likely scalpel.exe and scalpel.dat at the very least, but probably scalpel_readme.txt, ethel.txt, ethel.bat, and the scalpel_source directory as well. The whole shebang is only a couple hundred K, so you could also just stop using it and not worry about deleting it, but if you created the decomp.dat file - the decompressed version of skyrim.esm to do searches with - deleting that will free about 350M. ---------------------------------------------------------------- GETTING STARTED If you are brand-new to command line things but interested, read on, because one of my goals is to make this work as an introduction to command line utilities. If you're brand-new to the ESP file format as well - as I was when I started writing this thing - you've got some work ahead of you, but I muddled through it so you can too. If you're an old hand at this stuff, run ESP Scalpel for a Usage: message and have at it. ESP Scalpel is read-only without the -w switch, so as long as -w is not used, it's safe to play with. Follow the installation instructions above and get to the point where you can open up a command window to a C:\...\skyrim\data> prompt. Commands are entered simply by typing them and pressing the Enter key. The first command to know is "exit". That'll close the window. Getting in the habit of exiting that way instead of hitting the little red X will make sure that you never accidently abort a task that hadn't completed yet. The term "folder" can get off our lawns now, because they were "directories" before that. The next command to know is "dir", which will give you a list of files in your current directory. Like many command line commands, you can make dir do lots of other stuff by adding parameters and switches. Type "dir *.exe" and hit enter. The * is a wildcard, so that command will show you all of the files that end in ".exe" in the skyrim data directory. You should see scalpel.exe there. The third thing to know is that the up arrow will recall your last command. This is slightly broken in cmd.exe and sometimes it's the down arrow instead, but even so, it's a feature that endows command line users with the ability to get stuff done fast. Scalpel's command line parameters and switches have been designed with the up arrow in mind. The next things to know are "help", "help ", and that adding "/?" to the end of many commands will cause them to display a list of additional switches and parameters that they support. Type "dir /?" or "help dir" and it will show you that "dir /od" will sort a directory of files by their timestamps. "Help" will give you a list of other commands. Ones that I use a lot include findstr, cd, copy, and rename. I also use erase a lot, but save that one until you feel confident about it, because it erases things. "dir scalpel.*" should show you scalpel.exe and scalpel.dat. "dir skyrim.esm" should show you that skyrim.esm is there, and that it's roughly 250MB in size. Type "scalpel" and press Enter to run it without any parameters. It will print a usage message. Like many command line things, scalpel has a simple syntax. The less-than/greater-than brackets around parameters mean that they are required. Square brackets mean that they are optional. Scalpel's switches are prefixed with a dash, which is slightly different from the native MS-DOS commands which use a forward slash to prefix switches. (eg: "dir /?" versus "scalpel -?") The dash prefixing of switches is the Linux, Mac OS, and multiplatform command line way of doing things. The forward slash prefixing is the Microsoft way. Scalpel requires an input file. It does not care about the extension (the part after the ".") but only that the file contains some groups, records, or subrecords for it to manipulate. Type "scalpel skyrim.esm" and press Enter. 119 lines of data should be displayed, prefixed with A0000 through A0118. You can select a block to look at in greater detail with the -b switch. Press the up arrow key to recall "scalpel skyrim.esm" and then add "-b A0011" or "-ba11" to select the EYES group. It's not picky about the leading zeros. "scalpel skyrim.esm -ba11" and Enter will display the contents of the A0011 block. Now let's select the blue eyes. Again, up arrow to recall the last command and add "b7" to select the particular EYES record with "scalpel skyrim.esm -ba11b7". Finally, select the ICON subrecord for the selected EYES record with "scalpel skyrim.esm -ba11b7c2" and press Enter. You should see: C0002: ICON 0x00000022 / 34 bytes 00000000: 4163746f 72735c43 68617261 63746572 |Actors\Character| 00000010: 5c457965 735c4579 65426c75 652e6464 |\Eyes\EyeBlue.dd| 00000020: 7300 |s. | Now, try changing the selected subrecord by adding "-lhonk.dds" to get "scalpel skyrim.esm -ba11b7c2 -lhonk.dds". Don't worry: no changes to disk files will be made without the -w switch. You should see: C0002: ICON 0x00000008 / 8 bytes 00000000: 686f6e6b 2e646473 |honk.dds | Without the -w switch, ESP Scalpel displays the change that it has made to the subrecord in memory and then throws the change away without writing anything. That gives you a chance to see if that's really what you want to write to a file or not. In this case, you would probably wind up redoing it to add a "Actors\Chara.." path, and then again with doubled backslashes to get the backslashes inserted (see the section on LITERALS below for what that's about) and again to add the 0x00 NULL terminator to the string. Let's stop there though, because you don't want to be changing skyrim.esm anyway. What you just accomplished, however, was that you learned how to specify an input file, how to use the -b switch to select a group, record, or subrecord, and then once you have what you want selected, you can add a switch to perform an operation on that group, record, or subrecord. Finally, if you don't like what you see, you can just abandon it with no harm done. You saw that a typical way of working with ESP Scalpel is to hit the up arrow, add to the parameters or switches that you've already got, hit Enter to see the next result, and repeat until you like what you see. Try exploring some ESP files that are in your data directory, or if you'd like, check out... ---------------------------------------------------------------- The ETHEL.BAT Demo If you unzipped ethel.bat into your data directory, you can run it with the command "ethel". A bunch of stuff should scroll by for a few seconds as scalpel is run multiple times to extract bits and pieces of skyrim.esm and use them to construct ethel.esp. After it completes, you can list its contents with "scalpel ethel.esp" A0000: TES4 0x00000059 / 89 bytes A0001: GRUP NPC_ 0x0000020d / 525 bytes A0002: GRUP CELL 0x000001ab / 427 bytes I have included a file, ethel.txt, that documents my thought processes as I went about coming up with the commands that I used to construct ethel.esp. You can experiment with ethel.esp and try writing to it with -w ethel.esp. If it gets damaged beyond repair, you can just regenerate it by running ethel.bat again. An experiment might be to see if you can replace Ethel's iron dagger with something else. scalpel decomp.dat -tz -m OrcishDaggerShock could be used to begin the hunt for the ID of a more powerful replacement. An NPC's inventory begins with the COCT subrecord, which is a count of how many CNTO subrecords follow it. The CNTO subrecords are an item ID and a count, so if you find an item ID to use, you should be able to replace the ID in an existing CNTO record, or add a new CNTO record with the ID you want to add to the NPC's inventory. The -o switch works well to overwrite 4-byte ID values, and the -y switch works to insert new subrecords. For example, gold has an ID of 0000000f, so this should add 50 gold to Ethel's inventory: scalpel ethel.esp -ba1b0c8 -y CNTO0x08000f00000032000000 -w ethel.esp Annnnd that didn't work... until I remembered COCT and bumped the count: scalpel ethel.esp -ba1b0c6 -o 0x03 -w ethel.esp Annnnd it still didn't work because Ethel's CNTO d1461000 "Citizen Loot" item was replacing the 50 gold with a small random amount of gold. Swapping the order of the CNTOs didn't fix it, so whatever fills in the loot and clobbers existing inventory numbers must do that after all of the inventory slots are filled. I had to delete the loot item and put the COCT record count back before Ethel had 50 gold: scalpel ethel.esp -ba1b0c8 -d -w ethel.esp scalpel ethel.esp -ba1b0c6 -o 0x02 -w ethel.esp For me, the above flailing about is typical of Skyrim modding regardless of the tools being used. I tend to spend the majority of my time trying to understand why something that seemed reasonable to expect to work did not work, then after searching in vain for documentation, experimenting until an adequate workaround is found. ESP Scalpel was such an experiment as I struggled to try to get the CK to add one door triangle to an exterior cell without needlessly modifying the navmeshes of the surrounding cells. If you are hacking on ethel.esp, I recommend avoiding loading it up in-game and making a Save file with a character that you care about and then continuing to play long-term on that Save thread. The reason is that, while ESP Scalpel will certainly allow one to scramble an ESP file catastrophically by deleting the wrong GRUP or loading an entire CELL into the middle of some poor NPC's eyeball, it will also allow one to do something like forget to change an ID in a copied record and corrupt an ESP in a subtle way that will appear to work for awhile but fail later for reasons that are not obvious. ---------------------------------------------------------------- SELECTING GROUPS, RECORDS, and SUBRECORDS In general, the top level of an ESP file consists of a collection of groups. ESP Scalpel lists them as blocks of data labelled A0000, A0001, etc. Within any particular group may be one or more records, which ESP Scalpel will list as B0000, B0001, etc, but only if it has been asked to descend into that group with the -b switch. The selection of a group might look like "-b A0042" or "-ba42". The -b switch isn't picky about the space or leading zeros. Selecting a record is done by adding it after the group. The selection of a record might look like "-b A0042B0555" or "-ba42b555". Selecting a record will cause any subrecords that it contains to be listed as C0000, C0001, etc, and selecting one of those subrecords is done as above, by adding it after the record. Selecting a subrecord might look like "-ba42b555c0". A selected record must be preceded by a selected group, and a selected subrecord must be preceded by a selected record. ESP Scalpel will work with headerless ESP file fragments, labelling whatever structures it finds at the top level as A0000, A0001, etc. This is useful because it also supports extraction of fragments to standalone files and import of fragments from files. By default, the selected block is displayed as a hexdump, with a list of any subblocks beneath. The hexdump display can be modified with the -t and -q switches, and the block list display can be modified with the -f and -z switches. The -p switch will cause parent blocks that are normally hidden when drilling down to be included. In addition, the -p switch can be used to cause parent records to be included in lists, hexdumps, or fragment extracts when they would normally not be as a result of having selected a subblock. For an example, compare the outputs of "scalpel ethel.esp -pt -ba2b2c1" with and without the -p switch. ESP Scalpel relies on the external file scalpel.dat to tell it which codes are groups, records, or subrecords. It matters because groups and records are followed by 4-byte lengths while subrecords are followed by 2-byte lengths, along with a few other differences. scalpel.dat is just a text file, editable with notepad or whatever. Hopefully there will be no need ever to change it, but I did it as an external file in case I missed some codes or got some wrong. Some records are compressed, and are loaded into memory but left compressed unless selected with the -b switch. Compressed or decompressed records are shown as such. By default, records that were decompressed as a result of being selected will be recompressed upon being written. This can be overridden with the -n switch. ---------------------------------------------------------------- LITERALS Four of scalpel's switches take a as a parameter. A literal is a string of characters without any intervening spaces that can be used to specify an arbitrary sequence of bytes. "0x" will cause the parser to switch from character string to hexadecimal mode, and "\" (or encountering a non-hexadecimal character) will cause the parser to switch from hexadecimal mode back to character string mode. Thus, "The Bannered Mare" with two embedded spaces and a NULL terminator could be specified as "The0x20\Bannered0x20Mare0x00" and a TES4 header's MAST subrecord might be inserted with "-ba0b1 -y MAST0x0B00Skyrim.esm0x00" The "\" character also works to escape the "0x" sequence and the "\" itself, so if you need "0x00" entered as a 4-character string as opposed to a single byte with a value of zero, you can use "\0x00". If you need a single backslash, which commonly occur in DDS texture filename paths, use "\\". A single byte is 8 bits, and each hexadecimal character is 4 bits, so a byte requires 2 hexadecimal characters. When you're in 0x hexadecimal mode, remember to always enter hex digits in pairs. When mixing hex and strings, remember that "abcdef" are valid hex digits, so you'll need to switch out of hex mode with \ if the string begins with one of those letters. 0x20 is a space, so "Bannered" was a good example of this. Without the \ to exit hex mode and interpret the B literally, the "Ba" would have been interpreted as a single byte in hex. Scalpel's hex interpreter is not case-sensitive. The Intel or AMD processor in your PC stores numbers as "big endian" (as opposed to Motorola processors, which are "little endian". What that means is that things that are multiple-byte values, like Form IDs, will appear in reverse order, with their least significant byte first and their most significant byte last. If you are searching for Ysolda's ID 00013BAB, for example, you will need to do this: scalpel skyrim.esm -ba42 -tz -m 0xAB3B0100 because that is how the ID is stored in Ysolda's NPC_ record: scalpel skyrim.esm -ba42b4789 -tz B4789: NPC_ 0x00000449 / 1097 bytes (decompressed) 00000000: 00000000 ab3b0100 1b695500 28000c00 |.....;...iU.(...| 00000010: 45444944 07005973 6f6c6461 00564d41 |EDID..Ysolda.VMA| ... Here are some examples: "Ethel" : "Ethel0x00" (don't forget to null-terminate your strings!) "actor\swamp zombie\foo.dds" : "actor\\swamp0x20\zombie\\foo.dds0x00" "0x0x0\DethBlyd/0x0x0" : "\0x\0x0\\DethBlyd/\0x\0x\00x00" ---------------------------------------------------------------- SEARCHING Searching large files like skyrim.esm is complicated by the fact that ESP and ESM files can contain multiple individually-compressed records. While the -m switch is used to search for a string or binary value as described above, it does not by default decompress records to search them. This is often not a problem for Form IDs of cells or NPCs, which tend to be found in uncompressed record headers, but additional steps must be taken to search for strings. For ESP files, which tend to be much smaller than skyrim.esm, an easy solution is to use the -e switch to decompress all of the records in the file before searching them: scalpel ethel.esp -ezpm Ethel The -z switch inhibits display of records that did not match, and the -p switch forces parent records to be considered to have matched if a subrecord matched. That allows you to see which A0000,B0000 etc records to specify with the -b switch on a subsequent command to select the record that matched. Another approach is to use -e with the -n and -w switches to expand everything and write it to a decompressed version of the file, which is then used for fast searching even of large files like skyrim.esm. This is described in more detail in ethel.txt, in which the command scalpel skyrim.esm -eqzn -w decomp.dat is used to create a decompressed version of skyrim.esm. Here are some examples of searching for the IDs and records of NPCs, places, and items in decomp.dat and ethel.esp for you to try: scalpel ethel.esp -etzp -m Ethel (Shows you that -ba1b0 will display Ethel's NPC_ record) scalpel ethel.esp -ba1b0 (Display's Ethel's NPC_ record, along with a list of subrecords. If you know what COCT and CNTO records are, then you see that -ba1b0c7 and c8 will show you the item IDs of what's in Ethels' inventory.) scalpel ethel.esp -etzpm Banner (Shows you that -ba2b2 will display the Bannered Mare's CELL record and give you its ID, 0001605E) When working with IDs, remember that the PC is a "big endian" machine, meaning that you'll constantly be having to reverse bytes in your head. Don't worry.. it's not that bad, especially with Skyrim IDs which have certain characteristics that you'll start to recognize. If you see 5e600100 where the ID is supposed to be in a hexdump, read it as 0001605E. If you want to search for that ID, use 0x5e600100 as the -m literal value to search for since that's the order it is stored in memory and on disk. Beware that searches are case sensitive, so "ysolda" has no matches in decomp.dat, while "Ysolda" has several. scalpel decomp.dat -m Ysolda (list all of the groups containing Ysolda. Shows you that -ba42 is the NPC_ group) scalpel decomp.dat -zm Ysolda (don't display the groups she's not in) scalpel decomp.dat -ba42 -tzm Ysolda (add -t to print only the first 8 lines of the 4-megabyte A42 NPC_ group. Shows you that -ba42b4789 is Ysolda) scalpel decomp.dat -zm SleepingGiant (Same basic search, only for a place) scalpel decomp.dat -ba58 -tzm SleepingGiant (finds B29621) scalpel decomp.dat -ba58b29621 (selects the Sleeping Giant Inn cell) scalpel decomp.dat -ba58 -tzm DraugrMelee (finds 20 candidates for my wife's swamp zombies project) ---------------------------------------------------------------- DELETING Selected groups, records, or subrecords can be deleted with the -d switch. Without the -w switch, the -d switch displays a "Would delete ..." message with the selected block and any subblocks removed from the block list display. The sizes of structures that change as a result of a delete (for example, the GRUP NPC_ becoming smaller because it contained multiple NPCs and one was deleted) are updated. Deleting a group or record will not automatically cause the TES4 HEDR group+record count to be updated. A -u update will need to be done, but it can be done at any point; given on the same command line along with -d, or done later as a standalone operation. ESP Scalpel reassigns its A0000, B0000, etc labels to groups, records, and subrecords every time it is run. That means that if, for example, you wish to delete two consecutive records, the -b block selection may be same each time. For example, suppose you wished to delete two records that happened to have ESP Scalpel block labels B0007 and B0008 and you deleted B0007 first. The record that was B0008 would become B0007, so the second delete would need to delete B0007 rather than B0008. If it deleted B0007 and then B0008, it would actually be deleting whatever was B0009 at the time that B0007 was deleted. This isn't an issue if the higher numbered blocks are deleted first. You can see this being done both ways in ethel.bat. ---------------------------------------------------------------- EXTRACTING FRAGMENTS Selected groups, records, or subrecords can be extracted as an ESP fragment with the -x switch. An ESP file fragment created with the -w switch can be edited with ESP Scalpel even though it is not a complete ESP file. In fact, ESP Scalpel is intended for use with incomplete or work-in-progress ESP files. Selected records that were originally compressed will be recompressed before writing, but this can be overridden with the -n switch. The -p switch can be added to cause the parents of the selected block to be included in the output file. This is generally useful only to grab the parent GRUP header of a record when you are going to insert that record in an ESP that you're working on that does not yet have a GRUP header to contain that type of record. Recompression of records is inhibited by the use of -p (compressed records with single selected subrecords are pointless), but records can be compressed later with -c if desired. ---------------------------------------------------------------- IMPORTING FRAGMENTS The -i and -r switches accept filenames of ESP fragments to import. -i will insert the file's contents after the block selected with -b, and -r will replace the selected block with the file's contents. Structures that change size as a result of The -s switch is a standalone operation to swap the selected block with the block following it, but it can also be used in conjunction with -i to cause -i to effectively insert before the selected block rather than after it. ---------------------------------------------------------------- EDITING WITH LITERALS The -y and -l switches are analogous to -i and -r, with the primary difference being that they accept literals as command line parameters as the data to be inserted after the selected block or used to replace the selected block with, respectively. While it's technically possible to enter an entire GRUP or record with -y or -l, these switches are intended for entering subrecord data. The -s swap switch works with -y to cause it to insert before a selected record in the same way that it works with -i. Sizes of structures that change as a result of an insert or replace will be updated. The literal accompanying a -y or -l switch should start with the 4-character code, and must be followed by the length bytes, which are dummies and can be any value. They will be replaced with the actual length of the data that follows. Note that the -l switch can thus be used to change a selected subrecord code to a different code. The code is specified with -l, even though it is replacing an existing code. The -o data overwrite switch expects only the group, record, or subrecord data as the literal, and not a code or length as with the -y and -l switches. It overwrites starting at the beginning of a block's data up to the end of the block. It cannot change the size of a structure. A common use of -o is to change something's Form ID, which typically occurs 4 bytes into a record rather than at the beginning of it. The four bytes to be skipped must be provided and are not dummies - they will overwrite whatever is there on the way to what you want to change. ---------------------------------------------------------------- WRITING Without the -w switch, ESP Scalpel is read-only, performing all of its operations on linked lists in memory and displaying the results, but terminating without performing a write operation or changing anything on disk. The -w switch takes a filename. It can be a new file or it can be the name of a file to overwrite - including the input file. When using ESP Scalpel interactively, a common way to use it is to see what it's going to do without the -w switch before adding it. After a time, you might get comfortable enough with it to hit enter on scalpel myfile.esp -ba4b7c3 -y FULL0x0000New0x20Name0x00 -w myfile.esp without bothering to see what it looks like without the -w switch first, but I'm not there yet and I wrote the thing. :) ---------------------------------------------------------------- May your hard drive always spin up and all your mods be successful.