ArchiveInvalidation

Everything You Never Wanted to Know And Were Afraid to Ask

ArchiveInvalidation Destroyed
ArchiveInvalidation Explained

ArchiveInvalidation Destroyed

by devakm
Updated 1-29-2008

All of the original threads on the Elder Scrolls Forums have now expired and been purged from the system, but this summary of the great work done by the ArchiveInvalidation Explained Research Team lives on! The recent discovery of an entirely new method is now described as well.

Overview

Oblivion normally gets all of its media assets (artwork, spoken dialog, music, etc.) from within a few large .bsa files rather than in individual folders (such as Meshes, Textures, Sound, etc.). The original game and official plugins use these .bsa files (it stands for Bethesda Softworks Archive). These .bsa files are organized internally just like your "Data" folder is.

Although fan-made mods are starting to use .bsa files more often, most mods still place individual meshes, textures, etc. into folders within your "Data" directory. This isn't a problem for new items added by a mod, but it can be a problem for some "replacer" mods that alter original game items. For example, some of the most popular types of mods are "armor texture replacers" and "landscape texture replacer" mods that improve the look of the game by replacing the original "stock" textures.

Normally, this isn't a problem since Oblivion is designed to automatically load any meshes, textures, etc. that it finds in your Data folder, as long as the timestamp (Modification Date) on each of these files is more recent than the timestamp on the original .bsa files. Since the .bsa files are older, items in them get replaced by any newer items with the same name that exist in your Data folders.

In some cases, however, this doesn't work correctly. It now seems fairly certain that this is caused by a bug in Oblivion's ArchiveInvalidation system itself. After more than a year and several official game patches, it seems unlikely that Bethesda will ever correct this bug. This means you will have to use one of several workarounds if you want to install any "replacer" mods.

Further details on this subject, as well as extensive testing results for ArchiveInvalidation problems in general, can be found in the article ArchiveInvalidation Explained (the ESF threads on the topic have expired).

Each of the known workaround solutions has advantages and disadvantages. The first two methods in the list below -- BSA-alteration and BSA-redirection -- are the only recommended solutions. The last two methods -- BSA-invalidation and BSA-extraction -- have such serious drawbacks that they are not recommended solutions. Here's the list of methods along with some description of what's involved in each:

  • BSA-alteration: This was the first true solution to the problem. It is widely used, very reliable, fairly simple, and most Oblivion mod users are by now familiar with using it. The disadvantages are that you must use one of several external tools (not a big deal since you probably need one of these tools anyway), you must update the BSA changes after installing or removing any replacer mods (although most of the tools will do this in an automated way), and you must remember to remove BSA changes before applying any of the official patches from Bethesda.
  • BSA-redirection: This is a newer solution that has proven to be very effective. This method has advantages that make it preferable over BSA-alteration. The biggest advantage is that it does not require any external tools (although it does require you to install some files and you may want to use a tool to help you configure it) and once it's set up, you can pretty much forget about it (no intervention is required after adding or removing any replacer mods).
  • BSA-invalidation: This was the first solution discovered, and as a result was the most widely used method for a long time. However, it has severe limitations and problems and is rarely used anymore because of this. Unfortunately, many early mods still recommend this solution, so lots of users get fooled into thinking it's a good idea. In fact, it's a terrible idea under most circumstances and the only way to make it work reliably is to also use BSA-extraction (see below). You should avoid this method if at all possible.
  • BSA-extraction: This solution works well for mod creators (who often need to have the source files extracted from the BSAs anyway). However, it has major disadvantages for mod players and because of this it is not discussed in any great detail here.

Next we'll examine these methods in a bit more detail.

BSA-redirection

This method works by tricking the Oblivion game engine into using its bugged ArchiveInvalidation method on the wrong BSA file. By doing this, it allows all files in the main Textures BSA to be reliably superseded by any more recent files found in your Textures folder.

The easiest way to use this method is to use the BSA-redirection option in Oblivion Mod Manager if you already have it (see BSA-alteration below for more details). Here are the steps using OBMM:

  1. Start OBMM.
  2. Click Utilities.
  3. Select Archive invalidation. You'll get a popup window.
  4. Click BSA redirection (it's the default choice).
  5. Click Update Now.
  6. Close the Archive invalidation popup (click the red X in the upper-left corner).
  7. Quit OBMM or click Launch Oblivion.

Alternately, you can install a mod specifically designed to do BSA-redirection, such as ArchiveInvalidation Invalidated!.

You can also set up BSA-redirection fairly easily yourself if you don't mind a bit of tweaking. In the Oblivion.ini file, there is a configuration entry called the SArchiveList. It lists all of the standard BSA files. It normally looks like this:

SArchiveList=Oblivion - Meshes.bsa, Oblivion - Textures - Compressed.bsa, Oblivion - Sounds.bsa,
Oblivion - Voices1.bsa, Oblivion - Voices2.bsa, Oblivion - Misc.bsa

By changing the list to include another BSA containing at least one texture, you can trick the game engine into processing the bugged ArchiveInvalidation logic for that new dummy file rather than the real Textures BSA (called Oblivion - Textures - Compressed.bsa). So, the modified SArchiveList should look something like this:

SArchiveList=DUMMYFILE.bsa, Oblivion - Meshes.bsa, Oblivion - Textures - Compressed.bsa,
Oblivion - Sounds.bsa, Oblivion - Voices1.bsa, Oblivion - Voices2.bsa, Oblivion - Misc.bsa

To do this, you'll need to either use a BSA file that came with some other mod to be your DUMMYFILE.bsa (any BSA file containing at least one texture should work), create one yourself using a tool like BSA Commander (you only need to put one texture in the BSA).

BSA-alteration

BSA-alteration requires Wrye Bash (as of version 0.60), Vortex, or one of Timeslip's excellent utility programs: Oblivion Mod Manager (OBMM version 0.7.10 or later) or BSA Patcher. What Wrye Bash, OBMM and BSA Patcher do is to sidestep the ArchiveInvalidation problem by making Oblivion think it never had a copy of the textures you are replacing.

In other words, these utilities edit your Textures BSA so that Oblivion cannot find the original version of texture files you have replaced, thus forcing the game to load the replacements instead of the originals. Without getting too technical, this works much like taking a book and scrambling some entries from the table of contents so those chapters can't be found anymore, even though the chapters are really still there -- thus allowing the process to be easily reversed later by restoring the table of contents back to its original state.

This method of invalidation was initially tested by the ArchiveInvalidation Explained Research Team and has now been adopted as the de facto standard solution by the Oblivion mods community. It has proven extremely stable, but you should still make a backup of your BSA files if you have the hard drive space (or a DVD-burner) just in case something goes wrong (like you accidentally try to run both OBMM and BSA Patcher on your BSA files, which will result in a corrupt file).

OBMM and BSA Patcher will keep track of which files have been renamed in your BSAs and provide a "Remove BSA edits" function to rollback any changes they have made to your BSA.

Wrye Bash

Download Wrye Bash.

Note: The BSA-alteration features in Wrye Bash are designed for advanced users and provide more control over the process than OBMM's automated approach.

Technical Details

The ArchiveInvalidation problems described here are related to the way Oblivion invalidates entries in its BSA hash tables. In the Oblivion.ini, these are named:

A "hash table" is a very common method of improving performance when you've got a lot of things to keep track of. A directory structure full of files is a perfect candidate for this technology.

Unfortunately, Oblivion's invalidation routines don't seem to be very accurate about how they mark a file as needing to be replaced. It appears there is a "disconnect" in the intersection between these two tables that causes a file invalidated in one path to also get invalidated in all other paths where it exists, resulting in index "collisions" that cause Oblivion to mistakenly think a texture should not be loaded from the BSA.

Common Filename Problem: When multiple armor sets use files with the same names (like "shield.dds", "cuirass.dds", etc.) in different directories, invalidating files for one armor set can inadvertently invalidate those same filenames in other armor sets, causing missing textures.

Tools

Ultimately, the only really good solution is to use BSA-redirection or BSA-alteration, but if you really want to see for yourself then these tools will help you generate an ArchiveInvalidation.txt file.

Further Reading

Glossary

AI.txt is shorthand for ArchiveInvalidation.txt.

Credits

Man of the Year award goes to Timeslip for solving the whole AI mess by avoiding it entirely!

Aside from that, there's a very big list of very talented and somewhat crazy people who put a lot of hard work into figuring out the AI mess. The list includes:

Thank you all for this heroic effort!

ArchiveInvalidation Explained

by dev_akm

Preamble
This is adapted from my ESF thread: ArchiveInvalidation Explained (2). Unfortunately, like all the previous ESF threads on this subject, that thread has now expired and been purged from the forum. The summary lives on here.

Last Updated: 10-13-2006 11:40 AM CDT and 1/10/2026

OVERVIEW

There is a terrible beast wreaking mass destruction and chaos in the mod community. It is called ArchiveInvalidation. The purpose of our mission here was to slay this horrid creature and prevent it from respawning.

This goal has now been accomplished, but historians (and a few others, perhaps) may still wonder: From whence did this beast arise? What spurred it forth from its dank lair and into our peaceful community? The answers lie within this document and its related discussion threads.

The problems with ArchiveInvalidation described here most often appear right now for people using more than one armor texture replacer, but it could crop up anywhere Oblivion uses common filenames like "boots.dds" for more than one set of things. If you have not experienced any problems yet, it is probably because your configuration does not meet these criteria.

The extensive testing carried out by multiple users and reported in these threads makes it fairly clear that Bethesda probably never intended for anyone to have to use ArchiveInvalidation in the first place. The vast majority of textures seem to load correctly from the Data folders even without using an ArchiveInvalidation file, which explains why so many users get duped into believing that a simple three-line global/universal/generic ArchiveInvalidation has solved all their problems when in reality it has been proven to have no effect whatsoever (it's the same as having no AI file at all).

In fact, we are proposing that the only reason it's necessary to use ArchiveInvalidation at all is due to a bug in the game that causes some texture replacements to fail to load, while others load just fine. Based largely on Malifrax's contributions, we've isolated these problems to certain common directory names and perhaps even certain NIF files (meshes) that appear to have a different structure than ones which load textures correctly.

Note that you don't need to read all of this if you are using OBMM 0.7.8 or later and you select the option to "Directly Edit BSAs". Timeslip's workaround avoids the need for ArchiveInvalidation entirely in our tests.

If, on the other hand, you do intend to use ArchiveInvalidation, or just want to understand what led to such drastic measures as Timeslip's solution, then you do need to read this.

Hopefully, Bethesda will issue a patch for the textures bug at some point and all of this will become an odd historical footnote in the life of Oblivion. Until then, read on ...

ARCHIVEINVALIDATION TEST RESULTS

1) ArchiveInvalidation.txt (AI.txt) is only needed for replacement textures. Also, it is most often required for replacement textures that are specified from within a mesh (NIF) rather than specified in the CS (icons and landscape textures, for example, will usually load without being referenced in AI.txt).

2) Universal/global/generic ArchiveInvalidation methods do not work. Period. End of discussion.

3) Mesh replacers load regardless of whether they are specified in your ArchiveInvalidation.txt (AI.txt), so you don't need to list any .NIF files. If you do list them, try to only list the specific meshes that are giving you problems, and be aware of the ramifications, as described in #4 and #5 below.

4) A single entry of a common filename like "boots.dds" in AI.txt will invalidate all paths containing a "boots.dds" file that exist in both the BSA and your Data folder. It will do this regardless of whether those paths in your Data folder actually contain a "boots.dds" file and regardless of whether you have an AI.txt entry for that path. *

5) Listing a path in your AI.txt has the same effect as listing individual textures, in the sense that it causes all the items in that path to be invalidated. However, once you've listed one such folder, Oblivion will automatically invalidate all further instances of any common filenames it found in that first folder. *

If all this sounds really confusing, you're not alone. It's not easy to understand, even for computer experts, in part because it's such a crazy way of doing things that it seems in many ways like a bug in the invalidation code Bethesda implemented. Perhaps it is a bug, perhaps it is by design, we have no way of knowing.

* Credit for this discovery goes to Wonder Dog.

HOW DO I MAKE ONE?

Instructions can be found in the Oblivion Mods FAQ.

Once you get it started, you can list either just individual filenames, such as "boots.dds", one per line, or list entire directory paths, one per line, like this:

textures/armor/Iron/ \s
textures/armor/glass/ \s

In the end, I think discussions about which approach is better - listing files with no path versus listing only paths - probably won't get us anywhere, since both methods are highly imprecise and cause similar errors. Pick one and stick with it or use a mixture of approaches. I don't have enough data yet to make a recommendation one way or the other.

WHAT NOW?

This is a terrible situation any way you look at it. You can extract your textures BSA into your Data folder. That will solve most of the problems, but really it's a terrible solution. If you do this, be careful not to overwrite all the mod files you've already installed. You've been warned.

Probably the only halfway reasonable approach to even begin dealing with this problem is to have a tool like ACV, OBMM, or Wrye Bash check to make sure all replaced common filenames exist in all replaced paths and warn you when this isn't the case.

I think the best we can hope for is a combination of these approaches.

1) Modders creating texture replacers for things with common names in the BSA need to package all the related files with their mod, as many are already doing; if they don't, then their mods would be subject to warnings or blocks from utilities like OBMM and ACV.

This only applies if you are making a texture replacer that affects files with common filenames like "boots.dds", "cuirass.dds", etc. In this case, you must package all of the original textures that you did not change in the directory where the textures reside, such as "textures\armor\iron", along with the ones you did change. Otherwise, the ones you did not change in that directory path will be broken on the user's system.

We've seen this over and over again with mod-makers releasing a 1.0 version of a replacer that only includes their changed files, then (after getting deluged with hate-mail) having to follow up with a 1.1 version that includes all the items in that set regardless of whether they changed the files.

Another solution would be for modders to avoid creating texture-only replacers whenever possible. You can achieve the same result in many cases by altering the texture references in an item's mesh (using NifSkope) so it points to a new directory structure with texture files that don't use the same common filenames. Since meshes are loaded automatically without any need for entries in AI.txt, this avoids the problem entirely.

2) Utilities like OBMM might warn users before they install a mod that it may cause problems, assuming the user takes advantage of the omod format; then the user can decide whether they want to ignore the warning or stop the installation process.

3) Utilities like ACV might warn users after they have installed a mod that they are likely to have problems with certain files (like it already does for meshes); the user can then decide whether they want to ignore the warning or go back and uninstall the mod.

4) Utilities like Wrye Bash might provide some combination of #2 and #3, perhaps with more advanced users in mind.

TECHNICAL DETAILS

The ArchiveInvalidation problems described here are related to the way Oblivion invalidates entries in its BSA hash tables. In the Oblivion.ini, these are named:

- DirectoryStringTable
- FilenameStringTable

A "hash table" is a very common method of improving performance when you've got a lot of things to keep track of. A directory structure full of files is a perfect candidate for this technology.

Unfortunately, Oblivion's invalidation routines don't seem to be very accurate about how they mark a file as needing to be replaced. It appears there is a "disconnect" in the intersection between these two tables that causes a file invalidated in one path to also get invalidated in all other paths where it exists, resulting in index "collisions" that cause Oblivion to mistakenly think a texture should not be loaded from the BSA.

Here is my first attempt at a simplified explanation of how this works. Unfortunately, I just couldn't figure out a way to make the explanation any simpler than this. (I also couldn't get any colors to work, so we'll have to settle for underscores and strikethroughs instead.) If you have an idea on how to make this explanation any simpler or clearer, please post it or send me a PM.

The FilenameStringTable and DirectoryStringTable are initialized with all the textures in the BSA file, so all the paths and names of existing default textures get indexed in the hashes.

A small sample of these two indexes might look like this:

- FilenameStringTable= shield.dds, cuirass.dds, greaves.dds, greaves.dds, helm.dds
- DirectoryStringTable = textures/armor/glass/, textures/armor/iron/

where the underscored items reside in the BSA.

If I install one texture replacer for, say, all the Glass Armor, with AI.txt entries, then the hash entries for those textures get marked as being replaced by files in the Data folders. This means these items will now be invalidated, i.e. marked as being in Data the folders rather than the BSA, so now the hash index will look something like this:

- FilenameStringTable = shield.dds, cuirass.dds, greaves.dds, greaves.dds, helm.dds
- DirectoryStringTable = textures/armor/glass/, textures/armor/iron/

where the strikethrough items have been invalidated, but the underscored items still reside in the BSA.

If you're following any of this so far, you can probably already start to see where problems will arise, since all the common filenames shared between the two armor sets have now been invalidated from the FilenameStringTable simply because we wanted to replace them for the Glass armor set.

Now, if I want to also replace the texture for just the Iron cuirass, I'm going to run into problems because the hash table entries for all the common filenames in the Iron armor folder, including cuirass.dds, have already been invalidated (i.e., marked as existing in the Data folder rather than the BSA) and now the hash index will look something like this:

- FilenameStringTable = shield.dds, cuirass.dds, greaves.dds, greaves.dds, helm.dds
- DirectoryStringTable = textures/armor/glass/, textures/armor/iron/

where everything in both sets of armor has been invalidated.

Because the path "textures/armor/iron/" is now marked as existing in the Data folder, and the hash already knows that path is linked to a set of files that includes shield.dds, cuirass.dds, greaves.dds, gauntlet.dds, and helm.dds, then the hash will now expect to find all those same names in the Data folders even though I only wanted to replace the Iron cuirass.

In fact, the problem is even worse than this because Oblivion immediately recognizes the new directory path textures\armor\iron\ as soon as you add it to your Data folder. It doesn't even need to be listed in your AI.txt because the Oblivion automatically scans your Data folders and recognizes that directory path as being one from the BSA that has common filenames which have already been invalidated.

TOOLS

Ultimately, the only really good solution is to use BSA-redirection or BSA-alteration, but if you really want to see for yourself then these tools will help you generate an ArchiveInvalidation.txt file.

FURTHER READING

Oblivion Mods FAQ

GLOSSARY

AI.txt is shorthand for ArchiveInvalidation.txt.

CREDITS

Man of the Year award goes to Timeslip for solving the whole AI mess by avoiding it entirely!

Aside from that, there's a very big list of very talented and somewhat crazy people who put a lot of hard work into figuring out the AI mess. The list includes:
- DoomedMarauder
- Auctionmule
- Wonder Dog
- Scanti
- Malifrax
- Jarol
- Buerban
- Midnight Voyager
- Zanderat
- Martigen
- MadBat
- rdjeke
- DragoonWraith
- Oblivionmasta
- Quarn
- and many, many others ... Thank you all for this heroic effort!