Save disk space for ISOs

From Emulation General Wiki
Revision as of 05:52, 20 November 2018 by YusufT19 (talk | contribs) (CHD Compression: Demul --> DEmul)
Jump to navigation Jump to search

ISOs are faithful software recreations of game disks. However, at 700 MB (CD) / 1.4 GB (GC Mini-DVD) / 4.7 GB (single-layered DVD) / 25 GB (Blu-Ray), they can get pretty taxing to disk storage, as newer generations of consoles come.

It wouldn't be so bad if not for the fact that the actually useful game data is often only a fraction of that data size - for instance, the Mario 25th Anniversary Wii disk is a 4.7GB game with only a single SNES rom and nothing else (12 MB of useful data, to be precise). So naturally, one would want to trim the extra fat as much as possible. This is what this improved version of a previous guide aims to help for.

How does one lighten ISO / ROM dumps?

There are many ways. Some alter the dump copy forever. Some are playable on only some specific emulators. And many light dumps are unplayable on real hardware (though a bunch are). All depending on the method and the console. So you might want to consider all of this before.

This page was based partially on this guide.

Archive-quality dump means that the resulting compressed dump, when reverted back to its original state, will be the same checksum as the official uncompressed release. Compressions that can't be reversed, or those that can be but will have missing or altered content, whether it interferes with functionality (rebuilt table of content) or not, are not archive quality dumps. For example, the WBFS format is not archive quality since it will be missing padding content and upgrade partitions (which have their uses in 3DS/Wii modding) compared to an intact uncompressed dump.

Applicable to All Platforms

Audio-CD

Sega-CD, PC-Engine, PlayStation, Sega Saturn... what did these have in common was their reliance on the regular CD format. Game developers stored often orchestrated/Redbook music and occasionally voice acting, using the Audio-CD format. Of course, the CD contained also game data. But it was terribly inefficient when it comes to disk storage. Even a 700 MB CD containing nothing but Audio-CD data can hold at most around 80 minutes worth of sound data.

That's why devs no longer used it, preferring custom audio formats included in the "game data" part of the disk. By the time the PS1 gen came, the Audio-CD part was just used for messages like "Don't put this in a CD player, dumb user!" and little else (exceptions exist, of course!)

  • Full Dump:
    BIN/ISO + CUE


BIN/ISO is the full disk data, including Audio-CD sound data and game data
CUE is the datasheet file

  • Light Dump:
    ISO + MP3/WAV + CUE


ISO is the disk data with only the game data
MP3/WAV is the sound data from the Audio-CD, but these formats take much less disk space
CUE is the datasheet file

  • Archive-quality dump? No (unless audio is converted to and from uncompressed formats, which is unlikely)
  • Gain: Several hundreds of MBs to just a few dozens, depending on how much this specific game relies on the Audio-CD sound format
  • Tools Used: Load the BIN+CUE using a virtual drive, then use a CD dumping tool
  • Can be reverted? Yes, just burn the ISO+MP3/WAV+CUE again using a CD burner tool (ImgBurner) either to a physical disk or as an ISO+BIN file. Lossy audio formats will result in data loss.
  • Playable on Hardware? No, but can be reverted to be
  • Playable on Emulators? Yes (use virtual drive if needed). Some aren't compatible with MP3 so convert to WAV with MP32WAV if that's the case. You may need Sega Cue Maker.

Examples:

  • Princess Crown (SAT): 574 MB > 72 MB (Game Data) + 50 MB (Audio-CD data in MP3 format)
  • Captain Tsubasa (SCD): 512 MB > 146 MB (Game Data) + 3 MB (audio as MP3) > (as 7zip) 34 MB (Game Data) + 3 MB (audio)

M3U (playlist) files may be used too for this distribution scheme.

Sometimes dumps that come this way may not work on some emulators. This is often due to incorrect CUE files sometimes using wrong filenames or MP3 files instead of WAV files.

Padding

Devs often have their games much, much bigger than they need to be. They put garbage data in the disk. Garbage data isn't game data and just bloats the disk size. It's either a sequence of 00/FF (you know what's inside a file if you open it with a hex editor) or randomized garbage data.

Its purpose can be to fill in some spots in the disk so that specific parts of game data are in specific areas of the disk (like the borders) and hence the drive's reading speed is quick enough in these spots for the game to work properly. It's in the best of your interests not to mess with this data arrangement (referred to as LBA and TOC in the case of GC/Wii/PS2/PSP) or else the game might not even work in some cases (it might in others though).

BUT-- the most common bar none use for this is to screw with pirates and people who download ISOs off online sharing websites, by making the ISO bigger and harder to download. Some go a little step further and make that garbage data, not just a sequence of 00/FF to make the ISO much, much harder to compress using regular archive formats like zip/7zip/rar... You might be overjoyed to learn this has become the industry standard nowadays.

Many compression schemes remove or simplify padding patterns to allow for easier compression.

CHD Compression

  • Archive-quality dump? Yes
  • Gain: Immediate
  • Tools Used: chdman (included with MAME)
  • Can be reverted? Yes, using extractcd (included with MAME)
  • Playable on Hardware? No.
  • Playable on Emulators? Only MAME and DEmul. Some libretro cores for other emulators started adding support.

MAME uses the CHD format for disc images in general and includes tools to convert from and to it. It uses 7zip's LZMA compression on the game data and lossless FLAC compression for the audio data to optimize compression even further than with the BIN+CUE+MP3/WAV data separation alone.

Placing chdman and extractcd in the same directory as dumps in BIN+CUE format, the following command-line instructions can be used:

  • BIN/CUE to CHD: for %i in (*.cue) do chdman createcd -i "%i" -o "%~ni.chd" (Windows)
  • CHD to BIN/CUE: for %i in (*.chd) do chdman extractcd -i "%i" -o "%~ni.cue" (Windows)

If you have one of the European PSX games that feature LibCrypt copy protection, you have a .sbi file in addition to the .bin/cue file. The CHD creation process doesn't process the .sbi file. Thus, you will need the .sbi file in the same directory as the.CHD file for the game to run.

PlayStation 1

ECM

  • Archive-quality dump? No (removes error correction data permanently)
  • Gain: Not Immediate (ISO size doesn't change). ECM only zeroes out redundant error correction data (but in some rare cases this data may be used for anti-piracy, hence corrupting the dump!). When coupled to a compressed archive format (7zip/gzip/zip) however it achieves drastic size reductions.
  • Tools Used: PakkISO or ECM Tools (from emuparadise download page). Drag-and-drop the iso on the specific tool, then compress it with 7zip.
  • Can be reverted? Yes, using the same tools (unECM). However, there's data loss (error correction data) which can damage a select few games. Check the hash with Redump to make sure nothing was altered.
  • Playable on Hardware? No.
  • Playable on Emulators? Only recent SVN builds of PCSX-R and ePSXe (they also support 7zip/gzip/zip archives so use them with ECM). Other emulators don't, so you convert manually the ECM dump to an ISO dump whenever you want to play it.

PBP (PSP Format for PS1 Images)

  • Archive-quality dump? No
  • Gain: Immediate (ISO size decreases a lot). Official format used by Sony for PS1 dumps on PSP.
  • Tools Used: PSX2PSP. Generates an EBOOT.PBP file.
  • Can be reverted? Yes. Using the same tool, to generate BIN+CUE files. There's data loss, although it's negligible in terms of functionality.
  • Playable on Hardware? On PSP, not on PS1.
  • Playable on Emulators? Retroarch, PCSX-R, and ePSXe.

PlayStation 2

Manual Scrubbing + GZIP Compression

Often, you can open the ISO in UltraISO and find the dummy files. Sometimes they're obviously looking files like DUMMY.BIN, DATA0.BIN (or .DAT), DUMMY.DAT, etc. Or folders named "PADDING" and stuff like that. You can look into the files (with a hex editor) to see if they're obvious padding data (full of 00/FF, though sometimes it's not as obvious).

However, you must never mess with LBA and TOC when removing padding. So you try to change the size of the padding file inside the ISO to 0 MB or alter it directly with a hex editor so that it's all zeroed out.

  • Archive-quality dump? No (missing data)
  • Gain: Not Immediate (ISO dump size the same). Coupled with compression, however... the gain is really noticeable.
  • Tools Used: UltraISO
  • Can be reverted? Not really. Garbage data here is zeroed out, it's not important and you still have the same disk data structure.
  • Playable on Hardware? Yes.
  • Playable on Emulators? Yes.

You may forego the manual scrubbing part entirely. If you feel adventurous you might want to do it to enhance compression though. Let's get to the meaty part though...

The PCSX2 emulator supports opening compressed archives containing ISOs. The best format it supports would be GZIP.

Use 7zip ("Add to archive..." then choose to gzip) or Pigz (multi-threaded, much faster compression) to generate gzip archives containing the ISO file in question. PCSX2 will build an index of each gzip compressed game it loads (as a file in the same directory as the gzip archive), so after the first time where you'll have to wait for the decompression, in all subsequent times there is no speed difference between playing an uncompressed and compressed game. Of course, you can extract the ISO back from the GZIP archive.

CSO (aka CISO)

  • Archive-quality dump? No (missing data)
  • Gain: Immediate (ISO dump size the same). Coupled with manual scrubbing, it can be bigger. Not as much of a gain as GZIP though.
  • Tools Used: maxcso
  • Can be reverted? Yes, using the same tool. No data loss.
  • Playable on Hardware? NO
  • Playable on Emulators? Yes - PCSX2.

PlayStation Portable

CSO (aka CISO)

  • Archive-quality dump? No (changes disc structure, some gameplay issues)
  • Gain: Immediate (ISO dump size lowers). Substantial, depending on game. Increases load times on real hardware but not emulators.
  • Tools Used: CISO, CISO GUI, maxcso, UMDGen or PSP ISO Compressor


CISO GUI (CISO is command line-based): Drag-and-drop ISO file on program, Select compression level (1 to 9 (Max)), "Compress", and choose directory for saving the new CSO file.
UMDGen is a PSP ISO editor. You can save to either the CSO format with this.

  • Can be reverted? Yes, using the same tools and saving in ISO (uncompressed) format. No data loss.
  • Playable on Hardware? Yes. Longer load times though.
  • Playable on Emulators? Yes. PPSSPP.

Dreamcast

CHD Archive Format

  • Archive-quality dump? No (missing data, no gameplay issues)
  • Gain: Immediate (ROM dump size lowers).
  • Tools Used: GDI to CHD converter.
  • Can be reverted? No.
  • Playable on Hardware? No
  • Playable on Emulators? Yes - DEmul and reicast

GameCube / Wii

Some examples:

  • Super Mario Anniversary (Wii): 4.7 GB (original) > 12 MB (GCZ) (!!)
  • Super Smash Bros Brawl (Wii): 7.8 GB (original) > 7.1 GB (GCZ) (main cause is FMV's low compression)
  • Xenoblade PAL (Wii): 7.8 GB (original) > 6.3 GB (GCZ)
  • Super Mario Galaxy 2 (Wii): 4.7 GB (original) > 1.8 GB (GCZ)
  • Tales of Graces (Wii): 4.7 GB (original) > 4.2 GB (GCZ) (game data already fills the disk)
  • Animal Crossing (GC): 1.4 GB (original) > 26 MB (GCZ)
  • Zelda Four Swords Plus Japan (GC): 1.4 GB (original) > 480 MB (GCZ)
  • Zelda Twilight Princess (GC): 1.4 GB (original) > 1.4 GB (GCZ) (game data already fills the disk)
  • Megaman Collection (GC): 1.4 GB (original) > 1.1 GB (GCZ) (sound data is stored as uncompressed stream to fill disk)

People used to resort to WiiScrubber (Wii) and GCM Utility (GC) to scrub/trim games to end up with dumps that while they had no immediate size change, their randomized garbage data (like "dummy", "padding" or "znull") was still there but zeroed out making archived dumps using zip/7zip/rar formats have stunning gains (from 1.4GB uncompressed to 26MB zipped for Animal Crossing for example!). Of course, the file still needed to be uncompressed to its full size everytime you wanted to play it.

Trimming and scrubbing (in Wiiscrubber terms) aren't the same! While they both are terms for "zeroing garbage data" to make it more compression-friendly, trimming does not just that like scrubbing but takes the extra step of relocating the garbage data to the end of the file, hence altering its TOC and requiring the disc to be fakesigned, for a not-so-big compression gain. Hence why scrubbing is by far the most authentic and safe way to solve the garbage data problem.

However compressed formats incorporating the "padding zeroing" part were made since then, and Dolphin supports them! These are the WBFS and GCZ formats.

GCZ (Dolphin native archive format) - GC/Wii

Dolphin can't play games directly from compressed archives (7zip/zip/gzip/rar...). Instead, it utilizes its own compression method.

  • Archive-quality dump? Yes for GC games, No for Wii games. (checksum won't be the same, no gameplay issues)
  • Gain: Immediate (ISO dump size lowers). Considerable, depending on the game.
  • Tools Used: Dolphin; right-click the game(s) in the games list and select "Compress ISO..." ("Compressed selected ISOs..." if more than one is selected)
  • Can be reverted? Yes. You can right-click on the GCZ-compressed (in blue) ISOs in Dolphin and choose "Uncompress selected ISOs". Garbage data for Wii titles will be zeroed out and thus different from the official dump but will have no effect on gameplay in most cases.
  • Playable on Hardware? No
  • Playable on Emulators? Only Dolphin.

Scrubbing and Trimming - GC/Wii

  • Archive-quality dump? No (missing data)
  • Gain: Not Immediate for just scrubbing (ISO dump size the same); Immediate for trimming (ISO dump size lowers).
  • Tools Used: GameCube ISO Trimmer (scrubbing and trimming for GameCube), WiiScrubber (scrubbing for Wii, although compressing with Dolphin will scrub it anyways).
  • Can be reverted? No.
  • Playable on Hardware? Yes for scrubbing; some games won't work trimmed.
  • Playable on Emulators? Yes for scrubbing; some games won't work trimmed.

Scrubbing games zeros out garbage/dummy data in the ISO file. The resulting file will be the same size, but it will be able to compress better than unmodified ISOs. The difference can be huge depending on the game. Animal Crossing, for instance, will compress to just a 26 MB .gcz file after being scrubbed!

Trimming games implies scrubbing, but will also remove the garbage/dummy data outright, resulting in a smaller ISO file. The trimmed file can be used immediately at a smaller size without compression, but the game's internal structure will be wholly modified. Several games will break if you trim them.

WBFS - Wii only

  • Archive-quality dump? No (checksum will be different even when reverted)
  • Gain: Immediate (ISO dump size lowers). Considerable, depending on the game.
  • Tools Used: Wii Backup Manager; after adding your game, go to "Transfer" and select "WBFS".
  • Can be reverted? Yes. You "Add" the WBFS disk dump and "Transfer" it to the "ISO" format. Garbage data is still there but zeroed out, which won't affect gameplay.
  • Playable on Hardware? Yes!! WBFS dumps are playable on Wii.
  • Playable on Emulators? Yes - Dolphin

CSO (aka CISO) - Wii only

  • Archive-quality dump? No (checksum will be different even when reverted)
  • Gain: Immediate (ISO dump size lowers). Less than WBFS.
  • Tools Used: Wii Backup Manager; same as above, but select "CISO".
  • Can be reverted? Yes. You "Add" the CSO disk dump and "Transfer" it to the "ISO" format. Garbage data is still there but zeroed out, which won't affect gameplay.
  • Playable on Hardware? No
  • Playable on Emulators? Yes - Dolphin

Wii U

All uncompressed ISO dumps are 23.3GB, which makes compression needed de facto. Some examples:

  • Super Mario World 3D: 23.3GB > 2.61GB (88% gain)
  • New Super Mario Bros. U: 23.3GB > 1.8GB (92% gain)
  • Donkey Kong Country Tropical Freeze: 23.3GB > 11.7GB (49% gain)

WUD (Wii U Compressed Image Dump Archive)

  • Archive-quality dump? Yes (lossless, doesn't actually alter the data).
  • Gain: Immediate (ISO dump size lowers): Around 50% gain usually (depending on the game)
  • Tools Used: wud tool. To use, drag-and-drop the ISO on the executable.
  • Can be reverted? Yes. The wud tool includes a decompression utility, which recreates the original file.
  • Playable on Hardware? No (can be converted to an installable format using wud2app)
  • Playable on Emulators? Yes - Future releases of CEMU (and potentially other Wii U emulators)

By Exzap, CEMU's author (release thread). The tool detects duplicate sectors and only stores them once: all the empty ranges end up removed, storing only ranges which contain file or filesystem data. Of course, the original ISO can be reconstituted.

RPX/RPL (a.k.a. Spilling the WUD's Guts)

  • Archive-quality dump? No (dump is collapsed to individual files)
  • Gain: Immediate (Total size decreases). Can be less than a WUD depending on the game.
  • Tools Used: UWizard. Follow this guide for usage information. You'll need the title key for the game you're trying to extract, which you should have gotten with the WUD.
  • Can be reverted? Any padding information is lost.
  • Playable on Hardware? Yes, use Loadiine GX2
  • Playable on Emulators? Yes

GameBoy Advance / Nintendo DS / Nintendo 3DS

Trimming

  • Archive-quality dump? No (destructive)
  • Gain: Immediate (ROM dump size lowers).
  • Tools Used: NDSTokyoTrim (GBA/DS/3DS), rom_tool (3DS)
    NDSTokyoTrim: Drag-and-drop roms, and press "Trim". The original file will be overwritten!
  • Can be reverted? No. Sometimes it damages the game's functionality irreversibly.
  • Playable on Hardware? Yes, same file format.
  • Playable on Emulators? Yes

While it doesn't affect the game's functionality by much other than being a smaller regular ROM, it's not advisable though to trim in many cases, including GBA games in general. Rebuilding the internal structure of the rom might even destroy the game's functionality in the few games where it does matter (for example Golden Sun DS).

Xbox 360

It's certainly better than keeping 8.5GB images, but the conversion is too substantial and irreversible to be suitable for archival purposes since it affects data structure tables as well.

XEX (Xbox Executable) + Data folder (a.k.a Spilling The ISO Guts)

  • Archive-quality dump? No (dump is collapsed to individual files)
  • Gain: Immediate (total file size decreases). It still works with Xenia
  • Tools Used: XBOX 360 ISO Extract, Exiso-GUI or Exiso. (These might be useful for rom-hacking too I guess?)
  • Can be reverted? ISO could be rebuilt, though not accurately.
  • Playable on Hardware? Yes. Using a JTAG or RGH exploited console.
  • Playable on Emulators? Yes - Xenia.

Rebuilt ISO

  • Archive-quality dump? No (missing data)
  • Gain: Immediate (total file size decreases). It still works with Xenia.
  • Tools Used: ISO2GOD (also a "Games on Demand" X360 image converter).
    Under Settings, set the output and rebuild path to the same location. Check "Always save rebuilt ISO" and set Padding to "Full (ISO Rebuild)", then save changes.
    "Add ISO", and press "Convert". Keep generated ISO, and delete generated folder.
  • Can be reverted? The padding quantity information alongside the original data structure is lost forever.
  • Playable on Hardware? Yes. If the rebuilding process didn't damage anything vital.
  • Playable on Emulators? Yes - Xenia.

Other Tricks

Storage Tricks

These methods have the advantage of being compatible with EVERY emulator, even those without proper support for compressed ISO/archive formats.

NTFS Compression

You can enable filesystem-level compression (like "NTFS Compression" in Windows) for the directory containing your ISOs/ROMs. This has a very noticeable space gain and doesn't affect the emulator's functionality. It's surprisingly more effective than many people would like to give this credit.

  • Archive-quality dump? Yes

Decompression on Demand

You can keep your ROMs/ISOs compressed in a 7zip archive and use RocketLauncher or any other Frontends to decompress 7zip archives and pass the contents onto the emulator.

  • Archive-quality dump? Yes

Destructive Modification

Of course, there are also the devs who don't merely use garbage data for padding but also bloat the game data part. For instance, many PSP JRPGs have a specific FMV movie duplicated a dozen times, and the Megaman Collection on GC stores its sound data using an uncompressed format bringing the size of that portion alone to 1GB. In many of these cases, there's just not much you can do about it without destructively altering game data.

So what do some people do? Note when I say "delete" it often means replacing the file with a 1KB dummy file to prevent the ISO file structure from collapsing on itself, but sometimes such care isn't even put into ensuring it's still in a functional state. Some examples:

  • delete all videos: this brings Super Smash Bros Brawl to 4.7GB.
  • delete all voice acting and occasionally sound and music: this brings Xenoblade PAL to 4.2 GB with even only one of both dubs removed.
  • delete unused content you could get with emulator cheats like rooms and stuff
  • delete "extra" languages other than English
  • If a game has multiple quests, they delete some characters/quests they don't like. Especially seen in compilations and stuff with demos included.

Tools used for this are modding tools for rebuilding file trees, like UMDGen (PSP), Tinke (DS) and also regular ISO tools (PS1, Saturn)...

Since this results in most of the case in very noticeable detrimental effects in gameplay (if the game doesn't crash outright), this is nothing short of mutilating the game image. Some of these dumps make it to sharing sites sadly enough.

Avoid resorting to destructive modifications since it can lead to random crashes and unexpected behavior, especially in games with lots of shared assets. [Verify] your dumps to make sure you don't have these, and if you want to compress those prefer other methods or using decompression on demand. One reason to use a destructively modified dump is for burning your own Dreamcast games, as GD-ROMs were over a gigabyte in size and CD-Rs top out at 850MB.

  • Archive-quality dump? No (destructive mod)