One final (hopefully) "blog" post on the terminally-dull subject of data compression.
As we're rapidly approaching the end of the project, it's time to concentrate on a few of the little details that I've been ignoring for some time.
That includes the following warning message that both SamIAm and I see every time we run the script-insterter to convert the English scripts for LoX2.
META_BLOCK "ZAD", was 76KB, now 78KB.
META_BLOCK is bigger than original! Aborting!Whoops ... that's a "fatal" error that can't be recovered from, and should abort the whole conversion process.
That's what the message is for.
It's a problem ... that's why I put the test in there and made it print a scary message.
Then, because it was the only file in the two games with the problem, and it's not a huge show-stopper, I just disabled the "abort", and let the English conversion continue anyway.
I still have no idea what long-term bug we're creating by having that one file overwrite the start of the next file in the game ... but we've lived with it for over a year now.
No user-visible bug, no problem, right?
Well ... not really.
It's time to finally fix things properly before the release.
This all goes back to a whole bunch of blog posts from a long, long time ago.
We've talked here, and in many other threads, about how CD-based translations are a huge PITA because of the limited amount of free-memory available to change things (like the language), and how a Super-Duper System Card would make life a lot easier for translators.
In the case of the LoX games, I got around the need for that by figuring-out exactly where Falcom loaded every single asset in the game from, and then re-compressing every single graphic element or text script or code with a different and more-efficient compression algorithm, that made everything smaller in memory, and so freed-up the extra space that I needed to fit in SamIAm's wonderful-but-larger English translations.
Which was great ... a significant amount of work, but great ... except that I had a problem re-compressing a few files with my not-quite-as-brilliant-as-I'd-like compression algorithm.
Anyway ... we messed-around with this, messed-around with that, and eventually got to the point where there was only one file causing a problem, and we couldn't actually see that problem on-screen ... so we ignored it. :-"
The problem comes from the statistical-encoding. Let me explain in real English ...
Hudson, and Falcom, and everyone else in the 16-bit console era used some form of LZSS compression (except for the SNES's SDD-1 chip in Star Ocean).
My not-quite-as-brilliant-as-I'd-like compression algorithm is just a variant of that (like Falcom's enhanced "Falcom2" compression), that takes advantage of the fact that shorter repeat-lengths occur more-often than longer ones, and that "repeat" sequences are more-often closer to the current decompression-pointer than farther away.
I'm sorry if that sounds like gobbledygook ... but it's just-about as simply as I can explain it.
The huge stupid-problem with my compression tools was that they weren't originally designed to search for both of those cases, because the code that does the search-for-a-match was originally copied from the LZSS compressor in Mark Nelson's Data Compression book.
You'd always get the longest-possible match for a repeat ... but you had no idea if it was the longest-and-closest-possible match.
That threw away a number of opportunities for data compression ... and made that one file larger than it had been when Falcom compressed it.
I finally (after approx 20 years) fixed that, and now the compression code checks for both the longest match, and when it finds that, checks for the closest match of that exact size (no, it's not as trivial as it sounds).
I've avoided making that change for approx 20 years, because I was afraid of how it was going to slow-down the compressor, and make it unusable. At the end-of-the-day, I figured-out a reasonable way of doing it, and it only adds a few % to the compression time. Silly me. :oops:
Anyway ... where does that leave us?
Well, that one bad file ...
META_BLOCK "ZAD", was 76KB, now 78KB.
META_BLOCK is bigger than original! Aborting!Now shows ...
META_BLOCK "ZAD", was 76KB, now 74KB.That's much better!
And when it comes to the level data, which includes the scripts, which include the English text ...
// LoX1 GAME META-BLOCKS 176KB (22 8KB banks)
META_BLOCK "Z10", Falcom 160KB, simple 142KB, optimal 136KB.
META_BLOCK "Z11", Falcom 148KB, simple 132KB, optimal 128KB.
META_BLOCK "Z12", Falcom 176KB, simple 156KB, optimal 150KB.
META_BLOCK "Z13", Falcom 174KB, simple 154KB, optimal 150KB.
META_BLOCK "Z14", Falcom 166KB, simple 150KB, optimal 146KB.
META_BLOCK "Z15", Falcom 172KB, simple 156KB, optimal 152KB.
META_BLOCK "Z16", Falcom 168KB, simple 152KB, optimal 146KB.
META_BLOCK "Z17", Falcom 176KB, simple 156KB, optimal 150KB.
META_BLOCK "Z18", Falcom 158KB, simple 142KB, optimal 138KB.
META_BLOCK "Z19", Falcom 176KB, simple 154KB, optimal 150KB.
META_BLOCK "Z1A", Falcom 132KB, simple 116KB, optimal 112KB.
META_BLOCK "Z1B", Falcom 174KB, simple 142KB, optimal 136KB.
// LoX2 GAME META-BLOCKS 128KB (16 8KB banks) (only the largest blocks are shown)
META_BLOCK "Z25", Falcom 128KB, simple 124KB, optimal 122KB.
META_BLOCK "Z63", Falcom 128KB, simple 124KB, optimal 120KB.
META_BLOCK "Z22", Falcom 128KB, simple 122KB, optimal 118KB.
META_BLOCK "Z10", Falcom 128KB, simple 118KB, optimal 116KB.
META_BLOCK "Z40", Falcom 128KB, simple 118KB, optimal 116KB.
META_BLOCK "Z41", Falcom 128KB, simple 118KB, optimal 114KB.
META_BLOCK "Z42", Falcom 126KB, simple 120KB, optimal 116KB.
META_BLOCK "Z54", Falcom 126KB, simple 120KB, optimal 116KB.
META_BLOCK "Z84", Falcom 126KB, simple 120KB, optimal 116KB.
META_BLOCK "Z51", Falcom 124KB, simple 118KB, optimal 114KB.
META_BLOCK "Z74", Falcom 122KB, simple 118KB, optimal 114KB.
META_BLOCK "Z73", Falcom 120KB, simple 114KB, optimal 112KB.In summary ...
// LoX1 Summary
Uncompressed 49 files, 12562926 bytes.
Falcom 49 files, 6062815 bytes.
SWD5 simple 49 files, 5235179 bytes.
SWD5 optimal 49 files, 5057690 bytes.
// LoX2 Summary
Uncompressed 103 files, 23278006 bytes.
Falcom 103 files, 9026275 bytes.
SWD5 simple 103 files, 8381998 bytes.
SWD5 optimal 103 files, 8140408 bytes.So, again, we got a huge "win" with Lox1 ... but jeez ... we only-just won enough space in LoX2 in order to add the English translations.
We were really, really close to needing one of those Super-Duper System Cards that we've talked about for a while!
