Recreated Keen 4-6 Source Code

Here is where to post about the latest Commander Keen fangame or modification you've finished, a new website you've made, or another Keen-related creation.
User avatar
K1n9_Duk3
Vorticon Elite
Posts: 781
Joined: Mon Aug 25, 2008 9:30
Location: Germany
Contact:

Recreated Keen 4-6 Source Code

Post by K1n9_Duk3 »

Levellass wrote: Thu Sep 03, 2015 6:22I can't see any reason why we couldn't make our own source code (With blackjack! And hookers!)
Looks like we won't be seeing the original source code released to the public anytime soon, so I might as well share this with you (no blackjack or hookers, sorry if that quote got your hopes up).

If you played Return to the Shadowlands, then you have already seen an older incarnation of this code in action.

That old incarnation still had a couple of bugs/differences, but this new version is more accurate. So accurate, in fact, that you can compile the code and compress the new executables with LZEXE and get 100% identical copies of the original v1.4 executables! The source code may not be exactly the same as the original source code, but this is pretty much as close as you can get.

This source code is based on Catacomb 3-D, with some parts taken from Wolfenstein 3-D and Keens Dreams as well. It is released under the terms of the GNU General Public License, so please read and try to understand that license before you release any mods or other programs based on this source code.

Disclaimer: Although this code is an accurate recreation of the original code, that doesn't mean there aren't any bugs. Most of the bugs I know about have been marked with comments in the source code (search for "BUG"). I may or may not create a modified version that has most of these bugs fixed to provide a better base for other people to use for their source code mods. But that may need to wait until after Foray in the Forest is released.

PS: I feel like I'm gonna be disappointed by the reaction (or lack of reaction) to this achievement. Prove me wrong.
Hail to the K1n9, baby!
http://k1n9duk3.shikadi.net
User avatar
RoboBlue
It's that one guy.
Posts: 941
Joined: Wed Apr 23, 2008 10:26
Contact:

Re: Recreated Keen 4-6 Source Code

Post by RoboBlue »

Well this is a nice surprise.

Time for Keen 4-6 to get ported to everything under the sun.

I wonder if the Sega Game Gear can handle it.
Image
User avatar
Nospike
Keen Minecrafter
Posts: 1402
Joined: Tue Mar 30, 2010 13:56
Location: Czech Republic

Re: Recreated Keen 4-6 Source Code

Post by Nospike »

Fantastic work, I ought to study this code. It's a keener's dream come true.
User avatar
Multimania
Vortininja
Posts: 84
Joined: Sat Nov 10, 2007 8:10
Location: Hiding in a small, cramped corner of the BwB megarocket.
Contact:

Re: Recreated Keen 4-6 Source Code

Post by Multimania »

This is extremely awesome!

I was just able to get Keen 5 to build on the computer I first played it on, which is thoroughly neat! It only took about 10 minutes to get everything working, too: the only minor issue was that I had to boot without any TSRs loaded to get K5_SPEC.C to compile, otherwise Borland C++ ran out of memory. (I'm using Borland C++ 2.0). Even then: I guess that's probably what the folks at id had to do as well.

I'm looking forward to playing with it further. K1n9_Duk3 already found a couple of bugs in Omnispeak with it (now fixed), and I think I've found at least one more browsing the source, so I can't wait to see what else we can do with it.

Very cool!
User avatar
Tormentor667
Vortininja
Posts: 225
Joined: Sat Mar 29, 2008 14:08
Location: Germany
Contact:

Re: Recreated Keen 4-6 Source Code

Post by Tormentor667 »

I imagine adding new features and removing restrictions - some kind of a sourceport like GZDoom for Doom?
Ceilick
The Dude
Posts: 1670
Joined: Mon Sep 22, 2008 20:10
Location: Seattle

Re: Recreated Keen 4-6 Source Code

Post by Ceilick »

Fantastic! You know I"ll be using these, and Nisaba and I are currently developing a source modding tutorial and I think this would make the perfect base to teach others with. I think you're work here is going to be the foundation of whatever mods we see in the future. Looking forward to working with it!
User avatar
XkyRauh
Mortimer's RightHand Man
Posts: 418
Joined: Thu Nov 01, 2007 16:32
Location: San Diego, California

Re: Recreated Keen 4-6 Source Code

Post by XkyRauh »

This is neat. :)
User avatar
DarkAle
Vorticon Elite
Posts: 318
Joined: Wed Jul 26, 2017 17:40
Location: Coral Hideout
Contact:

Re: Recreated Keen 4-6 Source Code

Post by DarkAle »

That's awesome! It would be nice to see the originals as source code mods
Commander Keen in... Robot Apocalypse!
Episode 13: The Ultra Omegamatic viewtopic.php?f=4&t=6802
Episode 14: Secret of the Sorcerer viewtopic.php?t=13031
NY00123
Vorticon Elite
Posts: 508
Joined: Sun Sep 06, 2009 19:36

Re: Recreated Keen 4-6 Source Code

Post by NY00123 »

I should still check the archive, but I think it's clear that this is an important milestone!

I mean, I knew that the games were reverse-engineered beforehand (see e.g., Omnispeak), and that it shouldn't be impossible to recreate the original exes under the right conditions. Although I'm wondering how much problems there were with the orders of global variables in the exes. I think that there could maybe be more issues with this while using Borland C++ 2.0, but I'm unsure.

One thing to mention is that there can be more than one way for modding the games, as well as supporting mods via source ports.

Just to bring some examples:
1. First of all, CKPatch is still an option.
2. After it was open-sourced, a commonplace method of modding Wolf3D was source-modding. This is still done today, and there were a few examples of this for Keen as well (with RttS being just one example).
3. Doom got DeHackEd before it was open-sourced. Afterwards, there were other options for modding, with the BEX format being just one example. I think that Omnispeak gets close with its Galaxy Action (state) definitions.
4. Duke Nukem 3D supported CON scripting from the beginning. After the release of a derivative of Duke3D, titled NAM, another game, WW2GI, introduced additions to CON, like the gamevars. These were inherited by Enhanced Duke (i.e., EDuke), while it had its own additions for accessing game internals. Even more have been added to the EDuke32 port.
5. DECORATE was introduced to (G)ZDoom for actor definitions. ECWolf also got its own dialect of DECORATE, and LZWolf inherited it from ECWolf.
6. During the last 5 years, GZDoom got ZScript, adding even more flexibility over what's allowed by DECORATE. When it comes to actor definitions, it basically lets you write new code for them.
Last edited by NY00123 on Fri Jul 02, 2021 13:44, edited 1 time in total.
User avatar
K1n9_Duk3
Vorticon Elite
Posts: 781
Joined: Mon Aug 25, 2008 9:30
Location: Germany
Contact:

Re: Recreated Keen 4-6 Source Code

Post by K1n9_Duk3 »

NY00123 wrote: Thu Jul 01, 2021 20:43 Although I'm wondering how much problems there were with the orders of global variables in the exes. I think that there could maybe be more issues with this while using Borland C++ 2.0, but I'm unsure.
I was actually about to contact you and ask for some help or advice, but after browsing your recreation thread on the RGB Form once again and checking out the additional information in your bitbucket repository (and checking out disassemblies and the compiler settings of Keen Dreams and Catacomb 3-D), I was able to figure this out myself.

Fortunately, I only focussed on v1.4 of Keen 4-6 EGA (and also Keen 6 v1.5 EGA) and all of those were compiled wth Borland C++ 3.0 or 3.1, which appear to arrange the global variables in a way that's much easier to reverse engineer. Any earlier versions of Keen 4-6 were indeed compiled with Borland C++ 2.0, but I didn't bother trying to recreate exact copies of those executables.

I did run into an issue when compiling the "Return to the Shadowlands" source code with Borland C++ 2.0 where the code compiled without issues, but the executable would always quit with an "Abnormal program termination" error. The compiler appears to have forced the "grneeded" array into the main data segment instead of giving it its own far data segment. That meant the main data segment didn't have enough free space for the stack and the environment (command line parameters and such).
RoboBlue wrote: Thu Jul 01, 2021 5:39 Time for Keen 4-6 to get ported to everything under the sun.
Just to get this out of the way now, I do not intend to create any source ports based on this code myself, at least not now. But I planned for that before starting the code recreation, that's why I used SDL-style integer types throughout the CK_* and K?_* files. I hope this helps a little when porting the code.

I think that using this code to make the existing "ports" like Omnispeak more accurate (or to verify that they are accurate) might be the better course of action. I spent enough time re-inventing the wheel with this recreation as it is. In restrospect, using the old source modding package and applying the same semi-automated comparison techniques I used in the past couple of weeks might have saved me some time. But I had already recreated all of the code that wasn't part of the ID Engine long before I came up with these techniques, so whatever...

As a bonus, my recreated code uses names that are much closer (if not idectical) to the ones from the Keen Dreams code, so it should be much easier to go back and forth between these two codebases (and you can port enemies and gameplay mechanics from one game to the other more easily).
Hail to the K1n9, baby!
http://k1n9duk3.shikadi.net
User avatar
Flaose
Vorticon Elder
Posts: 568
Joined: Sat Oct 27, 2007 20:30
Location: The Frozen Hell
Contact:

Re: Recreated Keen 4-6 Source Code

Post by Flaose »

This is incredible, well done! I look forward to seeing where the community goes from here.
Cerebral Cortex 314 - For All of your Commander Keen Needs.
Eat at Joe's
NY00123
Vorticon Elite
Posts: 508
Joined: Sun Sep 06, 2009 19:36

Re: Recreated Keen 4-6 Source Code

Post by NY00123 »

ok, I tried to build the exes identified as v1.4, and it's indeed possible to byte-by-byte recreate the originals!

Note that if the patching utility is used, you should tell it to extract the data under KEEN4-6\static, and then run KEEN4-6\static\make.bat under a compatible DOS environment (e.g., DOSBox) in order to create the OBJ files for the linker, using Blzut3's makeobj program. These OBJ files should then be found under the separate KEEN4, KEEN5 and/or KEEN6 subdirs.

As expected from the readme, the recreated Keen 6 EGA v1.5 exe was more-or-less the same as the original, ignoring the referenced difference in CheckInTiles' layout and the workaround for it.

Maybe it'll remain unresolved, but it's really minor. I tried a few approaches to counter it without getting the desired result, unless a combination of these is maybe required. Details are purposefully somewhat-hidden with a different font color.

Examples I recall:
- Adding the hdrstop pragma.
- Redefining enum intiletype's values using separate macros.
- Changing the type of CheckInTiles' "intile" variable to intiletype or Sint16. In the latter case, I also tried to change the types of rowdelta and midx to Sint16. I still think that intile should be Uint16, due to its uses in the gamestate.keys array, and possibly also in the call to TileBonus (emphasis on the generated OBJ file's contents).
- Moving the definitions of a few global variables into CK_KEEN.C, in order to impact CK_KEEN.OBJ's layout.

The reason for the latter is that the function bodies are preceded by a list of extern functions/variables, which are used but not defined. Each referenced function appears to use 1 byte, while for a variable, the size in use is probably the variable's size minus 1.

I still haven't found a clear deterministic manner for determining if the compiler adds a byte of padding between a function body and a jump table or not. It does seem to be decided by the compiler, rather than the linker, thus it can be checked in the OBJ files. Such a byte seemed to get added in CK_TEXT.OBJ:_HandleCommand: and ID_US_2.OBJ:_USL_LoadCustom from what I recall, but (in the original exe) not in K6_ACT1.OBJ:_SpawnBlooglet.


I already encountered similar problems beforehand, and that's just with codebases built with Borland C++. A couple of examples I recall:
- The layout of at least one function in The Catacomb Armageddon v1.02/Apocalypse v1.01 wasn't fully matching, using the open-source release. I could get it to match in size by changing the order of addends in a sum IIRC, but it was still not 100% matching.
- For The Catacomb Abyss v1.24, you just had to change the value of STARTTILE8 in GFXE_ABS.EQU, but not in GFXE_ABS.H. I really don't know why was this the cause, other than using an outdated file, having an unnoticeable data corruption or going through a Borland C++ (assembler) bug. I'm quite sure this had no impact on the game due to the lack of 8x8 tiles, so no assembly drawing runtime depending on STARTTILE8 is used.
K1n9_Duk3 wrote: Thu Jul 01, 2021 21:54
NY00123 wrote: Thu Jul 01, 2021 20:43 Although I'm wondering how much problems there were with the orders of global variables in the exes. I think that there could maybe be more issues with this while using Borland C++ 2.0, but I'm unsure.
I was actually about to contact you and ask for some help or advice, but after browsing your recreation thread on the RGB Form once again and checking out the additional information in your bitbucket repository (and checking out disassemblies and the compiler settings of Keen Dreams and Catacomb 3-D), I was able to figure this out myself.
It's great to see that my information and posts were useful!
Fortunately, I only focussed on v1.4 of Keen 4-6 EGA (and also Keen 6 v1.5 EGA) and all of those were compiled wth Borland C++ 3.0 or 3.1, which appear to arrange the global variables in a way that's much easier to reverse engineer. Any earlier versions of Keen 4-6 were indeed compiled with Borland C++ 2.0, but I didn't bother trying to recreate exact copies of those executables.
So it is less of a problem with 3.0/3.1, as I suspected.
One may give the CGA versions a try now! (Just kidding...or am I?)
I did run into an issue when compiling the "Return to the Shadowlands" source code with Borland C++ 2.0 where the code compiled without issues, but the executable would always quit with an "Abnormal program termination" error. The compiler appears to have forced the "grneeded" array into the main data segment instead of giving it its own far data segment. That meant the main data segment didn't have enough free space for the stack and the environment (command line parameters and such).
That's a bit unexpected, given that grneeded is clearly defined as a part of far memory. Maybe it's a compiler bug or another related glitch; I recall having problems while using Borland C++ 3.0, where you could unexpectedly get compilations errors, but not in a consistent manner. Maybe it happened in my Wolf3D tree.
NY00123
Vorticon Elite
Posts: 508
Joined: Sun Sep 06, 2009 19:36

Re: Recreated Keen 4-6 Source Code

Post by NY00123 »

ok, I realize that this is technically another post, but (ignoring the "LZ91" signature), I think that I fixed the problem with the generation of Keen 6 v1.5's exe!

This didn't impact builds of other exes for me. Note that my testing was done without changing the indentation of most of the loop body.

Code: Select all

diff -ur keensource456/KEEN4-6/CK_KEEN.C keensource456_mod/KEEN4-6/CK_KEEN.C
--- keensource456/KEEN4-6/CK_KEEN.C	2021-07-01 02:44:06.000000000 +0300
+++ keensource456_mod/KEEN4-6/CK_KEEN.C	2021-07-02 16:33:08.819513300 +0300
@@ -2018,10 +2018,8 @@
 	{
 		for (x = ob->tileleft; x <= ob->tileright; x++, map++)
 		{
-			if ((intile = tinf[INTILE + *map] & (INTILE_FOREGROUND-1)) == 0)
-				continue;
-
-			switch (intile)
+			if ((intile = tinf[INTILE + *map] & (INTILE_FOREGROUND-1)) != 0)
+				switch (intile)
 			{
 			case INTILE_DEADLY:
 				KillKeen();
@@ -2095,15 +2093,6 @@
 
 void KeenSimpleReact(objtype *ob)
 {
-#ifdef KEEN6Ev15
-	// The original v1.5 executable has a padding byte at the end of the
-	// CheckInTiles routine (between the retf and the jump table). I could
-	// not get the compiler to generate that padding byte, so I inserted
-	// a nop in here to make all following routines start at the correct
-	// position. This means only the jump table offset and any refrerences
-	// to KeenSimpleReact will differ from the original Keen 6 v1.5 code.
-	asm nop;
-#endif
 	RF_PlaceSprite(&ob->sprite, ob->x, ob->y, ob->shapenum, 0, ob->priority);
 }
 
User avatar
K1n9_Duk3
Vorticon Elite
Posts: 781
Joined: Mon Aug 25, 2008 9:30
Location: Germany
Contact:

Re: Recreated Keen 4-6 Source Code

Post by K1n9_Duk3 »

Awesome! I just tested it myself. Compiling the modified code, compressing the executable and hex-editing the compressed file to get rid of the LZ91 string gave me a file with the same size and checksum as the original. Thank you for looking into this.

I'm currently going though the source once more to add a couple of things I forgot. RF_PlaceSprite calls now use 'spritedraw' instead of '0' as the drawtype parameter and I'm also adding some comments to indicate what the 'temp1' to 'temp4' fields are used for in each actor type's code.
Hail to the K1n9, baby!
http://k1n9duk3.shikadi.net
User avatar
ckguy
Bipship Engineer
Posts: 1169
Joined: Thu Nov 01, 2007 17:56
Location: Wakefield, RI, US
Contact:

Re: Recreated Keen 4-6 Source Code

Post by ckguy »

If there's one thing that's going to make me post after five years of not posting, it's this. Holy moly this is super amazing.

Thank you so much for all your work on this! I'm still trying to figure out what I want to do with this. Maybe revive my Keen 5 levelpack from ages ago? Who knows? I've just been playing around with this for a bit and making silly small changes to various sprite behavior. This is super super amazing.
Post Reply