Recreated Keen 4-6 Source Code

Here is where to post about the latest Commander Keen fangame or modification you've finished, a new website you've made, or another Keen-related creation.
Calvero
Vortininja
Posts: 98
Joined: Tue Jan 29, 2008 15:31

Re: Recreated Keen 4-6 Source Code

Post by Calvero »

NY00123 wrote: Fri Jul 02, 2021 12:01 One may give the CGA versions a try now! (Just kidding...or am I?)
How difficult is it to recreate the CGA versions?
We have the Keen Dreams CGA source code, right?
User avatar
K1n9_Duk3
Vorticon Elite
Posts: 781
Joined: Mon Aug 25, 2008 9:30
Location: Germany
Contact:

Re: Recreated Keen 4-6 Source Code

Post by K1n9_Duk3 »

I just uploaded a new version of the code package (same download link as before, it might take a while before the server caches clear and you get the new file instead of the old one).

This is what has been changed:
  • added NY00123's changes to allow 100% identical recreations of Keen 6 EGA v1.5
  • added some more comments to the source code, mostly explaining what the temp variables are used for in each actor's code and pointing out possible causes for bugs or crashes
  • changed the function names in the Keen 6 code from ...Think, ...Contact and ...React to T_..., C_... and R_... for more consistency with the Keen 4 and Keen 5 code, also changed some state and sprite names for more consistency
  • fixed some mistakes in the readme.txt file
  • added a patch script for removing the "LZ91" signature from the LZEXE-compressed Keen 6 v1.5 executable
Calvero wrote: Sat Jul 03, 2021 16:55 How difficult is it to recreate the CGA versions?
We have the Keen Dreams CGA source code, right?
Probably not too difficult. The main CGA code is already in this package. The Catacomb 3-D source code had all the CGA code in place, despite being an EGA-only game. There may be a couple of changes necessary to make enemies flash white when they get shot in Keen 5 and 6. I had to add the code responsible for these draw modes to the EGA files, so I would guess that it would need to be added for CGA as well.

I just browsed through a Keen 4 CGA v1.4 disassembly earlier today and found that almost all of the actor code is identical for the CGA version, only the HandleRiding routine in CK_KEEN.C differs. And of course there's a lot of code to disable, like the Star Wars text and the Terminator intro. The Keen 5 game over sequence would need to be reprogrammed for CGA, since the current version is low-level EGA code.

Honestly, I don't really see the point, except for some obsessive-compulsive desire to recreate every posible version of Keen 4-6 that ever existed. The CGA versions are clearly inferior versions of the games, both in terms of the number of colors and the smoothness of the movement in-game. If you want the CGA look, you could just fake that in EGA, so I'd consider CGA support a low priority.

Judging by NY00123's post, a CGA version might already be in the works, so for the moment I'm gonna wait and see what happens.
Hail to the K1n9, baby!
http://k1n9duk3.shikadi.net
User avatar
proYorp
Vorticon Elite
Posts: 472
Joined: Fri Mar 03, 2017 1:56
Location: Orbit
Contact:

Re: Recreated Keen 4-6 Source Code

Post by proYorp »

K1n9_Duk3 wrote: Thu Jul 01, 2021 1:38 you can compile the code and compress the new executables with LZEXE and get 100% identical copies of the original v1.4 executables!
NY00123 wrote: Fri Jul 02, 2021 12:01 ok, I tried to build the exes identified as v1.4, and it's indeed possible to byte-by-byte recreate the originals!
Now that's impressive. I think I've seen some very, very old forum threads wondering if, and hoping that, this was possible. This might just mark the beginning of the source modding era proper. Imagine that, twenty years after modding began and we are still making new advances.

I wonder about how usable/accessible this will be for the average modder, though. This sounds like it would require actual knowledge and understanding of low-level programming languages (namely C++) to use, which can take years to learn from what I hear. Whereas copy-pasting patches from KeenWiki (possibly with minimal alterations) does not take much skill to accomplish. This is just my at-a-glance impression (correct me if I'm wrong here), but for smaller, less ambitious projects (as far as code modification goes) I'm inclined to think it might be simpler to stick with that. (Certainly patching should still be an option. :p ) Nevertheless, this is undoubtedly a great achievement.
"Friendly. Very friendly. Too friendly." Image
Calvero
Vortininja
Posts: 98
Joined: Tue Jan 29, 2008 15:31

Re: Recreated Keen 4-6 Source Code

Post by Calvero »

K1n9_Duk3 wrote: Sat Jul 03, 2021 20:47 Honestly, I don't really see the point, except for some obsessive-compulsive desire to recreate every posible version of Keen 4-6 that ever existed. The CGA versions are clearly inferior versions of the games, both in terms of the number of colors and the smoothness of the movement in-game. If you want the CGA look, you could just fake that in EGA, so I'd consider CGA support a low priority.
I guess having the Keen CGA source code will make it easier to create a 320x200 16 color Tandy version. And Keen 6 in 16 color CGA composite mode.
NY00123
Vorticon Elite
Posts: 508
Joined: Sun Sep 06, 2009 19:36

Re: Recreated Keen 4-6 Source Code

Post by NY00123 »

Nice to see that you already got an updated archive ready.
K1n9_Duk3 wrote: Sat Jul 03, 2021 20:47I just browsed through a Keen 4 CGA v1.4 disassembly earlier today and found that almost all of the actor code is identical for the CGA version, only the HandleRiding routine in CK_KEEN.C differs. And of course there's a lot of code to disable, like the Star Wars text and the Terminator intro. The Keen 5 game over sequence would need to be reprogrammed for CGA, since the current version is low-level EGA code.
I didn't think of or know all differences, but obvious examples I recall are the star wars text and the palette fades; Also the less smooth horizontal scrolling. I still don't think it'll be too much work, especially as we already have the CGA drawing routines from the Keen Dreams and Catacomb 3-D sources. My general assumption was that CGA support was originally added to Keen Dreams in order to potentially increase the sales for Softdisk, while the ability to re-use the code in Keen 4-6 was essentially a bonus. I recall reading about the lower sales of the CGA versions of Keen 4-5 proving to Apogee that the CGA market was dead at the time. As things turned out later, the next id-owned title, Wolfenstein 3D, also abandoned the EGA, albeit Apogee still didn't, and a few later games published by Apogee still support the EGA.
Judging by NY00123's post, a CGA version might already be in the works, so for the moment I'm gonna wait and see what happens.
Hehe, I know that it was maybe a bit confusing, but I haven't been doing any related work on this; Doesn't mean that others can't, of course!

What I did check, was to see how compatible is Keen 4-6 v1.4/1.5's updated ID Engine with the available Catacomb 3-D codebase. Of course, it's mostly a novelty, and the same also applies to using this code with Keen Dreams. After all, none of these games originally had id's updates to the ID Engine in newer versions of the games, which were prepared by people not from id.

I found in keensource456's ID_US_1.C a kind of a small typo, where the "jerk" variable is read/written for all games, not just for Keen. However, the situation is a bit more complicated in ID_US_2.C (details are given below). There's also ID_VW_AE.ASM's use of the variable "jerk".

Regarding ID_US_2.C:
- The released Catacomb 3-D sources still technically have definitions for the options menu, covering the scorebox and SVGA compatibility for all games, including Catacomb 3-D. Only two-button firing is exclusive to Keen.
- However, the options menu itself is not accessible in Catacomb 3-D, even though it's technically present in the exe.
- Keen 4-6 v1.4 have two options which can be useful for Catacomb 3-D, namely the jerky motion fix and the gamepad toggle; Maybe the jerky motion fix is less relevant, as the game doesn't sync to vblank once per frame, unless you're using something like the "extra VBLs" debug option.
- Adding gamepad support to Catacomb 3-D might or might not be a bit more complicated, if only due to the need to decide which in-game actions can be mapped to them.
- Finally, even more work might be required for The Catacomb Adventure Series, since it had the whole menu replaced with separate message boxes / dialogs. Similar complications may arise for Keen Dreams. Yeah, you can use Keen 4-6's menus with it, but there are still missing features, like access to the help texts.
proYorp wrote: Sun Jul 04, 2021 7:14I wonder about how usable/accessible this will be for the average modder, though. This sounds like it would require actual knowledge and understanding of low-level programming languages (namely C++) to use, which can take years to learn from what I hear. Whereas copy-pasting patches from KeenWiki (possibly with minimal alterations) does not take much skill to accomplish. This is just my at-a-glance impression (correct me if I'm wrong here), but for smaller, less ambitious projects (as far as code modification goes) I'm inclined to think it might be simpler to stick with that. (Certainly patching should still be an option. :p ) Nevertheless, this is undoubtedly a great achievement.
For small or existing changes, patches can still be used, at least as long as nothing gets broken as a consequence. On the other hand, scripting and source modding should both be significantly more straightforward than creating new patches from scratch. The latter requires analysis of EXE's contents - either as machine mode or as x86 assembly - along with figuring out how to make changes within the original EXE's limits.
User avatar
K1n9_Duk3
Vorticon Elite
Posts: 781
Joined: Mon Aug 25, 2008 9:30
Location: Germany
Contact:

Re: Recreated Keen 4-6 Source Code

Post by K1n9_Duk3 »

Just to make it clear, you can't combine the old ways of patching with an already modified source code. CKPatch won't recognize the executable and abort. If you want to edit the source code, you'll have to stick to editing and compiling the source code. If you want to keep using patches, you can't use the source code (except for reference, I guess).
proYorp wrote: Sun Jul 04, 2021 7:14 I wonder about how usable/accessible this will be for the average modder, though. This sounds like it would require actual knowledge and understanding of low-level programming languages (namely C++) to use, which can take years to learn from what I hear.
I think the source code is far more accessible than patching. If you don't know anything about programming in C and you can't even be bothered to learn the basics (which is still much easier than trying to learn machine language for writing new patches), then you probably shouldn't be messing with the code in the first place. As was pointed out, the average user doesn't even (need to) know a thing about patching, they're just copying stuff from the KeenWiki and inserting it into their patch file. If the community wants to, we could have similar Wiki pages for commonly used source code changes.

And even those who do not know much about the code or programming in C can still browse the code and easily change a few numbers here and there. The vast majority of the patches on the KeenWiki are just changing numbers or skipping some code. Changing those numbers (or deleting some code) does not require you to learn or even master programming in C. Just one example: try to figure out which part of the code you would need to change to make the Shockshund wiggle its tail twice as many times every time it sits down. There is no such patch on the KeenWiki, but with the source code and the comments, it should be relatively easy to figure out, even for those who know nothing about programming in C. Or, for a slightly tougher challenge, try to find out how to make the Shockshund bark after it lands.

Perhaps the trickiest part is finding the right file if you want to change something but don't know where to find it. In that case, you should take a look at the header files (CK_DEF.H and K?_DEF.H), since these contain a list of nearly all the routines (and variables) from the CK_*.C and the K?_*.C files respectively. You generally don't need to edit the ID_* files for most mods, except maybe for changing the default high scores (found in ID_US_1.C) and the animated tiles that produce a sound (Keen 6 only, found in ID_RF.C).
Hail to the K1n9, baby!
http://k1n9duk3.shikadi.net
User avatar
ckguy
Bipship Engineer
Posts: 1169
Joined: Thu Nov 01, 2007 17:56
Location: Wakefield, RI, US
Contact:

Re: Recreated Keen 4-6 Source Code

Post by ckguy »

Is there a reason for the RCKx.BAT files to update the PATH before calling BC? Couldn't they just directly call C:\BC30\BIN\BC.EXE, etc. without updating the PATH and eliminate the batch files interacting poorly with each other when, say, switching between things using different compiler versions without restarting DOSBox?
User avatar
K1n9_Duk3
Vorticon Elite
Posts: 781
Joined: Mon Aug 25, 2008 9:30
Location: Germany
Contact:

Re: Recreated Keen 4-6 Source Code

Post by K1n9_Duk3 »

IIRC, the path to TASM.EXE must be in the PATH variable, otherwise the IDE wouldn't be able to compile the .ASM files. There might be a way around that by editing the TASM path in the IDE, but this is how I set up the compiler years ago and I kind of stuck to this batch file style.

If the PATH thing bothers you, you can change the batch files to set a completely new PATH instead of adding a directory to the old one. This would be a problem on real DOS/Windows systems, but not so much in DOSBox. Simply edit all the batch files and replace the "%PATH%" part with "Z:\" for DOSBox.

I should mention that I'm doing all of the Borland C++ related work on my trusty old Core 2 Quad Q6600 system running Windows XP (32 bit). To compile the code, I run the batch file directly in Windows using the built-in NT Virtual DOS Machine (NTVDM), which makes it compile the code a lot faster than under DOSBox. The modified PATH variable has never been an issue for me, since Windows resets the PATH setting when that command-line window closes (i.e. directly after exiting Borland C++).

In addition to that, I have my systems set up so that I can just right-click on any .BAT, .COM or .EXE file and select "Run in DOSBox" from the context menu, so that I usually don't have to drag-and-drop anything nor mount any directories manually in DOSBox.
Hail to the K1n9, baby!
http://k1n9duk3.shikadi.net
User avatar
ckguy
Bipship Engineer
Posts: 1169
Joined: Thu Nov 01, 2007 17:56
Location: Wakefield, RI, US
Contact:

Re: Recreated Keen 4-6 Source Code

Post by ckguy »

Yep, good call, it looks like the IDE does complain if it can't find TASM in the PATH. I wasn't hitting that before because when I was compiling without updating the PATH, I hadn't changed the .ASM files, and so it was using the compiled .OBJs for those without trying to run TASM and noticing that it couldn't find it.

I poked around the IDE's settings a bit and I found "Options > Transfer" which let me specify a full path for TASM (and this configuration is then saved in the current project), but then I got a message during building about not being able to find TASM2MSG. I poked around a bit more but didn't see how to specify where to find that without it being on the PATH. Oh well.

I don't think I have particularly strong opinions, then, about how this ought to be set up in the default batch files in the distributed archive. What's already here works well enough, and anyone who's doing anything more serious over an extended period is almost certainly going to want to have their own custom dev environment anyway.
User avatar
K1n9_Duk3
Vorticon Elite
Posts: 781
Joined: Mon Aug 25, 2008 9:30
Location: Germany
Contact:

Re: Recreated Keen 4-6 Source Code

Post by K1n9_Duk3 »

I just got the code to compile new CGA executables that are 100% identical to the original v1.4 CGA releases. Still need to check Keen 6 CGA v1.5 and then set set up a couple of new directories and project files to make it easier to compile CGA versions without interfering with the EGA versions.
Calvero wrote: Sun Jul 04, 2021 19:31 I guess having the Keen CGA source code will make it easier to create a 320x200 16 color Tandy version.
I don't know exactly how Tandy graphics work, but 16 colors means that it will need 4 bits per pixel (= 2 pixels per byte) instead of the 2 bits per pixel (= 4 pixels per byte) layout that CGA uses. You would probably need to rewrite all of the low-level CGA code to make this work. Using a 160x200 16 color mode would be closer to the CGA layout and probably make things easier to implement.

One thing to keep in mind is that the CGA code in Keen 4-6 has to copy a full-screen image from main memory into video memory every time the screen is updated or "scrolled". A Tandy machine might not even be fast enough to do this at acceptable speed with a 320x200 16 color screen, since the amount of data to copy would be twice the size of a CGA screen. The EGA version will always be faster when it comes to updating or scrolling the screen, as the whole engine was designed around the hardware capabilities of an EGA card with 256k of video memory.
Hail to the K1n9, baby!
http://k1n9duk3.shikadi.net
Calvero
Vortininja
Posts: 98
Joined: Tue Jan 29, 2008 15:31

Re: Recreated Keen 4-6 Source Code

Post by Calvero »

I guess in theory it's easier to modify the CGA version to support other graphics modes because it doesn't use hardware tricks unlike the EGA version. In practice, I have no idea how fast it will run.

Btw, somebody already created a 160x200 16 color Tandy version of Keen 4 by hacking the Keen 4 composite CGA hack.
NY00123
Vorticon Elite
Posts: 508
Joined: Sun Sep 06, 2009 19:36

Re: Recreated Keen 4-6 Source Code

Post by NY00123 »

From what I recall, most of the work done for the Composite CGA conversions of Keen 4 and 5 was in the graphics, rather than EXE modifications. More details can be found here: https://int10h.org/blog/2016/05/dopefis ... er-keen-4/
K1n9_Duk3 wrote: Tue Jul 06, 2021 22:34 I just got the code to compile new CGA executables that are 100% identical to the original v1.4 CGA releases. Still need to check Keen 6 CGA v1.5 and then set set up a couple of new directories and project files to make it easier to compile CGA versions without interfering with the EGA versions.
That's great! I knew that it would probably not take too much time at this point, given the circumstances, but still.
User avatar
ckguy
Bipship Engineer
Posts: 1169
Joined: Thu Nov 01, 2007 17:56
Location: Wakefield, RI, US
Contact:

Re: Recreated Keen 4-6 Source Code

Post by ckguy »

Yesterday I spent a while fiddling with stuff from a couple of disparate sources to create a version of ID_VW_AE.ASM with the changes from your ck456_screenfix.zip patch, and it was only once I succeeded that I noticed that this had already been done and was available in keensource456.zip as -ID_VW_AE.ASM! It's a small thing, but this could be documented a bit better. Combine them into one file with compiler macros to switch between them?

While I was working on that, I also noticed that, when building the whole project, reassembling changed .ASM files failed when the target .OBJ file already existed. The assembler would return "Error writing object file". (Recompiling .C files when the .OBJ already existed worked fine.) Is this because of the transfer settings for TASM in the project file? Is TASM refusing to overwrite the file unless it's given some command line option?
User avatar
K1n9_Duk3
Vorticon Elite
Posts: 781
Joined: Mon Aug 25, 2008 9:30
Location: Germany
Contact:

Re: Recreated Keen 4-6 Source Code

Post by K1n9_Duk3 »

Yeah, I wanted to add a note about -ID_VW_AE.ASM but it seems I never actually did that. All there is is this part of timeline.txt:

Code: Select all

2019-12-19:

- implemented Keen 4-6 version of VW_SetScreen (in _ID_VW_AE.ASM) and my
  customized version of the routine (in ID_VW_AE.ASM). swap the files if you
  want to use the original code
The whole point of this release was recreating something as close to the original sources as possible. My customized screen update code obviously wasn't part of the original source, so it had to be disabled somehow. Renaming the files seemed like the best solution at the time, since you can just rename the files without editing anything. Perhaps I cared a little too much about the source file's time stamps while I was coding all of this. Yes, you could set up the code to have both versions in the same file and select the one to use via macro definitions. But with the way the code is set up, you would either have to edit the TASM switches within each project or you would have to edit each project's ID_ASM.EQU files to change that macro, or you would have to put that macro definition directly into ID_VW_AE.ASM itself and edit it as needed. I think renaming the file is less of a hassle.

The original Keen 4-6 code has many more bugs, none of which have been fixed in this source code recreation (but most of them should at least be mentioned in comments). The only reason why I included this bug-fixed file was that it already existed in the source directory.

I'm not sure what's going on with your system regarding that TASM error. I used Borland C++ 3.1 (bundled with TASM 3.1, I think) for development and have never had that problem. Whenever I build the projects with BC30 (or BC20), I always delete all the .OBJ files in the OBJ directories anyway. And I run BC30 and BC31 directly in Windows XP instead of using DOSBox. The way I see it, the problem might be caused by some access rights issues (your system allows you to create new files, but not modify/overwrite any existing ones) or by the files being accessed by some other program, which prevented TASM from modifying it.
Hail to the K1n9, baby!
http://k1n9duk3.shikadi.net
User avatar
MoffD
Vorticon Elite
Posts: 1220
Joined: Thu Jul 05, 2012 17:30
Location: /dev/null
Contact:

Re: Recreated Keen 4-6 Source Code

Post by MoffD »

Awesome! I need to see if this will compile om my 98 box :D

Hopefully this will lead to some interesting new mods

Edit: Well, source mods. I've always been a bit hazy on proper terminology
mortimermcmirestinks wrote: Now I wish MoffD wasn't allergic to me.
Levellass wrote:You're an evil man.
Image
Post Reply