COSMO'S COSMIC ADVENTURE SOURCE CODE RECONSTRUCTED!

This is where you can post your Commander Keen related stories, artwork, or other stuff that is related to Commander Keen but otherwise doesn't belong in another forum.
User avatar
T-Squared
Vortininja
Posts: 205
Joined: Fri Nov 26, 2010 21:22
Location: Next Sunday, A.D.
Contact:

COSMO'S COSMIC ADVENTURE SOURCE CODE RECONSTRUCTED!

Post by T-Squared »

https://github.com/smitelli/cosmore

Yes, you read that right. This is NOT a joke! I am so grateful that this has happened.

One Mr. Scott Smitelli has taken the time to look through the hex assembly of the game and reconstitute it into C (or C++, I'm not sure because I am not versed in either language, although a variant of either of them is used in Marlin 3D printer configuration).

I am, however, able to get into the higher levels of the code and modify them (i.e. text box and image ordering), thankfully, because of the in-code comments and my very-long experience in figuring how this game works just under the surface.

Thanks to this, I'm slowly finding my footing once again. I will be able to put MOST of my ideas into this game AT LAST! (Except for that infamous "Sound Blaster- Talking Cosmo" idea that I had... unless...)

Even better, this is MUCH easier! (Much of the coding is put together in a way that the subroutines can be copied to add or subtract text, pictures, or ideas!)
Last edited by T-Squared on Sun Dec 06, 2020 20:31, edited 1 time in total.
Cosmo II: HUMANIZED!!! Progress:
Graphics: 15% Complete
Story: 100% Complete
Music: 5% Complete
Programming Modifications: 5% Complete (FINALLY! SOME PROGRESS!)

http://www.tsqproductions.com
User avatar
Tormentor667
Vortininja
Posts: 180
Joined: Sat Mar 29, 2008 14:08
Location: Germany
Contact:

Re: COSMO'S COSMIC ADVENTURE SOURCE CODE RECONSTRUCTED!

Post by Tormentor667 »

So is something like a modern Cosmo sourceport happening?
User avatar
Nisaba
Vorticon Elite
Posts: 1262
Joined: Fri Jan 01, 2016 23:34
Location: The Outpost
Contact:

Re: COSMO'S COSMIC ADVENTURE SOURCE CODE RECONSTRUCTED!

Post by Nisaba »

the readme on Github states:
Cosmore is a reconstruction of the source code of Cosmo's Cosmic Adventure version 1.20, using the original DOS compiler and toolchain. Its goal is to duplicate every detail, quirk, and bug of the original game as faithfully as possible. A player should not be able to distinguish a Cosmore binary from the original.

The reconstruction is 96.28% accurate by the following metric: The load images (skipping the EXE headers and relocation tables), when compared byte-by-byte against the corresponding uncompressed originals, have the same byte values in the same locations 96% of the time. The remaining 4% consists of uninitialized data segment memory addresses that have not (yet!) been arranged to match the original layout.
Foray in the Forest progress:
⏺⏺⏺⏺⏺ ⏺⏺⏺⏺⭘ | Programming
⏺⏺⏺⏺⏺ ⏺⏺⏺⏺⭘ | GFX
⏺⏺⏺⏺⏺ ⏺⏺⏺⭘⭘ | Levels
⏺⏺⏺⏺⏺ ⏺⏺⏺⏺⏺ | Creatures
⏺⏺⏺⏺⏺ ⏺⏺⭘⭘⭘ | Story
⏺⏺⏺⏺⏺ ⏺⏺⏺⏺⭘ | Music
⏺⏺⏺⏺⏺ ⏺⏺⏺⏺⭘ | SFX
⏺⏺⏺⏺⏺ ⏺⏺⏺⏺⭘ | Extras

Other projects:
- Play Commander Keen on a GameBoy
- The Lynx Nyx
User avatar
T-Squared
Vortininja
Posts: 205
Joined: Fri Nov 26, 2010 21:22
Location: Next Sunday, A.D.
Contact:

Re: COSMO'S COSMIC ADVENTURE SOURCE CODE RECONSTRUCTED!

Post by T-Squared »

I would be putting info of how the engine works up on the Modding Wiki, but all my avenues of logging in are gone.

I wasn't banned or anything. To be honest, the Modding Wiki is very broken. I lost my password, and the files on the server have so many viruses that Google has blocked the e-mail service there, so even making a new account is impossible.
Cosmo II: HUMANIZED!!! Progress:
Graphics: 15% Complete
Story: 100% Complete
Music: 5% Complete
Programming Modifications: 5% Complete (FINALLY! SOME PROGRESS!)

http://www.tsqproductions.com
User avatar
Nisaba
Vorticon Elite
Posts: 1262
Joined: Fri Jan 01, 2016 23:34
Location: The Outpost
Contact:

Re: COSMO'S COSMIC ADVENTURE SOURCE CODE RECONSTRUCTED!

Post by Nisaba »

which Modding Wiki are you referring to?
Foray in the Forest progress:
⏺⏺⏺⏺⏺ ⏺⏺⏺⏺⭘ | Programming
⏺⏺⏺⏺⏺ ⏺⏺⏺⏺⭘ | GFX
⏺⏺⏺⏺⏺ ⏺⏺⏺⭘⭘ | Levels
⏺⏺⏺⏺⏺ ⏺⏺⏺⏺⏺ | Creatures
⏺⏺⏺⏺⏺ ⏺⏺⭘⭘⭘ | Story
⏺⏺⏺⏺⏺ ⏺⏺⏺⏺⭘ | Music
⏺⏺⏺⏺⏺ ⏺⏺⏺⏺⭘ | SFX
⏺⏺⏺⏺⏺ ⏺⏺⏺⏺⭘ | Extras

Other projects:
- Play Commander Keen on a GameBoy
- The Lynx Nyx
User avatar
K1n9_Duk3
Vorticon Elite
Posts: 726
Joined: Mon Aug 25, 2008 9:30
Location: Germany
Contact:

Re: COSMO'S COSMIC ADVENTURE SOURCE CODE RECONSTRUCTED!

Post by K1n9_Duk3 »

This is good news, but it's also not totally new.

By all means, I do not want to discredit the author of this recreation. I know that it must have taken him a lot of time and effort to get to this point. But we already have Cosmo Engine, which is also a recreation of the code based on disassembling/decompiling the original executables, except for it being an SDL "port" of that game for modern operating systems.

Having a DOS source code recreation of the game is indeed extremely useful for one purpose, and that's verifying that the recreated code works exactly like the original code. When you translate machine code back into assembly code and/or higher-level language code (C code in this case), you have to do at least some of that work by hand. And whenever you do that, you are bound to make mistakes here and there. (If you don't believe me, go and write the first 1000 digits of Pi onto a piece of paper without making a mistake.) But compiling the new code with the exact same tools that were used to create the original game gives you the chance to compare the generated machine code directly to the original machine code. Getting a 100% match is pretty much the easiest (and perhaps the only) way to prove that the recreated code will always behave exactly like the original program.

Proving that two programs are in fact identical is one of the big problems in computer science. This problem is, in general, undecidable. Solving this problem would also solve the halting problem, which has been proven to be unsolvable. This doesn't mean that it's always indecidable whether two programs are identical, it's just not always possible to decide. If the machine code can be proven to be 100% identical, then it's obvious that the two programs have to be identical as well.

Anyway, back to Cosmo. I don't really see anything else that would be worth documenting on the ModdingWiki. The file formats are all there and their documentation appears to be complete. I'm not sure what you would like to add. Since the source code recreation exists, modders can just read that source code (it appears to be extremely well commented) and go from there. The ModdingWiki never had tutorials on how to actually perform certain modifications to the source code of Wolf3D, for example, so I don't think we would need anything like that for Cosmo. If you want to add something like this anyway, I'd recommend discussing this with Malvineous first.

And just in case: Please don't start documenting how to hex-edit the game's executable to change the behavior of certain enemies or how to change the story texts. You have access to source code. Use the source! Think! At this point, any hex-editing is a pointless waste of time. You can do things so much faster and easier if you just edit the source and re-compile it. Believe me, I've done a lot of "patching" work on Keen mods recently, while also working on a true source code mod. If I had any kind of power over this community, I would make patching illegal and force everybody to do source code mods.
Hail to the K1n9, baby!
http://k1n9duk3.shikadi.net
User avatar
Nisaba
Vorticon Elite
Posts: 1262
Joined: Fri Jan 01, 2016 23:34
Location: The Outpost
Contact:

Re: COSMO'S COSMIC ADVENTURE SOURCE CODE RECONSTRUCTED!

Post by Nisaba »

amen!
Foray in the Forest progress:
⏺⏺⏺⏺⏺ ⏺⏺⏺⏺⭘ | Programming
⏺⏺⏺⏺⏺ ⏺⏺⏺⏺⭘ | GFX
⏺⏺⏺⏺⏺ ⏺⏺⏺⭘⭘ | Levels
⏺⏺⏺⏺⏺ ⏺⏺⏺⏺⏺ | Creatures
⏺⏺⏺⏺⏺ ⏺⏺⭘⭘⭘ | Story
⏺⏺⏺⏺⏺ ⏺⏺⏺⏺⭘ | Music
⏺⏺⏺⏺⏺ ⏺⏺⏺⏺⭘ | SFX
⏺⏺⏺⏺⏺ ⏺⏺⏺⏺⭘ | Extras

Other projects:
- Play Commander Keen on a GameBoy
- The Lynx Nyx
User avatar
XkyRauh
Mortimer's RightHand Man
Posts: 381
Joined: Thu Nov 01, 2007 16:32
Location: San Diego, California

Re: COSMO'S COSMIC ADVENTURE SOURCE CODE RECONSTRUCTED!

Post by XkyRauh »

Can you guys please help me understand the difference between "patching" and "source code mods?"

Like... a source code mod would be changing the actual game code, changing the hard-coded values for, say, text and graphics. Modding would be... intercepting values as they are placed in memory, to change the values for, say, text and graphics? What's the operative difference and why are you so in favor of one and not the other? I feel lost and confused. :)
User avatar
K1n9_Duk3
Vorticon Elite
Posts: 726
Joined: Mon Aug 25, 2008 9:30
Location: Germany
Contact:

Re: COSMO'S COSMIC ADVENTURE SOURCE CODE RECONSTRUCTED!

Post by K1n9_Duk3 »

A source code mod means having full access to everything (the full source code), and most importantly, being able to easily add entirely new features. You don't have to worry about whether or not the code will fit in the existing executable, since the compiler generates a new executable for you. Also, the source code is written in a high-level language that a human being can actually read and learn to understand. If you can't figure out how to accomplish something, you can ask for help and are far more likely to actually find somebody who can help you. There are people out there who understand the programming language.

Patching means hacking/editing the game's executable by changing certain constants to different values or modifying the machine code to behave differently. Unless you actually understand machine language (or at least 8086 assembly), you won't be able to change the code at all. And even worse, since you're just modifying the existing executable, you cannot add any new code (or data), at least not as easily as you could when compiling source code. If you want to add something new, you'll first have to find some code/data that you can safely overwrite with the new additions without causing the game to crash. That means you'll have to know the executable inside and out. You must know exactly what each block of "zeroes and ones" is for and under which circumstances it is used. Otherwise you'll have no way of knowing for sure which parts can be overwritten and which parts need to stay intact.

I think the vast majority of Keen modders actually have little to no knowledge of 8086 machine language or assembly language and rely entirely on talented patchers to do the work for them. Taking patches from the KeenWiki and putting them into a patch file does not mean you know anything about patching. Modifying the patches according to the instructions on the KeenWiki does not mean you know anything about patching. And what's worse: While I was on my Galaxy Mod Bug Hunt, I came across a couple of bad patches that caused memory corruption when their code was executed. Some of those came straight from the KeenWiki. In at least one case, the cause of the problem was a miscalculated jump address. This is a mistake that you can only make when you're writing machine code by hand. Who knows how many other errors lurk within the massive amount of patches on the KeenWiki. I'm certainly not planning to go through all of them and check them. That's anyother reason why I'd recommend compiling source code instead of patching executables.

Let me give you a very specific example about the benefits of working with source code. Foray in the Forest (a source code mod) currently has 688 sprites in its EGAGRAPH file. Keen 4-6 each have less than 400 sprites. It would be incredibly difficult if not outright impossible to "patch" such a large amount of sprites into any existing Keen game because the number of sprites is hard-coded into the game and there are fixed-length arrays in the executable that rely on these numbers. And even if you could somehow manage to patch it in, you would hardly have enough space to actually implement all of the AI code to use each of these sprites in the game.

Sure, you could probably do some impressive stuff with patching if you really want to. But then you'd be rewriting half of the game. In machine code. By hand! That's insane. We invented assemblers and compilers so that we don't have to do that anymore.

And one last thing: Patching doesn't necessarily mean intercepting the values as they are loaded into memory. Sure, that's how the CKPATCH tools work, but there are other ways to achieve the same goal. T-Squared already released an early beta of his Cosmo mod, and that release came with a fully "patched" executable. I guess in this case "patched" actually means "hex-edited". And for Keen mods (or any other game, really), you can use my patching utility to generate fully patched executables based on the CKPATCH patch scripts.
Hail to the K1n9, baby!
http://k1n9duk3.shikadi.net
User avatar
T-Squared
Vortininja
Posts: 205
Joined: Fri Nov 26, 2010 21:22
Location: Next Sunday, A.D.
Contact:

Re: COSMO'S COSMIC ADVENTURE SOURCE CODE RECONSTRUCTED!

Post by T-Squared »

Yeah, I have NO experience with 8086 assembly language.

I had to go in with a hex editor and severely truncate my story to fit within the established parameters. Not fun. There was always a risk that I would corrupt the entire program by pressing Backspace, moving every instruction out of alignment. But now, with a 96% accurate engine in C, with easy-to-understand concepts distilled into commanding subroutines (like "CanBeExploded" or "DrawTextFrame [this position, this size]" or "DrawInFront"), I can break and change several limits that are within my ability, meaning I can put most of my ideas into the game that I had originally planned. (and then some, because it means I can add extra ideas that I had never thought about before or that the programmers never intended before. :D)
Cosmo II: HUMANIZED!!! Progress:
Graphics: 15% Complete
Story: 100% Complete
Music: 5% Complete
Programming Modifications: 5% Complete (FINALLY! SOME PROGRESS!)

http://www.tsqproductions.com
Benvolio
Vorticon Elite
Posts: 775
Joined: Sun May 29, 2011 12:43
Location: Ireland
Contact:

Re: COSMO'S COSMIC ADVENTURE SOURCE CODE RECONSTRUCTED!

Post by Benvolio »

K1n9_Duk3 wrote:rely entirely on talented patchers to do the work for them.
Any why not? History has space in its pages for Stradivarius and Mehuhin. The makers of patches presumably have done so because of a love for the hobby of doing so. And not under duress. (I always lament the thankless way some patchers have been spoken to by some modders in the past but that's another story).

I would discourage looking down on mods made without the author having programming expertise. One of the Keen community's sources of lifeblood has always been how accessible so many of its pursuits are to those with only slightly above average computer literacy (e.g. kids aged 11, and those same kids 20 years later having pursued non computer science livelihoods).

Modding is art and science. The collaboration of artists and scientists can yield beautiful things!
User avatar
XkyRauh
Mortimer's RightHand Man
Posts: 381
Joined: Thu Nov 01, 2007 16:32
Location: San Diego, California

Re: COSMO'S COSMIC ADVENTURE SOURCE CODE RECONSTRUCTED!

Post by XkyRauh »

Thank you, K1n9_Duk3, for that in-depth response!

You've pretty much identified exactly why I lost steam on XkyKeen4 and eventually gave up. I have no experience working with "The Stack" and assembler language, and all I was doing was poking around in the .EXE with a hex editor and trying to intuit what the hell was happening with the already-compiled program. A fool's errand. :)

I have some experience with C# and BASIC, and the thought of a "source code" version of Keen:Galaxy makes the prospect of a mod a whole lot more feasible, in my mind. I'm glad to hear that "Foray in the Forest" is making use of that!

One of my goals when I made XkyKeen1, 2, and 3 was to patch as little as possible. I didn't change enemy behaviors, I didn't change the amount of text in the story, I didn't change the number of collectibles necessary to complete the game. I just changed level layouts and graphics, working within the confines of the game as it existed--so in a sense, the "patching" version worked just fine for me--with the huge glaring exception of Andrew Durdin's patching that helped make the "time's up, you're dead!" thing possible in XkyKeen3. I'm still really grateful to him for figuring that out for me. :)

I've never used any of the a-la-carte patches from the KeenWiki, but I can see why one might be resentful or frustrated by a patch offering that calculates a JMP instruction incorrectly or otherwise causes memory leaks. :\ That's definitely something to be avoided.

I think, having heard your explanation, I agree with you on the big picture--but I have to say, without "relying on experts" like adurdin, XkyKeen3 wouldn't have happened (12 years ago lol).
User avatar
K1n9_Duk3
Vorticon Elite
Posts: 726
Joined: Mon Aug 25, 2008 9:30
Location: Germany
Contact:

Re: COSMO'S COSMIC ADVENTURE SOURCE CODE RECONSTRUCTED!

Post by K1n9_Duk3 »

There's nothing wrong with seeking help from experts per se. The thing is, when everybody is relying on a small number of active "experts" in the community, any progress will be tediously slow, depending on how much time those experts can afford to invest. And once those experts decide to stop helping out for whatever reason, any (advanced) modding will come to a dead stop.

Learning C is a lot easier than learning assembly language and figuring out how to disassemble the executable and then figuring out what the disassembled code actually does. If you know for a fact than an enemy takes 3 hits to stun, it's a lot easier to search the C code to track that number down and change it into something else for your mod. Trying to do the same with only a hex dump of the executable will drive you insane. A raw disassembly without any meaningful names for functions and variables is only slightly more helpful, assuming you understand assembly to begin with.

The experts have already gone through the hassle of figuring out what each part of the code actually does. That's why we now have source code recreations like Cosmore and Cosmo Engine for Cosmo and the source modding package for Keen: Galaxy style mods as well as the Omnispeak port of Keen 4-6. Heck, we even have the full, original source code of Keen Dreams publicly available since 2014, but in 2019, we still saw the release of a (rather buggy) Keen Dreams mod based entirely on patching.

I can only speak for myself, but I also don't see anybody else actively creating new patches at this point. I would rather help you write C code for your mods than write patches. I'm not fluent in machine code, which means I write all of my patches in (pseudo-)assembly and then manually translate the code into machine language for the patch file. Sometimes I write a short assembly program, assemble/compile it and then disassemble it to get the actual machine code for each instruction so that I can type the hex values into the patch scripts. Once the patch script is done, I apply it to the executable with my patching utility and run the fully patched executable through the disassembler to verify that the patch script actually generates the code I want. (A great way to detect miscalculated jump addresses btw.) That's a long and tedious process for me. Writing C code is a lot faster and easier. I, as a self-proclaimed "expert", could help a lot more in a lot less time if everybody was doing source modding instead of patching.
Last edited by K1n9_Duk3 on Thu Dec 10, 2020 21:18, edited 1 time in total.
Hail to the K1n9, baby!
http://k1n9duk3.shikadi.net
User avatar
Nisaba
Vorticon Elite
Posts: 1262
Joined: Fri Jan 01, 2016 23:34
Location: The Outpost
Contact:

Re: COSMO'S COSMIC ADVENTURE SOURCE CODE RECONSTRUCTED!

Post by Nisaba »

still off topic:
I absolutely venerate our patching veterans for all the reasons K1n9_Duk3 just mentioned. To me it is freaking mind blowing that some guys of this fine community seem to be able to read assembly code with ease. That's a very rare and a highly specialized skill.
At one point I went down that road as well and managed to learn the bare basics, digging through tons of hex dump of the executable, trying to arrange code segments using near calls and jumps, getting to ruffly understand the overall structure and conditions of game variables, and so on...
And boy, each time I was actually working on patches, my brain just started to explode. Understanding the structure is one thing. And I won't claim that I actually did. But keeping the concentration high all the time while digging for clues, decipher hex, shifting things around, and avoiding to make just the slightest mistakes (debugging is hell!), gives me the creeps.

So kudos to all our patching heroes! they did a darn good job. without them we as a community wouldn't be were we are today. I mean, look at all those fabulous mods and how they evolved over time. it was them who went to the time and effort of making patches available for all of us. and those patches truly pushed the boundaries of what was possible to achieve modding wise over time. (and yeah, it is true, many of these patches are buggy in one way or another. but c'mon, without the skill and patience of our patching elite we wouldn't even have any of those fine patch based mods.)

that all said, thank heaven we do have the option to work with the source code by now. it's a real game changer. Even a coding noob like me might be able to learn how to program in C without getting insane. So, yeah. I guess K1n9_Duk3 is right that we should shift towards source code modding. Although I guess that it depends on what you want achieve. Wiki is well documented in terms of existing patches. And if you have nothing fancy in mind, this should still do the trick.
however, for every more advances project I'd recommend a source code base. chances are way higher that you'll find someone capable of understanding C and stuff. to establish this new way of modding we do need to write a wiki, as this seems to be the starting point for most novice modders these days. I'd be happy to take care of the formatting and stuff, but I'd need an expert or two who can write down a pertinent tutorial on how to setup up files and folders, plus all that is to it to get started. just like always: to chance some you've gotta do some... I for one am in.
Foray in the Forest progress:
⏺⏺⏺⏺⏺ ⏺⏺⏺⏺⭘ | Programming
⏺⏺⏺⏺⏺ ⏺⏺⏺⏺⭘ | GFX
⏺⏺⏺⏺⏺ ⏺⏺⏺⭘⭘ | Levels
⏺⏺⏺⏺⏺ ⏺⏺⏺⏺⏺ | Creatures
⏺⏺⏺⏺⏺ ⏺⏺⭘⭘⭘ | Story
⏺⏺⏺⏺⏺ ⏺⏺⏺⏺⭘ | Music
⏺⏺⏺⏺⏺ ⏺⏺⏺⏺⭘ | SFX
⏺⏺⏺⏺⏺ ⏺⏺⏺⏺⭘ | Extras

Other projects:
- Play Commander Keen on a GameBoy
- The Lynx Nyx
User avatar
entropicdecay
Mad Mushroom
Posts: 511
Joined: Thu Nov 01, 2007 15:00
Contact:

Re: COSMO'S COSMIC ADVENTURE SOURCE CODE RECONSTRUCTED!

Post by entropicdecay »

Do we have a source recreation for Keen: Vorticons?
Post Reply