Getting Keen 4-6 and Dreams mods running on real DOS PCs

You can discuss anything about fan-made Commander Keen games here.
Post Reply
User avatar
K1n9_Duk3
Vorticon Elite
Posts: 779
Joined: Mon Aug 25, 2008 9:30
Location: Germany
Contact:

Getting Keen 4-6 and Dreams mods running on real DOS PCs

Post by K1n9_Duk3 »

I know I'm not the only one around here who tends to play some Keen games and mods on a real DOS PC from time to time. Unfortunately, most mods require very large amounts of memory, to the point where some of them won't even run without issues in DOSBox. Many mods even come with a disclaimer saying that they will only work in DOSBox and cannot be played on real DOS PCs.

The biggest problem (aside from the fact that you need to remove as many TSRs and device drivers as possible to free up memory) is the fact that the games appear to be unable to detect and use XMS for data storage, only EMS seems to work.
keenmaster486 wrote: Wed Jan 22, 2020 6:34 After all this, though, Keen refuses to detect both XMS and EMS at the same time. It grabs an extra 64K of RAM from the EMS, and refuses to get any more.
Well, today I decided to look up the specification of the XMS driver calls that the ID Memory Manager uses when trying to allocate XMS memory. That's where I found this:
Under DOS 5+, if CONFIG.SYS contains the line DOS=UMB, then no upper memory blocks will be available for allocation because all blocks have been grabbed by MS-DOS while booting.
And sure enough, after modifying the startup files of my primary DOS system, this is what the startup screen for Keen 4 looks like:
Image
It doesn't reach quite the amount of memory available to the game under DOSBox (which is 513k), but 507k is about as close as it gets. I am running MS-DOS 6.22 on that system, so perhaps you could gain a little more memory by replacing DOS 6.22's HIMEM.SYS and EMM386.EXE with third-party memory managers.

For reference, this is what my CONFIG.SYS looks like on that DOS 6.22 machine:

Code: Select all

DEVICE=C:\DOS\HIMEM.SYS /TESTMEM:OFF
DEVICE=C:\DOS\EMM386.EXE RAM I=B000-B7FF
DOS=HIGH,NOUMB
BUFFERS=15,0
FILES=30
LASTDRIVE=E
FCBS=4,0
My AUTOEXEC.BAT is empty, except for the usual PROMPT, PATH and SET BLASTER lines. So no DOSKEY, no CD-ROM or mouse drivers, nothing.
Hail to the K1n9, baby!
http://k1n9duk3.shikadi.net
User avatar
keenmaster486
Vorticon Elite
Posts: 542
Joined: Sun Mar 20, 2016 18:29
Location: Tranquility Base
Contact:

Re: Getting Keen 4-6 and Dreams mods running on real DOS PCs

Post by keenmaster486 »

Woah!

I vaguely suspected the UMB thing might affect it, but I was not prescient enough to actually try to remove it.

Thanks for figuring this out, K1n9_Duk3. I'll test it on my machines soon.
I flermmed the plootash just like you asked.
User avatar
thehackercat
Yorp Doctor
Posts: 669
Joined: Fri Aug 14, 2009 0:05
Location: Slug Village

Re: Getting Keen 4-6 and Dreams mods running on real DOS PCs

Post by thehackercat »

Very cool!

I had a working 486 until a couple years ago; after a hard drive failure I left it at my parents' house until I could find a replacement part. Turns out somebody got rid of it a while back.

I guess this means I can use this trick you've found as an excuse to get another one!
Image
User avatar
keenmaster486
Vorticon Elite
Posts: 542
Joined: Sun Mar 20, 2016 18:29
Location: Tranquility Base
Contact:

Re: Getting Keen 4-6 and Dreams mods running on real DOS PCs

Post by keenmaster486 »

OK, so I tested this trick on my 486 and my IBM PC AT (286).

On the 486 it works! But not nearly as well as your results, K1n9_Duk3... I wasn't able to get it to surpass 500K. It was grabbing enough memory for any of the big mods to function properly, but only barely. My best results came with JEMMEX, with something like 450K for Keen 4 (was only getting 30 or 40K of XMS for some reason). MEM /DEBUG doesn't reveal anything strange. Using HIMEM and EMM386 always results in less memory overall, and copying your CONFIG.SYS verbatim gets me about 440K if I remember right.

On the 286, it doesn't work at all. I get the same result as before. Not sure why. It has HIMEM.SYS and a proprietary EMM for my Everex RAM card (hardware XMS and EMS)
I flermmed the plootash just like you asked.
User avatar
K1n9_Duk3
Vorticon Elite
Posts: 779
Joined: Mon Aug 25, 2008 9:30
Location: Germany
Contact:

Re: Getting Keen 4-6 and Dreams mods running on real DOS PCs

Post by K1n9_Duk3 »

I guess the results may vary depending on the hardware configuration. The way the EMS and XMS memory allocation works in the Keen games is that the unused parts of the upper end of the address space are mapped to physical memory addresses. This area (anything between 640 kB and 1 MB) is usually reserved for accessing hardware like the video RAM (EGA/VGA memory starts at 640k a.k.a. 0xA000), BIOS ROM etc. If you have expansion cards or ROM chips installed that use parts of this address space, you obviously can't use those addresses to access additional memory.

I also only get around 42k of XMS used by Keen 4 when I run EMM386.EXE without the "I=B000-B7FF" part. Making these addresses usable gives me another 32k and thus boosts the total up to 74k.

Also, HIMEM.SYS from DOS 6.22 is unable to provide UMBs on its own, it relies on EMM386.EXE to provide UMBs. So if the proprietary EMM driver for your RAM card doesn't provide UMBs or is incompatible with HIMEM.SYS, the Keen games won't be able to use XMS. And since your system uses an actual RAM card, it's possible that the EMS driver doesn't provide any UMBs at all. EMM386 uses 386 code to remap the real-mode address lines to memory areas that can only be accessed in protected mode. Maybe that remapping is what's commonly referred to as "UMB". I think EMS cards don't need to remap anything, since they are real expansion cards with their own chips. They can just read from or write to their memory chips depending on which address lines are active on the ISA bus.

And just a heads up: While I was looking up the XMS commands used by Keen, I also found that XMS drivers don't have to implement these commands to be considered "XMS compliant". So when in doubt, go back to Microsoft's HIMEM.SYS.
Hail to the K1n9, baby!
http://k1n9duk3.shikadi.net
User avatar
krivulak
Grunt
Posts: 21
Joined: Tue Jun 16, 2015 18:48

Re: Getting Keen 4-6 and Dreams mods running on real DOS PCs

Post by krivulak »

I've given many, many hours looking in DOS memory management system and from what I can say, M$ memory manager is pure garbage. There is no way you can allocate more than 460k of memory because of... Well, I am not really sure.
I started fiddling with other, upper level modern memory managers like QEMM with no such luck, I tried alternatives to EMM386 which actually helped somewhat, but not really much. So far I am trying to make PTS-DOS to run on 486, but I am running into graphical problems with LCD - installation software is programmed with super odd horizontal frequency and LCDs aren't able to display that, so most attempts end with "Not supported display resolution" message.
User avatar
K1n9_Duk3
Vorticon Elite
Posts: 779
Joined: Mon Aug 25, 2008 9:30
Location: Germany
Contact:

Re: Getting Keen 4-6 and Dreams mods running on real DOS PCs

Post by K1n9_Duk3 »

krivulak wrote: Mon Apr 13, 2020 16:10There is no way you can allocate more than 460k of memory because of... Well, I am not really sure.
Could you please explain where that number comes from? Without context, that number is absolutely meaningless to me. Maybe you were thinking about the 640k limit for conventional memory under DOS and somehow swapped the 4 and the 6 in that number?
Hail to the K1n9, baby!
http://k1n9duk3.shikadi.net
User avatar
krivulak
Grunt
Posts: 21
Joined: Tue Jun 16, 2015 18:48

Re: Getting Keen 4-6 and Dreams mods running on real DOS PCs

Post by krivulak »

K1n9_Duk3 wrote: Mon Apr 13, 2020 20:22 Could you please explain where that number comes from? Without context, that number is absolutely meaningless to me. Maybe you were thinking about the 640k limit for conventional memory under DOS and somehow swapped the 4 and the 6 in that number?
True, last few days I have quite noticeable problems with speaking, don't know why. Maybe the quarantine and two months of complete isolation are starting to take their price... Anyways, five years ago when I was quite new to this problem I started new topic about troubles running Keen 9. (viewtopic.php?f=2&t=3302) Back then I found out that I can manage to make about 604k free conventional memory, but Keen never seems to allocate more than 350k with standard M$ memory manager. With 64k expanded memory it makes total of about 420k of memory used by Commander Keen, which is clearly not enough, but better than standard clean DOS boot which couldn't even manage to run music in the first level. With the super conservative way of booting DOS the music actually sometimes managed to play, but Defense Tower Surth still crashed as well as Trog Village from Terror From Outer Space. I think Atroxian Realm crashed too. Never finished any of those games since I don't play on anything other than the 486.
With himem from Freedos I actually managed to get 460k of total memory allocated, but even that was not enough. Just as I said, now I am trying to work with PTS-DOS, Russian clone of DOS with quite advanced memory manager (from what I heard you can manage to get about 630k conventional memory free with this), but I have never been able to make it work properly and about two weeks ago my rig started acting up - on certain memory addresses started somehow adding one bit in quite weird pattern, which sadly makes the computer unstable. And the worst thing is that I sold all of my 386 VLB boards so I don't have any spares to work on. Right now I am stuck with logic probe, vague manual and tears in my eyes, because I love that machine since I got it about six years ago...
Nevertheless, if I manage to make some progress, I will sure post a reply!
User avatar
K1n9_Duk3
Vorticon Elite
Posts: 779
Joined: Mon Aug 25, 2008 9:30
Location: Germany
Contact:

Re: Getting Keen 4-6 and Dreams mods running on real DOS PCs

Post by K1n9_Duk3 »

krivulak wrote: Mon Apr 13, 2020 21:22 Back then I found out that I can manage to make about 604k free conventional memory, but Keen never seems to allocate more than 350k with standard M$ memory manager.
I think you're comparing apples and oranges here. Those 604k of free conventional memory is how much free memory you have before starting the game. The number that's shown in the game's startup screen is how much free conventional memory you have after the game executable was loaded into memory. The executables usually take up over 200k of memory on their own, so this big difference is totally normal. Just check out the screenshot in the first post of this thread.

If you want to get this sorted out (assuming you can get your 486 back in working order), you should start with plain, unmodded Keen 4 and tweak your startup files to get the most out of your system. If you load DOS high and don't load any unnecessary drivers and TSRs, you should be able to get to a point where you have about 617k (or more) free conventional memory under DOS. That's the amount you see when you type MEM at the DOS prompt. That means an absolute bare-bones configuration without CD-ROM drivers or anything like that.

Try creating a boot disk with the CONFIG.SYS settings I listed in the first post of this thread. The AUTOEXEC.BAT file on that boot disk should be empty. Boot from that boot disk and see how much free conventional memory you have under DOS. Also check the startup screen of KEEN4 and compare it to the one I posted. The number for "Expanded" should always be either 0 (no EMS) or 64k (with EMS). The other values may vary. You can probalby get even more than 74k of "Extended" (XMS) memory, depending on your system, but you might also get less, as reported by keenmaster486.

Unless you can get vanilla KEEN4 to show a total of 513k or more (which is how much memory is available in DOSBox), you will probably run into memory problems when trying to play mods on the same system using the same memory configuration. If you can't get the game to use XMS as well as EMS, you won't be able to get anywhere near those 513k of total available memory and the chances of getting any mods to run properly are extremely slim. And in order to allow the game to use XMS, you cannot let DOS use UMBs, which is what this thread was all about in the first place.

The problem with most mods is that modders develop them primarily for DOSBox and keep putting more and more stuff into the game levels until they run out of memory in DOSBox and are forced to stop. Sometimes these issues get fixed before release, but not always.

There is one additional thing you can try to free up a few kilobytes of memory for mods, and that is to get rid of CKPATCH. All mods of Keen 4-6 and Dreams need CKPATCH to modify parts of the game's executable in order to run. The CKPATCH program takes up about 22k of memory, so any Keen mod that uses CKPATCH will have about 22k less available memory than the orignal games on the same system. You can use my patching utility to create a fully patched executable for each mod, so you don't have to start the mod via CKPATCH anymore and therefore free up those 22k for the game. But you'll have to modify the patch script a little for this to work, since my tool is not 100% compatible with CKPATCH.

And just FYI, the reason why conventional memory is limited to 640k is that the 8086/8088 processors only had 1 megabyte of address space. Anything above 640k was reserved for accessing hardware other than the main memory. As I mentioned before, the EGA (and VGA) video memory starts at 640k. If I remember correctly, the first IBM PCs shipped with only 64k of memory, so the famous claim that "640k should be enough for everybody" would have made sense back then. Fun fact: if your PC only has a CGA or MDA video card, you can actually have 704k of conventional memory.
Hail to the K1n9, baby!
http://k1n9duk3.shikadi.net
User avatar
K1n9_Duk3
Vorticon Elite
Posts: 779
Joined: Mon Aug 25, 2008 9:30
Location: Germany
Contact:

Re: Getting Keen 4-6 and Dreams mods running on real DOS PCs

Post by K1n9_Duk3 »

keenmaster486 wrote: Tue Jan 28, 2020 16:47was only getting 30 or 40K of XMS for some reason
I think this might just have been because of a minor bug in the ID Engine's XMS code. It's possible that the game actually allocated 94k or 104k of XMS.

TL/DR: The amount of "Extended" (XMS) memory shown on the startup screen is not reliable. As a result, the "Total" amount on the startup screen and in the memory usage window is also unreliable. The game may have allocated 64k (or 128k?) more than that.

While testing the current build of FitF on my 386 DX-40, I noticed that it would only use 58k of XMS, even though the system should have around 122k of free upper memory. At first I thought the game was only using 58k of XMS, but then I noticed that the memory usage window claimed that the game was using about 4 gigabytes(!) of memory, even though the engine can only address 1 megabyte (and the system only has 4 megabytes of RAM).

This was the first hint that something was wrong with the XMS code. The 4 gigabytes number indicated that there was probably a 32 bit integer overflow somewhere. My memory info code displays the amount of memory actually used by subtracting the "Free with purge" amount (which is always correct) from the "Total" amount reported at the startup screen. So this meant that the game must have had more memory than it was reporting. I quickly tracked this down to an issue of 16-bit vs. 32-bit integers, so this problem will be fixed in FitF.

I also created a tiny little patch for Keen 4 and tested the result on my 386:

Image

Yup, plain old Keen 4 has 556k of available memory on my 386. That's about 43k more memory than what you get in DOSBox (total in DOSBox is about 513k), meaning that it's theoretically possible to create Keen mods that can be played on real DOS PCs, but won't work in DOSBox!

Just in case you're confused: The patch I wrote for Keen 4 only means that the game will show the correct amount of available memory. Without the patch, the game will still use the full 122k of XMS on this system, even though it only reports 58k on the startup screen.

Okay, so how can you check how much XMS memory can actually be allocated on your system?

One way is to edit your CONFIG.SYS again and change DOS=HIGH,NOUMB into DOS=HIGH,UMB and reboot the system. Run MEM and see how much high memory you have in total. That's roughly how much XMS the Keen games should be able to allocate (after changing your CONFIG.SYS back to NOUMB and rebooting the system, of course).

Or you can try the patch I have written (for KEEN4E v1.4 only!):

Code: Select all

%patch $157C2
	$BB $0010w $F7 $E3 $01 $06 $A017w $11 $16 $A019w
The other ways involve using the memory usage window (F10+M). The world map of the unmodified EGA version 1.4 of Keen 4 requires about 218k (when playing with music and AdLib sound effects). This value is calculated as "Total" - "With purge". Compare that to whatever you get on your system. The difference will be the error in the XMS size calculation. You could also read the memory map that the game displays after the memory usage window, but that might be a bit too complicated.

Note:
The amount of XMS memory that Keen 4-6 and Dreams can use may vary from system to system. XMS memory is allocated via upper memory blocks, and the address space of those upper memory blocks is shared with other hardware (video RAM, ROM chips, etc.). For example, my Pentium MMX laptop can only use up to 74k of XMS for the Keen games (as shown in the first post of this thread), because it has more (or at least bigger) ROM chips that take up a lot of the address space.
Hail to the K1n9, baby!
http://k1n9duk3.shikadi.net
User avatar
Frenkel
Vortininja
Posts: 33
Joined: Sat Jan 01, 2022 11:25

Re: Getting Keen 4-6 and Dreams mods running on real DOS PCs

Post by Frenkel »

So the problem is in ID_MM.C function MML_SetupXMS()

Code: Select all

mminfo.XMSmem += size*16;
where mminfo.XMSmem is 32 bits but size is 16 bits, right?
User avatar
Frenkel
Vortininja
Posts: 33
Joined: Sat Jan 01, 2022 11:25

Re: Getting Keen 4-6 and Dreams mods running on real DOS PCs

Post by Frenkel »

BTW, the startup screen is 4000 bytes and is released from memory when it isn't needed anymore, but the total memory available counter doesn't get updated. So it's possible the game won't start and tell you you do not have enough memory to run Commander Keen, while you do have in fact just enough free memory.
Also, F10 + M should show about 4k more in the total memory field.
Post Reply