Tech Talk: ProRWTS Supercharges Nox Archaist

By: Mark Lemmert and Mike Reimer based on interviews with Peter Ferrie

During our adventures developing Nox Archaist we had the great pleasure of meeting Peter Ferrie (also known as qkumba).  After some discussions on the comp.sys.apple2 newsgroup, Nox Archaist became the test environment for a file system controller Ferrie recently developed called ProRWTS. We used the controller to incorporate hard drive support, design a game engine capable of supporting thousands of tiles graphics, and other features that would not have otherwise have been possible.

We sat down for an interview with Peter recently to learn more about ProRWTS.  We discovered that it is not only a ground breaking development for future Apple II games like Nox Archaist, but also for games written in the 1980s as well. ProRWTS allows virtually any disk-based game to obtain hard drive support via a port to ProDOS, which was not possible before for all games.

What is a File system Controller? Why Write One?

For many years, Peter Ferrie has been a top level anti-virus researcher for companies like Symantec and Microsoft by day, and a grand master 6502 assembly language programmer by night. He has worked on many present-day 6502 projects such as BitsyBye with John Brooks for ProDOS v2.4 (see previous issue) and 0boot with anonymous cracker 4am (see December 2015 issue).

Peter’s desire to create ProRWTS stemmed from his desire to make it possible for all Apple II games to be ported to ProDOS and ported in a way that doesn’t increase the system requirements to run the game. A port to ProDOS has been most practical way to add hard drive support to an existing game because hard drive support is built into the structure of the ProDOS file system and I/O controllers.  ProRWTS resolves some of the limitations of existing ProDOS ports including situations where titles were too large to be ported or the memory required to run the titles would increase.  Peter summed it up well when he remarked to us “Donkey Kong shouldn't need 128kb of RAM to run”. We couldn’t agree more!

A major complication in porting games to ProDOS is that ProDOS takes up a lot of memory. ProDOS usually resides in the 16k region of memory known as bank switched RAM (the language card), which leaves only 48k of main memory available for programs.

Many games in the 1980s were created using custom boot loaders, which didn’t load any operating system at all. The assembled code would run on the “bare metal”, as it was called, because any interaction between the code and the hardware had to be done directly. The benefit of these boot loaders was that a lot more memory was available to be used by the game code since an operating system was not loaded.

Games using a bare metal boot loader sometimes would occupy more than 48k of main memory. This made it impossible to port them to ProDOS without moving ProDOS into the 64k auxiliary memory bank, which is not available on all Apple II systems. This is how games such as Impossible Mission and Arkanoid could require 128k of RAM to run after porting even though they originally only required 64k. From Peter’s perspective, if the original version of a game could run on an Apple II+ with 48k then the ported version should be able to as well. We totally agree.   

Additionally, some games such as Airheart and Rad Warrior originally required 128k of RAM to run. Since 128k was the maximum memory supported on the systems games like these were designed for, there was no memory available to load the ProDOS operating system and the original game code into memory at the same time. As a result, hard drive support for these games was not possible before ProRWTS was created.

Modern games being developed for the Apple II platform face the same challenges. We initially wrote a custom bare metal bootloader for Nox Archaist because our game engine requires 128k to run. Using this bootloader would have eliminated any possibility of hard drive support for us, just like the large games from the past.

ProRWTS solves these problems. As a file system controller, it provides read/write access to the ProDOS file system, much like the native RWTS for ProDOS. However, what makes ProRWTS unique is that it is designed to do this without ProDOS or any other operating system in memory. Thus, for the first time, games requiring 128k of RAM can now be ported to the ProDOS file system and run from hard drives.

To our knowledge, with ProRWTS taking up only 1K of memory, games using this controller will have more memory available for game code than any of the titles of the 1980s that had read and write disk I/O routines. There were a few games, such as Captain Goodnight that did not write any data to disk (i.e. to save the game), which had I/O controllers smaller than 1K. ProRWTS represents an opportunity to create innovative new gameplay mechanics due to its smaller size and other features, especially for those that need to write data to disk.

ProRWTS Features

Some of the key features include:

·      Floppy disk and hard disk support

·      Read/write support with auxiliary and main memory

·      ProDOS directory and subdirectory support

·      Read/write length*

·      Seek forward in open file*

·      Get file size*

*Supported by ProRWTS2, currently in beta testing with Nox Archaist. Ferrie plans on a public release of this version in the future.

At 6502 Workshop, we wrote a front-end interface wrapper to ProRWTS for Nox Archaist that enables any of these features to be accessed with a single standardized call from any memory bank.  This interface is also planned to be publicly available in the future.

ProRWTS: Under the Hood

ProRWTS contains both a hard-disk controller and a floppy-disk controller to maintain backward compatibility. For Nox Archaist, the ProRWTS hard disk controller is launched by executing a file called NOXARCH.SYSTEM from a hard disk with ProDOS installed. Once loaded, NOXARCH.SYSTEM relocates itself into bank-switched RAM and then loads NOXARCH.MAIN into memory address $2000. NOXARCH.MAIN is the first file in the game launch process. NOXARCH.MAIN and the files it loads clobbers the ProDOS operating system because the game needs the memory occupied by ProDOS.

The floppy controller is designed to provide file system access on standalone bootable disks via its own boot loader.

All Apple II computers have a boot PROM that contains a small routine that reads a single sector from the floppy disk drive. When the computer is powered on, the boot PROM loads track $0, sector $0 from the floppy disk in drive 1 into the memory address $800. Accordingly, stage 1 of the boot loader is stored at this disk location.

Stage 1 uses the boot PROM to load stage 2 into $900. Stage 2 loads a small, read-only version of ProRWTS. Then, stage 2 uses the read-only version to load the ProDOS-compatible file called NOXARCH.SYSTEM into $2000. The file NOXARCH.SYSTEM is the full version of ProRWTS. Like the hard disk controller, once NOXARCH.SYSTEM is loaded it relocates itself into bank switched RAM and then loads NOXARCH.MAIN into $2000.

To access floppy drives, the full version of ProRWTS uses the latches at $C080–$C08F to which the disk hardware is connected. These are the latches described at the beginning of chapter six in the book Beneath Apple DOS, which the author discourages using—kind of like the emergency brake in Spaceballs with the “NEVER USE” sign. Unlike the sign, Beneath Apple DOS provides a reason to avoid using the latches. The timing of the access requests is so critical that even crossing a page boundary with the code can throw off the drive arm positioning.  

Beneath Apple DOS recommends using DOS RWTS for file access instead of the latches. DOS RWTS was written by Steve Wozniak using the latches, making it largely unnecessary for other programmers to deal with the complexities involved. However, since ProRWTS is an alternative to DOS RWTS and the I/O controller native to ProDOS, using the latches to interface directly with the disk drive hardware was necessary.

We experienced first-hand the extreme timing sensitivity described in Beneath Apple DOS during development and testing. Even though the floppy controller passed initial tests on drive1, it didn’t work correctly on drive2. In this case, drive1 was an Appledisk model A9M0107 and drive2 was a UniDisk model A9M0104. The slight difference between these two models was enough to throw off the timing of the floppy drive code. This problem by solved by adjusting the number of clock cycles consumed by the ProRWTS code between calls to the disk drive hardware. 

ProRWTS SUPERCHARGES NOX ARCHAIST

ProRWTS has helped us pursue our mission to push beyond the frontier at which Apple II tile based game development left off in the 1980s. In addition to facilitating hard drive support, ProRWTS has also enhanced or made possible many other features in Nox Archaist.

Multimedia Solution

The ProRWTS media auto-detect feature makes it possible for a single build of Nox Archaist to support both floppy and hard disks.

For example, when Nox Archaist is released (expected in 2017), it will be available as a free download containing a series of floppy disk images. The disk images will run on virtual or real floppy disk drives. The contents of the disk images will also be able to be run from a hard disk with ProDOS installed after copying the files from the disk image to a hard disk subdirectory.

Support for Thousands of Tile Graphics

Many Apple II RPGs organize their graphics into tiles. A tile is usually a region of the hi-res video screen that is 2 screen bytes by 16 lines in size. The map in a tile-based game is defined using an array of 8-bit hexadecimal numbers ($00–$FF). Each hexadecimal number represents a unique shape. For example, in Nox Archaist, the shape “grass” has a tile_id of $34; “shallow water” has a tile_id of $88.

Since an 8-bit number represents tile_ids, it is very difficult for a game to have more than 256 unique tile graphics shapes. Using a 16-bit number is usually impractical due to the speed constraints in the graphics plotting routines which render the map graphics that the player sees on the video screen. Tile sets are typically loaded from disk only during game launch to minimize gameplay delays.

Nox Archaist breaks past this barrier by leveraging the increased I/O performance of ProRWTS. Slight gameplay delays in tile-based RPGs are typical when a player enters a location that has its own map, such as castles, towns, villages, dungeons, and the outdoor world. We tested many different games and found that a delay of 4–5 seconds was typical using real floppy hardware and considered that a benchmark that we should strive to meet or exceed.

Using ProRWTS, Nox Archaist is able to load a unique set of 256 tiles when a player enters a location while limiting delay to roughly that 4–5 second benchmark. In contrast, we tested this feature using DOS 3.3 RWTS via our original boot loader and the gameplay delay was around 20 seconds. This time included copying some data from main to auxiliary memory, which is avoided by ProRWTS reading data from disk to auxiliary memory directly.

As a result of this feature, Nox Archaist can support thousands of unique tile shapes, bounded only by disk space. This aligns well with the recent expansion of development team, including Bill Giggie and Robert Padovan. Bill and Robert are professional graphics animators in the movie industry and are taking advantage of the game’s high capacity tile system to help bring Nox Archaist to life.

We do not know yet exactly how many tiles we will create. Our goal for Nox Archaist is to have more tile graphics than any Apple II RPG released in the 1980s. While many tiles will be shared between the various locations in the game, we expect to be able to offer a lot of variety. For example, the following is a likely outcome:

·      Castles with many unique tiles not found in towns or other locations.

·      Castles in different map regions that look very different from one another.

·      Different versions of the same tile. Perhaps the chairs in one town will look different from another town’s.

·      Many, many, many mobs. By dividing tile sets by location, Nox Archaist only stores those mob tiles in memory that are associated with the current player location.

High-Compression Speech Text

Nox Archaist has a comprehensive conversation system that enables the player to talk to NPCs.  Speech text is written with a high level programming language we developed called NTALK.

On the surface, the conversation format will likely appear very familiar to some players. For example, the player can input defined keywords, and the NPC will respond with a response text block. The player can then type in keywords found in the response text block, and the NPC might, or might not, respond with more information about those keywords.

By leveraging NTALK, we were able to add a lot of additional functionality to the conversation system beyond the basic keyword/response architecture. For example:

·      Players can communicate with an NPC using one of three different voice modes: Whisper, Normal, and Yell. The NPC may or may not respond differently depending on the voice mode used.

·      Certain NPC responses can be restricted based on the in-game time of day. Since NPCs move around the town map based on a schedule, this feature also effectively restricts responses based on the NPC’s location.

·      Certain NPC responses can be restricted based on an event flag or can set the value of an event flag. For example, an event-flag restriction could be used so that an NPC has something different to say after a quest is completed. Setting the value of an event flag can be used to make almost anything happen within the game as a result of the conversation. Enjoy!

After developing NTALK we were very excited to begin writing huge quantities of conversation text for Nox Archaist. And then a wise wizard, known to many as Steve Wozniak, hit us with the Wall of Text spell. As we began to comprehend the sheer amount of disk and memory consumed by text, relative to the capacity of the the Apple II, we realized that we would need a compression solution—a solution that not only stored the data compressed on disk, but that also minimized the amount of text loaded into memory at one time.

ProRWTS2 saved the day. After consulting with Ferrie, we deployed a solution in which the speech text is compressed on our development PC using ZX7, then decompressed in memory on the Apple II, but only for the specific NPC the player is talking to at that moment. The decompression is done using Peter’s 6502 ZX7 decompressor, which he graciously granted us permission to use.

The abilities of ProRWTS2 to read a specific number of bytes from a file and to seek to another position in a file are key elements that enables Nox Archaist to store only the decompressed text in memory for one NPC at a time. Here is a walkthrough of how it works:

·      In the game-build process, a separate binary file for each NPC’s speech text is created by the SBASM cross assembler.

·      A QB64 program we wrote collates the binary files together for each location (castle, town, etc.) with a two-byte length value and a one-byte NPC ID inserted between the data of each binary file.

·      In the conversation module, the data is read from the collated binary file for the location the player is in until the NPC ID is found that is associated with the NPC the player is talking to. Once an NPC ID match occurs, the length bytes of the associated speech text are passed to ProRWTS to read in the exact number of bytes needed.

·      Once the compressed speech text has been read into memory, it is decompressed into a buffer used by NTALK to access specific response text blocks.

Disk-Loaded Combat Effects

Nox Archaist has graphics plotting routines for combat effects, which provide capabilities not normally found in a tile graphics engine. These routines use bit shifting and AND/ORA masks to move shapes on the video screen off the tile grid with the background blended in. Here are some things that players can expect to see:

·      Projectile weapons such as arrows and throwing knives.

·      Fireballs that race across the screen and explode on impact.

Unfortunately, the memory that these routines use severely limits the number of shapes for combat effects that we can store in memory. However, we have developed an architecture where the assembly code for each spell and its associated shape data is stored in a “spell file” on disk.  

Loading combat spells from disk would normally not be practical due to disk I/O speed constraints. This approach worked out in Nox Archaist because our spell file is accessed via ProRWTS2. The code and shape data for the specific spell the player is casting is read into a buffer quickly enough to avoid material game delays. As a result, Nox Archaist will engage the player with many interesting combat effects.

ProRWTS is a great contribution to the ecosystem of the Apple II platform through its support of modern day development and by providing effective tools for preserving the past. The small size and flexible options ProRWTS provides for file access opens the door for programmers to create features in games and other software that would have previously been impractical.  We look forward to demonstrating this in Nox Archaist and seeing the innovations that other programmers create.

ProRWTS is available for download at:

https://github.com/peterferrie/prorwts

To follow the development of Nox Archaist stay tuned to Juiced.GS, our website, and Twitter for more updates!