Skip to content

ZMPP available on Amazon and Kindle Fire

I am overjoyed to announce that ZMPP is available on Amazon’s Appstore and for the Kindle Fire for a couple of days now. I wanted to take the opportunity to say thank you to Amazon’s Appstore team for the quick approval time, getting ZMPP approved makes me very proud.

I believe that the tablet form factor is a great way to enjoy Interactive Fiction and hope that ZMPP can contribute a little to introduce IF to a wider audience.

Remembering Six Years of ZMPP Development

Z-Machine Preservation Project

I am happy to announce that ZMPP can now be found and downloaded from the Android Market. Shipping a software project always fills a developer with relief and pride and for me there is no difference. I wanted to take the opportunity to reminisce about the process that led up to this point, as ZMPP has recently turned 6 years old . It’s amazing that nowadays, you can just search the web to help you track the proceedings of a project.

The Beginnings

In 2005, I decided to learn Test Driven Development, so I picked up Kent Beck’s “TDD By Example” and studied it. Along with that, I also decided to practice the techniques on a non-trivial project. Right around that time, I also remembered that I had never played any of the classic Infocom adventures – my first experiences with the IF until then had been “The Hobbit” for the C64 and “Fish” and “Guild of Thieves” for the Amiga.

So that was it, I got myself a copy of the “Masterpieces of Infocom” for something like 180 US Dollar from Ebay, my 12.1″ iBook G4 and the Z-Machine Specification and said to myself: “now TDD yourself a Z-Machine interpreter to play these”. I usually worked on our kitchen table or on the bus, because I did not have a desk. I still don’t use one, so “deskless ZMPP development” has been something of a tradition, just with the work places being extended to coffee shops and Barnes&Noble. I called the project “Z-Machine Preservation Project” to indicate that I was going to revive an extinct technology. Little did I know that the Z-Machine was still alive and well.

Even though I did not TDD everything, it was a large portion of the code and I learned a lot about both the strengths and limitations of TDD and the Z-Machine. The Z-Machine is good for this kind of project, because it has a good specification, which you can match up with tests (except perhaps the UI part). Within 3 weeks, I was able to get my reference game “Leather Goddesses of Phobos” to play the first turn, take the input and crash on an invalid instruction code. That’s somewhat the general approach I have taken to start implementing emulators and virtual machines: trying to keep the VM running longer between crashes. I benefitted a lot from only being able to afford programming in assembler on the Amiga and C64 as a teenager, reading long runs of Z-code mnemonics felt oddly familiar.

After 3 more weeks, I had the first stable version that could run V3 games, for a total of 6 weeks, always working in the evening. After having implemented 4-5 Z-Machine interpreters in different programming languages, I only need a fraction of that time to get V1-V3 running now.

Reality Check: Meet the IF community

After making the first release of ZMPP, I was trying to share my efforts, because I thought it could very useful if authors could publish their works on the web. But when your software hits users, things might turn out different than what you expected.

First of all, the Z-Machine version V3 is almost not used in modern Interactive Fiction, due to its limitations in terms of available memory and text representation. These weaknesses are also some of the motivations behind the Inform authors moving more and more to the Glulx platform, which ZMPP supports now, but is not available on Android – yet.

Another thing is that Inform 6 stories are in general more computational demanding than Infocom. I did not know this when I started and my tests were initially performed on Infocom games. The initial ZMPP had lots of classes and created lots of immutable objects, also because TDD encourages many classes and many methods. This often leads to better code, but in an interpreter that runs thousands of instructions per turn, this can hurt you, especially when combined with creating lots of small objects that need to be garbage collected.

Enter the Web – Smeagol

ZMPP was originally designed to run as a Java application first and as an applet second. Sun Microsystems brought out Java Applets in the mid-90′s and they were cool and exciting at the time, unfortunately they did not put enough effort into it, so Flash pretty much took the market, and nowadays, even Flash seems to be on its way out.

The one thing that I always disliked about applets was that they felt like a bolt-on in the browser and I also knew that you could achieve a lot with web browser technology alone. Look at the efforts behind Parchment to see how far this has come today, it’s working pretty well.

So, in 2006, I decided to port ZMPP to Ruby, with a Ruby on Rails interface. It worked quite well as a proof-of-concept, but would require special hosting, which I could not afford. From this experience, however, I derived ZMPP’s execution model, that the interpreter runs until it requires user interaction and is resumed from the user interface again.

Schmalz – An Erlang Port

Schmalz (german word for “Lard”), was an exercise in programming Erlang, which I did in late 2007. I taught myself the actor model as well as the Erlang OTP failure recovery model with it. Erlang provides some wonderful tools to make scalable and recoverable systems and I love the way you can express operations on binary data with pattern matching, but it is not really a great platform for Z-Machine/Glulx interpreters that mostly run on a single thread/process. When Erlang can not take advantage of its scalability, it is actually quite slow.

Still, I wrote the beginnings of the ZMPP Glulx implementation in Erlang, that’s why I mentioned it here.

ZMPP2 – A new home at the Scala

In 2010, I was toying with the idea of distilling all that experience I made over the years in a new implementation. I knew that I wanted to stay on the Java Virtual Machine, but wanted to use a different language than Java. This is probably one of the few advantages being the sole developer on an Open Source project: you can change technologies at will. The best alternatives at that time were (and still are in my opinion) Clojure and Scala.

I love Lisp and do most of my coding tasks in Emacs for that reason, but given my experiences with Erlang and Ruby, which are both dynamic languages, I decided to go with Scala. Scala is a language chamaeleon in many ways in that it is very flexible like a dynamic language, yet you have great control over the representation of data. You can easily use mutable objects if you need to and you can use primitive types. In other words, you can use Scala, as a “better, cleaner Java” if you need to, and use a more functional style in non-performance-critical code.

Yeah, reality is, in a Z-Machine or Glulx interpreter, a lot of code is performance-critical and CPU-bound as we see later in the performance section.

A nice side effect of using Scala is that the language does not try to force me into putting a three-line class into its own separate source file and so I can group related classes in a single file. ZMPP2 has now about the same size as the original ZMPP – but includes the more complicated Glulx and Glk implementations in addition to the Z-Machine. To be fair, the shorter code is due to choosing more efficient data representations first and due to using Scala second.

I was initially concerned about the overhead Scala introduces, especially about closures and the fact that access to member variables is usually wrapped into accessor methods. Closures, generated by for-expressions, did in fact make the code a lot slower on Android due to a lot of garbage collection invocations they caused, but Android’s Dalvik VM is actually pretty good – if you can avoid creating too many objects in your execution loop, performance is similar to its Java equivalent. I did, at one point experiment with rewriting parts in Java, but the performance gain was close to zero, so I switched back to pure Scala.

Another thing that led to increased garbage collection was that I created a large number of case class instances, with the additional problem that they are created without the “new” keyword, so you can not find the instantiation as easily. I use case classes all the time, but for the ZMPP core, I had to replace them with a mutable representation.

People recently have been talking all over the place about “never use this” and “never use that” in Scala, e.g. closures, Scala collections or what not, up to the ridiculous point where you could interpret that as “when programming Scala, never use any Scala features ever” (which was not what the original post said, btw). Well, I did not use Java collections in performance-critical code either, just plain integer arrays, because these are even faster, there you go. The emphasis is on “performance-critical”, nothing will ever stop you using the nicer Scala alternatives in the other places where they come in handy. Additionally, things change over time, I guess that there are still people out there thinking that Java is slow because that was true 15 years ago. But then, people might as well still think computers only have 8 K of memory and are as large as your fridge.

ZMPP/Android – ZMPP goes mobile

Fast-forward a year and a half to October 2011. I looked into interpreters for Interactive Fiction on Android and it seemed that the available ones are taking the route of using C implementations through the Android NDK. It sounds reasonable, because at the moment, IF interpreters are “embarassingly non-parallel” and it is harder to make a VM-in-a-VM perform comparably well as a C implementation.

I am taking a slightly different stance, though: I released “Logic Shrinker”  more than a year earlier. “Logic Shrinker” was developed on Android 1.6, which does not contain a JIT. There is some code in this application that is computational intensive, but the tuning improved over the initial performance by about 1000%, perfectly sufficient on my slow HTC Magic/myTouch.

Now the minimum Android version ZMPP supports is 2.2 Froyo, which includes a JIT. I don’t see any hope you could get reasonable performance running Inform 7 games without a JIT, so it does not make any sense to support lower Android versions.

The Android team did a great job on the Dalvik VM, given that it is not based on Hotspot and they had to practically write it from scratch. Startup times are fantastic, even when compared to iOS applications, something you would not expect from the typical Java desktop application. I took the gamble and just used my ZMPP2 Z-Machine core (I am certainly not going to go back and write a Z-Machine in C anyways).

My build setup

I had to fiddle with my build setup quite a bit initially, but having the android plugin for SBT made things relatively easy. I use just the plain vanilla setup, without putting pre-packaged Scala libraries on my device. The reason is that when I did that, build times were awesomely short, but I had to pay back in startup time on the emulator so the wait times were similar, around a minute or so per build, YMMV. As previously said, I use Emacs for all development, because I work with a lot of different programming languages all the time, from 68000 assembly to R and Emacs can handle all of these pretty well.

When I started programming in Scala, I used Maven, because it has become my tool of choice for Java projects, but recently, SBT has become the best Scala build tool available. The full configuration with a .scala project file has turned out to be better than the .sbt alternative for building ZMPP.

Performance

A little earlier, I promised some numbers, here some Infocom games, along with “Curses”, an Inform game:

Story Author Year Z-Machine Version avg. instructions/turn
Zork I Infocom 1980 3 ~500
A Mind Forever Voyaging Infocom 1985 4 ~2000
Border Zone Infocom 1987 5 ~1000
Curses! Graham Nelson 1994 5 ~4500

Curses is just a single example, but I would say, among pre-Inform 7 works, its instruction/turn count is probably the lower average.

Inform 7 games can increase the complexity even more – far more. Some examples:

1. In “Bronze” (Emily Short, 2006), you can see numbers of typically a couple hundred thousand instructions per turn. If I recall correctly, this is due to an Inform extension that dynamically computes directions and a compass rose.

2. “Reliques of Tolti-Aph” (Graham Nelson, 2006), what I call the “World Record holder for most instructions in the first turn”. It takes 3,132,785 instructions for the game to display the first input prompt. “Tolti-Aph” is a special case, it is actually more like an RPG than a typical work of Interactive Fiction.

Both Graham Nelson and Emily Short often push the limits of and demonstrate the power of the Inform authoring system, so as an interpreter author, their stories are a very good way to test the efficiency of your interpreter. As people who worked on tuning performance-critical code probably noticed by now, there is actually some potential room for optimization here: Initialization of non-random content could be precomputed at compile time, the same goes for path finding: unless you have a randomly generated map, you can probably get away with baked-in offline computed data. There is some ongoing optimization work on the Inform compiler side, but I have no knowledge about the status.

Back to ZMPP: the current ZMPP2 core takes something like 700 ms on Java 6 on my 4 year old Macbook Pro/2.4 Core 2 Duo to execute these 3 million+ instructions. I remember running “Tolti-Aph” for the first time in 2006, I thought my interpreter was running in an infinite loop, it took a long, long time. Eventually I learned how to get satisfactory performance without the need to use obscure hacks, even though “Tolti-Aph”‘s first turn still takes some time on mobile devices.

Well, enough of the rambling and reminiscing, working on ZMPP has been exciting so far and will hopefully continue to be. I have my roadmap laid out, but nothing set in stone, staying flexible has served me well over the years.

In closing, I wanted to say thanks to all those members of the IF community that I had the pleasure to meet over the years, either in person, through mail or on one of the forums. This has been the real blessing of working on ZMPP: To learn from people who really care about Interactive Fiction and who try to advance it. Most of them have been actively involved for decades. I will try to continue learning from you. Since the new year is approaching as well, I might as well wish you a Happy New Year.

 

Available in Android Market

Arr!Jay – Amiga File System Browser

I was recently trying to transfer some software from Aminet to my Amigas. The way I planned to do it was to use a null modem cable to transfer an ADF file onto a disk. I had previously done that for system disks. In this case however, I had to prepare an ADF file by myself. Now I could have used UAE for that – since it allows for mounting directories as a filesystem.

For some reason, it does not work on my Mac, and anyways, I did not want to boot into UAE every time I wanted to prepare an ADF image. An internet search for tools I could use, pointed to ADFLib by Laurent Clevy (he also has a fantastic FAQ which describes the details about Amiga filesystems). The library is written in C, and unfortunately relies on int being 32 bits wide. I was weighing in the pros and cons of fixing the data types in the source code or writing a Scala version by myself and decided that having an ADF library running on the JVM would not be a bad thing. On the JVM, data types always have the same size.

This little project was also a great opportunity to study the implementation of file systems and explore some of Scala’s advanced features.

Over the last three weeks (working only in the evenings), I implemented

  • an ADF library
  • a file system browser (called Arr!Jay)
  • an .info reader library

This is how it currently looks like:

Arr!Jay reading a Workbench 1.3

and with a 2.1 Workbench:

Arr!Jay reading a Workbench 2.1

I originally planned a much smaller feature set, but as soon as the browser started to do something useful, there were a couple of features I had to implement. One of them was the icon viewer which, luckily, was done in less than a day.

All in all, making the file system browser was fun because I could use my stack of Amiga literature that I keep around – in this case, “Amiga Intern” and the “Intuition Reference Manual”. Since I still work on my emulation project, it was also nice to revisit the structure of planar graphics again.

I am planning to provide the browser as a download, the source code, as always, is on github, using the BSD license. I started this project mainly because I had a need for it, yet I hope that it can be of use to other Amiga enthusiasts as well.

2010 – The Year of Retooling

Happy 2011 !

The new year has begun and I wanted to use the opportunity to review 2010 from my perspective. A lot of changes have happened in the past year. One has been switching from one fantastic employer to a new fantastic employer. This change means that I will

  1. continue making software in scientific research, an area that fascinates me
  2. work on Open Source software projects full-time

For me this is a killer combination, because I can use cutting edge technologies in a cutting edge application domain and do not have to worry about whether I will be able to share the code – I am always in “Open Source mode”.

Last year also meant a radical change in my tool chain, which used to be centered around Java (the language). In 2010, I managed to entirely switch over to Scala, with the implementation of ZMPP2. I haven’t regretted the decision so far and while my code is still quite Java-ish, I am learning new ways every day and adapt my style accortingly. Last year has certainly not been the best one for Java overall with all the politics creating a very uncertain climate. Scala is the main reason that I stay with the JVM, I have looked at a lot of different languages last year and for me there is currently no real alternative for the projects I am working on.

Another change was that I now do almost all development and text editing tasks in Emacs. This had two rather large consequences:

  • Due to less context switching, I feel that my whole workflow is much more efficient. IDEs don’t really work well for me, I don’t rely on Intellisense-like tools so much (which would probably be different if I worked in C/C++ every day) but instead like good user interface responsiveness.
  • Only a week after the change, I started to get symptoms of tendonitis. This was due to the increased typing amount and especially because I had to type the Control key with my pinkie much more often. If you own a MacBook, you know how small that key is.

I found a solution by following Steve Yegge’s advice of remapping the Caps/Lock key to Control, which by itself probably would have been sufficient. In addition, I changed my Mac to use the Colemak keyboard layout and bought a typing software to learn touch typing. While neither typing speed nor accuracy has significantly improved, now being able to really touch type and use a more economical keyboard layout feels incredibly satisfying while I type – I can feel how my fingers have to move much less than before while typing the same amount of characters.

There is one drawback with Scala: because the compiler has to do more work, compilation times are much slower here. As the ZMPP code grew larger over time, compilation with maven takes around a minute on my MacBook Pro. I was able to streamline my turnaround times by using sbt instead of maven, at the moment ZMPP can be built with either buildr, maven or sbt, but sbt is my personal preference after working with all three in parallel for a couple of weeks.

What’s planned for 2011 ?

First of all: TADS3 support in ZMPP is coming. I looked at the image format briefly last summer while I was working on the Glulx part and started to do real work on it recently. “Real” work means at the moment putting in a couple of minutes here and there when I find the time. But progress is surprisingly good, at a slow, steady pace. I’ll write some more about it in a future post.

For mobile platforms, I will probably entirely focus on Android. Yes there is the Oracle lawsuit (did I say there have been better years for Java as a platform ?),  but as a technology, it’s the one that appeals to me the most. I recently got myself a nook color, because it has a backlit touchscreen (trade-off power consumption vs. being able to read in the night without a lamp), is great at PDF’s but mostly because it runs Android, which I think is the perfect ebook-reader platform. I’ve never read as much as in the past month.

I have a couple of plans for ZMPP and IF interpretation in general, researching performance improvements is one thing, among others. Let’s see what the schedule will allow.

I Moved – What’s Next ?

Ok, time to reflect on the past, review what has been done and where to go in the future.

Blog migrated to boxofrats.com

I have moved my blog to the boxofrats.com and am hosting it by myself now. I made the decision in order to have a little more flexibility and to have everything under one main domain name, but am aware that the maintenance responsibilities are now on me. If I would not have had various reasons to migrate, I simply would have stayed there, WordPress.com is a great hoster.

I should mention that I undertook one fruitless attempt to migrate to a Tumblr blog – I gave up after I found out that this is a totally different world.

Second Revision of ZMPP in the Works

ZMPP will stay as my main project – as a consequence, I am now writing a Glulx VM and a Glk library. A couple of games already work pretty well, there is still quite a bit of work to do. There are also the improvements on the Z-Machine implementation that I want to do, among a couple of other ideas.

Scala is now  my main JVM language

I have given Java The Language the boot, but I still believe in Java The Platform and as “assembly code” for the JVM, as in Mahatma68k.

After tinkering with Clojure for a while, I have finally settled on Scala – my main focus is on virtual machines, where I feel much more comfortable using a statically typed language. It simply works for me. I wished it had as much tool support as Java, though.

DMPP – some time …

I still have the intention on continuing work on the Amiga emulation some time, but this will have to wait until ZMPP got to the point that I envision. Understanding how the Amiga works is one of the most fascinating topics for me, but for now, the pragmatist in me wins.

Logic Shrinker the 2nd: iPhone Version

Logic Shrinker iPhone

After having uploaded Logic Shrinker to Android Market a couple of days ago, I now decided to port it to the iPhone. The reason is that the set of people who are interested in using this kind of tool is non-empty and is not identical to the set that only contains myself. Let’s face it: Most people can get by just fine without ever dealing with Boolean Logic simplification, but some can’t. If you belong to this small group – this tool is for you and it’s free.

It will be a couple of days until I submit the iPhone version to the AppStore, since I just started the port today and I’d rather test it a little while to make sure that the submission will go smoothly.

Logic Shrinker – Digital Logic Simplifier for Android

I just published my first application on Android Market. “Logic Shrinker” is a little utility that takes a digital logic function and simplifies it using the Quine-McCluskey algorithm. This application actually comes out of my Amiga emulator project – in particular it is part of the logic function generator for the Blitter component. I did not see an App for this on Android or the iPhone, which is the reason I wrote one for Android first. If there is enough interest, I might release one version for the iPhone as well.

In the last couple of weeks I digged a little bit deeper into digital logic and having a portable simplifier has been very useful to me while standing in front of the whiteboard and simulating the Blitter functionality. It also helped me quite a bit in getting a better feel for developing mobile applications.

Logic Shrinker Android

The user interface is quite simple: It is mainly dominated by the truth table and the user can define the output of each row in the truth table. I decided to allow a maximum of 7 input variables, mainly for usability reasons. A push on the “Simplify !” button displays the result of the simplification. I had thought about aresult view that is always visible, but it takes simply too much space on the screen so I went with the output dialog for the first version.

All in all, I am pretty happy with the experience of releasing my first app on Android Market. The release process on Android Market is extremely simple and pleasant: in Eclipse it just requires the developer to export a signed application, and the wizard walks you through the process which results in a signed and aligned file that can be uploaded in the publisher.

One small caveat: When downloading my app from the Market, the installation failed with an “invalid certificate” message. I solved that by deinstalling my test application. The test certificate is probably conflicting with the one for the release version.

PS: As a hint to the fact that the functional core of Logic Shrinker was written for my Amiga emulator, I chose the blue background color of the Amiga Workbench 1.3 for the button panel.

DMPP Cymus – CIA Emulation Library

Now that I have a somewhat working M68000 emulation, I can finally focus on the rest of the Amiga system. Currently most of the system is stubbed out with a couple of dummy functions so I can see when something interesting happens.

I have reached now reached a point, where I can’t fake out the CIA functionality anymore, because the Amiga OS tries to use the timers and the TOD (Time-of-Day clock). Apropos TOD – in german the word “TOD” means “death”, so using that abbreviation in the source code gives me the shivers. In UAE’s CIA source code, I even found the words “TOD Hack” – which sounds to me like “chop to death” – (brrr ! horrible !).

However, I’ll stick with that word since “tz” (for “Tageszeit”) does not seem to be any better. I grabbed the MOS CIA 6526 datasheet linked from the Wikipedia article and the information from the Amiga HRM. As an additional reference, I am also looking into the CIA implementation of the MAME project (as I have done for the M68000 implementation). MAME  is a pretty large project, but from what I have seen so far, I really like the overall code quality and the CIA implementation is no exception.

From the combined information, I created a first class design:

DMPP/Cymus CIA Class Design

I find creating UML diagrams during software development very useful. I can verify and review the software design on different levels. When designing, I first create a rough draft and then try to verify it with a prototype (usually written in Ruby or Python) to get quick feedback, so I can feel how well it works. I then iterate a couple of times until the design seems to meet the requirements.

A while ago, I had created the project “Dream Machine Preservation Project”, which I currently use as an incubator to experiment with a couple of ideas, which will hopefully one day lead to a full Amiga emulation.

The CIA library is the first thing I checked into the public git repository on Sourceforge. The implementation is still incomplete, because features are added as they are needed in the Amiga emulator. It was originally written in Scala, but when I decided to make a separate library out of it, I changed the implementation language to Java. This was mainly because I did not want to force users of the CIA library (assuming that there could be someone else besides me) to have to add the Scala library or having to know Scala in order to compile it.

Introducing the DMPP Amiga Debugger

I had this actually running for some time now: It is a graphical debugger which helps me understand how the 68000 instructions in the Kickstart ROM affect my Amiga emulator’s state. Here is a screen shot:

DMPP Debugger

It is currently mainly a view to the Mahatma68k CPU, the two CIA-8520′s and the custom chip registers. It can execute a defined number of steps and any changes to the chip registers or CPU state will show there. I am currently implementing a function which displays a tooltip for each custom chip register to show a little more information than just its numerical value. The CIA’s and the CPU have only a few registers, so it is still feasible to display the role of each flag in the main view. The sheer amount of custom chip registers however, made it even hard to get them all in one frame.
By the way, compliments to the developers of Scala Swing – these are very nice wrappers ! I started with conventional Java Swing components and then realized that Scala Swing makes programming user interfaces much easier. As a long time user,  Swing does really seem to be all that difficult to me, still I like it when the code looks as tidy as it does when using Scala Swing.

What else is new today ? I have made my second file release to Mahatma68k today. As its main addition, it now comes with the Javadocs the distribution archive and made sure only to have the methods and classes in there that matter to the user of the library.

First File Release for Mahatma68k

I decided to build a package of the current code in the Mahatma68k repository and upload it with the little example program that comes with it. It can now be found on its project website. I am aware that not everyone likes or knows maven and so I provided an ant build file for the binary distribution. I like Maven a lot, and have therefore standardized all my builds on it, but it has a steeper learning curve.

I could have provided projects for Netbeans, Eclipse and IDEA, but I guess that this task is simple enough. For Mahatma68k, I wrote everything in Emacs and Vim by the way.

Oh yes, and finally, I have the most important reference manuals together that I will need for my emulation project:

Amiga Reference Manuals

Of these, the “Amiga Hardware Reference Manual” is the most important, of course. I got the Exec Manual and “Libraries and Devices” as well (DOS and Intuition are coming, too), because I might be able to clarify some details in case I get stuck.