Swift on Mac OS 9
Swift on Mac OS 9
It is April 1, and that means it is
both April Fools’ Day and the anniversary of the founding of Apple Inc. While
this year is a sober one due to current events, I think a lot of people still
appreciate what people are creating and sharing to keep spirits up, whether
that be music or art or…impractical programming projects. And while pranks on
April Fools’ seem less and less fun[1],
obvious jokes and whimsy, not at anyone’s expense, are still something I
believe in…and even better if they work.
Modern compiler, classic
linker
What is a Mac OS 9?
Twenty (!) years ago, before the macOS[2]
we know today, there was another operating system known as “Mac OS”.[3]
It was one of the first OSs to use a GUI at all, something that we pretty much
take for granted these days. It also dates from the days when only one program
could run at a time; because of that, even the latest version uses cooperative
multitasking to run multiple programs—that is, a program has to yield it's time
to let others run.[4] If
a program crashed or overwrote memory it wasn’t supposed to, there was a good
chance you’d have to restart the whole system.
Mac OS 9 ran on PowerPC processors, which were also used in the GameCube, PS3,
and Xbox 360; earlier versions of the OS had started on Motorola’s 68k CPU
series. Its successor Mac OS X[5]
also ran on PowerPC when it first launched; it was not until 10.4 that Apple
began to switch to Intel processors instead, and 10.6 when PowerPC was finally
dropped.
Mac OS X was a huge step forward from Mac OS 9 in several ways, including
preemptive multitasking so that you could run multiple things at once. But
Apple did not want to just leave OS 9 programs behind, so they did two things:
·
The Classic environment set up a sandbox that looked enough like Mac OS 9 to
run Classic Mac OS programs directly in Mac OS X. Because the Classic
environment was itself an app, all the programs that ran inside it were
protected from interfering with other Mac OS X programs and vice versa. It
really was quite effective and survived longer than booting into Mac OS 9
(which never received support for newer PowerPC processors). But its life ended
with the switch to Intel-powered Macs—Classic was built on running the
instructions in the original apps directly, only having to provide
compatibility shims for libraries. Think of it like Wine / CrossOver rather
than VirtualBox / Parallels.[6]
·
Carbon was a packaged-up version of the old Mac OS Toolbox APIs so that you
could write Mac OS X apps the same way you always had. You basically just
recompiled your app and added an extra annotation saying you were “Carbonized”.
(Sound familiar?[7])
Fun aside: This is the reason (one of the reasons?) Core Foundation exists—to
provide a common interface between Carbon and Cocoa. (h/t Marshall Elfstrand
for the video link.)
Classic ended with the switch to Intel processors back in the 2000s, but Carbon
worked all the way up to last year, macOS Mojave. Apple never released a 64-bit
version of Carbon, presumably to encourage developers to move to Cocoa, and
with last year’s macOS Catalina, support for 32-bit apps was dropped entirely
with very few exceptions.[8]
What is the goal?
Since I learned to program on
Classic Mac OS, and years later spent a good chunk of my career working on
Swift, I’ve had the tantalizing thought that I’d like to write a program in
Swift and run it on Mac OS 9. That is,
·
I write Swift source code that calls Carbon / Toolbox APIs.
·
I compile it for PowerPC
with (a version of) the Swift compiler.
·
I package it up as
necessary for Mac OS 9.
·
Profit!
Is this useful? No! Absolutely not!
But neither was ROSE-8, and yet I still learned a lot doing it.
As you probably guessed, I managed to accomplish this, or I would not be
writing this blog post. So, without further ado, here is a picture of a Swift
Toolbox app running on Mac OS 9.2, on my friend Nadine’s Power Mac G4. (Check
out that blazing fast 400MHz processor!)
 |
You can see BitPaint running in the middle, and the Classic version of Apple System Profiler showing that yes, this is Mac OS 9.2. |
I assume a good number of people reading this would like to know how to do it
too!
·
If you want to build your own PPC-capable Swift compiler, check out the
following repositories:
git clone https://belkadan.com/source/ppc-swift-project
cd ppc-swift-project
git clone -b ppc-swift
https://belkadan.com/source/swift
git clone -b ppc-swift
https://belkadan.com/source/llvm-project
git clone
https://github.com/apple/swift-cmark cmark
make #
quick start to build swiftc and the stripped-down stdlib
Note the
directory and branch names of the sub-repos and note that they should be nested
inside the ppc-swift-project repo. You will also need the mpw emulator and a
copy of the Macintosh Programmer’s Workshop tools[9] to build an actual app using modern
macOS.
·
If you want a prebuilt PPC-capable Swift toolchain, here is one:
ppc-swift-toolchain. Note that while I have put a built Swift.o in this
toolchain, you’ll probably only have success with optimized code that doesn’t
actually have any remaining links to the stdlib (i.e. everything is inlined
away). You will also need the mpw emulator and a copy of the Macintosh
Programmer’s Workshop tools.
You may still want to check out the example in the ppc-swift-project repo. The
required flags for swiftc, PPCLink, and Rez can be a little finicky. (And note
that the SIZE and carb resources are required for any Carbon app, so you cannot
just skip the Rez part if you want to run your app.)
·
If you just want to try a built version of BitPaint, here is one:
BitPaint-Swift.hqx.
 |
(".hqx. Now that's an extension I haven't heard in a long time.") |
I would like to hear about anything you make with these tools! Meanwhile, if you
would like to hear how I made this work, read on.
Gathering materials
The last time I was building Classic
Mac OS apps, I was using CodeWarrior. Calling that “building Classic Mac OS
apps” was a stretch; I was learning C and using CodeWarrior’s terminal I/O
library to get a stdin/stdout interface that Classic Mac OS did not have
natively. (Remember, no command line!) I could try to get some version of
CodeWarrior running again, but that did not seem like the most convenient
thing. I did not think I would be able to get the Swift compiler running on
Classic, so I would be shuttling object files back and forth between OSs to get
anything done.
Fortunately for me, I am not the only one interested in building Classic apps
on modern macOS. At some point I found about the mpw project: an emulator
specifically for running Apple’s Macintosh Programmer’s Workshop tools. And I
knew it was going to work, too, because Steve Troughton-Smith, (in)famous in
the Apple community for finding undocumented and prerelease features in Apple’s
OSs, had written up his experiences building an app with mpw that ran on System
1 all the way up to modern Mac OS X, just by building with the appropriate
compiler and against the appropriate libraries.
If you are interested in all this, I highly recommend checking out his blog
post. Not only was this the reference I used to get started, but the app you
see running in the above picture, BitPaint, is Troughton-Smith’s test app,
ported to Swift. (I did ask him ahead of time if it was okay to use his app for
a hobbyist project.) Longtime Mac developer Gwynne Raskind also gave a two-part
high-level tour of the Toolbox APIs on Mike Ash’s blog several years ago (part
1 | part 2); fortunately, Carbon takes care of a fair amount for us even on Mac
OS 9.
So okay. What does MPW give us?
·
A PowerPC compiler
·
A PowerPC assembler
·
A PowerPC linker
·
The Classic Mac OS header
files
·
The Classic Mac OS library
stubs, for linking against
·
A bunch of object and
binary inspection tools, which we do not need for the finished product but
which I made a lot of use of when trying to debug mystery misbehavior
That is pretty good; as
Troughton-Smith’s blog post shows, it is enough to build an entire app that
will run on Classic. My idea was to take object files produced by a modern
compiler and feed them to the PowerPC linker, which means I will additionally
need:
·
A modified version of the Swift compiler that supports emitting MPW-compatible
object files
·
Some stripped-down form of
the Swift standard library and runtime (enough to read in and interact with
Carbon headers, at least)
·
An actual machine running
Mac OS 9. I have one, but not the charger for it, and so I did most of my
testing using SheepShaver. My friend Nadine provided some testing on actual
machines once things were working.
And, well, that should be it! So,
off we go.
Modern compiler, classic linker
To make things more manageable, I
set an intermediate goal: build an app using Clang, the modern C compiler that
ships with Xcode. Clang uses the same LLVM infrastructure as the Swift
compiler, so I figured I could deal with all the object format and workflow
issues in Clang, and then move on to the Swift-specific parts.
The first thing I did was try to figure out what the file format was for
PowerPC object files. It turns out it is a format called XCOFF; searching for
modern documentation on this turned up an IBM reference doc. Pretty much no one
else uses this format, which was not encouraging. The first time I started
looking into this project, I was worried I’d have to have my compiler write out
assembly code and then send that through the MPW PowerPC assembler…after fixing
it up to account for the differences in how LLVM and MPW print PowerPC assembly.
However, when I checked to see if LLVM supported XCOFF, I was in for a stroke
of luck. It turns out IBM has started adding support for XCOFF to LLVM just
last year, as part of adding support for their AIX OS…which runs on PowerPC. So,
I could ask Clang to generate XCOFF files for AIX, which means it should only
be a short step to making it generate XCOFF files for Classic Mac OS.
At this point I remembered a bit of trivia. Apple and IBM used to have a close
partnership, along with Motorola. They had even made some common standards that
were used across platforms and CPUs, though perhaps with less impact than they
had hoped. Was it possible that AIX and Classic Mac OS used the same calling
conventions for their procedures, and they could just interoperate without any
extra work?
I got lucky: the answer is (nearly) yes. The AIX register conventions and stack
conventions match up with the ones in the Mac OS Runtime Architectures guide.
That meant I could feed object files produced by Clang directly into MPW’s
PPCLink and get a working Classic Mac OS binary out.
I am pretty sure my mouth fell open when I first saw this work.
% clang -c test.c \
-target powerpc-ibm-aix-xcoff \
-isystem ${MPW}/Interfaces/CIncludes \
-integrated-as \
-fpascal-strings
% mpw PPCLink test.o ${PPC_LIBRARIES} -o Test
That should work with just
top-of-master-branch LLVM/Clang (and a very simple test.c). I did end up
needing to change LLVM in a few ways in the end, but it is minimal, so much
thanks to the IBM folks for doing the hard part of the work for me!
Now do Swift
Being able to compile a simple test
program was a great milestone, but I had to do a fair bit of work before I
could get swiftc to compile a whole BitPaint. Here are some of the highlights:
·
Teach Swift about the PPC/AIX target. This mostly involved adding ppc
and AIX cases to switch statements across the Swift compiler, but also involved
making a simple description of the Swift calling conventions for Clang (which I
cribbed from the 32-bit ARM implementation) and then assuring the PPC/AIX
backend that this was an okay calling convention to be using. I got lucky in
the amount of work I had to do here because Swift already supports 32-bit ARM,
little-endian 64-bit PowerPC (when running Linux), and big-endian 64-bit s390x
(another IBM architecture); all the pieces were already in place.
·
Add support for Pascal strings. The Mac’s first high-level programming
language was Pascal, not C! As such, the default format for strings throughout
the Toolbox APIs was Pascal strings (a length byte followed by string data)
rather than C strings (string data followed by a null byte). With the
-fpascal-strings command-line flag, Clang supports static Pascal strings with
the syntax "\pHello World". The \p would be replaced by the length of
the string (which must be no more than 255 bytes) so that you did not have to
count it yourself. I hacked this into Swift as well, and while my
implementation probably has problems[10],
it was enough to get simple things working.
·
Turn off reflection support and nearly all runtime metadata. The Swift
runtime is very powerful, but I did not want to write much of a runtime for
this project, which was primarily about calling a bunch of C functions. Beyond
that, though, the default format for Swift metadata makes heavy use of relative
addressing (mainly to reduce startup time, but learn more here) as well as
symbols pointing inside of a global, and the LLVM XCOFF implementation doesn’t
(yet?) support either. So, to get to a working proof-of-concept, I aggressively
commented out parts of IRGen that made use of either feature. I would like to
get some of the static metadata back at some point, but reflection is not
something I am ever interested in. Probably.
·
Make a smol stdlib. The full Swift standard library has a lot of things
in it I do not need, and some that I would not even know how to implement. (What
is a String in a world that cannot assume Unicode?) But all the logic to
integrate with C code is based on having some basic types in the standard
library (like Int16 and UnsafeMutablePointer). What I ended up doing was taking
a subset of the standard library sources, and then adding additional files and
commenting things out until it worked.
…haha, nope, even that was not good enough. My early attempts at this compiled
okay, but they managed to crash PPCLink when I tried to write a test program,
presumably because there are just too many symbols in the standard library. So,
I cut things down to an even smaller subset, and that (eventually) worked. Of
course, I was working on this at the same time as I was modifying the compiler
to get to a working proof-of-concept, so I think I ultimately went further than
I needed to. (A bunch of symbols are only used for runtime metadata purposes.)
As mentioned above, non-optimized builds of non-trivial programs do not work
yet, so I do not know if I am in the danger zone or not, but I might try to add
a few more things back in.
Rather than modify the actual Swift repo for this, I decided to keep my
stripped-down standard library separate, so you can find it in the
ppc-swift-project repo. This might be a good reference for someone looking for
a C-compatible, runtime-less subset of Swift, perhaps for an embedded or other
resource-constrained environment.
·
Disable jump tables. LLVM optimizes switch statements into jump tables
when it looks like it will help performance and/or code size, but its default
implementations of jump tables also were not supported in the LLVM XCOFF
implementation. I imagine the AIX folks will get around to implementing this
sooner or later, but for now I just disabled jump tables entirely, forcing the
compiler to emit switches as a series of ifs instead.
You can check out all the changes in the swift and llvm-project repos if you
are curious. Very few are appropriate for upstreaming to their respective
projects, but I will try to get the ones that are relevant upstreamed at some
point.
A week of mysterious failures
Having made all the changes above, I
had an app that worked! In Swift!
…except, it only worked some of the time. I would change something arbitrary
and suddenly events would not register any more. It got so bad that I added a
counter: after any ten events, exit the app. Without that, I would get trapped,
unable to even quit without restarting the (virtual) machine. Even in a
seemingly working version, my friend Nadine reported that trying to use the
Reset command caused the app to crash. What was going on?
I decided I had to get to the bottom of something strange I had seen earlier:
even the Clang version of the program did not work correctly when I turned on
optimizations. It is possible that that was a bug in IBM’s newly added AIX
support, or 32-bit PowerPC support since it is not such a common platform, or
even LLVM’s optimizations. It could be that AIX and Classic Mac OS really were
not as similar as I thought they were, and so my code was not agreeing with the
system code on how things were supposed to work. And it could be that the
optimized code was using an instruction that SheepShaver did not support,
though that did not really seem to match the symptoms.
And the symptoms were weird. Some local variables were getting corrupted, but
others were not. So, I started testing everything I could think of:
·
was the stack aligned properly?
·
was the stack pointer
somehow not getting restored properly?
·
was the glue code for
cross-library calls trashing other data?[11]
(see the Mac OS Runtime Architectures guide)
·
was there something causing
the Code Fragment Manager (dynamic linker) to put the wrong address in for
cross-library calls?
Without being able to rely on
logging, I made the simplest textual debug output facility I could: modifying
the title of a menu. I wrote C functions that tracked the current stack pointer
to make sure it was getting restored properly; I made good use of the DumpXCOFF
and DumpPEF tools that came with MPW; I learned how PEF “pidata” (“pattern-initialized
data”) worked and tried to step through CFM relocations by hand (again, see the
Mac OS Runtime Architectures guide). I even started trying to decompile some of
the actual system libraries to see if they were doing anything suspicious, even
though a bug in the actual Mac OS 9 seemed incredibly unlikely. This led all
the way to learning about the “toolbox ROM”, which is not actually ROM at all: it
is a boot script and a compressed set of system libraries. (It’s called that
because it’s content that used to be in ROM.) Fortunately SheepShaver already
knows how to load it, which meant that I could do the same decompression and
then manually split out the individual libraries.
Yeah, I got way off in the weeds. I learned a lot, though!
Finally, I looked at the decompiled optimized code—the C version, not the Swift
version. I observed that the variable getting corrupted was in general-purpose
register 13. That’s supposed to be an okay place to put data in Classic Mac OS
(and in 32-bit AIX, and in 32-bit Mac OS X), but I decided I didn’t trust that,
particularly because that register had been used to track thread-local storage
in 64-bit AIX. So, I marked r13 as reserved…
…and the problems went away. Optimized, non-optimized, even with
-fstack-protector-all on. And Swift.
(Debugging this took about a week, unfortunately, which led to this project
being a little less ambitious than I originally wanted.)
“Future
directions”
What didn’t I get to? An awful lot.
·
There is no runtime at all, which means no dynamic allocation (among other
things).
·
There is no type metadata,
which means no generics (that are not optimized away).
·
There is no field metadata,
which means no key paths (that are not optimized away).
·
There is no Unicode
support, so no Strings. Arguably I could make a String without Characters, or a
String using MacRoman as the native encoding, but it would not necessarily look
much like today’s Swift.String.
·
There are a bunch of other
standard library things missing because I wanted to get the proof-of-concept
working, but also because PPCLink was choking on large object files. If I do
get more standard library stuff working, I will probably split it out of the
‘Swift’ module somehow.
·
I had to mess with linkage
in several ways to make the LLVM XCOFF backend happy, so I am not sure
multi-file builds would work. I did not even test it.
·
I am using Carbon, which
means that my program ought to work on older versions of Mac OS X as well, but
my friend Nadine tried and it did not, and it was not a priority to figure out.
·
I wanted to try making nice
abstractions on top of the some of the Toolbox APIs.
·
I wanted to make more
complicated example apps!
Maybe I’ll follow up on some of
these, but I’ve been putting a lot of effort into making sure I could finish
this by April 1, so I should probably get to some of the things I’ve been
neglecting in favor of this project instead.
Summary
This project took a lot of time,
even though I (1) know a lot about compilers and (2) hacked my way to success
instead of being careful and maintaining proper software development practices.
But I learned a lot, and I accomplished a goal I have had in the back of my
mind for a long time.
If you made it all the way to the end of the article, here is a reward:
BitPaint running under Classic on Mac OS X 10.2 (also courtesy of Nadine).
 |
You can see BitPaint running in the middle, and the Classic version of Apple System Profiler...but also the Mac OS X "About This Computer" box showing 10.2.8. |
Stay safe, everyone, and help the people around you when you can. And if
anybody makes something with this project, I want to hear about it!
[1] “Being mean is easy, young
man. Humor is harder.” – Annalee Flower Horne
[2] Née OS X, née Mac OS X.
[3] Née “System”, as in
“System 7”. The OS only got branded in Mac OS 7.6.
[4] If you are interested in
hearing more about this, check out the MultiFinder article on Wikipedia.
[5] Always “ten”, never “ex”.
[6] Apple did also make an
emulator for PowerPC apps when they switched to Intel chips, called Rosetta. In
a way this was an even more technically impressive feat than Classic, but why
couldn’t they run Classic through Rosetta? The Wikipedia article suggests that
it’s because Classic required lower-level system hooks that they didn’t want to
provide in Rosetta; I could also imagine it being too many layers to get good
performance through, or even just Apple trying to shed the maintenance burden
of a piece of software that, in general, was getting less and less use each year.
[7] While I could not resist
the opportunity to make a technical comparison to Catalyst, I do not think the
situations are really that similar. For a Mac programmer, Carbon/Toolbox would
have been the familiar API then, but for a Mac programmer, AppKit is the
familiar API now, and UIKit is the newly introduced thing.
[8] “This has made a lot of
people very angry and been widely regarded as a bad move.” The problem here
isn’t just with developers needing to port their existing apps to 64-bit (or to
Cocoa, if they were still on Carbon even after it stopped receiving updates);
it’s a concern with older apps whose developers have no plans to update them.
People are especially concerned about games, which cannot be “replaced” by a
similar app that handles the same data. On the other hand, there is a lot of
older code in the OS that Apple no longer must support, which means fewer
security vulnerabilities, fewer bugs, and faster development. In theory,
anyway. Meanwhile, people are resorting to bizarre things like running macOS
Mojave in a virtual machine, which I regret to inform you totally works,
mostly.
[9] Getting MPW these days is
increasingly tricky. I’m not comfortable hosting it myself, and the still-available
hosting linked at the bottom of the Wikipedia page contains the tools in the
form of an HFS disk image—the disk format Apple used before HFS+, which has
itself been superseded by APFS. HFS disk images are no longer supported on
macOS 10.15 Catalina, so to extract the disk image I ended up using my Mac OS 9
install and a hastily obtained install of Disk Copy, which for some reason
Apple still hosts. If you have a pre-Catalina machine around, that is probably
easier.
[10] Swift strings are
supposed to be valid UTF-8, and I am not sure if some part of the compiler will
choke if they are not. But if I ever have a string literal that is longer than
127 bytes, a length byte is going to show up as part of a multibyte UTF-8
sequence rather than a single Unicode scalar. Fortunately, all my test strings
have been short so far.
(Strings on Classic Mac OS were encoded as MacRoman by
default anyway, so I had also run into this problem if I tried to, say, put a
MacRoman ellipsis character in a static string.)
[11] As an aside, the code for
cross-library calls (“named indirect calls”) seems like it ought to be sharing
logic for calls through a function pointer. That would decrease code size at
the cost of one extra jump, but maybe that one extra jump has a significant
impact on performance.
Comments
Post a Comment