Generating History

As we get closer to milestone 3, there have been a lot of small things that need nailing down but one job has dominated in recent months, and it's not even strictly a part of milestone 3 at all. That is the work of "history generation."

What exactly is this, and why is it a bottleneck?

Most MMOGs have static content, designed by large teams of people. This makes it easier to give a good first impression but comes at a high price. Once you eliminate the possibility of players changing the game environment, the only way left for players to measure progress is to add more bits to their characters. Eventually gameplay becomes a matter of repeating quests and accumulating more and more known loot, keeping up with the virtual Joneses. Players inevitably grow bored with the same old routine, and they leave.

This is not what we want for Imperial Realms. The goal here is for players to be able to shape their worlds as they want (subject to natural laws, of course) so that the possibilities for future play change constantly. The best way to do this is with a procedural engine which simply generates stars, systems and planets as needed. To get estimates of how big the game world is, we then ran the Imperial Survey (mentioned in a previous entry).

Of course, the game environment can't be completely random — we have to have some idea where things are going! In fact, I wrote the backstory (with Bruce Byfield) before we had any game data at all. Now we have the basic game environment but with no obvious connection to the backstory. Fortunately, it is a big game universe, and that gap can be filled simply by analyzing the survey and looking for systems and planets which match what we wrote.

That is what history generation is really all about; generating game content.

The first stage of this is simply to assign actual locations to the early Human Federation worlds mentioned in the history. That was done back around November 2010. Then came a much more formidable job; mapping out the network of spacefolds that connects the worlds, in a way consistent with the backstory. This proved to be very difficult. Consider just the scope of it: with about 250 billion systems, the possible number of connections to examine is on the order of 250,000,000,000 X 250,000,000,000 — a crazy number way too big to address directly. So instead we have to work out strategies for exploring it all.

This is a good thing, though, because those same strategies are needed for regular play. In fact, the history generation as a whole is really an exercise of the simplest aspects of game logic. That is the reason why we're spending so much time on it now. It has turned up a few unanticipated quirks, too, which I may post about in another entry.

At the moment I write this, the history generator has progressed to about IY103, with another 6900 or so years to go and an estimated ETA of two months. It's no longer a bottleneck, though, and nothing will be held up waiting for it.

This is also just the first stage of it. After the locations of the worlds have been assigned, we still need to generate the worlds themselves, as well as their populations, factions and units. All that won't be done until the end of milestone five.

So, you will be hearing more on this subject! For those who are curious to see more closely how it's progressing, I'll tweet and post updates on Facebook. I may comment further here, also.

The serious business of writing the backstory

Although most work on Imperial Realms is mundane coding and testing, that's not all that's required to build a game like this. It also takes significant effort to develop a backstory that spans some fifteen thousand years, even more so since my goal from day one has been to present a plausible universe with believable ties to reality.

Much of the best written science fiction makes serious attempts to picture what the future might look like, then explores the dramatic possibilities suggested. Unfortunately, science fiction in films and on TV rarely aspires to this level (even after forty years, Stanley Kubrick's 2001 remains the noteworthy exception). Games rarely even reach the mediocre levels of TV.

With Imperial Realms I wanted to develop a science fiction game that meets the higher standards of written SF. It wouldn't be science fiction if it didn't contain some invented reality, but we do try to make it plausible and avoid cliches and outright nonsense such as force fields, wormholes and time travel. As well as the satisfaction in doing things right, this approach has another benefit; after taking the time to map out believable paths through which society and technology can evolve, game play can then follow those same paths, something of considerable value.

To achieve that, we've looked at three specific areas in the backstory. We need to know what technology might exist, both to decide where and how people might live. We need to know how society might be structured, to map out alliances and divisions. And less obviously, it's worth spending a bit of time considering language issues, not so much for their direct use in the backstory but because these have to relate to the practical problems of accommodating real-life languages.

Future Technology

Technology is actually the easiest of those three areas to address, in many ways. When we look around in real life, it seems that we're advancing at an exponential rate, and in future will develop faster than we can imagine. However, this may not be so. For the last few hundred years, almost all technological progress has come from mapping out what we could call the "electric domain." That is, it stems directly and indirectly from the twin achievements of understanding electromagnetism and, from that, of mapping out atomic structure. Those let us harness electricity and study material structure, from which progress has built upon progress, but ultimately we're still reaping the benefits of a couple nineteenth-century discoveries. There are further and great gains to be had from this, for sure; in a hundred years we might be able to generate and transmit power efficiently, and our understanding of biology may grow enough to create man-made life. However, barring some new leap in scientific understanding, we still won't have anything but the simplest ideas how to harness nuclear forces or gravity for our own purposes.

So, it might well be that our current spurt of development is an anomaly, similar to the great leap in progress that humans doubtless took after learning to control fire. Even if we do make the scientific discoveries to keep on with the technology race, there are social factors which might also slow technological progress, just as in real life. What really dramatic changes do occur in the near future will probably not be the gross scientific leaps SF writers wrote about back in the fifties, such as Asimov's "nucleonics." Rather, they are more likely to be the result of steady incremental improvements in existing technology - making our gizmos better, faster and smaller. In Imperial Realms, we assume humanity remains at this state for a period of several thousand years (about as long as current history), with steady scientific discoveries related to gravity gradually leading us to a new technological era that in turn lasts for thousands of years more. Both periods present many game and dramatic possibilities, without speculating so far abroad as to turn Imperial Realms into fantasy.

Future Society

Along with technological progress comes social change, which is much harder to picture. While it's sometimes tempting to think that humanity can't keep going at all unless human nature changes, actually "humanity" already includes a very wide range of customs and attitudes, and our nature has probably already changed since early days.

So, for inspiration here I look mainly to history and to other real cultures around the world. This isn't much help at plotting details but does provide an overarching principle; given infinite space for people to spread, they are likely to do so and then adapt to their new environments in infinitely many ways. So there likely would be people in different places with every conceivable sort of social organization, and some likely would wipe themselves out just as some fear we might do in real life.

On the other hand, even though IR postulates that people would have practically infinite space to spread, and that they would change, we also have natural constraints to that change. If people continue to live together in societies at all, then they will be subject to the same biological and social pressures as always. Sudden, sharp and dramatic changes are unlikely to progress too far unless people unlearn the habit of xenophobia, which seems unlikely. Outside trade will provide opportunities to those who do it, and so is always likely to be pursued by some residents of a planet even if shunned by most. Family ties will likely remain the basis of society because family relationships will be the most sure of all human relationships. People may evolve with technology to become more individualistic, or even sociopathic, but the ultimate reasons why societies evolve the way they do are practical rather than idealistic and probably not likely to change much. Taken all together, we picture humans changing only slightly and gradually for a long way into the future, but also with constant small pockets of extreme radical change.

Future Languages

Of all the ways in which people adapt to their environments, language is probably the most obvious. Two groups of the same people separated from each other to any significant degree will develop different dialects, which may in turn develop into different languages - all because of practical needs. To think this would not happen on a galactic scale would be naive; obviously it would. However. once we grant the existence of an overall ruling class, ties between worlds, and such, we open the door to common languages. What would those look like?

Mostly we don't try to impose any rules about language at all, in the backstory or in real play; players can speak in whichever language they want. There is a need to communicate official information, though, which is best met using an official language. In a civilization spanning star systems that would need to be translated into an incredible diversity of local languages. For such a language to work, it would have to be quite mechanistic and unequivocal - more like a computer languge than a human language, in many ways. This presents the obvious temptation to create such a system in real-life for use within the game as a way to centralize bits which must really be shared and translated into different languages.

Future Knowledge

Pondering all these things together leads to a minor matter, which is rather fun to tackle; How would future knowledge be organized? Surely it would not use the same distinctions we maintain in libraries and universities, any more than we maintain the same schools of thought that the ancient Greeks did. This, too, has a practical application; these categories are immediately useful for organizing the Imperial Library.

Physics is taught now as a single field of study, but in fact it has long ago fractured into several quite different areas. Particle physicists carry on the most "fundamental" scientific research while many others work in fields such as "condensed matter" physics, which are really applications with such large scope they count as areas of study in their own right. Some, such as meteorology, are already taught and organized separately, but not all. How long will we persist in the artificial distinction between "condensed matter physics" and chemistry? Wouldn't it make more sense to group them together under "material science" (as indeed some already do?) What about other areas?

I naturally think of Physics first because it's my own area of expertise, but the same phenomenon surely exists in other fields. Would Geography be one subject or several, once we have millions of worlds to study? How would Anthropology change when it had to include not only millions of different human societies but also aliens and mixed alien-human societies? If biology progressed to the point where all human behaviour could be directly tied to physical phenomena, what would this mean for psychology and sociology?

We can't spend all our time studying these philosophical matters and will doubtless make mistakes and inconsistencies while writing the backstory. However, I am proud of what we've done so far and consider it already far above the level that most games reach. Look for even better in future!

The Imperial Survey is done!

Last week, one of the major bottlenecks in ongoing development disappeared, when the survey jobs finally finished after months of running. What is this survey, you ask, and why does it matter so much? I'll explain...

The basic IR "gameboard" is a collection of planets orbiting stars. Obvious stuff, right? However, as I mentioned last post, we strive for realism here, so we want star systems that fit with what we know of real star systems, and they should be found in about the same numbers and places as real stars. They should also include actual known star data, too. The real galaxy is estimated to contain somewhere between 100-400 billion stars, depending on whose estimates you believe. We don't know how many planets there are, but early estimates are "a lot."

For comparison, this first phase of the Imperial Survey (which covers the galaxy and the Large Magellanic Cloud) contains 249 billion stars and just over 5 trillion orbitals (meaning planets, moons, large asteroids, dwarf planets; basically anything big enough to build a base on). It also contains a network of about 250 million C-tunnels which, while not important for actual game play, are quite important to game history and need to be mapped to generate pre-existing areas.

Although it's not impossible to store all the needed data on the servers, it is quite impractical. Just the bandwidth required to send it to users would be prohibitive - and besides, most of it will never be seen anyway. So instead we do it all procedurally; inside both the IR servers and client is code to generate volumes of space as needed.

The problem with doing this is that not only are there a lot of star systems out there but they can be really complex. Most of them are multiple-star systems, and one thing we've already learned from real exoplanet observations is that real planets come in a lot more interesting varieties than what we see in our own solar system.

We need to be sure that all those variations work within the game, and since the code that generates them is complex there's no easy way to see this in advance; the whole space has to be generated automatically and checked directly once. Obviously, it's a big job. In fact, it was big enough to take the full power of four servers (with 14 64-bit cores) a few weeks of constant operation, and all had to be repeated a few times due to tweaks and fixes.

It's finished now, though. Look for updates to the relevant encyclopedia articles some time soon.

Tough Questions, Easy Answers

People ask me one question about Imperial Realms much more than any other:

"When can we play?"

Well, the answer to that is easy. There is no set schedule; the milestone list is as close as we get. What's much more interesting is the question behind that one, which most people are too polite to ask:

"Why is it taking so long?"

Since I first conceived Imperial Realms back in 2004, that's a natural thing to wonder, but the answer to that one is also easy although it takes more space.

First, much of the time since 2004 has actually been eaten up by quite mundane things. IR was at first a spare-time project while I looked for work, and grew in scope with time. When combined with family obligations and other diversions, that's meant that 2008 and 2009 were really the first stretch of time when I work anything like full-time on it.

Then, IR development proceeds more slowly than another game might because we're trying to do some things which are quite new. For example, our planets are treated as proper spherical objects and divided up into zones of roughly equal area, connected in a real spherical mesh. In contrast, the traditional way to handle game maps is to divide up rectangular areas into squares or hexagons. Our approach will enhance the strategy aspects of the game, but introduces quite a lot of extra complexity.

Developing the graphic engine to draw such maps took months of work and experimentation all by itself, and there's an even bigger and less obvious problem; since there's nothing else I can point to and say "we're doing it like that," it's hard to explain to somebody else what is needed. That recurring problem makes it very hard to hire people to help with the work, and increases the time requirements - perhaps by a year just for that one aspect of the game!

As well as the need to produce new technology, I've always worked hard to do Imperial Realms right. For a space-themed game nowadays, this can be quite a challenge. When I laid out the basic design parameters back in 2004, the Milky Way was thought to have four spiral arms. A year or two ago the current wisdom changed and it's now thought to have two major spirals. That means a complete revamp of one of the basic data structures. In the same time period there's been a veritable avalanche of other astronomy results, changing our views on such basic parameters as the kinds of stars and their abundances to the range of possible planets and their orbits. While keeping on top of these discoveries is challenging and adds months of time, I think the extra work will make for a better game in the end.

Time-consuming as these matters are, they still aren't the end of it. Probably only about two-thirds of the working time spent on the game actually goes to direct development of the game itself. The rest is taken up developing supporting services. The web platform driving the sites, SEBCMS, represents several months of work at different times, Content development and simple administrative matters take months on top of that, but are worth the time: MMOs fly or die depending on their user communities, so we work hard to lay good community foundations.

Perhaps the best way to summarize my approach to IR is with a comparison, and since I am a big movie buff I like to compare game production to movie production. In that light, my goal is not to be like a big-name movie producer who pumps out summer blockbusters, but to be like Stanley Kubrick and produce a polished enduring classic. That means spending however much time is needed and not compromising, and that's all right because the result will be something I'm proud to be part of - and I hope players will share that pride.