The Parallel Universe Blog

February 13, 2014

Writing an MMO with Web Actors

A couple of weeks ago we launched Comsat, a set of libraries implementing standard Java web-realted APIs (servlet, JAX-RS, JDBC etc.) using Quasar fibers. The main benefit of using fibers as the foundation for a web application is the performance and scalability of the asynchronous (callback-based) programming model combined with the simplicity of the threaded model. I’ve discussed this in detail in a previous blog post.

In addition to fiber-based, standard API implementations, Comsat includes an optional API called Web Actors, that lets you write interactive web applications using the actor model. To see how these web actors might be used in production, we have ported our spaceships demo from a standalone simulation to an MMO:

Web Spaceships screenshot

The code for this game is found here as part of the Comsat examples project. Mind you, we are neither UI nor game designers, so please forgive the crude graphics and gameplay. Behind the scenes, however, a lot is going on. Every spacehip is running its simulation in a fiber (lightweight thread) of its own, updating its location in an in-memory spatial database, and querying the database for its surroundings many times per second. All simulation is done strictly on the server, which talks to the client via WebSockets. There are ships controlled by players as well as NPC ships. The server sends JSON updates to the clients with the state of the spaceships in their vicinity. The client merely renders the graphics (in WebGL) and performs simple extrapolation of the server state to provide smoother animation; it also sends command to control the player’s ship.

Because all processing is done on the server, user experience is heavily affected by your network latency (which is displayed at the top left of the screen). A real game will employ techniques like dead-reckoning to hide that latency.

We’re not going to discuss the code in depth. Instead, this short post will give a broad overview of the game architecture in two parts: the first will discuss how the client-server communication is done using Web Actors, and the second will discuss some core aspects of the simulation architecture.

Part 1 – Communicating with the Real World

When a new player first accesses the game’s page, a new instance of this web actor will be created. The web actor is responsible for all communication with the client. Messages may arrive in the form of HTTP request (when the player logs in), or websocket messages for game data. Whatever kind of message, it is all handled by the actor, which manages the player’s server-side state.

This is the web actor’s main loop:

@Override
protected Void doRun() throws InterruptedException, SuspendExecution {
    ActorRef<WebDataMessage> client = null;
    String loginName = "empty";

    for (;;) {
        Object message = receive();
        if (message instanceof HttpRequest) {
            HttpRequest msg = (HttpRequest) message;
            loginName = msg.getParameter("name");
            if (loginName == null) // not logged in - reply with the login page
                msg.getFrom().send(new HttpResponse(self(), ok(msg, nameFormHtml())));
            else { // logged in - reply with the game page
                loginName = truncate(loginName.replaceAll("[\"\'<>/\\\\]", ""), 10); // protect against js injection
                if (spaceships.getControlledAmmount().get() / spaceships.players > 0.9)
                    msg.getFrom().send(new HttpResponse(self(), ok(msg, noMoreSpaceshipsHtml())));
                else
                    msg.getFrom().send(new HttpResponse(self(), ok(msg, gameHtml())));
            }
        } else if (message instanceof WebSocketOpened) {
            client = ((WebSocketOpened) message).getFrom();
            watch(client);
        } else if (message instanceof WebDataMessage) {
            WebDataMessage msg = (WebDataMessage) message;
            if (spaceship == null) {
                spaceship = spaceships.spawnControlledSpaceship(client, "c." + loginName);
                if (spaceship == null) // too many players
                    break;

                watch(spaceship);
            }
            if (msg.getFrom() == client)
                spaceship.send(msg); // pipe messages from the client to the spaceship
        } else if (message instanceof ExitMessage) {
            ActorRef actor = ((ExitMessage) message).getActor();
            if (actor == spaceship) { // the spaceship is dead
                spaceships.notifyControlledSpaceshipDied();
                spaceship = null;
            } else { // the client is dead
                if (spaceship != null)
                    spaceship.send(new WebDataMessage(self(), "exit"));
            }
            break;
        }
    }

    return null;
}

The web actor communicates with two other actors: one representing the client browser, and one for the player’s spaceship. It is watching both so that it’s notified with an ExitMessage if either of them die for whatever reason.

Once the player is logged in, a new spaceship is spawned, and all further events received from the client are simply passed to the ship actor. The client actor is passed to the ship actor’s constructor, so that the ship can directly send messages to the client without going through the web actor.

The spaceship actor is used both for player spaceships and NPC ships; for NPCs, the client field is null. Whenever the spaceship receives a message from the client it replies with a JSON update containing the ship’s state as well as that of all neighboring ships. Such an update is also sent to the ship’s client every 100ms.

We didn’t use any HTML template libraries or any JSON marshalling libraries because for our game, both the little HTML we needed, and whatever JSON messages we used were trivial; of course in your own application you might decide to make use of either.

Part 2 – Simulating the Virtual World

The game world simulation is pretty much the same as in the standalone demo (you can follow the link for a more in-depth discussion of how concurrency is used in the simulation), but here I’d like to talk some more about modeling a domain with actors and an in-memory database.

Rich Hickey is right. Programming is all about controlling and communicating state. But because we’re talking about a game-world simulation, let’s see how state is managed in the real world. In the real world, I guess you could talk about private and public state. Private state is what I know about myself. If you want to know what I know – my name, for example – you have to ask me. There is also public state, which is what you can know about me without actively asking. For example, my location is part of my public state, known to all who can see me. Of course, I can choose to share other information publicly, for example by printing my name on my T-shirt, which will make it known to people who can see me without them asking me. Public state is obtained without me actively participating in an exchange; alternatively, you can think about public state as something I constantly broadcast to the entire world or to a portion of it. Also, the owner of the private state decides what to share and who to share it with, while when it comes to public state, it is the consumer that decides what information to retrieve.

To continue this example, we see there are more differences between public and private state. If you’re nearby and can see me, you know my location instantaenously (well, at the speed of light). But if you’re asking exactly where I am on the phone, by the time I tell you and you’ve processed my reply, I could have moved.

In our simulation, actors manage private state. Every ship is an actor, running concurrently on its own fiber (lightweight thread). You can tell it things (like, “I just shot you”), or ask it things by sending it messages. The actor model, however, is not enough. I’d say it’s hardly ever enough for any application: you also need public state. Heck, even Erlang has ETS tables.

But while the actor model takes care of any concurrency issues we may have managing private state, public state is more tricky, as it can be accessed at any time, even concurrently with updates. One solution is storing public state in persistent (immutable) data strcutures. This is a fine approach, and the one used by Clojure atoms and agents. It has, however, two drawbacks. One, it tends to generate garbage (memory) of the worst kind – one that’s neither short lived nor particularly long-lived . Two, and more importantly, it does not allow concurrent modifications (i.e. it supports only on update at a time). So for public state, our game uses SpaceBase, our in-memory, concurrent spatial database. Transactions (including queries), provide instantaneus (in virtual time) access to shared public state.

Every spaceship actor, holds its private state in its class fields, while its public state is kept in the state variable, which is of type Record<SpaceshipState>. It is this record that is inserted into the database, wrapped by an instance of TransactionalRecord.

The Record class provides a standard API for a simple, mutable data record. It is very fast (about as fast as direct field access), but it allows us to protect a record instance with decorators like TransactionalRecord, that control access to the record’s data. TransactionalRecord will make sure that state records returned by database queries are only read and/or modified within a transaction.

For example, when an NPC spaceship is searching for targets, it performs this query that returns the public state records of the ships that are in our radar range:

try (ResultSet<Record<SpaceshipState>> rs = 
      global.sb.query(
        new RadarQuery(state.get($x), state.get($y), state.get($vx), state.get($vy), 
                       toRadians(30), MAX_SEARCH_RANGE))) {
    Record<SpaceshipState> nearestShip = null;
    double minRange2 = MAX_SEARCH_RANGE_SQUARED;
    for (Record<SpaceshipState> s : rs.getResultReadOnly()) {
        final double dx = s.get($x) - state.get($x);
        final double dy = s.get($y) - state.get($y);

        final double rng2 = dx * dx + dy * dy;
        if (rng2 > 100 & rng2 <= minRange2) { // not me and not too close
            minRange2 = rng2;
            nearestShip = s;
        }
    }
    if (nearestShip != null)
        lockOnTarget(nearestShip);
}

Any attempt to store the state records and access them outside a transaction, will result in an immediate runtime exception.

Get Started Now

First, go play the game here.

Then feel free to browse through the Comsat and Quasar documentation, and to check out the Comsat examples.

Discuss on Hacker News

Join our mailing list

Sign up to receive news and updates.

Tags: , ,

comments powered by Disqus