Giving a fly's wiring a game to play

I came across ornata's fly project, which connects a simulated fruit fly network to video games, and wanted to try building something along similar lines in the browser. The part that interested me was being able to follow the whole process: what goes into the network, what happens to the signals, and how those signals become a move in a game.

That became Fly Plays. It uses 166,700 retained neurons and 25,582,938 weighted connections from a real fruit fly wiring map, with Space Invaders and Tetris as the games. Your browser runs the simulation and a small controller that starts without knowing how to play.

There are several different ideas packed into that description. I will try to explain them through examples, since I think following one signal through the system is more useful than just listing the components.

Bear in mind that this is an engineering experiment with simplified models of neural activity. The first comparisons haven't shown consistent improvement at the games. I will get into those results as well, because getting something to run and showing that it learns something useful are separate things to establish.

First, having a map

A connectome is a map of nerve cells and the connections between them. The dataset used here is called MaleCNS, and it covers a male fruit fly's brain and ventral nerve cord. The nerve cord is part of the system involved in controlling the body.

For an engineering analogy, picture a road map. It tells you which places are connected and which paths you could take. If you wanted to simulate traffic, though, you would still need to decide how cars behave, when they enter a road, and how they affect each other.

The map gives you structure. A simulation needs rules for what happens on that structure.

This distinction carries through the whole project. I can use the measured connections to decide where a signal goes, while still making a fairly rough assumption about what that signal does when it arrives.

The MaleCNS reconstruction shown in Google Research's article. The large image shows reconstructed anatomy; the flashing dots in Fly Plays show activity calculated by my simulation.

The work behind that map

MaleCNS came from a collaboration led by HHMI Janelia, with Google Research, the University of Cambridge, and the MRC Laboratory of Molecular Biology. The researchers used high-resolution electron microscopy to image the tissue, then reconstructed and checked the cells and their connections. Janelia's account of the project describes how the imaging, computation, proofreading, and cell labeling fit together.

The motivation is to understand how neural circuits produce behavior. Having the brain and nerve cord in one map lets researchers follow paths from sensory input toward the systems controlling movement. It also gives them a way to compare male and female circuitry, including circuits involved in social behavior. Those are the research questions behind the dataset; the arcade is something I added afterward.

Google's part: tracing the cells

Google Research helped turn microscope images into three-dimensional reconstructions of nerve cells using AI. This involves segmentation, which means identifying which parts of an image belong to which object. Human experts then checked and corrected the reconstruction. Google also developed Neuroglancer, a tool for exploring large datasets like these. Google's explanation of the MaleCNS work is a good starting point.

Take for instance a cell with a long, branching extension passing close to another cell. A reconstruction has to follow that extension without accidentally joining it to its neighbor. If it joins them, the resulting wiring map can contain connections that shouldn't be there.

One approach Google developed is the flood-filling network. It starts from a location and repeatedly predicts which surrounding image pixels belong to the same object, growing the region as it goes. You can think of this as following one thread through a tangled pile, using what you've already followed to guide the next step. The real method works on three-dimensional image data. Google's earlier explanation of flood-filling networks includes animations of that process.

This is an example of AI being used to help construct a scientific dataset. Once the map exists, other people can build experiments that use it. The AI doing the image reconstruction and the controller playing a game here have different jobs.

From the research files to something a browser can use

The official release includes cell annotations, neurotransmitter predictions, connection tables, and much larger imaging data. The connection-strength table alone is around 1.1 GB. Loading all the research files and preparing them on every visitor's device would make for a fairly unpleasant first visit, especially on a phone.

I prepare the runtime files ahead of time with a Python script. The process is reproducible, and after the original downloads it can run offline.

First, the script keeps cells with a nonempty broad cell-class annotation, including provisional annotations. It keeps every connection between those retained cells. That produces the same 166,700 cells and 25,582,938 weighted connections documented by the reference project's technical notes.

N.B. those 25.6 million weighted connections and the roughly 125 million synapses reported for MaleCNS count different things. Several synaptic contacts between the same source and destination can contribute to one weighted connection. The browser is working with a graph of connections between retained neurons, rather than a separate simulated object for each anatomical contact.

Each retained cell gets a compact numeric index. The connections are grouped by their source cell, so when cell 42 fires, the engine can go directly to the list of cells it connects to. There is no need to search the entire graph to find its neighbors.

The preparation also assigns positive or negative influence using the source cell's predicted neurotransmitter and normalizes the incoming connection magnitudes for each destination. In this model, GABA, glutamate, and histamine are treated as inhibitory. Other or unknown transmitters are treated as excitatory. This is a coarse modeling choice; the resulting numbers aren't measurements of the exact electrical effect at every synapse.

The output is a set of compact numeric arrays, compressed into 32 runtime chunks. They total about 139 MB, or 132 MiB in the units shown on the page. All retained connections remain in those arrays. I didn't remove weak connections to make the simulation faster.

Now let's follow one game picture

At a high level, the loop looks like this:

Game picture
    ↓
Simulated eye cells
    ↓
Signals through the fly's wiring
    ↓
Recent activity from output cells
    ↓
Controller chooses a button press
    ↓
Game changes and gives feedback
    ↓
Repeat

There are a few choices hidden inside each arrow, so let's go through them.

1. Give the network something to see

The game renders a small grayscale picture of its playfield, 64 pixels wide and 48 pixels tall. This is the picture you see in the vision panel. The score display, buttons, and explanatory text around the game stay out of it.

Imagine an enemy moving across an otherwise dark part of the Space Invaders screen. Some pixels get brighter and others get darker. The simulation uses both brightness and changes from the previous picture to stimulate 6,006 annotated photoreceptors, which are the cells serving as the simulated eyes.

The screen-to-cell mapping uses published information about visual columns where available, connections to assigned cells for others, and deterministic estimates for the remaining cells. It is approximate. A 64 by 48 pixel game screen doesn't suddenly become an accurate model of a fly's optics because I connected it to cells labeled as photoreceptors.

This is one of the first places I would look when trying to improve the experiment. A useful network can still receive a poor representation of the thing it is supposed to respond to.

2. Pass signals through the wiring

The cells use a model called leaky integrate-and-fire. The name sounds more complicated than the basic idea.

Picture a bucket with a small leak. Incoming activity adds to it, some activity fades over time, and when it reaches a threshold the cell fires and resets. That firing then affects the cells connected to it. Depending on the modeled connection, it can push a neighbor toward firing or pull it away.

For a made-up numerical example, suppose a cell is at 0.7 and its threshold is 1. After some activity fades, it is at 0.6. If the net incoming signal adds 0.5, it reaches 1.1, fires, and resets. If that incoming signal had been negative, it might instead have moved farther from the threshold.

The exact values in that example are just to build intuition. The implemented model has its own decay, gain, background input, and threshold settings. It also includes seeded background activity, so the visual input isn't the only thing capable of making a cell fire.

The engine advances in steps of 20 milliseconds of simulated time. At each step it updates every retained cell, and activity from cells that fired on the previous step travels through their outgoing connections. This lets the engine avoid propagating spikes from cells that didn't fire, while preserving the graph and every scheduled neural update.

The brain panel shows 4,096 sampled cells at measured positions, projected into two dimensions. Their brightness follows simulated firing with a short fade. The other cells are still running; they just aren't all being drawn.

3. Turn activity into a move

The controller reads recent firing activity from 1,316 descending neurons, cells that carry signals away from the brain. These rates summarize a short window of activity rather than just whether a cell fired at one instant.

Ten times per simulated second, the controller produces probabilities for the available actions. In Space Invaders, those include waiting, moving, firing, and moving while firing. In Tetris, they include moving, rotating, dropping, and waiting.

For example, suppose the controller gives moving left a 40% chance and firing a 20% chance. Moving left is more likely, but the controller can still pick another action. Trying different actions is part of the learning process. The action bars on the page show those probabilities.

The controller's inputs are these neural rates and a bias term, which lets it develop a general preference for an action. It doesn't get the enemy coordinates, a list of occupied squares, or a description saying that a Tetris piece would fit nicely in the gap on the right.

That restriction matters to the question I want to explore. Any useful information about the playfield has to reach the action chooser through the simulated neural activity.

What learning from scratch means here

The controller starts with small seeded random weights, without a policy already trained to play either game. Its decisions can then change as it gets feedback.

The method is called actor–critic. The actor chooses actions. The critic estimates how much future reward might follow from the current input. After a move, the learner compares the reward and its next estimate with what it expected before.

If the outcome is better than expected, the update generally makes the chosen action more likely for similar input. If it is worse, the update pushes in the other direction. The critic adjusts its estimate too. The implementation also encourages some exploration and bounds its updates, so a large reward doesn't produce an arbitrarily large change.

Take the earlier example of moving left and firing. If an enemy gets hit later, the learner receives a positive reward. But it still has to work out whether the earlier activity and action were useful predictors of that outcome. A single reward doesn't explain the rules of Space Invaders to it.

This gets harder when a useful action only pays off much later. In Tetris, placing one piece well may help you clear a line several pieces from now. Deciding which earlier choices deserve credit is a real part of the problem.

The fly network's connections stay fixed during all of this. The small readout controller is what learns. Reloading with a saved checkpoint also means the controller is continuing its practice; it only starts from scratch on a fresh browser or after Reset learning.

Giving feedback in each game

Space Invaders has relatively frequent events to reward. Destroying an enemy gives +1, completing a wave gives +3, and losing a life gives -5. The game includes waves, shields, shooting, and three lives.

Tetris uses a 10 by 20 board and a seven-piece bag, so each set of seven pieces contains each shape once in a shuffled order. It supports rotation, soft and hard drops, line clearing, and increasing gravity.

Clearing one, two, three, or four lines gives rewards of 1, 3, 5, or 8. Reaching the top gives -5. I also added smaller feedback based on changes in stack height and holes, bounded to keep those hints from growing arbitrarily large.

This is called reward shaping. You are giving the learner intermediate hints about progress toward the task.

Counterpoint: those hints reflect my assumptions about what is useful. Keeping a stack low is generally helpful, but a move that temporarily makes the stack taller could be part of a better plan. A reward function can encourage behavior that looks good by its own numbers while missing the thing you actually care about.

The game code can inspect the board to calculate a reward. The action chooser receives the resulting number, along with neural activity. Board features aren't secretly being passed in as extra instructions for its next move.

Did it actually improve?

The first evaluation was deliberately small: three training episodes, each capped at 90 simulated seconds. I then froze the trained controller and compared it with an untrained frozen controller and random actions across three fixed evaluation seeds. Each evaluation episode was capped at 45 simulated seconds.

Freezing the trained controller makes it easier to compare the behavior it learned without continuing to change it during the comparison. The same evaluation seeds make the starting conditions more comparable, although different actions can still lead to very different games.

These are the mean game scores, rounded to one decimal place:

The trained Space Invaders controller slightly exceeded the untrained one, but random actions scored higher than both. The Tetris results also didn't show a consistent advantage from training. None of the compared controllers cleared a Tetris line.

N.B. Tetris awards game score for dropping pieces as well. A positive score can therefore look encouraging even when the controller hasn't achieved the central task of clearing lines. Game score and the reward used for training aren't the same number. Looking at cleared lines changes how you interpret that result.

Three seeds and a few short training episodes are a small test. They don't establish long-term improvement or prove that improvement is impossible. They tell me that I haven't demonstrated reliable game learning with this setup yet.

I also checked that changing the visual input path changes downstream readout activity. In a 100-step comparison, connected and disconnected visual input produced different readouts on 90 steps. That shows the input has a causal effect in this model. It doesn't establish that the resulting activity contains a useful strategy for playing the game.

I think it's useful to keep those claims separate. The input affects the network, the learning rule updates the controller, and the games run. Whether the entire combination reliably learns to play better still needs more evidence.

Fitting it into a browser

There is a practical engineering problem here as well: running a large simulation without making the page stop responding.

The network, game state, and learner live together in one Web Worker. Think of it as a separate worker handling the experiment while the main browser thread deals with the page and draws the pictures. The worker sends small snapshots containing things like the game pixels, sampled activity, action probabilities, and scores. It doesn't keep copying the whole network back and forth.

The main loop works in bounded batches. If your device can't keep up, simulated time slows for both the game and the network. At 0.5× actual speed, one second in the simulation takes about two seconds on your clock. This preserves the sequence of neural updates instead of skipping parts of the simulation to match real time.

In the desktop browser checks on my M2 machine, a fresh load ran at roughly 1× simulation speed with 30 frames per second. The reconstructed graph arrays alone occupy about 196 MiB, before the rest of the application and browser overhead. I haven't tested physical iOS and Android devices, so I wouldn't generalize that desktop result to every phone.

Each game has its own controller and saved progress. The browser periodically writes the learned weights, counters, and recent scores to IndexedDB, its local database. Switching games starts a new round for that game while preserving its practice. Restart also preserves practice. Reset learning clears the selected game's progress.

If browser storage is unavailable or full, it can keep running for the session, with a notice explaining what won't persist. Switching away from the page pauses the experiment. Nothing about the saved learning syncs across devices.

The download is its own problem

The simulation happens on your device, but the initial wiring still has to get there. I host the prepared chunks on Cloudflare Pages, with the existing Vercel copy as a backup. The browser requests the files directly from Cloudflare, so the large download normally bypasses the site's Vercel bandwidth allowance.

Every chunk is checked against its expected size and SHA-256 checksum before use. Think of the checksum as a way to check that the bytes received match the prepared file. Completed chunks are cached locally when possible, and retries reuse them.

If the primary host fails, returns a damaged file, or stops sending data for 15 seconds, the loader switches to Vercel for the rest of that load. The cached files use the same keys whichever host supplied them. Moving hosting therefore doesn't force a returning visitor to download everything again.

This is a fairly ordinary piece of engineering, but it matters to whether someone can actually try the interesting part of the project.

Where I want to take it next

The next useful work is to run longer comparisons across more seeds and change one modeling choice at a time. I would start with the visual mapping, the activity reaching the readout, and the reward setup. It would also be useful to compare the measured wiring with controlled alternatives to see whether its particular structure contributes anything to game performance.

Adding more games is tempting, but understanding why the current setup does or doesn't improve would teach me more than another game tab by itself.

In So you want to learn AI, I wrote about continuing to build as you learn and paying attention to the limits of your tools. This project is an opportunity to do that. It brings together data preparation, simulation, reinforcement learning, and browser engineering, and gives me a concrete result to question at each stage.

You can try Fly Plays here. Watch the vision panel, the cell activity, and the action probabilities together. Give it a few rounds, then compare the scores. The point is to make the process visible enough that you can start asking which part is doing something useful, and which part still needs work.

The underlying MaleCNS data is credited to FlyEM / HHMI Janelia, Cambridge, MRC LMB, and Google Research under CC BY 4.0. My version modifies its indexing, weights, visual mapping, and packaging for this experiment. ornata/fly provided the inspiration; the browser engine and games are independently implemented in TypeScript.