The Darwin Results

Epistemic Status: True story (numbers are best recollections)

This is post three in the sequence Zbybpu’f Nezl.

Previously (required): The Darwin Game, The Darwin Pregame.

I

It was Friday night and time to play The Darwin Game. Excited players gathered around their computers to view the scoreboard and message board.

In the first round, my score went up slightly, to something like 109 from the starting 100. One other player had a similar score. A large group scored around 98. Others did poorly to varying degrees, with one doing especially poorly. That one played all 3s.

Three, including David, shot up to around 130.

If it isn’t obvious what happened, take a minute to think about it before proceeding.

II

The CliqueBots had scores of 98 or so. They quickly figured out what happened.

David lied. He sent the 2-0-2 signal, and cooperated with CliqueBots, but instead of playing all 3s against others, he and two others cooperated with others too.

Whoops.

CliqueBots had been betrayed by MimicBots. The three defectors prospered, and the CliqueBots would lose.

Without those three members, the CliqueBots lacked critical mass. Members would die slowly, then increasingly quickly. If the three defectors had submitted CliqueBots, the CliqueBots would have grown in the first round, reaching critical mass. The rest of us would have been wiped out.

Instead, the three defectors would take a huge early lead, and the remaining members would constitute, as our professor put it, their ‘packed lunch.’

The opening consisted of CliqueBots being wiped out, along with G-type weirdos, A-type attackers and D-style cooperators that got zero points from the CliqueBots.

Meanwhile, on the message board, the coalition members were pissed. 

III

Everyone who survived into the middle game cooperated with everyone else. Victory would come down to efficiency, and size boosted efficiency. Four players soon owned the entire pool: Me and the three defectors.

I thought I had won. The coalition members wasted three turns on 2-0-2. Nothing could make up for that. My self-cooperation was far stronger, and I would outscore them over the first two rounds when we met due to the 0. It wouldn’t go fast, but I would grind them out.

It did not work out that way. David had the least efficient algorithm and finished fourth, but I was slowly dying off as the game ended after round 200. Maybe there was a bug or mistake somewhere. Maybe I was being exploited a tiny bit in the early turns, in ways that seem hard to reconcile with the other program being efficient. I never saw their exact programs, so I’m not sure. I’d taken this risk, being willing to be slightly outscored in early turns to better signal and get cooperation, so that’s probably what cost me in the end. Either way, I didn’t win The Darwin Game, but did survive long enough to win the Golden Shark. If I hadn’t done as well as I did in the opening I might not have, so I was pretty happy.

IV

Many of us went to a class party at the professor’s apartment. I was presented with my prize, a wooden block with a stick glued on, at the top of which was a little plastic shark, with plaque on the front saying Golden Shark 2001.

Everyone wanted to talk about was how awful David was and how glad they were I had won while not being him. They loved that my core strategy was so simple and elegant.

I tried gently pointing out David’s actions were utterly predictable. I didn’t know about the CliqueBot agreement, but I was deeply confused how they didn’t see this ‘betrayal’ coming a mile away. Yes, the fact that they were only one or two CliqueBots short of critical mass had to sting, but was David really going to leave all that value on the table? Even if betraying them hadn’t been the plan all along?

They were having none of it. I didn’t press. Why spoil the party?

V

Several tipping points could have led to very different outcomes.

If there had been roughly two more loyal CliqueBots, the CliqueBots would have  snowballed. Everyone not sending 2-0-2 would have been wiped out in order of how much they gave in to the coalition (which in turn accelerates their victory). Betrayers would have bigger pools, but from there all would cooperate with all and victory would come down to if anyone tweaked their cooperation algorithms to be slightly more efficient. David’s betrayal may have cost him the Golden Shark.

If someone had said out loud “I notice that anyone who cares about winning is unlikely to submit the CliqueBot program, but instead will start 2-0-2 and then cooperate with others anyway” perhaps the CliqueBots reconsider.

If enough other players had played more 2s against the CliqueBots, as each of us was individually rewarded for doing, the CliqueBots would have won. If the signal had been 2-5-2 instead of 2-0-2, preventing rivals from scoring points on turn two, that might have been enough.

If I had arrived in the late game with a slightly larger pool, I would have snowballed and won. If another player submits my program, we each end up with half the pool.

Playing more 2s against attackers might have won me the entire game. It also might have handed victory to the CliqueBots.

If I had played a better split of 2s and 3s at the start, the result would have still depended on the exact response of other programs to starting 2s and 3s, but that too might have been enough.

Thus these paths were all possible:

The game ended mostly in MimicBots winning from the momentum they got from the CliqueBots.

It could have ended in an EquityBot (or even a DefenseBot) riding its efficiency edge in the first few turns to victory after the CliqueBots died out. Scenarios with far fewer CliqueBots end this way; without the large initial size boost, those first three signaling turns are a killer handicap.

It could have ended in MimicBots and CliqueBots winning together and dividing the pool. This could happen even if their numbers declined slightly early on, if they survived long enough while creating sufficient growth of FoldBot.

CliqueBots could have died early but sufficiently rewarded FoldBots to create a world where a BullyBot could succeed, and any BullyBot that survived could turn around and win.

It could have had MimicBots and CliqueBots wipe out everyone else, then ended in victory for very subtle MimicBots, perhaps that in fact played 3s against outsiders, that exploited the early setup turns to get a tiny edge. Choosing an algorithm that can’t be gamed this way would mean choosing a less efficient one.

In various worlds with variously sized initial groups of CliqueBots and associated MimicBots, and various other programs, the correct program to submit might be a CliqueBot, a MimicBot that attacks everyone else but cheats on the coordination algorithm, a MimicBot that also cooperates with others, a BullyBot with various tactics, an EquityBot with various levels of folding, or an FoldBot. There are even scenarios where all marginal submissions lose, because the program that would win without you is poisoning the pool for its early advantage, so adding another similar program kills you both. 

This is in addition to various tactical settings and methods of coordination that depend on exactly what else is out there.

Everyone’s short term interest in points directly conflicts with their long term goal of having a favorable pool. The more you poison the pool, the better you do now, but if bots like you poison the pool too much, you’ll all lose.

There is no ‘right’ answer, and no equilibrium.

What would have happened if the group had played again?

If we consider it only as a game, my guess is that this group would have been unable to trust each other enough to form a coalition, so cooperative bots in the second game would send no signal. Since cooperative bots won the first game, most entries would be cooperative bots. Victory would likely come down to who could get a slight edge during the coordination phase, and players would be tempted to enter true FoldBots and otherwise work with attackers, since they would expect attackers to die quickly. So there’s some chance a well-built BullyBot could survive long enough to win, and I’d have been tempted to try it.

If we include the broader picture, I would expect an attempt to use out-of-game incentives to enforce the rules of a coalition. The rise of a true CliqueBot.

VI

I spent so long on the Darwin Game story and my thinking process about it for several reasons.

One, it’s a fun true story.

Two, it’s an interesting game for its own sake.

Three, because it’s a framework we can extend and work with, that has a lot of nice properties. There’s lots to maximize and balance at different levels, no ‘right’ answer and no equilibrium. It isn’t obvious what to reward and what to punish.

Four, it naturally ties your current decisions to your future and past decisions, and to what the world looks like and what situations you will find yourself in.

Five, it was encountered ‘in the wild’ and doesn’t involve superhuman-level predictors. A natural objection to models is ‘you engineered that to give the answer you want’. Another is ‘let’s figure out how to fool the predictor.’ Hopefully minimizing such issues will help people take these ideas seriously.

There are many worthwhile paths forward. I have begun work on several. I am curious which ones seem most valuable and interesting, or where people think I will go next, and encourage such discussion and speculation.

This entry was posted in Uncategorized. Bookmark the permalink.

16 Responses to The Darwin Results

  1. Pingback: The Darwin Pregame | Don't Worry About the Vase

  2. Michael Crone says:

    “I was deeply confused how they didn’t see this ‘betrayal’ coming a mile away.”

    I see the pre-plan as a real life (not just within game) experience. In real life, defections are not OK. The confusion may come from being on the boundary of game-space.

    • michealvassar says:

      If the pre plan is real life, then in real life they are defecting in real life against the people playing the game according to the normal rules of sportsmanship according to which real life shouldn’t effect the in game world.

      Also, in real evolution, the idea of ‘OK’ is ‘not OK’. Unless you believe that all defection is the result of entropy or emergent pseudo-complexity, or you adopt a perspective which puts the locus of play somewhere strange so that viruses are secretly cooperative towards their hosts, evolution is FULL OF defection AND cooperation.

      If it looks to you like real life is different from real evolution, it might be, but I would put much lower priors on defection not being OK than I would on narratives spinning defection retroactively as virtuous or unvirtuous depending on whether they had worked out and on the (probably not status) signalling value in the present of presenting a given narrative. Actually, now that I think about it, in real life, a lot of what rationalists (and to a much lesser extent biologists) call signalling is very flagrantly defection, in particular, defection against the group’s ability to coordinate or to reach consensus regarding the facts the signals are supposed to be signals of.

      • Michael Crone says:

        Thanks for the response. What I meant when I said “in real life, defection is not OK.” is that in real life, making agreements that you don’t intend to keep is not OK. So I disagree with your first paragraph – the clique defectors acted outside the normal rules of sportsmanship, in view.

        Real life evolution, on the other hand, has the morals of I rock. In other words, it is completely amoral, and has no concept of OK. It sounds like we agree here, and I didn’t mean to suggest otherwise.

      • Michael Crone says:

        in my view*

  3. Pingback: Rational Feed – deluks917

  4. BloodyShrimp says:

    Before Nate Soares became executive director of MIRI, there was a somewhat similar bot-submitting game played at Less Wrong where he was the coordinator/betrayer and I was outside the coalition/beat his bot:

    http://lesswrong.com/lw/hmx/prisoners_dilemma_with_visible_source_code/980r
    http://lesswrong.com/lw/hxy/prisoners_dilemma_tournament_results/9c3q

  5. SquareBoys says:

    I wanted to test some ideas after reading the first post, so I was implementing the game, but I never did write any actual strategies. Posting here in case anyone else wanted to experiment with it. Really not built for efficiency, and I might have sort of skimmed the actual problem statement: https://gist.github.com/anonymous/7c5d32479f2612e3e1374ec1d6863f97

  6. Magister Ludi says:

    Zvi, I enjoyed this series immensely. Thank you for it!

    Back in part 1, you wrote “In the late game, how efficiently you cooperate with yourself is very important.” But then the issue of cooperation with a copy of yourself isn’t much developed in the posts or in the comments. The idea is touched upon in part 2 with “I figured opponents would start 2 more often than 3, so I decided to do a 50/50 split to take advantage of that, coordinating faster and with a slight edge, at the expense of doing slightly worse against myself, but this was probably just a mistake and I should have done an uneven split (but not quite the fully maximizing-for-self-play ratio). However, in an endgame against a similar program, you can definitely get an edge by being slightly more willing to play 3s early than your opponent.” Would you be willing to elaborate on that?

    • TheZvi says:

      If I play against another bot using the same basic algorithm, but I play 3 slightly more often, when we play each other I am more likely to play the first 3, and thus gain +1 on them in that turn, and +0.5 in expectation overall (since I don’t know the polarity of the last round). Over time that adds up more than the self-meeting penalty if we are relatively close to the ideal target of 69% 2s. As I noted, 50% was too big to serve this purpose efficiently, and this mistake was likely quite costly.

      • Magister Ludi says:

        Thank you for the fast response!

        I think I understand what you were getting at now (corrections to faulty logic appreciated). You assumed that most opponents would be fine with allowing you to extort an expected 0.5 points per game when playing the first 3. If that prediction was correct, then I think you made the right choice. If you limit yourself to only 2s and 3s for the initial handshake, in the infinite turn version of the game, you expect to be 3 points below a maximum score when playing yourself with the 50-50, and about 2.618 below maximum with the optimum split of playing 2s with prob (5-sqrt(5))/4 ~ 69.1%. This seems to give you an edge as long as the populations aren’t badly skewed against you and your opponent isn’t using a better opening cooperation strategy.

        Was it the attempt to make yourself secure against the first-3 +0.5 point edge that kept you from playing the optimal 1/2/3 signalling strategy to gain a few more points when playing yourself?

      • TheZvi says:

        There are two problems with going 1/2/3. First, while it offers better self-cooperation, playing a 1 against someone else means they outscore you. Second, there’s a serious risk they don’t know what a 1 means in that spot – if they played 2, are they now 2 or now 3? If they get it wrong it could take a while to coordinate and that could get costly.

  7. Qiaochu Yuan says:

    This is really astonishingly similar to some of the things that happen in Liar Game. You might get a kick out of it.

  8. Daniel says:

    2-0-2 is virtue signaling

  9. RG says:

    I worry you learned the wrong lesson from David. Being straightforward and not-sneaky has reputational benefits.

    If you can convincingly make a separation where pre-game betrayal counts as in-game so all’s fair, that’s fine. But from e.g. http://magic.tcgplayer.com/db/article.asp?ID=11101 it doesn’t sound like that’s the case.

  10. Pingback: Best of Don’t Worry About the Vase | Don't Worry About the Vase

Leave a Reply to Qiaochu Yuan Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s