It seems to be possible to get split-brain

Summary: I've only seen this one, for this Issue Tracking Space, and don't understand the details. But there were clearly two shards in existence for this Space.

Behaviors seemed very consistent with split-brain: I would create a Thing, and something like a third or half of the requests to access it would fail in ways consistent with not knowing that that Thing existed.

Note that the Space Monitor showed nothing amiss -- it thought there only existed one copy of the Space. (Might it overwrite other entries, though?)

This is, of course, deeply unacceptable -- split-brain is the worst possible thing to be happening. But so far, I don't know how to repro this.

Later: Lightbend pointed out that we had auto-downing in our application.conf. I found this deeply mysterious -- I have no idea when that snuck in, but it's obviously wrong. I believe that that was the reason for split-brain.

That said, removing auto-downing means that we currently have no downing, because it turns out that ConductR doesn't actually deal with that for you. So I have to write my own downing code. Humph.