When Does an Intelligence Become a Collaborator?

Here's something I've been thinking about. If we encountered an alien civilization a million times more intelligent than us, would they see us as something they could actually cooperate with? Or would we look more like chimpanzees do to us?

I think there's a chance the answer has less to do with how much smarter they are and more to do with whether we've crossed some minimum level of ability to communicate, understand agreements, and work toward shared goals. And if that distinction is real, it might apply to AI too.

Rather than just argue about it, I want to see whether we can describe that idea mathematically, and then figure out what would have to be true for it to hold up.

The part I think matters

We usually talk about intelligence as if everything belongs on one ladder. Chimpanzee, human, alien, superintelligent AI. But even if we could rank all four on one scale, would that tell us whether two of them could negotiate an agreement?

Not really. An alien could be wildly beyond us in physics and still understand our language and our goals. A chimpanzee is much closer to us biologically, but there's a limit to what kinds of abstract agreements we can make with one. That's the distinction I'm interested in.

Of course, the smarter participant can compensate for the weaker one. An alien might invent an interface we couldn't have imagined. So any useful model has to describe the relationship between two intelligences, not just assign each one a score.

Putting a model behind the idea

Start with two agents, A and B, attempting a task together. Define success carefully: they need to communicate a joint goal, agree on responsibilities, and actually follow through. Let the environment be e and the available communication and tools be r. Then:

That gives us something we could observe. It doesn't require us to know an alien's IQ. The next step is to try to predict it.

Give each agent a vector of independently tested capabilities: learning unfamiliar symbols, inferring another agent's goals, remembering commitments, planning across time, and adapting when the rules change. Call those vectors \(\mathbf{x}_A\) and \(\mathbf{x}_B\).

It looks complicated, but the idea is simple. The first terms represent what each participant can do. The \(W\) term asks whether their abilities work together. The final terms account for the difficulty of the task and the quality of their communication.

This is a candidate statistical model, not a discovered law of cognition. Its advantage is that it gives us something concrete to fit, criticize, and compare against alternatives.

Where would a threshold come from?

There's an important trap here. If I decide that a system needs a 90% success rate to count as a collaborator, I haven't discovered a threshold. I've just picked one.

The interesting possibility is that a threshold appears in the data without us putting it there. Maybe cooperation improves gradually as the weaker agent gets better. Or maybe there's a fairly narrow region where open-ended communication suddenly becomes much more reliable.

A steep curve on one benchmark wouldn't prove much. Smooth underlying changes can look abrupt if the tasks happen to cluster around a particular difficulty. We'd need the same pattern to survive different tasks, different partners, and different ways of measuring capability.

The prediction I'm most interested in

Imagine the human and alien both have the minimum abilities required to establish an agreement. Now make the alien smarter. And smarter. And smarter again.

Does the relationship eventually break down just because the gap gets enormous? My guess is that it doesn't necessarily have to. The two sides might have very unequal power and understanding, but they could still communicate and cooperate.

That suggests a specific test: after accounting for the weaker participant's capabilities and the communication channel, does the difference between their capabilities still strongly predict failure? We can test that instead of assuming the answer.

How I'd actually test it

AI systems are a practical place to start, since we can vary models, reasoning budgets, memory, tools, and communication channels. I'd pair systems with similar capabilities and systems with deliberately large gaps, then give them tasks requiring real coordination: negotiating with private information, teaching each other a new symbolic language, making commitments over multiple rounds, and adjusting plans when goals conflict.

I'd want held-out tasks and unfamiliar partners. I'd also separate actual mutual understanding from a case where the stronger agent quietly does everything. Success alone isn't enough to tell those apart.

Then fit the interaction model above and compare it with smoother models and explicit change-point models. Use held-out predictive accuracy and calibrated uncertainty, not whichever curve looks best. If the supposed boundary disappears when we change the benchmark, that matters. If a continuous model predicts better, we should accept that too.

What this would mean for AI

The obvious implication is that we might eventually have a more useful way of talking about when an AI system becomes something we can make reciprocal agreements with, rather than only something we give commands to.

But I want to keep a very important distinction. Being capable of cooperation doesn't automatically mean being conscious, deserving rights, or being safe to trust. A system could be an excellent negotiator and still be dangerous. Something could also deserve moral consideration without being capable of negotiation at all.

So this wouldn't be a personhood test. It would be a way to measure one specific thing that I think gets lost when we reduce intelligence to a ranking.

What would change my mind?

If cooperation turns out to be a smooth function of dozens of context-specific abilities, with no generalizable boundary, then the threshold idea is probably wrong. If the stronger participant can always compensate for a weaker one, that would also complicate the idea. And if the apparent threshold only exists for systems trained on human language, it might not tell us anything about aliens.

I don't think we need to know the answer before building the model. That's the whole point of making the idea testable.

What interests me is the possibility that humans could be incomprehensibly less intelligent than another civilization and still belong to the same broad category of beings capable of participating in one. If that's true, I want to know what that category actually consists of. And I think AI gives us a way to start looking.