Last month, a number of Artificial Intelligence experts signed an open letter, warning against the dangers of summoning
the demon. Are intelligent autonomous weapons going to get a will of their own, as is often depicted in
science fiction movies? Or, in more scientific terms, as we build ever more powerful systems, how can we assure
they don't fail to function according to their design, and that the design doesn't have an unintended
effect?
The open letter is accompanied by a research paper discussing the risks and proposing research
directions. It starts from mundane short-term topics like software security, goes on to discuss containment of
powerful systems (make sure each drone always has a human owner who controls its kill switch), and ends with a
discussion of the intelligence explosion.
Humanity is not human
My first reaction to this paper upon reading it yesterday, was on a meta-level, noticing how it is written
from a positive perspective, how it is well-crafted to make as few assumptions as possible about what dangerously
malfunctioning AI systems would look like in practice, and also how the AI research community itself is an "agent"
that (like any research community or institute) tries to survive and grow by explaining why it needs more
funding.
But let's not be cynical about the academia politics around this letter, because in fact (I
only saw this after reading the paper), the Future of Life institute that was founded around this open letter is
awarding (rather than requesting) 10 million dollars
in research funding. So we can read the paper as a guideline to what the Future of Life institute would want you
to work on if you want to be funded by them. The proposal submission deadline is in two weeks!
Observing academia politics is
interesting though, and it is a good Segway for where I want to take this blogpost. It is interesting how most
people don't care about existential risk. From a personal perspective of why each person doesn't invest more
time into researching the topic, purely because it may affect them as an individual, or certainly the children and
grandchildren they care about, it is probably purely a case of group responsibility: surely, someone else will
take care of that.
From a political point of view, we can ask why governments (or humanity as a whole)
are not acting more rationally, that is, why do we keep enacting results that nobody wants. This line of thought is an erroneously anthropomorphic way of looking at
humanity. Humanity is made up of actors, but it is not itself an actor whose behavior makes any
sense.
So group responsibility is the wrong word. Instead, there are individuals who are interested in
working on long-term goals such as humanity's survival, and these individuals can work together to try to have a
positive impact on the world. When you are yourself one of these individuals, you
will probably have noticed how few people
really think about what to work on.
Most people choose work they like to do, with people they enjoy
working with, in a place where they want to live, and for which they get paid well. When you hear people talking
about their job choice, it's often about which activity they want to be doing, and rarely about which industry
they think they should be contributing to in order to have a positive impact with the work they do.
I
often hear people say that working for a commercial company is a bit like prostituting 8 hours out of your day to
someone else's goals. But usually what they mean by this is about the income inequality between them as an
employee and the CEO or investor who they work for getting rich at their expense. This feels unfair, so they would
rather start their own commercial business and get rich themselves.
Working for money
From a point of view of direct economy, when you are working for money, you are working for the people who
ultimately consume the products you help produce. Teachers and nurses see this quite directly, but most computer
programmers don't. The impact of your work as an employee is anonymized over many consumers who each pay a tiny
fraction of the products they buy, ultimately to you.
If you work for Google or Facebook, then this is
even more indirect, because people generally don't pay directly for the product you are working on. They will pay
for sneakers, and then Nike will pay your employer for advertising, so in the end, what you are being paid for by
the end-consumer, your ultimate value impact on the consumption market which feeds into your pay check, is that
pair of sneakers and its nice luxurious branding.
Turning this around, when you buy a pair of sneakers, you are paying not only factory workers to produce
your sneakers for you, but you are also paying a portion of the price to some scalability engineer optimizing
Google's infrastructure.
Now, although Google's infrastructure has the pleasant side effect of improving the internet, and Google does sell some services directly to consumers instead of to advertisers, as the
proportion of the money consumers spend on actual sneakers, it is clear that a scalability engineer at Google is
contributing very little to the direct economy.
Adam Smith's utopian view of capitalism and the free
market, where we end up doing the work that other people need us to do, has obviously stopped working when more
and more excess capacity started to be allocated to bullshit jobs.
Put yourself in a rational agent's shoes
Now, how does this relate to AI security? Simple. Imagine you are a space traveller arriving at a foreign planet.
You notice how there is competition among the creatures on this planet. They live in a bit of a Prisoner's Dilemma
situation with each other - they generally behave peacefully, and cooperate, but there is an underlying survival
instinct in each creature, which means that ultimately they compete with each other, and use ownership to create
artificial scarcity (keep stuff to themselves) instead of sharing resources fairly with all other
creatures.
You decide to settle on this planet, and to optimize your behavior for your survival there,
treating this society of creatures as part of the terrain which you can mine for your own benefit. How would you
do it?
One option is to steal stuff from the creatures. This is quite dangerous, because your victim
will probably defend itself in the same way (or probably even more violently) as when a fellow creature would try
to rob it of its property. Also, the creatures will probably team up against you, to either put you in jail or
annihilate you. You could try to break out of this jail again by exploiting its security flaws, but unless you are
at least an order of magnitude more powerful than the agression the creatures unleash on you, then your stay at
this planet this is not going to be a pleasant one.
A much smarter option is to set up the creatures
against each other, to work for you. Simply extract money from their economy and feed on that. For simplicity,
let's assume the creatures use money, but it would probably work in any society where the individual creatures are
at least partially selfish. So you start by selling something to one of the creatures, to get some of their money.
This fundamentally changes your situation on this planet, since you are
now not merely a foreign actor, you are now an autonomous proprietor.
That gives you power to convince (rather than force) creatures to make shoes for you.
Non-human agents employing us
Translate this to AI security. We all know (I hope) that proprietary technology is, if not evil, at least
suboptimal when compared to free technology. But now consider proprietor
technology. A technological system that is the owner of something (!).
Commercial companies
are already legal persons in the sense that, in the eyes of the law, a company can own something. That is why
companies can tell us humans what to do. It is not your boss that tells you what
to work on. It is the company, telling your boss to tell you, what to work on. Even CEOs can be fired by
the board of investors, and the humans on this board are again employees of investment funds that act as legal
persons.
We have already seen how the tobacco industry, the weapon industry, and indeed the proprietary
technology industry can influence governments and use the law system to get what they, as legal persons, want us
physical persons to work on. Watch "Food, Inc." if you haven't heard of Monsanto, or listen to what the FSFE has
to say about the Microsoft monopoly with all its government contracts.
The NSA is agruably another
example, with its uncontrollable secret court (although I think many humans in the territory they control, outside
our little bubble of Snowden-followers, would vote in favor of the NSA's activities, so you could still say they
form part of a democratically elected authority).
What's mine is mine, what's its is its
The concept of a legal person can be stretched further as technology progresses. Right now, the property of
companies is defended by law and nation state armies. Because legal persons have no way of exerting physical
persuasion themselves, they ultimately depend on the law, to act at all.
If I suspect you are going to
steal my property, then I can personally take actions in the physical world (e.g. turn a key to lock a door), to
protect what's mine. I'm not much of a safeguard in terms of my physical power, but thanks to tools, I am powerful
enough in the physical world to (in a big enough percentage of cases) defend my own property.
For legal
persons, this is different. Even with tools, they cannot protect their property without the help of at least one
employee. This is where legal persons are fundamentally less powerful than physical persons.
But: legal
persons have the law on their side. If you steal money from a bank (even if you're one of its employees with
access to the safe), the cops come after you, and that's how the bank is able to possess money in the first place.
If the bank would not be a legal entity, then it would be more accurate to say the boss of the bank, as a human,
owns its assets.
Ownership without law
Now here's one scary insight: bitcoin changes that. For the first time in history, a computer program can
now autonomously own money, and thus exert power over humans. This computer program does not need to be a legal
entity, nor does it need to be owned by a human. Once a computer program owns bitcoins, it has become proprietor technology. It can rent computers to run on (i.e. pay humans to do
so), extract money from humanity in the same way industries like advertising, corporate law, and banking work by
extracting money from our economy.
And unlike a legal person, whose assets can be frozen by a judge, it
can not be shut down without shutting down bitcoin.
In reaction to the open letter from the Future of
Life institute, that's my proposal for what we should research: how can we avoid the rise of autonomous artificial
proprietors. Scary, huh?
EDIT: Although bitcoin does make it
much easier for non-humans to stash assets, it's not the only way. A computer program could actually also end up
owning normal US dollars in tax havens and numbered bank accounts, since shell companies make it quite easy to
hide even who ultimately owns a company. Thanks to Max Tegmark for pointing this out.