Last month, a number of Artificial Intelligence experts signed an open letter, warning against the dangers of summoning the demon. Are intelligent autonomous weapons going to get a will of their own, as is often depicted in science fiction movies? Or, in more scientific terms, as we build ever more powerful systems, how can we assure they don't fail to function according to their design, and that the design doesn't have an unintended effect?
The open letter is accompanied by a research paper discussing the risks and proposing research directions. It starts from mundane short-term topics like software security, goes on to discuss containment of powerful systems (make sure each drone always has a human owner who controls its kill switch), and ends with a discussion of the intelligence explosion.
Humanity is not human
My first reaction to this paper upon reading it yesterday, was on a meta-level, noticing how it is written from a positive perspective, how it is well-crafted to make as few assumptions as possible about what dangerously malfunctioning AI systems would look like in practice, and also how the AI research community itself is an "agent" that (like any research community or institute) tries to survive and grow by explaining why it needs more funding.
But let's not be cynical about the academia politics around this letter, because in fact (I only saw this after reading the paper), the Future of Life institute that was founded around this open letter is awarding (rather than requesting) 10 million dollars in research funding. So we can read the paper as a guideline to what the Future of Life institute would want you to work on if you want to be funded by them. The proposal submission deadline is in two weeks!
Observing academia politics is interesting though, and it is a good Segway for where I want to take this blogpost. It is interesting how most people don't care about existential risk. From a personal perspective of why each person doesn't invest more time into researching the topic, purely because it may affect them as an individual, or certainly the children and grandchildren they care about, it is probably purely a case of group responsibility: surely, someone else will take care of that.
From a political point of view, we can ask why governments (or humanity as a whole) are not acting more rationally, that is, why do we keep enacting results that nobody wants. This line of thought is an erroneously anthropomorphic way of looking at humanity. Humanity is made up of actors, but it is not itself an actor whose behavior makes any sense.
So group responsibility is the wrong word. Instead, there are individuals who are interested in working on long-term goals such as humanity's survival, and these individuals can work together to try to have a positive impact on the world. When you are yourself one of these individuals, you will probably have noticed how few people really think about what to work on.
Most people choose work they like to do, with people they enjoy working with, in a place where they want to live, and for which they get paid well. When you hear people talking about their job choice, it's often about which activity they want to be doing, and rarely about which industry they think they should be contributing to in order to have a positive impact with the work they do.
I often hear people say that working for a commercial company is a bit like prostituting 8 hours out of your day to someone else's goals. But usually what they mean by this is about the income inequality between them as an employee and the CEO or investor who they work for getting rich at their expense. This feels unfair, so they would rather start their own commercial business and get rich themselves.
Working for money
From a point of view of direct economy, when you are working for money, you are working for the people who ultimately consume the products you help produce. Teachers and nurses see this quite directly, but most computer programmers don't. The impact of your work as an employee is anonymized over many consumers who each pay a tiny fraction of the products they buy, ultimately to you.
If you work for Google or Facebook, then this is even more indirect, because people generally don't pay directly for the product you are working on. They will pay for sneakers, and then Nike will pay your employer for advertising, so in the end, what you are being paid for by the end-consumer, your ultimate value impact on the consumption market which feeds into your pay check, is that pair of sneakers and its nice luxurious branding.
Turning this around, when you buy a pair of sneakers, you are paying not only factory workers to produce your sneakers for you, but you are also paying a portion of the price to some scalability engineer optimizing Google's infrastructure.
Now, although Google's infrastructure has the pleasant side effect of improving the internet, and Google does sell some services directly to consumers instead of to advertisers, as the proportion of the money consumers spend on actual sneakers, it is clear that a scalability engineer at Google is contributing very little to the direct economy.
Adam Smith's utopian view of capitalism and the free market, where we end up doing the work that other people need us to do, has obviously stopped working when more and more excess capacity started to be allocated to bullshit jobs.
Put yourself in a rational agent's shoes
Now, how does this relate to AI security? Simple. Imagine you are a space traveller arriving at a foreign planet. You notice how there is competition among the creatures on this planet. They live in a bit of a Prisoner's Dilemma situation with each other - they generally behave peacefully, and cooperate, but there is an underlying survival instinct in each creature, which means that ultimately they compete with each other, and use ownership to create artificial scarcity (keep stuff to themselves) instead of sharing resources fairly with all other creatures.
You decide to settle on this planet, and to optimize your behavior for your survival there, treating this society of creatures as part of the terrain which you can mine for your own benefit. How would you do it?
One option is to steal stuff from the creatures. This is quite dangerous, because your victim will probably defend itself in the same way (or probably even more violently) as when a fellow creature would try to rob it of its property. Also, the creatures will probably team up against you, to either put you in jail or annihilate you. You could try to break out of this jail again by exploiting its security flaws, but unless you are at least an order of magnitude more powerful than the agression the creatures unleash on you, then your stay at this planet this is not going to be a pleasant one.
A much smarter option is to set up the creatures against each other, to work for you. Simply extract money from their economy and feed on that. For simplicity, let's assume the creatures use money, but it would probably work in any society where the individual creatures are at least partially selfish. So you start by selling something to one of the creatures, to get some of their money. This fundamentally changes your situation on this planet, since you are now not merely a foreign actor, you are now an autonomous proprietor. That gives you power to convince (rather than force) creatures to make shoes for you.
Non-human agents employing us
Translate this to AI security. We all know (I hope) that proprietary technology is, if not evil, at least suboptimal when compared to free technology. But now consider proprietor technology. A technological system that is the owner of something (!).
Commercial companies are already legal persons in the sense that, in the eyes of the law, a company can own something. That is why companies can tell us humans what to do. It is not your boss that tells you what to work on. It is the company, telling your boss to tell you, what to work on. Even CEOs can be fired by the board of investors, and the humans on this board are again employees of investment funds that act as legal persons.
We have already seen how the tobacco industry, the weapon industry, and indeed the proprietary technology industry can influence governments and use the law system to get what they, as legal persons, want us physical persons to work on. Watch "Food, Inc." if you haven't heard of Monsanto, or listen to what the FSFE has to say about the Microsoft monopoly with all its government contracts.
The NSA is agruably another example, with its uncontrollable secret court (although I think many humans in the territory they control, outside our little bubble of Snowden-followers, would vote in favor of the NSA's activities, so you could still say they form part of a democratically elected authority).
What's mine is mine, what's its is its
The concept of a legal person can be stretched further as technology progresses. Right now, the property of companies is defended by law and nation state armies. Because legal persons have no way of exerting physical persuasion themselves, they ultimately depend on the law, to act at all.
If I suspect you are going to steal my property, then I can personally take actions in the physical world (e.g. turn a key to lock a door), to protect what's mine. I'm not much of a safeguard in terms of my physical power, but thanks to tools, I am powerful enough in the physical world to (in a big enough percentage of cases) defend my own property.
For legal persons, this is different. Even with tools, they cannot protect their property without the help of at least one employee. This is where legal persons are fundamentally less powerful than physical persons.
But: legal persons have the law on their side. If you steal money from a bank (even if you're one of its employees with access to the safe), the cops come after you, and that's how the bank is able to possess money in the first place. If the bank would not be a legal entity, then it would be more accurate to say the boss of the bank, as a human, owns its assets.
Ownership without law
Now here's one scary insight: bitcoin changes that. For the first time in history, a computer program can now autonomously own money, and thus exert power over humans. This computer program does not need to be a legal entity, nor does it need to be owned by a human. Once a computer program owns bitcoins, it has become proprietor technology. It can rent computers to run on (i.e. pay humans to do so), extract money from humanity in the same way industries like advertising, corporate law, and banking work by extracting money from our economy.
And unlike a legal person, whose assets can be frozen by a judge, it can not be shut down without shutting down bitcoin.
In reaction to the open letter from the Future of Life institute, that's my proposal for what we should research: how can we avoid the rise of autonomous artificial proprietors. Scary, huh?
EDIT: Although bitcoin does make it much easier for non-humans to stash assets, it's not the only way. A computer program could actually also end up owning normal US dollars in tax havens and numbered bank accounts, since shell companies make it quite easy to hide even who ultimately owns a company. Thanks to Max Tegmark for pointing this out.