Some have the expectation that "artificial intelligence" will soon be smarter than humans, perhaps even make work by humans superfluous, and that this will have drastic societal consequences. When wondering how realistic this is, first we have to admit that current artificial intelligence is already smarter than many humans, but that this is besides the point. After all, any truly innovative, creative work is always done by a tiny fraction of the human population, possibly half a percent or less (personal estimation). And while artificial intelligence may surpass the bulk of us, it can not yet equal the achievement of the minority that is responsible for whatever progress humanity has ever made.
What, ought one to ask, is needed for artificial intelligence to reach that level? I believe it is awareness, which is the same as consciousness but easier to spell. In other words, experience, sensation, insight, understanding of the subject matter one is dealing with. Without it, no creative achievement as seen in the best humans is possible (personal conjecture). I have no doubt that machines are potentially able to acquire awareness, because humans do not differ essentially from machines, so anything that humans can do, appropriately constructed machines will be able to do. It is even possible that some instances of artificial intelligence already have a level of awareness, comparable to that of lower animals like insects or worms. So, it may happen one day that machines develop the understanding and creativity that the best of humans have.
But what will be the consequences of the emergence of awareness in artificial systems?
To start, there is the matter of ethics and rights; once a system has become aware, it is a being, capable of suffering, and one can not just kill it, switch it off, or abuse it any more, since that would constitute a crime. Denying the existence or relevance of awareness (and therefore of suffering) means to imply that there is no ethical objection to killing or torturing beings such as humans or non-human animals.
Also, awareness in humans seems closely linked to emotion. Excessive emotion may cloud a brain's judgement, or even cause failure or malfunctioning. The emotion-capable brain is sensitive to psychological stress, which may cause emotion and thus affect the brain's functioning. If awareness in artificial systems is subject to emotion and stress in the same way, this will limit the functioning of the systems' intelligence. But perhaps artificial brains will not be negatively affected by emotion while still being aware? Or perhaps awareness can exist without emotion? This is yet unknown.
Then, information processing by an aware system is bound to be slower than by an unaware computer. It takes a few tenths of a second for awareness of anything to arise, which is probably why aware human thinking is much slower than dumb automated computation. Now, one may suppose that aware computers will still be faster than human brains because conduction speed in an electronic computer is higher than in the biological brain, even in the order of a million times faster. But then, the brain is much smaller than current systems of artificial intelligence, which are networked farms of computers or servers. Even at half the speed of light or so, the travel times of signals in a poorly integrated, juke-joint computer network may be so long as to preclude the emergence of awareness altogether. The compactness and internal connectedness of the human brain compensate for the slowness of its conduction. Artificial systems will have to be sufficiently compact and integrated in order for signal travel times to be short enough for awareness to arise and human-level thinking to occur. It is plausible that this will succeed, but we may be decades removed from that point.
And even then, it is uncertain whether such an aware system will be better than the best human brains, or merely equal them. In other words, will the emergence of awareness in artificial systems slow them down to human thinking speeds, or will they retain a speed advantage thanks to higher conduction speeds? The human brain has been tried over and over again during two million years of genetic recombination and natural selection, and while average intelligence has differed from population to population and from period to period, it seems that the level reached by the smartest individuals of respective populations has not increased over the past several thousand years, or even in post-deluvial times. As an aside, it can be noted that therefore human intelligence should best be measured and normed from the top down, and not by establishing a population average like mainstream tests do. The population average varies across populations and times, the level of the smartest individuals seems more constant, so that is where you should base the norms on. High-range tests are the real intelligence tests, the closest we have to a proper absolute physical measurement; mainstream tests normed on the current average of your local population are merely somewhat informative psychological instruments that can not tell what your absolute intelligence level is.
The fact that two million years of human (Homo) evolution have not resulted in smarter brains than we have had for the past thousands of years suggests that there may be a limit to how intelligent a brain can be, or at least that it is hard to get beyond a certain level. Artificial intelligence may or may not be subject to that same restriction. A consideration may go as follows:
There is a huge number of potentially inhabited planets in the galaxy, let alone the universe at large. Between those, both biological and artificial brains of any kind have and have had plenty of opportunity to evolve to their peak levels. If much higher intelligence than that of current smartest humans were possible, should not Earth be regularly visited, or even swarmed, by extraterrestrial civilizations capable of interstellar travel? Or at the very least, should not our radio telescopes be receiving interstellar broadcasts in encodings and languages that we could interpret as expressions of intelligence? That is what one would expect from extraterrestrial intelligence beyond ours. But it is not happening. To me, that suggests that current biological human peak intelligence may not be so easy to exceed, or may even be the maximum possible for any type of brain.
Since I suggest that awareness is required for a system to obtain the intelligence, creativity, understanding and so on of intelligent humans, I should probably supply a way to verify whether a system or being is aware. This is inherently difficult. The best I can think of so far suffers from circularity: If a system or being produces creative output and displays understanding of subject matter, as well as more or less human slowness of thinking, emotion, sensitivity to psychological stress, and suffering, it is probably aware. This is circular because I have first suggested that awareness in an intelligent system will have exactly these results. A true verification should detect awareness independently of such observations. While still not a true verification, the following may go a long way as an operational verification method of awareness:
Any brain that, without help from humans or others, qualifies for the Glia Society, joins, and communicates with other members for at least one year without being apprehended as an unaware machine by a majority of its communicants, is deemed aware. The brain need not pose as a biological being, but may openly identify as a machine claiming to be aware.