Chapter Eleven
DEEP NEURAL NETWORK
In chapters 9 and 10, we examined two different brain regions, the prefrontal cortex and the precuneus, whose expansions may have been pivotal for the unique human capacity for high-level abstraction. Here, we will entertain the possibility that this human capacity is owed primarily to the overall organization of the brain rather than the expansion of a particular brain region. For this, let's turn our attention to artificial neural networks. As a major branch of artificial intelligence, they perform computations based on architectures that mimic biological neural networks. Historically, there have been four waves of artificial neural network research since the 1940s. The third one was in the 1980s. I began my neuroscience career as a graduate student in California around this time. I still vividly remember the excitement they generated at the time.
It was, and still is, difficult to understand the brain because of its complexity. The human brain has approximately a hundred billion neurons and a hundred trillion connections, and many different types of neurons form a vastly complex system of connections. It is therefore a daunting challenge to unravel the way this complicated system processes information to guide behavior in an ever-changing world. Experimental neuroscientists study the brain directly, many using animal models that have somewhat less complex brains. For example, neurophysiologists place microelectrodes in the brain to monitor activity of single neurons while an animal is engaged in certain behaviors, such as deciding which of two available levers to press. As an alternative approach to understand the brain, computational neuroscientists study what artificial neural networks can do and how they do the job.
In the 1980s, neuroscientists and engineers alike hoped that artificial neural networks would lead to significant advances in our understanding of the brain and offer novel solutions to challenging AI problems such as visual object classification. To their disappointment, the initial excitement gradually subsided because, despite their significant contributions, artificial neural networks did not live up to expectations. The fourth wave of artificial neural networks is currently in progress, and 'deep learning' is driving this wave.
DEEP LEARNING
Artificial neural networks are made up of layers of units (or nodes or artificial neurons) that behave according to simplified rules derived from biological neurons (such as 'integrate and fire,' in which each unit sums active inputs and emits outputs when a threshold is crossed). Typical artificial neural networks have an input layer, an output layer, and a variable number of hidden layers (see fig. 11.1). So, what exactly is deep learning in neural networks? It refers to learning performed by a multi-hidden-layer artificial neural network. Hidden layers enable a neural network to perform complex, nonlinear computations. The depth of a network refers to the number of hidden layers.
FIGURE 11.1. A schematic for an artificial neural network. Figure reproduced from Massimo Merenda, Carlo Porcaro, and Demetrio Iero, 'Edge Machine Learning for AI-Enabled IoT Devices: A Review,' Sensors (Basel) 20, no. 9 (April 2020): 2533 (CC BY).
In the early 2010s, scientists found that performance of artificial neural networks in certain applications could be enhanced dramatically by adding more hidden layers (i.e., making the networks deeper). Deep neural networks started outperforming other methods in prominent image analysis benchmarks, most notably the ImageNet Large Scale Visual Recognition Challenge in 2012, and these achievements sparked the artificial intelligence 'deep learning revolution.' Deep learning is now used in almost every field of AI research.
How do the hidden layers function? Multiple hidden layers allow for multiple levels of abstraction, which is useful for hierarchical feature extraction. Assume you want to train a neural network to recognize animal species from their photographs (see fig. 11.2). Once you train it with some sample animal images (elephants, seals, etc.), then you test it with new images of these animals. A shallow neural network (e.g., one hidden layer) can be trained to perform this task, but the number of required feature detectors in the hidden layer grows exponentially with the number of inputs. A deep neural network can alleviate this problem.
FIGURE 11.2. Learning feature hierarchy by a deep neural network. Figure reproduced with permission from Hannes Schulz and Sven Behnke, 'Deep Learning: Layer-Wise Learning of Feature Hierarchies,' Künstliche Intelligenz 26, no. 4 (2012): 357.
In the example shown in fig. 11.2, a six-layer feed-forward neural network successfully classifies animal species from new images by forming a feature hierarchy. Early hidden layers represent low-level features such as localized edges, which are progressively combined to represent more abstract features in late hidden layers, eventually representing high-level features for animal species prototypes (e.g., 'elephant-ness'). This is a powerful and efficient way of implementing visual object recognition. Note that no information about animal species was provided to the hidden layers during training. The neural network organizes itself to hierarchically represent features to achieve the task. In fact, this line of neural network research was inspired by findings from visual neuroscience. Neurons in the early stages of visual processing are responsive to low-level features such as edges, whereas neurons in the late stages of visual processing are responsive to more complex features such as faces. This example shows that high-level feature extraction can be readily implemented in a deep neural network.
UNSUPERVISED LEARNING
The artificial neural network in the preceding example performed abstraction as a result of supervised learning. The species of an animal in the image was explicitly given to the artificial neural network (although not to the hidden layers) during training. One might then argue that the artificial neural network performed abstraction because it was trained to do so. In other words, artificial neural networks can perform abstraction only when explicitly trained to do so in a narrow domain. For example, a network trained to classify animal species may perform poorly in other domains such as classifying vehicle types. If so, understanding learning-induced abstraction in deep artificial neural networks may provide no useful insight into the neural processes behind human abstraction. This, thankfully, is not the case. Deep neural networks can be trained to represent abstract concepts using unsupervised learning.
In 2012, the team led by Andrew Ng and Jeff Dean trained a nine-layered neural network with millions of unlabeled images (i.e., no information was provided about the images). Surprisingly, after three days of training, face-selective neurons emerged. The best-performing neuron in the network identified faces from new images with 81.7 percent accuracy. This study and subsequent ones have convincingly shown that deep neural networks can represent high-level features (the prototype of face in the above example) without a specific instruction about the features of interest.
A PRIORI ABSTRACTION
More recent studies demonstrated, even more surprisingly, that deep neural networks can represent abstract concepts even without any training whatsoever, further indicating that there is something special about the deep neural network structure in representing abstract concepts. Se-Bum Paik, my colleague at the Korea Advanced Institute of Science and Technology, constructed biologically inspired deep neural networks. In one study, his team found that neurons in a simulated multilayer neural network show number-specific responses even without any training. In other words, number sense emerges spontaneously from the statistical properties of multilayer neural network projections. These number-coding neurons of the simulated neural network showed similar response characteristics to those of number-coding neurons found in the brain (see chapter 9). For example, their sensitivity to discriminate between two numbers decreases as the size of the numbers increases (e.g., the sensitivity to discriminate between 1 and 2 is similar to the sensitivity to discriminate between 10 and 20), which obeys the Weber-Fechner law (the intensity of a sensation is proportional to the logarithm of the stimulus intensity).
In another study, the team found face-selective neurons in a multilayer neural network that simulated the monkey visual pathway without any training. These face-selective neurons showed many response characteristics that are similar to those of face-selective neurons found in the monkey brain. These studies tell us two important points. First, number selectivity and face selectivity may emerge spontaneously in a neural network without any learning. Second, such properties are found only in neural networks with a sufficient depth (i.e., enough hidden layers). Then, a sufficiently deep neural network, such as the primate brain, may be able to generate abstract concepts (such as the number sense) without any prior experience. In other words, the brain's multilayered network structure may be the source of innate cognitive functions such as abstraction.
CONCEPT CELLS
Studies using artificial neural networks suggest that the superb human capacity for high-level abstraction may be because the human brain has more layers of connections than other animals rather than having a dedicated neural network for high-level abstraction. In fact, the basic circuit structure is similar across all mammalian cortices. From this perspective, the key to higher cognitive functions of the prefrontal cortex (see chapter 9) may be that it is the highest-order association cortex rather than the existence of a specialized neural circuitry. In particular, the frontopolar cortex is at the top of the connection hierarchy (see fig. 9.4), which may be why it deals with particularly high-level abstract concepts in humans.
But what about the hippocampus? It can also be considered the highest-order association cortex. Thus, humans may be able to imagine freely using high-level abstract concepts because those concepts are formed spontaneously as sensory inputs go through multiple layers of the neural network before reaching the hippocampus. Note that counting the number of hidden layers between sensory receptors and the hippocampus is not as straightforward as counting the number of hidden layers in a feed-forward artificial neural network because the cortex has strong recurrent projections (i.e., a group of neurons project to themselves) and one area of the cortex is typically connected with multiple areas. Also, we only have limited data for a systematic, quantitative comparison of the number of cortical hidden layers across different animal species. Nevertheless, it is clear that the human brain has a particularly large number of cortical associational areas. For example, in the ventral visual stream (the pathway from the primary visual cortex to the hippocampus), the number of intervening cortical areas is way larger in humans than in mice, tree shrews, gray squirrels, and monkeys.
Consider the following study, which found abstraction-related neural activity in the human hippocampus. As part of a procedure to localize the source of epileptic seizures, neural activity was recorded from the hippocampus and surrounding areas (medial temporal lobe) of human epileptic patients. Neurons in the medial temporal lobe of these patients showed diverse activity patterns in response to diverse visual stimuli. Surprisingly, some of the neurons were only responsive to specific persons or objects. For example, the neuron shown in fig. 11.3 was responsive to the pictures of Jackie Chan, an actor, but not to the pictures of other people.
FIGURE 11.3. A human hippocampal neuron that is responsive to the actor Jackie Chan but not to another actor, Luciano Castro. Middle row, spike raster plot (each dot is a spike and each row is a trial). Bottom row, average response. Figure reproduced from Hernan G. Rey et al., 'Single Neuron Coding of Identity in the Human Hippocampal Formation,' Current Biology 30, no. 6 (March 2020): 1153 (CC BY).
Remarkably, this neuron was responsive not only to the pictures of him but also to his written and spoken names. This indicates that these person-selective neurons are not simply responsive to visual features. Rather, they are responsive to a personal identity, which is an abstract concept. These cells were therefore named concept cells.
Attempts to find similar concept cells in animals have not been successful. For example, hippocampal neurons in monkeys were independently activated by pictures and voices of other monkeys in the colony. This suggests that the human hippocampus may handle particularly high-level abstract concepts, although other interpretations (e.g., abstract identity coding may not be particularly advantageous for monkeys that do not form such large social communities as humans do) are equally plausible. Notably, the prefrontal cortex and hippocampus, which are the highest-order association cortices, have direct communication channels. The hippocampus sends direct projections to the prefrontal cortex, and conversely, the prefrontal cortex sends direct projections to the hippocampus. Of course, they are also connected indirectly via other brain regions such as the precuneus. It is entirely possible that the hippocampus and prefrontal cortex interact closely via these direct and indirect pathways and that such hippocampal-prefrontal cortical interactions underpin unbounded imagination using high-level abstract concepts in humans.
DEEP ARTIFICIAL NEURAL NETWORKS
If the depth of a neural network in the brain determines the level of abstract thinking, what would abstract concepts of very deep artificial neural networks be like? Nowadays, the depth of artificial neural networks routinely exceeds 100. Powerful graphics processing units, originally designed to meet the demands of modern computer games, facilitate the implementation of very deep learning algorithms. Will these deep artificial neural networks be able to form higher-level abstract concepts than human brains? Did AlphaGo, the first computer program to defeat a top-tier human Go master, prevail by using high-level abstract concepts beyond human comprehension?
It has been proposed that the human brain will reach its maximum processing capacity at approximately 3,500 cm³, which is approximately 2.5 times larger than the current one (1,350 cm³). Because of energetic and neural processing constraints, a larger brain would be less efficient. It requires more energy and powerful cooling, which would increase the volume used by the blood vessels. Furthermore, neurons have inherent limits in signal processing speed, which constrains the efficiency of information processing that a brain can achieve by increasing its size. If this estimate is correct, the human brain has room to evolve into a deeper neural network. If someone is born with a deeper network structure due to a mutation, how would they think? Would their artistic sense differ from ours? If we manipulate genes so that a dog is born with neural network connections as deep as in humans, would the dog have the same capacity for abstract thinking as humans? We currently have no clear answers to these questions. However, we are not without research along this line.
DEEP REAL NEURAL NETWORKS
As elaborated in chapter 8, the current human brain is the outcome of the sudden expansion of the neocortex in the course of evolution. This is most likely due to mutations in some of the genes that control neocortex development. ARHGAP11B is one of the genes critical for the development of the human neocortex. It plays an essential role in the proliferation of neural stem cells during early development. In other words, it promotes the production of neurons for the neocortex. What would happen if this gene were expressed in an animal?
Thus far, scientists have expressed this gene in the mouse, ferret, and marmoset. Its expression in mice increased neocortical size as well as memory flexibility. ARHGAP11B-expressing mice and control mice performed similarly in remembering a fixed rewarding location. However, ARHGAP11B-expressing mice outperformed control mice when the rewarding location changed daily, indicating enhanced behavioral flexibility. Remember from chapter 10 that people with prefrontal cortex damage have poor behavioral flexibility.
What about the marmoset, which is a small monkey? Would ARHGAP11B expression make the marmoset smarter? Would it allow the marmoset to think using high-level abstract concepts? As you might expect, these studies are not free from ethical issues. That's why the German and Japanese scientists who performed this work halted the project before the birth of the experimental marmoset. They removed the mutant embryos surgically and examined their brains. It is therefore unknown whether these marmosets would 'think' differently from ordinary ones. However, it was found that the gene expression did alter the development of the marmoset brain. Their neocortex was visibly enlarged and wrinkled (see fig. 11.4).
FIGURE 11.4. ARHGAP11B expression in the marmoset brain. (Left) A common marmoset. Panel reproduced from 'Common Marmoset (Callithrix jacchus),' NatureRules1 Wiki, accessed August 28, 2022, https://naturerules1.fandom.com/wiki/Common_Marmoset (CC BY-SA). (Right top) The brain of a normal marmoset fetus. (Right bottom) The brain of a marmoset expressing human ARHGAP11B. Panels reproduced with permission from Michael Heide et al., 'Human-Specific ARHGAP11B Increases Size and Folding of Primate Neocortex in the Fetal Marmoset,' Science 369, no. 6503 (July 2020): 547.
The top of the human brain has many folds, which increase the surface area of the neocortex. This appears to be a method of harnessing an enlarged neocortex within a limited space inside the skull. By contrast, normal marmosets' brains have a smooth surface. That the altered marmoset fetuses have an enlarged neocortex with some wrinkles on the surface suggests that their 'thinking,' provided they were born and survived until adulthood, may differ considerably from that of normal marmosets. Of course, the brains of these mutant marmosets are still far from the complexity of the human brain. Hence, their abstract-thinking capability is unlikely to have approached that of humans. Nonetheless, with the caveat of ethical issues, this line of research may one day come up with an animal brain that is as complex as, or even more complex than, the human brain.
To summarize, the neural basis of human high-level abstract thinking is unclear. However, studies directly investigating these issues may become popular in the future, though they may be accompanied by ethical concerns.