Geoffrey Hinton | On working with Ilya, choosing problems, and the power of intuition

Summary

The script features an interview with a renowned scientist discussing his journey from England to Carnegie Mellon, his early disappointment with traditional studies of the brain, and his shift to artificial intelligence (AI). He reflects on his experiences and collaborations in AI research, highlighting the development of neural networks and backpropagation. The discussion covers the evolution of AI, the importance of large models and multimodal data, and the potential societal impacts of AI advancements. The scientist emphasizes the importance of intuition, selecting talented students, and the future direction of AI research, including potential risks and ethical considerations.

Takeaways

• 🧠 The importance of environment: Moving from England to Carnegie Mellon, the speaker noticed a stark difference in work culture and environment, with students working late into the night believing their work would shape the future of computer science.
• 🔬 Disappointment in initial studies: The speaker found the study of the brain at Cambridge disappointing because it only covered basic neuron functions, leading him to switch to philosophy and eventually AI.
• 📘 Influential readings: Books by Donald Hebb and John von Neumann were pivotal in shaping the speaker's understanding and interest in how the brain learns and computes.
• 🤖 Collaborations: Key collaborations with Terry Sejnowski and Peter Brown were significant in the speaker's research, providing valuable insights and advancements in neural networks and speech recognition.
• 👨‍🎓 Notable students: Ilia, a standout student, impressed the speaker with his intuitive understanding of AI and neural networks, leading to impactful collaborations.
• 💡 Scale and innovation: Ilia's belief in the importance of scaling models was initially met with skepticism but proved crucial as larger models demonstrated significant improvements.
• 🧩 Understanding through prediction: The speaker believes that predicting the next symbol in language models forces a deeper understanding of context and reasoning.
• 🌐 Multimodal learning: Integrating multiple forms of data (text, images, video) into AI models will enhance their understanding and reasoning capabilities.
• ⚡ GPU revolution: The transition to using GPUs for training neural networks significantly accelerated AI research and development.
• 🔄 Digital immortality: Digital systems can share weights and knowledge efficiently, unlike human brains, leading to superior learning and knowledge dissemination.

Q & A

What was the speaker's initial impression of the academic environment at Carnegie Mellon compared to England?

The speaker found the environment at Carnegie Mellon to be very different and refreshing compared to England. Students at Carnegie Mellon were working late into the night because they believed their work was shaping the future of computer science, which was a stark contrast to the pub-going culture after 6:00 PM in England.

Why was the speaker disappointed with his initial studies in physiology and philosophy?

The speaker was disappointed because his studies in physiology only taught him about how neurons conduct action potentials, which didn't explain how the brain works as a whole. Similarly, philosophy didn't provide insights into how the mind worked, which was his ultimate interest.

What inspired the speaker to pursue AI research?

The speaker was inspired to pursue AI research after reading books by Donald Hebb and John von Neumann. Hebb's interest in learning connection strengths in neural nets and von Neumann's interest in brain computation intrigued the speaker and led him to Edinburgh to study AI.

How did the speaker's collaboration with Terry Sejnowski come about?

The speaker's collaboration with Terry Sejnowski began when they interacted frequently despite the distance between Pittsburgh and Baltimore. They would take turns visiting each other's city about once a month to work on Boltzmann machines, sharing a conviction that this was how the brain worked.

What was the significance of the speaker's collaboration with Peter Brown?

The collaboration with Peter Brown was significant because Peter, a statistician, taught the speaker about speech recognition and hidden Markov models. This collaboration was fruitful, with the speaker feeling that he learned more from Peter than Peter did from him.

How did Ilya Sutskever's initial interaction with the speaker influence their future collaboration?

Ilya Sutskever's initial interaction with the speaker demonstrated his eagerness and intuition for AI. Despite not understanding the paper on backpropagation initially, Ilya's question about why the gradient wasn't given to a sensible function optimizer showed his deep thinking, which led to a productive collaboration.

What was the speaker's view on the importance of scale in AI models?

The speaker believed that while new ideas like Transformers helped, the real shift in AI performance was due to the scale of data and computation. He mentioned that they didn't anticipate computers becoming a billion times faster, and with larger scale, models could achieve more without needing as many new ideas.

How does the speaker perceive the process of predicting the next word in language models?

The speaker believes that predicting the next word in language models is not just a mechanical process. It requires understanding the context, similar to how humans comprehend and generate language, which involves reasoning.

What role does the speaker see for multimodality in the future of AI models?

The speaker sees multimodality as a significant advancement for AI models. By incorporating images, video, and sound, models will improve in understanding spatial relationships and concepts that are difficult to grasp from language alone.

What was the speaker's intuition about the use of GPUs for training neural networks?

The speaker's intuition about using GPUs for training neural networks was based on their efficiency in performing matrix multiplications, which are fundamental to neural network computations. This led to significant speed improvements in training times.

How does the speaker view the relationship between language and cognition?

The speaker views language as a tool for cognition, where symbols are converted into embeddings that interact to predict subsequent symbols. This process of converting and interacting symbols is seen as central to both understanding and generating language.

What is the speaker's perspective on the potential of AI in healthcare?

The speaker sees AI in healthcare as a promising application, with the potential to significantly increase the availability and quality of medical care. AI could assist or replace doctors, leading to a situation where everyone could have personalized medical attention.

What is the speaker's approach to selecting research problems?

The speaker selects research problems based on intuition and a sense that a widely accepted idea might be wrong. He looks for opportunities to challenge conventional wisdom with simple demonstrations that can show why the prevailing view may not be accurate.

What does the speaker consider as the most promising direction in AI research today?

The speaker believes that training large models on multimodal data is a very promising direction. Even if the models are initially used for simple tasks like predicting the next word, the approach has great potential for future development.

What is the speaker's view on the importance of learning algorithms in achieving human-level intelligence?

The speaker believes that while backpropagation is a fundamentally correct and successful approach for learning, there may be alternative learning algorithms that could also achieve human-level intelligence. However, he acknowledges that backpropagation has proven to be highly effective.

What achievement from the speaker's career is he most proud of?

The speaker is most proud of the development of the learning algorithm for Boltzmann machines. He considers it elegant, even if it may not be practical, and it was a project he greatly enjoyed working on.