Evolving AI: Embracing Prospective Learning

InfoThis is a summary of the following YouTube video:

Joshua Vogelstein It's about time: learning in a dynamic world - IPAM at UCLA

Institute for Pure & Applied Mathematics (IPAM)

Nov 22, 2024

·

Science & Technology

Prospective learning adapts AI to dynamic changes

  1. Joshua Vogelstein discusses the concept of learning from both a neuroscientific and artificial intelligence perspective, highlighting the differences between biological and AI learning models.
  2. He introduces the idea of prospective learning, which allows the optimal hypothesis to change over time, contrasting with traditional PAC learning that assumes a fixed hypothesis.
  3. Vogelstein uses examples from nature, such as single-cell organisms and squirrels, to illustrate different learning processes, including evolutionary learning and imitation learning.
  4. He describes a video experiment where humans attempt to ride a bike with reversed handlebars, demonstrating reversal learning, a type of learning where previously learned skills must be adapted to new conditions.
  5. The talk emphasizes that learning is a universal trait across species, defined as the ability to use past experiences to improve future performance.
  6. Vogelstein argues that current AI models, which focus on retrospective learning, are inadequate for dynamic environments and proposes enhancements to empirical risk minimization to address this.
  7. Numerical experiments show that prospective learners outperform retrospective ones in tasks involving synthetic and visual recognition, suggesting potential improvements in AI problem-solving capabilities.

Prospective learning adapts to dynamic data

  1. The concept of learning is based on the idea that the past is related to the future in useful ways. If the future were completely different from the past, learning would be ineffective. This relationship is why learning has evolved across various organisms and mechanisms.
  2. In the context of artificial intelligence (AI), learning is modeled rather than directly replicated from biological phenomena. There is a distinction between actual biological learning and the formal models used in AI, which have been developed over the past century.
  3. Several formal notions of learning have been defined, including probably approximately correct (PAC) learning, reinforcement learning, and online learning. PAC learning is a foundational concept in AI, focusing on the idea that models can be approximately correct given enough data.
  4. A famous quote by George Box, "All models are wrong, but some are useful," highlights the idea that while models may not be perfect, they can still provide valuable insights. This is relevant in the context of learning models in AI.
  5. The historical development of learning models can be traced back to the 1930s with work by Italian researchers Gaveno and Celli. Their work laid the groundwork for later developments in machine learning and AI, including the influential work of Vapnik.
  6. The theorem from the 1930s by Gaveno and Celli states that with enough data, any distribution can be estimated arbitrarily well. This concept was foundational for later developments in AI, particularly in the area of PAC learning.
  7. PAC learning assumes that data are independent and identically distributed (IID) and that goals are fixed. However, these assumptions do not hold true in the real world, where data distributions and goals change over time.
  8. The limitations of PAC learning highlight the need for models that can adapt to changing data and goals. This has led to the development of prospective learning, which allows the optimal hypothesis to change over time, addressing the dynamic nature of real-world applications.

Prospective learning adapts AI to dynamic changes

  1. The text discusses the importance of adapting AI to dynamic changes over time, using the example of flu prediction by the CDC and Google. The CDC uses a simple method based on last week's data, while Google attempted a more complex AI approach that failed due to not accounting for time changes.
  2. Google's AI model, despite having vast data and computational resources, failed in flu prediction because it did not incorporate the dynamic nature of time, leading to significant errors. This highlights the need for AI models to adapt to changing conditions over time.
  3. The speaker argues that the failure of Google's AI in flu prediction is a result of not considering temporal changes, which is crucial for accurate predictions in real-world applications.
  4. The text also touches on the role of statistical learning theory in the development of deep learning, suggesting that early deep learning advancements relied on demonstrating capabilities beyond traditional statistical methods.
  5. The speaker emphasizes the importance of understanding intelligence and learning as fundamental aspects of human nature, suggesting that learning is a core property of thinking and essential for understanding ourselves.
  6. The text references a book by Leslie Valiant, published in 2013, which may relate to the discussion of learning and intelligence, although specific details about the book's content are not provided.

Dynamic learning adapts over time

  1. Joshua Vogelstein challenges the traditional AI learning theory known as Probably Approximately Correct (PAC) learning, which lacks a temporal component. He argues that this theory, which has been influential in AI, is incorrect in describing how learning occurs in the natural world.
  2. Vogelstein proposes a modification to PAC learning, introducing the concept of 'prospective learning' that incorporates time as a factor. This approach allows the optimal hypothesis to change over time, unlike the static nature of traditional PAC learning.
  3. The traditional PAC model assumes data pairs are identically and independently distributed (IID), but Vogelstein suggests that in reality, data is sequential and not IID. This shift requires viewing data as a stochastic process rather than a random variable.
  4. In machine learning, a hypothesis is a function that predicts an output from an input. Vogelstein modifies this by including a timestamp with the input, allowing the hypothesis to account for when the input occurred, thus integrating time into the learning process.
  5. Instead of producing a single fixed hypothesis, Vogelstein's approach outputs an infinite sequence of hypotheses. This sequence forecasts possible future outcomes, adapting to changes over time and providing a more dynamic learning model.
  6. The goal in PAC learning is to minimize loss by finding the best hypothesis. Vogelstein's prospective learning aims to output a sequence of hypotheses that can adapt over time, offering a more flexible and accurate model for real-world applications.

Prospective learning adapts AI to dynamic data

  1. The text discusses minimizing errors between predictions and truth over time, integrating future data, similar to reinforcement learning but simpler.
  2. The fundamental theorem of pattern recognition states that with enough data, a learner can select a hypothesis that is probably approximately correct.
  3. The paper introduces 'prospective learning,' which adapts the theorem to consider future data, allowing hypotheses to remain correct over time.
  4. Prospective learning is limited to certain stochastic processes, excluding completely random ones, but is still broadly applicable.
  5. The method involves encoding time into neural networks, using techniques like time encoding and concatenation with inputs.
  6. Empirical experiments show that the placement of time encoding in neural networks affects performance, highlighting its importance.
  7. The approach is inspired by early attention models, such as the Transformer, which encode time to enhance learning.

Prospective learning adapts to dynamic tasks

  1. The text discusses a dynamic learning scenario where tasks switch every 20 samples between two tasks: Task A and Task B. Task A involves identifying positive numbers as gray and negative numbers as black, while Task B is a sign flip of Task A.
  2. The challenge presented is that existing AI algorithms struggle with this dynamic task switching because they are not designed to anticipate changes over time. They perform well on one task but fail when the task switches, as they are not aware of the change.
  3. Several algorithms are compared, including Bayesian gradient descent, online stochastic gradient descent (SGD), and follow the leader. These algorithms attempt to learn which task they are in but fail to adapt quickly when the task switches, resulting in high error rates initially after each switch.
  4. The proposed solution, called prospective learning, incorporates time into the learning process, allowing the algorithm to predict when a task switch will occur. This enables the algorithm to adapt before the switch happens, reducing errors significantly.
  5. Prospective learning is contrasted with traditional temporal difference learning, which is retrospective and only adapts after an error is made. By predicting future changes, prospective learning minimizes errors over time.
  6. The text argues that the failure of modern AI to incorporate time is a fundamental flaw, similar to the XOR problem for independent and identically distributed (IID) data. The solution is simple: integrate time into the learning model, akin to adding a layer to a perceptron.
  7. The effectiveness of prospective learning is demonstrated through experiments, showing that after an initial learning period, the algorithm can predict task switches and maintain low error rates, unlike other algorithms that continue to struggle with task switching.

Prospective learning adapts to dynamic environments

  1. The discussion begins with an analogy about a fox learning to avoid a lion at a watering hole, illustrating how animals adapt to temporal patterns in their environment. This example highlights the importance of learning from time-based experiences to avoid danger.
  2. The speaker mentions designing animal experiments to study how creatures learn about time and adapt their behavior accordingly. This reflects the broader theme of understanding dynamic environments and the role of time in learning processes.
  3. A question is raised about using analysis of variance to detect underlying variables in changing environments. The speaker suggests that variance can be a useful indicator, but emphasizes that their approach relies on simple input-output pairs to learn effectively.
  4. The conversation shifts to traffic patterns as an example of learning based on time. The speaker explains that while time-based learning can predict peak traffic hours, it may not account for unexpected events like a football game, which can alter traffic patterns.
  5. The speaker asserts that their theoretical framework, which includes time as a factor, converges to an optimal solution regardless of the underlying distribution. This suggests that incorporating time into learning models can enhance adaptability to changing conditions.
  6. A concern is raised about the limitations of the theory if the underlying stochastic process changes unpredictably. The speaker acknowledges this challenge, noting that the field of online learning often assumes uncertainty about future events.
  7. The discussion concludes with the idea that the best strategy in unpredictable environments is to rely on past successful actions, though this is seen as a limited approach. The conversation underscores the need for models that can adapt to both predictable and unpredictable changes.

Prospective learning adapts to dynamic data

  1. The discussion begins with the challenge of predicting future data distributions, emphasizing that having partial knowledge is more realistic than having complete or no knowledge. This partial knowledge allows for leveraging time-dependent optimal solutions, making the problem complex.
  2. Existing methods in AI often fail because they do not incorporate time as an input, relying solely on data without temporal context. This omission limits their ability to learn from time-dependent changes.
  3. A simple one-dimensional simulation is used to illustrate the concept, where traditional algorithms perform at chance levels, while the proposed method shows significant improvement, even though MNIST and CIFAR datasets are not truly real-world data.
  4. The datasets are manipulated to include a temporal element by dividing tasks into classes that switch periodically, simulating a lifelong learning scenario. This setup allows for predictable task switching after enough samples are collected.
  5. To increase complexity, a non-deterministic task switching model is introduced using a non-Markov chain, where tasks switch probabilistically. This model challenges traditional methods like Hidden Markov Models (HMM) and Kalman filters, which struggle without deterministic task sequences.
  6. The four-task simulation demonstrates the proposed method's ability to handle non-deterministic task sequences, with tasks defined by simple quadrant-based rules, showcasing the method's adaptability to dynamic environments.

Prospective learning adapts to dynamic tasks

  1. The concept of design in learning involves understanding which quadrant and task you are in, as these can change dynamically. For example, a quadrant may be red in one task and blue in another, indicating the need for adaptable hypotheses.
  2. In dynamic tasks, there is no fixed hypothesis that works universally; adaptation is crucial for success. The task's nature dictates the necessary changes in approach to achieve optimal results.
  3. The discussion highlights the randomness in task switching, with a prior that influences the starting point, though specifics are not detailed.
  4. Prospective learning, after several thousand steps, can converge to a base optimal solution, outperforming other methods that do not adapt well to dynamic changes.
  5. Experiments with MNIST and CIFAR datasets show that prospective learning significantly outperforms other algorithms, which only achieve chance-level performance in dynamic test environments.
  6. The speaker explores the potential of large language models (LLMs) like Llama and Gamma to perform prospective learning, but finds them lacking in adaptability and performance compared to human-like inference.
  7. An experiment with LLMs using prompts based on Bernoulli distributions reveals their inability to accurately predict outcomes, highlighting their limitations in prospective learning tasks.
  8. The LLMs tested, such as Llama 7B and Gemma 7B, fail to learn effectively from given data, unlike more recent models that can use code to improve accuracy.
  9. The simplest inference tasks, like biased coin flips, are challenging for LLMs, which perform poorly compared to human capabilities, indicating a gap in their learning adaptability.

Prospective learning enhances AI adaptability

  1. The speaker argues that large language models (LLMs) cannot perform inference, reasoning, or causal inference, dismissing claims of them having AGI capabilities. They emphasize that LLMs cannot even perform simple tasks like flipping a coin, which is fundamental to more complex reasoning tasks.
  2. A discussion ensues about setting up a biased coin flip experiment with LLMs, highlighting the challenges in prompting these models effectively. The speaker notes that despite various attempts, including different tokenization strategies, LLMs failed to perform the task, suggesting that the issue might be with the prompts used.
  3. The speaker suggests that LLMs are better with language-based tasks rather than numerical ones, as their training data is primarily text. They experimented with different formats, such as using words instead of numbers, but found no improvement in performance.
  4. There is a suggestion to use a 'chain of thought' approach, where LLMs are shown examples of how to perform a task before attempting it themselves. The speaker is skeptical but open to trying this method, acknowledging that trivial prompts have not worked so far.
  5. The speaker distinguishes between inference and action, noting that LLMs are limited to predicting the next token without affecting the world or themselves. They express interest in exploring how prospective learning can be applied to scenarios where actions are involved.
  6. The concept of 'prospective foraging' is introduced as a new problem designed to explore prospective learning. This involves a sequential search for general resources, akin to how foraging societies operate, and is presented as preliminary work to understand how agents can learn and adapt over time.

Dynamic resource management optimizes learning

  1. The text discusses a dynamic resource management scenario using OpenAI Gym's Frozen Lake environment, where resources appear and decay over time at different locations, labeled A and B.
  2. In this setup, resources at location A initially have value but decay over time, disappearing by time step 10, while resources appear at location B at time step 10 and decay until time step 20, after which they reappear at A.
  3. The optimal strategy involves moving between locations A and B to maximize resource collection, specifically timing movements to coincide with peak resource availability at each location.
  4. Traditional foraging theory assumes static resources, leading to suboptimal strategies when resources change over time, as demonstrated by the marginal value theorem, which suggests leaving a patch when average resources elsewhere exceed current patch resources.
  5. The text critiques traditional foraging theory for its static assumptions and highlights the need for strategies that account for dynamic changes, proposing a straightforward optimal strategy of moving to the next resource location before resources peak.
  6. Reinforcement learning algorithms, particularly the advantage actor-critic method, are applied to solve this dynamic foraging problem, adapting the scenario to allow multiple episodes to learn optimal strategies over time.
  7. The performance of these algorithms is evaluated based on the number of episodes, demonstrating the effectiveness of reinforcement learning in adapting to dynamic resource environments.

Prospective learning addresses dynamic AI challenges

  1. The speaker discusses the challenge of achieving optimal performance in dynamic environments, where traditional methods struggle to adapt over time. The blue line represents the optimal performance, while the red line indicates the optimal foraging theory's recommendation, which performs well in certain settings.
  2. In a single long episode, akin to real life without retries, the algorithms fail to learn effectively, achieving only chance-level performance. This highlights the limitations of current methods in adapting to continuous, non-resetting environments.
  3. The speaker notes the absence of results showing that prospective foraging solves these issues, indicating that traditional foraging methods are ineffective for this problem. This suggests a need for new approaches to learning in dynamic settings.
  4. The concept of 'reset-free' or 'single episode' learning is introduced, where learning occurs continuously without resetting to an initial state. This approach is contrasted with multi-episode learning, where updates occur at the end of each episode.
  5. The speaker acknowledges the novelty of reset-free learning and its challenges, comparing it to the early stages of large language models (LLMs). Despite using recent algorithms, they fail to solve the problem, indicating a gap in current methodologies.
  6. The speaker's lab is developing a new model called 'perspective learning,' which aims to better describe biological processes and solve real-world AI problems. However, the model is still in its early stages and requires further development to work at scale.
  7. The speaker emphasizes the need for collaboration to advance perspective learning, inviting those with relevant expertise to contribute. They acknowledge the model's current limitations, being tested only on toy problems, and the lack of real-world examples.

Prospective learning adapts to dynamic data

  1. Joshua Vogelstein from Johns Hopkins University presented a talk titled 'It's about time: learning in a dynamic world' at IPAM's workshop on Modeling Multi-Scale Collective Intelligences.
  2. The presentation was recorded on November 22, 2024, and it focused on the evolving nature of data distribution and goals in real-world applications.
  3. Vogelstein highlighted the limitations of the traditional PAC (probably approximately correct) learning framework, which does not account for changes over time.
  4. He introduced the concept of 'prospective learning,' which allows the optimal hypothesis to change dynamically, unlike the static approach of PAC learning.
  5. The talk included proofs showing that traditional empirical risk minimization fails in certain prospective learning scenarios, while a prospective augmentation can solve these problems.
  6. Numerical experiments demonstrated the effectiveness of prospective learning on tasks using datasets like MNIST and CIFAR, outperforming retrospective methods.
  7. This new framework aims to improve AI solutions for complex problems and better understand naturally intelligent systems.