Memory is Not Search: Toward Proactive, Lifelong Memory in AI

Abstract

Memory is a foundational cognitive function for any artificial or biological intelligent system. Current AI approaches to long-term memory treat it as a problem largely of data storage, summarization, and search. Drawing inspiration from cognitive science and neuroscience, this position paper argues that search is insufficient for memory. Instead, memory requires distinct computational mechanisms that can proactively recall information. We analyze the specific limitations of current AI memory approaches and highlight the core requirements needed to achieve proactive, lifelong memory. Building AI systems with human-level lifelong memory will require new datasets, appropriate evaluation methods and benchmarks, and novel algorithms.

1. Introduction

Memory is inseparable from learning and integral to intelligence [1]. Any system that aspires to emulate or surpass human-like general intelligence must be able to encode and recall memories over a long horizon of experience [2]. To adapt to novel environments, interpret input contextually, and plan for the long term, biological and artificial agents need long-term memory that supports rapid recall, continual updates, multimodal data, veracity (no hallucinations), and contextual relevance.

Current AI has demonstrated enormous success across many general intelligence tasks. Frontier models memorize general knowledge by compacting large amounts of training data into their weights. However, the same models lack long-term memory after deployment. For example, a coding agent working in a specific code base must constantly (re-)remind itself, in each new conversation, about the idioms of that code base (e.g., [3, 4]).

The problem of lifelong memory in AI has been framed as continual learning (persistent updates to a model to adapt it to a new data distribution) [5–8]. When considering in-context information, memory has also been studied under the framework of test-time training (ephemeral updates for one context) [9–11]. Both continual learning and test-time training involve effectively adapting a (pre-trained) model to new test-time context(s). Thus, both frameworks also draw insights from meta-learning, i.e., learning to learn quickly in a new context [12, 10, 13]. Continual learning is an attractive conceptual framework and remains an open research area, especially in relation to frontier models. Test-time training focuses on learning in a context window, which is orders of magnitude smaller than lifelong memory such as long-horizon plans and actions, entire code bases, multimodal data, and personal life logs. Furthermore, current models do not fully utilize even their context windows, showing degraded performance when tasked with retrieving multiple pieces of information, synthesizing dispersed information, performing multi-hop retrieval, and reasoning over long contexts [14–18].

Instead, state-of-the-art methods for long-term AI memory rely on search. A typical approach is retrieval-augmented generation (RAG; [19, 20]), which involves 1) indexing documents using conventional information-retrieval methods (lexical index, (optionally learned) vector embeddings, and/or knowledge graphs) and 2) searching for related documents based on a query or prompt. Another common approach is agentic search, wherein an AI agent answers the prompt by interactively using tools (e.g., web search, app connectors, file system tools, sub-agents (e.g., deep research)) to look for relevant information [21, 22].

RAG and agentic search are both based on search, which we define here as:

In this position paper, we argue that proactive, lifelong memory is fundamentally unlike search. We remember without explicitly querying for information. Instead, the brain spontaneously surfaces memories relevant to the current situation (the ambience). Search starts with a query. Either a human in the loop must provide this query, or the system must know when to search and what to search for; both require the querying agent to already have some memory in the first place. To build AI systems that can emulate and even augment human memory, a paradigm other than search is needed. Memory systems must incorporate the ambience to automatically recall memories from lifelong data, which can be multimodal and up to petabytes in size (Section A). To make progress toward developing novel architectures for memory, it is critical to build new and more appropriate benchmarks and evaluation methods.

Below, we explicate the properties of lifelong memory (the memorome; Section 2), survey current AI memory algorithms and their limitations (Sections 3 & 4), articulate memory as proactive recall (Section 5), identify key gaps in current AI memory benchmarks and architectures and suggest ways to address them (Section 6), and discuss alternative views (Section 7).

2. The Memorome: Lifelong, Dynamic Data up to Multimodal Petabytes

Memory is a broad concept that encompasses working memory, short-term memory, long-term memory, and more. Here, we focus on the lifelong memory of a biological or artificial agent. We define the memorome: Analogous to the genome—the entirety of a person's genetic information—the memorome is the entire collection of memories from an individual lifetime. The memorome includes conversations, pictures, videos, emails, documents, and more. Thus, the memorome is naturally multimodal. In practice, we focus on the digital memorome, excluding touch, smell, etc., as well as never-recorded memories. The memorome is closely allied with the concept of a life log [23–25].

The memorome is individual-specific and excludes world knowledge. If an individual never knew who the prime minister of Cambodia is, this information is not part of that individual's memorome, even though this information is easily accessible on the internet. Conversely, foundational models, by definition trained on general-purpose data, lack individual-specific information, such as a particular conversation instance. The general-purpose knowledge stored in foundation models is distinct from the contents of an individual's memorome.

The size of a memorome varies from individual to individual. We estimate a typical person's memorome to be potentially up to petabytes of data (Appendix A). Even individuals with a small digital footprint have memoromes orders of magnitude larger than the context windows of current foundation models.

A person's memorome spans decades, in contrast with the lifetimes of current AI agents, which typically live from seconds to hours. Future agents will likely live longer and process more data in their lifetime, heightening the need for a memory system that can scale to large memoromes.

The memorome is dynamic and continually updated. Dynamic changes involve not only adding new knowledge and updating stale beliefs, but also reinterpreting and synthesizing accumulated information.

The vast size, time span, and dynamic nature of lifelong memory lead to challenges qualitatively different from typical concerns of current AI memory algorithms, which we review next.

3. Current Approaches to AI Memory Require Search to Scale

3.1 RAG and Agentic Search Attempt to Use Search to Scale to Large Memoromes

Current algorithms for AI memory require search to scale to life-sized memoromes. Consistent with the definition in Section 1, we use 'search-based memory' broadly to refer to retrieval using a query either written by a human or generated by an agent. Practical search-based AI memory algorithms fall into two main clusters: RAG and agentic search.

Retrieval-Augmented Generation (RAG). RAG uses conventional search methods to retrieve information from a database [19, 20, 26, 27]. Retrieval methods include semantic (dense) search, keyword (sparse) search, hybrid search, knowledge graph-based retrieval, and metadata filtering. To improve search efficiency, RAG indexes the database offline before processing the query, limiting the possible interactions between the query and memories; this is only partially mitigated by multi-stage and multi-hop retrieval. The RAG retrieval function can be learned [20, 28–30]. Practical RAG offerings are usually not adapted to the context of an individual user, further limiting their relevance.

Agentic search-based memory. Agentic search-based memory systems mimic how people organize and search for information using tools by using an LLM or an associated controller to decide what to store, summarize, link, retrieve, and update. Agentic search is often combined with a set of notes or a file-system-like organization of memories maintained by the agent [31–33, 3, 4]. Agentic search is costlier and slower than RAG. At the start of each session, the agent has no user-specific information except what is automatically injected into context or later retrieved. Thus, as the scale of the memorome increases, the agent is likely to encounter bottlenecks in how much context can be retrieved, incorporated, and used to dynamically steer the search.

3.2 Current Neural Memory Architectures Do Not Scale to Lifelong Memory

Research has also approached the memory problem with more tightly integrated neural mechanisms. To date, these mechanisms have not been demonstrated or designed to scale to lifelong memories.

Model fine-tuning. Foundation models can be fine-tuned on personal data [34, 35]; a prominent example is digital cloning [36]. Digital clones usually try to mimic the style and behavior of the cloned person. Digital clones do not explicitly solve memory, although they implicitly memorize some information about the cloned person. Model fine-tuning faces challenges in catastrophic forgetting [37] (losing, e.g., instruction following and general knowledge) and hallucinating memories.

In-context memory. The context window can be viewed as test-time memory [38–40], and neural architectures have been proposed to efficiently process long contexts [41, 42, 11, 43, 12, 44–46]. At the scale of current context windows of around 1 M tokens, memory performance remains inadequate when the context is long [14–18]. Further, in-context memory is usually considered ephemeral test-time learning rather than persistent continual learning. As such, this direction of research has not addressed scaling to lifelong memory. Long-context abilities require dedicated training [47, 48, 15], which can also be expensive to scale. Thus, it is unclear whether current approaches to in-context memory can extend to the petabyte scale of human lifelong memory.

4. Limitations of Search-Based Approaches to AI Memory

RAG and agentic search, the two main current approaches to supplement AI with long-term memory, are practical, but they are insufficient to serve as models of proactive memory.

Search requires a query or prompt. RAG and agentic search start with a search query or prompt. Formulating an effective query requires already knowing something about what to search for. This need for foreknowledge poses a bootstrapping problem. Ultimately, the user (or an agent suggesting a query) is left with the key function of formulating a query or prompt. Proactive memory recall adds the challenge of knowing when to search, i.e., whether there is useful information to search for, another piece of information that requires pre-existing memory. The brain spontaneously surfaces useful memories or suggests the need to search for information. Memories that are not spontaneously surfaced or suggested become irretrievable, which we refer to as the dark matter of memory—the memory content still exists, but it can only be triggered serendipitously, with the right cues. Search-based memory systems that rely on the user or an agent to originate a query have no natural mechanism for surfacing this dark matter and will systematically under-access it.

Search struggles to perform associative recall. Search similarity metrics, whether lexical or semantic, are primarily built on culturally shared meaning that does not reflect an individual's idiosyncratic associations. In contrast, much of human memory is associative, linking together pieces of information that share no surface-level similarity—no lexical overlap, no culturally shared semantic relevance, and no entity overlap. For example, if someone once heard a piece of shocking news, they can remember the details of where they were, what they were doing, etc., even though the news and the remembered details have no other culturally shared semantic linkage (flashbulb memory; [49]). Modeling associative recall requires a context-specific, i.e., personalized, model of relevance.

5. Properties of Proactive, Lifelong Memory

Several key properties of human memory can instruct the design of proactive, lifelong AI memory.

Memory is proactive. The human brain surfaces memories spontaneously without external prompting. When someone meets an acquaintance, they (ideally) automatically recall the acquaintance's name, relationship to them, recent interactions, and so on. When an author discusses related work, they (ideally) automatically remember the key papers. People search for information, but only with effort, and we search infrequently compared with how often we proactively remember. Similarly, AI memory should automatically retrieve information instead of chasing information.

While an AI agent can search proactively, coming up with personally relevant queries requires the AI agent to know what information can be looked for and how to look for it, which in turn requires memory. As discussed in Section 4, most memories are lost as dark matter, i.e., information that we once knew and that we could potentially access, but that we are no longer aware of. By definition, memory dark matter is not searchable via easy-to-identify queries. Rather, accessing the dark matter requires a proactive memory system that can infer, from the ambience, cues with personal relevance.

Memories are recalled in real time. The brain recalls memories alongside cognition in real time. This fast memory recall is necessary in many situations. A lawyer conducting a deposition, a surgeon facing a complication, or a politician debating an opponent cannot afford minutes formulating a search prompt, waiting for search results, digesting the hits, and only then continuing their thinking. For a practical AI memory system, a search loop, such as agentic search, that takes tens of seconds is inefficient and even impractical for proactively retrieving memories in real time.

Memories are lifelong. Human memory spans decades. Years after last meeting an old friend, people can often still readily remember what they know about the friend. In addition to the long time span, lifelong memories are large, up to petabytes of data. As AI agents become increasingly longer-lived, they will also need increasingly longer-horizon memory. Besides challenging the in-context capabilities of models (as discussed in Sections 3 & 4), lifelong memory also introduces unique challenges in terms of synthesizing dispersed information and understanding dynamic trajectories across time scales.

Memory involves a specific form of compression. Memory is not lossless storage, nor is memory a summarization. Rather, memories retain rich, abstract, and often-idiosyncratically specific aspects of the original experience. When someone meets an old friend decades later, they may not remember what their friend wore the last time they met, but they may still remember the restaurant they met at. If they revisit the restaurant, they may even recognize the table they sat at. What human memory retains and recalls is highly dependent on a person's existing knowledge and context.

6. The Way Forward: New Datasets, Benchmarks, and Architectures

To build AI with lifelong, proactive memory, a new paradigm of datasets, benchmarks, and architectures is needed to represent and handle large memoromes, realistic data, and proactive recall.

6.1 Benchmarks Need Larger, Realistic Memoromes and Need to Test Proactive Recall

In computer vision, the MNIST dataset [50] helped develop initial algorithms, while later progress relied on the transformative role of ImageNet [51]. Analogously, while existing memory benchmarks provide useful guidance, progress toward real-world memory systems requires more natural, more complex, and much larger datasets.

Benchmarks should test much larger memoromes relevant to lifelong memory. Current benchmarks for AI memory test O(1 M) tokens of context. These memory benchmarks form two clusters. One stems from testing long-context models and includes needle in a haystack [52], multiple needles, multi-hop retrieval on a graph [53], long-context instruction following, and long-context reasoning [18, 54]. Although some of these benchmarks are programmatically extensible, they are typically tested up to 1 M tokens, informed by the context capacity of current models.

The other cluster originates in testing memory for chat assistants and AI agents, including LoCoMo [55], LongMemEval [56], BEAM [57], MEMTRACK [58], MemoryAgentBench [59], and others. These datasets test memory sizes from 10 K tokens to over 1 M tokens, with the recent BEAM dataset extending up to 10 M tokens of context. The sizes of agentic memory benchmarks have increased over the years in concert with increasing model context windows.

Compared to long-context benchmarks, agentic memory benchmarks emphasize temporal grounding, incremental data updates, and a memory persistent across queries and tasks, all properties relevant to lifelong memory. However, the context sizes of current agentic memory benchmarks significantly overlap the context-window sizes of current LLMs. Indeed, in-context memory is a strong baseline on popular agentic memory benchmarks (Table 1; Figure 1). Further, agentic search without tuning nearly matches the performance of specialized memory systems. Some of the top performances reported by specialized memory systems used frontier models (Hindsight and Honcho used Gemini 3(.1) Pro; see full details in Appendix B.1), while others used efficiency-oriented models. Thus, while the benchmarks may be somewhat more discriminative if all existing memory systems employed full models, available results indicate that current benchmarks—in part due to the limited memory size tested—are insufficient for evaluating improvements in frontier long-term memory abilities.

Figure 1: Current memory benchmarks insufficiently distinguish general model capabilities represented by in-context and agentic-search baselines from specialized memory systems. The data correspond to Table 1. The in-context prompt was minimal and not optimized (prompts shown in Appendix D). BEAM-10M was not tested in context due to context-length limits. Error bars indicate standard error across question-answer pairs. For LoCoMo*, see notes in Table 1. Across benchmarks, agentic search and, in one case, in-context memory already match or exceed public specialized memory systems.

Lifelong memory is orders of magnitude larger than the millions of tokens tested in existing memory benchmarks. For example, knowledge workers can exchange 30–50 M tokens per year of email alone. An AI-assisted coder today orchestrates millions of tokens per day, or O(1 B) tokens per year; an organizational context is even larger. An individual's memorome can extend to petabytes of multimodal data (Section 2). Thus, benchmarks relevant to lifelong memory need to test at least 100 M to 1 B tokens per memory context (e.g., one user or one timeline).

Memory datasets need more realistic statistics. Existing memory benchmarks are largely built from synthetic data. While controlled synthetic datasets like MRCR (multi-round co-reference resolution; [69, 70]) are valuable for answering specific questions, synthetic data abstract away important aspects of real data. For example, real data show small-world networks of co-occurring entities (e.g., social networks in the case of people as entities). Real data cover more diverse topics, have messier content (typos, ambiguities, and complex associations), and combine heterogeneous sources. Topics in real data are fluid, blending into each other and evolving over time (e.g., a research idea may spawn several projects over years and evolve all the while). These differences limit the transferability of results from synthetic benchmarks to real-world applications.

To quantify how existing memory benchmark data diverge from data in the wild, we compared descriptive statistics between real personal memoromes and agentic memory benchmark data (methods detailed in Appendix C). In memory benchmarks, each independent context (e.g., a conversation) constitutes a memorome. Real data show a higher proportion of unique people and people mentions, have more diverse and higher-dimensional semantic topics, show less topical clustering, contain small-world co-occurrence graphs, and use longer-tailed vocabulary (Figure 2). These qualitative differences are robust despite significant variations across differently constructed synthetic datasets on the one hand, and heterogeneous data from different people on the other hand. The complexity of real data presents both challenges for search-based memory and opportunities for developing better models of memory.

Real data, or synthetic data that better match real-data statistics, are needed to build memory benchmarks that can guide translatable model improvements. Because personal memory data, such as life logs, contain inherently sensitive information, collecting datasets of real memoromes at scale will be challenging, although there are laudable efforts in this direction (e.g., [25]). Simulated societies of interacting agents have been used to study AI memory (e.g., [31, 71]) and could generate data with more realistic statistics than synthetic conversations. However, data derived from agents with limited memory may be fundamentally less coherent than real data.

Figure 2: Real memorome data differ statistically from current synthetic memory benchmarks. a, b, Proportions of entities in each category (person, product, organization (org), and document), separately for unique entities in each memorome (a) and entity mentions (b). c, Effective dimensionality of semantic embeddings of events extracted from each memorome, quantified by the number of principal components needed to explain 90% of variance. d, Unique people per 1 K extracted events. e, Number of out-of-vocabulary words per 1 K words. f, Number of entity pairs occurring in the same event per 1 K extracted events. g, Index σ [72] quantifying the small-worldness of entity co-occurrence graphs. In panels c–g, each dot corresponds to a memorome, either a personal corpus (real data) or a complete, independent history (synthetic datasets; see Appendix C for details). The dashed vertical line indicates the median across synthetic datasets. Real memoromes differ from synthetic data in entity composition, topic dimensionality, entity & lexical variety, and co-occurrence graph structure.

Benchmarks need to test proactive memory recall. Memory benchmarks almost exclusively test question answering. In contrast, proactive memory does not answer explicit questions but rather surfaces information based on the current holistic context (ambience) to inform the user's next action (Section 5). To test proactive memory recall, benchmarks need to include dynamic contexts as a model input, and evaluations should assess proactively surfaced memories for timeliness, informativeness, abstention, and non-redundancy. The evaluation should be relative: Can a future action be better predicted (supervised labels), or a better future action taken (reward function), with a surfaced memory vs. without it?

Additionally, benchmarks should address actual memory needs instead of artificially generated queries. More work (e.g., [73, 74]) is needed to quantify the actual memory needs of people and AI agents in realistic situations.

6.2 New Models and Architectures Are Required for Proactive, Lifelong Memory

Models need several new abilities beyond current AI memory architectures (Section 3). First, models need to be able to incorporate memoromes (i.e., test-time data) at least as large as 100 M–1 B tokens. Context sizes at this scale are beyond reach for current linear attention, test-time memory, and neural memory approaches. Ultimately, because petabyte-scale lifelong memoromes rival the scale of pretraining data, a promising approach may be to represent memory implicitly in model weights. If so, however, a key challenge is how to efficiently adapt the model weights to a new memorome without cost-prohibitive (pre)training or catastrophic forgetting of general abilities.

Second, models should recall memories contextually given a streaming ambience without requiring explicit queries. Ambient memory recall refers to

where $\boldsymbol{x}_{(t-\tau):t}$ is a (multimodal) ambience time series, $M_t$ is a user-specific, continually updated memory corpus, and $m_t$ is a piece of information grounded in $M_t$. For illustration, $M_t$ could be a set of valid memories, and $m_t \in M_t$; in general, a grounding model $G(m_{t_i}, E_{t_i}, M_t) \to \{0, 1\}$ can be used to determine whether memory $m_{t_i}$ is grounded in evidence $E_{t_i} \subseteq M_t$. Early and recent examples of ambient memory recall systems include [75–82].

The need for contextual recall limits the direct applicability of current search-based memory systems optimized for question answering. The memory model will ideally input multimodal context including vision, audition, location, time, company, etc., all of which are vital to human memory. In contrast, RAG, agentic search, and long-context models focus on content (occasionally with a thin layer of context represented by metadata). Models, like human memory, should be informed by context at multiple time scales.

As models are developed that can represent lifelong memory, it will also be vital to ensure controllability and the right to forget. Users should be able to ask the model to 'forget' specific information. The right to forget will be an especially important algorithmic consideration for implicit (neural) memory architectures. A particularly interesting combination is to use implicit memory for in-context-scale recent recall and an external evidence store for provenance, grounding verification, and user control.

To advance algorithms for proactive, lifelong memory, we first need appropriate benchmarks to measure progress. These benchmarks will also allow for testing our position against alternative views, detailed in the next section.

7. Falsifiability and Alternative Views

Our position makes testable predictions that contrast with alternative views. The first alternative to our position is: Search can solve (proactive, lifelong) memory. In principle, a model can be built that predicts a verbal or latent query based on the input context. The predicted query can then be passed into a question-answering system employing, e.g., RAG or agentic search. Such a system can be made proactive by searching continually, perhaps gated on the context input and the quality of the memory output [31, 71]. The utility of search engines demonstrates that there is no fundamental difficulty in scaling information retrieval to large-scale data. The search-based alternative is thus a feasible baseline deserving rigorous comparison. There could be a continuum of alternatives, including personalized search by fine-tuning embedding models, rerankers, generative retrieval, and retrieval using queries in a learned latent space [83, 84]. Our position predicts that a search-based alternative introduces bottlenecks limiting the quality of queries, the search relevance function, and proactivity. Potential limitations include cases involving vague queries, like tip-of-tongue questions; synthesis of information sparsely distributed over many documents; and inefficiency in determining when to search and what to search for. Circumventing these potential bottlenecks calls for an integrated model that internalizes a memorome to directly map ambience to proactive memories, without explicit search.

A second alternative view is: Current AI architectures can scale to lifelong memories. This alternative view does not necessarily contradict our position that memory is not search, although this view contradicts some of our deductions. Existing work that scales in-context memory uses fixed-size memory representations that are either recurrently accumulated (a latent state matrix (e.g., [44, 85]) or test-time learned (model weights for neural/soft memory; e.g., [12, 46]). In the limit of very large memoromes, finite-sized memories will approximately forget everything, and lifelong memoromes are significantly larger than the context sizes considered in current test-time memory work. Moreover, (meta-)learning to memorize (learn) at test time could require a combinatorial number of training examples for a model to learn to use all parts of the context. Instead, our position is that fundamental architecture changes are needed to add extensible memory capacity, inductive biases for effective generalization to very long contexts, and more efficient test-time learning to feasibly incorporate lifelong memoromes.

A third alternative view concerns our position on the need for a new kind of datasets (Section 6.1). Alternatively: Existing approaches suffice to construct memory datasets useful for training and testing proactive, lifelong memory. This alternative view holds that scaling methods similar to those that constructed MRCR [70] or BEAM [57] can create the necessary datasets. In contrast, our position is that real data, or significantly more realism in synthetic data starting with the statistics we analyzed, are required to evaluate and learn proactive, lifelong memory recall. In particular, the shapes of input data and output behavior need to be fundamentally different and oriented toward ambient intelligence.