ActMem: Bridging the Gap Between Memory Retrieval and Reasoning in LLM Agents

Memory management is a core capability for intelligent agents in long-term interactions Zhang et al. (2025c); Hu et al. (2025). Existing studies mainly fall into three groups: retrieval-centric memory, structured memory, and lightweight memory systems.

Synthetic benchmarks are widely used to evaluate LLMs in controlled settings. They have moved from simple information retention to more complex reasoning. Recent datasets like RULER Hsieh et al. (2024) and BABILong Kuratov et al. (2024) test variable tracking and logical chaining. Datasets like LoCoMo Maharana et al. (2024) and LongMemEval Wu et al. (2025) focus on long-term dialogue consistency and memory retrieval. However, these datasets are mostly QA-oriented. They test passive retrieval more than active use of memory for decision-making. We address this issue with ActMemEval. It evaluates the utility of memory for action via implicit constraints and causal conflicts. We use human verification to improve reliability.

Raw dialogue history often contains redundancy and noise that hinder effective reasoning. To address this, we first compress the raw history $\mathcal{D}$ into a set of atomic memory facts $\mathcal{F}$. We use an LLM-based information extractor to parse the dialogue and generate declarative sentences containing key entities and facts following Fang et al. (2026). During extraction, we resolve pronouns in person or place mentions to explicit entities, and normalize relative time expressions into absolute timestamps based on the dialogue time. We define an extraction function $f_{\text{ext}}(\cdot)$ powered by an LLM to map raw dialogues into a set of atomic facts $\mathcal{F}$. For each dialogue turn $(u_{t},a_{t})$, we extract a subset of facts:

where $p_{\text{ext}}$ denotes the extraction prompt designed to identify entities and events. The global fact set is the union of all turns: $\mathcal{F}=\bigcup_{t=1}^{N}\mathcal{F}_{t}$. Unlike traditional summarization which compresses the entire context into a single paragraph, our method focuses on atomicity. This granular extraction serves as the foundation for subsequent causal mining.

To model long-term interactions, we need to find relations between facts from different sessions. Besides, mining causal edges by computing relations across all pairs of facts in the global history is time-consuming. To mitigate this, we introduce a fact clustering module. Let $\phi(\cdot)$ be an embedding function, e.g., Qwen3-Embedding-8B Zhang et al. (2025b), that maps a textual fact $f_{i}$ to a $d$-dimensional vector $\mathbf{h}_{i}\in\mathbb{R}^{d}$. We compute semantic similarity with cosine similarity:

To partition $\mathcal{F}$ into disjoint clusters $\mathcal{C}=\{C_{1},\dots,C_{K}\}$ efficiently, we employ a single-pass incremental clustering algorithm. We maintain a centroid vector $\mathbf{c}_{k}$ for each cluster. For each incoming fact $f_{i}$, we calculate its similarity with existing centroids. If the maximum similarity exceeds a clustering threshold $\delta$, the fact is assigned to the corresponding cluster; otherwise, a new cluster is initialized with $f_{i}$ as its centroid. This strategy ensures that facts related to the same topic are consolidated dynamically. It lets us mine causal relations inside local clusters, which reduces computation while keeping topical coherence.

The core of ActMem is the construction of the memory KG, denoted as $\mathcal{G}=(\mathcal{V},\mathcal{E})$, where nodes $\mathcal{V}$ correspond to memory facts and edges $\mathcal{E}$ represent causal or semantic relations.

Traditional RAG systems retrieve memories solely based on the semantic similarity to the user’s current query $q$, often missing implicit constraints. We propose a retrieval-reasoning-refinement loop to utilize the memory KG effectively.

Figure 3 shows the construction pipeline, which has three steps. To avoid model self-bias and samples that are easy for the same model to solve, we use different models for data generation and validation. The generation model is also stronger than the model used in our method, which helps improve data quality. Specifically, we use Gemini 3.1 Pro for generation and Claude Sonnet 4.5 for validation. To improve diversity, we synthesize six categories: safety-health risks, feasibility limits, time-space-procedure mismatches, access or availability gaps, preference incompatibility, and benefit-reuse opportunities. Detailed definitions are in Appendix E.

We evaluate the dataset from two aspects: context structure and query-history similarity. We compare our ActMemEval with the retrieval-based benchmarks LongMemEval Wu et al. (2025) and LoCoMo. Table 1 reports the basic statistics, and Figure 4 shows the similarity distribution between each question and its supporting history context.

Table 1 shows that ActMemEval is structurally denser. It has the largest average number of sessions and much longer contexts. Its evidence sessions are more numerous and appear later in the dialogue. Compared with LongMemEval and LoCoMo, ActMemEval spreads supporting clues across more sessions.

Figure 4 compares semantic similarity distributions. ActMemEval has the lowest average similarity (0.232) and LongMemEval is highest at 0.347. The ActMemEval curve is shifted left, which means the question is often semantically far from the key history. In other words, the evidence cannot be found by simple similarity retrieval. Thus, ActMemEval is less retrieval-oriented and better tests memory reasoning.

We compare ActMem with six representative baselines: NaiveRAG, MemoryBank Zhong et al. (2024), Mem0 Chhikara et al. (2025), A-Mem Xu et al. (2025), SimpleMem Liu et al. (2026), and LightMem Fang et al. (2026). Detailed descriptions are provided in Appendix A.

We employ two key metrics to assess both the precision of recall and the effectiveness of reasoning:

• •

  Retrieval accuracy: It tests whether the agent can recall the memory facts needed to answer the query. Following Chen et al. (2025), retrieval accuracy is the recall rate of ground-truth evidence facts in our dataset.

• •

  Question-answering accuracy: This metric measures the final utility of the memory. It assesses whether the generated response correctly addresses the user intent while adhering to historical constraints.

We employ two LLMs as backbones for reasoning and generation: DeepSeek-V3, representing high-performance open-source models, and GPT-4o-mini, representing efficient lightweight models. For memory representation, we utilize Qwen3-Embedding-8B Zhang et al. (2025b) to generate semantic vectors. For hyperparameter settings, in the event clustering module, we set the clustering threshold $\delta$ to 0.2. The semantic-edge threshold $\tau$ is set to 0.3. For causal edge construction, the PMI validation threshold $\gamma$ is empirically set to 0.2. During the retrieval phase, we set the number of retrieved memory facts to 20 for the initial retrieval step. The number of counterfactual facts retrieved is set to 10. In the refinement step, we dynamically retrieve additional facts based on the generated KG. For the baseline configuration, we preserve each method’s original final QA prompt and make minimal incremental modifications. Specifically, we append brief instructions on reasoning to better adapt the baselines to our dataset. This setting keeps the original prompting design intact while making the required reasoning process more explicit. We use DeepSeek-V3 as the judge to evaluate retrieval results and generated responses, and we further verify its judgments through an agreement check with human annotation; see Appendix F.

Table 2 presents the results on ActMemEval. Although NaiveRAG achieves the highest retrieval accuracy by recalling extensive surface-level semantic matches, its inferior QA performance indicates a failure to synthesize these facts with noise to deduce the implicit logical constraints required for QA. Compared to LightMem, ActMem achieves substantial improvements. Specifically, when using DeepSeek-V3, it substantially improves QA accuracy over LightMem while also delivering consistently stronger retrieval quality across different categories. A similar trend is observed with GPT-4o-mini. In contrast, retrieval-based baselines exhibit relatively poor performance. We attribute their failure to the reasoning gap. These methods rely primarily on surface-level semantic matching, which struggles to retrieve historical events that are logically related but semantically distant from the current query. For structured memory baselines, A-Mem underperforms ActMem because it uses the graph merely as a retrieval aid, while ActMem employs it as a reasoning framework with explicit causal and semantic edges. In summary, the ability of ActMem to “reason before retrieving” allows it to identify implicit constraints that other methods miss, thereby achieving state-of-the-art performance in complex, action-oriented scenarios. A qualitative case study is provided in Appendix B.

We also observe an interesting gap across different LLM backbones. GPT-4o-mini attains high retrieval accuracy, yet its QA accuracy is markedly lower. In contrast, DeepSeek-V3 delivers balanced and consistently superior results on both metrics. We attribute this discrepancy to a reasoning bottleneck inherent in lightweight LLMs. The lower QA accuracy suggests that GPT-4o-mini struggles to integrate and reason over these memory clues. This phenomenon also reflects the difficulty of ActMemEval that requires reasoning. With stronger reasoning ability, DeepSeek-V3 can use the retrieved context to resolve conflicts, while weaker LLMs may still fail even with correct evidence.

To verify the contribution of each module, we design three variants by removing key modules:

• •

  w/o Reasoning: This variant removes the counterfactual reasoning module. The agent retrieves memory facts directly from the memory KG without intermediate reasoning.

• •

  w/o Causal: It constructs the memory KG excluding causal edges. The graph uses only semantic similarity to connect nodes.

• •

  w/o Semantic: This variant constructs the memory KG excluding semantic edges. The graph consists only of extracted causal chains, making it a sparse causal network.

Figure 5 presents the ablation results. First, removing the reasoning module (w/o Reasoning) with DeepSeek-V3 causes the largest drop in retrieval accuracy (about 9%), confirming our core motivation. In complex scenarios, relevant memories (e.g., “owning a puppy”) may be semantically distant from the query (e.g., “buying plants”). Without counterfactual reasoning and commonsense completion, the agent fails to uncover such “hidden” evidence. Second, removing causal edges (w/o Causal) yields the lowest QA accuracy. Even when some facts are retrieved, the absence of explicit causal links prevents the agent from forming coherent reasoning chains, leading to generic rather than conflict-aware decisions. We observe that when using GPT-4o-mini, removing causal edges actually yields better retrieval performance than the full model. We attribute this to the weaker reasoning capability of GPT-4o-mini, which may introduce errors during world-knowledge completion, adding more noise and degrading the retrieval quality of the full model. Third, the w/o Semantic variant also reduces retrieval accuracy, indicating that semantic edges serve as crucial connectors. They ensure that topic-related but non-causal facts remain reachable, preserving the completeness of the retrieved context. In summary, causal edges provide logical backbone, semantic edges ensure coverage, and the reasoning module guides inference. Their combination gives ActMem strong performance. Additional analyses of PMI validation and semantic threshold sensitivity are in Appendices C and D, respectively.

To test the generalization ability of ActMem, we run additional experiments on LongMemEval Wu et al. (2025) (LongMemEval-S) and LoCoMo Maharana et al. (2024). For fairness, we use GPT-4o-mini as the backbone LLM for all methods and keep the evaluation setting consistent. ActMem uses the same parameters on all three datasets, without dataset-specific tuning. As shown in Figure 6(a), ActMem achieves the best QA accuracy on LongMemEval. It outperforms the strongest baseline LightMem, and also exceeds NativeRAG and A-Mem. Figure 6(b) shows a similar trend on LoCoMo. ActMem reaches 83.57%, again ranking first among all methods. These consistent gains on both datasets indicate that our structured memory KG is also effective in conventional retrieval-oriented settings. By organizing scattered memories into connected fact structures, ActMem reduces retrieval noise and helps the model produce more accurate final answers.

Figure 7 compares efficiency and QA accuracy from multiple views. Subfigure (a) shows that heavier frameworks such as A-Mem and MemoryBank have much higher construction cost, but still do not achieve competitive performance. In contrast, ActMem achieves the best accuracy with moderate extra overhead. Although it needs more construction time than lightweight methods such as LightMem and SimpleMem, it is still much more efficient than the most expensive baselines. The second and third subfigures show the same trade-off in token usage. ActMem uses more input tokens than compression-based methods because building the causal graph needs more context. However, this extra cost brings clear gains in answer quality. At the same time, its output token usage stays close to LightMem and is much lower than SimpleMem. This suggests that the gain does not come from longer responses. Overall, ActMem gives a good balance between reasoning performance and computational cost. It adds moderate overhead for causal and semantic reasoning, and converts that cost into strong performance.