AAMAS-2025 Formatting Instructions]{Learning Hierarchical Procedural Memory for LLM Agents through Bayesian Selection and Contrastive Refinement

Saman Forouzandeh, Wei Peng, Parham Moradi, Xinghuo Yu, Mahdi Jalili

Abstract

We present MACLA, a framework that decouples reasoning from learning by maintaining a frozen large language model (LLM) while performing all adaptation in an external hierarchical procedural memory. MACLA extracts reusable procedures from trajectories, tracks reliability via Bayesian posteriors, selects actions through expected-utility scoring, and refines procedures by contrasting successes vs. failures. Across four benchmarks (ALFWorld, WebShop, TravelPlanner, InterCodeSQL), MACLA achieves 78.1% average performance, outperforming all baselines. On ALFWorld unseen tasks, MACLA reaches 90.3% with +3.1% positive generalization. The system constructs memory in 56 seconds (2,800× faster than the state-of-the-art LLM parameter-training baseline), compresses 2,851 trajectories into 187 procedures (15:1). Experimental results demonstrate that structured external memory with Bayesian selection and constrastive refinement enable sample-efficient, interpretable and continually improving agents without LLM parameter updates. Code is publicly available at https://github.com/S-Forouzandeh/MACLA-LLM-Agents-AAMAS-Conference{MACLA}.

Learning Hierarchical Procedural Memory for LLM Agents through Bayesian Selection and Contrastive Refinement

Saman Forouzandeh, Wei Peng, Parham Moradi, Xinghuo Yu, Mahdi Jalili

School of Engineering, Royal Melbourne Institute of Technology University

Melbourne, VIC, Australia its execution) to achieve the goal in the current context, deciding which information to gather, which tools to invoke, and which subroutines to chain in order to achieve completion [25, 26].

Early LLM agents used prompt-based planning [26] and selfcritique [15], but lack persistent 'how-to' procedures - when tasks are similar but not identical, agents must re-plan from scratch, increasing cost and latency. Fine-tuning approaches [2, 27, 29] adapt agents via supervised learning or RLHF, but typically treat entire trajectories as single units weighted by terminal success/failure, neglecting rich intermediate steps. In practice, failed trajectories often contain correct substeps (e.g., 'successfully navigating and retrieving an egg, but failing to boil it' [16]), while successful ones may include suboptimal actions that accidentally cancel out. Recent work [22] addresses this via step-level rewards, but requires repeated policy training on densely-labeled data, incurring substantial computational cost.

We introduce MACLA (Memory-Augmented Contrastive Learning Agent), a framework that disentangles reasoning from learning through the coupling a frozen LLM and a structured external procedural memory (Figure 1). Unlike fine-tuning approaches where reasoning and adaptation are entangled within billions of parameters, MACLA fixes the LLM as a stable semantic reasoner responsible for trajectory segmentation, abstraction, and action generation. All learning occurs externally through explicit, interpretable memory operations - maintaining human-readable procedures, updating Bayesian posteriors, and refining preconditions through contrastive analysis. MACLA operates through three core mechanisms:

(1) Bayesian procedure selection: Maintains Beta posteriors Beta ( 𝛼 𝑖 , 𝛽 𝑖 ) over procedure success rates and ranks candidates via expected-utility scoring that balances contextual relevance, success probability, failure risk, and information gain, providing principled exploration-exploitation. (2) Contrastive refinement: Compares successful and failed execution contexts to tighten preconditions, repair action sequences, and refine postconditions once procedures accumulate sufficient evidence (i.e., ≥ a threshold), progressively improving procedure quality through memory edits rather than gradient updates. (3) Meta-procedural learning: Composesfrequently co-occurring procedures into hierarchical 'playbooks' with conditional control policies (continue, skip, repeat, abort) for long-horizon tasks, enabling strategic reuse beyond atomic skills.

This architecture yields sample-efficient, interpretable agents with human-readable procedural knowledge, closed-form utility computation, and minimal LLM usage. Specifically, this work contributes:

· Online procedural memory adaptation: Continual updates to procedural and meta-procedural memory during and

Abstract.

We present MACLA , a framework that decouples reasoning from learning by maintaining a frozen large language model (LLM) while performing all adaptation in an external hierarchical procedural memory. MACLA extracts reusable procedures from trajectories, tracks reliability via Bayesian posteriors, selects actions through expected-utility scoring, and refines procedures by contrasting successes vs. failures. Across four benchmarks (ALFWorld, WebShop, TravelPlanner, InterCodeSQL), MACLA achieves 78.1% average performance , outperforming all baselines. On ALFWorld unseen tasks, MACLA reaches 90.3% with +3.1% positive generalization . The system constructs memory in 56 seconds (2,800× faster than the state-of-the-art LLM parameter-training baseline), compresses 2,851 trajectories into 187 procedures (15:1). Experimental results demonstrate that structured external memory with Bayesian selection and constrastive refinement enable sample-efficient, interpretable and continually improving agents without LLM parameter updates. Code is publicly available at MACLA.

Key Observations and Architectural Insights

Memory-augmented agents, Procedural memory, Bayesian decision making, Contrastive learning, LLM agents

Introduction

Large language model (LLM) agents can solve complex, interactive tasks such as web shopping [25] and embodied AI housekeeping [9], by transforming natural-language instructions into sequences of environment actions [26]. In these settings, agents navigate step-bystep through partially observable environments to pursue subgoals and ultimately complete the task [9, 22]. The resulting trajectory is the ordered record of an episode's interaction, typically written as ( 𝑇, 𝐴, 𝑂, 𝑅 ) , where 𝑇 represents a task to complete, 𝐴 are actions, 𝑂 stand for observations for the outcome of corresponding actions, and 𝑅 records step-level outcomes or rewards. Trajectories thus capture the full decision process, not merely terminal success or failure, and provide dense supervision for how an agent progresses through a task [16, 22]. When a new task arrives, the agent synthesizes an appropriate trajectory (that is, a step-by-step plan and

Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026), C. Amato, L. Dennis, V. Mascardi, J. Thangarajah (eds.), May 25 - 29, 2026, Paphos, Cyprus . © 2026 International Foundation for Autonomous Agents and Multiagent Systems (www.ifaamas.org). This work is licenced under the Creative Commons Attribution 4.0 International (CC-BY 4.0) licence.

VIACLA: Memory-Augmented Contrastive Learning Agent (Inference |ime Learning)

Figure 1: Comparison between existing LLM-based trajectory learning (top) and the proposed memory-augmented contrastive learning agent (MACLA, bottom). Existing methods train trajectories ( 𝑇, 𝐴, 𝑂, 𝑅 ) (Task, Action, Observation, Reward) into LLM parameters through post-training (finetuning and/or RLHF), whereas MACLA constructs procedural and meta-procedural memory externally through frozen LLM abstraction, segmentation, Bayesian selection, and contrastive refinement. Memories are learned during memory construction. Besides learning during memory construction, MACLA enables inference-time learning in which outputs are verified in the task environment, with feedback used for contrastive refinement on the retrieved memories. Meta-procedural learning enables the composition policy to be learned among procedures.

• after episodes, enabling adaptation without weight updates, compared with offline LLM post-training approaches [17, 22, 29] that remain static at inference. · Reasoning/learning decoupling: A frozen LLM for parsing and abstraction with all improvements occurring in an external, structured procedural memory, avoiding the computational cost and catastrophic forgetting risks of parameter fine-tuning. · Bayesian uncertainty-aware selection: A principled procedure selection module that maintains Beta posteriors over success rates with closed-form expected utility objectives balancing relevance, success probability, failure risk and information gain. · Contrastive procedural refinement: An algorithm leveraging paired successes and failures to tighten preconditions, repair action schemas, and refine postconditions of stored procedures without requiring expert demonstrations. · Hierarchical meta-procedural composition: Automatic discovery and maintenance of conditional playbooks with control policies (skip, repeat, abort) for long-horizon tasks, enabling compositional generalization.

We evaluate MACLA across four benchmarks (ALFWorld [9], WebShop [25], TravelPlanner [21], InterCodeSQL [24]), achieving 78.1% average performance - the highest among all methods, including those using models 10× larger (later in Table 1). On ALFWorld [16], MACLA reaches 87.2% on seen and 90.3% on unseen tasks, with a positive generalization gap (+3.1%) indicating compositional transfer rather than overfitting. The system achieves this with only 0.016 GPU-hours for one-time memory construction - 2,800× faster than the state-of-the-art LLM parameter-training baseline [22], which requires 44.8 GPU-hours of iterative training -while simultaneously producing human-interpretable procedural knowledge.

Related works

LLM agents have advanced rapidly in reasoning and decisionmaking, enabling multi-step interaction in embodied and webbased environments. Early frameworks such as ReAct[26] and Reflexion[14] integrate reasoning and acting within the same loop, while trajectory-tuning methods [2, 22] fine-tune models using expert demonstrations. However, fine-tuning is computationally expensive, requires offline data collection and training cycles, and does not support true online adaptation at inference time. To overcome this issue, a line of research augments LLM agents with memory for continuous reasoning. Memory is a foundational component of language agents, supporting competence across multiple timescales from transient working context to persistent long-term knowledge [6, 8, 31]. Research on memory for LLM agents can be usefully organized along two directions: where memory resides and what is stored. Along the first direction, some methods such as MemGPT [11] and MemoryBank [32], use buffer-based systems to store conversational or episodic traces and retrieve them with embedding search and simple heuristics. Some others, such as HiAgent[5], A-Mem[23], MemAgent [28] use hierarchical designs to separate working buffers from episodic and long-term stores to relieve context pressure and improve persistence. Recently, SAGE [7] used reflective multi-agent controllers to curate these stores while controlling growth. The second direction concerns what is stored. Many systems retain free-form text snippets such as notes, summaries, or dialogue chunks; these are easy to write but suffer from retrieval drift and weak compositionality as repositories scale [11, 32]. More structured artifacts appear as tuples and key-value frames (e.g., tool logs or entity/event graphs), which aid filtering but still lack executable semantics for reuse. A growing line of work targets skills and procedures: agents capture reusable action patterns, tool workflows, and instruction-like steps across related tasks [3, 19, 20]. Memp [4] advances this view by treating procedural memory as a first-class object and studying its construction, retrieval, and update across domains. However, several key limitations remain; (1) it represents know-how largely as monolithic text (scripts or full trajectories) with heuristic retrieval and simple updates; (2) it lacks uncertainty-aware selection or principled exploration-exploitation balance, preventing reason about reliability or risk of retrieved memory; and (3) it lacks a mechanism to refine procedures from paired successes and failures or abstract recurring patterns into meta-procedural compositions. Comparatively, we represent experience as structured, hierarchical procedures with explicit preconditions, action schemas, and postconditions, enabling interpretable reuse and safe composition and direct schema edits when evidence warrants change. The proposed approach enables the system to continuously adapt and improve.

The Preamble

You will be assigned a submission number when you register the abstract of your paper on OpenReview . Include this number in your document using the ' \acmSubmissionID ' command.

Then use the familiar commands to specify the title and authors of your paper in the preamble of the document. The title should be appropriately capitalised (meaning that every 'important' word in the title should start with a capital letter). For the final version of your paper, make sure to specify the affiliation and email address of each author using the appropriate commands. Specify an affiliation and email address separately for each author, even if two authors share the same affiliation. You can specify more than one affiliation for an author by using a separate ' \affiliation ' command for each affiliation.

Provide a short abstract using the ' abstract ' environment.

Finally, specify a small number of keywords characterising your work, using the ' \keywords ' command.

Proposed Method

The key components of MACLA are described in detail below.

LLM-based Procedural Abstraction

The first stage transforms raw episodic trajectories into structured, reusable procedural knowledge. Given a trajectory

𝜏 = {( 𝑜 𝑡 , 𝑎 𝑡 , 𝑟 𝑡 )} 𝑇 𝑡 = 0 consisting of textual observations 𝑜 𝑡 , primitive actions 𝑎 𝑡 , and rewards 𝑟 𝑡 , the frozen LLM L 𝜃 receives the full trajectory and identifies semantically coherent segments that correspond to meaningful sub-tasks:

$$

$$

where each segment 𝑘 spans time steps [ 𝑡 start 𝑘 , 𝑡 end 𝑘 ] and is summarized by a description 𝑑 𝑘 . For each segment, MACLA constructs a structured procedure Proc 𝑘 = ⟨G 𝑘 , Ψ 𝑘 , 𝜋 𝑘 , Φ 𝑘 ⟩ , where G 𝑘 is a natural-language goal, Ψ 𝑘 are precondition patterns inferred from the observations before the segment, 𝜋 𝑘 is an abstracted action sequence, and Φ 𝑘 are postcondition patterns extracted from the final observations. This decomposition produces interpretable 'how-to' skills that can be invoked whenever their preconditions are met. To support retrieval and merging, each procedure is embedded into a semantic vector space using an encoder 𝜙 , e 𝑘 = 𝜙 ([G 𝑘 ; Ψ 𝑘 ; Φ 𝑘 ]) ∈ R 𝑑 . When a new procedure is created, it is compared to existing ones via cosine similarity, 𝑖 ∗ = arg max 𝑖 sim ( e 𝑘 , e 𝑖 ) . If sim ( e 𝑘 , e 𝑖 ∗ ) > 𝜃 dup , the new procedure is merged into the existing one by expanding its condition sets; otherwise, a new entry is added. This process yields a continually growing procedural library M proc = {( Proc 𝑖 , e 𝑖 , 𝛼 𝑖 , 𝛽 𝑖 )} 𝑁 𝑝 𝑖 = 1 that forms the foundation for later Bayesian selection and refinement.

Bayesian Reliability and Utility Selection

Given the procedural library, the agent must decide which procedure to execute for the current observation. Each procedure Proc 𝑖 maintains a Beta posterior over its success probability 𝜌 𝑖 ∈ [ 0 , 1 ] :

$$

$$

where 𝛼 𝑖 and 𝛽 𝑖 accumulate successful and failed executions from history D 𝑖 . The posterior mean E [ 𝜌 𝑖 ] = 𝛼 𝑖 /( 𝛼 𝑖 + 𝛽 𝑖 ) estimates current reliability, while the variance Var [ 𝜌 𝑖 ] = 𝛼 𝑖 𝛽 𝑖 ( 𝛼 𝑖 + 𝛽 𝑖 ) 2 ( 𝛼 𝑖 + 𝛽 𝑖 + 1 ) quantifies epistemic uncertainty. For each candidate, we compute expected utility by integrating over the Beta posterior. Given utility 𝑈 ( 𝜌 | 𝑜 𝑡 , 𝑖 ) = Rel 𝑖 ( 𝑜 𝑡 ) · 𝜌 · 𝑅 max -Risk 𝑖 ( 𝑜 𝑡 ) · ( 1 -𝜌 ) · 𝐶 fail + 𝜆 info · 𝐼 ( 𝜌 ) , the expected utility is:

$$

$$

Exploiting E Beta ( 𝛼,𝛽 ) [ 𝜌 ] = 𝛼 𝛼 + 𝛽 and E [ 1 -𝜌 ] = 𝛽 𝛼 + 𝛽 , this simplifies to:

$$

$$

where Rel 𝑖 ( 𝑜 𝑡 ) = cos ( 𝜙 ( 𝑜 𝑡 ) , e 𝑖 ) is contextual similarity, Risk 𝑖 ( 𝑜 𝑡 ) is the fraction of past failures with similar contexts, and 𝐻 [·] is differential entropy encouraging exploration. The selected procedure

$$

$$

subject to confidence threshold 𝜃 conf . If max 𝑖 EU ( Proc 𝑖 | 𝑜 𝑡 ) < 𝜃 conf , the agent falls back to zero-shot LLM reasoning. This Bayesian selection mechanism balances exploitation (high 𝛼 𝛼 + 𝛽 procedures), risk aversion (avoiding contexts similar to past failures), and exploration (high entropy procedures). The expected utility formulation naturally handles the explore-exploit tradeoff: early in learning, high entropy dominates selection, while after sufficient evidence accumulates, expected reward becomes the primary driver.

Contrastive Refinement of Procedures

As experience accumulates, procedures with both successful and failed instances are subjected to contrastive refinement to improve their accuracy and robustness. For a procedure Proc 𝑖 with sets of successful and failed contexts S 𝑖 and F 𝑖 , the LLM performs discriminative comparison, D 𝑖 = ContrastiveExtract (S 𝑖 , F 𝑖 ) , identifying differences in three dimensions: (i) precondition patterns ( ΔΨ + 𝑖 and ΔΨ -𝑖 ) that distinguish successful from failed initial contexts, (ii) action discrepancies ( Δ 𝜋 𝑖 ) revealing missing or misordered actions, and (iii) postcondition mismatches ( ΔΦ 𝑖 ) that capture incomplete goal states. These discriminators drive explicit refinement operations

$$

$$

When distinct execution modes are detected, the procedure is specialized into separate variants with inherited reliability priors. This process progressively tightens applicability conditions and action precision, yielding interpretable improvements purely through memory edits rather than gradient updates.

Meta-procedural Composition

To extend reasoning beyond atomic skills, MACLA automatically discovers and learns meta-procedures that are structured compositions of procedures that capture recurrent long-horizon strategies. When a sequence of procedures ⟨ Proc 𝑖 1 , . . . , Proc 𝑖 𝑚 ⟩ repeatedly leads to success under a common high-level goal, the agent abstracts it as MP 𝑗 = ⟨G meta 𝑗 , Ψ meta 𝑗 , { Proc 𝑖 1 , . . . , Proc 𝑖 𝑚 } , Θ 𝑗 ⟩ . Here, Θ 𝑗 denotes a lightweight control policy governing conditional transitions among sub-procedures based on the current observation and execution context, Θ 𝑗 ( 𝑜 𝑡 , index ) ∈ { continue , skip , repeat , abort } . This policy is distilled by analyzing successful traces, where the LLM identifies observation patterns that triggered each branch-for example, repeating when postconditions are unmet, skipping when preconditions already hold, or aborting when failures recur. Each meta-procedure maintains its own Beta success posterior 𝑝 ( 𝜎 𝑗 |D 𝑗 ) = Beta ( 𝛼 𝑗 , 𝛽 𝑗 ) and is refined periodically to add new branches, reorder sub-procedures, or prune redundant ones. Through these hierarchical compositions, MACLA acquires flexible 'playbooks' that encapsulate extended strategies with conditional logic.

Ontological Semantic Grounding

To enable cross-context generalization (e.g., procedures learned on "mug" applying to "cup"), MACLA constructs a lightweight ontological semantic index during offline memory construction. We extract the 𝑘 𝑣𝑜𝑐𝑎𝑏 most frequent words from task descriptions and actions, then cluster semantically similar words using SentenceTransformer embeddings [12] to form an implicit domain ontology:

$$

$$

where each cluster C 𝑤 represents a semantic category (e.g., C container = { mug , cup , glass } ). During retrieval, observations are mapped to these ontological categories, allowing procedures to match across lexically different but semantically equivalent contexts. This ontological grounding enables domain-adaptive generalization without requiring explicit knowledge engineering.

System Efficiency and Memory Management

To ensure practical scalability, MACLA employs efficient retrieval, bounded growth, and strict control over LLM usage. All procedures and meta-procedures are embedded in an approximate nearestneighbor index supporting sublinear retrieval ( 𝑂 ( log 𝑁 𝑝 ) ) for semantic search. The episode buffer stores at most 𝑁 𝑏 = 1000 steps, providing local context for LLM prompts and post-episode updates. Each procedure maintains a failure index limited to 𝐾 fail = 15 entries, managed through success-based removal, redundancy-aware eviction, and temporal decay, ensuring that memory remains concise and informative. To prevent memory saturation, procedures and meta-procedures are periodically pruned using a multi-factor utility score that balances reliability, usage frequency, and temporal relevance:

$$

$$

where 𝛼 𝑖 𝛼 𝑖 + 𝛽 𝑖 is the Bayesian success rate (reliability), 𝑛 𝑖 is the execution count of procedure 𝑖 , 𝑁 total is the total invocations across all procedures in the current episode window, 𝑡 current is the current episode index, 𝑡 last 𝑖 is the episode when 𝑖 was last used, and 𝜏 is the temporal decay constant.

The weighting coefficients 𝜆 𝑟 = 0 . 5, 𝜆 𝑓 = 0 . 3, and 𝜆 𝑡 = 0 . 2 reflect the relative importance of each factor: reliability receives the highest weight (0.5) as it directly predicts future success; frequency receives moderate weight (0.3) to favor well-tested procedures while avoiding over-retention of obsolete frequently-used skills; recency receives the lowest weight (0.2) to provide soft temporal decay without aggressive forgetting. These values were determined through grid search over { 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 } × { 0 . 2 , 0 . 3 , 0 . 4 } × { 0 . 1 , 0 . 2 , 0 . 3 } on ALFWorld validation, with the constraint 𝜆 𝑟 + 𝜆 𝑓 + 𝜆 𝑡 = 1 . 0 for interpretability. The selected configuration (0.5, 0.3, 0.2) yielded the best balance between retaining high-quality procedures (>0.7 success rate) and pruning low-utility entries (<0.4 success rate), as validated later in Figure 4. Entries with the lowest utility are removed while ensuring diversity across goal clusters through stratified sampling. These operations keep the total memory footprint below 4 MB for hundreds of procedures.

Finally, MACLA limits LLM usage to a fixed budget of API calls per episode to cover segmentation, abstraction, and occasional refinement, while all retrieval, Bayesian scoring, and updates are symbolic or vectorized. As a result, per-step runtime remains effectively constant and inference cost does not scale with experience. This memory-first design ensures that MACLA remains efficient, interpretable, and deployable for continual learning across long interaction horizons.The theoretical foundations are provided in Appendix D.

Algorithm

At runtime, MACLA executes a new task by coupling frozen semantic reasoning with memory-driven decision making. The agent receives an initial observation 𝑜 0 (and, optionally, an instruction string) and embeds it as h 0 = 𝜙 ( 𝑜 0 ) . This embedding queries the semantic index of the external memory to retrieve a compact candidate set consisting of procedures { Proc 𝑖 } and meta-procedures { MP 𝑗 } whose embeddings are most similar to h 0 . Retrieval is approximate nearest neighbor over the concatenated descriptors of goals, preconditions, and postconditions, which keeps lookup sublinear in memory size.

Given the candidate set, MACLA ranks each item with a Bayesian expected-utility score that trades off contextual relevance, estimated success, risk, and information gain under the procedure's Beta posterior. The highest-scoring item above a confidence threshold is selected; otherwise the agent falls back to zero-shot LLM reasoning for that step, logs the outcome, and continues. If a meta-procedure is chosen, execution proceeds hierarchically under its composition policy Θ 𝑗 ( 𝑜 𝑡 , index ) ∈ { continue , skip , repeat , abort } until completion or abort; if an atomic procedure is chosen, the agent checks preconditions Ψ 𝑖 against 𝑜 𝑡 , invokes the action sketch 𝜋 𝑖 via the frozen LLM's action formatter, and verifies postconditions Φ 𝑖 to certify completion. In both cases the outcome updates ( 𝛼, 𝛽 ) and appends the initial context to the corresponding success or failure set for later analysis.

After each execution, the agent re-embeds the new observation and repeats retrieval and selection until the task is solved or a horizon is reached. When a procedure accumulates both successes and failures, a contrastive pass is triggered: the LLM proposes discriminators that tighten Ψ 𝑖 , repair 𝜋 𝑖 , and refine Φ 𝑖 , or if distinct modes are detected, specializes the procedure into variants that inherit prior counts. When successful episodes repeatedly traverse a small set of procedures in a stable order, the agent abstracts a meta-procedure with its own success posterior and a lightweight Θ 𝑗 distilled from divergence points across traces. Throughout, memory remains bounded by pruning with a utility that blends reliability, frequency, and recency, and the LLM-call budget is capped, as retrieval, scoring, and updates are vectorized operations. The complete runtime procedure is outlined in Algorithm 1.

Experiments

We evaluate MACLA on four challenging interactive agent benchmarks spanning diverse domains. All experiments use consistent hyperparameters across tasks to demonstrate generalization without task-specific tuning.

Algorithm 1 MACLA Runtime Procedure with Function Descriptions

Require: observation 𝑜 0 , memory M (procedures, meta-procedures, indices), horizon 𝐻 1: h ← 𝜙 ( 𝑜 0 ) ⊲ Embed observation 2: C ← RetrieveCandidates ( h , M ) ⊲ Top𝑘 ANN search 3: while not Terminal and 𝑡 < 𝐻 do 4: for all 𝑐 ∈ C do 5: EU [ 𝑐 ] ← ExpectedUtility ( 𝑐, 𝑜 𝑡 , M ) ⊲ Compute Eq. 4 6: end for 7: 𝑐 ★ ← arg max 𝑐 ∈C EU [ 𝑐 ] 8: if EU [ 𝑐 ★ ] < 𝜃 conf then 9: ( 𝑜 𝑡 + 1 , 𝑦 ) ← ZeroShotStep ( 𝑜 𝑡 ) ⊲ LLM generates action directly 10: else if 𝑐 ★ is MP 𝑗 then 11: ( 𝑜 𝑡 + 1 , 𝑦 ) ← ExecuteMeta ( MP 𝑗 , Θ 𝑗 , 𝑜 𝑡 ) ⊲ Run with control policy 12: ( 𝛼 𝑗 , 𝛽 𝑗 ) ← UpdateBeta ( ( 𝛼 𝑗 , 𝛽 𝑗 ) , 𝑦 ) ⊲ 𝛼 ← 𝛼 + 𝑦 , 𝛽 ← 𝛽 +( 1 -𝑦 ) 13: else ⊲ 𝑐 ★ is atomic Proc 𝑖 14: if CheckPre ( Ψ 𝑖 , 𝑜 𝑡 ) then ⊲ Verify preconditions match 𝑜 𝑡 15: ( 𝑜 𝑡 + 1 , 𝑦 ) ← ExecuteProc ( 𝜋 𝑖 , 𝑜 𝑡 ) ⊲ Instantiate & execute 𝜋 𝑖 16: 𝑦 ← 𝑦 ∧ CheckPost ( Φ 𝑖 , 𝑜 𝑡 + 1 ) ⊲ Verify postconditions in 𝑜 𝑡 + 1 17: else 18: ( 𝑜 𝑡 + 1 , 𝑦 ) ← ZeroShotStep ( 𝑜 𝑡 ) ⊲ Preconditions failed, fallback 19: end if 20: ( 𝛼 𝑖 , 𝛽 𝑖 ) ← UpdateBeta ( ( 𝛼 𝑖 , 𝛽 𝑖 ) , 𝑦 ) 21: RecordContext ( S 𝑖 , F 𝑖 , 𝑜 𝑡 , 𝑦 ) ⊲ Add to success/fail sets 22: end if 23: if RefineTrigger ( 𝑐 ★ ) then ⊲ If | S | , | F | ≥ 3 24: ContrastiveRefine ( 𝑐 ★ ) ⊲ LLM compares S vs. F (§4.3) 25: end if 26: h ← 𝜙 ( 𝑜 𝑡 + 1 ) ; C ← RetrieveCandidates ( h , M ) ; 𝑡 ← 𝑡 + 1 27: end while 28: if EligibleForMeta ( trace ) then ⊲ If ≥ 3 procs in stable order 29: ExtractOrRefineMeta ( trace , M ) ⊲ Create/update meta-proc 30: end if 31: PruneAndMaintain ( M ) ⊲ Remove low-utility via Eq. 8

Experimental Setting

Memory Architecture: Episode buffer 𝑁 𝑏𝑢𝑓 𝑓 𝑒𝑟 = 1000 (stores recent observations and actions for temporal context provision during action generation); procedural memory 𝑁 𝑝𝑟𝑜𝑐 = 200 (capacity for extracted reusable skills); meta-procedural memory 𝑁 𝑚𝑒𝑡𝑎 = 50 (capacity for hierarchical procedure compositions). Critically, MACLA does not store raw trajectories. Instead, the LLM segments each episode into coherent sub-tasks and extracts structured procedures (Section 4.1). Duplicate detection with similarity threshold 𝜃 𝑑𝑢𝑝 = 0 . 85 prevents redundant storage. Through this process, the 2,851 ALFWorld training trajectories compress into approximately 187 unique procedures-demonstrating efficient knowledge distillation from experience.

Bayesian Selection. Information gain weight 𝜆 𝑖𝑛𝑓 𝑜 = 0 . 1, failure cost 𝐶 𝑓 𝑎𝑖𝑙 = 0 . 5. These parameters balance exploration (trying uncertain procedures to reduce epistemic uncertainty) with exploitation (selecting high-posterior reliable procedures).

Contrastive Refinement. Minimum contexts 𝑛 𝑠 𝑚𝑖𝑛 = 𝑛 𝑓 𝑚𝑖𝑛 = 3. Refinement activates only when a procedure has accumulated at least 3 successes and 3 failures, ensuring sufficient statistical evidence for discriminative pattern extraction.

LLM Configuration. Llama-2-7B [18] via Ollama with 4-bit quantization and temperature 𝑇 = 0 . 7. The LLM parameters remain frozen throughout all experiments-learning occurs exclusively through external memory updates.

Benchmarks and Dataset Statistic: ALFWorld [16] (2,851 train, 274 test) is a text-based embodied environment with six household tasks (e.g., retrieval, placement). We follow the standard train/validation-seen/validation-unseen split, where test trajectories feature novel object-location configurations. WebShop [25] (1,624 train, 200 test) simulates e-commerce search over 12,087 products, requiring agents to follow natural-language instructions via multi-step navigation and filtering. TravelPlanner [21] (1,000 train, 180 validation, 45 test) involves multi-day itinerary planning under hard constraints (budget, dates) and soft preferences (cuisine, attractions). Evaluation uses Common Sense (CS) and Hard Constraint (HC) scores. InterCodeSQL [24] benchmarks interactive text-to-SQL generation over diverse schemas, requiring correct handling of schema relationships and varying query difficulty.

Experimental Results and Analysis

Table 1 compares MACLA against state-of-the-art baselines across all benchmarks. We organize baselines into three paradigms: promptbased methods using in-context learning, outcome refinement approaches optimizing trajectory-level rewards, and process refinement methods refining step-level generation. MACLA achieves the highest average performance (78.1%) while using a 7B parameter model, demonstrating that domain-agnostic procedural memory with Bayesian selection and contrastive refinement enables competitive performance without task-specific engineering.

In Table 1, MACLA achieves state-of-the-art results on TravelPlanner (83.3 CS) and ALFWorld-Unseen (90.3%), outperforming methods that rely on models 10× larger. Its strong performance across all benchmarks demonstrates cross-domain generalization, while the positive generalization gap on ALFWorld (+3.1 points for unseen vs. seen) indicates robust compositional transfer rather than memorization.

Conclusion

We presented MACLA, a framework that decouples reasoning from learning by maintaining a frozen LLM and performing all adaptation in an external hierarchical procedural memory through Bayesian selection, contrastive refinement, and meta-procedural composition. MACLA achieves 78.1% average performance across four benchmarks using only a 7B model, with state-of-the-art results on ALFWorld (87.2% seen; 90.3% unseen) and TravelPlanner (83.3%). The system compresses 2,851 ALFWorld training trajectories into 187 reusable procedures through semantic abstraction and duplicate detection, demonstrating efficient knowledge distillation.

Detailed Ablation Studies and Memory Analysis

This section provides comprehensive ablation studies examining MACLA's component contributions, memory scaling behavior, and task-specific effectiveness. These experiments address critical questions about system design choices and identify performance bottlenecks across different benchmarks. Table 4 systematically evaluates the contribution of each MACLA component by measuring performance degradation when individual modules are removed. Beyond success rates, we track memory dynamics (procedure/metaprocedure counts), behavioral patterns (reuse rate), and computational efficiency (LLM calls per episode).

Table 4: Component ablation and memory dynamics analysis on ALFWorld. All variants use Llama-2-7B.

Proc./Meta Count: final memory size after 200 episodes. Reuse Rate: % of actions from retrieved procedures vs. zero-shot LLM. LLM Calls: average per episode.

Synergistic Effects: Nosingle component accounts for MACLA's full performance. The combination of Bayesian selection (uncertaintyaware), contrastive learning (quality refinement), and meta-procedures (hierarchical composition) creates synergistic effects. Bayesian selection identifies reliable procedures, contrastive learning makes them more robust, and meta-procedures compose them efficiently.

Memory Capacity Scaling

Table 5 investigates the relationship between memory capacity and performance, addressing whether larger memory always yields better results or if there exists an optimal capacity.

Table 5: Impact of procedural memory capacity on performance. Results on ALFWorld after 200 training episodes.

Actual Proc.: number of procedures after training (may be less than capacity if not all slots filled). Avg 𝛼 𝛼 + 𝛽 : mean posterior success rate across all procedures.

Task-Specific Memory Effectiveness

Figure 5 explains SQL underperformance through three metrics. Lowreuse(51%): SQLqueries are schema-specific, e.g., customers.age does not apply to employees.experience . ALFWorld generalizes via placeholders (