I have been experimenting on an idea of separating short term vs long term memory combining filesystem and databases. I have worked as data engineer for 15 years in the industry, and observed every single time the analysts would start on their local filesystem.

They’d dump data into messy CSVs, hack together reports in notebooks, experiment like crazy. Once they found something that actually mattered, we’d sit down together, define the proper schema, and move it into the real data warehouse with clean tables, relationships, and all the structure it needed.

Learning from how big organizations work, I built a memory system around it. You onboard your agent with “these are the entities I actually want to remember long-term” (Contacts, Orders, Projects, Incidents, etc.). The agent learns the schema for each. Then when you’re talking about your dad, instead of just appending another blob of text to the flat memory, it goes “this belongs in the Contacts entity” and saves it as structured data there.

We still have BM25 full-text and vector search like everyone else. The real unlock is forcing that structured save. Suddenly you get real relationships between entities and retrieval becomes more like graph traversal instead of “hope the similarity score is good.”

I ran it on LongMemEval and LoCoMo and yeah, the numbers looked better, but honestly the bigger difference is how much more natural and reliable the agent feels now.

https://preview.redd.it/ngw3qayafkwg1.png?width=5286&format=png&auto=webp&s=b87898d6f8c7e16a846842dab17eb59ac7788019

Does this analogy click for anyone else, or am I just seeing data-engineer patterns everywhere? I wonder how superior is this approach compared to just giving agents a database access.

Still very much an experiment on my end, so I’m open to feedback, gotchas, or completely different ways to think about it.

Here is the project: https://github.com/lobu-ai/lobu

Also lobu.ai/memory explains how it works end-to-end. All open-source.

[留言]

为什么值得关注

能改变理解方式，而不只是重复常识；有直接可用的方法、工具或操作价值；它提供了新的理解或解释，而不只是表面观点

来源：reddit，领域：tech，保留分：0.64