LLM Memory Architecture

Portal, initially DejaVu at Hack the North 2023, is a wearable that allows you to search through anything you've done, in person or online. To do so, we should treat time as a tree of events, where left to right is progression in time. Events that happen are parent nodes of smaller leaf nodes, that are individual moments.

Visual Search

We used CLIP for text and image embeddings, from Whisper transcripts and indexing with YOLO object detection and custom face detection. The idea is that we can index information at every moment and larger event, which is searchable with an input query. To deal with the large amound of information, we used NLP to decompose input queries into people, objects or specific moments to make smaller queries to. For example, conversations with Ian about stickers would would find moments with Ian's face and visual/text mentions of stickers.

To do this indexing, I crated something called hierarchical tree search which creates latent summaries of smaller events that are similar, so that conversations are not just retrieved as the 10-15 second fragments but as the actual time length in which they happened. This means that generating summaries of these events can be done in a productive and useful way.

Online Search

Most people spend a large portion of their days on devices, so searching through those events would give Portal a more complete view of what happens. We indexed messages, files, browser bookmarks and our calendars. Asking what did i do this afternoon would use these signals to get a complete overview. It's important to store this information in knowledge graphs, so we can create relations between seemingly orthogonal data points.

By manually adding categories for people, calendar events, conversations, files, emails, browser, etc..., we actually force relations. Conversations and events have people, files and emails can be related. At search time, these relations are searched for and given as context for the retrieval.

Here's a demo!