One day, one feature: Automatically unloading models from memory

Today is release day: Mini Me v1.0.1 is out with a few new features:

• A message typed during a response is queued.
• Difficulty signals: when the model "struggles", Mini Me reports it (warnings are displayed).
• Display of the completion duration.
• autoPull: download the models at startup.
• Display the JSON response of a decider in the conversation
• unloadModels: true: before passing a question to an agent, Mini Me unloads from memory the model of the agent that is asking. When the answer comes back, it unloads the model of the agent that was asked.

It's this last feature, unloadModels: true, that matters.

Automatically unloading models from memory

Why does it matter? When you use an "LLM engine" (Ollama, Llmman, ...), the first time you run a completion, it loads the model into memory (and it will be unloaded after a while if it's no longer used). If you run a 2nd completion right after, but with a different model, you'll end up with two models in memory. And if your machine doesn't have much memory, this quickly becomes a problem: you'll run into heavy slowdowns, or even a frozen machine (my MacBook Air with 32GB of RAM went as far as freezing completely, with a forced reboot).

So I urgently needed to find a solution, because being able to use several agents with different models in a "small local models" context is particularly interesting (e.g.: choosing the most appropriate agent to answer, ...).

Actually, it was already possible to do it using hooks (you'll find an example here: /demos/coding-helpers), but today, you just need to declare unloadModels: true in the mmConfigs section (the multi-agent mode of Mini Me), in the main agent's configuration file, like this:

mmConfigs:
  go:
    path: agent.go.yaml
    description: >-
      Go specialist. Explains Go concepts and how to do something in Go,
      with short examples, like documentation. For any question about Go.
    yolo: true
    unloadModels: true
  rust:
    path: agent.rust.yaml
    description: >-
      Rust specialist. Explains Rust concepts and how to do something in Rust, 
      with short examples, like documentation. For any question about Rust.
    yolo: true
    unloadModels: true

And it's no more complicated than that. How it works is simple too:

You'll find an example to try here: one-day-one-feature/tree/main/05-unload-models.

Don't forget, you need the latest version of Mini Me v1.0.1.

That's all for today. And most likely see you tomorrow 👋.