Models

LM Studio or Local#

The switch at the top left of the window chooses who runs the model:

Where Local finds models. Roller looks for model files in two places:

  1. Its own models folder, inside the Acopa folder (the folder with acopa.command in it). In Local mode, the folder button next to the model list opens it in Finder.
  2. LM Studio's model folder, if you have downloaded models with LM Studio. So models you got with LM Studio show up in Local mode too, without copying anything.

Each side remembers its own model. Switching doesn't unload anything on the other side, so mind your Mac's memory if both have a model loaded.

What is Hugging Face?#

Hugging Face (huggingface.co) is a website where companies, researchers and hobbyists share AI models: a huge public library of models, a bit like an app store. Most models there are free to download, and many come in versions made for running on your own computer. The name comes from the 🤗 emoji, which is also its logo.

Roller doesn't download anything by itself. You download a model file from Hugging Face once, put it into Roller's models folder, and from then on it runs on your Mac without the internet. See Download a model from Hugging Face.

What is LM Studio?#

LM Studio (lmstudio.ai) is a free app from another company; it's not part of Roller. It finds and downloads models from Hugging Face for you, shows which ones fit your Mac, and runs them. It has its own chat window, and it can also serve a model to other apps on your Mac, such as Roller, through a small built-in server.

You don't need LM Studio to use Roller. It's simply a comfortable way to browse and download models; see Get models with LM Studio. Once downloaded, you can run those models either in Roller's Local mode, or through LM Studio with the LM Studio switch (LM Studio's server must then be running: in LM Studio, Developer tab, Start server).

Model sizes: what "12B" means#

Model names usually include a size, such as 4B, 12B or 27B. The B stands for billion parameters. Parameters are the numbers a model learned during its training; roughly, the more it has, the better it understands, remembers and writes, and the more memory (RAM) it needs.

For Roller, about 12B and up is where three characters stay clearly apart. Smaller models work, but the characters tend to sound alike.

How much fits on your Mac? Very roughly, a model file in the usual size (Q4, see below) takes about 0.6 GB of memory per billion parameters: a 12B model is a file of about 7 GB, a 27B model about 16 GB. On top of that the model needs some memory for the play itself (the context), and your Mac and other apps need theirs. A good rule: the model file should be no bigger than about half of your Mac's memory.

  • 8 GB of memory: models up to about 4B.
  • 16 GB: up to about 8B; 12B works if few other apps are open.
  • 24 GB: up to about 14B.
  • 32 GB: up to about 24–27B.
  • 64 GB or more: 27–32B comfortably; up to about 70B with little else running.

To see how much memory your Mac has: Apple menu → About This Mac → Memory.

If a model is too big, loading takes very long, or your Mac becomes very slow. Then pick a smaller model.

Model files: GGUF and "Q4_K_M"#

Roller's Local mode runs model files in the GGUF format: a single file (ending in .gguf) that holds the whole model. On Hugging Face, models in this format usually have GGUF in their name.

Most GGUF models come in several versions of different sizes, marked in the file name:

  • Q4_K_M: the numbers inside the model are squeezed to about 4 bits each. Much smaller and faster, and only slightly less precise. This is the one to pick in most cases.
  • Q5_K_M, Q6_K: a bit bigger, a bit more precise.
  • Q8_0: about twice the size of Q4, very close to the original.
  • Q3, Q2: smaller, but the quality drops noticeably.

Files with mmproj in their name are add-ons for looking at pictures; Roller doesn't need them. Models in the MLX format only run in LM Studio, and Local mode doesn't list them.

Loading a model#

A model that isn't loaded yet (○ in the list) is loaded when you send your first turn. Small models load in seconds; big ones can take minutes. Meanwhile a window shows the progress: what's happening, how much is done, and roughly how long it will take.

Cancel loading (or Esc) stops the loading. Your turn isn't lost: its text comes back into the boxes.

In Local, only one model is loaded at a time. Picking another one unloads the previous one when you send your next turn.

How long can a game get?#

A model can only keep a certain amount of text in mind at once: its context. Each turn, Roller sends it the whole play so far: the Protocol, the story, the characters, every turn, and your new turn. This is measured in tokens (a token is roughly three quarters of a word).

The token gauge at the bottom left shows how much of the context the game fills, for example Prompt ~1,563 / 16,384 tokens. A ~ means the number is an estimate; once the model is loaded, it's exact.

  • The bar turns yellow when the game gets close to the limit.
  • It turns red when the game no longer fits. The status line then says so, as soon as you open the game or pick the model.

Part of the context is always kept free for the characters' answer.

If a game doesn't fit, Roller stops your turn before it reaches the model and explains what to do. (Without this check, the model would only see part of the play, lose the story, and play something completely unrelated.) If the numbers are only an estimate, you can choose Load anyway: Roller then checks again exactly once the model is loaded.

What helps:

As a rough guide: the Protocol and cast take some 2,000–3,000 tokens, and every turn adds about 400–600. So 8,192 tokens last about ten turns, while 16,384–32,768 give room for long games.