This is an old revision of the document!
Table of Contents
Backends
By itself, an LLM is just a block of data. The software required to perform inference with that data is typically called a “Backend”.
Sometimes the interface includes a backend (e.g. LocalTavern), other times it is a strict “Frontend” that requires a separate “Backend” to perform inference (e.g. SillyTavern).
Finally, most interfaces can be connected to 3rd-party APIs which provide backend capability for you. This can be incredibly useful when you want to run models which require more hardware than you have locally available.
Local Backends
These backends run locally. As a result, their capabilities are directly related to the quality of the hardware you have available.
Additionally, each of these require a separate model to operate.
Inference Engines
Each of the following perform inference without needing additional software. User friendliness is not the first priority with these utilities.
Engine Manangers
These tools are designed to provide a user-friendly layer which handles backend needs and engine management simultaneously. If you're not sure what to pick, this is a good place to start.
| Name | Notes |
|---|---|
| text-generation-webui (Oobabooga) | Most popular backend, for good reason. |
| koboldcpp | Good UI, capable of managing itself. |
| LocalAI | Provides OpenAI-compatible API. |
| ollama | Wraps llama.cpp. |
| tabbyAPI | Official API server for ExLlama engines. |
Note: This list does not include backends that can directly accept Character Cards - those are considered interfaces.
3rd-Party API providers
These are essentially remote backends. Everything you send and receive is, at minimum, available to the provider(s). Censorship can be expected, to varying degrees.
| Name | Notes |
|---|---|
| AI Horde | Free, with limited performance and models. |
| OpenRouter | Large model selection. Low(er) cost. |
| mancer | Low/no censorship. Free tier available. |
| NovelAI | Low/no censorship. |
| Pollinations | Free tier available with ads. |
Additionally, most commercial APIs can be utilized such as ChatGPT, Claude, Perplexity, etc.