Local LLMs

I did a quick exploration for local llms and I found the following

If you search YouTube for local llms you’d find many videos about the matter. I found this to be the most useful one 6 Ways For Running A Local LLM

Table Of Contents

HuggingFace Transformers

• Go to HuggingFace.co the largest repository for open-source models
• Go to models and choose a conversational
• If you want chat download a conversational model like microsoft/dialogGPT-large
• Explore and follow instructions to use your model of choice
• Having Python, TensorFlow, PyTorch is usually a prerequisite for using the models
• Many models provide Python apis to interact with them
• The reliance on Python makes this option not the most performant one especially when you want to use models locally (I have an old laptop)
• Best for learning

LangChain

• Website: LangChain
• Framework for building language applications on top of models
• Can use local or remote models
• Oriented for software developers
• You will still need to install your tranformer library

Llama.cpp

• Github Repo: Llama.cpp
• C/C++ is a lot more performant than Python
• Can run bigger models on older hardware (Because C/C++ rocks)
• To use it you clone the repo and use it on a .gguf model
• You have to build the tool
• Download a .gguf model (for example a llama model from HuggingFace)
• You can chat with it through the command line

⭐ Llamafile

• Github Repo: Llamafile
• Built on top of llama.cpp
• A script that requires no compilcation
• Same benefits as llama.cpp with an easier install and UI
• Can embed your model in an executable file and share it
• Offers a browser interface that lets you tweak the model and chat with your chosen model
• My favorite tool as it’s performant, offers a UI for chatting, open-source, and can run .gguf models in contrast to easier to use tools that have a smaller library and options of llms to use

OllamaAI

• Website: OllamaAI
• It installs a command line tool where you can chat
• Easiest way to run a llama file
• Pull models using the cmd tool
• No much options to manage or tweak models
• No windows version

GPT4All

• Wesbite: GPT4ALL
• Most user friendly UI similar to ChatGPT
• Can add your own documents into the application so it indexes them and you can ‘talk’ to them
• Models can be tweaked
• Very limited amount of models