---
title: Ollama
date: 2024/02/19
description: 'Learn how to run AI models locally using Ollama'
tags:
  - ollama
  - guide
  - LLM
ogImage: /images/blog/2024-03-02_ollama.png
authorid: librechat
---

import { BlogHeader } from '@/components/blog/BlogHeader'

<BlogHeader />

# Unlock the Power of Ollama: Run Large Language Models on Your Local Hardware

Are you tired of relying on cloud-based solutions to run your language models? Do you want to tap into the potential of large language models without breaking the bank? Look no further than Ollama, a revolutionary platform that lets you run large language models on your local hardware.

## What Can Ollama Do?

With Ollama, you can:

* Run large language models on your local hardware, minus the hefty cloud computing costs
* Host multiple models with ease
* Dynamically load models upon request, streamlining your workflow

## Getting Started with Ollama

Ready to unlock the power of Ollama? Follow these simple steps to get started:

### Install Ollama

You have two options to install Ollama: via the Ollama app or using Docker.

<Tabs items={['Method 1: Ollama App Install', 'Method 2: Docker Install']}>
<Tabs.Tab>
For Mac, Linux, and Windows users, follow the instructions on the [Ollama Download](https://ollama.com/download) page to get started. Ollama supports GPU acceleration on Nvidia, AMD, and Apple Metal, so you can harness the power of your local hardware.
</Tabs.Tab>
<Tabs.Tab>
### Docker Install
If you're already using LibreChat, you can add Ollama to your container stack using the `docker-compose.override.yml` file. Simply create the `docker-compose.override.yml` with the following content at the root of your LibreChat folder, then use the command `docker compose up -d {:sh}`:
- See: [Docker Override](/docs/configuration/docker_override)

```yaml filename="docker-compose.override.yml"
version: '3.4'

services:
# USE LIBRECHAT CONFIG FILE
  api:
    volumes:
    - type: bind
      source: ./librechat.yaml
      target: /app/librechat.yaml

# ADD OLLAMA
  ollama:
    image: ollama/ollama:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [compute, utility]
    ports:
      - "11434:11434"
    volumes:
      - ./ollama:/root/.ollama
```
Once installed, you can access Ollama commands within the container using:
 -  `docker exec -it ollama /bin/bash {:bash}`.

</Tabs.Tab>
</Tabs>

## Load Models in Ollama

Now that you have Ollama installed, it's time to load your models. Here's how:

1. Browse the [Ollama Library](https://ollama.ai/library) to explore available models.
2. Copy the text from the Tags tab on the library website and paste it into your terminal. The command should begin with `ollama run {:bash}`.
3. Check the model size to ensure it can run in GPU memory for optimal performance.
4. Use `/bye{:bash}` to exit the terminal when you're done.

## Configure LibreChat

Finally, use your `librechat.yaml` configuration file to add Ollama as a separate endpoint. Follow our [Custom Endpoints & Configuration Guide](/docs/configuration/librechat_yaml) for a step-by-step walkthrough.

With Ollama, you can unlock the full potential of large language models on your local hardware. Say goodbye to cloud computing costs and hello to faster, more efficient workflows.



