---
title: LLMs on Apple Silicon with MLX
date: 2024/05/01
description: Unleash the power of Apple Silicon by running large language models locally with MLX.
tags:
  - apple
  - LLM
  - guide
  - MLX
ogImage: /images/blog/2024-05-01_mlx.png
authorid: anon
---

import { BlogHeader } from '@/components/blog/BlogHeader'

<BlogHeader />

# Running Large Language Models on Apple Silicon with MLX

In this post, we'll explore how to leverage the power of Apple Silicon hardware (M1, M2, M3) to run large language models locally using MLX. MLX is an open-source project that enables GPU acceleration on Apple's Metal backend, allowing you to harness the unified CPU/GPU memory for efficient model execution.

## Installing MLX on macOS

MLX supports GPU acceleration on Apple's Metal backend through the `mlx-lm` Python package. To get started, follow the instructions provided in the [mlx-lm package installation guide](https://github.com/ml-explore/mlx-examples/tree/main/llms).

**Note**: MLX is currently supported only on Mac MX series devices.

## Loading Models with MLX

While MLX supports common HuggingFace models directly, it is recommended to use converted and quantized models provided by the [mlx-community](https://huggingface.co/mlx-community). These models have been optimized for efficient performance on Apple Silicon hardware, depending on your device's capabilities.

To load a model with MLX, follow these steps:

1. Browse the available models on [HuggingFace](https://huggingface.co/models?search=mlx-community).
2. Copy the text from the model page in the format `<author>/<model_id>` (e.g., `mlx-community/Meta-Llama-3-8B-Instruct-4bit`).
3. Check the model size. Models that can run in CPU/GPU unified memory tend to perform better.
4. Follow the instructions to launch the model server [Run OpenAI Compatible Server Locally](https://github.com/ml-explore/mlx-examples/blob/main/llms/mlx_lm/SERVER.md) by running the command:

   ```sh filename="Launch the model server"
   mlx_lm.server --model <author>/<model_id>
   ```

## Configuring LibreChat

To use MLX with LibreChat, you'll need to add it as a separate endpoint in the `librechat.yaml` configuration file. An example configuration for the Llama-3 model is provided. Follow the [Custom Endpoints & Configuration Guide](/docs/configuration/librechat_yaml) for more details.

With MLX, you can now enjoy the benefits of running large language models locally on your Apple Silicon hardware, unlocking new possibilities for efficient and powerful natural language processing tasks.