centersl/dify-docs
1
0
Fork 0
mirror of https://github.com/langgenius/dify-docs.git synced 2026-03-27 13:28:32 +07:00
Code Issues Packages Projects Releases Wiki Activity

update the dify version, add the migration script link, and improve formatting

Browse Source
This commit is contained in:
Riskey
2025-12-16 13:17:36 +08:00
parent be4fd96603
commit dca5a24b25
1 changed files with 88 additions and 116 deletions
Download Patch File Download Diff File Expand all files Collapse all files

204
en/self-host/troubleshooting/weaviate-v4-migration.mdx
View File

@@ -6,13 +6,10 @@ title: Weaviate Migration Guide upgrading to Client v4 and Server 1.27+
## Overview
Starting with an upcoming Dify release, the weaviate-client has been upgraded from v3 to v4.17.0. This upgrade brings significant performance improvements and better stability, but requires **Weaviate server version 1.27.0 or higher**.
Starting with **Dify v1.9.2**, the weaviate-client has been upgraded from v3 to v4.17.0. This upgrade brings significant performance improvements and better stability, but requires **Weaviate server version 1.27.0 or higher**.
<Warning>
**BREAKING CHANGE:** The new weaviate-client v4 is NOT backward compatible
with Weaviate server versions below 1.27.0. If you are running a self-hosted
Weaviate instance on version 1.19.0 or older, you must upgrade your Weaviate
server before upgrading Dify.
**BREAKING CHANGE**: The new weaviate-client v4 is NOT backward compatible with Weaviate server versions below 1.27.0. If you are running a self-hosted Weaviate instance on version 1.19.0 or older, you must upgrade your Weaviate server before upgrading Dify.
</Warning>
### Who Is Affected?
@@ -23,7 +20,7 @@ This migration affects:
- Users currently on Weaviate server version 1.19.0-1.26.x
- Users upgrading to Dify versions with weaviate-client v4
**Not affected:**
**Not affected**:
- Cloud-hosted Weaviate users (Weaviate Cloud manages the server version)
- Users already on Weaviate 1.27.0+ can upgrade Dify without additional steps
@@ -35,35 +32,31 @@ This migration affects:
The weaviate-client v4 introduces several breaking changes:
1. **Minimum Server Version:** Requires Weaviate server 1.27.0 or higher
2. **API Changes:** New import structure (`weaviate.classes` instead of `weaviate.client`)
3. **gRPC Support:** Uses gRPC by default on port 50051 for improved performance
4. **Authentication Changes:** Updated authentication methods and configuration
1. **Minimum Server Version**: Requires Weaviate server 1.27.0 or higher
2. **API Changes**: New import structure (`weaviate.classes` instead of `weaviate.client`)
3. **gRPC Support**: Uses gRPC by default on port 50051 for improved performance
4. **Authentication Changes**: Updated authentication methods and configuration
### Why Upgrade?
- **Performance:** Significantly faster query and import operations via gRPC (50051)
- **Stability:** Better connection handling and error recovery
- **Future Compatibility:** Access to latest Weaviate features and ongoing support
- **Security:** Weaviate 1.19.0 is over a year old and no longer receives security updates
- **Performance**: Significantly faster query and import operations via gRPC (50051)
- **Stability**: Better connection handling and error recovery
- **Future Compatibility**: Access to latest Weaviate features and ongoing support
- **Security**: Weaviate 1.19.0 is over a year old and no longer receives security updates
## Version Compatibility Matrix
| Dify Version | weaviate-client Version | Compatible Weaviate Server Versions |
| Dify Version | Weaviate-client Version | Compatible Weaviate Server Versions |
| ------------ | ----------------------- | ----------------------------------- |
| ≤ 1.9.1 | v3.x | 1.19.0 - 1.26.x |
| ≥ 1.9.2\* | v4.17.0 | 1.27.0+ (tested up to 1.33.1) |
| ≥ 1.9.2 | v4.17.0 | 1.27.0+ (tested up to 1.33.1) |
<Info>
*The exact Dify version with weaviate-client v4 may vary. Check the release
notes for your specific version. This migration applies to any Dify version
using weaviate-client v4.17.0 or higher.
This migration applies to any Dify version using weaviate-client v4.17.0 or higher.
</Info>
<Info>
Weaviate server version 1.19.0 was released over a year ago and is now
outdated. Upgrading to 1.27.0+ provides access to numerous improvements in
performance, stability, and features.
Weaviate server version 1.19.0 was released over a year ago and is now outdated. Upgrading to 1.27.0+ provides access to numerous improvements in performance, stability, and features.
</Info>
## Prerequisites
@@ -101,26 +94,24 @@ Choose the migration path that matches your deployment setup and current Weaviat
### Choose Your Path
- **Path A – Migration with Backup (from 1.19):** Recommended if you are still on Weaviate 1.19. You will create a backup, upgrade to 1.27+, repair any orphaned data, and then migrate the schema.
- **Path B – Direct Recovery (already on 1.27+):** Use this if you already upgraded to 1.27+ and your knowledge bases stopped working. This path focuses on repairing the data layout and running the schema migration.
- **Path A – Migration with Backup (from 1.19)**: Recommended if you are still on Weaviate 1.19. You will create a backup, upgrade to 1.27+, repair any orphaned data, and then migrate the schema.
- **Path B – Direct Recovery (already on 1.27+)**: Use this if you already upgraded to 1.27+ and your knowledge bases stopped working. This path focuses on repairing the data layout and running the schema migration.
<Warning>
Do **not** attempt to downgrade back to 1.19. The schema format is
incompatible and will lead to data loss.
Do **not** attempt to downgrade back to 1.19. The schema format is incompatible and will lead to data loss.
</Warning>
### Path A: Migration with Backup (From 1.19)
<Info>
Safest path. Creates a backup before upgrading so you can restore if anything
goes wrong.
Safest path. Creates a backup before upgrading so you can restore if anything goes wrong.
</Info>
#### Prerequisites
- Currently running Weaviate 1.19
- Docker + Docker Compose installed
- Python 3.11+ available for the schema migration script
- Python 3.11+ available for the [schema migration script](https://github.com/langgenius/dify-docs/blob/main/assets/migrate_weaviate_collections.py)
#### Step A1: Enable the Backup Module on Weaviate 1.19
@@ -152,7 +143,7 @@ sleep 10
#### Step A2: Create a Backup
1. **List your collections:**
1. **List your collections**:
```bash
curl -s -H "Authorization: Bearer <WEAVIATE_API_KEY>" \
@@ -166,7 +157,7 @@ sleep 10
"
```
2. **Trigger the backup:** include specific collection names if you prefer.
2. **Trigger the backup**: include specific collection names if you prefer.
```bash
curl -X POST \
@@ -179,7 +170,7 @@ sleep 10
}'
```
3. **Check backup status:**
3. **Check backup status**:
```bash
sleep 5
@@ -188,7 +179,7 @@ sleep 10
python3 -m json.tool | grep status
```
4. **Verify backup files exist:**
4. **Verify backup files exist**:
```bash
ls -lh docker/volumes/weaviate_backups/kb-backup/
@@ -196,7 +187,7 @@ sleep 10
#### Step A3: Upgrade to Weaviate 1.27+
1. **Upgrade Dify to a version that ships Weaviate 1.27+:**
1. **Upgrade Dify to a version that ships Weaviate 1.27+**:
```bash
cd /path/to/dify
@@ -204,13 +195,13 @@ sleep 10
git checkout main # or a tagged release that includes the upgrade
```
2. **Confirm the new Weaviate image:**
2. **Confirm the new Weaviate image**:
```bash
grep "image: semitechnologies/weaviate" docker/docker-compose.yaml
```
3. **Restart with the new version:**
3. **Restart with the new version**:
```bash
cd docker
@@ -223,7 +214,7 @@ sleep 10
You can fix orphaned LSM data either from the host or inside the container:
**Option A: From host (if volumes are mounted):**
**Option A: From host (if volumes are mounted)**:
```bash
cd docker/volumes/weaviate
@@ -245,7 +236,7 @@ docker compose restart weaviate
sleep 15
```
**Option B: Inside Weaviate container (recommended):**
**Option B: Inside Weaviate container (recommended)**:
```bash
cd /path/to/dify/docker
@@ -282,17 +273,14 @@ sleep 15
pip install weaviate-client requests
```
2. **Run the migration script:**
You can run the script either locally or inside the Worker container:
**Option A: Run locally (if you have Python 3.11+ and dependencies installed):**
2. **Run the [migration script](https://github.com/langgenius/dify-docs/blob/main/assets/migrate_weaviate_collections.py)** either locally or inside the Worker container.\
**Option A: Run locally (if you have Python 3.11+ and dependencies installed)**:
```bash
python3 migrate_weaviate_collections.py
```
**Option B: Run inside Worker container (recommended for Docker setups):**
**Option B: Run inside Worker container (recommended for Docker setups)**:
```bash
# Copy script to storage directory
@@ -310,12 +298,10 @@ sleep 15
```
<Info>
The migration script uses environment variables for configuration, making
it suitable for running inside Docker containers. For Dify 1.11.0+, if you
encounter permission errors with `uv`, use `uv run --no-cache` instead.
The migration script uses environment variables for configuration, making it suitable for running inside Docker containers. For Dify 1.11.0+, if you encounter permission errors with `uv`, use `uv run --no-cache` instead.
</Info>
3. **Restart Dify services:**
3. **Restart Dify services**:
```bash
cd docker
@@ -323,32 +309,27 @@ sleep 15
sleep 15
```
4. **Verify in the UI:** open Dify, test retrieval against your migrated knowledge bases.
4. **Verify in the UI**: open Dify, test retrieval against your migrated knowledge bases.
<Warning>
For large collections (over 10,000 objects), verify that the object count
matches between old and new collections. The migration script will display
verification counts automatically.
For large collections (over 10,000 objects), verify that the object count matches between old and new collections. The migration script will display verification counts automatically.
</Warning>
<Info>
After confirming a healthy migration, you can delete `weaviate_migration_env`
and the backup files to reclaim disk space.
After confirming a healthy migration, you can delete `weaviate_migration_env` and the backup files to reclaim disk space.
</Info>
### Path B: Direct Recovery (Already on 1.27+)
<Warning>
Only use this path if you already upgraded to 1.27+ and your knowledge bases
stopped working. You cannot create a 1.19 backup anymore, so you must repair
the data in place.
Only use this path if you already upgraded to 1.27+ and your knowledge bases stopped working. You cannot create a 1.19 backup anymore, so you must repair the data in place.
</Warning>
#### Prerequisites
- Currently running Weaviate 1.27+ (including 1.33)
- Docker + Docker Compose installed
- Python 3.11+ for the migration script
- Python 3.11+ for the [migration script](https://github.com/langgenius/dify-docs/blob/main/assets/migrate_weaviate_collections.py)
#### Step B1: Repair Orphaned LSM Data
@@ -412,9 +393,9 @@ curl -s -H "Authorization: Bearer <WEAVIATE_API_KEY>" \
#### Step B2: Run the Schema Migration
Follow the same commands as [Step A5](#step-a5-migrate-the-schema). You can run the script locally or inside the Worker container:
Follow the same commands as [Step A5](#step-a5%3A-migrate-the-schema). You can run the script locally or inside the Worker container:
**To run inside Worker container:**
**To run inside Worker container**:
```bash
# Copy script to storage directory
@@ -446,23 +427,23 @@ docker compose restart api worker worker_beat
## Data Migration for Legacy Versions
<Warning>
### CRITICAL: Data Migration Required
**CRITICAL: Data Migration Required**
**Your existing knowledge bases will NOT work after upgrade without migration!**
### Why Migration is Needed:
**Why Migration is Needed**:
- Old data: Created with Weaviate v3 client (simple schema)
- New code: Requires Weaviate v4 format (extended schema)
- **Incompatible**: Old data missing required properties
### Migration Options:
**Migration Options**:
##### Option A: Use Weaviate Backup/Restore
- Option A: Use Weaviate Backup/Restore
##### Option B: Re-index from Original Documents
- Option B: Re-index from Original Documents
##### Option C: Keep Old Weaviate (Don't Upgrade Yet) If you can't afford downtime or data loss.
- Option C: Keep Old Weaviate (Don't Upgrade Yet) If you can't afford downtime or data loss.
</Warning>
@@ -500,9 +481,7 @@ curl -X POST "http://localhost:8080/v1/backups/filesystem/pre-migration-backup/r
```
<Info>
For comprehensive migration guidance, especially for complex schemas or large
datasets, refer to the official [Weaviate Migration
Guide](https://weaviate.io/developers/weaviate/installation/migration).
For comprehensive migration guidance, especially for complex schemas or large datasets, refer to the official [Weaviate Migration Guide](https://weaviate.io/developers/weaviate/installation/migration).
</Info>
## Configuration Changes
@@ -513,16 +492,16 @@ The following new environment variable is available in Dify versions with weavia
#### WEAVIATE_GRPC_ENDPOINT
**Description:** Specifies the gRPC endpoint for Weaviate connections. Using gRPC significantly improves performance for batch operations and queries.
**Description**: Specifies the gRPC endpoint for Weaviate connections. Using gRPC significantly improves performance for batch operations and queries.
**Format:** `hostname:port` (NO protocol prefix)
**Format**: `hostname:port` (NO protocol prefix)
**Default Ports:**
**Default Ports**:
- Insecure: 50051
- Secure (TLS): 443
**Examples:**
**Examples**:
```bash
# Docker Compose (internal network)
@@ -539,18 +518,17 @@ WEAVIATE_GRPC_ENDPOINT=your-instance.weaviate.cloud:443
```
<Warning>
Do NOT include protocol prefixes like `grpc://` or `http://` in the
WEAVIATE_GRPC_ENDPOINT value. Use only `hostname:port`.
Do NOT include protocol prefixes like `grpc://` or `http://` in the WEAVIATE_GRPC_ENDPOINT value. Use only `hostname:port`.
</Warning>
### Updated Environment Variables
All existing Weaviate environment variables remain the same:
- **WEAVIATE_ENDPOINT:** HTTP endpoint for Weaviate (e.g., `http://weaviate:8080`)
- **WEAVIATE_API_KEY:** API key for authentication (if enabled)
- **WEAVIATE_BATCH_SIZE:** Batch size for imports (default: 100)
- **WEAVIATE_GRPC_ENABLED:** Enable/disable gRPC (default: true in v4)
- **WEAVIATE_ENDPOINT**: HTTP endpoint for Weaviate (e.g., `http://weaviate:8080`)
- **WEAVIATE_API_KEY**: API key for authentication (if enabled)
- **WEAVIATE_BATCH_SIZE**: Batch size for imports (default: 100)
- **WEAVIATE_GRPC_ENABLED**: Enable/disable gRPC (default: true in v4)
### Complete Configuration Example
@@ -607,8 +585,7 @@ Look for messages indicating successful connection without "No module named 'wea
6. Check that status changes from "QUEUING" → "INDEXING" → "AVAILABLE"
<Info>
If documents get stuck in "QUEUING" status, check that the Celery worker is
running: `docker compose logs worker`
If documents get stuck in "QUEUING" status, check that the Celery worker is running: `docker compose logs worker`.
</Info>
### 4. Test Vector Search
@@ -631,17 +608,16 @@ docker compose logs -f api | grep -i "query_time\|duration"
```
<Info>
With gRPC properly configured, vector search queries should be 2-5x faster
compared to HTTP-only connections.
With gRPC properly configured, vector search queries should be 2-5x faster compared to HTTP-only connections.
</Info>
## Troubleshooting
### Issue: "No module named 'weaviate.classes'"
**Cause:** The weaviate-client v4 is not installed, or v3 is still being used.
**Cause**: The weaviate-client v4 is not installed, or v3 is still being used.
**Solution:**
**Solution**:
```bash
# For Docker installations, ensure you're running the correct Dify version
@@ -656,14 +632,14 @@ pip install weaviate-client==4.17.0
### Issue: Connection Refused on gRPC Port (50051)
**Cause:** Port 50051 is not exposed, not accessible, or Weaviate is not listening on it.
**Cause**: Port 50051 is not exposed, not accessible, or Weaviate is not listening on it.
**Solution:**
**Solution**:
1. **For Docker Compose users with bundled Weaviate:**
1. **For Docker Compose users with bundled Weaviate**:
The port is available internally between containers. No action needed unless you're connecting from outside Docker.
2. **For external Weaviate:**
2. **For external Weaviate**:
```bash
# Check if Weaviate is listening on 50051
@@ -674,7 +650,7 @@ pip install weaviate-client==4.17.0
docker run -p 8080:8080 -p 50051:50051 ...
```
3. **Check firewall rules:**
3. **Check firewall rules**:
```bash
# Linux
@@ -686,9 +662,9 @@ pip install weaviate-client==4.17.0
### Issue: Authentication Errors (401 Unauthorized)
**Cause:** API key mismatch or authentication configuration issue.
**Cause**: API key mismatch or authentication configuration issue.
**Solution:**
**Solution**:
1. Verify API key matches in both Weaviate and Dify:
@@ -714,9 +690,9 @@ pip install weaviate-client==4.17.0
### Issue: Documents Stuck in "QUEUING" Status
**Cause:** Celery worker not running or not connected to Redis.
**Cause**: Celery worker not running or not connected to Redis.
**Solution:**
**Solution**:
```bash
# Check if worker is running
@@ -735,9 +711,9 @@ docker compose restart worker
### Issue: Slow Performance After Migration
**Cause:** gRPC not enabled or configured incorrectly.
**Cause**: gRPC not enabled or configured incorrectly.
**Solution:**
**Solution**:
1. Verify gRPC configuration:
@@ -763,9 +739,9 @@ docker compose restart worker
### Issue: Schema Migration Errors
**Cause:** Incompatible schema changes between Weaviate versions or corrupted data.
**Cause**: Incompatible schema changes between Weaviate versions or corrupted data.
**Solution:**
**Solution**:
1. Check Weaviate logs for specific error messages:
@@ -792,15 +768,14 @@ docker compose restart worker
```
<Warning>
Deleting collections removes all data. Only do this if you have a backup and
are prepared to re-index all content.
Deleting collections removes all data. Only do this if you have a backup and are prepared to re-index all content.
</Warning>
### Issue: Docker Volume Permission Errors
**Cause:** User ID mismatch in Docker containers.
**Cause**: User ID mismatch in Docker containers.
**Solution:**
**Solution**:
```bash
# Check ownership of Weaviate data directory
@@ -815,9 +790,9 @@ docker compose restart weaviate
### Issue: Permission Denied When Running Migration Script (Dify 1.11.0+)
**Cause:** The `/home/dify` directory may not exist in newer Dify versions, causing `uv` cache creation to fail.
**Cause**: The `/home/dify` directory may not exist in newer Dify versions, causing `uv` cache creation to fail.
**Solution:**
**Solution**:
```bash
# Option 1: Use --no-cache flag (recommended)
@@ -878,8 +853,7 @@ docker compose logs | grep -i error
```
<Info>
Always test the rollback procedure in a staging environment first if possible.
Maintain multiple backup copies before attempting major migrations.
Always test the rollback procedure in a staging environment first if possible. Maintain multiple backup copies before attempting major migrations.
</Info>
## Additional Resources
@@ -889,15 +863,15 @@ docker compose logs | grep -i error
- [Weaviate Migration Guide](https://weaviate.io/developers/weaviate/installation/migration)
- [Weaviate v4 Client Documentation](https://weaviate.io/developers/weaviate/client-libraries/python)
- [Weaviate Backup and Restore](https://weaviate.io/developers/weaviate/configuration/backups)
- [Dify Self-Hosting Guide](/en/getting-started/install-self-hosted/docker-compose)
- [Dify Environment Variables](/en/getting-started/install-self-hosted/environments)
- [Dify Self-Hosting Guide](/en/self-host/quick-start/docker-compose)
- [Dify Environment Variables](/en/self-host/configuration/environments)
### Community Resources
- [Dify GitHub Repository](https://github.com/langgenius/dify)
- [Dify GitHub Issues - Weaviate](https://github.com/langgenius/dify/issues?q=is%3Aissue+weaviate)
- [Weaviate Community Forum](https://forum.weaviate.io/)
- [Dify Discord Community](https://discord.gg/dify)
- [Dify Community Forum](https://forum.dify.ai/)
### Migration Tools
@@ -908,18 +882,16 @@ docker compose logs | grep -i error
This migration brings important improvements to Dify's vector storage capabilities:
**Better Performance:** gRPC support dramatically improves query and import speeds (2-5x faster)
- **Better Performance**: gRPC support dramatically improves query and import speeds (2-5x faster)
**Improved Stability:** Enhanced connection handling and error recovery
- **Improved Stability**: Enhanced connection handling and error recovery
**Security:** Access to security updates and patches not available in Weaviate 1.19.0
- **Security**: Access to security updates and patches not available in Weaviate 1.19.0
**Future-Proof:** Access to latest Weaviate features and ongoing support
- **Future-Proof**: Access to latest Weaviate features and ongoing support
While this is a breaking change requiring server upgrade for users on old versions, the benefits significantly outweigh the migration effort. Most Docker Compose users can complete the migration in under 15 minutes with the automatic update.
<Info>
If you encounter any issues not covered in this guide, please report them on
the [Dify GitHub Issues page](https://github.com/langgenius/dify/issues) with
the label "weaviate" and "migration".
If you encounter any issues not covered in this guide, please report them on the [Dify GitHub Issues page](https://github.com/langgenius/dify/issues) with the label "weaviate" and "migration".
</Info>