milvus迁移及管理

https://github.com/zilliztech/vts

Prerequisites

• Docker installed
• Access to source and target databases
• Required credentials and permissions
• Milvus Version >= 2.3.6

Quick Start

1. Pull the VTS Image

docker pull zilliz/vector-transport-service:latest
docker run -it zilliz/vector-transport-service:latest /bin/bash

2. Configure Your Migration Create a configuration file (e.g., migration.conf):

env {
  parallelism = 1
  job.mode = "BATCH"
}

source {
  # Source configuration (e.g., Milvus, Elasticsearch, etc.)
  Milvus {
    url = "https://your-source-url:19530"
    token = "your-token"
    database = "default"
    collections = ["your-collection"]
    batch_size = 100
  }
}

sink {
  # Target configuration
  Milvus {
    url = "https://your-target-url:19530"
    token = "your-token"
    database = "default"
    batch_size = 10
  }
}

3. Run the Migration

Cluster Mode (Recommended):

# Start the cluster
mkdir -p ./logs
./bin/seatunnel-cluster.sh -d

# Submit the job
./bin/seatunnel.sh --config ./migration.conf

Local Mode:

./bin/seatunnel.sh --config ./migration.conf -m local

Configuration Tips

• Adjust parallelism based on your data volume
• Configure appropriate batch_size for optimal performance
• Set up proper authentication and security measures
• Monitor system resources during migration

Supported Connectors

VTS supports various connectors for data migration:

• Milvus (example config)
• Elasticsearch (example config)
• Pinecone (example config)
• Qdrant (example config)
• Postgres Vector (example config)
• Tencent VectorDB (example config)

Advanced Features

For more advanced features, refer to our Tutorial.md and the Apache SeaTunnel Documentation:

• Transformers (TablePathMapper, FieldMapper, Embedding)
• Cluster mode deployment
• RESTful API for job management
• Docker deployment
• Advanced configuration options

Attu 管理工具

https://github.com/zilliztech/attu

System Requirements

• Docker 20.10.0 or later
• Kubernetes 1.19 or later (if using K8s deployment)
• Modern web browser (Chrome, Firefox, Safari, Edge)
• For desktop application:
  • Windows 10/11
  • macOS 10.15 or later
  • Linux (Ubuntu 20.04 or later)

Quick Start

1. Start Milvus server (if not already running):

docker run -d --name milvus_standalone -p 19530:19530 -p 9091:9091 milvusdb/milvus:latest

2. Start Attu:

docker run -p 8000:3000 -e MILVUS_URL=localhost:19530 zilliz/attu:v2.6

3. Open your browser and navigate to http://localhost:8000

Running Attu from Docker

Here are the steps to start a container for running Attu:

docker run -p 8000:3000 -e MILVUS_URL={milvus server IP}:19530 zilliz/attu:v2.6

Make sure that the Attu container can access the Milvus IP address. After starting the container, open your web browser and enter http://{ Attu IP }:8000 to view the Attu GUI.

Optional Environment Variables for Running Attu Docker

Parameter	Example	Required	Description
MILVUS_URL	192.168.0.1:19530	false	Optional, Milvus server URL
DATABASE	your database	false	Optional, default database name
ATTU_LOG_LEVEL	info	false	Optional, sets the log level for Attu
ROOT_CERT_PATH	/path/to/root/cert	false	Optional, path to the root certificate
PRIVATE_KEY_PATH	/path/to/private/key	false	Optional, path to the private key
CERT_CHAIN_PATH	/path/to/cert/chain	false	Optional, path to the certificate chain
SERVER_NAME	your_server_name	false	Optional, name of your server
SERVER_PORT	Server listen port	false	Optional, 3000 by default if unset

Please note that the MILVUS_URL should be an address that the Attu Docker container can access. Therefore, “127.0.0.1” or “localhost” will not work.

To run the Docker container with these environment variables, use the following command:

Attu SSL Example

docker run -p 8000:3000 \
-v /your-tls-file-path:/app/tls \
-e ATTU_LOG_LEVEL=info  \
-e ROOT_CERT_PATH=/app/tls/ca.pem \
-e PRIVATE_KEY_PATH=/app/tls/client.key \
-e CERT_CHAIN_PATH=/app/tls/client.pem \
-e SERVER_NAME=your_server_name \
zilliz/attu:v2.6

Custom Server Port Example

This command lets you run the docker container with host networking, specifying a custom port for the server to listen on

docker run --network host \
-v /your-tls-file-path:/app/tls \
-e ATTU_LOG_LEVEL=info  \
-e SERVER_NAME=your_server_name \
-e SERVER_PORT=8080 \
zilliz/attu:v2.6

Running Attu within Kubernetes

Before you begin, make sure that you have Milvus installed and running within your K8’s Cluster. Note that Attu only supports Milvus 2.x.

Here are the steps to start a container for running Attu:

kubectl apply -f https://raw.githubusercontent.com/zilliztech/attu/main/attu-k8s-deploy.yaml

索引：

from pymilvus import MilvusClient

# 创建Milvus Client。
client = MilvusClient(
    uri="http://c-xxxx.milvus.aliyuncs.com:19530",  # Milvus实例的公网地址。
    token="<yourUsername>:<yourPassword>",  # 登录Milvus实例的用户名和密码。
    db_name="default"  # 待连接的数据库名称，本文示例为默认的default。
)

index_params = MilvusClient.prepare_index_params()

index_params.add_index(
    field_name="vector",
    index_type="HNSW",
    metric_type="L2",
    params= {"M": 100,"efConstruction": 64}
)

client.create_index(
    collection_name="src_collection_name",
    index_params=index_params
)

client.load_collection(
    collection_name="src_collection_name"
)

from pymilvus import MilvusClient

# 创建Milvus Client。
client = MilvusClient(
    uri="http://c-xxxx.milvus.aliyuncs.com:19530",  # Milvus实例的公网地址。
    token="<yourUsername>:<yourPassword>",  # 登录Milvus实例的用户名和密码。
    db_name="default"  # 待连接的数据库名称，本文示例为默认的default。
)

index_params = MilvusClient.prepare_index_params()

index_params.add_index(
    field_name="vector",
    index_type="IVF_FLAT",
    metric_type="L2",
    params={"nlist": 1024}
 )

client.create_index(
    collection_name="src_collection_name",
    index_params=index_params
)

client.load_collection(
    collection_name="src_collection_name"
)