A high-performance vector database and search engine built in Rust, designed for semantic search, document indexing, and AI-powered applications.
- π Semantic Search: Advanced vector similarity with multiple distance metrics (Cosine, Euclidean, Dot Product)
- β‘ SIMD Acceleration: AVX2-optimized vector operations (5-10x faster) with automatic CPU feature detection
- πΎ Memory-Mapped Storage: MMap support for datasets larger than RAM with efficient OS paging
- π GPU Acceleration: Metal GPU support for macOS (Apple Silicon) with cross-platform compatibility
- π¦ Product Quantization: PQ compression for 64x memory reduction with minimal accuracy loss
- πΎ Compact Storage: Unified
.vecdbformat with 20-30% space savings and automatic snapshots - π Master-Replica Replication: High availability with automatic failover and SDK routing support
- π Distributed Sharding: Horizontal scaling across multiple servers with automatic shard routing
- βοΈ HiveHub Cluster Mode: Multi-tenant cluster deployment with HiveHub.Cloud
- Tenant isolation with user-scoped collections
- Quota enforcement (collections, vectors, storage)
- Usage tracking and reporting
- Memory limits and MMap storage enforcement
- Operation logging with cloud integration
- Comprehensive audit trail and analytics
- Tenant migration API (export, transfer, clone, cleanup)
- π Document Conversion: Automatic conversion of PDF, DOCX, XLSX, PPTX, HTML, XML, and images
- π Qdrant Migration: Complete migration tools and full Qdrant 1.14.x API compatibility
- Snapshots API (create, list, delete, recover)
- Sharding API (create shard keys, distribute data)
- Cluster Management API (status, recovery, peer management, metadata)
- Query API (query, batch query, grouped queries with prefetch)
- Search Groups and Matrix API (grouped results, similarity matrices)
- Named Vectors support (partial)
- Quantization configuration (PQ and Binary)
- π― MCP Integration: 26 focused individual tools for AI model integration
- π UMICP Protocol: Native JSON types + Tool Discovery endpoint
- π GraphQL API: Full GraphQL API with async-graphql
- Complete REST API parity with flexible queries
- GraphiQL playground for interactive exploration
- Mutations for collections, vectors, and search
- π₯οΈ Web Dashboard: Modern React + TypeScript dashboard with complete graph management interface
- JWT-based authentication with login page and session management
- Create/delete edges with relationship types and weights
- Explore node neighbors and related nodes
- Find shortest paths between nodes
- Node-specific edge discovery with configurable parameters
- Real-time graph visualization with vis-network
- π₯οΈ Desktop GUI: Electron-based desktop application with vis-network graph visualization for visual database management
- β‘ High Performance: Sub-3ms search times with HNSW indexing
- π§ Multiple Embeddings: TF-IDF, BM25, FastEmbed (production), BERT/MiniLM (real or placeholder), and custom models
- π Hybrid Search: Dense + Sparse search with Reciprocal Rank Fusion (RRF)
- π Smart Summarization: Extractive, keyword, sentence, and abstractive (OpenAI GPT) methods
- π TLS/SSL Security: Full TLS 1.2/1.3 support with mTLS, configurable cipher suites, and ALPN
- β‘ Rate Limiting: Per-API-key rate limiting with configurable tiers and overrides
- π Quantization Cache: Cache hit ratio tracking with comprehensive metrics
- πΈοΈ Graph Relationships: Automatic relationship discovery and graph traversal with full GUI support for edge management, node exploration, and path finding
- π n8n Integration: Official n8n community node for no-code workflow automation (400+ node integrations)
- π¨ Langflow Integration: LangChain-compatible components for visual LLM app building
- π Security: JWT + API Key authentication with RBAC
Installs Vectorizer CLI and configures it as a system service that starts automatically on boot:
curl -fsSL https://raw.githubusercontent.com/hivellm/vectorizer/main/scripts/install.sh | bashAfter installation:
- β
CLI available:
vectorizer --help - β
Service running:
sudo systemctl status vectorizer - β Auto-starts on boot
- β
Service commands:
sudo systemctl restart vectorizer- Restart servicesudo systemctl stop vectorizer- Stop servicesudo journalctl -u vectorizer -f- View logs
Installs Vectorizer CLI and configures it as a Windows Service that starts automatically on boot:
powershell -c "irm https://raw.githubusercontent.com/hivellm/vectorizer/main/scripts/install.ps1 | iex"Note: Service installation requires Administrator privileges. If not running as admin, the script will provide instructions.
After installation:
- β
CLI available:
vectorizer --help - β
Service running:
Get-Service Vectorizer - β Auto-starts on boot
- β
Service commands:
Restart-Service Vectorizer- Restart serviceStop-Service Vectorizer- Stop serviceStart-Service Vectorizer- Start service
Basic Docker Run (with authentication):
docker run -d \
--name vectorizer \
-p 15002:15002 \
-v $(pwd)/vectorizer-data:/vectorizer/data \
-e VECTORIZER_AUTH_ENABLED=true \
-e VECTORIZER_ADMIN_USERNAME=admin \
-e VECTORIZER_ADMIN_PASSWORD=admin \
-e VECTORIZER_JWT_SECRET=change-this-secret-in-production \
--restart unless-stopped \
hivehub/vectorizer:latestProduction Docker Run (with custom credentials):
docker run -d \
--name vectorizer \
-p 15002:15002 \
-v $(pwd)/vectorizer-data:/vectorizer/data \
-e VECTORIZER_AUTH_ENABLED=true \
-e VECTORIZER_ADMIN_USERNAME=admin \
-e VECTORIZER_ADMIN_PASSWORD=your-secure-password \
-e VECTORIZER_JWT_SECRET=your-jwt-secret-key \
--restart unless-stopped \
hivehub/vectorizer:latestUsing Docker Compose:
# Copy .env.example to .env and customize
cp .env.example .env
# Edit .env with your credentials
# Start with docker-compose
docker-compose up -dDefault Credentials (CHANGE IN PRODUCTION!):
- Username:
admin - Password:
admin - JWT Secret:
change-this-secret-in-production
Available at:
- π³ Docker Hub -
hivehub/vectorizer:latest - π¦ GitHub Container Registry -
ghcr.io/hivellm/vectorizer:latest
git clone https://github.com/hivellm/vectorizer.git
cd vectorizer
# Basic build
cargo build --release
./target/release/vectorizer
# With GPU acceleration (macOS Metal)
cargo build --release --features hive-gpu
# With all features
cargo build --release --features full- Web Dashboard: http://localhost:15002/dashboard/ - Modern React dashboard with complete graph management interface (create/delete edges, explore neighbors, find paths, discover relationships)
- REST API: http://localhost:15002
- MCP Server: http://localhost:15002/mcp
- UMICP Discovery: http://localhost:15002/umicp/discover
- Health Check: http://localhost:15002/health
When using Docker, authentication is enabled by default:
Default Credentials (
- Username:
admin - Password:
admin - Login Endpoint:
POST http://localhost:15002/auth/login
Authentication Example:
# Login to get JWT token
curl -X POST http://localhost:15002/auth/login \
-H "Content-Type: application/json" \
-d '{"username":"admin","password":"admin"}'
# Use JWT token in requests
curl -X GET http://localhost:15002/collections \
-H "Authorization: Bearer YOUR_JWT_TOKEN"API Key Authentication:
API Keys can be created in the dashboard (/api-keys) or via REST API for programmatic access.
Bearer prefix. Use them directly in the Authorization header:
# Create API key (requires JWT authentication)
curl -X POST http://localhost:15002/auth/keys \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_JWT_TOKEN" \
-d '{
"name": "Production API Key",
"permissions": ["read", "write"],
"expires_in_days": 90
}'
# Use API key in requests (NO Bearer prefix!)
curl -X GET http://localhost:15002/collections \
-H "Authorization: YOUR_API_KEY"
# MCP Configuration (mcp.json)
{
"mcpServers": {
"vectorizer": {
"command": "npx",
"args": ["-y", "@hivellm/mcp-vectorizer"],
"env": {
"VECTORIZER_API_URL": "http://localhost:15002",
"VECTORIZER_API_KEY": "YOUR_API_KEY"
}
}
}
}Authentication Methods Comparison:
| Method | Header Format | Use Case |
|---|---|---|
| JWT Token | Authorization: Bearer YOUR_JWT_TOKEN |
Dashboard, short-lived sessions |
| API Key | Authorization: YOUR_API_KEY |
MCP, CLI, long-lived integrations |
Production Security:
- Change default credentials using environment variables
- Use strong passwords (minimum 32 characters)
- Generate secure JWT secret (minimum 48 characters)
- See Docker Authentication Guide for details
- Review Security Policy for best practices
| Metric | Value |
|---|---|
| Search Speed | < 3ms (CPU), < 1ms (Metal GPU) |
| Storage Reduction | 30-50% with normalization |
| Test Coverage | 95%+ coverage |
| Test Suite | 1514 passing, 101 ignored |
| MCP Tools | 26 focused individual tools |
| Document Formats | 14 formats supported |
Comprehensive benchmark comparing Vectorizer with Qdrant across multiple scenarios:
- Search Performance: Vectorizer is 4-5x faster than Qdrant in all test scenarios
- Average latency: 0.16-0.23ms (Vectorizer) vs 0.80-0.87ms (Qdrant)
- Throughput: 4,400-6,000 queries/sec (Vectorizer) vs 1,100-1,300 queries/sec (Qdrant)
- Insert Performance: Optimized with fire-and-forget pattern for non-blocking operations
- Configurable batch sizes and request body limits
- Background processing prevents API blocking
- Test Scenarios: 5 comprehensive scenarios tested
- Small (1K vectors), Medium (5K vectors), Large (10K vectors) datasets
- Multiple dimensions: 384, 512, 768
- Full benchmark reports available in
docs/directory
See Benchmark Documentation for detailed performance metrics and how to run benchmarks.
Comprehensive feature comparison with major vector database solutions:
| Feature | Vectorizer | Qdrant | pgvector | Pinecone | Weaviate | Milvus | Chroma |
|---|---|---|---|---|---|---|---|
| Core | |||||||
| Language | Rust | Rust | C (PostgreSQL) | C++/Go | Go | C++/Go | Python |
| License | Apache 2.0 | Apache 2.0 | PostgreSQL | Proprietary | BSD 3-Clause | Apache 2.0 | Apache 2.0 |
| Deployment | Standalone/Embedded | Standalone | PostgreSQL Extension | Cloud/Self-hosted | Standalone | Standalone | Standalone |
| APIs & Integration | |||||||
| REST API | β Full | β Full | β (via PostgreSQL) | β Full | β Full | β Full | β Full |
| gRPC API | β Qdrant-compatible | β | β | β | β | β | β |
| GraphQL API | β Full with GraphiQL | β | β | β | β | β | β |
| MCP Integration | β 26 tools | β | β | β | β | β | β |
| n8n Integration | β Official node | β | β | β | β | β | β |
| Langflow Integration | β LangChain components | β | β | β | β | β | β |
| Python SDK | β | β | β | β | β | β | β |
| TypeScript SDK | β | β | β | β | β | β | β |
| JavaScript SDK | β | β | β | β | β | β | β |
| Rust SDK | β | β | β | β | β | β | β |
| C# SDK | β | β | β | β | β | β | β |
| Go SDK | β | β | β | β | β | β | β |
| Performance | |||||||
| Search Latency | < 3ms (CPU) < 1ms (GPU) |
~1-5ms | ~5-50ms | ~50-100ms | ~10-50ms | ~5-20ms | ~10-100ms |
| SIMD Acceleration | β AVX2 | β | β | β | β | β | β |
| GPU Support | β Metal (macOS) | β CUDA | β | β Cloud GPU | β | β CUDA | β |
| Storage & Indexing | |||||||
| HNSW Index | β | β | β | β | β | β | β |
| Product Quantization | β 64x compression | β | β | β | β | β | β |
| Scalar Quantization | β | β | β | β | β | β | β |
| Memory-Mapped Storage | β | β | β | β | β | β | β |
| Persistent Storage | β .vecdb format | β | β | β Cloud | β | β | β |
| Distance Metrics | |||||||
| Cosine Similarity | β | β | β | β | β | β | β |
| Euclidean Distance | β | β | β | β | β | β | β |
| Dot Product | β | β | β | β | β | β | β |
| Advanced Features | |||||||
| Graph Relationships | β Auto-discovery | β | β | β | β | β | β |
| Document Processing | β 14 formats | β | β | β | β | β | β |
| Multi-Collection Search | β | β | β | β | β | β | β |
| Hybrid Search | β Dense + Sparse | β | β | β | β | β | β |
| Semantic Reranking | β | β | β | β | β | β | β |
| Query Expansion | β | β | β | β | β | β | β |
| Embedding Providers | |||||||
| Built-in Embeddings | β TF-IDF, BM25, BERT, MiniLM | β | β | β | β | β | β |
| Custom Models | β | β | β | β | β | β | β |
| Scalability | |||||||
| Horizontal Sharding | β (BETA) | β | β (PostgreSQL) | β Cloud | β | β | β |
| Replication | β Master-Replica (BETA) | β | β (PostgreSQL) | β Cloud | β | β | β |
| Auto-scaling | β | β | β | β Cloud | β | β | β |
| Management & UI | |||||||
| Web Dashboard | β React + Full Graph UI | β Basic | β (pgAdmin) | β Cloud | β | β | β Basic |
| Desktop GUI | β Electron + vis-network | β | β | β | β | β | β |
| Graph Visualization | β vis-network + Full Controls | β | β | β | β | β | β |
| Graph Management | β Create/Delete Edges, Path Finding | β | β | β | β Basic | β | β |
| CLI Tools | β | β | β | β | β | β | β |
| Migration & Compatibility | |||||||
| Qdrant Compatibility | β Full API | β | β | β | β | β | β |
| Migration Tools | β Qdrant β Vectorizer | β | β | β | β | β | β |
| Security | |||||||
| Authentication | β JWT + API Keys | β | β (PostgreSQL) | β Cloud | β | β | β |
| RBAC | β | β | β (PostgreSQL) | β Cloud | β | β | β |
| Encryption at Rest | β | β | β (PostgreSQL) | β Cloud | β | β | β |
| Cost & Licensing | |||||||
| Open Source | β | β | β | β | β | β | β |
| Self-Hosted | β | β | β | β (Enterprise) | β | β | β |
| Cloud Hosted | β (HiveHub.Cloud) | β (Qdrant Cloud) | β (Various) | β | β (Weaviate Cloud) | β (Zilliz Cloud) | β |
| Free Tier | β Unlimited | β | β | β Limited | β | β | β |
Vectorizer Advantages:
- β MCP Integration: Native Model Context Protocol support with 20 focused tools
- β Graph Relationships: Automatic relationship discovery with complete GUI management (create/delete edges, path finding, neighbor exploration)
- β No-Code Integrations: Official n8n node and Langflow components for visual workflow/LLM app building
- β GraphQL API: Full GraphQL API with GraphiQL playground and complete REST parity
- β Document Processing: Built-in support for 14 document formats (PDF, Office, images)
- β Desktop GUI: Electron-based desktop application with vis-network graph visualization
- β Qdrant Compatibility: Full API compatibility + migration tools + gRPC support
- β Performance: 4-5x faster search than Qdrant in benchmarks
- β
Unified Storage: Compact
.vecdbformat with 20-30% space savings - β Complete SDK Coverage: 6 official SDKs (Python, TypeScript, JavaScript, Rust, C#, Go)
Best Use Cases:
- Vectorizer: AI applications requiring MCP integration, no-code workflows, document processing, graph relationships, and high-performance search
- Qdrant: Production-ready vector search with good performance and cloud options
- pgvector: PostgreSQL-based applications needing vector search alongside relational data
- Pinecone: Managed cloud solution with minimal infrastructure management
- Weaviate: Applications requiring GraphQL and built-in ML models
- Milvus: Large-scale deployments requiring advanced scalability features
- Chroma: Python-first applications with simple setup requirements
- β
Dashboard Authentication: Complete authentication system for the dashboard
- Login page with username/password form and modern UI
- JWT token-based authentication via
/auth/loginendpoint - Session persistence with localStorage and automatic route protection
- β
HiveHub Cluster Integration: Multi-tenant cluster mode support
HubManagerfor HiveHub API integration with tenant isolation- API key validation, quota enforcement, and usage tracking
- Request signing and IP whitelist support for security
- β
Cluster Memory Limits: Enforce predictable memory usage in cluster mode
- Global cache memory limit (default: 1GB)
- MMap storage enforcement and file watcher auto-disable
- Comprehensive configuration validator at startup
- β
MMap Storage Deadlock Fix: Fixed deadlock during concurrent vector insertions
- Removed internal
Arc<RwLock<>>wrapper for proper lock management - Stable concurrent insert operations without blocking
- Removed internal
- β Dashboard SPA Routing Fix: Browser refresh now works on all dashboard routes
- β File Watcher Improvements: Uses default collection instead of creating empty collections
- β Empty Collection Management: New endpoints to list and cleanup empty collections
- β Dashboard Cache Headers: Proper caching for faster dashboard loading
- β Graph Dashboard Enhancements: Complete graph management interface
- β n8n Integration: Official community node for workflow automation
- β Langflow Integration: LangChain-compatible components for RAG pipelines
- β GraphQL API: Full GraphQL API with async-graphql and GraphiQL playground
- β SDK Master/Replica Routing: Automatic read/write routing for high availability
- β All core tests passing: 1514+ tests with comprehensive coverage
- RAG Systems: Semantic search for AI applications with automatic document conversion
- Document Search: Intelligent indexing and retrieval of PDFs, Office files, and web content
- Code Analysis: Semantic code search and navigation
- Knowledge Bases: Enterprise knowledge management with multi-format support
Cursor IDE configuration:
{
"mcpServers": {
"vectorizer": {
"url": "http://localhost:15002/mcp",
"type": "streamablehttp"
}
}
}Available MCP Tools (26 tools):
list_collections,create_collection,get_collection_infoinsert_text,get_vector,update_vector,delete_vectorsearch,multi_collection_search
search_intelligent- AI-powered search with query expansionsearch_semantic- Semantic search with rerankingsearch_extra- Combined search using multiple strategiessearch_hybrid- Hybrid dense + sparse vector search
filter_collections,expand_queriesget_file_content,list_files,get_file_chunksget_project_outline,get_related_files
graph_list_nodes,graph_list_edges,graph_find_relatedgraph_create_edge,graph_delete_edgegraph_discover_edges,graph_discover_status
Note: Cluster management operations are available via REST API only for security reasons.
All SDKs are synchronized with server version 1.8.5:
- Python:
pip install vectorizer-sdk(v1.8.5) - TypeScript:
npm install @hivellm/vectorizer-sdk(v1.8.5) - Rust:
cargo add vectorizer-sdk(v1.8.5) - JavaScript:
npm install @hivellm/vectorizer-sdk-js(v1.8.5) - C#:
dotnet add package Vectorizer.SDK(v1.8.5) - Go:
go get github.com/hivellm/vectorizer/sdks/go(v1.8.5)
Official n8n community node for no-code workflow automation.
Installation:
npm install @vectorizer/n8n-nodes-vectorizerFeatures:
- Collection management (create, delete, get, list)
- Vector operations (insert, batch insert, delete, get)
- Search operations (vector, semantic, hybrid)
- 400+ n8n node integrations available
- Visual workflow builder
Example Workflow:
Document Loader β Vectorizer (Insert) β Trigger β Vectorizer (Search) β Response
See n8n Integration Guide for detailed usage.
LangChain-compatible components for visual LLM application building.
Installation:
pip install vectorizer-langflowComponents:
VectorizerVectorStore- Full LangChain VectorStore implementationVectorizerRetriever- RAG pipeline retrieverVectorizerLoader- Document loader for existing vectors
Example:
from vectorizer_langflow import VectorizerVectorStore
from langchain.embeddings import OpenAIEmbeddings
vectorstore = VectorizerVectorStore(
host="http://localhost:15002",
collection_name="docs",
embedding=OpenAIEmbeddings()
)
# Add documents
vectorstore.add_texts(["Document 1", "Document 2"])
# Search
results = vectorstore.similarity_search("query", k=5)See Langflow Integration Guide for detailed usage.
Vectorizer provides comprehensive migration tools to help you migrate from Qdrant:
- Configuration Migration: Parse and convert Qdrant config files (YAML/JSON) to Vectorizer format
- Data Migration: Export collections from Qdrant and import into Vectorizer
- Validation: Validate exported data, check compatibility, and verify integrity after migration
- REST API Compatibility: Full Qdrant REST API compatibility at
/qdrant/*endpoints
Quick Migration Example:
use vectorizer::migration::qdrant::{QdrantDataExporter, QdrantDataImporter};
// Export from Qdrant
let exported = QdrantDataExporter::export_collection(
"http://localhost:6333",
"my_collection"
).await?;
// Import into Vectorizer
let result = QdrantDataImporter::import_collection(&store, &exported).await?;See Qdrant Migration Guide for detailed instructions.
Vectorizer supports multi-tenant cluster mode integration with HiveHub.Cloud for managed deployment:
- Multi-Tenant Isolation: Each user's collections are isolated with owner-based filtering
- Quota Management: Collection count, vector count, and storage quotas enforced per tenant
- Usage Tracking: Automatic tracking and reporting of resource usage
- User-Scoped Backups: Create, download, and restore backups per user
Enable HiveHub integration in config.yml:
hub:
enabled: true
api_url: "https://api.hivehub.cloud"
tenant_isolation: "collection"
usage_report_interval: 300Set the service API key:
export HIVEHUB_SERVICE_API_KEY="your-service-api-key"For internal HiveHub requests:
# Bypass authentication
curl -H "x-hivehub-service: true" \
http://localhost:15002/collections
# With user context (tenant scoping)
curl -H "x-hivehub-service: true" \
-H "x-hivehub-user-id: <user-uuid>" \
http://localhost:15002/collectionsSee HiveHub Integration Guide for detailed documentation.
When running Vectorizer in cluster mode, the following requirements are enforced:
| Requirement | Description | Default |
|---|---|---|
| MMap Storage | Memory storage is not allowed; MMap is required | Enforced |
| Cache Limit | Maximum cache memory across all caches | 1GB |
| File Watcher | Automatically disabled in cluster mode | Disabled |
| Strict Validation | Server fails to start on config violations | Enabled |
Example cluster configuration:
cluster:
enabled: true
node_id: "node-1"
memory:
max_cache_memory_bytes: 1073741824 # 1GB
enforce_mmap_storage: true
disable_file_watcher: true
strict_validation: trueSee Cluster Memory Limits for detailed configuration and troubleshooting.
- User Documentation - Installation guides and user tutorials
- API Reference - Complete REST API documentation
- Dashboard Integration - Web dashboard setup and integration guide
- Qdrant Compatibility - Qdrant API compatibility and migration guide
- HiveHub Integration - Multi-tenant cluster mode with HiveHub.Cloud
- Cluster Memory Limits - Cluster mode memory management and validation
- Technical Specifications - Architecture, performance, and implementation guides
- MCP Integration - Model Context Protocol guide
Apache License 2.0 - See LICENSE for details
Contributions are welcome! Please see our Contributing Guide for details.