🚀 Edge Multimodal Embeddings

Production-grade multimodal embedding system that runs on edge devices.
Handles text, images, video, audio, PDFs, code, and 40+ file formats — all under 4GB VRAM.

┌─────────────────────────────────────────────────────────────────────┐
│                    EdgeMultimodalEmbedder                           │
│                                                                     │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐           │
│  │   Text   │  │  Vision  │  │  Audio   │  │  Video   │           │
│  │ (Nomic)  │  │ (Nomic)  │  │  (CLAP)  │  │(VideoMAE)│           │
│  │  768d    │  │  768d    │  │  512d    │  │  384d    │           │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘           │
│       │              │             │              │                 │
│       └──────────────┴──────┬──────┴──────────────┘                 │
│                             │                                       │
│              ┌──────────────┴──────────────┐                        │
│              │   Projection Heads → 512d   │                        │
│              └──────────────┬──────────────┘                        │
│                             │                                       │
│                   L2-Normalized 512d                                │
└─────────────────────────────────────────────────────────────────────┘

✨ Key Features

📊 Model Stack

Text ↔ Image share the same 768d space via Nomic. No projection needed for text-image retrieval!

🏗️ Hardware Profiles

from edge_multimodal_embeddings import EdgeMultimodalEmbedder

# Quick start — auto-selects profile
embedder = EdgeMultimodalEmbedder.from_profile("edge_gpu")

# Profiles:
# "phone"        → 384d, INT4, 1.5GB budget, model swapping, MiniLM text
# "edge_gpu"     → 512d, INT8, 3.5GB budget, all models loaded
# "workstation"  → 768d, FP16, 8GB budget, no quantization
# "raspberry_pi" → 256d, INT4, 1GB budget, model swapping, no audio
# "jetson"       → 512d, FP16, 3GB budget

🚀 Quick Start

Install

pip install torch transformers Pillow numpy
# Optional: pip install onnxruntime onnx librosa soundfile opencv-python pymupdf

Embed Anything

from edge_multimodal_embeddings import EdgeMultimodalEmbedder

embedder = EdgeMultimodalEmbedder.from_profile("edge_gpu")

# Text
text_emb = embedder.embed("The quick brown fox")

# Image (auto-detected from file extension)
img_emb = embedder.embed("photo.jpg")

# Video
vid_emb = embedder.embed("clip.mp4")

# Audio
audio_emb = embedder.embed("song.wav")

# PDF (extracts text + images, embeds both)
pdf_emb = embedder.embed("paper.pdf")

# Code
code_emb = embedder.embed("main.py", modality="document")

# Cross-modal similarity
similarity = embedder.similarity(text_emb, img_emb)
print(f"Text-Image similarity: {similarity:.4f}")

Batch Embedding

# Mixed modality batch
inputs = ["Hello world", "photo.jpg", "clip.mp4"]
embeddings = embedder.embed_batch(inputs)
# → shape: (3, 512)

Phone Deployment

# Step 1: Export on workstation
embedder = EdgeMultimodalEmbedder.from_profile("phone")
embedder.export_onnx("./phone_models")

# Step 2: Run on phone (only needs onnxruntime + numpy)
from edge_multimodal_embeddings.runtime import EdgeRuntime

runtime = EdgeRuntime("./phone_models", num_threads=4)
emb = runtime.embed_text("Find photos of cats")
img_emb = runtime.embed_image("cat.jpg")
sim = runtime.similarity(emb, img_emb)

🔧 Custom Configuration

from edge_multimodal_embeddings.core.config import (
    EmbedderConfig, ModelConfig, QuantizationMode
)

config = EmbedderConfig(
    unified_dim=512,
    device="cuda",
    quantization=QuantizationMode.INT8_DYNAMIC,
    max_memory_mb=3500,
    lazy_load=True,
    model_swapping=False,
    max_batch_size=16,
    
    # Swap in different models
    text_model=ModelConfig(
        model_id="sentence-transformers/all-MiniLM-L6-v2",
        embedding_dim=384,
        max_input_size=512,
    ),
    # Disable modalities you don't need
    audio_model=ModelConfig("laion/clap-htsat-unfused", 512, enabled=False),
    video_model=ModelConfig("MCG-NJU/videomae-small-finetuned-kinetics", 384, enabled=False),
)

embedder = EdgeMultimodalEmbedder(config)

📁 Supported File Formats

Plus: PIL Images, numpy arrays, torch tensors, base64 strings, URLs, raw bytes.

🧠 Architecture Details

Why This Model Stack?

1. Nomic text + vision share a native 768d embedding space — no projection needed for text↔image retrieval, highest accuracy per byte.
2. CLAP is the only Apache-2.0 audio-text model with proper zero-shot capabilities.
3. VideoMAE-Small is remarkably tiny (84MB) while achieving 79% top-1 on Kinetics-400.
4. Projection heads are single Linear + LayerNorm layers (~0.5MB each) that unify all spaces to 512d.

Quantization Strategy

Based on MobileQuant (2408.13933) and EdgeVL (2403.04908):

• Dynamic INT8 (default for edge): Weights quantized, activations computed in FP32. <1% cosine similarity loss. No calibration needed.
• Static INT8: Both weights and activations quantized. Needs calibration data. Best throughput.
• INT4: 8× memory reduction. 1-3% quality loss. For extreme edge (phones, RPi).
• Per-tensor quantization for mobile NPU compatibility (not per-token).

Memory Management

• Lazy loading: Models loaded on first use, not at init
• LRU swapping: When memory budget is exceeded, least-recently-used model is evicted
• Adaptive batching: Automatically reduces batch size on OOM and retries
• Real-time monitoring: Track memory usage via embedder.get_status()

📱 Mobile/Edge Deployment

Android (ONNX Runtime Mobile)

// build.gradle
implementation 'com.microsoft.onnxruntime:onnxruntime-mobile:1.17.0'

// Load model
val session = env.createSession(modelBytes, SessionOptions().apply {
    setIntraOpNumThreads(4)
    addNnapi()  // Use Android Neural Networks API
})

iOS (CoreML)

// Use ONNX → CoreML conversion or CoreML EP
let config = MLModelConfiguration()
config.computeUnits = .all  // CPU + GPU + ANE

Raspberry Pi / ARM Linux

pip install onnxruntime  # ARM64 wheel available
python -c "from edge_multimodal_embeddings.runtime import EdgeRuntime; ..."

📊 Benchmarks

🔗 CLI

# Embed
edge-embed embed "Hello world"
edge-embed embed photo.jpg
edge-embed embed --modality video clip.mp4

# Compare
edge-embed compare "a cat" "a kitten"

# Benchmark
edge-embed benchmark -n 100

# Export
edge-embed export --format onnx -o ./exported/

# System info
edge-embed info

📄 License

Apache-2.0. All component models are Apache-2.0 or MIT licensed.

🙏 Credits

• Nomic AI — nomic-embed-text/vision-v1.5
• LAION — CLAP audio embeddings
• MCG-NJU — VideoMAE
• MobileQuant — Mobile quantization research
• EdgeVL — Edge visual-language models
• MobileCLIP — Architecture inspiration