01
Document pages
Example query
“Find the chart showing retention by customer segment.”
Encode rendered pages with their text, charts and layout so visual context can contribute to retrieval.
OmniSpectraMultimodal embedding model
Across modalities. One semantic space.
OmniSpectra is a multimodal embedding model. It encodes documents, images, video, audio and text into comparable vectors for cross-modal retrieval and content reuse.
Documents, images, video, audio and text are inputs to OmniSpectra, which encodes them into vectors. Content text and query text use the same encoder and are related by meaning in a shared semantic space. The content and vector positions are illustrative.
Content Query example· Text
- Wind energy Documents
-
Image - Video
- “Wind power” Audio
- Turbine checks Text
Offshore wind power
Shared semantic space
Different content. Connected by meaning.
A text query can relate to a document page, an image, a video or audio. The model produces embeddings; your application uses vector similarity to retrieve related content.
02
Images + video
Example query
“A wind turbine operating at sea.”
Relate text to visual subjects, scenes and actions across images and video shots.
03
Audio + speech
Example query
“Find the part of the lecture explaining customer retention.”
Relate the meaning of spoken content and audible events to text queries for recordings, interviews and lectures.
Shot-level retrieval. English and Chinese queries.
Compare model versions on shot-level video retrieval using English and Chinese queries. Results apply to this task and the model versions shown.
- TASK
- Shot-level semantic retrieval
- METRIC
- Average(Recall@5, Recall@10)
- QUERY LANGUAGES
- English / Chinese
- MODEL SHOWN
- OmniSpectra 2.5 2605
-
Text description
-
Jina CLIP v2
-
TwelveLabs Marengo 2.7
-
TwelveLabs Marengo 3.0
-
Amazon Nova · 1024-dim
-
Amazon Nova · 3072-dim
-
OmniSpectra 2.0 preview
-
OmniSpectra 2.5
ON-DEVICE INFERENCE
OmniSpectra Edge
OmniSpectra Edge is the on-device version of OmniSpectra. It encodes multimodal content locally for retrieval workflows on your own devices.
- Local model inference
- Multimodal encoding
- Local file retrieval
Model API
Build into your retrieval system.
Use the API or SDKs to obtain embeddings for your existing vector database, similarity search or knowledge system. The model encodes content; your application organizes storage and retrieval.
See the API reference for available endpoints, input formats and request examples.
Evaluate on the work that matters.
Evaluate with representative documents, images and recordings. Check whether relevant content is found, then assess latency and hardware for your deployment.
Discuss an evaluation