Applied AI
GitHub RepositoryMultimodal Semantic Search & RAG Engine
Search images and video with ordinary language, locally and with measurable retrieval results.
Context: Multimodal AI search system
Period: 2025-10 to Present
Summary
A FastAPI, React/Vite, and CLI retrieval system using CLIP embeddings, FAISS nearest-neighbour search, caption-aware reranking, and Ollama-powered local explanations.
From a natural-language query to ranked visual evidence
Query
A user describes the image or video they want to find
CLIP encoding
Text and media enter a shared embedding space
FAISS search
Nearest-neighbour indexes retrieve strong candidates
Reranking
Caption and retrieval signals refine the ordering
Results
The interface returns ranked media and explanations
What the system handled
Maps natural-language queries to ranked image and video results.
Uses CLIP text, image, and frame embeddings with FAISS nearest-neighbour search.
Supports image retrieval, video retrieval, text-to-video search, frame-level video search, and reverse video-to-text retrieval.
Uses Ollama for local explanation generation.
Measured retrieval benchmark
COCO-caption evaluation across 200 queries, with separate 1,000-image and 5,000-image index runs. Latency belongs to the evaluated 1,000-image run.
Recall@10
0.88
1,000-image index
Recall@10
0.57
5,000-image index
Recall@1 / @5 / @10
0.46 / 0.755 / 0.88
200 COCO-caption queries
Average query latency
6.92ms
1,000-image index
Technology
Stack
Areas
Retrieval benchmark
Committed benchmark results
How retrieval changes as the index grows
Switch between the evaluated index sizes to compare Recall@K and controlled query latency. The application latency below comes from a separate runtime log and includes more of the full workflow.Recall comparison
200 evaluation queriesRecall@10: 88% of relevant matches recovered
Application runtime context
209 logged events- Average logged search
- 68.07 ms
- Average local-LLM explanation
- 13.74 s
The 6.9 ms evaluation query and 68.07 ms logged search measure different paths, so each retains its own label and timing context.
Inspect the benchmark JSON files on GitHubConclusions
- A natural-language query can return ranked image or video evidence directly from the concepts a user remembers.
- Benchmark runs make retrieval quality and search latency visible, supporting objective comparisons between changes.
- The same workflow is available through a browser interface and a CLI, which makes experiments easier to repeat and inspect.
Want to inspect the implementation?
The repository contains the source, setup notes, and development history.