Image to Text AI

Image to Text AI

Posted on by Toma Velev

There are open-source (or freely usable) AI or OCR systems that convert image to text. What “image to text” exactly means can vary (e.g. OCR, image captioning, scene text recognition). Here are some good ones and pointers:

Types of “Image → Text” tasks

Before listing, it’s useful to distinguish:

  1. OCR (Optical Character Recognition) — extracting text characters/words that are present in the image (e.g. a scanned page, signboard).
  2. Image captioning / vision → description — generating descriptive sentences about “what’s in the image,” not just extracting printed text.

Which of these do you mean? I’ll list examples for both.

Open-source OCR / text extraction models & libraries

These are some of the well-known, open or free tools for extracting text from images:

Name Description / Features
Tesseract OCR Probably the most widely used open-source OCR engine. Supports many languages. (GitHub)
EasyOCR Python library built on PyTorch. Supports ~80+ languages. (Modal)
PaddleOCR OCR toolkit from the PaddlePaddle ecosystem; supports layout-aware OCR and works well with structured documents. (Modal)
docTR A document recognition library, with components for detection + recognition of text. (OpenNews)
GOCR Older open-source OCR; works with basic fonts; less accurate on messy or complex images. (Wikipedia)
CuneiForm OCR Opened source; supports multiple languages (including Bulgarian) for document/print text extraction. (Wikipedia)

Open / research models for vision → text (captioning / multimodal models)

If your goal is generation of descriptions (not just reading printed words), there are open models and models on repositories like Hugging Face:

  • On Hugging Face, you can filter for “image-to-text” models. (Hugging Face)
  • The paper GIT: Generative Image-to-text Transformer describes a unified vision + text model for tasks like captioning, scene text recognition, etc. Its code is released. (arXiv)
  • Some recent document understanding models (which combine layout, visual, and text) are published as open research. For example, a “Universal Document Processing” model that unifies vision, text, layout. (arXiv)

✅ Here are the best open-source generative image-to-text models you can use right now:

🧠 Top Open-Source Image → Text Models (Captioning / Vision-Language)

Model Description Licence Notes / Links
BLIP / BLIP-2 (Salesforce) State-of-the-art open models for image captioning, visual question answering, etc. BLIP-2 uses a pre-trained image encoder (ViT) + language model (e.g. Flan-T5). MIT 📦 BLIP GitHub — 📦 BLIP-2 on Hugging Face
GIT (Generative Image-to-Text Transformer) (Microsoft) Unified transformer for captioning and VQA. Good performance, simple interface. MIT 📦 GIT GitHub — Hugging Face GIT models
OFA (One For All) (Microsoft Research) Multi-modal model for captioning, VQA, OCR, etc. Open source and flexible. MIT 📦 OFA GitHub
Kosmos-2 (Microsoft) Vision-language model that can describe images and ground phrases to regions. Slightly heavier but more powerful. MIT 📦 Kosmos-2 GitHub
PaLI & PaLI-Gemma (Google DeepMind) Open weights for smaller PaLI-Gemma models are now on Hugging Face. Great for multilingual captioning. Ajax 2.0 📦 PaLI-Gemma models

⚙️ Example: BLIP-2 (Python, Hugging Face)

You can use it with just a few lines of code:

from transformers import Blip2Processor, Blip2ForConditionalGeneration
from PIL import Image
import torch

# Load model and processor
processor = Blip2Processor.from_pretrained("Salesforce/blip2-flan-t5-xl")
model = Blip2ForConditionalGeneration.from_pretrained("Salesforce/blip2-flan-t5-xl", torch_dtype=torch.float16).to("cuda")

# Load image
img = Image.open("your_image.jpg")

# Generate caption
inputs = processor(img, return_tensors="pt").to("cuda", torch.float16)
out = model.generate(**inputs, max_new_tokens=50)
caption = processor.decode(out[0], skip_special_tokens=True)

print("Caption:", caption)

💡 This will output something like:

"A small golden retriever puppy playing with a red ball on the grass."

🧪 Where to Explore / Demo Online

📊 Tips for Choosing

  • Lightweight / easiest to run locally: GIT or BLIP (base).
  • Most accurate / powerful: BLIP-2 or Kosmos-2 (larger, but best quality captions).
  • Multilingual support: PaLI-Gemma.
  • General-purpose multimodal tasks (beyond captioning): OFA or Kosmos-2.

Would you like me to help you set up one locally (e.g. Python script you can run on your machine), or are you more interested in no-code tools / online demos?

Does any of these models have an. example – http server – accepting an image and returning text

Yes — several of these open-source models can be wrapped in a simple HTTP server to accept an image and return generated text. Some already have ready-to-use demo servers or example scripts, and others can be easily adapted with FastAPI, Flask, or Gradio. Let me break it down:

1️⃣ BLIP / BLIP-2

  • BLIP-base / BLIP-small is small enough to run locally on your RTX 2060.
  • Salesforce BLIP GitHub includes examples for inference, but not a full HTTP server out-of-the-box.
  • It’s easy to wrap with FastAPI or Gradio.

Example with FastAPI:

from fastapi import FastAPI, File, UploadFile
from PIL import Image
import torch
from transformers import BlipProcessor, BlipForConditionalGeneration

app = FastAPI()

device = "cuda" if torch.cuda.is_available() else "cpu"
processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base").to(device)

@app.post("/caption")
async def caption_image(file: UploadFile = File(...)):
    img = Image.open(file.file)
    inputs = processor(images=img, return_tensors="pt").to(device)
    out = model.generate(**inputs, max_new_tokens=50)
    caption = processor.decode(out[0], skip_special_tokens=True)
    return {"caption": caption}
  • Send a POST request to /caption with an image file, and you get back JSON:
{"caption": "A cat sitting on a windowsill looking outside."}

2️⃣ GIT (Generative Image-to-Text Transformer)

  • Hugging Face provides GIT models with examples.
  • You can wrap them in Gradio, which creates an HTTP endpoint automatically.

Example with Gradio:

import gradio as gr
from transformers import AutoProcessor, AutoModelForCausalLM
from PIL import Image
import torch

processor = AutoProcessor.from_pretrained("microsoft/git-base-captioning")
model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-captioning").to("cuda")

def generate_caption(image):
    inputs = processor(images=image, return_tensors="pt").to("cuda")
    out = model.generate(**inputs, max_new_tokens=50)
    return processor.decode(out[0], skip_special_tokens=True)

gr.Interface(fn=generate_caption, inputs=gr.Image(type="pil"), outputs="text").launch(server_name="0.0.0.0")
  • This gives you a web server accessible from your LAN, returning captions for uploaded images.

3️⃣ OFA (One For All)

  • OFA GitHub provides scripts for image captioning, VQA, and more.
  • Can be wrapped in FastAPI / Flask exactly like BLIP.
  • Works well with 12 GB VRAM if you pick OFA-base.

⚡ Recommendations for Your Setup (RTX 2060 + DDR3)

  1. BLIP-base / BLIP-small – easiest, runs well, good quality captions.
  2. GIT-base – slightly heavier, works well with fp16.
  3. Use Gradio if you want a quick HTTP server without coding FastAPI endpoints.
  4. Resize images (224–384 px) to save GPU memory and speed up inference.