Local Open-Source LLMs

Why Run LLMs Locally?

Cloud APIs are the fastest way to get started, but local models become attractive once any of these apply:

The Five LLM Roles in Power RAG

Before picking a model, identify which role it needs to fill. Each role has a different demand profile:

Role 1 — Embedding Models

# You would remove OllamaEmbeddingAutoConfiguration from spring.autoconfigure.exclude,
# then align dimensions, e.g.:
spring:
  ai:
    ollama:
      embedding:
        options:
          model: mxbai-embed-large
    vectorstore:
      qdrant:
        dimensions: 1024

Role 2 — Safety Guardrail Models

Shipped default: Gemini 2.5 Flash (gemini-2.5-flash) — fast, low temperature, and suitable for a short rubric response. The table below lists local Ollama classifiers if you replace the geminiGuard client with an Ollama-based flow.

// inputModelId defaults to gemini-2.5-flash; override with POWERRAG_GUARDRAIL_MODEL
.options(GoogleGenAiChatOptions.builder()
        .model(inputModelId)
        .temperature(0.0)
        .build())

Role 3 — General RAG Chat Models

The general chat model handles document Q&A — the core use case of the application. Power RAG sends it up to 24,000 characters of context (retrieved document chunks) plus the user's question. The model must follow citation instructions ([SOURCE N] inline) and maintain coherence across the full context window.

Role 4 — Text-to-SQL Models

Text-to-SQL is the most demanding task for a local model. TextToSqlService sends a full schema description (table names, column types, enum values) plus a natural language question, and expects a correct, syntactically valid PostgreSQL SELECT statement in return. Errors are immediately visible to the user as execution failures.

// Current: uses Gemini 2.5 Pro (cloud)
public TextToSqlService(SchemaIntrospector schemaIntrospector,
                        SqlValidator sqlValidator,
                        JdbcTemplate jdbcTemplate,
                        @Qualifier("geminiPro") ChatClient chatClient) {

// To use local deepseek-coder-v2:16b instead:
//   1. Register a dedicated bean in SpringAiConfig.java:
//      @Bean @Qualifier("ollamaDeepSeekSql")
//      public ChatClient ollamaDeepSeekSqlClient(OllamaChatModel model) {
//          return ChatClient.builder(model).build();
//      }
//   2. Change qualifier here:
//        @Qualifier("ollamaDeepSeekSql") ChatClient chatClient
//   3. Override model at call time:
//        .options(OllamaChatOptions.builder()
//            .model("deepseek-coder-v2:16b").build())
}

Role 5 — Image Description (Vision Models)

ImageParser uses a multimodal LLM to generate text descriptions of image files during document ingestion. The description becomes a text chunk that is embedded and indexed — making image content searchable via RAG. This role runs only during ingestion, not on every query.

Hardware Requirements

GPU VRAM Tiers

Apple Silicon (Unified Memory)

Apple M-series chips use unified memory shared between CPU and GPU. This is excellent for LLMs because the full memory pool is available for model weights:

RAM (System Memory) — CPU-Only

If no dedicated GPU is available, Ollama falls back to CPU inference. Minimum requirements:

Recommended Configurations by Use Case

Configuration A — Minimum Viable (Developer Laptop, No Dedicated GPU)

# Default stack: set GOOGLE_API_KEY (embeddings + input guard + Gemini chat as needed)
# Ollama optional — only if you want local chat models, e.g.:
# ollama pull qwen2.5-coder:7b

# Hybrid: cloud embeddings/guardrails + local Ollama chat — no nomic/llama-guard pulls required

Configuration B — Balanced Local (16 GB VRAM / M3 Pro 32 GB)

# If fully local beyond chat, add your own embedding + guard Ollama models and reconfigure.
ollama pull qwen2.5:14b             # General RAG chat (primary)
ollama pull deepseek-coder-v2:16b   # Text-to-SQL
ollama pull qwen2.5vl:7b            # Image description (ingestion)

Full local isolation requires replacing Google embedding and guardrail beans, not just emptying API keys. For chat-only local default while keeping Google embeddings/guardrails, point Ollama at your preferred model:

spring:
  ai:
    ollama:
      base-url: http://localhost:11434
      chat:
        options:
          model: qwen2.5:14b
    google:
      genai:
        api-key: ${GOOGLE_API_KEY}  # still required for gemini-embedding-001 + guardrails

Configuration C — Maximum Quality (24 GB VRAM / M3 Max 64 GB)

ollama pull qwen2.5:32b              # General RAG chat
ollama pull qwen2.5-coder:32b        # Text-to-SQL (excellent accuracy)
ollama pull qwen2.5vl:7b             # Image description

Performance Tuning for User Experience

A RAG response chains several steps: input guard (Gemini by default) → cache lookup (embedding API) → retrieval (embedding again) → main LLM. When using Ollama for chat, keep models warm — latency compounds.

services:
  ollama:
    environment:
      OLLAMA_KEEP_ALIVE: "24h"   # keep all recently-used models in VRAM

powerrag:
  models:
    cache:
      similarity-threshold: 0.88   # was 0.92 — more aggressive caching

Quick Reference — Role to Model Mapping