mirror of
https://github.com/photoprism/photoprism.git
synced 2026-07-17 16:49:04 +00:00
- Default Service.Think to "false" for the Ollama engine via a new EngineInfo.DefaultThink applied in ApplyEngineDefaults; re-enable reasoning explicitly with Service.Think: "true". - Replace the retired cloud default qwen3-vl:235b-instruct-cloud with minimax-m3:cloud, and refresh the self-hosted default from gemma3:latest to gemma4:latest. - Warn on HTTP 404/410 (retired/renamed model) and other 4xx statuses in ollamaParser.Parse so failures are diagnosable instead of silent. - Defensively strip a leading, tag-delimited <think>...</think> block from the response body. - Update tests, regenerate testdata/vision.yml, and refresh package docs.
This commit is contained in:
Michael Mayer
parent
017e24927e
commit
311929ab27
13 changed files with 223 additions and 42 deletions
|
|
@ -1,13 +1,13 @@
|
|||
## PhotoPrism — Vision Package
|
||||
|
||||
**Last Updated:** May 21, 2026
|
||||
**Last Updated:** July 16, 2026
|
||||
|
||||
### Overview
|
||||
|
||||
`internal/ai/vision` provides the shared model registry, request builders, and parsers that power PhotoPrism’s caption, label, face, NSFW, and future generate workflows. It reads `vision.yml`, normalizes models, and dispatches calls to one of three engines:
|
||||
|
||||
- **TensorFlow (built‑in)** — default Nasnet / NSFW / Facenet models, no remote service required. Long-running TensorFlow inference can accumulate C-allocated tensor memory until GC finalizers run, so PhotoPrism periodically triggers garbage collection to return that memory to the OS; tune with `PHOTOPRISM_TF_GC_EVERY` (default **200**, `0` disables). Lower values reduce peak RSS but increase GC overhead and can slow indexing, so keep the default unless memory pressure is severe.
|
||||
- **Ollama** — local or proxied multimodal LLMs. See [`ollama/README.md`](ollama/README.md) for tuning and schema details. The engine defaults to `${OLLAMA_BASE_URL:-http://ollama:11434}/api/generate`, trimming any trailing slash on the base URL; set `OLLAMA_BASE_URL=https://ollama.com` to opt into cloud defaults.
|
||||
- **Ollama** — local or proxied multimodal LLMs. See [`ollama/README.md`](ollama/README.md) for tuning and schema details. The engine defaults to `${OLLAMA_BASE_URL:-http://ollama:11434}/api/generate`, trimming any trailing slash on the base URL; set `OLLAMA_BASE_URL=https://ollama.com` to opt into cloud defaults. The default model is `gemma4:latest` (self-hosted) or `minimax-m3:cloud` (cloud), and reasoning is disabled by default (`Service.Think: "false"`) so thinking-capable models do not leak reasoning into results.
|
||||
- **OpenAI** — cloud Responses API. See [`openai/README.md`](openai/README.md) for prompts, schema variants, and header requirements.
|
||||
|
||||
### Configuration
|
||||
|
|
@ -93,18 +93,18 @@ The model `Options` adjust model parameters such as temperature, top-p, and sche
|
|||
|
||||
Configures the endpoint URL, method, format, and authentication for [Ollama](ollama/README.md), [OpenAI](openai/README.md), and other engines that perform remote HTTP requests:
|
||||
|
||||
| Field | Default | Notes |
|
||||
|:-----------------------------------|:-----------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `Uri` | required for remote | Endpoint base. Empty keeps model local (TensorFlow). Ollama alias fills `${OLLAMA_BASE_URL}/api/generate`, defaulting to `http://ollama:11434`. |
|
||||
| `Method` | `POST` | Override verb if provider needs it. |
|
||||
| `Key` | `""` | Bearer token; prefer env expansion (OpenAI: `OPENAI_API_KEY`, Ollama: `OLLAMA_API_KEY`). |
|
||||
| `Username` / `Password` | `""` | Injected as basic auth when URI lacks userinfo. |
|
||||
| `Model` | `""` | Endpoint-specific override; wins over model/name. |
|
||||
| `Org` / `Project` | `""` | OpenAI headers (org/proj IDs). |
|
||||
| `Think` | `""` | Optional reasoning hint passed as `think` in service requests. Supports levels like `low`, `medium`, `high`; string values `true`/`false` are normalized to JSON booleans on output. Omitted when empty. |
|
||||
| `RequestFormat` / `ResponseFormat` | set by engine alias | Explicit values win over alias defaults. |
|
||||
| `FileScheme` | set by engine alias (`data` or `base64`) | Controls image transport. |
|
||||
| `Disabled` | `false` | Disable the endpoint without removing the model. |
|
||||
| Field | Default | Notes |
|
||||
|:-----------------------------------|:-----------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `Uri` | required for remote | Endpoint base. Empty keeps model local (TensorFlow). Ollama alias fills `${OLLAMA_BASE_URL}/api/generate`, defaulting to `http://ollama:11434`. |
|
||||
| `Method` | `POST` | Override verb if provider needs it. |
|
||||
| `Key` | `""` | Bearer token; prefer env expansion (OpenAI: `OPENAI_API_KEY`, Ollama: `OLLAMA_API_KEY`). |
|
||||
| `Username` / `Password` | `""` | Injected as basic auth when URI lacks userinfo. |
|
||||
| `Model` | `""` | Endpoint-specific override; wins over model/name. |
|
||||
| `Org` / `Project` | `""` | OpenAI headers (org/proj IDs). |
|
||||
| `Think` | `""` (Ollama engine: `"false"`) | Optional reasoning hint passed as `think` in service requests. The Ollama engine defaults it to `"false"` (reasoning off); re-enable with `"true"`. Supports levels like `low`, `medium`, `high`; string values `true`/`false` are normalized to JSON booleans on output. Omitted when empty. |
|
||||
| `RequestFormat` / `ResponseFormat` | set by engine alias | Explicit values win over alias defaults. |
|
||||
| `FileScheme` | set by engine alias (`data` or `base64`) | Controls image transport. |
|
||||
| `Disabled` | `false` | Disable the endpoint without removing the model. |
|
||||
|
||||
> **Authentication:** All credentials and identifiers support `${ENV_VAR}` expansion. `Service.Key` sets `Authorization: Bearer <token>`; `Username`/`Password` injects HTTP basic authentication into the service URI when it is not already present. When `Service.Key` is empty, PhotoPrism defaults to `OPENAI_API_KEY` (OpenAI engine) or `OLLAMA_API_KEY` (Ollama engine), also honoring their `_FILE` counterparts. Key and schema file paths must reference readable regular files (directories are ignored/rejected).
|
||||
|
||||
|
|
@ -114,7 +114,7 @@ Configures the endpoint URL, method, format, and authentication for [Ollama](oll
|
|||
- Env expansion runs for all `Service` credentials and `Model` overrides; empty or disabled models return empty identifiers.
|
||||
- Options merging: engine defaults fill missing fields; explicit values always win. Temperature is capped at `MaxTemperature`.
|
||||
- Authentication: `Service.Key` sets `Authorization: Bearer <token>`; `Username`/`Password` inject HTTP basic auth into the service URI when not already present.
|
||||
- Reasoning control: `Service.Think` maps to `ApiRequest.Think` and is serialized only when non-empty (`omitempty`). During JSON encoding, `"true"` / `"false"` are converted to boolean `true` / `false`; other non-empty values are sent as strings.
|
||||
- Reasoning control: `Service.Think` maps to `ApiRequest.Think` and is serialized only when non-empty (`omitempty`). The Ollama engine defaults it to `"false"` via its engine alias (applied when `Service.Think` is empty), so reasoning is off out of the box; other engines leave it empty. During JSON encoding, `"true"` / `"false"` are converted to boolean `true` / `false`; other non-empty values are sent as strings.
|
||||
|
||||
### Minimal Examples
|
||||
|
||||
|
|
|
|||
|
|
@ -3,10 +3,12 @@ package vision
|
|||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
|
||||
"github.com/photoprism/photoprism/internal/ai/vision/ollama"
|
||||
"github.com/photoprism/photoprism/pkg/fs"
|
||||
)
|
||||
|
||||
|
|
@ -15,6 +17,23 @@ func TestOptions(t *testing.T) {
|
|||
var configFile = filepath.Join(configPath, "vision.yml")
|
||||
|
||||
t.Run("Save", func(t *testing.T) {
|
||||
// Regenerate the committed fixture deterministically, independent of any ambient
|
||||
// OLLAMA_* env, so it always reflects the self-hosted defaults (gemma4 + Think: "false").
|
||||
origModel := CaptionModel.Model
|
||||
origService := CaptionModel.Service
|
||||
t.Cleanup(func() {
|
||||
CaptionModel.Model = origModel
|
||||
CaptionModel.Service = origService
|
||||
ensureEnvOnce = sync.Once{}
|
||||
registerOllamaEngineDefaults()
|
||||
})
|
||||
t.Setenv(ollama.APIKeyEnv, "")
|
||||
t.Setenv(ollama.BaseUrlEnv, ollama.DefaultBaseUrl)
|
||||
ensureEnvOnce = sync.Once{}
|
||||
CaptionModel.Model = ""
|
||||
CaptionModel.Service = Service{}
|
||||
registerOllamaEngineDefaults()
|
||||
|
||||
_ = os.Remove(configFile)
|
||||
options := NewConfig()
|
||||
err := options.Save(configFile)
|
||||
|
|
|
|||
|
|
@ -78,6 +78,7 @@ type EngineInfo struct {
|
|||
DefaultModel string
|
||||
DefaultResolution int
|
||||
DefaultKey string // Optional placeholder key (e.g., ${OPENAI_API_KEY}); applied only when Service.Key is empty.
|
||||
DefaultThink string // Optional reasoning hint (e.g., "false"); applied only when Service.Think is empty.
|
||||
}
|
||||
|
||||
// RegisterEngineAlias maps a logical engine name (e.g., "ollama") to a
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ package vision
|
|||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
|
|
@ -53,6 +54,7 @@ func registerOllamaEngineDefaults() {
|
|||
DefaultModel: defaultModel,
|
||||
DefaultResolution: ollama.DefaultResolution,
|
||||
DefaultKey: ollama.APIKeyPlaceholder,
|
||||
DefaultThink: ollama.DefaultThink,
|
||||
})
|
||||
|
||||
// Keep the default caption model config aligned with the defaults.
|
||||
|
|
@ -156,6 +158,16 @@ func (ollamaParser) Parse(ctx context.Context, req *ApiRequest, raw []byte, stat
|
|||
return nil, err
|
||||
}
|
||||
|
||||
// Surface upstream failures so they are diagnosable instead of silently yielding no labels or caption.
|
||||
if status >= http.StatusBadRequest {
|
||||
switch status {
|
||||
case http.StatusNotFound, http.StatusGone:
|
||||
log.Warnf("vision: ollama model %s is unavailable (status %d), it may have been retired or renamed", clean.Log(req.Model), status)
|
||||
default:
|
||||
log.Warnf("vision: ollama request for model %s failed (status %d)", clean.Log(req.Model), status)
|
||||
}
|
||||
}
|
||||
|
||||
response := &ApiResponse{
|
||||
Id: req.GetId(),
|
||||
Code: status,
|
||||
|
|
@ -169,7 +181,7 @@ func (ollamaParser) Parse(ctx context.Context, req *ApiRequest, raw []byte, stat
|
|||
parsedLabels := len(response.Result.Labels) > 0
|
||||
|
||||
// Qwen3-VL models stream their JSON payload in the "Thinking" field.
|
||||
fallbackResponse := strings.TrimSpace(ollamaResp.Response)
|
||||
fallbackResponse := strings.TrimSpace(stripReasoningBlock(ollamaResp.Response))
|
||||
fallbackThinking := strings.TrimSpace(ollamaResp.Thinking)
|
||||
|
||||
fallbackJSON := fallbackResponse
|
||||
|
|
@ -226,6 +238,27 @@ func (ollamaParser) Parse(ctx context.Context, req *ApiRequest, raw []byte, stat
|
|||
return response, nil
|
||||
}
|
||||
|
||||
// stripReasoningBlock removes a single leading, well-delimited <think>...</think> block from s and
|
||||
// returns the trimmed remainder. It is a defensive fallback for thinking-capable models that ignore
|
||||
// the think=false hint and emit tagged reasoning inline; untagged text and an unterminated opening tag
|
||||
// are left unchanged so a legitimate caption is never truncated.
|
||||
func stripReasoningBlock(s string) string {
|
||||
trimmed := strings.TrimSpace(s)
|
||||
|
||||
const openTag = "<think>"
|
||||
|
||||
if len(trimmed) < len(openTag) || !strings.EqualFold(trimmed[:len(openTag)], openTag) {
|
||||
return s
|
||||
}
|
||||
|
||||
end := strings.Index(strings.ToLower(trimmed), "</think>")
|
||||
if end < 0 {
|
||||
return s
|
||||
}
|
||||
|
||||
return strings.TrimSpace(trimmed[end+len("</think>"):])
|
||||
}
|
||||
|
||||
func convertOllamaLabels(payload []ollama.LabelPayload) []LabelResult {
|
||||
if len(payload) == 0 {
|
||||
return nil
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@ package vision
|
|||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"os"
|
||||
"sync"
|
||||
"testing"
|
||||
|
|
@ -17,6 +18,7 @@ func TestRegisterOllamaEngineDefaults(t *testing.T) {
|
|||
testCaptionModel := CaptionModel.Clone()
|
||||
testCaptionModel.Model = ""
|
||||
testCaptionModel.Service.Uri = ""
|
||||
testCaptionModel.Service.Think = ""
|
||||
cloudToken := "moo9yaiS4ShoKiojiathie2vuejiec2X.Mahl7ewaej4ebi7afq8f_vwe" //nolint:gosec
|
||||
|
||||
t.Cleanup(func() {
|
||||
|
|
@ -31,7 +33,8 @@ func TestRegisterOllamaEngineDefaults(t *testing.T) {
|
|||
t.Run("SelfHosted", func(t *testing.T) {
|
||||
ensureEnvOnce = sync.Once{}
|
||||
CaptionModel = testCaptionModel.Clone()
|
||||
_ = os.Unsetenv(ollama.APIKeyEnv)
|
||||
t.Setenv(ollama.APIKeyEnv, "")
|
||||
t.Setenv(ollama.BaseUrlEnv, ollama.DefaultBaseUrl)
|
||||
|
||||
registerOllamaEngineDefaults()
|
||||
|
||||
|
|
@ -55,6 +58,10 @@ func TestRegisterOllamaEngineDefaults(t *testing.T) {
|
|||
if CaptionModel.Service.Uri != ollama.DefaultUri {
|
||||
t.Fatalf("expected caption model uri %s, got %s", ollama.DefaultUri, CaptionModel.Service.Uri)
|
||||
}
|
||||
|
||||
if CaptionModel.Service.Think != ollama.DefaultThink {
|
||||
t.Fatalf("expected caption model think %s, got %s", ollama.DefaultThink, CaptionModel.Service.Think)
|
||||
}
|
||||
})
|
||||
t.Run("Cloud", func(t *testing.T) {
|
||||
ensureEnvOnce = sync.Once{}
|
||||
|
|
@ -83,11 +90,16 @@ func TestRegisterOllamaEngineDefaults(t *testing.T) {
|
|||
if CaptionModel.Service.Uri != ollama.DefaultUri {
|
||||
t.Fatalf("expected caption model uri %s, got %s", ollama.DefaultUri, CaptionModel.Service.Uri)
|
||||
}
|
||||
|
||||
if CaptionModel.Service.Think != ollama.DefaultThink {
|
||||
t.Fatalf("expected caption model think %s, got %s", ollama.DefaultThink, CaptionModel.Service.Think)
|
||||
}
|
||||
})
|
||||
t.Run("ApiKeyAloneKeepsLocalDefaults", func(t *testing.T) {
|
||||
ensureEnvOnce = sync.Once{}
|
||||
CaptionModel = testCaptionModel.Clone()
|
||||
t.Setenv(ollama.APIKeyEnv, cloudToken)
|
||||
t.Setenv(ollama.BaseUrlEnv, ollama.DefaultBaseUrl)
|
||||
|
||||
registerOllamaEngineDefaults()
|
||||
|
||||
|
|
@ -129,6 +141,10 @@ func TestRegisterOllamaEngineDefaults(t *testing.T) {
|
|||
if model.Resolution != ollama.DefaultResolution {
|
||||
t.Fatalf("expected resolution %d, got %d", ollama.DefaultResolution, model.Resolution)
|
||||
}
|
||||
|
||||
if model.Service.Think != ollama.DefaultThink {
|
||||
t.Fatalf("expected service think %s, got %s", ollama.DefaultThink, model.Service.Think)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
|
|
@ -251,4 +267,72 @@ func TestOllamaParserFallbacks(t *testing.T) {
|
|||
t.Fatalf("expected response field caption, got %q", resp.Result.Caption.Text)
|
||||
}
|
||||
})
|
||||
t.Run("StripsLeadingReasoningBlock", func(t *testing.T) {
|
||||
req := &ApiRequest{}
|
||||
payload := ollama.Response{
|
||||
Response: "<think>The user wants a concise caption.</think>A tabby cat stares upward.",
|
||||
}
|
||||
raw, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal: %v", err)
|
||||
}
|
||||
|
||||
parser := ollamaParser{}
|
||||
resp, err := parser.Parse(context.Background(), req, raw, 200)
|
||||
if err != nil {
|
||||
t.Fatalf("parse failed: %v", err)
|
||||
}
|
||||
|
||||
if resp.Result.Caption == nil {
|
||||
t.Fatal("expected caption result")
|
||||
}
|
||||
if resp.Result.Caption.Text != "A tabby cat stares upward." {
|
||||
t.Fatalf("expected reasoning block to be stripped, got %q", resp.Result.Caption.Text)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestOllamaParserUnavailableStatus(t *testing.T) {
|
||||
t.Run("Gone", func(t *testing.T) {
|
||||
req := &ApiRequest{Model: ollama.CloudModel}
|
||||
payload := ollama.Response{}
|
||||
raw, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal: %v", err)
|
||||
}
|
||||
|
||||
parser := ollamaParser{}
|
||||
resp, err := parser.Parse(context.Background(), req, raw, http.StatusGone)
|
||||
if err != nil {
|
||||
t.Fatalf("parse should not error on upstream status, got %v", err)
|
||||
}
|
||||
|
||||
if resp.Code != http.StatusGone {
|
||||
t.Fatalf("expected code %d, got %d", http.StatusGone, resp.Code)
|
||||
}
|
||||
if len(resp.Result.Labels) != 0 || resp.Result.Caption != nil {
|
||||
t.Fatalf("expected empty result, got %+v", resp.Result)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestStripReasoningBlock(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
in string
|
||||
want string
|
||||
}{
|
||||
{"LeadingBlock", "<think>reasoning here</think>Actual caption.", "Actual caption."},
|
||||
{"CaseInsensitive", "<THINK>reasoning</THINK>\n Caption text.", "Caption text."},
|
||||
{"UntaggedUnchanged", "The user wants a concise description of the image.", "The user wants a concise description of the image."},
|
||||
{"UnterminatedUnchanged", "<think>reasoning without a close tag and no caption", "<think>reasoning without a close tag and no caption"},
|
||||
{"Empty", "", ""},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
if got := stripReasoningBlock(tc.in); got != tc.want {
|
||||
t.Fatalf("stripReasoningBlock(%q) = %q, want %q", tc.in, got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -81,12 +81,15 @@ func TestGenerateLabelsRequestShapingForStructuredOutputIdea(t *testing.T) {
|
|||
})
|
||||
t.Run("OllamaUsesJsonFormatWithSchemaPromptInstructions", func(t *testing.T) {
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
var req ApiRequest
|
||||
// Decode into a map because the serialized "think" value is a JSON boolean
|
||||
// (the Ollama engine now disables reasoning by default), not the string form.
|
||||
var req map[string]any
|
||||
require.NoError(t, json.NewDecoder(r.Body).Decode(&req))
|
||||
|
||||
assert.Equal(t, FormatJSON, req.Format)
|
||||
assert.Contains(t, req.Prompt, "Return JSON that matches this schema:")
|
||||
assert.Empty(t, req.Schema, "Ollama structured schema payload is not sent yet")
|
||||
assert.Equal(t, FormatJSON, req["format"])
|
||||
assert.Contains(t, req["prompt"], "Return JSON that matches this schema:")
|
||||
assert.NotContains(t, req, "schema", "Ollama structured schema payload is not sent yet")
|
||||
assert.Equal(t, false, req["think"], "reasoning is disabled by default for the Ollama engine")
|
||||
|
||||
require.NoError(t, json.NewEncoder(w).Encode(ollama.Response{
|
||||
Model: "gemma3:4b",
|
||||
|
|
|
|||
|
|
@ -505,6 +505,10 @@ func (m *Model) ApplyEngineDefaults() {
|
|||
if strings.TrimSpace(m.Service.Key) == "" && info.DefaultKey != "" {
|
||||
m.Service.Key = info.DefaultKey
|
||||
}
|
||||
|
||||
if strings.TrimSpace(m.Service.Think) == "" && info.DefaultThink != "" {
|
||||
m.Service.Think = info.DefaultThink
|
||||
}
|
||||
}
|
||||
|
||||
m.Engine = engine
|
||||
|
|
|
|||
|
|
@ -274,6 +274,18 @@ func TestModelApplyEngineDefaultsSetsServiceDefaults(t *testing.T) {
|
|||
assert.Equal(t, ApiFormatOllama, model.Service.ResponseFormat)
|
||||
assert.Equal(t, scheme.Base64, model.Service.FileScheme)
|
||||
assert.Equal(t, ollama.APIKeyPlaceholder, model.Service.Key)
|
||||
assert.Equal(t, ollama.DefaultThink, model.Service.Think)
|
||||
})
|
||||
t.Run("OllamaPreservesExplicitThink", func(t *testing.T) {
|
||||
model := &Model{
|
||||
Type: ModelTypeLabels,
|
||||
Engine: ollama.EngineName,
|
||||
Service: Service{Think: "true"},
|
||||
}
|
||||
|
||||
model.ApplyEngineDefaults()
|
||||
|
||||
assert.Equal(t, "true", model.Service.Think)
|
||||
})
|
||||
t.Run("PreserveExistingService", func(t *testing.T) {
|
||||
model := &Model{
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
## PhotoPrism — Ollama Engine Integration
|
||||
|
||||
**Last Updated:** May 21, 2026
|
||||
**Last Updated:** July 16, 2026
|
||||
|
||||
### Overview
|
||||
|
||||
|
|
@ -8,8 +8,9 @@ This package provides PhotoPrism’s native adapter for Ollama-compatible multim
|
|||
|
||||
#### Constraints
|
||||
|
||||
- Engine defaults live in `internal/ai/vision/ollama` and are applied whenever a model sets `Engine: ollama`. Aliases map to `ApiFormatOllama`, `scheme.Base64`, and a default 720 px thumbnail. Cloud defaults are only selected when `OLLAMA_BASE_URL` equals `https://ollama.com`.
|
||||
- Responses may arrive as newline-delimited JSON chunks. `decodeOllamaResponse` keeps the most recent chunk, while the parser now supports both `response` and `thinking` fallbacks for captions and labels.
|
||||
- Engine defaults live in `internal/ai/vision/ollama` and are applied whenever a model sets `Engine: ollama`. Aliases map to `ApiFormatOllama`, `scheme.Base64`, and a default 720 px thumbnail. The default model is `gemma4:latest` for self-hosted instances and `minimax-m3:cloud` when `OLLAMA_BASE_URL` equals `https://ollama.com` (cloud defaults are only selected on that exact match).
|
||||
- Reasoning is disabled by default (`DefaultThink = "false"`, applied to `Service.Think` when empty) so thinking-capable models do not leak their reasoning into captions or invalidate label JSON. Re-enable it explicitly with `Service.Think: "true"`.
|
||||
- Responses may arrive as newline-delimited JSON chunks. `decodeOllamaResponse` keeps the most recent chunk, while the parser supports both `response` and `thinking` fallbacks for captions and labels and strips a leading, well-delimited `<think>...</think>` block from the response body as a defensive fallback.
|
||||
- Structured JSON is optional for captions but enforced for labels when `Format: json` (default for label models targeting the Ollama engine).
|
||||
- The adapter never overwrites TensorFlow defaults. If an Ollama call fails, downstream code still has Nasnet, NSFW, and Face models available.
|
||||
- Workers assume a single-image payload per request. Run `photoprism vision run` to validate multi-image prompts before changing that invariant.
|
||||
|
|
@ -53,18 +54,32 @@ This package provides PhotoPrism’s native adapter for Ollama-compatible multim
|
|||
|
||||
### Supported Ollama Vision Models
|
||||
|
||||
| Model (Ollama Tag) | Size & Footprint | Strengths | JSON & Language Notes | When To Use |
|
||||
|:------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `gemma3:4b / 12b / 27b` | 4B/12B/27B parameters, ~3.3 GB → 17 GB downloads, 128 K context | Multimodal text+image reasoning with SigLIP encoder, handles OCR/long documents, supports tool/function calling | Emits structured JSON reliably; >140 languages with strong default English output | High-quality captions + multilingual labels when you have ≥12 GB VRAM (4B works on 8 GB with Q4_K_M) |
|
||||
| `qwen2.5vl:7b` | 8.29 B params (Q4_K_M) ≈6 GB download, 125 K context | Excellent charts, GUI grounding, DocVQA, multi-image reasoning, agentic tool use | JSON mode tuned for schema compliance; supports 20+ languages with strong Chinese/English parity | Label extraction for mixed-language archives or UI/diagram analysis |
|
||||
| `qwen3-vl:2b / 4b / 8b` | Dense 2B/4B/8B tiers (~3 GB, ~3.5 GB, ~6 GB downloads) with native 256 K context extendable to 1 M; fits single 12–24 GB GPUs or high-end CPUs (2B) | Spatial + video reasoning upgrades (Interleaved-MRoPE, DeepStack), 32-language OCR, GUI/agent control, long-document ingest | Emits JSON reliably when prompts specify schema; multilingual captions/labels with Thinking variants boosting STEM reasoning | General-purpose captions/labels when you need long-context doc/video support without cloud APIs; 2B for CPU/edge, 4B as balanced default, 8B when accuracy outweighs latency |
|
||||
| `llama3.2-vision:11b` | 11 B params, ~7.8 GB download, requires ≥8 GB VRAM; 90 B variant needs ≥64 GB | Strong general reasoning, captioning, OCR, supported by Meta ecosystem tooling | Vision tasks officially supported in English; text-only tasks cover eight major languages | Keep captions consistent with Meta-compatible prompts or when teams already standardize on Llama 3.x |
|
||||
| `minicpm-v:8b-2.6` | 8 B params, ~5.5 GB download, 32 K context | Optimized for edge GPUs, high OCR accuracy, multi-image/video support, low token count (≈640 tokens for 1.8 MP) | Multilingual (EN/ZH/DE/FR/IT/KR). Emits concise JSON but may need stricter stopping sequences | Memory-constrained deployments that still require NSFW/OCR-aware label output |
|
||||
> Recommended defaults: `gemma4:latest` for self-hosted instances (a drop-in replacement for `gemma3` at similar latency) and `minimax-m3:cloud` for Ollama Cloud. Both return reliable JSON and handle multiple languages well. The table below lists additional models that work well for specific use cases.
|
||||
|
||||
| Model (Ollama Tag) | Size & Footprint | Strengths | JSON & Language Notes | When To Use |
|
||||
|:----------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `gemma4:e2b / e4b (latest)` | ~7.2 GB (e2b) / ~9.6 GB (e4b) downloads; e2b fits 8 GB VRAM, e4b needs >=10 GB VRAM or CPU offload | Multimodal successor to Gemma 3; very consistent, reliable JSON; drop-in replacement at similar latency (~2 s labels on an RTX 4060) | Emits clean JSON (any code fences stripped by `clean.JSON`); strong multilingual output | **Recommended self-hosted default.** General captions + labels; `gemma4:latest` (e4b) for best quality, `gemma4:e2b` for a faster single-primary-subject classifier |
|
||||
| `gemma3:4b / 12b / 27b` | 4B/12B/27B parameters, ~3.3 GB → 17 GB downloads, 128 K context | Multimodal text+image reasoning with SigLIP encoder, handles OCR/long documents, supports tool/function calling | Emits structured JSON reliably; >140 languages with strong default English output | High-quality captions + multilingual labels when you have ≥12 GB VRAM (4B works on 8 GB with Q4_K_M) |
|
||||
| `qwen2.5vl:7b` | 8.29 B params (Q4_K_M) ≈6 GB download, 125 K context | Excellent charts, GUI grounding, DocVQA, multi-image reasoning, agentic tool use | JSON mode tuned for schema compliance; supports 20+ languages with strong Chinese/English parity | Label extraction for mixed-language archives or UI/diagram analysis |
|
||||
| `qwen3-vl:2b / 4b / 8b` | Dense 2B/4B/8B tiers (~3 GB, ~3.5 GB, ~6 GB downloads) with native 256 K context extendable to 1 M; fits single 12–24 GB GPUs or high-end CPUs (2B) | Spatial + video reasoning upgrades (Interleaved-MRoPE, DeepStack), 32-language OCR, GUI/agent control, long-document ingest | Emits JSON reliably when prompts specify schema; multilingual captions/labels with Thinking variants boosting STEM reasoning | General-purpose captions/labels when you need long-context doc/video support without cloud APIs; 2B for CPU/edge, 4B as balanced default, 8B when accuracy outweighs latency |
|
||||
| `llama3.2-vision:11b` | 11 B params, ~7.8 GB download, requires ≥8 GB VRAM; 90 B variant needs ≥64 GB | Strong general reasoning, captioning, OCR, supported by Meta ecosystem tooling | Vision tasks officially supported in English; text-only tasks cover eight major languages | Keep captions consistent with Meta-compatible prompts or when teams already standardize on Llama 3.x |
|
||||
| `minicpm-v:8b-2.6` | 8 B params, ~5.5 GB download, 32 K context | Optimized for edge GPUs, high OCR accuracy, multi-image/video support, low token count (≈640 tokens for 1.8 MP) | Multilingual (EN/ZH/DE/FR/IT/KR). Emits concise JSON but may need stricter stopping sequences | Memory-constrained deployments that still require NSFW/OCR-aware label output |
|
||||
| `qwen3.5:4b` | ~3.4 GB download, fits 8 GB VRAM | Qwen3.5 family; accepts images in recent Ollama builds; fast, clean single-pass `response` | Thinking-capable, so keep `Service.Think: "false"` (the default); concise JSON labels | Smaller Qwen-family alternative to Gemma 4; verify multimodal support on your Ollama version |
|
||||
|
||||
> Tip: pull models inside the dev container with `docker compose --profile ollama up -d` and then `docker compose exec ollama ollama pull gemma3:4b`. Keep the profile stopped when you do not need extra GPU/CPU load.
|
||||
|
||||
> Qwen3-VL models can stream structured output via `thinking` while leaving `response` empty. The parser checks `response` first and falls back to `thinking`, so captions/labels continue to work with either field.
|
||||
|
||||
#### Ollama Cloud Models
|
||||
|
||||
Set `OLLAMA_BASE_URL=https://ollama.com` and provide `OLLAMA_API_KEY` to use hosted models (no local download or GPU required). The default cloud model is `minimax-m3:cloud`. The cloud catalog changes over time and models are occasionally retired without notice, so treat this as a snapshot and consult <https://ollama.com/search?c=cloud> for the current list; PhotoPrism logs a clear warning when a configured cloud model returns HTTP 404/410. Latencies below are approximate single-image timings observed on the hosted service.
|
||||
|
||||
| Model (Cloud Tag) | Latency (caption / labels) | Strengths | Notes |
|
||||
|:-------------------|:---------------------------|:-----------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------|
|
||||
| `minimax-m3:cloud` | ~3 s / ~7 s | Fast, multilingual (verified English, German, Arabic), 12-17 labels | **Recommended default.** Streams reasoning into the separate `thinking` field; `response` stays clean |
|
||||
| `gemma4:31b` | ~2 s / ~15-19 s | Consistent Gemma-family JSON, cloud sibling of the self-hosted default | Slower on labels than `minimax-m3:cloud`; good when you prefer Gemma output |
|
||||
| `qwen3.5:397b` | ~4-5 s / ~5-7 s | Strong multilingual captions and labels | Large Qwen3.5 MoE; thinking-capable, so keep `Service.Think: "false"` (the default) |
|
||||
|
||||
### Configuration
|
||||
|
||||
#### Environment Variables
|
||||
|
|
@ -112,8 +127,8 @@ Guidelines:
|
|||
|
||||
- Place new entries after the default TensorFlow models so they take precedence while Nasnet/NSFW remain as fallbacks.
|
||||
- Always specify the exact Ollama tag (`model:version`) so upgrades are deliberate.
|
||||
- `Service.Think` is optional and is sent only when non-empty. Keep it quoted (for example `"false"` or `"low"`) so YAML preserves it as a string; PhotoPrism serializes `"true"` / `"false"` as JSON booleans for Ollama compatibility.
|
||||
- Model support is not universal: `think:true` may fail on models that do not implement reasoning, and `think:false` can still yield empty `response` fields on some reasoning-capable models.
|
||||
- `Service.Think` defaults to `"false"` for the Ollama engine (reasoning off) and is sent whenever non-empty. Keep it quoted (for example `"false"`, `"true"`, or `"low"`) so YAML preserves it as a string; PhotoPrism serializes `"true"` / `"false"` as JSON booleans for Ollama compatibility. Set `Service.Think: "true"` to re-enable reasoning for a model that benefits from it.
|
||||
- Model support is not universal: `think:true` may fail on models that do not implement reasoning, and `think:false` can still yield empty `response` fields on some reasoning-capable models (which then stream their JSON via the `thinking` field — the parser handles this).
|
||||
- Keep option flags before positional arguments in CLI snippets (`photoprism vision run -m labels --count 1`).
|
||||
- If you proxy requests (e.g., through Traefik), set `Service.Key` to `Bearer <token>` and configure the proxy to inject/validate it.
|
||||
|
||||
|
|
@ -146,7 +161,7 @@ Guidelines:
|
|||
- `internal/ai/vision/engine_ollama.go` — Builder/parser glue plus label/caption normalization.
|
||||
- `internal/ai/vision/api_ollama.go` — Base64 payload builder.
|
||||
- `internal/ai/vision/api_client.go` — Streaming decoder shared among engines.
|
||||
- `internal/ai/vision/models.go` — Default caption model definition (`gemma3`).
|
||||
- `internal/ai/vision/models.go` — Default caption model definition (`gemma4:latest`, `minimax-m3:cloud` for Ollama Cloud).
|
||||
- `compose*.yaml` — Ollama service profile, Traefik labels, and persistent volume wiring.
|
||||
- `frontend/src/common/util.js` — Maps `src="ollama"` to the correct badge; keep it updated when adding new source strings.
|
||||
|
||||
|
|
|
|||
|
|
@ -21,10 +21,14 @@ const (
|
|||
CloudBaseUrl = "https://ollama.com"
|
||||
// DefaultUri is the default service URI for self-hosted Ollama instances.
|
||||
DefaultUri = BaseUrlPlaceholder + "/api/generate"
|
||||
// DefaultModel names the default caption model bundled with our adapter defaults.
|
||||
DefaultModel = "gemma3:latest"
|
||||
// CloudModel names the default caption for the Ollama cloud service, see https://ollama.com/cloud.
|
||||
CloudModel = "qwen3-vl:235b-instruct-cloud"
|
||||
// DefaultModel names the default vision model bundled with our adapter defaults, see https://ollama.com/library/gemma4.
|
||||
DefaultModel = "gemma4:latest"
|
||||
// CloudModel names the default vision model for the Ollama cloud service, see https://ollama.com/cloud.
|
||||
CloudModel = "minimax-m3:cloud"
|
||||
// DefaultThink is the default reasoning hint for the Ollama engine. Reasoning is disabled by default so
|
||||
// thinking-capable models do not leak their reasoning into captions or invalidate label JSON; users can
|
||||
// re-enable it explicitly with Service.Think: "true".
|
||||
DefaultThink = "false"
|
||||
// CaptionPrompt instructs Ollama caption models to emit a single, active-voice sentence.
|
||||
CaptionPrompt = "Create a caption with exactly one sentence in the active voice that describes the main visual content. Begin with the main subject and clear action. Avoid text formatting, meta-language, and filler words."
|
||||
// LabelConfidenceDefault is used when the model omits the confidence field.
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ type Service struct {
|
|||
Key string `yaml:"Key,omitempty" json:"-"`
|
||||
Org string `yaml:"Org,omitempty" json:"org,omitempty"` // Optional organization header (e.g. OpenAI).
|
||||
Project string `yaml:"Project,omitempty" json:"project,omitempty"` // Optional project header (e.g. OpenAI).
|
||||
Think string `yaml:"Think,omitempty" json:"think,omitempty"` // Optional reasoning hint for compatible engines (e.g. Ollama, GPT-OSS).
|
||||
Think string `yaml:"Think,omitempty" json:"think,omitempty"` // Optional reasoning hint for compatible engines (e.g. Ollama, GPT-OSS); defaults to "false" for the Ollama engine.
|
||||
FileScheme string `yaml:"FileScheme,omitempty" json:"fileScheme,omitempty"`
|
||||
RequestFormat ApiFormat `yaml:"RequestFormat,omitempty" json:"requestFormat,omitempty"`
|
||||
ResponseFormat ApiFormat `yaml:"ResponseFormat,omitempty" json:"responseFormat,omitempty"`
|
||||
|
|
|
|||
|
|
@ -1,6 +1,8 @@
|
|||
package vision
|
||||
|
||||
import "testing"
|
||||
import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestServiceEndpoint(t *testing.T) {
|
||||
//nolint:gosec // G101: Credential-style URLs are intentional test fixtures.
|
||||
|
|
@ -50,8 +52,11 @@ func TestServiceEndpoint(t *testing.T) {
|
|||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
if tt.name == "ExpandsBaseUrlEnv" {
|
||||
switch tt.name {
|
||||
case "ExpandsBaseUrlEnv":
|
||||
t.Setenv("OLLAMA_BASE_URL", "http://custom:11434")
|
||||
case "FallbacksWhenEnvMissing":
|
||||
t.Setenv("OLLAMA_BASE_URL", "http://ollama:11434")
|
||||
}
|
||||
|
||||
uri, method := tt.svc.Endpoint()
|
||||
|
|
|
|||
3
internal/ai/vision/testdata/vision.yml
vendored
3
internal/ai/vision/testdata/vision.yml
vendored
|
|
@ -65,13 +65,14 @@ Models:
|
|||
Name: embeddings
|
||||
Outputs: 512
|
||||
- Type: caption
|
||||
Model: gemma3:latest
|
||||
Model: gemma4:latest
|
||||
Engine: ollama
|
||||
Run: manual
|
||||
Resolution: 720
|
||||
Service:
|
||||
Uri: ${OLLAMA_BASE_URL}/api/generate
|
||||
Key: ${OLLAMA_API_KEY}
|
||||
Think: "false"
|
||||
FileScheme: base64
|
||||
RequestFormat: ollama
|
||||
ResponseFormat: ollama
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue