阶段 4：自适应与专业引擎（spike-first）——研究¶

# Singing variant — zero new deps
# Just adds a model entry; VoiceStudioSingingBackend reuses the existing omnivoice library

# GGUF variant — adds a bundled binary, not a Python dep
# Built once per platform in CI and stored alongside the Tauri installer:
#   bin/omnivoice-tts-{darwin-arm64,darwin-x86_64,windows-x86_64,linux-x86_64,linux-aarch64}
# Quants pulled at first use via huggingface_hub

# Confirmed live on PyPI (verified 2026-05-18 via WebFetch):
#   omnivoice 0.1.5 — released 2026-04-28, Apache-2.0
# Already a transitive/direct dependency of VoiceStudio (per tts_backend.py:VoiceStudioBackend)

# Confirmed live on HuggingFace (verified 2026-05-18 via WebFetch):
#   Serveurperso/OmniVoice-GGUF — 10,603 downloads last month, 4 quant variants
#   ModelsLab/omnivoice-singing — historical download count; commercial rights unresolved
#   k2-fsa/OmniVoice (upstream) — separate code/weight terms; arXiv 2604.00688

# Confirmed live on GitHub (verified 2026-05-18 via WebFetch):
#   ServeurpersoCom/omnivoice.cpp — MIT, 38 stars, 59 commits on master, 6 open issues

                                     ┌──────────────────────────────────────┐
User opens app / first run           │  Settings → Engines → Default        │
        │                            │  ┌──────────────────────────────┐    │
        ▼                            │  │ Auto-selected quant: Q8_0    │    │
┌──────────────────────────┐         │  │ [▼ Override: F32/BF16/Q4_K_M]│    │
│ detect_capabilities()    │ ─────►  │  └──────────────────────────────┘    │
│   - CPU / RAM            │         │  Source of truth: SQLite settings    │
│   - GPU + VRAM (MB)      │         └──────────────────────────────────────┘
│   - compute_class bucket │                          │
│     (CPU/low/mid/high)   │                          │ persisted choice
└──────────────────────────┘                          ▼
        │                            ┌──────────────────────────────────────┐
        │ probe result               │ quant_map.json (shippable, not       │
        ▼                            │ pinned-in-binary):                   │
┌──────────────────────────┐         │   CPU       → Q4_K_M                 │
│ Quant resolver           │ ◄────── │   low-VRAM  → Q4_K_M (≤4 GB VRAM)    │
│ (compute_class → quant)  │         │   mid-VRAM  → Q8_0  (4-12 GB)        │
└──────────────────────────┘         │   high-VRAM → BF16  (≥12 GB)         │
        │                            └──────────────────────────────────────┘
        │ selected quant
        ▼
┌──────────────────────────┐         ┌──────────────────────────────────────┐
│ huggingface_hub.download │ ─────►  │ $HF_HUB_CACHE/                       │
│   Serveurperso/OmniVoice-GGUF      │   omnivoice-base-{Q}.gguf            │
│   pinned-by-SHA          │         │   omnivoice-tokenizer-{Q}.gguf       │
└──────────────────────────┘         └──────────────────────────────────────┘
        │                                          │
        ▼                                          ▼
┌─────────────────────────────────────────────────────────────────────────┐
│ SubprocessBackend (Phase 2 primitive)                                   │
│                                                                         │
│  spawn:  bin/omnivoice-tts-{darwin-arm64|x86_64|win|linux|              │
│                              linux-aarch64}                             │
│          --model    $HF_HUB_CACHE/.../omnivoice-base-{Q}.gguf           │
│          --codec    $HF_HUB_CACHE/.../omnivoice-tokenizer-{Q}.gguf      │
│          --lang     <user lang>                                         │
│          --ref-wav  <voice clone reference, optional>                   │
│          --ref-text <voice clone reference text, optional>              │
│          -o         /tmp/omnivoice-job-<id>.wav                         │
│          < <stdin: prompt text>                                         │
│                                                                         │
│  stdout/stderr → backend.log (filtered for HF token per AUTH-05)        │
│  output file   → read back into torch.Tensor for downstream consumers   │
└─────────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────────┐
│ Dubbing job UI — new "Singing mode" toggle in job-creation form         │
└─────────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
┌─────────────────────────────────────────────────────────────────────────┐
│ dub_pipeline.py (existing) — extended with singing routing              │
│                                                                         │
│  Source audio  ──► Demucs (already in pipeline)                         │
│                    ├── vocals stem ────┐                                │
│                    └── instrumental ───┼──► preserved untouched         │
│                                        │    in final mix                │
│                                        ▼                                │
│                    Segment detector (NEW, SING-03 heuristic):           │
│                      pitch stability  + energy                          │
│                      → list[(start, end, kind ∈ {speech, sing})]        │
│                                        │                                │
│                                        ▼                                │
│                    Route segments:                                      │
│                      kind=speech → VoiceStudioBackend (existing default)  │
│                      kind=sing   → VoiceStudioSingingBackend (NEW)        │
│                                        │                                │
│                                        ▼                                │
│                    Re-attach to instrumental stem                       │
│                                        │                                │
│                                        ▼                                │
│                              Final dubbed mix                           │
└─────────────────────────────────────────────────────────────────────────┘

VoiceStudioSingingBackend (NEW class, ≤ 30 lines):
  class VoiceStudioSingingBackend(VoiceStudioBackend):
      id            = "omnivoice-singing"
      display_name  = "VoiceStudio (singing)"
      model_id      = "ModelsLab/omnivoice-singing"
      def generate(self, text, **kw):
          text_with_tag = f"[singing] {text}" if not text.startswith("[") else text
          return super().generate(text_with_tag, **kw)

backend/engines/
├── omnivoice_gguf/
│   ├── __init__.py
│   ├── backend.py            # VoiceStudioGGUFBackend(TTSBackend) — SubprocessBackend host
│   ├── quant_map.json        # compute_class → quant filename (updatable without release)
│   ├── hardware_probe.py     # extends services/gpu_sandbox.py probe
│   └── README.md             # engine card content (license, source URL, hardware notes)
└── omnivoice_singing/
    ├── __init__.py
    ├── backend.py            # VoiceStudioSingingBackend(VoiceStudioBackend) — subclass
    ├── segment_detector.py   # SING-03 pitch + energy heuristic
    └── README.md             # engine card content

backend/services/
├── tts_backend.py            # existing — register both new backends in _REGISTRY
└── dub_pipeline.py           # existing — extend with singing routing (SING-02, SING-03)

bin/                          # bundled in installer (per platform)
├── omnivoice-tts-darwin-arm64
├── omnivoice-tts-darwin-x86_64
├── omnivoice-tts-windows-x86_64.exe
├── omnivoice-tts-linux-x86_64
└── omnivoice-tts-linux-aarch64

.planning/decisions/
├── SPIKE-01-gguf.md          # ADR (this research → human review → ratified)
└── SPIKE-02-singing.md       # ADR

# Source: synthesized from Phase 2 SubprocessBackend contract + GGUF-01..05 requirements
# Confidence: HIGH for shape, MEDIUM for default-quant boundaries (no published benchmark)

def select_default_engine() -> str:
    """Run at first launch (and on Settings → Engines → 'Re-probe hardware')."""
    caps = detect_capabilities()  # extends backend/services/gpu_sandbox.py

    # Hardware bucket → quant from quant_map.json (shippable, not pinned in code)
    quant_map = json.load(open(GGUF_DIR / "quant_map.json"))
    quant = quant_map.get(caps.compute_class)  # one of: Q4_K_M, Q8_0, BF16, F32

    if quant is None:
        # CPU-only on a 4 GB-RAM box, etc. — fall back gracefully
        return "omnivoice"  # the existing in-process default

    # Probe load (sub-second timeout) — if the runtime binary or quant won't load,
    # don't make it default; keep falling back to the in-process VoiceStudioBackend
    try:
        VoiceStudioGGUFBackend.probe_load(quant=quant, timeout=5.0)
        return "omnivoice-gguf"
    except (RuntimeError, FileNotFoundError, subprocess.TimeoutExpired) as e:
        logger.warning("GGUF probe failed; falling back to in-process VoiceStudio: %s", e)
        return "omnivoice"

# Source: omnivoice.cpp README CLI invocation [VERIFIED via WebFetch 2026-05-18]
# Combined with SubprocessBackend contract from Phase 2 ROADMAP

class VoiceStudioGGUFBackend(TTSBackend):
    id = "omnivoice-gguf"
    display_name = "VoiceStudio (GGUF)"

    @property
    def sample_rate(self) -> int:
        return 24_000  # Higgs Audio v2, same as base VoiceStudio

    def generate(self, text, *, ref_audio=None, ref_text=None, language="en", **kw):
        out_path = self._tmp_wav_path()
        args = [
            self._binary_path,                           # bin/omnivoice-tts-<plat>
            "--model",  str(self._base_quant_path),      # omnivoice-base-Q8_0.gguf
            "--codec",  str(self._tokenizer_quant_path), # omnivoice-tokenizer-Q8_0.gguf
            "--lang",   _iso_to_omnivoice_lang(language),
            "-o",       str(out_path),
        ]
        if ref_audio:
            args += ["--ref-wav", ref_audio]
            if ref_text:
                args += ["--ref-text", ref_text]

        # SubprocessBackend handles env scrubbing (HF_TOKEN injection from token_resolver,
        # log redaction per AUTH-05, mp.get_context("spawn") IPC supervision)
        self._subprocess_backend.run(args, stdin=text, timeout=120)

        wav, sr = soundfile.read(out_path)
        return torch.from_numpy(wav).unsqueeze(0)  # (1, n_samples)

# Source: extends backend/services/gpu_sandbox.py (already detects CUDA/MPS/ROCm/CPU)
# Confidence: HIGH for shape (existing pattern); MEDIUM for the exact VRAM thresholds
#             (no published OmniVoice-GGUF quality-per-VRAM benchmark)

from dataclasses import dataclass
from typing import Literal

ComputeClass = Literal["cpu", "low-vram", "mid-vram", "high-vram"]

@dataclass(frozen=True)
class HardwareCapabilities:
    backend: Literal["cuda", "mps", "rocm", "cpu"]
    vram_mb: int          # 0 for CPU/MPS-unified-memory bookkeeping; MPS reports via mps.current_allocated_memory()
    compute_class: ComputeClass

def detect_capabilities() -> HardwareCapabilities:
    if torch.cuda.is_available():
        free, total = torch.cuda.mem_get_info()  # bytes
        vram_mb = total // (1024 * 1024)
        return HardwareCapabilities(
            backend="cuda",
            vram_mb=vram_mb,
            compute_class=_bucket(vram_mb),
        )
    if torch.backends.mps.is_available():
        # MPS unified memory: treat half of system RAM as effective VRAM ceiling
        ram_mb = psutil.virtual_memory().total // (1024 * 1024)
        vram_mb = ram_mb // 2
        return HardwareCapabilities(
            backend="mps",
            vram_mb=vram_mb,
            compute_class=_bucket(vram_mb),
        )
    # CPU
    return HardwareCapabilities(backend="cpu", vram_mb=0, compute_class="cpu")

def _bucket(vram_mb: int) -> ComputeClass:
    if vram_mb >= 12_000:  return "high-vram"   # → BF16
    if vram_mb >=  4_000:  return "mid-vram"    # → Q8_0  (recommended default)
    if vram_mb >=  1_000:  return "low-vram"    # → Q4_K_M
    return "cpu"                                # → Q4_K_M (CPU inference)

// Source: synthesized from OmniVoice-GGUF model card quant table + recommended-defaults heuristic
// Shippable JSON so the table can update without an app release
{
  "_meta": {
    "schema_version": 1,
    "source_model": "Serveurperso/OmniVoice-GGUF",
    "source_commit_sha": "<PIN-AT-PHASE-4-PLAN-TIME>",
    "runtime": "omnivoice.cpp",
    "runtime_commit_sha": "<PIN-AT-PHASE-4-PLAN-TIME>"
  },
  "high-vram": {
    "base": "omnivoice-base-BF16.gguf",
    "tokenizer": "omnivoice-tokenizer-BF16.gguf",
    "rationale": "≥12 GB VRAM — quality-first; ~1.6 GB total VRAM use"
  },
  "mid-vram": {
    "base": "omnivoice-base-Q8_0.gguf",
    "tokenizer": "omnivoice-tokenizer-Q8_0.gguf",
    "rationale": "4-12 GB VRAM — recommended balance; ~945 MB VRAM use"
  },
  "low-vram": {
    "base": "omnivoice-base-Q4_K_M.gguf",
    "tokenizer": "omnivoice-tokenizer-Q4_K_M.gguf",
    "rationale": "1-4 GB VRAM — minimal footprint; ~659 MB VRAM use"
  },
  "cpu": {
    "base": "omnivoice-base-Q4_K_M.gguf",
    "tokenizer": "omnivoice-tokenizer-Q4_K_M.gguf",
    "rationale": "CPU-only — Q4_K_M to keep latency tolerable"
  }
}

# Source: synthesized from REQUIREMENTS.md SING-03 ("pitch-stability + energy heuristic")
# Confidence: MEDIUM — heuristic is feasible from existing toolkit (librosa-style pitch + RMS),
#             but no published benchmark for this exact heuristic on VoiceStudio's dubbing corpus.
#             User override per segment is the safety net.

from dataclasses import dataclass
from typing import Literal

SegmentKind = Literal["speech", "sing"]

@dataclass(frozen=True)
class Segment:
    start_s: float
    end_s: float
    kind: SegmentKind
    confidence: float  # 0..1 — surface in UI for user override

def detect_singing_segments(
    vocal_stem_wav: "np.ndarray",
    sr: int,
    *,
    min_segment_s: float = 1.0,
) -> list[Segment]:
    """
    Per-frame analysis on the Demucs vocal stem:
      • pitch via librosa.yin (or torch equivalent already in deps)
      • energy via RMS
    Sing-flag if pitch is sustained > N frames AND energy is above threshold.
    Merge adjacent flagged frames into segments ≥ min_segment_s.
    Return segments with a coverage gap = inferred speech segment.
    """
    ...  # implementation in segment_detector.py

# Source: extends backend/services/tts_backend.py _REGISTRY (existing)
# Confidence: HIGH (existing pattern preserved)

# In tts_backend.py (additive):
class VoiceStudioGGUFBackend(TTSBackend):
    id = "omnivoice-gguf"
    display_name = "VoiceStudio (GGUF, hardware-adaptive)"
    # … see Pattern 2 above

class VoiceStudioSingingBackend(VoiceStudioBackend):
    id = "omnivoice-singing"
    display_name = "VoiceStudio (singing)"
    # See Architecture Diagram inset

_REGISTRY.update({
    "omnivoice-gguf":    VoiceStudioGGUFBackend,
    "omnivoice-singing": VoiceStudioSingingBackend,
})

# SPIKE-01: Adopt `Serveurperso/OmniVoice-GGUF` as hardware-adaptive default cloning engine

**Status:** Proposed (research-supported) — awaiting Phase 2 SubprocessBackend merge
**Date:** 2026-05-18
**Decision-makers:** [maintainer]
**Related:** ROADMAP Phase 4; REQUIREMENTS GGUF-01..06; this Phase 4 RESEARCH.md

## Context

VoiceStudio v0.2.7 ships `k2-fsa/OmniVoice` (separate code/weight terms; 0.6B Qwen3 backbone, Higgs Audio v2 codec) as its default voice-cloning engine via `backend/services/tts_backend.py:VoiceStudioBackend`. The Python in-process path requires PyTorch + CUDA / MPS / CPU.

`Serveurperso/OmniVoice-GGUF` publishes 4 quantizations of the same model (Q4_K_M / Q8_0 / BF16 / F32) consumable through the MIT-licensed `omnivoice.cpp` runtime (a custom GGML-based C++ inference binary). This decision is whether to integrate the GGUF engine as a hardware-adaptive default with overridable fallback.

## Decision

**Technical GO only** — integrate per GGUF-01..06; engine acceptance remains conditional on the rights gate in `docs/engine-acceptance.md`.

## Consequences

**Positive:**
- 4 GB-VRAM GPUs (currently falling back to CPU on the in-process path) get GPU-backed cloning via Q4_K_M.
- Smaller VRAM footprint = stays out of the way of other engines when users run multiple in one session.
- Model licensing needs separate review from the runtime licence; see the correction above.

**Negative / risk:**
- Adds a maintained-by-others C++ runtime to the dependency graph (`omnivoice.cpp`, 38 stars at decision time).
- Adds 12-16 MB of platform binaries to the installer.
- macOS code signing scope expands by 4 binaries (track via REL-05).

**Mitigations:**
- Pin `omnivoice.cpp` by commit SHA; rebuild from pinned SHA in CI.
- In-process `VoiceStudioBackend` remains and is the fallback if any GGUF step fails.
- macOS Apple Silicon Metal build is verified in Wave 1; if blocked, downgrade SPIKE-01 default on macOS to in-process path.

## Sources

- `.planning/phases/04-adaptive-specialty-engines-spike-first/04-RESEARCH.md` (this research)
- https://huggingface.co/Serveurperso/OmniVoice-GGUF (verified 2026-05-18)
- https://github.com/ServeurpersoCom/omnivoice.cpp (verified 2026-05-18)
- https://huggingface.co/k2-fsa/OmniVoice (upstream; separate code/weight terms)

# SPIKE-02: Adopt `ModelsLab/omnivoice-singing` as singing variant of the existing engine

**Status:** Proposed (research-supported) — awaiting Phase 2 SubprocessBackend merge
**Date:** 2026-05-18
**Decision-makers:** [maintainer]
**Related:** ROADMAP Phase 4; REQUIREMENTS SING-01..05; this Phase 4 RESEARCH.md

## Context

`ModelsLab/omnivoice-singing` is a finetune of `k2-fsa/OmniVoice` (commercial rights unresolved; same Qwen3-0.6B backbone, same Higgs Audio v2 codec, same `omnivoice` PyPI library) trained on additional singing + emotion-tagged data. Activated by a `[singing]` text control tag at generation time.

VoiceStudio's dubbing pipeline currently routes vocal stems (via Demucs) through the default TTS engine, which produces speech output even on sung source material. This decision is whether to integrate the singing finetune as a routed alternative for sung segments.

## Decision

**GO with reduced scope** — integrate per SING-01..05, with SING-02 scope-decided after a Wave 2 read of `dub_pipeline.py`.

## Consequences

**Positive:**
- Sung segments of dubbed content produce sung output (currently produces unsuitable speech-like output).
- Zero new Python deps — same `omnivoice` library already shipping.
- ≤30-line backend subclass.

**Negative / risk:**
- Heuristic segmentation (SING-03 pitch-stability + energy) is approximate; user override per segment is the safety net.
- Cross-language singing quality is acknowledged by the model card as variable.

**Mitigations:**
- Per-segment override available in dubbing UI before commit.
- SING-05 acceptance scoped to native-language singing pass; cross-language flagged as best-effort.
- Model-based classifier deferred to v2 per REQUIREMENTS.md.

## Sources

- `.planning/phases/04-adaptive-specialty-engines-spike-first/04-RESEARCH.md` (this research)
- https://huggingface.co/ModelsLab/omnivoice-singing (verified 2026-05-18)
- https://huggingface.co/k2-fsa/OmniVoice (upstream)
- https://pypi.org/project/omnivoice/ (0.1.5, 2026-04-28)