mirror of
https://github.com/Misterio77/Foundry.git
synced 2026-08-24 10:04:09 -05:00
feat(overlays): make llama.cpp router HF-cache scan opt-in
The router unconditionally scans the HF cache (server-models.cpp load_from_cache) and auto-registers every downloaded model under a forced repo:tag id, creating untuned twins of preset-defined models and blocking free naming. The scan is just a listing convenience - auto-download on first use goes through the child's --hf-repo and is unaffected. Patch gates it behind LLAMA_ROUTER_SCAN_CACHE (off by default) so --models-preset is the single source of truth and models can be named freely. Assisted-by: pi (opus-4.8)
This commit is contained in:
@@ -63,6 +63,12 @@ in {
|
||||
})
|
||||
];
|
||||
|
||||
# Make the llama.cpp router's HF-cache scan opt-in (LLAMA_ROUTER_SCAN_CACHE)
|
||||
# so --models-preset is the single source of truth and models can be named
|
||||
# freely without untuned repo:tag twins. Patch the base so the -vulkan and
|
||||
# -rocm variants (llama-cpp.override) inherit it.
|
||||
llama-cpp = addPatches prev.llama-cpp [./llama-cpp-optional-cache-scan.patch];
|
||||
|
||||
wl-clipboard = addPatches prev.wl-clipboard [./wl-clipboard-secrets.diff];
|
||||
|
||||
pass = addPatches prev.pass [./pass-wlclipboard-secret.diff];
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
Make the router's HF-cache scan opt-in.
|
||||
|
||||
By default `llama-server` in router mode scans the HF cache and auto-registers
|
||||
every downloaded model under a forced `repo:tag` id, producing untuned
|
||||
duplicates of models already defined via --models-preset and preventing free
|
||||
naming. The scan is purely a listing convenience and is independent of
|
||||
auto-download on first use (that goes through the child process's --hf-repo).
|
||||
|
||||
Gate it behind LLAMA_ROUTER_SCAN_CACHE so presets are the single source of
|
||||
truth by default; set the env var to restore upstream behaviour.
|
||||
|
||||
--- a/tools/server/server-models.cpp
|
||||
+++ b/tools/server/server-models.cpp
|
||||
@@ -280,8 +280,13 @@
|
||||
|
||||
void server_models::load_models() {
|
||||
// Phase 1: load presets from all sources — pure I/O, no lock needed
|
||||
- // 1. cached models
|
||||
- common_presets cached_models = ctx_preset.load_from_cache();
|
||||
+ // 1. cached models (opt-in). Scanning the HF cache auto-registers untuned
|
||||
+ // twins of models already defined as presets and forces their id to
|
||||
+ // repo:tag. Off by default; set LLAMA_ROUTER_SCAN_CACHE to restore it.
|
||||
+ common_presets cached_models;
|
||||
+ if (std::getenv("LLAMA_ROUTER_SCAN_CACHE")) {
|
||||
+ cached_models = ctx_preset.load_from_cache();
|
||||
+ }
|
||||
SRV_INF("Loaded %zu cached model presets\n", cached_models.size());
|
||||
// 2. local models from --models-dir
|
||||
common_presets local_models;
|
||||
Reference in New Issue
Block a user