diff --git a/.ai/lessons.md b/.ai/lessons.md index 980e572c45..9408a4d308 100644 --- a/.ai/lessons.md +++ b/.ai/lessons.md @@ -7,15 +7,22 @@ - Test serialized worker tasks with the real process driver. An in-process test does not prove that captured models or nested closures survive serialization. - Use the same regression test before and after the production change so the result shows the behavior difference. - Several dev instances can run from one checkout (see `./scripts/dev urls`). After you add a migration, run it on each running instance of that checkout, or those instances fail on the new code. -- The checkout is shared with other sessions. When the full suite fails, rerun each failing file alone and compare with a clean `git archive HEAD` copy before you connect a failure to your change. +- The checkout is shared with other sessions. When the full suite fails, rerun each failing file alone and compare with a clean `git archive HEAD` copy before you connect a failure to your change. The dev instance of the checkout runs its live files, so a live repro also runs the uncommitted edits of other sessions: run `git diff` on the code under test before you trust the result. - Run seeders and other artisan commands that write SSH keys in a dev container as `www-data` (`docker exec -u www-data`). Root-owned files in `storage/app/ssh` make every SSH call from the web process fail. +- Files written on a server with `tee` get the server umask (027/077 on hardened hosts). Set the mode explicitly (`chmod 644`) for files that a non-root container user reads through a read-only bind mount. - `sshd` keeps the login shell of an open multiplexed SSH connection. Call `SshMultiplexingHelper::removeMuxFile()` before a live test of SSH or login-shell behavior. +- Remote command batches run in `bash -se` (`sh -se` only when the host has no bash). A command that can fail without being an error (such as `cat` of an optional file) stops the batch unless it ends with `|| true`. `Process::fake()` cannot show this, so run such a command once in a real `bash -se` on the target host. - SSH retries replay the full command batch. Only exit code 255 is an SSH transport failure; other exit codes come from the remote command, so do not retry them. Keep batches safe to run twice (for example `docker rm -f || true` before `docker run --name `). +- Do not put `pkill -f "playwright run-server"` in the same shell command as other work; the pattern matches that shell and kills it. Run it as its own command. - Redirect browser test output to a file (`> /tmp/x.log 2>&1`); piping it (`| tail`) hangs because the Playwright server keeps the pipe open. - Browser `click()`/`assertSee()` wait forever when nothing matches, so a bad selector hangs the run instead of failing it. `click('Text')` can match a hidden element first (use `button:visible:has-text("Text")`); `:text-is()` on an element with child elements and `assertSee()` text with double quotes never match. Prefer plain CSS selectors. - A new worktree has no `vendor/` or `public/build/`. Run `composer install --ignore-platform-req=ext-zip` on the host and copy `public/build` from the main checkout before page-rendering tests; otherwise they fail with a Vite manifest error. - Never run `php artisan test` inside a dev container (`./scripts/dev exec` or `docker exec coolify-dev-*`). The container sets `DB_CONNECTION=pgsql` as a real environment variable, which wins over `phpunit.xml`, so `RefreshDatabase` runs `migrate:fresh` on the dev database and wipes it. Run tests on the host only. +- The helper image runs `docker compose` for deployments and for the Coolify upgrade itself. Before you bump its Docker CLI or Compose, test them against the oldest supported Docker Engine (for example a `docker:-dind` container); newer clients can reject data from older daemons. - Do not symlink `vendor` into another worktree. Composer resolves the base path from the real `vendor` path, so the tests load the app classes of the other checkout. Run `composer install --ignore-platform-req=ext-zip` in the worktree. +- To assert that the unsaved-changes bar shows, check `el.classList.contains('is-dirty')`. Playwright treats the hidden bar (`opacity-0`) as visible, and its class attribute always contains the text `is-dirty` from Tailwind variants. +- Do not call `sleep()` in a browser test. The app server runs in the test process, so `sleep()` stops it; use `$page->wait()`. +- In tests, turning on `is_metrics_enabled` (and other Sentinel settings) with `save()`/`update()` runs the settings hook, which starts Sentinel over a real SSH connection on the sync queue. Use `saveQuietly()` or `Queue::fake()`. - Call `visit()` directly in each `tests/v4/Browser` test body; Pest does not mark a test that only uses helper-wrapped `visit()` as a browser test, so it fails with `sendText() on null`. - SQLite returns `0`/`1` for boolean columns without a `boolean` cast, while Postgres returns real booleans. A strict `=== false` check on such a column never matches in tests, so tests can pass on behavior that Postgres blocks. Add the cast or compare loosely. @@ -116,3 +123,9 @@ - `StandaloneDocker`/`SwarmDocker` `server` uses `Server::findCached()`, a per-process identity map that is flushed only when the `servers` row updates. A `ServerSetting` save does not flush it, so pass the current `Server` instance (`setRelation('server', $server)`) when generated output depends on just-saved settings. - Application deployments use the stored `custom_labels` when they exist. When a server-level setting changes the generated labels, regenerate the stored labels of read-only (`is_container_label_readonly_enabled`) applications, or a redeploy keeps the old labels. - Tailwind `text-base` is a colour utility here (`--color-base`, near-black), not a font size. Use `text-[16px]` for size. + +## Keep browser form restore away from app state +- Firefox session restore writes saved input values back and fires a trusted `input` event. Add `autocomplete="off"` to inputs that only mirror client state (for example an Alpine-bound color picker), or their handlers run without a user action. + +## Session writes are last-writer-wins +- Laravel saves the whole session payload at the end of every request, and pages run concurrent `wire:poll` requests. A value that one request changes can be overwritten by a request that started earlier. `KeepConcurrentTeamSwitch` protects `currentTeam`; protect any new session value that changes mid-session the same way, or keep it out of the session. diff --git a/.gitignore b/.gitignore index 3b583795bf..7f342543fe 100644 --- a/.gitignore +++ b/.gitignore @@ -54,3 +54,6 @@ ref # Multi-instance local Coolify env files (scripts/dev-instances) .dev-instances/ + +# Service parser template snapshots: generated locally by scripts/parser-snapshot-diff, not committed +/tests/Fixtures/parser/expected/templates/ diff --git a/AGENTS.md b/AGENTS.md index a57f2c2c4c..3016656c89 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -66,6 +66,22 @@ npm run dev # vite dev server npm run build # production build ``` +## Testing the Service Parser + +`tests/Feature/ServiceParserReferenceTest.php` is a golden test for both compose parsers (`serviceParser` with `updateCompose`, and `applicationParser`) and the proxy label and magic-variable helpers. It parses `tests/Fixtures/parser/service-reference.yaml` and `application-reference.yaml` (one service per parser feature, with setting variations and error fixtures) and every template in `templates/compose`, then compares the output with `tests/Fixtures/parser/expected/`. Only the reference expected files are committed. Template expected files are gitignored, so CI checks only that each template parses. + +```bash +./scripts/parser-snapshot-diff save # before a parser change: save the output of all cases (~90 s) +./scripts/parser-snapshot-diff baseline # after the change: one line per changed case, full diff in storage/framework/testing/parser-snapshot-diff.diff +php artisan test --compact tests/Feature/ServiceParserReferenceTest.php # before you commit +UPDATE_PARSER_SNAPSHOTS=1 php artisan test --compact tests/Feature/ServiceParserReferenceTest.php # accept an intended change, then review git diff tests/Fixtures/parser/ +``` + +- A refactor must give "No differences". For an intended change, check every changed line in the diff file. +- When you add a parser feature, add a small service that uses it to `service-reference.yaml` or `application-reference.yaml` and regenerate the expected files. +- `./scripts/parser-snapshot-diff ` compares with a commit instead of a saved baseline. +- It covers only the latest compose parsing version. Older parsing versions are not covered. + ## Browser Tests (Pest Browser Plugin) Uses `pestphp/pest-plugin-browser` (Playwright). Browser tests go in `tests/v4/Browser/`. @@ -175,6 +191,22 @@ Because the "server" and the test share one PHP process, they share the phpunit - **Project/Environment** — Organizational hierarchy: Team → Project → Environment → Resources. - **Proxy** — Traefik reverse proxy managed per server. +### Server transfer (keep it complete) + +A server transfer moves a server and every record that belongs to it to another Coolify instance: `ServerTransferExporter` writes the bundle, `ServerTransferImporter` reads it (`app/Services/ServerTransfer/`). `ServerTransferCoverage` records the transfer decision for every model and every column. + +- When you add a model, a column, or a relation that belongs to a server or to a resource on a server (application, preview, database, service, volume, environment variable, backup, scheduled task, tag, or setting), decide whether a transfer moves it: + - **Moved:** export and import it, and add a round-trip assertion to `tests/Unit/ServerTransfer/ServerTransferExporterImporterTest.php`. For a new model, also add it to `ServerTransferCoverage::TRANSFERRED`. + - **Not moved:** add the model to `ServerTransferCoverage::NOT_TRANSFERRED`, or the column to `NOT_EXPORTED_COLUMNS`, with the reason. If the data is lost for the user, also add an export warning. +- `tests/Unit/ServerTransfer/ServerTransferCoverageTest.php` exports a server with one record of every transferred model. It fails when a model or a column has no decision. When you add a transferred model, add a record of it to the fixture of that test. +- `docs/server-transfer-full-coverage.md` lists what to set up on a source server for a manual transfer test that covers every transferred model and setting. Add new transferred models and settings to it. +- `scripts/transfer-test/` has live end-to-end transfer tests between two dev instances (see its README). Run them after a change to the transfer flow. When you add transferred data that the unit tests cannot prove (remote commands, provider calls, queue behavior), add a check to `extras-e2e.sh`. +- Application, Service, and the standalone databases export their `$fillable` columns, plus hand-written lists for hidden columns. A column that is not fillable is lost unless you add it to those lists. +- Do not copy a foreign key as a raw ID. IDs are different on the target. Link the record by UUID, or leave the column out with a reason. +- Team credentials that a moved record needs (integration tokens, S3 storages, git sources, private keys, cloud provider tokens) move with it. +- When the user must do a step after the transfer (for example, change a webhook URL at GitHub), add an item type to `TransferFinishChecklist`: the "Finish the transfer" list on the server page of the instance that manages the server now. Give it a check where Coolify can verify the step, and a fix where it can do the step. Also add it to the `post_transfer_checklist` of the bundle (`{target_url}` stands for the URL of the target), which the source shows as text. +- On the source, a transferred server (`Server::isTransferredAway()`) is only an archive, because another instance manages the host. Do not run remote commands, cloud provider calls, or DNS provider calls for it, also not when the archive is deleted. + ### Instance sentinels (`id = 0`) Coolify seeds **instance-owned** rows at primary key `0`. That value is a sentinel meaning “this is the Coolify instance itself”, not a normal autoincrement id. Do not migrate, resequence, or “fix” these to a positive id. diff --git a/DESIGN.md b/DESIGN.md index a482fa3e50..02757f53f7 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -229,8 +229,13 @@ sub-header. The dashboard is a compact overview, not a metrics wall. It has no page header: live active deployments come first as a compact table, followed by traffic analytics when a server has it enabled, then two full-width sections (Projects, -then Servers) that follow the projects-page grid pattern. Each section uses -`x-section-heading` linking to its full index. Communicate server health with +then Servers). Each section uses `x-section-heading` with a small "View all" +link to its full index (next to the switcher) and a table / grid `x-canvas-view-toggle` (no canvas +link). Table is the default; each section remembers its own choice. A page-level +List / Canvas switch at the top opens `/dashboard/canvas`: the servers canvas +with a "Deploying" / "Queued" pill on each application card that has an active +deployment, and a foldable Recent deployments panel at the bottom left (active, then +recent; a row shows its card, the status opens the deployment). Communicate server health with the shared status badge. ### Top-level dashboard destinations diff --git a/app/Actions/Database/StartClickhouse.php b/app/Actions/Database/StartClickhouse.php index bcf5e051f7..86545e010f 100644 --- a/app/Actions/Database/StartClickhouse.php +++ b/app/Actions/Database/StartClickhouse.php @@ -192,7 +192,7 @@ class StartClickhouse $environment_variables->push("CLICKHOUSE_DB={$this->database->clickhouse_db}"); } - add_coolify_default_environment_variables($this->database, $environment_variables, $environment_variables); + $this->addCoolifyEnvironmentVariables($environment_variables); return $environment_variables->all(); } diff --git a/app/Actions/Database/StartDragonfly.php b/app/Actions/Database/StartDragonfly.php index 0bef866db7..2602de63d1 100644 --- a/app/Actions/Database/StartDragonfly.php +++ b/app/Actions/Database/StartDragonfly.php @@ -269,6 +269,8 @@ class StartDragonfly $environment_variables->push("REDIS_PASSWORD={$this->database->dragonfly_password}"); } + $this->addCoolifyEnvironmentVariables($environment_variables); + return $environment_variables->all(); } diff --git a/app/Actions/Database/StartKeydb.php b/app/Actions/Database/StartKeydb.php index f7066fc1f8..dddb86e2c2 100644 --- a/app/Actions/Database/StartKeydb.php +++ b/app/Actions/Database/StartKeydb.php @@ -269,7 +269,7 @@ class StartKeydb $environment_variables->push("REDIS_PASSWORD={$this->database->keydb_password}"); } - add_coolify_default_environment_variables($this->database, $environment_variables, $environment_variables); + $this->addCoolifyEnvironmentVariables($environment_variables); return $environment_variables->all(); } diff --git a/app/Actions/Database/StartMariadb.php b/app/Actions/Database/StartMariadb.php index 6f67500801..234583a538 100644 --- a/app/Actions/Database/StartMariadb.php +++ b/app/Actions/Database/StartMariadb.php @@ -268,7 +268,7 @@ class StartMariadb $environment_variables->push("MARIADB_PASSWORD={$this->database->mariadb_password}"); } - add_coolify_default_environment_variables($this->database, $environment_variables, $environment_variables); + $this->addCoolifyEnvironmentVariables($environment_variables); return $environment_variables->all(); } @@ -282,5 +282,7 @@ class StartMariadb $content = $this->database->mariadb_conf; $content_base64 = base64_encode($content); $this->commands[] = "echo '{$content_base64}' | base64 -d | tee $this->configuration_dir/{$filename} > /dev/null"; + // The container user must read the read-only mount; a strict server umask leaves the file unreadable. + $this->commands[] = "chmod 644 $this->configuration_dir/{$filename}"; } } diff --git a/app/Actions/Database/StartMongodb.php b/app/Actions/Database/StartMongodb.php index 5d748cd4f4..0064b43e38 100644 --- a/app/Actions/Database/StartMongodb.php +++ b/app/Actions/Database/StartMongodb.php @@ -325,7 +325,7 @@ class StartMongodb $environment_variables->push("MONGO_INITDB_DATABASE={$this->database->mongo_initdb_database}"); } - add_coolify_default_environment_variables($this->database, $environment_variables, $environment_variables); + $this->addCoolifyEnvironmentVariables($environment_variables); return $environment_variables->all(); } @@ -372,6 +372,8 @@ class StartMongodb $content = $this->database->mongo_conf; $content_base64 = base64_encode($content); $this->commands[] = "echo '{$content_base64}' | base64 -d | tee $this->configuration_dir/{$filename} > /dev/null"; + // The container user must read the read-only mount; a strict server umask leaves the file unreadable. + $this->commands[] = "chmod 644 $this->configuration_dir/{$filename}"; } private function add_default_database() diff --git a/app/Actions/Database/StartMysql.php b/app/Actions/Database/StartMysql.php index 01e8ff5edf..e8c24d5179 100644 --- a/app/Actions/Database/StartMysql.php +++ b/app/Actions/Database/StartMysql.php @@ -280,7 +280,7 @@ class StartMysql $environment_variables->push("MYSQL_PASSWORD={$this->database->mysql_password}"); } - add_coolify_default_environment_variables($this->database, $environment_variables, $environment_variables); + $this->addCoolifyEnvironmentVariables($environment_variables); return $environment_variables->all(); } @@ -294,5 +294,7 @@ class StartMysql $content = $this->database->mysql_conf; $content_base64 = base64_encode($content); $this->commands[] = "echo '{$content_base64}' | base64 -d | tee $this->configuration_dir/{$filename} > /dev/null"; + // The container user must read the read-only mount; a strict server umask leaves the file unreadable. + $this->commands[] = "chmod 644 $this->configuration_dir/{$filename}"; } } diff --git a/app/Actions/Database/StartPostgresql.php b/app/Actions/Database/StartPostgresql.php index ca22dcfebb..88bb59a88b 100644 --- a/app/Actions/Database/StartPostgresql.php +++ b/app/Actions/Database/StartPostgresql.php @@ -291,7 +291,7 @@ class StartPostgresql $environment_variables->push("POSTGRES_DB={$this->database->postgres_db}"); } - add_coolify_default_environment_variables($this->database, $environment_variables, $environment_variables); + $this->addCoolifyEnvironmentVariables($environment_variables); return $environment_variables->all(); } @@ -342,5 +342,7 @@ class StartPostgresql } $content_base64 = base64_encode($content); $this->commands[] = "echo '{$content_base64}' | base64 -d | tee $config_file_path > /dev/null"; + // The container user must read the read-only mount; a strict server umask leaves the file unreadable. + $this->commands[] = "chmod 644 $config_file_path"; } } diff --git a/app/Actions/Database/StartRedis.php b/app/Actions/Database/StartRedis.php index 6eecbec902..b0ac2e7001 100644 --- a/app/Actions/Database/StartRedis.php +++ b/app/Actions/Database/StartRedis.php @@ -284,7 +284,7 @@ class StartRedis } } - add_coolify_default_environment_variables($this->database, $environment_variables, $environment_variables); + $this->addCoolifyEnvironmentVariables($environment_variables); return $environment_variables->all(); } diff --git a/app/Actions/Database/StartSqlite.php b/app/Actions/Database/StartSqlite.php index 6cc806dc5e..b63598f998 100644 --- a/app/Actions/Database/StartSqlite.php +++ b/app/Actions/Database/StartSqlite.php @@ -153,7 +153,7 @@ class StartSqlite $environment_variables->push("SQLITE_DATABASES={$this->database->sqlite_databases}"); } - add_coolify_default_environment_variables($this->database, $environment_variables, $environment_variables); + $this->addCoolifyEnvironmentVariables($environment_variables); return $environment_variables->all(); } diff --git a/app/Actions/Server/CleanupDocker.php b/app/Actions/Server/CleanupDocker.php index c49c057060..ff4df515f4 100644 --- a/app/Actions/Server/CleanupDocker.php +++ b/app/Actions/Server/CleanupDocker.php @@ -131,10 +131,7 @@ class CleanupDocker implements ShouldBeUnique } } $helperImageVersion = getHelperVersion(); - $helperImage = coolifyHelperImage(); - $helperImageWithVersion = "$helperImage:$helperImageVersion"; - $helperImageWithoutPrefix = 'coollabsio/coolify-helper'; - $helperImageWithoutPrefixVersion = "coollabsio/coolify-helper:$helperImageVersion"; + $helperImageWithVersion = coolifyHelperImage().":$helperImageVersion"; $cleanupLog = []; @@ -163,8 +160,7 @@ class CleanupDocker implements ShouldBeUnique 'docker builder prune -af', // The prune fails on a stopped builder, so start it first. railpackBuilderHelperCommand(railpackBuildxMetadataVolume($server), $helperImageWithVersion, 'docker buildx inspect --bootstrap coolify-railpack >/dev/null 2>&1; docker buildx prune --builder coolify-railpack -af'), - "docker images --filter before=$helperImageWithVersion --filter reference=$helperImage | grep $helperImage | awk '{print $3}' | xargs -r docker rmi -f", - "docker images --filter before=$helperImageWithoutPrefixVersion --filter reference=$helperImageWithoutPrefix | grep $helperImageWithoutPrefix | awk '{print $3}' | xargs -r docker rmi -f", + $this->buildCoolifyImageCleanupCommand($helperImageVersion), ]; if ($deleteUnusedVolumes) { @@ -219,6 +215,22 @@ class CleanupDocker implements ShouldBeUnique "awk '{print $1}' | xargs -r docker rm"; } + /** + * Build a command that removes unused images of the Coolify helper, realtime and sentinel, + * from any registry. They have the coolify.managed=true label, so the generic image prune + * skips them. The current helper version stays, and docker rmi (without -f) keeps images + * that a container uses, for example the running sentinel. + */ + private function buildCoolifyImageCleanupCommand(string $helperImageVersion): string + { + $escapedHelperVersion = preg_replace('/([.\\\\+*?\[\]^$(){}|])/', '\\\\$1', $helperImageVersion); + + return "docker images --format '{{.Repository}}:{{.Tag}}' | ". + "grep -E '(^|/)coollabsio/(coolify-helper|coolify-realtime|sentinel):' | ". + "grep -v -E ':$|(^|/)coollabsio/coolify-helper:{$escapedHelperVersion}$' | ". + 'xargs -r docker rmi 2>/dev/null || true'; + } + /** * Build a docker image prune command that excludes application image repositories. * @@ -254,17 +266,18 @@ class CleanupDocker implements ShouldBeUnique // - Don't have coolify.managed=true label // Images in use by containers will fail silently with docker rmi // Pattern matches both uuid:tag and uuid_servicename:tag (Docker Compose with build) + // Images pulled by digest have no tag and are not dangling, so they are listed as repo@digest $grepCommands = "grep -v ''"; // Add application repo exclusion if there are applications if ($applicationImageRepos->isNotEmpty()) { - $grepCommands .= " | grep -v -E '^({$appExcludePatterns})[_:].+'"; + $grepCommands .= " | grep -v -E '^({$appExcludePatterns})[_:@].+'"; } // Add infrastructure image exclusion (matches any registry prefix) $grepCommands .= " | grep -v -E '{$infraExcludePattern}'"; - $commands[] = "docker images --format '{{.Repository}}:{{.Tag}}' | ". + $commands[] = "docker images --digests --format '{{if eq .Tag \"\"}}{{.Repository}}@{{.Digest}}{{else}}{{.Repository}}:{{.Tag}}{{end}}' | ". $grepCommands.' | '. "xargs -r -I {} sh -c 'docker inspect --format \"{{index .Config.Labels \\\"coolify.managed\\\"}}\" \"{}\" 2>/dev/null | grep -q true || docker rmi \"{}\" 2>/dev/null' || true"; diff --git a/app/Actions/Server/StartSentinel.php b/app/Actions/Server/StartSentinel.php index 3dcc2650ac..b888892cea 100644 --- a/app/Actions/Server/StartSentinel.php +++ b/app/Actions/Server/StartSentinel.php @@ -148,7 +148,9 @@ class StartSentinel ? '-v '.escapeshellarg("{$trafficLogDirectory}:{$trafficLogDirectory}:ro").' ' : ''; $network = $server->isLocalhost() ? ' --network coolify' : ''; - $dockerCommand = "docker run -d$network $dockerEnvironments --name coolify-sentinel --restart unless-stopped -v /var/run/docker.sock:/var/run/docker.sock -v $mountDir:/app/db {$trafficMount}--pid host --health-cmd \"curl --fail http://127.0.0.1:8888/api/health || exit 1\" --health-start-period 120s --health-interval 10s --health-retries 3 --add-host=host.docker.internal:host-gateway --label $dockerLabels $image"; + // A userns-remap daemon refuses --pid host unless the container opts out of remapping. + // On other daemons --userns host has no effect. https://docs.docker.com/engine/security/userns-remap/ + $dockerCommand = "docker run -d$network $dockerEnvironments --name coolify-sentinel --restart unless-stopped -v /var/run/docker.sock:/var/run/docker.sock -v $mountDir:/app/db {$trafficMount}--userns host --pid host --health-cmd \"curl --fail http://127.0.0.1:8888/api/health || exit 1\" --health-start-period 120s --health-interval 10s --health-retries 3 --add-host=host.docker.internal:host-gateway --label $dockerLabels $image"; $server->sentinelHeartbeat(isReset: true); $server->forceFill(['sentinel_waiting_since' => now()])->save(); diff --git a/app/Actions/Service/DeployServiceApplication.php b/app/Actions/Service/DeployServiceApplication.php index 4db4834d4c..2a475cd462 100644 --- a/app/Actions/Service/DeployServiceApplication.php +++ b/app/Actions/Service/DeployServiceApplication.php @@ -47,24 +47,30 @@ class DeployServiceApplication $commands->push("docker network inspect {$safeProjectName} >/dev/null 2>&1 || docker network create --attachable {$safeProjectName}"); } + $connectToPredefinedNetwork = (bool) data_get($service, 'connect_to_docker_network'); $upCommand = "docker compose --project-directory {$safeWorkdir} -f {$safeComposeFile} --project-name {$safeProjectName} up -d --no-deps"; if ($forceRebuild) { $upCommand .= ' --build'; } + if ($connectToPredefinedNetwork) { + $upCommand .= ' --no-start'; + } // No stdin for Compose prompts: they would read the next script lines (see StartService). $upCommand .= " {$safeComposeServiceName} < /dev/null"; $commands->push('echo Starting service container.'); $commands->push($upCommand); - $commands->push("docker network connect {$safeProjectName} coolify-proxy >/dev/null 2>&1 || true"); - - if (data_get($service, 'connect_to_docker_network')) { - $network = escapeshellarg($service->destination->network); - $containerName = escapeshellarg("{$composeServiceName}-{$service->uuid}"); - $networkAlias = escapeshellarg("{$composeServiceName}-{$service->uuid}"); - $commands->push("docker network connect --alias {$networkAlias} {$network} {$containerName} >/dev/null 2>&1 || true"); + if ($connectToPredefinedNetwork) { + // Connect to the predefined network before the container starts (see StartService). + $serviceConfig = data_get(parseDockerComposeYaml($service->docker_compose ?? ''), "services.{$composeServiceName}"); + if ($command = StartService::predefinedNetworkConnectCommand($service, $composeServiceName, $serviceConfig)) { + $commands->push($command); + } + $commands->push('docker start '.escapeshellarg("{$composeServiceName}-{$service->uuid}").' >/dev/null'); } + $commands->push("docker network connect {$safeProjectName} coolify-proxy >/dev/null 2>&1 || true"); + return remote_process($commands->toArray(), $service->server, type_uuid: $service->uuid, callEventOnFinish: 'ServiceStartFinished', callEventData: $service->id, queue: deployment_queue()); } } diff --git a/app/Actions/Service/StartService.php b/app/Actions/Service/StartService.php index 3823c71661..5f71218e0c 100644 --- a/app/Actions/Service/StartService.php +++ b/app/Actions/Service/StartService.php @@ -55,17 +55,22 @@ class StartService $commands[] = "docker network inspect $service->uuid >/dev/null 2>&1 || docker network create --attachable $service->uuid"; } $commands[] = 'echo Starting service.'; - $commands[] = "docker compose --project-directory {$workdir} -f {$workdir}/docker-compose.yml --project-name {$service->uuid} up -d --remove-orphans --force-recreate --build < /dev/null"; - $commands[] = "docker network connect $service->uuid coolify-proxy >/dev/null 2>&1 || true"; + $compose = "docker compose --project-directory {$workdir} -f {$workdir}/docker-compose.yml --project-name {$service->uuid}"; if (data_get($service, 'connect_to_docker_network')) { - $compose = data_get($service, 'docker_compose', []); - $safeNetwork = escapeshellarg($service->destination->network); - $serviceNames = data_get(parseDockerComposeYaml($compose), 'services', []); - foreach ($serviceNames as $serviceName => $serviceConfig) { - $containerName = escapeshellarg("{$serviceName}-{$service->uuid}"); - $commands[] = "docker network connect --alias {$containerName} {$safeNetwork} {$containerName} >/dev/null 2>&1 || true"; + // Connect the created containers to the predefined network before they start. A container + // that needs this network during a service_healthy wait never gets it after `up -d`. + $commands[] = "{$compose} up -d --remove-orphans --force-recreate --build --no-start < /dev/null"; + $composeServices = data_get(parseDockerComposeYaml(data_get($service, 'docker_compose', '')), 'services', []); + foreach ($composeServices as $serviceName => $serviceConfig) { + if ($command = self::predefinedNetworkConnectCommand($service, $serviceName, $serviceConfig)) { + $commands[] = $command; + } } + $commands[] = "{$compose} start < /dev/null"; + } else { + $commands[] = "{$compose} up -d --remove-orphans --force-recreate --build < /dev/null"; } + $commands[] = "docker network connect $service->uuid coolify-proxy >/dev/null 2>&1 || true"; $commands = array_merge($commands, $this->logDrainNetworkConnectCommands($service)); if ($service->is_maintenance_enabled) { // Domains can change with the compose file, so the maintenance routes follow them. @@ -79,6 +84,24 @@ class StartService return remote_process($commands, $service->server, type_uuid: $service->uuid, callEventOnFinish: 'ServiceStartFinished', callEventData: $service->id, queue: deployment_queue()); } + /** + * Connects a container to the predefined network of the destination, unless it is already + * connected (declared in the compose file, or a running container that was not recreated). + * Containers with a network_mode cannot join other networks and are skipped. + */ + public static function predefinedNetworkConnectCommand(Service $service, string $serviceName, mixed $serviceConfig): ?string + { + if (data_get($serviceConfig, 'network_mode') !== null) { + return null; + } + $network = escapeshellarg($service->destination->network); + $containerName = escapeshellarg("{$serviceName}-{$service->uuid}"); + $warning = escapeshellarg("Warning: Could not connect {$serviceName} to the {$service->destination->network} network."); + $connectedNetworks = "docker container inspect --format '{{range \$name, \$_ := .NetworkSettings.Networks}}{{println \$name}}{{end}}' {$containerName}"; + + return "{$connectedNetworks} | grep -qxF {$network} || docker network connect --alias {$containerName} {$network} {$containerName} || echo {$warning}"; + } + /** * Shows the volume warnings of the last parse (for example an external volume that the * service does not use yet) in the start log. diff --git a/app/Console/Commands/Generate/OpenApi.php b/app/Console/Commands/Generate/OpenApi.php index 64d83e1bf9..0a91e6e9cf 100644 --- a/app/Console/Commands/Generate/OpenApi.php +++ b/app/Console/Commands/Generate/OpenApi.php @@ -21,9 +21,6 @@ class OpenApi extends Command $process = Process::env(['XDEBUG_MODE' => 'off'])->run([ './vendor/bin/openapi', 'app', - // Server transfer endpoints are development-only (isDev()), so they stay out of the public docs. - '--exclude', - 'Http/Controllers/Api/ServerTransferController.php', '-o', 'openapi.yaml', '--version', diff --git a/app/Console/Kernel.php b/app/Console/Kernel.php index a88ba64a81..7f3e5fb8bf 100644 --- a/app/Console/Kernel.php +++ b/app/Console/Kernel.php @@ -25,6 +25,8 @@ use Illuminate\Foundation\Console\Kernel as ConsoleKernel; class Kernel extends ConsoleKernel { + private const CDN_JITTER_SECONDS = 600; + private Schedule $scheduleInstance; private InstanceSettings $settings; @@ -91,8 +93,8 @@ class Kernel extends ConsoleKernel $this->scheduleInstance->command('horizon:snapshot')->everyFiveMinutes(); $this->scheduleInstance->command('cleanup:unreachable-servers')->daily()->onOneServer(); - $this->scheduleInstance->job(new PullTemplatesFromCDN)->cron($this->updateCheckFrequency)->timezone($this->instanceTimezone)->onOneServer(); - $this->scheduleInstance->job(new PullChangelog)->cron($this->updateCheckFrequency)->timezone($this->instanceTimezone)->onOneServer(); + $this->scheduleInstance->job($this->withCdnJitter(new PullTemplatesFromCDN))->cron($this->updateCheckFrequency)->timezone($this->instanceTimezone)->onOneServer(); + $this->scheduleInstance->job($this->withCdnJitter(new PullChangelog))->cron($this->updateCheckFrequency)->timezone($this->instanceTimezone)->onOneServer(); $this->scheduleInstance->job(new CleanupInstanceStuffsJob)->everyTwoMinutes()->onOneServer(); $this->scheduleUpdates(); @@ -117,7 +119,7 @@ class Kernel extends ConsoleKernel private function pullImages(): void { - $this->scheduleInstance->job(new CheckHelperImageJob) + $this->scheduleInstance->job($this->withCdnJitter(new CheckHelperImageJob)) ->cron($this->updateCheckFrequency) ->timezone($this->instanceTimezone) ->onOneServer(); @@ -162,20 +164,29 @@ class Kernel extends ConsoleKernel private function scheduleUpdates(): void { - $this->scheduleInstance->job(new CheckForUpdatesJob) + $this->scheduleInstance->job($this->withCdnJitter(new CheckForUpdatesJob)) ->cron($this->updateCheckFrequency) ->timezone($this->instanceTimezone) ->onOneServer(); if ($this->settings->is_auto_update_enabled) { $autoUpdateFrequency = $this->settings->auto_update_frequency; - $this->scheduleInstance->job(new UpdateCoolifyJob) + $this->scheduleInstance->job($this->withCdnJitter(new UpdateCoolifyJob)) ->cron($autoUpdateFrequency) ->timezone($this->instanceTimezone) ->onOneServer(); } } + /** + * All instances share the same default cron, so without a random delay they all request the CDN + * at the same second and Cloudflare rejects the burst (error 1200). + */ + private function withCdnJitter(object $job): object + { + return $job->delay(random_int(0, self::CDN_JITTER_SECONDS)); + } + protected function commands(): void { $this->load(__DIR__.'/Commands'); diff --git a/app/Data/FileEntry.php b/app/Data/FileEntry.php new file mode 100644 index 0000000000..7e01793ea3 --- /dev/null +++ b/app/Data/FileEntry.php @@ -0,0 +1,43 @@ +type === 'dir'; + } + + public function isEditableCandidate(): bool + { + return ! $this->isDir(); + } + + /** + * @param array $entries + * @return array + */ + public static function sort(array $entries): array + { + usort($entries, function (self $a, self $b) { + if ($a->isDir() !== $b->isDir()) { + return $a->isDir() ? -1 : 1; + } + + return strcasecmp($a->name, $b->name); + }); + + return $entries; + } +} diff --git a/app/Data/Traffic/TrafficBreakdownData.php b/app/Data/Traffic/TrafficBreakdownData.php index a98e5323a4..47114f1362 100644 --- a/app/Data/Traffic/TrafficBreakdownData.php +++ b/app/Data/Traffic/TrafficBreakdownData.php @@ -10,14 +10,23 @@ class TrafficBreakdownData extends Data public string $value, public int $requests, public int $bytesOut, + public ?float $lat = null, + public ?float $lon = null, ) {} public static function fromSentinel(array $row): self { + $lat = data_get($row, 'lat'); + $lon = data_get($row, 'lon'); + $hasCoordinates = is_numeric($lat) && is_numeric($lon) + && abs((float) $lat) <= 90 && abs((float) $lon) <= 180; + return new self( value: (string) data_get($row, 'value', ''), requests: (int) data_get($row, 'requests', 0), bytesOut: (int) data_get($row, 'bytes_out', 0), + lat: $hasCoordinates ? (float) $lat : null, + lon: $hasCoordinates ? (float) $lon : null, ); } } diff --git a/app/Enums/TrafficIpMode.php b/app/Enums/TrafficIpMode.php index 6f2b09d89a..4b9f5f48d1 100644 --- a/app/Enums/TrafficIpMode.php +++ b/app/Enums/TrafficIpMode.php @@ -44,7 +44,8 @@ enum TrafficIpMode: string } /** - * Drop the `ip` dimension when Sentinel does not record it. + * Drop the `ip` and `city` dimensions when Sentinel does not record them + * (Sentinel finds the city from the client IP, so `off` stores no cities). * * @param array $dimensions * @return array @@ -55,7 +56,7 @@ enum TrafficIpMode: string return $dimensions; } - return array_values(array_filter($dimensions, fn (string $dimension): bool => $dimension !== 'ip')); + return array_values(array_filter($dimensions, fn (string $dimension): bool => ! in_array($dimension, ['ip', 'city'], true))); } public function breakdownLabel(): string diff --git a/app/Events/DockerCleanupDone.php b/app/Events/DockerCleanupDone.php index 2bd7fee198..f229c66451 100644 --- a/app/Events/DockerCleanupDone.php +++ b/app/Events/DockerCleanupDone.php @@ -21,4 +21,18 @@ class DockerCleanupDone implements ShouldBroadcast new PrivateChannel('team.'.$this->execution->server->team->id), ]; } + + /** + * Keep the payload under Reverb's max message size: the cleanup log can be large. + * + * @return array{executionId: int, serverId: int, status: string} + */ + public function broadcastWith(): array + { + return [ + 'executionId' => $this->execution->id, + 'serverId' => $this->execution->server_id, + 'status' => $this->execution->status, + ]; + } } diff --git a/app/Events/TransferFinishChanged.php b/app/Events/TransferFinishChanged.php new file mode 100644 index 0000000000..15fd906b90 --- /dev/null +++ b/app/Events/TransferFinishChanged.php @@ -0,0 +1,26 @@ +teamId = $server->team_id; + $this->serverUuid = $server->uuid; + } +} diff --git a/app/Http/Controllers/Api/ApplicationsController.php b/app/Http/Controllers/Api/ApplicationsController.php index 4096ef94e5..2904a61806 100644 --- a/app/Http/Controllers/Api/ApplicationsController.php +++ b/app/Http/Controllers/Api/ApplicationsController.php @@ -421,7 +421,7 @@ class ApplicationsController extends Controller ], ), ], - 'watch_paths' => ['type' => 'string', 'description' => 'The watch paths.'], + 'watch_paths' => ['type' => 'string', 'description' => 'Watch path patterns, one per line (newline-separated). Commas do not separate patterns.'], 'use_build_server' => ['type' => 'boolean', 'nullable' => true, 'description' => 'Use build server.'], 'use_build_secrets' => ['type' => 'boolean', 'default' => false, 'description' => 'Use Docker Build Secrets for build-time environment variables.'], 'is_git_submodules_enabled' => ['type' => 'boolean', 'description' => 'Clone Git submodules.'], @@ -622,7 +622,7 @@ class ApplicationsController extends Controller ], ), ], - 'watch_paths' => ['type' => 'string', 'description' => 'The watch paths.'], + 'watch_paths' => ['type' => 'string', 'description' => 'Watch path patterns, one per line (newline-separated). Commas do not separate patterns.'], 'use_build_server' => ['type' => 'boolean', 'nullable' => true, 'description' => 'Use build server.'], 'use_build_secrets' => ['type' => 'boolean', 'default' => false, 'description' => 'Use Docker Build Secrets for build-time environment variables.'], 'is_git_submodules_enabled' => ['type' => 'boolean', 'description' => 'Clone Git submodules.'], @@ -823,7 +823,7 @@ class ApplicationsController extends Controller ], ), ], - 'watch_paths' => ['type' => 'string', 'description' => 'The watch paths.'], + 'watch_paths' => ['type' => 'string', 'description' => 'Watch path patterns, one per line (newline-separated). Commas do not separate patterns.'], 'use_build_server' => ['type' => 'boolean', 'nullable' => true, 'description' => 'Use build server.'], 'use_build_secrets' => ['type' => 'boolean', 'default' => false, 'description' => 'Use Docker Build Secrets for build-time environment variables.'], 'is_git_submodules_enabled' => ['type' => 'boolean', 'description' => 'Clone Git submodules.'], @@ -1276,6 +1276,9 @@ class ApplicationsController extends Controller $this->authorize('create', Application::class); + // removeUnnecessaryFieldsFromRequest() strips force_domain_override before the docker_compose_domains checks run. + $forceDomainOverride = $request->boolean('force_domain_override'); + if ($request->instant_deploy) { abort_unless($request->user()->tokenCan('deploy') || $request->user()->tokenCan('root'), 403, 'Missing required permissions: deploy'); } @@ -1531,7 +1534,7 @@ class ApplicationsController extends Controller }); $duplicates = $urls->duplicates()->unique()->values(); - if ($duplicates->isNotEmpty() && ! $request->boolean('force_domain_override')) { + if ($duplicates->isNotEmpty() && ! $forceDomainOverride) { $errors[] = 'The current request contains conflicting URLs: '.implode(', ', $duplicates->toArray()).' Use force_domain_override=true to proceed.'; } @@ -1552,7 +1555,7 @@ class ApplicationsController extends Controller ], 422); } - if ($result['hasConflicts'] && ! $request->boolean('force_domain_override')) { + if ($result['hasConflicts'] && ! $forceDomainOverride) { return response()->json([ 'message' => 'Domain conflicts detected. Use force_domain_override=true to proceed.', 'conflicts' => $result['conflicts'], @@ -1577,7 +1580,7 @@ class ApplicationsController extends Controller $application->domain_port_overrides = $domainPortOverrides; } $gitRepository = $application->git_repository; - $httpsRepository = scpStyleGitUrlToHttps($gitRepository); + $httpsRepository = sshGitUrlToHttps($gitRepository); if (is_string($httpsRepository)) { $gitRepository = $httpsRepository; $application->git_repository = $httpsRepository; @@ -1799,7 +1802,7 @@ class ApplicationsController extends Controller }); $duplicates = $urls->duplicates()->unique()->values(); - if ($duplicates->isNotEmpty() && ! $request->boolean('force_domain_override')) { + if ($duplicates->isNotEmpty() && ! $forceDomainOverride) { $errors[] = 'The current request contains conflicting URLs: '.implode(', ', $duplicates->toArray()).' Use force_domain_override=true to proceed. '; } @@ -1820,7 +1823,7 @@ class ApplicationsController extends Controller ], 422); } - if ($result['hasConflicts'] && ! $request->boolean('force_domain_override')) { + if ($result['hasConflicts'] && ! $forceDomainOverride) { return response()->json([ 'message' => 'Domain conflicts detected. Use force_domain_override=true to proceed.', 'conflicts' => $result['conflicts'], @@ -2032,7 +2035,7 @@ class ApplicationsController extends Controller }); $duplicates = $urls->duplicates()->unique()->values(); - if ($duplicates->isNotEmpty() && ! $request->boolean('force_domain_override')) { + if ($duplicates->isNotEmpty() && ! $forceDomainOverride) { $errors[] = 'The current request contains conflicting URLs: '.implode(', ', $duplicates->toArray()).' Use force_domain_override=true to proceed.'; } @@ -2053,7 +2056,7 @@ class ApplicationsController extends Controller ], 422); } - if ($result['hasConflicts'] && ! $request->boolean('force_domain_override')) { + if ($result['hasConflicts'] && ! $forceDomainOverride) { return response()->json([ 'message' => 'Domain conflicts detected. Use force_domain_override=true to proceed.', 'conflicts' => $result['conflicts'], @@ -3093,14 +3096,10 @@ class ApplicationsController extends Controller schema: new OA\Schema( type: 'object', properties: [ - 'project_uuid' => ['type' => 'string', 'description' => 'The project UUID.'], - 'server_uuid' => ['type' => 'string', 'description' => 'The server UUID.'], - 'environment_name' => ['type' => 'string', 'description' => 'The environment name.'], 'github_app_uuid' => ['type' => 'string', 'description' => 'The Github App UUID.'], 'git_repository' => ['type' => 'string', 'description' => 'The git repository URL.'], 'git_branch' => ['type' => 'string', 'description' => 'The git branch.'], 'ports_exposes' => ['type' => 'string', 'description' => 'The ports to expose.'], - 'destination_uuid' => ['type' => 'string', 'description' => 'The destination UUID.'], 'build_pack' => ['type' => 'string', 'enum' => ['nixpacks', 'railpack', 'static', 'dockerfile', 'dockercompose'], 'description' => 'The build pack type.'], 'name' => ['type' => 'string', 'description' => 'The application name.'], 'description' => ['type' => 'string', 'description' => 'The application description.'], @@ -3168,7 +3167,7 @@ class ApplicationsController extends Controller ], ), ], - 'watch_paths' => ['type' => 'string', 'description' => 'The watch paths.'], + 'watch_paths' => ['type' => 'string', 'description' => 'Watch path patterns, one per line (newline-separated). Commas do not separate patterns.'], 'use_build_server' => ['type' => 'boolean', 'nullable' => true, 'description' => 'Use build server.'], 'use_build_secrets' => ['type' => 'boolean', 'description' => 'Use Docker Build Secrets for build-time environment variables.'], 'http_basic_auth_hash_algorithm' => ['type' => 'string', 'enum' => ['bcrypt', 'argon2id'], 'description' => 'Hash algorithm for the HTTP Basic Authentication password. Argon2id needs the Caddy proxy, version 2.11 or newer.'], @@ -3301,10 +3300,12 @@ class ApplicationsController extends Controller $dnsCleanup = app(ManagedDnsRecordCleanup::class); $previousDnsHostnames = $dnsCleanup->hostnamesOf($application); $server = $application->destination->server; - $allowedFields = ['name', 'description', 'is_static', 'is_spa', 'is_auto_deploy_enabled', 'is_force_https_enabled', 'is_preview_deployments_enabled', 'domains', 'noindex_domains', 'git_repository', 'git_branch', 'git_commit_sha', 'docker_registry_image_name', 'docker_registry_image_tag', 'build_pack', 'static_image', 'install_command', 'build_command', 'start_command', 'ports_exposes', 'ports_mappings', 'custom_network_aliases', 'base_directory', 'publish_directory', 'health_check_enabled', 'health_check_type', 'health_check_command', 'health_check_path', 'health_check_port', 'health_check_host', 'health_check_method', 'health_check_return_code', 'health_check_scheme', 'health_check_response_text', 'health_check_interval', 'health_check_timeout', 'health_check_retries', 'health_check_start_period', 'limits_memory', 'limits_memory_swap', 'limits_memory_swappiness', 'limits_memory_reservation', 'limits_cpus', 'limits_cpuset', 'limits_cpu_shares', 'custom_labels', 'custom_docker_run_options', 'post_deployment_command', 'post_deployment_command_container', 'pre_deployment_command', 'pre_deployment_command_container', 'watch_paths', 'manual_webhook_secret_github', 'manual_webhook_secret_gitlab', 'manual_webhook_secret_bitbucket', 'manual_webhook_secret_gitea', 'dockerfile_location', 'dockerfile_target_build', 'docker_compose_location', 'docker_compose_custom_start_command', 'docker_compose_custom_build_command', 'docker_compose_domains', 'redirect', 'instant_deploy', 'use_build_server', 'use_build_secrets', 'custom_nginx_configuration', 'is_http_basic_auth_enabled', 'http_basic_auth_username', 'http_basic_auth_password', 'http_basic_auth_hash_algorithm', 'http_basic_auth_bcrypt_cost', 'http_basic_auth_argon2id_memory_cost', 'http_basic_auth_argon2id_time_cost', 'connect_to_docker_network', 'force_domain_override', 'is_container_label_escape_enabled', 'is_preserve_repository_enabled', 'preview_url_template', 'max_restart_count', ...self::APPLICATION_SETTING_FIELDS]; + $allowedFields = ['name', 'description', 'is_static', 'is_spa', 'is_auto_deploy_enabled', 'is_force_https_enabled', 'is_preview_deployments_enabled', 'domains', 'noindex_domains', 'git_repository', 'git_branch', 'git_commit_sha', 'docker_registry_image_name', 'docker_registry_image_tag', 'build_pack', 'static_image', 'install_command', 'build_command', 'start_command', 'ports_exposes', 'ports_mappings', 'custom_network_aliases', 'base_directory', 'publish_directory', 'health_check_enabled', 'health_check_type', 'health_check_command', 'health_check_path', 'health_check_port', 'health_check_host', 'health_check_method', 'health_check_return_code', 'health_check_scheme', 'health_check_response_text', 'health_check_interval', 'health_check_timeout', 'health_check_retries', 'health_check_start_period', 'limits_memory', 'limits_memory_swap', 'limits_memory_swappiness', 'limits_memory_reservation', 'limits_cpus', 'limits_cpuset', 'limits_cpu_shares', 'custom_labels', 'custom_docker_run_options', 'post_deployment_command', 'post_deployment_command_container', 'pre_deployment_command', 'pre_deployment_command_container', 'watch_paths', 'manual_webhook_secret_github', 'manual_webhook_secret_gitlab', 'manual_webhook_secret_bitbucket', 'manual_webhook_secret_gitea', 'dockerfile_location', 'dockerfile_target_build', 'docker_compose_location', 'docker_compose_custom_start_command', 'docker_compose_custom_build_command', 'docker_compose_domains', 'redirect', 'instant_deploy', 'use_build_server', 'use_build_secrets', 'custom_nginx_configuration', 'is_http_basic_auth_enabled', 'http_basic_auth_username', 'http_basic_auth_password', 'http_basic_auth_hash_algorithm', 'http_basic_auth_bcrypt_cost', 'http_basic_auth_argon2id_memory_cost', 'http_basic_auth_argon2id_time_cost', 'connect_to_docker_network', 'force_domain_override', 'is_container_label_escape_enabled', 'is_preserve_repository_enabled', 'preview_url_template', 'max_restart_count', 'github_app_uuid', 'dockerfile', ...self::APPLICATION_SETTING_FIELDS]; $validationRules = [ 'name' => 'string|max:255', + 'github_app_uuid' => 'string', + 'dockerfile' => 'string', 'description' => 'string|nullable', 'watch_paths' => 'string|nullable', 'docker_compose_domains' => 'array|nullable', @@ -3364,6 +3365,28 @@ class ApplicationsController extends Controller 'custom_nginx_configuration' => $customNginxConfiguration, ]); } + if ($request->has('dockerfile')) { + if (blank($application->dockerfile)) { + return response()->json([ + 'message' => 'Validation failed.', + 'errors' => [ + 'dockerfile' => 'The dockerfile can only be updated for applications created from a Dockerfile.', + ], + ], 422); + } + $dockerfile = is_string($request->dockerfile) && isBase64Encoded($request->dockerfile) ? base64_decode($request->dockerfile) : false; + if ($dockerfile === false || blank($dockerfile) || mb_detect_encoding($dockerfile, 'UTF-8', true) === false) { + return response()->json([ + 'message' => 'Validation failed.', + 'errors' => [ + 'dockerfile' => 'The dockerfile should be base64 encoded.', + ], + ], 422); + } + $request->merge([ + 'dockerfile' => $dockerfile, + ]); + } $return = $this->validateDataApplications($request, $server); if ($return instanceof JsonResponse) { return $return; @@ -3383,6 +3406,24 @@ class ApplicationsController extends Controller ], 422); } + $githubApp = null; + if ($request->has('github_app_uuid')) { + if ($application->build_pack === 'dockerimage' || filled($application->dockerfile) || $application->deploymentType() === 'deploy_key') { + return response()->json([ + 'message' => 'Validation failed.', + 'errors' => [ + 'github_app_uuid' => 'The github_app_uuid can only be changed for applications deployed from a GitHub App or public repository.', + ], + ], 422); + } + $githubApp = GithubApp::where('uuid', $request->github_app_uuid) + ->where(fn ($query) => $query->where('team_id', $teamId)->orWhere('is_system_wide', true)) + ->first(); + if (! $githubApp?->isConnected()) { + return response()->json(['message' => 'Github App not found.'], 404); + } + } + $applicationSettings = $this->applicationSettingsFromRequest($request); if ($prefixValidation = $this->containerNamePrefixValidationResponse($applicationSettings, $application->destination->server, $application)) { return $prefixValidation; @@ -3642,6 +3683,12 @@ class ApplicationsController extends Controller $requestHasNoindexDomains = $request->has('noindex_domains'); data_forget($data, 'noindex_domains'); $application->fill($data); + if ($githubApp && ($application->source_id !== $githubApp->id || $application->source_type !== $githubApp->getMorphClass())) { + $application->source_id = $githubApp->id; + $application->source_type = $githubApp->getMorphClass(); + // The next deployment fetches the repository ID from the new source. + $application->repository_project_id = null; + } if ($requestHasNoindexDomains) { // Must run after fqdn is filled: flags are kept only for domains the app still has. $application->setNoindexDomains($request->input('noindex_domains') ?? []); @@ -3650,6 +3697,9 @@ class ApplicationsController extends Controller $application->custom_labels = str(implode('|coolify|', generateLabelsApplication($application)))->replace('|coolify|', "\n"); } $application->withoutAuditLogging(fn () => $application->save()); + if ($application->wasChanged('dockerfile')) { + $application->parseHealthcheckFromDockerfile($application->dockerfile); + } $dnsCleanup->queueReleaseOfRemovedHostnames($application, $previousDnsHostnames, (int) $teamId); auditLog('api.application.updated', [ diff --git a/app/Http/Controllers/Api/DatabasesController.php b/app/Http/Controllers/Api/DatabasesController.php index fd7c53e90a..634f20720d 100644 --- a/app/Http/Controllers/Api/DatabasesController.php +++ b/app/Http/Controllers/Api/DatabasesController.php @@ -556,6 +556,7 @@ class DatabasesController extends Controller 'limits_cpus' => ['type' => 'string', 'description' => 'CPU limit of the database'], 'limits_cpuset' => ['type' => 'string', 'description' => 'CPU set of the database'], 'limits_cpu_shares' => ['type' => 'integer', 'description' => 'CPU shares of the database'], + 'instant_deploy' => ['type' => 'boolean', 'description' => 'Restart the database after the update. Requires the deploy ability.'], 'postgres_user' => ['type' => 'string', 'description' => 'PostgreSQL user'], 'postgres_password' => ['type' => 'string', 'description' => 'PostgreSQL password'], 'postgres_db' => ['type' => 'string', 'description' => 'PostgreSQL database'], @@ -644,6 +645,7 @@ class DatabasesController extends Controller 'limits_cpus' => 'string', 'limits_cpuset' => 'string|nullable', 'limits_cpu_shares' => 'numeric', + 'instant_deploy' => 'boolean', 'redact_all_env_values_in_logs' => 'boolean', ]); @@ -654,6 +656,7 @@ class DatabasesController extends Controller ], 422); } $uuid = $request->uuid; + $instantDeploy = $request->boolean('instant_deploy'); $redactAllEnvValuesInLogs = $request->has('redact_all_env_values_in_logs') ? $request->boolean('redact_all_env_values_in_logs') : null; removeUnnecessaryFieldsFromRequest($request); $database = queryDatabaseByUuidWithinTeam($uuid, $teamId); @@ -663,6 +666,11 @@ class DatabasesController extends Controller $this->authorize('update', $database); + if ($instantDeploy) { + abort_unless($request->user()->tokenCan('deploy') || $request->user()->tokenCan('root'), 403, 'Missing required permissions: deploy'); + $this->authorize('manage', $database); + } + if ($redactAllEnvValuesInLogs !== null) { $this->authorize('manageEnvironment', $database); } @@ -890,6 +898,17 @@ class DatabasesController extends Controller 'errors' => $errors, ], 422); } + $restartReservation = null; + if ($instantDeploy) { + $prerequisiteError = StartDatabase::prerequisiteError($database); + if ($prerequisiteError !== null) { + return response()->json(['message' => $prerequisiteError], 422); + } + $restartReservation = StartDatabase::reserveOperation($database); + if ($restartReservation === null) { + return response()->json(['message' => ResourceStartActivity::DATABASE_OPERATION_IN_PROGRESS_MESSAGE], 409); + } + } $whatToDoWithDatabaseProxy = null; if ($request->is_public === false && $database->is_public === true) { $whatToDoWithDatabaseProxy = 'stop'; @@ -903,12 +922,22 @@ class DatabasesController extends Controller $database->redact_all_env_values_in_logs = $redactAllEnvValuesInLogs; } $database->update($request->only($allowedFields)); + // The Redis password is stored in the REDIS_PASSWORD variable, not in a column. + if ($database instanceof StandaloneRedis && filled($request->redis_password)) { + $database->runtime_environment_variables()->updateOrCreate( + ['key' => 'REDIS_PASSWORD'], + ['value' => $request->redis_password, 'resourceable_id' => $database->id] + ); + } if ($whatToDoWithDatabaseProxy === 'start') { StartDatabaseProxy::dispatch($database); } elseif ($whatToDoWithDatabaseProxy === 'stop') { StopDatabaseProxy::dispatch($database); } + if ($restartReservation !== null) { + StartDatabase::dispatchReserved(RestartDatabase::class, $database, $restartReservation); + } auditLog('api.database.updated', [ 'team_id' => $teamId, diff --git a/app/Http/Controllers/Api/HostingerController.php b/app/Http/Controllers/Api/HostingerController.php index 84017e7ded..af4173e39d 100644 --- a/app/Http/Controllers/Api/HostingerController.php +++ b/app/Http/Controllers/Api/HostingerController.php @@ -122,6 +122,7 @@ class HostingerController extends Controller responses: [ new OA\Response(response: 201, description: 'Hostinger VPS purchased and linked to a Coolify server.'), new OA\Response(response: 401, ref: '#/components/responses/401'), + new OA\Response(response: 403, description: 'The token needs the deploy ability to set post_install_script_id or public_key_ids.'), new OA\Response(response: 422, description: 'Validation failed.'), new OA\Response(response: 429, description: 'Hostinger rate limit exceeded.'), ] @@ -182,6 +183,10 @@ class HostingerController extends Controller ], 422); } + if ($request->filled('post_install_script_id') || $request->array('public_key_ids')) { + abort_unless($request->user()->tokenCan('deploy') || $request->user()->tokenCan('root'), 403, 'Missing required permissions: deploy'); + } + $team = Team::find($teamId); if (Team::serverLimitReached($team)) { return response()->json(['message' => 'Server limit reached for your subscription.'], 400); diff --git a/app/Http/Controllers/Api/ServerCanvasController.php b/app/Http/Controllers/Api/ServerCanvasController.php index e48b8c4aff..79c33553ad 100644 --- a/app/Http/Controllers/Api/ServerCanvasController.php +++ b/app/Http/Controllers/Api/ServerCanvasController.php @@ -29,6 +29,22 @@ class ServerCanvasController extends Controller return response()->json($this->canvas->payload($teamId, $servers, null, $this->canReadSensitive($request), $this->canEdit($request, $teamId))); } + /** + * Dashboard canvas: every server of the team with its resources and their running or queued deployments. + */ + public function dashboard(Request $request): JsonResponse + { + $teamId = getTeamIdFromToken(); + if (is_null($teamId)) { + return invalidTokenResponse(); + } + $this->authorize('viewAny', Server::class); + + $servers = Server::query()->whereTeamId($teamId)->with('settings')->orderBy('name')->get(); + + return response()->json($this->canvas->dashboardPayload($teamId, $servers, $this->canReadSensitive($request), $this->canEdit($request, $teamId))); + } + public function show(Request $request, string $uuid): JsonResponse { $teamId = getTeamIdFromToken(); @@ -73,6 +89,27 @@ class ServerCanvasController extends Controller return response()->json(['nodes' => $this->canvas->unmanagedContainerNodes($server)->all()]); } + /** + * Latest CPU, memory and disk usage of one server for its zone header on the canvas. + */ + public function usage(string $uuid): JsonResponse + { + $teamId = getTeamIdFromToken(); + if (is_null($teamId)) { + return invalidTokenResponse(); + } + $server = Server::query()->whereTeamId($teamId)->whereUuid($uuid)->with('settings')->first(); + if (! $server) { + return response()->json(['message' => 'Server not found.'], 404); + } + $this->authorize('view', $server); + if (! $server->settings->is_metrics_enabled) { + return response()->json(['cpu' => null, 'memory' => null, 'disk' => null]); + } + + return response()->json($this->canvas->usage($server)); + } + private function canReadSensitive(Request $request): bool { return (bool) $request->attributes->get('can_read_sensitive', false); diff --git a/app/Http/Controllers/Api/ServerTransferController.php b/app/Http/Controllers/Api/ServerTransferController.php index f43189abf8..ffc86b8641 100644 --- a/app/Http/Controllers/Api/ServerTransferController.php +++ b/app/Http/Controllers/Api/ServerTransferController.php @@ -3,10 +3,12 @@ namespace App\Http\Controllers\Api; use App\Http\Controllers\Controller; +use App\Models\GithubApp; +use App\Models\GitlabApp; use App\Models\Server; +use App\Rules\ValidServerIp; use App\Services\ServerTransfer\ServerTransferBundle; use App\Services\ServerTransfer\ServerTransferClaimer; -use App\Services\ServerTransfer\ServerTransferExporter; use App\Services\ServerTransfer\ServerTransferImporter; use App\Services\ServerTransfer\ServerTransferMigrator; use Illuminate\Http\JsonResponse; @@ -18,17 +20,14 @@ use Throwable; class ServerTransferController extends Controller { public function __construct( - private ServerTransferExporter $exporter, private ServerTransferImporter $importer, private ServerTransferClaimer $claimer, private ServerTransferMigrator $migrator, - ) { - abort_unless(isDev(), 404); - } + ) {} #[OA\Post( summary: 'Migrate server to another Coolify instance', - description: 'One-shot handoff: export this server, import+claim on the target instance (using the provided token), then disable automations here. Requires read:sensitive and write.', + description: 'Start a transfer in the background: export this server, import and claim it on the target instance (with the given token), then disable automations here. Follow the progress with GET /servers/{uuid}/transfer. The target URL must use https. Requires read:sensitive and write.', path: '/servers/{uuid}/migrate', operationId: 'migrate-server-between-instances', security: [['bearerAuth' => []]], @@ -43,13 +42,11 @@ class ServerTransferController extends Controller properties: [ new OA\Property(property: 'target_url', type: 'string', example: 'https://coolify-b.example.com'), new OA\Property(property: 'target_token', type: 'string', description: 'API token on the target instance (root or write)'), - new OA\Property(property: 'preserve_uuids', type: 'boolean', default: true), - new OA\Property(property: 'adopt_mode', type: 'boolean', default: true), ] ) ), responses: [ - new OA\Response(response: 200, description: 'Migrated'), + new OA\Response(response: 202, description: 'Transfer started'), new OA\Response(response: 403, description: 'Missing sensitive permission'), new OA\Response(response: 404, ref: '#/components/responses/404'), new OA\Response(response: 422, description: 'Validation or remote import failed'), @@ -83,10 +80,8 @@ class ServerTransferController extends Controller $validator = customApiValidator($request->all(), [ 'target_url' => 'required|string|url', 'target_token' => 'required|string', - 'preserve_uuids' => 'boolean|nullable', - 'adopt_mode' => 'boolean|nullable', ]); - $allowedFields = ['target_url', 'target_token', 'preserve_uuids', 'adopt_mode']; + $allowedFields = ['target_url', 'target_token']; $extraFields = array_diff(array_keys($request->all()), $allowedFields); if ($validator->fails() || $extraFields !== []) { $errors = $validator->errors(); @@ -101,59 +96,52 @@ class ServerTransferController extends Controller } try { - $result = $this->migrator->migrate( + $this->migrator->start( server: $server, targetUrl: $request->string('target_url')->toString(), targetToken: $request->string('target_token')->toString(), - preserveUuids: $request->boolean('preserve_uuids', true), - adoptMode: $request->boolean('adopt_mode', true), ); } catch (Throwable $e) { return response()->json(['message' => $e->getMessage()], 422); } + $server->refresh(); auditLog('api.server.migrate', [ 'team_id' => $teamId, 'server_uuid' => $server->uuid, - 'export_id' => $result['export_id'], - 'target_url' => $result['target_url'], + 'target_url' => data_get($server->server_metadata, 'migration.target_url'), ]); - return response()->json($result); + return response()->json([ + 'server_uuid' => $server->uuid, + 'migration' => data_get($server->server_metadata, 'migration'), + 'message' => 'Transfer started. Follow it with GET /api/v1/servers/'.$server->uuid.'/transfer.', + ], 202); } #[OA\Get( - summary: 'Export server transfer bundle', - description: 'Export a server and all resources hosted on it as a versioned transfer bundle for moving between Coolify instances. Requires read:sensitive.', - path: '/servers/{uuid}/export', - operationId: 'export-server-transfer-bundle', + summary: 'Get server transfer status', + description: 'Show the transfer state of a server on this instance: the import/claim state (transfer) and the state of a running or finished transfer to another instance (migration).', + path: '/servers/{uuid}/transfer', + operationId: 'get-server-transfer-status', security: [['bearerAuth' => []]], tags: ['Servers'], parameters: [ new OA\Parameter(name: 'uuid', in: 'path', required: true, schema: new OA\Schema(type: 'string')), - new OA\Parameter(name: 'encrypt', in: 'query', required: false, description: 'If true and passphrase is provided, return an encrypted envelope.', schema: new OA\Schema(type: 'boolean')), - new OA\Parameter(name: 'passphrase', in: 'query', required: false, description: 'Passphrase used when encrypt=true.', schema: new OA\Schema(type: 'string')), ], responses: [ - new OA\Response(response: 200, description: 'Transfer bundle'), + new OA\Response(response: 200, description: 'Transfer status'), new OA\Response(response: 401, ref: '#/components/responses/401'), - new OA\Response(response: 403, description: 'Missing sensitive permission'), new OA\Response(response: 404, ref: '#/components/responses/404'), ] )] - public function export(Request $request, string $uuid): JsonResponse + public function status(string $uuid): JsonResponse { $teamId = getTeamIdFromToken(); if (is_null($teamId)) { return invalidTokenResponse(); } - if (! $this->canReadSensitive($request)) { - return response()->json([ - 'message' => 'Exporting a server requires a token with read:sensitive (or root) ability and an admin/owner team role.', - ], 403); - } - $server = Server::whereTeamId($teamId)->whereUuid($uuid)->first(); if (! $server) { return response()->json(['message' => 'Server not found.'], 404); @@ -161,25 +149,106 @@ class ServerTransferController extends Controller $this->authorize('view', $server); - try { - $bundle = $this->exporter->export($server, includeSensitive: true); - } catch (Throwable $e) { - return response()->json(['message' => $e->getMessage()], 422); - } - - auditLog('api.server.export', [ - 'team_id' => $teamId, + return response()->json([ 'server_uuid' => $server->uuid, - 'export_id' => data_get($bundle, 'export_id'), + 'transfer' => collect((array) data_get($server->server_metadata, 'transfer', [])) + ->only(['status', 'export_id', 'imported_at', 'claimed_at', 'transferred_at', 'target_instance_url']) + ->all(), + 'migration' => data_get($server->server_metadata, 'migration'), ]); + } - if ($request->boolean('encrypt') && $request->filled('passphrase')) { - return response()->json( - ServerTransferBundle::encryptWithPassphrase($bundle, $request->string('passphrase')->toString()) - ); + #[OA\Get( + summary: 'Check server import', + description: 'Check that this instance can import a server, before the source instance starts the transfer. It checks the token, the bundle version, the server limit, servers with the same UUID or IP, and that this instance can reach the SSH port of the server.', + path: '/servers/import/check', + operationId: 'check-server-import', + security: [['bearerAuth' => []]], + tags: ['Servers'], + parameters: [ + new OA\Parameter(name: 'server_uuid', in: 'query', required: true, schema: new OA\Schema(type: 'string')), + new OA\Parameter(name: 'ip', in: 'query', required: true, schema: new OA\Schema(type: 'string')), + new OA\Parameter(name: 'port', in: 'query', required: false, schema: new OA\Schema(type: 'integer', default: 22)), + new OA\Parameter(name: 'schema_version', in: 'query', required: false, schema: new OA\Schema(type: 'integer')), + new OA\Parameter(name: 'system_github_apps[]', in: 'query', required: false, description: 'UUIDs of system-wide GitHub Apps that the server uses. The response lists the ones this instance does not have.', schema: new OA\Schema(type: 'array', items: new OA\Items(type: 'string'))), + new OA\Parameter(name: 'system_gitlab_apps[]', in: 'query', required: false, description: 'UUIDs of system-wide GitLab Apps that the server uses. The response lists the ones this instance does not have.', schema: new OA\Schema(type: 'array', items: new OA\Items(type: 'string'))), + ], + responses: [ + new OA\Response(response: 200, description: 'The server can be imported. Lists every check with its result.'), + new OA\Response(response: 401, ref: '#/components/responses/401'), + new OA\Response(response: 422, description: 'One or more checks failed. Lists every check with its result.'), + ] + )] + public function check(Request $request): JsonResponse + { + $teamId = getTeamIdFromToken(); + if (is_null($teamId)) { + return invalidTokenResponse(); } - return response()->json($bundle); + $this->authorize('create', Server::class); + + $validator = customApiValidator($request->query(), [ + 'server_uuid' => 'required|string', + 'ip' => ['required', 'string', new ValidServerIp], + 'port' => 'nullable|integer|between:1,65535', + 'schema_version' => 'nullable|integer', + 'system_github_apps' => 'nullable|array|max:100', + 'system_github_apps.*' => 'string|max:64', + 'system_gitlab_apps' => 'nullable|array|max:100', + 'system_gitlab_apps.*' => 'string|max:64', + ]); + if ($validator->fails()) { + return response()->json([ + 'message' => 'Validation failed.', + 'errors' => $validator->errors(), + ], 422); + } + + // System-wide git sources are not transferred. Applications re-link only to one with the same uuid here. + $missingSystemSources = [ + 'github' => array_values(array_diff((array) $request->query('system_github_apps', []), GithubApp::query()->where('is_system_wide', true)->whereIn('uuid', (array) $request->query('system_github_apps', []))->pluck('uuid')->all())), + 'gitlab' => array_values(array_diff((array) $request->query('system_gitlab_apps', []), GitlabApp::query()->where('is_system_wide', true)->whereIn('uuid', (array) $request->query('system_gitlab_apps', []))->pluck('uuid')->all())), + ]; + + $schemaVersion = (int) $request->query('schema_version', ServerTransferBundle::SCHEMA_VERSION); + $ip = trim((string) $request->query('ip')); + $port = (int) $request->query('port', 22); + + // The target connects to the server over SSH when it takes over management, so an unreachable + // server would leave an imported server here that this instance cannot manage. + $connection = @fsockopen($ip, $port, timeout: 5); + if ($connection !== false) { + fclose($connection); + } + + $checks = [ + [ + 'label' => 'the transfer bundle version is supported', + 'ok' => $schemaVersion === ServerTransferBundle::SCHEMA_VERSION, + 'message' => $schemaVersion === ServerTransferBundle::SCHEMA_VERSION ? null + : 'This instance reads transfer bundle version '.ServerTransferBundle::SCHEMA_VERSION.", but the source sends version {$schemaVersion}. Upgrade both instances to the same Coolify version.", + ], + ...collect($this->importer->importChecks((string) $request->query('server_uuid'), $ip, $teamId)) + ->map(fn (array $check) => collect($check)->only(['label', 'ok', 'message'])->all()) + ->all(), + [ + 'label' => "the target can reach the SSH port of the server ({$ip}:{$port})", + 'ok' => $connection !== false, + 'message' => $connection !== false ? null + : "This instance cannot reach the server at {$ip}:{$port}. Allow SSH from this instance to the server, then transfer again.", + ], + ]; + + $failed = collect($checks)->where('ok', false); + + return response()->json([ + 'message' => $failed->isEmpty() + ? 'This instance can import the server.' + : $failed->pluck('message')->implode(' '), + 'checks' => $checks, + 'missing_system_sources' => $missingSystemSources, + ], $failed->isEmpty() ? 200 : 422); } #[OA\Post( @@ -193,8 +262,7 @@ class ServerTransferController extends Controller required: true, content: new OA\JsonContent( properties: [ - new OA\Property(property: 'bundle', type: 'object', description: 'Plain or encrypted transfer bundle'), - new OA\Property(property: 'passphrase', type: 'string', nullable: true), + new OA\Property(property: 'bundle', type: 'object', description: 'Transfer bundle'), new OA\Property(property: 'dry_run', type: 'boolean', default: false), new OA\Property(property: 'preserve_uuids', type: 'boolean', default: true), new OA\Property(property: 'adopt_mode', type: 'boolean', default: true, description: 'Import without forcing redeploy; keep statuses for adoption'), @@ -224,12 +292,11 @@ class ServerTransferController extends Controller $validator = customApiValidator($request->all(), [ 'bundle' => 'required|array', - 'passphrase' => 'string|nullable', 'dry_run' => 'boolean|nullable', 'preserve_uuids' => 'boolean|nullable', 'adopt_mode' => 'boolean|nullable', ]); - $allowedFields = ['bundle', 'passphrase', 'dry_run', 'preserve_uuids', 'adopt_mode']; + $allowedFields = ['bundle', 'dry_run', 'preserve_uuids', 'adopt_mode']; $extraFields = array_diff(array_keys($request->all()), $allowedFields); if ($validator->fails() || $extraFields !== []) { $errors = $validator->errors(); @@ -243,34 +310,21 @@ class ServerTransferController extends Controller ], 422); } - $bundle = $request->input('bundle', []); - if (data_get($bundle, 'encrypted')) { - if (! $request->filled('passphrase')) { - return response()->json(['message' => 'Passphrase is required for encrypted bundles.'], 422); - } - try { - $bundle = ServerTransferBundle::decryptWithPassphrase($bundle, $request->string('passphrase')->toString()); - } catch (Throwable $e) { - return response()->json(['message' => $e->getMessage()], 422); - } - } - try { $result = $this->importer->import( - bundle: $bundle, + bundle: $request->input('bundle', []), teamId: $teamId, dryRun: $request->boolean('dry_run', false), preserveUuids: $request->boolean('preserve_uuids', true), adoptMode: $request->boolean('adopt_mode', true), ); } catch (Throwable $e) { - $status = $e instanceof ValidationException ? 422 : 422; $payload = ['message' => $e->getMessage()]; if ($e instanceof ValidationException) { $payload['errors'] = $e->errors(); } - return response()->json($payload, $status); + return response()->json($payload, 422); } auditLog('api.server.import', [ @@ -285,7 +339,7 @@ class ServerTransferController extends Controller #[OA\Post( summary: 'Mark server transferred', - description: 'Source-instance step: disable automations after a successful export/import handoff.', + description: 'Source-instance recovery step: disable automations here when the target already manages the server but the transfer could not finish on this instance.', path: '/servers/{uuid}/transfer/complete', operationId: 'complete-server-transfer', security: [['bearerAuth' => []]], @@ -350,77 +404,6 @@ class ServerTransferController extends Controller return response()->json($result); } - #[OA\Post( - summary: 'Write transfer bundle to server mailbox', - description: 'Write an export bundle to /data/coolify/exports on the managed host for air-gapped import.', - path: '/servers/{uuid}/export/mailbox', - operationId: 'export-server-transfer-mailbox', - security: [['bearerAuth' => []]], - tags: ['Servers'], - parameters: [ - new OA\Parameter(name: 'uuid', in: 'path', required: true, schema: new OA\Schema(type: 'string')), - ], - requestBody: new OA\RequestBody( - content: new OA\JsonContent( - properties: [ - new OA\Property(property: 'passphrase', type: 'string', nullable: true), - ] - ) - ), - responses: [ - new OA\Response(response: 200, description: 'Mailbox write result'), - new OA\Response(response: 403, description: 'Missing sensitive permission'), - new OA\Response(response: 404, ref: '#/components/responses/404'), - ] - )] - public function writeMailbox(Request $request, string $uuid): JsonResponse - { - $teamId = getTeamIdFromToken(); - if (is_null($teamId)) { - return invalidTokenResponse(); - } - - if (! $this->canReadSensitive($request)) { - return response()->json([ - 'message' => 'Writing a transfer mailbox requires read:sensitive (or root) ability and an admin/owner team role.', - ], 403); - } - - $server = Server::whereTeamId($teamId)->whereUuid($uuid)->first(); - if (! $server) { - return response()->json(['message' => 'Server not found.'], 404); - } - - $this->authorize('view', $server); - - try { - $bundle = $this->exporter->export($server, includeSensitive: true); - $result = $this->claimer->writeMailbox( - $server, - $bundle, - $request->filled('passphrase') ? $request->string('passphrase')->toString() : null, - ); - } catch (Throwable $e) { - return response()->json(['message' => $e->getMessage()], 422); - } - - auditLog('api.server.export_mailbox', [ - 'team_id' => $teamId, - 'server_uuid' => $server->uuid, - 'export_id' => data_get($bundle, 'export_id'), - 'path' => $result['path'], - ]); - - return response()->json([ - 'export_id' => data_get($bundle, 'export_id'), - 'path' => $result['path'], - 'written' => $result['written'], - 'message' => $result['written'] - ? 'Transfer bundle written to server mailbox.' - : 'Failed to write mailbox on remote host.', - ], $result['written'] ? 200 : 422); - } - private function canReadSensitive(Request $request): bool { return (bool) $request->attributes->get('can_read_sensitive', false); diff --git a/app/Http/Controllers/Api/ServicesController.php b/app/Http/Controllers/Api/ServicesController.php index c858c1bc8c..0a86606518 100644 --- a/app/Http/Controllers/Api/ServicesController.php +++ b/app/Http/Controllers/Api/ServicesController.php @@ -210,6 +210,7 @@ class ServicesController extends Controller $application->fqdn = $containerUrls; $application->save(); + updateCompose($application); }); if (! empty($errors)) { @@ -308,6 +309,7 @@ class ServicesController extends Controller 'destination_uuid' => ['type' => 'string', 'description' => 'Destination UUID. Required if server has multiple destinations.'], 'instant_deploy' => ['type' => 'boolean', 'default' => false, 'description' => 'Start the service immediately after creation.'], 'docker_compose_raw' => ['type' => 'string', 'description' => 'The base64 encoded Docker Compose content.'], + 'connect_to_docker_network' => ['type' => 'boolean', 'default' => false, 'description' => 'Connect the service to the predefined docker network.'], 'urls' => [ 'type' => 'array', 'description' => 'Array of URLs to be applied to containers of a service.', @@ -389,7 +391,7 @@ class ServicesController extends Controller )] public function create_service(Request $request) { - $allowedFields = ['type', 'name', 'description', 'project_uuid', 'environment_name', 'environment_uuid', 'server_uuid', 'destination_uuid', 'instant_deploy', 'docker_compose_raw', 'urls', 'force_domain_override', 'is_container_label_escape_enabled', 'redact_all_env_values_in_logs', 'tags']; + $allowedFields = ['type', 'name', 'description', 'project_uuid', 'environment_name', 'environment_uuid', 'server_uuid', 'destination_uuid', 'instant_deploy', 'docker_compose_raw', 'connect_to_docker_network', 'urls', 'force_domain_override', 'is_container_label_escape_enabled', 'redact_all_env_values_in_logs', 'tags']; $teamId = getTeamIdFromToken(); if (is_null($teamId)) { @@ -417,6 +419,7 @@ class ServicesController extends Controller 'name' => 'string|max:255', 'description' => 'string|nullable', 'instant_deploy' => 'boolean', + 'connect_to_docker_network' => 'boolean', 'urls' => 'array|nullable', 'urls.*' => 'array:name,url', 'urls.*.name' => 'string|required', @@ -546,7 +549,7 @@ class ServicesController extends Controller 'destination_id' => $destination->id, 'destination_type' => $destination->getMorphClass(), ]; - if (in_array($oneClickServiceName, NEEDS_TO_CONNECT_TO_PREDEFINED_NETWORK)) { + if (in_array($oneClickServiceName, NEEDS_TO_CONNECT_TO_PREDEFINED_NETWORK) || $request->boolean('connect_to_docker_network')) { data_set($servicePayload, 'connect_to_docker_network', true); } $service = new Service($servicePayload); diff --git a/app/Http/Controllers/FileBrowserDownloadController.php b/app/Http/Controllers/FileBrowserDownloadController.php new file mode 100644 index 0000000000..17bed2392e --- /dev/null +++ b/app/Http/Controllers/FileBrowserDownloadController.php @@ -0,0 +1,53 @@ +id(), 404); + + Gate::authorize('canAccessTerminal'); + $resource = $download['resource_type']::findOrFail($download['resource_id']); + Gate::authorize('view', $resource); + abort_unless((int) $resource->environment->project->team_id === (int) auth()->user()->currentTeam()->id, 403); + + $filesystem = new ContainerFilesystemService(Server::findOrFail($download['server_id']), $download['container']); + + return response()->streamDownload(function () use ($filesystem, $download) { + set_time_limit(0); + $filesystem->streamDownload($download['path'], $download['is_dir'], function (string $chunk) { + echo $chunk; + if (ob_get_level() > 0) { + ob_flush(); + } + flush(); + }); + }, $download['name'], [ + 'Content-Type' => 'application/octet-stream', + // Stop nginx from buffering the response to a temp file on the Coolify host. + 'X-Accel-Buffering' => 'no', + ]); + } +} diff --git a/app/Http/Controllers/Webhook/Bitbucket.php b/app/Http/Controllers/Webhook/Bitbucket.php index e06a14e0bf..3138d168a3 100644 --- a/app/Http/Controllers/Webhook/Bitbucket.php +++ b/app/Http/Controllers/Webhook/Bitbucket.php @@ -11,6 +11,7 @@ use App\Http\Controllers\Webhook\Concerns\ReadsWebhookPushPayload; use App\Http\Controllers\Webhook\Concerns\ValidatesPreviewDeploymentRepository; use App\Models\Application; use App\Models\ApplicationPreview; +use App\Services\ServerTransfer\TransferFinishChecklist; use Exception; use Illuminate\Http\Request; @@ -84,7 +85,11 @@ class Bitbucket extends Controller $failure_key = $this->manualWebhookFailureRateLimitKey($request, 'bitbucket', $full_name, $branch); // A redelivery of the same signed payload is one guess. $failure_attempt = $this->manualWebhookSignedPayloadAttempt($request, $x_bitbucket_token); - $applications = $this->manualWebhookApplications(Application::query()->where('git_branch', $branch), $full_name); + $applications = Application::query(); + if ($x_bitbucket_event !== 'pullrequest:rejected' && $x_bitbucket_event !== 'pullrequest:fulfilled') { + $applications->where('git_branch', $branch); + } + $applications = $this->manualWebhookApplications($applications, $full_name); if ($applications->isEmpty()) { return $this->unauthenticatedManualWebhookResponse($failure_key, $failure_attempt); } @@ -128,6 +133,7 @@ class Bitbucket extends Controller continue; } + TransferFinishChecklist::recordWebhook('application', $application->uuid); $isFunctional = $application->destination->server->isFunctional(); if (! $isFunctional) { $return_payloads->push([ diff --git a/app/Http/Controllers/Webhook/Gitea.php b/app/Http/Controllers/Webhook/Gitea.php index b3c0b22a48..05ceebeabd 100644 --- a/app/Http/Controllers/Webhook/Gitea.php +++ b/app/Http/Controllers/Webhook/Gitea.php @@ -11,6 +11,7 @@ use App\Http\Controllers\Webhook\Concerns\ReadsWebhookPushPayload; use App\Http\Controllers\Webhook\Concerns\ValidatesPreviewDeploymentRepository; use App\Models\Application; use App\Models\ApplicationPreview; +use App\Services\ServerTransfer\TransferFinishChecklist; use Exception; use Illuminate\Http\Request; use Illuminate\Support\Str; @@ -84,7 +85,10 @@ class Gitea extends Controller } } if ($x_gitea_event === 'pull_request') { - $applications = $this->manualWebhookApplications($applications->where('git_branch', $base_branch), $full_name); + if ($action !== 'closed') { + $applications->where('git_branch', $base_branch); + } + $applications = $this->manualWebhookApplications($applications, $full_name); if ($applications->isEmpty()) { return $this->unauthenticatedManualWebhookResponse($failure_key, $failure_attempt); } @@ -114,6 +118,7 @@ class Gitea extends Controller continue; } + TransferFinishChecklist::recordWebhook('application', $application->uuid); $isFunctional = $application->destination->server->isFunctional(); if (! $isFunctional) { $return_payloads->push([ diff --git a/app/Http/Controllers/Webhook/Github.php b/app/Http/Controllers/Webhook/Github.php index ba270cf688..e615abd3c6 100644 --- a/app/Http/Controllers/Webhook/Github.php +++ b/app/Http/Controllers/Webhook/Github.php @@ -19,6 +19,7 @@ use App\Models\GithubRunnerExecution; use App\Models\PrivateKey; use App\Services\GithubRunner\GithubRunnerApi; use App\Services\GithubRunner\GithubRunnerContainer; +use App\Services\ServerTransfer\TransferFinishChecklist; use Exception; use Illuminate\Http\Exceptions\HttpResponseException; use Illuminate\Http\RedirectResponse; @@ -71,7 +72,6 @@ class Github extends Controller 'before_sha' => $before_sha, 'after_sha' => $after_sha, 'commit_sha' => $commit_sha, - 'author_association' => $author_association, 'is_fork_pull_request' => $is_fork_pull_request, ] = $this->readPullRequestPayload($payload); $full_name = data_get($payload, 'repository.full_name'); @@ -136,6 +136,7 @@ class Github extends Controller continue; } + TransferFinishChecklist::recordWebhook('application', $application->uuid); $isFunctional = $application->destination->server->isFunctional(); if (! $isFunctional) { $return_payloads->push([ @@ -239,7 +240,6 @@ class Github extends Controller beforeSha: $before_sha, afterSha: $after_sha, commitSha: $commit_sha, - authorAssociation: $author_association, fullName: $full_name, isForkPullRequest: $is_fork_pull_request, ); @@ -400,7 +400,12 @@ class Github extends Controller // Just pong return response('pong'); } - $github_app = GithubApp::where('app_id', $x_github_hook_installation_target_id)->first(); + // One GitHub App can be installed on several accounts, each added as its own source. + $installation_id = data_get($payload, 'installation.id'); + $github_app = is_numeric($installation_id) + ? GithubApp::where('app_id', $x_github_hook_installation_target_id)->where('installation_id', (int) $installation_id)->first() + : null; + $github_app ??= GithubApp::where('app_id', $x_github_hook_installation_target_id)->first(); if (is_null($github_app)) { return response('Nothing to do. No GitHub App found.'); } @@ -428,6 +433,7 @@ class Github extends Controller return response('Invalid signature.'); } } + TransferFinishChecklist::recordWebhook('github_app', $github_app->uuid); if ($x_github_event === 'installation' || $x_github_event === 'installation_repositories') { // Installation handled by setup redirect url. Repositories queried on-demand. $action = data_get($payload, 'action'); @@ -459,7 +465,6 @@ class Github extends Controller 'before_sha' => $before_sha, 'after_sha' => $after_sha, 'commit_sha' => $commit_sha, - 'author_association' => $author_association, 'is_fork_pull_request' => $is_fork_pull_request, ] = $this->readPullRequestPayload($payload); $full_name = $this->manualWebhookRepositoryFullName(data_get($payload, 'repository.full_name')); @@ -484,6 +489,9 @@ class Github extends Controller if ($applications->isEmpty()) { return response("Nothing to do. No applications found with branch '$branch'."); } + if ($applications->contains(fn (Application $application): bool => filled($application->watch_paths))) { + $changed_files = $this->githubPushComparedFiles($github_app, $payload) ?? $changed_files; + } } if ($x_github_event === 'pull_request') { if ($action !== 'closed') { @@ -598,7 +606,6 @@ class Github extends Controller beforeSha: $before_sha, afterSha: $after_sha, commitSha: $commit_sha, - authorAssociation: $author_association, fullName: $full_name, isForkPullRequest: $is_fork_pull_request, ); @@ -620,10 +627,39 @@ class Github extends Controller } } + /** + * Files that the push changes between its before and after commits, from + * the GitHub compare API. + * + * The file lists of the pushed commits are not the net change: a merge + * commit names the files that it changes relative to its first parent, + * also when the push does not change them. Returns null for a new branch, + * a missing repository name or an API failure; the caller then uses the + * file lists of the pushed commits. + * + * @return Collection|null + * + * @throws InvalidWebhookPayloadException When the before or after value is not a commit SHA. + */ + private function githubPushComparedFiles(GithubApp $githubApp, mixed $payload): ?Collection + { + $before = $this->webhookCommitSha($payload, 'before'); + $after = $this->webhookCommitSha($payload, 'after'); + $fullName = $this->manualWebhookRepositoryFullName(data_get($payload, 'repository.full_name')); + if ($before === null || $after === null || $fullName === null || preg_match('/\A0+\z/', $before) === 1) { + return null; + } + + [$owner, $repo] = explode('/', $fullName, 2); + $files = getGithubCommitRangeFiles($githubApp, $owner, $repo, $before, $after); + + return $files === null ? null : collect($files); + } + /** * Read and validate the pull_request payload fields that the handlers use. * - * @return array{action: string, pull_request_id: int, pull_request_html_url: string, pull_request_title: ?string, branch: ?string, base_branch: ?string, before_sha: ?string, after_sha: ?string, commit_sha: string, author_association: ?string, is_fork_pull_request: bool} + * @return array{action: string, pull_request_id: int, pull_request_html_url: string, pull_request_title: ?string, branch: ?string, base_branch: ?string, before_sha: ?string, after_sha: ?string, commit_sha: string, is_fork_pull_request: bool} * * @throws InvalidWebhookPayloadException When a field has a wrong type or format. */ @@ -641,7 +677,6 @@ class Github extends Controller 'before_sha' => $this->webhookCommitSha($payload, 'before'), 'after_sha' => $this->webhookCommitSha($payload, 'after') ?? $head_sha, 'commit_sha' => $head_sha ?? 'HEAD', - 'author_association' => $this->webhookPayloadString($payload, 'pull_request.author_association'), 'is_fork_pull_request' => $this->isForkPullRequest($payload), ]; } @@ -649,13 +684,15 @@ class Github extends Controller /** * Determine whether a pull_request webhook payload originates from a fork. * - * GitHub's `author_association` is not a reliable trust signal (it grants - * CONTRIBUTOR to anyone who has merely opened an issue/PR before), so fork - * detection is gated on whether the PR crosses repository boundaries. + * A same-repository PR branch can only be pushed by users with write access, + * so this is the trust signal for preview deployments. GitHub's + * `author_association` is not reliable for this: users with access through a + * team or a private organization membership arrive as CONTRIBUTOR or NONE. * * The repository id comparison is the canonical signal; the `head.repo.fork` * flag and a case-insensitive full_name comparison are fallbacks for payloads - * where the ids are unavailable (e.g. a deleted head repository). + * where the ids are unavailable. A PR whose head repository is unknown (e.g. + * a deleted fork) is treated as a fork. * * @throws InvalidWebhookPayloadException When a repository id is not a positive integer. */ @@ -679,7 +716,7 @@ class Github extends Controller return Str::lower($headRepoFullName) !== Str::lower($baseRepoFullName); } - return false; + return true; } public function redirect(Request $request) diff --git a/app/Http/Controllers/Webhook/Gitlab.php b/app/Http/Controllers/Webhook/Gitlab.php index c135aef328..892408976d 100644 --- a/app/Http/Controllers/Webhook/Gitlab.php +++ b/app/Http/Controllers/Webhook/Gitlab.php @@ -13,8 +13,10 @@ use App\Livewire\Source\Gitlab\Change as GitlabSource; use App\Models\Application; use App\Models\ApplicationPreview; use App\Models\GitlabApp; +use App\Services\ServerTransfer\TransferFinishChecklist; use Exception; use Illuminate\Http\Request; +use Illuminate\Support\Collection; use Illuminate\Support\Facades\Cache; use Illuminate\Support\Facades\Http; @@ -84,6 +86,9 @@ class Gitlab extends Controller try { $return_payloads = collect([]); $payload = $request->collect(); + if ($request->header('X-Gitlab-Event') === 'System Hook') { + $payload = $this->normalizeSystemHookPayload($payload); + } $x_gitlab_token = $request->header('X-Gitlab-Token'); $object_kind = $this->webhookString(data_get($payload, 'object_kind')); @@ -118,6 +123,7 @@ class Gitlab extends Controller ], 401); } + TransferFinishChecklist::recordWebhook('gitlab_app', $gitlab_app->uuid); $project_id = $this->webhookPayloadDatabaseId($payload, 'project.id', required: true); $applications = Application::where('source_id', $gitlab_app->id) ->where('source_type', GitlabApp::class) @@ -231,7 +237,10 @@ class Gitlab extends Controller 'skip_deploy' => $skip_deploy_pr, ] = $this->readMergeRequestPayload($payload); - $applications = $applications->where('git_branch', $base_branch)->get(); + if (! in_array($action, ['closed', 'close', 'merge'])) { + $applications->where('git_branch', $base_branch); + } + $applications = $applications->get(); foreach ($applications as $application) { if (! $application->destination->server->isFunctional()) { @@ -440,7 +449,10 @@ class Gitlab extends Controller } } if ($x_gitlab_event === 'merge_request') { - $applications = $this->manualWebhookApplications($applications->where('git_branch', $base_branch), $full_name); + if (! in_array($action, ['closed', 'close', 'merge'])) { + $applications->where('git_branch', $base_branch); + } + $applications = $this->manualWebhookApplications($applications, $full_name); if ($applications->isEmpty()) { return $this->unauthenticatedManualWebhookResponse($failure_key, $failure_attempt); } @@ -469,6 +481,7 @@ class Gitlab extends Controller continue; } + TransferFinishChecklist::recordWebhook('application', $application->uuid); $isFunctional = $application->destination->server->isFunctional(); if (! $isFunctional) { $return_payloads->push([ @@ -661,6 +674,30 @@ class Gitlab extends Controller } } + /** + * Give a system hook payload the project webhook shape. Current GitLab sends + * the project webhook data to system hooks, but the documented push payload + * has only `event_name` and a top-level `project_id`. + * + * @see https://docs.gitlab.com/administration/system_hooks/#push-events + * + * @param Collection $payload + * @return Collection + */ + private function normalizeSystemHookPayload(Collection $payload): Collection + { + if (! $payload->has('object_kind') && $payload->has('event_name')) { + $payload->put('object_kind', $payload->get('event_name')); + } + + $project = $payload->get('project'); + if (is_array($project) && ! array_key_exists('id', $project) && $payload->has('project_id')) { + $payload->put('project', ['id' => $payload->get('project_id')] + $project); + } + + return $payload; + } + /** * Read and validate the merge_request payload fields that the handlers use. * diff --git a/app/Http/Kernel.php b/app/Http/Kernel.php index 2e6c36b722..48710f88fe 100644 --- a/app/Http/Kernel.php +++ b/app/Http/Kernel.php @@ -16,6 +16,7 @@ use App\Http\Middleware\EncryptCookies; use App\Http\Middleware\EnsureMcpEnabled; use App\Http\Middleware\EnsureTeamMcpEnabled; use App\Http\Middleware\EnsureTokenBelongsToCurrentTeamMember; +use App\Http\Middleware\KeepConcurrentTeamSwitch; use App\Http\Middleware\PreventRequestsDuringMaintenance; use App\Http\Middleware\RedirectIfAuthenticated; use App\Http\Middleware\RequirePassword; @@ -73,6 +74,7 @@ class Kernel extends HttpKernel EncryptCookies::class, AddQueuedCookiesToResponse::class, StartSession::class, + KeepConcurrentTeamSwitch::class, ShareErrorsFromSession::class, VerifyCsrfToken::class, SubstituteBindings::class, diff --git a/app/Http/Middleware/ApiIpAllowed.php b/app/Http/Middleware/ApiIpAllowed.php index da2d009576..ede32567d0 100644 --- a/app/Http/Middleware/ApiIpAllowed.php +++ b/app/Http/Middleware/ApiIpAllowed.php @@ -14,13 +14,9 @@ class ApiIpAllowed return $next($request); } - $allowedIps = instanceSettings()->allowed_ips; - if (! $allowedIps) { - return $next($request); - } - - // Check for special case: 0.0.0.0 means allow all - if (trim($allowedIps) === '0.0.0.0') { + $allowedIps = trim((string) instanceSettings()->allowed_ips); + // Empty or 0.0.0.0 means allow all + if ($allowedIps === '' || $allowedIps === '0.0.0.0') { return $next($request); } diff --git a/app/Http/Middleware/KeepConcurrentTeamSwitch.php b/app/Http/Middleware/KeepConcurrentTeamSwitch.php new file mode 100644 index 0000000000..7f9a796ade --- /dev/null +++ b/app/Http/Middleware/KeepConcurrentTeamSwitch.php @@ -0,0 +1,37 @@ +session()->get('currentTeam'), 'id'); + + $response = $next($request); + + $session = $request->session(); + if (data_get($session->get('currentTeam'), 'id') !== $teamIdAtStart) { + return $response; + } + + $storedSession = clone $session; + $storedSession->start(); + $storedTeam = $storedSession->get('currentTeam'); + if ($storedTeam && data_get($storedTeam, 'id') !== $teamIdAtStart) { + $session->put('currentTeam', $storedTeam); + } + + return $response; + } +} diff --git a/app/Jobs/ApplicationDeploymentJob.php b/app/Jobs/ApplicationDeploymentJob.php index 710dcb823c..ec95267419 100644 --- a/app/Jobs/ApplicationDeploymentJob.php +++ b/app/Jobs/ApplicationDeploymentJob.php @@ -260,6 +260,8 @@ class ApplicationDeploymentJob implements ShouldBeEncrypted, ShouldQueue $source = data_get($this->application, 'source'); if ($source) { $this->source = $source->getMorphClass()::where('id', $this->application->source->id)->first(); + // Share one source instance so the job reuses its in-memory GitHub installation token. + $this->application->setRelation('source', $this->source); } $this->server = Server::find($this->application_deployment_queue->server_id); $this->timeout = $this->server->settings->dynamic_timeout; @@ -975,18 +977,35 @@ class ApplicationDeploymentJob implements ShouldBeEncrypted, ShouldQueue return $composeFile; } + /** + * Pulls the registry images of the Compose services before the current deployment stops. Images that + * this deployment builds are not pulled: a pull fails for an image that is only local, and a pull of + * an image that is also in a registry replaces the new build with the old registry image. + */ private function pull_docker_compose_images(): void { + if ($this->docker_compose_custom_build_command && ! $this->use_build_server) { + $this->application_deployment_queue->addLogEntry('Skipping the image pull before stopping the current deployment, because the custom build command can build images outside of Docker Compose.'); + + return; + } + + $services = $this->registry_image_compose_services(); + if ($services->isEmpty()) { + return; + } + $this->application_deployment_queue->addLogEntry('Pulling image-based services before stopping the current deployment.'); + $services = $services->map(fn (string $service) => escapeshellarg($service))->implode(' '); if ($this->use_build_server) { $this->write_deployment_configurations(); $this->server = $this->mainServer; $workdir = $this->application->workdir(); - $command = "{$this->coolify_variables} docker compose --env-file {$workdir}/.env --project-name {$this->application->uuid} --project-directory {$workdir} -f {$workdir}{$this->docker_compose_location} pull --ignore-buildable"; + $command = "{$this->coolify_variables} docker compose --env-file {$workdir}/.env --project-name {$this->application->uuid} --project-directory {$workdir} -f {$workdir}{$this->docker_compose_location} pull --ignore-buildable {$services}"; } else { $workdir = $this->workdir; - $command = executeInDocker($this->deployment_uuid, "{$this->coolify_variables} docker compose --env-file {$workdir}/.env --project-name {$this->application->uuid} --project-directory {$workdir} -f {$workdir}{$this->docker_compose_location} pull --ignore-buildable"); + $command = executeInDocker($this->deployment_uuid, "{$this->coolify_variables} docker compose --env-file {$workdir}/.env --project-name {$this->application->uuid} --project-directory {$workdir} -f {$workdir}{$this->docker_compose_location} pull --ignore-buildable {$services}"); } $this->execute_remote_command([ @@ -995,6 +1014,26 @@ class ApplicationDeploymentJob implements ShouldBeEncrypted, ShouldQueue ]); } + /** + * Returns the names of the Compose services that get their image from a registry. A service with a + * `build` section, or a service that uses the image of such a service, gets its image from the build. + * + * @return Collection + */ + private function registry_image_compose_services(): Collection + { + $services = collect(data_get(Yaml::parse($this->generated_docker_compose), 'services', [])); + $builtImages = $services->filter(fn ($service) => filled(data_get($service, 'build')))->pluck('image')->filter(); + + return $services + ->filter(fn ($service) => filled(data_get($service, 'image')) + && blank(data_get($service, 'build')) + && ! $builtImages->contains(data_get($service, 'image'))) + ->keys() + ->map(fn ($service) => (string) $service) + ->values(); + } + /** * Starts the Compose containers. Content files are written first, because `docker compose up` * creates a missing bind source as an empty directory. @@ -1711,7 +1750,7 @@ class ApplicationDeploymentJob implements ShouldBeEncrypted, ShouldQueue $ports = $this->application->main_port(); $coolify_envs = $this->generate_coolify_env_variables(); $coolify_envs->each(function ($item, $key) use ($envs) { - $envs->push($key.'='.$item); + $envs->push($key.'='.escapeComposeEnvFileValue($item)); }); if ($this->pull_request_id === 0) { // Generate SERVICE_ variables first for dockercompose @@ -3189,13 +3228,12 @@ class ApplicationDeploymentJob implements ShouldBeEncrypted, ShouldQueue private function clone_repository() { - $importCommands = $this->generate_git_import_commands(); $this->application_deployment_queue->addLogEntry("\n----------------------------------------"); $this->application_deployment_queue->addLogEntry("Importing {$this->customRepository}:{$this->application->git_branch} (commit sha {$this->commit}) to {$this->basedir}."); if ($this->pull_request_id !== 0) { $this->application_deployment_queue->addLogEntry("Checking out tag pull/{$this->pull_request_id}/head."); } - $this->execute_remote_command(...$this->gitCommandDefinitions($importCommands)); + $this->import_repository(); $this->create_workdir(); $this->execute_remote_command( [ @@ -3213,6 +3251,37 @@ class ApplicationDeploymentJob implements ShouldBeEncrypted, ShouldQueue } } + /** + * SSH exits with 255 when the connection drops, which can happen during a long clone. Git exits with + * 128 for its own errors, so only 255 is retried. The interrupted clone can still run in the helper + * container, so stop it and remove the partial checkout before cloning again. + */ + private function import_repository(): void + { + $maxAttempts = config('constants.ssh.max_retries'); + + for ($attempt = 1; ; $attempt++) { + try { + $this->execute_remote_command(...$this->gitCommandDefinitions($this->generate_git_import_commands())); + + return; + } catch (DeploymentException $e) { + if ($e->getCode() !== 255 || $attempt >= $maxAttempts) { + throw $e; + } + + $delay = $this->calculateRetryDelay($attempt - 1); + $this->application_deployment_queue->addLogEntry('Lost the SSH connection while importing the repository. Retrying in '.$delay.'s (attempt '.($attempt + 1)."/{$maxAttempts}).", 'stderr'); + Sleep::for($delay)->seconds(); + $this->execute_remote_command([ + executeInDocker($this->deployment_uuid, 'pkill git; rm -rf '.escapeshellarg($this->basedir)), + 'hidden' => true, + 'ignore_errors' => true, + ]); + } + } + } + private function generate_git_import_commands() { ['commands' => $commands, 'branch' => $this->branch, 'fullRepoUrl' => $this->fullRepoUrl] = $this->application->generateGitImportCommands( @@ -3910,7 +3979,10 @@ COPY ./nginx.conf /etc/nginx/conf.d/default.conf"); } } - add_coolify_default_environment_variables($this->application, $coolify_envs, $this->application->environment_variables_preview); + // Runtime only, so a project or server change does not break the Docker build cache + if (! $forBuildTime) { + $coolify_envs = $coolify_envs->merge(coolifyDefaultEnvironmentVariables($this->application, $this->application->environment_variables_preview->pluck('key'), $this->mainServer)); + } } else { // Only add SOURCE_COMMIT for runtime OR when explicitly enabled for build-time @@ -3954,7 +4026,10 @@ COPY ./nginx.conf /etc/nginx/conf.d/default.conf"); } } - add_coolify_default_environment_variables($this->application, $coolify_envs, $this->application->environment_variables); + // Runtime only, so a project or server change does not break the Docker build cache + if (! $forBuildTime) { + $coolify_envs = $coolify_envs->merge(coolifyDefaultEnvironmentVariables($this->application, $this->application->environment_variables->pluck('key'), $this->mainServer)); + } } diff --git a/app/Jobs/CheckHelperImageJob.php b/app/Jobs/CheckHelperImageJob.php index 6d76da8eb7..56118a238a 100644 --- a/app/Jobs/CheckHelperImageJob.php +++ b/app/Jobs/CheckHelperImageJob.php @@ -9,6 +9,7 @@ use Illuminate\Foundation\Bus\Dispatchable; use Illuminate\Queue\InteractsWithQueue; use Illuminate\Queue\SerializesModels; use Illuminate\Support\Facades\Http; +use Illuminate\Support\Facades\Log; class CheckHelperImageJob implements ShouldBeEncrypted, ShouldQueue { @@ -20,20 +21,18 @@ class CheckHelperImageJob implements ShouldBeEncrypted, ShouldQueue public function handle(): void { - try { - $response = Http::retry(3, 1000)->get(config('constants.coolify.versions_url')); - if ($response->successful()) { - $versions = $response->json(); - $settings = instanceSettings(); - $latest_version = data_get($versions, 'coolify.helper.version'); - $current_version = $settings->helper_version; - if (version_compare($latest_version, $current_version, '>')) { - $settings->update(['helper_version' => $latest_version]); - } - } - } catch (\Throwable $e) { - send_internal_notification('CheckHelperImageJob failed with: '.$e->getMessage()); - throw $e; + $response = Http::retry(3, 1000, throw: false)->get(config('constants.coolify.versions_url')); + if ($response->failed()) { + // The CDN can be briefly unavailable. The next scheduled run tries again, so do not notify. + Log::warning('CheckHelperImageJob could not fetch versions', ['status' => $response->status()]); + + return; + } + + $latest_version = data_get($response->json(), 'coolify.helper.version'); + $settings = instanceSettings(); + if (version_compare($latest_version, $settings->helper_version, '>')) { + $settings->update(['helper_version' => $latest_version]); } } } diff --git a/app/Jobs/CheckTransferFinishStepsJob.php b/app/Jobs/CheckTransferFinishStepsJob.php new file mode 100644 index 0000000000..ebd47f5871 --- /dev/null +++ b/app/Jobs/CheckTransferFinishStepsJob.php @@ -0,0 +1,39 @@ +server->id; + } + + public function handle(TransferFinishChecklist $checklist): void + { + $checklist->checkAll($this->server); + } +} diff --git a/app/Jobs/CleanupGithubRunnerJob.php b/app/Jobs/CleanupGithubRunnerJob.php index baca8504f0..2caf75f505 100644 --- a/app/Jobs/CleanupGithubRunnerJob.php +++ b/app/Jobs/CleanupGithubRunnerJob.php @@ -36,6 +36,14 @@ class CleanupGithubRunnerJob implements ShouldQueue return; } + // Another Coolify instance manages a transferred server and its runners: never touch them from here. + if ($execution->server?->isTransferredAway()) { + Log::info('Skipped GitHub runner cleanup: the server was transferred to another instance', ['execution' => $execution->uuid]); + ProvisionGithubRunnerJob::dispatchQueued($execution->github_app_id); + + return; + } + if ($execution->server) { instant_remote_process(GithubRunnerContainer::cleanupCommands($execution->uuid), $execution->server, false); } diff --git a/app/Jobs/CleanupStaleMultiplexedConnections.php b/app/Jobs/CleanupStaleMultiplexedConnections.php index 69e151d24f..3162ae9c41 100644 --- a/app/Jobs/CleanupStaleMultiplexedConnections.php +++ b/app/Jobs/CleanupStaleMultiplexedConnections.php @@ -28,8 +28,14 @@ class CleanupStaleMultiplexedConnections implements ShouldQueue /** * Kill backgrounded ssh master processes that lost the ControlPath socket * race. Such processes are not masters, so ControlPersist never reaps them - * and they leak memory until the container restarts. A legitimate master - * always owns its socket file; an orphan has none. + * and they hold an authenticated session until the container restarts. + * + * Every `ssh -fN` for a server carries the same ControlPath, so the socket + * file existing says nothing about which process owns it. Ownership is read + * from the socket itself (`ssh -O check` reports the master pid); every + * other process for that ControlPath is an orphan. When the socket file is + * missing, no process owns it. When it exists but does not answer, the + * owner is unknown and nothing is reaped. * * Processes younger than the minimum age are skipped: a freshly forked * master creates its socket a few milliseconds after starting, so a young @@ -39,6 +45,7 @@ class CleanupStaleMultiplexedConnections implements ShouldQueue { $muxDir = storage_path('app/ssh/mux'); $minAge = (int) config('constants.ssh.mux_orphan_min_age'); + $socketOwners = []; foreach ($this->listProcesses() as $process) { // Backgrounded ssh master: current `ssh -fN` or legacy `ssh -fNM`. @@ -51,20 +58,44 @@ class CleanupStaleMultiplexedConnections implements ShouldQueue continue; } - if ($process['etimes'] >= $minAge - && ! file_exists($pathMatch[1]) - && ! SshMultiplexingHelper::isMuxProcessRetiring($process['pid'], $pathMatch[1])) { + $muxSocket = $pathMatch[1]; + if ($process['etimes'] < $minAge + || SshMultiplexingHelper::isMuxProcessRetiring($process['pid'], $muxSocket)) { + continue; + } + + $socketOwners[$muxSocket] ??= $this->muxSocketOwner($muxSocket); + if ($socketOwners[$muxSocket] !== false && $socketOwners[$muxSocket] !== $process['pid']) { $this->reapOrphan('ssh', $process); } } } + /** + * Pid of the master that owns a mux socket. Returns null when the socket + * file does not exist, and false when it exists but its owner is unknown. + */ + private function muxSocketOwner(string $muxSocket): string|false|null + { + if (! file_exists($muxSocket)) { + return null; + } + + $check = Process::run('timeout 10 ssh -O check -o ControlPath='.escapeshellarg($muxSocket).' localhost'); + + return preg_match('/pid=(\d+)/', $check->output().$check->errorOutput(), $matches) + ? $matches[1] + : false; + } + /** * Kill orphaned `cloudflared access ssh` proxy processes. Each is spawned * as the SSH ProxyCommand transport for a Cloudflare Tunnel server and must * die with its parent ssh. When that ssh is killed or orphaned (e.g. a lost * mux master), the cloudflared process can leak and accumulate. A legitimate - * proxy always has a live ssh parent; one without is safe to reap. + * proxy always has a live ssh at the other end of its stdin pipe. Its parent + * is not enough: `ssh -f` forks after connecting, so the proxy of a live + * master is reparented away from ssh. * * Processes younger than the minimum age are skipped so a proxy whose parent * ssh is still starting up, or a transient `ssh -O check` proxy mid-exit, is @@ -84,22 +115,49 @@ class CleanupStaleMultiplexedConnections implements ShouldQueue } foreach ($processes as $process) { - // `cloudflared access ssh`, never the `cloudflared tunnel` daemon. - if (! str_contains($process['args'], 'cloudflared access ssh')) { + // `cloudflared access ssh`, never the `cloudflared tunnel` daemon, nor + // an ssh whose ProxyCommand option contains the same words. + if (isset($sshPids[$process['pid']]) || ! str_contains($process['args'], 'cloudflared access ssh')) { continue; } - // Orphaned when no live ssh process is its parent. - if ($process['etimes'] >= $minAge && ! isset($sshPids[$process['ppid']])) { + if ($process['etimes'] >= $minAge + && ! isset($sshPids[$process['ppid']]) + && ! $this->stdinPipeHeldByAny($process['pid'], array_keys($sshPids))) { $this->reapOrphan('cloudflared', $process); } } } /** - * Reap a detected orphan process. When orphan reaping is disabled (the - * default), the orphan is only logged — a dry-run mode that lets operators - * verify what would be killed before enabling it for real. + * Whether one of the given processes holds the other end of a process's + * stdin pipe. An unreadable stdin counts as held, so it is never reaped. + * + * @param list $holderPids + */ + private function stdinPipeHeldByAny(string $pid, array $holderPids): bool + { + $stdin = @readlink("/proc/{$pid}/fd/0"); + if ($stdin === false || ! str_starts_with($stdin, 'pipe:')) { + return true; + } + + foreach ($holderPids as $holderPid) { + foreach (glob("/proc/{$holderPid}/fd/*") ?: [] as $fd) { + if (@readlink($fd) === $stdin) { + return true; + } + } + } + + return false; + } + + /** + * Reap a detected orphan process. When orphan reaping is disabled + * (SSH_MUX_ORPHAN_REAP_ENABLED=false, the default), the orphan is only + * logged — a dry-run mode that lets operators verify what would be killed + * before enabling it for real. * * @param array{pid: string, ppid: string, etimes: int, args: string} $process */ @@ -115,7 +173,10 @@ class CleanupStaleMultiplexedConnections implements ShouldQueue return; } - Process::run('kill '.escapeshellarg($process['pid'])); + // SIGKILL ssh: a former master unlinks its ControlPath on SIGTERM, even + // when a newer master owns that path now. + $signal = $kind === 'ssh' ? 'kill -9 ' : 'kill '; + Process::run($signal.escapeshellarg($process['pid'])); Log::info("Killed orphaned {$kind} process", [ 'pid' => $process['pid'], 'etimes' => $process['etimes'], @@ -195,8 +256,9 @@ class CleanupStaleMultiplexedConnections implements ShouldQueue /** * Close and delete a stale mux socket file. When orphan reaping is disabled - * (the default), the file is only logged — a dry-run mode that lets operators - * verify what would be removed before enabling it for real. + * (SSH_MUX_ORPHAN_REAP_ENABLED=false, the default), the file is only logged — + * a dry-run mode that lets operators verify what would be removed before + * enabling it for real. */ private function removeMultiplexFile(string $muxFile, string $reason): void { diff --git a/app/Jobs/DeleteFileDownloadJob.php b/app/Jobs/DeleteFileDownloadJob.php new file mode 100644 index 0000000000..598a2c2041 --- /dev/null +++ b/app/Jobs/DeleteFileDownloadJob.php @@ -0,0 +1,32 @@ +s3StorageId); + if ($storage && str_starts_with($this->key, FileDownloadStorageService::PREFIX)) { + $downloads->delete($storage, $this->key); + } + } +} diff --git a/app/Jobs/DeleteResourceJob.php b/app/Jobs/DeleteResourceJob.php index d5e8aaf4ff..4749891550 100644 --- a/app/Jobs/DeleteResourceJob.php +++ b/app/Jobs/DeleteResourceJob.php @@ -56,7 +56,7 @@ class DeleteResourceJob implements ShouldBeEncrypted, ShouldQueue return; } - if ($this->deleteFromCoolifyOnly && $this->resource instanceof Service) { + if ($this->deleteFromCoolifyOnly) { // No remote calls on this path: remove only the schedule records, so the scheduler does // not keep running backups of volumes that no longer exist in Coolify. $this->volumeBackupSchedules()->each->delete(); diff --git a/app/Jobs/DockerCleanupJob.php b/app/Jobs/DockerCleanupJob.php index ed430a600c..146f751b23 100644 --- a/app/Jobs/DockerCleanupJob.php +++ b/app/Jobs/DockerCleanupJob.php @@ -41,6 +41,8 @@ class DockerCleanupJob implements ShouldBeEncrypted, ShouldQueue public ?DockerCleanupExecution $execution_log = null; + public const INTERRUPTED_MESSAGE = 'Docker cleanup was interrupted: its worker stopped before the cleanup finished.'; + /** * Shares the per-server lock with queued CleanupDocker runs, so two cleanups never run on * one server at once. A scheduled run that finds the lock held marks its occurrence @@ -232,6 +234,40 @@ class DockerCleanupJob implements ShouldBeEncrypted, ShouldQueue return $jobUuid ? 'docker-cleanup-execution:'.$jobUuid : null; } + /** + * Fails running executions that are older than the server lock. The worker kills a cleanup at the + * job timeout, which is shorter than the lock, so such a run lost its worker before failed() ran + * (for example a killed container) and would otherwise stay running. + */ + public static function failInterruptedExecutions(): void + { + $executions = DockerCleanupExecution::query() + ->with('server.team') + ->where('status', 'running') + ->whereNull('finished_at') + ->where('created_at', '<', now()->subSeconds(CleanupDocker::overlapLockExpiresAfter())) + ->get(); + + foreach ($executions as $execution) { + $updated = DockerCleanupExecution::query() + ->whereKey($execution->id) + ->where('status', 'running') + ->whereNull('finished_at') + ->update([ + 'status' => 'failed', + 'message' => self::INTERRUPTED_MESSAGE, + 'finished_at' => Carbon::now()->toImmutable(), + ]); + + if ($updated === 0) { + continue; + } + + event(new DockerCleanupDone($execution->refresh())); + $execution->server?->team?->notify(new DockerCleanupFailed($execution->server, 'Docker cleanup job failed with the following error: '.self::INTERRUPTED_MESSAGE)); + } + } + public function failed(?\Throwable $exception): void { if ($this->occurrenceUuid) { diff --git a/app/Jobs/MigrateServerToInstanceJob.php b/app/Jobs/MigrateServerToInstanceJob.php new file mode 100644 index 0000000000..cba109fb04 --- /dev/null +++ b/app/Jobs/MigrateServerToInstanceJob.php @@ -0,0 +1,69 @@ +onQueue('high'); + } + + public function uniqueId(): string + { + return (string) $this->server->id; + } + + public function handle(ServerTransferMigrator $migrator): void + { + try { + $result = $migrator->migrate($this->server, $this->targetUrl, $this->targetToken); + $migrator->recordMigration($this->server, 'succeeded', [ + 'export_id' => $result['export_id'], + 'warnings' => $result['warnings'], + 'checklist' => $result['checklist'], + 'target_server_url' => $result['target_server_url'] ?? null, + ]); + } catch (Throwable $e) { + $migrator->recordMigration($this->server, 'failed', ['error' => $e->getMessage()]); + } + } + + /** + * The worker stopped the job, for example at its timeout. The target can manage the server already, + * so the migrator asks the target before it records the result. + */ + public function failed(?Throwable $exception): void + { + app(ServerTransferMigrator::class)->recordStoppedMigration( + $this->server, + $this->targetUrl, + $this->targetToken, + $exception?->getMessage() ?: 'The transfer job stopped before it finished.', + ); + } +} diff --git a/app/Jobs/PrepareFileDownloadJob.php b/app/Jobs/PrepareFileDownloadJob.php new file mode 100644 index 0000000000..b36d5ccd96 --- /dev/null +++ b/app/Jobs/PrepareFileDownloadJob.php @@ -0,0 +1,81 @@ +whereTeamId(0)->where('is_usable', true)->find($this->s3StorageId); + if (! $storage) { + throw new \RuntimeException('The file download storage is not available.'); + } + $server = Server::findOrFail($this->serverId); + + $key = FileDownloadStorageService::newObjectKey(); + // Queue the delete first, so a partial upload is removed as well. + DeleteFileDownloadJob::dispatch($storage->id, $key)->delay(now()->addHours(FileDownloadStorageService::OBJECT_TTL_HOURS)); + + $upload = $downloads->uploadUrl($storage, $key, $this->downloadName); + if ($this->container === null) { + instant_remote_process([FileDownloadStorageService::buildHostFileUploadCommand($this->path, $upload['url'], $upload['headers'])], $server, timeout: 3600); + } else { + (new ContainerFilesystemService($server, $this->container))->uploadToS3($this->path, $this->isDir, $upload['url'], $upload['headers']); + } + + $this->store(['status' => 'ready', 'url' => $downloads->downloadUrl($storage, $key)]); + } catch (\Throwable $e) { + // The message can contain the S3 endpoint or a presigned URL; keep it in the log only. + Log::warning('S3 download failed', ['download_id' => $this->downloadId, 'error' => $e->getMessage()]); + $this->store(['status' => 'failed', 'message' => 'Download failed. The file could not be uploaded to the download storage. Try again, or ask your administrator to check the storage settings.']); + } + } + + /** + * @param array $result + */ + private function store(array $result): void + { + Cache::put(self::cacheKey($this->downloadId), ['user_id' => $this->userId, ...$result], now()->addMinutes(FileDownloadStorageService::LINK_TTL_MINUTES)); + } +} diff --git a/app/Jobs/ProcessGithubPullRequestWebhook.php b/app/Jobs/ProcessGithubPullRequestWebhook.php index 362ad0a2d8..1602b576a2 100644 --- a/app/Jobs/ProcessGithubPullRequestWebhook.php +++ b/app/Jobs/ProcessGithubPullRequestWebhook.php @@ -14,6 +14,7 @@ use Illuminate\Contracts\Queue\ShouldQueue; use Illuminate\Foundation\Bus\Dispatchable; use Illuminate\Queue\InteractsWithQueue; use Illuminate\Queue\SerializesModels; +use Illuminate\Support\Facades\Log; use Throwable; class ProcessGithubPullRequestWebhook implements ShouldBeEncrypted, ShouldQueue @@ -37,7 +38,6 @@ class ProcessGithubPullRequestWebhook implements ShouldBeEncrypted, ShouldQueue public ?string $beforeSha, public ?string $afterSha, public string $commitSha, - public ?string $authorAssociation, public string $fullName, public bool $isForkPullRequest = false, ) { @@ -109,39 +109,46 @@ class ProcessGithubPullRequestWebhook implements ShouldBeEncrypted, ShouldQueue return; } - // Check if PR deployments from public contributors are restricted - if (! $application->settings->is_pr_deployments_public_enabled) { - // Fork PRs carry untrusted code from a repository outside our control. - // GitHub's author_association cannot be trusted to gate these (it grants - // CONTRIBUTOR to anyone who has merely opened an issue/PR before), so fork - // PRs are never deployed automatically when public previews are off. - if ($this->isForkPullRequest) { - return; - } + // Fork PRs carry untrusted code, so they deploy only when public previews are on. + // Same-repository branches can only be pushed by users with write access. + if ($this->isForkPullRequest && ! $application->settings->is_pr_deployments_public_enabled) { + Log::info('Skipped GitHub pull request preview deployment from a fork because public PR deployments are disabled.', [ + 'application_uuid' => $application->uuid, + 'pull_request_id' => $this->pullRequestId, + 'action' => $this->action, + ]); - // Same-repo (non-fork) branch PRs require push access to the base repo, - // so only trusted associations are allowed to trigger a deployment. - $trustedAssociations = ['OWNER', 'MEMBER', 'COLLABORATOR']; - if (! in_array($this->authorAssociation, $trustedAssociations)) { - return; - } + return; } - // Get changed files for watch path filtering - $changed_files = collect(); + // Get changed files for watch path filtering (null means the full list is unknown) + $changed_files = []; if ($this->action === 'synchronize' && $this->beforeSha && $this->afterSha) { // For synchronize events, get files changed between before and after commits - $changed_files = collect(getGithubCommitRangeFiles($githubApp, $owner, $repo, $this->beforeSha, $this->afterSha)); + $changed_files = getGithubCommitRangeFiles($githubApp, $owner, $repo, $this->beforeSha, $this->afterSha); } elseif ($this->action === 'opened' || $this->action === 'reopened') { // For opened/reopened events, get all files in the PR - $changed_files = collect(getGithubPullRequestFiles($githubApp, $owner, $repo, $this->pullRequestId)); + $changed_files = getGithubPullRequestFiles($githubApp, $owner, $repo, $this->pullRequestId); } // Apply watch path filtering - $is_watch_path_triggered = $application->isWatchPathsTriggered($changed_files); - if (! $is_watch_path_triggered && ! blank($application->watch_paths)) { - return; + if (! blank($application->watch_paths)) { + $logContext = [ + 'application_uuid' => $application->uuid, + 'pull_request_id' => $this->pullRequestId, + 'action' => $this->action, + ]; + + if (is_null($changed_files)) { + Log::warning('Could not get the full list of changed files from GitHub. Deploying the pull request preview without the watch paths check.', $logContext); + } elseif (! $application->isWatchPathsTriggered(collect($changed_files))) { + Log::info('Skipped GitHub pull request preview deployment because no changed file matches the watch paths.', $logContext + [ + 'changed_files_count' => count($changed_files), + ]); + + return; + } } // Create ApplicationPreview if not exists diff --git a/app/Jobs/ProvisionGithubRunnerJob.php b/app/Jobs/ProvisionGithubRunnerJob.php index e4415a81c1..ae9e5e15e7 100644 --- a/app/Jobs/ProvisionGithubRunnerJob.php +++ b/app/Jobs/ProvisionGithubRunnerJob.php @@ -85,7 +85,8 @@ class ProvisionGithubRunnerJob implements ShouldBeEncrypted, ShouldQueue if ($this->stopIfFinished($execution)) { return; } - instant_remote_process($container->startCommands($runner['encoded_jit_config']), $server); + // stopIfFinished() reloaded the server, so a transfer that just finished stops the start command. + instant_remote_process($container->startCommands($runner['encoded_jit_config']), $execution->server); if ($this->stopIfFinished($execution)) { return; @@ -159,11 +160,15 @@ class ProvisionGithubRunnerJob implements ShouldBeEncrypted, ShouldQueue return true; } + // Reloads the server too: a transfer to another instance can finish while this job runs. $execution->refresh(); - if ($execution->isActive()) { + if ($execution->isActive() && ! $execution->server?->isTransferredAway()) { return false; } + if ($execution->isActive()) { + $execution->finish(GithubRunnerStatus::Cancelled, 'The server was transferred to another Coolify instance.'); + } CleanupGithubRunnerJob::dispatch($execution->id); return true; diff --git a/app/Jobs/PullTemplatesFromCDN.php b/app/Jobs/PullTemplatesFromCDN.php index ec09285e1e..2125581f50 100644 --- a/app/Jobs/PullTemplatesFromCDN.php +++ b/app/Jobs/PullTemplatesFromCDN.php @@ -44,13 +44,11 @@ class PullTemplatesFromCDN implements ShouldBeEncrypted, ShouldQueue 'status' => $response->status(), 'body' => str($response->body())->limit(500)->toString(), ]); - send_internal_notification('PullTemplatesAndVersions failed with: '.$response->status().' '.$response->body()); } } catch (\Throwable $e) { Log::error('PullTemplatesFromCDN exception', [ 'message' => $e->getMessage(), ]); - send_internal_notification('PullTemplatesAndVersions failed with: '.$e->getMessage()); } } } diff --git a/app/Jobs/ScheduledJobManager.php b/app/Jobs/ScheduledJobManager.php index c218a53652..7e137dadaa 100644 --- a/app/Jobs/ScheduledJobManager.php +++ b/app/Jobs/ScheduledJobManager.php @@ -191,6 +191,7 @@ class ScheduledJobManager implements ShouldQueue // Process Docker cleanups - don't let failures stop the job manager try { if ($this->includes('docker-cleanups')) { + DockerCleanupJob::failInterruptedExecutions(); $this->processDockerCleanups(); } } catch (\Throwable $e) { diff --git a/app/Jobs/ScheduledTaskJob.php b/app/Jobs/ScheduledTaskJob.php index 56c5bb7821..765e6f89d2 100644 --- a/app/Jobs/ScheduledTaskJob.php +++ b/app/Jobs/ScheduledTaskJob.php @@ -85,6 +85,17 @@ class ScheduledTaskJob implements ShouldBeEncrypted, ShouldQueue $this->executionUuid = new_public_id(); } + /** + * The lock outlives the worker timeout, so a run killed at the timeout still blocks a new run + * while its remote command can still be running. + */ + public function middleware(): array + { + $expireAfter = ($this->task->timeout ?? 300) + 300; + + return [ScheduledJobDeliveryService::withoutOverlapping('scheduled-task-'.$this->task->id, $this->occurrenceUuid)->expireAfter($expireAfter)->dontRelease()]; + } + private function initializeExecutionContext(): void { $this->task->loadMissing([ diff --git a/app/Jobs/ServerConnectionCheckJob.php b/app/Jobs/ServerConnectionCheckJob.php index ab8d56a98a..841e915b78 100644 --- a/app/Jobs/ServerConnectionCheckJob.php +++ b/app/Jobs/ServerConnectionCheckJob.php @@ -5,6 +5,7 @@ namespace App\Jobs; use App\Events\ServerReachabilityChanged; use App\Helpers\SshMultiplexingHelper; use App\Models\Server; +use App\Services\ServerTransfer\ServerTransferClaimer; use Illuminate\Bus\Queueable; use Illuminate\Contracts\Queue\ShouldBeEncrypted; use Illuminate\Contracts\Queue\ShouldQueue; @@ -44,6 +45,9 @@ class ServerConnectionCheckJob implements ShouldBeEncrypted, ShouldQueue private ?string $connectionError = null; + /** Owner marker that the SSH probe read from the host; empty when the host has none. */ + private ?string $ownerMarker = null; + public function __construct( public Server $server, public bool $disableMux = true @@ -108,6 +112,10 @@ class ServerConnectionCheckJob implements ShouldBeEncrypted, ShouldQueue return; } + if ($this->releasedToAnotherInstance()) { + return; + } + // Server is reachable, check if Docker is available $isUsable = $this->checkDockerAvailability(); @@ -218,9 +226,11 @@ class ServerConnectionCheckJob implements ShouldBeEncrypted, ShouldQueue { try { $process = Process::timeout(self::SSH_TIMEOUT_SECONDS) - ->run($this->sshCommand('ls -la /', disableMultiplexing: true)); + ->run($this->sshCommand(self::probeCommand(), disableMultiplexing: true)); if ($process->exitCode() !== 0) { $this->connectionError = 'ssh exit '.$process->exitCode().': '.Str::limit(trim($process->errorOutput()), 300); + } else { + $this->ownerMarker = trim($process->output()); } return $process->exitCode() === 0; @@ -235,6 +245,25 @@ class ServerConnectionCheckJob implements ShouldBeEncrypted, ShouldQueue } } + /** + * The SSH probe also reads the owner marker, so the marker check needs no extra SSH connection. + * The remote shell runs with `set -e` and most servers have no marker, so a missing file must not fail the probe. + */ + public static function probeCommand(): string + { + return 'cat '.ServerTransferClaimer::OWNER_MARKER_PATH.' 2>/dev/null || true; ls -la / > /dev/null'; + } + + /** + * After a transfer, another instance can claim this server. Its owner marker on the host is the only + * record that both instances share, so this instance stops managing the server when the marker names + * another claim. + */ + private function releasedToAnotherInstance(): bool + { + return app(ServerTransferClaimer::class)->releaseIfClaimedElsewhere($this->server, $this->ownerMarker); + } + private function checkDockerAvailability(): bool { try { diff --git a/app/Jobs/UpdateCoolifyJob.php b/app/Jobs/UpdateCoolifyJob.php index f0e43cbc0a..e2310f0bee 100644 --- a/app/Jobs/UpdateCoolifyJob.php +++ b/app/Jobs/UpdateCoolifyJob.php @@ -44,8 +44,7 @@ class UpdateCoolifyJob implements ShouldBeEncrypted, ShouldQueue Log::info('Starting Coolify update process...'); UpdateCoolify::run(false); // false means it's not a manual update - $settings->update(['new_version_available' => false]); - Log::info('Coolify update completed successfully.'); + Log::info('Coolify upgrade script started. Result: /data/coolify/source/.upgrade-status, log: /data/coolify/source/upgrade-*.log.'); } catch (\Throwable $e) { Log::error('UpdateCoolifyJob failed: '.$e->getMessage()); // Consider implementing a notification to administrators diff --git a/app/Jobs/VolumeBackupJob.php b/app/Jobs/VolumeBackupJob.php index 9c36f67e53..6439942e1a 100644 --- a/app/Jobs/VolumeBackupJob.php +++ b/app/Jobs/VolumeBackupJob.php @@ -99,8 +99,7 @@ class VolumeBackupJob implements ShouldBeEncrypted, ShouldQueue ? 'docker volume inspect '.escapeshellarg($source).' >/dev/null' : 'test -d '.escapeshellarg($source); - $compressorCommand = BackupCompression::compressorCommand($compressionCpuPercentage); - $archiveScript = "compressor=\$({$compressorCommand}); tar -I \"\$compressor\" -cf - -C /volume ."; + $archiveScript = self::archiveScript($compressionCpuPercentage); if ($streamToS3) { $this->execution->update(['local_storage_deleted' => true]); $archiveCommand = $this->streamToS3Command($destinations->first(), $archiveScript, $backupLocation, $source, $containerName, $image); @@ -303,6 +302,17 @@ class VolumeBackupJob implements ShouldBeEncrypted, ShouldQueue )); } + /** + * GNU tar exits with 1 when a file changes while it is read (for example, an app that writes to the volume + * during a live backup). The archive is still valid, so only exit codes above 1 fail the backup. + */ + public static function archiveScript(int $compressionCpuPercentage): string + { + $compressorCommand = BackupCompression::compressorCommand($compressionCpuPercentage); + + return "compressor=\$({$compressorCommand}); { tar -I \"\$compressor\" -cf - -C /volume . || { status=\$?; [ \$status -eq 1 ] || exit \$status; }; }"; + } + /** * Redact the key and secret of every S3 destination of the backup from text that is stored or sent in notifications. */ @@ -323,12 +333,15 @@ class VolumeBackupJob implements ShouldBeEncrypted, ShouldQueue } /** + * Containers can exit and be removed (e.g. `--rm` backup helpers) between `docker ps` and + * `docker inspect`. Skip those, but still fail when an existing container cannot be inspected. + * * @return array */ private function containersUsingVolume(string $source, Server $server): array { $output = instant_remote_process( - ["containers=\$(docker ps -q) || exit 1; for container in \$containers; do paused=\$(docker inspect --format '{{.State.Paused}}' \"\$container\") || exit 1; [ \"\$paused\" = true ] && continue; mounts=\$(docker inspect --format '{{range .Mounts}}{{println .Source}}{{println .Name}}{{end}}' \"\$container\") || exit 1; if printf '%s\\n' \"\$mounts\" | grep -Fqx -- ".escapeshellarg($source).'; then echo "$container"; fi; done'], + ["containers=\$(docker ps -q) || exit 1; for container in \$containers; do mounts=\$(docker inspect --format '{{if not .State.Paused}}{{range .Mounts}}{{println .Source}}{{println .Name}}{{end}}{{end}}' \"\$container\" 2>/dev/null) || { remaining=\$(docker ps -aq --filter \"id=\$container\") || exit 1; [ -z \"\$remaining\" ] && continue; docker inspect \"\$container\" >/dev/null; exit 1; }; if printf '%s\\n' \"\$mounts\" | grep -Fqx -- ".escapeshellarg($source).'; then echo "$container"; fi; done'], $server, disableMultiplexing: true, ); @@ -449,9 +462,13 @@ class VolumeBackupJob implements ShouldBeEncrypted, ShouldQueue ->implode(' '); $resolveOptions = $resolveOptions === '' ? '' : ' '.$resolveOptions; $destination = 'temporary/'.$s3->bucket.$backupLocation; + // `mc pipe --quiet` panics in recent mc releases, so keep its progress bar out of the error output instead. + // Each progress frame starts with "\r", so printing the last field per line keeps only the final frame and real errors. + $mcPipeLog = '/tmp/mc-pipe.log'; $streamScript = 'set -o pipefail; mc alias set'.$resolveOptions.' temporary ' .escapeshellarg($s3->endpoint).' '.escapeshellarg($s3->key).' '.escapeshellarg($s3->secret) - .' >/dev/null && ('.$archiveScript.' | mc pipe --quiet'.$resolveOptions.' '.escapeshellarg($destination).' >/dev/null)' + .' >/dev/null && ('.$archiveScript.' | mc pipe'.$resolveOptions.' '.escapeshellarg($destination).' >/dev/null 2>'.$mcPipeLog + .' || { status=$?; awk -F '.escapeshellarg('\r').' '.escapeshellarg('{ print $NF }').' '.$mcPipeLog.' >&2; exit $status; })' .' && mc stat --json'.$resolveOptions.' '.escapeshellarg($destination) .' | sed -n '.escapeshellarg('s/.*"size":\([0-9][0-9]*\).*/\1/p'); diff --git a/app/Listeners/CloudflareTunnelChangedNotification.php b/app/Listeners/CloudflareTunnelChangedNotification.php index 34e713f954..3cc44c2f82 100644 --- a/app/Listeners/CloudflareTunnelChangedNotification.php +++ b/app/Listeners/CloudflareTunnelChangedNotification.php @@ -26,7 +26,7 @@ class CloudflareTunnelChangedNotification for ($i = 1; $i <= $attempts; $i++) { \Log::debug("Cloudflare health check attempt {$i}/{$attempts}", ['server_id' => $server_id]); - $result = instant_remote_process_with_timeout(['docker inspect coolify-cloudflared | jq -e ".[0].State.Health.Status == \"healthy\""'], $this->server, false, 10); + $result = instant_remote_process_with_timeout(['docker inspect coolify-cloudflared | jq -e ".[0].State.Health.Status == \"healthy\""'], $this->server, false); if (blank($result)) { \Log::debug("Cloudflare Tunnels container not found on attempt {$i}", ['server_id' => $server_id]); diff --git a/app/Livewire/Analytics.php b/app/Livewire/Analytics.php index e4400f58ec..007618501c 100644 --- a/app/Livewire/Analytics.php +++ b/app/Livewire/Analytics.php @@ -109,7 +109,7 @@ class Analytics extends Component private const MAX_LEADERBOARD_APPS = 200; /** @var array */ - protected array $breakdownDimensions = ['country', 'referer', 'browser', 'os', 'device', 'protocol', 'cache', 'status', 'agent', 'ip', 'useragent']; + protected array $breakdownDimensions = ['country', 'city', 'referer', 'browser', 'os', 'device', 'protocol', 'cache', 'status', 'agent', 'ip', 'useragent']; /** * Per-request cache of Sentinel key => display metadata, so resolving a name/domain/link diff --git a/app/Livewire/Concerns/BuildsTrafficChartPayload.php b/app/Livewire/Concerns/BuildsTrafficChartPayload.php index eb57c855fb..d1c16672be 100644 --- a/app/Livewire/Concerns/BuildsTrafficChartPayload.php +++ b/app/Livewire/Concerns/BuildsTrafficChartPayload.php @@ -4,7 +4,7 @@ namespace App\Livewire\Concerns; /** * Shared derivations for the traffic chart payload: per-bucket sparkline series, - * the device-donut labels/series, and the globe's per-country marker data. Consumed + * the device-donut labels/series, and the globe's marker data. Consumed * by both the global and per-application analytics components, which expose * `$series` (status buckets) and `$breakdowns` (dimension rows). */ @@ -77,22 +77,13 @@ trait BuildsTrafficChartPayload } /** - * Per-country marker data for the globe: [{code, requests}] over known ISO-A2 rows. + * Marker data for the globe: country markers and city markers, one set for each view. * - * @return array + * @return array{countries: array, cities: array} */ protected function geoMarkers(): array { - $out = []; - foreach (($this->breakdowns['country'] ?? []) as $row) { - $code = strtoupper((string) ($row['value'] ?? '')); - $requests = (int) ($row['requests'] ?? 0); - if (preg_match('/^[A-Z]{2}$/', $code) && $requests > 0) { - $out[] = ['code' => $code, 'requests' => $requests]; - } - } - - return $out; + return trafficGeoMarkers($this->breakdowns['country'] ?? [], trafficCityRows($this->breakdowns['city'] ?? [])); } /** diff --git a/app/Livewire/Concerns/ManagesDatabaseBackup.php b/app/Livewire/Concerns/ManagesDatabaseBackup.php new file mode 100644 index 0000000000..64eaffa1a1 --- /dev/null +++ b/app/Livewire/Concerns/ManagesDatabaseBackup.php @@ -0,0 +1,85 @@ +authorize('manageBackups', $backup->database); + + $backup->enabled = ! $backup->enabled; + $changedFields = auditChangedFields($backup); + $backup->save(); + + $database = $backup->database; + auditLog('ui.database.backup_schedule_updated', [ + 'team_id' => $database?->team()?->id, + 'database_uuid' => $database?->uuid, + 'database_name' => $database?->name, + 'backup_uuid' => $backup->uuid, + 'changed_fields' => $changedFields, + ]); + $this->dispatch('success', $backup->enabled ? 'Backup enabled.' : 'Backup disabled.'); + + return $backup->enabled; + } + + /** + * Queue a backup run now and redirect to the executions page of the backup. + */ + protected function queueDatabaseBackup(ScheduledDatabaseBackup $backup): Redirector|RedirectResponse|null + { + $this->authorize('manageBackups', $backup->database); + + $database = $backup->database->refresh(); + if ($database->id !== 0 && ! str($database->status)->startsWith('running')) { + $this->dispatch('error', 'The database must be running to start a backup.'); + + return null; + } + + DatabaseBackupJob::dispatch($backup); + auditLog('ui.database.backup_started', [ + 'team_id' => $database->team()?->id, + 'database_uuid' => $database->uuid, + 'database_name' => $database->name, + 'backup_uuid' => $backup->uuid, + ]); + $this->dispatch('success', 'Backup queued. It will be available in a few minutes.'); + + if ($database instanceof ServiceDatabase) { + return redirect()->route('project.service.database.backup.executions', [ + 'project_uuid' => $database->service->project()->uuid, + 'environment_uuid' => $database->service->environment->uuid, + 'service_uuid' => $database->service->uuid, + 'stack_service_uuid' => $database->uuid, + 'backup_uuid' => $backup->uuid, + ]); + } + + // Instance databases (e.g. coolify-db) have no project/environment. + // Stay on the current page (settings.backup) instead of redirecting. + $project = $database->project(); + $environment = $database->environment; + if (! $project || ! $environment) { + return null; + } + + return redirect()->route('project.database.backup.executions', [ + 'project_uuid' => $project->uuid, + 'environment_uuid' => $environment->uuid, + 'database_uuid' => $database->uuid, + 'backup_uuid' => $backup->uuid, + ]); + } +} diff --git a/app/Livewire/Concerns/RemembersCanvasView.php b/app/Livewire/Concerns/RemembersCanvasView.php index d8bad86d4d..4d7e4f29d8 100644 --- a/app/Livewire/Concerns/RemembersCanvasView.php +++ b/app/Livewire/Concerns/RemembersCanvasView.php @@ -13,7 +13,7 @@ use Livewire\Attributes\Renderless; */ trait RemembersCanvasView { - private const array CANVAS_VIEW_PAGES = ['servers', 'server_resources', 'projects', 'project', 'environment', 'tags', 'team_variables', 'project_variables', 'environment_variables', 'server_variables']; + private const array CANVAS_VIEW_PAGES = ['servers', 'server_resources', 'projects', 'project', 'environment', 'tags', 'team_variables', 'project_variables', 'environment_variables', 'server_variables', 'dashboard', 'dashboard_projects', 'dashboard_servers']; private const array LIST_VIEWS = ['table', 'grid', 'list']; @@ -60,9 +60,13 @@ trait RemembersCanvasView */ protected function listView(string $page, string $default): string { - $view = request()->query('view') ?? request()->cookie($this->viewCookieName($page)); + foreach ([request()->query('view'), request()->cookie($this->viewCookieName($page))] as $view) { + if (in_array($view, ['table', 'grid'], true)) { + return $view; + } + } - return in_array($view, ['table', 'grid'], true) ? $view : $default; + return $default; } private function viewCookieName(string $page): string diff --git a/app/Livewire/Dashboard.php b/app/Livewire/Dashboard.php index 2fc2553512..aa32fe6770 100644 --- a/app/Livewire/Dashboard.php +++ b/app/Livewire/Dashboard.php @@ -2,6 +2,7 @@ namespace App\Livewire; +use App\Livewire\Concerns\RemembersCanvasView; use App\Models\PrivateKey; use App\Models\Project; use App\Models\Server; @@ -11,6 +12,8 @@ use Livewire\Component; class Dashboard extends Component { + use RemembersCanvasView; + public Collection $projects; public Collection $servers; @@ -19,6 +22,12 @@ class Dashboard extends Component public function mount() { + if ($this->prefersCanvasView('dashboard')) { + $this->redirectRoute('dashboard.canvas', navigate: true); + + return; + } + $this->privateKeys = PrivateKey::ownedByCurrentTeamCached(); $this->servers = Server::ownedByCurrentTeamCached(); $this->projects = Project::ownedByCurrentTeam() diff --git a/app/Livewire/Dashboard/Canvas.php b/app/Livewire/Dashboard/Canvas.php new file mode 100644 index 0000000000..8aba6c5f54 --- /dev/null +++ b/app/Livewire/Dashboard/Canvas.php @@ -0,0 +1,33 @@ +teamId = currentTeam()->id; + $this->rememberView('dashboard', 'canvas'); + } + + public function render(): View + { + return view('livewire.dashboard.canvas')->layout('layouts.app', ['fullBleed' => true]); + } +} diff --git a/app/Livewire/Dashboard/ServerMetricsChart.php b/app/Livewire/Dashboard/ServerMetricsChart.php index fce9460064..83748052ce 100644 --- a/app/Livewire/Dashboard/ServerMetricsChart.php +++ b/app/Livewire/Dashboard/ServerMetricsChart.php @@ -4,14 +4,23 @@ namespace App\Livewire\Dashboard; use App\Models\Server; use Illuminate\Contracts\View\View; +use Livewire\Attributes\Isolate; use Livewire\Component; +/** + * Each chart sends its own request, so one slow server does not delay the charts of the other servers. + */ +#[Isolate] class ServerMetricsChart extends Component { public Server $server; public function loadData(): void { + if (! $this->server->isFunctional()) { + return; + } + try { $this->dispatch("dashboard-server-metrics-{$this->server->uuid}", [ 'cpu' => $this->server->getCpuMetrics(10), diff --git a/app/Livewire/DeploymentsIndicator.php b/app/Livewire/DeploymentsIndicator.php index e6b68a62b7..190da1a097 100644 --- a/app/Livewire/DeploymentsIndicator.php +++ b/app/Livewire/DeploymentsIndicator.php @@ -5,14 +5,26 @@ namespace App\Livewire; use App\Models\ApplicationDeploymentQueue; use App\Models\Server; use Livewire\Attributes\Computed; +use Livewire\Attributes\Locked; use Livewire\Component; class DeploymentsIndicator extends Component { /** - * Where the indicator is rendered: 'sidebar' (desktop sidebar footer) or 'mobile' (mobile top bar). + * Where the indicator is rendered: 'topbar' (desktop top bar) or 'mobile' (mobile top bar). */ - public string $variant = 'sidebar'; + public string $variant = 'topbar'; + + /** + * The application on the current page. Its heading has its own "Deploying…" indicator. + */ + #[Locked] + public ?string $currentApplicationUuid = null; + + public function mount(): void + { + $this->currentApplicationUuid = request()->route('application_uuid'); + } #[Computed] public function deployments() @@ -41,6 +53,16 @@ class DeploymentsIndicator extends Component return $this->deployments->count(); } + /** + * Hide the counter when it would only repeat the current page's "Deploying…" indicator. + */ + #[Computed] + public function isOnlyCurrentApplicationDeploying(): bool + { + return $this->currentApplicationUuid !== null + && $this->deployments->every(fn ($deployment) => $deployment->application?->uuid === $this->currentApplicationUuid); + } + public function render() { return view('livewire.deployments-indicator'); diff --git a/app/Livewire/Destination/Show.php b/app/Livewire/Destination/Show.php index e887abed9e..05b295b88b 100644 --- a/app/Livewire/Destination/Show.php +++ b/app/Livewire/Destination/Show.php @@ -20,10 +20,10 @@ class Show extends Component #[Validate(['string', 'required'])] public string $name; - #[Validate(['string', 'required', 'max:255', 'regex:/^[a-zA-Z0-9][a-zA-Z0-9._-]*$/'])] + #[Locked] public string $network; - #[Validate(['string', 'required'])] + #[Locked] public string $serverIp; public function mount(string $destination_uuid) @@ -49,8 +49,6 @@ class Show extends Component if ($toModel) { $this->validate(); $this->destination->name = $this->name; - $this->destination->network = $this->network; - $this->destination->server->ip = $this->serverIp; $changedFields = auditChangedFields($this->destination); $this->destination->save(); if ($changedFields !== []) { diff --git a/app/Livewire/Fleet/Metrics.php b/app/Livewire/Fleet/Metrics.php index bd9580a8cc..bcf6917499 100644 --- a/app/Livewire/Fleet/Metrics.php +++ b/app/Livewire/Fleet/Metrics.php @@ -29,6 +29,9 @@ class Metrics extends Component /** @var array uuid => name */ public array $serverOptions = []; + /** @var array Names of the servers where the user can turn metrics on; filled only when no server has metrics. */ + public array $enableableServers = []; + #[Url(as: 'range')] public string $range = '24h'; @@ -87,9 +90,13 @@ class Metrics extends Component private function refreshServers(): void { abort_unless(currentTeam() !== null, 403); - $this->servers = Server::ownedByCurrentTeam()->get() + $teamServers = Server::ownedByCurrentTeam()->get(); + $this->servers = $teamServers ->filter(fn (Server $server) => $server->isMetricsEnabled()) ->values(); + $this->enableableServers = $this->servers->isEmpty() + ? $this->serversWithoutMetrics($teamServers)->pluck('name')->all() + : []; foreach ($this->servers as $server) { $this->authorize('view', $server); @@ -104,6 +111,65 @@ class Metrics extends Component } } + /** + * Servers that can report metrics, have them off, are reachable, and that the user can change. + * An unreachable server cannot start Sentinel, so it stays out. + * + * @param Collection $teamServers + * @return Collection + */ + private function serversWithoutMetrics(Collection $teamServers): Collection + { + return $teamServers + ->filter(fn (Server $server) => ! $server->isMetricsEnabled() + && $server->isSentinelEnabled() + && auth()->user()->can('update', $server) + && $server->isFunctional()) + ->values(); + } + + public function enableMetrics(): void + { + try { + abort_unless(currentTeam() !== null, 403); + $servers = $this->serversWithoutMetrics(Server::ownedByCurrentTeam()->get()); + if ($servers->isEmpty()) { + return; + } + + // One failed server must not stop the others, so each server has its own try. + $failed = []; + foreach ($servers as $server) { + try { + $server->settings->is_metrics_enabled = true; + // Saved quietly: Sentinel is started below, so the settings hook must not start it a second time. + $server->settings->saveQuietly(); + auditLog('ui.server.sentinel.updated', [ + 'team_id' => $server->team_id, + 'server_uuid' => $server->uuid, + 'server_name' => $server->name, + 'changed_fields' => ['is_metrics_enabled'], + ]); + $server->restartSentinel(); + } catch (\Throwable $e) { + report($e); + $failed[] = $server->name; + } + } + + $this->loadData(); + $enabled = $servers->count() - count($failed); + if ($enabled > 0) { + $this->dispatch('success', 'Metrics enabled on '.$enabled.' '.str('server')->plural($enabled).'. Sentinel is starting, data shows in about one minute.'); + } + if ($failed !== []) { + $this->dispatch('error', 'Could not enable metrics on '.implode(', ', $failed).'. Check the server settings and try again.'); + } + } catch (\Throwable $e) { + handleError($e, $this); + } + } + public function updatedServerUuid(): void { $this->loadData(); diff --git a/app/Livewire/Project/Application/Backup/Index.php b/app/Livewire/Project/Application/Backup/Index.php index 7ca097db96..e53a3cdf88 100644 --- a/app/Livewire/Project/Application/Backup/Index.php +++ b/app/Livewire/Project/Application/Backup/Index.php @@ -4,6 +4,7 @@ namespace App\Livewire\Project\Application\Backup; use App\Models\Application; use App\Models\ScheduledVolumeBackup; +use App\Traits\ListensToTeamChannel; use Illuminate\Contracts\View\View; use Illuminate\Foundation\Auth\Access\AuthorizesRequests; use Livewire\Component; @@ -11,6 +12,7 @@ use Livewire\Component; class Index extends Component { use AuthorizesRequests; + use ListensToTeamChannel; public Application $application; @@ -18,7 +20,13 @@ class Index extends Component public string $search = ''; - protected $listeners = ['refreshVolumeBackups' => '$refresh']; + public function getListeners(): array + { + return [ + 'refreshVolumeBackups' => '$refresh', + ...$this->teamChannelListeners(['BackupCreated' => '$refresh']), + ]; + } public function mount(): void { diff --git a/app/Livewire/Project/Application/Backup/Show.php b/app/Livewire/Project/Application/Backup/Show.php index 5e10f0c8ac..c07e49852a 100644 --- a/app/Livewire/Project/Application/Backup/Show.php +++ b/app/Livewire/Project/Application/Backup/Show.php @@ -40,8 +40,6 @@ class Show extends Component ->firstOrFail(); $this->parameters = get_route_parameters(); $this->section = match (request()->route()?->getName()) { - 'project.application.backup.s3' => 's3', - 'project.application.backup.retention' => 'retention', 'project.application.backup.executions' => 'executions', 'project.application.backup.danger' => 'danger', default => 'general', diff --git a/app/Livewire/Project/Database/Backup/Execution.php b/app/Livewire/Project/Database/Backup/Execution.php index 726ee3b698..c96c7e5217 100644 --- a/app/Livewire/Project/Database/Backup/Execution.php +++ b/app/Livewire/Project/Database/Backup/Execution.php @@ -2,12 +2,23 @@ namespace App\Livewire\Project\Database\Backup; +use App\Livewire\Concerns\ManagesDatabaseBackup; +use App\Livewire\Project\Database\BackupEdit; use App\Models\S3Storage; use App\Models\ScheduledDatabaseBackup; +use App\Traits\ListensToTeamChannel; +use Illuminate\Foundation\Auth\Access\AuthorizesRequests; +use Illuminate\Http\RedirectResponse; +use Illuminate\Support\Facades\Auth; use Livewire\Component; +use Livewire\Features\SupportRedirects\Redirector; class Execution extends Component { + use AuthorizesRequests; + use ListensToTeamChannel; + use ManagesDatabaseBackup; + public $database; public ?ScheduledDatabaseBackup $backup; @@ -20,6 +31,28 @@ class Execution extends Component public string $section = 'general'; + public function getListeners(): array + { + // Keep the header backup actions in sync when the database starts/stops. + $listeners = [ + 'databaseUpdated' => '$refresh', + 'backupEnabledChanged' => 'syncBackupEnabled', + ]; + + $user = Auth::user(); + if ($user) { + $listeners["echo-private:user.{$user->id},DatabaseStatusChanged"] = '$refresh'; + } + + return [ + ...$listeners, + ...$this->teamChannelListeners([ + 'ServiceChecked' => '$refresh', + 'ServiceStatusChanged' => '$refresh', + ]), + ]; + } + public function mount() { $backup_uuid = request()->route('backup_uuid'); @@ -50,14 +83,38 @@ class Execution extends Component ->get(); $this->parameters = get_route_parameters(); $this->section = match (request()->route()?->getName()) { - 'project.database.backup.s3' => 's3', - 'project.database.backup.retention' => 'retention', 'project.database.backup.executions' => 'executions', 'project.database.backup.danger' => 'danger', default => 'general', }; } + public function backupNow(): Redirector|RedirectResponse|null + { + try { + return $this->queueDatabaseBackup($this->backup); + } catch (\Throwable $e) { + handleError($e, $this); + + return null; + } + } + + public function toggleEnabled(): void + { + try { + $this->toggleDatabaseBackup($this->backup); + $this->dispatch('backupEnabledChanged')->to(BackupEdit::class); + } catch (\Throwable $e) { + $this->dispatch('error', $e->getMessage()); + } + } + + public function syncBackupEnabled(): void + { + $this->backup->refresh(); + } + public function render() { return view('livewire.project.database.backup.execution'); diff --git a/app/Livewire/Project/Database/BackupEdit.php b/app/Livewire/Project/Database/BackupEdit.php index a7c49eeb58..97da01715e 100644 --- a/app/Livewire/Project/Database/BackupEdit.php +++ b/app/Livewire/Project/Database/BackupEdit.php @@ -2,7 +2,8 @@ namespace App\Livewire\Project\Database; -use App\Jobs\DatabaseBackupJob; +use App\Livewire\Concerns\ManagesDatabaseBackup; +use App\Livewire\Project\Database\Backup\Execution; use App\Models\S3Storage; use App\Models\ScheduledDatabaseBackup; use App\Models\ServiceDatabase; @@ -21,11 +22,15 @@ class BackupEdit extends Component { use AuthorizesRequests; use ListensToTeamChannel; + use ManagesDatabaseBackup; public ScheduledDatabaseBackup $backup; public string $section = 'general'; + /** The page shows the backup actions in the top header bar instead. */ + public bool $actionsInHeader = false; + #[Locked] public $availableS3Storages; @@ -96,7 +101,10 @@ class BackupEdit extends Component public function getListeners(): array { // Keep "Backup Now" in sync when the database starts/stops without a full page refresh. - $listeners = ['databaseUpdated' => 'refreshStatus']; + $listeners = [ + 'databaseUpdated' => 'refreshStatus', + 'backupEnabledChanged' => 'syncBackupEnabled', + ]; $user = Auth::user(); if (! $user) { @@ -107,7 +115,10 @@ class BackupEdit extends Component return [ ...$listeners, - ...$this->teamChannelListeners(['ServiceChecked' => 'refreshStatus']), + ...$this->teamChannelListeners([ + 'ServiceChecked' => 'refreshStatus', + 'ServiceStatusChanged' => 'refreshStatus', + ]), ]; } @@ -127,6 +138,15 @@ class BackupEdit extends Component } } + public function hydrate(): void + { + // Livewire restores a deleted backup as an empty model. Requests that arrive after the delete + // (for example status events before the redirect completes) must not render it. + if (! $this->backup->exists) { + $this->skipRender(); + } + } + public function refreshStatus(): void { $database = $this->backup->database; @@ -268,50 +288,9 @@ class BackupEdit extends Component public function backupNow() { try { - $this->authorize('manageBackups', $this->backup->database); + $this->status = $this->backup->database->refresh()->status; - $database = $this->backup->database->refresh(); - $this->status = $database->status; - if ($database->id !== 0 && ! str($database->status)->startsWith('running')) { - $this->dispatch('error', 'The database must be running to start a backup.'); - - return; - } - - DatabaseBackupJob::dispatch($this->backup); - $database = $this->backup->database; - auditLog('ui.database.backup_started', [ - 'team_id' => $database->team()?->id, - 'database_uuid' => $database->uuid, - 'database_name' => $database->name, - 'backup_uuid' => $this->backup->uuid, - ]); - $this->dispatch('success', 'Backup queued. It will be available in a few minutes.'); - - if ($database instanceof ServiceDatabase) { - return redirect()->route('project.service.database.backup.executions', [ - 'project_uuid' => $database->service->project()->uuid, - 'environment_uuid' => $database->service->environment->uuid, - 'service_uuid' => $database->service->uuid, - 'stack_service_uuid' => $database->uuid, - 'backup_uuid' => $this->backup->uuid, - ]); - } - - // Instance databases (e.g. coolify-db) have no project/environment. - // Stay on the current page (settings.backup) instead of redirecting. - $project = $database->project(); - $environment = $database->environment; - if (! $project || ! $environment) { - return null; - } - - return redirect()->route('project.database.backup.executions', [ - 'project_uuid' => $project->uuid, - 'environment_uuid' => $environment->uuid, - 'database_uuid' => $database->uuid, - 'backup_uuid' => $this->backup->uuid, - ]); + return $this->queueDatabaseBackup($this->backup); } catch (\Throwable $e) { return handleError($e, $this); } @@ -332,19 +311,18 @@ class BackupEdit extends Component public function toggleEnabled(): void { try { - $this->authorize('manageBackups', $this->backup->database); - - $this->backupEnabled = ! $this->backupEnabled; - $this->backup->enabled = $this->backupEnabled; - $changedFields = auditChangedFields($this->backup); - $this->backup->save(); - $this->auditBackupScheduleUpdated($changedFields); - $this->dispatch('success', $this->backupEnabled ? 'Backup enabled.' : 'Backup disabled.'); + $this->backupEnabled = $this->toggleDatabaseBackup($this->backup); + $this->dispatch('backupEnabledChanged')->to(Execution::class); } catch (\Throwable $e) { $this->dispatch('error', $e->getMessage()); } } + public function syncBackupEnabled(): void + { + $this->backupEnabled = $this->backup->refresh()->enabled; + } + public function updatedS3StorageIds(): void { $this->instantSave(); diff --git a/app/Livewire/Project/Database/BackupExecutions.php b/app/Livewire/Project/Database/BackupExecutions.php index f9d4658688..168076238f 100644 --- a/app/Livewire/Project/Database/BackupExecutions.php +++ b/app/Livewire/Project/Database/BackupExecutions.php @@ -5,6 +5,7 @@ namespace App\Livewire\Project\Database; use App\Models\ScheduledDatabaseBackup; use App\Models\ServiceDatabase; use App\Traits\ListensToTeamChannel; +use App\Traits\PreparesBackupDownloads; use Illuminate\Foundation\Auth\Access\AuthorizesRequests; use Illuminate\Support\Collection; use Livewire\Component; @@ -13,6 +14,7 @@ class BackupExecutions extends Component { use AuthorizesRequests; use ListensToTeamChannel; + use PreparesBackupDownloads; public ?ScheduledDatabaseBackup $backup = null; @@ -126,9 +128,25 @@ class BackupExecutions extends Component return true; } - public function download_file($exeuctionId) + public function download_file($executionId): void { - return redirect()->route('download.backup', $exeuctionId); + try { + $this->authorize('manageBackups', $this->database); + $execution = $this->backup->executions()->whereKey($executionId)->first(); + if (! $execution) { + $this->dispatch('error', 'Backup execution not found.'); + + return; + } + + $this->prepareBackupDownload( + $execution->local_storage_deleted ? null : $this->server(), + $execution->filename, + route('download.backup', $execution->id), + ); + } catch (\Throwable $e) { + handleError($e, $this); + } } public function refreshBackupExecutions(): void @@ -223,6 +241,8 @@ class BackupExecutions extends Component public function render() { - return view('livewire.project.database.backup-executions'); + return view('livewire.project.database.backup-executions', [ + 's3Downloads' => $this->s3BackupDownloadsEnabled(), + ]); } } diff --git a/app/Livewire/Project/Database/Clickhouse/General.php b/app/Livewire/Project/Database/Clickhouse/General.php index ea8d8f197b..2cd0bfecee 100644 --- a/app/Livewire/Project/Database/Clickhouse/General.php +++ b/app/Livewire/Project/Database/Clickhouse/General.php @@ -122,8 +122,11 @@ class General extends Component $this->validate(); $this->database->name = $this->name; $this->database->description = $this->description; - $this->database->clickhouse_admin_user = $this->clickhouseAdminUser; - $this->database->clickhouse_admin_password = $this->clickhouseAdminPassword; + // After the first start the credentials are read-only, so autofilled values must not be saved. + if (! $this->database->started_at) { + $this->database->clickhouse_admin_user = $this->clickhouseAdminUser; + $this->database->clickhouse_admin_password = $this->clickhouseAdminPassword; + } $this->database->image = $this->image; $this->database->ports_mappings = $this->portsMappings; $this->database->is_public = $this->isPublic; diff --git a/app/Livewire/Project/Database/Dragonfly/General.php b/app/Livewire/Project/Database/Dragonfly/General.php index 1505aad781..b3c5bcc945 100644 --- a/app/Livewire/Project/Database/Dragonfly/General.php +++ b/app/Livewire/Project/Database/Dragonfly/General.php @@ -110,7 +110,10 @@ class General extends Component $this->validate(); $this->database->name = $this->name; $this->database->description = $this->description; - $this->database->dragonfly_password = $this->dragonflyPassword; + // After the first start the password is read-only, so autofilled values must not be saved. + if (! $this->database->started_at) { + $this->database->dragonfly_password = $this->dragonflyPassword; + } $this->database->image = $this->image; $this->database->ports_mappings = $this->portsMappings; $this->database->is_public = $this->isPublic; diff --git a/app/Livewire/Project/Database/Keydb/General.php b/app/Livewire/Project/Database/Keydb/General.php index 2ae64f3121..fd84bad209 100644 --- a/app/Livewire/Project/Database/Keydb/General.php +++ b/app/Livewire/Project/Database/Keydb/General.php @@ -114,7 +114,10 @@ class General extends Component $this->database->name = $this->name; $this->database->description = $this->description; $this->database->keydb_conf = $this->keydbConf; - $this->database->keydb_password = $this->keydbPassword; + // After the first start the password is read-only, so autofilled values must not be saved. + if (! $this->database->started_at) { + $this->database->keydb_password = $this->keydbPassword; + } $this->database->image = $this->image; $this->database->ports_mappings = $this->portsMappings; $this->database->is_public = $this->isPublic; diff --git a/app/Livewire/Project/Database/ScheduledBackups.php b/app/Livewire/Project/Database/ScheduledBackups.php index 2071337b4b..b9da614ea3 100644 --- a/app/Livewire/Project/Database/ScheduledBackups.php +++ b/app/Livewire/Project/Database/ScheduledBackups.php @@ -3,6 +3,7 @@ namespace App\Livewire\Project\Database; use App\Models\ServiceDatabase; +use App\Traits\ListensToTeamChannel; use Illuminate\Contracts\View\View; use Illuminate\Foundation\Auth\Access\AuthorizesRequests; use Livewire\Component; @@ -10,6 +11,7 @@ use Livewire\Component; class ScheduledBackups extends Component { use AuthorizesRequests; + use ListensToTeamChannel; public $database; @@ -19,7 +21,13 @@ class ScheduledBackups extends Component public string $custom_type = 'mysql'; - protected $listeners = ['refreshScheduledBackups']; + public function getListeners(): array + { + return [ + 'refreshScheduledBackups', + ...$this->teamChannelListeners(['BackupCreated' => '$refresh']), + ]; + } public function mount(): void { diff --git a/app/Livewire/Project/New/GithubPrivateRepositoryDeployKey.php b/app/Livewire/Project/New/GithubPrivateRepositoryDeployKey.php index d55972e97d..5a777d7a59 100644 --- a/app/Livewire/Project/New/GithubPrivateRepositoryDeployKey.php +++ b/app/Livewire/Project/New/GithubPrivateRepositoryDeployKey.php @@ -248,6 +248,15 @@ class GithubPrivateRepositoryDeployKey extends Component return; } + // Keep ssh:// URLs as-is so a custom SSH port is preserved for deployments. + if (str_starts_with($this->repository_url, 'ssh://')) { + $this->git_host = parse_url($this->repository_url, PHP_URL_HOST); + $this->git_repository = $this->repository_url; + $this->git_source = 'other'; + + return; + } + $this->repository_url_parsed = Url::fromString($this->repository_url); $this->git_host = $this->repository_url_parsed->getHost(); $this->git_repository = $this->repository_url_parsed->getSegment(1).'/'.$this->repository_url_parsed->getSegment(2); diff --git a/app/Livewire/Project/New/PublicGitRepository.php b/app/Livewire/Project/New/PublicGitRepository.php index c14c9dcd37..a898204fc2 100644 --- a/app/Livewire/Project/New/PublicGitRepository.php +++ b/app/Livewire/Project/New/PublicGitRepository.php @@ -139,7 +139,7 @@ class PublicGitRepository extends Component throw new \RuntimeException('Invalid repository URL: '.$validator->errors()->first('repository_url')); } - $httpsRepositoryUrl = scpStyleGitUrlToHttps($this->repository_url); + $httpsRepositoryUrl = sshGitUrlToHttps($this->repository_url); if (is_string($httpsRepositoryUrl)) { $this->repository_url = $httpsRepositoryUrl; } diff --git a/app/Livewire/Project/New/Select.php b/app/Livewire/Project/New/Select.php index 9986bfa1f5..2549cf5239 100644 --- a/app/Livewire/Project/New/Select.php +++ b/app/Livewire/Project/New/Select.php @@ -118,6 +118,7 @@ class Select extends Component 'name' => str($serviceKey)->headline(), 'docsSlug' => str($serviceKey)->lower()->value(), 'templateLastUpdated' => $templateLastUpdatedMap[$serviceKey] ?? null, + 'website' => $this->serviceWebsiteUrl(data_get($service, 'documentation')), ] + service_logo_urls(data_get($service, 'logo')) + (array) $service; })->all(); @@ -288,6 +289,29 @@ class Select extends Component } } + /** + * Derive a website link from the documentation URL. Shared hosts keep the + * path that identifies the project, so the link does not open the host root. + */ + private function serviceWebsiteUrl(?string $documentation): ?string + { + $url = parse_url((string) $documentation); + if (! in_array($url['scheme'] ?? null, ['http', 'https'], true) || blank($url['host'] ?? null)) { + return null; + } + + $projectPathSegments = [ + 'github.com' => 2, + 'gitlab.com' => 2, + 'codeberg.org' => 2, + 'hub.docker.com' => 3, + ][str($url['host'])->lower()->chopStart('www.')->value()] ?? 0; + + $projectPath = str($url['path'] ?? '')->explode('/')->filter()->take($projectPathSegments)->implode('/'); + + return rtrim("{$url['scheme']}://{$url['host']}/{$projectPath}", '/'); + } + private function serviceTemplatesLastUpdated(): ?string { $fetchedAt = get_service_templates_fetched_at(); diff --git a/app/Livewire/Project/Service/BackupExecutions.php b/app/Livewire/Project/Service/BackupExecutions.php index b319f66412..d4b63e4766 100644 --- a/app/Livewire/Project/Service/BackupExecutions.php +++ b/app/Livewire/Project/Service/BackupExecutions.php @@ -9,6 +9,7 @@ use App\Models\ScheduledVolumeBackupExecution; use App\Models\Service; use App\Models\ServiceDatabase; use App\Traits\ListensToTeamChannel; +use App\Traits\PreparesBackupDownloads; use Illuminate\Contracts\View\View; use Illuminate\Database\Query\Builder; use Illuminate\Foundation\Auth\Access\AuthorizesRequests; @@ -20,6 +21,7 @@ class BackupExecutions extends Component { use AuthorizesRequests; use ListensToTeamChannel; + use PreparesBackupDownloads; use WithPagination; public Service $service; @@ -62,6 +64,29 @@ class BackupExecutions extends Component $this->executionModalOpen = true; } + public function downloadExecution(string $executionUuid): void + { + $this->authorize('update', $this->service); + $row = $this->executionQuery($executionUuid)->first(); + abort_unless($row, 404); + + try { + if ($row->type === 'database') { + $execution = ScheduledDatabaseBackupExecution::query()->with('scheduledDatabaseBackup')->findOrFail($row->id); + $server = $execution->scheduledDatabaseBackup->server(); + $streamUrl = route('download.backup', $execution->id); + } else { + $execution = ScheduledVolumeBackupExecution::query()->with('scheduledVolumeBackup')->findOrFail($row->id); + $server = $execution->scheduledVolumeBackup->server(); + $streamUrl = route('download.volume-backup', $execution->id); + } + + $this->prepareBackupDownload($execution->local_storage_deleted ? null : $server, $execution->filename, $streamUrl); + } catch (\Throwable $e) { + handleError($e, $this); + } + } + public function closeExecutionModal(): void { $this->executionModalOpen = false; @@ -80,6 +105,7 @@ class BackupExecutions extends Component return view('livewire.project.service.backup-executions', [ 'executions' => $executions, + 's3Downloads' => $this->s3BackupDownloadsEnabled(), ]); } @@ -114,11 +140,11 @@ class BackupExecutions extends Component private function formatExecutions(Collection $rows): Collection { $databaseExecutions = ScheduledDatabaseBackupExecution::query() - ->with(['scheduledDatabaseBackup.database', 'scheduledDatabaseBackup.s3', 'scheduledDatabaseBackup.s3Storages']) + ->with(['scheduledDatabaseBackup.database', 'scheduledDatabaseBackup.s3', 'scheduledDatabaseBackup.s3Storages', 's3Replicas']) ->whereIn('id', $rows->where('type', 'database')->pluck('id')) ->get()->keyBy('id'); $volumeExecutions = ScheduledVolumeBackupExecution::query() - ->with(['scheduledVolumeBackup.backupable.resource', 's3']) + ->with(['scheduledVolumeBackup.backupable.resource', 's3', 's3Replicas']) ->whereIn('id', $rows->where('type', 'storage')->pluck('id')) ->get()->keyBy('id'); @@ -156,7 +182,7 @@ class BackupExecutions extends Component 'size' => $execution->size, 'message' => $execution->message, 'filename' => $execution->filename, - 'download_url' => $execution->status === 'success' && ! $execution->local_storage_deleted + 'download_url' => $execution->status === 'success' && $execution->isDownloadable() ? route($isDatabase ? 'download.backup' : 'download.volume-backup', $execution->id) : null, ]; diff --git a/app/Livewire/Project/Service/DatabaseBackups.php b/app/Livewire/Project/Service/DatabaseBackups.php index 71885094aa..c778572f94 100644 --- a/app/Livewire/Project/Service/DatabaseBackups.php +++ b/app/Livewire/Project/Service/DatabaseBackups.php @@ -84,8 +84,6 @@ class DatabaseBackups extends Component ->get(); $this->backupParameters = [...$this->parameters, 'backup_uuid' => $this->backup->uuid]; $this->section = match (request()->route()?->getName()) { - 'project.service.database.backup.s3' => 's3', - 'project.service.database.backup.retention' => 'retention', 'project.service.database.backup.executions' => 'executions', 'project.service.database.backup.danger' => 'danger', default => 'general', diff --git a/app/Livewire/Project/Service/ResourceCard.php b/app/Livewire/Project/Service/ResourceCard.php index 22ed772a80..5b67baab01 100644 --- a/app/Livewire/Project/Service/ResourceCard.php +++ b/app/Livewire/Project/Service/ResourceCard.php @@ -24,6 +24,7 @@ class ResourceCard extends Component public function getListeners(): array { return $this->teamChannelListeners([ + 'ServiceStatusChanged' => 'refreshResource', 'ServiceChecked' => 'refreshResource', ]); } diff --git a/app/Livewire/Project/Service/VolumeBackup/Index.php b/app/Livewire/Project/Service/VolumeBackup/Index.php index 64dbbb8df2..e73bc12216 100644 --- a/app/Livewire/Project/Service/VolumeBackup/Index.php +++ b/app/Livewire/Project/Service/VolumeBackup/Index.php @@ -45,6 +45,7 @@ class Index extends Component 'modalClosed' => 'closeScheduleModal', ...$this->teamChannelListeners([ 'ServiceChecked' => '$refresh', + 'ServiceStatusChanged' => '$refresh', 'BackupCreated' => '$refresh', ]), ]; diff --git a/app/Livewire/Project/Service/VolumeBackup/Show.php b/app/Livewire/Project/Service/VolumeBackup/Show.php index 10abeb3bf4..8ca50d458a 100644 --- a/app/Livewire/Project/Service/VolumeBackup/Show.php +++ b/app/Livewire/Project/Service/VolumeBackup/Show.php @@ -37,8 +37,6 @@ class Show extends Component ->firstOrFail(); $this->parameters = get_route_parameters(); $this->section = match (request()->route()?->getName()) { - 'project.service.volume-backups.s3' => 's3', - 'project.service.volume-backups.retention' => 'retention', 'project.service.volume-backups.executions' => 'executions', 'project.service.volume-backups.danger' => 'danger', default => 'general', diff --git a/app/Livewire/Project/Shared/EnvironmentVariable/All.php b/app/Livewire/Project/Shared/EnvironmentVariable/All.php index daad018b2b..736075dc7c 100644 --- a/app/Livewire/Project/Shared/EnvironmentVariable/All.php +++ b/app/Livewire/Project/Shared/EnvironmentVariable/All.php @@ -57,6 +57,8 @@ class All extends Component public bool $use_build_secrets = false; + public bool $include_coolify_default_variables = true; + public bool $redact_all_env_values_in_logs = true; /** @@ -111,6 +113,8 @@ class All extends Component { $this->is_env_sorting_enabled = data_get($this->resource, 'settings.is_env_sorting_enabled', false); $this->use_build_secrets = data_get($this->resource, 'settings.use_build_secrets', false); + $coolifyDefaultVariablesOwner = $this->resource instanceof Application ? $this->resource->settings : $this->resource; + $this->include_coolify_default_variables = data_get($coolifyDefaultVariablesOwner, 'include_coolify_default_variables') !== false; if ($this->hasLogRedactionSetting()) { $this->redact_all_env_values_in_logs = $this->resource->redactsAllEnvValuesInLogs(); } @@ -201,6 +205,25 @@ class All extends Component } } + public function saveCoolifyDefaultVariables() + { + try { + $this->authorize('manageEnvironment', $this->resource); + + if ($this->resource instanceof Application) { + $this->resource->settings->include_coolify_default_variables = $this->include_coolify_default_variables; + $this->saveApplicationSettingsWithAudit($this->resource); + } else { + $this->resource->include_coolify_default_variables = $this->include_coolify_default_variables; + $this->resource->save(); + } + $this->dispatch('success', 'Environment variable settings updated.'); + $this->dispatch('configurationChanged'); + } catch (\Throwable $e) { + return handleError($e, $this); + } + } + public function getEnvironmentVariablesProperty() { if (! $this->readyToLoad) { diff --git a/app/Livewire/Project/Shared/ExecuteContainerCommand.php b/app/Livewire/Project/Shared/ExecuteContainerCommand.php index d8ce2e03fc..a3ddb9dc2e 100644 --- a/app/Livewire/Project/Shared/ExecuteContainerCommand.php +++ b/app/Livewire/Project/Shared/ExecuteContainerCommand.php @@ -76,9 +76,6 @@ class ExecuteContainerCommand extends Component $this->type = 'service'; $this->resource = Service::ownedByCurrentTeam()->where('uuid', $this->parameters['service_uuid'])->firstOrFail(); $this->authorize('view', $this->resource); - if (! $this->resource->isRunning()) { - $this->containersLoaded = true; - } if ($this->resource->server->isFunctional()) { $this->servers = $this->servers->push($this->resource->server); } @@ -140,7 +137,7 @@ class ExecuteContainerCommand extends Component } }); $this->resource->databases()->get()->each(function ($database) { - if ($database->isRunning()) { + if ($database->isRunning() && $this->resource->server->isTerminalEnabled()) { $this->containers->push([ 'server' => $this->resource->server, 'container' => [ diff --git a/app/Livewire/Project/Shared/FileBrowser.php b/app/Livewire/Project/Shared/FileBrowser.php new file mode 100644 index 0000000000..09f8f0c6b9 --- /dev/null +++ b/app/Livewire/Project/Shared/FileBrowser.php @@ -0,0 +1,545 @@ + */ + #[Locked] + public array $availableContainers = []; + + public string $currentPath = ''; + + /** @var array */ + public array $entries = []; + + #[Locked] + public ?string $editingPath = null; + + public bool $editorOpen = false; + + public string $editorContent = ''; + + public string $editorLanguage = ''; + + public ?TemporaryUploadedFile $upload = null; + + /** @var array */ + #[Locked] + public array $pendingDownloads = []; + + /** @var array */ + public array $parameters = []; + + public function mount(?Model $resource = null): void + { + $this->parameters = get_route_parameters(); + $resource = $resource ?? $this->resolveResourceFromRoute(); + + // Security gate first, before any server or container resolution. + $this->authorize('canAccessTerminal'); + $this->authorize('view', $resource); + abort_unless((int) $resource->environment->project->team_id === (int) auth()->user()->currentTeam()->id, 403); + + $this->resource = $resource; + $this->type = $this->resolveType($resource); + $this->containerRunning = (bool) $resource->isRunning(); + + $this->filesLoaded = ! $this->containerRunning; + } + + public function loadFiles(): void + { + $this->authorizeFileAccess(); + if ($this->filesLoaded) { + return; + } + $this->loadFailed = false; + $this->containerRunning = (bool) $this->resource->isRunning(); + + try { + $this->availableContainers = $this->containerRunning ? $this->resolveContainers() : []; + $this->container = $this->availableContainers[0] ?? null; + if (is_null($this->container)) { + $this->containerRunning = false; + } else { + $this->currentPath = $this->service()->defaultRoot(); + $this->loadEntries(); + } + $this->filesLoaded = true; + } catch (\Throwable $e) { + $this->container = null; + $this->loadFailed = true; + handleError($e, $this); + } + } + + public function selectContainer(string $name): void + { + $this->guard(); + if (! in_array($name, $this->availableContainers, true)) { + return; + } + $this->closeEditor(); + $this->container = $name; + $this->currentPath = $this->service()->defaultRoot(); + $this->loadEntries(); + } + + public function open(string $name): void + { + $this->guard(); + $this->currentPath = $this->childPath($name); + $this->loadEntries(); + } + + public function goTo(string $path): void + { + $this->guard(); + if (! str_starts_with($path, '/')) { + return; + } + $this->currentPath = $path === '/' ? '/' : rtrim($path, '/'); + $this->loadEntries(); + } + + public function refresh(): void + { + $this->guard(); + $this->loadEntries(); + } + + public function createDirectory(string $name): void + { + $this->guard(); + try { + $this->service()->makeDirectory($this->childPath($name)); + $this->loadEntries(); + $this->dispatch('success', 'Folder created.'); + } catch (\Throwable $e) { + handleError($e, $this); + } + } + + public function createFile(string $name): void + { + $this->guard(); + try { + $this->service()->createFile($this->childPath($name)); + $this->loadEntries(); + $this->dispatch('success', 'File created.'); + } catch (\Throwable $e) { + handleError($e, $this); + } + } + + public function chmodEntry(string $name, string $mode): void + { + $this->guard(); + try { + $this->service()->chmod($this->childPath($name), $mode); + $this->loadEntries(); + $this->dispatch('success', 'Permissions updated.'); + } catch (\Throwable $e) { + handleError($e, $this); + } + } + + public function renameEntry(string $from, string $newName): void + { + $this->guard(); + try { + $this->service()->rename($this->childPath($from), $this->childPath($newName)); + $this->loadEntries(); + $this->dispatch('success', 'Renamed.'); + } catch (\Throwable $e) { + handleError($e, $this); + } + } + + public function deleteEntry(string $name): void + { + $this->guard(); + try { + $this->service()->delete($this->childPath($name)); + $this->loadEntries(); + $this->dispatch('success', 'Deleted.'); + } catch (\Throwable $e) { + handleError($e, $this); + } + } + + public function openEditor(string $name): void + { + $this->guard(); + $path = $this->childPath($name); + try { + // read() guards binary/oversized files internally and throws. + $this->editorContent = $this->service()->read($path); + $this->editorLanguage = $this->guessLanguage($name); + $this->editingPath = $path; + $this->editorOpen = true; + // Push content + language into the mounted Monaco editor. It lives + // behind wire:ignore, so a dispatched event is how it receives them. + $this->dispatch('load-file-editor', content: $this->editorContent, language: $this->editorLanguage); + } catch (\RuntimeException $e) { + $this->dispatch('error', "Can't edit this file - it's binary or larger than 5 MB. Download it instead."); + } catch (\Throwable $e) { + handleError($e, $this); + } + } + + public function saveEditor(): void + { + $this->guard(); + if (is_null($this->editingPath)) { + return; + } + try { + $this->service()->write($this->editingPath, $this->editorContent); + $this->editingPath = null; + $this->editorOpen = false; + $this->editorContent = ''; + $this->loadEntries(); + $this->dispatch('success', 'Saved.'); + } catch (\Throwable $e) { + handleError($e, $this); + } + } + + public function closeEditor(): void + { + $this->editingPath = null; + $this->editorOpen = false; + $this->editorContent = ''; + } + + public function uploadFile(): void + { + $this->guard(); + $this->validate(['upload' => 'required|file|max:1048576']); // 1 GB (KB units) + try { + $localTmp = $this->upload->getRealPath(); + $filename = $this->sanitizeUploadName($this->upload->getClientOriginalName()); + $this->service()->upload($localTmp, $this->childPath($filename)); + $this->upload = null; + $this->loadEntries(); + $this->dispatch('success', 'Uploaded.'); + } catch (\Throwable $e) { + handleError($e, $this); + } + } + + public function download(string $name): void + { + $this->guard(); + try { + $path = $this->childPath($name); + $target = $this->service()->downloadInfo($path); + $downloadName = $target['isDir'] ? $name.'.tar.gz' : $name; + $s3Storage = FileDownloadStorageService::configuredStorage(); + $maxSize = $s3Storage ? FileDownloadStorageService::MAX_SIZE : ContainerFilesystemService::MAX_DOWNLOAD_SIZE; + if ($target['size'] > $maxSize) { + $this->dispatch('error', ($target['isDir'] ? 'This folder' : 'This file').' is larger than '.formatBytes($maxSize).' and cannot be downloaded here. Use the terminal or a volume backup instead.'); + + return; + } + + if ($s3Storage) { + if (count($this->pendingDownloads) >= self::MAX_PENDING_S3_DOWNLOADS + || ! RateLimiter::attempt('file-browser-s3-download:'.auth()->id(), 10, fn () => true, 3600)) { + $this->dispatch('error', 'Too many downloads are in progress. Wait for them to finish, then try again.'); + + return; + } + $downloadId = (string) Str::uuid(); + $this->pendingDownloads[] = $downloadId; + PrepareFileDownloadJob::dispatch($downloadId, auth()->id(), $this->server()->id, $this->container, $path, $target['isDir'], $downloadName, $s3Storage->id); + $this->dispatch('info', 'Preparing '.$downloadName.'. The download starts when it is ready.'); + + return; + } + + $token = Str::random(64); + Cache::put(FileBrowserDownloadController::cacheKey($token), [ + 'user_id' => auth()->id(), + 'resource_type' => $this->resource::class, + 'resource_id' => $this->resource->getKey(), + 'server_id' => $this->server()->id, + 'container' => $this->container, + 'path' => $path, + 'name' => $downloadName, + 'is_dir' => $target['isDir'], + ], FileBrowserDownloadController::TOKEN_TTL_SECONDS); + + $this->redirect(route('project.files.download', $token)); + } catch (\Throwable $e) { + handleError($e, $this); + } + } + + /** + * Polled while S3 downloads are prepared. Opens each finished link once. + */ + public function checkDownloads(): void + { + $this->authorizeFileAccess(); + foreach ($this->pendingDownloads as $index => $downloadId) { + $result = Cache::get(PrepareFileDownloadJob::cacheKey($downloadId)); + if (! is_array($result) || ($result['user_id'] ?? null) !== auth()->id()) { + continue; + } + + unset($this->pendingDownloads[$index]); + Cache::forget(PrepareFileDownloadJob::cacheKey($downloadId)); + if ($result['status'] === 'ready') { + $this->dispatch('open-file-download', url: $result['url']); + } else { + $this->dispatch('error', $result['message']); + } + } + $this->pendingDownloads = array_values($this->pendingDownloads); + } + + /** + * Reduce a client-supplied upload name to a single safe basename, + * stripping any directory traversal before it reaches the container. + */ + protected function sanitizeUploadName(string $name): string + { + $name = basename(str_replace('\\', '/', $name)); + if ($name === '' || preg_match('/^\.+$/', $name) === 1) { + throw new \RuntimeException('Invalid file name.'); + } + + return $name; + } + + /** + * Map a filename to a Monaco language id for the editor. Falls back to + * well-known whole-file names so extension-less files still highlight. + */ + protected function guessLanguage(string $name): string + { + $extMap = [ + 'js' => 'javascript', 'mjs' => 'javascript', 'cjs' => 'javascript', 'ts' => 'typescript', + 'json' => 'json', 'php' => 'php', 'py' => 'python', 'rb' => 'ruby', 'go' => 'go', + 'rs' => 'rust', 'java' => 'java', 'sh' => 'shell', 'bash' => 'shell', 'zsh' => 'shell', + 'yml' => 'yaml', 'yaml' => 'yaml', 'toml' => 'ini', 'ini' => 'ini', 'env' => 'ini', + 'conf' => 'ini', 'cnf' => 'ini', 'md' => 'markdown', 'markdown' => 'markdown', 'html' => 'html', + 'htm' => 'html', 'xml' => 'xml', 'svg' => 'xml', 'css' => 'css', 'scss' => 'scss', + 'sql' => 'sql', 'dockerfile' => 'dockerfile', + ]; + + $base = strtolower(basename($name)); + + // Whole-name matches for common extension-less/dotfiles. + $nameMap = [ + 'dockerfile' => 'dockerfile', 'containerfile' => 'dockerfile', + '.gitignore' => 'ini', '.dockerignore' => 'ini', '.editorconfig' => 'ini', + '.npmrc' => 'ini', '.gitconfig' => 'ini', + '.bashrc' => 'shell', '.bash_profile' => 'shell', '.bash_aliases' => 'shell', + '.profile' => 'shell', '.zshrc' => 'shell', '.zprofile' => 'shell', + ]; + if (isset($nameMap[$base])) { + return $nameMap[$base]; + } + if ($base === '.env' || str_starts_with($base, '.env.')) { + return 'ini'; + } + + $ext = strtolower(pathinfo($name, PATHINFO_EXTENSION)); + + return $extMap[$ext] ?? ''; + } + + protected function childPath(string $name): string + { + if ($name === '' || $name === '.' || $name === '..' || str_contains($name, '/') || str_contains($name, '\\') || str_contains($name, "\0")) { + throw new \InvalidArgumentException('Invalid file name.'); + } + validateShellSafePath($name, 'file name'); + + return rtrim($this->currentPath, '/').'/'.$name; + } + + protected function authorizeFileAccess(): void + { + $this->authorize('canAccessTerminal'); + $this->resource->refresh(); + $this->authorize('view', $this->resource); + abort_unless((int) $this->resource->environment->project->team_id === (int) auth()->user()->currentTeam()->id, 403); + } + + protected function guard(): void + { + $this->authorizeFileAccess(); + $this->type = $this->resolveType($this->resource); + $this->containerRunning = (bool) $this->resource->isRunning(); + $this->availableContainers = $this->containerRunning ? $this->resolveContainers() : []; + if (! $this->containerRunning || is_null($this->container) || ! in_array($this->container, $this->availableContainers, true)) { + throw new \RuntimeException('Container is not running.'); + } + } + + /** + * @return array + */ + public function breadcrumbs(): array + { + $crumbs = [['label' => '/', 'path' => '/']]; + $accumulated = ''; + foreach (array_filter(explode('/', $this->currentPath)) as $segment) { + $accumulated .= '/'.$segment; + $crumbs[] = ['label' => $segment, 'path' => $accumulated]; + } + + return $crumbs; + } + + protected function loadEntries(): void + { + $entries = $this->service()->list($this->currentPath); + $this->entries = array_map(fn ($e) => [ + 'name' => $e->name, + 'type' => $e->type, + 'size' => $e->size, + 'mtime' => $e->mtime, + 'perms' => $e->perms, + 'owner' => $e->owner, + 'group' => $e->group, + ], $entries); + } + + protected function service(): ContainerFilesystemService + { + return new ContainerFilesystemService($this->server(), $this->container); + } + + protected function server(): Server + { + return match ($this->type) { + 'service' => $this->resource->server, + default => $this->resource->destination->server, + }; + } + + protected function resolveResourceFromRoute(): Model + { + $parameters = get_route_parameters(); + $teamId = data_get(auth()->user()->currentTeam(), 'id'); + + if ($uuid = data_get($parameters, 'application_uuid')) { + return Application::ownedByCurrentTeam()->where('uuid', $uuid)->firstOrFail(); + } + if ($uuid = data_get($parameters, 'service_uuid')) { + return Service::ownedByCurrentTeam()->where('uuid', $uuid)->firstOrFail(); + } + if ($uuid = data_get($parameters, 'database_uuid')) { + $resource = getResourceByUuid($uuid, $teamId); + if (is_null($resource)) { + abort(404); + } + + return $resource; + } + + abort(404); + } + + protected function resolveType(Model $resource): string + { + return match (true) { + $resource instanceof Application => 'application', + $resource instanceof Service => 'service', + default => 'database', + }; + } + + /** + * @return array + */ + protected function resolveContainers(): array + { + if ($this->type === 'service') { + $names = []; + $this->resource->applications()->get()->each(function ($application) use (&$names) { + if ($application->isRunning()) { + $names[] = $application->name.'-'.$this->resource->uuid; + } + }); + $this->resource->databases()->get()->each(function ($database) use (&$names) { + if ($database->isRunning()) { + $names[] = $database->name.'-'.$this->resource->uuid; + } + }); + + return array_values(array_filter($names, [ValidationPatterns::class, 'isValidContainerName'])); + } + + if ($this->type === 'application') { + $server = $this->server(); + $names = getCurrentApplicationContainerStatus($server, $this->resource, includePullrequests: false) + ->filter(fn ($c) => data_get($c, 'State') === 'running') + ->map(fn ($c) => data_get($c, 'Names')) + ->filter() + ->values() + ->all(); + + return array_values(array_filter($names, [ValidationPatterns::class, 'isValidContainerName'])); + } + + // Standalone database: the container name is the resource uuid. + $name = $this->resource->uuid; + + return ValidationPatterns::isValidContainerName($name) ? [$name] : []; + } + + public function render(): View + { + return view('livewire.project.shared.file-browser'); + } +} diff --git a/app/Livewire/Project/Shared/ResourceTrafficAnalytics.php b/app/Livewire/Project/Shared/ResourceTrafficAnalytics.php index d936333a92..02f1718ede 100644 --- a/app/Livewire/Project/Shared/ResourceTrafficAnalytics.php +++ b/app/Livewire/Project/Shared/ResourceTrafficAnalytics.php @@ -65,7 +65,7 @@ abstract class ResourceTrafficAnalytics extends Component public array $series = []; /** @var array */ - protected array $breakdownDimensions = ['country', 'referer', 'browser', 'os', 'device', 'protocol', 'cache', 'status', 'agent', 'ip', 'useragent']; + protected array $breakdownDimensions = ['country', 'city', 'referer', 'browser', 'os', 'device', 'protocol', 'cache', 'status', 'agent', 'ip', 'useragent']; /** * Name of the public model property and lazy-load parameter ('application' / 'service'). diff --git a/app/Livewire/Project/Shared/ScheduledTask/All.php b/app/Livewire/Project/Shared/ScheduledTask/All.php index b58e4f97a9..124c907210 100644 --- a/app/Livewire/Project/Shared/ScheduledTask/All.php +++ b/app/Livewire/Project/Shared/ScheduledTask/All.php @@ -2,13 +2,18 @@ namespace App\Livewire\Project\Shared\ScheduledTask; +use Illuminate\Contracts\View\View; +use Illuminate\Foundation\Auth\Access\AuthorizesRequests; use Illuminate\Support\Collection; use Livewire\Attributes\Locked; use Livewire\Attributes\On; +use Livewire\Attributes\Url; use Livewire\Component; class All extends Component { + use AuthorizesRequests; + #[Locked] public $resource; @@ -19,6 +24,18 @@ class All extends Component public ?string $variables = null; + #[Url(as: 'task_uuid', except: '')] + public string $taskUuid = ''; + + public bool $taskModalOpen = false; + + public function getListeners(): array + { + return [ + 'modalClosed' => 'closeTask', + ]; + } + public function mount() { $this->parameters = get_route_parameters(); @@ -34,6 +51,30 @@ class All extends Component $this->containerNames = collect([]); } } + + if ($this->taskUuid !== '') { + $this->openTask($this->taskUuid); + } + } + + public function openTask(string $taskUuid): void + { + $this->authorize('view', $this->resource); + $this->taskUuid = $this->resource->scheduled_tasks()->where('uuid', $taskUuid)->firstOrFail()->uuid; + $this->taskModalOpen = true; + } + + public function closeTask(): void + { + $this->taskUuid = ''; + $this->taskModalOpen = false; + $this->resource->refresh(); + } + + #[On('scheduledTaskDeleted')] + public function taskDeleted(): void + { + $this->closeTask(); } #[On('refreshTasks')] @@ -41,4 +82,13 @@ class All extends Component { $this->resource->refresh(); } + + public function render(): View + { + return view('livewire.project.shared.scheduled-task.all', [ + 'selectedTask' => $this->taskUuid === '' + ? null + : $this->resource->scheduled_tasks()->where('uuid', $this->taskUuid)->first(), + ]); + } } diff --git a/app/Livewire/Project/Shared/ScheduledTask/Executions.php b/app/Livewire/Project/Shared/ScheduledTask/Executions.php index 76e0735ff4..f7e5ca1d1f 100644 --- a/app/Livewire/Project/Shared/ScheduledTask/Executions.php +++ b/app/Livewire/Project/Shared/ScheduledTask/Executions.php @@ -6,6 +6,7 @@ use App\Models\ScheduledTask; use App\Traits\ListensToTeamChannel; use Illuminate\Support\Collection; use Illuminate\Support\Facades\Auth; +use Livewire\Attributes\Computed; use Livewire\Attributes\Locked; use Livewire\Component; @@ -13,9 +14,6 @@ class Executions extends Component { use ListensToTeamChannel; - #[Locked] - public ScheduledTask $task; - #[Locked] public int $taskId; @@ -48,11 +46,11 @@ class Executions extends Component { try { $this->taskId = $taskId; - $this->task = ScheduledTask::where('team_id', Auth::user()->currentTeam()->id)->findOrFail($taskId); - $this->executions = $this->task->executions()->take(20)->get(); - $this->serverTimezone = data_get($this->task, 'application.destination.server.settings.server_timezone'); + $task = ScheduledTask::where('team_id', Auth::user()->currentTeam()->id)->findOrFail($taskId); + $this->executions = $task->executions()->take(20)->get(); + $this->serverTimezone = data_get($task, 'application.destination.server.settings.server_timezone'); if (! $this->serverTimezone) { - $this->serverTimezone = data_get($this->task, 'service.destination.server.settings.server_timezone'); + $this->serverTimezone = data_get($task, 'service.destination.server.settings.server_timezone'); } if (! $this->serverTimezone) { $this->serverTimezone = 'UTC'; @@ -62,8 +60,22 @@ class Executions extends Component } } + /** + * Load the task on each request instead of storing it as a model property. A stored model fails + * with a 404 when the poll runs after the task is deleted and before the page navigates away. + */ + #[Computed] + public function task(): ?ScheduledTask + { + return ScheduledTask::find($this->taskId); + } + public function refreshExecutions(): void { + if (! $this->task) { + return; + } + $this->executions = $this->task->executions()->take(20)->get(); if ($this->selectedKey) { $this->selectedExecution = $this->task->executions()->find($this->selectedKey); @@ -84,7 +96,7 @@ class Executions extends Component return; } $this->selectedKey = $key; - $this->selectedExecution = $this->task->executions()->find($key); + $this->selectedExecution = $this->task?->executions()->find($key); $this->currentPage = 1; // Start polling if task is running @@ -157,7 +169,9 @@ class Executions extends Component */ public function canReadOutput(): bool { - return auth()->user()?->can('update', $this->task->application ?? $this->task->service) ?? false; + $resource = $this->task?->application ?? $this->task?->service; + + return $resource && auth()->user()?->can('update', $resource); } public function hasMoreLogs() diff --git a/app/Livewire/Project/Shared/ScheduledTask/Show.php b/app/Livewire/Project/Shared/ScheduledTask/Show.php index 4419b2e701..5f5689b5de 100644 --- a/app/Livewire/Project/Shared/ScheduledTask/Show.php +++ b/app/Livewire/Project/Shared/ScheduledTask/Show.php @@ -45,46 +45,13 @@ class Show extends Component #[Validate(['integer', 'required', 'min:60', 'max:36000'])] public $timeout = 300; - #[Locked] - public ?string $application_uuid; - - #[Locked] - public ?string $service_uuid; - - #[Locked] - public string $task_uuid; - - public function mount() + public function mount(Application|Service $resource, ScheduledTask $task): void { - try { - $task_uuid = request()->route('task_uuid'); - $project_uuid = request()->route('project_uuid'); - $environment_uuid = request()->route('environment_uuid'); - $application_uuid = request()->route('application_uuid'); - $service_uuid = request()->route('service_uuid'); - - $this->task_uuid = $task_uuid; - if ($application_uuid) { - $this->type = 'application'; - $this->application_uuid = $application_uuid; - $this->resource = Application::ownedByCurrentTeam()->where('uuid', $application_uuid)->firstOrFail(); - } elseif ($service_uuid) { - $this->type = 'service'; - $this->service_uuid = $service_uuid; - $this->resource = Service::ownedByCurrentTeamCached()->where('uuid', $service_uuid)->firstOrFail(); - } - $this->parameters = [ - 'environment_uuid' => $environment_uuid, - 'project_uuid' => $project_uuid, - 'application_uuid' => $application_uuid, - 'service_uuid' => $service_uuid, - ]; - - $this->task = $this->resource->scheduled_tasks()->where('uuid', $task_uuid)->firstOrFail(); - $this->syncData(); - } catch (\Exception $e) { - return handleError($e); - } + $this->resource = $resource; + $this->type = $resource->type(); + $this->parameters = get_route_parameters(); + $this->task = $resource->scheduled_tasks()->whereKey($task->id)->firstOrFail(); + $this->syncData(); } /** @@ -178,12 +145,11 @@ class Show extends Component $this->authorize('delete', $this->task); $this->task->delete(); auditLog('ui.scheduled_task.deleted', $this->taskAuditContext()); + $this->dispatch('success', 'Scheduled task deleted.'); + $this->dispatch('scheduledTaskDeleted')->to(All::class); - if ($this->type === 'application') { - return redirectRoute($this, 'project.application.scheduled-tasks.show', $this->parameters); - } else { - return redirectRoute($this, 'project.service.scheduled-tasks.show', $this->parameters); - } + // Without true, the confirmation modal refreshes this deleted task. + return true; } catch (\Exception $e) { return handleError($e); } diff --git a/app/Livewire/Project/Shared/Storages/VolumeBackups.php b/app/Livewire/Project/Shared/Storages/VolumeBackups.php index 083585fee6..bfa122d0e2 100644 --- a/app/Livewire/Project/Shared/Storages/VolumeBackups.php +++ b/app/Livewire/Project/Shared/Storages/VolumeBackups.php @@ -11,6 +11,8 @@ use App\Models\LocalPersistentVolume; use App\Models\S3Storage; use App\Models\ScheduledVolumeBackup; use App\Models\Service; +use App\Traits\ListensToTeamChannel; +use App\Traits\PreparesBackupDownloads; use Illuminate\Database\Eloquent\Builder; use Illuminate\Foundation\Auth\Access\AuthorizesRequests; use Illuminate\Http\RedirectResponse; @@ -24,6 +26,8 @@ use Throwable; class VolumeBackups extends Component { use AuthorizesRequests; + use ListensToTeamChannel; + use PreparesBackupDownloads; use WithPagination; public LocalPersistentVolume|LocalFileVolume $storage; @@ -34,6 +38,9 @@ class VolumeBackups extends Component public string $section = 'general'; + /** The page shows the backup actions in the top header bar instead. */ + public bool $actionsInHeader = false; + public string $frequency = 'daily'; public bool $enabled = false; @@ -67,6 +74,15 @@ class VolumeBackups extends Component public int $perPage = 10; + public function getListeners(): array + { + if ($this->section !== 'executions') { + return []; + } + + return $this->teamChannelListeners(['BackupCreated' => '$refresh']); + } + public function updatedPerPage(): void { $this->perPage = max(1, min(100, $this->perPage)); @@ -329,6 +345,28 @@ class VolumeBackups extends Component $this->dispatch('success', 'Backup recovery queued.'); } + public function downloadBackup(int $executionId): void + { + $this->authorize('update', $this->resource); + + $execution = $this->backup?->executions()->whereKey($executionId)->first(); + if (! $execution) { + $this->dispatch('error', 'Backup execution not found.'); + + return; + } + + try { + $this->prepareBackupDownload( + $execution->local_storage_deleted ? null : $this->backup->server(), + $execution->filename, + route('download.volume-backup', $execution->id), + ); + } catch (Throwable $e) { + handleError($e, $this); + } + } + public function deleteBackup(int $executionId, string $password, array $selectedActions = []): bool|string { $this->authorize('update', $this->resource); @@ -383,6 +421,7 @@ class VolumeBackups extends Component return view('livewire.project.shared.storages.volume-backups', [ 'executions' => $executions ?? collect(), 'latestExecution' => $this->backup?->executions()->first(), + 's3Downloads' => $this->s3BackupDownloadsEnabled(), 'deleteScheduleCheckboxes' => [ ['id' => 'delete_associated_backups_locally', 'label' => 'Delete all local archives created by this schedule.'], ['id' => 'delete_associated_backups_s3', 'label' => 'Delete all S3 archives created by this schedule.'], diff --git a/app/Livewire/Project/Shared/Webhooks.php b/app/Livewire/Project/Shared/Webhooks.php index 172d91fc5e..4a21654838 100644 --- a/app/Livewire/Project/Shared/Webhooks.php +++ b/app/Livewire/Project/Shared/Webhooks.php @@ -2,11 +2,14 @@ namespace App\Livewire\Project\Shared; +use App\Models\Application; +use App\Models\GitlabApp; use App\Rules\ManualWebhookSecret; use Illuminate\Foundation\Auth\Access\AuthorizesRequests; use Illuminate\Support\Str; use Illuminate\Validation\ValidationException; use Livewire\Component; +use Spatie\Url\Url; // Refactored ✅ class Webhooks extends Component @@ -25,6 +28,12 @@ class Webhooks extends Component public ?string $giteaManualWebhook; + /** + * Source-level endpoint for GitLab App sources. GitLab has no app-level webhooks, + * so users add this URL to each GitLab project themselves. + */ + public ?string $gitlabAppWebhook = null; + public ?string $githubManualWebhookSecret = null; public ?string $gitlabManualWebhookSecret = null; @@ -33,6 +42,14 @@ class Webhooks extends Component public ?string $giteaManualWebhookSecret = null; + /** + * Stored secret per component property. The unsaved bar compares the fields + * with these values, because Livewire syncs unsaved field edits with every request. + * + * @var array + */ + public array $savedSecrets = []; + /** * Maps each manual webhook provider to its component property and model attribute. * @@ -54,12 +71,17 @@ class Webhooks extends Component $this->gitlabManualWebhookSecret = data_get($this->resource, 'manual_webhook_secret_gitlab'); $this->bitbucketManualWebhookSecret = data_get($this->resource, 'manual_webhook_secret_bitbucket'); $this->giteaManualWebhookSecret = data_get($this->resource, 'manual_webhook_secret_gitea'); + $this->syncSavedSecrets(); } $this->githubManualWebhook = generateGitManualWebhook($this->resource, 'github'); $this->gitlabManualWebhook = generateGitManualWebhook($this->resource, 'gitlab'); $this->bitbucketManualWebhook = generateGitManualWebhook($this->resource, 'bitbucket'); $this->giteaManualWebhook = generateGitManualWebhook($this->resource, 'gitea'); + + if ($this->resource instanceof Application && $this->resource->source instanceof GitlabApp) { + $this->gitlabAppWebhook = Url::fromString(base_url()).'/webhooks/source/gitlab/events'; + } } public function canViewSecrets(): bool @@ -78,6 +100,7 @@ class Webhooks extends Component 'manual_webhook_secret_bitbucket' => $this->bitbucketManualWebhookSecret, 'manual_webhook_secret_gitea' => $this->giteaManualWebhookSecret, ]); + $this->syncSavedSecrets(); $this->dispatch('success', 'Webhook secrets saved.'); } catch (ValidationException $e) { throw $e; @@ -87,10 +110,10 @@ class Webhooks extends Component } /** - * Return a random secret for the provider's field. The browser fills the field - * as an unsaved change, and the user saves it with the form. + * Generate a random secret for the provider and save it immediately. + * Other unsaved secret edits in the form are kept as they are. */ - public function generateSecret(string $provider): ?string + public function generateSecret(string $provider): void { try { $this->authorize('update', $this->resource); @@ -99,11 +122,21 @@ class Webhooks extends Component throw new \InvalidArgumentException('Unknown webhook provider.'); } - return Str::random(40); + $field = self::SECRET_FIELDS[$provider]; + $secret = Str::random(40); + $this->resource->update([$field['attribute'] => $secret]); + $this->{$field['property']} = $secret; + $this->savedSecrets[$field['property']] = $secret; + $this->dispatch('success', "New {$field['label']} generated and saved."); } catch (\Exception $e) { handleError($e, $this); + } + } - return null; + private function syncSavedSecrets(): void + { + foreach (self::SECRET_FIELDS as $field) { + $this->savedSecrets[$field['property']] = data_get($this->resource, $field['attribute']); } } diff --git a/app/Livewire/Security/ApiTokens.php b/app/Livewire/Security/ApiTokens.php index bd7189bd24..0a255370e5 100644 --- a/app/Livewire/Security/ApiTokens.php +++ b/app/Livewire/Security/ApiTokens.php @@ -2,7 +2,6 @@ namespace App\Livewire\Security; -use App\Models\InstanceSettings; use Illuminate\Foundation\Auth\Access\AuthorizesRequests; use Illuminate\Validation\Rule; use Laravel\Sanctum\PersonalAccessToken; @@ -35,10 +34,6 @@ class ApiTokens extends Component 365 => '1 year', ]; - public $isApiEnabled; - - public $isMcpEnabled; - #[Locked] public bool $canUseRootPermissions = false; @@ -58,9 +53,6 @@ class ApiTokens extends Component public function mount() { - $settings = InstanceSettings::get(); - $this->isApiEnabled = $settings->is_api_enabled; - $this->isMcpEnabled = $settings->is_mcp_server_enabled; $this->canUseRootPermissions = auth()->user()->can('useRootPermissions', PersonalAccessToken::class); $this->canUseWritePermissions = auth()->user()->can('useWritePermissions', PersonalAccessToken::class); $this->canUseDeployPermissions = auth()->user()->can('useDeployPermissions', PersonalAccessToken::class); diff --git a/app/Livewire/Server/Delete.php b/app/Livewire/Server/Delete.php index c3539aa282..2b5051c65e 100644 --- a/app/Livewire/Server/Delete.php +++ b/app/Livewire/Server/Delete.php @@ -5,6 +5,7 @@ namespace App\Livewire\Server; use App\Actions\Server\DeleteServer; use App\Jobs\DeleteResourceJob; use App\Models\Server; +use App\Services\ServerTransfer\ServerTransferClaimer; use Illuminate\Foundation\Auth\Access\AuthorizesRequests; use Livewire\Component; @@ -46,6 +47,16 @@ class Delete extends Component $this->delete_from_hostinger = in_array('delete_from_hostinger', $selectedActions); $this->force_delete_resources = in_array('force_delete_resources', $selectedActions); } + // Another Coolify instance manages a transferred server. This instance deletes only its own records: + // never the VM, and never the containers that the other instance runs. + $isArchive = $this->server->isTransferredAway(); + if ($isArchive) { + $this->delete_from_hetzner = false; + $this->delete_from_vultr = false; + $this->delete_from_digitalocean = false; + $this->delete_from_hostinger = false; + $this->force_delete_resources = true; + } try { $this->authorize('delete', $this->server); if ($this->server->is_coolify_host) { @@ -59,9 +70,12 @@ class Delete extends Component return; } + if ($isArchive) { + app(ServerTransferClaimer::class)->forgetDnsRecords($this->server); + } if ($this->force_delete_resources) { foreach ($this->server->definedResources() as $resource) { - DeleteResourceJob::dispatch($resource); + DeleteResourceJob::dispatch($resource, deleteFromCoolifyOnly: $isArchive); } } @@ -90,7 +104,7 @@ class Delete extends Component { $checkboxes = []; - if ($this->server->hasDefinedResources()) { + if ($this->server->hasDefinedResources() && ! $this->server->isTransferredAway()) { $resourceCount = $this->server->definedResources()->count(); $checkboxes[] = [ 'id' => 'force_delete_resources', @@ -99,7 +113,9 @@ class Delete extends Component ]; } - if ($this->server->hetzner_server_id) { + $canDeleteFromProvider = ! $this->server->isTransferredAway(); + + if ($canDeleteFromProvider && $this->server->hetzner_server_id) { $checkboxes[] = [ 'id' => 'delete_from_hetzner', 'label' => 'Also delete server from Hetzner Cloud', @@ -107,7 +123,7 @@ class Delete extends Component ]; } - if ($this->server->vultr_instance_id) { + if ($canDeleteFromProvider && $this->server->vultr_instance_id) { $checkboxes[] = [ 'id' => 'delete_from_vultr', 'label' => 'Also delete server from Vultr', @@ -115,7 +131,7 @@ class Delete extends Component ]; } - if ($this->server->digitalocean_droplet_id) { + if ($canDeleteFromProvider && $this->server->digitalocean_droplet_id) { $checkboxes[] = [ 'id' => 'delete_from_digitalocean', 'label' => 'Also delete droplet from DigitalOcean', @@ -123,7 +139,7 @@ class Delete extends Component ]; } - if ($this->server->hostinger_virtual_machine_id) { + if ($canDeleteFromProvider && $this->server->hostinger_virtual_machine_id) { $checkboxes[] = [ 'id' => 'delete_from_hostinger', 'label' => 'Also delete VPS from Hostinger (at the end of the paid period)', diff --git a/app/Livewire/Server/Show.php b/app/Livewire/Server/Show.php index 4cb59e1996..ff5751ed2a 100644 --- a/app/Livewire/Server/Show.php +++ b/app/Livewire/Server/Show.php @@ -395,43 +395,46 @@ class Show extends Component } } - public function toggleManagement(ServerTransferClaimer $claimer): void + /** + * Take back a server that was transferred to another instance. + */ + public function enableManagement(ServerTransferClaimer $claimer): void { - abort_unless(isDev(), 404); $this->authorize('update', $this->server); - if ($this->server->isLocalhost()) { - $this->dispatch('error', 'The Coolify host cannot be transferred.'); - + if ($this->server->isLocalhost() || ! $this->server->isTransferredAway()) { return; } - if ($this->server->isTransferredAway() && $this->server->team->serverOverflow()) { + if ($this->server->team->serverOverflow()) { $this->dispatch('error', 'Your team is over its server limit. Upgrade your subscription or remove a server first.'); return; } - if ($this->server->isTransferredAway()) { + try { $claimer->claim($this->server); - $event = 'ui.server.management_enabled'; - $message = 'This Coolify instance now manages the server.'; - } else { - $claimer->markTransferred($this->server, managementDisabled: true); - $event = 'ui.server.management_disabled'; - $message = 'Server automations are disabled on this Coolify instance.'; + } catch (\Throwable $e) { + handleError($e, $this); + + return; } + $claimer->restoreAfterTakeBack($this->server->refresh()); $this->server->refresh(); + // Sentinel must report here. While it reports to the other instance, that instance does not run + // the connection check that reads the owner marker and makes it stop managing the server. + $this->server->restartSentinel(); $this->syncData(); + $this->dispatch('transferFinishStarted')->to(TransferFinish::class); - auditLog($event, [ + auditLog('ui.server.management_enabled', [ 'team_id' => $this->server->team_id, 'server_uuid' => $this->server->uuid, 'server_name' => $this->server->name, ]); - $this->dispatch('success', $message); + $this->dispatch('success', 'This Coolify instance now manages the server.'); } public function checkLocalhostConnection() diff --git a/app/Livewire/Server/Transfer.php b/app/Livewire/Server/Transfer.php index 8ae349ec3f..22bb6d0685 100644 --- a/app/Livewire/Server/Transfer.php +++ b/app/Livewire/Server/Transfer.php @@ -3,8 +3,6 @@ namespace App\Livewire\Server; use App\Models\Server; -use App\Services\ServerTransfer\ServerTransferBundle; -use App\Services\ServerTransfer\ServerTransferExporter; use App\Services\ServerTransfer\ServerTransferMigrator; use Illuminate\Foundation\Auth\Access\AuthorizesRequests; use Livewire\Component; @@ -21,25 +19,27 @@ class Transfer extends Component public string $targetToken = ''; - /** Manual transfer */ - /** Encrypts the downloaded file when filled; empty or whitespace means no encryption. */ - public ?string $passphrase = null; - public ?string $exportId = null; + /** Transfer back fields, shown for an archive */ + public string $managerUrl = ''; + + public string $managerToken = ''; + + public string $thisInstanceUrl = ''; + /** @var list */ public array $lastWarnings = []; - public ?string $lastResultJson = null; - public function mount(string $server_uuid): void { - $this->ensureDevelopmentAvailability(); - try { $this->server = Server::ownedByCurrentTeam()->whereUuid($server_uuid)->firstOrFail(); - $this->authorize('view', $this->server); + // The transfer exports every credential of the server, so only admins and owners may open it. + $this->authorize('update', $this->server); $this->exportId = data_get($this->server->server_metadata, 'transfer.export_id'); + $this->managerUrl = (string) data_get($this->server->server_metadata, 'transfer.target_instance_url', ''); + $this->thisInstanceUrl = rtrim((string) (instanceSettings()->fqdn ?: config('app.url')), '/'); } catch (Throwable $e) { handleError($e, $this); $this->redirect(route('server.index'), navigate: true); @@ -51,72 +51,86 @@ class Transfer extends Component return (int) $this->server->id === 0; } + /** + * State of the queued transfer to another instance, read by the page poll. + * + * @return array|null + */ + public function getMigrationProperty(): ?array + { + $migration = data_get($this->server->server_metadata, 'migration'); + + return is_array($migration) ? $migration : null; + } + + /** + * Manual steps of a finished transfer that the target instance cannot do. + * + * @return list + */ + public function getPostTransferChecklistProperty(): array + { + if (data_get($this->migration, 'status') !== 'succeeded') { + return []; + } + + return array_values(array_filter((array) data_get($this->migration, 'checklist', []), 'is_string')); + } + + public function refreshMigration(): void + { + $this->server->refresh(); + $this->lastWarnings = array_values((array) data_get($this->migration, 'warnings', $this->lastWarnings)); + } + public function migrateServer(ServerTransferMigrator $migrator): void { - $this->ensureDevelopmentAvailability(); - try { $this->authorize('update', $this->server); if ($this->isLocalhost) { throw new \RuntimeException('The Coolify host (localhost) cannot be transferred.'); } - $result = $migrator->migrate( + $migrator->start( server: $this->server, targetUrl: $this->targetUrl, targetToken: $this->targetToken, ); + auditLog('ui.server.migrate', [ + 'team_id' => $this->server->team_id, + 'server_uuid' => $this->server->uuid, + 'target_url' => $this->targetUrl, + ]); + $this->server->refresh(); - $this->exportId = $result['export_id'] ?? $this->exportId; - $this->lastWarnings = array_values((array) data_get($result, 'warnings', [])); - // Never echo the target token in the result dump. - $safe = $result; - unset($safe['target_token']); - $this->lastResultJson = json_encode($safe, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES); + $this->lastWarnings = []; $this->targetToken = ''; - $this->dispatch('success', $result['message'] ?? 'Server transferred.'); + $this->dispatch('success', 'All checks passed. The transfer started.'); } catch (Throwable $e) { + // A failed check is saved in the transfer log, so show it. + $this->server->refresh(); handleError($e, $this); } } - public function exportBundle(ServerTransferExporter $exporter) + public function transferBack(ServerTransferMigrator $migrator): void { - $this->ensureDevelopmentAvailability(); - try { - $this->authorize('view', $this->server); - if ($this->isLocalhost) { - throw new \RuntimeException('The Coolify host (localhost) cannot be transferred.'); - } + $this->authorize('update', $this->server); + $migrator->requestTransferBack($this->server, auth()->user(), $this->managerUrl, $this->managerToken, $this->thisInstanceUrl); - $bundle = $exporter->export($this->server, includeSensitive: true); - $this->exportId = data_get($bundle, 'export_id'); - $this->lastWarnings = array_values((array) data_get($bundle, 'warnings', [])); - $this->lastResultJson = null; - - $payload = $bundle; - $fileName = 'server-transfer-'.$this->server->uuid.'.json'; - if (filled($this->passphrase)) { - $payload = ServerTransferBundle::encryptWithPassphrase($bundle, $this->passphrase); - $fileName = 'server-transfer-'.$this->server->uuid.'.encrypted.json'; - } - - $json = json_encode($payload, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES); - if ($json === false) { - throw new \RuntimeException('Failed to encode transfer bundle.'); - } - - $this->dispatch('success', 'Transfer bundle ready for download.'); - - return response()->streamDownload(function () use ($json) { - echo $json; - }, $fileName, [ - 'Content-Type' => 'application/json', + auditLog('ui.server.transfer_back', [ + 'team_id' => $this->server->team_id, + 'server_uuid' => $this->server->uuid, + 'from' => $this->managerUrl, ]); + + $this->server->refresh(); + $this->managerToken = ''; + $this->dispatch('success', 'The other instance started the transfer back.'); } catch (Throwable $e) { - return handleError($e, $this); + handleError($e, $this); } } @@ -124,9 +138,4 @@ class Transfer extends Component { return view('livewire.server.transfer'); } - - private function ensureDevelopmentAvailability(): void - { - abort_unless(isDev(), 404); - } } diff --git a/app/Livewire/Server/TransferFinish.php b/app/Livewire/Server/TransferFinish.php new file mode 100644 index 0000000000..e60664ed0c --- /dev/null +++ b/app/Livewire/Server/TransferFinish.php @@ -0,0 +1,122 @@ +server->team_id},TransferFinishChanged" => 'handleTransferFinishChanged', + ]; + } + + /** + * The server page sends this after a take back, which starts a new list. + */ + #[On('transferFinishStarted')] + public function reloadServer(): void + { + $this->server->refresh(); + } + + /** + * A background check, or a user in another tab, changed a step of this list. + */ + public function handleTransferFinishChanged(array $event): void + { + if (($event['serverUuid'] ?? null) === $this->server->uuid) { + $this->server->refresh(); + } + } + + /** + * Start the background checks when they are stale. Called when the panel loads and by its poll. + */ + public function refreshSteps(TransferFinishChecklist $checklist): void + { + $this->server->refresh(); + if (auth()->user()?->can('update', $this->server)) { + $checklist->checkWhenStale($this->server); + } + } + + public function checkAllNow(): void + { + $this->authorize('update', $this->server); + CheckTransferFinishStepsJob::dispatch($this->server); + } + + public function checkStep(TransferFinishChecklist $checklist, string $key): void + { + $this->authorize('update', $this->server); + $checklist->check($this->server, $key); + } + + public function fixStep(TransferFinishChecklist $checklist, string $key): void + { + $this->authorize('update', $this->server); + $item = $checklist->fix($this->server, $key); + $toast = match ($item['status']) { + 'ok' => 'success', + 'pending' => 'info', + default => 'error', + }; + $this->dispatch($toast, $item['message'] ?? 'Done.'); + } + + public function markStepDone(TransferFinishChecklist $checklist, string $key): void + { + $this->authorize('update', $this->server); + $checklist->markDone($this->server, $key); + } + + public function finishTransfer(TransferFinishChecklist $checklist): void + { + $this->authorize('update', $this->server); + $skipped = count($checklist->openItems($this->server)); + $checklist->finish($this->server, auth()->user()); + + auditLog('ui.server.transfer_finished', [ + 'team_id' => $this->server->team_id, + 'server_uuid' => $this->server->uuid, + 'skipped_steps' => $skipped, + ]); + + $this->dispatch('success', 'The transfer is finished.'); + } + + public function render() + { + $checklist = app(TransferFinishChecklist::class); + $finish = $checklist->isOpen($this->server) ? $checklist->of($this->server) : null; + $items = collect($finish['items'] ?? []); + $openItems = $items->filter(fn (array $item) => in_array($item['status'], TransferFinishChecklist::OPEN_STATUSES, true)); + + return view('livewire.server.transfer-finish', [ + 'checklist' => $checklist, + 'finish' => $finish, + 'openGroups' => collect(TransferFinishChecklist::GROUPS) + ->map(fn (string $label, string $group) => ['label' => $label, 'items' => $openItems->filter(fn (array $item) => $checklist->groupOf($item) === $group)->values()]) + ->filter(fn (array $group) => $group['items']->isNotEmpty()), + 'doneItems' => $items->reject(fn (array $item) => in_array($item['status'], TransferFinishChecklist::OPEN_STATUSES, true))->values(), + 'openCount' => $openItems->count(), + 'totalCount' => $items->count(), + ]); + } +} diff --git a/app/Livewire/Server/TransferImport.php b/app/Livewire/Server/TransferImport.php deleted file mode 100644 index 97bb6c5c7d..0000000000 --- a/app/Livewire/Server/TransferImport.php +++ /dev/null @@ -1,150 +0,0 @@ -|null */ - public ?array $lastResult = null; - - /** @var list */ - public array $lastWarnings = []; - - public ?string $importedServerUuid = null; - - public function mount(): void - { - $this->ensureDevelopmentAvailability(); - $this->authorize('create', Server::class); - } - - public function updatedBundleFile(): void - { - $this->ensureDevelopmentAvailability(); - - if (! $this->bundleFile) { - return; - } - try { - $contents = $this->bundleFile->get(); - if (! is_string($contents) || blank($contents)) { - throw new \RuntimeException('Uploaded file is empty.'); - } - $this->bundleJson = $contents; - $this->dispatch('success', 'Bundle file loaded into the form.'); - } catch (Throwable $e) { - handleError($e, $this); - } - } - - /** - * @return array - */ - private function resolveBundle(): array - { - $raw = trim($this->bundleJson); - if ($raw === '') { - throw new \RuntimeException('Paste a transfer bundle JSON or upload a file.'); - } - - $decoded = json_decode($raw, true); - if (! is_array($decoded)) { - throw new \RuntimeException('Bundle is not valid JSON.'); - } - - if (data_get($decoded, 'encrypted')) { - if (blank($this->passphrase)) { - throw new \RuntimeException('Passphrase is required for encrypted bundles.'); - } - - return ServerTransferBundle::decryptWithPassphrase($decoded, $this->passphrase); - } - - return $decoded; - } - - public function dryRun(ServerTransferImporter $importer): void - { - $this->ensureDevelopmentAvailability(); - $this->runImport($importer, dryRun: true); - } - - public function importBundle(ServerTransferImporter $importer): void - { - $this->ensureDevelopmentAvailability(); - $this->runImport($importer, dryRun: false); - } - - private function runImport(ServerTransferImporter $importer, bool $dryRun): void - { - try { - $this->authorize('create', Server::class); - $teamId = currentTeam()->id; - $bundle = $this->resolveBundle(); - - $result = $importer->import( - bundle: $bundle, - teamId: $teamId, - dryRun: $dryRun, - preserveUuids: $this->preserveUuids, - adoptMode: $this->adoptMode, - ); - - $this->lastResult = $result; - $this->lastWarnings = array_values((array) data_get($result, 'warnings', [])); - $this->importedServerUuid = $dryRun ? null : data_get($result, 'server_uuid'); - - if (! $dryRun) { - auditLog('ui.server.imported', [ - 'team_id' => $teamId, - 'server_uuid' => $this->importedServerUuid, - 'claimed' => (bool) data_get($result, 'claimed'), - 'adopt_mode' => $this->adoptMode, - ]); - } - - if ($dryRun) { - $this->dispatch('success', 'Dry run completed — nothing was written.'); - } elseif (data_get($result, 'claimed')) { - $this->dispatch('success', 'Server imported. This instance now manages it.'); - } else { - $this->dispatch('success', 'Server imported, but management could not be enabled. Check the warnings, then click Enable management on the server page.'); - } - } catch (Throwable $e) { - handleError($e, $this); - } - } - - public function render() - { - return view('livewire.server.transfer-import'); - } - - private function ensureDevelopmentAvailability(): void - { - abort_unless(isDev(), 404); - } -} diff --git a/app/Livewire/Settings/Advanced.php b/app/Livewire/Settings/Advanced.php index 28cbc971d2..86060ddd3f 100644 --- a/app/Livewire/Settings/Advanced.php +++ b/app/Livewire/Settings/Advanced.php @@ -6,6 +6,7 @@ use App\Models\InstanceSettings; use App\Models\S3Storage; use App\Rules\ValidDnsServers; use App\Rules\ValidIpOrCidr; +use App\Services\FileDownloadStorageService; use Illuminate\Foundation\Auth\Access\AuthorizesRequests; use Livewire\Attributes\Validate; use Livewire\Component; @@ -63,6 +64,8 @@ class Advanced extends Component public array $avatar_storage_options = []; + public string $file_download_storage = 'stream'; + public function rules() { return [ @@ -123,6 +126,9 @@ class Advanced extends Component 'label' => $storage->name.' (S3)', ])->all(), ]; + $this->file_download_storage = $this->settings->file_download_s3_storage_id + ? 's3:'.$this->settings->file_download_s3_storage_id + : 'stream'; } public function submit() @@ -228,6 +234,7 @@ class Advanced extends Component $this->settings->webhook_allow_localhost = $this->webhook_allow_localhost; $this->settings->image_cdn_url = filled($this->image_cdn_url) ? rtrim($this->image_cdn_url, '/') : null; $this->saveAvatarStorageSetting(); + $this->saveFileDownloadStorageSetting(); $this->saveAndAudit(); $this->dispatch('success', 'Settings updated!'); } catch (\Exception $e) { @@ -258,6 +265,48 @@ class Advanced extends Component $this->settings->avatar_s3_storage_id = $storage->id; } + /** + * Selecting a storage adds the auto-delete lifecycle rule to its bucket. + * The setting is refused when the bucket is listable by anonymous users + * or does not accept the rule. + */ + private function saveFileDownloadStorageSetting(): void + { + if ($this->file_download_storage === 'stream') { + $this->settings->file_download_s3_storage_id = null; + + return; + } + + $storageId = (int) str($this->file_download_storage)->after('s3:')->value(); + if ($storageId === $this->settings->file_download_s3_storage_id) { + return; + } + + $storage = S3Storage::query() + ->whereTeamId(0) + ->where('is_usable', true) + ->find($storageId); + + if (! $storage || $this->file_download_storage !== 's3:'.$storage->id) { + $this->file_download_storage = 'stream'; + throw new \InvalidArgumentException('The selected file download storage is not available.'); + } + + try { + $downloads = app(FileDownloadStorageService::class); + $downloads->ensureBucketIsPrivate($storage); + $downloads->ensureLifecycleRule($storage); + } catch (\RuntimeException $e) { + $this->file_download_storage = $this->settings->file_download_s3_storage_id + ? 's3:'.$this->settings->file_download_s3_storage_id + : 'stream'; + throw $e; + } + + $this->settings->file_download_s3_storage_id = $storage->id; + } + public function clearDomainConnectPrivateKey(): void { try { diff --git a/app/Livewire/Settings/Index.php b/app/Livewire/Settings/Index.php index 75826f2d4e..cd047b1f43 100644 --- a/app/Livewire/Settings/Index.php +++ b/app/Livewire/Settings/Index.php @@ -201,6 +201,31 @@ class Index extends Component } } + /** + * Look up the public IPs of this host again and save them. + */ + public function detectPublicIps() + { + try { + $this->authorize('update', $this->settings); + + $detected = detect_public_ips(); + if ($detected === []) { + $this->dispatch('error', 'Could not detect a public IP address.'); + + return; + } + + $this->settings->fill($detected); + $this->saveAndAudit(); + $this->public_ipv4 = $this->settings->public_ipv4; + $this->public_ipv6 = $this->settings->public_ipv6; + $this->dispatch('success', 'Detected '.implode(' and ', $detected).'.'); + } catch (\Throwable $e) { + return handleError($e, $this); + } + } + public function buildHelperImage() { try { diff --git a/app/Livewire/Settings/Updates.php b/app/Livewire/Settings/Updates.php index 10b3180640..aba7521dd9 100644 --- a/app/Livewire/Settings/Updates.php +++ b/app/Livewire/Settings/Updates.php @@ -130,7 +130,7 @@ class Updates extends Component if ($this->is_auto_update_enabled && ! validate_cron_expression($this->auto_update_frequency)) { $this->dispatch('error', 'Invalid Cron / Human expression for Auto Update Frequency.'); if (empty($this->auto_update_frequency)) { - $this->auto_update_frequency = '0 0 * * *'; + $this->auto_update_frequency = InstanceSettings::randomAutoUpdateFrequency(); } return; diff --git a/app/Livewire/SettingsBackup.php b/app/Livewire/SettingsBackup.php index 0f92b2ad34..d7eaf7d596 100644 --- a/app/Livewire/SettingsBackup.php +++ b/app/Livewire/SettingsBackup.php @@ -31,21 +31,9 @@ class SettingsBackup extends Component #[Locked] public $executions = []; - #[Validate(['required'])] - public string $uuid; - - #[Validate(['required'])] - public string $name; - #[Validate(['nullable'])] public ?string $description = null; - #[Validate(['required'])] - public string $postgres_user; - - #[Validate(['required'])] - public string $postgres_password; - public bool $is_backup_before_update_enabled; public function mount() @@ -59,11 +47,7 @@ class SettingsBackup extends Component $this->database = StandalonePostgresql::find(0); $s3s = S3Storage::whereTeamId(0)->get() ?? []; if ($this->database) { - $this->uuid = $this->database->uuid; - $this->name = $this->database->name; $this->description = $this->database->description; - $this->postgres_user = $this->database->postgres_user; - $this->postgres_password = $this->database->postgres_password; if ($this->database->status !== 'running') { $this->database->status = 'running'; @@ -127,11 +111,7 @@ class SettingsBackup extends Component } $this->s3s = S3Storage::whereTeamId(0)->get(); - $this->uuid = $this->database->uuid; - $this->name = $this->database->name; $this->description = $this->database->description; - $this->postgres_user = $this->database->postgres_user; - $this->postgres_password = $this->database->postgres_password; $this->executions = $this->backup->executions; } catch (\Exception $e) { @@ -166,11 +146,9 @@ class SettingsBackup extends Component $this->authorize('update', $this->settings); $this->validate(); + // Credentials come from the coolify-db container and are display-only, so only the description is saved. $this->database->update([ - 'name' => $this->name, 'description' => $this->description, - 'postgres_user' => $this->postgres_user, - 'postgres_password' => $this->postgres_password, ]); $this->dispatch('success', 'Backup updated.'); } catch (\Throwable $e) { diff --git a/app/Livewire/SharedVariables/Index.php b/app/Livewire/SharedVariables/Index.php index 61c31e9f83..fe8ccc1802 100644 --- a/app/Livewire/SharedVariables/Index.php +++ b/app/Livewire/SharedVariables/Index.php @@ -2,12 +2,17 @@ namespace App\Livewire\SharedVariables; +use App\Services\SharedVariableCanvas; +use Illuminate\View\View; use Livewire\Component; class Index extends Component { - public function render() + public function render(): View { - return view('livewire.shared-variables.index'); + // A "my team" page: the overview of the session team's shared variables (keys and counts only). + return view('livewire.shared-variables.index', [ + 'overview' => app(SharedVariableCanvas::class)->overview(currentTeam()), + ]); } } diff --git a/app/Livewire/Source/Github/Change.php b/app/Livewire/Source/Github/Change.php index 7e0753ad69..ca59577ccb 100644 --- a/app/Livewire/Source/Github/Change.php +++ b/app/Livewire/Source/Github/Change.php @@ -372,14 +372,7 @@ class Change extends Component public function getGithubAppNameUpdatePath() { - $name = encodeGithubPathSegment($this->github_app->name); - $organization = normalizeGithubOrganization($this->github_app->organization); - - if (filled($organization)) { - return rtrim($this->github_app->html_url, '/').'/organizations/'.encodeGithubPathSegment($organization)."/settings/apps/{$name}"; - } - - return rtrim($this->github_app->html_url, '/')."/settings/apps/{$name}"; + return $this->github_app->settingsUrl(); } public function updateGithubAppName() diff --git a/app/Livewire/Tags/Show.php b/app/Livewire/Tags/Show.php index e2f4980d75..01450abe9a 100644 --- a/app/Livewire/Tags/Show.php +++ b/app/Livewire/Tags/Show.php @@ -7,6 +7,7 @@ use App\Livewire\Concerns\CreatesTags; use App\Livewire\Concerns\RemembersCanvasView; use App\Models\ApplicationDeploymentQueue; use App\Models\Tag; +use App\Services\TagCanvas; use Illuminate\Foundation\Auth\Access\AuthorizesRequests; use Illuminate\Support\Collection; use Livewire\Attributes\Locked; @@ -41,6 +42,9 @@ class Show extends Component #[Locked] public ?array $deploymentsPerTagPerServer = null; + /** The "Connect to resources" modal on small screens, where the tags canvas is not shown. */ + public bool $connectResourcesOpen = false; + public function mount() { if (str($this->tagName)->isEmpty() && $this->prefersCanvasView('tags')) { @@ -123,9 +127,39 @@ class Show extends Component } } + public function openConnectResources(): void + { + $this->connectResourcesOpen = true; + } + /** - * Creates a tag without resources. Tag resources on the tags canvas or on the resource. + * Adds the tag to a resource of the tag's team, or removes it when the resource already has it. */ + public function toggleResource(string $uuid): void + { + try { + $resource = app(TagCanvas::class)->resources($this->tag->team_id)->get($uuid); + if (! $resource) { + $this->dispatch('error', 'Resource not found.'); + + return; + } + $this->authorize('update', $resource); + + if ($resource->tags->contains('id', $this->tag->id)) { + $resource->tags()->detach($this->tag->id); + } else { + $resource->tags()->attach($this->tag->id); + } + + $this->applications = $this->tag->applications()->get(); + $this->services = $this->tag->services()->get(); + $this->getDeployments(); + } catch (\Exception $e) { + handleError($e, $this); + } + } + /** * Deletes the tag and removes it from every resource. */ @@ -158,6 +192,30 @@ class Show extends Component 'resourceCount' => $applicationsCount + $servicesCount, ]; })->values()->toArray(), + 'connectableResources' => $this->connectableResources(), ]); } + + /** + * Resources of the tag's team for the "Connect to resources" modal, by "Project / Environment". + * Loaded only while the modal is open. + * + * @return Collection> + */ + private function connectableResources(): Collection + { + if (! $this->connectResourcesOpen || ! $this->tag) { + return collect(); + } + + return app(TagCanvas::class)->resources($this->tag->team_id) + ->groupBy(fn ($resource) => $resource->environment->project->name.' / '.$resource->environment->name) + ->sortKeys(SORT_NATURAL | SORT_FLAG_CASE) + ->map(fn (Collection $resources) => $resources->map(fn ($resource) => [ + 'uuid' => $resource->uuid, + 'name' => $resource->name, + 'tagged' => $resource->tags->contains('id', $this->tag->id), + 'can_update' => auth()->user()->can('update', $resource), + ])->values()); + } } diff --git a/app/Models/Application.php b/app/Models/Application.php index fc9085752c..a2dc1d6f62 100644 --- a/app/Models/Application.php +++ b/app/Models/Application.php @@ -790,7 +790,7 @@ class Application extends BaseModel return "{$this->source->html_url}/{$this->git_repository}/tree/{$this->git_branch}{$base_dir}"; } - $httpsRepository = $this->httpsUrlFromScpStyleGitRepository(); + $httpsRepository = $this->httpsUrlFromSshGitRepository(); if (is_string($httpsRepository)) { if (str($this->git_repository)->contains('bitbucket')) { return "{$httpsRepository}/src/{$this->git_branch}{$base_dir}"; @@ -811,7 +811,7 @@ class Application extends BaseModel if (! is_null($this->source?->html_url) && ! is_null($this->git_repository) && ! is_null($this->git_branch)) { return "{$this->source->html_url}/{$this->git_repository}/settings/hooks"; } - $httpsRepository = $this->httpsUrlFromScpStyleGitRepository(); + $httpsRepository = $this->httpsUrlFromSshGitRepository(); if (is_string($httpsRepository)) { return "{$httpsRepository}/settings/hooks"; } @@ -828,7 +828,7 @@ class Application extends BaseModel if (! is_null($this->source?->html_url) && ! is_null($this->git_repository) && ! is_null($this->git_branch)) { return "{$this->source->html_url}/{$this->git_repository}/commits/{$this->git_branch}"; } - $httpsRepository = $this->httpsUrlFromScpStyleGitRepository(); + $httpsRepository = $this->httpsUrlFromSshGitRepository(); if (is_string($httpsRepository)) { return "{$httpsRepository}/commits/{$this->git_branch}"; } @@ -849,11 +849,9 @@ class Application extends BaseModel } $git_repository = $this->git_repository; - $httpsRepository = scpStyleGitUrlToHttps($git_repository); + $httpsRepository = sshGitUrlToHttps($git_repository); if (is_string($httpsRepository)) { $git_repository = $httpsRepository; - } elseif (str($this->git_repository)->startsWith('ssh://')) { - $git_repository = 'https://'.parse_url($git_repository, PHP_URL_HOST).parse_url($git_repository, PHP_URL_PATH); } if (! filter_var($git_repository, FILTER_VALIDATE_URL) || ! in_array(parse_url($git_repository, PHP_URL_SCHEME), ['http', 'https'], true)) { @@ -868,9 +866,9 @@ class Application extends BaseModel return $url->__toString(); } - private function httpsUrlFromScpStyleGitRepository(): ?string + private function httpsUrlFromSshGitRepository(): ?string { - $httpsRepository = scpStyleGitUrlToHttps($this->git_repository); + $httpsRepository = sshGitUrlToHttps($this->git_repository); if (! is_string($httpsRepository)) { return null; @@ -1032,11 +1030,11 @@ class Application extends BaseModel $additional_servers_status = $this->additional_servers->pluck('pivot.status'); foreach ($additional_servers_status as $status) { $server_status = str($status)->before(':')->value(); - $server_health = str($status)->after(':')->value() ?? 'unhealthy'; + $server_health = str($status)->contains(':') ? str($status)->after(':')->value() : null; if ($main_server_status !== $server_status) { $complex_status = 'degraded'; } - if ($main_server_health !== $server_health) { + if ($server_health !== null && $main_server_health !== $server_health) { $complex_health = 'unhealthy'; } } @@ -2193,7 +2191,7 @@ class Application extends BaseModel continue; } if ($source->startsWith('.')) { - $source = str($workdir.$source->after('.')); + $source = replaceLocalSource($source, str($workdir)); } $mkdirCommand = rawComposeBindMkdirCommand($source->value()); if ($mkdirCommand !== null) { diff --git a/app/Models/ApplicationPreview.php b/app/Models/ApplicationPreview.php index 4a0f6d78f0..17b791d733 100644 --- a/app/Models/ApplicationPreview.php +++ b/app/Models/ApplicationPreview.php @@ -54,10 +54,13 @@ class ApplicationPreview extends BaseModel if (data_get($preview, 'application.build_pack') === 'dockercompose') { $composeFile = $application->parse(pull_request_id: $preview->pull_request_id); - foreach ($preview->generatedComposeVolumeNames($application, collect(data_get($composeFile, 'volumes', []))) as $volumeName) { + $volumeNames = $preview->generatedComposeVolumeNames($application, collect(data_get($composeFile, 'volumes', []))); + foreach ($volumeNames as $volumeName) { // Docker does not remove a volume that a container still uses, also with -f. instant_remote_process(['docker volume rm -f '.escapeshellarg($volumeName)], $server, false); } + // The compose parser stores preview volumes on the application, not on the preview + $application->persistentStorages()->whereIn('name', $volumeNames)->delete(); $preview->removeComposePreviewNetwork($application, $server); } else { // Regular application volume cleanup diff --git a/app/Models/ApplicationSetting.php b/app/Models/ApplicationSetting.php index 41249a5201..abe317530f 100644 --- a/app/Models/ApplicationSetting.php +++ b/app/Models/ApplicationSetting.php @@ -36,6 +36,7 @@ use OpenApi\Attributes as OA; 'custom_container_name_prefix' => ['type' => 'string', 'nullable' => true], 'is_container_label_escape_enabled' => ['type' => 'boolean'], 'is_env_sorting_enabled' => ['type' => 'boolean'], + 'include_coolify_default_variables' => ['type' => 'boolean'], 'is_container_label_readonly_enabled' => ['type' => 'boolean'], 'is_preserve_repository_enabled' => ['type' => 'boolean'], 'disable_build_cache' => ['type' => 'boolean'], @@ -88,6 +89,7 @@ class ApplicationSetting extends Model 'is_stripprefix_enabled' => 'boolean', 'connect_to_docker_network' => 'boolean', 'is_env_sorting_enabled' => 'boolean', + 'include_coolify_default_variables' => 'boolean', 'redact_all_env_values_in_logs' => 'boolean', 'disable_build_cache' => 'boolean', ]; @@ -120,6 +122,7 @@ class ApplicationSetting extends Model 'custom_container_name_prefix', 'is_container_label_escape_enabled', 'is_env_sorting_enabled', + 'include_coolify_default_variables', 'is_container_label_readonly_enabled', 'is_preserve_repository_enabled', 'disable_build_cache', diff --git a/app/Models/GithubApp.php b/app/Models/GithubApp.php index 0052232fb1..49e32e0b85 100644 --- a/app/Models/GithubApp.php +++ b/app/Models/GithubApp.php @@ -57,6 +57,34 @@ class GithubApp extends BaseModel 'webhook_secret', ]; + /** + * Installation token kept in memory, so one job does not request a new token for each GitHub call. + * + * @var array{credentials: string, token: string, expires_at: int}|null + */ + private ?array $installationTokenInMemory = null; + + public function installationToken(): string + { + $credentials = implode('|', [$this->api_url, $this->app_id, $this->installation_id, $this->private_key_id]); + $cached = $this->installationTokenInMemory; + + if ($cached && $cached['credentials'] === $credentials && $cached['expires_at'] > now()->getTimestamp()) { + return $cached['token']; + } + + $token = generateGithubToken($this, 'installation'); + + // GitHub installation tokens expire after one hour; renew them well before that. + $this->installationTokenInMemory = [ + 'credentials' => $credentials, + 'token' => $token, + 'expires_at' => now()->addMinutes(50)->getTimestamp(), + ]; + + return $token; + } + protected static function booted(): void { static::deleting(function (GithubApp $github_app) { @@ -165,4 +193,19 @@ class GithubApp extends BaseModel return filled($this->app_id) && filled($this->installation_id); } + + /** + * The settings page of this App at GitHub. + */ + public function settingsUrl(): string + { + $name = encodeGithubPathSegment($this->name); + $organization = normalizeGithubOrganization($this->organization); + + if (filled($organization)) { + return rtrim($this->html_url, '/').'/organizations/'.encodeGithubPathSegment($organization)."/settings/apps/{$name}"; + } + + return rtrim($this->html_url, '/')."/settings/apps/{$name}"; + } } diff --git a/app/Models/InstanceSettings.php b/app/Models/InstanceSettings.php index 1d08078176..8726a9f0f6 100644 --- a/app/Models/InstanceSettings.php +++ b/app/Models/InstanceSettings.php @@ -59,6 +59,7 @@ class InstanceSettings extends Model 'avatar_storage_type', 'avatar_s3_storage_id', 'image_cdn_url', + 'file_download_s3_storage_id', 'is_dashboard_force_https_enabled', 'is_passkey_enabled', ]; @@ -108,6 +109,12 @@ class InstanceSettings extends Model protected static function booted(): void { + static::creating(function ($settings) { + if (blank($settings->getAttributes()['auto_update_frequency'] ?? null)) { + $settings->auto_update_frequency = self::randomAutoUpdateFrequency(); + } + }); + static::created(function () { Once::flush(); }); @@ -125,6 +132,14 @@ class InstanceSettings extends Model }); } + /** + * Daily auto update at a random minute after midnight, so instances do not all pull images from the registry in the same second. + */ + public static function randomAutoUpdateFrequency(): string + { + return random_int(0, 59).' 0 * * *'; + } + public function isPasswordRegistrationAllowed(): bool { if (! $this->is_registration_enabled) { diff --git a/app/Models/Server.php b/app/Models/Server.php index fa4c454885..da29b7ac02 100644 --- a/app/Models/Server.php +++ b/app/Models/Server.php @@ -11,6 +11,7 @@ use App\Enums\ApplicationDeploymentStatus; use App\Enums\ProxyTypes; use App\Enums\ServerRole; use App\Events\ServerReachabilityChanged; +use App\Helpers\SshMultiplexingHelper; use App\Helpers\SslHelper; use App\Jobs\CheckAndStartSentinelJob; use App\Jobs\CheckTraefikVersionForServerJob; @@ -189,6 +190,9 @@ class Server extends BaseModel static::saved(function ($server) { if ($server->wasChanged('private_key_id') || $server->privateKey?->isDirty()) { refresh_server_connection($server->privateKey); + } elseif ($server->wasChanged(['user', 'ip', 'port'])) { + // An open multiplexed connection keeps the old user and host until it is closed. + SshMultiplexingHelper::removeMuxFile($server); } }); static::created(function ($server) { @@ -252,7 +256,13 @@ class Server extends BaseModel $destination->delete(); }); // Leftover active runner rows would block deleting the GitHub App. - GithubRunnerExecution::deleteAndDeregister(GithubRunnerExecution::query()->where('server_id', $server->id)); + // The runners of a transferred server now belong to the other instance, so keep them registered at GitHub. + $executions = GithubRunnerExecution::query()->where('server_id', $server->id); + if ($server->isTransferredAway()) { + $executions->delete(); + } else { + GithubRunnerExecution::deleteAndDeregister($executions); + } $server->githubRunnerConfig()->delete(); $server->settings()->delete(); $server->sslCertificates()->delete(); @@ -1376,15 +1386,6 @@ $siteAddress { return data_get($this->server_metadata, 'transfer.status') === 'transferred'; } - /** - * Management was disabled manually on this instance; the server is ready to be transferred. - */ - public function isManagementDisabled(): bool - { - return $this->isTransferredAway() - && (bool) data_get($this->server_metadata, 'transfer.management_disabled', false); - } - /** * Whether this server may be validated / installed against from this instance. */ @@ -2071,25 +2072,34 @@ $siteAddress { public function validateOS(): bool|Stringable { $os_release = instant_remote_process(['cat /etc/os-release'], $this); - $releaseLines = collect(explode("\n", $os_release)); - $collectedData = collect([]); - foreach ($releaseLines as $line) { - $item = str($line)->trim(); - $collectedData->put($item->before('=')->value(), $item->after('=')->lower()->replace('"', '')->value()); - } - $ID = data_get($collectedData, 'ID'); - // $ID_LIKE = data_get($collectedData, 'ID_LIKE'); - // $VERSION_ID = data_get($collectedData, 'VERSION_ID'); - $supported = collect(SUPPORTED_OS)->filter(function ($supportedOs) use ($ID) { - if (str($supportedOs)->contains($ID)) { - return str($ID); + + return self::supportedOsFromOsRelease($os_release); + } + + /** + * Match /etc/os-release against SUPPORTED_OS: ID first, then each ID_LIKE token (e.g. Manjaro: ID=manjaro, ID_LIKE=arch). + */ + public static function supportedOsFromOsRelease(?string $osRelease): bool|Stringable + { + $collectedData = collect(explode("\n", (string) $osRelease)) + ->mapWithKeys(function ($line) { + $item = str($line)->trim(); + + return [$item->before('=')->value() => $item->after('=')->lower()->replace(['"', "'"], '')->value()]; + }); + + $candidates = collect([data_get($collectedData, 'ID')]) + ->merge(explode(' ', (string) data_get($collectedData, 'ID_LIKE'))) + ->filter(); + + foreach ($candidates as $candidate) { + $supported = collect(SUPPORTED_OS)->filter(fn ($supportedOs) => str($supportedOs)->contains($candidate)); + if ($supported->count() === 1) { + return str($supported->first()); } - }); - if ($supported->count() === 1) { - return str($supported->first()); - } else { - return false; } + + return false; } public function gatherServerMetadata(): ?array @@ -2128,7 +2138,13 @@ $siteAddress { 'collected_at' => now()->toIso8601String(), ]; - $this->update(['server_metadata' => $metadata]); + // Merge into the saved row: server_metadata also holds the transfer and migration state. + DB::transaction(function () use ($metadata) { + $locked = self::query()->lockForUpdate()->findOrFail($this->id); + $locked->update(['server_metadata' => array_merge($locked->server_metadata ?? [], $metadata)]); + $this->server_metadata = $locked->server_metadata; + $this->syncOriginalAttribute('server_metadata'); + }); try { $detectedDockerVersion = parseDockerEngineVersion($sections['DOCKER'] ?? null); diff --git a/app/Models/ServerSetting.php b/app/Models/ServerSetting.php index 706d90c517..c1dfc5bf2b 100644 --- a/app/Models/ServerSetting.php +++ b/app/Models/ServerSetting.php @@ -331,7 +331,9 @@ class ServerSetting extends Model { $url = $this->sentinel_custom_url; - if ($this->server->isLocalhost() && $url === 'http://host.docker.internal:8000') { + // The local Sentinel runs on the coolify network. A host gateway URL leaves that network + // through the host INPUT chain, where host firewalls such as UFW drop it. + if ($this->server->isLocalhost() && strtolower((string) parse_url((string) $url, PHP_URL_HOST)) === 'host.docker.internal') { $url = null; } diff --git a/app/Models/Service.php b/app/Models/Service.php index 1f413491bd..2ebff2ad9c 100644 --- a/app/Models/Service.php +++ b/app/Models/Service.php @@ -71,6 +71,7 @@ class Service extends BaseModel protected $appends = ['server_status', 'status']; protected $casts = [ + 'include_coolify_default_variables' => 'boolean', 'redact_all_env_values_in_logs' => 'boolean', ]; @@ -753,7 +754,7 @@ class Service extends BaseModel ]); } $fields->put('Supabase', $data->toArray()); - case $image->contains('minio'): + case $image->contains(['minio', 'pgsty/silo']): $data = collect([]); $console_url = $this->environment_variables()->where('key', 'MINIO_BROWSER_REDIRECT_URL')->first(); $s3_api_url = $this->environment_variables()->where('key', 'MINIO_SERVER_URL')->first(); diff --git a/app/Models/StandaloneClickhouse.php b/app/Models/StandaloneClickhouse.php index 147dab9832..eeaa59c97c 100644 --- a/app/Models/StandaloneClickhouse.php +++ b/app/Models/StandaloneClickhouse.php @@ -74,6 +74,7 @@ class StandaloneClickhouse extends BaseModel ]; protected $casts = [ + 'include_coolify_default_variables' => 'boolean', 'health_check_enabled' => 'boolean', 'health_check_interval' => 'integer', 'health_check_timeout' => 'integer', diff --git a/app/Models/StandaloneDragonfly.php b/app/Models/StandaloneDragonfly.php index 7de46038c6..23d7131244 100644 --- a/app/Models/StandaloneDragonfly.php +++ b/app/Models/StandaloneDragonfly.php @@ -70,6 +70,7 @@ class StandaloneDragonfly extends BaseModel ]; protected $casts = [ + 'include_coolify_default_variables' => 'boolean', 'health_check_enabled' => 'boolean', 'health_check_interval' => 'integer', 'health_check_timeout' => 'integer', diff --git a/app/Models/StandaloneKeydb.php b/app/Models/StandaloneKeydb.php index 3257a9c108..d2aa105bd3 100644 --- a/app/Models/StandaloneKeydb.php +++ b/app/Models/StandaloneKeydb.php @@ -71,6 +71,7 @@ class StandaloneKeydb extends BaseModel ]; protected $casts = [ + 'include_coolify_default_variables' => 'boolean', 'health_check_enabled' => 'boolean', 'health_check_interval' => 'integer', 'health_check_timeout' => 'integer', diff --git a/app/Models/StandaloneMariadb.php b/app/Models/StandaloneMariadb.php index ea3bc39ce4..be5c464e0d 100644 --- a/app/Models/StandaloneMariadb.php +++ b/app/Models/StandaloneMariadb.php @@ -73,6 +73,7 @@ class StandaloneMariadb extends BaseModel ]; protected $casts = [ + 'include_coolify_default_variables' => 'boolean', 'health_check_enabled' => 'boolean', 'health_check_interval' => 'integer', 'health_check_timeout' => 'integer', diff --git a/app/Models/StandaloneMongodb.php b/app/Models/StandaloneMongodb.php index 41bf67678b..208b5e917a 100644 --- a/app/Models/StandaloneMongodb.php +++ b/app/Models/StandaloneMongodb.php @@ -72,6 +72,7 @@ class StandaloneMongodb extends BaseModel ]; protected $casts = [ + 'include_coolify_default_variables' => 'boolean', 'health_check_enabled' => 'boolean', 'health_check_interval' => 'integer', 'health_check_timeout' => 'integer', diff --git a/app/Models/StandaloneMysql.php b/app/Models/StandaloneMysql.php index 2593bafa32..68a73ae573 100644 --- a/app/Models/StandaloneMysql.php +++ b/app/Models/StandaloneMysql.php @@ -74,6 +74,7 @@ class StandaloneMysql extends BaseModel ]; protected $casts = [ + 'include_coolify_default_variables' => 'boolean', 'health_check_enabled' => 'boolean', 'health_check_interval' => 'integer', 'health_check_timeout' => 'integer', diff --git a/app/Models/StandalonePostgresql.php b/app/Models/StandalonePostgresql.php index 9e6f66c958..7829ab0d0e 100644 --- a/app/Models/StandalonePostgresql.php +++ b/app/Models/StandalonePostgresql.php @@ -76,6 +76,7 @@ class StandalonePostgresql extends BaseModel ]; protected $casts = [ + 'include_coolify_default_variables' => 'boolean', 'health_check_enabled' => 'boolean', 'health_check_interval' => 'integer', 'health_check_timeout' => 'integer', diff --git a/app/Models/StandaloneRedis.php b/app/Models/StandaloneRedis.php index 7fe86c9974..d3c8004ae3 100644 --- a/app/Models/StandaloneRedis.php +++ b/app/Models/StandaloneRedis.php @@ -72,6 +72,7 @@ class StandaloneRedis extends BaseModel ]; protected $casts = [ + 'include_coolify_default_variables' => 'boolean', 'health_check_enabled' => 'boolean', 'health_check_interval' => 'integer', 'health_check_timeout' => 'integer', diff --git a/app/Models/StandaloneSqlite.php b/app/Models/StandaloneSqlite.php index e3490c1031..2f6993865c 100644 --- a/app/Models/StandaloneSqlite.php +++ b/app/Models/StandaloneSqlite.php @@ -60,6 +60,7 @@ class StandaloneSqlite extends BaseModel protected $appends = ['database_type', 'server_status']; protected $casts = [ + 'include_coolify_default_variables' => 'boolean', 'health_check_enabled' => 'boolean', 'health_check_interval' => 'integer', 'health_check_timeout' => 'integer', diff --git a/app/Models/User.php b/app/Models/User.php index 743871fcfc..1257f3c857 100644 --- a/app/Models/User.php +++ b/app/Models/User.php @@ -263,7 +263,7 @@ class User extends Authenticatable implements PasskeyUser, SendsEmail return $new_team; } - public function createToken(string $name, array $abilities = ['read'], ?DateTimeInterface $expiresAt = null) + public function createToken(string $name, array $abilities = ['read'], ?DateTimeInterface $expiresAt = null, ?int $teamId = null) { $plainTextToken = sprintf( '%s%s%s', @@ -277,7 +277,7 @@ class User extends Authenticatable implements PasskeyUser, SendsEmail 'token' => hash('sha256', $plainTextToken), 'abilities' => $abilities, 'expires_at' => $expiresAt, - 'team_id' => session('currentTeam')->id, + 'team_id' => $teamId ?? session('currentTeam')->id, ]); return new NewAccessToken($token, $token->getKey().'|'.$plainTextToken); diff --git a/app/Rules/ValidGitRepositoryUrl.php b/app/Rules/ValidGitRepositoryUrl.php index 9a04409046..8b2d484e54 100644 --- a/app/Rules/ValidGitRepositoryUrl.php +++ b/app/Rules/ValidGitRepositoryUrl.php @@ -29,7 +29,7 @@ class ValidGitRepositoryUrl implements ValidationRule // Brackets are only allowed around an IPv6 host, so check the rest of the URL without it. $valueWithoutIpv6Host = preg_replace_callback( - '/^((?:https?|git):\/\/|[a-zA-Z0-9._-]+@)\[([0-9A-Fa-f:.]+)\]/', + '/^((?:https?|git):\/\/|(?:ssh:\/\/)?[a-zA-Z0-9._-]+@)\[([0-9A-Fa-f:.]+)\]/', fn (array $matches): string => filter_var($matches[2], FILTER_VALIDATE_IP, FILTER_FLAG_IPV6) !== false ? $matches[1].'ipv6-host' : $matches[0], $value, ); @@ -96,6 +96,19 @@ class ValidGitRepositoryUrl implements ValidationRule if ($scp === null || preg_match('/^(\[[0-9A-Fa-f:.]+\]|[a-zA-Z0-9.-]+)$/', $scp['host']) !== 1 || preg_match('/^[a-zA-Z0-9\-_\/.~]+$/', $scp['path']) !== 1) { $fail('The :attribute is not a valid SSH repository URL.'); + return; + } + } elseif (str_starts_with($value, 'ssh://')) { + if (! $this->allowSSH) { + $fail('SSH URLs are not allowed.'); + + return; + } + + // Validate ssh:// URL format (ssh://user@host[:port]/user/repo.git) + if (! preg_match('/^ssh:\/\/[a-zA-Z0-9._-]+@(\[[0-9A-Fa-f:.]+\]|[a-zA-Z0-9.-]+)(:[0-9]{1,5})?\/[a-zA-Z0-9\-_\/.~]+$/', $value)) { + $fail('The :attribute is not a valid SSH repository URL.'); + return; } } elseif (str_starts_with($value, 'http://') || str_starts_with($value, 'https://')) { @@ -160,7 +173,7 @@ class ValidGitRepositoryUrl implements ValidationRule return; } } else { - $fail('The :attribute must start with https://, http://, git://, or be an SSH URL (user@host:path).'); + $fail('The :attribute must start with https://, http://, git://, ssh://, or be an SSH URL (user@host:path).'); return; } diff --git a/app/Services/ContainerFilesystemService.php b/app/Services/ContainerFilesystemService.php new file mode 100644 index 0000000000..e7178ed765 --- /dev/null +++ b/app/Services/ContainerFilesystemService.php @@ -0,0 +1,426 @@ + | base64 -d` writes travel as a single execve + * argument, capped at ~128 KiB by MAX_ARG_STRLEN. base64 inflates by 4/3, + * so keep inline content below that limit with room for command quoting. + */ + private const MAX_INLINE_WRITE_SIZE = 64 * 1024; + + /** Largest file, or uncompressed folder, the file browser may download. */ + public const MAX_DOWNLOAD_SIZE = 1024 * 1024 * 1024; + + public function __construct( + private Server $server, + private string $container, + ) {} + + public function buildListCommand(string $path): string + { + $escapedPath = $this->escapePath($path, 'list path'); + + // Emit the whole directory through batched stat calls (one for most folders) + // instead of a per-entry loop. %F is the human file type ("directory", + // "symbolic link", "regular file"), mapped to a token by parseListing. + // Fields: typesizemtimepermsownergroupname. + // A real tab is embedded in the format because stat's -c does NOT expand + // a backslash \t escape (unlike printf) - it would emit the literal text. + // POSIX `find ... -prune -exec {} +` lists hidden files without recursing + // and batches names, so a huge directory cannot overflow the argument + // limit like a `* .*` glob would. Names arrive as ./name; parseListing + // strips that prefix. Names containing a tab/newline are skipped (v1 scope). + $tab = "\t"; + $format = '%F'.$tab.'%s'.$tab.'%Y'.$tab.'%a'.$tab.'%U'.$tab.'%G'.$tab.'%n'; + $inner = 'cd '.$escapedPath.' 2>/dev/null || exit 0; ' + .'find . ! -name . -prune -exec stat -c "'.$format.'" -- {} + 2>/dev/null'; + + return $this->dockerExecShell($inner); + } + + /** + * @return array + */ + public function list(string $path): array + { + $raw = instant_remote_process([$this->buildListCommand($path)], $this->server, throwError: false); + + return $this->parseListing($raw); + } + + /** + * Read an editable text file in a single SSH round trip. + * + * The remote command folds the size cap, binary probe and base64 read into + * one docker exec: it prints TOOBIG or BINARY for a rejected file, otherwise + * an "OK\n" header followed by the base64 payload (empty for a 0-byte file, + * which is valid editable text). base64 keeps exact bytes safe from the + * trailing-whitespace trim in instant_remote_process. + * + * @throws \RuntimeException when the file is binary or larger than the cap + */ + public function read(string $path): string + { + $escaped = $this->escapePath($path, 'read path'); + $max = LocalFileVolume::MAX_CONTENT_SIZE; + + $cmd = '[ -f '.$escaped.' ] && [ -r '.$escaped.' ] || exit 1; ' + .'sz=$(stat -Lc %s -- '.$escaped.') || exit 1; ' + .'if [ "$sz" -gt '.$max.' ]; then echo TOOBIG; exit 0; fi; ' + .'if [ -s '.$escaped.' ] && ! grep -qI \'\' -- '.$escaped.'; then echo BINARY; exit 0; fi; ' + .'echo OK; base64 -- '.$escaped.' || exit 1'; + + $raw = (string) instant_remote_process( + [$this->dockerExecShell($cmd)], + $this->server, + ); + + $newline = strpos($raw, "\n"); + $status = $newline === false ? trim($raw) : substr($raw, 0, $newline); + if ($status === 'TOOBIG' || $status === 'BINARY') { + throw new \RuntimeException('File is not editable (binary or too large).'); + } + + $payload = $newline === false ? '' : substr($raw, $newline + 1); + + $content = base64_decode(trim($payload), true); + if ($status !== 'OK' || $content === false || strlen($content) > $max) { + throw new \RuntimeException('Could not read file.'); + } + + return $content; + } + + public function buildWriteCommand(string $path, string $content): string + { + $escaped = $this->escapePath($path, 'write path'); + $b64 = base64_encode($content); + + return $this->dockerExecShell("echo {$b64} | base64 -d > {$escaped}"); + } + + public function buildMkdirCommand(string $path): string + { + $escaped = $this->escapePath($path, 'mkdir path'); + + return $this->dockerExecShell("mkdir -p -- {$escaped}"); + } + + public function buildCreateFileCommand(string $path): string + { + $escaped = $this->escapePath($path, 'create file path'); + + // Create only if absent so an existing file is never truncated. + return $this->dockerExecShell("[ -e {$escaped} ] || : > {$escaped}"); + } + + public function buildChmodCommand(string $path, string $mode): string + { + $mode = $this->normalizeMode($mode); + $escaped = $this->escapePath($path, 'chmod path'); + + return $this->dockerExecShell("chmod {$mode} -- {$escaped}"); + } + + public function buildRenameCommand(string $from, string $to): string + { + $escapedFrom = $this->escapePath($from, 'rename source'); + $escapedTo = $this->escapePath($to, 'rename target'); + + return $this->dockerExecShell("mv -- {$escapedFrom} {$escapedTo}"); + } + + public function buildDeleteCommand(string $path): string + { + $escaped = $this->escapePath($path, 'delete path'); + + return $this->dockerExecShell("rm -rf -- {$escaped}"); + } + + public function write(string $path, string $content): void + { + if (strlen($content) > self::MAX_INLINE_WRITE_SIZE) { + $this->writeViaUpload($path, $content); + + return; + } + instant_remote_process([$this->buildWriteCommand($path, $content)], $this->server); + } + + /** + * Write content too large for a single shell argument by staging it to a + * local temp file and reusing the docker cp upload path. + */ + protected function writeViaUpload(string $path, string $content): void + { + validateShellSafePath($path, 'write path'); + $localTmp = storage_path('app/tmp/'.Str::random(16)); + @mkdir(dirname($localTmp), 0755, true); + file_put_contents($localTmp, $content); + try { + $this->upload($localTmp, $path); + } finally { + @unlink($localTmp); + } + } + + public function makeDirectory(string $path): void + { + instant_remote_process([$this->buildMkdirCommand($path)], $this->server); + } + + public function createFile(string $path): void + { + instant_remote_process([$this->buildCreateFileCommand($path)], $this->server); + } + + public function chmod(string $path, string $mode): void + { + instant_remote_process([$this->buildChmodCommand($path, $mode)], $this->server); + } + + protected function normalizeMode(string $mode): string + { + $mode = trim($mode); + if (! preg_match('/^[0-7]{3,4}$/', $mode)) { + throw new \InvalidArgumentException('Invalid permission mode. Use octal like 644 or 0755.'); + } + + return $mode; + } + + public function rename(string $from, string $to): void + { + instant_remote_process([$this->buildRenameCommand($from, $to)], $this->server); + } + + public function delete(string $path): void + { + instant_remote_process([$this->buildDeleteCommand($path)], $this->server); + } + + public function defaultRoot(): string + { + $escapedContainer = escapeshellarg($this->container); + $workingDir = instant_remote_process( + ["docker inspect --format '{{.Config.WorkingDir}}' {$escapedContainer}"], + $this->server, + throwError: false, + ); + + $workingDir = trim((string) $workingDir); + + return $workingDir !== '' ? $workingDir : '/'; + } + + /** + * @return array + */ + public function parseListing(?string $raw): array + { + if (blank($raw)) { + return []; + } + + $entries = []; + foreach (explode("\n", trim($raw)) as $line) { + if ($line === '') { + continue; + } + $parts = explode("\t", $line, 7); + $owner = ''; + $group = ''; + // Accept the current 7-field format (with owner/group) and the + // legacy 5-field (perms) and 4-field formats. + if (count($parts) === 7) { + [$type, $size, $mtime, $perms, $owner, $group, $name] = $parts; + } elseif (count($parts) === 5) { + [$type, $size, $mtime, $perms, $name] = $parts; + } elseif (count($parts) === 4) { + [$type, $size, $mtime, $name] = $parts; + $perms = ''; + } else { + continue; + } + // find prefixes names with ./; a name itself can never contain a slash. + if (str_starts_with($name, './')) { + $name = substr($name, 2); + } + if ($name === '' || $name === '.' || $name === '..') { + continue; + } + $entries[] = new FileEntry( + $name, + $this->normalizeType($type), + (int) $size, + (int) $mtime, + trim($perms), + trim($owner), + trim($group), + ); + } + + return FileEntry::sort($entries); + } + + /** + * Map a raw type field to a file/dir/symlink token. Accepts both the + * already-tokenized legacy value and stat's human %F string. + */ + protected function normalizeType(string $raw): string + { + $t = strtolower(trim($raw)); + if ($t === 'dir' || str_contains($t, 'directory')) { + return 'dir'; + } + if ($t === 'symlink' || str_contains($t, 'symbolic link')) { + return 'symlink'; + } + + return 'file'; + } + + public function upload(string $localTmpPath, string $destPath): void + { + validateShellSafePath($destPath, 'upload path'); + $serverTmp = '/tmp/coolify-upload-'.Str::random(16); + $escapedServerTmp = escapeshellarg($serverTmp); + $escapedContainer = escapeshellarg($this->container); + $escapedDest = escapeshellarg($destPath); + + try { + instant_scp($localTmpPath, $serverTmp, $this->server); + instant_remote_process( + ["docker cp {$escapedServerTmp} {$escapedContainer}:{$escapedDest}"], + $this->server, + ); + } finally { + instant_remote_process(["rm -f {$escapedServerTmp}"], $this->server, throwError: false); + } + } + + /** + * Inspect a download target in one round trip. Folder size is the + * uncompressed disk usage, so the archive is never larger than reported. + * + * @return array{isDir: bool, size: int} + * + * @throws \RuntimeException when the path does not exist + */ + public function downloadInfo(string $path): array + { + $escaped = $this->escapePath($path, 'download path'); + $cmd = 'if [ -d '.$escaped.' ]; then echo "dir $(du -sk '.$escaped.' | cut -f1)"; ' + .'elif [ -e '.$escaped.' ]; then echo "file $(stat -L -c %s '.$escaped.')"; fi'; + + $raw = trim((string) instant_remote_process([$this->dockerExecShell($cmd)], $this->server)); + [$kind, $size] = array_pad(explode(' ', $raw, 2), 2, ''); + if (! in_array($kind, ['dir', 'file'], true) || ! ctype_digit($size)) { + throw new \RuntimeException('File not found.'); + } + + return [ + 'isDir' => $kind === 'dir', + 'size' => $kind === 'dir' ? (int) $size * 1024 : (int) $size, + ]; + } + + public function buildDownloadCommand(string $path, bool $isDir): string + { + $escaped = $this->escapePath($path, 'download path'); + $escapedContainer = escapeshellarg($this->container); + + return $isDir + ? "docker exec {$escapedContainer} tar czf - -C {$escaped} ." + : "docker exec {$escapedContainer} cat -- {$escaped}"; + } + + /** + * Pipe a file, or a folder as .tar.gz, from the container to $write in + * chunks. Nothing is stored on the Coolify host or the managed server. + * + * @param callable(string): void $write + */ + public function streamDownload(string $path, bool $isDir, callable $write): void + { + $command = $this->buildDownloadCommand($path, $isDir); + if ($this->server->isNonRoot()) { + $command = parseCommandsByLineForSudo(collect([$command]), $this->server)[0]; + } + + $process = proc_open($this->sshCommand($command), [1 => ['pipe', 'w'], 2 => ['file', '/dev/null', 'w']], $pipes); + if (! is_resource($process)) { + throw new \RuntimeException('Could not start the download.'); + } + + try { + while (! feof($pipes[1])) { + $chunk = fread($pipes[1], 64 * 1024); + if ($chunk === false || connection_aborted()) { + break; + } + if ($chunk !== '') { + $write($chunk); + } + } + } finally { + fclose($pipes[1]); + proc_terminate($process); + proc_close($process); + } + } + + /** + * Stage the file, or the folder as .tar.gz, on the managed server and PUT + * it to a presigned S3 URL. S3 needs a Content-Length, so a pipe without a + * known size cannot be uploaded directly. The temp file is always removed. + * + * @param array $headers + */ + public function buildS3UploadCommand(string $path, bool $isDir, string $url, array $headers): string + { + $headerArgs = FileDownloadStorageService::curlHeaderArgs($headers); + $script = 'tmp=$(mktemp /tmp/coolify-download-XXXXXXXXXX) || exit 1; ' + .'trap \'rm -f "$tmp"\' EXIT; ' + .$this->buildDownloadCommand($path, $isDir).' > "$tmp" && ' + .'curl -fsS --retry 2 '.$headerArgs.' -T "$tmp" '.escapeshellarg($url); + + return 'sh -c '.escapeshellarg($script); + } + + /** + * @param array $headers + */ + public function uploadToS3(string $path, bool $isDir, string $url, array $headers): void + { + instant_remote_process([$this->buildS3UploadCommand($path, $isDir, $url, $headers)], $this->server, timeout: 3600); + } + + protected function sshCommand(string $command): string + { + return SshMultiplexingHelper::generateSshCommand($this->server, $command); + } + + protected function escapePath(string $path, string $label): string + { + validateShellSafePath($path, $label); + + return escapeshellarg($path); + } + + protected function dockerExecShell(string $innerShell): string + { + $escapedContainer = escapeshellarg($this->container); + + return "docker exec {$escapedContainer} sh -c ".escapeshellarg($innerShell); + } +} diff --git a/app/Services/Dns/CloudflareDnsProvider.php b/app/Services/Dns/CloudflareDnsProvider.php index d3c57b78b5..9cd21a4fbf 100644 --- a/app/Services/Dns/CloudflareDnsProvider.php +++ b/app/Services/Dns/CloudflareDnsProvider.php @@ -330,6 +330,85 @@ class CloudflareDnsProvider return $this->forgetDeletedRecord($record, ManagedDnsDeletionResult::Deleted, $resource); } + /** + * Takes ownership of a record that another Coolify instance created, after a server transfer. The provider + * comment changes from $previousComment to the ownership comment of this instance only when the record still + * has the expected type, name, value and comment. Never throws; returns false when the record was not adopted. + */ + public function adoptRecord(ManagedDnsRecord $record, string $previousComment): bool + { + try { + $record->loadMissing(['zone', 'integrationToken']); + if ($record->zone === null || $record->integrationToken?->provider !== 'cloudflare') { + return false; + } + + return $this->hostnameLock->run( + (int) $record->integrationToken->team_id, + $record->name, + fn (): bool => $this->adoptRecordLocked($record, $previousComment), + ); + } catch (Throwable) { + return false; + } + } + + /** + * Takes a record back after a take back, when another Coolify instance may have adopted it meanwhile. The + * provider comment must be an ownership comment for this record uuid, from this or any other instance. + * Never throws; returns false when the record was not taken back. + */ + public function reclaimRecord(ManagedDnsRecord $record): bool + { + try { + $record->loadMissing(['zone', 'integrationToken']); + if ($record->zone === null || $record->integrationToken?->provider !== 'cloudflare') { + return false; + } + + return $this->hostnameLock->run( + (int) $record->integrationToken->team_id, + $record->name, + function () use ($record): bool { + $url = "https://api.cloudflare.com/client/v4/zones/{$record->zone->provider_zone_id}/dns_records/{$record->provider_record_id}"; + $comment = $this->client($record->integrationToken)->get($url)->json('result.comment'); + $pattern = '/^'.preg_quote(ManagedDnsRecord::OWNERSHIP_COMMENT_PREFIX, '/').' \\S+\\/'.preg_quote($record->uuid, '/').'$/'; + if (! is_string($comment) || preg_match($pattern, $comment) !== 1) { + return false; + } + + return $comment === $record->ownershipComment() || $this->adoptRecordLocked($record, $comment); + }, + ); + } catch (Throwable) { + return false; + } + } + + private function adoptRecordLocked(ManagedDnsRecord $record, string $previousComment): bool + { + $url = "https://api.cloudflare.com/client/v4/zones/{$record->zone->provider_zone_id}/dns_records/{$record->provider_record_id}"; + $response = $this->client($record->integrationToken)->get($url); + $remote = $response->json('result'); + if (! $response->successful() || ! is_array($remote)) { + return false; + } + if (($remote['type'] ?? null) !== $record->type + || strtolower((string) ($remote['name'] ?? '')) !== $record->name + || ! DnsRecordHints::sameAddress(is_string($remote['content'] ?? null) ? $remote['content'] : null, $record->content) + || ($remote['comment'] ?? null) !== $previousComment) { + return false; + } + + $patchResponse = $this->client($record->integrationToken)->patch($url, ['comment' => $record->ownershipComment()]); + if (! $patchResponse->successful()) { + return false; + } + $this->auditDnsRecord('ownership_adopted', $record->zone, $record->name, null); + + return true; + } + private function forgetDeletedRecord(ManagedDnsRecord $record, ManagedDnsDeletionResult $result, ?Model $resource): ManagedDnsDeletionResult { $record->delete(); diff --git a/app/Services/Dns/ManagedDnsRecordCleanup.php b/app/Services/Dns/ManagedDnsRecordCleanup.php index d8d02dfd73..59b75a55a6 100644 --- a/app/Services/Dns/ManagedDnsRecordCleanup.php +++ b/app/Services/Dns/ManagedDnsRecordCleanup.php @@ -149,6 +149,71 @@ class ManagedDnsRecordCleanup return $successful; } + /** + * Stops tracking the records of these resources without touching the provider. Used when another Coolify + * instance takes over the resources: their records still serve live traffic there. A record that another + * resource of this instance still uses keeps that resource's references. + * + * @param iterable $resources + */ + public function forgetResources(iterable $resources): void + { + $keys = collect($resources) + ->flatMap(fn (Model $resource) => $this->resourceKeys($resource->getMorphClass(), $resource->getKey())) + ->all(); + if ($keys === []) { + return; + } + + $records = ManagedDnsRecord::query() + ->whereHas('references', fn (Builder $query) => $this->whereResourceKeys($query, $keys)) + ->get(); + + foreach ($records as $record) { + $this->deleteReferences($record, $keys); + if (! $record->references()->exists()) { + $this->forget($record, 'The resource was transferred to another Coolify instance, which uses the record now.', 'info'); + } + } + } + + /** + * Keeps the records of resources that another Coolify instance manages now, for a later take back, but marks + * them not owned, so this instance never changes or deletes them at the provider. A record that another + * resource of this instance also uses only loses the references of these resources. + * + * @param iterable $resources + * @return list ids of the records that were owned before + */ + public function parkResources(iterable $resources): array + { + $keys = collect($resources) + ->flatMap(fn (Model $resource) => $this->resourceKeys($resource->getMorphClass(), $resource->getKey())) + ->all(); + if ($keys === []) { + return []; + } + + $parked = []; + $records = ManagedDnsRecord::query() + ->whereHas('references', fn (Builder $query) => $this->whereResourceKeys($query, $keys)) + ->get(); + foreach ($records as $record) { + $sharedWithOthers = $record->references()->whereNot(fn (Builder $query) => $this->whereResourceKeys($query, $keys))->exists(); + if ($sharedWithOthers) { + $this->deleteReferences($record, $keys); + + continue; + } + if ($record->owned) { + $record->update(['owned' => false]); + $parked[] = $record->id; + } + } + + return $parked; + } + /** * Releases orphaned records: records whose references all point to deleted (or soft-deleted) resources, and * records without references that are older than the grace period. Records with a live reference are not touched. diff --git a/app/Services/EnvironmentCanvas.php b/app/Services/EnvironmentCanvas.php index ff42ebf6bd..8372af3371 100644 --- a/app/Services/EnvironmentCanvas.php +++ b/app/Services/EnvironmentCanvas.php @@ -4,6 +4,7 @@ namespace App\Services; use App\Models\Application; use App\Models\Environment; +use App\Models\Project; use App\Models\ResourceLink; use App\Models\Server; use App\Models\Service; @@ -22,6 +23,9 @@ class EnvironmentCanvas { public const MAX_COORDINATE = 100000; + /** Zone id of the placeholder cards of environments without resources. */ + public const EMPTY_ZONE = 'empty'; + /** * Environment relations that hold standalone databases, keyed by relation name. */ @@ -108,6 +112,8 @@ class EnvironmentCanvas $servers->put($server->id, $server); } unset($node['servers']); + // Other canvases keep the arrangement saved on the environment canvas inside each environment group. + $node['saved'] = $this->filterPositions($resource->environment?->canvas_positions ?? [], [$resource->uuid])[$resource->uuid] ?? null; return $node; })->values(); @@ -146,6 +152,65 @@ class EnvironmentCanvas ]; } + /** + * Zone of the server, project and dashboard canvases for environments without resources (they have no server). + * + * @return array{id: string, name: string, icon: string} + */ + public function emptyZone(): array + { + return ['id' => self::EMPTY_ZONE, 'name' => 'Empty environments', 'icon' => 'service']; + } + + /** + * Zone for the placeholder cards of a project: the zone with most of the project's cards, or `$fallback` + * when the project has no cards. + * + * @param Collection> $nodes cards of the canvas + * @return callable(Project): string + */ + public function projectZones(Collection $nodes, string $fallback): callable + { + $zones = $nodes + ->filter(fn (array $node) => filled($node['zone'])) + ->groupBy(fn (array $node) => data_get($node, 'project.uuid')) + ->map(fn (Collection $cards) => $cards->countBy('zone')->sortDesc()->keys()->first()); + + return fn (Project $project) => $zones->get($project->uuid, $fallback); + } + + /** + * Placeholder card of an environment without resources, so the grouped canvases still show it. + * Without an environment, it stands for a project without environments. + * `$canAdd`: the card opens the deploy canvas for the environment instead of the environment page. + * + * @return array + */ + public function emptyNode(Project $project, ?Environment $environment, string $zone, string $context, bool $canAdd = false): array + { + $page = $environment + ? route('project.resource.index', ['project_uuid' => $project->uuid, 'environment_uuid' => $environment->uuid]) + : route('project.show', ['project_uuid' => $project->uuid]); + $add = $canAdd && $environment; + + return [ + 'id' => 'empty:'.($environment?->uuid ?? $project->uuid), + 'type' => 'empty', + 'name' => $environment ? 'No resources yet' : 'No environments yet', + 'subtitle' => $add ? 'Click to add a resource' : 'Open to see this '.($environment ? 'environment' : 'project'), + 'status' => null, + 'zone' => $zone, + 'extra_zones' => [], + 'urls' => [], + 'children' => [], + 'link' => $add ? route('deploy-canvas', ['project' => $project->uuid, 'environment' => $environment->uuid]) : $page, + 'context' => $context, + 'context_link' => $page, + 'project' => ['uuid' => $project->uuid, 'name' => $project->name], + 'environment' => $environment ? ['uuid' => $environment->uuid, 'name' => $environment->name] : null, + ]; + } + /** * Top-level resources of the environment (applications, services, standalone databases), keyed by uuid, * with the relations the canvas reads. @@ -157,19 +222,19 @@ class EnvironmentCanvas $project = $environment->project; $environment->setRelation('project', $project); $applications = $environment->applications() - ->with(['destination.server.settings', 'additional_servers.settings', 'persistentStorages']) + ->with(['destination.server.settings', 'additional_servers.settings']) ->get(); $services = $environment->services() ->with([ 'server.settings', 'destination.server.settings', - 'applications.persistentStorages', - 'databases.persistentStorages', + 'applications', + 'databases', ]) ->get(); $databases = collect(self::DATABASE_RELATIONS) ->flatMap(fn (string $relation) => $environment->{$relation}() - ->with(['destination.server.settings', 'persistentStorages']) + ->with(['destination.server.settings']) ->get()); return collect() @@ -251,7 +316,6 @@ class EnvironmentCanvas 'kind' => $buildPack, 'subtitle' => $this->subtitle([self::BUILD_PACK_NAMES[$buildPack] ?? str($buildPack)->headline()->value(), $detail]), 'children' => [], - 'volumes' => $application->persistentStorages->pluck('name')->filter()->values()->all(), 'link' => $application->link(), 'logo' => resource_logo($application), 'servers' => collect([$server])->concat($extraServers)->filter()->all(), @@ -285,12 +349,6 @@ class EnvironmentCanvas 'kind' => null, 'subtitle' => $this->subtitle(['Service', $containerCount.' '.str('container')->plural($containerCount)]), 'children' => $children->all(), - 'volumes' => $service->applications->concat($service->databases) - ->flatMap(fn ($child) => $child->persistentStorages->pluck('name')) - ->filter() - ->unique() - ->values() - ->all(), 'link' => $service->link(), 'logo' => resource_logo($service), 'servers' => collect([$server])->filter()->all(), @@ -316,7 +374,6 @@ class EnvironmentCanvas 'kind' => $engine, 'subtitle' => $this->subtitle([self::DATABASE_ENGINE_NAMES[$engine] ?? str($engine)->headline()->value(), $database->image]), 'children' => [], - 'volumes' => $database->persistentStorages->pluck('name')->filter()->values()->all(), 'link' => $database->link(), 'logo' => resource_logo($database), 'servers' => collect([$server])->filter()->all(), diff --git a/app/Services/FileDownloadStorageService.php b/app/Services/FileDownloadStorageService.php new file mode 100644 index 0000000000..f342873c9c --- /dev/null +++ b/app/Services/FileDownloadStorageService.php @@ -0,0 +1,207 @@ +file_download_s3_storage_id; + if (! $storageId) { + return null; + } + + return S3Storage::query()->whereTeamId(0)->where('is_usable', true)->find($storageId); + } + + public static function newObjectKey(): string + { + return self::PREFIX.Str::random(48); + } + + /** + * Add or replace the rule that deletes download objects after one day. + * S3 lifecycle rules have a one-day minimum, so this is only a safety net + * for the six-hour delete job. Other rules on the bucket are kept. + * + * @throws RuntimeException when the bucket does not accept the rule + */ + public function ensureLifecycleRule(S3Storage $storage): void + { + try { + $client = $storage->filesystem()->getClient(); + try { + $rules = $client->getBucketLifecycleConfiguration(['Bucket' => $storage->bucket])['Rules'] ?? []; + } catch (S3Exception $e) { + if ($e->getAwsErrorCode() !== 'NoSuchLifecycleConfiguration') { + throw $e; + } + $rules = []; + } + + $rules = array_values(array_filter($rules, fn (array $rule) => ($rule['ID'] ?? null) !== self::LIFECYCLE_RULE_ID)); + $rules[] = [ + 'ID' => self::LIFECYCLE_RULE_ID, + 'Filter' => ['Prefix' => self::PREFIX], + 'Status' => 'Enabled', + 'Expiration' => ['Days' => 1], + ]; + + $client->putBucketLifecycleConfiguration([ + 'Bucket' => $storage->bucket, + 'LifecycleConfiguration' => ['Rules' => $rules], + ]); + } catch (\Throwable $e) { + throw new RuntimeException('Could not add the auto-delete lifecycle rule to bucket "'.$storage->bucket.'". The S3 provider must support lifecycle rules, and the key must be allowed to change them. Error: '.$e->getMessage(), previous: $e); + } + } + + /** + * Download links are only safe while object keys stay secret. Refuse a + * bucket that lets anonymous users list its objects, and an endpoint that + * only Coolify can reach (servers upload to it, browsers download from it). + * + * @throws RuntimeException when the endpoint is internal, unreachable or listable + */ + public function ensureBucketIsPrivate(S3Storage $storage): void + { + if ($storage->trustedInternalHosts() !== []) { + throw new RuntimeException('"'.$storage->name.'" uses an internal Docker host name. Your servers and browsers cannot reach it. Use an S3 endpoint with a public or LAN address.'); + } + + try { + $this->anonymousClient($storage)->listObjectsV2(['Bucket' => $storage->bucket, 'Prefix' => self::PREFIX, 'MaxKeys' => 1]); + } catch (S3Exception $e) { + if ($e->getResponse() === null && $e->getAwsErrorCode() === null) { + throw new RuntimeException('Could not connect to "'.$storage->name.'". Check the endpoint. Your servers and browsers must reach it too. Error: '.$e->getMessage(), previous: $e); + } + + return; + } catch (\Throwable $e) { + throw new RuntimeException('Could not connect to "'.$storage->name.'". Check the endpoint. Your servers and browsers must reach it too. Error: '.$e->getMessage(), previous: $e); + } + + throw new RuntimeException('Bucket "'.$storage->bucket.'" allows anonymous listing. Make the bucket private before you use it for file downloads.'); + } + + protected function anonymousClient(S3Storage $storage): S3Client + { + return new S3Client([ + 'version' => 'latest', + 'region' => $storage->region, + 'endpoint' => $storage->endpoint, + 'use_path_style_endpoint' => true, + 'credentials' => false, + 'retries' => 0, + 'http' => array_merge(SafeWebhookUrl::httpClientOptions($storage->endpoint, $storage->trustedInternalHosts()), [ + 'connect_timeout' => 15, + 'timeout' => 15, + ]), + ]); + } + + /** + * @return array{url: string, headers: array} + */ + public function uploadUrl(S3Storage $storage, string $key, string $downloadName): array + { + $upload = $storage->filesystem()->temporaryUploadUrl($key, now()->addHour(), [ + 'ContentDisposition' => self::contentDisposition($downloadName), + ]); + + return [ + 'url' => $upload['url'], + 'headers' => collect($upload['headers']) + ->map(fn ($value) => is_array($value) ? implode(', ', $value) : (string) $value) + ->except(['Host', 'host']) + ->all(), + ]; + } + + public function downloadUrl(S3Storage $storage, string $key): string + { + return $storage->filesystem()->temporaryUrl($key, now()->addMinutes(self::LINK_TTL_MINUTES)); + } + + public function delete(S3Storage $storage, string $key): void + { + $storage->filesystem()->delete($key); + } + + /** + * Size in bytes of a file on the server host (for example a backup), or + * null when the file does not exist. + */ + public static function hostFileSize(Server $server, string $path): ?int + { + $size = trim((string) instant_remote_process([self::buildHostFileSizeCommand($path)], $server)); + + return ctype_digit($size) ? (int) $size : null; + } + + public static function buildHostFileSizeCommand(string $path): string + { + validateShellSafePath($path, 'download path'); + $escaped = escapeshellarg($path); + + return 'sh -c '.escapeshellarg('if [ -f '.$escaped.' ]; then stat -L -c %s '.$escaped.'; fi'); + } + + /** + * PUT a file on the server host to a presigned S3 URL. + * + * @param array $headers + */ + public static function buildHostFileUploadCommand(string $path, string $url, array $headers): string + { + validateShellSafePath($path, 'download path'); + + return 'sh -c '.escapeshellarg('curl -fsS --retry 2 '.self::curlHeaderArgs($headers).' -T '.escapeshellarg($path).' '.escapeshellarg($url)); + } + + /** + * @param array $headers + */ + public static function curlHeaderArgs(array $headers): string + { + return collect($headers) + ->map(fn (string $value, string $name) => '-H '.escapeshellarg("{$name}: {$value}")) + ->implode(' '); + } + + public static function contentDisposition(string $name): string + { + $fallback = preg_replace('/[^A-Za-z0-9._-]/', '_', $name) ?: 'download'; + + return 'attachment; filename="'.$fallback.'"; filename*=UTF-8\'\''.rawurlencode($name); + } +} diff --git a/app/Services/ProjectCanvas.php b/app/Services/ProjectCanvas.php index 20e537a8c4..57619b2e30 100644 --- a/app/Services/ProjectCanvas.php +++ b/app/Services/ProjectCanvas.php @@ -6,6 +6,7 @@ use App\Models\Environment; use App\Models\Project; use App\Models\ResourceLink; use Illuminate\Database\Eloquent\Model; +use Illuminate\Support\Collection; /** * Builds the project canvas: every resource of the project's environments, one zone per server, @@ -19,13 +20,14 @@ class ProjectCanvas /** * The canvas of every project of a team: one zone per project, cards grouped by environment. * - * @return array{title: string, title_link: string, can_edit: bool, layout: 'server', zones: array>, nodes: array>, links: array>, positions: array} + * @return array{title: string, title_link: string, can_edit: bool, can_move: false, can_move_groups: bool, layout: 'server', zones: array>, nodes: array>, links: array>, positions: array{}} */ public function teamPayload(int $teamId, bool $canReadSensitive, bool $canEdit = false): array { $projects = Project::query()->where('team_id', $teamId)->orderBy('name')->get(['id', 'uuid', 'name', 'team_id']); $environments = Environment::query()->whereIn('project_id', $projects->pluck('id')) - ->get(['id', 'uuid', 'name', 'project_id', 'canvas_positions']); + ->orderBy('id') + ->get(['id', 'uuid', 'name', 'project_id']); $resources = $this->servers->loadResources($environments->pluck('id')); $links = ResourceLink::query()->whereIn('environment_id', $environments->pluck('id'))->orderBy('id')->get(); $graph = $this->canvas->graph($resources, $links, $canReadSensitive); @@ -42,6 +44,10 @@ class ProjectCanvas 'title' => 'Projects', 'title_link' => route('project.index'), 'can_edit' => $canEdit, + // Environment groups and cards keep their environment canvas arrangement. + 'can_move' => false, + // Admins move groups and cards here; the canvas saves them on their environment canvas. + 'can_move_groups' => $canEdit, // Same grouped layout as the server canvas, with projects as the zones. 'layout' => 'server', 'zones' => $projects->map(fn (Project $project) => [ @@ -50,21 +56,20 @@ class ProjectCanvas 'icon' => 'projects', 'link' => route('project.show', ['project_uuid' => $project->uuid]), ])->values()->all(), - 'nodes' => collect($graph['nodes'])->map(fn (array $node) => [...$node, ...$grouping->get($node['id'])])->values()->all(), + 'nodes' => collect($graph['nodes'])->map(fn (array $node) => [...$node, ...$grouping->get($node['id'])]) + ->concat($projects->flatMap(fn (Project $project) => $this->emptyNodes($project, $environments->where('project_id', $project->id), $resources, $project->uuid, $canEdit))) + ->values()->all(), 'links' => $graph['links'], - 'positions' => $this->canvas->filterPositions( - $environments->reduce(fn (array $positions, Environment $environment) => [...$positions, ...($environment->canvas_positions ?? [])], []), - $resources->keys()->all(), - ), + 'positions' => [], ]; } /** - * @return array{title: string, title_link: string, can_edit: bool, layout: 'server', zones: array>, nodes: array>, links: array>, positions: array} + * @return array{title: string, title_link: string, can_edit: bool, can_move: false, can_move_groups: bool, layout: 'server', zones: array>, nodes: array>, links: array>, positions: array{}} */ public function payload(Project $project, bool $canReadSensitive, bool $canEdit = false): array { - $environments = $project->environments()->get(['id', 'uuid', 'name', 'project_id', 'canvas_positions']); + $environments = $project->environments()->orderBy('id')->get(['id', 'uuid', 'name', 'project_id']); $resources = $this->servers->loadResources($environments->pluck('id')); $links = ResourceLink::query()->whereIn('environment_id', $environments->pluck('id'))->orderBy('id')->get(); $graph = $this->canvas->graph($resources, $links, $canReadSensitive); @@ -74,25 +79,50 @@ class ProjectCanvas 'context_link' => route('project.resource.index', ['project_uuid' => $project->uuid, 'environment_uuid' => $resource->environment->uuid]), ...$this->owner($resource), ]); + $nodes = collect($graph['nodes'])->map(fn (array $node) => [...$node, ...$context->get($node['id'])]); + // Placeholder cards sit on the server of the project's other environments. + $zone = $this->canvas->projectZones($nodes, $graph['zones'][0]['id'] ?? EnvironmentCanvas::EMPTY_ZONE)($project); + $empty = $this->emptyNodes($project, $environments, $resources, $zone, $canEdit); return [ 'title' => $project->name, 'title_link' => route('project.show', ['project_uuid' => $project->uuid]), 'can_edit' => $canEdit, + 'can_move' => false, + // Admins move groups and cards here; the canvas saves them on their environment canvas. + 'can_move_groups' => $canEdit, // Groups cards by environment inside each server zone, like the server canvas. 'layout' => 'server', - 'zones' => $graph['zones'], - 'nodes' => collect($graph['nodes'])->map(fn (array $node) => [...$node, ...$context->get($node['id'])])->values()->all(), + 'zones' => $empty->isNotEmpty() && $zone === EnvironmentCanvas::EMPTY_ZONE ? [...$graph['zones'], $this->canvas->emptyZone()] : $graph['zones'], + 'nodes' => $nodes->concat($empty)->values()->all(), 'links' => $graph['links'], - 'positions' => $this->canvas->filterPositions( - $environments->reduce(fn (array $positions, Environment $environment) => [...$positions, ...($environment->canvas_positions ?? [])], []), - $resources->keys()->all(), - ), + 'positions' => [], ]; } /** - * Project and environment of a card: the canvas saves moves and links on that environment canvas. + * Placeholder cards of the project's environments without resources, or one for the project when it has no environments. + * + * @param Collection $environments environments of the project + * @param Collection $resources + * @return Collection> + */ + private function emptyNodes(Project $project, Collection $environments, Collection $resources, string $zone, bool $canAdd): Collection + { + if ($environments->isEmpty()) { + return collect([$this->canvas->emptyNode($project, null, $zone, 'No environments')]); + } + + $used = $resources->pluck('environment_id')->flip(); + + return $environments + ->reject(fn (Environment $environment) => $used->has($environment->id)) + ->map(fn (Environment $environment) => $this->canvas->emptyNode($project, $environment, $zone, $environment->name, $canAdd)) + ->values(); + } + + /** + * Project and environment of a card: the canvas saves links on that environment canvas. * * @return array{project: array{uuid: string, name: string}, environment: array{uuid: string, name: string}} */ diff --git a/app/Services/ProjectIconStorageService.php b/app/Services/ProjectIconStorageService.php index b7a381e109..0c6d314816 100644 --- a/app/Services/ProjectIconStorageService.php +++ b/app/Services/ProjectIconStorageService.php @@ -9,6 +9,14 @@ use RuntimeException; class ProjectIconStorageService extends AvatarStorageService { public function storeProject(Project $project, UploadedFile $upload): void + { + $this->storeProjectContents($project, $this->compress($upload)); + } + + /** + * Stores an icon that is already a compressed JPEG, for example one from a server transfer bundle. + */ + public function storeProjectContents(Project $project, string $contents): void { $settings = instanceSettings(); $storageType = $settings->avatar_storage_type === 's3' && $settings->avatar_s3_storage_id ? 's3' : 'local'; @@ -16,7 +24,7 @@ class ProjectIconStorageService extends AvatarStorageService $disk = $this->disk($storageType, $s3StorageId); $path = "project-icons/{$project->uuid}/icon.jpg"; - if (! $disk->put($path, $this->compress($upload))) { + if (! $disk->put($path, $contents)) { throw new RuntimeException('Unable to store the project icon.'); } diff --git a/app/Services/ScheduledJobDeliveryService.php b/app/Services/ScheduledJobDeliveryService.php index 0d5ddd633a..b2dc7446ea 100644 --- a/app/Services/ScheduledJobDeliveryService.php +++ b/app/Services/ScheduledJobDeliveryService.php @@ -627,8 +627,8 @@ class ScheduledJobDeliveryService return false; } - // An open previous occurrence usually means the previous run is still running. Backup jobs - // then skip this one without running it (see withoutOverlapping()). + // An open previous occurrence usually means the previous run is still running. Backup and + // task jobs then skip this one without running it (see withoutOverlapping()). $previousOpen = ScheduledJobDelivery::query() ->where('schedule_key', $occurrence->schedule_key) ->where('scheduled_for', '<', $occurrence->scheduled_for) diff --git a/app/Services/SentinelTrafficClient.php b/app/Services/SentinelTrafficClient.php index 73c2b70556..1c5c28e2d8 100644 --- a/app/Services/SentinelTrafficClient.php +++ b/app/Services/SentinelTrafficClient.php @@ -48,7 +48,7 @@ class SentinelTrafficClient /** @var array */ private const ALLOWED_DIMENSIONS = [ - 'status', 'method', 'country', 'referer', 'browser', 'os', 'device', 'protocol', 'scheme', 'tls', 'cache', 'bot', 'agent', 'ip', 'useragent', + 'status', 'method', 'country', 'city', 'referer', 'browser', 'os', 'device', 'protocol', 'scheme', 'tls', 'cache', 'bot', 'agent', 'ip', 'useragent', ]; public function __construct(protected Server $server) {} diff --git a/app/Services/ServerCanvas.php b/app/Services/ServerCanvas.php index 061bd2ad2c..d822920a54 100644 --- a/app/Services/ServerCanvas.php +++ b/app/Services/ServerCanvas.php @@ -2,18 +2,22 @@ namespace App\Services; +use App\Enums\ApplicationDeploymentStatus; use App\Models\Application; +use App\Models\ApplicationDeploymentQueue; use App\Models\Environment; use App\Models\ResourceLink; use App\Models\Server; use App\Models\Service; use Illuminate\Database\Eloquent\Model; use Illuminate\Support\Collection; +use Illuminate\Support\Facades\Cache; /** * Builds the server canvas: one zone per server with every resource of the team on it, - * in the same shape as the environment canvas. Cards keep the positions and links saved on their environment canvas; - * admins change them through the environment canvas endpoints of each card. + * in the same shape as the environment canvas. Cards and project / environment groups use the positions saved on + * their environment canvas (node.saved); admins move whole groups and single cards here, which saves the group's cards on that environment + * canvas, and change links through the environment canvas endpoints of each card. */ class ServerCanvas { @@ -21,15 +25,17 @@ class ServerCanvas /** * @param Collection $servers servers of the team to show - * @return array{title: string, title_link: ?string, can_edit: bool, layout: 'server', zones: array>, nodes: array>, links: array>, positions: array} + * @return array{title: string, title_link: ?string, can_edit: bool, can_move: false, can_move_groups: bool, layout: 'server', zones: array>, nodes: array>, links: array>, positions: array{}} */ public function payload(int $teamId, Collection $servers, ?Server $single, bool $canReadSensitive, bool $canEdit = false): array { $environments = Environment::query() ->whereHas('project', fn ($query) => $query->where('team_id', $teamId)) - ->get(['id', 'canvas_positions']); - $resources = $this->loadResources($environments->pluck('id')); - $links = ResourceLink::query()->whereIn('environment_id', $environments->pluck('id'))->orderBy('id')->get(); + ->with('project') + ->get(['id', 'uuid', 'name', 'project_id']); + $environmentIds = $environments->pluck('id'); + $resources = $this->loadResources($environmentIds); + $links = ResourceLink::query()->whereIn('environment_id', $environmentIds)->orderBy('id')->get(); $graph = $this->canvas->graph($resources, $links, $canReadSensitive); $serverUuids = $servers->pluck('uuid')->all(); @@ -52,6 +58,13 @@ class ServerCanvas }) ->values(); $shown = $nodes->pluck('id')->flip(); + // Every-server canvases: environments without resources show as placeholder cards next to the other environments of their project. + $used = $resources->pluck('environment_id')->flip(); + $zoneOf = $this->canvas->projectZones($nodes, $serverUuids[0] ?? EnvironmentCanvas::EMPTY_ZONE); + $empty = $single ? collect() : $environments + ->reject(fn (Environment $environment) => $used->has($environment->id)) + ->map(fn (Environment $environment) => $this->canvas->emptyNode($environment->project, $environment, $zoneOf($environment->project), $environment->project->name.' / '.$environment->name, $canEdit)) + ->values(); return [ 'title' => $single ? $single->name : 'Servers', @@ -59,16 +72,123 @@ class ServerCanvas // Unmanaged containers need a command on the server, so the canvas loads them after it renders. 'containers_endpoint' => $single ? '/servers/'.$single->uuid.'/canvas/containers' : null, 'can_edit' => $canEdit, + 'can_move' => false, + // Admins drag the project / environment group boxes; the canvas saves the moved cards on their environment canvas. + 'can_move_groups' => $canEdit, // Tells the canvas to group cards by project / environment and place linked cards side by side. 'layout' => 'server', - 'zones' => $servers->map(fn (Server $server) => $this->canvas->zone($server, $canReadSensitive))->values()->all(), - 'nodes' => $nodes->all(), + 'zones' => $servers->map(fn (Server $server) => [...$this->canvas->zone($server, $canReadSensitive), 'details' => $this->details($server)]) + ->when($empty->contains('zone', EnvironmentCanvas::EMPTY_ZONE), fn (Collection $zones) => $zones->push($this->canvas->emptyZone())) + ->values()->all(), + 'nodes' => $nodes->concat($empty)->all(), 'links' => collect($graph['links'])->filter(fn (array $link) => $shown->has($link['source']) && $shown->has($link['target']))->values()->all(), - // Card positions saved on the environment canvases; the canvas keeps their arrangement per project / environment group. - 'positions' => $this->canvas->filterPositions( - $environments->reduce(fn (array $positions, Environment $environment) => [...$positions, ...($environment->canvas_positions ?? [])], []), - $nodes->pluck('id')->all(), - ), + 'positions' => [], + ]; + } + + /** + * Host facts and proxy of a server for its zone header. Live usage loads after the canvas renders + * (usage_endpoint), because it runs a command on the server. + * + * @return array{os: ?string, arch: ?string, cpus: ?int, memory_bytes: ?int, uptime_since: ?string, proxy: ?array{type: string, status: ?string}, usage_endpoint: ?string} + */ + public function details(Server $server): array + { + $meta = $server->server_metadata ?? []; + $proxyType = $server->proxyType(); + $live = $server->settings?->is_reachable && $server->settings?->is_metrics_enabled; + + return [ + 'os' => $meta['os'] ?? null, + 'arch' => $meta['arch'] ?? null, + 'cpus' => isset($meta['cpus']) ? (int) $meta['cpus'] : null, + 'memory_bytes' => isset($meta['memory_bytes']) ? (int) $meta['memory_bytes'] : null, + 'uptime_since' => $meta['uptime_since'] ?? null, + 'proxy' => $proxyType && $proxyType !== 'NONE' ? ['type' => $proxyType, 'status' => data_get($server->proxy, 'status')] : null, + 'usage_endpoint' => $live ? '/servers/'.$server->uuid.'/canvas/usage' : null, + ]; + } + + /** + * Latest CPU, memory and root disk usage in percent from Sentinel, cached briefly so that + * canvas refreshes do not run a command on the server each time. Null values: no data. + * + * @return array{cpu: ?float, memory: ?float, disk: ?float} + */ + public function usage(Server $server): array + { + return Cache::remember("server-canvas-usage:{$server->id}", 30, function () use ($server) { + $latest = function (callable $read): ?float { + try { + $value = data_get(last($read() ?: [null]), 1); + + return is_numeric($value) ? round((float) $value, 1) : null; + } catch (\Throwable) { + return null; + } + }; + + return [ + 'cpu' => $latest(fn () => $server->getCpuMetrics(1)), + 'memory' => $latest(fn () => $server->getMemoryMetrics(1)), + 'disk' => $latest(fn () => $server->getDiskMetrics(1)), + ]; + }); + } + + /** + * Dashboard canvas: the server canvas of the team, on each application card its + * running or queued deployment (the oldest first, as the deployment queue runs them), + * and the list of active and recent deployments. + * + * @param Collection $servers servers of the team to show + * @return array + */ + public function dashboardPayload(int $teamId, Collection $servers, bool $canReadSensitive, bool $canEdit = false): array + { + $payload = $this->payload($teamId, $servers, null, $canReadSensitive, $canEdit); + + $columns = ['deployment_uuid', 'application_id', 'application_name', 'status', 'deployment_url', 'pull_request_id', 'server_name', 'created_at', 'finished_at']; + $query = fn () => ApplicationDeploymentQueue::query()->whereIn('server_id', $servers->pluck('id')); + $activeStatuses = [ApplicationDeploymentStatus::IN_PROGRESS->value, ApplicationDeploymentStatus::QUEUED->value]; + $active = $query()->whereIn('status', $activeStatuses)->orderBy('status')->orderBy('id')->get($columns); + $recent = $query()->whereNotIn('status', $activeStatuses)->orderByDesc('id')->limit(8)->get($columns); + $applicationUuids = Application::query() + ->whereIn('id', $active->concat($recent)->pluck('application_id')->unique()) + ->pluck('uuid', 'id'); + $uuidOf = fn (ApplicationDeploymentQueue $deployment) => $applicationUuids->get((int) $deployment->application_id); + $summary = fn (ApplicationDeploymentQueue $deployment) => [ + 'status' => $deployment->status, + 'link' => $deployment->deployment_url, + 'pull_request_id' => (int) $deployment->pull_request_id ?: null, + ]; + $byUuid = $active + ->filter($uuidOf) + ->unique('application_id') + ->mapWithKeys(fn (ApplicationDeploymentQueue $deployment) => [$uuidOf($deployment) => $summary($deployment)]); + + return [ + ...$payload, + 'title' => 'Dashboard', + 'title_link' => route('dashboard'), + 'nodes' => collect($payload['nodes']) + ->map(fn (array $node) => $node['type'] === 'application' && $byUuid->has($node['id']) + ? [...$node, 'deployment' => $byUuid->get($node['id'])] + : $node) + ->all(), + // The deployment panel: running and queued deployments first, then the latest finished ones. + 'deployments' => $active->concat($recent) + ->map(fn (ApplicationDeploymentQueue $deployment) => [ + 'id' => $deployment->deployment_uuid, + 'application' => $uuidOf($deployment), + 'name' => $deployment->application_name, + 'server' => $deployment->server_name, + 'created_at' => $deployment->created_at?->toIso8601String(), + 'finished_at' => $deployment->finished_at?->toIso8601String(), + ...$summary($deployment), + ]) + ->values() + ->all(), ]; } @@ -101,7 +221,6 @@ class ServerCanvas 'urls' => [], 'subtitle' => (string) data_get($container, 'Image'), 'children' => [], - 'volumes' => [], 'link' => route('server.resources', ['server_uuid' => $server->uuid]), 'context' => 'Unmanaged containers', 'context_link' => route('server.resources', ['server_uuid' => $server->uuid]), @@ -120,21 +239,21 @@ class ServerCanvas public function loadResources(Collection $environmentIds): Collection { $applications = Application::query()->whereIn('environment_id', $environmentIds) - ->with(['environment.project', 'destination.server.settings', 'additional_servers.settings', 'persistentStorages']) + ->with(['environment.project', 'destination.server.settings', 'additional_servers.settings']) ->get(); $services = Service::query()->whereIn('environment_id', $environmentIds) ->with([ 'environment.project', 'server.settings', 'destination.server.settings', - 'applications.persistentStorages', - 'databases.persistentStorages', + 'applications', + 'databases', ]) ->get(); $databases = collect(EnvironmentCanvas::DATABASE_RELATIONS) ->flatMap(fn (string $relation) => (new Environment)->{$relation}()->getRelated()->newQuery() ->whereIn('environment_id', $environmentIds) - ->with(['environment.project', 'destination.server.settings', 'persistentStorages']) + ->with(['environment.project', 'destination.server.settings']) ->get()); return collect()->concat($applications)->concat($services)->concat($databases)->keyBy('uuid'); diff --git a/app/Services/ServerTransfer/ServerTransferBundle.php b/app/Services/ServerTransfer/ServerTransferBundle.php index 692842d085..aa5936d14e 100644 --- a/app/Services/ServerTransfer/ServerTransferBundle.php +++ b/app/Services/ServerTransfer/ServerTransferBundle.php @@ -2,16 +2,12 @@ namespace App\Services\ServerTransfer; -use Illuminate\Support\Facades\Crypt; use Illuminate\Validation\ValidationException; -use RuntimeException; class ServerTransferBundle { public const SCHEMA_VERSION = 1; - public const MAILBOX_DIR = '/data/coolify/exports'; - /** * @param array $payload */ @@ -19,6 +15,7 @@ class ServerTransferBundle { return array_merge([ 'schema_version' => self::SCHEMA_VERSION, + 'coolify_version' => config('constants.coolify.version'), 'exported_at' => now()->toIso8601String(), 'export_id' => new_public_id(), ], $payload); @@ -39,6 +36,14 @@ class ServerTransferBundle $errors[] = 'Unsupported schema_version '.$bundle['schema_version'].'. Expected '.self::SCHEMA_VERSION.'.'; } + $bundleVersion = data_get($bundle, 'coolify_version'); + $instanceVersion = (string) config('constants.coolify.version'); + if (! is_string($bundleVersion) || $bundleVersion === '') { + $errors[] = 'Missing coolify_version. Export the server again with Coolify '.$instanceVersion.' or newer.'; + } elseif (version_compare($bundleVersion, $instanceVersion, '>')) { + $errors[] = "This bundle was exported from Coolify {$bundleVersion}. Upgrade this instance (Coolify {$instanceVersion}) to {$bundleVersion} or newer before you import it."; + } + if (! isset($bundle['export_id']) || ! is_string($bundle['export_id']) || $bundle['export_id'] === '') { $errors[] = 'Missing export_id.'; } @@ -90,90 +95,4 @@ class ServerTransferBundle ]); } } - - /** - * Encrypt an entire bundle for mailbox/API transport with a user passphrase. - * - * @param array $bundle - * @return array{encrypted: true, schema_version: int, payload: string} - */ - public static function encryptWithPassphrase(array $bundle, string $passphrase): array - { - if ($passphrase === '') { - throw new RuntimeException('Passphrase must not be empty.'); - } - - $json = json_encode($bundle, JSON_THROW_ON_ERROR); - $key = hash('sha256', $passphrase, true); - $iv = random_bytes(16); - $cipher = openssl_encrypt($json, 'AES-256-CBC', $key, OPENSSL_RAW_DATA, $iv); - if ($cipher === false) { - throw new RuntimeException('Failed to encrypt transfer bundle.'); - } - - $mac = hash_hmac('sha256', $iv.$cipher, $key); - - return [ - 'encrypted' => true, - 'schema_version' => self::SCHEMA_VERSION, - 'payload' => base64_encode($iv.$mac.$cipher), - ]; - } - - /** - * @param array $encrypted - * @return array - */ - public static function decryptWithPassphrase(array $encrypted, string $passphrase): array - { - if (! data_get($encrypted, 'encrypted')) { - throw new RuntimeException('Bundle is not encrypted.'); - } - - $raw = base64_decode((string) data_get($encrypted, 'payload'), true); - if ($raw === false || strlen($raw) < 48) { - throw new RuntimeException('Invalid encrypted payload.'); - } - - $key = hash('sha256', $passphrase, true); - $iv = substr($raw, 0, 16); - $mac = substr($raw, 16, 64); - $cipher = substr($raw, 80); - $expectedMac = hash_hmac('sha256', $iv.$cipher, $key); - if (! hash_equals($expectedMac, $mac)) { - throw new RuntimeException('Invalid passphrase or corrupted bundle.'); - } - - $json = openssl_decrypt($cipher, 'AES-256-CBC', $key, OPENSSL_RAW_DATA, $iv); - if ($json === false) { - throw new RuntimeException('Failed to decrypt transfer bundle.'); - } - - /** @var array $bundle */ - $bundle = json_decode($json, true, 512, JSON_THROW_ON_ERROR); - - return $bundle; - } - - /** - * Optionally re-seal secrets with Laravel's app key for intermediate storage. - * Prefer passphrase encryption for cross-instance transfers. - * - * @param array $bundle - */ - public static function sealWithAppKey(array $bundle): string - { - return Crypt::encryptString(json_encode($bundle, JSON_THROW_ON_ERROR)); - } - - /** - * @return array - */ - public static function unsealWithAppKey(string $sealed): array - { - /** @var array $bundle */ - $bundle = json_decode(Crypt::decryptString($sealed), true, 512, JSON_THROW_ON_ERROR); - - return $bundle; - } } diff --git a/app/Services/ServerTransfer/ServerTransferClaimer.php b/app/Services/ServerTransfer/ServerTransferClaimer.php index 593bdd91ed..f479b1fb49 100644 --- a/app/Services/ServerTransfer/ServerTransferClaimer.php +++ b/app/Services/ServerTransfer/ServerTransferClaimer.php @@ -2,7 +2,16 @@ namespace App\Services\ServerTransfer; +use App\Enums\GithubRunnerStatus; +use App\Jobs\DeleteResourceJob; +use App\Models\GithubRunnerExecution; +use App\Models\ManagedDnsRecord; use App\Models\Server; +use App\Notifications\Internal\GeneralNotification; +use App\Services\Dns\CloudflareDnsProvider; +use App\Services\Dns\ManagedDnsRecordCleanup; +use Illuminate\Database\Eloquent\Model; +use Illuminate\Support\Collection; use Illuminate\Support\Facades\DB; use Illuminate\Support\Facades\Log; use RuntimeException; @@ -10,11 +19,20 @@ use Throwable; class ServerTransferClaimer { + /** + * The managed host records which claim owns it. Both instances can reach the host, so it is the one + * place where an instance can learn that another instance took the server over. + */ + public const OWNER_MARKER_DIR = '/data/coolify/transfer'; + + public const OWNER_MARKER_PATH = self::OWNER_MARKER_DIR.'/owner.json'; + /** * Take management of a server on this Coolify instance and bind Sentinel to it. * * Database ownership (metadata + Sentinel settings) is committed in one transaction so a - * mid-flight failure rolls back. + * mid-flight failure rolls back. The owner marker on the host is written last in that + * transaction: if the host cannot be reached, the claim fails and nothing changes here. * * @return array{ * server_uuid: string, @@ -36,6 +54,7 @@ class ServerTransferClaimer $exportId = data_get($server->server_metadata, 'transfer.export_id'); $claim = [ + 'claim_id' => new_public_id(), 'instance_url' => $instanceUrl, 'server_uuid' => $server->uuid, 'team_id' => $server->team_id, @@ -47,13 +66,16 @@ class ServerTransferClaimer return DB::transaction(function () use ($server, $claim, $instanceUrl) { $server = Server::query()->with('settings')->lockForUpdate()->findOrFail($server->id); + // A disabled server refuses SSH, so the owner marker could not be written. + if ($server->team->serverOverflow()) { + throw new RuntimeException('Your team is over its server limit. Upgrade your subscription or remove a server first.'); + } + if ($server->settings) { $server->settings->sentinel_custom_url = $instanceUrl; $server->settings->ensureValidSentinelToken(); $server->settings->is_sentinel_enabled = true; - if (! $server->team->serverOverflow()) { - $server->settings->force_disabled = false; - } + $server->settings->force_disabled = false; $server->settings->save(); } @@ -66,6 +88,8 @@ class ServerTransferClaimer $server->server_metadata = $metadata; $server->save(); + $this->writeOwnerMarker($server, $claim); + return [ 'server_uuid' => $server->uuid, 'claim' => $claim, @@ -81,18 +105,15 @@ class ServerTransferClaimer * All database writes run in one transaction so partial disable state cannot stick on failure. * Local SSH key cache cleanup runs after commit (filesystem; not transactional). * - * $managementDisabled marks a manual "Disable management" action: the server is not moved yet, - * only ready to be transferred to another instance. - * * @return array{server_uuid: string, message: string} */ - public function markTransferred(Server $server, ?string $exportId = null, ?string $targetInstanceUrl = null, bool $managementDisabled = false): array + public function markTransferred(Server $server, ?string $exportId = null, ?string $targetInstanceUrl = null): array { if ($server->id === 0) { throw new RuntimeException('Cannot transfer the Coolify host itself.'); } - $result = DB::transaction(function () use ($server, $exportId, $targetInstanceUrl, $managementDisabled) { + $result = DB::transaction(function () use ($server, $exportId, $targetInstanceUrl) { $server = Server::query()->with('settings')->lockForUpdate()->findOrFail($server->id); if ($server->settings) { @@ -107,8 +128,15 @@ class ServerTransferClaimer 'export_id' => $exportId ?? data_get($metadata, 'transfer.export_id'), 'target_instance_url' => $targetInstanceUrl, 'transferred_at' => now()->toIso8601String(), - 'management_disabled' => $managementDisabled, ]); + unset($metadata['transfer']['finish'], $metadata['transfer']['taken_back_at']); + $server->server_metadata = $metadata; + $server->save(); + + $this->releaseGithubRunners($server); + + // Kept for a take back. Not owned, so this instance never changes or deletes them at the provider. + $metadata['transfer']['parked_dns_record_ids'] = app(ManagedDnsRecordCleanup::class)->parkResources($this->dnsResources($server)); $server->server_metadata = $metadata; $server->save(); @@ -123,6 +151,144 @@ class ServerTransferClaimer return $result; } + /** + * The runners on the server belong to the target now. Queued runners are cancelled. Started runners are marked + * finished without removing them at GitHub: their jobs keep running, and deleting the archive here later must + * not deregister them. A disabled config also stops new runners for this server from GitHub webhooks. + */ + /** + * The other instance serves the hostnames of a transferred server. Without this, deleting the archive here, + * or the orphaned-record cleanup after it, would delete live DNS records at the provider. + */ + public function forgetDnsRecords(Server $server): void + { + app(ManagedDnsRecordCleanup::class)->forgetResources($this->dnsResources($server)); + } + + /** + * Delete the archive of a transferred server with its resources, records only: another instance manages the + * host, so nothing runs on the server and nothing changes at a provider. + */ + public function deleteArchive(Server $server): void + { + if (! $server->isTransferredAway()) { + throw new RuntimeException('Only the archive of a transferred server can be deleted this way.'); + } + + $this->forgetDnsRecords($server); + foreach ($server->definedResources() as $resource) { + (new DeleteResourceJob($resource, deleteFromCoolifyOnly: true))->handle(); + } + $server->forceDelete(); + } + + /** + * @return Collection + */ + private function dnsResources(Server $server): Collection + { + return $server->applications() + ->concat($server->services()->with('applications')->get()->flatMap->applications); + } + + /** + * After a take back: own the parked DNS records again where Cloudflare confirms them, then start the steps that + * finish the take back (webhooks, runners, records that this instance could not take back). Provider calls run + * here, after the claim committed. + * + * @return list> the steps + */ + public function restoreAfterTakeBack(Server $server): array + { + $dns = app(CloudflareDnsProvider::class); + $parkedIds = (array) data_get($server->server_metadata, 'transfer.parked_dns_record_ids', []); + foreach (ManagedDnsRecord::query()->whereIn('id', $parkedIds)->get() as $record) { + if ($dns->reclaimRecord($record)) { + $record->update(['owned' => true]); + } + } + + DB::transaction(function () use ($server) { + $locked = Server::query()->lockForUpdate()->findOrFail($server->id); + $metadata = $locked->server_metadata ?? []; + unset($metadata['transfer']['parked_dns_record_ids']); + $metadata['transfer']['taken_back_at'] = now()->toIso8601String(); + $locked->server_metadata = $metadata; + $locked->save(); + $server->server_metadata = $metadata; + }); + + return app(TransferFinishChecklist::class)->start($server); + } + + private function releaseGithubRunners(Server $server): void + { + $server->githubRunnerConfig()->update(['is_enabled' => false]); + + $message = 'The server was transferred to another Coolify instance. This instance stopped managing its runners.'; + GithubRunnerExecution::query() + ->where('server_id', $server->id) + ->whereIn('status', GithubRunnerStatus::active()) + ->get() + ->each(fn (GithubRunnerExecution $execution) => $execution->finish(GithubRunnerStatus::Cancelled, $message)); + } + + /** + * Stop managing a claimed server when its owner marker names a claim of another instance. + * A server that was added here without a claim ignores the marker: a marker can be left over on a + * host that an old instance deleted, and the next claim overwrites it. + * + * @return bool true when this instance released the server + */ + public function releaseIfClaimedElsewhere(Server $server, ?string $markerJson): bool + { + if ($server->id === 0 || data_get($server->server_metadata, 'transfer.status') !== 'claimed') { + return false; + } + + $marker = json_decode((string) $markerJson, true); + $markerClaimId = data_get($marker, 'claim_id'); + if (! is_string($markerClaimId) || $markerClaimId === '') { + return false; + } + + if ($markerClaimId === data_get($server->server_metadata, 'transfer.claim.claim_id')) { + return false; + } + + $owner = data_get($marker, 'instance_url'); + $owner = is_string($owner) && $owner !== '' ? $owner : null; + + $this->markTransferred($server, targetInstanceUrl: $owner); + + Log::warning('Server is claimed by another Coolify instance; management stopped here', [ + 'server_uuid' => $server->uuid, + 'owner_instance_url' => $owner, + ]); + $server->team?->notify(new GeneralNotification( + "Server '{$server->name}' is now managed by ".($owner ?? 'another Coolify instance').'. This instance stopped managing it.', + success: false, + )); + + return true; + } + + /** + * @param array $claim + */ + private function writeOwnerMarker(Server $server, array $claim): void + { + $marker = json_encode([ + 'claim_id' => $claim['claim_id'], + 'instance_url' => $claim['instance_url'], + 'server_uuid' => $claim['server_uuid'], + 'claimed_at' => $claim['claimed_at'], + ], JSON_THROW_ON_ERROR | JSON_UNESCAPED_SLASHES); + + instant_remote_process(['mkdir -p '.self::OWNER_MARKER_DIR], $server); + instant_remote_write_file($server, self::OWNER_MARKER_PATH, $marker); + } + /** * Best-effort local SSH mux/key cleanup after transfer (filesystem; not part of DB rollback). * forceDisableServer is idempotent for force_disabled and also clears cached SSH key material. @@ -139,51 +305,4 @@ class ServerTransferClaimer ]); } } - - /** - * Write export bundle to the managed host mailbox (air-gapped transfer). - * - * @param array $bundle - * @return array{path: string, written: bool} - */ - public function writeMailbox(Server $server, array $bundle, ?string $passphrase = null): array - { - ServerTransferBundle::assertValid($bundle); - - $exportId = (string) data_get($bundle, 'export_id', new_public_id()); - $filename = "server-transfer-{$exportId}.coolify.json"; - $path = ServerTransferBundle::MAILBOX_DIR.'/'.$filename; - - $payload = $passphrase - ? ServerTransferBundle::encryptWithPassphrase($bundle, $passphrase) - : $bundle; - - $json = json_encode($payload, JSON_THROW_ON_ERROR | JSON_UNESCAPED_SLASHES); - $b64 = base64_encode($json); - - $written = false; - try { - instant_remote_process([ - 'mkdir -p '.escapeshellarg(ServerTransferBundle::MAILBOX_DIR), - // tee, not a redirect: the SSH user's shell opens a redirect, and a non-root user cannot enter /data/coolify. - 'echo '.escapeshellarg($b64).' | base64 -d | tee '.escapeshellarg($path).' > /dev/null', - 'chmod 600 '.escapeshellarg($path), - 'chown 9999:root '.escapeshellarg($path).' || true', - ], $server, true); - $written = true; - } catch (Throwable $e) { - Log::warning('Failed to write server transfer mailbox', [ - 'server_uuid' => $server->uuid, - 'error' => $e->getMessage(), - ]); - if (! app()->runningUnitTests()) { - throw $e; - } - } - - return [ - 'path' => $path, - 'written' => $written, - ]; - } } diff --git a/app/Services/ServerTransfer/ServerTransferCoverage.php b/app/Services/ServerTransfer/ServerTransferCoverage.php new file mode 100644 index 0000000000..168ffae7c0 --- /dev/null +++ b/app/Services/ServerTransfer/ServerTransferCoverage.php @@ -0,0 +1,347 @@ +> + */ + public const TRANSFERRED = [ + Server::class => ['server'], + ServerSetting::class => ['server.settings'], + PrivateKey::class => ['private_key', 'private_keys.*'], + StandaloneDocker::class => ['destinations.*'], + SwarmDocker::class => ['destinations.*'], + CloudProviderToken::class => ['cloud_provider_tokens.*'], + IntegrationToken::class => ['integration_tokens.*'], + DnsProviderZone::class => ['dns_zones.*'], + ManagedDnsRecord::class => ['managed_dns_records.*'], + ManagedDnsRecordReference::class => ['managed_dns_records.*.references.*'], + SecretManagerLink::class => [ + 'projects.*.environments.*.applications.*.secret_manager_link', + 'projects.*.environments.*.databases.*.secret_manager_link', + 'projects.*.environments.*.services.*.secret_manager_link', + ], + SslCertificate::class => ['ssl_certificates.*'], + GithubApp::class => ['github_apps.*'], + GithubRunnerConfig::class => ['github_runner_config'], + GitlabApp::class => ['gitlab_apps.*'], + S3Storage::class => ['s3_storages.*'], + Project::class => ['projects.*'], + Environment::class => ['projects.*.environments.*'], + ResourceLink::class => ['projects.*.environments.*.resource_links.*'], + SharedEnvironmentVariable::class => [ + 'shared_environment_variables.server.*', + 'shared_environment_variables.team.*', + 'projects.*.shared_environment_variables.*', + 'projects.*.environments.*.shared_environment_variables.*', + ], + Application::class => ['projects.*.environments.*.applications.*'], + ApplicationSetting::class => ['projects.*.environments.*.applications.*.settings'], + ApplicationPreview::class => ['projects.*.environments.*.applications.*.previews.*'], + EnvironmentVariable::class => [ + 'projects.*.environments.*.applications.*.environment_variables.*', + 'projects.*.environments.*.applications.*.environment_variables_preview.*', + 'projects.*.environments.*.databases.*.environment_variables.*', + 'projects.*.environments.*.services.*.environment_variables.*', + ], + LocalPersistentVolume::class => [ + 'projects.*.environments.*.applications.*.persistent_storages.*', + 'projects.*.environments.*.databases.*.persistent_storages.*', + ], + LocalFileVolume::class => [ + 'projects.*.environments.*.applications.*.file_storages.*', + 'projects.*.environments.*.databases.*.file_storages.*', + ], + ScheduledTask::class => [ + 'projects.*.environments.*.applications.*.scheduled_tasks.*', + 'projects.*.environments.*.services.*.scheduled_tasks.*', + ], + Tag::class => [ + 'projects.*.environments.*.applications.*.tags.*', + 'projects.*.environments.*.databases.*.tags.*', + 'projects.*.environments.*.services.*.tags.*', + ], + ScheduledDatabaseBackup::class => ['projects.*.environments.*.databases.*.scheduled_backups.*'], + ScheduledVolumeBackup::class => ['volume_backups.*'], + Service::class => ['projects.*.environments.*.services.*'], + ServiceApplication::class => ['projects.*.environments.*.services.*.applications.*'], + ServiceDatabase::class => ['projects.*.environments.*.services.*.databases.*'], + StandalonePostgresql::class => ['projects.*.environments.*.databases.*'], + StandaloneMysql::class => ['projects.*.environments.*.databases.*'], + StandaloneMariadb::class => ['projects.*.environments.*.databases.*'], + StandaloneMongodb::class => ['projects.*.environments.*.databases.*'], + StandaloneRedis::class => ['projects.*.environments.*.databases.*'], + StandaloneKeydb::class => ['projects.*.environments.*.databases.*'], + StandaloneDragonfly::class => ['projects.*.environments.*.databases.*'], + StandaloneClickhouse::class => ['projects.*.environments.*.databases.*'], + StandaloneSqlite::class => ['projects.*.environments.*.databases.*'], + ]; + + private const LINKED = 'The bundle links the record by UUID or by its place in the bundle; the target sets its own ID.'; + + private const TEAM = 'The target imports into one of its own teams.'; + + private const RUNTIME = 'Runtime state: the target sets it again when it manages the server.'; + + /** + * Columns of transferred models that are not in the bundle under their own name, with the reason. + * Columns in ALWAYS_NOT_EXPORTED are left out of every model. + * + * @var array> + */ + public const NOT_EXPORTED_COLUMNS = [ + Server::class => [ + 'team_id' => self::TEAM, 'private_key_id' => self::LINKED, 'cloud_provider_token_id' => self::LINKED, + 'unreachable_count' => self::RUNTIME, 'high_disk_usage_notification_sent' => self::RUNTIME, + 'log_drain_notification_sent' => self::RUNTIME, 'validation_logs' => self::RUNTIME, + 'sentinel_updated_at' => self::RUNTIME, 'sentinel_waiting_since' => self::RUNTIME, 'ip_previous' => self::RUNTIME, + 'is_validating' => self::RUNTIME, 'detected_traefik_version' => self::RUNTIME, 'traefik_outdated_info' => self::RUNTIME, + 'hetzner_server_status' => self::RUNTIME, 'vultr_instance_status' => self::RUNTIME, + 'digitalocean_droplet_status' => self::RUNTIME, 'hostinger_virtual_machine_status' => self::RUNTIME, + 'server_metadata' => 'Facts about the host and the transfer state of the source; the target collects its own.', + ], + ServerSetting::class => [ + 'server_id' => self::LINKED, 'is_reachable' => self::RUNTIME, 'is_usable' => self::RUNTIME, 'force_disabled' => self::RUNTIME, + 'is_sentinel_enabled' => 'The target enables Sentinel when it claims the server.', + 'is_sentinel_debug_enabled' => 'The target enables Sentinel when it claims the server.', + 'sentinel_token' => 'The target makes a new Sentinel token when it claims the server.', + 'sentinel_custom_url' => 'The target sets its own URL when it claims the server.', + 'cloudflare_tunnel_integration_token_id' => 'Exported as cloudflare_tunnel_integration_token_uuid. '.self::LINKED, + ], + PrivateKey::class => ['team_id' => self::TEAM], + StandaloneDocker::class => ['server_id' => self::LINKED], + SwarmDocker::class => ['server_id' => self::LINKED], + CloudProviderToken::class => ['team_id' => self::TEAM], + IntegrationToken::class => ['team_id' => self::TEAM], + DnsProviderZone::class => ['integration_token_id' => self::LINKED], + ManagedDnsRecord::class => ['team_id' => self::TEAM, 'integration_token_id' => self::LINKED, 'dns_provider_zone_id' => self::LINKED], + ManagedDnsRecordReference::class => [ + 'managed_dns_record_id' => 'The reference is part of its record in the bundle.', + 'resource_type' => self::LINKED, 'resource_id' => self::LINKED, + ], + SecretManagerLink::class => ['resourceable_type' => self::LINKED, 'resourceable_id' => self::LINKED, 'integration_token_id' => self::LINKED], + SslCertificate::class => ['server_id' => self::LINKED, 'resource_type' => self::LINKED, 'resource_id' => self::LINKED], + GithubApp::class => ['team_id' => self::TEAM, 'private_key_id' => self::LINKED], + GithubRunnerConfig::class => [ + 'server_id' => self::LINKED, + 'github_app_id' => 'Exported as github_app_uuid. '.self::LINKED, + ], + GitlabApp::class => ['team_id' => self::TEAM, 'private_key_id' => self::LINKED], + S3Storage::class => ['team_id' => self::TEAM, 'unusable_email_sent' => self::RUNTIME], + Project::class => [ + 'team_id' => self::TEAM, + 'icon_path' => 'The icon file is exported as `icon` (base64 content). The target stores it in its own icon storage and sets the path.', + 'icon_storage_type' => 'The target uses its own default icon storage. See icon_path.', + 'icon_s3_storage_id' => 'The target uses its own default icon storage. See icon_path.', + ], + Environment::class => ['project_id' => self::LINKED], + ResourceLink::class => [ + 'environment_id' => 'The link is part of its environment in the bundle.', + 'source_type' => 'Exported as source.kind and source.uuid. '.self::LINKED, + 'source_id' => 'Exported as source.kind and source.uuid. '.self::LINKED, + 'target_type' => 'Exported as target.kind and target.uuid. '.self::LINKED, + 'target_id' => 'Exported as target.kind and target.uuid. '.self::LINKED, + 'created_by' => 'A user of the source instance; the target leaves it empty.', + ], + SharedEnvironmentVariable::class => [ + 'type' => 'The scope follows from the place of the variable in the bundle.', + 'team_id' => self::TEAM, 'project_id' => self::LINKED, 'environment_id' => self::LINKED, 'server_id' => self::LINKED, + 'version' => 'The Coolify version that created the variable; the target sets its own.', + ], + Application::class => [ + 'environment_id' => self::LINKED, 'destination_type' => self::LINKED, 'destination_id' => self::LINKED, + 'source_type' => self::LINKED, 'source_id' => self::LINKED, 'private_key_id' => self::LINKED, + ], + ApplicationSetting::class => ['application_id' => self::LINKED], + ApplicationPreview::class => [ + 'application_id' => self::LINKED, 'last_online_at' => self::RUNTIME, 'domain_dns_statuses' => self::RUNTIME, + 'restart_count' => self::RUNTIME, 'restart_limit_reached' => self::RUNTIME, 'last_restart_at' => self::RUNTIME, + 'last_restart_type' => self::RUNTIME, + ], + EnvironmentVariable::class => ['resourceable_type' => self::LINKED, 'resourceable_id' => self::LINKED], + LocalPersistentVolume::class => [ + 'resource_type' => self::LINKED, 'resource_id' => self::LINKED, 'standalone_sqlite_id' => self::LINKED, + 'container_id' => 'Not used: only listed in $fillable.', + ], + LocalFileVolume::class => ['resource_type' => self::LINKED, 'resource_id' => self::LINKED], + ScheduledTask::class => ['team_id' => self::TEAM, 'application_id' => self::LINKED, 'service_id' => self::LINKED], + Tag::class => ['team_id' => self::TEAM], + ScheduledDatabaseBackup::class => [ + 'team_id' => self::TEAM, 'database_type' => self::LINKED, 'database_id' => self::LINKED, 's3_storage_id' => self::LINKED, + 'last_execution_at' => 'Execution history of the source.', + ], + ScheduledVolumeBackup::class => [ + 'team_id' => self::TEAM, 'backupable_type' => self::LINKED, 'backupable_id' => self::LINKED, 's3_storage_id' => self::LINKED, + ], + Service::class => [ + 'environment_id' => self::LINKED, 'server_id' => self::LINKED, 'destination_type' => self::LINKED, 'destination_id' => self::LINKED, + ], + ServiceApplication::class => ['service_id' => self::LINKED], + ServiceDatabase::class => ['service_id' => self::LINKED], + StandalonePostgresql::class => [ + 'destination_type' => self::LINKED, 'destination_id' => self::LINKED, 'environment_id' => self::LINKED, + 'config_hash' => 'Calculated again when the target starts the database.', + ], + StandaloneMysql::class => [ + 'destination_type' => self::LINKED, 'destination_id' => self::LINKED, 'environment_id' => self::LINKED, + 'config_hash' => 'Calculated again when the target starts the database.', + ], + StandaloneMariadb::class => [ + 'destination_type' => self::LINKED, 'destination_id' => self::LINKED, 'environment_id' => self::LINKED, + 'config_hash' => 'Calculated again when the target starts the database.', + ], + StandaloneMongodb::class => [ + 'destination_type' => self::LINKED, 'destination_id' => self::LINKED, 'environment_id' => self::LINKED, + 'config_hash' => 'Calculated again when the target starts the database.', + ], + StandaloneRedis::class => [ + 'destination_type' => self::LINKED, 'destination_id' => self::LINKED, 'environment_id' => self::LINKED, + 'config_hash' => 'Calculated again when the target starts the database.', + ], + StandaloneKeydb::class => [ + 'destination_type' => self::LINKED, 'destination_id' => self::LINKED, 'environment_id' => self::LINKED, + 'config_hash' => 'Calculated again when the target starts the database.', + ], + StandaloneDragonfly::class => [ + 'destination_type' => self::LINKED, 'destination_id' => self::LINKED, 'environment_id' => self::LINKED, + 'config_hash' => 'Calculated again when the target starts the database.', + ], + StandaloneClickhouse::class => [ + 'destination_type' => self::LINKED, 'destination_id' => self::LINKED, 'environment_id' => self::LINKED, + 'config_hash' => 'Calculated again when the target starts the database.', + ], + StandaloneSqlite::class => [ + 'destination_type' => self::LINKED, 'destination_id' => self::LINKED, 'environment_id' => self::LINKED, + 'config_hash' => 'Calculated again when the target starts the database.', + 'is_public' => 'Not used: SQLite has no public port.', + ], + ]; + + /** + * Created on the target instance for every record. + * + * @var list + */ + public const ALWAYS_NOT_EXPORTED = ['id', 'created_at', 'updated_at', 'deleted_at']; + + /** + * Models that a transfer does not move, with the reason. + * + * @var array + */ + public const NOT_TRANSFERRED = [ + Team::class => 'Instance-wide: the target imports into one of its own teams.', + User::class => 'Instance-wide: users are not moved with a server.', + TeamInvitation::class => 'Team data, not server data.', + Subscription::class => 'Billing of the source instance.', + InstanceSettings::class => 'Settings of the source instance.', + OauthSetting::class => 'Settings of the source instance.', + OauthIdentity::class => 'User login data of the source instance.', + PersonalAccessToken::class => 'API tokens of the source instance.', + UserChangelogRead::class => 'User data of the source instance.', + AuditEvent::class => 'Audit log of the source instance.', + NotificationThrottle::class => 'Runtime state of the source instance.', + ScheduledJobState::class => 'Scheduler state of the source instance.', + ScheduledJobDelivery::class => 'Scheduler state of the source instance.', + EmailNotificationSettings::class => 'Team notification settings, not server data.', + DiscordNotificationSettings::class => 'Team notification settings, not server data.', + SlackNotificationSettings::class => 'Team notification settings, not server data.', + TelegramNotificationSettings::class => 'Team notification settings, not server data.', + PushoverNotificationSettings::class => 'Team notification settings, not server data.', + WebhookNotificationSettings::class => 'Team notification settings, not server data.', + CloudInitScript::class => 'Team server-creation templates, not server data.', + ProjectSetting::class => 'Created with the project on the target; it has no settings of its own yet.', + ApplicationDeploymentQueue::class => 'Deployment history.', + ScheduledTaskExecution::class => 'Execution history.', + ScheduledDatabaseBackupExecution::class => 'Execution history.', + ScheduledVolumeBackupExecution::class => 'Execution history.', + DatabaseBackupS3Replica::class => 'Backup history.', + VolumeBackupS3Replica::class => 'Backup history.', + DockerCleanupExecution::class => 'Execution history.', + GithubRunnerExecution::class => 'Execution history. The source stops tracking the runners of the server when it is transferred.', + ]; +} diff --git a/app/Services/ServerTransfer/ServerTransferExporter.php b/app/Services/ServerTransfer/ServerTransferExporter.php index 176aa8d30d..66895c4209 100644 --- a/app/Services/ServerTransfer/ServerTransferExporter.php +++ b/app/Services/ServerTransfer/ServerTransferExporter.php @@ -5,12 +5,17 @@ namespace App\Services\ServerTransfer; use App\Models\Application; use App\Models\ApplicationPreview; use App\Models\CloudProviderToken; +use App\Models\DnsProviderZone; use App\Models\Environment; use App\Models\EnvironmentVariable; use App\Models\GithubApp; +use App\Models\GithubRunnerConfig; use App\Models\GitlabApp; +use App\Models\IntegrationToken; use App\Models\LocalFileVolume; use App\Models\LocalPersistentVolume; +use App\Models\ManagedDnsRecord; +use App\Models\ManagedDnsRecordReference; use App\Models\PrivateKey; use App\Models\Project; use App\Models\S3Storage; @@ -23,14 +28,22 @@ use App\Models\SharedEnvironmentVariable; use App\Models\StandaloneDocker; use App\Models\SwarmDocker; use App\Models\Tag; +use App\Services\ProjectIconStorageService; use Illuminate\Database\Eloquent\Model; use Illuminate\Support\Collection; use Illuminate\Support\Facades\DB; use Illuminate\Support\Facades\Schema; +use Illuminate\Support\Str; use RuntimeException; +use Throwable; class ServerTransferExporter { + /** + * Larger project icons are not put in the bundle. Icons that Coolify stores are small JPEG files. + */ + public const MAX_PROJECT_ICON_BYTES = 2 * 1024 * 1024; + /** * Settings that are instance-bound or runtime health state and must not transfer. * @@ -48,6 +61,8 @@ class ServerTransferExporter 'sentinel_custom_url', 'is_sentinel_enabled', 'is_sentinel_debug_enabled', + // Exported as cloudflare_tunnel_integration_token_uuid. + 'cloudflare_tunnel_integration_token_id', ]; /** @@ -86,6 +101,9 @@ class ServerTransferExporter if (! $privateKey instanceof PrivateKey) { throw new RuntimeException('Server has no private key to export.'); } + if ((int) $privateKey->team_id !== (int) $server->team_id) { + throw new RuntimeException('The server private key does not belong to the server team. Select a private key of this team first.'); + } if (! $includeSensitive) { throw new RuntimeException('Server transfer export requires sensitive data access (read:sensitive).'); @@ -100,15 +118,36 @@ class ServerTransferExporter $this->assertNoAdditionalDestinations($server, $applications); $projectIds = $this->collectProjectIds($applications, $databases, $services); - $projects = Project::query()->whereIn('id', $projectIds)->orderBy('name')->get(); + $projects = Project::query()->where('team_id', $server->team_id)->whereIn('id', $projectIds)->orderBy('name')->get(); + if ($projects->count() !== collect($projectIds)->unique()->count()) { + throw new RuntimeException('Some resources on this server belong to a project of another team. Move them to a project of this team first.'); + } $destinationUuidByKey = $this->destinationUuidMap($server); - $dependencies = $this->collectDependencies($server, $applications, $databases, $services); + $runnerConfig = $server->githubRunnerConfig()->with('githubApp')->first(); + $dependencies = $this->collectDependencies($server, $applications, $databases, $services, $runnerConfig); $sourceInstanceUrl = rtrim((string) (instanceSettings()->fqdn ?: config('app.url')), '/'); - $warnings = $this->buildWarnings($applications, $dependencies, $sourceInstanceUrl); - if ($secretManagerWarning = $this->secretManagerWarning($applications->concat($databases)->concat($services))) { - $warnings[] = $secretManagerWarning; + $managedDns = $this->collectManagedDnsRecords($server, $applications, $services); + $integrationTokens = $this->collectIntegrationTokens($server, $applications->concat($databases)->concat($services), $managedDns['records']); + + $serverSharedEnvVars = $this->exportSharedEnvVars( + SharedEnvironmentVariable::query() + ->where('type', 'server') + ->where('server_id', $server->id) + ->get() + ); + $projectWarnings = []; + $leftOutLinks = 0; + $projectPayloads = $projects->map(function (Project $project) use ($destinationUuidByKey, $applications, $databases, $services, &$projectWarnings, &$leftOutLinks) { + return $this->exportProject($project, $destinationUuidByKey, $applications, $databases, $services, $projectWarnings, $leftOutLinks); + })->values()->all(); + + $warnings = $this->buildWarnings($applications, $dependencies); + $warnings = array_merge($warnings, $managedDns['warnings'], $projectWarnings); + if ($leftOutLinks > 0) { + $warnings[] = $leftOutLinks.' '.Str::plural('link', $leftOutLinks).' between resources on the environment canvas ' + .($leftOutLinks === 1 ? 'is' : 'are').' not transferred, because one of the linked resources stays on this instance.'; } $payload = ServerTransferBundle::wrap([ @@ -117,6 +156,8 @@ class ServerTransferExporter 'name' => config('app.name'), ], 'warnings' => $warnings, + // Manual steps after the transfer. The source replaces {target_url} with the URL of the target. + 'post_transfer_checklist' => $this->postTransferChecklist($dependencies, $runnerConfig, $applications), // Back-compat: single server key 'private_key' => $this->exportPrivateKey($privateKey), // All keys needed by server, apps, and git sources @@ -125,21 +166,19 @@ class ServerTransferExporter 'gitlab_apps' => $dependencies['gitlab_apps']->map(fn (GitlabApp $app) => $this->exportGitlabApp($app))->values()->all(), 's3_storages' => $dependencies['s3_storages']->map(fn (S3Storage $storage) => $this->exportS3Storage($storage))->values()->all(), 'cloud_provider_tokens' => $dependencies['cloud_provider_tokens']->map(fn (CloudProviderToken $token) => $this->exportCloudProviderToken($token))->values()->all(), + 'integration_tokens' => $integrationTokens->map(fn (IntegrationToken $token) => $this->exportIntegrationToken($token))->values()->all(), + 'dns_zones' => $managedDns['records']->pluck('zone')->unique('id')->map(fn (DnsProviderZone $zone) => $this->exportDnsZone($zone))->values()->all(), + 'managed_dns_records' => $managedDns['records']->map(fn (ManagedDnsRecord $record) => $this->exportManagedDnsRecord($record, $managedDns['resource_keys']))->values()->all(), 'ssl_certificates' => $this->exportSslCertificates($server), 'volume_backups' => $this->exportVolumeBackups($applications, $databases, $services), 'server' => $this->exportServer($server), 'destinations' => $this->exportDestinations($server), + 'github_runner_config' => $runnerConfig ? $this->exportGithubRunnerConfig($runnerConfig) : null, 'shared_environment_variables' => [ - 'server' => $this->exportSharedEnvVars( - SharedEnvironmentVariable::query() - ->where('type', 'server') - ->where('server_id', $server->id) - ->get() - ), + 'server' => $serverSharedEnvVars, + 'team' => $this->exportReferencedTeamSharedEnvVars($server->team_id, $serverSharedEnvVars, $projectPayloads), ], - 'projects' => $projects->map(function (Project $project) use ($server, $destinationUuidByKey, $applications, $databases, $services) { - return $this->exportProject($project, $server, $destinationUuidByKey, $applications, $databases, $services); - })->values()->all(), + 'projects' => $projectPayloads, ]); return $payload; @@ -157,47 +196,46 @@ class ServerTransferExporter * cloud_provider_tokens: Collection * } */ - private function collectDependencies(Server $server, Collection $applications, Collection $databases, Collection $services): array + private function collectDependencies(Server $server, Collection $applications, Collection $databases, Collection $services, ?GithubRunnerConfig $runnerConfig): array { + $teamId = $server->team_id; $privateKeys = collect([$server->privateKey])->filter(); $githubApps = collect(); $gitlabApps = collect(); $s3Ids = collect(); + // System-wide GitHub Apps are instance-owned; do not copy them. The application source and the runner + // config still record the App uuid, so the import can re-link to a system-wide App of the target. + $collectGithubApp = function (int|string $id) use ($teamId, $githubApps, $privateKeys): void { + $gh = $this->findGitSource(GithubApp::class, $id, $teamId); + if ($gh && ! $gh->is_system_wide) { + $githubApps->push($gh); + if ($gh->private_key_id) { + $privateKeys->push($this->findTeamRecord(PrivateKey::class, $gh->private_key_id, $teamId, 'private key')); + } + } + }; + + if ($runnerConfig) { + $collectGithubApp($runnerConfig->github_app_id); + } + foreach ($applications as $application) { if ($application->private_key_id) { - $key = PrivateKey::find($application->private_key_id); - if ($key) { - $privateKeys->push($key); - } + $privateKeys->push($this->findTeamRecord(PrivateKey::class, $application->private_key_id, $teamId, 'private key')); } if ($application->source_type === GithubApp::class && $application->source_id !== null) { - $gh = GithubApp::find($application->source_id); - // System-wide GitHub Apps are instance-owned; do not copy them. - // Application export still records source.uuid so import can re-link - // to an existing system-wide app on the target instance. - if ($gh && ! $gh->is_system_wide) { - $githubApps->push($gh); - if ($gh->private_key_id) { - $pk = PrivateKey::find($gh->private_key_id); - if ($pk) { - $privateKeys->push($pk); - } - } - } + $collectGithubApp($application->source_id); } if ($application->source_type === GitlabApp::class && $application->source_id !== null) { - $gl = GitlabApp::find($application->source_id); + $gl = $this->findGitSource(GitlabApp::class, $application->source_id, $teamId); // System-wide GitLab Apps are instance-owned; re-link by UUID on import. if ($gl && ! $gl->is_system_wide) { $gitlabApps->push($gl); if ($gl->private_key_id) { - $pk = PrivateKey::find($gl->private_key_id); - if ($pk) { - $privateKeys->push($pk); - } + $privateKeys->push($this->findTeamRecord(PrivateKey::class, $gl->private_key_id, $teamId, 'private key')); } } } @@ -264,37 +302,75 @@ class ServerTransferExporter $cloudTokens = collect(); if ($server->cloud_provider_token_id) { - $token = CloudProviderToken::find($server->cloud_provider_token_id); - if ($token) { - $cloudTokens->push($token); - } + $cloudTokens->push($this->findTeamRecord(CloudProviderToken::class, $server->cloud_provider_token_id, $teamId, 'cloud provider token')); } return [ 'private_keys' => $privateKeys->unique('id')->values(), 'github_apps' => $githubApps->unique('id')->values(), 'gitlab_apps' => $gitlabApps->unique('id')->values(), - 's3_storages' => S3Storage::query()->whereIn('id', $s3Ids->unique()->filter()->all())->get(), + 's3_storages' => $this->findTeamS3Storages($s3Ids->unique()->filter()->values()->all(), $teamId), 'cloud_provider_tokens' => $cloudTokens->unique('id')->values(), ]; } + /** + * Load a credential record that must belong to the server team. A record of another team is + * never exported, so a broken reference stops the export instead of leaking its secrets. + * + * @template T of Model + * + * @param class-string $modelClass + * @return T + */ + private function findTeamRecord(string $modelClass, int|string $id, int $teamId, string $label): Model + { + $record = $modelClass::query()->where('team_id', $teamId)->find($id); + if (! $record) { + throw new RuntimeException("A {$label} used on this server does not belong to the server team. Select a {$label} of this team first."); + } + + return $record; + } + + /** + * System-wide git sources are returned as they are (they are only linked by UUID, never exported). + * + * @param class-string $modelClass + */ + private function findGitSource(string $modelClass, int|string $id, int $teamId): GithubApp|GitlabApp|null + { + $source = $modelClass::find($id); + if ($source && ! $source->is_system_wide && (int) $source->team_id !== $teamId) { + throw new RuntimeException('A git source used on this server does not belong to the server team. Select a git source of this team first.'); + } + + return $source; + } + + /** + * @param list $ids + * @return Collection + */ + private function findTeamS3Storages(array $ids, int $teamId): Collection + { + $storages = S3Storage::query()->where('team_id', $teamId)->whereIn('id', $ids)->get(); + if ($storages->count() !== count($ids)) { + throw new RuntimeException('An S3 storage used by a backup on this server does not belong to the server team. Select an S3 storage of this team first.'); + } + + return $storages; + } + /** * @param Collection $applications * @param array $dependencies * @return list */ - private function buildWarnings(Collection $applications, array $dependencies, string $sourceInstanceUrl): array + private function buildWarnings(Collection $applications, array $dependencies): array { $warnings = []; - if ($dependencies['github_apps']->isNotEmpty()) { - $warnings[] = 'Team-scoped GitHub Apps were exported with credentials (system-wide apps are skipped). After import, update each GitHub App webhook URL (and setup/callback URL if used) to this Coolify instance so push/PR automations keep working. ' - ."Webhook path: {$sourceInstanceUrl}/webhooks/source/github/events " - .'(replace host with the target instance FQDN after import). ' - .'Also reinstall/refresh the GitHub App installation if installation_id is invalid on the new host.'; - } - $usesSystemWideGithub = $applications->contains(function (Application $app) { if ($app->source_type !== GithubApp::class || $app->source_id === null) { return false; @@ -319,11 +395,6 @@ class ServerTransferExporter $warnings[] = 'Some applications use a system-wide GitLab App, which is not exported. On import, Coolify will re-link to a matching system-wide GitLab App on the target instance (by UUID) if one exists.'; } - if ($dependencies['gitlab_apps']->isNotEmpty()) { - $warnings[] = 'Team-scoped GitLab Apps/OAuth sources were exported (system-wide apps are skipped). After import, update GitLab webhook URLs to the target Coolify instance ' - ."({$sourceInstanceUrl}/webhooks/source/gitlab/events — use the new FQDN) and refresh OAuth tokens if needed."; - } - if ($dependencies['s3_storages']->isNotEmpty()) { $warnings[] = 'S3 storage credentials were exported. Confirm endpoint access from the target instance and that bucket policies still apply.'; } @@ -332,24 +403,180 @@ class ServerTransferExporter } /** - * Secret manager links point to integration tokens of this instance, which are not exported. + * Managed DNS records that only exported resources use. A record that a resource staying on this instance + * also uses is not exported: this instance keeps managing it. * - * @param Collection $resources + * @param Collection $applications + * @param Collection $services + * @return array{ + * records: Collection, + * resource_keys: array, + * warnings: list + * } */ - private function secretManagerWarning(Collection $resources): ?string + private function collectManagedDnsRecords(Server $server, Collection $applications, Collection $services): array { - $linkedNames = $resources - ->filter(fn (Model $resource) => method_exists($resource, 'secretManagerLink') && $resource->secretManagerLink()->exists()) - ->pluck('name') - ->unique() - ->values(); - - if ($linkedNames->isEmpty()) { - return null; + // Keyed by "morph type:id". Soft-deleted previews are covered, but their references are not exported. + $resourceKeys = []; + foreach ($applications as $application) { + $resourceKeys[$application->getMorphClass().':'.$application->id] = ['kind' => 'application', 'uuid' => $application->uuid]; + foreach (ApplicationPreview::withTrashed()->where('application_id', $application->id)->get() as $preview) { + $resourceKeys[$preview->getMorphClass().':'.$preview->id] = $preview->trashed() ? null : ['kind' => 'preview', 'uuid' => $preview->uuid]; + } + } + foreach ($services as $service) { + foreach ($service->applications as $serviceApplication) { + $resourceKeys[$serviceApplication->getMorphClass().':'.$serviceApplication->id] = ['kind' => 'service_application', 'uuid' => $serviceApplication->uuid]; + } + } + if ($resourceKeys === []) { + return ['records' => collect(), 'resource_keys' => [], 'warnings' => []]; } - return 'Some resources load environment variables from a secret manager: '.$linkedNames->implode(', ').'. ' - .'Secret manager tokens and links are not exported. After import, add the integration token on the target instance and link the secret manager source again, or deployments of these resources fail.'; + $records = ManagedDnsRecord::query() + ->whereHas('references', function ($query) use ($resourceKeys) { + $query->where(function ($query) use ($resourceKeys) { + foreach (array_keys($resourceKeys) as $key) { + [$type, $id] = explode(':', $key, 2); + $query->orWhere(fn ($query) => $query->where('resource_type', $type)->where('resource_id', $id)); + } + }); + }) + ->with(['references', 'zone.integrationToken', 'integrationToken']) + ->orderBy('name') + ->get(); + + $exported = collect(); + $warnings = []; + foreach ($records as $record) { + $teamId = (int) $server->team_id; + if ((int) $record->team_id !== $teamId + || (int) $record->integrationToken?->team_id !== $teamId + || (int) $record->zone?->integrationToken?->team_id !== $teamId) { + $warnings[] = "DNS record {$record->name} is not transferred, because its DNS provider token belongs to another team. The target instance does not manage it."; + + continue; + } + $sharedWithOtherResources = $record->references + ->contains(fn (ManagedDnsRecordReference $reference) => ! array_key_exists($reference->resource_type.':'.$reference->resource_id, $resourceKeys)); + if ($sharedWithOtherResources) { + $warnings[] = "DNS record {$record->name} stays managed by this instance, because a resource that is not transferred also uses it. The target instance does not update or delete it."; + + continue; + } + $hasLiveReference = $record->references + ->contains(fn (ManagedDnsRecordReference $reference) => $resourceKeys[$reference->resource_type.':'.$reference->resource_id] !== null); + if ($hasLiveReference) { + $exported->push($record); + } + } + + return ['records' => $exported, 'resource_keys' => $resourceKeys, 'warnings' => $warnings]; + } + + /** + * Integration tokens that the exported resources need: secret manager links, managed DNS records, and the + * Cloudflare tunnel of the server. + * + * @param Collection $resources + * @param Collection $dnsRecords + * @return Collection + */ + private function collectIntegrationTokens(Server $server, Collection $resources, Collection $dnsRecords): Collection + { + $tokenIds = collect([$server->settings?->cloudflare_tunnel_integration_token_id]); + foreach ($resources as $resource) { + if (method_exists($resource, 'secretManagerLink')) { + $tokenIds->push($resource->secretManagerLink()->value('integration_token_id')); + } + } + foreach ($dnsRecords as $record) { + $tokenIds->push($record->integration_token_id, $record->zone->integration_token_id); + } + + return $tokenIds->filter()->unique()->values() + ->map(fn (int|string $id) => $this->findTeamRecord(IntegrationToken::class, $id, (int) $server->team_id, 'integration token')); + } + + /** + * Manual steps after the transfer. {target_url} stands for the URL of the target instance. + * + * @param array{github_apps: Collection, gitlab_apps: Collection} $dependencies + * @param Collection $applications + * @return list + */ + private function postTransferChecklist(array $dependencies, ?GithubRunnerConfig $runnerConfig, Collection $applications): array + { + $steps = []; + // A manual webhook is set in the repository itself, so its URL still points to this instance. Every + // application has the secrets, so list the git applications that do not deploy through a private App. + $manualWebhookApps = $applications + ->filter(fn (Application $application) => $application->build_pack !== 'dockerimage' && filled($application->git_repository) + && (! $application->source || data_get($application->source, 'is_public') === true)) + ->pluck('name'); + if ($manualWebhookApps->isNotEmpty()) { + $steps[] = 'If a repository of these applications has a manual webhook to Coolify, change its URL to the target: ' + .$manualWebhookApps->implode(', ').'. Use {target_url}/webhooks/source/github/events/manual, ' + .'{target_url}/webhooks/source/gitlab/events/manual, {target_url}/webhooks/source/bitbucket/events/manual, ' + .'or {target_url}/webhooks/source/gitea/events/manual.'; + } + foreach ($dependencies['github_apps'] as $githubApp) { + $steps[] = "On GitHub, open the settings of GitHub App {$githubApp->name}. Set the Webhook URL to {target_url}/webhooks/source/github/events, " + .'the Setup URL to {target_url}/webhooks/source/github/install, and the Callback URL to {target_url}/login/github/app.'; + } + foreach ($dependencies['gitlab_apps'] as $gitlabApp) { + $steps[] = "In GitLab, open Settings > Webhooks of every project that uses GitLab App {$gitlabApp->name}, and set the webhook URL to {target_url}/webhooks/source/gitlab/events."; + } + if ($runnerConfig) { + $steps[] = 'After the GitHub App sends webhooks to {target_url}, open this server there, check the runner requirements, and enable GitHub runners. ' + .'Enabling them stops runners that another Coolify instance started on this server.'; + } + + return $steps; + } + + /** + * Team variables are exported only when an exported variable references them as {{team.KEY}}. + * + * @param list> $serverVariables + * @param list> $projects + * @return list> + */ + private function exportReferencedTeamSharedEnvVars(int $teamId, array $serverVariables, array $projects): array + { + $values = collect($serverVariables)->pluck('value'); + foreach ([ + '*.shared_environment_variables.*.value', + '*.environments.*.shared_environment_variables.*.value', + '*.environments.*.applications.*.environment_variables.*.value', + '*.environments.*.applications.*.environment_variables_preview.*.value', + '*.environments.*.databases.*.environment_variables.*.value', + '*.environments.*.services.*.environment_variables.*.value', + '*.environments.*.services.*.applications.*.environment_variables.*.value', + ] as $path) { + $values = $values->merge(data_get($projects, $path, [])); + } + + $keys = $values + ->filter(fn ($value) => is_string($value)) + ->flatMap(function (string $value) { + preg_match_all('/{{\s*team\.(.+?)\s*}}/', $value, $matches); + + return $matches[1]; + }) + ->unique() + ->values(); + if ($keys->isEmpty()) { + return []; + } + + return $this->exportSharedEnvVars( + SharedEnvironmentVariable::query() + ->where('type', 'team') + ->where('team_id', $teamId) + ->whereIn('key', $keys) + ->get() + ); } /** @@ -359,7 +586,7 @@ class ServerTransferExporter { try { $material = $key->private_key; - } catch (\Throwable $e) { + } catch (Throwable $e) { throw new RuntimeException( "Cannot decrypt private key \"{$key->name}\" (uuid={$key->uuid}). " .'This instance APP_KEY does not match the key that encrypted the data ' @@ -395,6 +622,9 @@ class ServerTransferExporter // Read through cast so encrypted fields are plaintext for re-encryption on import. $settingsPayload[$key] = $settings->{$key}; } + $settingsPayload['cloudflare_tunnel_integration_token_uuid'] = $settings->cloudflare_tunnel_integration_token_id + ? IntegrationToken::query()->whereKey($settings->cloudflare_tunnel_integration_token_id)->value('uuid') + : null; } return [ @@ -402,12 +632,18 @@ class ServerTransferExporter 'name' => $server->name, 'description' => $server->description, 'ip' => (string) $server->ip, - 'port' => (int) $server->port, + 'port' => $server->port ?: 22, 'user' => (string) $server->user, 'proxy' => $server->proxy?->toArray() ?? [], 'server_role' => $server->settings->effectiveServerRole()->value, 'is_build_server' => $server->isBuildServer(), 'cloud_provider_token_uuid' => $server->cloudProviderToken?->uuid, + 'hetzner_server_id' => $server->hetzner_server_id, + 'vultr_instance_id' => $server->vultr_instance_id, + 'digitalocean_droplet_id' => $server->digitalocean_droplet_id, + 'hostinger_virtual_machine_id' => $server->hostinger_virtual_machine_id, + 'proxy_error_page' => $server->proxy_error_page, + 'swarm_cluster' => $server->swarm_cluster, 'settings' => $settingsPayload, ]; } @@ -443,29 +679,37 @@ class ServerTransferExporter * @param Collection $databases * @param Collection $services * @param array $destinationUuidByKey + * @param list $warnings * @return array */ private function exportProject( Project $project, - Server $server, array $destinationUuidByKey, Collection $applications, Collection $databases, Collection $services, + array &$warnings, + int &$leftOutLinks, ): array { - $environments = $project->environments()->orderBy('name')->get(); + // Only environments with a resource of this server. The other environments stay on this instance. + $environmentIds = $applications->pluck('environment_id') + ->merge($databases->pluck('environment_id')) + ->merge($services->pluck('environment_id')) + ->unique(); + $environments = $project->environments()->whereIn('id', $environmentIds)->orderBy('name')->get(); return [ 'uuid' => $project->uuid, 'name' => $project->name, 'description' => $project->description, + 'icon' => $this->exportProjectIcon($project, $warnings), 'shared_environment_variables' => $this->exportSharedEnvVars( SharedEnvironmentVariable::query() ->where('type', 'project') ->where('project_id', $project->id) ->get() ), - 'environments' => $environments->map(function (Environment $environment) use ($destinationUuidByKey, $applications, $databases, $services) { + 'environments' => $environments->map(function (Environment $environment) use ($destinationUuidByKey, $applications, $databases, $services, &$leftOutLinks) { $envApps = $applications->where('environment_id', $environment->id)->values(); $envDbs = $databases->where('environment_id', $environment->id)->values(); $envServices = $services->where('environment_id', $environment->id)->values(); @@ -474,6 +718,10 @@ class ServerTransferExporter 'uuid' => $environment->uuid, 'name' => $environment->name, 'description' => $environment->description, + // Keyed by resource uuid; only the positions of exported resources. + 'canvas_positions' => collect($environment->canvas_positions ?? []) + ->only($envApps->pluck('uuid')->merge($envDbs->pluck('uuid'))->merge($envServices->pluck('uuid'))) + ->all(), 'shared_environment_variables' => $this->exportSharedEnvVars( SharedEnvironmentVariable::query() ->where('type', 'environment') @@ -483,11 +731,105 @@ class ServerTransferExporter 'applications' => $envApps->map(fn (Application $app) => $this->exportApplication($app, $destinationUuidByKey))->all(), 'databases' => $envDbs->map(fn (Model $db) => $this->exportDatabase($db, $destinationUuidByKey))->all(), 'services' => $envServices->map(fn (Service $service) => $this->exportService($service, $destinationUuidByKey))->all(), + 'resource_links' => $this->exportResourceLinks($environment, $envApps, $envDbs, $envServices, $leftOutLinks), ]; })->values()->all(), ]; } + /** + * The icon file travels in the bundle; the target stores it in its own icon storage. + * + * @param list $warnings + * @return array{content_base64: string, extension: string, mime: string}|null + */ + private function exportProjectIcon(Project $project, array &$warnings): ?array + { + if (blank($project->icon_path)) { + return null; + } + + try { + $contents = app(ProjectIconStorageService::class)->projectContents($project); + } catch (Throwable) { + $contents = null; + } + if (! is_string($contents) || $contents === '') { + $warnings[] = "The icon of project {$project->name} cannot be read, so it is not transferred. Upload it again on the target instance."; + + return null; + } + if (strlen($contents) > self::MAX_PROJECT_ICON_BYTES) { + $warnings[] = "The icon of project {$project->name} is larger than 2 MB, so it is not transferred. Upload it again on the target instance."; + + return null; + } + + return [ + 'content_base64' => base64_encode($contents), + 'extension' => strtolower(pathinfo($project->icon_path, PATHINFO_EXTENSION)) ?: 'jpg', + 'mime' => (new \finfo(FILEINFO_MIME_TYPE))->buffer($contents) ?: 'application/octet-stream', + ]; + } + + /** + * Canvas links whose two resources are both exported. Each end is the kind and uuid of a resource of the + * environment, the same keys that the import uses for its resource maps. + * + * @param Collection $applications + * @param Collection $databases + * @param Collection $services + * @return list + */ + private function exportResourceLinks(Environment $environment, Collection $applications, Collection $databases, Collection $services, int &$leftOut): array + { + $ends = collect(); + foreach (['application' => $applications, 'database' => $databases, 'service' => $services] as $kind => $resources) { + foreach ($resources as $resource) { + $ends->put($resource->getMorphClass().':'.$resource->getKey(), ['kind' => $kind, 'uuid' => $resource->uuid]); + } + } + + $out = []; + foreach ($environment->resourceLinks()->orderBy('id')->get() as $link) { + $source = $ends->get($link->source_type.':'.$link->source_id); + $target = $ends->get($link->target_type.':'.$link->target_id); + if (! $source || ! $target) { + $leftOut++; + + continue; + } + $out[] = ['uuid' => $link->uuid, 'label' => $link->label, 'source' => $source, 'target' => $target]; + } + + return $out; + } + + /** + * The target imports the config disabled. Enabling it there removes the runner containers of the source. + * + * @return array + */ + private function exportGithubRunnerConfig(GithubRunnerConfig $config): array + { + return [ + 'uuid' => $config->uuid, + 'github_app_uuid' => $config->githubApp?->uuid, + 'labels' => $config->labels ?? [], + 'is_enabled' => (bool) $config->is_enabled, + 'is_dedicated' => (bool) $config->is_dedicated, + 'allow_pull_requests' => (bool) $config->allow_pull_requests, + 'max_runners' => $config->max_runners, + 'docker_mode' => $config->docker_mode?->value, + 'runner_image' => $config->runner_image, + 'cpu_limit' => $config->cpu_limit, + 'memory_limit' => $config->memory_limit, + 'capacity_wait_timeout' => $config->capacity_wait_timeout, + 'idle_timeout' => $config->idle_timeout, + 'job_timeout' => $config->job_timeout, + ]; + } + /** * @param array $destinationUuidByKey * @return array @@ -530,6 +872,8 @@ class ServerTransferExporter 'docker_compose', 'docker_compose_raw', 'custom_labels', + 'is_maintenance_enabled', + 'maintenance_page', ] as $hidden) { if (in_array($hidden, $columns, true)) { $attrs[$hidden] = $application->{$hidden}; @@ -550,9 +894,9 @@ class ServerTransferExporter $source = null; if ($application->source_type === GithubApp::class && $application->source) { - $source = ['type' => 'github_app', 'uuid' => $application->source->uuid]; + $source = ['type' => 'github_app', 'uuid' => $application->source->uuid, 'system_wide' => (bool) $application->source->is_system_wide]; } elseif ($application->source_type === GitlabApp::class && $application->source) { - $source = ['type' => 'gitlab_app', 'uuid' => $application->source->uuid]; + $source = ['type' => 'gitlab_app', 'uuid' => $application->source->uuid, 'system_wide' => (bool) $application->source->is_system_wide]; } return [ @@ -568,6 +912,7 @@ class ServerTransferExporter 'scheduled_tasks' => $this->exportScheduledTasks($application->scheduled_tasks()->get()), 'tags' => $this->exportTags($application->tags), 'previews' => $this->exportPreviews($application->previews), + 'secret_manager_link' => $this->exportSecretManagerLink($application), 'source' => $source, 'private_key_uuid' => $application->private_key?->uuid, 'had_git_source' => filled($application->source_id), @@ -612,6 +957,11 @@ class ServerTransferExporter } } + // Not fillable + if (in_array('include_coolify_default_variables', $columns, true)) { + $attrs['include_coolify_default_variables'] = $database->include_coolify_default_variables; + } + $destinationKey = $database->destination_type.':'.$database->destination_id; return [ @@ -635,6 +985,7 @@ class ServerTransferExporter 'scheduled_backups' => method_exists($database, 'scheduledBackups') ? $this->exportScheduledBackups($database->scheduledBackups) : [], + 'secret_manager_link' => $this->exportSecretManagerLink($database), ]; } @@ -668,8 +1019,8 @@ class ServerTransferExporter $attrs[$field] = $service->{$field}; } - // Hidden compose fields - foreach (['docker_compose', 'docker_compose_raw'] as $hidden) { + // Hidden compose fields, maintenance mode and the Coolify default variables switch (not fillable) + foreach (['docker_compose', 'docker_compose_raw', 'is_maintenance_enabled', 'maintenance_page', 'include_coolify_default_variables'] as $hidden) { if (in_array($hidden, $columns, true)) { $attrs[$hidden] = $service->{$hidden}; } @@ -685,6 +1036,7 @@ class ServerTransferExporter 'environment_variables' => $this->exportEnvVars($service->environment_variables()->get()), 'scheduled_tasks' => $this->exportScheduledTasks($service->scheduled_tasks()->get()), 'tags' => $this->exportTags($service->tags), + 'secret_manager_link' => $this->exportSecretManagerLink($service), 'applications' => $service->applications ->map(fn ($app) => $this->exportServiceApplication($app)) ->values() @@ -861,8 +1213,11 @@ class ServerTransferExporter 'is_shared' => (bool) $variable->is_shared, 'is_required' => (bool) ($variable->is_required ?? false), 'uses_legacy_escaping' => (bool) $variable->uses_legacy_escaping, + 'preserve_compose_null' => (bool) $variable->preserve_compose_null, 'comment' => $variable->comment, 'order' => $variable->order, + // The Coolify version that created the variable. + 'version' => $variable->version, ]; } @@ -979,6 +1334,7 @@ class ServerTransferExporter 'database_backup_retention_max_storage_s3' => $backup->database_backup_retention_max_storage_s3, 'timeout' => $backup->timeout, 'disable_local_backup' => (bool) ($backup->disable_local_backup ?? false), + 'missing_backup_notification_days' => $backup->missing_backup_notification_days, 's3_storage_uuid' => $backup->s3?->uuid, 's3_storage_uuids' => $backup->selectedS3Storages()->pluck('uuid')->values()->all(), 'had_s3_storage' => filled($backup->s3_storage_id), @@ -1012,6 +1368,10 @@ class ServerTransferExporter 'metadata' => $app->metadata, 'pull_requests' => $app->pull_requests, 'administration' => $app->administration, + 'organization_self_hosted_runners' => $app->organization_self_hosted_runners, + 'actions' => $app->actions, + 'webhook_events' => $app->webhook_events, + 'runner_group_id' => $app->runner_group_id, 'private_key_uuid' => $app->privateKey?->uuid, 'webhook_url_hint' => '/webhooks/source/github/events', ]; @@ -1082,6 +1442,85 @@ class ServerTransferExporter ]; } + /** + * The token is read through its cast, so the bundle has it in plaintext like the other credentials. + * + * @return array + */ + private function exportIntegrationToken(IntegrationToken $token): array + { + return [ + 'uuid' => $token->uuid, + 'provider' => $token->provider, + 'name' => $token->name, + 'token' => $token->token, + 'capabilities' => $token->capabilities, + 'metadata' => $token->metadata, + ]; + } + + /** + * @return array + */ + private function exportDnsZone(DnsProviderZone $zone): array + { + return [ + 'uuid' => $zone->uuid, + 'integration_token_uuid' => $zone->integrationToken->uuid, + 'provider_zone_id' => $zone->provider_zone_id, + 'name' => $zone->name, + 'account_id' => $zone->account_id, + 'account_name' => $zone->account_name, + ]; + } + + /** + * The ownership comment proves at the provider that this instance created the record. The target compares it + * with the remote comment before it takes ownership. + * + * @param array $resourceKeys + * @return array + */ + private function exportManagedDnsRecord(ManagedDnsRecord $record, array $resourceKeys): array + { + return [ + 'uuid' => $record->uuid, + 'integration_token_uuid' => $record->integrationToken->uuid, + 'dns_provider_zone_uuid' => $record->zone->uuid, + 'provider_record_id' => $record->provider_record_id, + 'type' => $record->type, + 'name' => $record->name, + 'content' => $record->content, + 'owned' => (bool) $record->owned, + 'ownership_comment' => $record->ownershipComment(), + 'references' => $record->references + ->map(fn (ManagedDnsRecordReference $reference) => $resourceKeys[$reference->resource_type.':'.$reference->resource_id] ?? null) + ->filter() + ->values() + ->all(), + ]; + } + + /** + * @return array{uuid: string, integration_token_uuid: string, settings: array|null}|null + */ + private function exportSecretManagerLink(Model $resource): ?array + { + if (! method_exists($resource, 'secretManagerLink')) { + return null; + } + $link = $resource->secretManagerLink()->with('integrationToken')->first(); + if (! $link) { + return null; + } + + return [ + 'uuid' => $link->uuid, + 'integration_token_uuid' => $link->integrationToken->uuid, + 'settings' => $link->settings, + ]; + } + /** * @return list> */ @@ -1138,6 +1577,8 @@ class ServerTransferExporter 'git_type' => $preview->git_type, 'docker_compose_domains' => $preview->docker_compose_domains, 'docker_registry_image_tag' => $preview->docker_registry_image_tag, + 'domain_port_overrides' => $preview->domain_port_overrides, + 'max_restart_count' => $preview->max_restart_count, 'persistent_storages' => method_exists($preview, 'persistentStorages') ? $this->exportPersistentStorages($preview->persistentStorages) : [], @@ -1192,6 +1633,7 @@ class ServerTransferExporter 'retention_days_s3' => $backup->retention_days_s3, 'retention_max_storage_s3' => $backup->retention_max_storage_s3, 'timeout' => $backup->timeout, + 'missing_backup_notification_days' => $backup->missing_backup_notification_days, 's3_storage_uuid' => $backup->s3?->uuid, 's3_storage_uuids' => $backup->selectedS3Storages()->pluck('uuid')->values()->all(), 'had_s3_storage' => filled($backup->s3_storage_id), diff --git a/app/Services/ServerTransfer/ServerTransferImporter.php b/app/Services/ServerTransfer/ServerTransferImporter.php index 4c175fc8af..053d043d0d 100644 --- a/app/Services/ServerTransfer/ServerTransferImporter.php +++ b/app/Services/ServerTransfer/ServerTransferImporter.php @@ -2,22 +2,30 @@ namespace App\Services\ServerTransfer; +use App\Enums\GithubRunnerDockerMode; use App\Enums\ServerRole; +use App\Jobs\ValidateAndInstallServerJob; use App\Models\Application; use App\Models\ApplicationPreview; use App\Models\CloudProviderToken; +use App\Models\DnsProviderZone; use App\Models\Environment; use App\Models\EnvironmentVariable; use App\Models\GithubApp; +use App\Models\GithubRunnerConfig; use App\Models\GitlabApp; +use App\Models\IntegrationToken; use App\Models\LocalFileVolume; use App\Models\LocalPersistentVolume; +use App\Models\ManagedDnsRecord; use App\Models\PrivateKey; use App\Models\Project; +use App\Models\ResourceLink; use App\Models\S3Storage; use App\Models\ScheduledDatabaseBackup; use App\Models\ScheduledTask; use App\Models\ScheduledVolumeBackup; +use App\Models\SecretManagerLink; use App\Models\Server; use App\Models\Service; use App\Models\ServiceApplication; @@ -37,9 +45,18 @@ use App\Models\StandaloneSqlite; use App\Models\SwarmDocker; use App\Models\Tag; use App\Models\Team; +use App\Notifications\Internal\GeneralNotification; +use App\Rules\DockerImageFormat; +use App\Rules\ValidServerIp; +use App\Services\Dns\CloudflareDnsProvider; +use App\Services\ProjectIconStorageService; +use App\Support\ValidationPatterns; use Illuminate\Database\Eloquent\Model; use Illuminate\Support\Facades\DB; +use Illuminate\Support\Facades\Log; use Illuminate\Support\Facades\Schema; +use Illuminate\Support\Facades\Validator; +use Illuminate\Validation\Rule; use Illuminate\Validation\ValidationException; use RuntimeException; use Throwable; @@ -76,15 +93,43 @@ class ServerTransferImporter /** @var array */ private array $cloudTokenMap = []; + /** @var array */ + private array $integrationTokenMap = []; + + /** @var array */ + private array $dnsZoneMap = []; + + /** + * Imported owned DNS records that carry the ownership comment of the source instance. This instance takes + * ownership at the provider after the import transaction commits. + * + * @var list + */ + private array $pendingDnsAdoptions = []; + + /** + * Icons of imported projects. They are stored after the import transaction commits, so a storage error + * cannot roll back DB rows. + * + * @var list}> + */ + private array $pendingProjectIcons = []; + /** @var array */ private array $volumeMap = []; /** @var array */ private array $applicationMap = []; + /** @var array */ + private array $previewMap = []; + /** @var array */ private array $serviceMap = []; + /** @var array */ + private array $serviceApplicationMap = []; + /** @var array */ private array $databaseMap = []; @@ -95,6 +140,21 @@ class ServerTransferImporter */ private array $pendingSqliteLinks = []; + /** + * Warnings about references of imported resources that this instance cannot resolve. + * + * @var list + */ + private array $importWarnings = []; + + /** + * Applications whose git source needs a check after the transfer: a system-wide source of the source + * instance, or a source that this instance does not have. + * + * @var list + */ + private array $gitSourceApplications = []; + /** * @param array $bundle * @return array{ @@ -106,7 +166,8 @@ class ServerTransferImporter * preserved_uuids: bool, * export_id: string|null, * claimed: bool, - * claim: array|null + * claim: array|null, + * checklist: list * } */ public function import( @@ -117,6 +178,13 @@ class ServerTransferImporter bool $adoptMode = true, ): array { ServerTransferBundle::assertValid($bundle); + $this->assertValidServerPayload((array) data_get($bundle, 'server', [])); + + if ($adoptMode && ! $preserveUuids) { + throw ValidationException::withMessages([ + 'preserve_uuids' => ['Adopt mode keeps the running containers, so it needs the original UUIDs. Import with preserve_uuids=true.'], + ]); + } $validation = ServerTransferBundle::validate($bundle); $warnings = $validation['warnings']; @@ -127,27 +195,20 @@ class ServerTransferImporter $serverUuid = (string) data_get($bundle, 'server.uuid'); $serverIp = (string) data_get($bundle, 'server.ip'); - $uuidConflict = Server::withTrashed()->where('uuid', $serverUuid)->exists() && $preserveUuids; - $existingIp = Server::where('ip', Server::normalizeIp($serverIp))->first(); - $ipConflict = $existingIp !== null; + // A server that comes back replaces the archived copy that this instance kept from the first transfer. + $archive = $preserveUuids ? $this->replaceableArchive($serverUuid, $teamId) : null; + $uuidConflict = $archive === null && $preserveUuids && Server::withTrashed()->where('uuid', $serverUuid)->exists(); + $ipConflict = Server::where('ip', Server::normalizeIp($serverIp))->when($archive, fn ($query) => $query->whereKeyNot($archive->id))->exists(); - if (! $dryRun && $uuidConflict) { - throw ValidationException::withMessages([ - 'server.uuid' => ["A server with UUID {$serverUuid} already exists on this instance. Delete/transfer it first, or import with preserve_uuids=false."], - ]); - } - - if (! $dryRun && $ipConflict) { - throw ValidationException::withMessages([ - 'server.ip' => ["A server with IP/domain {$serverIp} already exists (uuid={$existingIp->uuid}). Complete handoff on the source instance first."], - ]); + if (! $dryRun) { + $this->assertCanImport($serverUuid, $serverIp, $teamId, $preserveUuids); } if ($uuidConflict) { $warnings[] = "A server with UUID {$serverUuid} already exists on this instance."; } if ($ipConflict) { - $warnings[] = "A server with IP/domain {$serverIp} already exists (uuid={$existingIp->uuid})."; + $warnings[] = "A server with IP/domain {$serverIp} already exists on this instance."; } $created = [ @@ -162,6 +223,8 @@ class ServerTransferImporter 'gitlab_apps' => 0, 's3_storages' => 0, 'cloud_provider_tokens' => 0, + 'integration_tokens' => 0, + 'managed_dns_records' => 0, 'ssl_certificates' => 0, 'volume_backups' => 0, 'previews' => 0, @@ -189,18 +252,11 @@ class ServerTransferImporter $created['gitlab_apps'] = count(data_get($bundle, 'gitlab_apps', [])); $created['s3_storages'] = count(data_get($bundle, 's3_storages', [])); $created['cloud_provider_tokens'] = count(data_get($bundle, 'cloud_provider_tokens', [])); + $created['integration_tokens'] = count(data_get($bundle, 'integration_tokens', [])); + $created['managed_dns_records'] = count(data_get($bundle, 'managed_dns_records', [])); $created['ssl_certificates'] = count(data_get($bundle, 'ssl_certificates', [])); $created['volume_backups'] = count(data_get($bundle, 'volume_backups', [])); - // Append transfer hints for the target instance FQDN. - $targetUrl = rtrim((string) (instanceSettings()->fqdn ?: config('app.url')), '/'); - if (count(data_get($bundle, 'github_apps', [])) > 0) { - $warnings[] = "After import, set GitHub App webhook URL to {$targetUrl}/webhooks/source/github/events (and update setup/callback URLs in GitHub if needed) so automations work on this instance."; - } - if (count(data_get($bundle, 'gitlab_apps', [])) > 0) { - $warnings[] = "After import, set GitLab webhook URL to {$targetUrl}/webhooks/source/gitlab/events so automations work on this instance."; - } - return [ 'dry_run' => true, 'warnings' => $warnings, @@ -211,10 +267,15 @@ class ServerTransferImporter 'export_id' => data_get($bundle, 'export_id'), 'claimed' => false, 'claim' => null, + 'checklist' => $this->postTransferChecklist($bundle), ]; } - $result = DB::transaction(function () use ($bundle, $teamId, $preserveUuids, $adoptMode, $warnings, &$created) { + $result = DB::transaction(function () use ($bundle, $teamId, $preserveUuids, $adoptMode, $warnings, $archive, &$created) { + // Removed in the same transaction as the import, so a failed import keeps the archive. + if ($archive) { + app(ServerTransferClaimer::class)->deleteArchive($archive); + } Team::ensureServerCapacity($teamId); $this->privateKeyMap = []; @@ -222,11 +283,19 @@ class ServerTransferImporter $this->gitlabAppMap = []; $this->s3StorageMap = []; $this->cloudTokenMap = []; + $this->integrationTokenMap = []; + $this->dnsZoneMap = []; + $this->pendingDnsAdoptions = []; + $this->pendingProjectIcons = []; $this->volumeMap = []; $this->applicationMap = []; + $this->previewMap = []; $this->serviceMap = []; + $this->serviceApplicationMap = []; $this->databaseMap = []; $this->pendingSqliteLinks = []; + $this->importWarnings = []; + $this->gitSourceApplications = []; // Import shared dependencies first $keyPayloads = data_get($bundle, 'private_keys', []); @@ -276,6 +345,13 @@ class ServerTransferImporter $created['cloud_provider_tokens']++; } + foreach (data_get($bundle, 'integration_tokens', []) as $tokenPayload) { + $token = $this->importIntegrationToken($tokenPayload, $teamId, $preserveUuids); + $this->integrationTokenMap[(string) data_get($tokenPayload, 'uuid', $token->uuid)] = $token; + $this->integrationTokenMap[$token->uuid] = $token; + $created['integration_tokens']++; + } + $serverKeyUuid = (string) data_get($bundle, 'private_key.uuid', ''); $privateKey = $this->privateKeyMap[$serverKeyUuid] ?? $this->importPrivateKey(data_get($bundle, 'private_key', []), $teamId, $preserveUuids); @@ -283,8 +359,10 @@ class ServerTransferImporter $server = $this->importServer(data_get($bundle, 'server', []), $privateKey, $teamId, $preserveUuids, $adoptMode); $destinationMap = $this->importDestinations(data_get($bundle, 'destinations', []), $server, $preserveUuids); $created['destinations'] = count($destinationMap); + $this->importGithubRunnerConfig(data_get($bundle, 'github_runner_config'), $server, $teamId, $preserveUuids); $this->importServerSharedEnvVars(data_get($bundle, 'shared_environment_variables.server', []), $server, $teamId); + $this->importTeamSharedEnvVars(data_get($bundle, 'shared_environment_variables.team', []), $teamId); foreach (data_get($bundle, 'projects', []) as $projectPayload) { $project = $this->importProject($projectPayload, $teamId, $preserveUuids, $created); @@ -311,21 +389,20 @@ class ServerTransferImporter $this->serviceMap[(string) data_get($servicePayload, 'uuid', $service->uuid)] = $service; $created['services']++; } + + $this->importCanvasPositions((array) data_get($environmentPayload, 'canvas_positions', []), $environment); + $this->importResourceLinks((array) data_get($environmentPayload, 'resource_links', []), $environment, $preserveUuids); } } $warnings = array_merge($warnings, $this->linkSqliteVolumes()); + $this->importDnsZones(data_get($bundle, 'dns_zones', []), $preserveUuids); + $created['managed_dns_records'] = $this->importManagedDnsRecords(data_get($bundle, 'managed_dns_records', []), $teamId, $preserveUuids); + $created['ssl_certificates'] = $this->importSslCertificates(data_get($bundle, 'ssl_certificates', []), $server); $created['volume_backups'] = $this->importVolumeBackups(data_get($bundle, 'volume_backups', []), $teamId); - - $targetUrl = rtrim((string) (instanceSettings()->fqdn ?: config('app.url')), '/'); - if ($created['github_apps'] > 0) { - $warnings[] = "GitHub Apps imported. Update webhook URL to {$targetUrl}/webhooks/source/github/events (and setup/callback URLs in GitHub) so automations work on this instance."; - } - if ($created['gitlab_apps'] > 0) { - $warnings[] = "GitLab sources imported. Update webhook URL to {$targetUrl}/webhooks/source/gitlab/events so automations work on this instance."; - } + $warnings = array_merge($warnings, $this->importWarnings); $metadata = $server->server_metadata ?? []; $metadata['transfer'] = [ @@ -334,6 +411,7 @@ class ServerTransferImporter 'source_instance_url' => data_get($bundle, 'source_instance.url'), 'imported_at' => now()->toIso8601String(), 'adopt_mode' => $adoptMode, + 'git_source_applications' => array_values(array_unique($this->gitSourceApplications)), ]; $server->server_metadata = $metadata; $server->save(); @@ -348,9 +426,13 @@ class ServerTransferImporter 'export_id' => data_get($bundle, 'export_id'), 'claimed' => false, 'claim' => null, + 'checklist' => $this->postTransferChecklist($bundle), ]; }); + // Provider and storage calls run after the import transaction commits, so an error cannot roll back DB rows. + $result['warnings'] = array_values(array_unique(array_merge($result['warnings'], $this->adoptManagedDnsRecords(), $this->storeProjectIcons()))); + // Claim after the import transaction commits so a claim failure cannot roll back DB rows. if (filled(data_get($result, 'server_uuid'))) { $server = Server::where('uuid', $result['server_uuid'])->where('team_id', $teamId)->first(); @@ -358,6 +440,10 @@ class ServerTransferImporter try { $result['claim'] = app(ServerTransferClaimer::class)->claim($server); $result['claimed'] = true; + // Same check as the Validate button. A running proxy and Docker are kept as they are. + ValidateAndInstallServerJob::dispatch($server); + $this->startFinishChecklist($server); + $result['server_url'] = rtrim((string) (instanceSettings()->fqdn ?: config('app.url')), '/').'/server/'.$server->uuid; } catch (Throwable $e) { $result['claimed'] = false; $result['claim'] = null; @@ -369,6 +455,130 @@ class ServerTransferImporter return $result; } + /** + * Check the server connection fields with the same rules as the create server API. + * + * @param array $server + */ + private function assertValidServerPayload(array $server): void + { + Validator::make($server, [ + 'name' => 'nullable|string|max:255', + 'ip' => ['required', 'string', new ValidServerIp], + 'port' => 'required|integer|between:1,65535', + 'user' => ValidationPatterns::serverUsernameRules(required: false), + ], [], [ + 'ip' => 'server.ip', + 'port' => 'server.port', + 'user' => 'server.user', + ])->validate(); + } + + /** + * Checks that a server can be imported into the team, without the bundle. Every check runs, so the + * source instance can show which checks are OK and which failed. + * + * @return list + */ + public function importChecks(string $serverUuid, string $serverIp, int $teamId, bool $preserveUuids = true): array + { + $team = Team::find($teamId); + $archive = $preserveUuids ? $this->replaceableArchive($serverUuid, $teamId) : null; + // The archive is replaced, so the import does not add a server. + $limitReached = ! $team || ($archive === null && Team::serverLimitReached($team)); + + $existingServer = $preserveUuids && $archive === null ? Server::withTrashed()->where('uuid', $serverUuid)->first() : null; + $uuidMessage = match (true) { + $existingServer === null => null, + $existingServer->trashed() => "The server with UUID {$serverUuid} is still being deleted on this instance. Wait a minute, then transfer again.", + $existingServer->isTransferredAway() => "Another team of this instance has the archived copy of server {$existingServer->name}. Transfer the server into that team, or delete the archive there first.", + default => "A server with UUID {$serverUuid} already exists on this instance. Delete/transfer it first, or import with preserve_uuids=false.", + }; + + $ipExists = Server::where('ip', Server::normalizeIp($serverIp))->when($archive, fn ($query) => $query->whereKeyNot($archive->id))->exists(); + + return [ + [ + 'field' => 'server', + 'label' => 'the team can have one more server', + 'ok' => ! $limitReached, + 'message' => $limitReached ? 'Server limit reached for your subscription.' : null, + ], + [ + 'field' => 'server.uuid', + 'label' => $archive ? 'the archived copy of this server on this instance is replaced' : 'no server with the same UUID exists', + 'ok' => $uuidMessage === null, + 'message' => $uuidMessage, + ], + [ + 'field' => 'server.ip', + 'label' => 'no server with the same IP exists', + 'ok' => ! $ipExists, + 'message' => $ipExists ? "A server with IP/domain {$serverIp} already exists on this instance. Remove it first." : null, + ], + ]; + } + + /** + * The steps that finish the transfer here, and one notification that points the team to them. + * A failure here never fails the import: the server is imported and claimed already. + */ + private function startFinishChecklist(Server $server): void + { + try { + $items = app(TransferFinishChecklist::class)->start($server); + $server->team?->notify(new GeneralNotification( + "Server '{$server->name}' was transferred to this instance. Open the server to finish the transfer: ".count($items).' steps to check.' + )); + } catch (Throwable $e) { + Log::warning('Could not start the transfer finish checklist', ['server_uuid' => $server->uuid, 'error' => $e->getMessage()]); + } + } + + /** + * The archive that this instance kept when it transferred the server away. A transfer back into the same team + * replaces it. + */ + private function replaceableArchive(string $serverUuid, int $teamId): ?Server + { + $server = Server::query()->where('uuid', $serverUuid)->where('team_id', $teamId)->first(); + + return $server?->isTransferredAway() ? $server : null; + } + + /** + * @throws ValidationException with the first failed import check + */ + public function assertCanImport(string $serverUuid, string $serverIp, int $teamId, bool $preserveUuids = true): void + { + foreach ($this->importChecks($serverUuid, $serverIp, $teamId, $preserveUuids) as $check) { + if (! $check['ok']) { + throw ValidationException::withMessages([$check['field'] => [$check['message']]]); + } + } + } + + /** + * Running containers are found by the resource UUID. In adopt mode a new UUID would leave them + * without a resource, so a conflict stops the import instead. + * + * @param class-string $modelClass + */ + private function resolveResourceUuid(string $modelClass, string $uuid, bool $adoptMode, string $label): string + { + if (! $modelClass::withTrashed()->where('uuid', $uuid)->exists()) { + return $uuid; + } + + if ($adoptMode) { + throw ValidationException::withMessages([ + 'bundle' => ["The {$label} UUID {$uuid} is already used on this instance. Delete that {$label} permanently first, because adopt mode needs the same UUID to keep the running containers."], + ]); + } + + return new_public_id(); + } + /** * @param array $payload */ @@ -450,6 +660,12 @@ class ServerTransferImporter 'private_key_id' => $privateKey->id, 'team_id' => $teamId, 'cloud_provider_token_id' => $cloudTokenId, + 'hetzner_server_id' => data_get($payload, 'hetzner_server_id'), + 'vultr_instance_id' => data_get($payload, 'vultr_instance_id'), + 'digitalocean_droplet_id' => data_get($payload, 'digitalocean_droplet_id'), + 'hostinger_virtual_machine_id' => data_get($payload, 'hostinger_virtual_machine_id'), + 'proxy_error_page' => data_get($payload, 'proxy_error_page'), + 'swarm_cluster' => data_get($payload, 'swarm_cluster'), ]); $server->uuid = $uuid; $server->save(); @@ -479,9 +695,16 @@ class ServerTransferImporter 'id', 'server_id', 'created_at', 'updated_at', 'is_reachable', 'is_usable', 'force_disabled', 'sentinel_token', 'sentinel_custom_url', + // A foreign key of the source instance; older bundles still contain it. + 'cloudflare_tunnel_integration_token_id', + 'cloudflare_tunnel_integration_token_uuid', ]) ->all(); $server->settings->fill($safe); + $server->settings->cloudflare_tunnel_integration_token_id = $this->importedIntegrationTokenId( + data_get($settingsPayload, 'cloudflare_tunnel_integration_token_uuid'), + 'The Cloudflare tunnel of this server: its Cloudflare API token was not found on this instance; select it again in the Cloudflare tunnel settings.', + ); // Imported servers start not validated; claim will rebind sentinel. $server->settings->is_reachable = false; $server->settings->is_usable = false; @@ -593,6 +816,38 @@ class ServerTransferImporter } } + /** + * A team variable that already exists keeps the value of this instance. + * + * @param list> $variables + */ + private function importTeamSharedEnvVars(array $variables, int $teamId): void + { + foreach ($variables as $variable) { + $key = (string) data_get($variable, 'key'); + $exists = SharedEnvironmentVariable::query() + ->where('type', 'team') + ->where('team_id', $teamId) + ->where('key', $key) + ->exists(); + if ($exists) { + $this->importWarnings[] = "Team shared variable {$key} already exists on this instance. It keeps its value here, which can differ from the value on the source instance."; + + continue; + } + SharedEnvironmentVariable::create([ + 'key' => $key, + 'value' => data_get($variable, 'value'), + 'is_multiline' => (bool) data_get($variable, 'is_multiline', false), + 'is_literal' => (bool) data_get($variable, 'is_literal', false), + 'is_shown_once' => (bool) data_get($variable, 'is_shown_once', false), + 'comment' => data_get($variable, 'comment'), + 'type' => 'team', + 'team_id' => $teamId, + ]); + } + } + /** * @param array $payload * @param array $created @@ -603,9 +858,14 @@ class ServerTransferImporter ? (string) data_get($payload, 'uuid') : new_public_id(); + $icon = data_get($payload, 'icon'); $existing = Project::where('uuid', $uuid)->where('team_id', $teamId)->first(); if ($existing) { $this->importProjectSharedEnvVars(data_get($payload, 'shared_environment_variables', []), $existing, $teamId); + // An icon that the project already has on this instance is kept. + if (is_array($icon) && blank($existing->icon_path)) { + $this->pendingProjectIcons[] = ['project' => $existing, 'icon' => $icon]; + } return $existing; } @@ -624,7 +884,124 @@ class ServerTransferImporter $this->importProjectSharedEnvVars(data_get($payload, 'shared_environment_variables', []), $project, $teamId); - return $project->fresh(['environments']); + $project = $project->fresh(['environments']); + if (is_array($icon)) { + $this->pendingProjectIcons[] = ['project' => $project, 'icon' => $icon]; + } + + return $project; + } + + /** + * Stores the icons of the imported projects in the icon storage of this instance. Coolify stores every + * icon as a JPEG file and serves it as one, so other content is not stored. + * + * @return list warnings + */ + private function storeProjectIcons(): array + { + $warnings = []; + foreach ($this->pendingProjectIcons as ['project' => $project, 'icon' => $icon]) { + $contents = base64_decode((string) data_get($icon, 'content_base64', ''), true); + $problem = match (true) { + ! is_string($contents) || $contents === '' => 'its content is not valid', + strlen($contents) > ServerTransferExporter::MAX_PROJECT_ICON_BYTES => 'it is larger than 2 MB', + (new \finfo(FILEINFO_MIME_TYPE))->buffer($contents) !== 'image/jpeg' => 'it is not a JPEG image', + default => null, + }; + if ($problem === null) { + try { + app(ProjectIconStorageService::class)->storeProjectContents($project, $contents); + + continue; + } catch (Throwable) { + $problem = 'the icon storage of this instance is not available'; + } + } + $warnings[] = "The icon of project {$project->name} was not imported, because {$problem}. Upload it again in the project settings."; + } + $this->pendingProjectIcons = []; + + return $warnings; + } + + /** + * Manual steps after the transfer, with the URL of this instance in place of {target_url}. + * + * @param array $bundle + * @return list + */ + private function postTransferChecklist(array $bundle): array + { + $targetUrl = rtrim((string) (instanceSettings()->fqdn ?: config('app.url')), '/'); + + return array_values(array_unique(array_map( + fn (string $step) => str_replace('{target_url}', $targetUrl, $step), + array_filter((array) data_get($bundle, 'post_transfer_checklist', []), 'is_string'), + ))); + } + + /** + * The config is imported disabled: an enabled config makes this instance remove every runner container on + * the server that it does not know, also the runners that the source started. The user enables it after + * the GitHub App sends its webhooks here. + */ + private function importGithubRunnerConfig(mixed $payload, Server $server, int $teamId, bool $preserveUuids): void + { + if (! is_array($payload)) { + return; + } + + $appUuid = (string) data_get($payload, 'github_app_uuid', ''); + $githubApp = $appUuid === '' ? null : ($this->githubAppMap[$appUuid] + ?? GithubApp::query()->where('uuid', $appUuid)->where('is_system_wide', true)->first() + ?? GithubApp::query()->where('uuid', $appUuid)->where('team_id', $teamId)->first()); + if (! $githubApp) { + $this->importWarnings[] = 'The GitHub runner settings of this server were not imported, because their GitHub App was not found on this instance. Set up GitHub runners on the server again.'; + + return; + } + + $validator = Validator::make($payload, [ + 'labels' => ['required', 'array', 'min:1'], + 'labels.*' => ['string', 'regex:/\A[a-z0-9][a-z0-9._-]{0,99}\z/'], + 'max_runners' => ['required', 'integer', 'min:1', 'max:32'], + 'docker_mode' => ['required', Rule::enum(GithubRunnerDockerMode::class)], + 'runner_image' => ['nullable', 'string', 'max:255', new DockerImageFormat], + 'cpu_limit' => ['nullable', 'string', 'regex:/\A\d+(\.\d+)?\z/'], + 'memory_limit' => ['nullable', 'string', 'regex:/\A\d+[bkmg]?\z/i'], + 'capacity_wait_timeout' => ['required', 'integer', 'min:1', 'max:1440'], + 'idle_timeout' => ['required', 'integer', 'min:1', 'max:1440'], + 'job_timeout' => ['required', 'integer', 'min:1', 'max:7200'], + ]); + if ($validator->fails()) { + $this->importWarnings[] = 'The GitHub runner settings of this server were not imported, because they are not valid: '.$validator->errors()->first().' Set up GitHub runners on the server again.'; + + return; + } + + $uuid = (string) data_get($payload, 'uuid', ''); + $config = new GithubRunnerConfig; + $config->forceFill([ + 'server_id' => $server->id, + 'github_app_id' => $githubApp->id, + 'labels' => array_values($payload['labels']), + 'is_enabled' => false, + 'is_dedicated' => (bool) data_get($payload, 'is_dedicated', false), + 'allow_pull_requests' => (bool) data_get($payload, 'allow_pull_requests', false), + 'max_runners' => (int) $payload['max_runners'], + 'docker_mode' => GithubRunnerDockerMode::from($payload['docker_mode']), + 'runner_image' => data_get($payload, 'runner_image'), + 'cpu_limit' => data_get($payload, 'cpu_limit'), + 'memory_limit' => data_get($payload, 'memory_limit'), + 'capacity_wait_timeout' => (int) $payload['capacity_wait_timeout'], + 'idle_timeout' => (int) $payload['idle_timeout'], + 'job_timeout' => (int) $payload['job_timeout'], + ]); + if ($preserveUuids && $uuid !== '' && ! GithubRunnerConfig::where('uuid', $uuid)->exists()) { + $config->uuid = $uuid; + } + $config->save(); } /** @@ -733,6 +1110,86 @@ class ServerTransferImporter } } + /** + * Canvas positions are keyed by resource uuid. Resources may get a new uuid on import. + * + * @param array $positions + */ + private function importCanvasPositions(array $positions, Environment $environment): void + { + $imported = []; + foreach ($positions as $uuid => $position) { + $resource = $this->applicationMap[$uuid] ?? $this->databaseMap[$uuid] ?? $this->serviceMap[$uuid] ?? null; + if ($resource) { + $imported[$resource->uuid] = $position; + } + } + if ($imported === []) { + return; + } + + $environment->canvas_positions = array_merge($environment->canvas_positions ?? [], $imported); + $environment->save(); + } + + /** + * Links between two imported resources of the environment. The user who drew the link is a user of the + * source instance, so created_by stays empty. + * + * @param list> $links + */ + private function importResourceLinks(array $links, Environment $environment, bool $preserveUuids): void + { + foreach ($links as $link) { + $source = $this->linkedResource(data_get($link, 'source')); + $target = $this->linkedResource(data_get($link, 'target')); + if (! $source || ! $target || $source->is($target) + || (int) $source->environment_id !== $environment->id || (int) $target->environment_id !== $environment->id) { + $this->importWarnings[] = "A link between resources on the canvas of environment {$environment->name} was not imported, because one of its resources was not imported."; + + continue; + } + + $attributes = [ + 'environment_id' => $environment->id, + 'source_type' => $source->getMorphClass(), + 'source_id' => $source->getKey(), + 'target_type' => $target->getMorphClass(), + 'target_id' => $target->getKey(), + ]; + if (ResourceLink::query()->where($attributes)->exists()) { + continue; + } + + $label = data_get($link, 'label'); + $resourceLink = new ResourceLink([ + ...$attributes, + 'label' => is_string($label) && trim($label) !== '' ? mb_substr(trim($label), 0, 64) : null, + 'created_by' => null, + ]); + $uuid = (string) data_get($link, 'uuid', ''); + if ($preserveUuids && $uuid !== '' && ! ResourceLink::where('uuid', $uuid)->exists()) { + $resourceLink->uuid = $uuid; + } + $resourceLink->save(); + } + } + + /** + * An end of a canvas link: the kind and the original uuid of an imported resource. + */ + private function linkedResource(mixed $end): ?Model + { + $uuid = (string) data_get($end, 'uuid', ''); + + return match (data_get($end, 'kind')) { + 'application' => $this->applicationMap[$uuid] ?? null, + 'database' => $this->databaseMap[$uuid] ?? null, + 'service' => $this->serviceMap[$uuid] ?? null, + default => null, + }; + } + /** * @param array $payload * @param array $destinationMap @@ -750,9 +1207,7 @@ class ServerTransferImporter ? (string) data_get($payload, 'uuid') : new_public_id(); - if (Application::withTrashed()->where('uuid', $uuid)->exists()) { - $uuid = new_public_id(); - } + $uuid = $this->resolveResourceUuid(Application::class, $uuid, $adoptMode, 'application'); $attributes = (array) data_get($payload, 'attributes', []); unset($attributes['id'], $attributes['environment_id'], $attributes['destination_id'], $attributes['destination_type'], $attributes['source_id'], $attributes['source_type'], $attributes['private_key_id']); @@ -763,6 +1218,8 @@ class ServerTransferImporter $attributes['destination_id'] = $destination->id; $attributes['destination_type'] = $destination->getMorphClass(); + $name = (string) data_get($attributes, 'name', $uuid); + // Re-link git source $source = data_get($payload, 'source'); if (is_array($source) && filled(data_get($source, 'uuid'))) { @@ -786,12 +1243,27 @@ class ServerTransferImporter } } } + $sourceMissing = data_get($payload, 'had_git_source') && ! isset($attributes['source_id']); + if ($sourceMissing || data_get($payload, 'source.system_wide') === true) { + $this->gitSourceApplications[] = $uuid; + } + if ($sourceMissing) { + $sourceLabel = match (data_get($payload, 'source.type')) { + 'github_app' => 'GitHub App', + 'gitlab_app' => 'GitLab App', + default => 'git source', + }; + $this->importWarnings[] = "Application {$name}: its {$sourceLabel} was not found on this instance; set the source again."; + } // Re-link deploy key $appKeyUuid = data_get($payload, 'private_key_uuid'); if ($appKeyUuid && isset($this->privateKeyMap[$appKeyUuid])) { $attributes['private_key_id'] = $this->privateKeyMap[$appKeyUuid]->id; } + if (data_get($payload, 'had_private_key') && ! isset($attributes['private_key_id'])) { + $this->importWarnings[] = "Application {$name}: its private key was not found on this instance; select the private key again."; + } if ($adoptMode) { // Keep runtime status if provided so inventory can match, but default exited if empty. @@ -800,7 +1272,10 @@ class ServerTransferImporter $attributes['status'] = 'exited'; } - $application = Application::create($attributes); + $application = new Application($attributes); + // Maintenance mode is not fillable. + $application->forceFill(collect($attributes)->only(['is_maintenance_enabled', 'maintenance_page'])->all()); + $application->save(); $settings = (array) data_get($payload, 'settings', []); if ($settings !== [] && $application->settings) { @@ -815,6 +1290,7 @@ class ServerTransferImporter $this->importScheduledTasks(data_get($payload, 'scheduled_tasks', []), $application, $teamId); $this->importTags(data_get($payload, 'tags', []), $application, $teamId); $this->importPreviews(data_get($payload, 'previews', []), $application, $preserveUuids, $adoptMode, $teamId); + $this->importSecretManagerLink(data_get($payload, 'secret_manager_link'), $application, "Application {$name}"); return $application; } @@ -832,12 +1308,9 @@ class ServerTransferImporter int $teamId = 0, ): ?Model { $type = (string) data_get($payload, 'type', 'StandalonePostgresql'); - $modelClass = self::DATABASE_MODELS[$type] - ?? (is_string(data_get($payload, 'model')) && class_exists((string) data_get($payload, 'model')) - ? (string) data_get($payload, 'model') - : null); + $modelClass = self::DATABASE_MODELS[$type] ?? null; - if (! $modelClass || ! class_exists($modelClass)) { + if (! $modelClass) { throw new RuntimeException("Unsupported database type: {$type}"); } @@ -846,9 +1319,7 @@ class ServerTransferImporter ? (string) data_get($payload, 'uuid') : new_public_id(); - if ($modelClass::withTrashed()->where('uuid', $uuid)->exists()) { - $uuid = new_public_id(); - } + $uuid = $this->resolveResourceUuid($modelClass, $uuid, $adoptMode, 'database'); $attributes = (array) data_get($payload, 'attributes', []); unset($attributes['id'], $attributes['environment_id'], $attributes['destination_id'], $attributes['destination_type']); @@ -878,6 +1349,7 @@ class ServerTransferImporter $this->importTags(data_get($payload, 'tags', []), $database, $teamId); } $this->importScheduledBackups(data_get($payload, 'scheduled_backups', []), $database, $teamId); + $this->importSecretManagerLink(data_get($payload, 'secret_manager_link'), $database, "Database {$database->name}"); return $database; } @@ -900,9 +1372,7 @@ class ServerTransferImporter ? (string) data_get($payload, 'uuid') : new_public_id(); - if (Service::withTrashed()->where('uuid', $uuid)->exists()) { - $uuid = new_public_id(); - } + $uuid = $this->resolveResourceUuid(Service::class, $uuid, $adoptMode, 'service'); $attributes = (array) data_get($payload, 'attributes', []); unset($attributes['id'], $attributes['environment_id'], $attributes['destination_id'], $attributes['destination_type'], $attributes['server_id']); @@ -913,14 +1383,19 @@ class ServerTransferImporter $attributes['destination_id'] = $destination->id; $attributes['destination_type'] = $destination->getMorphClass(); - $service = Service::create($attributes); + $service = new Service($attributes); + // Maintenance mode and the Coolify default variables switch are not fillable. + $service->forceFill(collect($attributes)->only(['is_maintenance_enabled', 'maintenance_page', 'include_coolify_default_variables'])->all()); + $service->save(); $this->importEnvVars(data_get($payload, 'environment_variables', []), $service, false); $this->importScheduledTasks(data_get($payload, 'scheduled_tasks', []), $service, $teamId); $this->importTags(data_get($payload, 'tags', []), $service, $teamId); + $this->importSecretManagerLink(data_get($payload, 'secret_manager_link'), $service, "Service {$service->name}"); foreach (data_get($payload, 'applications', []) as $appPayload) { - $this->importServiceApplication($appPayload, $service, $preserveUuids, $adoptMode); + $serviceApp = $this->importServiceApplication($appPayload, $service, $preserveUuids, $adoptMode); + $this->serviceApplicationMap[(string) data_get($appPayload, 'uuid', $serviceApp->uuid)] = $serviceApp; } foreach (data_get($payload, 'databases', []) as $dbPayload) { @@ -1033,11 +1508,16 @@ class ServerTransferImporter 'is_required' => (bool) data_get($variable, 'is_required', false), // Exports from older versions have no flag; their values use the old escaping. 'uses_legacy_escaping' => (bool) data_get($variable, 'uses_legacy_escaping', true), + 'preserve_compose_null' => (bool) data_get($variable, 'preserve_compose_null', false), 'comment' => data_get($variable, 'comment'), 'order' => data_get($variable, 'order'), 'resourceable_type' => $resource->getMorphClass(), 'resourceable_id' => $resource->id, ]); + // Older bundles have no version; the column default applies. + if (filled(data_get($variable, 'version'))) { + $env->version = (string) data_get($variable, 'version'); + } $env->uuid = $uuid; $env->save(); }); @@ -1241,6 +1721,7 @@ class ServerTransferImporter $s3Ids = $this->importedBackupS3StorageIds($backup); $s3Id = $s3Ids[0] ?? null; + $this->warnAboutMissingS3Storage($backup, $s3Id, "database {$database->name}"); $importedBackup = ScheduledDatabaseBackup::create([ 'uuid' => $uuid, @@ -1259,6 +1740,7 @@ class ServerTransferImporter 'database_backup_retention_max_storage_s3' => data_get($backup, 'database_backup_retention_max_storage_s3'), 'timeout' => data_get($backup, 'timeout'), 'disable_local_backup' => (bool) data_get($backup, 'disable_local_backup', false), + 'missing_backup_notification_days' => (int) data_get($backup, 'missing_backup_notification_days', 0), 's3_storage_id' => $s3Id, 'database_type' => $database->getMorphClass(), 'database_id' => $database->id, @@ -1267,6 +1749,20 @@ class ServerTransferImporter } } + /** + * @param array $backup + */ + private function warnAboutMissingS3Storage(array $backup, ?int $s3Id, string $backupOf): void + { + if (! data_get($backup, 'had_s3_storage') || $s3Id !== null) { + return; + } + + $this->importWarnings[] = data_get($backup, 'save_s3') + ? "Backup of {$backupOf}: S3 upload was turned off because its S3 storage was not found." + : "Backup of {$backupOf}: its S3 storage was not found; select it again."; + } + /** * Maps the exported S3 destinations of a backup to imported storages, primary first. Older bundles only have * `s3_storage_uuid`. @@ -1327,6 +1823,10 @@ class ServerTransferImporter 'metadata' => data_get($payload, 'metadata'), 'pull_requests' => data_get($payload, 'pull_requests'), 'administration' => data_get($payload, 'administration'), + 'organization_self_hosted_runners' => data_get($payload, 'organization_self_hosted_runners'), + 'actions' => data_get($payload, 'actions'), + 'webhook_events' => data_get($payload, 'webhook_events'), + 'runner_group_id' => data_get($payload, 'runner_group_id'), ]); $app->uuid = $uuid; $app->save(); @@ -1464,6 +1964,222 @@ class ServerTransferImporter return $token; } + /** + * A token with the same UUID in the target team is reused. A UUID that another team uses gets a new UUID. + * + * @param array $payload + */ + private function importIntegrationToken(array $payload, int $teamId, bool $preserveUuids): IntegrationToken + { + $uuid = $preserveUuids && filled(data_get($payload, 'uuid')) + ? (string) data_get($payload, 'uuid') + : new_public_id(); + + $existing = IntegrationToken::where('uuid', $uuid)->where('team_id', $teamId)->first(); + if ($existing) { + return $existing; + } + if (IntegrationToken::where('uuid', $uuid)->exists()) { + $uuid = new_public_id(); + } + + $token = new IntegrationToken; + $token->forceFill([ + 'team_id' => $teamId, + 'provider' => (string) data_get($payload, 'provider'), + 'name' => data_get($payload, 'name') ?: 'Imported integration token', + 'token' => (string) data_get($payload, 'token'), + 'capabilities' => (array) data_get($payload, 'capabilities', []), + 'metadata' => data_get($payload, 'metadata'), + ]); + $token->uuid = $uuid; + $token->save(); + + return $token; + } + + /** + * The ID of the imported token, or null with a warning when the bundle names a token that it does not contain. + */ + private function importedIntegrationTokenId(mixed $tokenUuid, string $missingWarning): ?int + { + if (! is_string($tokenUuid) || $tokenUuid === '') { + return null; + } + if (! isset($this->integrationTokenMap[$tokenUuid])) { + $this->importWarnings[] = $missingWarning; + + return null; + } + + return $this->integrationTokenMap[$tokenUuid]->id; + } + + private function importSecretManagerLink(mixed $payload, Model $resource, string $label): void + { + if (! is_array($payload) || ! method_exists($resource, 'secretManagerLink')) { + return; + } + + $tokenId = $this->importedIntegrationTokenId( + data_get($payload, 'integration_token_uuid'), + "{$label}: its secret manager token was not found on this instance; link the secret manager source again.", + ); + if ($tokenId === null) { + return; + } + + $uuid = filled(data_get($payload, 'uuid')) ? (string) data_get($payload, 'uuid') : new_public_id(); + if (SecretManagerLink::where('uuid', $uuid)->exists()) { + $uuid = new_public_id(); + } + + $link = new SecretManagerLink; + $link->forceFill([ + 'resourceable_type' => $resource->getMorphClass(), + 'resourceable_id' => $resource->getKey(), + 'integration_token_id' => $tokenId, + 'settings' => data_get($payload, 'settings'), + ]); + $link->uuid = $uuid; + $link->save(); + } + + /** + * A zone of the same token and provider zone that the target team already has is reused. + * + * @param list> $zones + */ + private function importDnsZones(array $zones, bool $preserveUuids): void + { + foreach ($zones as $payload) { + $token = $this->integrationTokenMap[(string) data_get($payload, 'integration_token_uuid')] ?? null; + if (! $token) { + continue; + } + + $zone = DnsProviderZone::query() + ->where('integration_token_id', $token->id) + ->where('provider_zone_id', (string) data_get($payload, 'provider_zone_id')) + ->first(); + if (! $zone) { + $uuid = $preserveUuids && filled(data_get($payload, 'uuid')) ? (string) data_get($payload, 'uuid') : new_public_id(); + if (DnsProviderZone::where('uuid', $uuid)->exists()) { + $uuid = new_public_id(); + } + $zone = new DnsProviderZone; + $zone->forceFill([ + 'integration_token_id' => $token->id, + 'provider_zone_id' => (string) data_get($payload, 'provider_zone_id'), + 'name' => strtolower((string) data_get($payload, 'name')), + 'account_id' => data_get($payload, 'account_id'), + 'account_name' => data_get($payload, 'account_name'), + ]); + $zone->uuid = $uuid; + $zone->save(); + } + + $this->dnsZoneMap[(string) data_get($payload, 'uuid', $zone->uuid)] = $zone; + } + } + + /** + * Imports the managed DNS records and links them to the imported resources. An owned record keeps the ownership + * comment of the source at the provider until adoptManagedDnsRecords() replaces it. + * + * @param list> $records + */ + private function importManagedDnsRecords(array $records, int $teamId, bool $preserveUuids): int + { + $count = 0; + foreach ($records as $payload) { + $name = (string) data_get($payload, 'name'); + $zone = $this->dnsZoneMap[(string) data_get($payload, 'dns_provider_zone_uuid')] ?? null; + $resources = collect(data_get($payload, 'references', [])) + ->map(fn ($reference) => match (data_get($reference, 'kind')) { + 'application' => $this->applicationMap[(string) data_get($reference, 'uuid')] ?? null, + 'preview' => $this->previewMap[(string) data_get($reference, 'uuid')] ?? null, + 'service_application' => $this->serviceApplicationMap[(string) data_get($reference, 'uuid')] ?? null, + default => null, + }) + ->filter() + ->values(); + if (! $zone || $resources->isEmpty()) { + $this->importWarnings[] = "DNS record {$name}: its DNS zone or resources were not imported, so this instance does not manage it."; + + continue; + } + + $record = ManagedDnsRecord::query() + ->where('dns_provider_zone_id', $zone->id) + ->where('provider_record_id', (string) data_get($payload, 'provider_record_id')) + ->first(); + if (! $record) { + $uuid = $preserveUuids && filled(data_get($payload, 'uuid')) ? (string) data_get($payload, 'uuid') : new_public_id(); + if (ManagedDnsRecord::where('uuid', $uuid)->exists()) { + $uuid = new_public_id(); + } + $previousComment = data_get($payload, 'ownership_comment'); + // Without the source comment there is no proof of ownership at the provider. + $owned = (bool) data_get($payload, 'owned', false) && is_string($previousComment) && $previousComment !== ''; + + $record = ManagedDnsRecord::query()->create([ + 'uuid' => $uuid, + 'team_id' => $teamId, + 'integration_token_id' => $zone->integration_token_id, + 'dns_provider_zone_id' => $zone->id, + 'provider_record_id' => (string) data_get($payload, 'provider_record_id'), + 'type' => (string) data_get($payload, 'type'), + 'name' => $name, + 'content' => (string) data_get($payload, 'content'), + 'owned' => $owned, + ]); + if ($owned && $previousComment !== $record->ownershipComment()) { + $this->pendingDnsAdoptions[] = ['record_id' => $record->id, 'previous_comment' => $previousComment]; + } + $count++; + } + + foreach ($resources as $resource) { + $record->addReference($resource); + } + } + + return $count; + } + + /** + * Replaces the ownership comment of the source instance with the comment of this instance at the provider. + * A record that cannot be adopted is no longer owned, so this instance never changes or deletes it. + * + * @return list warnings + */ + private function adoptManagedDnsRecords(): array + { + $warnings = []; + foreach ($this->pendingDnsAdoptions as $adoption) { + $record = ManagedDnsRecord::query()->find($adoption['record_id']); + if (! $record) { + continue; + } + + try { + $adopted = app(CloudflareDnsProvider::class)->adoptRecord($record, $adoption['previous_comment']); + } catch (Throwable) { + $adopted = false; + } + if ($adopted) { + continue; + } + + $record->forceFill(['owned' => false])->save(); + $warnings[] = "DNS record {$record->name}: this instance could not take ownership; it will not change or delete it."; + } + $this->pendingDnsAdoptions = []; + + return $warnings; + } + /** * @param list> $previews */ @@ -1531,6 +2247,8 @@ class ServerTransferImporter 'git_type' => data_get($previewPayload, 'git_type'), 'docker_compose_domains' => data_get($previewPayload, 'docker_compose_domains'), 'docker_registry_image_tag' => data_get($previewPayload, 'docker_registry_image_tag'), + 'domain_port_overrides' => data_get($previewPayload, 'domain_port_overrides'), + 'max_restart_count' => (int) data_get($previewPayload, 'max_restart_count', 0), ]; if ($preview) { @@ -1551,6 +2269,8 @@ class ServerTransferImporter $preview->save(); } + $this->previewMap[(string) data_get($previewPayload, 'uuid', $preview->uuid)] = $preview; + // Avoid duplicating volumes when re-importing the same preview. if ($preview->wasRecentlyCreated || $preview->persistentStorages()->count() === 0) { $this->importPersistentStorages(data_get($previewPayload, 'persistent_storages', []), $preview); @@ -1630,6 +2350,7 @@ class ServerTransferImporter $s3Ids = $this->importedBackupS3StorageIds($backupPayload); $s3Id = $s3Ids[0] ?? null; + $this->warnAboutMissingS3Storage($backupPayload, $s3Id, 'volume '.($volume->name ?? $volume->mount_path)); $importedBackup = ScheduledVolumeBackup::create([ 'uuid' => $uuid, @@ -1649,6 +2370,7 @@ class ServerTransferImporter 'retention_days_s3' => data_get($backupPayload, 'retention_days_s3'), 'retention_max_storage_s3' => data_get($backupPayload, 'retention_max_storage_s3'), 'timeout' => data_get($backupPayload, 'timeout'), + 'missing_backup_notification_days' => (int) data_get($backupPayload, 'missing_backup_notification_days', 0), ]); $importedBackup->syncS3Storages($s3Ids); $count++; diff --git a/app/Services/ServerTransfer/ServerTransferMigrator.php b/app/Services/ServerTransfer/ServerTransferMigrator.php index 1c2571e3d0..bb51b563f5 100644 --- a/app/Services/ServerTransfer/ServerTransferMigrator.php +++ b/app/Services/ServerTransfer/ServerTransferMigrator.php @@ -2,23 +2,402 @@ namespace App\Services\ServerTransfer; +use App\Enums\ApplicationDeploymentStatus; +use App\Jobs\MigrateServerToInstanceJob; +use App\Models\Application; +use App\Models\ApplicationDeploymentQueue; +use App\Models\Environment; +use App\Models\GithubApp; +use App\Models\GitlabApp; +use App\Models\Project; use App\Models\Server; +use App\Models\Service; +use App\Models\User; use App\Rules\SafeWebhookUrl; use App\Services\Security\SensitiveDataRedactor; +use Carbon\Carbon; +use Illuminate\Database\Eloquent\Model; use Illuminate\Database\QueryException; use Illuminate\Http\Client\ConnectionException; +use Illuminate\Http\Client\PendingRequest; +use Illuminate\Support\Facades\DB; use Illuminate\Support\Facades\Http; use Illuminate\Support\Facades\Validator; +use Illuminate\Support\Sleep; +use Illuminate\Support\Str; use RuntimeException; use Throwable; class ServerTransferMigrator { + /** Time the target gets for one import request. Its nginx can stop waiting earlier (99 s). */ + public const IMPORT_TIMEOUT_SECONDS = 300; + + /** How often and how long to ask the target for the result after the import request failed. */ + public const STATUS_PROBE_ATTEMPTS = 20; + + public const STATUS_PROBE_INTERVAL_SECONDS = 15; + + public const STATUS_REQUEST_TIMEOUT_SECONDS = 15; + + public const CHECK_TIMEOUT_SECONDS = 15; + + /** + * Time the queued job gets. It must be longer than the import request and all status probes together, + * plus time for the export, so the worker does not stop the job after the target took the server. + */ + public const MIGRATION_TIMEOUT_SECONDS = 1500; + + /** Longer than the job timeout, so a running state older than this one is stale. */ + public const STALE_MIGRATION_SECONDS = 1800; + public function __construct( private ServerTransferExporter $exporter, private ServerTransferClaimer $claimer, ) {} + /** The UI shows this many of the latest log lines. */ + public const MAX_LOG_ENTRIES = 100; + + /** + * System-wide git sources that the target of the last precheck does not have. + * + * @var array{github: list, gitlab: list} + */ + private array $missingSystemSources = ['github' => [], 'gitlab' => []]; + + /** + * Run the prechecks now, so the user sees problems at once, then run the transfer in a queued job. + * Every check writes a line to the transfer log; a failed check stops the transfer before it starts. + */ + public function start(Server $server, string $targetUrl, string $targetToken): void + { + $targetUrl = $this->normalizeTargetUrl($targetUrl); + $token = $this->normalizeToken($targetToken); + if ($this->hasRunningMigration($server->fresh())) { + throw new RuntimeException('A transfer of this server is already running.'); + } + + $startedAt = now()->toIso8601String(); + $log = [$this->logEntry("Running checks before the transfer to {$targetUrl}.")]; + $checks = [ + ...$this->migrationChecks($server), + $this->sshCheck($server), + ...$this->targetChecks($server, $targetUrl, $token), + ]; + foreach ($checks as $check) { + $log[] = match ($check['ok']) { + true => ['at' => $check['at'], 'level' => 'success', 'message' => "OK: {$check['label']}."], + false => ['at' => $check['at'], 'level' => 'error', 'message' => "FAILED: {$check['label']}. {$check['message']}"], + null => ['at' => $check['at'], 'level' => 'warning', 'message' => "NOT CHECKED: {$check['label']}. {$check['message']}"], + }; + } + foreach ([...$this->sharedProjectNotes($server), ...$this->systemSourceNotes($server)] as $note) { + $log[] = $this->logEntry($note); + } + + $failed = array_values(array_filter($checks, fn (array $check) => $check['ok'] === false)); + if ($failed !== []) { + $error = count($failed).' of '.count($checks).' checks failed: '.implode(' ', array_column($failed, 'message')); + $log[] = $this->logEntry('The transfer did not start, because '.count($failed).' check(s) failed. Fix them, then transfer again.', 'error'); + DB::transaction(function () use ($server, $targetUrl, $startedAt, $error, $log) { + $this->writeMigration(Server::query()->lockForUpdate()->findOrFail($server->id), [ + 'status' => 'failed', + 'phase' => 'checks', + 'target_url' => $targetUrl, + 'started_at' => $startedAt, + 'finished_at' => now()->toIso8601String(), + 'error' => $error, + 'log' => $log, + ]); + }); + + throw new RuntimeException($error); + } + + $log[] = $this->logEntry('All checks passed. The transfer is queued.'); + DB::transaction(function () use ($server, $targetUrl, $startedAt, $log) { + $locked = Server::query()->lockForUpdate()->findOrFail($server->id); + if ($this->hasRunningMigration($locked)) { + throw new RuntimeException('A transfer of this server is already running.'); + } + $this->writeMigration($locked, [ + 'status' => 'running', + 'target_url' => $targetUrl, + 'started_at' => $startedAt, + 'log' => $log, + ]); + }); + + MigrateServerToInstanceJob::dispatch($server, $targetUrl, $token); + } + + /** + * Save the state of the queued transfer on the source server, so the UI and the API can show it. + * + * @param array $details + */ + public function recordMigration(Server $server, string $status, array $details = []): void + { + $entry = match ($status) { + 'succeeded' => $this->logEntry('Transfer finished. The target instance manages the server now.', 'success'), + 'unknown' => $this->logEntry('Transfer result unknown: '.data_get($details, 'error', 'unknown error'), 'warning'), + default => $this->logEntry('Transfer failed: '.data_get($details, 'error', 'unknown error'), 'error'), + }; + + DB::transaction(function () use ($server, $status, $details, $entry) { + $locked = Server::query()->lockForUpdate()->find($server->id); + if (! $locked) { + return; + } + $migration = array_merge((array) data_get($locked->server_metadata, 'migration', []), $details, [ + 'status' => $status, + 'finished_at' => now()->toIso8601String(), + ]); + $migration['log'] = $this->appendToLog((array) data_get($migration, 'log', []), $entry); + $this->writeMigration($locked, $migration); + }); + } + + /** + * Record the result of a transfer job that the worker stopped, for example at its timeout. When the export + * was already sent, ask the target once, because the target can manage the server already. + */ + public function recordStoppedMigration(Server $server, string $targetUrl, string $targetToken, string $error): void + { + $exportId = data_get($server->fresh()?->server_metadata, 'migration.export_id'); + if (! is_string($exportId)) { + $this->recordMigration($server, 'failed', ['error' => $error]); + + return; + } + + $targetUrl = $this->normalizeTargetUrl($targetUrl); + $status = $this->targetTransferStatus($targetUrl, $this->normalizeToken($targetToken), $server->uuid, $exportId); + + if ($status === 'claimed') { + try { + $this->claimer->markTransferred($server, exportId: $exportId, targetInstanceUrl: $targetUrl); + $this->recordMigration($server, 'succeeded', [ + 'warnings' => ['The transfer job stopped before it finished. The target status confirms that the target manages the server.'], + ]); + } catch (Throwable) { + $this->recordMigration($server, 'unknown', [ + 'error' => "{$error} The target manages the server, but this instance could not mark it as transferred. Call POST /api/v1/servers/{$server->uuid}/transfer/complete on this instance.", + ]); + } + + return; + } + + // The target answered that it does not manage this export, so this instance still manages the server. + if (in_array($status, ['imported', 'absent'], true)) { + $this->recordMigration($server, 'failed', ['error' => $error]); + + return; + } + + $this->recordMigration($server, 'unknown', [ + 'error' => "{$error} This instance could not ask {$targetUrl} whether it imported the server. ". + "Check the server on the target. If the target manages it, call POST /api/v1/servers/{$server->uuid}/transfer/complete on this instance. ". + 'If not, transfer again.', + ]); + } + + /** + * Add one line to the transfer log, so the UI can show where the transfer is. + * + * @param array $fields values to save in the transfer state with the line + */ + public function log(Server $server, string $message, string $level = 'info', array $fields = []): void + { + $entry = $this->logEntry($message, $level); + + DB::transaction(function () use ($server, $entry, $fields) { + $locked = Server::query()->lockForUpdate()->find($server->id); + if (! $locked) { + return; + } + $migration = array_merge((array) data_get($locked->server_metadata, 'migration', []), $fields); + $migration['log'] = $this->appendToLog((array) data_get($migration, 'log', []), $entry); + $this->writeMigration($locked, $migration); + }); + } + + /** + * @return array{at: string, level: string, message: string} + */ + private function logEntry(string $message, string $level = 'info'): array + { + return ['at' => now()->toIso8601String(), 'level' => $level, 'message' => $message]; + } + + /** + * @param list> $log + * @param array $entry + * @return list> + */ + private function appendToLog(array $log, array $entry): array + { + return array_slice([...$log, $entry], -self::MAX_LOG_ENTRIES); + } + + /** + * @return array{label: string, ok: bool|null, message: string|null, at: string} + */ + private function check(string $label, ?bool $ok, ?string $message = null): array + { + return ['label' => $label, 'ok' => $ok, 'message' => $message, 'at' => now()->toIso8601String()]; + } + + /** + * @return array{label: string, ok: bool|null, message: string|null, at: string} + */ + private function sshCheck(Server $server): array + { + ['uptime' => $reachable, 'error' => $error] = $server->validateConnection(); + + return $this->check( + 'this instance can connect to the server over SSH', + (bool) $reachable, + $reachable ? null : 'This instance cannot connect to the server over SSH: '.rtrim($error ?: 'unknown error', '. ').'. Fix the connection, then transfer again.', + ); + } + + /** + * A running state older than the job can run is left over from a stopped worker, so it does not block a new start. + */ + private function hasRunningMigration(Server $server): bool + { + $startedAt = data_get($server->server_metadata, 'migration.started_at'); + + return data_get($server->server_metadata, 'migration.status') === 'running' + && is_string($startedAt) + && Carbon::parse($startedAt)->gt(now()->subSeconds(self::STALE_MIGRATION_SECONDS)); + } + + /** + * @param array $migration + */ + private function writeMigration(Server $server, array $migration): void + { + $metadata = $server->server_metadata ?? []; + $metadata['migration'] = $migration; + $server->server_metadata = $metadata; + $server->save(); + } + + /** + * @return list + */ + private function migrationChecks(Server $server): array + { + $activeDeployments = ApplicationDeploymentQueue::query() + ->where('server_id', $server->id) + ->whereIn('status', [ApplicationDeploymentStatus::QUEUED->value, ApplicationDeploymentStatus::IN_PROGRESS->value]) + ->count(); + + return [ + $this->check('the server is not the Coolify host', $server->id !== 0, + 'The Coolify host (localhost) cannot be transferred between instances.'), + $this->check('this instance manages the server', ! $server->isTransferredAway(), + 'This server was already transferred to another instance.'), + $this->check('no deployment is queued or running on the server', $activeDeployments === 0, + "{$activeDeployments} deployment(s) on this server are queued or running. Wait until they finish or cancel them, then transfer the server."), + ]; + } + + /** + * System-wide git sources are not transferred: an application re-links only to one with the same uuid on the + * target. One note per source that the target does not have, with the applications that lose their source. + * + * @return list + */ + private function systemSourceNotes(Server $server): array + { + $notes = []; + foreach ($this->systemSources($server) as $kind => $sources) { + foreach ($sources as $uuid => $source) { + if (in_array($uuid, $this->missingSystemSources[$kind], true)) { + $label = $kind === 'github' ? 'GitHub App' : 'GitLab App'; + $notes[] = "NOTE: the system-wide {$label} '{$source['name']}' of this instance is not transferred, and the target has no matching one. " + .'These applications need a new git source on the target: '.implode(', ', $source['applications']).'. The finish list on the target shows them.'; + } + } + } + + return $notes; + } + + /** + * @return array{github: array}>, gitlab: array}>} + */ + private function systemSources(Server $server): array + { + $sources = ['github' => [], 'gitlab' => []]; + foreach ($server->applications()->unique('id') as $application) { + $source = $application->source; + $kind = match (true) { + $source instanceof GithubApp => 'github', + $source instanceof GitlabApp => 'gitlab', + default => null, + }; + if ($kind === null || ! $source->is_system_wide) { + continue; + } + $sources[$kind][$source->uuid]['name'] = $source->name; + $sources[$kind][$source->uuid]['applications'][] = $application->name; + } + + return $sources; + } + + /** + * One note for every project that has resources on this server and on other servers. Only the + * resources of this server are transferred, so the user must know that the project is split. + * + * @return list + */ + private function sharedProjectNotes(Server $server): array + { + $resourcesOnServer = collect([ + ...$server->applications(), + ...$server->databases(), + ...$server->services()->get(), + ]); + $keysOnServer = $resourcesOnServer->map(fn (Model $resource) => $resource::class.':'.$resource->getKey())->flip(); + + $projectIds = Environment::query() + ->whereIn('id', $resourcesOnServer->pluck('environment_id')->filter()->unique()) + ->distinct() + ->pluck('project_id'); + $projectIdByEnvironment = Environment::query()->whereIn('project_id', $projectIds)->pluck('project_id', 'id'); + + $otherServerCounts = collect([Application::class, Service::class, ...array_values(STANDALONE_DATABASE_MODELS)]) + ->flatMap(fn (string $model) => $model::query() + ->whereIn('environment_id', $projectIdByEnvironment->keys()) + ->pluck('environment_id', 'id') + ->reject(fn (int|string $environmentId, int|string $id) => $keysOnServer->has("{$model}:{$id}"))) + ->countBy(fn (int|string $environmentId) => $projectIdByEnvironment[$environmentId]); + + return Project::query() + ->whereIn('id', $otherServerCounts->keys()) + ->orderBy('name') + ->get(['id', 'name']) + ->map(fn (Project $project) => "NOTE: project '{$project->name}' also has {$otherServerCounts[$project->id]} ". + Str::plural('resource', $otherServerCounts[$project->id]).' on other servers. They stay on this instance; only the resources of this server are transferred.') + ->all(); + } + + private function assertCanMigrate(Server $server): void + { + foreach ($this->migrationChecks($server) as $check) { + if (! $check['ok']) { + throw new RuntimeException($check['message']); + } + } + } + /** * One-shot migrate: export on this instance → import+claim on target → complete on this instance. * @@ -29,37 +408,41 @@ class ServerTransferMigrator * import: array, * complete: array, * warnings: list, + * checklist: list, * message: string * } */ - public function migrate( - Server $server, - string $targetUrl, - string $targetToken, - bool $preserveUuids = true, - bool $adoptMode = true, - ): array { - if ($server->id === 0) { - throw new RuntimeException('The Coolify host (localhost) cannot be transferred between instances.'); - } + public function migrate(Server $server, string $targetUrl, string $targetToken): array + { + $server->refresh(); + $this->assertCanMigrate($server); $targetUrl = $this->normalizeTargetUrl($targetUrl); $token = $this->normalizeToken($targetToken); + $this->log($server, 'Exporting the server configuration and its resources.'); $bundle = $this->exporter->export($server, includeSensitive: true); $exportId = data_get($bundle, 'export_id'); $warnings = array_values((array) data_get($bundle, 'warnings', [])); + $checklist = $this->postTransferChecklist($bundle, $targetUrl); + // A job that the worker stops after this line reads the export ID to ask the target for the result. + $this->log($server, "Export {$exportId} created. Sending it to {$targetUrl}. The target imports the server and takes over management.", fields: ['export_id' => $exportId]); // Target import+claim runs in its own DB transaction on the remote instance. // Source DB is unchanged until markTransferred below — so a failed remote import leaves source intact. $importBody = $this->postImportToTarget( + server: $server, targetUrl: $targetUrl, token: $token, bundle: $bundle, - preserveUuids: $preserveUuids, - adoptMode: $adoptMode, ); + // The target fills in its own public URL; the URL typed here can be an internal address. + $targetChecklist = array_values(array_filter((array) data_get($importBody, 'checklist', []), 'is_string')); + if ($targetChecklist !== []) { + $checklist = $targetChecklist; + } + if (is_array(data_get($importBody, 'warnings'))) { $warnings = array_values(array_unique(array_merge($warnings, $importBody['warnings']))); } @@ -68,13 +451,15 @@ class ServerTransferMigrator if (data_get($importBody, 'claimed') !== true) { throw new RuntimeException( "Server was imported on {$targetUrl}, but the target could not take management of it. ". - 'This instance still manages the server. Fix the problem on the target, click Enable management there, then click Disable management here. '. + 'This instance still manages the server. Delete the imported server on the target, fix the problem, then transfer again. '. 'Target warnings: '.(implode(' ', (array) data_get($importBody, 'warnings', [])) ?: 'none') ); } + $this->log($server, 'The target imported the server and manages it now. Disabling management on this instance.', 'success'); + // Cross-instance 2PC is impossible: if complete fails after a successful import, the target - // already owns the server. Surface that clearly so the operator can disable management here. + // already owns the server. Surface that clearly so the operator can complete the transfer here. try { $complete = $this->claimer->markTransferred( $server, @@ -86,7 +471,7 @@ class ServerTransferMigrator $reason = $e instanceof QueryException ? 'a database error occurred' : remove_iip($e->getMessage()); throw new RuntimeException( "Server was imported on {$targetUrl}, but this instance could not mark it as transferred: {$reason}. ". - 'Disable management of this server here (or API: POST /api/v1/servers/{uuid}/transfer/complete) so automations stay disabled. Do not re-import on the target.', + 'Call POST /api/v1/servers/{uuid}/transfer/complete on this instance so automations stay disabled here. Do not re-import on the target.', previous: $e ); } @@ -98,10 +483,77 @@ class ServerTransferMigrator 'import' => app(SensitiveDataRedactor::class)->redactValue($importBody), 'complete' => $complete, 'warnings' => $warnings, + 'checklist' => $checklist, + 'target_server_url' => is_string(data_get($importBody, 'server_url')) ? data_get($importBody, 'server_url') : null, 'message' => 'Server migrated to '.$targetUrl.'. Automations disabled on this instance.', ]; } + /** + * Manual steps the user must do after the transfer. The exporter writes {target_url} where the URL of the target belongs. + * + * @param array $bundle + * @return list + */ + private function postTransferChecklist(array $bundle, string $targetUrl): array + { + $steps = array_filter((array) data_get($bundle, 'post_transfer_checklist', []), 'is_string'); + + return array_values(array_unique(array_map( + fn (string $step) => str_replace('{target_url}', $targetUrl, $step), + $steps, + ))); + } + + /** + * Undo a transfer: ask the instance that manages the server now to transfer it back to this instance. That + * instance runs its prechecks at once and then a normal transfer; the import here replaces the archive. + * This instance creates a short-lived token for the import, scoped to the team of the archive. + */ + public function requestTransferBack(Server $archive, User $user, string $managerUrl, string $managerToken, string $thisInstanceUrl): void + { + if (! $archive->isTransferredAway()) { + throw new RuntimeException('Only a server that was transferred to another instance can be transferred back.'); + } + + $managerUrl = $this->normalizeTargetUrl($managerUrl); + $managerToken = $this->normalizeToken($managerToken); + $thisInstanceUrl = rtrim(trim($thisInstanceUrl), '/'); + if (! filter_var($thisInstanceUrl, FILTER_VALIDATE_URL)) { + throw new RuntimeException('Enter the URL at which the other instance reaches this instance.'); + } + + $importToken = $user->createToken("transfer back {$archive->uuid}", ['read', 'write'], now()->addHours(3), $archive->team_id); + $migrateUrl = $managerUrl.'/api/v1/servers/'.rawurlencode($archive->uuid).'/migrate'; + + try { + $response = $this->targetRequest($migrateUrl, $managerToken, self::CHECK_TIMEOUT_SECONDS * 4) + ->post($migrateUrl, ['target_url' => $thisInstanceUrl, 'target_token' => $importToken->plainTextToken]); + } catch (ConnectionException $e) { + $importToken->accessToken->delete(); + throw new RuntimeException('This instance cannot reach '.remove_iip($managerUrl).': '.remove_iip($e->getMessage()), previous: $e); + } + + if ($response->status() !== 202) { + $importToken->accessToken->delete(); + $message = remove_iip((string) (data_get($response->json(), 'message') ?: 'no details')); + throw new RuntimeException(match ($response->status()) { + 401 => 'The other instance does not accept the API token.', + 403 => 'The API token of the other instance needs root, or read:sensitive and write, from an admin or owner. '.$message, + 404 => 'The other instance does not know this server, or does not support transfers. '.$message, + default => "The other instance did not start the transfer back (HTTP {$response->status()}): {$message}", + }); + } + + DB::transaction(function () use ($archive, $managerUrl) { + $locked = Server::query()->lockForUpdate()->findOrFail($archive->id); + $metadata = $locked->server_metadata ?? []; + $metadata['transfer']['transfer_back'] = ['requested_at' => now()->toIso8601String(), 'from' => $managerUrl]; + $locked->server_metadata = $metadata; + $locked->save(); + }); + } + private function normalizeTargetUrl(string $targetUrl): string { $targetUrl = rtrim(trim($targetUrl), '/'); @@ -121,6 +573,10 @@ class ServerTransferMigrator return $targetUrl; } + if (strtolower((string) parse_url($targetUrl, PHP_URL_SCHEME)) !== 'https') { + throw new RuntimeException('The target instance URL must use https, because the transfer sends credentials.'); + } + Validator::make(['target_url' => $targetUrl], [ 'target_url' => ['required', new SafeWebhookUrl], ])->validate(); @@ -152,31 +608,21 @@ class ServerTransferMigrator * @param array $bundle * @return array */ - private function postImportToTarget( - string $targetUrl, - string $token, - array $bundle, - bool $preserveUuids, - bool $adoptMode, - ): array { + private function postImportToTarget(Server $server, string $targetUrl, string $token, array $bundle): array + { $importUrl = $targetUrl.'/api/v1/servers/import'; try { - $response = Http::timeout(120) - ->withOptions($this->isLocalDevelopmentTarget($importUrl) - ? ['allow_redirects' => false] - : SafeWebhookUrl::httpClientOptions($importUrl)) - ->acceptJson() - ->withToken($token) - ->asJson() + $response = $this->targetRequest($importUrl, $token, self::IMPORT_TIMEOUT_SECONDS) ->post($importUrl, [ 'bundle' => $bundle, 'dry_run' => false, - 'preserve_uuids' => $preserveUuids, - 'adopt_mode' => $adoptMode, + 'preserve_uuids' => true, + 'adopt_mode' => true, ]); } catch (ConnectionException $e) { - throw new RuntimeException('Could not reach target instance at '.remove_iip($importUrl).': '.remove_iip($e->getMessage()), previous: $e); + return $this->resultFromTargetStatus($server, $targetUrl, $token, $bundle, 'the request could not reach the target or timed out') + ?? throw new RuntimeException('Could not reach target instance at '.remove_iip($importUrl).': '.remove_iip($e->getMessage()), previous: $e); } catch (Throwable $e) { throw new RuntimeException('Transfer to target failed: '.remove_iip($e->getMessage()), previous: $e); } @@ -186,6 +632,18 @@ class ServerTransferMigrator $importBody = ['raw' => remove_iip($response->body())]; } + if ($response->status() === 404) { + throw new RuntimeException('The target instance does not support server transfer. Upgrade it to Coolify '.config('constants.coolify.version').' or newer.'); + } + + // A proxy timeout (502/504) or a server error does not mean that the import failed: the target can still finish it. + if ($response->serverError()) { + $result = $this->resultFromTargetStatus($server, $targetUrl, $token, $bundle, "the target answered with HTTP {$response->status()}"); + if ($result !== null) { + return $result; + } + } + if (! $response->successful()) { $message = data_get($importBody, 'message') ?? data_get($importBody, 'errors') @@ -198,4 +656,155 @@ class ServerTransferMigrator return $importBody; } + + /** + * Ask the target whether it accepts the token and can import this server, so a wrong token or a + * conflict on the target fails before the transfer starts instead of in the queued job. When a + * check fails that the target checks depend on, those checks are reported as not checked. + * + * @return list + */ + private function targetChecks(Server $server, string $targetUrl, string $token): array + { + $checkUrl = $targetUrl.'/api/v1/servers/import/check'; + $reachLabel = 'the target instance at '.remove_iip($targetUrl).' can be reached'; + $notChecked = fn (string $label) => $this->check('the checks on the target instance', null, "They did not run, because \"{$label}\" failed."); + + $systemSources = $this->systemSources($server); + $this->missingSystemSources = ['github' => [], 'gitlab' => []]; + try { + $response = $this->targetRequest($checkUrl, $token, self::CHECK_TIMEOUT_SECONDS) + ->get($checkUrl, [ + 'server_uuid' => $server->uuid, + 'ip' => $server->ip, + 'port' => $server->port, + 'schema_version' => ServerTransferBundle::SCHEMA_VERSION, + 'system_github_apps' => array_keys($systemSources['github']), + 'system_gitlab_apps' => array_keys($systemSources['gitlab']), + ]); + } catch (ConnectionException $e) { + return [ + $this->check($reachLabel, false, 'This instance cannot reach the target instance: '.remove_iip($e->getMessage())), + $notChecked($reachLabel), + ]; + } + + $checks = [$this->check($reachLabel, true)]; + $status = $response->status(); + $message = remove_iip((string) (data_get($response->json(), 'message') ?: 'no details')); + + // An unknown route answers 404 before the token is checked, so this check comes first. + $gates = [ + [404, 'the target instance supports server transfer', 'The target instance does not support server transfer. Upgrade it to Coolify '.config('constants.coolify.version').' or newer.'], + [401, 'the target instance accepts the API token', 'The target instance does not accept the API token. Create a new token on the target instance and paste it here.'], + [403, 'the API token can create servers on the target instance', 'Use a token with the write permission from an admin or owner. Target response: '.$message], + ]; + foreach ($gates as [$failStatus, $label, $failMessage]) { + if ($status === $failStatus) { + return [...$checks, $this->check($label, false, $failMessage), $notChecked($label)]; + } + $checks[] = $this->check($label, true); + } + + foreach (['github', 'gitlab'] as $kind) { + $this->missingSystemSources[$kind] = array_values(array_filter((array) data_get($response->json(), "missing_system_sources.{$kind}", []), 'is_string')); + } + + $remoteChecks = data_get($response->json(), 'checks'); + if (! in_array($status, [200, 422], true) || ! is_array($remoteChecks)) { + return [...$checks, $this->check('the target instance answered the checks', false, "The target instance answered with HTTP {$status}: {$message}")]; + } + + foreach ($remoteChecks as $remoteCheck) { + $checks[] = $this->check( + 'on the target: '.data_get($remoteCheck, 'label', 'unknown check'), + (bool) data_get($remoteCheck, 'ok'), + is_string(data_get($remoteCheck, 'message')) ? remove_iip(data_get($remoteCheck, 'message')) : null, + ); + } + + return $checks; + } + + /** + * Ask the target whether it imported this export. Returns null when the target does not know it. + * + * @param array $bundle + * @return array|null + */ + private function resultFromTargetStatus(Server $server, string $targetUrl, string $token, array $bundle, string $reason): ?array + { + $serverUuid = (string) data_get($bundle, 'server.uuid'); + $exportId = (string) data_get($bundle, 'export_id'); + + $this->log($server, "The import request did not finish ({$reason}). The target can still finish the import, so this instance asks the target for the result.", 'warning'); + + for ($attempt = 1; $attempt <= self::STATUS_PROBE_ATTEMPTS; $attempt++) { + Sleep::for(self::STATUS_PROBE_INTERVAL_SECONDS)->seconds(); + $this->log($server, "Asking the target for the import result (attempt {$attempt} of ".self::STATUS_PROBE_ATTEMPTS.').'); + + $status = $this->targetTransferStatus($targetUrl, $token, $serverUuid, $exportId); + + // A rejected token stays rejected, so more attempts cannot confirm the import. + if ($status === 'rejected') { + return null; + } + + if (in_array($status, ['claimed', 'imported'], true)) { + return [ + 'server_uuid' => $serverUuid, + 'export_id' => $exportId, + 'claimed' => $status === 'claimed', + 'warnings' => ['The import request to the target timed out. The target status confirms the import.'], + ]; + } + } + + return null; + } + + /** + * Ask the target once what it knows about this export: 'claimed' or 'imported', 'absent' when the target + * does not have it, 'rejected' when the target rejects the token, or 'unknown' when there is no clear answer. + * + * @return 'claimed'|'imported'|'absent'|'rejected'|'unknown' + */ + private function targetTransferStatus(string $targetUrl, string $token, string $serverUuid, string $exportId): string + { + $statusUrl = $targetUrl.'/api/v1/servers/'.rawurlencode($serverUuid).'/transfer'; + + try { + $response = $this->targetRequest($statusUrl, $token, self::STATUS_REQUEST_TIMEOUT_SECONDS)->get($statusUrl); + } catch (Throwable) { + return 'unknown'; + } + + if (in_array($response->status(), [401, 403], true)) { + return 'rejected'; + } + if ($response->status() === 404) { + return 'absent'; + } + if (! $response->successful()) { + return 'unknown'; + } + if (data_get($response->json(), 'transfer.export_id') !== $exportId) { + return 'absent'; + } + + $status = data_get($response->json(), 'transfer.status'); + + return in_array($status, ['claimed', 'imported'], true) ? $status : 'unknown'; + } + + private function targetRequest(string $url, string $token, int $timeout): PendingRequest + { + return Http::timeout($timeout) + ->withOptions($this->isLocalDevelopmentTarget($url) + ? ['allow_redirects' => false] + : SafeWebhookUrl::httpClientOptions($url)) + ->acceptJson() + ->withToken($token) + ->asJson(); + } } diff --git a/app/Services/ServerTransfer/TransferFinishChecklist.php b/app/Services/ServerTransfer/TransferFinishChecklist.php new file mode 100644 index 0000000000..ee8d93dd79 --- /dev/null +++ b/app/Services/ServerTransfer/TransferFinishChecklist.php @@ -0,0 +1,581 @@ + */ + public const GROUPS = [ + 'server' => 'Server', + 'git' => 'Git and webhooks', + 'dns' => 'Domains and DNS', + ]; + + private const GROUP_OF_TYPE = [ + 'server_connection' => 'server', 'sentinel' => 'server', 's3_storage' => 'server', + 'github_app_webhook' => 'git', 'github_app_urls' => 'git', 'gitlab_webhook' => 'git', 'manual_webhook' => 'git', + 'git_source' => 'git', 'github_runners' => 'git', + 'dns_record' => 'dns', 'domain' => 'dns', + ]; + + /** Types that the user can only mark as done. */ + private const MANUAL_TYPES = ['github_app_urls']; + + /** Types that Coolify can fix with one click. */ + private const FIX_LABELS = [ + 'github_app_webhook' => 'Set webhook URL', + 'sentinel' => 'Restart Sentinel', + 'dns_record' => 'Take ownership', + ]; + + /** + * Build the list for a server that this instance manages now. It replaces an earlier list. + * + * @return list> + */ + public function start(Server $server): array + { + $items = $this->buildItems($server); + $this->save($server, [ + 'started_at' => now()->toIso8601String(), + 'finished_at' => null, + 'finished_by' => null, + 'skipped' => [], + 'items' => $items, + ]); + + return $items; + } + + /** + * @return array|null + */ + public function of(Server $server): ?array + { + $finish = data_get($server->server_metadata, 'transfer.finish'); + + return is_array($finish) ? $finish : null; + } + + public function isOpen(Server $server): bool + { + $finish = $this->of($server); + + return $finish !== null && $finish['finished_at'] === null && ! $server->isTransferredAway(); + } + + /** + * @return list> + */ + public function openItems(Server $server): array + { + return array_values(array_filter((array) data_get($this->of($server), 'items', []), fn (array $item) => in_array($item['status'], self::OPEN_STATUSES, true))); + } + + public function groupOf(array $item): string + { + return self::GROUP_OF_TYPE[$item['type']] ?? 'server'; + } + + /** + * Run the checks of the open steps in a queued job when the last run is older than RECHECK_AFTER_SECONDS. + */ + public function checkWhenStale(Server $server): void + { + $finish = $this->of($server); + if (! $this->isOpen($server) || $this->openItems($server) === []) { + return; + } + $lastCheckedAt = data_get($finish, 'last_checked_at'); + if (is_string($lastCheckedAt) && Carbon::parse($lastCheckedAt)->isAfter(now()->subSeconds(self::RECHECK_AFTER_SECONDS))) { + return; + } + + CheckTransferFinishStepsJob::dispatch($server); + } + + public function canCheck(array $item): bool + { + return ! in_array($item['type'], self::MANUAL_TYPES, true); + } + + /** + * Where the user does the step. GitHub App links are built on each render, so lists saved before a rename of + * the App, or before the link existed, still point to the right page. + * + * @return array{0: string, 1: string}|null + */ + public function linkOf(array $item): ?array + { + if (in_array($item['type'], ['github_app_webhook', 'github_app_urls'], true)) { + $githubApp = GithubApp::where('uuid', $item['subject'])->first(); + + return $githubApp ? [$githubApp->settingsUrl(), 'Open on GitHub'] : null; + } + + return filled($item['link'] ?? null) ? [$item['link'], $item['link_label'] ?? 'Open'] : null; + } + + public function fixLabel(array $item): ?string + { + return self::FIX_LABELS[$item['type']] ?? null; + } + + public function check(Server $server, string $key): array + { + return $this->update($server, $key, fn (array $item) => $this->runCheck($server, $item)); + } + + public function checkAll(Server $server): void + { + foreach ($this->openItems($server) as $item) { + if ($this->canCheck($item)) { + $this->check($server, $item['key']); + } + } + + $finish = (array) $this->of($server->fresh()); + $finish['last_checked_at'] = now()->toIso8601String(); + $this->save($server, $finish); + } + + public function fix(Server $server, string $key): array + { + return $this->update($server, $key, function (array $item) use ($server) { + if ($this->fixLabel($item) === null) { + throw new RuntimeException('Coolify cannot fix this step. Do it by hand, then mark it as done.'); + } + $this->runFix($server, $item); + + // Sentinel restarts in the background, so a check now would always fail. + return $item['type'] === 'sentinel' + ? ['pending', 'Sentinel restarts. The step turns OK when Sentinel sends data.', null] + : $this->runCheck($server, $item); + }); + } + + public function markDone(Server $server, string $key): array + { + return $this->update($server, $key, fn (array $item) => ['done', 'Marked as done by '.(auth()->user()?->name ?? 'a user').'.', null]); + } + + /** + * The user finished the transfer. Open steps are not checked again; they are kept in the list as skipped. + */ + public function finish(Server $server, User $user): void + { + $finish = $this->of($server) ?? throw new RuntimeException('This server has no transfer to finish.'); + $finish['finished_at'] = now()->toIso8601String(); + $finish['finished_by'] = $user->id; + $finish['skipped'] = array_column($this->openItems($server), 'key'); + $this->save($server, $finish); + } + + /** + * Webhook controllers call this when a verified webhook arrives, so a check can see that the new URL works. + */ + public static function recordWebhook(string $kind, ?string $uuid): void + { + if (filled($uuid)) { + Cache::put("transfer-finish:webhook:{$kind}:{$uuid}", now()->toIso8601String(), now()->addDays(60)); + } + } + + /** + * @return list> + */ + private function buildItems(Server $server): array + { + $applications = $server->applications()->unique('id')->values(); + $serviceApplications = $server->services()->with('applications')->get()->flatMap->applications; + $runnerConfig = $server->githubRunnerConfig()->with('githubApp')->first(); + $items = [ + $this->item('server_connection', 'server', 'Connection to the server'), + $this->item('sentinel', 'server', 'Sentinel sends its data to this instance'), + ]; + + $githubApps = $applications->map(fn (Application $application) => $application->source) + ->push($runnerConfig?->githubApp) + ->filter(fn ($source) => $source instanceof GithubApp && ! $source->is_system_wide && ! $source->is_public) + ->unique('id'); + foreach ($githubApps as $githubApp) { + $items[] = $this->item('github_app_webhook', $githubApp->uuid, "Webhook URL of GitHub App {$githubApp->name}", + 'GitHub must send webhooks to this instance.', [['label' => 'Webhook URL', 'value' => $this->githubWebhookUrl()]]); + $items[] = $this->item('github_app_urls', $githubApp->uuid, "Setup and Callback URL of GitHub App {$githubApp->name}", + 'Set both in the GitHub App settings. GitHub has no API for them.', [ + ['label' => 'Setup URL', 'value' => $this->instanceUrl().'/webhooks/source/github/install'], + ['label' => 'Callback URL', 'value' => $this->instanceUrl().'/login/github/app'], + ]); + } + + $gitlabApps = $applications->map(fn (Application $application) => $application->source) + ->filter(fn ($source) => $source instanceof GitlabApp && ! $source->is_system_wide) + ->unique('id'); + foreach ($gitlabApps as $gitlabApp) { + $items[] = $this->item('gitlab_webhook', $gitlabApp->uuid, "GitLab webhooks of GitLab App {$gitlabApp->name}", + 'Set the webhook URL in each GitLab project that uses it, then push once.', + [['label' => 'Webhook URL', 'value' => $this->instanceUrl().'/webhooks/source/gitlab/events']]); + } + + // These applications get a git source step instead: their source is not on this instance. + $gitSourceApplications = (array) data_get($server->server_metadata, 'transfer.git_source_applications', []); + foreach ($applications as $application) { + // A Docker image application keeps a default git_repository, but it never deploys from git. + if (! in_array($application->uuid, $gitSourceApplications, true) && $application->build_pack !== 'dockerimage' && filled($application->git_repository) && (! $application->source || data_get($application->source, 'is_public') === true)) { + $items[] = $this->item('manual_webhook', $application->uuid, "Manual webhook of {$application->name}", + 'If the repository has a webhook to Coolify, change its URL, then push once.', + array_map(fn (string $provider, string $label) => ['label' => $label, 'value' => "{$this->instanceUrl()}/webhooks/source/{$provider}/events/manual"], + ['github', 'gitlab', 'bitbucket', 'gitea'], ['GitHub', 'GitLab', 'Bitbucket', 'Gitea'])); + } + } + + foreach ($this->dnsRecords($applications, $serviceApplications) as $record) { + $items[] = $this->item('dns_record', $record->uuid, "DNS record {$record->name}"); + } + + $hosts = $applications->flatMap(fn (Application $application) => $application->fqdns) + ->concat($serviceApplications->flatMap(fn (ServiceApplication $application) => explode(',', (string) $application->fqdn))) + ->map(fn ($url) => strtolower((string) parse_url(trim((string) $url), PHP_URL_HOST))) + ->filter() + ->unique(); + foreach ($hosts as $host) { + $items[] = $this->item('domain', $host, "Domain {$host} points to the server"); + } + + foreach ($this->s3Storages($server) as $storage) { + $items[] = $this->item('s3_storage', $storage->uuid, "S3 storage {$storage->name} works from this instance"); + } + + if ($runnerConfig) { + $items[] = $this->item('github_runners', $runnerConfig->uuid, 'GitHub runners on this server', + 'Disabled after the transfer. Enable them when the GitHub App sends its webhooks here.', [], + route('server.github-runners', ['server_uuid' => $server->uuid], false), 'Open runners'); + } + + // Links are relative: the list is built during the import request, whose host can be an internal one. + // The importer lists applications that used a system-wide git source, or a source that this instance lacks. + foreach ($applications->whereIn('uuid', $gitSourceApplications) as $application) { + $environment = $application->environment; + $items[] = $this->item('git_source', $application->uuid, "Git source of {$application->name}", + 'System-wide git sources are not transferred. Select a source of this instance.', [], + $environment?->project ? route('project.application.source', [ + 'project_uuid' => $environment->project->uuid, + 'environment_uuid' => $environment->uuid, + 'application_uuid' => $application->uuid, + ], false) : null, 'Open source'); + } + + return $items; + } + + /** + * @return array{0: string, 1: string, 2: string|null} status, short message, technical reason + */ + private function runCheck(Server $server, array $item): array + { + $startedAt = Carbon::parse((string) data_get($this->of($server), 'started_at', now()->toIso8601String())); + + return match ($item['type']) { + 'server_connection' => ($result = $server->validateConnection())['uptime'] + ? ['ok', 'This instance connects to the server over SSH.', null] + : ['action', 'This instance cannot connect to the server over SSH.', $result['error']], + 'sentinel' => $server->sentinel_updated_at && Carbon::parse($server->sentinel_updated_at)->isAfter($startedAt) + ? ['ok', 'Sentinel sent data at '.Carbon::parse($server->sentinel_updated_at)->format('H:i').'.', null] + : ['action', 'No data from Sentinel since the transfer.', null], + 'github_app_webhook' => $this->checkGithubAppWebhook($item['subject'], $startedAt), + 'gitlab_webhook' => $this->checkWebhookReceipt('gitlab_app', $item['subject'], $startedAt), + 'manual_webhook' => $this->checkWebhookReceipt('application', $item['subject'], $startedAt), + 'dns_record' => $this->checkDnsRecord($item['subject']), + 'domain' => $this->checkDomain($server, $item['subject']), + 's3_storage' => $this->checkS3Storage($item['subject']), + 'git_source' => $this->checkGitSource($item['subject']), + 'github_runners' => $server->githubRunnerConfig()->value('is_enabled') + ? ['ok', 'GitHub runners are enabled.', null] + : ['action', 'Disabled after the transfer. Enable them when the GitHub App sends its webhooks here.', null], + default => throw new RuntimeException('Coolify cannot check this step. Do it by hand, then mark it as done.'), + }; + } + + private function runFix(Server $server, array $item): void + { + match ($item['type']) { + 'github_app_webhook' => $this->githubAppRequest(GithubApp::where('uuid', $item['subject'])->firstOrFail(), 'patch', '/app/hook/config', ['url' => $this->githubWebhookUrl()]), + 'sentinel' => $server->restartSentinel(), + 'dns_record' => app(CloudflareDnsProvider::class)->reclaimRecord(ManagedDnsRecord::where('uuid', $item['subject'])->firstOrFail()) + ? ManagedDnsRecord::where('uuid', $item['subject'])->update(['owned' => true]) + : throw new RuntimeException('Coolify could not take ownership: the record changed at Cloudflare, or Cloudflare cannot be reached.'), + }; + } + + /** + * @return array{0: string, 1: string, 2: string|null} + */ + private function checkGithubAppWebhook(string $uuid, Carbon $startedAt): array + { + $githubApp = GithubApp::where('uuid', $uuid)->first(); + if (! $githubApp) { + return ['unchecked', 'The GitHub App does not exist on this instance anymore.', null]; + } + + try { + $url = (string) data_get($this->githubAppRequest($githubApp, 'get', '/app/hook/config'), 'url'); + } catch (Throwable $e) { + // Without the API, a webhook that arrives here still proves the URL. + [$status, $message] = $this->checkWebhookReceipt('github_app', $uuid, $startedAt); + + return [$status, $status === 'ok' ? $message : 'GitHub cannot be asked with the App key. '.$message, $e->getMessage()]; + } + + return rtrim($url, '/') === $this->githubWebhookUrl() + ? ['ok', 'GitHub sends webhooks to this instance.', null] + : ['action', 'GitHub sends webhooks to another URL.', 'Current URL: '.($url ?: 'none')]; + } + + /** + * @return array{0: string, 1: string, 2: string|null} + */ + private function checkWebhookReceipt(string $kind, string $uuid, Carbon $startedAt): array + { + $receivedAt = Cache::get("transfer-finish:webhook:{$kind}:{$uuid}"); + if (is_string($receivedAt) && Carbon::parse($receivedAt)->isAfter($startedAt)) { + return ['ok', 'A webhook arrived at '.Carbon::parse($receivedAt)->format('H:i').'.', null]; + } + + return ['action', 'No webhook arrived since the transfer.', null]; + } + + /** + * @return array{0: string, 1: string, 2: string|null} + */ + private function checkDnsRecord(string $uuid): array + { + $record = ManagedDnsRecord::where('uuid', $uuid)->first(); + if (! $record) { + return ['unchecked', 'This instance does not track this record anymore.', null]; + } + if ($record->owned && app(CloudflareDnsProvider::class)->reclaimRecord($record)) { + return ['ok', 'This instance manages the record at Cloudflare.', null]; + } + + return $record->owned + ? ['action', 'Cloudflare does not confirm the record.', 'The record changed at Cloudflare, or Cloudflare cannot be reached with the token.'] + : ['action', 'This instance does not own the record yet.', 'Until it owns the record, this instance does not change or delete it.']; + } + + /** + * @return array{0: string, 1: string, 2: string|null} + */ + private function checkDomain(Server $server, string $host): array + { + $serverIps = filter_var($server->ip, FILTER_VALIDATE_IP) ? [$server->ip] : (gethostbynamel($server->ip) ?: []); + $records = @dns_get_record($host, DNS_A | DNS_AAAA) ?: []; + $ips = array_values(array_filter(array_map(fn (array $record) => $record['ip'] ?? $record['ipv6'] ?? null, $records))); + if ($ips === []) { + return ['action', 'The domain does not resolve.', null]; + } + if (array_intersect($ips, $serverIps) !== []) { + return ['ok', 'The domain points to the server.', null]; + } + + return ['action', 'The domain points to another address.', 'Resolves to '.implode(', ', $ips).'; the server is '.implode(', ', $serverIps ?: [$server->ip]).'. If a proxy or load balancer is in front, mark it as done.']; + } + + /** + * @return array{0: string, 1: string, 2: string|null} + */ + private function checkS3Storage(string $uuid): array + { + $storage = S3Storage::where('uuid', $uuid)->first(); + if (! $storage) { + return ['unchecked', 'The S3 storage does not exist on this instance anymore.', null]; + } + try { + $storage->testConnection(shouldSave: true); + } catch (Throwable $e) { + return ['action', 'This instance cannot use the S3 storage.', $e->getMessage()]; + } + + return $storage->fresh()->is_usable ? ['ok', 'This instance can use the S3 storage.', null] : ['action', 'This instance cannot use the S3 storage.', null]; + } + + private function githubAppRequest(GithubApp $githubApp, string $method, string $path, array $data = []): array + { + $response = Http::GitSource($githubApp->api_url) + ->withHeaders(['Authorization' => 'Bearer '.generateGithubJwt($githubApp), 'Accept' => 'application/vnd.github+json']) + ->{$method}(rtrim($githubApp->api_url, '/').$path, $data); + if (! $response->successful()) { + throw new RuntimeException('GitHub answered with HTTP '.$response->status().': '.data_get($response->json(), 'message', 'no details')); + } + + return (array) $response->json(); + } + + /** + * @param Collection $applications + * @param Collection $serviceApplications + * @return Collection + */ + private function dnsRecords(Collection $applications, Collection $serviceApplications): Collection + { + $ids = ManagedDnsRecordReference::query() + ->where(function ($query) use ($applications, $serviceApplications) { + $query->where(fn ($q) => $q->where('resource_type', (new Application)->getMorphClass())->whereIn('resource_id', $applications->pluck('id'))) + ->orWhere(fn ($q) => $q->where('resource_type', (new ServiceApplication)->getMorphClass())->whereIn('resource_id', $serviceApplications->pluck('id'))); + }) + ->pluck('managed_dns_record_id'); + + return ManagedDnsRecord::query()->whereIn('id', $ids)->orderBy('name')->get(); + } + + /** + * @return Collection + */ + private function s3Storages(Server $server): Collection + { + $ids = collect($server->databases()) + ->flatMap(fn ($database) => method_exists($database, 'scheduledBackups') ? $database->scheduledBackups()->get() : []) + ->flatMap(fn ($backup) => [$backup->s3_storage_id, ...$backup->s3Storages()->pluck('s3_storages.id')->all()]) + ->filter() + ->unique(); + + return S3Storage::query()->whereIn('id', $ids)->where('team_id', $server->team_id)->get(); + } + + /** + * @param callable(array): array{0: string, 1: string} $change + * @return array + */ + private function update(Server $server, string $key, callable $change): array + { + $finish = $this->of($server) ?? throw new RuntimeException('This server has no transfer to finish.'); + $index = array_search($key, array_column($finish['items'], 'key'), true); + if ($index === false) { + throw new RuntimeException('This step is not in the list.'); + } + + try { + [$status, $message, $error] = $change($finish['items'][$index]) + [2 => null]; + } catch (Throwable $e) { + [$status, $message, $error] = ['action', 'Coolify could not do this step.', $e->getMessage()]; + } + + return DB::transaction(function () use ($server, $key, $status, $message, $error) { + $finish = (array) data_get(Server::query()->lockForUpdate()->findOrFail($server->id)->server_metadata, 'transfer.finish'); + $index = array_search($key, array_column($finish['items'], 'key'), true); + $finish['items'][$index] = [...$finish['items'][$index], 'status' => $status, 'message' => $message, 'error' => $error, 'checked_at' => now()->toIso8601String()]; + $this->save($server, $finish); + + return $finish['items'][$index]; + }); + } + + /** + * Writes only transfer.finish, on the current row: callers can hold an older copy of the server, for example + * from before the claim, and must not write its other metadata back. + */ + private function save(Server $server, array $finish): void + { + DB::transaction(function () use ($server, $finish) { + $locked = Server::query()->lockForUpdate()->findOrFail($server->id); + $metadata = $locked->server_metadata ?? []; + $metadata['transfer']['finish'] = $finish; + $locked->server_metadata = $metadata; + $locked->save(); + $server->server_metadata = $metadata; + $server->syncOriginalAttribute('server_metadata'); + }); + TransferFinishChanged::dispatch($server); + } + + /** + * @return array{0: string, 1: string} + */ + private function checkGitSource(string $applicationUuid): array + { + $application = Application::where('uuid', $applicationUuid)->first(); + $source = $application?->source; + if (! $source instanceof GithubApp && ! $source instanceof GitlabApp) { + return ['action', 'No git source on this instance, so deployments from git fail.', null]; + } + if ($source instanceof GithubApp && ! $source->is_public) { + try { + generateGithubInstallationToken($source); + } catch (Throwable $e) { + return ['action', "GitHub App {$source->name} cannot read the repository.", $e->getMessage()]; + } + } + + // Push webhooks of a system-wide App go to the instance where the App is registered. + return ['ok', "Uses {$source->name}.", $source->is_system_wide ? 'For push-to-deploy, use a manual webhook if the system-wide App is registered on another instance.' : null]; + } + + /** + * @return array + */ + /** + * @param list $details + */ + private function item(string $type, string $subject, string $label, ?string $hint = null, array $details = [], ?string $link = null, ?string $linkLabel = null): array + { + return [ + 'key' => "{$type}:{$subject}", + 'type' => $type, + 'subject' => $subject, + 'label' => $label, + 'status' => 'pending', + 'hint' => $hint, + 'message' => null, + 'details' => $details, + 'error' => null, + 'link' => $link, + 'link_label' => $linkLabel, + 'checked_at' => null, + ]; + } + + private function instanceUrl(): string + { + return rtrim((string) (instanceSettings()->fqdn ?: config('app.url')), '/'); + } + + private function githubWebhookUrl(): string + { + return $this->instanceUrl().'/webhooks/source/github/events'; + } +} diff --git a/app/Services/SharedVariableCanvas.php b/app/Services/SharedVariableCanvas.php index 4e35da400a..775faa8a48 100644 --- a/app/Services/SharedVariableCanvas.php +++ b/app/Services/SharedVariableCanvas.php @@ -30,6 +30,12 @@ class SharedVariableCanvas public const RESOURCES_ZONE = 'resources'; + /** + * Server variables that Coolify creates on every server. Resources get them only through a reference, + * but every server has them, so the canvas and the overview leave them out to stay readable. + */ + public const BUILT_IN_SERVER_KEYS = ['COOLIFY_SERVER_UUID', 'COOLIFY_SERVER_NAME']; + public function __construct(private EnvironmentCanvas $canvas, private ServerCanvas $serverCanvas) {} /** @@ -37,9 +43,14 @@ class SharedVariableCanvas */ public function payload(string $scope, Model $owner, bool $canEdit): array { - $variables = $this->variables($owner); + $hidden = $scope === 'server' ? self::BUILT_IN_SERVER_KEYS : []; + $variables = $this->variables($owner)->reject(fn (SharedEnvironmentVariable $variable) => in_array($variable->key, $hidden, true))->values(); $resources = $this->resources($scope, $owner); - $links = $this->links($scope, $variables, $resources); + // A reference to a hidden variable is not a reference to a deleted one: leave its link out too. + $links = collect($this->links($scope, $variables, $resources)) + ->reject(fn (array $link) => in_array($link['key'], $hidden, true)) + ->values() + ->all(); $missing = collect($links) ->where('missing', true) @@ -240,6 +251,81 @@ class SharedVariableCanvas return 'variable:'.$variable->id; } + /** + * The shared variables overview of a team: counts per scope, recently changed variables, + * and how many resources use each variable (from the {{scope.KEY}} references in the + * saved resource variables). Keys and counts only; values are never read out. + * + * @return array{scopes: array, recent: Collection>, used: Collection>, unused: Collection>, missing: int} + */ + public function overview(Team $team): array + { + $variables = SharedEnvironmentVariable::query() + ->where('team_id', $team->id) + ->with(['project', 'environment.project', 'server']) + ->get() + // Built-in server variables; the server variables page hides them too. + ->reject(fn (SharedEnvironmentVariable $variable) => $variable->type === 'server' && in_array($variable->key, self::BUILT_IN_SERVER_KEYS, true)); + + $scopes = collect(self::SCOPES)->mapWithKeys(function (string $scope) use ($variables) { + $ofScope = $variables->where('type', $scope); + $ownerColumn = $scope === 'team' ? 'team_id' : $scope.'_id'; + + return [$scope => ['variables' => $ofScope->count(), 'owners' => $ofScope->pluck($ownerColumn)->unique()->count()]]; + })->all(); + + // Resources per variable: "scope|owner id|KEY" => resource uuids. + $idByReference = $variables->mapWithKeys(fn (SharedEnvironmentVariable $variable) => [ + $variable->type.'|'.($variable->type === 'team' ? $variable->team_id : $variable->{$variable->type.'_id'}).'|'.$variable->key => $variable->id, + ]); + $resources = $this->resources('team', $team); + $resourceByMorph = $resources->mapWithKeys(fn (Model $resource) => [$resource->getMorphClass().'#'.$resource->getKey() => $resource]); + $usedBy = []; + $missing = []; + foreach ($this->resourceVariables($resources) as $environmentVariable) { + $resource = $resourceByMorph->get($environmentVariable->resourceable_type.'#'.$environmentVariable->resourceable_id); + $owners = [ + 'team' => $team->id, + 'project' => $resource->environment->project_id, + 'environment' => $resource->environment_id, + 'server' => $this->mainServer($resource)?->id, + ]; + foreach ($owners as $scope => $ownerId) { + foreach ($this->referencedKeys($scope, $environmentVariable->value) as $key) { + $reference = $scope.'|'.$ownerId.'|'.$key; + if ($idByReference->has($reference)) { + $usedBy[$idByReference->get($reference)][$resource->uuid] = true; + } else { + $missing[$reference] = true; + } + } + } + } + + $rows = $variables->map(fn (SharedEnvironmentVariable $variable) => [ + 'key' => $variable->key, + 'scope' => $variable->type, + 'owner' => match ($variable->type) { + 'team' => 'Team', + 'project' => $variable->project?->name, + 'environment' => $variable->environment ? $variable->environment->project?->name.' / '.$variable->environment->name : null, + 'server' => $variable->server?->name, + default => null, + }, + 'link' => $this->connectLink($variable), + 'resources' => count($usedBy[$variable->id] ?? []), + 'updated_at' => $variable->updated_at, + ]); + + return [ + 'scopes' => $scopes, + 'recent' => $rows->sortByDesc('updated_at')->take(6)->values(), + 'used' => $rows->where('resources', '>', 0)->sortByDesc('resources')->take(6)->values(), + 'unused' => $rows->where('resources', 0)->sortBy('key')->values(), + 'missing' => count($missing), + ]; + } + /** * The team of the scope owner. */ diff --git a/app/Services/TrafficAnalyticsAggregator.php b/app/Services/TrafficAnalyticsAggregator.php index e58a78bc3b..66ef76432e 100644 --- a/app/Services/TrafficAnalyticsAggregator.php +++ b/app/Services/TrafficAnalyticsAggregator.php @@ -161,6 +161,12 @@ class TrafficAnalyticsAggregator $this->breakdowns[$dimension][$value] ??= ['value' => $value, 'requests' => 0, 'bytesOut' => 0]; $this->breakdowns[$dimension][$value]['requests'] += (int) ($data['requests'] ?? 0); $this->breakdowns[$dimension][$value]['bytesOut'] += (int) ($data['bytesOut'] ?? 0); + + // City rows carry coordinates; the first server that has them wins. + if (isset($data['lat'], $data['lon'])) { + $this->breakdowns[$dimension][$value]['lat'] ??= (float) $data['lat']; + $this->breakdowns[$dimension][$value]['lon'] ??= (float) $data['lon']; + } } } @@ -205,7 +211,7 @@ class TrafficAnalyticsAggregator } /** - * @return array> + * @return array> */ public function breakdowns(int $limit = 50): array { diff --git a/app/Traits/ExecuteRemoteCommand.php b/app/Traits/ExecuteRemoteCommand.php index 86ed88e9b8..21912fb82f 100644 --- a/app/Traits/ExecuteRemoteCommand.php +++ b/app/Traits/ExecuteRemoteCommand.php @@ -248,7 +248,22 @@ trait ExecuteRemoteCommand $error = $this->redact_sensitive_info($error); - return "Command execution failed (exit code {$exitCode}): {$commandText}\nError: {$error}"; + return "Command execution failed (exit code {$exitCode}): {$commandText}\nError: {$error}".$this->dockerNetworkGatewayHint($error); + } + + /** + * Docker Engine 27.0 to 28.0.0 reports an IPv6 network gateway with its prefix length (fd00::1/64) until the + * daemon restarts, and newer Docker CLI and Compose versions fail to parse it. + * + * @see https://github.com/moby/moby/pull/49520 + */ + private function dockerNetworkGatewayHint(string $error): string + { + if (! preg_match('~ParseAddr\("[0-9A-Fa-f:.]+/\d+"\)~', $error)) { + return ''; + } + + return "\nDocker Engine on this server reports a Docker network gateway with a prefix length, which is a Docker Engine bug fixed in 28.0.1. Run 'systemctl restart docker' on the server (this restarts all containers) or upgrade Docker Engine, then deploy again."; } /** diff --git a/app/Traits/ExecutesDatabaseStartCommands.php b/app/Traits/ExecutesDatabaseStartCommands.php index d61cfb3146..dadc4dd68c 100644 --- a/app/Traits/ExecutesDatabaseStartCommands.php +++ b/app/Traits/ExecutesDatabaseStartCommands.php @@ -27,4 +27,23 @@ trait ExecutesDatabaseStartCommands { return $environmentVariables->contains(fn ($variable) => str($variable)->before('=')->value() === $key); } + + /** + * Adds the Coolify variables that applications and services also get. A key that the user + * defines wins. + * + * @param Collection $environmentVariables + */ + private function addCoolifyEnvironmentVariables(Collection $environmentVariables): void + { + $definedKeys = $environmentVariables->map(fn ($variable) => str($variable)->before('=')->value()); + $coolifyVariables = collect([ + 'COOLIFY_RESOURCE_UUID' => $this->database->uuid, + 'COOLIFY_CONTAINER_NAME' => $this->database->uuid, + ])->except($definedKeys->all())->merge(coolifyDefaultEnvironmentVariables($this->database, $definedKeys)); + + foreach ($coolifyVariables as $key => $value) { + $environmentVariables->push($key.'='.escapeDollarSign($value)); + } + } } diff --git a/app/Traits/HasS3Replicas.php b/app/Traits/HasS3Replicas.php index 81e935e174..0d63823cd6 100644 --- a/app/Traits/HasS3Replicas.php +++ b/app/Traits/HasS3Replicas.php @@ -2,6 +2,7 @@ namespace App\Traits; +use App\Services\FileDownloadStorageService; use Illuminate\Database\Eloquent\Builder; use Illuminate\Database\Eloquent\Relations\HasMany; @@ -44,6 +45,47 @@ trait HasS3Replicas return $this->s3Replicas()->where('s3_uploaded', true)->where('s3_storage_deleted', false)->exists(); } + /** + * Whether a local file or a live S3 copy is left to download. + */ + public function isDownloadable(): bool + { + return ! $this->local_storage_deleted + || $this->s3Replicas->contains(fn ($replica): bool => $replica->s3_uploaded === true && $replica->s3_storage_deleted === false); + } + + /** + * A short-lived presigned download URL for the first live copy in a usable S3 storage of the given team, the + * primary storage first. Null when no such copy exists. + */ + public function s3DownloadUrl(int $teamId, ?int $primaryStorageId): ?string + { + if (blank($this->filename)) { + return null; + } + + $replica = $this->s3Replicas() + ->with('s3') + ->where('s3_uploaded', true) + ->where('s3_storage_deleted', false) + ->whereHas('s3', fn (Builder $query) => $query->where('team_id', $teamId)->where('is_usable', true)) + ->orderBy('id') + ->get() + ->sortByDesc(fn ($replica): bool => $replica->s3_storage_id === $primaryStorageId) + ->first(); + + if (! $replica) { + return null; + } + + // The upload copies the file to "/", so the object key is the local file path. + return $replica->s3->filesystem()->temporaryUrl( + $this->filename, + now()->addMinutes(FileDownloadStorageService::LINK_TTL_MINUTES), + ['ResponseContentDisposition' => FileDownloadStorageService::contentDisposition(basename($this->filename))], + ); + } + /** * Writes the replica states to the summary columns: uploaded when every copy uploaded, deleted when every uploaded * copy is deleted. diff --git a/app/Traits/PreparesBackupDownloads.php b/app/Traits/PreparesBackupDownloads.php new file mode 100644 index 0000000000..118813fa5f --- /dev/null +++ b/app/Traits/PreparesBackupDownloads.php @@ -0,0 +1,82 @@ + */ + public array $pendingDownloads = []; + + /** + * Authorize the user and load the backup from a record scoped to the + * component before you call this. Pass a null server or file name when + * the local file is gone; the download route then answers the request. + */ + protected function prepareBackupDownload(?Server $server, ?string $filename, string $streamUrl): void + { + $storage = FileDownloadStorageService::configuredStorage(); + $size = $storage && $server && filled($filename) ? FileDownloadStorageService::hostFileSize($server, $filename) : null; + if ($size === null || $size > FileDownloadStorageService::MAX_SIZE) { + $this->dispatch('open-backup-download', url: $streamUrl); + + return; + } + + if (count($this->pendingDownloads) >= self::MAX_PENDING_S3_DOWNLOADS + || ! RateLimiter::attempt('file-browser-s3-download:'.auth()->id(), 10, fn () => true, 3600)) { + $this->dispatch('error', 'Too many downloads are in progress. Wait for them to finish, then try again.'); + + return; + } + + $downloadId = (string) Str::uuid(); + $this->pendingDownloads[] = $downloadId; + PrepareFileDownloadJob::dispatch($downloadId, auth()->id(), $server->id, null, $filename, false, basename($filename), $storage->id); + $this->dispatch('info', 'Preparing '.basename($filename).'. The download starts when it is ready.'); + } + + /** + * Polled while S3 downloads are prepared. Opens each finished link once. + */ + public function checkDownloads(): void + { + foreach ($this->pendingDownloads as $index => $downloadId) { + $result = Cache::get(PrepareFileDownloadJob::cacheKey($downloadId)); + if (! is_array($result) || ($result['user_id'] ?? null) !== auth()->id()) { + continue; + } + + unset($this->pendingDownloads[$index]); + Cache::forget(PrepareFileDownloadJob::cacheKey($downloadId)); + if ($result['status'] === 'ready') { + $this->dispatch('open-backup-download', url: $result['url']); + } else { + $this->dispatch('error', $result['message']); + } + } + $this->pendingDownloads = array_values($this->pendingDownloads); + } + + protected function s3BackupDownloadsEnabled(): bool + { + return FileDownloadStorageService::configuredStorage() !== null; + } +} diff --git a/bootstrap/helpers/constants.php b/bootstrap/helpers/constants.php index 4ab7d74698..543901932f 100644 --- a/bootstrap/helpers/constants.php +++ b/bootstrap/helpers/constants.php @@ -81,6 +81,8 @@ const SPECIFIC_SERVICES = [ 'minio/minio', 'ghcr.io/coollabsio/minio', 'coollabsio/minio', + 'pgsty/silo', + 'docker.io/pgsty/silo', 'svhd/logto', 'dxflrs/garage', ]; diff --git a/bootstrap/helpers/docker.php b/bootstrap/helpers/docker.php index faf3e43acb..787fce986a 100644 --- a/bootstrap/helpers/docker.php +++ b/bootstrap/helpers/docker.php @@ -7,6 +7,7 @@ use App\Models\Server; use App\Models\Service; use App\Models\ServiceApplication; use App\Models\ServiceDatabase; +use App\Support\DomainPortOverrides; use App\Support\ValidationPatterns; use Illuminate\Database\Eloquent\Model; use Illuminate\Support\Collection; @@ -675,7 +676,7 @@ function generateServiceSpecificFqdns(ServiceApplication|Application $resource) $variables = collect($environment_variables); $payload = collect([]); switch ($type) { - case $type?->contains('minio'): + case $type?->contains(['minio', 'pgsty/silo']): $MINIO_BROWSER_REDIRECT_URL = $variables->where('key', 'MINIO_BROWSER_REDIRECT_URL')->first(); $MINIO_SERVER_URL = $variables->where('key', 'MINIO_SERVER_URL')->first(); @@ -765,9 +766,12 @@ function isNoindexDomain(string $domain, ?Collection $noindex_domains): bool return false; } + // Flags are stored without a port, so compare portless: https://app.example.com:3000 matches https://app.example.com. + $normalize = fn (string $url) => DomainPortOverrides::withoutPort(ValidationPatterns::normalizeApplicationDomainUrl($url)); + return $noindex_domains - ->map(fn (string $noindex_domain) => ValidationPatterns::normalizeApplicationDomainUrl($noindex_domain)) - ->contains(ValidationPatterns::normalizeApplicationDomainUrl($domain)); + ->map($normalize) + ->contains($normalize($domain)); } /** diff --git a/bootstrap/helpers/domains.php b/bootstrap/helpers/domains.php index 8be6d9a77a..704c90988f 100644 --- a/bootstrap/helpers/domains.php +++ b/bootstrap/helpers/domains.php @@ -3,6 +3,7 @@ use App\Models\Application; use App\Models\Service; use App\Models\ServiceApplication; +use App\Models\ServiceDatabase; use Illuminate\Support\Collection; /** @@ -591,6 +592,47 @@ function findServiceApplicationForEnvName(Service $resource, string $envServiceN return $resource->applications()->where('name', $resolved)->first(); } +/** + * Check if a service copies a SERVICE_FQDN_ / SERVICE_URL_ declaration of the application named in it. + * The copy (same name, port and value; URL or FQDN) must not generate a domain or overwrite the variable (#11924). + * Declarations with another port or path share the host on purpose (for example Appwrite). + * + * @param array|Collection $services Parsed compose services + */ +function isCopiedServiceUrlDeclaration( + Service $resource, + array|Collection $services, + ServiceApplication|ServiceDatabase $current, + string $key, +): bool { + $owner = findServiceApplicationForEnvName($resource, parseServiceEnvironmentVariable($key)['service_name']); + if (! $owner || $owner->name === $current->name) { + return false; + } + + // Same declaration sources as serviceParser: environment and build arguments. + $declarationsOf = fn (string $serviceName) => convertToKeyValueCollection( + collect(data_get($services, [$serviceName, 'environment'], [])) + ->merge(data_get($services, [$serviceName, 'build', 'args'], [])) + ); + $ownerDeclarations = $declarationsOf($owner->name); + $currentDeclarations = $declarationsOf($current->name); + + if (! $currentDeclarations->has($key)) { + return false; + } + + // SERVICE_URL_X_3000 and SERVICE_FQDN_X_3000 write the same variable pair, so either one counts. + $twinKey = str_starts_with($key, 'SERVICE_URL_') + ? 'SERVICE_FQDN_'.substr($key, strlen('SERVICE_URL_')) + : 'SERVICE_URL_'.substr($key, strlen('SERVICE_FQDN_')); + $value = (string) $currentDeclarations->get($key); + + return collect([$key, $twinKey])->contains( + fn (string $ownerKey) => $ownerDeclarations->has($ownerKey) && (string) $ownerDeclarations->get($ownerKey) === $value + ); +} + /** * Put/update a service domain entry under the original compose service name, * removing legacy twin keys that resolve unambiguously to the same name. diff --git a/bootstrap/helpers/github.php b/bootstrap/helpers/github.php index d66ffd88b8..edcbca88f6 100644 --- a/bootstrap/helpers/github.php +++ b/bootstrap/helpers/github.php @@ -4,6 +4,8 @@ use App\Models\GithubApp; use App\Models\PrivateKey; use Carbon\Carbon; use Carbon\CarbonImmutable; +use Illuminate\Http\Client\ConnectionException; +use Illuminate\Http\Client\RequestException; use Illuminate\Support\Facades\Cache; use Illuminate\Support\Facades\Http; use Illuminate\Support\Str; @@ -195,13 +197,20 @@ function generateGithubToken(GithubApp $source, string $type) return match ($type) { 'jwt' => $jwt, 'installation' => (function () use ($source, $jwt) { + // GitHub sometimes answers with a 5xx page or drops the connection; a new token request is harmless. $response = Http::GitSource($source->api_url)->withHeaders([ 'Authorization' => "Bearer $jwt", 'Accept' => 'application/vnd.github.machine-man-preview+json', - ])->post("{$source->api_url}/app/installations/{$source->installation_id}/access_tokens"); + ])->retry( + 3, + fn (int $attempt) => $attempt * 1000, + fn (Throwable $exception) => $exception instanceof ConnectionException + || ($exception instanceof RequestException && $exception->response->serverError()), + throw: false, + )->post("{$source->api_url}/app/installations/{$source->installation_id}/access_tokens"); if (! $response->successful()) { - $error = data_get($response->json(), 'message', 'no error message found'); + $error = data_get($response->json(), 'message', "HTTP {$response->status()}"); if ($error === 'Not Found') { $error = 'Repository not found. Is it moved or deleted?'; } @@ -216,7 +225,7 @@ function generateGithubToken(GithubApp $source, string $type) function generateGithubInstallationToken(GithubApp $source) { - return generateGithubToken($source, 'installation'); + return $source->installationToken(); } function generateGithubJwt(GithubApp $source) @@ -397,28 +406,34 @@ function loadRepositoryByPage(GithubApp $source, string $token, int $page) 'repositories' => $json['repositories'], ]; } -function getGithubCommitRangeFiles(?GithubApp $source, string $owner, string $repo, string $beforeSha, string $afterSha): array +/** + * Returns null when the full list of changed files is unknown (API error or truncated list). + */ +function getGithubCommitRangeFiles(?GithubApp $source, string $owner, string $repo, string $beforeSha, string $afterSha): ?array { try { if (! $source) { // Manual webhooks don't have GitHub App authentication - // Return empty array so watch paths are ignored (current behavior) return []; } $endpoint = "/repos/{$owner}/{$repo}/compare/{$beforeSha}...{$afterSha}"; - $response = githubApi($source, $endpoint, 'get', null, false); - - if (! $response) { - return []; - } - + $response = githubApi($source, $endpoint); $files = collect(data_get($response, 'data.files', [])); - return $files->pluck('filename')->filter()->values()->toArray(); - } catch (Exception $e) { + // The compare API returns at most 300 files and cannot paginate them. + if ($files->count() >= 300) { + return null; + } - return []; + // A renamed file also changes its previous path. + return $files->flatMap(fn (mixed $file): array => [data_get($file, 'filename'), data_get($file, 'previous_filename')]) + ->filter(fn (mixed $filename): bool => is_string($filename) && $filename !== '') + ->unique() + ->values() + ->toArray(); + } catch (Exception $e) { + return null; } } @@ -444,27 +459,34 @@ function getGithubCommitMessage(?GithubApp $source, string $owner, string $repo, } } -function getGithubPullRequestFiles(?GithubApp $source, string $owner, string $repo, int $pullRequestId): array +/** + * Returns null when the full list of changed files is unknown (API error or truncated list). + */ +function getGithubPullRequestFiles(?GithubApp $source, string $owner, string $repo, int $pullRequestId): ?array { try { if (! $source) { // Manual webhooks don't have GitHub App authentication - // Return empty array so watch paths are ignored (current behavior) return []; } - $endpoint = "/repos/{$owner}/{$repo}/pulls/{$pullRequestId}/files"; - $response = githubApi($source, $endpoint, 'get', null, false); + // GitHub returns at most 100 files per page and 3000 files per pull request. + $perPage = 100; + $maxPages = 30; + $files = collect(); - if (! $response) { - return []; + for ($page = 1; $page <= $maxPages; $page++) { + $endpoint = "/repos/{$owner}/{$repo}/pulls/{$pullRequestId}/files?per_page={$perPage}&page={$page}"; + $pageFiles = collect(data_get(githubApi($source, $endpoint), 'data', [])); + $files = $files->merge($pageFiles->pluck('filename')); + + if ($pageFiles->count() < $perPage) { + return $files->filter()->values()->toArray(); + } } - $files = collect(data_get($response, 'data', [])); - - return $files->pluck('filename')->filter()->values()->toArray(); + return null; } catch (Exception $e) { - - return []; + return null; } } diff --git a/bootstrap/helpers/parsers.php b/bootstrap/helpers/parsers.php index 104497ed92..1a8f09c529 100644 --- a/bootstrap/helpers/parsers.php +++ b/bootstrap/helpers/parsers.php @@ -2031,7 +2031,9 @@ function applicationParser(Application $resource, int $pull_request_id = 0, ?int $coolifyEnvironments->put('COOLIFY_URL', $fqdns->map(fn ($fqdn) => getFqdnWithoutPort($fqdn))->implode(',')); $coolifyEnvironments->put('COOLIFY_FQDN', $fqdns->map(fn ($fqdn) => getHostWithoutPort($fqdn))->implode(',')); } - add_coolify_default_environment_variables($resource, $coolifyEnvironments, $resource->environment_variables); + $userDefinedKeys = $resource->environment_variables->pluck('key')->merge($environment->keys()); + coolifyDefaultEnvironmentVariables($resource, $userDefinedKeys) + ->each(fn ($value, $key) => $coolifyEnvironments->put($key, escapeDollarSign($value))); if ($environment->count() > 0) { $environment = $environment->filter(function ($value, $key) { return ! str($key)->startsWith('SERVICE_FQDN_'); @@ -2547,6 +2549,10 @@ function serviceParser(Service $resource): Collection $port = $parsed['port']; $fqdnFor = $parsed['service_name']; + if (isCopiedServiceUrlDeclaration($resource, $services, $savedService, $key->value())) { + continue; + } + // Only ServiceApplication has fqdn column, ServiceDatabase does not $isServiceApplication = $savedService instanceof ServiceApplication; @@ -2667,8 +2673,8 @@ function serviceParser(Service $resource): Collection // Also check if a port-suffixed version exists (e.g., SERVICE_FQDN_UMAMI_3000) $portSuffixedExists = $resource->environment_variables() ->where('key', 'LIKE', $key->value().'_%') - ->whereRaw('key ~ ?', ['^'.$key->value().'_[0-9]+$']) - ->exists(); + ->pluck('key') + ->contains(fn ($envKey) => preg_match('/^'.preg_quote($key->value(), '/').'_[0-9]+$/', $envKey) === 1); $serviceExists = findServiceApplicationForEnvName($resource, (string) $fqdnFor); // Check if FQDN already has a port set (contains ':' after the domain) $fqdnHasPort = $serviceExists && str($serviceExists->fqdn)->contains(':') && str($serviceExists->fqdn)->afterLast(':')->isMatch('/^\d+$/'); @@ -2712,8 +2718,8 @@ function serviceParser(Service $resource): Collection // Also check if a port-suffixed version exists (e.g., SERVICE_URL_DASHBOARD_6791) $portSuffixedExists = $resource->environment_variables() ->where('key', 'LIKE', $key->value().'_%') - ->whereRaw('key ~ ?', ['^'.$key->value().'_[0-9]+$']) - ->exists(); + ->pluck('key') + ->contains(fn ($envKey) => preg_match('/^'.preg_quote($key->value(), '/').'_[0-9]+$/', $envKey) === 1); $serviceExists = findServiceApplicationForEnvName($resource, (string) $urlFor); // Check if FQDN already has a port set (contains ':' after the domain) $fqdnHasPort = $serviceExists && str($serviceExists->fqdn)->contains(':') && str($serviceExists->fqdn)->afterLast(':')->isMatch('/^\d+$/'); @@ -3333,7 +3339,9 @@ function serviceParser(Service $resource): Collection $coolifyEnvironments->put('COOLIFY_FQDN', $fqdns->map(fn ($fqdn) => getHostWithoutPort($fqdn))->implode(',')); $coolifyEnvironments->put('COOLIFY_URL', $fqdns->map(fn ($fqdn) => getFqdnWithoutPort($fqdn))->implode(',')); } - add_coolify_default_environment_variables($resource, $coolifyEnvironments, $resource->environment_variables); + $userDefinedKeys = $resource->environment_variables->pluck('key')->merge($environment->keys()); + coolifyDefaultEnvironmentVariables($resource, $userDefinedKeys) + ->each(fn ($value, $key) => $coolifyEnvironments->put($key, escapeDollarSign($value))); if ($environment->count() > 0) { $environment = $environment->filter(function ($value, $key) { return ! str($key)->startsWith('SERVICE_FQDN_'); diff --git a/bootstrap/helpers/services.php b/bootstrap/helpers/services.php index 66154ce7e8..667b9b18e3 100644 --- a/bootstrap/helpers/services.php +++ b/bootstrap/helpers/services.php @@ -409,8 +409,16 @@ function updateCompose(ServiceApplication|ServiceDatabase $resource) // Those belong to other services and will be updated when THOSE services are updated } - // Remove duplicates - $templateVariableNames = array_unique($templateVariableNames); + // Remove duplicates and copies of another application's declaration + $templateVariableNames = array_filter( + array_unique($templateVariableNames), + fn ($variableName) => ! isCopiedServiceUrlDeclaration( + $resource->service, + collect(data_get($dockerCompose, 'services', [])), + $resource, + $variableName, + ), + ); // Extract unique service names to process (preserving the original case from template) // This allows us to create both URL and FQDN pairs regardless of which one is in the template diff --git a/bootstrap/helpers/shared.php b/bootstrap/helpers/shared.php index 482c77f40a..19049f6426 100644 --- a/bootstrap/helpers/shared.php +++ b/bootstrap/helpers/shared.php @@ -935,6 +935,15 @@ function s3_image_url(?int $storageId, ?string $path, int $version): ?string return null; } + // S3-compatible endpoints (e.g. Cloudflare R2) require authenticated requests for + // every object, including plain GETs, so an unauthenticated direct link only works + // when a public CDN has been configured in front of the bucket. Without one, callers + // must fall back to the app's own authenticated proxy route. + $cdnUrl = instanceSettings()->image_cdn_url; + if (blank($cdnUrl)) { + return null; + } + $storage = S3Storage::query() ->whereKey($storageId) ->whereTeamId(0) @@ -945,9 +954,7 @@ function s3_image_url(?int $storageId, ?string $path, int $version): ?string return null; } - $baseUrl = instanceSettings()->image_cdn_url ?: $storage->awsUrl(); - - return rtrim($baseUrl, '/').'/'.ltrim($path, '/').'?v='.$version; + return rtrim($cdnUrl, '/').'/'.ltrim($path, '/').'?v='.$version; } /** @@ -1517,7 +1524,25 @@ function sslip(Server $server) } $ip = $server->ip === 'host.docker.internal' ? base_ip() : $server->ip; - return 'http://'.sslipHostLabel($ip).'.sslip.io'; + return 'http://'.sslipHostLabel(resolveHostToIp($ip)).'.sslip.io'; +} + +/** + * Resolve a hostname to its IPv4 (or else IPv6) address. sslip.io only answers for names that + * contain an IP address. Returns the input unchanged if it is already an IP or does not resolve. + */ +function resolveHostToIp(string $host): string +{ + if (filter_var(normalizeIpAddress($host), FILTER_VALIDATE_IP) !== false) { + return $host; + } + $ipv4 = gethostbyname($host); + if (filter_var($ipv4, FILTER_VALIDATE_IP, FILTER_FLAG_IPV4) !== false) { + return $ipv4; + } + $ipv6 = data_get(@dns_get_record($host, DNS_AAAA), '0.ipv6'); + + return is_string($ipv6) && $ipv6 !== '' ? $ipv6 : $host; } function service_templates_cache_key(): string @@ -2683,44 +2708,53 @@ function deduplicateAllowlist(array $entries): array return $result; } -function get_public_ips() +/** + * Detect the instance public IPs with ifconfig.io. Only empty settings are filled, so an address saved in Settings is kept. + */ +function get_public_ips(): void { try { - [$first, $second] = Process::concurrently(function (Pool $pool) { - $pool->path(__DIR__)->command('curl -4s https://ifconfig.io'); - $pool->path(__DIR__)->command('curl -6s https://ifconfig.io'); - }); - $ipv4 = $first->output(); - if ($ipv4) { - $ipv4 = trim($ipv4); - $validate_ipv4 = filter_var($ipv4, FILTER_VALIDATE_IP, FILTER_FLAG_IPV4); - if ($validate_ipv4 == false) { - echo "Invalid ipv4: $ipv4\n"; - - return; - } - InstanceSettings::get()->update(['public_ipv4' => $ipv4]); + $settings = InstanceSettings::get(); + $columns = array_values(array_filter(['public_ipv4', 'public_ipv6'], fn (string $column) => blank($settings->$column))); + if ($columns === []) { + return; } - } catch (Exception $e) { - echo "Error: {$e->getMessage()}\n"; - } - try { - $ipv6 = $second->output(); - if ($ipv6) { - $ipv6 = trim($ipv6); - $validate_ipv6 = filter_var($ipv6, FILTER_VALIDATE_IP, FILTER_FLAG_IPV6); - if ($validate_ipv6 == false) { - echo "Invalid ipv6: $ipv6\n"; - return; - } - InstanceSettings::get()->update(['public_ipv6' => $ipv6]); + $detected = detect_public_ips($columns); + if ($detected !== []) { + $settings->update($detected); } } catch (Throwable $e) { echo "Error: {$e->getMessage()}\n"; } } +/** + * Look up the public IPs of this host with ifconfig.io, without saving them. + * + * @param list<'public_ipv4'|'public_ipv6'> $columns + * @return array<'public_ipv4'|'public_ipv6', string> valid addresses only + */ +function detect_public_ips(array $columns = ['public_ipv4', 'public_ipv6']): array +{ + $versions = array_intersect_key(['public_ipv4' => 4, 'public_ipv6' => 6], array_flip($columns)); + $results = Process::concurrently(function (Pool $pool) use ($versions) { + foreach ($versions as $column => $version) { + $pool->as($column)->path(__DIR__)->command("curl -{$version}s https://ifconfig.io"); + } + }); + + $detected = []; + foreach ($versions as $column => $version) { + $ip = trim($results[$column]->output()); + if (filter_var($ip, FILTER_VALIDATE_IP, $version === 4 ? FILTER_FLAG_IPV4 : FILTER_FLAG_IPV6) !== false) { + $detected[$column] = $ip; + } + } + + return $detected; +} + function isAnyDeploymentInprogress(bool $showAll = false) { $runningJobs = ApplicationDeploymentQueue::where('horizon_job_worker', gethostname())->where('status', ApplicationDeploymentStatus::IN_PROGRESS->value)->get(); @@ -3624,19 +3658,8 @@ function parseDockerComposeFile(Service|Application $resource, bool $isNew = fal $parsedServiceVariables->put('COOLIFY_RESOURCE_UUID', "{$resource->uuid}"); $parsedServiceVariables->put('COOLIFY_CONTAINER_NAME', "$serviceName-{$resource->uuid}"); - // TODO: move this in a shared function - if (! $parsedServiceVariables->has('COOLIFY_APP_NAME')) { - $parsedServiceVariables->put('COOLIFY_APP_NAME', "\"{$resource->name}\""); - } - if (! $parsedServiceVariables->has('COOLIFY_SERVER_IP')) { - $parsedServiceVariables->put('COOLIFY_SERVER_IP', "\"{$resource->destination->server->ip}\""); - } - if (! $parsedServiceVariables->has('COOLIFY_ENVIRONMENT_NAME')) { - $parsedServiceVariables->put('COOLIFY_ENVIRONMENT_NAME', "\"{$resource->environment->name}\""); - } - if (! $parsedServiceVariables->has('COOLIFY_PROJECT_NAME')) { - $parsedServiceVariables->put('COOLIFY_PROJECT_NAME', "\"{$resource->project()->name}\""); - } + coolifyDefaultEnvironmentVariables($resource, $parsedServiceVariables->keys()) + ->each(fn ($value, $key) => $parsedServiceVariables->put($key, escapeDollarSign($value))); $parsedServiceVariables = $parsedServiceVariables->map(function ($value, $key) use ($envs_from_coolify) { if (! str($value)->startsWith('$')) { @@ -4509,56 +4532,32 @@ function isAssociativeArray($array) } /** - * This method adds the default environment variables to the resource. - * - COOLIFY_APP_NAME - * - COOLIFY_PROJECT_NAME - * - COOLIFY_SERVER_IP - * - COOLIFY_ENVIRONMENT_NAME + * The variables that tell a container where it runs. Values are raw: each caller escapes them + * for its output (compose file, .env file, or build arguments). Keys in $definedKeys are + * skipped, so a variable that the user defines wins. Pass $server when the container runs on + * another server than the main destination (an application on an additional server). * - * Theses variables are added in place to the $where_to_add array. + * @param iterable $definedKeys + * @return Collection */ -function add_coolify_default_environment_variables(StandaloneRedis|StandalonePostgresql|StandaloneMongodb|StandaloneMysql|StandaloneMariadb|StandaloneKeydb|StandaloneDragonfly|StandaloneClickhouse|StandaloneSqlite|Application|Service $resource, Collection &$where_to_add, ?Collection $where_to_check = null) +function coolifyDefaultEnvironmentVariables(StandaloneRedis|StandalonePostgresql|StandaloneMongodb|StandaloneMysql|StandaloneMariadb|StandaloneKeydb|StandaloneDragonfly|StandaloneClickhouse|StandaloneSqlite|Application|Service $resource, iterable $definedKeys = [], ?Server $server = null): Collection { - // Currently disabled - return; - if ($resource instanceof Service) { - $ip = $resource->server->ip; - } else { - $ip = $resource->destination->server->ip; - } - if (isAssociativeArray($where_to_add)) { - $isAssociativeArray = true; - } else { - $isAssociativeArray = false; - } - if ($where_to_check != null && $where_to_check->where('key', 'COOLIFY_APP_NAME')->isEmpty()) { - if ($isAssociativeArray) { - $where_to_add->put('COOLIFY_APP_NAME', "\"{$resource->name}\""); - } else { - $where_to_add->push("COOLIFY_APP_NAME=\"{$resource->name}\""); - } - } - if ($where_to_check != null && $where_to_check->where('key', 'COOLIFY_SERVER_IP')->isEmpty()) { - if ($isAssociativeArray) { - $where_to_add->put('COOLIFY_SERVER_IP', "\"{$ip}\""); - } else { - $where_to_add->push("COOLIFY_SERVER_IP=\"{$ip}\""); - } - } - if ($where_to_check != null && $where_to_check->where('key', 'COOLIFY_ENVIRONMENT_NAME')->isEmpty()) { - if ($isAssociativeArray) { - $where_to_add->put('COOLIFY_ENVIRONMENT_NAME', "\"{$resource->environment->name}\""); - } else { - $where_to_add->push("COOLIFY_ENVIRONMENT_NAME=\"{$resource->environment->name}\""); - } - } - if ($where_to_check != null && $where_to_check->where('key', 'COOLIFY_PROJECT_NAME')->isEmpty()) { - if ($isAssociativeArray) { - $where_to_add->put('COOLIFY_PROJECT_NAME', "\"{$resource->project()->name}\""); - } else { - $where_to_add->push("COOLIFY_PROJECT_NAME=\"{$resource->project()->name}\""); - } + // A model created in this request has no column default loaded yet (null), which means on. + $isEnabled = $resource instanceof Application + ? $resource->settings?->include_coolify_default_variables + : $resource->include_coolify_default_variables; + if ($isEnabled === false) { + return collect(); } + + $server ??= $resource instanceof Service ? $resource->server : $resource->destination?->server; + + return collect([ + 'COOLIFY_APP_NAME' => $resource->name, + 'COOLIFY_PROJECT_NAME' => $resource->project()?->name, + 'COOLIFY_ENVIRONMENT_NAME' => $resource->environment?->name, + 'COOLIFY_SERVER_IP' => $server?->ip, + ])->whereNotNull()->map(fn ($value) => (string) $value)->except(collect($definedKeys)->all()); } function convertToKeyValueCollection($environment) @@ -4620,6 +4619,20 @@ function wireNavigate(): string } } +/** + * Registers a resource menu that the main sidebar shows in place of its main menu on wide screens + * (see components/navbar.blade.php). The menu teleports itself into #main-nav-context-{depth}. + * Back on a menu shows the menu one depth lower, and Back at depth 0 shows the main menu. + */ +function registerMainNavContext(int $depth, string $label, string $backTitle): void +{ + $levels = request()->attributes->get('mainNavContext', []); + $levels[$depth] = ['label' => $label, 'backTitle' => $backTitle]; + ksort($levels); + + request()->attributes->set('mainNavContext', $levels); +} + /** * Flattens a grouped settings sidebar into the items that the command palette * shows for the current page: every page, its child pages, and its in-page sections. @@ -4869,8 +4882,15 @@ function parseScpStyleGitUrl(?string $gitRepository): ?array ]; } -function scpStyleGitUrlToHttps(?string $gitRepository): ?string +function sshGitUrlToHttps(?string $gitRepository): ?string { + if (is_string($gitRepository) && str_starts_with($gitRepository, 'ssh://')) { + $host = parse_url($gitRepository, PHP_URL_HOST); + $path = ltrim((string) parse_url($gitRepository, PHP_URL_PATH), '/'); + + return filled($host) && $path !== '' ? "https://{$host}/{$path}" : null; + } + $parts = parseScpStyleGitUrl($gitRepository); if ($parts === null) { @@ -4886,8 +4906,8 @@ function gitRepositorySlug(?string $gitRepository): string return ''; } - if (($scp = parseScpStyleGitUrl($gitRepository)) !== null) { - $gitRepository = $scp['path']; + if (($httpsRepository = sshGitUrlToHttps($gitRepository)) !== null) { + $gitRepository = (string) parse_url($httpsRepository, PHP_URL_PATH); } elseif (str($gitRepository)->startsWith('http') || str($gitRepository)->contains('github.com')) { $gitRepository = str($gitRepository)->replace('https://', '')->replace('http://', '')->replace('github.com/', ''); } @@ -5753,6 +5773,115 @@ function deviceLabel(?string $device): string }; } +/** + * Prepare aggregated `city` breakdown rows (`CC|subdivision|City`, from + * TrafficAnalyticsAggregator::breakdowns(), not raw Sentinel rows) for display, + * largest first. `name` and `detail` are the two parts of `label` ("Berlin" and + * "BE, Germany"). `CC||` (sent by older Sentinel builds) is an unknown city of + * that country. `__other__` and malformed values are merged into one "Other" + * row, which stays last. Returns [] when no row has a real city. + * + * @param iterable $rows + * @return array + */ +function trafficCityRows(iterable $rows): array +{ + $out = []; + $hasCity = false; + + foreach ($rows as $row) { + $requests = (int) data_get($row, 'requests', 0); + if ($requests <= 0) { + continue; + } + + $value = (string) data_get($row, 'value', ''); + $parts = explode('|', $value, 3); + $parts[0] = strtoupper($parts[0]); + $isCityKey = count($parts) === 3 && preg_match('/^[A-Z]{2}$/', $parts[0]) === 1; + + $key = '__other__'; + $country = ''; + $label = 'Other'; + $name = 'Other'; + $detail = ''; + $lat = null; + $lon = null; + + if ($isCityKey) { + [$country, $subdivision, $city] = array_map('trim', $parts); + $key = $value; + + if ($city === '') { + $label = 'Unknown, '.$country; + $name = 'Unknown'; + $detail = countryName($country); + } else { + $hasCity = true; + $region = preg_match('/^[A-Z]{2,3}$/', $subdivision) === 1 ? $subdivision.', ' : ''; + $label = $city.', '.$region.$country; + $name = $city; + $detail = $region.countryName($country); + + $rawLat = data_get($row, 'lat'); + $rawLon = data_get($row, 'lon'); + if (is_numeric($rawLat) && is_numeric($rawLon)) { + $lat = (float) $rawLat; + $lon = (float) $rawLon; + } + } + } + + $out[$key] ??= ['key' => $key, 'country' => $country, 'label' => $label, 'name' => $name, 'detail' => $detail, 'requests' => 0, 'bytesOut' => 0, 'lat' => $lat, 'lon' => $lon]; + $out[$key]['requests'] += $requests; + $out[$key]['bytesOut'] += (int) data_get($row, 'bytesOut', 0); + } + + if (! $hasCity) { + return []; + } + + $other = $out['__other__'] ?? null; + unset($out['__other__']); + $out = array_values($out); + usort($out, fn (array $a, array $b): int => $b['requests'] <=> $a['requests']); + + return $other === null ? $out : [...$out, $other]; +} + +/** + * Marker data for the analytics globe, one set for each view of the Locations + * card: a marker per country with a valid ISO-A2 code, and a marker per city + * that has coordinates. + * + * @param iterable $countryRows + * @param iterable $cityRows + * @return array{countries: array, cities: array} + */ +function trafficGeoMarkers(iterable $countryRows, iterable $cityRows): array +{ + $countries = []; + foreach ($countryRows as $row) { + $code = strtoupper((string) data_get($row, 'value', '')); + $requests = (int) data_get($row, 'requests', 0); + if ($requests > 0 && preg_match('/^[A-Z]{2}$/', $code) === 1) { + $countries[] = ['code' => $code, 'requests' => $requests]; + } + } + + $cities = []; + foreach ($cityRows as $row) { + $requests = (int) data_get($row, 'requests', 0); + $lat = data_get($row, 'lat'); + $lon = data_get($row, 'lon'); + if ($requests > 0 && is_numeric($lat) && is_numeric($lon)) { + $cities[] = ['lat' => (float) $lat, 'lon' => (float) $lon, 'requests' => $requests]; + } + } + + return ['countries' => $countries, 'cities' => $cities]; +} + /** * Resolve an ISO 3166-1 alpha-2 country code to its English country name. * diff --git a/config/constants.php b/config/constants.php index 0da39cc3d2..728ba9c8ee 100644 --- a/config/constants.php +++ b/config/constants.php @@ -5,7 +5,7 @@ $sshCommandTimeout = (int) env('SSH_COMMAND_TIMEOUT', 3600); return [ 'coolify' => [ - 'version' => env('COOLIFY_VERSION') ?: '4.4.6', + 'version' => env('COOLIFY_VERSION') ?: '4.5', 'helper_version' => '1.0.18', 'railpack_version' => '0.40.1', 'self_hosted' => env('SELF_HOSTED', true), diff --git a/database/migrations/2026_10_14_105230_add_file_download_storage_to_instance_settings_table.php b/database/migrations/2026_10_14_105230_add_file_download_storage_to_instance_settings_table.php new file mode 100644 index 0000000000..ff1a43faba --- /dev/null +++ b/database/migrations/2026_10_14_105230_add_file_download_storage_to_instance_settings_table.php @@ -0,0 +1,28 @@ +foreignId('file_download_s3_storage_id')->nullable()->constrained('s3_storages')->nullOnDelete(); + }); + } + + /** + * Reverse the migrations. + */ + public function down(): void + { + Schema::table('instance_settings', function (Blueprint $table) { + $table->dropConstrainedForeignId('file_download_s3_storage_id'); + }); + } +}; diff --git a/database/migrations/2026_10_15_213357_spread_default_auto_update_frequency.php b/database/migrations/2026_10_15_213357_spread_default_auto_update_frequency.php new file mode 100644 index 0000000000..9648ab50ec --- /dev/null +++ b/database/migrations/2026_10_15_213357_spread_default_auto_update_frequency.php @@ -0,0 +1,24 @@ +where('auto_update_frequency', '0 0 * * *') + ->update(['auto_update_frequency' => InstanceSettings::randomAutoUpdateFrequency()]); + } + + public function down(): void + { + // + } +}; diff --git a/database/migrations/2026_10_18_103517_add_include_coolify_default_variables_to_resources.php b/database/migrations/2026_10_18_103517_add_include_coolify_default_variables_to_resources.php new file mode 100644 index 0000000000..5059e44005 --- /dev/null +++ b/database/migrations/2026_10_18_103517_add_include_coolify_default_variables_to_resources.php @@ -0,0 +1,40 @@ +boolean('include_coolify_default_variables')->default(true); + }); + } + } + + public function down(): void + { + foreach (self::TABLES as $tableName) { + Schema::table($tableName, function (Blueprint $table) { + $table->dropColumn('include_coolify_default_variables'); + }); + } + } +}; diff --git a/docker-compose.dev-multi.yml b/docker-compose.dev-multi.yml index 153e1c9a1b..1b5e3fad10 100644 --- a/docker-compose.dev-multi.yml +++ b/docker-compose.dev-multi.yml @@ -181,7 +181,10 @@ services: volumes: - minio_data:/data networks: - - coolify + coolify: + # The seeded "Local MinIO" storage (S3StorageSeeder) uses http://coolify-minio:9000. + aliases: + - coolify-minio minio-init: profiles: ["minio"] diff --git a/docker-compose.dev-side-by-side.yml b/docker-compose.dev-side-by-side.yml new file mode 100644 index 0000000000..5c8404495a --- /dev/null +++ b/docker-compose.dev-side-by-side.yml @@ -0,0 +1,9 @@ +# Added by scripts/dev for side-by-side instances (COOLIFY_DEV_SIDE_BY_SIDE=1). Two instances of one +# checkout share storage/, and both have a server with the uuid "localhost", so they would share the +# SSH key and multiplexing files. This gives the instance its own SSH folder. +services: + coolify: + volumes: + - ssh_data:/var/www/html/storage/app/ssh +volumes: + ssh_data: diff --git a/docker/development/Dockerfile b/docker/development/Dockerfile index aa178765ad..06e779d4ae 100644 --- a/docker/development/Dockerfile +++ b/docker/development/Dockerfile @@ -70,6 +70,7 @@ RUN apk add --no-cache \ git-lfs \ jq \ lsof \ + procps \ vim # Install PHP extensions diff --git a/docker/development/etc/s6-overlay/s6-rc.d/user/contents.d/horizon b/docker/development/etc/s6-overlay/user-bundles.d/user/contents.d/horizon similarity index 100% rename from docker/development/etc/s6-overlay/s6-rc.d/user/contents.d/horizon rename to docker/development/etc/s6-overlay/user-bundles.d/user/contents.d/horizon diff --git a/docker/development/etc/s6-overlay/s6-rc.d/user/contents.d/init-setup b/docker/development/etc/s6-overlay/user-bundles.d/user/contents.d/init-setup similarity index 100% rename from docker/development/etc/s6-overlay/s6-rc.d/user/contents.d/init-setup rename to docker/development/etc/s6-overlay/user-bundles.d/user/contents.d/init-setup diff --git a/docker/development/etc/s6-overlay/s6-rc.d/user/contents.d/reverb b/docker/development/etc/s6-overlay/user-bundles.d/user/contents.d/reverb similarity index 100% rename from docker/development/etc/s6-overlay/s6-rc.d/user/contents.d/reverb rename to docker/development/etc/s6-overlay/user-bundles.d/user/contents.d/reverb diff --git a/docker/development/etc/s6-overlay/s6-rc.d/user/contents.d/scheduler-worker b/docker/development/etc/s6-overlay/user-bundles.d/user/contents.d/scheduler-worker similarity index 100% rename from docker/development/etc/s6-overlay/s6-rc.d/user/contents.d/scheduler-worker rename to docker/development/etc/s6-overlay/user-bundles.d/user/contents.d/scheduler-worker diff --git a/docker/development/etc/s6-overlay/s6-rc.d/user/contents.d/terminal-server b/docker/development/etc/s6-overlay/user-bundles.d/user/contents.d/terminal-server similarity index 100% rename from docker/development/etc/s6-overlay/s6-rc.d/user/contents.d/terminal-server rename to docker/development/etc/s6-overlay/user-bundles.d/user/contents.d/terminal-server diff --git a/docker/production/Dockerfile b/docker/production/Dockerfile index 9ddff6476d..5060ef6b13 100644 --- a/docker/production/Dockerfile +++ b/docker/production/Dockerfile @@ -135,6 +135,7 @@ RUN --mount=type=cache,target=/var/cache/apk \ git-lfs \ jq \ lsof \ + procps \ vim # Install PHP extensions @@ -186,6 +187,8 @@ COPY --chown=www-data:www-data resources/views ./resources/views COPY --chown=www-data:www-data artisan artisan COPY --chown=www-data:www-data openapi.yaml ./openapi.yaml COPY --chown=www-data:www-data changelogs/ ./changelogs/ +# Version data for Traefik, Sentinel, and helper checks until CheckForUpdatesJob refreshes it +COPY --chown=www-data:www-data versions.json ./versions.json RUN composer dump-autoload diff --git a/docs/server-transfer-full-coverage.md b/docs/server-transfer-full-coverage.md new file mode 100644 index 0000000000..af6636b8f0 --- /dev/null +++ b/docs/server-transfer-full-coverage.md @@ -0,0 +1,139 @@ +# Full-coverage server for transfer tests + +Set up one source server with everything below, transfer it, then compare each item on the target. +Use values that are NOT the default, so that a lost value is visible after the transfer. + +Use the kit in `scripts/transfer-test/` (see its README): it starts the source (A) and the target (B) and seeds A with a basic set of deployed resources. + +Legend: ✅ `scripts/transfer-test/seed.sh` or `seed-extras.php` already creates it · ➕ add it · ⚠️ needs special infrastructure + +The models come from `ServerTransferCoverage::TRANSFERRED` (`app/Services/ServerTransfer/ServerTransferCoverage.php`). +When you add a model or a setting to the transfer, add it to this file too. + +## 1. Server (Server → General / Proxy) + +| ✓ | What | Value to set | +|---|---|---| +| ✅ | Non-root SSH user | a dev VM with the `ubuntu-non-root` profile | +| ➕ | Name, description | any text | +| ➕ | Port | other than 22, if the VM allows it | +| ➕ | Proxy: redirect | redirect enabled + target URL | +| ✅ | Proxy: custom error page | custom HTML (`proxy_error_page`) | +| ➕ | Server role / build server | mark a second server as a build server | +| ⚠️ | Cloud provider link | Hetzner/Vultr/DigitalOcean/Hostinger server ID + cloud provider token (needs a real cloud server) | +| ⚠️ | Swarm | swarm manager / worker + swarm cluster + swarm destination (needs a swarm server) | + +## 2. Server settings (Server → Advanced, Log drains, Sentinel/Metrics, Traffic, Cloudflare tunnel) + +| ✓ | What | Value to set | +|---|---|---| +| ➕ | Wildcard domain | `http://{ip}.sslip.io` style, not empty | +| ➕ | Concurrent builds, dynamic timeout, deployment queue limit, connection timeout | 3, 1800, 5, 20 | +| ➕ | Server timezone | `Europe/Budapest` | +| ➕ | Docker cleanup | force cleanup on, frequency `0 4 * * *`, threshold 70, delete unused volumes on, delete unused networks on | +| ➕ | Disk usage notification | threshold 70, check frequency `*/30 * * * *`, interval 12 h | +| ➕ | Terminal | terminal access off | +| ➕ | Image retention | "disable application image retention" on | +| ➕ | Labels / Traefik | generate exact labels on, Traefik certificate resolver `custom` | +| ➕ | Backup compression | CPU percentage 50 | +| ➕ | Metrics / Sentinel | metrics on, refresh rate 10 s, history 14 days, push interval 30 s | +| ➕ | Traffic analytics | on, top N 20, sample threshold, retention 1h=3 / 1d=60 days, IP mode `anonymized` | +| ➕ | GeoIP | on, MaxMind license key, refresh 14 days | +| ➕ | Log drains | New Relic (key + base URI), Highlight (project ID), Axiom (dataset + key), custom (config + parser), one of them enabled | +| ✅ | Cloudflare tunnel | tunnel on, tunnel ID, account ID, Cloudflare integration token selected | +| ➕ | Jump server flag | on (on a server that is only a jump host) | + +## 3. Server-level records + +| ✓ | What | Notes | +|---|---|---| +| ✅ | Default destination | created automatically | +| ➕ | Second standalone destination | extra Docker network, one resource on it | +| ➕ | Server shared variables | 2 variables, one multiline, one literal, one with a comment | +| ➕ | SSL certificates | CA certificate (created automatically) + a database with SSL on (creates a server certificate) | +| ✅ | GitHub runner config | labels, max runners 2, docker mode `dind`, image, CPU/memory limits, 3 timeouts, PR jobs off, dedicated on, uses the team GitHub App (see 4). Expected: on the target it is **disabled** and the checklist says to enable it | + +## 4. Team credentials the server uses + +| ✓ | What | Notes | +|---|---|---| +| ✅ | Server private key | | +| ➕ | Deploy key | a second private key used by an application (git over SSH) | +| ✅ | GitHub App (team, not system-wide) | with runner fields (`organization_self_hosted_runners`, `actions`, `webhook_events`, `runner_group_id`). Expected: checklist step for Webhook/Setup/Callback URL | +| ➕ | GitLab App (team) | used by one application. Expected: checklist step for the webhook URL | +| ➕ | S3 storage | the dev MinIO service (the VM must reach it), used by a database backup and a volume backup | +| ✅ | Cloudflare integration token | for DNS zones and the tunnel | +| ✅ | Secret manager token | Infisical, Doppler, or Vault | +| ⚠️ | Cloud provider token | only with a real cloud server | + +## 5. Projects and environments + +| ✓ | What | Notes | +|---|---|---| +| ✅ | 2 projects, 2 environments | for example production and staging | +| ✅/➕ | Project / environment shared variables | the seed tries to create them; check that they exist | +| ✅ | Team shared variable | `TEAM_KEY`, used as `{{team.TEAM_KEY}}` in an application variable. Expected: created on the target | +| ✅ | Project description + icon (JPEG) | Expected: the icon is on the target | +| ✅ | Canvas positions | drag the nodes on the environment canvas | +| ✅ | Resource links | a link between 2 resources on this server, and a link to a resource on another server. Expected: the first moves, the second does not, and the export warns | +| ✅ | Environment with resources only on another server | Expected: it is not exported, and the prechecks show a `NOTE:` line | + +## 6. Applications (one per build pack and source) + +| ✓ | What | Notes | +|---|---|---| +| ✅ | Docker image app (whoami) | | +| ✅ | Dockerfile app | | +| ➕ | Nixpacks app, public GitHub repo | Expected: checklist step for manual webhooks | +| ➕ | Static app (`is_static`, `is_spa`) | | +| ➕ | Docker Compose app | raw compose on, one variable with an empty value (`preserve_compose_null`) | +| ➕ | App with GitHub App source, app with GitLab App source, app with deploy key | | +| ➕ | Domains | 2 domains, noindex domains, domain port overrides, preview URL template | +| ➕ | Health check | custom path, port, method, return code, interval, retries, start period | +| ➕ | Limits | memory, swap, swappiness, reservation, CPUs, cpuset, CPU shares | +| ➕ | Advanced | custom labels, custom docker run options, pre/post deployment command + container, manual webhook secrets | +| ✅ | Maintenance | maintenance mode on + custom maintenance page | +| ➕ | All application settings that are not the default | git submodules, LFS, shallow clone, auto deploy off, force HTTPS, debug, preview deployments on, PR deployments public, log drain, GPU (driver, count, device IDs, options), include timestamps, swarm-only worker nodes, build server, consistent container name, gzip off, strip prefix off, connect to Docker network, custom internal name, container label escape off, env sorting, read-only labels, preserve repository, disable build cache, use build secrets, stop grace period 30, inject build args off, include source commit, Docker images to keep 5, custom container name prefix, redact env values in logs, Coolify default variables off, Traefik certificate resolver | +| ➕ | Environment variables | normal, multiline, literal, shown once, build time only, runtime only, required, with comment, preview-only, and references to `{{team.*}}`, `{{project.*}}`, `{{environment.*}}`, `{{server.*}}` | +| ✅/➕ | Persistent volume | the seed has one; add a host-path bind mount and preview suffix on | +| ➕ | File mount + directory mount | content, chown, chmod | +| ➕ | Volume backup | on the persistent volume: S3 on, stop during backup, all retention values, timeout, missing-backup days | +| ✅/➕ | Scheduled tasks | the seed has one; add a disabled one with container + timeout | +| ✅ | Tag | | +| ➕ | Preview deployment | PR preview with domain port overrides + max restart count (needs the GitHub App, or create it in the database) | +| ✅ | Secret manager link | the app loads variables from the secret manager token | +| ✅ | Managed DNS record | owned record in the Cloudflare zone for the app domain. Expected: the target takes ownership (provider comment changes) | + +## 7. Databases (one of each of the 9 types) + +| ✓ | Type | Extra settings | +|---|---|---| +| ✅ | PostgreSQL | + init script, `postgres_conf`, host auth method, SSL on + mode | +| ➕ | MySQL | + `mysql_conf`, SSL | +| ✅ | MariaDB | + `mariadb_conf`, SSL | +| ➕ | MongoDB | + `mongo_conf`, SSL | +| ✅ | Redis | + `redis_conf`, SSL | +| ➕ | KeyDB, Dragonfly, ClickHouse | defaults are enough, plus the common settings below | +| ➕ | SQLite | database files list, volume link | + +Common settings on at least one database: custom image, public port + public on + public port timeout, limits, custom docker run options, health check values, log drain, include timestamps, redact env values, Coolify default variables off, environment variables, persistent volume + volume backup, tags, secret manager link (if the type supports it). + +Database backups (✅ one exists): add S3 on, dump all, databases list, local and S3 retention (amount, days, max storage), timeout, disable local backup, missing-backup days. + +## 8. Services + +| ✓ | What | Notes | +|---|---|---| +| ✅ | One-click service (uptime-kuma) | | +| ➕ | Custom compose service with an app and a database part | | +| ➕ | Service app settings | domain, gzip off, strip prefix off, force HTTPS, exclude from status, redirect, port overrides, log drain, custom image, Traefik resolver | +| ➕ | Service database settings | public port + public on, custom type | +| ➕ | Service level | connect to network, label escape off, redact env values, Coolify default variables off, maintenance mode + page, environment variables, scheduled task, tags, volume backup on a service volume | + +## 9. After the transfer, check + +- Transfer log: every check `OK`, the `NOTE:` line for the shared project. +- "After the transfer" checklist: GitHub App URLs, GitLab webhook, manual webhooks, GitHub runners. All URLs use the public URL of the target. +- Target: every item above has the same value. Runner config disabled. DNS records owned by the target. +- Source: the server is an archive, the runner config is disabled, and no DNS records are tracked. +- Delete the archive on the source: the containers, DNS records, and GitHub runners are not changed. diff --git a/openapi.json b/openapi.json index 102181f7c0..db7af84045 100644 --- a/openapi.json +++ b/openapi.json @@ -846,7 +846,7 @@ }, "watch_paths": { "type": "string", - "description": "The watch paths." + "description": "Watch path patterns, one per line (newline-separated). Commas do not separate patterns." }, "use_build_server": { "type": "boolean", @@ -1476,7 +1476,7 @@ }, "watch_paths": { "type": "string", - "description": "The watch paths." + "description": "Watch path patterns, one per line (newline-separated). Commas do not separate patterns." }, "use_build_server": { "type": "boolean", @@ -2106,7 +2106,7 @@ }, "watch_paths": { "type": "string", - "description": "The watch paths." + "description": "Watch path patterns, one per line (newline-separated). Commas do not separate patterns." }, "use_build_server": { "type": "boolean", @@ -3586,18 +3586,6 @@ "application\/json": { "schema": { "properties": { - "project_uuid": { - "type": "string", - "description": "The project UUID." - }, - "server_uuid": { - "type": "string", - "description": "The server UUID." - }, - "environment_name": { - "type": "string", - "description": "The environment name." - }, "github_app_uuid": { "type": "string", "description": "The Github App UUID." @@ -3614,10 +3602,6 @@ "type": "string", "description": "The ports to expose." }, - "destination_uuid": { - "type": "string", - "description": "The destination UUID." - }, "build_pack": { "type": "string", "enum": [ @@ -3887,7 +3871,7 @@ }, "watch_paths": { "type": "string", - "description": "The watch paths." + "description": "Watch path patterns, one per line (newline-separated). Commas do not separate patterns." }, "use_build_server": { "type": "boolean", @@ -7302,6 +7286,10 @@ "type": "integer", "description": "CPU shares of the database" }, + "instant_deploy": { + "type": "boolean", + "description": "Restart the database after the update. Requires the deploy ability." + }, "postgres_user": { "type": "string", "description": "PostgreSQL user" @@ -13120,6 +13108,9 @@ "401": { "$ref": "#\/components\/responses\/401" }, + "403": { + "description": "The token needs the deploy ability to set post_install_script_id or public_key_ids." + }, "422": { "description": "Validation failed." }, @@ -17655,6 +17646,304 @@ ] } }, + "\/servers\/{uuid}\/migrate": { + "post": { + "tags": [ + "Servers" + ], + "summary": "Migrate server to another Coolify instance", + "description": "Start a transfer in the background: export this server, import and claim it on the target instance (with the given token), then disable automations here. Follow the progress with GET \/servers\/{uuid}\/transfer. The target URL must use https. Requires read:sensitive and write.", + "operationId": "migrate-server-between-instances", + "parameters": [ + { + "name": "uuid", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + } + ], + "requestBody": { + "required": true, + "content": { + "application\/json": { + "schema": { + "required": [ + "target_url", + "target_token" + ], + "properties": { + "target_url": { + "type": "string", + "example": "https:\/\/coolify-b.example.com" + }, + "target_token": { + "description": "API token on the target instance (root or write)", + "type": "string" + } + }, + "type": "object" + } + } + } + }, + "responses": { + "202": { + "description": "Transfer started" + }, + "403": { + "description": "Missing sensitive permission" + }, + "404": { + "$ref": "#\/components\/responses\/404" + }, + "422": { + "description": "Validation or remote import failed" + } + }, + "security": [ + { + "bearerAuth": [] + } + ] + } + }, + "\/servers\/{uuid}\/transfer": { + "get": { + "tags": [ + "Servers" + ], + "summary": "Get server transfer status", + "description": "Show the transfer state of a server on this instance: the import\/claim state (transfer) and the state of a running or finished transfer to another instance (migration).", + "operationId": "get-server-transfer-status", + "parameters": [ + { + "name": "uuid", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "Transfer status" + }, + "401": { + "$ref": "#\/components\/responses\/401" + }, + "404": { + "$ref": "#\/components\/responses\/404" + } + }, + "security": [ + { + "bearerAuth": [] + } + ] + } + }, + "\/servers\/import\/check": { + "get": { + "tags": [ + "Servers" + ], + "summary": "Check server import", + "description": "Check that this instance can import a server, before the source instance starts the transfer. It checks the token, the bundle version, the server limit, servers with the same UUID or IP, and that this instance can reach the SSH port of the server.", + "operationId": "check-server-import", + "parameters": [ + { + "name": "server_uuid", + "in": "query", + "required": true, + "schema": { + "type": "string" + } + }, + { + "name": "ip", + "in": "query", + "required": true, + "schema": { + "type": "string" + } + }, + { + "name": "port", + "in": "query", + "required": false, + "schema": { + "type": "integer", + "default": 22 + } + }, + { + "name": "schema_version", + "in": "query", + "required": false, + "schema": { + "type": "integer" + } + }, + { + "name": "system_github_apps[]", + "in": "query", + "description": "UUIDs of system-wide GitHub Apps that the server uses. The response lists the ones this instance does not have.", + "required": false, + "schema": { + "type": "array", + "items": { + "type": "string" + } + } + }, + { + "name": "system_gitlab_apps[]", + "in": "query", + "description": "UUIDs of system-wide GitLab Apps that the server uses. The response lists the ones this instance does not have.", + "required": false, + "schema": { + "type": "array", + "items": { + "type": "string" + } + } + } + ], + "responses": { + "200": { + "description": "The server can be imported. Lists every check with its result." + }, + "401": { + "$ref": "#\/components\/responses\/401" + }, + "422": { + "description": "One or more checks failed. Lists every check with its result." + } + }, + "security": [ + { + "bearerAuth": [] + } + ] + } + }, + "\/servers\/import": { + "post": { + "tags": [ + "Servers" + ], + "summary": "Import server transfer bundle", + "description": "Import a server transfer bundle into this Coolify instance (adopt mode by default). This instance then manages the server, and Sentinel reports to it.", + "operationId": "import-server-transfer-bundle", + "requestBody": { + "required": true, + "content": { + "application\/json": { + "schema": { + "properties": { + "bundle": { + "description": "Transfer bundle", + "type": "object" + }, + "dry_run": { + "type": "boolean", + "default": false + }, + "preserve_uuids": { + "type": "boolean", + "default": true + }, + "adopt_mode": { + "description": "Import without forcing redeploy; keep statuses for adoption", + "type": "boolean", + "default": true + } + }, + "type": "object" + } + } + } + }, + "responses": { + "200": { + "description": "Dry-run result" + }, + "201": { + "description": "Imported" + }, + "401": { + "$ref": "#\/components\/responses\/401" + }, + "422": { + "description": "Validation failed" + } + }, + "security": [ + { + "bearerAuth": [] + } + ] + } + }, + "\/servers\/{uuid}\/transfer\/complete": { + "post": { + "tags": [ + "Servers" + ], + "summary": "Mark server transferred", + "description": "Source-instance recovery step: disable automations here when the target already manages the server but the transfer could not finish on this instance.", + "operationId": "complete-server-transfer", + "parameters": [ + { + "name": "uuid", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + } + ], + "requestBody": { + "content": { + "application\/json": { + "schema": { + "properties": { + "export_id": { + "type": [ + "string", + "null" + ] + }, + "target_instance_url": { + "type": [ + "string", + "null" + ] + } + }, + "type": "object" + } + } + } + }, + "responses": { + "200": { + "description": "Marked transferred" + }, + "404": { + "$ref": "#\/components\/responses\/404" + } + }, + "security": [ + { + "bearerAuth": [] + } + ] + } + }, "\/servers": { "get": { "tags": [ @@ -19651,6 +19940,11 @@ "type": "string", "description": "The base64 encoded Docker Compose content." }, + "connect_to_docker_network": { + "type": "boolean", + "default": false, + "description": "Connect the service to the predefined docker network." + }, "urls": { "type": "array", "description": "Array of URLs to be applied to containers of a service.", diff --git a/openapi.yaml b/openapi.yaml index 49cc639bd7..cbfb3b6131 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -542,7 +542,7 @@ paths: items: { properties: { name: { type: string, description: 'The service name as defined in docker-compose.' }, domain: { type: string, description: 'Comma-separated list of URLs (e.g. "https://app.coolify.io,https://app2.coolify.io")' }, redirect: { type: string, nullable: true, description: 'Per-service www/non-www redirect for this compose service.', enum: [www, non-www, both] } }, type: object } watch_paths: type: string - description: 'The watch paths.' + description: 'Watch path patterns, one per line (newline-separated). Commas do not separate patterns.' use_build_server: type: boolean nullable: true @@ -958,7 +958,7 @@ paths: items: { properties: { name: { type: string, description: 'The service name as defined in docker-compose.' }, domain: { type: string, description: 'Comma-separated list of URLs (e.g. "https://app.coolify.io,https://app2.coolify.io")' }, redirect: { type: string, nullable: true, description: 'Per-service www/non-www redirect for this compose service.', enum: [www, non-www, both] } }, type: object } watch_paths: type: string - description: 'The watch paths.' + description: 'Watch path patterns, one per line (newline-separated). Commas do not separate patterns.' use_build_server: type: boolean nullable: true @@ -1374,7 +1374,7 @@ paths: items: { properties: { name: { type: string, description: 'The service name as defined in docker-compose.' }, domain: { type: string, description: 'Comma-separated list of URLs (e.g. "https://app.coolify.io,https://app2.coolify.io")' }, redirect: { type: string, nullable: true, description: 'Per-service www/non-www redirect for this compose service.', enum: [www, non-www, both] } }, type: object } watch_paths: type: string - description: 'The watch paths.' + description: 'Watch path patterns, one per line (newline-separated). Commas do not separate patterns.' use_build_server: type: boolean nullable: true @@ -2369,15 +2369,6 @@ paths: application/json: schema: properties: - project_uuid: - type: string - description: 'The project UUID.' - server_uuid: - type: string - description: 'The server UUID.' - environment_name: - type: string - description: 'The environment name.' github_app_uuid: type: string description: 'The Github App UUID.' @@ -2390,9 +2381,6 @@ paths: ports_exposes: type: string description: 'The ports to expose.' - destination_uuid: - type: string - description: 'The destination UUID.' build_pack: type: string enum: [nixpacks, railpack, static, dockerfile, dockercompose] @@ -2572,7 +2560,7 @@ paths: items: { properties: { name: { type: string, description: 'The service name as defined in docker-compose.' }, domain: { type: string, description: 'Comma-separated list of URLs (e.g. "https://app.coolify.io,https://app2.coolify.io")' }, redirect: { type: string, nullable: true, description: 'Per-service www/non-www redirect for this compose service.', enum: [www, non-www, both] } }, type: object } watch_paths: type: string - description: 'The watch paths.' + description: 'Watch path patterns, one per line (newline-separated). Commas do not separate patterns.' use_build_server: type: boolean nullable: true @@ -4715,6 +4703,9 @@ paths: limits_cpu_shares: type: integer description: 'CPU shares of the database' + instant_deploy: + type: boolean + description: 'Restart the database after the update. Requires the deploy ability.' postgres_user: type: string description: 'PostgreSQL user' @@ -8411,6 +8402,8 @@ paths: description: 'Hostinger VPS purchased and linked to a Coolify server.' '401': $ref: '#/components/responses/401' + '403': + description: 'The token needs the deploy ability to set post_install_script_id or public_key_ids.' '422': description: 'Validation failed.' '429': @@ -11144,6 +11137,204 @@ paths: security: - bearerAuth: [] + '/servers/{uuid}/migrate': + post: + tags: + - Servers + summary: 'Migrate server to another Coolify instance' + description: 'Start a transfer in the background: export this server, import and claim it on the target instance (with the given token), then disable automations here. Follow the progress with GET /servers/{uuid}/transfer. The target URL must use https. Requires read:sensitive and write.' + operationId: migrate-server-between-instances + parameters: + - + name: uuid + in: path + required: true + schema: + type: string + requestBody: + required: true + content: + application/json: + schema: + required: + - target_url + - target_token + properties: + target_url: + type: string + example: 'https://coolify-b.example.com' + target_token: + description: 'API token on the target instance (root or write)' + type: string + type: object + responses: + '202': + description: 'Transfer started' + '403': + description: 'Missing sensitive permission' + '404': + $ref: '#/components/responses/404' + '422': + description: 'Validation or remote import failed' + security: + - + bearerAuth: [] + '/servers/{uuid}/transfer': + get: + tags: + - Servers + summary: 'Get server transfer status' + description: 'Show the transfer state of a server on this instance: the import/claim state (transfer) and the state of a running or finished transfer to another instance (migration).' + operationId: get-server-transfer-status + parameters: + - + name: uuid + in: path + required: true + schema: + type: string + responses: + '200': + description: 'Transfer status' + '401': + $ref: '#/components/responses/401' + '404': + $ref: '#/components/responses/404' + security: + - + bearerAuth: [] + /servers/import/check: + get: + tags: + - Servers + summary: 'Check server import' + description: 'Check that this instance can import a server, before the source instance starts the transfer. It checks the token, the bundle version, the server limit, servers with the same UUID or IP, and that this instance can reach the SSH port of the server.' + operationId: check-server-import + parameters: + - + name: server_uuid + in: query + required: true + schema: + type: string + - + name: ip + in: query + required: true + schema: + type: string + - + name: port + in: query + required: false + schema: + type: integer + default: 22 + - + name: schema_version + in: query + required: false + schema: + type: integer + - + name: 'system_github_apps[]' + in: query + description: 'UUIDs of system-wide GitHub Apps that the server uses. The response lists the ones this instance does not have.' + required: false + schema: + type: array + items: + type: string + - + name: 'system_gitlab_apps[]' + in: query + description: 'UUIDs of system-wide GitLab Apps that the server uses. The response lists the ones this instance does not have.' + required: false + schema: + type: array + items: + type: string + responses: + '200': + description: 'The server can be imported. Lists every check with its result.' + '401': + $ref: '#/components/responses/401' + '422': + description: 'One or more checks failed. Lists every check with its result.' + security: + - + bearerAuth: [] + /servers/import: + post: + tags: + - Servers + summary: 'Import server transfer bundle' + description: 'Import a server transfer bundle into this Coolify instance (adopt mode by default). This instance then manages the server, and Sentinel reports to it.' + operationId: import-server-transfer-bundle + requestBody: + required: true + content: + application/json: + schema: + properties: + bundle: + description: 'Transfer bundle' + type: object + dry_run: + type: boolean + default: false + preserve_uuids: + type: boolean + default: true + adopt_mode: + description: 'Import without forcing redeploy; keep statuses for adoption' + type: boolean + default: true + type: object + responses: + '200': + description: 'Dry-run result' + '201': + description: Imported + '401': + $ref: '#/components/responses/401' + '422': + description: 'Validation failed' + security: + - + bearerAuth: [] + '/servers/{uuid}/transfer/complete': + post: + tags: + - Servers + summary: 'Mark server transferred' + description: 'Source-instance recovery step: disable automations here when the target already manages the server but the transfer could not finish on this instance.' + operationId: complete-server-transfer + parameters: + - + name: uuid + in: path + required: true + schema: + type: string + requestBody: + content: + application/json: + schema: + properties: + export_id: + type: [string, 'null'] + target_instance_url: + type: [string, 'null'] + type: object + responses: + '200': + description: 'Marked transferred' + '404': + $ref: '#/components/responses/404' + security: + - + bearerAuth: [] /servers: get: tags: @@ -12429,6 +12620,10 @@ paths: docker_compose_raw: type: string description: 'The base64 encoded Docker Compose content.' + connect_to_docker_network: + type: boolean + default: false + description: 'Connect the service to the predefined docker network.' urls: type: array description: 'Array of URLs to be applied to containers of a service.' diff --git a/other/nightly/install.sh b/other/nightly/install.sh index 5b5d74f1b1..ebd309a021 100755 --- a/other/nightly/install.sh +++ b/other/nightly/install.sh @@ -7,7 +7,7 @@ ## ROOT_USER_PASSWORD - Predefined root user password ## DOCKER_ADDRESS_POOL_BASE - Custom Docker address pool base (default: 10.0.0.0/8) ## DOCKER_ADDRESS_POOL_SIZE - Custom Docker address pool size (default: 24) -## DOCKER_POOL_FORCE_OVERRIDE - Force override Docker address pool configuration (default: false) +## DOCKER_POOL_FORCE_OVERRIDE - Force override Docker address pool configuration, also when Docker must restart with running containers (default: false) ## AUTOUPDATE - Set to "false" to disable auto-updates ## REGISTRY_URL - Custom registry URL for Docker images (default: docker.io) @@ -31,14 +31,21 @@ fi # Resource directories hold data whose owner and mode the resource images or a non-root SSH user need. # Set their owner and mode only when they are created, never on an upgrade (#11945). -for COOLIFY_DIRECTORY in /data/coolify /data/coolify/{applications,databases,backups,services,proxy,proxy/dynamic,sentinel}; do - if [ ! -d "$COOLIFY_DIRECTORY" ]; then - mkdir -p "$COOLIFY_DIRECTORY" - chown 9999:root "$COOLIFY_DIRECTORY" - chmod 700 "$COOLIFY_DIRECTORY" +create_data_directory() { + if [ ! -d "$1" ]; then + mkdir -p "$1" + chown 9999:root "$1" + chmod 700 "$1" fi -done -mkdir -p /data/coolify/source /data/coolify/images /data/coolify/ssh/{keys,mux} +} + +# Run only after the checks and downloads succeed, so a failed run does not change the data directories (#11973). +create_coolify_directories() { + for COOLIFY_DIRECTORY in /data/coolify/{applications,databases,backups,services,proxy,proxy/dynamic,sentinel}; do + create_data_directory "$COOLIFY_DIRECTORY" + done + mkdir -p /data/coolify/images /data/coolify/ssh/{keys,mux} +} # Coolify's own directories, used by the coolify container (UID 9999) set_coolify_directory_permissions() { @@ -46,7 +53,9 @@ set_coolify_directory_permissions() { chmod -R 700 /data/coolify/{source,ssh,images} } -set_coolify_directory_permissions +# The installation log and the downloaded files go to /data/coolify/source, so only it is created before the checks. +create_data_directory /data/coolify +mkdir -p /data/coolify/source INSTALLATION_LOG_WITH_DATE="/data/coolify/source/installation-${DATE}.log" @@ -828,131 +837,39 @@ step_done "${DOCKER_SERVER_VERSION:+v$DOCKER_SERVER_VERSION}" step_start "Configuring Docker daemon" "log rotation · address pools" -echo " - Network pool configuration: ${DOCKER_ADDRESS_POOL_BASE}/${DOCKER_ADDRESS_POOL_SIZE}" -echo " - To override existing configuration: DOCKER_POOL_FORCE_OVERRIDE=true" - -mkdir -p /etc/docker - -# Backup original daemon.json if it exists -if [ -f /etc/docker/daemon.json ]; then - cp /etc/docker/daemon.json /etc/docker/daemon.json.original-"$DATE" -fi - -# Create coolify configuration with or without address pools based on whether they were explicitly provided +# Changing daemon.json needs a Docker restart, which stops every running container. +# Coolify sets log rotation and the network pool only when no pool is configured or DOCKER_POOL_FORCE_OVERRIDE=true, +# keeps all other keys, and does not restart a Docker daemon with running containers unless DOCKER_POOL_FORCE_OVERRIDE=true. if [ "$DOCKER_POOL_FORCE_OVERRIDE" = true ] || [ "$EXISTING_POOL_CONFIGURED" = false ]; then - # First check if the configuration would actually change anything - if [ -f /etc/docker/daemon.json ]; then - CURRENT_POOL_BASE=$(jq -r '.["default-address-pools"][0].base' /etc/docker/daemon.json 2>/dev/null) - CURRENT_POOL_SIZE=$(jq -r '.["default-address-pools"][0].size' /etc/docker/daemon.json 2>/dev/null) - - if [ "$CURRENT_POOL_BASE" = "$DOCKER_ADDRESS_POOL_BASE" ] && [ "$CURRENT_POOL_SIZE" = "$DOCKER_ADDRESS_POOL_SIZE" ]; then - echo " - Network pool configuration unchanged, skipping update" - NEED_MERGE=false + mkdir -p /etc/docker + CURRENT_DAEMON_CONFIG='{}' + if [ -s /etc/docker/daemon.json ] && ! CURRENT_DAEMON_CONFIG=$(jq -S . /etc/docker/daemon.json 2>/dev/null); then + CURRENT_DAEMON_CONFIG="" + fi + if [ -z "$CURRENT_DAEMON_CONFIG" ]; then + warn "Docker daemon configuration not changed: /etc/docker/daemon.json is not valid JSON." + else + NEW_DAEMON_CONFIG=$(jq -S --arg base "$DOCKER_ADDRESS_POOL_BASE" --argjson size "$DOCKER_ADDRESS_POOL_SIZE" \ + '. * {"log-driver": "json-file", "log-opts": {"max-size": "10m", "max-file": "3"}, "default-address-pools": [{"base": $base, "size": $size}]}' <<<"$CURRENT_DAEMON_CONFIG") + RUNNING_CONTAINERS=$({ docker ps -q 2>/dev/null || true; } | wc -l) + if [ "$NEW_DAEMON_CONFIG" = "$CURRENT_DAEMON_CONFIG" ]; then + echo " - Configuration is up to date" + elif [ "$RUNNING_CONTAINERS" -gt 0 ] && [ "$DOCKER_POOL_FORCE_OVERRIDE" != true ]; then + warn "Docker daemon configuration not changed: log rotation and the ${DOCKER_ADDRESS_POOL_BASE} (size ${DOCKER_ADDRESS_POOL_SIZE}) network pool need a Docker restart, which stops ${RUNNING_CONTAINERS} running container(s) and can change the docker0 bridge IP. To apply it, run the installer again with DOCKER_POOL_FORCE_OVERRIDE=true." + UI_STEP_DETAIL="skipped · containers running" else - # If force override is enabled or no existing configuration exists, - # create a new configuration with the specified address pools - echo " - Creating new Docker configuration with network pool: ${DOCKER_ADDRESS_POOL_BASE}/${DOCKER_ADDRESS_POOL_SIZE}" - cat >/etc/docker/daemon.json </etc/docker/daemon.json </dev/null 2>&1; then - echo " - Log configuration is up to date" - NEED_MERGE=false - else - # Create a configuration without address pools to preserve existing ones - cat >/etc/docker/daemon.json.coolify </etc/docker/daemon.json </etc/docker/daemon.json + echo " - Network pool configuration: ${DOCKER_ADDRESS_POOL_BASE} (size ${DOCKER_ADDRESS_POOL_SIZE})" echo " - Configuration updated - restarting Docker daemon..." restart_docker_service - else - echo " - Configuration is up to date" fi - else - echo " - Configuration is up to date" fi else - if [ "$NEED_MERGE" = true ]; then - echo " - Configuration updated - restarting Docker daemon..." - restart_docker_service - else - echo " - Configuration is up to date" - fi + echo " - Using the existing network pool ${DOCKER_ADDRESS_POOL_BASE} (size ${DOCKER_ADDRESS_POOL_SIZE})" + echo " - To override it: DOCKER_POOL_FORCE_OVERRIDE=true" fi step_done @@ -1070,6 +987,7 @@ step_done step_start "Checking SSH key for localhost access" SSH_KEY_DETAIL="existing key" +create_coolify_directories if [ ! -f ~/.ssh/authorized_keys ]; then mkdir -p ~/.ssh chmod 700 ~/.ssh diff --git a/other/nightly/upgrade.sh b/other/nightly/upgrade.sh index ce8fdb6589..4d594cb9e9 100644 --- a/other/nightly/upgrade.sh +++ b/other/nightly/upgrade.sh @@ -38,6 +38,65 @@ write_status() { echo "${step}|${message}|$(date -Iseconds)" > "$STATUS_FILE" } +# Print the first gateway of a network that Docker reports with a prefix length (fd00::1/64). +# Docker Engine 27.0 to 28.0.0 does this for IPv6 until the daemon restarts (https://github.com/moby/moby/pull/49520). +# The Docker CLI and Compose in the helper image fail on such a network with a ParseAddr error. +network_gateway_with_prefix() { + docker network inspect "$1" --format '{{range .IPAM.Config}}{{.Gateway}} {{end}}' 2>&1 | grep -oE '[0-9A-Fa-f:.]+/[0-9]+' | head -n1 +} + +# Make sure the coolify network exists and the helper image can read it. +# It is created when starting Coolify with docker compose +ensure_coolify_network() { + log "Checking Docker network 'coolify'..." + if ! docker network inspect coolify >/dev/null 2>&1; then + log "Network 'coolify' does not exist, creating..." + if ! docker network create --attachable --ipv6 coolify 2>/dev/null; then + log "Failed to create network with IPv6, trying without IPv6..." + docker network create --attachable coolify 2>/dev/null + log "Network 'coolify' created without IPv6" + elif [ -n "$(network_gateway_with_prefix coolify)" ]; then + # No container uses the new network yet, so it is safe to recreate it. + log "Docker reports the IPv6 gateway of network 'coolify' with a prefix length, recreating it without IPv6..." + docker network rm coolify >/dev/null 2>&1 + docker network create --attachable coolify 2>/dev/null + log "Network 'coolify' created without IPv6" + else + log "Network 'coolify' created with IPv6 support" + fi + return 0 + fi + + log "Network 'coolify' already exists" + local gateway + gateway=$(network_gateway_with_prefix coolify) + if [ -n "$gateway" ]; then + local message="Docker Engine $(docker version --format '{{.Server.Version}}' 2>/dev/null) reports the gateway of the 'coolify' network as ${gateway}, which Coolify cannot read (Docker Engine bug, fixed in 28.0.1). Run 'systemctl restart docker' (this restarts all containers) or upgrade Docker Engine, then upgrade Coolify again." + log "ERROR: ${message}" + write_status "error" "$message" + echo " ERROR: ${message}" + return 1 + fi +} + +# Pull an image. Retry with backoff, because the registry can rate limit many instances that pull at the same time. +pull_image() { + local image="$1" + local attempt=1 + local max_attempts=5 + local delay=15 + until docker pull "$image" >>"$LOGFILE" 2>&1; do + if [ "$attempt" -ge "$max_attempts" ]; then + return 1 + fi + log "Pull of $image failed (attempt $attempt/$max_attempts). Retrying in ${delay}s..." + echo " Pull failed (attempt $attempt/$max_attempts). Retrying in ${delay}s..." + sleep "$((delay + RANDOM % 10))" + attempt=$((attempt + 1)) + delay=$((delay * 2)) + done +} + echo "" echo "==========================================" echo " Coolify Upgrade - ${DATE}" @@ -159,21 +218,7 @@ fi log "Environment variables check complete" echo " Done." -# Make sure coolify network exists -# It is created when starting Coolify with docker compose -log "Checking Docker network 'coolify'..." -if ! docker network inspect coolify >/dev/null 2>&1; then - log "Network 'coolify' does not exist, creating..." - if ! docker network create --attachable --ipv6 coolify 2>/dev/null; then - log "Failed to create network with IPv6, trying without IPv6..." - docker network create --attachable coolify 2>/dev/null - log "Network 'coolify' created without IPv6" - else - log "Network 'coolify' created with IPv6 support" - fi -else - log "Network 'coolify' already exists" -fi +ensure_coolify_network || exit 1 mkdir -p /data/coolify/images/{avatars,project-icons} chown -R 9999:root /data/coolify/images @@ -205,7 +250,7 @@ echo " This may take a few minutes depending on your connection." HELPER_IMAGE="${REGISTRY_URL:-docker.io}/coollabsio/coolify-helper:${LATEST_HELPER_VERSION}" echo " - Pulling $HELPER_IMAGE..." log "Pulling image: $HELPER_IMAGE" -if docker pull "$HELPER_IMAGE" >>"$LOGFILE" 2>&1; then +if pull_image "$HELPER_IMAGE"; then log "Successfully pulled $HELPER_IMAGE" else log "ERROR: Failed to pull $HELPER_IMAGE" @@ -220,7 +265,7 @@ for IMAGE in $IMAGES; do if [ -n "$IMAGE" ]; then echo " - Pulling $IMAGE..." log "Pulling image: $IMAGE" - if docker pull "$IMAGE" >>"$LOGFILE" 2>&1; then + if pull_image "$IMAGE"; then log "Successfully pulled $IMAGE" else log "ERROR: Failed to pull $IMAGE" @@ -298,8 +343,16 @@ nohup bash -c " fi log 'Running docker compose up...' - docker run -v /data/coolify/source:/data/coolify/source -v /var/run/docker.sock:/var/run/docker.sock \${DOCKER_CONFIG_MOUNT} --rm \${REGISTRY_URL:-docker.io}/coollabsio/coolify-helper:\${LATEST_HELPER_VERSION} bash -c \"LATEST_IMAGE=\${LATEST_IMAGE} docker compose --env-file /data/coolify/source/.env \${COMPOSE_FILES} up -d --remove-orphans --wait --wait-timeout 60\" >>\"\$LOGFILE\" 2>&1 - log 'Docker compose up completed' + if docker run -v /data/coolify/source:/data/coolify/source -v /var/run/docker.sock:/var/run/docker.sock \${DOCKER_CONFIG_MOUNT} --rm \${REGISTRY_URL:-docker.io}/coollabsio/coolify-helper:\${LATEST_HELPER_VERSION} bash -c \"LATEST_IMAGE=\${LATEST_IMAGE} docker compose --env-file /data/coolify/source/.env \${COMPOSE_FILES} up -d --remove-orphans --wait --wait-timeout 60\" >>\"\$LOGFILE\" 2>&1; then + log 'Docker compose up completed' + elif [ \"\$(docker inspect --format '{{.State.Running}}' coolify 2>/dev/null)\" = 'true' ]; then + # A slow first start (for example long migrations) can pass the wait timeout. Coolify is still starting. + log 'Docker compose up did not finish waiting, but the coolify container is running' + else + log 'ERROR: Docker compose up failed and the coolify container is not running' + write_status 'error' \"Coolify did not start. Check the upgrade log: \$LOGFILE\" + exit 1 + fi # Final log entry echo '' >>\"\$LOGFILE\" diff --git a/other/nightly/versions.json b/other/nightly/versions.json index 4165d6831e..7dc784d8e6 100644 --- a/other/nightly/versions.json +++ b/other/nightly/versions.json @@ -1,10 +1,10 @@ { "coolify": { "v4": { - "version": "4.4.6" + "version": "4.5" }, "nightly": { - "version": "4.5-rc.1" + "version": "4.6-rc.1" }, "helper": { "version": "1.0.18" diff --git a/package-lock.json b/package-lock.json index fd2834a7b9..4c99d1fa38 100644 --- a/package-lock.json +++ b/package-lock.json @@ -684,6 +684,72 @@ "node": ">=14.0.0" } }, + "node_modules/@tailwindcss/oxide-wasm32-wasi/node_modules/@emnapi/core": { + "version": "1.11.1", + "dev": true, + "inBundle": true, + "license": "MIT", + "optional": true, + "dependencies": { + "@emnapi/wasi-threads": "1.2.2", + "tslib": "^2.4.0" + } + }, + "node_modules/@tailwindcss/oxide-wasm32-wasi/node_modules/@emnapi/runtime": { + "version": "1.11.1", + "dev": true, + "inBundle": true, + "license": "MIT", + "optional": true, + "dependencies": { + "tslib": "^2.4.0" + } + }, + "node_modules/@tailwindcss/oxide-wasm32-wasi/node_modules/@emnapi/wasi-threads": { + "version": "1.2.2", + "dev": true, + "inBundle": true, + "license": "MIT", + "optional": true, + "dependencies": { + "tslib": "^2.4.0" + } + }, + "node_modules/@tailwindcss/oxide-wasm32-wasi/node_modules/@napi-rs/wasm-runtime": { + "version": "1.1.4", + "dev": true, + "inBundle": true, + "license": "MIT", + "optional": true, + "dependencies": { + "@tybys/wasm-util": "^0.10.1" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + }, + "peerDependencies": { + "@emnapi/core": "^1.7.1", + "@emnapi/runtime": "^1.7.1" + } + }, + "node_modules/@tailwindcss/oxide-wasm32-wasi/node_modules/@tybys/wasm-util": { + "version": "0.10.2", + "dev": true, + "inBundle": true, + "license": "MIT", + "optional": true, + "dependencies": { + "tslib": "^2.4.0" + } + }, + "node_modules/@tailwindcss/oxide-wasm32-wasi/node_modules/tslib": { + "version": "2.8.1", + "dev": true, + "inBundle": true, + "license": "0BSD", + "optional": true + }, "node_modules/@tailwindcss/oxide-win32-arm64-msvc": { "version": "4.3.3", "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-win32-arm64-msvc/-/oxide-win32-arm64-msvc-4.3.3.tgz", @@ -1479,9 +1545,9 @@ } }, "node_modules/source-map-js": { - "version": "1.2.1", - "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz", - "integrity": "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==", + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.2.tgz", + "integrity": "sha512-KGj/8Y43x35aZVDtt+J4mK1hoLGHULMYfSkODJNQjNDC3oW1PqPoxMwo0pLUsWM/UEGzON/NxeHywEfNXNP3Vw==", "dev": true, "license": "BSD-3-Clause", "engines": { @@ -1568,14 +1634,6 @@ "url": "https://github.com/sponsors/SuperchupuDev" } }, - "node_modules/tslib": { - "version": "2.8.1", - "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz", - "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==", - "dev": true, - "license": "0BSD", - "optional": true - }, "node_modules/tw-animate-css": { "version": "1.4.0", "resolved": "https://registry.npmjs.org/tw-animate-css/-/tw-animate-css-1.4.0.tgz", diff --git a/resources/css/app.css b/resources/css/app.css index d752416566..e21e9611ea 100644 --- a/resources/css/app.css +++ b/resources/css/app.css @@ -1951,6 +1951,15 @@ html[data-theme="custom"] textarea:disabled { grid-column: 2; } + /* A resource menu that the main sidebar shows instead (navbar #main-nav-context-{depth}). */ + .application-settings-navigation.in-main-nav { + display: none; + } + + .application-settings-navigation.in-main-nav ~ * { + grid-column: 1 / -1; + } + html:not(.dark) .application-settings-navigation { --rail-fill: var(--coollabs-base); } @@ -1978,7 +1987,7 @@ html[data-theme="custom"] textarea:disabled { z-index: 20; } - .app-main-full .application-settings-navigation ~ [data-environment-canvas] { + .app-main-full .application-settings-navigation:not(.in-main-nav) ~ [data-environment-canvas] { margin-left: calc(210px + 2.5rem); } @@ -2097,6 +2106,17 @@ html[data-theme="custom"] textarea:disabled { background: transparent; } +/* Main sidebar: only the menu scrolls on short or zoomed-in screens; the + footer stays visible. Hide the scrollbar so it does not take width from the + narrow collapsed rail and push the icons aside. */ +.sidebar-menu { + scrollbar-width: none; /* Firefox */ +} + +.sidebar-menu::-webkit-scrollbar { + display: none; /* Chrome/Safari */ +} + /* Unified resource heading bar: hide tab-strip scrollbars; menus/actions stay overflow-visible for dropdowns */ .resource-heading-navbar { overflow: visible; @@ -2226,7 +2246,7 @@ html[data-theme="custom"] textarea:disabled { .application-heading-actions .button, .application-heading-actions .app-tab, .application-heading-actions .relative > button[x-ref='trigger'] { - height: 1.75rem; + height: 2rem; display: inline-flex; align-items: center; gap: 0.375rem; @@ -2383,11 +2403,6 @@ html[data-theme="custom"] textarea:disabled { box-shadow: 0 0 0 1px var(--color-accent); } -.application-heading-actions .split-action-main, -.application-heading-actions .split-action-caret { - height: 1.75rem; -} - /* Custom listbox (replaces native diff --git a/resources/views/components/top-breadcrumb.blade.php b/resources/views/components/top-breadcrumb.blade.php index d2751296dd..7d18f67b17 100644 --- a/resources/views/components/top-breadcrumb.blade.php +++ b/resources/views/components/top-breadcrumb.blade.php @@ -102,33 +102,6 @@ request()->routeIs('admin.*') => 'Admin', default => null, }; - // Workspace destinations require an active plan on cloud; unsubscribed users - // only keep profile/appearance and must not see this switcher. - $canUseWorkspaceNav = isSubscribed() || ! isCloud(); - $pageDestinations = $canUseWorkspaceNav - ? collect([ - ['label' => 'Dashboard', 'href' => url('/')], - ['label' => 'Projects', 'href' => url('/projects')], - auth()->user()?->can('canAccessTerminal') - ? ['label' => 'Terminal', 'href' => route('terminal')] - : null, - ['label' => 'Servers', 'href' => url('/servers')], - ['label' => 'Sources', 'href' => route('source.all')], - ['label' => 'Destinations', 'href' => route('destination.index')], - auth()->user()?->isAdmin() - ? ['label' => 'Registries', 'href' => route('registries.index')] - : null, - ['label' => 'S3 Storage', 'href' => route('storage.index')], - ['label' => 'Shared Variables', 'href' => route('shared-variables.index')], - ['label' => 'Team', 'href' => route('team.index')], - ['label' => 'Notifications', 'href' => route('notifications.email')], - ['label' => 'Keys & Tokens', 'href' => route('security.private-key.index')], - ['label' => 'Tags', 'href' => route('tags.show')], - isInstanceAdmin() - ? ['label' => 'Settings', 'href' => route('settings.index')] - : null, - ])->filter() - : collect(); @endphp
{{-- Team --}} @@ -136,33 +109,7 @@
- @if (!$currentProject && $dashboardContext && $canUseWorkspaceNav) - / -
- -
-
- Pages -
- @foreach ($pageDestinations as $destination) - - {{ $destination['label'] }} - - @endforeach -
-
- @elseif (!$currentProject && $dashboardContext) - {{-- Static context label only — no links to paid workspace pages. --}} + @if (!$currentProject && $dashboardContext) / {{ $dashboardContext }} @@ -247,7 +194,7 @@ {{-- Project switcher --}}
-
diff --git a/resources/views/layouts/app.blade.php b/resources/views/layouts/app.blade.php index 256717db01..3b2a560974 100644 --- a/resources/views/layouts/app.blade.php +++ b/resources/views/layouts/app.blade.php @@ -9,51 +9,22 @@ @auth
$fullBleed ?? false])> {{-- ============ DESKTOP TOP BAR ============ --}} @@ -68,15 +39,15 @@ {{-- Brand (width tracks sidebar) --}}
-
+ @if (isInstanceAdmin() && !isCloud())
@@ -98,6 +69,7 @@ {{-- Page toolbars (the canvases) dock here on desktop. --}}
+ {{-- Resource actions dock here on desktop. --}}
@@ -149,8 +121,8 @@ {{-- ============ DESKTOP SIDEBAR (below top bar) ============ --}} diff --git a/resources/views/livewire/analytics.blade.php b/resources/views/livewire/analytics.blade.php index c326ac29d5..f383f56506 100644 --- a/resources/views/livewire/analytics.blade.php +++ b/resources/views/livewire/analytics.blade.php @@ -254,14 +254,15 @@ $appListboxOptions = array_merge( icon-name="unordered-list" /> @else @php $maxAppRequests = max(1, (int) collect($topApps)->max('requests')); @endphp -
-
- Application / service - - Requests - +
+
+ @include('livewire.traffic._sort-header', ['column' => 'name', 'label' => 'Application / service', 'text' => true, 'class' => 'min-w-0 flex-1']) + + @include('livewire.traffic._sort-header', ['column' => 'requests', 'label' => 'Requests', 'class' => 'w-16 shrink-0']) + @include('livewire.traffic._sort-header', ['column' => 'bytesOut', 'label' => 'Bandwidth', 'class' => 'hidden w-16 shrink-0 sm:inline-flex', 'title' => 'Total response data sent'])
+
@foreach ($topApps as $row) @php $appHref = $row['link'] ?? null; @@ -269,11 +270,9 @@ $appListboxOptions = array_merge( @endphp <{{ $appHref ? 'a' : 'div' }} wire:key="analytics-app-{{ $row['uuid'] }}" @if ($appHref) href="{{ $appHref }}" {{ wireNavigate() }} @endif - x-show="{{ $loop->index }} >= page * per && {{ $loop->index }} < (page + 1) * per" - @class([ - 'flex min-h-11 items-center gap-3 border-b border-neutral-200 px-4 py-2 last:border-b-0 dark:border-white/[0.07]', - 'transition-colors hover:bg-neutral-50 dark:hover:bg-white/[0.03]' => $appHref, - ])> + data-row="{{ json_encode(['name' => (string) $row['name'], 'requests' => (int) $row['requests'], 'bytesOut' => (int) $row['bandwidth']]) }}" + x-show="shown($el)" + class="flex min-h-11 items-center gap-3 border-t border-neutral-200 px-4 py-2 transition-colors hover:bg-neutral-50 dark:border-white/[0.07] dark:hover:bg-white/[0.03]"> {{ $row['name'] }} @if (! empty($row['isService'])) @@ -299,6 +298,7 @@ $appListboxOptions = array_merge( @endif @endforeach +
@include('livewire.traffic._pager')
@endif @@ -312,14 +312,12 @@ $appListboxOptions = array_merge( @include('livewire.traffic._paths-list', ['paths' => $topPaths]) - {{-- Countries --}} - - @include('livewire.traffic._geo', [ - 'countries' => data_get($breakdowns, 'country', []), - 'attribution' => $attribution, - ]) - + {{-- Locations (countries and cities) --}} + @include('livewire.traffic._geo', [ + 'countries' => data_get($breakdowns, 'country', []), + 'cities' => trafficCityRows(data_get($breakdowns, 'city', [])), + 'attribution' => $attribution, + ]) {{-- Requests by device type (donut) + HTTP versions / cache / status. --}}
diff --git a/resources/views/livewire/dashboard.blade.php b/resources/views/livewire/dashboard.blade.php index 5de545df33..352060701a 100644 --- a/resources/views/livewire/dashboard.blade.php +++ b/resources/views/livewire/dashboard.blade.php @@ -12,6 +12,38 @@ $dashboardProjects = $projects->sortBy('name', SORT_NATURAL)->take($dashboardItemLimit); $dashboardServers = $servers->sortBy('name', SORT_NATURAL)->take($dashboardItemLimit); $hasTrafficAnalytics = $servers->contains(fn ($server) => $server->isTrafficAnalyticsEnabled()); + $projectResourceCounts = $dashboardProjects->mapWithKeys(fn ($project) => [$project->id => collect([ + $project->applications_count, + $project->services_count, + $project->postgresqls_count, + $project->redis_count, + $project->keydbs_count, + $project->dragonflies_count, + $project->clickhouses_count, + $project->mongodbs_count, + $project->mysqls_count, + $project->mariadbs_count, + $project->sqlites_count, + ])->sum()]); + $serverStatuses = $dashboardServers->mapWithKeys(function ($server) { + $proxyNeedsAttention = $server->proxySet() && ($server->proxy->status !== 'running' || $server->hasCurrentTraefikOutdatedInfo()); + $sentinelNeedsAttention = $server->isSentinelEnabled() && $server->sentinelStatus() === 'out_of_sync'; + + return [$server->id => match (true) { + $server->settings->force_disabled => ['Disabled', 'error'], + ! $server->settings->is_reachable && ! $server->settings->is_usable => ['Unavailable', 'error'], + ! $server->settings->is_reachable => ['Unreachable', 'error'], + ! $server->settings->is_usable => ['Not ready', 'warning'], + $proxyNeedsAttention || $sentinelNeedsAttention => ['Attention required', 'warning'], + default => ['Ready', 'success'], + }]; + }); + $iconButton = 'flex size-6.5 items-center justify-center rounded-md text-neutral-400 transition-colors hover:bg-neutral-100 hover:text-black dark:text-fg-faint dark:hover:bg-white/[0.06] dark:hover:text-fg'; + // Same look as $iconButton, sized to the 20px section title it sits next to. + $titleIconButton = 'flex size-5 items-center justify-center rounded-md text-neutral-400 transition-colors hover:bg-neutral-100 hover:text-black dark:text-fg-faint dark:hover:bg-white/[0.06] dark:hover:text-fg'; + $tableShell = 'overflow-x-auto rounded-xl border border-neutral-200 bg-white shadow-sm dark:border-white/[0.08] dark:bg-white/[0.05]'; + $tableHead = 'border-b border-neutral-200 bg-neutral-50 px-4 py-2.5 text-[11px] font-medium text-neutral-500 dark:border-white/[0.08] dark:bg-white/[0.05] dark:text-fg-faint'; + $tableRow = 'min-h-14 items-center border-b border-neutral-200 px-4 py-2.5 text-[12px] transition-colors last:border-b-0 hover:bg-neutral-50 dark:border-white/[0.07] dark:hover:bg-white/[0.025]'; @endphp
@@ -32,38 +64,59 @@
@endif + {{-- Wide screens: activity takes the left third, projects and servers the other two thirds. + It stays one column while the left side has no section (no deployments, no traffic). --}} +
+ -
- +
+
+ + {{-- Projects has no sidebar entry, so the dashboard is where new projects start. --}} + @can('createAnyResource') + + + + + + + + + @endcan + {{-- The dashboard canvas link lives here, so it never adds a row above the page. --}} + + + + @if ($dashboardProjects->isEmpty()) @else -
+
@foreach ($dashboardProjects as $project) @php $firstEnvironment = $project->environments->first(); - $resourceCount = collect([ - $project->applications_count, - $project->services_count, - $project->postgresqls_count, - $project->redis_count, - $project->keydbs_count, - $project->dragonflies_count, - $project->clickhouses_count, - $project->mongodbs_count, - $project->mysqls_count, - $project->mariadbs_count, - $project->sqlites_count, - ])->sum(); + $resourceCount = $projectResourceCounts[$project->id]; @endphp
$firstEnvironment->uuid, ]) }}" {{ wireNavigate() }} - class="flex size-6.5 items-center justify-center rounded-md text-neutral-400 transition-colors hover:bg-neutral-100 hover:text-black dark:text-fg-faint dark:hover:bg-white/[0.06] dark:hover:text-fg" + class="{{ $iconButton }}" title="Add resource" aria-label="Add resource to {{ $project->name }}"> @@ -128,7 +181,7 @@ @can('update', $project) @@ -139,12 +192,101 @@
@endforeach
+ +
+
+
Project
+
Contents
+ +
+
+ @foreach ($dashboardProjects as $project) + @php + $firstEnvironment = $project->environments->first(); + @endphp +
+ +
+
+ @if ($project->icon_path) + {{ $project->name }} icon + @else + + @endif +
+

+ {{ $project->name }} +

+
+
+ + + {{ $project->environments->count() }} + + + + {{ $projectResourceCounts[$project->id] }} + +
+ +
+ @if ($firstEnvironment) + @can('createAnyResource') + + + + @endcan + @endif + @can('update', $project) + + + + @endcan +
+
+ @endforeach +
@endif
-
- +
+ + @can('createAnyResource') + + + + + + @endcan + @if ($dashboardServers->isNotEmpty()) + + + + @endif + @if ($dashboardServers->isEmpty()) @if ($privateKeys->isEmpty()) @@ -177,20 +319,11 @@ @endif @else -
+
@foreach ($dashboardServers as $server) @php - $proxyNeedsAttention = $server->proxySet() && ($server->proxy->status !== 'running' || $server->hasCurrentTraefikOutdatedInfo()); - $sentinelNeedsAttention = $server->isSentinelEnabled() && $server->sentinelStatus() === 'out_of_sync'; - - [$serverStatus, $serverStatusType] = match (true) { - $server->settings->force_disabled => ['Disabled', 'error'], - ! $server->settings->is_reachable && ! $server->settings->is_usable => ['Unavailable', 'error'], - ! $server->settings->is_reachable => ['Unreachable', 'error'], - ! $server->settings->is_usable => ['Not ready', 'warning'], - $proxyNeedsAttention || $sentinelNeedsAttention => ['Attention required', 'warning'], - default => ['Ready', 'success'], - }; + [$serverStatus, $serverStatusType] = $serverStatuses[$server->id]; @endphp @endforeach
+ @endif
+
+
diff --git a/resources/views/livewire/dashboard/active-deployments.blade.php b/resources/views/livewire/dashboard/active-deployments.blade.php index cea0489a20..7d2e88b2ff 100644 --- a/resources/views/livewire/dashboard/active-deployments.blade.php +++ b/resources/views/livewire/dashboard/active-deployments.blade.php @@ -24,10 +24,11 @@ $hasAnyDeployments = $hasActiveDeployments || $hasRecentDeployments; @endphp -
$hasAnyDeployments, 'hidden' => ! $hasAnyDeployments])> +
$hasAnyDeployments, 'hidden' => ! $hasAnyDeployments])> @if ($hasAnyDeployments) -
- + {{-- Container breakpoints: the table needs room, which the dashboard column may not have. --}} +
+
@if ($hasActiveDeployments) @@ -43,7 +44,7 @@