- When pulling warmup images for concurrent tests, save
the images to a cache directory under the application root.
- Serial tests that aren't testing pull can save time by restoring
a cached warmup image.
- Closes#2001.
- Handle "container exists" error gracefully instead
of failing, when trying to start the buildkit container.
- Move build tests to parallel suites, while the builder
lifecycle tests remain serial. Parallel builds don't
use the fixture lock that deletes and restarts the
builder and runs a build block in isolation.
- Part of #1833.
- CLI progress and registry test migrations were inadventently reverted
by #1857.
- Migrate TestCLINoParallelCases to TestCLIImagePruneSerial and
TestCLINetworkPruneSerial.
- Clean up test selection patterns in Makefile.
- Remove all legacy CLITests files.
- Use swift-testing `withKnownIssue` to run but ignore failures on flaky
`testCreateNameLongestValid` and `testIsolatedNetwork`.
- Extracts a fixture helper for tests requiring a retry loop.
- Part of #1833.
- Replace old targets with new ones.
- Try increased parallelism after test tweaks in #1857.
- Exclude test files from coverage analysis.
- Part of #1833.
- Adds `ContainerFixture` with scoped resource lifecycle and cleanup in
place of implementation inheritance for test support functions. The
fixture also handles resource prefixing and uses a more ergonomic
`CommandResult` in place of a tuple for return values.
- `ImageWarmup` suite pre-pulls well-known images, and
`copyWarmupImage()` tags test-local refs, keeping the canonical image
store untouched.
- Three-phase `integration-new`: warmup, followed by concurrent tests
(managed by the swift test
`--experimental-maximum-parallelization-width` flag), followed by
serialized tests.
- `coverage-new` merges unit + integration-new profraw, replacing
`coverage` in CI as a migration progress indicator.
- Updates GH workflow so non-coverage invokes both the `integration` and
`integration-new` Makefile targets, while coverage runs invoke the
`coverage-new` target.
- Closes#1812.
- The network plugin is the source of truth for the variant, if any,
that applies to the network. Resolving a missing variant configuration
option in the API server can create a situation where the variant the
runtime uses for interface selection is incorrect.
- Adds serial suites trait to tests to see whether it helps current CI
issues.
## Type of Change
- [x] Bug fix
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
## Motivation and Context
Fixes a flaw in our interface strategy logic.
## Testing
- [x] Tested locally
- [ ] Added/updated tests
- [ ] Added/updated docs
- PRs are backed up. We need to rework the CLI tests
to shorten test time and fix conflicts between tests.
## Testing
- [ ] Tested locally
- [x] Added/updated tests
- [ ] Added/updated docs
- Closes#1750.
- Applies permission code used for the `--ssh` mount to all
host-to-container socket mounts.
- Adds a user option to the `doExec` test support function.
- Updates the `testRunCommandUnixSocketMount` to install `nc` in the
test container, and check the socket permission, and check the mounted
socket using `nc` as the guest user.
- Closes#1647.
- `id` will become a system assigned (Docker-like) identifier
for the managed resource, and `configuration.name` is the
user-assigned name.
Normalize JSON output for network and volume resources. (#1624)
- Reworks both JSON output presets to use sorted keys, ISO timestamps.
`compact` is used for `ls` output, and `pretty` is used for `inspect`.
- Extracts non-DRY option configuration into presets.
- Closes#1623.
- Refactor network model types: replace `NetworkState` enum and
phase-based NetworkStatus with a flat `NetworkStatus` struct.
- Simplify API server ↔ plugin protocol: plugin `status()` returns
runtime status only, API server owns configuration.
- `NetworksService` `list()`/`create()` now return `NetworkResource`
directly.
- Remove lifecycle phase checks and state machine guards throughout CLI
and API server.
- `variant` is plugin-specific, it's not a required property. This PR
replaces `NetworkPluginInfo` with a `plugin` name property on
`NetworkConfiguration` and an `options` list similar to that for
volumes.
- Moved `variant` to the option list.
- Part of #1404.
- Updates containerization to 0.33.2.
- Reorganizes network plugin targets into:
- `ContainerNetworkClient` - network plugin client and default types
- `ContainerNetworkServer` - separate protocols for `Network` which
manages the underlying virtual network, `NetworkService`, which takes a
network and implements the API, and an actor `NetworkHarness` that
marshals between the API and the XPC protocol. The service-harness
separation will help us ensure XPC protocol compatibility in both
directions as we evolve the plugin APIs.
- Removes `disableAllocator()` which is no longer used since #1545
switched over to using XPC connections between runtime and network
plugin instances to track whether a network has attached containers.
- Closes#1610.
- Discovered, and originally filed as a security advisory, by:
PresidentL <131139636+liyander@users.noreply.github.com>.
- `PublishPort` currently can store invalid combinations of starting
port and range that can overflow UInt16 values when summed, crashing the
process.
- Updates `PublishPort` to validate inputs on initialization.
- Closes#1557.
- Replaces `executableURL` utility function for getting app executable
path with `executablePath`.
- Adds `FilePath.resolvingSymlinks()` extension.
- Also converts for FilePath for types in `ContainerVersion` target.
- Closes#1540.
- Use a published vminit image in the custom init example.
- Refine the integration test example in BUILDING.md to build in an
isolated app-root directory.
## Type of Change
- [ ] Bug fix
- [ ] New feature
- [ ] Breaking change
- [x] Documentation update
## Motivation and Context
We don't make `latest` tags for vminit images, so the example would
fail.
## Testing
- [ ] Tested locally
- [ ] Added/updated tests
- [x] Added/updated docs
- Closes#1565.
- `runtime-config.toml` suggested the config
pertains to just the runtime (managing container
workloads) aspect of the system.
- Also renames `ADDR` columns to `IP` in
`container ls` and `container builder status`.
- Closes#1566.
- Currently on `system start` we only remove the read-only user config
if the user config file exists. This means that changes to the config
are processed properly on restart, but a removal is ignored, and the old
read-only settings linger.
- Closes#1318.
- Closes#1378.
- Reduces the complexity and coupling for IP allocation.
- Runtimes connect to networks for the life of the running container.
The runtime shuts down on connection loss.
- Networks automatically deallocate a runtime's IP address and hostname
record on connection loss.
- Removes AllocatedAttachment as this is no longer necessary. The
`bootstrap()` XPC now takes a `NetworkBootstrapInfo` array which
parallels the attachments in the bundle config and provides the network
plugin attributes needed to create VM network interface configurations.
## Type of Change
- [x] Bug fix
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
## Motivation and Context
Simplify IP allocation and make deallocation more reliable.
## Testing
- [x] Tested locally
- [ ] Added/updated tests
- [ ] Added/updated docs
- Part of #1318.
- Part of #1378.
- Removes network plugin `deallocate()`, and allocate takes an
`XPCServerSession` that registers an `onDisconnect` handler that
performs deallocation.
- ContainerService now tracks `networkSessions` for allocations.
- Part of #1318.
- Part of #1378.
- `RouteHandler` now accepts an additional `XPCServerSession` which
include a disconnect handler.
- `XPCServer` configures a session for each client connection and passes
it to every route handler.
- `XPCServer.route(fn:)` can be used to wrap existing route handlers
that don't care about sessions.
- `XPCClient` adds functions to support client-side session handling.
- Update all existing routes.
- Currently we need to use `pfctl` to create the packet filter rule for
container-to-host networking (`host.docker.internal`). This approach
unfortunately has a few limitations that require documentation.
- Part of #1404.
- Evolves API to use NetworkResource that conforms to ManagedResource.
- BREAKING CHANGE - compile time impact due to changes to Swift client
API signatures. No change to persistent data or API server XPC
protocols.
- Don't prefix match localhost, use exact
match only.
- Only check default local domain if one is
defined.
- Use IPv4Address for private IP CIDR tests.
- Closes#1450.
- If we require that bundle-packaged installs must populate a
Unix-structured `bin` and `libexec`, we don't need the code that
searches the app bundle install root. As it stands, we enumerate both
the app bundle and the Linux paths, meaning we're looking at the same
plugins (in different paths) twice.
- Fixes single file mounts. Closes#1251.
- Adds journal modes to `EXT4.Formatter`.
- vminitd fix for client timeouts resulting in EBADF and other minor
improvements.
- Reenable block size for EXT4 formatter, and other fixes.
- Update vminit client and server to grpc-swift-2.
- Closes#1433.
- Renames `env` parameter to `dynamicEnv` to differentiate these
special-case environment variables from the standard container
environment.
- Use `[String: String]` instead of `[String: String]?`.
- Default to `[:]` when an down-revision client calls bootstrap without
supplying `dynamicEnv`.
- Closes#1308.
- Applies dependency and code changes similar to
apple/containerization#578.
- Upgrades hawkeye to latest version.
- Update StderrLogHandler not to create a (non-Sendable) Swift time
formatter object for every log message.
- Part of #1404.
- COMPATIBILITY: Breaking change to client API. XPC protocol and
persistent data remain fully compatibile.
- Renames `ClientNetwork` to `NetworkClient`, aligning with
`ContainerClient`.
- Client uses instance methods instead of utility functions, with an
`init(serviceIdentifier:)` for the XPC service name. The value is
currently unused.
- Closes#1359.
- Create a ResourceLabels type and extract the label validation from
NetworkConfiguration into the new type.
- Create a base AppError type that is compatible with structured logging
and delegates message presentation to the error receiver.
- Define LabelError over AppError for label validation.
- Slightly reworks NetworkConfiguration entity migration code in
NetworksService.
- Nothing uses this function, and it does not perform secure extraction.
Clients should instead use the `ArchiveReader` from the
ContainerizationArchive library.
- Closes#656.
- The SandboxService automatically sets the DNS server as part of
bootstrap if no server is given. Setting it in the container config
means that the builder will fail if the system restarts and the network
subnet changes.
- Part of #1295.
- Needed for #1342.
- If APP_ROOT is defined, remove everything but kernel from data
directory before starting integration test.
- Adds `--enable-kernel-install` for system start on integration test.
- Closes#1268.
- The types we were using weren't very usable with Swift 6 structured
concurrency.
- Implements just the subset of records that we use.
- Use notImplemented instead of formatError for unknown record types.
- Use pure actor for LocalhostDNSHandler now that we have sendable
types.
- Use DNSName as key for table lookups in LocalhostDNSHandler and
HostTableResolver.
- Utilize dot-suffixed domain names everywhere in the lookup chain.
- Huge thanks to @manojmahapatra and @katiewasnothere for their
diligent and patient review :D
- Closes#1308.
- Applies dependency and code changes similar to
apple/containerization#578.
- Upgrades hawkeye to latest version.
- Update StderrLogHandler not to create a (non-Sendable) Swift time
formatter object for every log message.
- Closes#1270.
- The current test is a bit baroque. It tries to run `nc`, either to
`8.8.8.8:53` or a proxy server port if `HTTP_PROXY` is in the
environment. It seems that for an alpine image, a proxy-aware
`Dockerfile` and a `RUN apk install curl` directive is a more natural
test of "network access".
- This reverts commit 1942a399e3.
- This approach to getting variant into the runtime helper was fine for
proof of concept work but we need a persistent runtime config file so
that the variant is available for `container start`.
- Adds plugin variant selection to bootstrap() on the Container API
service in such a way that we can revert the change soon without
compatibility issues when we work out a more permanent approach, and
requires no persistent data migration.
- Restore lexical ordering on ManagementFlags (except `--runtime`, will
take care of that next PR).
- Add a couple plugin loader tests to improve coverage.
- Closes#1236.
- As with other resources, default to tabular output, with `--format
json` alternative.
- Also changes common workflow to collect logs even when test step times
out.
- Adds a a `--log-root` option to `swift system start`, propagating the
value as `CONTAINER_LOG_ROOT` to services for logging to files instead
of the OS log facility. This is not a "production" capability as it
neither merges nor rotates logs.
- Currently we don't collect logs on CI builds, and we don't have
permission to run the `log` command there. The PR adds `--log-root` to
the CI test phase, archives the results, and uploads the archive as an
artifact.
- Use FilePath from swift-system for the log root. Foundation URL is a
bit of a footgun for filesystem paths, so unless we identify a
showstopper, we should incrementally transition to this type everywhere
except where we really need network URLs.
- Output the hostname of the CI runner at the start of the test phase so
we can identify runner-specific issues where they exist.
- Fix formatting for log messages with multiple metadata items, and fix
unstructured messages on instances that weren't found using `grep -r
'log\.' Sources`.
- Adds command reference documentation for `--log-root`.
- Closes#1206.
- Closes#1185.
- Closes#507.
- Addresses existing log messages for #642.
- Nondeterministic CI errors are resulting from very slow launch times
for the first runtime helper, which causes ContainersService to be
locked for longer than our 20 sec timeout. Bumping the timeout to 60
seconds addresses this case for now.
- Since many log messages needed to be changed to troubleshoot the
issue, updated all log messages to use structured logging, and
implemented consistent entry/exit logging for all service operations.
- Added logging for ContainerService lock acquisition to help with
finding root cause for the slow service startup.
- Plumbed the `--debug` flag on both `container system start` and
`container system logs` so that the flag is actually useful.
- Updated the `install-init.sh` script so that can install in a custom
app root directory.
## Type of Change
- [x] Bug fix
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
## Motivation and Context
- CI build is failing because runners don't have an /etc/resolver
directory, causing the directory monitor to fail. This occurs while the
install-kernel make target is running, so it appears that kernel
download is failing when it's just that the API server is dying. Created
#1207 for the issue.
- Fixing the initial scan for the directory just moves the problem to
the filesystem watch loop, likely because we're not testing the result
of `open()` for errors.
- Right now the priority is getting CI running and PRs merged, so just
commenting out the realhost DNS server setup.
- Also seeing that under some conditions it can take quite a while for
launchd to start the helper for the default network (8 seconds or more).
With the 10 second health check timeout after API server registration,
this means that some CI runs can exhibit this failure mode. Added a
`--timeout` option to SystemStart and set a 60 second timeout for
install-kernel and integration Makefile targets.
- Fixed a bug where `--debug` was being placed in the wrong location in
the api server startup args.
- Disabled all network CLI tests due to container bootstrap errors when
trying to run the container immediately after creating the network. The
slow network helper launch could be the reason behind the failures that
drove us to serialize these tests. Filed #1206 for this issue.
## Testing
- [x] Tested locally
- [ ] Added/updated tests
- [ ] Added/updated docs
- Closes#1150.
- The problem seems to be that the selinux label attribute enforcement
doesn't play well with overlayfs.
- Solution is to set the `lsm` boot line argument. The new value
corresponds to what one would see in a Lima VM with Colima. The Kata
kernel we use doesn't provide any of those modules, so now if you run
`dmesg | grep -i lsm` in a container you will just see: `LSM:
initializing lsm=capability`
- Closes#1122.
- Adds placeholder ManagedResource and unit tests. Nothing is using
these yet.
- Adds system-defined resource labels for owning plugin and resource
role. The system discriminates the builtin network using role "builtin".
- Adds builtin role when creating builtin network at startup, and
ensures that a preexisting network with ID "default" gets updated with
the role label.
- Replace all network ID checks for "default" with the builtin role
check.
- Adds "builder" role to builder VM.
## Type of Change
- [ ] Bug fix
- [x] New feature
- [ ] Breaking change
- [ ] Documentation update
## Motivation and Context
Role and owner labels should make cross-cutting resource policy easier
to implement.
## Testing
- [x] Tested locally
- [x] Added/updated tests
- [ ] Added/updated docs
- Closes#1113.
- This is the newest we can do until we address #767.
- Slight change to PacketFilter error handling so unit tests work more
reliably.
- Try making CLINetworkTests serialized to see if parallel execution is
causing flakes.
- ContainerResources shouldn't need to know anything about CLI stuff.
- Move ExpressibleByArgument protocol conformance to an extension in the
package where it's needed.
J LoganandGitHub
