Closes#1225
Add a flag to signify that we'd like to run a minimal init process that
can reap zombie processes. The actual support for this is in the
Containerization library so the plumbing here is very simple.
- Adds a a `--log-root` option to `swift system start`, propagating the
value as `CONTAINER_LOG_ROOT` to services for logging to files instead
of the OS log facility. This is not a "production" capability as it
neither merges nor rotates logs.
- Currently we don't collect logs on CI builds, and we don't have
permission to run the `log` command there. The PR adds `--log-root` to
the CI test phase, archives the results, and uploads the archive as an
artifact.
- Use FilePath from swift-system for the log root. Foundation URL is a
bit of a footgun for filesystem paths, so unless we identify a
showstopper, we should incrementally transition to this type everywhere
except where we really need network URLs.
- Output the hostname of the CI runner at the start of the test phase so
we can identify runner-specific issues where they exist.
- Fix formatting for log messages with multiple metadata items, and fix
unstructured messages on instances that weren't found using `grep -r
'log\.' Sources`.
- Adds command reference documentation for `--log-root`.
- Closes#1206.
- Closes#1185.
- Closes#507.
- Addresses existing log messages for #642.
- Nondeterministic CI errors are resulting from very slow launch times
for the first runtime helper, which causes ContainersService to be
locked for longer than our 20 sec timeout. Bumping the timeout to 60
seconds addresses this case for now.
- Since many log messages needed to be changed to troubleshoot the
issue, updated all log messages to use structured logging, and
implemented consistent entry/exit logging for all service operations.
- Added logging for ContainerService lock acquisition to help with
finding root cause for the slow service startup.
- Plumbed the `--debug` flag on both `container system start` and
`container system logs` so that the flag is actually useful.
- Updated the `install-init.sh` script so that can install in a custom
app root directory.
## Type of Change
- [x] Bug fix
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
## Motivation and Context
- CI build is failing because runners don't have an /etc/resolver
directory, causing the directory monitor to fail. This occurs while the
install-kernel make target is running, so it appears that kernel
download is failing when it's just that the API server is dying. Created
#1207 for the issue.
- Fixing the initial scan for the directory just moves the problem to
the filesystem watch loop, likely because we're not testing the result
of `open()` for errors.
- Right now the priority is getting CI running and PRs merged, so just
commenting out the realhost DNS server setup.
- Also seeing that under some conditions it can take quite a while for
launchd to start the helper for the default network (8 seconds or more).
With the 10 second health check timeout after API server registration,
this means that some CI runs can exhibit this failure mode. Added a
`--timeout` option to SystemStart and set a 60 second timeout for
install-kernel and integration Makefile targets.
- Fixed a bug where `--debug` was being placed in the wrong location in
the api server startup args.
- Disabled all network CLI tests due to container bootstrap errors when
trying to run the container immediately after creating the network. The
slow network helper launch could be the reason behind the failures that
drove us to serialize these tests. Filed #1206 for this issue.
## Testing
- [x] Tested locally
- [ ] Added/updated tests
- [ ] Added/updated docs
Send the hash of entire tar file in the first BuildTransfer packet to
prevent container-builder-shim from using stale cached contents.
This PR resolves#1143.
This PR relies on apple/container-builder-shim#64.
## Type of Change
- [X] Bug fix
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
## Motivation and Context
Current container-builder-shim uses only first few bytes of tar file as
checksum, which leads to the usage of stale cached contents if the
change of build context is not included in the first bytes of tar file.
## Testing
- [X] Tested locally
- [ ] Added/updated tests
- [ ] Added/updated docs
---------
Co-authored-by: Ronit Sabhaya <ronitsabhaya75@gmail.com>
Co-authored-by: J Logan <john_logan@apple.com>
Co-authored-by: saehejkang <saehej.kang@gmail.com>
Co-authored-by: Anthony DePasquale <anthony@depasquale.org>
- Bump `containerization` to `0.25.0`
- Updates for parameter changes on containerization
registry access API.
- Updates for change to containerization
`cleanUpOrphanedBlobs` function.
- Closes#1113.
- This is the newest we can do until we address #767.
- Slight change to PacketFilter error handling so unit tests work more
reliably.
- Try making CLINetworkTests serialized to see if parallel execution is
causing flakes.
- Refer to #862
- Updated `SIZE` field to `FULL SIZE`, as it seemed more appropriate so
it does not get mixed up with the `descriptor size` field
- Closes#860
- Fixed#892.
- By contrast with `rm`, `prune` should display
the amount of reclaimed storage, so added code
to retrieve it.
Signed-off-by: ChengHao Yang <17496418+tico88612@users.noreply.github.com>
- Closes#977.
- Closes#1058.
- Prevents unexpected removal of containers on
bootstrapping and starting failures, by reorganizing
error handling for container `run`, `start`, and
`exec` so that error handling only unwinds that
which was done in the current scope.
- Relies on apple/containerization#495.
- Closes#1037.
- Adds a `--mode` flag that has `nat` and `hostOnly` options.
The host-only option selects the vmnet host-only mode,
where containers attached to the network can reach each
other and the host, but not external systems.
- Runner fleet is on 26.3 now.
- Integration tests started flaking and it appears that we've been misconfiguring/not configuring proxy variables where we needed to be and it finally caught up with us. Workflow now adds appropriate exclusions for host-to-container and container-to-container network requests so they aren't all rammed through the proxy.
- Update image load and build to handle rejected paths during tar
extraction. For the image load command there is now a `--force` function
that fails extractions with rejected paths when false, and just warns
about the rejected paths when true.
- Update `container stats` for statistics API properties now all being
optional.
## Type of Change
- [x] Bug fix
- [ ] New feature
- [ ] Breaking change
- [x] Documentation update
## Motivation and Context
See above
## Testing
- [x] Tested locally
- [x] Added/updated tests
- [x] Added/updated docs
- Closes#639.
- Adds swift format configuration that removes lint checks so we can use
`swift lint` to perform format-only tests.
- Adds `check` target that invokes format and header checks.
- Adds pre-commit script that runs `make check`.
- Adds `pre-commit` target that installs the check script as a
pre-commit hook.
## Type of Change
- [ ] Bug fix
- [x] New feature
- [ ] Breaking change
- [x] Documentation update
## Motivation and Context
Avoids wasting time and commit rewrites.
## Testing
- [x] Tested locally
- [ ] Added/updated tests
- [x] Added/updated docs
- TestCLIRunCommand now run so many tests concurrently that the API
server gets swamped and tests randomly time out.
- The parallelism options on `swift test` only work for XCTest, not
swift-testing.
- Work around this while retaining some parallelism (good for stress
testing) by breaking the tests into two suites.
- The container fails to start with a generic "permission denied"
error when attempting to publish privileged ports (ports below
1024) without root privileges. This provides a confusing user
experience as the error doesn't explain why permission was
denied.
- Closes#1005.
- Adapt everything to use MACAddress type from containerization 0.20.0.
- Allocate MAC addresses for every container so that we have
deterministic IPv6 link local addresses.
- Add AAAA handling to ContainerDNSHandler.
- NOTE: Only works on Tahoe. On Sequoia, we don't have a good way to set
or determine the IPv6 network prefix when networks are created, so we
can't infer the IPv6 link local addresses for AAAA responses and we
instead return `NODATA`.
- Closes#461.
- Extract core types into ContainerResources target.
- Extract ContainerNetworkServiceClient from ContainerNetworkService.
- Relocate sandbox client from ContainerClient to
ContainerSandboxServiceClient.
- Relocate ContainerClient to ContainerAPIServiceClient.
- Common structure from services and clients under Source/Services.
Updated project hierarchy:
```
Sources/CAuditToken - audit token access wrapper
Sources/CLI - CLI executable
Sources/ContainerBuild - builder
Sources/ContainerCommands - CLI command implementations
Sources/ContainerLog - logging helpers
Sources/ContainerPersistence - persistent data and system property helpers
Sources/ContainerPlugin - plugin system
Sources/ContainerResource - resource (container, image, volume, network) types
Sources/ContainerVersion - version helpers
Sources/ContainerXPC - XPC helpers
Sources/CVersion - injected project version
Sources/DNSServer - container DNS resolver
Sources/Helpers - service executables
Sources/Services/*/Client - service clients
Sources/Services/*/Server - service implementations
Sources/SocketForwarder - port forwarding
Sources/TerminalProgress - progress bar
```
## Type of Change
- [ ] Bug fix
- [ ] New feature
- [x] Breaking change
- [ ] Documentation update
## Motivation and Context
The ContainerClient library was a bit of a grab bag. This refactor
applies a more sensible project and library structure for resource data
types, services, and clients.
## Testing
- [x] Tested locally
- [x] Added/updated tests
- [ ] Added/updated docs
This is a fix for
[issue#956](https://github.com/apple/container/issues/956)
`FileManager.default.contents(atPath:)` returns `nil` for named pipes
(FIFOs)
and process substitutions like `/dev/fd/XX` because:
1. It expects regular files with a known size
2. Named pipes are stream-based and block until data arrives
## Solution
Use `FileHandle(forReadingFrom:)` instead, which:
- Properly handles blocking I/O
- Works with named pipes, process substitutions, and regular files
(mentioned in the
[doc](https://developer.apple.com/documentation/foundation/filehandle))
Co-authored-by: Bortniak Volodymyr <Bortnyak@users.noreply.github.com>
- Part of work for #460.
- With CZ release 0.17.0, the IP and CIDR address
types changed from String to IPv4Address and
CIDRv4, respectively. This PR applies the corresponding
adaptations to container.
- Updates to `image prune` for consistency with how
other `prune` commands are done. Added missing
test cases as well for the command
- Relates to the discussion from #914
- Refactor the `volume prune` command to follow a client-side approach.
The `volumeDiskUsage` is calculated in the service file, so it made
sense to leave that there.
- Relates to the discussion from #914
- Fixes#943.
- Use images other than alpine:3.20 for image concurrency test so as not
to interfere with tests using that image.
- Rename test files to match suite names.
Seems `run` was changed to return a data blob variant of stdout, which
made the function have a tuple of size 4 instead of 3 now. The stats
tests still used the old 3 wide variant which broke the build.
Closes#824
This implements statistics gathering across the various components, but
ultimately this is for implementing a new CLI command: `container
stats`. This shows memory usage, cpu usage, network and block i/o and
the number of processes in the container. The new command can inspect
stats for 1-N containers and by default continuously updates in a `top`
like stream.
## Type of Change
- [ ] Bug fix
- [x] New feature
- [ ] Breaking change
- [ ] Documentation update
## Testing
- [x] Tested locally
- [x] Added/updated tests
- [x] Added/updated docs
When working on test cases for #827 and #734 the following issues
occurred
1. There was no support to pass in `Data` for `stdin` (input pipe added
+ new argument for the data)
2. Pipes were getting blocked during the `try process.run()` and
`process.waitUntilExit()` (order of operations were fixed)
3. We need the binary data for directly piping the `stdout` data to
`stdin`, and converting the data to a `string`, returns `""` (added a
new value to the tuple)
I opened a new PR here for this as I did not want to bloat the other PRs
with so many updates. At first I thought of creating a new function
named `runStdinStdout`, but that seemed redundant as it pretty much was
going to do the same thing as `run`.
## Testing
- [x] Tested locally
- [x] Added/updated tests
- [ ] Added/updated docs
---------
Co-authored-by: J Logan <john_logan@apple.com>
- Fixes#736.
- BuildKit returns nil ref for Dockerfiles containing only metadata
(`ENV/ARG/LABEL`) directives without filesystem operations
(`RUN/COPY/ADD`). Previously, this caused builds to fail with "no build
directives" error.
- Builder-shim 0.6.3 [fixes this](https://github.com/apple/container-builder-shim/pull/47) by
creating a minimal marker layer when ref is nil but image config is
valid, satisfying OCI manifest requirements.
- Also, added some tests for this behavior.
Danny Canter
J Logan
jwhur
ChengHao Yang
Manu Schiller
Saehej Kang
Amir Alperin
Kathryn Baldauf
Volodymyr BortniakandBortniak Volodymyr
karen heckel
wangxiaoleiandfatelei
Santosh BhavaniandClaude
caztanj
Raj