Also cleans up the target in rpm and deb Makfiles because
the tar target already output a tar.
Signed-off-by: Jose Bigio <jose.bigio@docker.com>
(cherry picked from commit 8a4744f48aa13bea0b4e9174165be4add0a5a42f)
Signed-off-by: Jose Bigio <jose.bigio@docker.com>
Upstream-commit: 8e4a92dc667e2143b6d5707e19c5564b7da46ba1
Component: packaging
… this removes a whole lot of dependencies from people depending on docker/cli…
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
(cherry picked from commit 2d344b2f61)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: fc3dc8f058
Component: cli
… this is, for now, the only platform that is supported
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
(cherry picked from commit a3a955f204)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 2a46a3d46c
Component: cli
This will make it easier for anybody to build with a specific version of a distro they want to build on.
They could lock into a specific sha of a docker image if they wanted, as an example.
Signed-off-by: corbin-coleman <corbin.coleman@docker.com>
Upstream-commit: 7afc35b95ac46bfef383788341ddb0b157af14eb
Component: packaging
This should accomodate the new versioning fix
Signed-off-by: Daniel Hiltgen <daniel.hiltgen@docker.com>
Upstream-commit: be00095e1d0d5674665a63b6e5932178632feb0e
Component: packaging
Removes the need for the offline installer to install the shim process
and instead installs the shim process as part of the packaging.
May be easier in the future to just package the shim process on it's own
but that'll come after this 18.09 release
Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
(cherry picked from commit f8bd366d58f8bdf8a82b9a033353ca5bf4eda948)
Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
Upstream-commit: 31d0cb047b98ab43f661bd026bdd63deef62543d
Component: packaging
This should test that
- all the messages produced are delivered (i.e. not lost)
- followLogs() exits
Loosely based on the test having the same name by Brian Goff, see
https://gist.github.com/cpuguy83/e538793de18c762608358ee0eaddc197
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit f845d76d047760c91dc0c7076aea840291fbdbc5)
Upstream-commit: 2a82480df9ad91593d59be4b5283917dbea2da39
Component: engine
When daemon.ContainerLogs() is called with options.follow=true
(as in "docker logs --follow"), the "loggerutils.followLogs()"
function never returns (even then the logs consumer is gone).
As a result, all the resources associated with it (including
an opened file descriptor for the log file being read, two FDs
for a pipe, and two FDs for inotify watch) are never released.
If this is repeated (such as by running "docker logs --follow"
and pressing Ctrl-C a few times), this results in DoS caused by
either hitting the limit of inotify watches, or the limit of
opened files. The only cure is daemon restart.
Apparently, what happens is:
1. logs producer (a container) is gone, calling (*LogWatcher).Close()
for all its readers (daemon/logger/jsonfilelog/jsonfilelog.go:175).
2. WatchClose() is properly handled by a dedicated goroutine in
followLogs(), cancelling the context.
3. Upon receiving the ctx.Done(), the code in followLogs()
(daemon/logger/loggerutils/logfile.go#L626-L638) keeps to
send messages _synchronously_ (which is OK for now).
4. Logs consumer is gone (Ctrl-C is pressed on a terminal running
"docker logs --follow"). Method (*LogWatcher).Close() is properly
called (see daemon/logs.go:114). Since it was called before and
due to to once.Do(), nothing happens (which is kinda good, as
otherwise it will panic on closing a closed channel).
5. A goroutine (see item 3 above) keeps sending log messages
synchronously to the logWatcher.Msg channel. Since the
channel reader is gone, the channel send operation blocks forever,
and resource cleanup set up in defer statements at the beginning
of followLogs() never happens.
Alas, the fix is somewhat complicated:
1. Distinguish between close from logs producer and logs consumer.
To that effect,
- yet another channel is added to LogWatcher();
- {Watch,}Close() are renamed to {Watch,}ProducerGone();
- {Watch,}ConsumerGone() are added;
*NOTE* that ProducerGone()/WatchProducerGone() pair is ONLY needed
in order to stop ConsumerLogs(follow=true) when a container is stopped;
otherwise we're not interested in it. In other words, we're only
using it in followLogs().
2. Code that was doing (logWatcher*).Close() is modified to either call
ProducerGone() or ConsumerGone(), depending on the context.
3. Code that was waiting for WatchClose() is modified to wait for
either ConsumerGone() or ProducerGone(), or both, depending on the
context.
4. followLogs() are modified accordingly:
- context cancellation is happening on WatchProducerGone(),
and once it's received the FileWatcher is closed and waitRead()
returns errDone on EOF (i.e. log rotation handling logic is disabled);
- due to this, code that was writing synchronously to logWatcher.Msg
can be and is removed as the code above it handles this case;
- function returns once ConsumerGone is received, freeing all the
resources -- this is the bugfix itself.
While at it,
1. Let's also remove the ctx usage to simplify the code a bit.
It was introduced by commit a69a59ffc7e3d ("Decouple removing the
fileWatcher from reading") in order to fix a bug. The bug was actually
a deadlock in fsnotify, and the fix was just a workaround. Since then
the fsnofify bug has been fixed, and a new fsnotify was vendored in.
For more details, please see
https://github.com/moby/moby/pull/27782#issuecomment-416794490
2. Since `(*filePoller).Close()` is fixed to remove all the files
being watched, there is no need to explicitly call
fileWatcher.Remove(name) anymore, so get rid of the extra code.
Should fix https://github.com/moby/moby/issues/37391
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 916eabd459fe707b5c4a86377d12e2ad1871b353)
Upstream-commit: 84a5b528aede5579861201e869870d10fc98c07c
Component: engine
This test case checks that followLogs() exits once the reader is gone.
Currently it does not (i.e. this test is supposed to fail) due to #37391.
[kolyshkin@: test case Brian Goff, changelog and all bugs are by me]
Source: https://gist.github.com/cpuguy83/e538793de18c762608358ee0eaddc197
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit d37a11bfbab83ab42b1160f116e863daac046192)
Upstream-commit: 511741735e0aa2fe68a66d99384c00d187d1a157
Component: engine
This code has many return statements, for some of them the
"end logs" or "end stream" message was not printed, giving
the impression that this "for" loop never ended.
Make sure that "begin logs" is to be followed by "end logs".
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 2e4c2a6bf9cb47fd07e42f9c043024ed3dbcd04d)
Upstream-commit: 2b8bc86679b7153bb4ace063a858637df0f16a2e
Component: engine
The code in Close() that removes the watches was not working,
because it first sets `w.closed = true` and then calls w.close(),
which starts with
```
if w.closed {
return errPollerClosed
}
```
Fix by setting w.closed only after calling w.remove() for all the
files being watched.
While at it, remove the duplicated `delete(w.watches, name)` code.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit fffa8958d00860b4e3563327a2cc6836a12d4ba9)
Upstream-commit: 4e2dbfa1af48191126b0910b9463bf94d8371886
Component: engine
There is no need to wait for up to 200ms in order to close
the file descriptor once the chClose is received.
This commit might reduce the chances for occasional "The process
cannot access the file because it is being used by another process"
error on Windows, where an opened file can't be removed.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit dfbb64ea7d042d5b2bb0c1c2b88e3682b7069b10)
Upstream-commit: 3a3bfcbf47e98212abfc9cfed860d9e99fc41cdc
Component: engine
In case of errors, the file descriptor is never closed. Fix it.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 88bcf1573ca2eaffc15da346a1651a3749567554)
Upstream-commit: 7be43586af6824c1e55cb502d9d2bab45c9b4505
Component: engine
To include https://github.com/vbatts/tar-split/pull/48 which
fixes the issue of creating an image with >8GB file in it.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 92e75439037d205c218208945c70cfb3633e87aa)
Upstream-commit: d7085abec2e445630bedd3e79782c5ec33f62682
Component: engine
Add a test case for creating a 8GB file inside a container.
Due to a bug in tar-split this was failing in Docker 18.06.
The file being created is sparse, so there's not much I/O
happening or disk space being used -- meaning the test is
fast and does not require a lot of disk space.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit b3165f5b2d7c1063c2a76a7ed5c2dd18f868276b)
Upstream-commit: fc1d808c44f77e3929c4eeba7066501890aecd4e
Component: engine
docker-unir[bot]
GordonTheTurtle
Silvin Lubecki
Jose Bigio
Vincent Demeester
corbin-coleman
Eli Uriegas
Daniel Hiltgen
Eli Uriegas
Kir Kolyshkin
Brian Goff
Andrew Hsu
Michael Crosby