docker_ubuntu_18_04 TestFunctional/parallel/MySQL: CrashLoopBackOff

[medyagh]

for docker drivers

=== RUN   TestFunctional/parallel/MySQL
=== PAUSE TestFunctional/parallel/MySQL
=== CONT  TestFunctional/parallel/MySQL
TestFunctional/parallel/MySQL: functional_test.go:719: (dbg) Run:  kubectl --context functional-20200403T003626.019269334-5539 replace --force -f testdata/mysql.yaml
TestFunctional/parallel/MySQL: functional_test.go:724: (dbg) TestFunctional/parallel/MySQL: waiting 10m0s for pods matching "app=mysql" in namespace "default" ...
TestFunctional/parallel/MySQL: helpers.go:289: "mysql-78ff7d6cf9-j9p8v" [81bfa3a5-9f2a-4cbd-9227-a0b0c6cf272a] Pending / Ready:ContainersNotReady (containers with unready status: [mysql]) / ContainersReady:ContainersNotReady (containers with unready status: [mysql])
TestDockerFlags: docker_test.go:40: (dbg) Done: ./minikube-linux-amd64 start -p docker-flags-20200403T010036.924791143-5539 --cache-images=false --memory=1800 --install-addons=false --wait=false --docker-env=FOO=BAR --docker-env=BAZ=BAT --docker-opt=debug --docker-opt=icc=true --alsologtostderr -v=5 --driver=docker: (54.403443947s)
TestDockerFlags: docker_test.go:45: (dbg) Run:  ./minikube-linux-amd64 -p docker-flags-20200403T010036.924791143-5539 ssh "sudo systemctl show docker --property=Environment --no-pager"
TestDockerFlags: docker_test.go:56: (dbg) Run:  ./minikube-linux-amd64 -p docker-flags-20200403T010036.924791143-5539 ssh "sudo systemctl show docker --property=ExecStart --no-pager"
TestDockerFlags: helpers.go:170: (dbg) Run:  ./minikube-linux-amd64 delete -p docker-flags-20200403T010036.924791143-5539
TestDockerFlags: helpers.go:170: (dbg) Done: ./minikube-linux-amd64 delete -p docker-flags-20200403T010036.924791143-5539: (6.863687419s)
--- FAIL: TestFunctional/parallel/MySQL (281.46s)

I feel like this could be fixed by giving it a bit more time.

[tstromberg]

I noticed this bug was still open, so I've commandeered it for what may be the same issue, but seen differently with the longer timeout:

Apr 08 16:37:47 functional-20200408T155430.596471605-5413 kubelet[2026]: E0408 16:37:47.928613 2026 pod_workers.go:191] Error syncing pod fa3c80b8-f828-447e-a4f5-a9b5aad8edd4 ("mysql-78ff7d6cf9-kzxxv_default(fa3c80b8-f828-447e-a4f5-a9b5aad8edd4)"), skipping: failed to "StartContainer" for "mysql" with CrashLoopBackOff: "back-off 2m40s restarting failed container=mysql pod=mysql-78ff7d6cf9-kzxxv_default(fa3c80b8-f828-447e-a4f5-a9b5aad8edd4)

        	NAMESPACE              NAME                                                                READY   STATUS             RESTARTS   AGE     LABELS
        	default                busybox                                                             1/1     Running            0          4m27s   integration-test=busybox
        	default                busybox-mount                                                       0/1     Completed          0          4m34s   integration-test=busybox-mount
        	default                hello-node-677b9cfc6b-5ml4r                                         1/1     Running            0          5m6s    app=hello-node,pod-template-hash=677b9cfc6b
        	default                mysql-78ff7d6cf9-kzxxv                                              0/1     CrashLoopBackOff   5          5m54s   app=mysql,pod-template-hash=78ff7d6cf9
        	default                nginx-svc                                                           1/1     Running            0          5m23s   run=nginx-svc
        	kube-system            coredns-66bff467f8-thct4                                            1/1     Running            1          42m     k8s-app=kube-dns,pod-template-hash=66bff467f8
        	kube-system            coredns-66bff467f8-tzs57                                            1/1     Running            1          42m     k8s-app=kube-dns,pod-template-hash=66bff467f8
        	kube-system            etcd-functional-20200408t155430.596471605-5413                      1/1     Running            2          42m     component=etcd,tier=control-plane
        	kube-system            kube-apiserver-functional-20200408t155430.596471605-5413            1/1     Running            2          42m     component=kube-apiserver,tier=control-plane
        	kube-system            kube-controller-manager-functional-20200408t155430.596471605-5413   1/1     Running            2          42m     component=kube-controller-manager,tier=control-plane
        	kube-system            kube-proxy-9p9kz                                                    1/1     Running            1          42m     controller-revision-hash=c8bb659c5,k8s-app=kube-proxy,pod-template-generation=1
        	kube-system            kube-scheduler-functional-20200408t155430.596471605-5413            1/1     Running            2          42m     component=kube-scheduler,tier=control-plane
        	kube-system            storage-provisioner                                                 1/1     Running            3          42m     addonmanager.kubernetes.io/mode=Reconcile,integration-test=storage-provisioner
        	kubernetes-dashboard   dashboard-metrics-scraper-84bfdf55ff-q7hmb                          1/1     Running            0          4m19s   k8s-app=dashboard-metrics-scraper,pod-template-hash=84bfdf55ff
        	kubernetes-dashboard   kubernetes-dashboard-bc446cc64-k4hsf                                1/1     Running            0          4m18s   k8s-app=kubernetes-dashboard,pod-template-hash=bc446cc64

[tstromberg]

This still happens - we should add kubectl logs to the integration test helper to isolate what is going on.

[govargo]

This still happens. https://github.com/kubernetes/minikube/runs/595876638#step:7:419

I found test helper already has func for pods'log. I'll use it

/assign

[govargo]

https://github.com/kubernetes/minikube/runs/597343929#step:7:441

In result of e2e, MySQL's crashLoop's cause is Can't read dir of '/etc/mysql/conf.d/' (Errcode: 13 - Permission denied).

    TestFunctional/parallel/MySQL: functional_test.go:778: (dbg) kubectl --context functional-20200418054831-10280 logs mysql-78ff7d6cf9-rnchq -n default:
        2020-04-18 05:54:21+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.6.47-1debian9 started.
        2020-04-18 05:54:21+00:00 [ERROR] [Entrypoint]: mysqld failed while attempting to check config
        	command was: mysqld --verbose --help
        	�mysqld: Can't read dir of '/etc/mysql/conf.d/' (Errcode: 13 - Permission denied)
        Fatal error in defaults handling. Program aborted
    TestFunctional/parallel/MySQL: functional_test.go:779: failed to exec 'mysql -ppassword -e show databases;': exit status 1

[govargo]

============================ directory ============================

root@mysql-78ff7d6cf9-9k5zb:/# ls -la etc/mysql/conf.d/
total 20
drwxr-xr-x 2 root root 4096 Apr 17 16:58 .
drwxr-xr-x 4 root root 4096 Apr 17 16:58 ..
-rw-r--r-- 1 root root   43 Apr 17 16:58 docker.cnf
-rw-r--r-- 1 root root    8 Jul  9  2016 mysql.cnf
-rw-r--r-- 1 root root   55 Jul  9  2016 mysqldump.cnf

============================ entrypoint.sh ============================

mysql_check_config() {
        local toRun=( "$@" --verbose --help ) errors
        if ! errors="$("${toRun[@]}" 2>&1 >/dev/null)"; then
                mysql_error $'mysqld failed while attempting to check config\n\tcommand was: '"${toRun[*]}"$'\n\t'"$errors"
        fi
}

_main() {
        # if command starts with an option, prepend mysqld
        if [ "${1:0:1}" = '-' ]; then
                set -- mysqld "$@"
        fi

        # skip setup if they aren't running mysqld or want an option that stops mysqld
        if [ "$1" = 'mysqld' ] && ! _mysql_want_help "$@"; then
                mysql_note "Entrypoint script for MySQL Server ${MYSQL_VERSION} started."

                mysql_check_config "$@"       ⇦ this problem is happening here

[govargo]

I cannot understand why mysql can't read dir of '/etc/mysql/conf.d/ because it's container user is root.

Anyway I'll investigate more in my PR(#7761).

[govargo]

I found this docker mysql issue. moby/moby#7512

This is caused by AppArmor on Ubuntu(AppArmor is like selinux).
Apparmor restricts docker privileged mysql.

I'd like to try to disable AppArmor for mysql.
moby/moby#7512 (comment)

Could I add to .github/workflows/main.yml? @tstromberg @medyagh

sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld

If it's OK, I'll add above to #7761.