lxc containers hangs after upgrade to 13.10
- by doug123
I have 3 lxc containers. They were all working fine on 12.10 and I upgraded the containers with do-release-upgrade on the containers to 13.04 and 13.10 and that worked great. Then I upgraded the host to 13.04 and then 13.10 and now the 3 containers hang with this:
>lxc-start -n as1 -l DEBUG -o $(tty)
lxc-start 1383145786.513 INFO lxc_start_ui - using rcfile /var/lib/lxc/as1/config
lxc-start 1383145786.513 WARN lxc_log - lxc_log_init called with log already initialized
lxc-start 1383145786.513 INFO lxc_apparmor - aa_enabled set to 1
lxc-start 1383145786.514 DEBUG lxc_conf - allocated pty '/dev/pts/2' (5/6)
lxc-start 1383145786.514 DEBUG lxc_conf - allocated pty '/dev/pts/13' (7/8)
lxc-start 1383145786.514 DEBUG lxc_conf - allocated pty '/dev/pts/14' (9/10)
lxc-start 1383145786.514 DEBUG lxc_conf - allocated pty '/dev/pts/15' (11/12)
lxc-start 1383145786.514 DEBUG lxc_conf - allocated pty '/dev/pts/17' (13/14)
lxc-start 1383145786.514 DEBUG lxc_conf - allocated pty '/dev/pts/18' (15/16)
lxc-start 1383145786.514 DEBUG lxc_conf - allocated pty '/dev/pts/19' (17/18)
lxc-start 1383145786.514 DEBUG lxc_conf - allocated pty '/dev/pts/20' (19/20)
lxc-start 1383145786.514 INFO lxc_conf - tty's configured
lxc-start 1383145786.514 DEBUG lxc_start - sigchild handler set
lxc-start 1383145786.514 DEBUG lxc_console - opening /dev/tty for console peer
lxc-start 1383145786.514 DEBUG lxc_console - using '/dev/tty' as console
lxc-start 1383145786.514 DEBUG lxc_console - 6242 got SIGWINCH fd 25
lxc-start 1383145786.514 DEBUG lxc_console - set winsz dstfd:22 cols:177 rows:53
lxc-start 1383145786.514 INFO lxc_start - 'as1' is initialized
lxc-start 1383145786.522 DEBUG lxc_start - Not dropping cap_sys_boot or watching utmp
lxc-start 1383145786.524 DEBUG lxc_conf - mac address of host interface 'vethB4L35W' changed to private fe:7c:96:a0:ae:29
lxc-start 1383145786.525 DEBUG lxc_conf - instanciated veth 'vethB4L35W/vethVC61K2', index is '26'
lxc-start 1383145786.529 DEBUG lxc_cgroup - cgroup 'memory.limit_in_bytes' set to '20G'
lxc-start 1383145786.529 DEBUG lxc_cgroup - cgroup 'cpuset.cpus' set to '12-23'
lxc-start 1383145786.529 INFO lxc_cgroup - cgroup has been setup
lxc-start 1383145786.555 DEBUG lxc_conf - move 'eth0' to '6249'
lxc-start 1383145786.555 INFO lxc_conf - 'as1' hostname has been setup
lxc-start 1383145786.575 DEBUG lxc_conf - 'eth0' has been setup
lxc-start 1383145786.575 INFO lxc_conf - network has been setup
lxc-start 1383145786.575 INFO lxc_conf - looking at .44 42 252:0 / / rw,relatime - ext4 /dev/mapper/limitorderbook1-root rw,errors=remount-ro,data=ordered
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /.
lxc-start 1383145786.575 INFO lxc_conf - looking at .52 44 0:5 / /dev rw,relatime - devtmpfs udev rw,size=32961632k,nr_inodes=8240408,mode=755
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /dev.
lxc-start 1383145786.575 INFO lxc_conf - looking at .61 52 0:11 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,mode=600,ptmxmode=000
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /dev/pts.
lxc-start 1383145786.575 INFO lxc_conf - looking at .68 44 0:15 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=6594456k,mode=755
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /run.
lxc-start 1383145786.575 INFO lxc_conf - looking at .69 68 0:18 / /run/lock rw,nosuid,nodev,noexec,relatime - tmpfs none rw,size=5120k
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /run/lock.
lxc-start 1383145786.575 INFO lxc_conf - looking at .72 68 0:19 / /run/shm rw,nosuid,nodev,relatime - tmpfs none rw
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /run/shm.
lxc-start 1383145786.575 INFO lxc_conf - looking at .73 68 0:21 / /run/user rw,nosuid,nodev,noexec,relatime - tmpfs none rw,size=102400k,mode=755
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /run/user.
lxc-start 1383145786.575 INFO lxc_conf - looking at .76 44 0:14 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /sys.
lxc-start 1383145786.575 INFO lxc_conf - looking at .77 76 0:16 / /sys/fs/cgroup rw,relatime - tmpfs none rw,size=4k,mode=755
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /sys/fs/cgroup.
lxc-start 1383145786.575 INFO lxc_conf - looking at .78 77 0:20 / /sys/fs/cgroup/cpuset rw,relatime - cgroup cgroup rw,cpuset,clone_children
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /sys/fs/cgroup/cpuset.
lxc-start 1383145786.575 INFO lxc_conf - looking at .79 77 0:23 / /sys/fs/cgroup/cpu rw,relatime - cgroup cgroup rw,cpu
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /sys/fs/cgroup/cpu.
lxc-start 1383145786.575 INFO lxc_conf - looking at .80 77 0:24 / /sys/fs/cgroup/cpuacct rw,relatime - cgroup cgroup rw,cpuacct
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /sys/fs/cgroup/cpuacct.
lxc-start 1383145786.575 INFO lxc_conf - looking at .81 77 0:25 / /sys/fs/cgroup/memory rw,relatime - cgroup cgroup rw,memory
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /sys/fs/cgroup/memory.
lxc-start 1383145786.575 INFO lxc_conf - looking at .82 77 0:26 / /sys/fs/cgroup/devices rw,relatime - cgroup cgroup rw,devices
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /sys/fs/cgroup/devices.
lxc-start 1383145786.575 INFO lxc_conf - looking at .83 77 0:27 / /sys/fs/cgroup/freezer rw,relatime - cgroup cgroup rw,freezer
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /sys/fs/cgroup/freezer.
lxc-start 1383145786.575 INFO lxc_conf - looking at .84 77 0:28 / /sys/fs/cgroup/blkio rw,relatime - cgroup cgroup rw,blkio
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /sys/fs/cgroup/blkio.
lxc-start 1383145786.575 INFO lxc_conf - looking at .85 77 0:29 / /sys/fs/cgroup/perf_event rw,relatime - cgroup cgroup rw,perf_event
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /sys/fs/cgroup/perf_event.
lxc-start 1383145786.575 INFO lxc_conf - looking at .94 77 0:30 / /sys/fs/cgroup/hugetlb rw,relatime - cgroup cgroup rw,hugetlb
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /sys/fs/cgroup/hugetlb.
lxc-start 1383145786.575 INFO lxc_conf - looking at .95 77 0:31 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime - cgroup systemd rw,name=systemd
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /sys/fs/cgroup/systemd.
lxc-start 1383145786.575 INFO lxc_conf - looking at .96 76 0:17 / /sys/fs/fuse/connections rw,relatime - fusectl none rw
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /sys/fs/fuse/connections.
lxc-start 1383145786.575 INFO lxc_conf - looking at .98 76 0:6 / /sys/kernel/debug rw,relatime - debugfs none rw
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /sys/kernel/debug.
lxc-start 1383145786.575 INFO lxc_conf - looking at .101 76 0:10 / /sys/kernel/security rw,relatime - securityfs none rw
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /sys/kernel/security.
lxc-start 1383145786.575 INFO lxc_conf - looking at .102 76 0:22 / /sys/fs/pstore rw,relatime - pstore none rw
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /sys/fs/pstore.
lxc-start 1383145786.575 INFO lxc_conf - looking at .103 44 0:3 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /proc.
lxc-start 1383145786.575 INFO lxc_conf - looking at .104 44 9:2 / /data rw,relatime - ext4 /dev/md2 rw,errors=remount-ro,data=ordered
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /data.
lxc-start 1383145786.575 INFO lxc_conf - looking at .105 44 8:1 / /boot rw,relatime - ext2 /dev/sda1 rw,errors=continue
.
lxc-start 1383145786.575 INFO lxc_conf - now p is . /boot.
lxc-start 1383145786.576 DEBUG lxc_conf - mounted '/data/srv/lxc/as1' on '/usr/lib/x86_64-linux-gnu/lxc'
lxc-start 1383145786.576 DEBUG lxc_conf - mounted 'none' on '/usr/lib/x86_64-linux-gnu/lxc//dev/pts', type 'devpts'
lxc-start 1383145786.576 DEBUG lxc_conf - mounted 'none' on '/usr/lib/x86_64-linux-gnu/lxc//proc', type 'proc'
lxc-start 1383145786.576 DEBUG lxc_conf - mounted 'none' on '/usr/lib/x86_64-linux-gnu/lxc//sys', type 'sysfs'
lxc-start 1383145786.576 DEBUG lxc_conf - mounted 'none' on '/usr/lib/x86_64-linux-gnu/lxc//run', type 'tmpfs'
lxc-start 1383145786.576 INFO lxc_conf - mount points have been setup
lxc-start 1383145786.577 INFO lxc_conf - console has been setup
lxc-start 1383145786.577 INFO lxc_conf - 8 tty(s) has been setup
lxc-start 1383145786.577 INFO lxc_conf - rootfs path is ./data/srv/lxc/as1., mount is ./usr/lib/x86_64-linux-gnu/lxc.
lxc-start 1383145786.577 INFO lxc_apparmor - I am 1, /proc/self points to 1
lxc-start 1383145786.577 DEBUG lxc_conf - created '/usr/lib/x86_64-linux-gnu/lxc/lxc_putold' directory
lxc-start 1383145786.577 DEBUG lxc_conf - mountpoint for old rootfs is '/usr/lib/x86_64-linux-gnu/lxc/lxc_putold'
lxc-start 1383145786.577 DEBUG lxc_conf - pivot_root syscall to '/usr/lib/x86_64-linux-gnu/lxc' successful
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold/dev/pts'
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold/run/lock'
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold/run/shm'
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold/run/user'
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold/sys/fs/cgroup/cpuset'
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold/sys/fs/cgroup/cpu'
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold/sys/fs/cgroup/cpuacct'
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold/sys/fs/cgroup/memory'
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold/sys/fs/cgroup/devices'
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold/sys/fs/cgroup/freezer'
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold/sys/fs/cgroup/blkio'
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold/sys/fs/cgroup/perf_event'
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold/sys/fs/cgroup/hugetlb'
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold/sys/fs/cgroup/systemd'
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold/sys/fs/fuse/connections'
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold/sys/kernel/debug'
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold/sys/kernel/security'
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold/sys/fs/pstore'
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold/proc'
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold/data'
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold/boot'
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold/dev'
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold/run'
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold/sys/fs/cgroup'
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold/sys'
lxc-start 1383145786.577 DEBUG lxc_conf - umounted '/lxc_putold'
lxc-start 1383145786.577 INFO lxc_conf - created new pts instance
lxc-start 1383145786.578 DEBUG lxc_conf - drop capability 'sys_boot' (22)
lxc-start 1383145786.578 DEBUG lxc_conf - capabilities have been setup
lxc-start 1383145786.578 NOTICE lxc_conf - 'as1' is setup.
lxc-start 1383145786.578 DEBUG lxc_cgroup - cgroup 'memory.limit_in_bytes' set to '20G'
lxc-start 1383145786.578 DEBUG lxc_cgroup - cgroup 'cpuset.cpus' set to '12-23'
lxc-start 1383145786.578 INFO lxc_cgroup - cgroup has been setup
lxc-start 1383145786.578 INFO lxc_apparmor - setting up apparmor
lxc-start 1383145786.578 INFO lxc_apparmor - changed apparmor profile to lxc-container-default
lxc-start 1383145786.578 NOTICE lxc_start - exec'ing '/sbin/init'
lxc-start 1383145786.578 INFO lxc_conf - looking at .15 20 0:14 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw
.
lxc-start 1383145786.578 INFO lxc_conf - now p is . /sys.
lxc-start 1383145786.578 INFO lxc_conf - looking at .16 20 0:3 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
.
lxc-start 1383145786.578 INFO lxc_conf - now p is . /proc.
lxc-start 1383145786.578 INFO lxc_conf - looking at .17 20 0:5 / /dev rw,relatime - devtmpfs udev rw,size=32961632k,nr_inodes=8240408,mode=755
.
lxc-start 1383145786.578 INFO lxc_conf - now p is . /dev.
lxc-start 1383145786.578 INFO lxc_conf - looking at .18 17 0:11 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,mode=600,ptmxmode=000
.
lxc-start 1383145786.578 INFO lxc_conf - now p is . /dev/pts.
lxc-start 1383145786.578 INFO lxc_conf - looking at .19 20 0:15 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=6594456k,mode=755
.
lxc-start 1383145786.578 INFO lxc_conf - now p is . /run.
lxc-start 1383145786.578 INFO lxc_conf - looking at .20 1 252:0 / / rw,relatime - ext4 /dev/mapper/limitorderbook1-root rw,errors=remount-ro,data=ordered
.
lxc-start 1383145786.578 INFO lxc_conf - now p is . /.
lxc-start 1383145786.578 INFO lxc_conf - looking at .22 15 0:16 / /sys/fs/cgroup rw,relatime - tmpfs none rw,size=4k,mode=755
.
lxc-start 1383145786.578 INFO lxc_conf - now p is . /sys/fs/cgroup.
lxc-start 1383145786.578 INFO lxc_conf - looking at .23 15 0:17 / /sys/fs/fuse/connections rw,relatime - fusectl none rw
.
lxc-start 1383145786.578 INFO lxc_conf - now p is . /sys/fs/fuse/connections.
lxc-start 1383145786.578 INFO lxc_conf - looking at .24 15 0:6 / /sys/kernel/debug rw,relatime - debugfs none rw
.
lxc-start 1383145786.579 INFO lxc_conf - now p is . /sys/kernel/debug.
lxc-start 1383145786.579 INFO lxc_conf - looking at .25 15 0:10 / /sys/kernel/security rw,relatime - securityfs none rw
.
lxc-start 1383145786.579 INFO lxc_conf - now p is . /sys/kernel/security.
lxc-start 1383145786.579 INFO lxc_conf - looking at .26 19 0:18 / /run/lock rw,nosuid,nodev,noexec,relatime - tmpfs none rw,size=5120k
.
lxc-start 1383145786.579 INFO lxc_conf - now p is . /run/lock.
lxc-start 1383145786.579 INFO lxc_conf - looking at .27 19 0:19 / /run/shm rw,nosuid,nodev,relatime - tmpfs none rw
.
lxc-start 1383145786.579 INFO lxc_conf - now p is . /run/shm.
lxc-start 1383145786.579 INFO lxc_conf - looking at .28 22 0:20 / /sys/fs/cgroup/cpuset rw,relatime - cgroup cgroup rw,cpuset,clone_children
.
lxc-start 1383145786.579 INFO lxc_conf - now p is . /sys/fs/cgroup/cpuset.
lxc-start 1383145786.579 INFO lxc_conf - looking at .29 19 0:21 / /run/user rw,nosuid,nodev,noexec,relatime - tmpfs none rw,size=102400k,mode=755
.
lxc-start 1383145786.579 INFO lxc_conf - now p is . /run/user.
lxc-start 1383145786.579 INFO lxc_conf - looking at .30 15 0:22 / /sys/fs/pstore rw,relatime - pstore none rw
.
lxc-start 1383145786.579 INFO lxc_conf - now p is . /sys/fs/pstore.
lxc-start 1383145786.579 INFO lxc_conf - looking at .31 22 0:23 / /sys/fs/cgroup/cpu rw,relatime - cgroup cgroup rw,cpu
.
lxc-start 1383145786.579 INFO lxc_conf - now p is . /sys/fs/cgroup/cpu.
lxc-start 1383145786.579 INFO lxc_conf - looking at .32 22 0:24 / /sys/fs/cgroup/cpuacct rw,relatime - cgroup cgroup rw,cpuacct
.
lxc-start 1383145786.579 INFO lxc_conf - now p is . /sys/fs/cgroup/cpuacct.
lxc-start 1383145786.579 INFO lxc_conf - looking at .33 22 0:25 / /sys/fs/cgroup/memory rw,relatime - cgroup cgroup rw,memory
.
lxc-start 1383145786.579 INFO lxc_conf - now p is . /sys/fs/cgroup/memory.
lxc-start 1383145786.579 INFO lxc_conf - looking at .34 22 0:26 / /sys/fs/cgroup/devices rw,relatime - cgroup cgroup rw,devices
.
lxc-start 1383145786.579 INFO lxc_conf - now p is . /sys/fs/cgroup/devices.
lxc-start 1383145786.579 INFO lxc_conf - looking at .35 22 0:27 / /sys/fs/cgroup/freezer rw,relatime - cgroup cgroup rw,freezer
.
lxc-start 1383145786.579 INFO lxc_conf - now p is . /sys/fs/cgroup/freezer.
lxc-start 1383145786.579 INFO lxc_conf - looking at .36 22 0:28 / /sys/fs/cgroup/blkio rw,relatime - cgroup cgroup rw,blkio
.
lxc-start 1383145786.579 INFO lxc_conf - now p is . /sys/fs/cgroup/blkio.
lxc-start 1383145786.579 INFO lxc_conf - looking at .37 22 0:29 / /sys/fs/cgroup/perf_event rw,relatime - cgroup cgroup rw,perf_event
.
lxc-start 1383145786.579 INFO lxc_conf - now p is . /sys/fs/cgroup/perf_event.
lxc-start 1383145786.579 INFO lxc_conf - looking at .38 22 0:30 / /sys/fs/cgroup/hugetlb rw,relatime - cgroup cgroup rw,hugetlb
.
lxc-start 1383145786.579 INFO lxc_conf - now p is . /sys/fs/cgroup/hugetlb.
lxc-start 1383145786.579 INFO lxc_conf - looking at .39 20 9:2 / /data rw,relatime - ext4 /dev/md2 rw,errors=remount-ro,data=ordered
.
lxc-start 1383145786.579 INFO lxc_conf - now p is . /data.
lxc-start 1383145786.579 INFO lxc_conf - looking at .40 20 8:1 / /boot rw,relatime - ext2 /dev/sda1 rw,errors=continue
.
lxc-start 1383145786.579 INFO lxc_conf - now p is . /boot.
lxc-start 1383145786.579 INFO lxc_conf - looking at .41 22 0:31 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime - cgroup systemd rw,name=systemd
.
lxc-start 1383145786.579 INFO lxc_conf - now p is . /sys/fs/cgroup/systemd.
lxc-start 1383145786.579 NOTICE lxc_start - '/sbin/init' started with pid '6249'
lxc-start 1383145786.579 WARN lxc_start - invalid pid for SIGCHLD
<4>init: ureadahead main process (7) terminated with status 5
<4>init: console-font main process (94) terminated with status 1
And it will just sit there like that for hours at least. The container becomes pingable but I can't ssh and if I try lxc-console -n as1 I get a blank screen.
If I do lxc-stop -n as1 or ^C in the window where it has hung I get:
^CTERM environment variable not set.
<4>init: plymouth-upstart-bridge main process (192) terminated with status 1
<4>init: hwclock-save main process (187) terminated with status 70
* Asking all remaining processes to terminate...
...done.
* All processes ended within 1 seconds...
...done.
* Deactivating swap...
...fail!
mount: cannot mount block device /dev/md2 read-only
* Will now restart
But after 20 minutes it hasn't restarted. Any ideas why these containers are hanging?
