Home > 11.10 > LVM logical volumes fail to activate on boot after 11.04 to 11.10 live upgrade

LVM logical volumes fail to activate on boot after 11.04 to 11.10 live upgrade

December 29Hits:1
Advertisement

I have upgraded my server from 11.04 to 11.10 (64 bit) using sudo do-release-upgrade

The machine now halts during boot because it can't find certain logical volumes in /mnt. When this happens, I hit "m" to drop down to a root shell, and I see the following (forgive me for inaccuracies, I'm recreating this):

$ lvs   LV       VG   Attr   LSize   Origin Snap%  Move Log Copy%  Convert   audio    vg   -wi--- 372.53g                                         home     vg   -wi-ao 186.26g                                         swap     vg   -wi-ao   3.72g                                       

The corresponding block devices in /dev are missing for "audio".

If I run:

$ vgchange -a y $ lvs   LV       VG   Attr   LSize   Origin Snap%  Move Log Copy%  Convert   audio    vg   -wi-ao 372.53g                                         home     vg   -wi-ao 186.26g                                         swap     vg   -wi-ao   3.72g                                       

Then all LVs are activated and the system continues to boot perfectly after exiting from the root maintenance shell.

What is going on, and how would I set the LVs to always be active on boot up?



Update to answer questions raised:

There is one volume group:

# vgs   VG   #PV #LV #SN Attr   VSize VFree   vg     1   6   0 wz--n- 1.68t    0  # pvs   PV         VG   Fmt  Attr PSize PFree   /dev/md2   vg   lvm2 a-   1.68t    0  

Over a RAID1 MD array consisting of a matched pair of SATA hard disks:

 # cat /proc/mdstat   Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]   md2 : active raid1 sda3[0] sdb3[1]        1806932928 blocks [2/2] [UU]   md1 : active raid1 sda2[0] sdb2[1]        146484160 blocks [2/2] [UU]   md3 : active raid1 sda4[0] sdb4[1]        95168 blocks [2/2] [UU]   unused devices: <none> 

So:

 # mount  /dev/md1 on / type ext4 (rw,errors=remount-ro)  proc on /proc type proc (rw,noexec,nosuid,nodev)  sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)  fusectl on /sys/fs/fuse/connections type fusectl (rw)  none on /sys/kernel/debug type debugfs (rw)  none on /sys/kernel/security type securityfs (rw)  udev on /dev type devtmpfs (rw,mode=0755)  devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)  tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)  none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)  none on /run/shm type tmpfs (rw,nosuid,nodev)  /dev/md3 on /boot type ext3 (rw)  /dev/mapper/vg-home on /home type reiserfs (rw)  /dev/mapper/vg-audio on /mnt/audio type reiserfs (rw)  rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)  nfsd on /proc/fs/nfsd type nfsd (rw) 


Update with full lvdisplay - as suspected the partitions that work are first in the list. Can't see anything odd myself. I've included the full list here - this is all my LVM partitions.

This is from the running machine, if output from the broken state is useful that'll need some time to obtain.

# lvdisplay    --- Logical volume ---   LV Name                /dev/vg/swap   VG Name                vg   LV UUID                INuOTR-gwB8-Z0RW-lGHM-qtRF-Xc7D-Bv43ah   LV Write Access        read/write   LV Status              available   # open                 2   LV Size                3.72 GiB   Current LE             953   Segments               1   Allocation             inherit   Read ahead sectors     auto   - currently set to     256   Block device           252:0    --- Logical volume ---   LV Name                /dev/vg/home   VG Name                vg   LV UUID                7L34YS-Neh0-V5OL-bFfd-TmO4-8CkV-GwXuRL   LV Write Access        read/write   LV Status              available   # open                 2   LV Size                186.26 GiB   Current LE             47683   Segments               1   Allocation             inherit   Read ahead sectors     auto   - currently set to     256   Block device           252:1    --- Logical volume ---   LV Name                /dev/vg/audio   VG Name                vg   LV UUID                AX1ZG5-vwyk-mYVl-DBHt-Rgp2-DSwg-oDZlbS   LV Write Access        read/write   LV Status              available   # open                 2   LV Size                372.53 GiB   Current LE             95367   Segments               1   Allocation             inherit   Read ahead sectors     auto   - currently set to     256   Block device           252:2    --- Logical volume ---   LV Name                /dev/vg/vmware   VG Name                vg   LV UUID                bj0m1h-jndV-GWU8-aePm-gaoo-Q0pE-cWhWj2   LV Write Access        read/write   LV Status              available   # open                 2   LV Size                372.53 GiB   Current LE             95367   Segments               1   Allocation             inherit   Read ahead sectors     auto   - currently set to     256   Block device           252:3    --- Logical volume ---   LV Name                /dev/vg/backup   VG Name                vg   LV UUID                PHDnjD-8uT8-yHB2-8SBW-d7E1-1Zws-Qx0Tp8   LV Write Access        read/write   LV Status              available   # open                 1   LV Size                93.13 GiB   Current LE             23841   Segments               1   Allocation             inherit   Read ahead sectors     auto   - currently set to     256   Block device           252:4    --- Logical volume ---   LV Name                /dev/vg/download   VG Name                vg   LV UUID                64Your-pvNG-7EvG-exns-eK9A-vMDD-eozIBM   LV Write Access        read/write   LV Status              available   # open                 2   LV Size                695.05 GiB   Current LE             177934   Segments               2   Allocation             inherit   Read ahead sectors     auto   - currently set to     256   Block device           252:5 


Update 2012/01/28: a reboot has given an opportunity to look at the machine in its broken state.

I don't know if relevant, but the machine was shut down cleanly but the file systems were not clean on starting up again.

# lvs   LV       VG   Attr   LSize   Origin Snap%  Move Log Copy%  Convert   audio    vg   -wi--- 372.53g                                         backup   vg   -wi---  93.13g                                         download vg   -wi--- 695.05g                                         home     vg   -wi-ao 186.26g                                         swap     vg   -wi-ao   3.72g                                         vmware   vg   -wi--- 372.53g 

Although perhaps of interest is (note download):

# lvs --segments   LV       VG   Attr   #Str Type   SSize     audio    vg   -wi---    1 linear 372.53g   backup   vg   -wi---    1 linear  93.13g   download vg   -wi---    1 linear 508.79g   download vg   -wi---    1 linear 186.26g   home     vg   -wi-ao    1 linear 186.26g   swap     vg   -wi-ao    1 linear   3.72g   vmware   vg   -wi---    1 linear 372.53g 

More:

# lvdisplay    --- Logical volume ---   LV Name                /dev/vg/swap   VG Name                vg   LV UUID                INuOTR-gwB8-Z0RW-lGHM-qtRF-Xc7D-Bv43ah   LV Write Access        read/write   LV Status              available   # open                 2   LV Size                3.72 GiB   Current LE             953   Segments               1   Allocation             inherit   Read ahead sectors     auto   - currently set to     256   Block device           252:0    --- Logical volume ---   LV Name                /dev/vg/home   VG Name                vg   LV UUID                7L34YS-Neh0-V5OL-bFfd-TmO4-8CkV-GwXuRL   LV Write Access        read/write   LV Status              available   # open                 2   LV Size                186.26 GiB   Current LE             47683   Segments               1   Allocation             inherit   Read ahead sectors     auto   - currently set to     256   Block device           252:1    --- Logical volume ---   LV Name                /dev/vg/audio   VG Name                vg   LV UUID                AX1ZG5-vwyk-mYVl-DBHt-Rgp2-DSwg-oDZlbS   LV Write Access        read/write   LV Status              NOT available   LV Size                372.53 GiB   Current LE             95367   Segments               1   Allocation             inherit   Read ahead sectors     auto    --- Logical volume ---   LV Name                /dev/vg/vmware   VG Name                vg   LV UUID                bj0m1h-jndV-GWU8-aePm-gaoo-Q0pE-cWhWj2   LV Write Access        read/write   LV Status              NOT available   LV Size                372.53 GiB   Current LE             95367   Segments               1   Allocation             inherit   Read ahead sectors     auto    --- Logical volume ---   LV Name                /dev/vg/backup   VG Name                vg   LV UUID                PHDnjD-8uT8-yHB2-8SBW-d7E1-1Zws-Qx0Tp8   LV Write Access        read/write   LV Status              NOT available   LV Size                93.13 GiB   Current LE             23841   Segments               1   Allocation             inherit   Read ahead sectors     auto    --- Logical volume ---   LV Name                /dev/vg/download   VG Name                vg   LV UUID                64Your-pvNG-7EvG-exns-eK9A-vMDD-eozIBM   LV Write Access        read/write   LV Status              NOT available   LV Size                695.05 GiB   Current LE             177934   Segments               2   Allocation             inherit   Read ahead sectors     auto   # vgs   VG   #PV #LV #SN Attr   VSize VFree   vg     1   6   0 wz--n- 1.68t    0   # pvs   PV         VG   Fmt  Attr PSize PFree   /dev/md2   vg   lvm2 a-   1.68t    0  

From dmesg:

[    0.908322] ata3.00: ATA-8: ST32000542AS, CC34, max UDMA/133 [    0.908325] ata3.00: 3907029168 sectors, multi 16: LBA48 NCQ (depth 0/32) [    0.908536] ata3.01: ATA-8: ST32000542AS, CC34, max UDMA/133 [    0.908538] ata3.01: 3907029168 sectors, multi 16: LBA48 NCQ (depth 0/32) [    0.924307] ata3.00: configured for UDMA/133 [    0.940315] ata3.01: configured for UDMA/133 [    0.940408] scsi 2:0:0:0: Direct-Access     ATA      ST32000542AS     CC34 PQ: 0 ANSI: 5 [    0.940503] sd 2:0:0:0: [sda] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB) [    0.940541] sd 2:0:0:0: Attached scsi generic sg0 type 0 [    0.940544] sd 2:0:0:0: [sda] Write Protect is off [    0.940546] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00 [    0.940564] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [    0.940611] scsi 2:0:1:0: Direct-Access     ATA      ST32000542AS     CC34 PQ: 0 ANSI: 5 [    0.940699] sd 2:0:1:0: Attached scsi generic sg1 type 0 [    0.940728] sd 2:0:1:0: [sdb] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB) [    0.945319] sd 2:0:1:0: [sdb] Write Protect is off [    0.945322] sd 2:0:1:0: [sdb] Mode Sense: 00 3a 00 00 [    0.945660] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [    0.993794]  sda: sda1 sda2 sda3 sda4 [    1.023974] sd 2:0:0:0: [sda] Attached SCSI disk [    1.024277]  sdb: sdb1 sdb2 sdb3 sdb4 [    1.024529] sd 2:0:1:0: [sdb] Attached SCSI disk   [    1.537688] md: bind<sdb3> [    1.538922] bio: create slab <bio-1> at 1 [    1.538983] md/raid1:md2: active with 2 out of 2 mirrors [    1.539005] md2: detected capacity change from 0 to 1850299318272 [    1.540678]  md2: unknown partition table [    1.540851] md: bind<sdb4> [    1.542231] md/raid1:md3: active with 2 out of 2 mirrors [    1.542245] md3: detected capacity change from 0 to 97452032 [    1.543867] md: bind<sdb2> [    1.544680]  md3: unknown partition table [    1.545627] md/raid1:md1: active with 2 out of 2 mirrors [    1.545642] md1: detected capacity change from 0 to 149999779840 [    1.556008]    generic_sse:  9824.000 MB/sec [    1.556010] xor: using function: generic_sse (9824.000 MB/sec) [    1.556721] md: raid6 personality registered for level 6 [    1.556723] md: raid5 personality registered for level 5 [    1.556724] md: raid4 personality registered for level 4 [    1.560491] md: raid10 personality registered for level 10 [    1.571416]  md1: unknown partition table   [    1.935835] EXT4-fs (md1): INFO: recovery required on readonly filesystem [    1.935838] EXT4-fs (md1): write access will be enabled during recovery [    2.901833] EXT4-fs (md1): orphan cleanup on readonly fs [    2.901840] EXT4-fs (md1): ext4_orphan_cleanup: deleting unreferenced inode 4981215 [    2.901904] EXT4-fs (md1): ext4_orphan_cleanup: deleting unreferenced inode 8127848 [    2.901944] EXT4-fs (md1): 2 orphan inodes deleted [    2.901946] EXT4-fs (md1): recovery complete [    3.343830] EXT4-fs (md1): mounted filesystem with ordered data mode. Opts: (null) [   64.851211] Adding 3903484k swap on /dev/mapper/vg-swap.  Priority:-1 extents:1 across:3903484k   [   67.600045] EXT4-fs (md1): re-mounted. Opts: errors=remount-ro [   68.459775] EXT3-fs: barriers not enabled [   68.460520] kjournald starting.  Commit interval 5 seconds [   68.461183] EXT3-fs (md3): using internal journal [   68.461187] EXT3-fs (md3): mounted filesystem with ordered data mode [  130.280048] REISERFS (device dm-1): found reiserfs format "3.6" with standard journal [  130.280060] REISERFS (device dm-1): using ordered data mode [  130.284596] REISERFS (device dm-1): journal params: device dm-1, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30 [  130.284918] REISERFS (device dm-1): checking transaction log (dm-1) [  130.450867] REISERFS (device dm-1): Using r5 hash to sort names 

Answers

Nothing error/warning wise in the logs is standing out. However, I did notice that you've created 3 MDs, all using the same backing store, with essentially 3 different filesystems (ext3,ext4,reiserfs (maybe raw VMware too?)). I wonder if all the competing I/O writeback strategies, combined with multiple locks/queuing to the same backing store is crowding out your attempts to use the adjacent MDs under certain conditions?

This would blowback on the filesystem as a failed journal or writeback and manifest in LVM as a failure to assemble or activate the mapping.

Ideally, you're setup should look like this:

sda [sda1 {SPAN entire disk - 1-2%}]

sdb [sdb1 {SPAN entire disk - 1-2%}]

md1 (RAID1) [sda1 sdb1]

vg [md1]

You can boot directly from a root lvm backed by an md, I do this myself. The reason you don't span the entire disk is if you ever get bad blocks and use the vendor low level repair tool, you can sometimes run out of free blocks and the tool commender's blocks that are in use, which changes the net size of the disk, and breaking your partition table and certainly MD in the process. See MDADM Superblock Recovery

If you still have problems with one MD, instead of 3, then either one of your filesystems or VMware is being naughty and someone's starving out your siblings, or there's a real problem with your backing store which makes everyone else a victim.

The machine gradually became worse and worse as updates were installed. It was rebooted this morning and didn't boot back up - /home wouldn't activate properly at all. And then dbus errors, etc. etc.

9 hours later, I've done a full reinstall of 11.10 onto those same partitions and they now work fine. Very odd, but this appears to have been the solution.

Thanks ppetraki, I agree with your points on the basis of performance will look at this when the machine is replaced.

Tags:lvm, 11.10

Related Articles

  • LVM logical volumes fail to activate on boot after 11.04 to 11.10 live upgradeDecember 29

    I have upgraded my server from 11.04 to 11.10 (64 bit) using sudo do-release-upgrade The machine now halts during boot because it can't find certain logical volumes in /mnt. When this happens, I hit "m" to drop down to a root shell, and I see th

  • LVM logical volumes do not activate on boot on 13.10January 15

    I am running an UbuntuGNOME 13.10 and I set it up without an LVM during the installation process. I am now running it with to encrypted partitions, one is containing a LVM. This all works fine, except, that the logical volumes are not activated durin

  • "fdisk -l" like list of partitions and their types for LVM logical volumes?June 20

    You know how "fdisk -l" lists drive partition tables and shows the partition id/types for each partition? Is there a similar way to get the partition id for LVM logical volumes? EDIT: I'm aware of "lvs", which is mostly what I'm lookin

  • Mapping between KVM guest virtual discs and LVM logical volumesDecember 21

    The question is, how do I know which LVM partition corresponds to which virtual KVM disc? I am trying to install a KVM guest (Ubuntu server) on LVM logical volumes. I have used lvcreate to create partitions for boot, root, home, tmp, usr, var and swa

  • How can I minimise the size of an ext3 partition (and its LVM logical volume)?December 22

    I am using LVM on Ubuntu 9.10 (Karmic). I have a single LVM physical volume (and a single volume group). I have an ext3 filesystem inside an LVM logical volume which I no longer use, but for the time being I would like not to delete it. I'm trying to

  • How can I copy a LVM logical volume to a loopback file on another system?August 9

    I am trying to copy an LVM logical volume with an ext3 filesystem from one system to a file on another system that I will be able to mount as a loopback device. I have tried the following command to copy the volume: dd if=/dev/VolGroup01/example-volu

  • How to configure and run hadoop cluster on LVM(Logical Volume Manager)March 14

    Well i am new to the concepts of LVM(Local Volume Manager) and have create volumes in Ubuntu Actually I am using windows and putty.Now i have also created LVM in that putty. My question is whether it is possible to configure and run hadoop cluster in

  • Does linux lvm(logic volum manager) has API for C programming?May 25

    Hi, Does lvm(logic volum manager) in Linux(not UNIX) has any API that corresponds to commands like "vgdisplay" or "lvcreate"? And how to call these API? Thank you! --------------Solutions------------- http://fedoraproject.org/wiki/LVM/

  • Is there something analogue to LVM logical volumes in BTRFS?August 21

    I like the way BTRFS does snapshots and subvolumes but it does not feel right to use LVM just because I want my logical volume to be of certain size. Is there something analogue to LVM logical volumes in BTRFS? The closest thing to it seems to me to

  • NFS export configuration for multiple LVM logical volumesSeptember 18

    I have setup a file server that contains two large file systems, each consisting of a single LVM logical volume (10TB and 16TB) within a single volume group. The physical volumes making up the LV are SAN LUNs connected via iSCSI/multipath. Now, I wou

  • Is It Possible To Recover A Partial LVM Logical Volume?November 19

    Background It is an Ubuntu 12.04 VirtualBox VM with 5 virtual HDDs (VDI), NOTE this is just a test VM, so not well planned ahead: ubuntu.vdi for / (/dev/mapper/ubuntu-root AKA /dev/ubuntu/root) and /home (/dev/mapper/ubuntu-home) weblogic.vdi - /dev/

  • device mapper on RHEL6 unable to create devs for LVM logical volumeApril 3

    I have XEN guest running RHEL6, and it has a LUN presented from the Dom0. This contains an LVM volume groups called vg_ALHINT (INT for Integration and ALH is an abbreviation of its Oracle database name). The data is Oracle 11g. The VG was imported, a

  • Physical volumes, logical volumes, superblocks gone- how to boot?November 22

    A story fairly similar to this one happened to me yesterday. Update: My only hard drive /dev/sda was originally partitioned this way: Device Boot Start End Blocks Id System /dev/sda1 * 2048 1026047 512000 83 Linux /dev/sda2 1026048 976773119 48787353

  • Any way to recover ext4 filesystems from a deleted LVM logical volume?October 8

    The other day I had a proper brain fart moment while expanding a disk on a Linux guest under Vmware. I stretched the Vmware disk file to the desired size and then I did what I usually do on Linux guests without LVM: I deleted the LVM partition and re

  • User reduced LVM logical volume without resizing filesystemJune 18

    I received an email yesterday that one of our users was trying to make room for a heartbeat/clustering package which requires its own partition to act as a voting disk. To do this, he attempted to reduce the size of the root partition's logical volum

  • How should I configure TRIM Support for LVM logical volumes?September 22

    I am setting up a notebook for software demo purpose. The machine has a Intel Core i7 CPU, 8GB RAM, a 128GB SSD, and runs Ubuntu 12.04 LTS 64bit desktop. As it is, the SSD is configured to have a single volume group, with /boot, /swap, and / all in t

  • Does swap need to be on a contiguous LVM logical volumeDecember 12

    I am setting up a machine to run a number of virtual machines. I am using a single HDD with a boot partition and an LVM partition. I read on the Arch wiki that logcial volumes which will be used for swap should be setup with -C y To create a contiguo

  • Does bcache support use of an LVM logical volume as a cache device?January 15

    I've got a 64GB SSD and a 3TB hard drive in my system running Ubuntu 14.04. The SSD has a small root partition with rest of the device allocated to an LVM physical volume. From that LVM physical volume I have two logical volumes allocated, one for /u

  • How can I recreate removed LVM Logical Volumes?January 20

    I accidentally removed all my Logical Volumes with lvremove. After reboot the lvm backups were lost (live CD) so I can't use vgcfgrestore. testdisk can find the partitions by analysis so they are still there. Is there a way to restore the logical vol

  • Mount Logical Volume failingMarch 5

    I'm trying to mount my logical volume and get the following error: [email protected]:/home/jon# mount /dev/mediaserver/homeserver /media mount: you must specify the filesystem type So I tried to specify ext4 type: [email protected]:/home/jon# mount -t

Copyright (C) 2018 ceus-now.com, All Rights Reserved. webmaster#ceus-now.com 14 q. 0.859 s.