【LVM】系统重装后LVM分区挂载的坑

遇到的问题：

系统重装前有个LVM 做过扩容。重装后后手动挂载是可以的。系统重启后挂载失败。

原ubuntu12.04 server 版本分区如下。

磁盘配置

由于在升级glibc 导致系统重启后无法启动，做系统修复操作后还是错。于是重装系统。

重装工程中对原 ext4分区进行格式化，LVM 未做改动。

新系统磁盘分配

系统安装后安装完。执行以下命令后可以正常挂载访问。

e2label /dev/vgjenkins/lvjenkins /jenkinse2label /dev/vgbuild/lvlocaldata /localdatamkdir  /mnt/jenkins/mkdir  /mnt/localdatamount -L /jenkins /mnt/jenkinsmount -L  /localdata /mnt/localdata

修改 /etc/fstab

vim /etc/fstab#--add below
LABEL=/localdata  /mnt/localdata ext4        defaults,relatime,usrquota,data=writeback                    0       2
LABEL=/jenkins       /mnt/jenkins  ext4        defaults                    0       2

重启系统提示 /locadaldata 无法挂载。按S后跳过挂载开机。检查LVM 配置如下

 vgdisplayCouldn't find device with uuid n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc.--- Volume group ---VG Name               vgbuildSystem IDFormat                lvm2Metadata Areas        1Metadata Sequence No  5VG Access             read/writeVG Status             resizableMAX LV                0Cur LV                1Open LV               0Max PV                0Cur PV                2Act PV                1VG Size               6.94 TiBPE Size               4.00 MiBTotal PE              1819968Alloc PE / Size       1819592 / 6.94 TiBFree  PE / Size       376 / 1.47 GiBVG UUID               qM5sXD-YUFC-Ylwq-0VM1-qpI7-OvAc-CLdFJt--- Volume group ---VG Name               vgjenkinsSystem IDFormat                lvm2Metadata Areas        1Metadata Sequence No  2VG Access             read/writeVG Status             resizableMAX LV                0Cur LV                1Open LV               1Max PV                0Cur PV                1Act PV                1VG Size               465.66 GiBPE Size               4.00 MiBTotal PE              119209Alloc PE / Size       119209 / 465.66 GiBFree  PE / Size       0 / 0VG UUID               dVGlj7-nF52-csCj-8oMg-tddl-1WtY-Zn7IeWroot@xmbuilder02:~# vgscanReading all physical volumes.  This may take a while...Couldn't find device with uuid n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc.Found volume group "vgbuild" using metadata type lvm2Found volume group "vgjenkins" using metadata type lvm2
root@xmbuilder02:~# lvscanCouldn't find device with uuid n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc.inactive          '/dev/vgbuild/lvlocaldata' [6.94 TiB] inheritACTIVE            '/dev/vgjenkins/lvjenkins' [465.66 GiB] inherit

查看磁盘的物理卷组

# pvdisplayCouldn't find device with uuid n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc.--- Physical volume ---PV Name               /dev/sda9VG Name               vgbuildPV Size               4.22 TiB / not usable 1.00 MiBAllocatable           yes (but full)PE Size               4.00 MiBTotal PE              1105249Free PE               0Allocated PE          1105249PV UUID               0YRTc9-dBEp-Q6Oc-sqyX-PTzh-oqzj-AVUXHM--- Physical volume ---PV Name               unknown deviceVG Name               vgbuildPV Size               2.73 TiB / not usable 4.00 MiBAllocatable           yesPE Size               4.00 MiBTotal PE              714719Free PE               376Allocated PE          714343PV UUID               n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc--- Physical volume ---PV Name               /dev/sda8VG Name               vgjenkinsPV Size               465.66 GiB / not usable 1.00 MiBAllocatable           yes (but full)PE Size               4.00 MiBTotal PE              119209Free PE               0Allocated PE          119209PV UUID               cTpMy1-ml54-jMFu-UN09-KR9E-4XT7-gStZ49

可以确定是 /dev/sdb 丢了

#lvscanCouldn't find device with uuid n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc.inactive          '/dev/vgbuild/lvlocaldata' [6.94 TiB] inheritACTIVE            '/dev/vgjenkins/lvjenkins' [465.66 GiB] inherit#  pvdisplayCouldn't find device with uuid n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc.--- Physical volume ---PV Name               unknown deviceVG Name               vgbuildPV Size               2.73 TiB / not usable 4.00 MiBAllocatable           yesPE Size               4.00 MiBTotal PE              714719Free PE               376Allocated PE          714343PV UUID               n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc

fdisk 命令查看磁盘

 #fdisk -lWARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted.Disk /dev/sdb: 2997.8 GB, 2997752954880 bytes
255 heads, 63 sectors/track, 364456 cylinders, total 5854986240 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1  4294967295  2147483647+  ee  GPTWARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.Disk /dev/sda: 5396.0 GB, 5395955318784 bytes
255 heads, 63 sectors/track, 656020 cylinders, total 10538975232 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1  4294967295  2147483647+  ee  GPTDisk /dev/mapper/vgjenkins-lvjenkins: 500.0 GB, 499998785536 bytes
255 heads, 63 sectors/track, 60788 cylinders, total 976560128 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000Disk /dev/mapper/vgjenkins-lvjenkins doesn't contain a valid partition table

分析：

   1.   找了不少信息，也在不少技术群里问了，都没人回答，其实 想知道我要是用pvcreate 创建/dev/sdb1会导致之前数据丢失么。要不我很想试试这个命令 来新建丢失的PV

pvcreate /dev/sdb1 --u n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc --norestorefile

2.  或者考虑 vgreduce或者 pvremove  来移除这个  PV Name   unknown devicec，然后再添加上去。

但是vgreduce和命令不是太熟悉 pvremove 

3. 害怕数据丢失，还是来回顾一下之前添加扩容的过程吧

解决方法：

1  单纯的移除 vgbuild 中丢失的部分是不行的因为 VG vgbuild 中有一部分扩容，出问题的刚好是扩容的那部分。

root@xmbuilder02:/etc/lvm/backup# pvsCouldn't find device with uuid n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc.PV             VG        Fmt  Attr PSize   PFree/dev/sda8      vgjenkins lvm2 a--  465.66g    0/dev/sda9      vgbuild   lvm2 a--    4.22t    0unknown device vgbuild   lvm2 a-m    2.73t 1.47g
root@xmbuilder02:/etc/lvm/backup# vgreduce --removemissing vgbuildCouldn't find device with uuid n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc.WARNING: Partial LV lvlocaldata needs to be repaired or removed.There are still partial LVs in VG vgbuild.To remove them unconditionally use: vgreduce --removemissing --force.Proceeding to remove empty missing PVs.

打算重建PV了,用pvcreate 后sdb 是有了。

#pvcreate -u n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc --restorefile /etc/lvm/backup/vgbuild /dev/sdbCouldn't find device with uuid n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc.Physical volume "/dev/sdb" successfully created
root@xmbuilder02:~# pvsPV         VG        Fmt  Attr PSize   PFree/dev/sda8  vgjenkins lvm2 a--  465.66g    0/dev/sda9  vgbuild   lvm2 a--    4.22t    0/dev/sdb   vgbuild   lvm2 a-m    2.73t 1.47g

于是在做个reteore试试

root@xmbuilder02:~#vgcfgrestore vgbuild
root@xmbuilder02:~# vgcfgrestore vgbuildRestored volume group vgbuild
root@xmbuilder02:~# pvsPV         VG        Fmt  Attr PSize   PFree/dev/sda8  vgjenkins lvm2 a--  465.66g    0/dev/sda9  vgbuild   lvm2 a--    4.22t    0/dev/sdb   vgbuild   lvm2 a--    2.73t 1.47g

最终的记录

root@xmbuilder02:~# pvsPV         VG        Fmt  Attr PSize   PFree/dev/sda8  vgjenkins lvm2 a--  465.66g    0/dev/sda9  vgbuild   lvm2 a--    4.22t    0/dev/sdb   vgbuild   lvm2 a--    2.73t 1.47g
root@xmbuilder02:~# vgscanReading all physical volumes.  This may take a while...Found volume group "vgbuild" using metadata type lvm2Found volume group "vgjenkins" using metadata type lvm2
root@xmbuilder02:~# lvscanACTIVE            '/dev/vgbuild/lvlocaldata' [6.94 TiB] inheritACTIVE            '/dev/vgjenkins/lvjenkins' [465.66 GiB] inherit
root@xmbuilder02:~# vgdisplay--- Volume group ---VG Name               vgbuildSystem IDFormat                lvm2Metadata Areas        2Metadata Sequence No  10VG Access             read/writeVG Status             resizableMAX LV                0Cur LV                1Open LV               0Max PV                0Cur PV                2Act PV                2VG Size               6.94 TiBPE Size               4.00 MiBTotal PE              1819968Alloc PE / Size       1819592 / 6.94 TiBFree  PE / Size       376 / 1.47 GiBVG UUID               qM5sXD-YUFC-Ylwq-0VM1-qpI7-OvAc-CLdFJt--- Volume group ---VG Name               vgjenkinsSystem IDFormat                lvm2Metadata Areas        1Metadata Sequence No  2VG Access             read/writeVG Status             resizableMAX LV                0Cur LV                1Open LV               1Max PV                0Cur PV                1Act PV                1VG Size               465.66 GiBPE Size               4.00 MiBTotal PE              119209Alloc PE / Size       119209 / 465.66 GiBFree  PE / Size       0 / 0VG UUID               dVGlj7-nF52-csCj-8oMg-tddl-1WtY-Zn7IeWroot@xmbuilder02:~# pvdisplay--- Physical volume ---PV Name               /dev/sda9VG Name               vgbuildPV Size               4.22 TiB / not usable 1.00 MiBAllocatable           yes (but full)PE Size               4.00 MiBTotal PE              1105249Free PE               0Allocated PE          1105249PV UUID               0YRTc9-dBEp-Q6Oc-sqyX-PTzh-oqzj-AVUXHM--- Physical volume ---PV Name               /dev/sdbVG Name               vgbuildPV Size               2.73 TiB / not usable 4.00 MiBAllocatable           yesPE Size               4.00 MiBTotal PE              714719Free PE               376Allocated PE          714343PV UUID               n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc--- Physical volume ---PV Name               /dev/sda8VG Name               vgjenkinsPV Size               465.66 GiB / not usable 1.00 MiBAllocatable           yes (but full)PE Size               4.00 MiBTotal PE              119209Free PE               0Allocated PE          119209PV UUID               cTpMy1-ml54-jMFu-UN09-KR9E-4XT7-gStZ49

最终的结果，东西没丢，舒心一笑。

root@xmbuilder02:~# e2label /dev/vgbuild/lvlocaldata /localdata
root@xmbuilder02:~# mount -L  /localdata /mnt/localdata
root@xmbuilder02:~# df -h
Filesystem                       Size  Used Avail Use% Mounted on
udev                              63G   12K   63G   1% /dev
tmpfs                             13G  1.9M   13G   1% /run
/dev/sda2                        184G  400M  174G   1% /
none                             4.0K     0  4.0K   0% /sys/fs/cgroup
none                             5.0M     0  5.0M   0% /run/lock
none                              63G     0   63G   0% /run/shm
none                             100M     0  100M   0% /run/user
/dev/sda5                        9.1G   21M  8.6G   1% /tmp
/dev/sda3                        181M   38M  131M  23% /boot
/dev/sda6                        9.1G  557M  8.1G   7% /usr
/dev/sda7                        9.1G  350M  8.3G   4% /var
/dev/mapper/vgjenkins-lvjenkins  459G  324G  136G  71% /mnt/jenkins
/dev/mapper/vgbuild-lvlocaldata  6.9T  6.8T  156G  98% /mnt/localdata