本文主要是介绍【k8s存储--使用OpenEBS做持久化存储】，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！
1、简介
使用OpenEBS，你可以将有持久化数据的容器，像对待其他普通容器一样来对待。OpenEBS本身也是通过容器来部署的，支持Kubernetes、Swarm、Mesos、Rancher编排调度，存储服务可以分派给每个pod、应用程序、集群或者容器级别，包括：
- 跨节点的数据持久化
- 跨可用区和云厂商的数据同步
- 使用商业硬件和容器引擎来提供高可扩展的块存储
- 与容器编排引擎集成，开发者的应用程序可以自动的配置OpenEBS
- 基于CloudByte在BSD的容器化经验，为用户提供OpenEBS的QoS保证
2、架构
OpenEBS存储控制器本身就运行在容器中。OpenEBS Volume由一个或多个以微服务方式运行的容器组成。这种存储控制器功能基于微服务架构——每个卷的数据由其自己的一组容器来提供，而不是由一个统一的同时为多个卷提供控制的，单体（monolithic）存储控制器来提供。这就是OpenEBS与传统存储设备的本质区别。
OpenEBS的架构可以分为数据平面（Data Plane）和控制平面（Control Plane）两部分：
数据平面：为应用程序提供数据存储
控制平面：管理OpenEBS卷容器，这通常会用到容器编排软件的功能
3、数据平面
下图是OpenEBS对应在Kubernetes集群上部署的架构图。其中，黄色部分是OpenEBS持久化存储卷，通过Kubernetes的PV来创建，使用iSCSI来实现，数据保存在node节点上或者云中的卷（如EBS、GPD等），这取决于您的集群部署在哪里。OpenEBS的卷完全独立于用户的应用的生命周期来管理，这也是Kuberentes中的PV的基本思路。
OpenEBS卷为容器提供持久化存储，具有针对系统故障的弹性，更快地访问存储，快照和备份功能。此外，它还提供了监控使用情况和执行QoS策略的机制。
存储数据的磁盘称为存储后端，可以是主机目录，附加块设备或远程磁盘。每个OpenEBS卷包含一个iSCSI目标容器（在上图中表示为openebs-vol1）和一个或多个副本容器（openebs-vol1-R1和openebs-vol1-R2）。
应用程序pod通过iSCSI目标容器访问存储，iSCSI目标容器将数据复制到其所有副本。在发生节点故障时，iSCSI目标容器将从剩余的其中一个在线节点上启动，并通过连接到可用副本容器来提供数据。
该部分的实现包括两个容器：
openebs/jiva：存储控制功能，包括复制逻辑
openebs/gotgt：由openebs/jiva使用的iSCSI目标功能
4、控制平面
OpenEBS控制平面又叫做存储编排或maya。目的是为了创建超融合的OpenEBS，将其挂载到如Kubernetes、Swarm、Nomad等容器编排调度引擎上，用来扩展特定的容器编排系统提供的存储功能。
OpenEBS的控制平面也是基于微服务的，它的服务可以分成以下几个部分：
- 容器编排插件，用于增加强容器编排框架的功能：
- Kubernetes动态配置：openebs-provisioner
- Kubernetes-dashboard：openebs-dashboard
- 扩展的schema：基于Kubernetes的CRD（自定义资源类型），存储OpenEBS相关的配置数据
- 集群服务，提供OpenEBS特定的存储智能，如：
- maya-apiserver：包含执行卷操作的API，可将请求转换为容器编排系统特定的操作
- maya-mulebot：使用收集的信息来建议优化的布局和事件处理提示
- maya-connect：允许将监控数据上传到maya-cloud，以便进一步进行存储访问模式分析
- 节点服务，提供OpenEBS特定的随kubelet一起运行的存储智能，如：
- maya-agent：包括存储管理功能
通过使用prometheus、heapster、grafana和jaegar进行上述服务，可以添加监控和跟踪功能。
5、先决条件
OpenEBS依赖与iSCSI做存储管理，因此需要先确保您的集群上已有安装openiscsi。
注意：如果您使用kubeadm，容器方式安装的kublet，那么其中会自带iSCSI，不需要再手动安装，如果是直接使用二进制形式在裸机上安装的kubelet，则需要自己安装iSCSI。
iSCSI( Internet Small Computer System Interface 互联网小型计算机系统接口)是一种基于TCP/IP 的协议，用来建立和管理IP存储设备、主机和客户机等之间的相互连接，并创建存储区域网络（SAN）。SAN 使得SCSI 协议应用于高速数据传输网络成为可能，这种传输以数据块级别（block-level）在多个数据存储网络间进行。SCSI 结构基于C/S模式，其通常应用环境是：设备互相靠近，并且这些设备由SCSI 总线连接。
OpenEBS需要使用iSCSI作为存储协议，而CentOS上默认是没有安装该软件的，因此我们需要手动安装。
iSCSI中包括两种类型的角色：
- target：用来提供存储（server）
- initiator：使用存储的客户端（client）
下图在Kubernetes中使用iSCSI的架构图
6、安装OpenEBS服务
- wget https://raw.githubusercontent.com/openebs/openebs/master/k8s/openebs-operator.yaml
6.1 安装iSCSI服务
安装iscsi十分简单，不需要额外的配置，只要安装后启动服务即可。在每个node节点上执行下面的命令：
yum -y install iscsi-initiator-utils
systemctl enable iscsid
systemctl start iscsid
6.2 使用Operator运行OpenEBS服务
- wget https://raw.githubusercontent.com/openebs/openebs/master/k8s/openebs-operator.yaml
- kubectl apply -f openebs-operator.yaml
#
# DEPRECATION NOTICE
# This operator file is deprecated in 2.11.0 in favour of individual operators
# for each storage engine and the file will be removed in version 3.0.0
#
# Further specific components can be deploy using there individual operator yamls
#
# To deploy cStor:
# https://github.com/openebs/charts/blob/gh-pages/cstor-operator.yaml
#
# To deploy Jiva:
# https://github.com/openebs/charts/blob/gh-pages/jiva-operator.yaml
#
# To deploy Dynamic hostpath localpv provisioner:
# https://github.com/openebs/charts/blob/gh-pages/hostpath-operator.yaml
#
#
# This manifest deploys the OpenEBS control plane components, with associated CRs & RBAC rules
# NOTE: On GKE, deploy the openebs-operator.yaml in admin context# Create the OpenEBS namespace
apiVersion: v1
kind: Namespace
metadata:name: openebs
---
# Create Maya Service Account
apiVersion: v1
kind: ServiceAccount
metadata:name: openebs-maya-operatornamespace: openebs
---
# Define Role that allows operations on K8s pods/deployments
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: openebs-maya-operator
rules:
- apiGroups: ["*"]resources: ["nodes", "nodes/proxy"]verbs: ["*"]
- apiGroups: ["*"]resources: ["namespaces", "services", "pods", "pods/exec", "deployments", "deployments/finalizers", "replicationcontrollers", "replicasets", "events", "endpoints", "configmaps", "secrets", "jobs", "cronjobs"]verbs: ["*"]
- apiGroups: ["*"]resources: ["statefulsets", "daemonsets"]verbs: ["*"]
- apiGroups: ["*"]resources: ["resourcequotas", "limitranges"]verbs: ["list", "watch"]
- apiGroups: ["*"]resources: ["ingresses", "horizontalpodautoscalers", "verticalpodautoscalers", "certificatesigningrequests"]verbs: ["list", "watch"]
- apiGroups: ["*"]resources: ["storageclasses", "persistentvolumeclaims", "persistentvolumes"]verbs: ["*"]
- apiGroups: ["volumesnapshot.external-storage.k8s.io"]resources: ["volumesnapshots", "volumesnapshotdatas"]verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["apiextensions.k8s.io"]resources: ["customresourcedefinitions"]verbs: [ "get", "list", "create", "update", "delete", "patch"]
- apiGroups: ["openebs.io"]resources: [ "*"]verbs: ["*" ]
- apiGroups: ["cstor.openebs.io"]resources: [ "*"]verbs: ["*" ]
- apiGroups: ["coordination.k8s.io"]resources: ["leases"]verbs: ["get", "watch", "list", "delete", "update", "create"]
- apiGroups: ["admissionregistration.k8s.io"]resources: ["validatingwebhookconfigurations", "mutatingwebhookconfigurations"]verbs: ["get", "create", "list", "delete", "update", "patch"]
- nonResourceURLs: ["/metrics"]verbs: ["get"]
- apiGroups: ["*"]resources: ["poddisruptionbudgets"]verbs: ["get", "list", "create", "delete", "watch"]
---
# Bind the Service Account with the Role Privileges.
# TODO: Check if default account also needs to be there
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: openebs-maya-operator
subjects:
- kind: ServiceAccountname: openebs-maya-operatornamespace: openebs
roleRef:kind: ClusterRolename: openebs-maya-operatorapiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:name: maya-apiservernamespace: openebslabels:name: maya-apiserveropenebs.io/component-name: maya-apiserveropenebs.io/version: dev
spec:selector:matchLabels:name: maya-apiserveropenebs.io/component-name: maya-apiserverreplicas: 1strategy:type: RecreaterollingUpdate: nulltemplate:metadata:labels:name: maya-apiserveropenebs.io/component-name: maya-apiserveropenebs.io/version: devspec:serviceAccountName: openebs-maya-operatorcontainers:- name: maya-apiserverimagePullPolicy: IfNotPresentimage: openebs/m-apiserver:ciports:- containerPort: 5656env:# OPENEBS_IO_KUBE_CONFIG enables maya api service to connect to K8s# based on this config. This is ignored if empty.# This is supported for maya api server version 0.5.2 onwards#- name: OPENEBS_IO_KUBE_CONFIG# value: "/home/ubuntu/.kube/config"# OPENEBS_IO_K8S_MASTER enables maya api service to connect to K8s# based on this address. This is ignored if empty.# This is supported for maya api server version 0.5.2 onwards#- name: OPENEBS_IO_K8S_MASTER# value: "http://172.28.128.3:8080"# OPENEBS_NAMESPACE provides the namespace of this deployment as an# environment variable- name: OPENEBS_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace# OPENEBS_SERVICE_ACCOUNT provides the service account of this pod as# environment variable- name: OPENEBS_SERVICE_ACCOUNTvalueFrom:fieldRef:fieldPath: spec.serviceAccountName# OPENEBS_MAYA_POD_NAME provides the name of this pod as# environment variable- name: OPENEBS_MAYA_POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name# If OPENEBS_IO_CREATE_DEFAULT_STORAGE_CONFIG is false then OpenEBS default# storageclass and storagepool will not be created.- name: OPENEBS_IO_CREATE_DEFAULT_STORAGE_CONFIGvalue: "true"# OPENEBS_IO_INSTALL_DEFAULT_CSTOR_SPARSE_POOL decides whether default cstor sparse pool should be# configured as a part of openebs installation.# If "true" a default cstor sparse pool will be configured, if "false" it will not be configured.# This value takes effect only if OPENEBS_IO_CREATE_DEFAULT_STORAGE_CONFIG# is set to true- name: OPENEBS_IO_INSTALL_DEFAULT_CSTOR_SPARSE_POOLvalue: "false"# OPENEBS_IO_INSTALL_CRD environment variable is used to enable/disable CRD installation# from Maya API server. By default the CRDs will be installed# - name: OPENEBS_IO_INSTALL_CRD# value: "true"# OPENEBS_IO_BASE_DIR is used to configure base directory for openebs on host path.# Where OpenEBS can store required files. Default base path will be /var/openebs# - name: OPENEBS_IO_BASE_DIR# value: "/var/openebs"# OPENEBS_IO_CSTOR_TARGET_DIR can be used to specify the hostpath# to be used for saving the shared content between the side cars# of cstor volume pod.# The default path used is /var/openebs/sparse#- name: OPENEBS_IO_CSTOR_TARGET_DIR# value: "/var/openebs/sparse"# OPENEBS_IO_CSTOR_POOL_SPARSE_DIR can be used to specify the hostpath# to be used for saving the shared content between the side cars# of cstor pool pod. This ENV is also used to indicate the location# of the sparse devices.# The default path used is /var/openebs/sparse#- name: OPENEBS_IO_CSTOR_POOL_SPARSE_DIR# value: "/var/openebs/sparse"# OPENEBS_IO_JIVA_POOL_DIR can be used to specify the hostpath# to be used for default Jiva StoragePool loaded by OpenEBS# The default path used is /var/openebs# This value takes effect only if OPENEBS_IO_CREATE_DEFAULT_STORAGE_CONFIG# is set to true#- name: OPENEBS_IO_JIVA_POOL_DIR# value: "/var/openebs"# OPENEBS_IO_LOCALPV_HOSTPATH_DIR can be used to specify the hostpath# to be used for default openebs-hostpath storageclass loaded by OpenEBS# The default path used is /var/openebs/local# This value takes effect only if OPENEBS_IO_CREATE_DEFAULT_STORAGE_CONFIG# is set to true#- name: OPENEBS_IO_LOCALPV_HOSTPATH_DIR# value: "/var/openebs/local"- name: OPENEBS_IO_JIVA_CONTROLLER_IMAGEvalue: "openebs/jiva:ci"- name: OPENEBS_IO_JIVA_REPLICA_IMAGEvalue: "openebs/jiva:ci"- name: OPENEBS_IO_JIVA_REPLICA_COUNTvalue: "3"- name: OPENEBS_IO_CSTOR_TARGET_IMAGEvalue: "openebs/cstor-istgt:ci"- name: OPENEBS_IO_CSTOR_POOL_IMAGEvalue: "openebs/cstor-pool:ci"- name: OPENEBS_IO_CSTOR_POOL_MGMT_IMAGEvalue: "openebs/cstor-pool-mgmt:ci"- name: OPENEBS_IO_CSTOR_VOLUME_MGMT_IMAGEvalue: "openebs/cstor-volume-mgmt:ci"- name: OPENEBS_IO_VOLUME_MONITOR_IMAGEvalue: "openebs/m-exporter:ci"- name: OPENEBS_IO_CSTOR_POOL_EXPORTER_IMAGEvalue: "openebs/m-exporter:ci"- name: OPENEBS_IO_HELPER_IMAGEvalue: "openebs/linux-utils:ci"# OPENEBS_IO_ENABLE_ANALYTICS if set to true sends anonymous usage# events to Google Analytics- name: OPENEBS_IO_ENABLE_ANALYTICSvalue: "false"- name: OPENEBS_IO_INSTALLER_TYPEvalue: "openebs-operator"# OPENEBS_IO_ANALYTICS_PING_INTERVAL can be used to specify the duration (in hours)# for periodic ping events sent to Google Analytics.# Default is 24h.# Minimum is 1h. You can convert this to weekly by setting 168h#- name: OPENEBS_IO_ANALYTICS_PING_INTERVAL# value: "24h"livenessProbe:exec:command:- /usr/local/bin/mayactl- versioninitialDelaySeconds: 30periodSeconds: 60readinessProbe:exec:command:- /usr/local/bin/mayactl- versioninitialDelaySeconds: 30periodSeconds: 60
---
apiVersion: v1
kind: Service
metadata:name: maya-apiserver-servicenamespace: openebslabels:openebs.io/component-name: maya-apiserver-svc
spec:ports:- name: apiport: 5656protocol: TCPtargetPort: 5656selector:name: maya-apiserversessionAffinity: None
---
apiVersion: apps/v1
kind: Deployment
metadata:name: openebs-provisionernamespace: openebslabels:name: openebs-provisioneropenebs.io/component-name: openebs-provisioneropenebs.io/version: dev
spec:selector:matchLabels:name: openebs-provisioneropenebs.io/component-name: openebs-provisionerreplicas: 1strategy:type: RecreaterollingUpdate: nulltemplate:metadata:labels:name: openebs-provisioneropenebs.io/component-name: openebs-provisioneropenebs.io/version: devspec:serviceAccountName: openebs-maya-operatorcontainers:- name: openebs-provisionerimagePullPolicy: IfNotPresentimage: openebs/openebs-k8s-provisioner:cienv:# OPENEBS_IO_K8S_MASTER enables openebs provisioner to connect to K8s# based on this address. This is ignored if empty.# This is supported for openebs provisioner version 0.5.2 onwards#- name: OPENEBS_IO_K8S_MASTER# value: "http://10.128.0.12:8080"# OPENEBS_IO_KUBE_CONFIG enables openebs provisioner to connect to K8s# based on this config. This is ignored if empty.# This is supported for openebs provisioner version 0.5.2 onwards#- name: OPENEBS_IO_KUBE_CONFIG# value: "/home/ubuntu/.kube/config"- name: NODE_NAMEvalueFrom:fieldRef:fieldPath: spec.nodeName- name: OPENEBS_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace# OPENEBS_MAYA_SERVICE_NAME provides the maya-apiserver K8s service name,# that provisioner should forward the volume create/delete requests.# If not present, "maya-apiserver-service" will be used for lookup.# This is supported for openebs provisioner version 0.5.3-RC1 onwards#- name: OPENEBS_MAYA_SERVICE_NAME# value: "maya-apiserver-apiservice"# LEADER_ELECTION_ENABLED is used to enable/disable leader election. By default# leader election is enabled.#- name: LEADER_ELECTION_ENABLED# value: "true"# Process name used for matching is limited to the 15 characters# present in the pgrep output.# So fullname can't be used here with pgrep (>15 chars).A regular expression# that matches the entire command name has to specified.# Anchor `^` : matches any string that starts with `openebs-provis`# `.*`: matches any string that has `openebs-provis` followed by zero or more charlivenessProbe:exec:command:- sh- -c- test `pgrep -c "^openebs-provisi.*"` = 1initialDelaySeconds: 30periodSeconds: 60
---
apiVersion: apps/v1
kind: Deployment
metadata:name: openebs-snapshot-operatornamespace: openebslabels:name: openebs-snapshot-operatoropenebs.io/component-name: openebs-snapshot-operatoropenebs.io/version: dev
spec:selector:matchLabels:name: openebs-snapshot-operatoropenebs.io/component-name: openebs-snapshot-operatorreplicas: 1strategy:type: Recreatetemplate:metadata:labels:name: openebs-snapshot-operatoropenebs.io/component-name: openebs-snapshot-operatoropenebs.io/version: devspec:serviceAccountName: openebs-maya-operatorcontainers:- name: snapshot-controllerimage: openebs/snapshot-controller:ciimagePullPolicy: IfNotPresentenv:- name: OPENEBS_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace# Process name used for matching is limited to the 15 characters# present in the pgrep output.# So fullname can't be used here with pgrep (>15 chars).A regular expression# that matches the entire command name has to specified.# Anchor `^` : matches any string that starts with `snapshot-contro`# `.*`: matches any string that has `snapshot-contro` followed by zero or more charlivenessProbe:exec:command:- sh- -c- test `pgrep -c "^snapshot-contro.*"` = 1initialDelaySeconds: 30periodSeconds: 60# OPENEBS_MAYA_SERVICE_NAME provides the maya-apiserver K8s service name,# that snapshot controller should forward the snapshot create/delete requests.# If not present, "maya-apiserver-service" will be used for lookup.# This is supported for openebs provisioner version 0.5.3-RC1 onwards#- name: OPENEBS_MAYA_SERVICE_NAME# value: "maya-apiserver-apiservice"- name: snapshot-provisionerimage: openebs/snapshot-provisioner:ciimagePullPolicy: IfNotPresentenv:- name: OPENEBS_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace# OPENEBS_MAYA_SERVICE_NAME provides the maya-apiserver K8s service name,# that snapshot provisioner should forward the clone create/delete requests.# If not present, "maya-apiserver-service" will be used for lookup.# This is supported for openebs provisioner version 0.5.3-RC1 onwards#- name: OPENEBS_MAYA_SERVICE_NAME# value: "maya-apiserver-apiservice"# LEADER_ELECTION_ENABLED is used to enable/disable leader election. By default# leader election is enabled.#- name: LEADER_ELECTION_ENABLED# value: "true"# Process name used for matching is limited to the 15 characters# present in the pgrep output.# So fullname can't be used here with pgrep (>15 chars).A regular expression# that matches the entire command name has to specified.# Anchor `^` : matches any string that starts with `snapshot-provis`# `.*`: matches any string that has `snapshot-provis` followed by zero or more charlivenessProbe:exec:command:- sh- -c- test `pgrep -c "^snapshot-provis.*"` = 1initialDelaySeconds: 30periodSeconds: 60
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:annotations:controller-gen.kubebuilder.io/version: v0.5.0creationTimestamp: nullname: blockdevices.openebs.io
spec:group: openebs.ionames:kind: BlockDevicelistKind: BlockDeviceListplural: blockdevicesshortNames:- bdsingular: blockdevicescope: Namespacedversions:- additionalPrinterColumns:- jsonPath: .spec.nodeAttributes.nodeNamename: NodeNametype: string- jsonPath: .spec.pathname: Pathpriority: 1type: string- jsonPath: .spec.filesystem.fsTypename: FSTypepriority: 1type: string- jsonPath: .spec.capacity.storagename: Sizetype: string- jsonPath: .status.claimStatename: ClaimStatetype: string- jsonPath: .status.statename: Statustype: string- jsonPath: .metadata.creationTimestampname: Agetype: datename: v1alpha1schema:openAPIV3Schema:description: BlockDevice is the Schema for the blockdevices APIproperties:apiVersion:description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'type: stringkind:description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'type: stringmetadata:type: objectspec:description: DeviceSpec defines the properties and runtime status of a BlockDeviceproperties:aggregateDevice:description: AggregateDevice was intended to store the hierarchical information in cases of LVM. However this is currently not implemented and may need to be re-looked into for better design. To be deprecatedtype: stringcapacity:description: Capacityproperties:logicalSectorSize:description: LogicalSectorSize is blockdevice logical-sector size in bytesformat: int32type: integerphysicalSectorSize:description: PhysicalSectorSize is blockdevice physical-Sector size in bytesformat: int32type: integerstorage:description: Storage is the blockdevice capacity in bytesformat: int64type: integerrequired:- storagetype: objectclaimRef:description: ClaimRef is the reference to the BDC which has claimed this BDproperties:apiVersion:description: API version of the referent.type: stringfieldPath:description: 'If referring to a piece of an object instead of an entire object, this string should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2]. For example, if the object reference is to a container within a pod, this would take on a value like: "spec.containers{name}" (where "name" refers to the name of the container that triggered the event) or if no container name is specified "spec.containers[2]" (container with index 2 in this pod). This syntax is chosen only to have some well-defined way of referencing a part of an object. TODO: this design is not final and this field is subject to change in the future.'type: stringkind:description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'type: stringname:description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'type: stringnamespace:description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'type: stringresourceVersion:description: 'Specific resourceVersion to which this reference is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency'type: stringuid:description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'type: stringtype: objectdetails:description: Details contain static attributes of BD like model,serial, and so forthproperties:compliance:description: Compliance is standards/specifications version implemented by device firmware such as SPC-1, SPC-2, etctype: stringdeviceType:description: DeviceType represents the type of device like sparse, disk, partition, lvm, cryptenum:- disk- partition- sparse- loop- lvm- crypt- dm- mpathtype: stringdriveType:description: DriveType is the type of backing drive, HDD/SSDenum:- HDD- SSD- Unknown- ""type: stringfirmwareRevision:description: FirmwareRevision is the disk firmware revisiontype: stringhardwareSectorSize:description: HardwareSectorSize is the hardware sector size in bytesformat: int32type: integerlogicalBlockSize:description: LogicalBlockSize is the logical block size in bytes reported by /sys/class/block/sda/queue/logical_block_sizeformat: int32type: integermodel:description: Model is model of disktype: stringphysicalBlockSize:description: PhysicalBlockSize is the physical block size in bytes reported by /sys/class/block/sda/queue/physical_block_sizeformat: int32type: integerserial:description: Serial is serial number of disktype: stringvendor:description: Vendor is vendor of disktype: stringtype: objectdevlinks:description: DevLinks contains soft links of a block device like /dev/by-id/... /dev/by-uuid/...items:description: DeviceDevLink holds the mapping between type and links like by-id type or by-path type linkproperties:kind:description: Kind is the type of link like by-id or by-path.enum:- by-id- by-pathtype: stringlinks:description: Links are the soft linksitems:type: stringtype: arraytype: objecttype: arrayfilesystem:description: FileSystem contains mountpoint and filesystem typeproperties:fsType:description: Type represents the FileSystem type of the block devicetype: stringmountPoint:description: MountPoint represents the mountpoint of the block device.type: stringtype: objectnodeAttributes:description: NodeAttributes has the details of the node on which BD is attachedproperties:nodeName:description: NodeName is the name of the Kubernetes node resource on which the device is attachedtype: stringtype: objectparentDevice:description: "ParentDevice was intended to store the UUID of the parent Block Device as is the case for partitioned block devices. \n For example: /dev/sda is the parent for /dev/sda1 To be deprecated"type: stringpartitioned:description: Partitioned represents if BlockDevice has partitions or not (Yes/No) Currently always default to No. To be deprecatedenum:- "Yes"- "No"type: stringpath:description: Path contain devpath (e.g. /dev/sdb)type: stringrequired:- capacity- devlinks- nodeAttributes- pathtype: objectstatus:description: DeviceStatus defines the observed state of BlockDeviceproperties:claimState:description: ClaimState represents the claim state of the block deviceenum:- Claimed- Unclaimed- Releasedtype: stringstate:description: State is the current state of the blockdevice (Active/Inactive/Unknown)enum:- Active- Inactive- Unknowntype: stringrequired:- claimState- statetype: objecttype: objectserved: truestorage: truesubresources: {}
status:acceptedNames:kind: ""plural: ""conditions: []storedVersions: []---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:annotations:controller-gen.kubebuilder.io/version: v0.5.0creationTimestamp: nullname: blockdeviceclaims.openebs.io
spec:group: openebs.ionames:kind: BlockDeviceClaimlistKind: BlockDeviceClaimListplural: blockdeviceclaimsshortNames:- bdcsingular: blockdeviceclaimscope: Namespacedversions:- additionalPrinterColumns:- jsonPath: .spec.blockDeviceNamename: BlockDeviceNametype: string- jsonPath: .status.phasename: Phasetype: string- jsonPath: .metadata.creationTimestampname: Agetype: datename: v1alpha1schema:openAPIV3Schema:description: BlockDeviceClaim is the Schema for the blockdeviceclaims APIproperties:apiVersion:description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'type: stringkind:description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'type: stringmetadata:type: objectspec:description: DeviceClaimSpec defines the request details for a BlockDeviceproperties:blockDeviceName:description: BlockDeviceName is the reference to the block-device backing this claimtype: stringblockDeviceNodeAttributes:description: BlockDeviceNodeAttributes is the attributes on the node from which a BD should be selected for this claim. It can include nodename, failure domain etc.properties:hostName:description: HostName represents the hostname of the Kubernetes node resource where the BD should be presenttype: stringnodeName:description: NodeName represents the name of the Kubernetes node resource where the BD should be presenttype: stringtype: objectdeviceClaimDetails:description: Details of the device to be claimedproperties:allowPartition:description: AllowPartition represents whether to claim a full block device or a device that is a partitiontype: booleanblockVolumeMode:description: 'BlockVolumeMode represents whether to claim a device in Block mode or Filesystem mode. These are use cases of BlockVolumeMode: 1) Not specified: VolumeMode check will not be effective 2) VolumeModeBlock: BD should not have any filesystem or mountpoint 3) VolumeModeFileSystem: BD should have a filesystem and mountpoint. If DeviceFormat is specified then the format should match with the FSType in BD'type: stringformatType:description: Format of the device required, eg:ext4, xfstype: stringtype: objectdeviceType:description: DeviceType represents the type of drive like SSD, HDD etc.,nullable: truetype: stringhostName:description: Node name from where blockdevice has to be claimed. To be deprecated. Use NodeAttributes.HostName insteadtype: stringresources:description: Resources will help with placing claims on Capacity, IOPSproperties:requests:additionalProperties:anyOf:- type: integer- type: stringpattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$x-kubernetes-int-or-string: truedescription: 'Requests describes the minimum resources required. eg: if storage resource of 10G is requested minimum capacity of 10G should be available TODO for validating'type: objectrequired:- requeststype: objectselector:description: Selector is used to find block devices to be considered for claimingproperties:matchExpressions:description: matchExpressions is a list of label selector requirements. The requirements are ANDed.items:description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.properties:key:description: key is the label key that the selector applies to.type: stringoperator:description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.type: stringvalues:description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.items:type: stringtype: arrayrequired:- key- operatortype: objecttype: arraymatchLabels:additionalProperties:type: stringdescription: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.type: objecttype: objecttype: objectstatus:description: DeviceClaimStatus defines the observed state of BlockDeviceClaimproperties:phase:description: Phase represents the current phase of the claimtype: stringrequired:- phasetype: objecttype: objectserved: truestorage: truesubresources: {}
status:acceptedNames:kind: ""plural: ""conditions: []storedVersions: []
---
# This is the node-disk-manager related config.
# It can be used to customize the disks probes and filters
apiVersion: v1
kind: ConfigMap
metadata:name: openebs-ndm-confignamespace: openebslabels:openebs.io/component-name: ndm-config
data:# udev-probe is default or primary probe which should be enabled to run ndm# filterconfigs contails configs of filters - in their form fo include# and exclude comma separated stringsnode-disk-manager.config: |probeconfigs:- key: udev-probename: udev probestate: true- key: seachest-probename: seachest probestate: false- key: smart-probename: smart probestate: truefilterconfigs:- key: os-disk-exclude-filtername: os disk exclude filterstate: trueexclude: "/,/etc/hosts,/boot"- key: vendor-filtername: vendor filterstate: trueinclude: ""exclude: "CLOUDBYT,OpenEBS"- key: path-filtername: path filterstate: trueinclude: ""exclude: "/dev/loop,/dev/fd0,/dev/sr0,/dev/ram,/dev/md,/dev/dm-,/dev/rbd,/dev/zd"
---
apiVersion: apps/v1
kind: DaemonSet
metadata:name: openebs-ndmnamespace: openebslabels:name: openebs-ndmopenebs.io/component-name: ndmopenebs.io/version: dev
spec:selector:matchLabels:name: openebs-ndmopenebs.io/component-name: ndmupdateStrategy:type: RollingUpdatetemplate:metadata:labels:name: openebs-ndmopenebs.io/component-name: ndmopenebs.io/version: devspec:# By default the node-disk-manager will be run on all kubernetes nodes# If you would like to limit this to only some nodes, say the nodes# that have storage attached, you could label those node and use# nodeSelector.## e.g. label the storage nodes with - "openebs.io/nodegroup"="storage-node"# kubectl label node <node-name> "openebs.io/nodegroup"="storage-node"#nodeSelector:# "openebs.io/nodegroup": "storage-node"serviceAccountName: openebs-maya-operatorhostNetwork: true# host PID is used to check status of iSCSI Service when the NDM# API service is enabled#hostPID: truecontainers:- name: node-disk-managerimage: openebs/node-disk-manager:ciargs:- -v=4# The feature-gate is used to enable the new UUID algorithm.- --feature-gates="GPTBasedUUID"# Detect mount point and filesystem changes wihtout restart.# Uncomment the line below to enable the feature.# --feature-gates="MountChangeDetection"# The feature gate is used to start the gRPC API service. The gRPC server# starts at 9115 port by default. This feature is currently in Alpha state# - --feature-gates="APIService"# The feature gate is used to enable NDM, to create blockdevice resources# for unused partitions on the OS disk# - --feature-gates="UseOSDisk"imagePullPolicy: IfNotPresentsecurityContext:privileged: truevolumeMounts:- name: configmountPath: /host/node-disk-manager.configsubPath: node-disk-manager.configreadOnly: true- name: udevmountPath: /run/udev- name: procmountmountPath: /host/procreadOnly: true- name: devmountmountPath: /dev- name: basepathmountPath: /var/openebs/ndm- name: sparsepathmountPath: /var/openebs/sparseenv:# namespace in which NDM is installed will be passed to NDM Daemonset# as environment variable- name: NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace# pass hostname as env variable using downward API to the NDM container- name: NODE_NAMEvalueFrom:fieldRef:fieldPath: spec.nodeName# specify the directory where the sparse files need to be created.# if not specified, then sparse files will not be created.- name: SPARSE_FILE_DIRvalue: "/var/openebs/sparse"# Size(bytes) of the sparse file to be created.- name: SPARSE_FILE_SIZEvalue: "10737418240"# Specify the number of sparse files to be created- name: SPARSE_FILE_COUNTvalue: "1"# Process name used for matching is limited to the 15 characters# present in the pgrep output.# So fullname can be used here with pgrep (cmd is < 15 chars).livenessProbe:exec:command:- pgrep- "ndm"initialDelaySeconds: 30periodSeconds: 60volumes:- name: configconfigMap:name: openebs-ndm-config- name: udevhostPath:path: /run/udevtype: Directory# mount /proc (to access mount file of process 1 of host) inside container# to read mount-point of disks and partitions- name: procmounthostPath:path: /proctype: Directory- name: devmount# the /dev directory is mounted so that we have access to the devices that# are connected at runtime of the pod.hostPath:path: /devtype: Directory- name: basepathhostPath:path: /var/openebs/ndmtype: DirectoryOrCreate- name: sparsepathhostPath:path: /var/openebs/sparse
---
apiVersion: apps/v1
kind: Deployment
metadata:name: openebs-ndm-operatornamespace: openebslabels:name: openebs-ndm-operatoropenebs.io/component-name: ndm-operatoropenebs.io/version: dev
spec:selector:matchLabels:name: openebs-ndm-operatoropenebs.io/component-name: ndm-operatorreplicas: 1strategy:type: Recreatetemplate:metadata:labels:name: openebs-ndm-operatoropenebs.io/component-name: ndm-operatoropenebs.io/version: devspec:serviceAccountName: openebs-maya-operatorcontainers:- name: node-disk-operatorimage: openebs/node-disk-operator:ciimagePullPolicy: IfNotPresentenv:- name: WATCH_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name# the service account of the ndm-operator pod- name: SERVICE_ACCOUNTvalueFrom:fieldRef:fieldPath: spec.serviceAccountName- name: OPERATOR_NAMEvalue: "node-disk-operator"- name: CLEANUP_JOB_IMAGEvalue: "openebs/linux-utils:ci"livenessProbe:httpGet:path: /healthzport: 8585initialDelaySeconds: 15periodSeconds: 20readinessProbe:httpGet:path: /readyzport: 8585initialDelaySeconds: 5periodSeconds: 10
---
apiVersion: apps/v1
kind: Deployment
metadata:name: openebs-admission-servernamespace: openebslabels:app: admission-webhookopenebs.io/component-name: admission-webhookopenebs.io/version: dev
spec:replicas: 1strategy:type: RecreaterollingUpdate: nullselector:matchLabels:app: admission-webhooktemplate:metadata:labels:app: admission-webhookopenebs.io/component-name: admission-webhookopenebs.io/version: devspec:serviceAccountName: openebs-maya-operatorcontainers:- name: admission-webhookimage: openebs/admission-server:ciimagePullPolicy: IfNotPresentargs:- -alsologtostderr- -v=2- 2>&1env:- name: OPENEBS_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: ADMISSION_WEBHOOK_NAMEvalue: "openebs-admission-server"- name: ADMISSION_WEBHOOK_FAILURE_POLICYvalue: "Fail"# Process name used for matching is limited to the 15 characters# present in the pgrep output.# So fullname can't be used here with pgrep (>15 chars).A regular expression# Anchor `^` : matches any string that starts with `admission-serve`# `.*`: matche any string that has `admission-serve` followed by zero or more char# that matches the entire command name has to specified.livenessProbe:exec:command:- sh- -c- test `pgrep -c "^admission-serve.*"` = 1initialDelaySeconds: 30periodSeconds: 60
---
apiVersion: apps/v1
kind: Deployment
metadata:name: openebs-localpv-provisionernamespace: openebslabels:name: openebs-localpv-provisioneropenebs.io/component-name: openebs-localpv-provisioneropenebs.io/version: dev
spec:selector:matchLabels:name: openebs-localpv-provisioneropenebs.io/component-name: openebs-localpv-provisionerreplicas: 1strategy:type: Recreatetemplate:metadata:labels:name: openebs-localpv-provisioneropenebs.io/component-name: openebs-localpv-provisioneropenebs.io/version: devspec:serviceAccountName: openebs-maya-operatorcontainers:- name: openebs-provisioner-hostpathimagePullPolicy: IfNotPresentimage: openebs/provisioner-localpv:cienv:# OPENEBS_IO_K8S_MASTER enables openebs provisioner to connect to K8s# based on this address. This is ignored if empty.# This is supported for openebs provisioner version 0.5.2 onwards#- name: OPENEBS_IO_K8S_MASTER# value: "http://10.128.0.12:8080"# OPENEBS_IO_KUBE_CONFIG enables openebs provisioner to connect to K8s# based on this config. This is ignored if empty.# This is supported for openebs provisioner version 0.5.2 onwards#- name: OPENEBS_IO_KUBE_CONFIG# value: "/home/ubuntu/.kube/config"- name: NODE_NAMEvalueFrom:fieldRef:fieldPath: spec.nodeName- name: OPENEBS_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace# OPENEBS_SERVICE_ACCOUNT provides the service account of this pod as# environment variable- name: OPENEBS_SERVICE_ACCOUNTvalueFrom:fieldRef:fieldPath: spec.serviceAccountName- name: OPENEBS_IO_ENABLE_ANALYTICSvalue: "false"- name: OPENEBS_IO_INSTALLER_TYPEvalue: "openebs-operator"- name: OPENEBS_IO_HELPER_IMAGEvalue: "openebs/linux-utils:2.3.0"# LEADER_ELECTION_ENABLED is used to enable/disable leader election. By default# leader election is enabled.#- name: LEADER_ELECTION_ENABLED# value: "true"# Process name used for matching is limited to the 15 characters# present in the pgrep output.# So fullname can't be used here with pgrep (>15 chars).A regular expression# that matches the entire command name has to specified.# Anchor `^` : matches any string that starts with `provisioner-loc`# `.*`: matches any string that has `provisioner-loc` followed by zero or more charlivenessProbe:exec:command:- sh- -c- test `pgrep -c "^provisioner-loc.*"` = 1initialDelaySeconds: 30periodSeconds: 60
---6.3 查看资源的状态
[root@k8s-master kubesphere]# kubectl get all -n openebs
NAME READY STATUS RESTARTS AGE
pod/openebs-localpv-provisioner-787859df9f-q8tzs 1/1 Running 0 115m
pod/openebs-ndm-2rtg8 1/1 Running 0 115m
pod/openebs-ndm-cluster-exporter-65b6cd58c5-nmqsr 1/1 Running 0 115m
pod/openebs-ndm-l76rv 1/1 Running 0 115m
pod/openebs-ndm-node-exporter-frr6v 1/1 Running 0 115m
pod/openebs-ndm-node-exporter-scvqf 1/1 Running 0 115m
pod/openebs-ndm-operator-75b8886666-bhkbv 1/1 Running 0 115mNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/openebs-ndm-cluster-exporter-service ClusterIP None <none> 9100/TCP 115m
service/openebs-ndm-node-exporter-service ClusterIP None <none> 9101/TCP 115mNAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/openebs-ndm 2 2 2 2 2 <none> 115m
daemonset.apps/openebs-ndm-node-exporter 2 2 2 2 2 <none> 115mNAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/openebs-localpv-provisioner 1/1 1 1 115m
deployment.apps/openebs-ndm-cluster-exporter 1/1 1 1 115m
deployment.apps/openebs-ndm-operator 1/1 1 1 115mNAME DESIRED CURRENT READY AGE
replicaset.apps/openebs-localpv-provisioner-787859df9f 1 1 1 115m
replicaset.apps/openebs-ndm-cluster-exporter-65b6cd58c5 1 1 1 115m
replicaset.apps/openebs-ndm-operator-75b8886666 1 1 1 115m
6.4 查看storageclass存储器状态
[root@k8s-master kubesphere]# kubectl get storageclasses
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
managed-nfs-storage fuseim.pri/ifs Delete Immediate false 4h43m
openebs-device openebs.io/local Delete WaitForFirstConsumer false 118m
openebs-hostpath openebs.io/local Delete WaitForFirstConsumer false 118m
这篇关于【k8s存储--使用OpenEBS做持久化存储】的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！
