Cgroups是什么

本文主要是介绍Cgroups是什么，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！

原文地址：https://www.cnblogs.com/lisperl/archive/2013/01/14/2860353.html

作者：王喆锋 zhefwang@gmail.com（侵删）

Cgroups是什么？

Cgroups是control groups的缩写，是Linux内核提供的一种可以限制、记录、隔离进程组（process groups）所使用的物理资源（如：cpu,memory,IO等等）的机制。最初由google的工程师提出，后来被整合进Linux内核。Cgroups也是LXC为实现虚拟化所使用的资源管理手段，可以说没有cgroups就没有LXC。

Cgroups可以做什么？

Cgroups最初的目标是为资源管理提供的一个统一的框架，既整合现有的cpuset等子系统，也为未来开发新的子系统提供接口。现在的cgroups适用于多种应用场景，从单个进程的资源控制，到实现操作系统层次的虚拟化（OS Level Virtualization）。Cgroups提供了一下功能：

1.限制进程组可以使用的资源数量（Resource limiting ）。比如：memory子系统可以为进程组设定一个memory使用上限，一旦进程组使用的内存达到限额再申请内存，就会出发OOM（out of memory）。
2.进程组的优先级控制（Prioritization ）。比如：可以使用cpu子系统为某个进程组分配特定cpu share。
3.记录进程组使用的资源数量（Accounting ）。比如：可以使用cpuacct子系统记录某个进程组使用的cpu时间
4.进程组隔离（isolation）。比如：使用ns子系统可以使不同的进程组使用不同的namespace，以达到隔离的目的，不同的进程组有各自的进程、网络、文件系统挂载空间。
5.进程组控制（control）。比如：使用freezer子系统可以将进程组挂起和恢复。

Cgroups相关概念及其关系

相互关系

1.每次在系统中创建新层级时，该系统中的所有任务都是那个层级的默认 cgroup（我们称之为 root cgroup ，此cgroup在创建层级时自动创建，后面在该层级中创建的cgroup都是此cgroup的后代）的初始成员。
2.一个子系统最多只能附加到一个层级。
3.一个层级可以附加多个子系统。
4.一个任务可以是多个cgroup的成员，但是这些cgroup必须在不同的层级。
5.系统中的进程（任务）创建子进程（任务）时，该子任务自动成为其父进程所在 cgroup 的成员。然后可根据需要将该子任务移动到不同的 cgroup 中，但开始时它总是继承其父任务的cgroup。

Cgroups子系统介绍

blkio -- 这个子系统为块设备设定输入/输出限制，比如物理设备（磁盘，固态硬盘，USB 等等）。
cpu -- 这个子系统使用调度程序提供对 CPU 的 cgroup 任务访问。
cpuacct -- 这个子系统自动生成 cgroup 中任务所使用的 CPU 报告。
cpuset -- 这个子系统为 cgroup 中的任务分配独立 CPU（在多核系统）和内存节点。
devices -- 这个子系统可允许或者拒绝 cgroup 中的任务访问设备。
freezer -- 这个子系统挂起或者恢复 cgroup 中的任务。
memory -- 这个子系统设定 cgroup 中任务使用的内存限制，并自动生成由那些任务使用的内存资源报告。
net_cls -- 这个子系统使用等级识别符（classid）标记网络数据包，可允许 Linux 流量控制程序（tc）识别从具体 cgroup 中生成的数据包。
ns -- 名称空间子系统。

Cgroups如何实现

数据结构

我们从进程出发来剖析cgroups相关数据结构之间的关系。在Linux中，管理进程的数据结构是task_struct，其中与cgroups有关的：

#ifdef CONFIG_CGROUPS/* Control Group info protected by css_set_lock: */struct css_set __rcu		*cgroups;/* cg_list protected by css_set_lock and tsk->alloc_lock: */struct list_head		cg_list;
#endif
#ifdef CONFIG_CGROUPS/* disallow userland-initiated cgroup migration */unsigned			no_cgroup_migration:1;
#endif

其中cgroups指针指向了一个css_set结构，而css_set存储了与进程相关的cgroups信息。cg_list是一个嵌入的list_head结构，用于将连到同一个css_set的进程组织成一个链表。下面我们来看css_set的结构：

/** A css_set is a structure holding pointers to a set of* cgroup_subsys_state objects. This saves space in the task struct* object and speeds up fork()/exit(), since a single inc/dec and a* list_add()/del() can bump the reference count on the entire cgroup* set for a task.*/
struct css_set {/** Set of subsystem states, one for each subsystem. This array is* immutable after creation apart from the init_css_set during* subsystem registration (at boot time).*/struct cgroup_subsys_state *subsys[CGROUP_SUBSYS_COUNT];/* reference count */refcount_t refcount;/** For a domain cgroup, the following points to self.  If threaded,* to the matching cset of the nearest domain ancestor.  The* dom_cset provides access to the domain cgroup and its csses to* which domain level resource consumptions should be charged.*/struct css_set *dom_cset;/* the default cgroup associated with this css_set */struct cgroup *dfl_cgrp;/* internal task count, protected by css_set_lock */int nr_tasks;/** Lists running through all tasks using this cgroup group.* mg_tasks lists tasks which belong to this cset but are in the* process of being migrated out or in.  Protected by* css_set_rwsem, but, during migration, once tasks are moved to* mg_tasks, it can be read safely while holding cgroup_mutex.*/struct list_head tasks;struct list_head mg_tasks;/* all css_task_iters currently walking this cset */struct list_head task_iters;/** On the default hierarhcy, ->subsys[ssid] may point to a css* attached to an ancestor instead of the cgroup this css_set is* associated with.  The following node is anchored at* ->subsys[ssid]->cgroup->e_csets[ssid] and provides a way to* iterate through all css's attached to a given cgroup.*/struct list_head e_cset_node[CGROUP_SUBSYS_COUNT];/* all threaded csets whose ->dom_cset points to this cset */struct list_head threaded_csets;struct list_head threaded_csets_node;/** List running through all cgroup groups in the same hash* slot. Protected by css_set_lock*/struct hlist_node hlist;/** List of cgrp_cset_links pointing at cgroups referenced from this* css_set.  Protected by css_set_lock.*/struct list_head cgrp_links;/** List of csets participating in the on-going migration either as* source or destination.  Protected by cgroup_mutex.*/struct list_head mg_preload_node;struct list_head mg_node;/** If this cset is acting as the source of migration the following* two fields are set.  mg_src_cgrp and mg_dst_cgrp are* respectively the source and destination cgroups of the on-going* migration.  mg_dst_cset is the destination cset the target tasks* on this cset should be migrated to.  Protected by cgroup_mutex.*/struct cgroup *mg_src_cgrp;struct cgroup *mg_dst_cgrp;struct css_set *mg_dst_cset;/* dead and being drained, ignore for migration */bool dead;/* For RCU-protected deletion */struct rcu_head rcu_head;
};

其中refcount是该css_set的引用数，因为一个css_set可以被多个进程共用，只要这些进程的cgroups信息相同，比如：在所有已创建的层级里面都在同一个cgroup里的进程。 hlist是嵌入的hlist_node，用于把所有css_set组织成一个hash表，这样内核可以快速查找特定的css_set。

tasks指向所有连到此css_set的进程连成的链表。 cg_links指向一个由struct cg_cgroup_link连成的链表。 Subsys是一个指针数组，存储一组指向cgroup_subsys_state的指针。一个cgroup_subsys_state就是进程与一个特定子系统相关的信息。通过这个指针数组，进程就可以获得相应的cgroups控制信息了。下面我们就来看cgroup_subsys_state的结构：


/** Per-subsystem/per-cgroup state maintained by the system.  This is the* fundamental structural building block that controllers deal with.** Fields marked with "PI:" are public and immutable and may be accessed* directly without synchronization.*/
struct cgroup_subsys_state {/* PI: the cgroup that this css is attached to */struct cgroup *cgroup;/* PI: the cgroup subsystem that this css is attached to */struct cgroup_subsys *ss;/* reference count - access via css_[try]get() and css_put() */struct percpu_ref refcnt;/* siblings list anchored at the parent's ->children */struct list_head sibling;struct list_head children;/* flush target list anchored at cgrp->rstat_css_list */struct list_head rstat_css_node;/** PI: Subsys-unique ID.  0 is unused and root is always 1.  The* matching css can be looked up using css_from_id().*/int id;unsigned int flags;/** Monotonically increasing unique serial number which defines a* uniform order among all csses.  It's guaranteed that all* ->children lists are in the ascending order of ->serial_nr and* used to allow interrupting and resuming iterations.*/u64 serial_nr;/** Incremented by online self and children.  Used to guarantee that* parents are not offlined before their children.*/atomic_t online_cnt;/* percpu_ref killing and RCU release */struct work_struct destroy_work;struct rcu_work destroy_rwork;/** PI: the parent css.	Placed here for cache proximity to following* fields of the containing structure.*/struct cgroup_subsys_state *parent;
};

cgroup指针指向了一个cgroup结构，也就是进程属于的cgroup。进程受到子系统的控制，实际上是通过加入到特定的cgroup实现的，因为cgroup在特定的层级上，而子系统又是附加到曾经上的。通过以上三个结构，进程就可以和cgroup连接起来了：task_struct->css_set->cgroup_subsys_state->cgroup。

下面我们再来看cgroup的结构：


struct cgroup {/* self css with NULL ->ss, points back to this cgroup */struct cgroup_subsys_state self;unsigned long flags;		/* "unsigned long" so bitops work *//** idr allocated in-hierarchy ID.** ID 0 is not used, the ID of the root cgroup is always 1, and a* new cgroup will be assigned with a smallest available ID.** Allocating/Removing ID must be protected by cgroup_mutex.*/int id;/** The depth this cgroup is at.  The root is at depth zero and each* step down the hierarchy increments the level.  This along with* ancestor_ids[] can determine whether a given cgroup is a* descendant of another without traversing the hierarchy.*/int level;/* Maximum allowed descent tree depth */int max_depth;/** Keep track of total numbers of visible and dying descent cgroups.* Dying cgroups are cgroups which were deleted by a user,* but are still existing because someone else is holding a reference.* max_descendants is a maximum allowed number of descent cgroups.*/int nr_descendants;int nr_dying_descendants;int max_descendants;/** Each non-empty css_set associated with this cgroup contributes* one to nr_populated_csets.  The counter is zero iff this cgroup* doesn't have any tasks.** All children which have non-zero nr_populated_csets and/or* nr_populated_children of their own contribute one to either* nr_populated_domain_children or nr_populated_threaded_children* depending on their type.  Each counter is zero iff all cgroups* of the type in the subtree proper don't have any tasks.*/int nr_populated_csets;int nr_populated_domain_children;int nr_populated_threaded_children;int nr_threaded_children;	/* # of live threaded child cgroups */struct kernfs_node *kn;		/* cgroup kernfs entry */struct cgroup_file procs_file;	/* handle for "cgroup.procs" */struct cgroup_file events_file;	/* handle for "cgroup.events" *//** The bitmask of subsystems enabled on the child cgroups.* ->subtree_control is the one configured through* "cgroup.subtree_control" while ->child_ss_mask is the effective* one which may have more subsystems enabled.  Controller knobs* are made available iff it's enabled in ->subtree_control.*/u16 subtree_control;u16 subtree_ss_mask;u16 old_subtree_control;u16 old_subtree_ss_mask;/* Private pointers for each registered subsystem */struct cgroup_subsys_state __rcu *subsys[CGROUP_SUBSYS_COUNT];struct cgroup_root *root;/** List of cgrp_cset_links pointing at css_sets with tasks in this* cgroup.  Protected by css_set_lock.*/struct list_head cset_links;/** On the default hierarchy, a css_set for a cgroup with some* susbsys disabled will point to css's which are associated with* the closest ancestor which has the subsys enabled.  The* following lists all css_sets which point to this cgroup's css* for the given subsystem.*/struct list_head e_csets[CGROUP_SUBSYS_COUNT];/** If !threaded, self.  If threaded, it points to the nearest* domain ancestor.  Inside a threaded subtree, cgroups are exempt* from process granularity and no-internal-task constraint.* Domain level resource consumptions which aren't tied to a* specific task are charged to the dom_cgrp.*/struct cgroup *dom_cgrp;struct cgroup *old_dom_cgrp;		/* used while enabling threaded *//* per-cpu recursive resource statistics */struct cgroup_rstat_cpu __percpu *rstat_cpu;struct list_head rstat_css_list;/* cgroup basic resource statistics */struct cgroup_base_stat pending_bstat;	/* pending from children */struct cgroup_base_stat bstat;struct prev_cputime prev_cputime;	/* for printing out cputime *//** list of pidlists, up to two for each namespace (one for procs, one* for tasks); created on demand.*/struct list_head pidlists;struct mutex pidlist_mutex;/* used to wait for offlining of csses */wait_queue_head_t offline_waitq;/* used to schedule release agent */struct work_struct release_agent_work;/* used to track pressure stalls */struct psi_group psi;/* used to store eBPF programs */struct cgroup_bpf bpf;/* If there is block congestion on this cgroup. */atomic_t congestion_count;/* ids of the ancestors at each level including self */int ancestor_ids[];
};

sibling,children和parent三个嵌入的list_head负责将同一层级的cgroup连接成一颗cgroup树。 subsys是一个指针数组，存储一组指向cgroup_subsys_state的指针。这组指针指向了此cgroup跟各个子系统相关的信息，这个跟css_set中的道理是一样的。
root指向了一个cgroupfs_root的结构，就是cgroup所在的层级对应的结构体。这样以来，之前谈到的几个cgroups概念就全部联系起来了。 top_cgroup指向了所在层级的根cgroup，也就是创建层级时自动创建的那个cgroup。 css_set指向一个由struct cg_cgroup_link连成的链表，跟css_set中cg_links一样。
下面我们来分析一个css_set和cgroup之间的关系。我们先看一下 cg_cgroup_link的结构