PostgreSQL技术内幕5：PostgreSQL存储引擎从磁盘到内存的读取

本文主要是介绍PostgreSQL技术内幕5：PostgreSQL存储引擎从磁盘到内存的读取，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！

0.简介

本篇内容介绍PG从磁盘到内存的加载流程，经过那些层级，各层级作用以及源码分析，主要包括共享缓存（Buffer），存储管理器，磁盘管理器，虚拟文件管理器以及部分物理文件介绍。

1.背景知识

1.1 计算机存储结构

计算机存储层级如下图，速度层级越往下越慢。

1.2 数据库常见的磁盘和内存访问形式

常见的访问方式为在磁盘以page形式存储，在内存中存储到Buffer Pool中，如下图：

2. 整体获取层次

PG整体获取一个元组涉及的层级如图，下面将对每一层进行详细说明：

3.元组介绍

Tuple结构如下

Header信息：

struct HeapTupleHeaderData
{union{HeapTupleFields t_heap;DatumTupleFields t_datum;}           t_choice;ItemPointerData t_ctid;     /* current TID of this or newer tuple (or a* speculative insertion token) *//* Fields below here must match MinimalTupleData! */#define FIELDNO_HEAPTUPLEHEADERDATA_INFOMASK2 2uint16      t_infomask2;    /* number of attributes + various flags */#define FIELDNO_HEAPTUPLEHEADERDATA_INFOMASK 3uint16      t_infomask;     /* various flag bits, see below */#define FIELDNO_HEAPTUPLEHEADERDATA_HOFF 4uint8       t_hoff;         /* sizeof header incl. bitmap, padding *//* ^ - 23 bytes - ^ */#define FIELDNO_HEAPTUPLEHEADERDATA_BITS 5bits8       t_bits[FLEXIBLE_ARRAY_MEMBER];  /* bitmap of NULLs *//* MORE DATA FOLLOWS AT END OF STRUCT */
};

4. Buffer管理

4.1 Buffer组成

PG中buffer 由三部分组成，如下图，创建函数可见CreateSharedMemoryAndSemaphores。

1）Buffer table layer:一个hash表，记录buffer描述符和buff pool的映射信息（即为tag->bufferid（buffer pool的下标）的映射）
2）Buffer descriptors layer:buffer描述符
3）Buffer pool:真正的buffer数据

Buffer descriptors结构如下：

typedef struct BufferDesc
{BufferTag   tag;            /* ID of page contained in buffer */int         buf_id;         /* buffer's index number (from 0) *//* state of the tag, containing flags, refcount and usagecount */pg_atomic_uint32 state;int         wait_backend_pid;   /* backend PID of pin-count waiter */int         freeNext;       /* link in freelist chain */LWLock      content_lock;   /* to lock access to buffer contents */
} BufferDesc;
//对应到页
typedef struct buftag
{RelFileNode rnode;          /* physical relation identifier */ForkNumber  forkNum;    /*file type*/BlockNumber blockNum;       /* blknum relative to begin of reln */
} BufferTag;//唯一确定表
typedef struct RelFileNode
{Oid         spcNode;        /* tablespace */Oid         dbNode;         /* database */Oid         relNode;        /* relation */
} RelFileNode;

4.2 修改后落盘

 可见MarkBufferDirty函数，其主要作用就是增加该页的BM_DIRTY状态，该状态的页面淘汰前会落盘。

4.3 获取buffer页的流程

可见ReadBuffer_common函数

4.4 Buffer的淘汰策略

PG中使用的使用ClockSweep淘汰策略，其相关结构如下

typedef struct
{/* Spinlock: protects the values below */slock_t     buffer_strategy_lock;/** Clock sweep hand: index of next buffer to consider grabbing. Note that* this isn't a concrete buffer - we only ever increase the value. So, to* get an actual buffer, it needs to be used modulo NBuffers.*/pg_atomic_uint32 nextVictimBuffer;int         firstFreeBuffer;    /* Head of list of unused buffers */int         lastFreeBuffer; /* Tail of list of unused buffers *//** NOTE: lastFreeBuffer is undefined when firstFreeBuffer is -1 (that is,* when the list is empty)*//** Statistics.  These counters should be wide enough that they can't* overflow during a single bgwriter cycle.*/uint32      completePasses; /* Complete cycles of the clock sweep */pg_atomic_uint32 numBufferAllocs;   /* Buffers allocated since last reset *//** Bgworker process to be notified upon activity or -1 if none. See* StrategyNotifyBgWriter.*/int         bgwprocno;
} BufferStrategyControl;

buffer descriptor是循环链表，数组标识usage_count，nextVictimBuffer是32位usigned int，总是指向某个buffer descriptor并按顺时针顺序旋转。淘汰页面代码：StrategyGetBuffer。主要做到事情就是引用数为0，使用数减一，使用为0可以淘汰。

5.存储管理器（SMGR)

 负责统一管理不同存储介质的对接，通过虚函数表来调用。每个backend使用SMGR的SMgrRelationHash管理SMgrRelationData,实现加速访问,SMgrRelationData记录一张打开的表的信息。

typedef struct SMgrRelationData
{/* rnode is the hashtable lookup key, so it must be first! */RelFileNodeBackend smgr_rnode;  /* relation physical identifier *//* pointer to owning pointer, or NULL if none */struct SMgrRelationData **smgr_owner;/** These next three fields are not actually used or manipulated by smgr,* except that they are reset to InvalidBlockNumber upon a cache flush* event (in particular, upon truncation of the relation).  Higher levels* store cached state here so that it will be reset when truncation* happens.  In all three cases, InvalidBlockNumber means "unknown".*/BlockNumber smgr_targblock; /* current insertion target block */BlockNumber smgr_fsm_nblocks;  /* last known size of fsm fork */BlockNumber smgr_vm_nblocks;  /* last known size of vm fork *//* additional public fields may someday exist here *//** Fields below here are intended to be private to smgr.c and its* submodules.  Do not touch them from elsewhere.*/int      smgr_which;    /* storage manager selector *//** for md.c; per-fork arrays of the number of open segments* (md_num_open_segs) and the segments themselves (md_seg_fds).*/int      md_num_open_segs[MAX_FORKNUM + 1];struct _MdfdVec *md_seg_fds[MAX_FORKNUM + 1];/* if unowned, list link in list of all unowned SMgrRelations */dlist_node  node;
} SMgrRelationData;

外部调用流程如下：
1）外部buffer pool调用smgrread获取数据
2）smgr调用smgrrsw去获取
3）smgrsw调用底层MD接口，具体接口使用函数表定义

static const f_smgr smgrsw[] = {/* magnetic disk */{mdinit, NULL, mdclose, mdcreate, mdexists, mdunlink, mdextend,mdprefetch, mdread, mdwrite, mdwriteback, mdnblocks, mdtruncate,mdimmedsync, mdpreckpt, mdsync, mdpostckpt}
};

具体代码文件：smgr.c

6.磁盘管理器（MD)

Md是berkeyley开源的磁盘管理器，可以访问磁盘和ssd，可以看md.c文件。负责磁盘文件的打开，创建，删除。

7.虚拟文件管理器（VFD)

虚拟文件管理器的作用是为了防止句柄数量超过操作系统的限制，如果超过就需要进行淘汰。采用的是LRU的淘汰机制，可以看fd.c。

8.物理文件存储介绍

在这里和MySQL InnoDB做个比较，InnoDB采用的是段页式管理，而PG采用的只有8k分页，代码上来说更为简洁。

MySQL InnoDB:

PG是按8k分页，可以看到其实是将变与不变部分进行了分离，item为大小不变部分（也就是单个记录的元数据），tuple是大小可变部分。

//Header信息如下：
typedef struct PageHeaderData
{/* XXX LSN is member of *any* block, not only page-organized ones */PageXLogRecPtr pd_lsn;      /* LSN: next byte after last byte of xlog* record for last change to this page */uint16      pd_checksum;    /* checksum */uint16      pd_flags;       /* flag bits, see below */LocationIndex pd_lower;     /* offset to start of free space */LocationIndex pd_upper;     /* offset to end of free space */LocationIndex pd_special;   /* offset to start of special space */uint16      pd_pagesize_version;TransactionId pd_prune_xid; /* oldest prunable XID, or zero if none */ItemIdData  pd_linp[FLEXIBLE_ARRAY_MEMBER]; /* line pointer array */
} PageHeaderData;typedef PageHeaderData *PageHeader;//ItemData信息如下：
typedef struct ItemIdData
{unsigned    lp_off:15,      /* offset to tuple (from start of page) */lp_flags:2,     /* state of item pointer, see below */lp_len:15;      /* byte length of tuple */
} ItemIdData;typedef ItemIdData *ItemId;

9. 总结

本篇介绍了元组获取的五个层级，从发出获取元组的请求，依次经过Buffer(真正的页数据建立起的映射），存储管理器（SMGR，支持扩展对接不同的存储介质，目前只有磁盘)，磁盘管理器（MD，真正访问磁盘，read,write操作)，虚拟文件管理器（VFD，用以防止文件打开数目超过限制)，最终获得磁盘的页。

这篇关于PostgreSQL技术内幕5：PostgreSQL存储引擎从磁盘到内存的读取的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！

---