MySQL5.7 : Metadata Lock关键类及函数

MySQL5.7.4中,对Server层的Metadata Lock做了颠覆性的修改,在正常负载下几乎完全消除了MDL子系统部分的锁开销.

 

基于最新的MySQL5.7.5, 我们来看看MDL锁的具体实现思路。注意在5.7.5里,MDL子系统的代码和之前版本已经发生了非常大的差异,具体表现在:

1.    针对DDL DML做了区分,引入了fast path的概念,对于走fast path路径的MDL锁,无需任何读写锁操作,使用类似LOCK WORD的方式来标记赋予权限

2.    MDL的存储采用LOCK FREE HASH的方式,这样5.6所谓的mdl分区在5.7里都全部移除了

3.    THR_LOCK被移除掉了,改而全部使用MDL锁来代替,为了适应这种变化,引入新的MDL锁类型来处理类似LOCK TABLE READ这样的场景,同时例如DML饥饿处理也做了些变化

 

另外,鉴于MDL子系统的复杂度,本文只是做一些简单的描述,后续还会单独分篇来对其中的某些细节进行探讨。很早之前写了篇博客来介绍MDL锁的发展历程,有兴趣的可以看一下:http://mysqllover.com/?p=985

 

先画个图,简单的理一下各个类之间的关系

 mdl-class

 

0.background

在开始之前,先了解下MDL锁划分为哪几种类型,这有助于理解mdl,并且代码的注释也非常清楚,值得一读.锁类型enum_mdl_type:

name

简写

description

MDL_INTENTION_EXCLUSIVE An   intention exclusive metadata lock. Used only for scoped locks.Owner   of this type of lock can acquire upgradable exclusive locks on individual   objects.Compatible   with other IX locks, but is incompatible with scoped S andX locks.
MDL_SHARED S To   be used in cases when we are interested in object metadata only and there is   no intention to access object data (e.g. for stored routines or during   preparing prepared statements)
MDL_SHARED_HIGH_PRIO SH high   priority shared metadata lock “High   priority” means that, unlike other shared locks, it is granted ignoring   pending requests for exclusive locks. Intended for use incases when we only   need to access metadata and not data, e.g. whenfilling   an INFORMATION_SCHEMA table. 

Since   SH lock is compatible with SNRW lock, the connection that holds SH lock lock   should not try to acquire any kind of table-level or row-level lock, as this   can lead to a deadlock. Moreover, after acquiring SH lock, the connection   should not wait for any other resource, as it might cause starvation for X   locks and a potential deadlock during upgrade of SNW or SNRW to X lock (e.g.   if the upgrading connection holds the resource that is being waited for)

MDL_SHARED_READ SR A   shared metadata lock for cases when there is an intention to read data from   table. A   connection holding this kind of lock can read table metadata and read table   data (after acquiring appropriate table and row-level locks).This   means that one can only acquire TL_READ, TL_READ_NO_INSERT, and similar   table-level locks on table if one holds SR MDL lock on it. To   be used for tables in SELECTs, subqueries, and LOCK TABLE …  READ   statements.
MDL_SHARED_WRITE SW A   shared metadata lock for cases when there is an intention to modify (and not   just read) data in the table. A   connection holding SW lock can read table metadata and modify or read table   data (after acquiring appropriate table and row-level locks).To   be used for tables to be modified by INSERT, UPDATE, DELETE statements, but   not LOCK TABLE … WRITE or DDL). Also taken by SELECT … FOR UPDATE.
MDL_SHARED_WRITE_LOW_PRIO  A   version of MDL_SHARED_WRITE lock which has lower priority than   MDL_SHARED_READ_ONLY locks. Used by DML statements modifying tables and using   the LOW_PRIORITY clause.
MDL_SHARED_UPGRADABLE SU  An   upgradable shared metadata lock which allows concurrent updates and reads of   table data. A   connection holding this kind of lock can read table metadata and read table   data. It should not modify data as this lock is compatible with SRO locks.Can   be upgraded to SNW, SNRW and X locks. Once SU lock is upgraded to X or SNRW   lock data modification can happen freely. To be used for the first phase of   ALTER TABLE.
MDL_SHARED_READ_ONLY SRO  A   shared metadata lock for cases when we need to read data from table and block   all concurrent modifications to it (for both data and metadata). Used by LOCK   TABLES READ statement.
MDL_SHARED_NO_WRITE SNW An   upgradable shared metadata lock which blocks all attempts to update table   data, allowing reads.  A connection holding this kind of lock can read   table metadata and read table data.Can   be upgraded to X metadata lock. Note,   that since this type of lock is not compatible with SNRW or SW lock types,   acquiring appropriate engine-level locks for reading (TL_READ* for MyISAM,   shared row locks in InnoDB) should be contention-free. To   be used for the first phase of ALTER TABLE, when copying data between tables,   to allow concurrent SELECTs from the table, but not UPDATEs.
MDL_SHARED_NO_READ_WRITE SNRW An   upgradable shared metadata lock which allows other connections to access   table metadata, but not data.It   blocks all attempts to read or update table data, while allowing   INFORMATION_SCHEMA and SHOW queries.A   connection holding this kind of lock can read table metadata modify and read   table data.Can   be upgraded to X metadata lock. To be used for LOCK TABLES WRITE statement.Not   compatible with any other lock type except S and SH.
MDL_EXCLUSIVE An   exclusive metadata lock. A connection holding this lock can modify both   table’s metadata and data.No   other type of metadata lock can be granted while this lock is held. To be   used for CREATE/DROP/RENAME TABLE statements and for execution of certain   phases of other DDL statements.

 

enum_mdl_duration : DML duration类型

Name Description
MDL_STATEMENT 语句级别,SQL结束后自动释放
MDL_TRANSACTION 事务提交后释放
MDL_EXPLICIT 需要显式释放锁

 

2.构建一个MDL锁请求

 

在尝试加锁之前,总是需要先构建一个mdl锁请求,对应的类对象为MDL_request,主要成员包括:

enum_mdl_type   type 请求的MDL锁类型
enum   enum_mdl_duration duration duration级别
MDL_request   *next_in_listMDL_request   **prev_in_list; 维持当前session请求的MDL锁的链表结构
MDL_ticket   *ticket 后文描述
MDL_key   key; 锁请求的key一般是基于库表名或者类型
init_with_sourceinit_by_key_with_source 初始化锁请求
bool   is_write_lock_request() 判断是否是写锁请求type   >= MDL_SHARED_WRITE &&              type != MDL_SHARED_READ_ONLY
bool   is_ddl_or_lock_tables_lock_request() 判断是否是一个强优先级的,例如DDL, LOCK TABLE之类的请求 type   >= MDL_SHARED_UPGRADABLE;

 

MDL_key 用于确定一个具体的MDL锁对象,该类主要包括:

    enum enum_mdl_namespace { GLOBAL=0,                                SCHEMA,                                TABLE,                                FUNCTION,                                PROCEDURE,

                                TRIGGER,

                                EVENT,

                                COMMIT,

                                USER_LEVEL_LOCK,

                                /* This should be the last ! */

                                NAMESPACE_END };

针对不同的对象类型,定义不同的MDL锁:TABLE:   用于tableviewFUNCTION:   用于stored   functionPROCEDURE:用于stored   procedureTRIGGER:   trigger

EVENT:    event scheduler event

USER_LEVEL_LOCK:   用户自定义锁,通过GET_LOCK获得

mdl_key_init 初始化key的函数,根据namespace,   库名,及对象名构建key,实际上是一个简单的字符串,字符串首个字符表示namespace类型,大概分布为:/namespace/dbname/objname
m_length key值长度
m_db_name_length db名长度
char   m_ptr[MAX_MDLKEY_LENGTH] 存储的key

 

例如在执行一条简单的SELECT时候,就会在语法解析阶段将需要请求的MDL锁存储到对象TABLE_LIST::mdl_request中:

函数:TABLE_LIST *st_select_lex::add_table_to_list

  if (!MY_TEST(table_options & TL_OPTION_ALIAS))

  {

    MDL_REQUEST_INIT(& ptr->mdl_request,

                     MDL_key::TABLE, ptr->db, ptr->table_name, mdl_type,

                     MDL_TRANSACTION);

  }

 

注:几乎所有的mdl锁请求初始化,都是通过宏MDL_REQUEST_INIT来完成的。另一种是MDL_REQUEST_INIT_BY_KEY,实际上是对MDL_request::init_with_source init_by_key_with_source的封装

 

请求的MDL锁类型,在词法/语法解析阶段完成,参考sql_yacc.yy文件,搜m_mdl_type,可以看到针对不同的token,预先赋值好锁类型

 

3.请求加MDL

MDL锁的加锁操作,通过名为MDL_context的类来进行,该类对象挂在THD::mdl_context下面.当连接创建时就会实例化一个MDL_context对象

 

MDL_context类描述如下:

try_acquire_lock(MDL_request   *mdl_request) 尝试加锁
acquire_locks(MDL_request_list   *requests, ulong lock_wait_timeout)acquire_lock(MDL_request   *mdl_request, ulong lock_wait_timeout); 获取MDL锁,前者可以一次操作多个MDL锁请求
upgrade_shared_lock(MDL_ticket   *mdl_ticket,                               enum_mdl_type new_type,                               ulong lock_wait_timeout); 升级共享锁
release_all_locks_for_name(MDL_ticket   *ticket);release_lock(MDL_ticket   *ticket);
owns_equal_or_stronger_lock(MDL_key::enum_mdl_namespace   mdl_namespace,                                       const char *db, const char *name,                                       enum_mdl_type mdl_type) 判断当前session是否持有等同或更强的MDL
MDL_wait   m_wait 当锁请求被调度或者被死锁检测模块中断时,结果被记录在这里
Ticket_list   m_tickets[MDL_DURATION_END]; 所有被该连接获取到的MDL ticket,针对不同的duration级别,数组被划分成了三个链表
MDL_context_owner   *m_owner MDL_context的拥有者,用来和THD解除联系
bool   m_needs_thr_lock_abort 看起来只在handler接口被调用,暂时忽略
bool   m_force_dml_deadlock_weight Indicates   that we need to use DEADLOCK_WEIGHT_DML deadlock weight for this context and   ignore the deadlock weight provided by the MDL_wait_for_subgraph object which   we are waiting for
MDL_wait_for_subgraph   *m_waiting_for 告诉死锁检测模块当前session正在等待的mdl lock或者table   define cache entry.总的来说,这是冗余的,因为这些信息可以从等待队列中获得
LF_PINS   *m_pins; LOCK-FREE实现的pin,   后面单独介绍
m_rand_state  State   for pseudo random numbers generator (PRNG) which output is used to perform   random dives into MDL_lock objects hash when searching for unused objects to   free
find_deadlock() 检测死锁

 

MDL ticket ,如其名,相当于一张获得锁的门票,用于代表每一个请求的mdl锁,类描述如下:

MDL_ticket   *next_in_context;MDL_ticket   **prev_in_context; 维护在当前Context中的链表结构
MDL_ticket   *next_in_lock;MDL_ticket   **prev_in_lock; 维护在MDL_lock中的链表结构

has_pending_conflicting_lock()

检查是否有冲突的锁类型

downgrade_lock(enum_mdl_type   type) 对锁降级
has_stronger_or_equal_type(enum_mdl_type   type) 判断是否有等同或更强优先级的锁类型
bool   is_incompatible_when_granted(enum_mdl_type type)bool   is_incompatible_when_waiting(enum_mdl_type type)
enum   enum_mdl_type m_type mdl锁类型
enum_mdl_duration   m_duration; duration类型

MDL_context *m_ctx

拥有当前ticketMDL_context

MDL_lock   *m_lock 当前ticket对应的MDL_lock对象
bool   m_is_fast_path 用于标示是否是使用”fast path”来进行加锁,这也意味着没有被纳入MDL_lock::m_granted   bitmap/list,而是简单的进行计数

 

MDL_lock也是mdl子系统的核心类之一,在获取一个MDL锁时创建,对于一个给定的名字,只会存在一个实例,并且只有锁已经被grant后才存在。你可以把它视为MDL子系统中的TABLE_SHARE:

MDL_lock类描述如下:

Ticket_list Ticket_list类用于维护该lock上的ticket链表,两个成员:List   m_list; 存储ticket链表bitmap_t   m_bitmap; 链表上ticket类型的bitmap
struct   MDL_lock_strategy { 辅助结构体,来处理不同的锁类型,通常有两种:1.一种称为scoped   lock,包括GLOBAL, COMMIT, SCHEMA 三种namespace类型对应对象为m_scoped_lock_strategy2.另外一种称为object   lock对应的对象为m_object_lock_strategy

 

 

bitmap_t   m_granted_incompatible[MDL_TYPE_END] 权限矩阵,用于存储哪些已经grant的类型的锁和当前申请的不相容
bitmap_t   m_waiting_incompatible[4][MDL_TYPE_END] 指明哪些等待的锁类型和当前请求的不相容。出于防止DML饿死的原因,我们需要四个数组处理不同的情况:0)   in “normal” situation.1)   in situation when the number of successively granted “piglet”   requests exceeds the max_write_lock_count limit.2)   in situation when the number of successively granted “hog” requests   exceeds the max_write_lock_count limit.3)   in situation when both “piglet” and “hog” counters exceed   limit.

 

fast_path_state_t   m_unobtrusive_lock_increment[MDL_TYPE_END] 用于存储unobtrusive   类型的锁请求 ?? m_object_lock_strategy:{ 0,   1, 1, 1ULL << 20, 1ULL << 40, 1ULL << 40, 0, 0, 0, 0, 0 } 

对于每种object类型的锁:

“unobtrusive”   types: S, SH, SR and SW

“obtrusive”   types: SU, SRO, SNW, SNRW, X

这里可以理解成一个lock word, 当前的请求某种锁类型对应的值,被加到m_fast_path_state   , 这也是实现FAST-PATH的核心

注释:

Number   of locks acquired using “fast path” are encoded in the following bits   of MDL_lock::m_fast_path_state:

–   bits 0 .. 19  – S and SH (we don’t differentiate them once acquired)

–   bits 20 .. 39 – SR

–   bits 40 .. 59 – SW and SWLP (we don’t differentiate them once acquired)

 bool   m_is_affected_by_max_write_lock_count 是否受到max_write_lock_count参数限制的影响m_scoped_lock_strategy:   falsem_object_lock_strategy:   true
bool   (*m_needs_notification)(const MDL_ticket *ticket) 以下均为函数指针,目的是针对两种类型不同的处理方式 m_scoped_lock_strategy:   NULLm_object_lock_strategy:MDL_lock::object_lock_needs_notification
void   (*m_notify_conflicting_locks)(MDL_context *ctx, MDL_lock *lock); m_scoped_lock_strategy:   NULLm_object_lock_strategy:MDL_lock::object_lock_notify_conflicting_locks
bitmap_t   (*m_fast_path_granted_bitmap)(const MDL_lock &lock); m_scoped_lock_strategy:&MDL_lock::scoped_lock_fast_path_granted_bitmap m_object_lock_strategy:MDL_lock::object_lock_fast_path_granted_bitmap

 

参考函数:MDL_lock::can_grant_lock

bool   (*m_needs_connection_check)(const MDL_lock *lock); m_scoped_lock_strategy:NULLm_object_lock_strategy:MDL_lock::object_lock_needs_connection_check 等待MDL锁时进行连接检查(MDL_context::acquire_lock
};############end   of MDL_lock_strategy
 const   MDL_lock_strategy *m_strategy;
static   const MDL_lock_strategy m_scoped_lock_strategy;static   const MDL_lock_strategy m_object_lock_strategy; 两个核心成员!!!分别代表两种不同的策略.大多数情况下,我们只关注m_object_lock_strategy
 MDL_key   key; 当前的key
bool   count_piglets_and_hogs(enum_mdl_type type)
uint   m_obtrusive_locks_granted_waiting_count; Number   of granted or waiting lock requests of “obtrusive” type. Also   includes “obtrusive” lock requests for which we about to check if   they can be granted
Ticket_list   m_granted; 已被赋权的ticket
Ticket_list   m_waiting 正在等待的ticket
volatile   fast_path_state_t m_fast_path_state fast-path特性的核心变量,通过该变量决定了是走fast path 还是slow   path,实际上这里fast_path_state_t是一个long   long类型,被划分成了好几段来代表不同的含义,通过逻辑运算来获取当前MDL_lock的状态
static   const fast_path_state_t IS_DESTROYED=  1ULL << 62 标示该MDL_lock即将被销毁
static   const fast_path_state_t HAS_OBTRUSIVE= 1ULL << 61 标示该MDL_lock已经被赋予给一个OBSTRUSIVE
static   const fast_path_state_t HAS_SLOW_PATH= 1ULL << 60 标示有走slow   path的锁已经被grant,正在等待或者检查是否可以被grant

 

MDL_map用于管理所有的MDL锁(也即MDL_lock对象),因此使用一个全局static对象mdl_locks来进行存储。 MDL_map类描述如下:

MDL_lock   *find() 查找指定MDL_keylock
MDL_lock   *find_or_insert() 查找并插入
lock_object_used() 计数器m_unused_lock_objects1
lock_object_unused(MDL_context   *ctx, LF_PINS *pins) 增加未使用的对象的计数器,如果这些对象的计数值超过某些阀值,并且unused/total超过1/4,就娶尝试释放一些对象。 释放的方式随机选取,这不同于早期LRU的淘汰方式 低水位阀值默认为1000
LF_HASH   m_locks lock   freehash对象
 MDL_lock   *m_global_lock; 全局锁,如全局读锁
MDL_lock   *m_commit_lock; namespaceCOMMIT的锁对象预先分配的MDL_lock
volatile   int32 m_unused_lock_objects; 未使用的MDL_lock对象个数

 

关于这里使用到的LOCK-FREE算法,后面再单独描述。

 

MDL_wait表示MDL锁的等待状态

enum   enum_wait_status { EMPTY = 0, GRANTED, VICTIM, TIMEOUT, KILLED } 等待状态值
 enum_wait_status   timed_wait(MDL_context_owner *owner,                                  struct timespec *abs_timeout,                                  bool signal_timeout,                                  const PSI_stage_info *wait_state_name) 进入等待MDL锁的调用函数

 

MDL_context::acquire_lock流程:

 

Step1: 调用try_acquire_lock_impl尝试去获取mdl

在该函数了实现了fast-patch slow-path.这也是在5.7里对MDL锁的优化核心

 

a.    首先检查当前sessionMDL_context中是否已经获取的了相同或者更强的mdl

MDL_context::find_ticket

如果存在,赋值给mdl_request->ticket= ticket

 

如果不存在,则继续

 

b.fix_pins())

 

c.创建ticket

MDL_ticket::create

 

d.判断是否走slow path

 

unobtrusive_lock_increment=

    MDL_lock::get_unobtrusive_lock_increment(mdl_request);

 

force_slow= ! unobtrusive_lock_increment || m_needs_thr_lock_abort;

 

即当前的锁请求类型是否是unobtrusive的(S, SH, SR and SW),另外从代码来看,m_needs_thr_lock_abort只在通过HANDLER接口时才会被使用到.对于HANDLER接口调用,看起来总会走slow path

 

if (! unobtrusive_lock_increment)

    materialize_fast_path_locks();

 

e.根据key值尝试向mdl_locks对象中查找或插入key,并返回MDL_lock对象

lock= mdl_locks.find_or_insert(m_pins, key, &pinned)

 

f. force_slowfalse,则先走fast path

 

这里的判断都是无锁操作,通过MDL_lock::m_fast_path_state来判断是否有不相容的MDL

MDL_lock::IS_DESTROYED:表示这个MDL_lock对象被销毁了,需要重新插入(mdl_locks.find_or_insert)

 

MDL_lock::HAS_OBTRUSIVE: 存在Obstrusive类型的锁,需要走slow path

否则修改m_fast_path_state值,增加一定的增量值(有点类似Lock word的概念)

如果通过fast path赋予锁,无需在MDL_lock::m_granted列表中加上当前ticket

 

 

g.否则走slow path

slow path会持有读写锁来进行操作,因此相对前者会略慢点.

通过slow path进行加锁操作时,会设置m_fast_path_state变量状态值为MDL_lock::HAS_SLOW_PATH | MDL_lock::HAS_OBTRUSIVE

 

显而易见,如果我们所有的负载都走fast-path,那么就可以完全实现无锁的操作.

 

slow-path走和以前版本类似的常规检查

根据类型检查是否可以赋锁:MDL_lock::can_grant_lock

如果可以,则加入链表lock->m_granted.add_ticket(ticket)

并做一次mdl锁请求饥饿检查。

 

 

Step2: 无论是否成功,都会分配ticket,如果无法获得MDL锁,那么就进入等待状态

  lock= ticket->m_lock;

 

  lock->m_waiting.add_ticket(ticket);  当前等待的ticket会被加入到MDL_lock::m_waiting队列中

 

进行死锁检测

will_wait_for(ticket);

find_deadlock();

然后进入condition wait

done_waiting_for()

 

4. 释放MDL

解锁函数为:

MDL_context::release_lock(enum_mdl_duration duration, MDL_ticket *ticket)

同样这里针对fast-path slow-path有不同的处理.

fast-path的解锁依旧是无锁操作

slow-path则需要从m_granted中移除ticket.

 

TODO

1.    LOCK-FREE MDL HASH的具体实现和应用

2.    MDL如何进行死锁检测

3.    如何防止锁等待饿死

 

原创文章,转载请注明: 转载自Simple Life

本文链接地址: MySQL5.7 : Metadata Lock关键类及函数

Post Footer automatically generated by wp-posturl plugin for wordpress.


Comments

Leave a Reply

Your email address will not be published. Name and email are required


Current month ye@r day *