本文主要是介绍PUPPET部署篇,希望对大家解决编程问题提供一定的参考价值,需要的开发者们随着小编来一起学习吧!
一、简介
puppet是一种Linux、Unix、windows平台的集中配置管理系统,使用自有的puppet描述语言,可管理配置文件、用户、cron任务、软件包、系统服务等。puppet把这些系统实体称之为资源,puppet的设计目标是简化对这些资源的管理以及妥善处理资源间的依赖关系
puppet采用C/S星状的结构,所有的客户端和一个或几个服务器交互。每个客户端周期的(默认半个小时)向服务器发送请求,获得其最新的配置信息,保证和该配置信息同步。每个puppet客户端每半小时(可以设置)连接一次服务器端, 下载最新的配置文件,并且严格按照配置文件来配置客户端. 配置完成以后,puppet客户端可以反馈给服务器端一个消息. 如果出错,也会给服务器端反馈一个消息
Puppet 是唯一结合基于模型和基于任务的功能的解决方案,使组织能够随着自动化足迹的增长扩展其多云基础架构。PUPPET提出了基础设施即代码的概念,我们相信这将有助于客户更快地进行创新
二、单节点部署
2.1 总览
单节点部署使用一台Master+多台Agent的单节点网络形式,呈现出一对多的表现形式
实例 | IP | 域名(主机名) | 软件 |
---|---|---|---|
Master | 192.168.44.100 | master.testpp.com | puppet server |
Agent | 192.168.44.101 | agent.testpp.com | puppet |
2.2 Master服务器
1)配置系统环境,关闭防火墙,SELinux,设置NTP时间同步等
2)安装PUPPET软件包
#安装epel
yum install epel-release -y
#安装puppet客户端
yum install puppet-server -y
3)配置puppet主配置文件
vim /etc/puppet/puppet.conf
[main]# The Puppet log directory.# The default value is '$vardir/log'.logdir = /var/log/puppet# Where Puppet PID files are kept.# The default value is '$vardir/run'.rundir = /var/run/puppet# Where SSL certificates are kept.# The default value is '$confdir/ssl'.ssldir = $vardir/sslpluginsync = trueenvironmentpath = $confdir/environmentsdefault_manifest = ./manifestsbasemodulepath = $confdir/modules
[master]# The file in which puppetd stores a list of the classes# associated with the retrieved configuratiion. Can be loaded in# the separate ``puppet`` executable using the ``--loadclasses``# option.# The default value is '$confdir/classes.txt'.classfile = $vardir/classes.txt# Where puppetd caches the local configuration. An# extension indicating the cache format is added automatically.# The default value is '$confdir/localconfig'.localconfig = $vardir/localconfigfileserverconfig = /etc/puppet/fileserver.confreportdir = /home/logs/puppet/reportsmasterhttplog = /home/logs/puppet/masterhttp.log#masterlog = /home/logs/puppet/puppetmaster.logreports = log#templatedir = /etc/puppet/templatesca = trueautosign=trueautosign = /etc/puppet/autosign.confcertname=master.testpp.comssl_client_verify_header = HTTP_X_CLIENT_VERIFYssl_client_header = HTTP_X_CLIENT_DN
4)配置autosign.conf
*.testpp.com
5)配置auth.conf(在原文件末尾添加)
##auth config
path /certificate_status/ca
auth anypath /certificate_status/master.testpp.com
auth anypath /certificate_status
auth any
allow 192.168.44.100# deny everything else; this ACL is not strictly necessary, but
# illustrates the default policy.
path /
auth any
6)启动Master服务
systemctl start puppetmaster
(可选)如需要查看实时日志打印信息,可使用puppet master -v --no-daemonize
启动puppet master进程
2.3 Agent客户端
1)配置系统环境,关闭防火墙,SELinux,设置NTP时间同步等,切记还需要配置hosts,名称与主机名一致
2)安装PUPPET软件包
#安装epel
yum install epel-release -y
#安装puppet客户端
yum install puppet -y
3)配置puppet主配置文件
[main]# The Puppet log directory.# The default value is '$vardir/log'.logdir = /var/log/puppet# Where Puppet PID files are kept.# The default value is '$vardir/run'.rundir = /var/run/puppet# Where SSL certificates are kept.# The default value is '$confdir/ssl'.ssldir = $vardir/sslpluginsync = trueenvironmentpath = $confdir/environmentsdefault_manifest = ./manifestsbasemodulepath = $confdir/modules
[agent]# The file in which puppetd stores a list of the classes# associated with the retrieved configuratiion. Can be loaded in# the separate ``puppet`` executable using the ``--loadclasses``# option.# The default value is '$confdir/classes.txt'.classfile = $vardir/classes.txt# Where puppetd caches the local configuration. An# extension indicating the cache format is added automatically.# The default value is '$confdir/localconfig'.ignorecache = true#user = root#group = root#listen = falseserver = master.testpp.compuppetdlog = /var/log/puppet/puppetd.logruninterval = 60#report = trueconfig = /etc/puppet/puppet.conf#report_server = puppetmaster.idc.puppet.cnhttp_compression = true#modulepath = /etc/puppet/modules:/usr/share/puppet/modules
4)测试连接,需要用root用户,因为puppet很多配置需要root权限执行
puppet agent --test
若成功至此单节点部署完毕,可在小规模主机的情况下使用
三、分布式集群部署
本章以生产环境为例,应对大规模主机的情况下,需要对已有架构进行拓展,搭建分布式集群,以应对大量的同步请求
3.1 网络环境
集群网络拓扑
实例 | IP | 域名(主机名) | 软件 |
---|---|---|---|
CA | 10.60.22.160 | puppetca-22-160.osp-pup-pe.lin.idc.puppet.cn | puppet server [nginx] |
master1 | 10.60.22.161 | puppetca-22-161.osp-pup-pe.lin.idc.puppet.cn | puppet server [nginx] |
master2 | 10.60.22.162 | puppetca-22-162.osp-pup-pe.lin.idc.puppet.cn | puppet server [nginx] |
puppet-server(LB) | 10.60.22.161(测试用) | puppetca-22-161.osp-pup-pe.lin.idc.puppet.cn | nginx |
puppet-agent | 10.60.22.162(测试用) | puppetca-22-162.osp-pup-pe.lin.idc.puppet.cn | puppet |
首先排雷:puppet最大的坑,就是客户端必须配置自己的hosts,且必须和主机名保持一致,否则证书无法正常生成
总体架构 (一个CA,两台Master):
1、使用Nginx+Passenger替换Master服务,提高多机器环境的服务可用性
2、使用Gerrit进行puppet配置文件托管与审核服务,Master每间隔五分钟直接拉取Gerrit仓库的配置文件
3、集群流量入口为PP的LB集群,由LB负责反向代理请求至后端服务器
3.2 前置公共部分(一台CA、两台Master节点)
1)设置yum:vi /etc/yum.repos.d/puppet.repo
需要配合公网yum,需申请公网代理访问,也可使用阿里的公网yum源
[puppetlabs-products]
name=Puppet Labs Products El 7 - $basearch
baseurl=http://yum.puppetlabs.com/el/7/products/$basearch
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-puppetlabs
enabled=1
gpgcheck=0
[puppetlabs-deps]
name=Puppet Labs Dependencies El 7 - $basearch
baseurl=http://yum.puppetlabs.com/el/7/dependencies/$basearch
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-puppetlabs
enabled=1
gpgcheck=0
[puppetlabs-devel]
name=Puppet Labs Devel El 7 - $basearch
baseurl=http://yum.puppetlabs.com/el/7/devel/$basearch
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-puppetlabs
enabled=0
gpgcheck=1
[puppetlabs-products-source]
name=Puppet Labs Products El 7 - $basearch - Source
baseurl=http://yum.puppetlabs.com/el/7/products/SRPMS
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-puppetlabs
failovermethod=priority
enabled=0
gpgcheck=1
[puppetlabs-deps-source]
name=Puppet Labs Source Dependencies El 7 - $basearch - Source
baseurl=http://yum.puppetlabs.com/el/7/dependencies/SRPMS
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-puppetlabs
enabled=0
gpgcheck=1
[puppetlabs-devel-source]
name=Puppet Labs Devel El 7 - $basearch - Source
baseurl=http://yum.puppetlabs.com/el/7/devel/SRPMS
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-puppetlabs
enabled=0
gpgcheck=1
2)yum初始化
yum clean all && yum makecache
3)安装软件依赖包
yum install -y ruby-irb ruby-devel ruby ruby-shadow rubygems facter hiera libselinux-ruby ruby-augeas ruby-rgen gcc gcc-c++ curl-devel openssl-devel zlib-devel ruby-devel pcre pcre-devel make rubygem-net-ldap git puppet puppet-server -y
4)安装gem模块
gem安装Passenger/Rake/Rack,若下载失败可本地下载后上传至服务器安装
gem sources --add https://gems.ruby-china.com/ --remove https://rubygems.org/
gem install rake --version=0.9.6
gem install rack --version=1.6.4
gem install passenger --version=5.3.1
wget http://rubygems.org/downloads/rake-0.9.6.gem
wget http://rubygems.org/downloads/rack-1.6.4.gem
wget http://rubygems.org/downloads/passenger-5.3.1.gem
5)安装内置nginx, 设置prefix directory为 /usr/local/nginx,如无法下载找到脚本的下载地址将https修改成http,可执行以下命令
sed -i "s#https://nginx.org/download/#http://nginx.org/download/#g" /usr/local/share/gems/gems/passenger-5.3.1/bin/passenger-install-nginx-module
该方式为编译安装,附加了passenger模块
/usr/local/share/gems/gems/passenger-5.3.1/bin/passenger-install-nginx-module
一路回车,需注意指定nginx安装目录为 /usr/local/nginx
设置rack,nginx工作目录权限
mkdir -p /etc/puppet/rack/public
cp /usr/share/puppet/ext/rack/config.ru /etc/puppet/rack
chown -R puppet:puppet /etc/puppet/rack
3.3 CA服务器
1)生成ssl证书,非必须,可复用历史证书,保证证书名称与在用的一致即可
若复用,拷贝原CA服务器/var/lib/puppet/ssl文件夹至新CA服务器的/var/lib/puppet/文件夹下
#注意ppmaster7.puppet.cn后的空格,非换行
puppet cert --generate --allow-dns-alt-names --dns_alt_names puppetmaster.idc.puppet.cn,puppetmaster7.idc.puppet.cn,pre.ppmaster.puppet.cn,pre.ppmaster7.puppet.cn,ppmaster7.puppet.cn ppmaster.puppet.cn
2)修改puppet主配置文件 /etc/puppet/puppet.conf
[main]logdir = /var/log/puppetrundir = /var/run/puppetssldir = $vardir/ssl
[master]confdir = /etc/puppetcertname = ppmaster.puppet.cnca = trueautosign = trueautosign = /etc/puppet/autosign.conf
3)编辑/etc/puppet/auth.conf文件,添加到末尾倒数第一个配置前
path /certificate_revocation_list/ca
auth any
method find
allow *path /certificate_status/ca
auth anypath /certificate_status/ppmaster.puppet.cn
auth anypath /certificate_status/puppetmaster.idc.puppet.cn
auth anypath /certificate_status
auth any
allow [master和lb的ip]# deny everything else; this ACL is not strictly necessary, but
# illustrates the default policy.
path /
auth any
4)配置证书自动注册 /etc/puppet/autosign.conf
*.puppet.com
5)不要启动puppet-server,现在配置前面安装的nginx
修改nginx主配置文件 /usr/local/nginx/conf/nginx.conf
user root;
worker_processes 32;
worker_rlimit_nofile 65535;
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#error_log /home/logs/nginx/error.log info;
pid var/run/nginx.pid;
events {use epoll;worker_connections 65535;
}
http {passenger_root /usr/local/share/gems/gems/passenger-5.3.1;passenger_ruby /usr/bin/ruby;passenger_max_pool_size 120;passenger_max_requests 4000;passenger_pool_idle_time 100;
include mime.types;default_type application/octet-stream;log_format main '$remote_addr $host $scheme [$time_local] "$request" ''$status $body_bytes_sent "$http_referer" ''"$http_user_agent" "$http_x_forwarded_for" $upstream_response_time $upstream_addr $upstream_status $request_time';sendfile on;tcp_nopush on;tcp_nodelay on;keepalive_timeout 65;gzip on;gzip_min_length 1k;gzip_comp_level 6;gzip_buffers 4 16k;gzip_http_version 1.1;gzip_types text/plain text/css application/x-javascript text/xml application/xml+rss text/javascript;gzip_vary on;gzip_proxied any;client_max_body_size 100m;client_body_buffer_size 1024k;proxy_buffer_size 100m;proxy_buffers 8 100m;proxy_busy_buffers_size 100m;proxy_temp_file_write_size 100m;proxy_read_timeout 500;proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;include vhost/*.conf;
}
6)添加vhost,/usr/local/nginx/conf/vhost/puppet.conf
server{listen 8140;server_name puppet-master1.testpp.com 192.168.43.140;root /etc/puppet/rack/public;passenger_enabled on;access_log /home/logs/nginx/puppet.access.log main;error_log /home/logs/nginx/puppet.error.log debug;
}
7)配置nginx的log目录与pid文件,生成启动service
mkdir -p /home/logs/nginx/
touch /home/logs/nginx/puppet.access.log
touch /home/logs/nginx/puppet.error.log
mkdir -p /usr/local/nginx/var/run/ && touch /usr/local/nginx/var/run/nginx.pid
创建/usr/lib/systemd/system/nginx.service
[Unit]
Description=A high performance web server and a reverse proxy server
After=network.target
[Service]
Type=forking
ExecStart=/usr/local/nginx/sbin/nginx -g 'pid /usr/local/nginx/var/run/nginx.pid;'
PIDFile=/usr/local/nginx/var/run/nginx.pid
PrivateDevices=yes
SyslogLevel=err
ExecReload=/usr/bin/kill -HUP $MAINPID
KillSignal=SIGQUIT
KillMode=mixed
[Install]
WantedBy=multi-user.target
8)启动nginx
systemctl start nginx
9)端口启动验证,可以看到nginx进程监听了8140端口并成功启用了服务
3.4 Master服务器
1)证书同步
拷贝CA服务器/var/lib/puppet/ssl文件夹至Master服务器的/var/lib/puppet/文件夹下
2)修改puppet主配置文件 /etc/puppet/puppet.conf
[main]# The Puppet log directory.# The default value is '$vardir/log'.logdir = /var/log/puppet# Where Puppet PID files are kept.# The default value is '$vardir/run'.rundir = /var/run/puppet# Where SSL certificates are kept.# The default value is '$confdir/ssl'.ssldir = $vardir/ssl
[master]classfile = $vardir/classes.txtlocalconfig = $vardir/localconfigfileserverconfig = /etc/puppet/fileserver.confreportdir = /home/logs/puppet/reportsmasterhttplog = /home/logs/puppet/masterhttp.logreports = logca = falsecertname = ppmaster.puppet.cn#ssl_client_verify_header = HTTP_X_CLIENT_VERIFY#ssl_client_header = HTTP_X_CLIENT_DN
3)不要启动puppet-server,现在配置前面安装的nginx
修改nginx主配置文件 /usr/local/nginx/conf/nginx.conf
user root;
worker_processes 32;
worker_rlimit_nofile 65535;
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#error_log /home/logs/nginx/error.log info;
pid var/run/nginx.pid;
events {use epoll;worker_connections 65535;
}
http {passenger_root /usr/local/share/gems/gems/passenger-5.3.1;passenger_ruby /usr/bin/ruby;passenger_max_pool_size 120;passenger_max_requests 4000;passenger_pool_idle_time 100;
include mime.types;default_type application/octet-stream;log_format main '$remote_addr $host $scheme [$time_local] "$request" ''$status $body_bytes_sent "$http_referer" ''"$http_user_agent" "$http_x_forwarded_for" $upstream_response_time $upstream_addr $upstream_status $request_time';sendfile on;tcp_nopush on;tcp_nodelay on;keepalive_timeout 65;gzip on;gzip_min_length 1k;gzip_comp_level 6;gzip_buffers 4 16k;gzip_http_version 1.1;gzip_types text/plain text/css application/x-javascript text/xml application/xml+rss text/javascript;gzip_vary on;gzip_proxied any;client_max_body_size 100m;client_body_buffer_size 1024k;proxy_buffer_size 100m;proxy_buffers 8 100m;proxy_busy_buffers_size 100m;proxy_temp_file_write_size 100m;proxy_read_timeout 500;proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;include vhost/*.conf;
}
4)添加vhost,/usr/local/nginx/conf/vhost/puppet.conf
server{listen 8140;server_name puppet-master1.testpp.com 192.168.43.140;root /etc/puppet/rack/public;passenger_enabled on;access_log /home/logs/nginx/puppet.access.log main;error_log /home/logs/nginx/puppet.error.log debug;
}
5)配置nginx的log目录与pid文件,生成启动service
mkdir -p /home/logs/nginx/
touch /home/logs/nginx/puppet.access.log
touch /home/logs/nginx/puppet.error.log
mkdir -p /usr/local/nginx/var/run/ && touch /usr/local/nginx/var/run/nginx.pid
创建/usr/lib/systemd/system/nginx.service
[Unit]
Description=A high performance web server and a reverse proxy server
After=network.target
[Service]
Type=forking
ExecStart=/usr/local/nginx/sbin/nginx -g 'pid /usr/local/nginx/var/run/nginx.pid;'
PIDFile=/usr/local/nginx/var/run/nginx.pid
PrivateDevices=yes
SyslogLevel=err
ExecReload=/usr/bin/kill -HUP $MAINPID
KillSignal=SIGQUIT
KillMode=mixed
[Install]
WantedBy=multi-user.target
6)启动nginx
systemctl start nginx
7)端口启动验证
3.5 集群入口LB服务器
拷贝CA服务器生成的用于LB的证书 testppmaster.puppet.com文件夹,证书存放在在CA服务器下/var/lib/puppet/ssl文件夹下
1)还需单独安装nginx,然后配置反向代理服务
yum install -y nginx
2)修改nginx主配置文件 /usr/local/nginx/conf/nginx.conf
user root;
worker_processes 32;
worker_rlimit_nofile 65535;
error_log /home/logs/nginx/error.log info;
events {use epoll;worker_connections 65535;
}
http {#passenger_root /usr/local/share/gems/gems/passenger-5.3.1;#passenger_ruby /usr/bin/ruby;#passenger_max_pool_size 120;#passenger_max_requests 4000;#passenger_pool_idle_time 100;include mime.types;default_type application/octet-stream;log_format main '$remote_addr $host $scheme [$time_local] "$request" ''$status $body_bytes_sent "$http_referer" ''"$http_user_agent" "$http_x_forwarded_for" $upstream_response_time $upstream_addr $upstream_status $request_time';sendfile on;tcp_nopush on;tcp_nodelay on;keepalive_timeout 65;
3)新建puppet的vhost文件,/usr/local/nginx/conf/vhost/puppet.conf,需注意四个证书的存放路径,按需修改
upstream osp_ca {server 10.60.22.160:8140 weight=5;}
upstream osp_master {server 10.60.22.161:8140 weight=5;server 10.60.22.162:8140 weight=5;}
server {server_name ppmaster.puppet.cn;listen 443 default;ssl on ;ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;ssl_ciphers ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP;ssl_prefer_server_ciphers on;ssl_session_cache shared:SSL:512m;ssl_session_timeout 20m;#----start----#请按实际路径填写ssl_certificate /var/lib/puppet/ssl/certs/ppmaster.puppet.cn.pem;ssl_certificate_key /var/lib/puppet/ssl/ppmaster.puppet.cn.pem;ssl_crl /var/lib/puppet/ssl/ca/ca_crl.pem;ssl_client_certificate /var/lib/puppet/ssl/certs/ca.pem;#----end----ssl_verify_client optional;ssl_verify_depth 1;proxy_set_header X-SSL-Subject $ssl_client_s_dn;proxy_set_header X-Client-DN $ssl_client_s_dn;proxy_set_header X-Client-Verify $ssl_client_verify;proxy_buffers 256 8k;proxy_connect_timeout 2;proxy_read_timeout 5;proxy_send_timeout 15;proxy_http_version 1.1;proxy_set_header Connection "";proxy_headers_hash_max_size 2048;proxy_headers_hash_bucket_size 128;proxy_set_header Host $host;#proxy_set_header X-Real-IP $remote_addr;#proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;client_max_body_size 20m;location ~* /.+/(certificate|certificate_request|certificate_revocation_list)/ {proxy_pass http://osp_ca;}location / {proxy_pass http://osp_master;}
4)配置nginx的log目录与pid文件
mkdir -p /home/logs/nginx/
touch /home/logs/nginx/error.log
mkdir -p /usr/local/nginx/var/run/ && touch /usr/local/nginx/var/run/nginx.pid
5)创建/usr/lib/systemd/system/nginx.service
[Unit]
Description=A high performance web server and a reverse proxy server
After=network.target
[Service]
Type=forking
ExecStart=/usr/local/nginx/sbin/nginx -g 'pid /usr/local/nginx/var/run/nginx.pid;'
PIDFile=/usr/local/nginx/var/run/nginx.pid
PrivateDevices=yes
SyslogLevel=err
ExecReload=/usr/bin/kill -HUP $MAINPID
KillSignal=SIGQUIT
KillMode=mixed
[Install]
WantedBy=multi-user.target
6)启动nginx
systemctl start nginx
7)端口启动验证
3.6 Agent客户端
1)安装puppet软件
yum install -y puppet
2)修改/etc/puppet/puppet.conf
[main]logdir = /var/log/puppetrundir = /var/run/puppetssldir = $vardir/ssllisten = truepluginsync = true
[agent]classfile = $vardir/classes.txtlocalconfig = $vardir/localconfigruninterval = 10ignorecache = trueuser = rootgroup = rootlisten = falsepuppetdlog = /var/log/puppet/puppetd.logreport = falseconfig = /etc/puppet/puppet.confhttp_compression = trueserver = ppmaster.puppet.cnmasterport = 443
3)配置域名解析
需将ppmaster.puppet.cn的域名解析到LB节点的ip上,实验环境可用hosts代替,即在客户端绑定ppmaster.puppet.cn为LB的ip地址
4)客户端同步内容测试,出现Notice: Finished catalog run in 0.09 seconds即表示同步成功
puppet agent --test
四、集群测试
本章将创建一个简单的配置demo来确定集群是否能够正常同步配置
4.1 Master
master端新建测试配置
1)创建module
module的目录结构是固定的,目录的结构一般如下所示:
├── files
├── manifests
└── templates
- files: 属于模块的文件
- manifests: 脚本文件
- templates:模板文件
mkdir -p /etc/puppet/modules/helloworld/{files,templates,manifests}
2)新建模块的init.pp文件
vi /etc/puppet/modules/helloworld/manifests/init.ppclass helloworld{file { '/tmp/hello.txt':owner => 'root',group => 'root',mode => '0440',source => 'puppet:///modules/helloworld/hello_old.txt'}
}
3)配置file
在 /etc/puppet/modules/helloworld/files 预先新建hello_old文件
echo 'helloworld' > /etc/puppet/modules/helloworld/files/hello_old.txt
4)配置入口
编辑入口文件 vi /etc/puppet/manifests/site.pp ,无则新建
node default{notify{ 'show':name => "I am running in default!"}
}
node 'Hostname(agent端的主机名)' {notify{ 'show':name => "I am running in agent!"}include helloworld
}
4.2 Agent
客户端进行puppet agent -t测试,推荐root用户执行
五、总结与展望
puppet是一个优秀的自动化配置管理运维工具,可通过文件的形式管理配置,客户端一条命令即可调用执行。为提升管理能力,后续PUPPET还可以通过foreman平台接入管理,拥有较强的拓展性,因此puppet具体较强的延展能力,深受广大企业信赖
这篇关于PUPPET部署篇的文章就介绍到这儿,希望我们推荐的文章对编程师们有所帮助!