Docker下Greenplum集群安装配置-网络改装版

2018年7月29日

docker下greenplum 集群安装配置

作者 版本 备注
方后国 1.0 初建文档

文档说明
本安装测试环境Centos7,其他系统版本未做测试。请自行测试。
本文部分内容来自网络,如有抄袭,请联系作者删除

创建容器

docker run –hostname mdw –name gp_master –privileged -p 2345:2345 -it centos /bin/bash
docker run –hostname sdw1 –name gp_db1 –privileged -p 2346:2345 -it centos /bin/bash
docker run –hostname sdw2 –name gp_db2 –privileged -p 2347:2345 -it centos /bin/bash

注意:
如果上面的命令不加–hostname参数启动容器,会出现在容器内容是无法修改主机名称的,具体报错如下。
修改/etc/hostname也是无效的。
修改/etc/sysconfig/network也是无效的。

[gpadmin@mdw conf]$ hostnamectl set-hostname ddd
Failed to create bus connection: No such file or directory

进入容器,在每一个容器做如下配置
yum install -y net-tools which openssh-clients openssh-server less zip unzip iproute.x86_64

docker中默认没有启动ssh,为了方便各节点之间的互连,启动docker的每个节点里面的ssh,并创建相关的认证key

ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key
ssh-keygen -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key
ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key
/usr/sbin/sshd

配置主机域名映射关系

vi /etc/hosts

172.17.0.2 mdw
172.17.0.3 sdw1
172.17.0.4 sdw2

修改所有节点里面的 /etc/sysconfig/network 文件,保持主机名一致

vi /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=mdw

创建greenplum的用户和用户组

在每个节点中创建greenplum的用户和用户组,方便安装greenplum集群。另
groupadd -g 530 gpadmin
useradd -g 530 -u 530 -m -d /home/gpadmin -s /bin/bash gpadmin
chown -R gpadmin:gpadmin /home/gpadmin
passwd gpadmin
输入gpadmin用户名对应的密码

修改每个节点上的文件打开数量限制

vi /etc/security/limits.conf
End of file
soft nofile 65536
hard nofile 65536
soft nproc 131072
hard nproc 131072

关闭每个节点上的防火墙,关闭selinux(目前没有发现防火墙的配置,此处没有处理)
vi /etc/selinux/config
SELINUX=disabled
SELINUXTYPE=targeted

在master节点上安装greenplum, 以下过程只需要在master节点处理
mkdir gpfoler
chown -R gpadmin:gpadmin gpfoler
请自行自行下载greenplum zip安装包。

复制greenplum-db-5.3.0-rhel7-x86_64.zip到gpfolder
chown -R gpadmin:gpadmin greenplum-db-5.3.0-rhel7-x86_64.zip
切换到gpadmin账号
su – gpadmin
解压下载后的zip文件
cd /gpfoler
unzip greenplum-db-5.3.0-rhel7-x86_64.zip
赋予文件执行权限
chmod +x greenplum-db-5.3.0-rhel7-x86_64.bin
执行安装文件
./greenplum-db-5.3.0-rhel7-x86_64.bin
yes –> /home/gpadmin/greenplum-db-5.3.0–> yes –>yes
安装期间需要修改默认安装目录,输入/home/gpadmin/greenplum-db-5.3.0
为了方便安装集群,greenplum提供了批量操作的指令,通过创建配置文件,以使用批处理命令, 以下的当前路径是/home/gpadmin
note: 在gpadmin下可能不能创建这两个文件,需要切换到root创建
exit –切换到root账户下
cd /home/gpadmin/
mkdir conf
cd conf
touch hostlist

vi hostlist并输入以下内如
mdw
sdw1
sdw2

touch seg_hosts

vi seg_hosts并输入以下内容

sdw1
sdw2

chown -R gpadmin:gpadmin /home/gpadmin/conf
[gpadmin@mdw ~]$ cat ./conf/hostlist
mdw
sdw1
sdw2
[gpadmin@mdw ~]$ cat ./conf/seg_hosts
sdw1
sdw2

设置环境变量,打通所有节点
greenplum-db/greenplum_path.sh中保存了运行greenplum的一些环境变量设置,包括GPHOOME、PYTHONHOME等设置,在gpadmin账号下设置环境变量生效,然后打通所有节点
[gpadmin@mdw ~]$ source /home/gpadmin/greenplum-db/greenplum_path.sh
[gpadmin@mdw ~]$ gpssh-exkeys -f /home/gpadmin/conf/hostlist — 中间会插入一个密码123456
[STEP 1 of 5] create local ID and authorize on local host
… /home/gpadmin/.ssh/id_rsa file exists … key generation skipped
[STEP 2 of 5] keyscan all hosts and update known_hosts file

[STEP 3 of 5] authorize current user on remote hosts
… send to sdw1
… send to sdw2

[STEP 4 of 5] determine common authentication file content

[STEP 5 of 5] copy authentication files to all remote hosts
… finished key exchange with sdw1
… finished key exchange with sdw2

[INFO] completed successfully

成功打通,后面就可以使用以下命令开启批量操作
注意:使用gpssh-exkeys命令时一定要使用gpadmin身份,因为会在/home/gpadmin/.ssh中生成ssh的免密码登录秘钥,如果使用了其它账号登录,则会生成密钥在其它账号下,后面在gpadmin账号下就无法使用gpssh的批处理命令

使用gpssh的批处理命令
[gpadmin@mdw ~]$ gpssh -f /home/gpadmin/conf/hostlist
Note: command history unsupported on this machine …
=> pwd
[sdw] /home/gpadmin
[ mdw] /home/gpadmin
=> exit
这里pwd命令是linux中的查看路径命令,在这里也是查看批量操作时所在的路径,从中可以看到已经成功连通了三个节点
打包master节点上的安装包
cd /home/gpadmin
tar -cf gp.5.3.tar *
使用gpscp命令将这个文件复制到每一台机器上
gpscp -f /home/gpadmin/conf/hostlist gp.5.3.tar =:/home/gpadmin/ –可以看到其他的两个节点有/home/pgadmin/gp.5.4.tar

批量解压,并建立软链接
cd /home/gpadmin/conf
[gpadmin@mdw conf]$ gpssh -f hostlist
=> cd /home/gpadmin
[sdw]
[mdw]
=> tar -xf gp.5.4.tar
[sdw]
[mdw]
=> ln -s ./greenplum-db-5.3.0 greenplum-db
[sdw]
[mdw]

初始化安装数据库
批量创建数据目录 结果是/home/gpadmin/gpdata
[gpadmin@mdw conf]$ gpssh -f hostlist
=> mkdir gpdata
[ mdw]
[sdw2]
[sdw1]
=> cd gpdata
[ mdw]
[sdw2]
[sdw1]
=> mkdir gpmaster gpdatap1 gpdatap2 gpdatam1 gpdatam2
[ mdw]
[sdw2]
[sdw1]
=> exit

在每个节点上配置.bash_profile环境变量 对于这一步,查看数据节点的greenplum_path.sh的确切位置 使用 su – gpadmin
[gpadmin@mdw ~]$ cd
[gpadmin@mdw ~]$ cat .bash_profile
.bash_profile
Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
source /home/gpadmin/greenplum-db/greenplum_path.sh
export MASTER_DATA_DIRECTORY=/home/gpadmin/gpdata/gpmaster/gpseg-1
export PGPORT=2345
export PGDATABASE=postgres
[gpadmin@mdw ~]$ source .bash_profile

在主节点上初始化配置文件
ARRAY_NAME=”Greenplum”
MACHINE_LIST_FILE=/home/gpadmin/conf/seg_hosts
SEG_PREFIX=gpseg
PORT_BASE=33000
declare -a DATA_DIRECTORY=(/home/gpadmin/gpdata/gpdatap1 /home/gpadmin/gpdata/gpdatap2)
MASTER_HOSTNAME=mdw
MASTER_DIRECTORY=/home/gpadmin/gpdata/gpmaster
MASTER_PORT=2345
TRUSTED_SHELL=/usr/bin/ssh
MIRROR_PORT_BASE=43000
REPLICATION_PORT_BASE=34000
MIRROR_REPLICATION_PORT_BASE=44000
declare -a MIRROR_DATA_DIRECTORY=(/home/gpadmin/gpdata/gpdatam1 /home/gpadmin/gpdata/gpdatam2)

chown -R gpadmin:gpadmin /home/gpadmin/conf/gpinitsystem_config

[gpadmin@mdw ~]$ cat /home/gpadmin/conf/gpinitsystem_config

初始化配置文件解读:
ARRAY_NAME=”Greenplum”
MACHINE_LIST_FILE=/home/gpadmin/conf/seg_hosts
Segment 的名称前缀
SEG_PREFIX=gpseg
Primary Segment 起始的端口号
PORT_BASE=33000
指定 Primary Segment 的数据目录
declare -a DATA_DIRECTORY=(/home/gpadmin/gpdata/gpdatap1 /home/gpadmin/gpdata/gpdatap2)
Master 所在机器的 Hostname
MASTER_HOSTNAME=mdw
指定 Master 的数据目录
MASTER_DIRECTORY=/home/gpadmin/gpdata/gpmaster
Master 的端口
MASTER_PORT=2345
指定Bash的版本
TRUSTED_SHELL=/usr/bin/ssh
Mirror Segment起始的端口号
MIRROR_PORT_BASE=43000
Primary Segment 主备同步的起始端口号
REPLICATION_PORT_BASE=34000
Mirror Segment 主备同步的起始端口号
MIRROR_REPLICATION_PORT_BASE=44000
Mirror Segment 的数据目录
declare -a MIRROR_DATA_DIRECTORY=(/home/gpadmin/gpdata/gpdatam1 /home/gpadmin/gpdata/gpdatam2)

在主节点初始化数据库
gpinitsystem -c /home/gpadmin/conf/gpinitsystem_config
20180729:03:37:12:029533 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args:
20180729:03:37:12:029533 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: ‘postgres (Greenplum Database) 5.3.0 build commit:2155c5a8cf8bb7f13f49c6e248fd967a74fed591’
20180729:03:37:13:029533 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: ‘PostgreSQL 8.3.23 (Greenplum Database 5.3.0 build commit:2155c5a8cf8bb7f13f49c6e248fd967a74fed591) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Dec 5 2017 19:10:20’
20180729:03:37:13:029533 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master…
20180729:03:37:13:029533 gpstate:mdw:gpadmin-[INFO]:-Gathering data from segments…
……….
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:-Greenplum instance status summary
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:—————————————————–
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Master instance = Active
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Master standby = No master standby configured
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Total segment instance count from metadata = 8
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:—————————————————–
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Primary Segment Status
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:—————————————————–
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Total primary segments = 4
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Total primary segment valid (at master) = 4
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Total primary segment failures (at master) = 0
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Total number of postmaster.pid files missing = 0
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Total number of postmaster.pid files found = 4
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Total number of postmaster.pid PIDs missing = 0
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Total number of postmaster.pid PIDs found = 4
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Total number of /tmp lock files missing = 0
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Total number of /tmp lock files found = 4
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Total number postmaster processes missing = 0
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Total number postmaster processes found = 4
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:—————————————————–
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Mirror Segment Status
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:—————————————————–
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Total mirror segments = 4
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Total mirror segment valid (at master) = 4
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Total mirror segment failures (at master) = 0
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Total number of postmaster.pid files missing = 0
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Total number of postmaster.pid files found = 4
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Total number of postmaster.pid PIDs missing = 0
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Total number of postmaster.pid PIDs found = 4
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Total number of /tmp lock files missing = 0
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Total number of /tmp lock files found = 4
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Total number postmaster processes missing = 0
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Total number postmaster processes found = 4
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Total number mirror segments acting as primary segments = 0
20180729:03:37:23:029533 gpstate:mdw:gpadmin-[INFO]:- Total number mirror segments acting as mirror segments = 4
输出以上的内容,表示gp的集群创建成功了。
如果gpinitsystem不能成功,就在master节点的/home/gpadmin/gpAdminLogs/的gpinitsystem_*.log文件中查看日志信息,找出原因进行修改,然后再重新执行gpinitsystem进行初始化安装。
出现如下问题:ping: socket: Operation not permitted
命令行执行 su – gpadmin
setcap cap_net_raw=ep $(which ping)

安装完成后使用 su – gpadmin
psql
[gpadmin@mdw conf]$ psql
psql (8.3.23)
Type “help” for help.
postgres=#

下一期我将用Python实现Linux批量安装软件。

发表评论

电子邮件地址不会被公开。 必填项已用*标注