MySQLwithDRBD/Pacemaker/CorosynconLinux
DefinitionofDRBD:DRBD(DistributedReplicatedBlockDevice)
DRBDsynchronizesdataattheblockdevice(typicallyaspinningorsolidstatedisk)transparenttothe
application,databaseandeventhefilesystem.DRBDrequirestheuseofajournalingfilesystemsuchas
ext3orext4.Forthissolutionitactsinanactivestandbymodethismeansthatatanypointintimethe
directoriesbeingmanagedbyDRBDareaccessibleforreadsandwritesonexactlyoneofthetwohostsand
inaccessible(evenforreads)ontheother.Anychangesmadeontheactivehostaresynchronouslyreplicated
tothestandbyhostbyDRBD.
:TheConcept:
Theconceptofanactive/passivefailoverClusteristhefollowing:
Twoservers(nodes).
Theycommunicateoveraclustersoftware(Heartbeat
,Corosync,Pacemaker)
TheyarerunningonDRBDfailoverstoragesystem.
MySQLisonlyrunninginMASTERnode(active),theotheristhePASIVEnode.
YoureachMySQLoveraVirtualIP(ClusterIP)
IncaseofaproblemtheclusterfailovertheresourcesincludingtheVIPtothepassivenode.
Thisfailoveristransparentfortheapplication(aliteSERVICEDOWN).
NetworkandServersettings:
NetworkstaticipsetonBothNode
Deactivateselinux(/etc/sysconfig/selinux)
Tosimplificationtheconfigurationusehostnames.(/etc/sysconfig/network)
Addthetwonodesinthe/etc/hostsfileonBothNode
likethis:
127.0.0.1localhost
192.168.0.251Cluster1
192.168.0.252Cluster2
PasswordlesscommunicationwithBothNode
InstallmysqlwiththehelpofYum(OnBothNodeformountingDRBDcreateddevice)
DRBDInstallation&ConfigurationStepbyStep :
SimpleDownloadelreporelease65.el6.elrepo.noarch.rpmforDRBDpackageinstallation.
RequiredPackages:
1. drbd83utils.x86_64
2. kmoddrbd83.x86_64
Note:Thispackageinstallationwiththehelpofyum
Copythedistconfiguration:
#cppr/etc/drbd.conf/etc/drbd.confDIST
The/etc/drbd.confistheconfigurationfilefordrbd,heremyconfiguration:
looklikethis:
#Youcanfindanexamplein/usr/share/doc/drbd.../drbd.conf.example
include"drbd.d/global_common.conf";
resourceDISK1{
protocolC;
net{
cramhmacalgsha1;
sharedsecret"8a6cxxxxxxxxxxxxxxxxxxxxx49xxxxxxxxfb3";
aftersb0pridiscardzerochanges;
aftersb1pridiscardsecondary;
aftersb2pridisconnect;
rrconflictdisconnect;
}
device/dev/drbd0;
disk/dev/sda5;
metadiskinternal;
onCluster1{
address192.168.0.251:7789;
}
onCluster2{
address192.168.0.252:7789;
}
}
AsyoucanseeIspecifythisparameters:
RESOURCE:Thenameoftheresource
PROTOCOL:InthiscaseCmeanssynchronous
NET:TheSHA1key,thathavethesameinthetwonodes
aftersb0pri:WhenaSplitBrainocurrs,andnodatahavechanged,thetwonodesconnect
normally.
aftersb1pri:Ifsomedatahavebeenchanged,discardthesecondarydataandsynchronizewith
theprimary
aftersb2pri:Ifthepreviousoptionisimpossibledisconnectthetwonodes,inthiscasemanually
SplitBrainsolutionisrequired
rrconflict:Incasethatthepreviousstatementsdontapplyandthedrbdsystemhavearoleconflict,
thesystemdisconnectautomatically.
DEVICE:Virtualdevice,thepatchtothefisicaldevice.
DISK:Fisicaldevice
METADISK:Metadataarestoredinthesamedisk(sdc1)
ON<NODE>:Thenodesthatformthecluster
Creatingtheresource
Thiscommandsinbothnodes
#fdisk/dev/sda
Createpartition
Createthepartitionwithoutformat
[root@node1~]#drbdadmcreatemdDISK1
Writingmetadata...
initializingactivitylog
NOTinitializingbitmap
Newdrbdmetadatablocksuccessfullycreated.
[root@node2~]#drbdadmcreatemdDISK1
Writingmetadata...
initializingactivitylog
NOTinitializingbitmap
Newdrbdmetadatablocksuccessfullycreated.
ActivatetheResource
Besurethatthedrbdmoduleisload(lsmod),ifnotloadit:
#modprobedrbd(OnBothNode)
NowactivatetheresourceDISK1:
[root@node1~]#drbdadmupDISK1
[root@node2~]#drbdadmupDISK1
Synchronize
Onlyinthemasternode,wellsaythatthenode1istheprimary:
#drbdadmoverwritedataofpeerprimaryDISK1
Wellseethatthediskssynchronizationareinprogress,adnthestateisUpToDate/Inconsistent
CheckSynchronizationstatus
#/etc/init.d/drbdstatus(OnBothnode)
FormattheResource
Onlyinthemasternode:
#mkfs.ext4/dev/drbd0
Testing
Mounttheresourceinthenode1
[root@node1~]#mount/dev/drbd0/var/lib/mysql
Ok,nowumountandmarkthenode1likesecondary
[root@node1~]#umount/var/lib/mysql
[node1~]#drbdadmsecondaryDISK1
MarkNode2likePrimaryandmount:
[root@node2~]#drbdadmprimaryDISK1
[root@node2~]#mount/dev/drbd0/var/lib/mysql
IntroductionofCorosync&Pacemaker
PacemakerandCorosynccombinetoprovidetheclusteringlayerthatsitsbetweentheservicesandthe
underlyinghostsandoperatingsystems.Pacemakerisresponsibleforstartingandstoppingservices
ensuringthattheyrerunningonexactlyonehost,deliveringhighavailabilityandavoidingdatacorruption.
CorosyncprovidestheunderlyingmessaginginfrastructurebetweenthenodesthatenablesPacemakertodo
itsjob;italsohandlesthenodesmembershipwithintheclusterandinformsPacemakerofanychanges.
InstallationofCorosync&Pacemakerstepbystep
elreporelease65.el6.elrepo.noarch.rpm(.repofile)forcorosync&Pacemakerpackage
installation
Packageinstallwiththehelpofyum
#yuminstallpacemaker.x86_64corosync.x86_64
configurecorosynctousecryptographictechniquestoensureauthenticityandprivacyofthe
messages,youwillneedtogenerateaprivatekey.CorosyncClusterEngineAuthenticationkey
generator.Gathering1024bitsforkeyfrom/dev/random.
CorosyncKeyInonenodecreatethecorosyncsecuritycomunicationkey.
Corosyncforthefirsttime,youneedtocreatetheauthkeyfileforauthenticationwithincluster
communication.
[root@node1]#corosynckeygen
CorosyncClusterEngineAuthenticationkeygenerator.
Gathering1024bitsforkeyfrom/dev/random.
Presskeysonyourkeyboardtogenerateentropy.
Writingcorosynckeyto/etc/corosync/authkey.
needtocopythatfiletoallofyournodesandputitin/etc/corosync/withuser=root,
group=rootandmode0400.permissionsto400
[root@node1]#scp/etc/corosync/authkeynode2:/etc/corosync/
[root@node1]#ll/etc/corosync/authkey
r1rootroot128july1710:26/etc/corosync/authkey
[root@node2]#ll/etc/corosync/authkey
r1rootroot128july1710:27/etc/corosync/authkey
Nowconfigurethe/etc/corosync/corosync.conf
Looklikethis:
[root@node1]#vi/etc/corosync/corosync.conf
#Pleasereadthecorosync.conf.5manualpage
compatibility:whitetank
totem{
version:2
secauth:off
interface{
member{
memberaddr:192.168.0.251
}
member{
memberaddr:192.168.0.252
}
ringnumber:0
mcastaddr:239.255.255.255
bindnetaddr:192.168.0.0
mcastport:5405
ttl:1
}
transport:udpu
}
logging{
fileline:off
to_logfile:yes
to_syslog:yes
debug:on
logfile:/var/log/cluster/corosync.log
debug:off
timestamp:on
logger_subsys{
subsys:AMF
debug:off
}
}
NowCopycorosync.confconfiguredfileonNode02
[root@node1]#scp/etc/corosync/corosync.confnode2:/etc/corosync/.
Corosynctoloadthequorumandmessaginginterfacesneededbypacemaker,create
/etc/corosync/service.d/pcmkwiththefollowingfragment.
service{
#LoadthePacemakerClusterResourceManager
name:pacemaker
ver:1
}
NowcopyonNode02
[root@node1]#scp/etc/corosync/service.d/pcmknode2:/etc/corosync/service.d/.
crmcommand(clustermanagmentforpacemaker)notfoundinlatestCentos6
#yuminstallcrmsh.x86_64(OnBothNode)
NowcreateLogDirectoryandfileunder/var/log/(BothNode)
#mkdirp/var/log/cluster
#vi/var/log/cluster/corosync.log
Corosyncprocesswouldlaunchpacemaker,thisisnolongerthecase.Pacemakermustbe
launchedafterCorosynchassuccessfullystarted.(BothNode)
#servicecorosyncstart
#servicepacemakerstart
#chkconfigcorosyncon
#chkconfigpacemakeron
crm_monisrunwiththe1optiontoindicatethatitshouldreportonceandthenreturn.A
recommendationwouldbetoalsorunitwithouttheoption(onbothservers)sothatyougeta
continuallyrefreshedviewofthestateoftheclusterincludinganymanagedresources.
CheckOnlineconfiguredClusternodewithcrmcommand(OnBothNode)
[root@node01]#crm_mon1
============
Outputshowlooklikethis:
Lastupdated:MonFeb2717:51:102012
Lastchange:MonFeb2717:50:252012viacrmdonhost1.localdomain
Stack:openais
CurrentDC:host1.localdomainpartitionwithquorum
Version:1.1.63.el6a02c0f19a00c1eb2527ad38f146ebc0834814558
2Nodesconfigured,2expectedvotes
0Resourcesconfigured.
============
Online:[Node01Node02]
ConfiguringandManagingClusterResources(CommandLine)
Usethefollowingcommandstosettheoptionsforatwonodeclustersonly:
[root@node01~]#crmconfigurepropertynoquorumpolicy=ignore
Pacemakerusesresourcestickinessparameterstodeterminewhenresourcesshouldbe
migratedbetweennodestheabsolutevaluesarenotimportant,ratherhowtheycomparewith
thevaluesthatwillsubsequentlybeconfiguredagainstspecificevents;herewesetthestickiness
to100:
[root@node01~]#crmconfigurersc_defaultsresourcestickiness=100
STONITH(ShootTheOtherNodeInTheHead)otherwiseknownasfencingreferstoone
nodetryingtokillanotherintheeventhatitbelievestheotherhaspartiallyfailedandshouldbe
stoppedinordertoavoidanyriskofasplitbrainscenario.Weturnthisoffasthissolutionwill
relyoneachnodeshuttingitselfdownintheeventthatitlosesconnectivitywiththeindependent
host:
[root@node01~]#crmconfigurepropertystonithenabled=false
NowconfigureVirtualIPresourceforthisnodes.Nodeshavetocheckeachotherevery20
seconds:
[root@node1]#crmconfigure
crm(live)configure#primitiveClusterIPocf:heartbeat:IPaddr2paramsip=192.168.1.100
cidr_netmask=32
DRBD
Nowthefilesystem,addDISK1tothecluster
crm(live)configure#primitivedrbd_mysqlocf:linbit:drbdparamsdrbd_resource="DISK1"op
monitorinterval="15s"opstarttimeout="240s"
Definethemountpoint
crm(live)configure#primitivefs_mysqlocf:heartbeat:Filesystemparamsdevice="/dev/drbd0"
directory="/var/lib/mysql"fstype="ext4"
DefineonlyoneMasternode
crm(live)configure#msms_drbd_mysqldrbd_mysqlmetamastermax="1"masternodemax="1"
clonemax="2"clonenodemax="1"notify="true"
MySQL
Nowthemysqlserver
crm(live)configure#primitivemysqldocf:heartbeat:mysqlparamsbinary="/usr/bin/mysqld_safe"
config="/etc/my.cnf"user="mysql"group="mysql"log="/var/log/mysqld.log"
pid="/var/run/mysqld/mysqld.pid"datadir="/var/lib/mysql"socket="/var/lib/mysql/mysql.sock"op
monitorinterval="60s"timeout="60s"opstartinterval="0"timeout="180"opstopinterval="0"
timeout="240"
Groups&Colocations
Withthisgroupweensurethatthedrbd,mysqlandVIPareinthesamenode(master)andtheorder
tostopandstartiscorrectly:
start:fs_mysql>mysqld>ClusterIP
stop:ClusterIP>mysqld>fs_mysql
crm(live)configure#groupgroup_mysqlfs_mysqlmysqldClusterIPmetamigrationthreshold="5"
Thegroupgroup_mysqlallwaysintheMASTERnode
crm(live)configure#colocationmysql_on_drbdinf:group_mysqlms_drbd_mysql:Master
MysqlstartallwaysafterdrbdMASTER
crm(live)configure#ordermysql_after_drbdinf:ms_drbd_mysql:promotegroup_mysql:start
Howtoconfigurepacemakerapachefailoversystemwithpacemaker
crm(live)configure#crmconfigureprimitiveAPACHEocf:heartbeat:apacheparams
configfile="/etc/httpd/conf/httpd.conf"statusurl="http://localhost/serverstatus"opmonitor
interval="40s"
###APACHEresourcename
###configfilepathtoapacheconfigurationfile
###statusurlurltostatuspage(belowhowtoconfiguresone)
###intervaltimebetweenchecks
###InApacheConfigurationfile:
#vi/etc/httpd/conf/httpd.conf
ExtendedStatusOn
<Location/serverstatus>
SetHandlerserverstatus
Orderdeny,allow
Allowfromall
</Location>
Topreventsituationwhenresourceapachemigratetonode002andresourceIPstaysat
node001(Ithappenswhenapacheatnode001hungbutnetworkstackworkswell)weneedto
makecolocation
[root@node1]#crmconfigurecolocationWEB_SITEinf:APACHEClusterIP
TomakepacemakerstartupapacheonlyafterIPissetup.Inotherwordsdescribestartup
orderrun:
[root@node1]#crmconfigureorderSTART_ORDERinf:ClusterIPAPACHE
NowcheckconfigurationonbothNode
#crm_mon1
#crmconfigureshow
#crm_mon
By:AlokRaj
SystemAdministrator
SaigunTechnologiesPvtLtd
