Microsoft Word - SmartX超融合环境搭建oracleRAC实践部署手册.docx

Transcription

1 SmartX ZBS 分布式存储搭建 ORACLE-RAC 实践手册

2 目录 1. 概述 配置 ORACLE-RAC 基础环境 为 Oracle RAC 虚拟机添加存储控制器 创建共享磁盘 将共享磁盘添加到一个或多个虚拟机 安装 ORACLE-RAC 操作系统 安装 VMware Tools 时钟同步 准备其他虚拟机和 RAC 节点 虚机克隆 配置仲裁盘 使用 fdisk 格式化仲裁盘 : 安装 Oracle Grid 软件 验证 Grid 安装的正确性 增加数据盘和 REDO 盘 以 grid 用户创建 DATA 和 REDO 磁盘组 安装 RAC 数据库... 34

3 1. 概述 借助 SmartX 超融合架构, 我们可以利用 3 个节点来搭建 ORACLE-RAC 环境, 实现数据库高可用 在此我们利用 VMware vsphere6.0u2 平台 ( 目前使用比较广泛的虚拟化平台 ) 来部署 ORACLE-RAC 在 SMARTX ZBS 分布式数据存储上配置 Oracle RAC 群集 此过程需要执行以下步骤 : 以立刻置零模式创建共享虚拟磁盘并为磁盘启用多写入器模式 将共享磁盘附加到一个或多个虚拟机 警告 : 继续操作之前, 所有将成为 Oracle RAC 节点的虚拟机都必须处于关闭电源状态 2. 配置 ORACLE-RAC 基础环境 2.1 为 Oracle RAC 虚拟机添加存储控制器 建议 : 在每个 Oracle RAC 虚拟机上, 创建类型相同并处于相同位置 (SCSI 地址 ) 的控 制器 1. 在 vsphere Web Client 中右键单击虚拟机, 然后选择编辑设置 2. 在后续窗口的底部, 从新设备 : 下拉菜单中选择 SCSI 控制器, 然后单击添加 3. 展开新的 SCSI 控制器条目 4. 使总线共享模式保持为 无, 然后选择所需的类型 (LSI Parallel LSI SAS 或 Paravirtual) 注意 : 一般建议将准虚拟控制器类型 (VMware Paravirtual) 用于 RAC 共享磁盘 5. 单击确定 6. 对其余 RAC 虚拟机重复以上过程 2.2 创建共享磁盘

4 利用 vsphere webclient 创建独立模式持久写入的磁盘, 虚拟设备节点建议选择 (1:0), 这样会关联到新的 scsi 控制器, 磁盘选择空间大小后, 设置为立即置零, 磁盘存储位置请选择到 SmartX ZBS 共享 datastore 接下来需要为新增加的共享盘设置为多写入器模式, 如下图 : 对要创建的其余共享磁盘重复以上过程 注意 : 多写入器模式的共享磁盘最多支持 8 个节点同时写入, 对于只读模式, 可以支持最 多 32 个节点同时读取 2.3 将共享磁盘添加到一个或多个虚拟机 创建共享磁盘后, 您需要将其添加到其余 RAC 虚拟机 您可以通过 vsphere Web Client 执行此操作 虚拟磁盘应添加到每个虚拟机上的相同 SCSI 位置 如果某个磁盘位于一个虚拟机上的位置 1:0, 则在 RAC 群集中, 该磁盘应位于所有虚拟机上的位置 1:0 1. 右键单击相应的虚拟机, 然后选择编辑设置 2. 在后续窗口中, 从新设备下拉菜单中选择现有硬盘, 然后单击添加 3. 导航到 SmartX ZBS 共享存储, 然后选择所需的磁盘 4. 单击确定 5. 展开新硬盘条目, 然后适当地修改虚拟设备节点 6. 在共享下拉菜单中, 选择多写入器选项 7. 将磁盘模式更改为独立持久 8. 单击确定以保存更改 9. 对其余虚拟机和共享磁盘重复以上过程

5 3. 安装 ORACLE-RAC 操作系统 将 OEL6.4 安装介质 ISO 文件上传到 datastore, 然后进行安装 Note: This document is specific to Linux guest operating system. Oracle RAC and above versions can be deployed and supported on the list of guest operation systems using VMware multi-writer flag ( 在虚拟机上安装 Linux 操作系统 1. In vsphere Client, select a virtual machine. 2. Click Edit settings. 3. In the Hardware tab of the Virtual Machine Properties pane, select CD/DVD Drive In the Device Status section of this pane, select the Connect at power on check box. 5. In the Device Type section of this pane, select Datastore ISO File and enter the path to your Linux ISO image file. 6. In the Virtual Device Node section of this pane, select CD/DVD Drive 1 from the drop-down menu. 7. Click OK. 8. Power on the virtual machine. 9. Install Linux with the following settings. Custom partitioning scheme: /dev/sda1 ext3 150GB mounted on / Installed packages use the default package selection, plus the following: kernel-uek.x86_64 kmod-oracleasm oracleasm el el5.x86_64 (download from oracleasm-support el5.x86_64 systat el5.x86_ Use below script to install remaining packages: yum install binutils \ compat-libcap1 \

6 compat-libstdc++-33 \ compat-libstdc++-33.i686 \ gcc.x86_64 \ gcc-c++.x86_64 \ glibc.x86_64 \ glibc.i686 \ glibc-devel.x86_64 \ glibc-devel.i686 \ unixodbc.i686 \ unixodbc.x86_64 \ unixodbc-devel.x86_64 \ unixodbc-devel.i686 \ libgcc.x86_64 \ libgcc.i686 \ libstdc++.x86_64 \ libstdc++.i686 \ libstdc++-devel.x86_64 \ libstdc++-devel.i686 \ compat-libstdc++-33.x86_64 \ compat-libstdc++-33.i686 \ libaio.x86_64 \ libaio.i686 \ libaio-devel.x86_64 \ libaio-devel.i686 \ sysstat.x86_64 \ ksh \ libxtst-devel.x86_64 \ libxtst.i686 \ compat-db.x86_64 \ xorg-x11-utils.x86_64 \ xorg-x11-xinit.x86_64 \ make

7 11. View the installed packages with this command: run command rpm q <package name> 12. Confirm that the following packages have been installed: binutils el6 (x86_64) compat-libcap (x86_64) compat-libstdc el6 (x86_64) compat-libstdc el6.i686 gcc el6 (x86_64) gcc-c el6 (x86_64) glibc el6 (i686) glibc el6 (x86_64) glibc-devel el6 (x86_64) glibc-devel el6.i686 ksh libgcc el6 (i686) libaio el6 (x86_64) libaio el6.i686 libaio-devel el6 (x86_64) libaio-devel el6.i686 make el6 sysstat el6 (x86_64) unixodbc el6 (x86_64) or later unixodbc el6.i686 or later unixodbc-devel el6 (x86_64) or later unixodbc-devel el6.i686 or later libgcc el6 (x86_64) libstdc el6 (x86_64) libstdc el6.i686 libstdc++-devel el6 (x86_64) libstdc++-devel el6.i686

8 13. 最后安装下面两个包 : oracle-rdbms-server-11gr2-preinstall ( 从 YUM 安装, 这个包会配置 Oracle RAC 所需系统参数和环 境变量 ) cvuqdisk (Oracle 安装介质中提供这个包, 用于安装前检查 ) 14. 检查配置文件 /etc/sysctl.conf kernel.shmmni = 4096 kernel.sem = fs.file-max = (512 x processes for example for processes) net.ipv4.ip_local_port_range = net.core.rmem_default = net.core.rmem_max = net.core.wmem_default = net.core.wmem_max = kernel.shmall = physical RAM size / pagesize For most systems, this will be the value See Note: for more information. eg: kernel.shmall= (62gb) kernel.shnmax= (32GB) kernel.shmmax = RAM times 0.5 (or higher at customer's discretion - seenote: ) /etc/profile ulimit u ulimit n 依照 oracle 传统, 创建 clusterware 主用户 grid : [root@racnode1 ~]# groupadd -g 1000 oinstall [root@racnode1 ~]# groupadd -g 1200 asmadmin [root@racnode1 ~]# groupadd -g 1201 asmdba [root@racnode1 ~]# groupadd -g 1202 asmoper [root@racnode1 ~]# useradd -m -u g oinstall -G asmadmin,asmdba,asmoper -d /home/grid -s /bin/bash -c "Grid Infrastructure Owner" grid [root@racnode1 ~]# id grid uid=1100(grid) gid=1000(oinstall) groups=1000(oinstall), 1200(asmadmin), 1201(asmdba),1202(asmoper) 设置 grid 帐户的口令 : [root@racnode1 ~]# passwd grid Changing password for user grid. New UNIX password:xxxxxxxxxxx

9 Retype new UNIX password:xxxxxxxxxxx passwd: all authentication tokens updated successfully. 为 grid 用户创建初始化脚本 (/home/grid/.bash_profile): #.bash_profile # OS User: grid # Application: Oracle Grid Infrastructure # Version: Oracle 11g release 2 # Get the aliases and functions if [ -f ~/.bashrc ]; then. ~/.bashrc fi alias ls="ls -FA" # ORACLE_SID # Specifies the Oracle system identifier (SID) # for the Automatic Storage Management (ASM)instance # running on this node. # Each RAC node must have a unique ORACLE_SID. # (i.e. +ASM1, +ASM2,...) ORACLE_SID=+ASM1; export ORACLE_SID # JAVA_HOME # Specifies the directory of the Java SDK and Runtime # Environment. JAVA_HOME=/usr/local/java; export JAVA_HOME # ORACLE_BASE # Specifies the base of the Oracle directory structure # for Optimal Flexible Architecture (OFA) compliant

10 # installations. The Oracle base directory for the # grid installation owner is the location where # diagnostic and administrative logs, and other logs # associated with Oracle ASM and Oracle Clusterware # are stored. ORACLE_BASE=/u01/app/grid; export ORACLE_BASE # ORACLE_HOME # Specifies the directory containing the Oracle # Grid Infrastructure software. For grid # infrastructure for a cluster installations, the Grid # home must not be placed under one of the Oracle base # directories, or under Oracle home directories of # Oracle Database installation owners, or in the home # directory of an installation owner. During # installation, ownership of the path to the Grid # home is changed to root. This change causes # permission errors for other installations. ORACLE_HOME=/u01/app/11.2.0/grid; export ORACLE_HOME # ORACLE_PATH # Specifies the search path for files used by Oracle # applications such as SQL*Plus. If the full path to # the file is not specified, or if the file is not # in the current directory, the Oracle application # uses ORACLE_PATH to locate the file. # This variable is used by SQL*Plus, Forms and Menu. ORACLE_PATH=/u01/app/oracle/common/oracle/sql; export ORACLE_PATH # SQLPATH # Specifies the directory or list of directories that # SQL*Plus searches for a login.sql file. # SQLPATH=/u01/app/common/oracle/sql; export SQLPATH

11 # ORACLE_TERM # Defines a terminal definition. If not set, it # defaults to the value of your TERM environment # variable. Used by all character mode products. ORACLE_TERM=xterm; export ORACLE_TERM # NLS_DATE_FORMAT # Specifies the default date format to use with the # TO_CHAR and TO_DATE functions. The default value of # this parameter is determined by NLS_TERRITORY. The # value of this parameter can be any valid date # format mask, and the value must be surrounded by # double quotation marks. For example: # # NLS_DATE_FORMAT = "MM/DD/YYYY" # NLS_DATE_FORMAT="DD-MON-YYYY HH24:MI:SS"; export NLS_DATE_FORMAT # TNS_ADMIN # Specifies the directory containing the Oracle Net # Services configuration files like listener.ora, # tnsnames.ora, and sqlnet.ora. TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN # ORA_NLS11 # Specifies the directory where the language, # territory, character set, and linguistic definition # files are stored. ORA_NLS11=$ORACLE_HOME/nls/data; export ORA_NLS11 # PATH # Used by the shell to locate executable programs;

12 # must include the $ORACLE_HOME/bin directory. PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin PATH=${PATH}:/u01/app/common/oracle/bin export PATH # LD_LIBRARY_PATH # Specifies the list of directories that the shared # library loader searches to locate shared object # libraries at runtime. LD_LIBRARY_PATH=$ORACLE_HOME/lib LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib export LD_LIBRARY_PATH # CLASSPATH # Specifies the directory or list of directories that # contain compiled Java classes. CLASSPATH=$ORACLE_HOME/JRE CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib export CLASSPATH # THREADS_FLAG # All the tools in the JDK use green threads as a # default. To specify that native threads should be # used, set the THREADS_FLAG environment variable to # "native". You can revert to the use of green # threads by setting THREADS_FLAG to the value # "green". THREADS_FLAG=native; export THREADS_FLAG

13 # TEMP, TMP, and TMPDIR # Specify the default directories for temporary # files; if set, tools that create temporary files # create them in one of these directories. export TEMP=/tmp export TMPDIR=/tmp # UMASK # Set the default file mode creation mask # (umask) to 022 to ensure that the user performing # the Oracle software installation creates files # with 644 permissions. umask 创建 oracle 账号 oracle : [root@racnode1 ~]# groupadd -g 1300 dba [root@racnode1 ~]# groupadd -g 1301 oper [root@racnode1 ~]# useradd -m -u g oinstall -G dba,oper,asmdba -d /home/oracle -s /bin/bash -c "Oracle Software Owner" oracle [root@racnode1 ~]# id oracle uid=1101(oracle) gid=1000(oinstall) groups=1000(oinstall),1201(asmdba),1300(dba),1301(oper) 然后为 oracle 用户创建登录环境 (/home/oracle/.bash_profile) #.bash_profile # OS User: oracle # Application: Oracle Database Software Owner # Version: Oracle 11g release 2 # Get the aliases and functions if [ -f ~/.bashrc ]; then. ~/.bashrc

14 fi alias ls="ls -FA" # ORACLE_SID # Specifies the Oracle system identifier (SID) for # the Oracle instance running on this node. # Each RAC node must have a unique ORACLE_SID. # (i.e. racdb1, racdb2,...) ORACLE_SID=racdb1; export ORACLE_SID # ORACLE_UNQNAME # In previous releases of Oracle Database, you were # required to set environment variables for # ORACLE_HOME and ORACLE_SID to start, stop, and # check the status of Enterprise Manager. With # Oracle Database 11g release 2 (11.2) and later, you # need to set the environment variables ORACLE_HOME # and ORACLE_UNQNAME to use Enterprise Manager. # Set ORACLE_UNQNAME equal to the database unique # name. ORACLE_UNQNAME=racdb; export ORACLE_UNQNAME # JAVA_HOME # Specifies the directory of the Java SDK and Runtime # Environment. JAVA_HOME=/usr/local/java; export JAVA_HOME # ORACLE_BASE # Specifies the base of the Oracle directory structure # for Optimal Flexible Architecture (OFA) compliant # database software installations.

15 ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE # ORACLE_HOME # Specifies the directory containing the Oracle # Database software. ORACLE_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1; export ORACLE_HOME # ORACLE_PATH # Specifies the search path for files used by Oracle # applications such as SQL*Plus. If the full path to # the file is not specified, or if the file is not # in the current directory, the Oracle application # uses ORACLE_PATH to locate the file. # This variable is used by SQL*Plus, Forms and Menu. ORACLE_PATH=/u01/app/common/oracle/sql; export ORACLE_PATH # SQLPATH # Specifies the directory or list of directories that # SQL*Plus searches for a login.sql file. # SQLPATH=/u01/app/common/oracle/sql; export SQLPATH # ORACLE_TERM # Defines a terminal definition. If not set, it # defaults to the value of your TERM environment # variable. Used by all character mode products. ORACLE_TERM=xterm; export ORACLE_TERM # NLS_DATE_FORMAT # Specifies the default date format to use with the # TO_CHAR and TO_DATE functions. The default value of

16 # this parameter is determined by NLS_TERRITORY. The # value of this parameter can be any valid date # format mask, and the value must be surrounded by # double quotation marks. For example: # # NLS_DATE_FORMAT = "MM/DD/YYYY" # NLS_DATE_FORMAT="DD-MON-YYYY HH24:MI:SS"; export NLS_DATE_FORMAT # TNS_ADMIN # Specifies the directory containing the Oracle Net # Services configuration files like listener.ora, # tnsnames.ora, and sqlnet.ora. TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN # ORA_NLS11 # Specifies the directory where the language, # territory, character set, and linguistic definition # files are stored. ORA_NLS11=$ORACLE_HOME/nls/data; export ORA_NLS11 # PATH # Used by the shell to locate executable programs; # must include the $ORACLE_HOME/bin directory. PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin PATH=${PATH}:/u01/app/common/oracle/bin export PATH # LD_LIBRARY_PATH # Specifies the list of directories that the shared # library loader searches to locate shared object # libraries at runtime.

17 LD_LIBRARY_PATH=$ORACLE_HOME/lib LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib export LD_LIBRARY_PATH # CLASSPATH # Specifies the directory or list of directories that # contain compiled Java classes. CLASSPATH=$ORACLE_HOME/JRE CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib export CLASSPATH # THREADS_FLAG # All the tools in the JDK use green threads as a # default. To specify that native threads should be # used, set the THREADS_FLAG environment variable to # "native". You can revert to the use of green # threads by setting THREADS_FLAG to the value # "green". THREADS_FLAG=native; export THREADS_FLAG # TEMP, TMP, and TMPDIR # Specify the default directories for temporary # files; if set, tools that create temporary files # create them in one of these directories. export TEMP=/tmp export TMPDIR=/tmp # UMASK # Set the default file mode creation mask # (umask) to 022 to ensure that the user performing

18 # the Oracle software installation creates files # with 644 permissions. umask 创建 RAC 安装所需目录结构 : # mkdir -p /u01/app/11.2.0/grid # chown grid:oinstall /u01/app/11.2.0/grid # chmod -R 775 /u01/app/11.2.0/grid # mkdir -p /u01/app/oracle # chown -R oracle:oinstall /u01/app/oracle # chmod -R 775 /u01/app/oracle 3.1 安装 VMware Tools To install VMware Tools 1. Confirm that the Linux virtual machine is powered on. 2. In vsphere Client, right-click the virtual machine and select Guest > Install VMware Tools. 3. In the Linux guest OS, create a mount point and run the following command: mkdir /mnt/cdrom 4. Mount the CD-ROM with the following command: mount /dev/cdrom /mnt/cdrom 5. Install VMware Tools using RPM, with the following command: rpm -ivh /mnt/cdrom/vmwaretools rpm As an example, is the version of VMware Tools used here. 6. Configure VMware Tools, by running the following command: /usr/bin/vmware-config-tools.pl 7. Unmount the CD-ROM with the following command: umount /mnt/cdrom 8. Click VM in the virtual machine menu, and click Guest > End VMware Tools Install.

19 3.2 时钟同步 按照 VMware 官方文档 Timekeeping best practices for Linux guests ( 来配置在 Linux 上配置 NTP, 同时停掉虚拟机中 VMware tools 与 ESX 主机的时钟同步. 有两种方法停止 VMware tools 的时钟同步进程 : Set tools.synctime = "False" in the configuration file (.vmx file) of the virtual machine. Run the vmware-guestd --cmd "vmx.set_option synctime 1 0" command in the guest operating system. 编辑 /etc/sysconfig/ntpd 增加 -x 选项, 如下所示 : # Drop root to id 'ntp:ntp' by default. OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid" /etc/ntp.conf 示例 : tinker panic 0 driftfile /var/lib/ntp/drift statistics loopstats statsdir /var/log/ntp/ logfile /var/log/ntp/ntpd.log filegen peerstats file peers type day link enable filegen loopstats file loops type day link enable server

20 4. 准备其他虚拟机和 RAC 节点 现在第一个 RAC 节点 (RAC1) 所用的虚拟机就创建好了, 现在可以用这个虚拟机克隆出其他 RAC 节点所用的虚拟机, 如果前面的步骤都执行正确, 现在就可以把这个虚拟机制成模板, 留作以后使用 大概步骤如下 : In vcenter, clone RAC1. 在新创建的虚拟机里删除 RAC1 所带的共享磁盘 重新添加仲裁盘, 仲裁盘同样需要开启 multi-writer 开关 修改客户机 IP 地址. 4.1 虚机克隆 1. RAC1 停机 2. In vsphere Client, select the virtual machine, RAC1. 3. Right-click the name of the virtual machine and select Clone. 4. In the Clone Virtual Machine wizard on the Name and Location page, enter RAC2 for the Name. Click Next. 5. On the Host / Cluster page, select a host or cluster. Click Next. 6. On the Specific Host page, select a unique ESX host (that is, one not running a RAC node virtual machine). Click Next. 7. On the Datastore page, select a VMFS datastore. Click Next. 8. On the Disk Format page, select Same format as source. Click Next. 9. On the Guest Customization page, select customize using the customization wizard. 10. config IP address for eth0 and eth1

21 11. On the Ready to Complete page, click Finish.

22 4.2 配置仲裁盘 虚拟机克隆之后, 指向仲裁盘的路径会失效, 所以需要先在虚机配置里删除这个硬盘重建, 同时把 SCSI 控制器类型改为 paravirtual. 按照下述步骤配置 RAC2: 删除共享盘 1. In vsphere Client, select a virtual machine. 2. Right-click the name of the virtual machine and select Edit Settings to display the Virtual Machine Properties pane. 3. Select hard disk 2 and click Remove. In the Removal Options section, select Remove from virtual machine and click OK. 再新建一次 1. In vsphere Client, select a virtual machine. 2. Right-click the name of a virtual machine and select Edit Settings to display the Virtual Machine Properties pane. 3. Click Add to display the Add Hardware wizard. 4. In the Add Hardware wizard select Hard Disk for the Device Type, and click Next. 5. In the Disk section of the Select a Disk page, select Use an existing virtual disk and click Next. 6. On the Select Existing Disk page, enter the path to the first VMDK of the virtual machine (selected in step 1) used for the CRS and voting datastores. Click Next. 7. On the Advanced Options page, in the Virtual Device Node section, select SCSI (1:0). In the Mode section, select Independent and Persistent. Click Next. 8. On the Ready to Complete page, click Finish. 9. Click OK.

23 修改 SCSI 控制器类型为 paravirtual 1. In vsphere Client, select the virtual machine that was selected in the preceding step Right-click the name of the virtual machine and select Edit Settings to display the Virtual Machine Properties pane. 3. In the device list, select SCSI controller In the SCSI Controller Type section, click Change Type. 5. Select VMware Paravirtual. 6. Click OK, and click OK again.

24 4.3 使用 fdisk 格式化仲裁盘 : 必须在一个节点上对共享盘进行分区和格式化 在 Linux 上可以使用系统自带的 fdisk 工具进行 步骤如下 : 仲裁盘分区 : 1. Log in as root on node1. 2. Enter the command: fdisk /dev/sdb 3. Enter n to create a new partition. 4. Enter p to create a primary partition. 5. Enter 1 to create the first partition. 6. Press Enter to take the default value of 1 for the first cylinder. 7. Press Enter to take the default value of for the last cylinder. 8. Enter w to write the new partition table to disk and exit fdisk. The output from the fdisk command should like somewhat like the following: root@vmorarac1 ~# fdisk /dev/sdb Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable. Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite) Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder ( , default 1): Using default value 1 Last cylinder or +size or +sizem or +sizek ( , default 20480): Using default value Command (m for help): w The partition table has been altered!

25 Calling ioctl() to re-read partition table. Syncing disks.

26 5. 安装 Oracle Grid 软件 现在我们有两个虚机, 可以开始安装 grid 环境了. 安装 Oracle Grid Infrastructure 1. 用 oracleasm 在 /dev/sdb1 创建 griddisk CRS1. 2. 将 Oracle RAC 安装包 ( ) 上传到 RAC1 并解压 3. 切换到 grid 用户启动 Oracle Grid 安装程序./runInstaller 4. 在第一页安装选项页, 选择 Install and Configure Infrastructure for a Cluster. Click Next. 5. On the Installation Type page, select Advanced Installation and click Next. 6. On the Product Languages page, select English and click Next. 7. On the Grid Plug and Play page, enter rac for the Cluster Name, rac-scan.localdomain for the SCAN Name and 1521 for the SCAN Port. Click Next. 8. On the Cluster Node Information page, click Add to enter a Hostname and Virtual IP Name. Repeat the process for each of the four hosts, as follows: Hostname RAC1 RAC2 Virtual IP Name RAC1-vip RAC2-vip 9. Click Next. 10. On the Network Interface Usage page, enter the following information for the Interface Name, Subnet, and Interface Type: Interface Name Subnet Interface Type eth Public eth Private 11. Click Next. 12. On the Storage Option page, select Automatic Storage Management (ASM), and click Next. 13. On the Create ASM Disk Group page, enter CRS for the Disk Group Name, and select Normal for Redundancy.

27 14. In the Add Disks section of the Create ASM Disk Group page, select Candidate Disks. Make the following entries: Disk Path Size (in MB) Status ORCL:CRSDISK Candidate 15. Select the check box to the left of each of these entries, and click Next. 16. On the ASM Password page, select Use same passwords for these accounts and enter the password oracle in the Specify Password and Confirm Password fields. Click Next. 17. On the Failure Isolation page, select Do not use Intelligent Platform Management Interface (IPMI). Click Next. 18. On the Operating System Groups page, make the following entries: Field ASM Database Administrator (OSDBA) Group ASM Instance Administration Operator (OSOPER) Group ASM Instance Administrator (OSASM) Group Selection asmdba asmoper asmadmin 19. Click Next. 20. On the Installation Location page, enter /u01/app/grid for the Oracle Base, and /u01/app/11.2.0/grid for the Software Location. Click Next 21. On the Create Inventory page, enter /u01/app/orainventory for the Inventory Directory. Click Next. 22. Wait while the setup application conducts the prerequisite checks and installs Grid on each node. After setup is complete, execute the following configuration scripts on the four nodes VMORARAC1, VMORARAC2. These scripts must be executed in a terminal window as root. /u01/app/orainventory/orinstroot.sh /u01/app/11.2.0/grid/root.sh Note: don t run root.sh scripts on 2 nodes at the same time, or else It will fail. 23. Click OK, and click Finish.

28 6. 验证 Grid 安装的正确性 The following figures show how to run various RAC commands in the guest OS to check the status of the cluster. Figure 1. Check the CRS Status on the First Node as Grid User Figure 2. Check for Cluster Nodes

29 Figure 3. Check for Cluster Registry (OCR) Figure 4. Check Voting Disk

30 7. 增加数据盘和 REDO 盘 In this section, the database disks for data and redo logs are added to the virtual machine RAC nodes in the following way: Data and redo datastores are created with the assigned LUNs, and corresponding VMDKs are added to the virtual machine RAC1. The SCSI controller type is set to VMware Paravirtual. The same Data and Redo VMDKs are added to the remaining node (RAC2) by selecting the VMDKs that were created for RAC1. Again, the SCSI controller type is set to VMware Paravirtual. The added disks are formatted using the Linux fdisk utility. Configuration parameters are added for each virtual machine. The procedures in this section describe the steps needed for one virtual machine. Repeat the steps in this section for each of the four virtual machines named RAC1, RAC2. To add DATA or REDO disks to the first virtual machine 1. In vsphere Client, select the virtual machine. 2. Right-click the name of the virtual machine and select Edit Settings to display the Virtual Machine Properties pane. 3. Click Add to display the Add Hardware wizard. 4. In the Add Hardware wizard select Hard Disk for the Device Type, and click Next. 5. In the Disk section of the Select a Disk page, select Create a new virtual disk. Set the disk size to 1000GB for DATA or 900GB for REDO, and select the Support clustering features such as Fault Tolerance check box. Also, for vsphere 5, in the Disk Provisioning section, select Thick Provision Eager Zeroed. Select DATA01 for DATA, or REDO01 for REDO to specify a datastore. Click Next. 6. On the Compatibility Mode page, click Next. 7. On the Advanced Options page, in the Virtual Device Node section, select SCSI (1:1) for REDO, or SCSI (2:0) for DATA. Set the Mode to Independent. Click Next 8. On the Ready to Complete page, click Finish. 9. Click OK. 10. Repeat steps 1 through 9 for all the DATA datastores. Change step 7 to use SCSI (2:1)/ SCSI (3:0)/ SCSI (3:1) for the other DATA datastores.

31 To set the SCSI controller type to paravirtual 1. Select the virtual machine that was selected in the preceding step Right-click the name of the virtual machine and select Edit Settings to display the Virtual Machine Properties pane. 3. In the device list, select SCSI controller In the SCSI Bus Sharing section, select None, keep the default selection. 5. In the SCSI Controller Type section, click Change Type. 6. Select VMware Paravirtual. 7. Click OK, and click OK again. To format disks using the Linux fdisk utility 1. Follow the steps in Section 4.3 to format one disk. 2. Repeat, using fdisk to format each disk. To add configuration parameters for each Oracle RAC virtual machine Add Configuration parameters for each Oracle RAC virtual machine as per Disabling simultaneous write protection provided by VMFS using the multi-writer flag ( To add configuration parameters to the virtual machines 1. In vsphere Client, select a virtual machine. 2. Right-click the name of the virtual machine and select Edit Settings to display the Virtual Machine Properties pane. 3. Click the Options tab, click Advanced, and click General. 4. Click Configuration Parameters. 5. On the Configuration Parameters page, add rows and entries for the shared disks in the virtual machine s configuration parameters as shown in the following screenshot.

32 The devices for the virtual machine should look like the following screenshot.

33 8. 以 grid 用户创建 DATA 和 REDO 磁盘组 为安装数据库做准备 1. 使用 oracleasm 创建 4 个数据盘 DATA1 到 DATA4, 使用的设备为 /dev/sdd1 到 /dev/sdg1 用设备 /dev/sdc1 创建 REDO 磁盘 REDO1. 2. 以 grid 用户登录节点 1 (RAC1), 启动 ASM 配置助手 (asmca), 创建两个磁盘组 DATADG ( 选择所有 DATA 标签盘 DATA1 到 DATA4) 和 REDODG ( 选择磁盘 REDO1). 3. In the ASM Configuration Assistant, on the Create Disk Group page, enter DATADG for the Disk Group Name. In the Redundancy section, select External (None). Click OK, and click OK again. 4. On the Create Disk Group page, in the Select Member Disks section, select ORCL:DATA1, ORCL:DATA2, ORCL:DATA3, and ORCL:DATA4. Set the size of each to 1000GB. Click OK. 5. On the Create Disk Group page, enter REDODG for the Disk Group Name. In the Redundancy section, select External (None). Click OK, and click OK again. 6. On the Create Disk Group page, in the Select Member Disks section, select ORCL:REDO1,,. Set the size of each to 900GB. Click OK.

34 9. 安装 RAC 数据库 To install and create the Oracle Database 11g R2 RAC 1. Log in into RAC1 as oracle user and launch runinstaller from the Oracle software location to install the Oracle binaries. 2. In the Oracle Installer wizard, on the Configure Security Updates page, leave the field blank, and do not select I wish to receive security updates via My Oracle Support. Click Next. 3. On the Installation Option page, select Install database software only. Click Next. 4. On the Grid Options page, select Real Application Clusters database installation. Select the four nodes RAC1, RAC2. Click Next. 5. On the Product Languages page, select English. Click Next. 6. On the Database Edition page, select Enterprise Edition (4.29GB). Click Next. 7. On the Installation Location page, enter /u01/app/oracle for Oracle Base, and /u01/app/oracle/product/11.2.0/dbhome_1 for Software Location. Click Next. 8. On the Operating System Groups page, select dba for the Database Administrator (OSDBA) Group and oinstall for the Database Operator (OSOPER) Group. Click Next. 9. Review the configuration on the Summary page, and click Finish. 10. After the installation is complete, open a terminal window, log in as root, and run the following script: /u01/app/oracle/product/11.2.0/dbhome_1/root.sh Repeat for each of the four nodes, and click OK. 11. On the Finish page, click Close. 12. Create the Oracle RAC database using Oracle Database Configuration Assistant (DBCA).