###############################################
在cs6.5,cs7.2上安装grid11.2.0.1 和database11.2.0.1还是有问题
1.i386与i686的问题
2.cs7.2中的semmni内核参数问题 已经设置了kernel.sem = 250 32000 100 128这四个值，但安装界面不认识，只认识semmni参数名
所以选择cs5.5老版本
CRS-4664: Node oracle successfully pinned.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
ADVM/ACFS is not supported on centos-release-7-2.1511.el7.centos.2.10.x86_64
http://www.cnblogs.com/taowang2016/p/3316307.html ADVM/ACFS is not supported on centos-release-5-8.el5.centos
从Oracle11.2开始，ASM不仅是一个磁盘组，他还提供了了一个卷管理器，称为ADVM（asm dynamic volume manager）。ADVM向用户提供卷管理服务，并提供标准的磁盘设备驱动程序。利用ADVM，可以在卷组中创建一个或多个卷，每个卷对应操作系统中的 一个设备文件，这些卷是可以动态扩展的，就像操作系统中的卷或者利用第三方软件创建的卷一样，应用程序也可以对ADVM卷中的数据进行读写操作。
在 ADVM卷中可以创建ACFS文件系统，ACFS是一种跨平台的、可扩展的集群文件系统，多个节点可以同时访问asfs中的文件。ACFS不仅可以作为 oracle数据库软件的安装路径，还可以用来存储数据库中的警告文件和跟踪文件，还可以存储诸如视频、声音、图像、文本等类型的文件。
但是在CentOS5.8上面安装rac的时候，却发现 ADVM/ACFS is not supported on centos-release-5-8.el5.centos(同样适用于单机环境的asm安装，在安装grid的最后执行root.sh的时候出现 的。)
ADVM/ACFS is not supported on centos-release-5-5.el5.centos 解决方法
centOS 5.5 安装 ORACLE 11G RAC 问题汇总########################################################
oracle grid infrastrucure GI
提供了自动重启功能和asm功能
1、下载linux.x64_11gR2_grid.zip
http://www.oracle.com/technetwork/database/enterprise-edition/downloads/112010-linx8664soft-100572.html
2、上传到目标机上的独立目录
mkdir /mnt
chmod -R 777 oracle
cd /mnt;ls
cksum linux.x64_11gR2_grid.zip
unzip linux.x64_11gR2_grid.zip
3、在虚拟机里添加两块20g硬盘做asm,centos5.8,2G内存
4、环境配置
vi /etc/security/limits.conf
#for oracle
oracle soft nproc 2048
oracle hard nproc 16384
oracle soft nofile 2048
oracle hard nofile 65536
oracle soft stack 10240
oracle hard stack 32768
#for grid
grid soft nproc 2048
grid hard nproc 16384
grid soft nofile 2048
grid hard nofile 65536
grid soft stack 10240
grid hard stack 32768
vi /etc/security/limits.d/90-nproc.conf
* soft nproc 1024
将上面改为
* - nproc 16384
vi /etc/hosts
#public ip
192.168.30.31 rac01
192.168.30.41 rac02
#private ip
10.10.10.31 rac01prv
10.10.10.32 rac02prv
#vip
192.168.30.32 rac01vip
192.168.30.42 rac01vip
#scanip
192.168.30.50 scanip
vi /etc/sysctl.conf
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
下面两个参数默认是有的，在cs6.8desktop中，所以不需要设置了
kernel.shmall = 2097152
kernel.shmmax = 4294967295
sysctl -p
创建用户和组
oracle
grid
oracle restart + oracle asm
(grid软件的两个功能1 ASM 2 restart）
grid软件就是相当于一个单机版的群集，当你的系统重启的时候数据库会自动跟着启动。这个软件在oracle里面必须要装。
首先做2个用户：oracle和grid
建立5个组：
1数据库软件组dba（管理数据库）和oper
2grid软件组asmadmin（管理ASM），asmdba（可以访问ASM）和asmoper
在oracle组中的用户所拥有的权限各不相同，并且非常严格。
用户及组处理
groupadd -g 501 oinstall
groupadd -g 502 dba
groupadd -g 503 oper
groupadd -g 504 asmadmin
groupadd -g 505 asmdba
groupadd -g 506 asmoper
useradd -g oinstall -G asmadmin,asmdba,asmoper,oper,dba grid
useradd -g oinstall -G dba,asmdba,oper oracle
或
usermod -g oinstall -G dba,asmdba,oper oracle
passwd oracle
passwd grid
id oracle
id grid
创建安装目录
mkdir -p /u01/app/oracle/product/11.2.0/dbhome_1
chown -R oracle:oinstall /u01
mkdir -p /u01/app/oracle/product/11.2.0/grid
chown -R grid:oinstall /u01/app/oracle/product/11.2.0/grid
chmod -R 775 /u01
下面的这种处理方法
mkdir -p /oracle/app/grid
mkdir -p /oracle/app/11.2.0/grid
chown -R grid:oinstall /oracle
mkdir -p /oracle/app/oraInventory
chown -R grid:oinstall /oracle/app/oraInventory
mkdir -p /oracle/app/oracle
mkdir -p /oracle/app/oracle/product/11.2.0/db_1
chown -R oracle:oinstall /oracle/app/oracle/
chmod -R 775 /oracle/
目录处理
这里需说明一点，很多时候，Grid Infrastructure安装中出现问题，都和 grid与oracle的目录以及用户组混乱造成。
为了避免混乱，对grid用户的ORACLE_BASE目录，设置为/u02/app/grid
mkdir -p /u02/app/oracle/product/11.2.0/grid
chown -R grid:oinstall /u02
chmod -R 775 /u02
mkdir -p /u01/app/oracle/product/11.2.0/dbhome_1
chown -R oracle:oinstall /u01
chmod -R 775 /u01
设置用户环境变量
oracle 和grid 两个用户都要配置。
vi /home/oracle/.bash_profile
加入：
export ORACLE_BASE=/u01/app/oracle
export ORACLE_SID=orcl (数据库实例的名字)
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1 (oracle软件的安装目录)
export PATH=$PATH:$ORACLE_HOME/bin
vi /home/grid/.bash_profile
加入：
export ORACLE_BASE=/u01/app/oracle
export ORACLE_SID=+ASM (asm实例名)
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/grid (grid安装目录)
export PATH=$PATH:$ORACLE_HOME/bin
5、安装asm
下载包并安装：本来有三个包，在cs6下官网只有两个包，另外一个包叫kmod-oracleasm
yum install kmod-oracleasm
rpm -ivh oracleasm*
===================================== rpm -ivh http://mirror.symnds.com/distributions/CentOS-vault/6.7/os/x86_64/Packages/compat-libcap1-1.10-1.x86_64.rpm yum install kmod-oracleasm -y rpm -ivh oracleasm* fdisk /dev/sdb n,p,1,w /etc/init.d/oracleasm configure grid,asmadmin,y,y /etc/init.d/oracleasm createdisk crs /dev/sdb1 /etc/init.d/oracleasm createdisk data1 /dev/sdc1 /etc/init.d/oracleasm createdisk data2 /dev/sdd1 /etc/init.d/oracleasm createdisk fra /dev/sde1 oracleasm listdisks mkfs.ext4 /dev/sdd1 ===================================
分区之前添加的两块盘，然后初始化
/etc/init.d/oracleasm configure
grid
oinstall
y
y
oracle user login
unzip p13390677_112040_Linux-x86-64_1of7.zip
unzip p13390677_112040_Linux-x86-64_2of7.zip
cd database/
./runInstaller
su -
2g到16g内存的话，swap空间等于内存大小，16G以上，等于内存大小
grep SwapTotal /proc/meminfo
dd if=/dev/zero of=swapfile bs=1M count=1024
mkswap swapfile
swapon swapfile
free
vi /etc/fstab
/root/swapfile swap swap defaults 0 0
yum install gcc
yum install libaio-devel
yum install elfutils-libelf-devel
yum install gcc-c++
yum install compat-libstdc++-33
rpm -qa|grep ksh
rpm -ivh http://mirror.symnds.com/distributions/CentOS-vault/5.8/os/x86_64/CentOS/pdksh-5.2.14-37.el5.x86_64.rpm
rpm -ivh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
yum install rlwrap
rpm -ivh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm yum install rlwrap 有时会报下面这个错， [Errno -1] repomd.xml does not match metalink for epel Trying other mirror. 这时有两个解决办法， 一是直接下载下面的文件，但有可能在linux下用wget与curl报404错误，这时，在图形界面下的浏览器直接可以下载，然后传到服务器上，再rpm -ivh 就可以了 http://mirrors.sohu.com/fedora-epel/6Server/x86_64/rlwrap-0.42-1.el6.x86_64.rpm 二是修改epel.repo文件，将mirrors行注释，启用baseurl行，也能下 2016年10月29日 星期六 18时45分45秒 才解决，花了大约一个小时解决这个小问题，唉，太坑了
grid need to install follow packages,and oracle might not need to install it.
rpm -ivh http://mirror.symnds.com/distributions/CentOS-vault/6.7/os/x86_64/Packages/compat-libcap1-1.10-1.x86_64.rpm
yum install kmod-oracleasm
rpm -ivh oracleasm*
mount /dev/sr0 /media
rpm -ivh /media/Packages/compat-libcap1-1.10-1.x86_64.rpm
rpm -ivh http://mirror.symnds.com/distributions/CentOS-vault/5.8/os/x86_64/CentOS/pdksh-5.2.14-37.el5.x86_64.rpm
yum install gcc
yum install libaio-devel
yum install elfutils-libelf-devel
yum install gcc-c++
yum install compat-libstdc++-33
或者下面一条命令
yum install gcc libaio-devel elfutils-libelf-devel gcc-c++ compat-libstdc++-33
编辑文件 vi /etc/pam.d/login，添加如下行(11gR1有要求，R2没有要求)：
session required pam_limits.so
编辑 vi /etc/profile 文件，添加如下行(11gR1有要求，R2没有要求)：
if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi
该配置在用户oracle登录时会立即生效，如果当前 oracle 用户已经登录可退出后重新登录使之生效。/etc/init.d/oracleasm configure
grid
asmadmin
y
y
只在一个节点上创建就可以了，另一个扫描一下就可以了
/etc/init.d/oracleasm createdisk rac /dev/mapper/mpathbp1
/etc/init.d/oracleasm deletedisk rac
oracleasm scandisks
oracleasm listdisks
6、用grid用户安装
用grid登录
./runInstaller
出现界面选第二个，单机
可能不出现asm磁盘，首先前面的环境设置先做好，其次可以更改路径/dev/oracleasm/disks.
high 有三份相同的数据
normal 有两份相同的数据
external 没有冗余
硬盘30G,swap10G,不然这里检测条件时会出问题。
安装依赖包，下面这个包一定要安装，如果不提前装好，会出现下面的问题
[root@oracle Packages]# rpm -ivh compat-libcap1-1.10-1.x86_64.rpm
Preparing... ########################################### [100%]
1:compat-libcap1 ########################################### [100%]
[root@oracle Packages]# pwd
/media/cdrom/Packages
过程为，之前没有装包---》报错后根据报错安装这个包---》再次运行脚本会报下面的错---》根据提示还是报错（加-deconfig -force参数）---》所以最好提前装好
[root@rac1 11.2.0]# /g01/oraInventory/orainstRoot.sh
Changing permissions of /g01/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /g01/oraInventory to oinstall.
The execution of the script is complete.
[root@rac1 11.2.0]# /g01/app/11.2.0/grid/root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /g01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2013-10-10 03:41:35: Parsing the host name
2013-10-10 03:41:35: Checking for super user privileges
2013-10-10 03:41:35: User has super user privileges
Using configuration parameter file: /g01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
/g01/app/11.2.0/grid/bin/clscfg.bin: error while loading shared libraries: libcap.so.1: cannot open shared object file: No such file or directory
Failed to create keys in the OLR, rc = 127, 32512
OLR configuration failed第二次执行root.sh时，出现下面的报错。
/u01/app/11.2.0/grid/crs/install/roothas.pl -deconfig -force -verbose
[root@oracle Packages]# /u02/app/grid/product/11.2.0/grid/root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u02/app/grid/product/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2016-09-02 11:39:34: Checking for super user privileges
2016-09-02 11:39:34: User has super user privileges
2016-09-02 11:39:34: Parsing the host name
Using configuration parameter file: /u02/app/grid/product/11.2.0/grid/crs/install/crsconfig_params
Improper Oracle Clusterware configuration found on this host
Deconfigure the existing cluster configuration before starting
to configure a new Clusterware
run '/u02/app/grid/product/11.2.0/grid/crs/install/rootcrs.pl -deconfig'
to configure existing failed configuration and then rerun root.sh
[root@oracle Packages]#
可以忽略i386的包
root用户下
1、以文件系统增加(以下是以增加512M的文件系统为例)
dd if=/dev/zero of=/home/oracle/myswaps/swapfile1 bs=1M count=512
注：of后面的路径一定不能是/dev下，否则在激活swap文件时报参数无效
2、创建swap文件
mkswap /home/oracle/myswaps/swapfile1
3、激活swap文件
swapon /home/oracle/myswaps/swapfile1
4、此时用free -m就可以看到swap空间已经增加了，但是重启后就不会存在了
5、添加重启后还会生效，vi /etc/fstab添加如下内容：
/home/oracle/myswaps/swapfile1 swap swap defaults 0 0
执行两个脚本，并安装完成后
注意：如果之前安装过gi，在执行root.sh的时候需要删除之前的配置信息：命令如下：
/u01/app/11.2.0/grid/crs/install/roothas.pl -deconfig -force -verbose
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
ohasd failed to start: Inappropriate ioctl for device
ohasd failed to start at/u01/app/11.2.0/grid/crs/install/rootcrs.pl line 443.
第一次安装11gR2 RAC的时候就遇到了这个11.0.2.1的经典问题，上网一查才知道这是个oracle bug，
具体可以参考：https://forums.oracle.com/thread/2352285
解决办法也很简单，
就是在执行root.sh之前执行以下命令
/bin/dd if=/var/tmp/.oracle/npohasd of=/dev/null bs=1024 count=1
如果出现
/bin/dd: opening`/var/tmp/.oracle/npohasd': No such file or directory
的时候文件还没生成就继续执行，直到能执行为止，一般出现Adding daemon to inittab这条信息的时候执行dd命令。
另外还有一种解决方法就是更改文件权限
chown root:oinstall /var/tmp/.oracle/npohasd
重新执行root.sh之前别忘了删除配置：/u01/app/11.2.0/grid/crs/install/roothas.pl -deconfig -force -verbose
接下来验证一下
ps -ef|grep asm
su - grid
sqlplus / as sysasm
select status from v$instance
由于在安装GI的时候只能创建一个磁盘组，如果需要增加磁盘组的话，在GI安装完成后，使用ASMCA命令增加磁盘组。
grid安装完成后，重启操作系统，启动ASM实例报错如下：
[grid@oragrid ~]$ sqlplus / as sysasm SQL*Plus: Release 11.2.0.4.0 Production on Sun Jun 7 16:21:31 2015 Copyright (c) 1982, 2013, Oracle. All rights reserved. Connected to an idle instance. SQL> startup ORA-01078: failure in processing system parameters ORA-29701: unable to connect to Cluster Synchronization Service [grid@oragrid ~]$ crsctl check css CRS-4639: Could not contact Oracle High Availability Services CRS-4000: Command Check failed, or completed with errors. [grid@oragrid ~]$ crsctl check has CRS-4639: Could not contact Oracle High Availability Services 第一种方法：
切换到root用户执行启动init.ohasd [root@oragrid ~]# /etc/init.d/init.ohasd run & 再切换到grid用户check，可能有点慢，等一会就会online [grid@oragrid ~]$ crsctl check has CRS-4638: Oracle High Availability Services is online [grid@oragrid ~]$ crsctl check css CRS-4529: Cluster Synchronization Services is online [grid@oragrid ~]$ crsctl stat resource -t -init -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.DATA.dg ONLINE OFFLINE oragrid ora.LISTENER.lsnr ONLINE OFFLINE oragrid STARTING ora.asm ONLINE OFFLINE oragrid Instance Shutdown ora.ons OFFLINE OFFLINE oragrid -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.cssd 1 ONLINE OFFLINE ora.diskmon 1 OFFLINE OFFLINE ora.evmd 1 ONLINE OFFLINE STARTING 原来是开机启动的init.ohasd没有起作用，重新启动一次就好了。 直接杀掉该进程，会自动重新生成一个。 [root@oragrid ~]# ps -ef | grep init.ohasd root 972 1 0 22:01 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run root 2474 2450 0 22:04 pts/0 00:00:00 grep init.ohasd [root@oragrid ~]# kill -9 972 [root@oragrid ~]# ps -ef | grep init.ohasd root 2538 1 0 22:05 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run root 2638 2450 0 22:05 pts/0 00:00:00 grep init.ohasd [root@oragrid ~]# su - grid [grid@oragrid ~]$ crsctl check has CRS-4638: Oracle High Availability Services is online [grid@oragrid ~]$ crsctl check css CRS-4529: Cluster Synchronization Services is online [grid@oragrid ~]$ crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.DATA.dg ora....up.type ONLINE ONLINE oragrid ora....ER.lsnr ora....er.type ONLINE ONLINE oragrid ora.asm ora.asm.type ONLINE ONLINE oragrid ora.cssd ora.cssd.type ONLINE ONLINE oragrid ora.diskmon ora....on.type OFFLINE OFFLINE ora.evmd ora.evm.type ONLINE ONLINE oragrid ora.ons ora.ons.type OFFLINE OFFLINE 再进入sqlplus就可以看到ASM实例已经启动了。但是每次操作系统重启都会出现这个问题，上述方法治标不治本，且没有找到最终的原因，请教高手解决。 第二种方法： 在网上找到这篇文章，http://blog.csdn.net/tianlesoftware/article/details/8207629，说这个问题是11.2.0.1的bug，但是我现在安装的版本是11.2.0.4，使用这篇文章的解决办法，在root用户下执行下面的命令： [root@oragrid ~]# /bin/dd if=/var/tmp/.oracle/npohasd of=/dev/null bs=1024 count=1 ^C0+0 records in 0+0 records out 0 bytes (0 B) copied, 13.6369 s, 0.0 kB/s 然后在grid用户下查看has和css状态，可以看到已经启动了 [grid@oragrid ~]$ crsctl check has CRS-4638: Oracle High Availability Services is online [grid@oragrid ~]$ crsctl check css CRS-4529: Cluster Synchronization Services is online [grid@oragrid ~]$ crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.DATA.dg ora....up.type ONLINE ONLINE oragrid ora.FRA.dg ora....up.type ONLINE ONLINE oragrid ora....ER.lsnr ora....er.type ONLINE ONLINE oragrid ora.asm ora.asm.type ONLINE ONLINE oragrid ora.cssd ora.cssd.type ONLINE ONLINE oragrid ora.diskmon ora....on.type OFFLINE OFFLINE ora.evmd ora.evm.type ONLINE ONLINE oragrid ora.ons ora.ons.type OFFLINE OFFLINE ora.stone.db ora....se.type OFFLINE OFFLINE 居然这个方法管用，但是系统重启后问题依旧，难道这个bug在11.2.0.4还存在吗？
以grid用户安装grid完成后。
接下来，以oracle用户的身份，安装oracle数据库。