Uninstalling OEM agent via CLI Mode

Below Steps to needs to done on OMS Server::

[oracle@sivaOMS001 bin]$ ./emcli sync
Synchronized successfully
[oracle@sivaOMS001 bin]$ ./emcl delete_target -name="nj09mhf6213:3872" -type="oracle_emd" -delete_monitored_targets
-bash: ./emcl: No such file or directory
[oracle@sivaOMS001 bin]$ ./emcli delete_target -name="target001 :3872" -type="oracle_emd" -delete_monitored_targets
Target "target001 :3872:oracle_emd" deleted successfully
[oracle@sivaOMS001 bin]$

Below Steps to needs to done on Target Server where agent needs to install::

[oracle@target001  12.1.0.3.0]$ echo $ORACLE_HOME
/u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0
[oracle@target001  12.1.0.3.0]$
[oracle@target001  12.1.0.3.0]$
[oracle@target001  12.1.0.3.0]$ export AGENT_HOME=/u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0
[oracle@target001  12.1.0.3.0]$ perl $AGENT_HOME/sysman/install/AgentDeinstall.pl -agentHome $AGENT_HOME


 Agent Oracle Home: /u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0


NOTE: The agent base directory: /u01/app/oracle/product/OEM_HOME will be removed after successful deinstallation of agent home.


 DetachHome Command executed:/u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/oui/bin/runInstaller -detachHome -force -depHomesOnly -silent ORACLE_HOME=/u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0 -waitForCompletion -invPtrLoc /u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/oraInst.loc
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 103422 MB    Passed
The inventory pointer is located at /u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/oraInst.loc
'DetachHome' was successful.
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 103422 MB    Passed
The inventory pointer is located at /u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/oraInst.loc
The Oracle home '/u01/app/oracle/product/OEM_HOME/sbin' could not be updated as it does not exist.


Deinstall Command executed:/u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/oui/bin/runInstaller -deinstall -silent "REMOVE_HOMES={/u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0}" -waitForCompletion -removeAllFiles -invPtrLoc /u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/oraInst.loc
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 103422 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2017-03-03_04-55-28AM. Please wait ...Oracle Universal Installer, Version 11.1.0.11.0 Production
Copyright (C) 1999, 2013, Oracle. All rights reserved.

Starting deinstall


Deinstall in progress (Friday, March 3, 2017 4:55:32 AM EST)
Configuration assistant "Agent Deinstall Assistant" succeeded
............................................................... 100% Done.

Deinstall successful

End of install phases.(Friday, March 3, 2017 4:55:58 AM EST)
End of deinstallations
Please check '/u01/app/oraInventory/logs/silentInstall2017-03-03_04-55-28AM.log' for more details.

Do you want us to delete the old oracle home [yes/no] :yes
cannot fetch initial working directory: No such file or directory at /u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/sysman/install/AgentDeinstall.pl line 136.

NOTE: The targets monitored by this Management Agent will not be deleted in the Enterprise Manager Repository by this deinstall script. Make sure to delete the targets manually from the Cloud Control Console for a successful deinstallation.
[oracle@target001  12.1.0.3.0]$

Agent installation Command Line Interface (CLI) Method

Below Steps to needs to done on OMS Server::

[oracle@sivaOMS001 bin]$ ./emcli login -username=SYSMAN
Enter password :

Login successful
[oracle@sivaOMS001 bin]$  ./emcli get_supported_platforms
-----------------------------------------------
Version = 12.1.0.3.0
 Platform = Linux x86-64
-----------------------------------------------
Platforms list displayed successfully.
[oracle@sivaOMS001 bin]$ df -h /tmp/
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-LogVol00
                       99G  8.0G   86G   9% /
[oracle@sivaOMS001 bin]$ ./emcli get_agentimage -destination=/tmp -platform="Linux x86-64" -version=12.1.0.3.0
Destination directory /tmp is not empty. Make sure you pass an empty destination directory to download the agent image.
[oracle@sivaOMS001 bin]$ pwd
/u01/app/oracle/product/OEM_HOME/MIDDLEWARE/oms/bin
[oracle@sivaOMS001 bin]$ cd /tmp
[oracle@sivaOMS001 tmp]$ mkdir agent_soft
[oracle@sivaOMS001 tmp]$ cd -
/u01/app/oracle/product/OEM_HOME/MIDDLEWARE/oms/bin
[oracle@sivaOMS001 bin]$ ./emcli get_agentimage -destination=/tmp/agent_soft/ -platform="Linux x86-64" -version=12.1.0.3.0
 === Partition Detail ===
Space free : 90 GB
Space required : 1 GB
Check the logs at /u01/app/oracle/product/OEM_HOME/gc_inst/em/EMGC_OMS1/sysman/emcli/setup/.emcli/get_agentimage_2017-03-03_10-07-56-AM.log
Downloading /tmp/agent_soft//12.1.0.3.0_AgentCore_226.zip
File saved as /tmp/agent_soft//12.1.0.3.0_AgentCore_226.zip
Downloading /tmp/agent_soft//12.1.0.3.0_PluginsOneoffs_226.zip
File saved as /tmp/agent_soft//12.1.0.3.0_PluginsOneoffs_226.zip
Downloading /tmp/agent_soft//unzip
File saved as /tmp/agent_soft//unzip
Agent Image Download completed successfully.
[oracle@sivaOMS001 bin]$


[oracle@sivaOMS001 agent_soft]$ ls -ltrh
total 235M
-rw-r--r-- 1 oracle oinstall 235M Mar  3 10:15 12.1.0.3.0_AgentCore_226.zip
[oracle@sivaOMS001 agent_soft]$ scp 12.1.0.3.0_AgentCore_226.zip oracle@target001:/tmp/agent_software/
oracle@target001's password:
12.1.0.3.0_AgentCore_226.zip                                                                                                                                                                               100%  235MB  15.7MB/s   00:15
[oracle@sivaOMS001 agent_soft]$

Below Steps to needs to done on Target Server where agent needs to install::

Step 1:: verify the pre-check logs

./agentDeploy.sh AGENT_BASE_DIR=/u01/app/oracle/product/OEM_HOME OMS_HOST=sivaOMS001 EM_UPLOAD_PORT=4903 AGENT_REGISTRATION_PASSWORD=siva_1

Step 2:: use ignore precheck if it needs to ignore any of the pre-request from above

./agentDeploy.sh -ignorePrereqs AGENT_BASE_DIR=/u01/app/oracle/product/OEM_HOME OMS_HOST=sivaOMS001 EM_UPLOAD_PORT=4903 AGENT_REGISTRATION_PASSWORD=siva_1
-ignorePrereqs 


[oracle@target001 agent_software]$ ./agentDeploy.sh -ignorePrereqs AGENT_BASE_DIR=/u01/app/oracle/product/OEM_HOME OMS_HOST=sivaOMS001 EM_UPLOAD_PORT=4903 AGENT_REGISTRATION_PASSWORD=define_1

Validating the OMS_HOST & EM_UPLOAD_PORT
Executing command : /u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/jdk/bin/java -classpath /u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/jlib/agentInstaller.jar:/u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/oui/jlib/OraInstaller.jar oracle.sysman.agent.installer.AgentInstaller /u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0 /tmp/agent_software /u01/app/oracle/product/OEM_HOME -prereq

Validating oms host & port with url: http://sivaOMS001:4903/empbs/genwallet
Validating oms host & port with url: https://sivaOMS001:4903/empbs/genwallet
Return status:3-oms https port is passed
Unzipping the agentcoreimage.zip to /u01/app/oracle/product/OEM_HOME ....
12.1.0.3.0_PluginsOneoffs_226.zip
Executing command : /tmp/agent_software/unzip -o /tmp/agent_software/12.1.0.3.0_PluginsOneoffs_226.zip -d /u01/app/oracle/product/OEM_HOME
Executing command : /u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/jdk/bin/java -classpath /u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/oui/jlib/OraInstaller.jar:/u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/oui/jlib/xmlparserv2.jar:/u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/oui/jlib/srvm.jar:/u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/oui/jlib/emCfg.jar:/u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/jlib/agentInstaller.jar:/u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/oui/jlib/share.jar oracle.sysman.agent.installer.AgentInstaller /u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0 /tmp/agent_software /u01/app/oracle/product/OEM_HOME /u01/app/oracle/product/OEM_HOME/agent_inst AGENT_BASE_DIR=/u01/app/oracle/product/OEM_HOME


Cloning the agent home...
Executing command: /u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/oui/bin/runInstaller -debug   -ignoreSysPrereqs -clone -forceClone -silent -waitForCompletion -nowait ORACLE_HOME=/u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0   AGENT_BASE_DIR=/u01/app/oracle/product/OEM_HOME AGENT_BASE_DIR=/u01/app/oracle/product/OEM_HOME OMS_HOST=sivaOMS001 EM_UPLOAD_PORT=4903 AGENT_REGISTRATION_PASSWORD=define_1 -noconfig  ORACLE_HOME_NAME=agent12c1 -force b_noUpgrade=true AGENT_PORT=-1
Clone Action Logs Location:/u01/app/oraInventory/logs/cloneActions<timestamp>.log
Cloning of agent home completed successfully

Attaching sbin home...
Executing command: /u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/oui/bin/runInstaller -debug   -ignoreSysPrereqs -attachHome -waitForCompletion -nowait ORACLE_HOME=/u01/app/oracle/product/OEM_HOME/sbin ORACLE_HOME_NAME=sbin12c1 -force
Attach Home Logs Location:/u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/cfgtoollogs/agentDeploy/AttachHome<timestamp>.log
Attach home for sbin home completed successfully.

Updating home dependencies...
Executing command: /u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/oui/bin/runInstaller -debug   -ignoreSysPrereqs -updateHomeDeps -waitForCompletion HOME_DEPENDENCY_LIST={/u01/app/oracle/product/OEM_HOME/sbin:/u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0} -invPtrLoc /u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/oraInst.loc -force
Update Home Dependencies Location:/u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/cfgtoollogs/agentDeploy/UpdateHomeDeps<timestamp>.log
Update home dependency completed successfully.

Performing the agent configuration...
Executing command: /u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/oui/bin/runConfig.sh ORACLE_HOME=/u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0 RESPONSE_FILE=/u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/agent.rsp ACTION=configure MODE=perform COMPONENT_XML={oracle.sysman.top.agent.11_1_0_1_0.xml} RERUN=true
Configuration Log Location:/u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/cfgtoollogs/cfgfw/CfmLogger<timestamp>.log
Agent Configuration completed successfully

The following configuration scripts need to be executed as the "root" user.
#!/bin/sh
#Root script to run
 /u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/root.sh
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts
Agent Deployment Successful.
Agent deployment log location:
/u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/cfgtoollogs/agentDeploy/agentDeploy_2017-03-03_10-36-05-AM.log
Agent deployment completed successfully.
[oracle@target001 agent_software]$ sudo /u01/app/oracle/product/OEM_HOME/core/12.1.0.3.0/root.sh
Finished product-specific root actions.
/etc exist
Finished product-specific root actions.
[oracle@target001 agent_software]$ sudo id
uid=0(root) gid=0(root) groups=0(root)
[oracle@target001 agent_software]$

Agent Unreachable due to Agent was blocked by OMS

Resolving steps for "Agent Unreachable due to Agent was blocked by OMS"

1. Once you have seen or Received alert for Agent Unreachable, Check the agent status in EM , if it shows that agent is blocked , then follow the below steps to unblock the agent from OMS.  

2. Navigate to Agent Home page and click Agent drop down list and click Resynchronization. 

3. Click Continue on the Next Page.

4. Click Ok once you confirmed about the Agent Name and click Ok to Proceed.

 5. Once the Job created, click the Job Name and Monitor the Job of Agent Resync.

6. After Few Mins the Agent Resync Job will complete Successfully and you also can see the same.

7. Now Come to the agent Home page and click the glass button near to status of Agent which is below, which will run Diagnostic Test

 8. Once Diagnostic Test completed , the agent will back to normal status like below.

Schema Comparison for 2 different database via OEM

Schema Comparison for 2 different database via OEM

Below steps for comparing multiple Schemas using OEM:

1.  In Cloud Control, Search the Database target and go to DB Home page, from the Schema menu click the Change Management and click the Schema Comparison as highlighted below

2.       On Next Window you can see the Schema Comparison Window, from here you can click the create button as highlighted below.

3.       On Next Window we need to give the comparison name, database name for left Source database as below window and click Next.

(Note: Have Schema password or sys password for the database, while adding database to job, it will ask password for authentication)

4.       On Next Window we need to give the database name for the Right Source and click next.

5.       On Next Window select the object type which we need to compare and select non-objects like grants and role if we need to compare, and select the Multiple Schemas that we need to compare, and Click Next to proceed.

6.       On Next Window you can check or uncheck the option which you need to add for this comparison, this window is optional and we can leave as it is default and we can proceed further by clicking Next.

7.       On Next Window we need to provide the credentials for the both database to start the comparison job.

8.       And also you can schedule this job at any time or immediately and we can enable the email notification with job results once it is completed successfully and click Next to proceed further.

9.       On Next Window you can review all the details and submit the job to start the comparison.

10.   Once you submit the job, you will get the confirmation like below.

11.   And also you can monitor the job by clicking the View job details.

11a. Once job completed successfully you can get the mail like below

12.   You can view the job details in the same schema Comparison page at any time.

13.   Below are the Schema Comparison result windows from OEM

Exclude specifi ORA error from Database target Metrics

Exclude ORA Error from Database target Metrics

To exclude an ORA Error from OEM, follow these steps:

1.  In Cloud Control, go to the target database and navigate to Oracle Database à Monitoring à Metrics and collection Settings.

2.       On Next Window click Edit icon for the “Generic Alert log Error”

3.       On Next Window we can see the Alert log filter expression with default values, there we need to add our errors which we need to ignore

4.       Below are the errors which we need to exclude as per dream environment , so I am creating the exception expression as “New one”

5.       For Creating the expression we can use the below site to check the expression values is correct or not.

http://www.regular-expressions.info/javascriptexample.html

6.       Create the expression and give the error message in the text box like above and below image and you can see the show match, it need to match, if it is matches the ORA error will exclude in OEM Metrics.

7.       On Next Window paste the tested expression in the alert log filtering box and click continue.

8.       On Next Window click ok, unless you click ok here, the metrics will save the new changes.

9.On Next Window You can see the saved metric confirmation result.

OEM Agent issues and solution

Issue 1::
---------

Error : EMD upload error: uploadXMLFiles skipped :: OMS version not checked yet...*

a) Please use the below steps

$ORACLE_HOME/bin/emctl stop agent
rm -r $ORACLE_HOME/sysman/emd/state/*
rm -r $ORACLE_HOME/sysman/emd/collection/*
rm -r $ORACLE_HOME/sysman/emd/upload/*
*rm $ORACLE_HOME/sysman/emd/lastupld.xml*
*rm $ORACLE_HOME/sysman/emd/agntstmp.txt*
$ORACLE_HOME/bin/emctl clearstate agent
$ORACLE_HOME/bin/emctl start agent
$ORACLE_HOME/bin/emctl upload

If option a doesnt work,

b) Please try below
    emctl stop agent
cd $AGENT_HOME
rm sysman/emd/agntstmp.txt
rm sysman/emd/blackouts.xml
rm -r sysman/emd/state/*
rm -r sysman/emd/collection/*
rm -r sysman/emd/upload/*
rm sysman/emd/lastupld.xml
rm sysman/emd/protocol.ini
$ORACLE_HOME/bin/emctl clearstate agent
$ORACLE_HOME/bin/emctl start agent
$ORACLE_HOME/bin/emctl upload


Issue 2::

Move targets to the new OMS::
-----------------------------

To be done on every hosts that should be connected to the additional OMS.

   - Login as oracle account
      - Setup Agent environment
         - *# set the oms/agent environment*
      - Stop the Management Agent.
         - *# emctl stop agent*
      - Adapt the $oh/sysman/config/emd.properties to point to the new OMS
      host REPOSITORY_URL==http://*<new fq oms name>*:4889/em/upload
      emdWalletSrcUrl=http://*<new fq oms name>*:4889/em/wallets/emd
      emdWalletDest=*<OMS HOME>*/sysman/config/server
      - Delete all the files in the following directories:
         - *rm $oh/sysman/emd/upload/*{}*
         - *rm $oh/sysman/emd/state/*
      - Restart the Management Agent
         - *# emctl start agent*
      - Force an upload
         - *# emctl upload agent*
      - Check the status

         - *# emctl status agent*

Issue 3::
---------

When you find "Metric collection error", or "Target is broken state. Reason
- Computer dynamic property takes too long", usually you can find them in
unkown targets, you can use the following method to fix:

Find the metric error target name and type in
$AGENT_HOME/sysman/emd/targets.xml, then

emctl reload agent dynamicproperties <Target name>:<Target type>

Example:

emctl reload agent dynamicproperties sivadb.localhost.com:oracle_database

Adding/Removing disk to ASM Diskgroup via OEM

Adding/Removing disk to ASM Diskgroup via OEM

To add a disk to an ASM disk group, follow these steps:

1.       In Cloud Control, Search the ASM target and go to ASM Home page, from the ASM menu click the Disk Groups as highlighted below.

2.  On Next Window you can see all the disk groups, then you can Click the diskgroup where you need the space. (ex : click P---N_DBF01)

3.       On Next Window we can see the all disk which is belongs to the Diskgroup, to add the disk, click Add button in the righ-side corner.

4.       Once we click the Add on Next Window we can see the all Candidate disk which is available to add, in that select the disk which we need to add, if needed we can select the multiple disk also and select rebalance value depend on the environment.

5.       On Same Window we can see the show SQL button to see the SQL command which is going to execute if we want to check and then come back and Click Ok to add disk.

6.       On Next Window we will get the update message that the disk added successfully.

7.       Then we can check the Disk group Members used % status, if all the disk are in the same size then the Rebalance operation is completed successfully. On upcoming screen shots we can see the rebalance operation completion status.

8.       Below is the latest disk group usage after adding the disk to the diskgroup.

Removing disk from ASM Diskgroup via OEM

To remove a disk from an ASM disk group, follow these steps:

1.       In Cloud Control, Search the ASM target and go to ASM Home page, from the ASM menu click the Disk Groups, from that page click the disk group which u need to alter, the below page will open , there you can find the list of disk added to Disk group.

2.       Now you can select the disk which you need to remove and click the Remove button

3.       On Next Window you can give the rebalance power limit and click Yes to Remove the Disk from Disk Group

4.       Crosscheck the Query with show SQL Option.

5.  Now the Disk is successfully removed from the Disk Group.

6.       Now the Disk available in the disk group after dropping the disk

7.       The Removed Disk went to Former disk state as highlighted below.

8.       And Crosscheck the Disk group size in sqlplus level also.