Case of the Unexplained Support Issue – Troubleshooting steps for diagnosing HFM, Workspace and FDM

Case of the Unexplained Support Issue – Troubleshooting steps for diagnosing HFM, Workspace and FDM

• 1.
  #Kscope Case of theUnexplained Support Issue Charles Beyer Troubleshooting steps for diagnosing HFM, Workspace, and FDM
• 2.
  #Kscope About the Speaker 15+ years professional development experience in various languages/technologies (6 years w/ Hyperion products)  Significant experience reverse engineering software  Loves to help others solve problems  Posts under beyerch2 on OTN Discussion Boards
• 3.
  #Kscope Before we Begin….. To expedite session, we will use both screen caps and live demos  ‘All-in-one’ Virtual Machine created for session, available upon request*  Feel free to ask questions
• 4.
  #Kscope Session Goals  Discoversome new tools to be used for troubleshooting (Oracle / Third Party)  Acquire some new troubleshooting ideas
• 5.
  #Kscope Challenges  Responsible fora large amount of products! ● Foundation Services (OHS, Weblogic, Shared Services, Workspace, EPMA, Calc Manager, LCM) ● HFM (HFM Web, Client, Smartview), FDM ● Reporting and Analysis (FR, IR, SQLPR, WA) ● Planning, Performance Scorecard, Strategic Finance, Probability Cost, Disclosure Management, Financial Close Management, Data Relationship Management, Essbase, Crystal Ball, …….. Hyperion Kitchen Sink.
• 6.
  #Kscope Challenges (continued)  Needto understand underlying technologies! ● OS Servers (Windows/Unix) ● Application Servers (Weblogic, Tomcat), Web Servers (IIS, Apache, OHS) ● Authentication Options (AD, Native, Custom) ● Languages (Java, .NET, ASP Classic, JSP, VBScript) ● Communication Frameworks (DCOM, Web Services) ● Database (SQL Server, Oracle, Berkeley DB) ● Network Principles (Routing, Firewall, VPN)
• 7.
  #Kscope Challenges (continued)  EndUsers! ● Quantity – Intermittent issues due to number of users stressing system ● Internal vs. External ● Less control over environment ● More points of failure ● Remote troubleshooting more difficult ● User Experience ● New Users - More hands on, but easier problems. ● Power Users – Likes to experiment
• 8.
  #Kscope Challenges (continued)  InternalIssues a.k.a. Politics! ● Separation of Duties ● Complicates troubleshooting efforts due to lack of access ● Delays resolution due to co-ordination efforts ● Restrictive Policies ● Could prevent/limit certain types of troubleshooting ● Network/HTTP sniffing ● No monitor programs on production
• 9.
  #Kscope Challenges (continued)  CrypticError Messages! But I didn’t click on a folder!
• 10.
  #Kscope Challenges (continued)  CrypticError Messages!
• 11.
  #Kscope Challenges (continued)  CrypticError Messages!
• 12.
  #Kscope Challenges (continued)  CrypticError Messages!
• 13.
  #Kscope Challenges (continued)  CrypticError Messages!
• 14.
  #Kscope Challenges (continued)  CrypticError Messages!
• 15.
  #Kscope Challenges (continued)  LimitedOOTB (Out Of The Box) Tools! ● Mostly consists of Log Files ● System 11 has expanded diagnostic capability ● Support.Oracle.Com
• 16.
  #Kscope Troubleshooting Strategy
• 17.
  #Kscope Troubleshooting Strategy Final Solution Follow Up Evaluate Possible Solutions Clearly Defined Problem
• 18.
  #Kscope Troubleshooting Strategy (continued) Clearly Define the Problem ● Ask the right questions ● Verify Information Where Possible  Research, Prioritize, and Evaluate potential solutions ● Use Initial Information and Resources to propose possible causes/solutions ● Prioritize in order of likelihood ● Evaluate Potential Solutions ● Determine Final Solution or Escalate
• 19.
  #Kscope Troubleshooting Strategy (continued) Close Out The Problem ● Document / Get Sign-Off ● Add Information To Internal Knowledgebase
• 20.
  #Kscope Troubleshooting Strategy (continued) Notes ● Process Is Easier As You Learn More About The System ● Development / Test Environment Makes This Easier ● Don’t Fear Leveraging Other Resources ● Practice, Practice, Practice
• 21.
  #Kscope Troubleshooting Strategy (continued) Clearly Define The Problem – Answer the Who / What / When / Where / How ● Who? (is affected) ● All Users or Select Group? ● If subset, is there commonality? (Geographic, Hardware/Software, Data Provider) ● What ? (is happening / not happening) ● What was the attempted procedure? ● What has changed?
• 22.
  #Kscope Troubleshooting Strategy (continued) Clearly Define The Problem – Answer the Who / What / When / Where / How ● When? (does the issue occur) ● Consistently? Intermittently? Pattern? ● Where ? (are failures occurring) ● Client Machine(s)? ● Database Server(s)? ● Web Server(s)? ● App Server(s)?
• 23.
  #Kscope Troubleshooting Strategy (continued) Clearly Define The Problem – Answer the Who / What / When / Where / How ● How? (should the process work) ● Do you know how it should work? ● Important to understand expected operation to help determine point of failure, probable cause, and solution. ● Research this if you are not sure!
• 24.
  #Kscope Troubleshooting Strategy (continued) Research, Prioritize, and Evaluate potential solutions ● Iterative Process ● Defined Problem + Internal Knowledge + External Knowledge + Leveraged Resources + Tools = Potential Solutions ● Prioritize Potential Solutions ● KISS / Occam’s Razor – Start with most likely / obvious first!
• 25.
  #Kscope Troubleshooting Strategy (continued) Research, Prioritize, and Evaluate potential solutions ● Evaluate Potential Solutions ● In Test/Dev if possible ● Promote to production after verifying your fix doesn’t cause additional problems ● If you exhaust all possible causes, escalate. Don’t spin wheels. ● Do not discount workarounds!
• 26.
  #Kscope Troubleshooting Strategy (continued) Close Out The Problem ● Get Positive Confirmation That Issue Is Resolved ● Add Issue to Internal Knowledgebase For Future Review ● SharePoint ● Track-It ● Shared Outlook Folder ● Share Information When Necessary
• 27.
  #Kscope Internal Tools
• 28.
  #Kscope Troubleshooting Tools -Internal  Knowledge Tools ● http://support.oracle.com – Knowledgebase, Patches & Update, and “Communities”
• 29.
  #Kscope Troubleshooting Tools –Internal (cont)  Knowledge Tools ● OTN Discussion Forum (http://forums.oracle.com)
• 30.
  #Kscope Troubleshooting Tools –Internal (cont)  Knowledge Tools ● Product Documentation ● System 11.1.2.1 - http://docs.oracle.com/cd/E24674_01/index.htm ● System 9.3.x - http://docs.oracle.com/cd/E10530_01/welcome.html  NOTE : These move periodically! Google ‘Hyperion documentation library’ if you cannot find what you are looking for
• 31.
  #Kscope Troubleshooting Tools -Internal
• 32.
  #Kscope Troubleshooting Tools –Internal (cont.)  Knowledge Tools ● Tips and Tricks from EPM System Infrastructure Development ● Issues 1 to 72 - http://www.oracle.com/technetwork/middleware/bi- foundation/epm-tips-issues-1-72-427329.pdf ● Issues 73 and up - http://www.oracle.com/technetwork/middleware/bi- foundation/epm-tips-issues-73-up-399995.pdf ● NOTE: Constantly updating, so there will be more, etc.
• 33.
  #Kscope Troubleshooting Tools –Internal (cont.)
• 34.
  #Kscope Troubleshooting Tools –Internal (cont.)  Data Gathering ● Log Files (see Table) ● HFMErrorLookup.exe – Resolves Error Codes to Text Description ● System 11- <EPM ORACLE HOME>EPMSystem11R1productsFinancialManagementConsultant Utilities ● System 9 - <HYPERION HOME>FinancialManagementConsultant Utilities
• 35.
  #Kscope Troubleshooting Tools –Internal (cont.)  Data Gathering ● HFMErrorLogViewer.exe – Utility to view HFM error logs
• 36.
  #Kscope Troubleshooting Tools –Internal (cont.)  Data Gathering ● FRConfig.cmd – FR utility that shows app Mem / CPU utilization ● System 11 - <EPM_ORACLE_HOME>productsfinancialreportingbin
• 37.
  #Kscope Troubleshooting Tools –Internal (cont.)
• 38.
  #Kscope Troubleshooting Tools –Internal (cont.)  Data Gathering ● ManageUserPOV.cmd – View/Edit POV for users
• 39.
  #Kscope Troubleshooting Tools –Internal (cont.)  Data Gathering ● Remote Diagnostic Agent (RDA) – Gathers system information about environment. [See KB ID 1304885.1] ● oPatch [System 11 Only] – Install and list installed patches ● Location - <EPM ORACLE HOME>Opatch ● List Installed products = Opatch lsinventory ● NOTE: If you get an error running about OUI Home, make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
• 40.
  #Kscope Troubleshooting Tools –Internal (cont.)
• 41.
  #Kscope Troubleshooting Tools –Internal (cont.)  Data Gathering ● Comps.xml [System 11 Only] – List of all installed components on machine ● <EPM ORACLE HOME>inventoryContentsXML
• 42.
  #Kscope Troubleshooting Tools –Internal (cont.)  Data Gathering ● Epmsys_registry.bat [System 11 Only] – Generates a detailed report of everything registered in Shared Services Registry. ● Program - <MIDDLEWARE HOME>user_projectsepmsystem1bin ● Reports - <MIDDLEWARE HOME>user_projectsdiagnosticsreportsregistry.html
• 43.
  #Kscope Troubleshooting Tools –Internal (cont.)
• 44.
  #Kscope Troubleshooting Tools –Internal (cont.)  Data Gathering ● ‘About Us’ pages – Shows currently installed programs and specific version info. (Applies to Workspace, FDM, Shared Service web pages, etc.) ● Click on the ‘Show Details’ button to see version info per product ● NOTE : This version info *MAY NOT* change even if you have installed patches!
• 45.
  #Kscope Troubleshooting Tools –Internal (cont.)
• 46.
  #Kscope Troubleshooting Tools –Internal (cont.)  Data Gathering ● Informational Web Pages ● http://hostname:port/workspace/browse/configInfo - Returns detailed information about products installed and links to other configuration files
• 47.
  #Kscope Troubleshooting Tools –Internal (cont.)  Data Gathering ● Informational Web Pages ● http://<shared_services_server>:28080/interop/frameowrk/g etCSSConfigFile – Returns configuration information for Shared Services ● System 9 – Port Number is 58080 ● System 11 – You may need to administratively enable this page for it to be visible!
• 48.
  #Kscope Troubleshooting Tools –Internal (cont.)
• 49.
  #Kscope Troubleshooting Tools –Internal (cont.)  Data Gathering ● Informational Web Pages ● http://<hfm_web_server>:80/hfm – Returns ‘hfm’ if HFM is running properly and integrated with workspace. (else opens HFM Web)
• 50.
  #Kscope Troubleshooting Tools –Internal (cont.)  Utility Programs ● Validate.bat [System 11 Only] - Kicks off process that will test installed components on the machine for proper operation. Generates detailed report. ● Can be run on product by product basis or all at once ● Program - <MIDDLEWARE HOME>user_projectsepmsystem1bin ● Report - <MIDDLEWARE HOME>user_projectsepmsystem1diagnosticsreports ● NOTE : If you are building an environment, you should be running this after your complete install/config!
• 51.
  #Kscope Troubleshooting Tools –Internal (cont.)
• 52.
  #Kscope Troubleshooting Tools –Internal (cont.)
• 53.
  #Kscope Troubleshooting Tools –Internal (cont.)  Utility Programs ● DB_Verify / DB_Recover [Pre-System 11 Only] - Checks and corrects Shared Services OpenLDAP database issues.
• 54.
  #Kscope 3rd Party Tools
• 55.
  #Kscope Troubleshooting Tools –3rd Party  Data Gathering ● WebEx / GoTo Meeting / Log Me In / VNC / Lync – Remote Access Software. A must for remote support. ● Windows Event Log
• 56.
  #Kscope Troubleshooting Tools –3rd Party  Basic Connectivity Tools ● Ping / Tracert – Simple connectivity test tools
• 57.
  #Kscope Troubleshooting Tools –3rd Party  Basic Connectivity Tools ● Telnet – Connectivity tool for any port
• 58.
  #Kscope Troubleshooting Tools –3rd Party (cont.)  Advanced Connectivity Tools ● Fiddler – Web proxy that allows you to monitor (and simulate) HTTP/HTTPS traffic. http://fiddler2.com/fiddler2/ ● Great Tool for HTTP/HTTPS traffic ● Warning : You can sniff / decode HTTPS traffic ● Warning : Some programs may send logins / passwords in plain text.
• 59.
  #Kscope Troubleshooting Tools –3rd Party (cont.)
• 60.
  #Kscope Troubleshooting Tools –3rd Party (cont.)
• 61.
  #Kscope Troubleshooting Tools –3rd Party (cont.)  Advanced Connectivity Tools ● Wireshark – Network traffic analyzer – http://www.wireshark.org ● Network packet analyzer ● Monitors ALL network traffic, not just HTTP/HTTPS ● Warning : Can intercept all types of traffic such as: ● Streaming Audio / Video ● VoiP ● Types of Encrypted Data ● Plain Text User Name / Password
• 62.
  #Kscope Troubleshooting Tools –3rd Party (cont.)  Advanced Connectivity Tools ● Wireshark – Network traffic analyzer – http://www.wireshark.org ● Network packet analyzer ● Monitors ALL network traffic, not just HTTP/HTTPS ● Warning : Can intercept all types of traffic such as: ● Streaming Audio / Video ● VoiP ● Types of Encrypted Data ● Plain Text User Name / Password
• 63.
  #Kscope Troubleshooting Tools –3rd Party (cont.)
• 64.
  #Kscope Troubleshooting Tools –3rd Party (cont.)  Server Tools ● Service Manager (services.msc) – View services to see which are running ● Process Monitor – Allows you to monitor processes’ file use, registry access, memory utilization, etc. http://technet.microsoft.com/en-us/sysinternals/bb896645
• 65.
  #Kscope Troubleshooting Tools –3rd Party (cont.)  Server Tools ● SQL Profiler – Allows you to monitor database activity in SQL Server – http://msdn.microsoft.com/en- us/library/ms181091.aspx ● [Oracle Enterprise Manager has similar functionality for Oracle databases]
• 66.
  #Kscope Troubleshooting Tools –3rd Party (cont.)
• 67.
  #Kscope Troubleshooting Problems
• 68.
  #Kscope Migrate VM ServerError
• 69.
  #Kscope Migrate VM ServerError  To aid with session, built VM/Virtual PC System 11 “All In One” Environment
• 70.
  #Kscope Migrate VM ServerError (cont)  After copying VM to another machine…
• 71.
  #Kscope Migrate VM ServerError (cont)  But not everything was dead… VM to another machine…
• 72.
  #Kscope Migrate VM ServerError (cont)  IDEAS?? ●What changed?
• 73.
  #Kscope Migrate VM ServerError (cont)  Lets get some machine info
• 74.
  #Kscope Migrate VM ServerError (cont)  Lets get some config (epmsys_registry.bat)
• 75.
  #Kscope Migrate VM ServerError (cont)  Lets get some config (epmsys_registry.bat)
• 76.
  #Kscope Migrate VM ServerError (cont)  IDEAS?? ●Anyone see anything suspect so far? ●I did, and have a suspicion, but need more information to confirm.
• 77.
  #Kscope Migrate VM ServerError (cont)  Lets do some sniffing…(Wireshark)
• 78.
  #Kscope Migrate VM ServerError (cont)  IDEAS?? ●Why would there be calls to an IP address that does not belong to the machine?
• 79.
  #Kscope Migrate VM ServerError (cont)  Lets do some sniffing…(WinGrep)
• 80.
  #Kscope Migrate VM ServerError (cont)  Lets do some more sniffing…(SQLLocator)
• 81.
  #Kscope Migrate VM ServerError (cont)  Lets do some more sniffing…(SQLLocator)
• 82.
  #Kscope Migrate VM ServerError (cont)  IDEAS?? ●Looks pretty reasonable that hardcode IP addresses are breaking things since the machine no longer has the IP address buried in config / database!
• 83.
  #Kscope Migrate VM ServerError (cont)  How To Fix ●Reinstall? ●Reconfigure? ●Manually Search and Replace?
• 84.
  #Kscope Manual Repair! (Because wecan)
• 85.
  #Kscope Migrate VM ServerError (cont)  Lets fix the SQL Damage First
• 86.
  #Kscope Migrate VM ServerError (cont)  Lets fix the SQL Damage First
• 87.
  #Kscope Migrate VM ServerError (cont)  Lets fix config file damage ● Content.XML ● Config.XML ● GlobalContext.XML ● httpd.conf ● mod_wl_ohs.conf ● Server.xml  Warning: In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier. That could be bad.
• 88.
  #Kscope Migrate VM ServerError (cont)
• 89.
  #Kscope Migrate VM ServerError (cont)  Reboot
• 90.
  #Kscope Migrate VM ServerError (cont)  Mission Accomplished!
• 91.
  #Kscope Questions?  If youhave any questions with the sample code or presentation, feel free to ask/contact me ● [email protected]