Presentation racsig 090730

Managing Performance and Availability for
25,000 Siebel Contact Center Users with
Oracle Real Application Clusters
Roland Höhl, Perot Systems / Deutsche
Telekom AG
30rd
of July, 2009
RAC SIG Webseminar
roland.hoehl@ps.net
roland.hoehl@telekom.de

2
 Introductions
 Project Overview & Business Drivers
 Scope and Usage of the system
 Technical Footprint - Handling 25,000 Users
 Architecture & Infrastructure
 Session Management
 RAC Performance
 System Monitoring
 Reporting
 System Interfaces
 Lessons Learned
 Future plans
Agenda

3
Who am I / Biography Roland Höhl
 46 years old
 Studied at the university in Darmstadt / Germany
 Perot Systems employee
Manager Application Development
 Working for Deutsche Telekom as
external consultant in the CRM-T
project since 2006
 In the Deutsche Telekom CRM project responsible for
 technical architecture
 infrastructure sizing and planning
 release management

4
Company overview – Deutsche Telekom
 Internationalization:
Deutsche Telekom is represented in ~ 50 countries
worldwide..
 Employees:
Approx. 242,000 employees worldwide (December 31,
2007)
 Revenue:
62.5 billion euros (2007 financial year)
 Figures T-Home
 37 Million Fixed Line User (PSTN)
 No.1 in German DSL-Market
(total amount12, 5 Mio Customer)
 150.000 new Customer for IPTV (Entertain) in 2007
 1.800 Sale points (Biggest integrated sales- and
Service
organization for Fixed and Mobile products)

5
 Introductions
 RAC Performance
 Reporting
 Lessons Learned
 Future plans
Agenda

6
Facts and figures
 Project scope
 Siebel upgrade from 7.0 to 7.8.2.5
 Change from DB2 Host (OS390) to Oracle 10g RAC Unix
 Reimplementation of customized Siebel business processes
 Data migration from DB2 to Oracle 10g (based on EIM load)
 Building a ESB around Siebel CRM (based on MQSeries)
 Project staffing:
 Oracle – Software vendor (Siebel / Database)
 IBM Global Services – Development Siebel / EAI
 T-Systems - Operations and hosting
 Duration:
 Project start: 11/2005
 First release: 04/2008
 Involved employees: ~800

7
Project Goals
Strategic Goals
Improve customer
satisfaction and raise
call center efficiency
Time reduction to
develop new
products
and services
Cost reduction
for operations,
maintenance and
enhancements
Standardization,
risk mitigation and
protection of
investment
Functional Scale
identical with CRM-C V2.130
(KM, KKM, OM)
Upgrade to version:
Siebel Standard SW 7.8
Align reduction of the application to
Siebel Standard
Reduction of interface complexity
(Service oriented integration platform)
Hardware change –
Change of database platform if applicable
Master function of CRM-T for customer and
contract data (End of parallel operation, KoSi)
Improved application operation,
incident and problem management
Operative Goals

8
Project Business Drivers
 Advantages for the customers:
 Improved customer service
 Increased customer satisfaction
 Advantages for the employees:
 Enhanced graphical user interface  Simplification of IT systems
 Single and completed view of customer information
 Transparency of processes
 Decreased administrative complexity
 Advantages for the Deutsche Telekom:
 Simplification of IT processes
 Improvement of IT stability
 Increased flexibility through open standards
 Retirement of legacy systems
 Achieved saving potentials in operations, support and development
 1. Milestone in redesign of the IT architecture

9
 Introductions
 RAC Performance
 Reporting
 Lessons Learned
 Future plans
Agenda

10
Deutsche Telekom – T-Home - CRM Landscape
2 Data Centers Data Warehouse
Legacy Systems
T-ShopT-Shop
Call CentersCall Centers
PartnerPartner
T-ESB

11
Scope and usage
 89 call center within Germany
 20000 named call center users
 7000 named T-Shop and partner users
 80 million accounts
 up to 40.000 customer contacts per day
(Inbound items, i.e. Calls, Mails, Docs)
 ~ up to 10.000 orders / hour
 ~ 19 TB Data in the Database
 ~ 40.000.000 user requests / hour
 ~ 50.000.000 disk reads / hour
 Up to 7.000.000.000 buffer gets / hour
 42 Legacy systems connected
 several million messages per day transferred

12
 Introductions
 RAC Performance
 Reporting
 Lessons Learned
 Future plans
Agenda

13
Infrastructure Architecture

14
Web Server Infrastructure
Software Configuration
• AIX 5.3 ML8
• IBM IHS 2.0
• Siebel Web Server Extension 7.8.2.5
Hardware Configuration
• 12 x IBM P5 9133-55A
•4 x 1.65 GHz P5 CPUs
•24 GB Memory
• CISCO CSS 1150
•Frontend and Backend Load balancer

15
• AIX 5.3 ML8
• Siebel 7.8.2.5
• WebSphere MQ V6 (on EAI Servers only)
• IBM HACMP (Cluster) on EAI Servers
• Genesys CTI Driver
Siebel Application Infrastructure
Users related Servers
• 6 x IBM P595 (eCommunications)
•48 x 5 GHz P5 CPUs
•336 GB Memory
• 3 x IBM P595 (eCommunications)
•128 GB Memory
• 2 x IBM P590 (Siebel eConfigurator)
EAI Server
• 8 x IBM P590 (Active)
•8 x 2.1 GHz CPUs
•24 GB Memory
• 8 x IBM P590 (passive – Virtual Servers)
•0.1 x 2.1 GHz CPUs
•24 GB Memory
Batch/Read Audit Servers
• 4 x IBM P590 (active + passive)
•8 x 2.1 GHz CPUs
•32 GB Memory

16
Oracle RAC Infrastructure
• AIX 5.3 ML8
• ORACLE 10.2.0.3
• ASM, Streams
Oracle RAC Nodes
• 4 x IBM P595
•312 GB Memory
SAN Storage 160 TB
• 10 x IBM DS8300
•RAID 10
•16 x 146 GB Disks
•2 x 2 GB/sec FC
Solid State Disks
• 2 x Texas Memory Systems RamSan-500
•2 TB Flash RAID
•2 GB/sec Bandwidth

17
 Introductions
 RAC Performance
 Reporting
 Lessons Learned
 Future plans
Agenda

18
Session Management for Performance, Availability,
and Scalability Sessions Handling
• Up to 20.000 user session + several hundred sessions
from EAI Components will be handled on the Siebel
System
• 2 Options to handle them on the Database
•Oracle MTS
•Siebel Connection Pooling

19
and Scalability Sessions Management
Siebel Connection Pooling of 5:1 for User Sessions
• Session are connected via DB Services
•Up to now each session will connect via the same
service
•Multiply Services allow to
•Separate – Distribute – Prioritize - …
load over the different RAC Nodes.
Siebel
Connection
Pooling

20
and Scalability
Siebel
Connection
Pooling
Resource Management
• Siebel implements Transparent Application Failover TAF
internal
• Bad SQL statements have been hard to stop
• Solution was Resource Profiles
•CPU limitation

21
 Introductions
 RAC Performance
 Reporting
 Lessons Learned
 Future plans
Agenda

22
RAC Performance & Availability Aspects
 General memory considerations
 Session size up to 15 MB
 Keep enough memory for the operating system
 10-20% more memory on RAC environments
 Amount of cluster nodes
 Expected load and hardware limitations defines
the minimum # of nodes
 More nodes provide better availability / less
failure impact (MTBF, upgrades, …)
 More nodes allows dedicated resource
assignment
 Less nodes reduce interconnect traffic,
management overhead
•Interconnect
•Dedicated infrastructure
•Failover Interfaces / Channel bundling

23
RAC Performance & Availability Aspects II
 Disaster Scenarios and Stretch Cluster Limitations
 20 – 50 km Max
 Disaster Recovery
 Consider Rolling Upgrades
 Increase the availability – reduce planned down time
 Define database services as soon as possible
 Group Servers
 Separate load
 Control performance and availability
 Watch out for Architectural bottlenecks
 Centralized Tables, Sequences
 Prevent, Tune, or Invest

24
RAC Performance Configuration
 Consider to use Partitioning
 Try to spread out load but keep the data local
 Consider that i.e. DB Triggers will be fired on each node
 Watch out for High frequently changing DB objects
 Consider less records per block
 Less records decreases block concurrency
 Less nodes request the same block at a single point in time
 Use caching
 Increase cache size for sequences
 Pin small tables into the cache
 Oracle features
 ASSM, ASM, Reverse Indexes, Function Based Indexes, Partitioning
 Follow best practices

25
 Introductions
 RAC Performance
 Lessons Learned
 Future plans
Agenda

26
System Monitoring
 Joint Team of Admins
 IBM Tivoli
 Oracle GRID Control
 Quest Foglight Experience Monitor
 Interface monitoring

27
Joint Team of Admins
 Joint team of Siebel , EAI and Database Admins
 Total overview of the system
 Database monitoring
 Siebel view (functional SQL process monitoring)
 Database view (technical monitoring)
 Queue Monitoring (Siebel inbound and outbound)
 Technical Monitoring
 Logs, CPU, Memory, Storage etc.
 Propose technical improvements

28
System Monitoring
 IBM Tivoli

29
IBM Tivoli
 Operational Management
 System Automation
 Monitoring (CPU, Memory…)
 Workload Scheduling
 Service Level Advisor
 Storage Management:
 Tivoli Storage Manager
 SAN Volume Controller
 Alerting

30
System Monitoring
 IBM Tivoli

31
Oracle GRID Control
 RAC monitoring
 Long running transactions
 Wait States

32
System Monitoring
 IBM Tivoli
 Quest Foglight Experience Monitoring

33
Quest Foglight Experience Monitor
Coverage:
 Round about 60 business
processes are being monitored
(90% coverage)
 Various report sets are available
to users
Applicable Fields:
 Monitoring of software quality
after deployment
 Alarm based control of the
production environment
 Information base for sizing
recommendations and server
consolidation
Acceptance:
 FxM is accepted as an objective
measuring instrument which
reflects “end user reality” and is
used to support SLA monitoring
and reporting

34
System Monitoring
 IBM Tivoli
 Interface Monitoring

35
Interface Monitoring
 The Telekom Enterprise
Service Bus

36
Interface Monitoring
 Evaluation of auditing
messages
 Monitor traffic and response
times on interfaces

37
 Introductions
 RAC Performance
 Reporting
 Lessons Learned
 Future plans
Agenda

38
Reporting
 OA Mirror (Operative
analysis)
 RO Mirror (Read only)
Logshipment
(arch and
redo)
CRM RAC
Database
Stage DB
Capture process
Propagation process
RO Mirror DB
Physical Standby
read only
Logshipment
(arch and redo)
OA Mirror DB
Apply processes (16
CDC-capture
CDC-apply (8)
Data warehouse
CRM OA
Reports

39
CRM technical achievements
 Reduced Response times
 Increased Automation
 Faster time to market and increased flexibility for new products
 Less development more configuration
 Service Oriented Architecture is implemented
 Host platform retired
 OLAP database with realtime mirroring based on Streams technology

40
 Introductions
 RAC Performance
 Reporting
 Lessons Learned
 Future plans
Agenda

42
 Introductions
 RAC Performance
 Reporting
 Lessons Learned
 Future plans
Agenda

43
Lessons Learned
 Get a 360° view on the system
 Test with full blown database as early as possible
 Test the infrastructure with worst case scenarios
 Test the system infrastructure carefully before roll out
 Test your migration with real data
 Always have a look at system and application performance
 Avoid load mixing of batch and OLTP
 Keep commit sizes low (Batch Jobs)
 Keep work on a single node when possible
 Be prepared for long running queries
 Slim line tables with highly changing content

44
 Introductions
 RAC Performance
 Reporting
 Lessons Learned
 Future plans
Agenda

45
Future Plans
 Additional customer groups will be integrated and served through the new
platform
 Small business customers
 Medium business customers
 Additional product types will be integrated to process the complete portfolio
 Internet access and value added services
 Next generation network services
 Service Architecture will be enhanced
 Additional services will be externalized
 Further automation improvements
 Continuous systems consolidation and renewal of IT architecture
 Upgrades to Oracle 11g and Siebel 8.x

46
 Introductions
 RAC Performance
 Reporting
 Lessons Learned
 Future plans
 CRM-T Infrastructure details
Infrastructure in detail

47
CRM-T Infrastructure details
Web Server and Reporting

48
Quest Monitoring

49
Object manager and Batch Server

50
EAI Server and ESB

51
Database

52
Storage

Presentation racsig 090730

More Related Content

What's hot (19)

Viewers also liked (20)

Similar to Presentation racsig 090730 (20)

More from maclean liu (20)

Recently uploaded (20)

Presentation racsig 090730

Editor's Notes