Real time data integration best practices and architecture

Real time data integration best practices and architecture

• 1.
  Real-Time Data Integration BestPractices and Architectures
• 2.
  Real-time Data Quality Missed Opportunities Lower Customer Service Levels TheNeed for Timely Information 2 Customers Indirect Sales Channel Sales & Marketing Enterprise Product Designers Enterprise Information System Retail Stores Call Center Increased Costs Market Share Erosion Increased Exposure to Risk RT Data Warehouse Data Synchronization Operational Data Hub Data Warehouse Operational Data Hub
• 3.
  3 Applications created in2012 using traditional architecture models will be an IT-constraining legacy by 2016 The leading business applications of 2016 are designed today using Nexus-enabled application architecture principles
• 4.
  Changing Perspectives onData • It is no longer sufficient to view information “after the fact”. • Business demands information sooner, with more accuracy, in order to meet competitive and regulatory demands. • Business needs to respond to “threats” and “opportunities sooner. • Reduce decision latency. • Proactive Alerts and Notifications 4
• 5.
  What does “Real-Time”mean to you? 5 Days Performance Management Sales by Business Unit Hour Inventory Management Compliance Minutes Order Confirmation Operational Metrics Second Straight-through Processing Call-center Analytics Point-of-entry Applications Instantaneous Securities Exchange Operational Data Integration Analytical Data Integration
• 6.
  Application 2 Traditional Integration 6 Application1 Application Logic Data Application Logic Data Application 3 Application Logic Data Custom Extracts and Interfaces Query/pull Interfaces Interfaces Query/pull “Siloed” Applications, Embedded Interfaces and Transformation Interfaces
• 7.
  Traditional Integration: Result… 7
• 8.
  Critical Considerations • Complexityof source and targets. • Data formats. • Level of quality. • Availability of source/target interfaces. • Volume, velocity; and variety. • Delivery requirements. • Real-time, near real-time, batch/scheduled. • Loose coupling/reusability. • Performance/Availability. 8
• 9.
  Real-time Data IntegrationPatterns • Transactional Data Processing • Data Replication • Data Integration Hub (DIH) • Event Driven Architecture (EDA) 9 Applications Processes BI Tools Portals Web Services Batch Trickle Feed Real Time Applications Databases Messages Flat Files XML Unstructured Data Mainframe Data Consumers Data Sources Operational Data (Field Devices, Applications, etc.) Location Context (GIS) BI, DW, Other Targets Event Based Applications Communication “Cloud” Data Receipt / Transformation & Mapping Real Time Event Transport / Delivery Event Processing Real Time Web Content Delivery IDRPWC/ PX Source Applications/ Technologies Interaction management (Publications & subscriptions) Data Governance Persistence Hub Management Archiving Delivery & Schedule Catalog Service Data & Event Monitoring Validation & Quality Rules On-boarding Monitoring & Auditing Data Transformation & Enrichment Cloud App (HR) POS App CRM Planning Master Data Data Warehouse Big Data (Analytics) Finance
• 10.
  Transactional Data Processing 10 1 OperationalBI & Real-Time DW Provide freshest information to business w/o impact, off-load reporting, & do high volume updates (ODS/OLTP/DW/Appliance) 2 Operational (Sync) Ensure all users are seeing the same, trusted, up-to-date data that is synchronized across the organization. Transactional/Production Applications Merge/Apply, Reports & Queries Source System Target Systems ODS/OLTP/ DW/Appliance Transactional Applications Merge/Apply, Reports & Queries Source System Target Systems ODS/OLTP/DW
• 11.
  Transactional Data Processing 11 EXTRACT SERVER MANAGER SERVER MANAGER http:// APPLY Console SourceSystem Target System SQL Apply Merge Apply Audit Apply Intermediate Files Committed Checkpoint Checkpoint High Speed Extraction High Speed Parallel Apply JMS*
• 12.
  In-Memory Cache Synchronization 12 CacheNode Cache Node Cache Node Data Sources Real Time Data Replication In Memory Grid Consuming Applications
• 13.
  Data Integration Hub 13 Data Integration Hub (Publish/ Subscribe) DefineTopic Define Publishers (Connection properties, Mode, Frequency) Define Transformation and validation rules Define Security & Access Policies Define Subscribers (Connection properties, Mode, Frequency) Deploy (Instantiate persistence and data integration flows) Monitor, Govern, Alert & Audit Cloud App (HR) POS App CRM Planning Master Data Data Warehouse Big Data (Analytics) Finance
• 14.
  Data Integration Hub 14 Pub.Identification DataQualityFirewall Statelogging, Error handling XMLNormalization Sub.Transformation Sync-upControl Sub.Identification Sub.Delivery Sub. Acknowledgement Command Interface Monitor and Alert Interface Transfer Interface Applications Applications Transfer Interface Enterprise Scheduler Enterprise Monitor File Manager Connector Connector Prompt:$ SNMP Prompt:$
• 15.
  Event Driven Architecture •An architecture in which the activity is driven by changes in state within an environment. • Events drive the execution of logic. 15 EventSources(Producers) DetectedSituations(Consumers) Integration/Direct Integration/Direct Pre- Process Process Post- Process Patterns Alerts Event Streams Actions Devices Systems Applications People
• 16.
  Sample EDA ReferenceArchitecture 16 Operational Data (Field Devices, Applications, etc.) Location Context (GIS) BI, DW, Other Targets Event Based Applications Connectivity & Data Replication Transformation & Mapping Real Time Delivery Event Processing Real Time Web Content Delivery
• 17.
  Event Enablement andTransformation 17 • Event-enable and collect events from underlying systems and applications. • Data Replication • PowerExchange/PowerCenter • Transform events to normalized format for downstream processing. • B2B Data Transformation • Enrich events as needed. Transformation & Mapping Connectivity & Data Replication
• 18.
  Event Transport 18 • Delivertransformed events to downstream applications, systems, and event processors. • Ultra Messaging • Dynamic Routing • Parallel Persistence • Guarantees delivery, and decouples event producers from event consumers. Real Time Delivery
• 19.
  Event Rules andDelivery 19 • Processes events, applies user/business defined rules, and generates responses. • RulePoint • Rules may be applied across events and over time/space. • Alerts/notifications may be sent to BI environments, web applications, etc. Real Time Web Content Delivery Event Processing
• 20.
  Architectural Implications 20 Information and DataArchitecture Application Architecture and Development Infrastructure Architecture • Shift to modeling events as enterprise assets. Incorporate the processing of immediate, individual events. • Deliver quality data as it is needed by the business. • Complement existing SOA strategies with event driven applications. • Move to a single environment that supports different integration delivery models. • Shift from centralized, database-centric client-server applications to distributed systems. • Deploy infrastructure that can support data delivery over different latencies.
• 21.
  Summary • Augment traditionalintegration solutions with real-time data integration to: • Reduce decision latency. • Improve decision making and responsiveness. • Increase visibility. • Build upon the principles of SOA and an event-driven architecture (EDA). • Informatica provides the capabilities for adding real-time data integration to your environment. 21
• 22.
  22 1 2 3 Join our exclusive Potentialat Work Communities. Visit the kiosk. Get a PowerCenter Express demo! Visit the Pavilion kiosk. Tell us what you think. Click on Evaluations in the IW13 Mobile App.