Application Design
1. Users (Actors) of the system
2. Requirements
- Functional (input / Output /Read to Write)
- Non Functional (Data Partitioning & Replication / Load Balancing /Security / Caching)
- Extended (Analytics)
3. System API's (CRUDs) (api_dev_key)
4. Database Design
- SQL - Highly scalable vertically / ACID Compliant / Reliable
- NoSQL - (Key Value Pair) - Metadata/ No-fixed Scheme / Highly Scalable Horizontally / No
Downtime is required/ good where changes happen frequently
- File Storage (Amazon S3/ GCS) - On Cloud Storage / Can be coupled with CDNs / Scaling,
availability taken care of by vendor
5. Database Schema - Define Tables and relationships / Audit & Logging
5. High Level Design - Abstract figure showing the flow of design
6. Data Partitioning
- Horizontal Partitioning (Different DB split based on Key values) - Leads to unbalanced servers
- Vertical Partitioning (Different columns) - When system grows, eventually needs to be
partitioned again
- IP-Hash based Partitioning (Calculate Hash value based on IP address)
- Consistent Hashing
7. Caching
- What to Cache, 20/80 Rule 20% data is fetched by 80% traffic
- Cache Invalidation Scheme - Write Through/ Write Around / Write Back
- Distributed Caching - Distribute Caches among many servers, Leads to Cache misses
- Global Caching - Looks for cache on all servers, if not fetches from DB
- CDN - Caching Static media like html, pictures, Videos
- Cache Eviction Policy - FIFO / LIFO / LRU / LFU / MRU / RR (Random Replacement)
8. Load Balancing
- Health Checks (Heartbeat)
- Least Connection Method / Least Response time/ Least Bandwidth / Round Robin / Weighted
Round Robin / IP Hash
9. Data Replication
- Eliminate Single Point of failures
- Leader & Follower Election, Use of Heartbeat Checks
- Disaster Recovery
10. Security
- Authentication - Multifactor Authentication (MFA) / SSO (SAML 2.0 or OAuth 2.0 or Open ID
Connect) / Service Accounts
- Authorization - Role based functions
- Encryption - Wherever PII (Symmetric (Base 62 / 64) / Asymmetric Encryption (Public/Private
Key))
11. Network & Protocols
- IP - Transfer of packets
- TCP - Built over IP - Transfer of packets in order
- HTTP - Text Transfer Protocol
- HTTPS - Secure, SSL certificate provided by third party, Public/private key encryption
- FTP - File Transfer Protocol
System Integration
1. Requirements
- Functional (input / Output /Read to Write)
- Non Functional (Network)
- Extended (Analytics - ETL- Extract, Transform, Load)
2. Divide and Conquer - High Level - Abstract components
3. Real time / Batch / Bulk
4. Synchronous / Asynchronous
5. Fit Patterns - File Transfer / Shared Database / REST or SOAP Api's / Messaging Systems
- Messaging patterns *
- Recipient List vs Pub sub
6. Exception Handling
- Control Bus
- Wire tap
7. Reprocessing / Full Load + Delta Load
8. Authentication - API keys
9. Security & Privacy
- Authorization - Service Accounts
10. API Details (Idempotent - Persistence)
11. Resilience
Integration Patterns:
1. File transfer
2. Shared Database
3. Real time calls (REST/SOAP)
4. Messages
Messaging Patterns:
1. Message
2. Channel
3. Endpoints
4. Channel Adapter
5. Transformation
6. Point to Point Channel
7. Publisher Subscriber
8. Invalid Message Channel
9. Dead Letter Channel
10. Messaging Bus
11. Document Message
12. Event Message
13. Request Reply
14. Return Address
15. Message Sequence
16. Correlation ID
17. Message Expiration
18. Content Based Router
19. Message Filter
20. Recipient List
21. Splitter
22. Aggregation
23. Re sequencer
24. Composite Message Processor (Splits, Routes, Aggregates)
25. Scatter Gather (Recipient List + Aggregator)
26. Routing List (Sequence Based on Route)
27. Process Manager (Sequence Based on result)
28. Envelop wrapper
29. Envelop un-wrapper
30. Content Enricher
31. Content Filter
32. Claim Check
33. Normalizer
34. Durable Subscriber
35. Control Bus
36. Detour
37. Wire tap
38. Message History
39. Smart Proxy
40. Test Message Generator
41. Monitor / Management Console
42. Test Data Verifier
43. Dynamic Recipient List
44. Dynamic Router
Popular Patterns:
1. File Based
2. Quote from Vendors
3. Test Messages
4. Analytics
5. Monitoring
6. Order to Item
