Wtu 2014 ibm web sphere mq for zos - shared queues

Wtu 2014 ibm web sphere mq for zos - shared queues

• 1.
  IBM WebSphere MQfor z/OS Shared Queues Alexander Ross ([email protected]) Software Developer
• 2.
  Please Note IBM’sstatements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here. © 2014 IBM Corporation 2
• 3.
  3 Agenda •What are Shared Queues? • Large messages with DB2 • Shared Message Data Sets (SMDS) • Structures – Persistence and recovery
• 4.
  Application Application 5 Shared queues QMgr QMgr Shared Coupling Facility (CF) Queue Same Sysplex
• 5.
  7 Queue-sharing groups(QSGs) Queue-Sharing Group Shared Objects Msg data > 63KB DB2 Data Sharing Group CF Shared Queues Channel Initiator QMGR Private Queues Private Objects Channel Initiator QMGR Private Queues Private Objects DB2A DB2B DB2C
• 6.
  11 Shared MessageData Sets (SMDS) Queue-Sharing Group DB2 Data Sharing Group SMDS SMDS Shared Objects CF Msg data > 63KB Shared Queues Channel Initiator QMGR Private Queues Private Objects Channel Initiator QMGR Private Queues Private Objects DB2A DB2B DB2C V7.1
• 7.
  13 CF structuresfor shared queues { Coupling Facility Administration structure Application structures Information for unit-of-work recovery { Administration Information for unit-of-work recovery structure Application Queue Queue Queue structures Structures for QSG 1 Structures for QSG 2 x 63 Queue Queue Queue x 512
• 8.
  Creating CF structuresand shared queues • Define a structure to z/OS by updating the CFRM policy: – The structure is known to MQ by its 12-character str-name. – The structure is known to z/OS by the 16-character name formed by: • Application structures: qsg-name || str-name • Administration structures: qsg-name || CSQ_ADMIN • Application structures are defined to MQ using the DEFINE CFSTRUCT command on any queue manager in the QSG • Shared queues are defined using the DEFINE QLOCAL command on any queue manager in the QSG: – DEFINE QLOCAL(queue-name) QSGDISP(SHARED) CFSTRUCT(str-name) • z/OS allocates the structure in the CF when required (first use) • MQ creates the queue when required (first use) 15
• 9.
  17 Large sharedqueue messages (using SMDS) QM1 Ptr to 100K message QM1 SMDS QM2 SMDS QM2 APP MQPUT APP MQGET 1 2 3 4 Shared queue V7.1 • One SMDS per queue manager per CF structure • Each queue manager only writes large messages to its own SMDS • Queue managers can get messages from any queue manager’s SMDS
• 10.
  3 LPAR Test- SMDS 64KB Non-Persistent Messages In-Syncpoint - SMDS 1 2 3 4 5 6 7 8 9 10 19 SMDS performance improvement 3 LPAR Test - DB2 64KB Non-Persistent Messages In-Syncpoint - DB2 1 2 3 4 5 6 7 8 9 10 400 350 300 250 200 150 100 50 0 NP SIS Scaling – 3 qmgr Queue Pairs NP SIS Scaling – 6 qmgr NP SIS Scaling – 9 qmgr Transactions / Second 7000 6000 5000 4000 3000 2000 1000 0 NP SIS Scaling – 3 qmgr Queue Pairs NP SIS Scaling – 6 qmgr NP SIS Scaling – 9 qmgr Transactions / Second • Tests show comparable CPU savings, making SMDS a more usable feature for managing your CF storage • SMDS per CF structure provides better scaling than DB2 BLOB storage
• 11.
  Creating a SharedMessage Data Set • SMDS is defined as a VSAM data set using DEFINE CLUSTER – Requires LINEAR option – Control interval size must be 4096, which is the default for linear data sets – Requires SHAREOPTIONS(2 3), that allows one queue manager to write and other queue managers to read at the same time – If maximum size may need to exceed 4GB, requires SMS data class which has VSAM extended addressability attribute – If automatic expansion is to be supported, requires an appropriate secondary space allocation, although a default of 20% will be used if an expansion attempt fails because of no secondary allocation • Can optionally be pre-formatted, for example using CSQJUFMT – Otherwise formatted automatically when first opened 21
• 12.
  Access to SharedMessage Data Sets • Shared message data sets must be on shared direct access storage accessible to all queue managers in the QSG. • Normal running: – Queue manager opens own data set read/write. • Requires UPDATE access to own data set. – Queue manager opens other data sets read-only. • Requires READ access to all other data sets. • Media recovery processing: – Queue manager performing recovery opens own data set and all other 23 data sets for read/write access. • Requires UPDATE access to all data sets.
• 13.
  Shared Message DataSet capacity considerations • Each shared message data set only contains data for large messages written by its owning queue manager • Message size calculation: – Each stored message includes standard headers (usually 352 bytes) – Each message is stored as one or more message blocks – Each message block is stored in a range of consecutive 4KB pages on the data set, with a very small header (32 bytes) – Approximate data set space required per large message, in bytes, is given by size of message plus the header rounded up to the next 4KB • Multiply by the maximum anticipated backlog of messages written by that queue manager (plus some safety margin) to estimate the required data set size 24
• 14.
  Shared Message DataSet expansion • Data set can be automatically expanded when necessary – Normally configured by the DSEXPAND(YES|NO) option on the CFSTRUCT object, which specifies the default option for the data set group. – Can also be overridden for individual data sets using ALTER SMDS with the DSEXPAND option • Expansion attempt is automatically triggered when 90% full – If no secondary allocation was specified a VSAM error message will appear, but the queue manager will retry using a default secondary allocation of 20% of the existing size – If expansion fails because there is not enough space available the queue manager sets DSEXPAND(NO) to prevent further attempts. The operator can use ALTER SMDS to restore DSEXPAND(YES) once the problem has been resolved – If the maximum number of extents are reached the data set cannot be expanded further. However, it could be marked unavailable then copied to a larger data set, which is subsequently renamed to the original name 25
• 15.
  26 Selecting whichmessages to offload • Messages too large for CF entry (> 63KB) are always offloaded • Other messages may be selectively offloaded using offload rules – On each CFSTRUCT definition, you can specify 3 off-load rules – Each rule specifies: • Message size in KB – OFFLDnSZ(size) • Structure usage threshold – OFFLDnTH(percentage) where n = 1, 2, 3. • Data for new messages exceeding the specified size is offloaded (as for a large message) when the structure usage exceeds the specified threshold • Default rules are provided which should be useful in most cases • Rules can be set to dummy values if not required Offloading CF structure size Message size Number of messages No 100GB 63KB 1.25M Yes 100GB Any 140M
• 16.
  0% 70% 80%90% ~ 140,000 offloaded messages 28 Storage benefits of offloading • 1GB structure • 20KB messages being put ~ 35,000 messages in CF Thresholds for % of CF structure full ~ 5,000 messages ~ 140,000 offloaded messages > 63KB > 32KB > 4KB > 0KB Message sizes (default rules) • ~ 320,000 messages using offloading versus ~ 50,000 without offloading
• 17.
  SCM Storage (Flash)– Planned Emergency Storage • z/OS 2.1 and zEC12 GA2 introduce Coupling Facility Flash storage – Additional capacity for maintenance windows or consumer outages • Consider CFSTRUCT OFFLOAD rules: – OFFLD1TH(70), OFFLD1SZ(32K) • Offload messages larger than 32KB if the structure is 70% full – OFFLD2TH(80), OFFLD2SZ(4K) • Offload messages larger than 4KB if the structure is 80% full – OFFLD3TH(90), OFFLD3SZ(0K) • Offload all message data if the structure is 90% full • Progressively smaller messages written to SMDS as the structure fills • At 90% full the queue manager stores the minimum data per message to squeeze as many message references as possible into the remaining storage • CF Flash algorithm also starts moving the middle of the queue out to flash storage, keeping the faster 'real' storage for messages most likely to be got next 30
• 18.
  SCM Storage (Flash)– Maximum speed • Quicker to access messages stored in the CF than in SMDS – Keep high performance messages in the CF •CFSTRUCT OFFLOAD rules disabled using OFFLD1SZ(64K) – Maybe one rule to reduce large message threshold below 63KB • At 90% full the CF Flash algorithm starts moving the middle of the queue out to flash storage, keeping the faster 'real' storage for messages most likely to be got next. • As messages are got and deleted the CF Flash algorithm attempts to pre-fetch the next messages from Flash storage they are rapidly available for MQGET. • In this scenario the Flash storage acts like an extension to 'real' CF structure storage. However it is consumed more rapidly in this scenario since all small message data is stored in it. 32
• 19.
  34 SCM Storage(Flash) – Comparison Scenario Message Size Total Messages # Messages in 4GB 'real‘ SMDS Space # Messages in 200GB Flash Augmented (limit 30GB) No SMDS No Flash 1KB 3M 3M 4KB 900,000 900,000 16KB 250,000 250,000 SMDS No Flash 1KB 3.2M 3.2M 800MB 4KB 1.8M 1.8M 5GB 16KB 1.3M 1.3M 20GB Emergency Scenario 1KB 190M 2M 270GB 190M 30GB 4KB 190M 600,000 850GB 190M 30GB 16KB 190M 150,000 3TB 190M 30GB Speed Scenario 1KB 150M 2M 150M 26GB 4KB 48M 600,000 48M 8GB 16KB 12M 150,000 12M 2GB
• 20.
  Queue manager Private queues 36 Failure and persistence Coupling Facility failure Shared queues Queue manager failure Queue manager Private queues Queue manager Private queues Nonpersistent messages on private queues OK (kept) Messages on shared queues OK (kept) Persistent messages on shared queues restored from log Queue manager Private queues Private queues Queue manager Private queues Queue manager Private queues Shared queues Nonpersistent messages on private queues lost (deleted) Nonpersistent messages on shared queues lost (deleted)
• 21.
  38 Administration structurerecovery • In a disaster recovery situation (e.g. loss of CF) – Prior to version 7.0.1 • Each queue manager rebuilds its own admin structure entries • Therefore, you need to start all queue managers to rebuild admin structure • Once recovered, application structures can be recovered – From version 7.0.1 • Active queue managers notice if other queue managers don’t have entries in admin structure and initiate rebuild of admin structure entries on their behalf • Application structures can be recovered before all queue managers are restarted • A version 7.0.1 or higher, queue managers can recover entries on behalf of any queue manager at the same or lower version of MQ – Not all queue managers in the QSG need to be at V7.0.1, or higher
• 22.
  40 CF lossof connectivity CF QM2 QM1 QM3 Pre-V7.1 Queue Managers A failure of the Coupling Facility is most likely going to be presented to connectors as a Loss of Connectivity Prior to V7.1 if a queue manager receives a loss of connectivity it will terminate. If a Coupling Facility failure occurs this results in an outage for the entire QSG, unless protected by CF Duplexing
• 23.
  41 CF lossof connectivity tolerance V7.1 or later queue CF2 managers will not terminate. They will automatically attempt to re-establish access to the affected structures. CF1 V7.1 QM2 QM1 QM3 V7.1+ Queue Managers If a total loss of connectivity occurs the queue managers can automatically recover (RECOVER CFSTRUCT) the structures that were on the failed CF in to an alternative CF if one is available).
• 24.
  43 CF lossof connectivity tolerance CCFF12 V7.1 QM2 QM1 QM3 V7.1+ Queue Managers If a partial loss of connectivity occurs a System Managed Rebuild is automatically initiated by the queue managers to rebuild the structures in to a more available CF. This means both persistent and non-persistent messages are retained.
• 25.
  CF admin structureloss of connectivity tolerance • Queue managers will tolerate loss of connectivity to the admin structure without terminating if – The QMGR CFCONLOS attribute is set to TOLERATE – All the queue managers in the QSG are at version 7.1 or later • All queue managers in the QSG will disconnect from the admin structure, then attempt to reconnect and rebuild their own admin structure data • If a queue manager cannot reconnect to the admin structure, for example because there is no CF available with better connectivity, some shared queue operations remain unavailable until the queue manager can successfully reconnect to the admin structure and rebuild its admin structure data • The queue manager will automatically reconnect to the admin structure when a suitable CF becomes available on the system • Failure to connect to the admin structure during queue manager start-up is not tolerated, regardless of the value of CFCONLOS 45
• 26.
  47 CF structuredefinition considerations MQ CFSTRUCT definition CFRM policy structure definition CFSTRUCT(TEST1) STRUCTURE NAME(SQ27TEST1) CFLEVEL(5) SIZE(50000) CFCONLOS(TOLERATE) INITSIZE(20000) RECAUTO(YES) DUPLEX(DISABLED) OFFLOAD(SMDS) ALLOWAUTOALT(YES) PREFLIST(P5CF01,P5CF02) • If using CFCONLOS(TOLERATE) you should also consider: – RECAUTO – automatic recovery of application structures – PREFLIST – to allow the structure to be allocated in multiple Coupling Facilities • ALLOWAUTOALT(YES) – Enables CF to adjust entry/element ratio, and also automatically resize structure up to a maximum SIZE value – MINSIZE can be set to automatically adjust the structure size down • CF duplexing can be used to duplex MQ structures – Makes most types of failures transparent to MQ
• 27.
  For Additional Information IBM Training http://www.ibm.com/training IBM WebSphere http://www.ibm.com/software/websphere/ http://www.ibm.com/software/products/ibm-mq IBM developerWorks http://www.ibm.com/developerworks/websphere https://www.ibm.com/developerworks/community/blogs/messaging
• 28.
  IBM MQ Sessionsthis week 10:30 - 12:00 13:15 - 14:15 14:30 - 15:30 16:00 - 17:00 17:15 - 18:15 Tuesday Opening General Session- IBM Digital Experience and WebSphere Technical University Session A31: IBM MQ CHLAUTH rules – with MQ V8 updates Speaker: Morag Hughson Room 02 Session A4: WebSphere MQ for z/OS: Performance and Accounting Speaker: Alex Ross Room 8 Session I26: DataPower-MQ Connectivity Deep Dive (Theory) Speaker: Robin Wiley Room 27 Session Z1: WebSphere MQ for z/OS V8: Latest Features Deep Dive Speaker: Damon Cross Room 6 9:00 - 10:00 10:30 - 11:30 11:45 - 12:45 14:00 - 15:00 15:15 - 16:15 16:45 - 17:45 Wednesday Session Z5: WebSphere MQ for z/OS: Security Speaker: Damon Cross Room 02 Session A21: What's New in IBM Messaging Speaker: Morag Hughson Room 8 Session C7: Messaging in the Cloud with IBM MQ Light and IBM Bluemix Speaker: Rob Nicholson Room 27 Session A17: Managing work-loads, scaling and availability with IBM MQ clusters Speaker: David Ware Room 6 Lab IL5: DataPower-MQ Connectivity Deep Dive (Hands-On) Speaker: Robin Wiley Room 7b Session A9: WebSphere MQ for z/OS: The Inside Story Speaker: Damon Cross Room 6 Thursday Session A35: How to Develop Responsive Applications with IBM MQ Light Speaker: Rob Nicholson Room 27 Session A22: New IBM MQ V8 Security Features Speaker: Morag Hughson Room 01 Session A3: WebSphere MQ for z/OS: Shared Queues Speaker: Alex Ross Room 6 Session A18: Using Publish /Subscribe with IBM MQ Speaker: David Ware Room 27 Friday Lab AL6: Developing a First Application with IBM WebSphere MQ Light Speakers: Robert Nicholson, Alex Ross Room 7b Session A16: Using IBM MQ Pub/Sub in an MQ network Speaker: David Ware Room 6
• 29.
  Questions? © 2014IBM Corporation 50