Testing at scale with redis queues
Download as PPTX, PDF1 like62 views
This document discusses using Redis queues and data structures to manage test concurrency and store test results from automated testing. It describes implementing a queue using Redis lists to manage the ordering of tests as they move from pending to active status. Redis sets are proposed to avoid duplicate test sessions, and sorted sets can implement priority queues. Hashes are recommended for storing test metadata and results associated with each test session. The use of Redis JSON and RediSearch modules are presented for storing and querying more complex test data. Finally, opportunities for analyzing historical test data using machine learning techniques are mentioned.
More Related Content
Similar to Testing at scale with redis queues (20)
Testing at scale with redis queues
- 1. PRESENTED BY Resilient Testing at Scale using Redis Queues to Manage Concurrency Aaron Evans Sauce Labs, Solutions Architect
- 2. PRESENTED BY 1 Test Automation with Selenium & Sauce Labs Agenda 2 How to implement a queue to manage concurrency 3 Redis data types (STRING, LIST, SET, HASH, ZSET) 4 Notifications with PUBLISH / SUBSCRIBE 5 Using RedisJSON to store complex data (test results) 6 Analyzing historical data with RediSearch 7 Predicting test failures with machine learning
- 3. PRESENTED BY • My name is Aaron Evans • Accidental software tester • Burned out and went to Fiji to live on the beach • Came back to buy a sailboat but got married instead • Moved to Ecuador and started freelance consulting • Traveled around the USA with my family in an RV • Live in Montana on a farm in the woods & built a yurt About me
- 4. PRESENTED BY • Sauce Labs is a testing infrastructure company • Founded by creator of Selenium, Jason Huggins • Allows you to run test automation in the cloud on different OS, browsers & mobile devices • Using Selenium & Appium • Essentially a managed remote Selenium grid • Also provides a nice UI, dashboard, reports, manual testing via remote control, and test archives & analytics About Sauce Labs
- 5. PRESENTED BY • Many tests run in parallel • Need to manage test concurrency • Across multiple teams & users • Triggered by CI server or manually by developers • Flaky / Slow Tests • Analysis & Prediction The Problem
- 6. PRESENTED BY • Use Redis as a single source of truth • Keep track of how many tests are running • Create a "pool" of sessions • Move from pending, to active, to complete – Like a ledger • Check out a session at the start of each test • Check in when complete • Store test data in Redis for future analysis A Solution
- 7. PRESENTED BY • KEY/VALUE store – Memcached, Berkeley DB • Relational database – MySQL, PostgreSQL, Oracle • NoSQL database / document store – MongoDB, Cassandra • Shared File System – NFS, SMB • Web Service Alternatives to Redis
- 8. PRESENTED BY • It’s fast! • Simple API with available client for all major platforms • Nice middle ground between Key/Value and NoSQL • I didn’t want the hassle of setting up hosting or managing infrastructure • Redis Labs has a free service tier that is sufficient • I wanted to learn something new Why Redis
- 10. PRESENTED BY • Use a STRING to keep a count of concurrent sessions • Numeric values can be incremented/decremented • Subtract from counter when each session starts • Add to counter after each session finishes Keep track of available sessions with a counter
- 11. PRESENTED BY String operations for simple counter redis.test:6379>setavailable_sessions 100 OK redis.test:6379>GETavailable_sessions "100" redis.test:6379>DECRavailable_sessions (integer) 99 redis.test:6379>INCRavailable_sessions (integer) 100 redis.test:6379>DECRBYavailable_sessions50 (integer) 50 redis.test:6379>INCRBYavailable_sessions25 (integer) 75 redis.test:6379>GETSETavailable_sessions 150 "75"
- 12. PRESENTED BY • Add session_id to LIST when started • Remove session_id from LIST when complete • Get length of active_sessions to make sure you don’t exceed concurrency • Trim sessions to appropriate size for cleanup Create a LIST of active sessions
- 13. PRESENTED BY List operations for active sessions redis.test:6379>LPUSHactive_sessions session:123 (integer) 1 redis.test:6379>LPUSHactive_sessions session:124 (integer) 2 redis.test:6379>LPUSHactive_sessions session:125 (integer) 3 redis.test:6379>LREMactive_sessions 1 session:124 (integer) 1 redis.test:6379>LLENactive_sessions (integer) 2 redis.test:6379>LRANGEactive_sessions 0 -1 1) "session:125" 2) "session:123" redis.test:6379>LTRIMactive_sessions 0 99 OK
- 14. PRESENTED BY • Use a separate list for requested test sessions • First In / First Out (FIFO) queue • LPUSH when a test session is requested • RPOP when an active session becomes available • RPUSHLPOP to atomically push from pending to active Create a queue of pending tests
- 15. PRESENTED BY List operations for requested sessions queue redis.test:6379>LPUSHrequested_sessionsrequested:126 (integer) 1 redis.test:6379>LPUSHrequested_sessionsrequested:127 (integer) 2 redis.test:6379>RPOPrequested_sessions "requested:126" redis.test:6379>LPUSHactive_sessions session:126 (integer) 4 redis.test:6379>RPOPLPUSHrequested_sessions active_sessions "requested:127" redis.test:6379>LSETactive_sessions 0 session:127 OK redis.test:6379>LRANGEactive_sessions 0 1 1) "session:127" 2) "session:126"
- 16. PRESENTED BY Transaction for moving from requested to active session redis.test:6379>MULTI OK redis.test:6379>LREMactive_sessions 1 session:126 QUEUED redis.test:6379>BRPOPLPUSHrequested_sessionsactive_sessions 60 QUEUED redis.test:6379>LSETactive_sessions 1 session:128 QUEUED redis.test:6379>EXEC 1) (integer) 1 2) "requested:128" 3) OK redis.test:6379>LINDEXactive_sessions 0 "session:128"
- 17. PRESENTED BY • Notify client when sessions are available • PUBLISH sends to all clients listening to channel • SUBSCRIBE listens (may be blocking) • Single worker publishes • Avoids need for constant polling for available sessions Notifications with PUBLISH / SUBSCRIBE
- 18. PRESENTED BY PUBLISH / SUBSCRIBE operations # CLIENTLISTENS redis.test:6379>SUBSCRIBEsessions_available Readingmessages...(pressCtrl-Ctoquit) 1) "subscribe" 2) "sessions_available" 3) (integer) 1 1) "message" 2) "sessions_available" 3) "requested:130" # WORKERPUBLISHES redis.test:6379>PUBLISHsessions_availablerequested:131 (integer) 1
- 19. PRESENTED BY • Use a SET instead of a LIST • Keys must be unique • Sets aren’t ordered (no push / pop) * SPOP is random • Ok because active_sessions are not necessarily in sequence • Can move elements from one set to another – (e.g. active to completed, passed, failed, errored) • Allows grouping with SUNION, SINTER, SDIFF • Can SORT if needed (and STORE as a list) Avoid duplicate sessions with a SET
- 20. PRESENTED BY Set operations for managing sessions redis.test:6379>SADDactive_sessions session:123 (integer) 1 redis.test:6379>SADDactive_sessions session:123 (integer) 0 redis.test:6379>SCARDactive_sessions (integer) 1 redis.test:6379>SMOVEactive_sessions completed_sessionssession:123 (integer) 1 redis.test:6379>SADDerrored_sessionssession:124 (integer) 1 redis.test:6379>SUNIONcompleted_sessions errored_sessions 1) "session:124" 2) "session:123"
- 21. PRESENTED BY Move errored tests to retry queue redis.test:6379>SMEMBERSerrored_sessions 1) "session:126" 2) "session:124" 3) "session:125” redis.test:6379>SORTerrored_sessions ALPHASTOREretry_queue (integer) 3 redis.test:6379>LRANGEretry_queue0 -1 1) "session:124" 2) "session:125" 3) "session:126”
- 22. PRESENTED BY • Use a ZSET (Sorted Set) • Combines the best features of both LIST and SET – *but it comes at a cost • Unique keys • SCORE allows you to rank tests to specify priority • POP elements by highest or lowest score – ZPOPMAX / ZPOPMIN Implement a priority queue with a sorted test
- 23. PRESENTED BY ZSET commands for priority queue redis.test:6379>ZADDpriority_queue1 session:1 (integer) 1 redis.test:6379>ZADDpriority_queue2 session:2 (integer) 1 redis.test:6379>ZINCRBYpriority_queue2 session:1 "3" redis.test:6379>ZRANGEpriority_queue0 -1WITHSCORES 1) "session:2" 2) "2" 3) "session:1" 4) "3" redis.test:6379>ZPOPMAX priority_queue 1) "session:1" 2) "3"
- 24. PRESENTED BY • Store multiple Name/Value pairs • Use session ID as HASH key • Store test metadata – Execution environment capabilities (e.g. OS, browser, version) – Test name, build id, tags – Test status (Passed/Failed, Completed/Errored) Capture test data with a HASH
- 25. PRESENTED BY Hash operations for storing test data redis.test:6379>HSETsession:123name"login test"platform"Windows10"browser"IE"version "11"tag "regression" (integer) 5 redis.test:6379>HEXISTSsession:123status (integer) 0 redis.test:6379>HSETsession:123statuspassed (integer) 0 redis.test:6379>HGETsession:123 status "passed" redis.test:6379>HGETALLsession:123 1) "name" 2) "login test" … 11) "status" 12) "passed"
- 26. PRESENTED BY • Hashes only store 1 dimension • Can fake it with multipart keys – foo.bar.baz=quux • Use a common key between records for relationships – testdata:{123} – testresult:{123} • Store as a JSON blob in a STRING • Use Redis JSON • Use Redis Search Storing complex (hierarchical) test data
- 27. PRESENTED BY • Additional Module • Can be compiled or used with RedisLabs • Allows for storing and querying JSON data Using RedisJSON
- 28. PRESENTED BY RedisJSON operations redis.test:6379>JSON.SETsession:123. '{"name":"logintest","status":"failed"}' OK redis.test:6379>JSON.GETsession:123status ""failed""
- 29. PRESENTED BY • Additional Module • Can be compiled or use with RedisLabs • Allows for full text search • Numeric or Tags • Add unstructured data – Test steps – Log files – Error messages & stack traces Using RediSearch
- 30. PRESENTED BY RediSearch for test results redis.test:6379>FT.CREATEtestresultsSCHEMA nameTEXTplatformTEXTbrowserTEXTversion TEXT exec_timeNUMERICstatusTAG OK redis.test:6379>FT.ADDtestresultstest:1231 FIELDSname"LoginTest"browser"Chrome"exec_time 123.456statusPASSED OK redis.test:6379>FT.SEARCHtestresultslogin 1) (integer) 1 2) "test:123" 3) 1) name 2) "LoginTest" 3) browser 4) "Chrome" 5) exec_time 6) "123.456"
- 31. PRESENTED BY RediSearch for test results by status tag redis.test:6379>FT.SEARCHtestresults"@status:{PASSED|COMPLETE}" 1) (integer) 1 2) "test:123" 3) 1) name 2) "LoginTest" 3) browser 4) "Chrome" 5) exec_time 6) "123.456" 7) status 8) "PASSED"
- 32. PRESENTED BY • Finding patterns and anticipating failures • Group By – test failures – platform / browser / device – feature or tags – release / over time series – other data points that you can't predict • Using Machine Learning Looking forwards
- 33. Thank you!
- 34. PRESENTED BY