Get more than a cache back! The Microsoft Azure Redis Cache (NDC Oslo)
Download as PPTX, PDF3 likes2,788 views
The document discusses Azure Cache and Redis. It provides an overview of Redis, including its data types, transactions, pub/sub capabilities, scripting, and sharding/partitioning. It then discusses common patterns for using Redis, such as caching, counting likes on Facebook, getting the latest reviews, rate limiting, and autocompletion. The document emphasizes that Redis is very flexible and can be used for more than just caching, acting as a general datastore. It concludes by recommending a Redis reference book for further learning.
![Data types
Type Example Key Example value
String cache:/home <html><head><title>Home page</title>
Hash categories:1 Field
name
description
numproducts
Member
Books
Books for sale
50000
Set categories:1:products 20 11 56 89 32 4
List products:20:comments [0] -> “...”
[1] -> “...”
[2] -> “...”
Sorted set products:bestsellers Field
Eye Patch
Parrot
Score
84632
82120
+ Bitmap, Hyperloglog](https://image.slidesharecdn.com/getmorethanacachebackthemicrosoftazurerediscache-ndcoslo-150622113614-lva1-app6892/85/Get-more-than-a-cache-back-The-Microsoft-Azure-Redis-Cache-NDC-Oslo-13-320.jpg)
![Data types
Type Example Key Example value
String cache:/home
user:nextid
<html><head><title>Home page</title>
2
Hash user:1 Field
name
handle
Member
Maarten
@maartenballiauw
Set user:1:followers 10 42 99 85
List user:1:timeline [0] -> { ... }
[1] -> { ... }
[2] -> { ... }
Sorted set trending:no:tags Field
#ndcoslo
#redis
Score
84632
82120
+ Bitmap, Hyperloglog](https://image.slidesharecdn.com/getmorethanacachebackthemicrosoftazurerediscache-ndcoslo-150622113614-lva1-app6892/85/Get-more-than-a-cache-back-The-Microsoft-Azure-Redis-Cache-NDC-Oslo-15-320.jpg)
Get more than a cache back! The Microsoft Azure Redis Cache (NDC Oslo)
- 1. Get more than a cache back! The Microsoft Azure (Redis) Cache Maarten Balliauw @maartenballiauw
- 2. Who am I? Maarten Balliauw Antwerp, Belgium Software Engineer, Microsoft Founder, MyGet AZUG Focus on web ASP.NET MVC, Azure, SignalR, ... Former MVP Azure & ASPInsider Big passion: Azure http://blog.maartenballiauw.be @maartenballiauw Shameless self promotion: Pro NuGet - http://amzn.to/pronuget2
- 3. Agenda Azure Cache Redis Data types Transactions Pub/sub Scripting Sharding/partitioning Patterns
- 4. Azure Cache
- 5. Caching on Azure Windows Server: AppFabric (“Velocity”) (2009-2010) Azure (ASP.NET cache) “Shared Cache” (AppFabric cache, multi-tenant) Self-Hosted Dedicated Cache Managed Cache (the above, but managed by Microsoft and with an SLA) A bit of history...
- 6. Caching on Azure 1920: Cache used to be a cache 2012: Cache became the go-to datastore Twitter stores 800 tweets / timeline in Redis New application types on Azure: PHP, Node, Java, ... Too much work to write AppFabric clients for all! http://www.redis.io/clients Way easier: just make it work on Windows and Azure New ways to develop applications
- 7. Redis
- 8. Redis “Redis is an open source, BSD licensed, networked, single-threaded, in-memory key- value cache and store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets, sorted sets, bitmaps and hyperloglogs.” REmote DIctionary Server
- 9. Redis Key-value cache and store (value can be a couple of things) In-memory (no persistence, but you can) Single-threaded (atomic operations & transactions) Networked (it’s a server and it does master/slave) Some other stuff (scripting, pub/sub, Sentinel, snapshot, …) Things to remember
- 10. So no persistence? Possible using a lot of I/O AOF (append-only files) RDB (Redis DB snapshots) With or without: all your data must fit in memory
- 11. Redis 101 demo
- 12. The Little Redis Book http://openmymind.net/redis.pdf By Karl Seguin
- 13. Data types Type Example Key Example value String cache:/home <html><head><title>Home page</title> Hash categories:1 Field name description numproducts Member Books Books for sale 50000 Set categories:1:products 20 11 56 89 32 4 List products:20:comments [0] -> “...” [1] -> “...” [2] -> “...” Sorted set products:bestsellers Field Eye Patch Parrot Score 84632 82120 + Bitmap, Hyperloglog
- 14. Data types demo
- 15. Data types Type Example Key Example value String cache:/home user:nextid <html><head><title>Home page</title> 2 Hash user:1 Field name handle Member Maarten @maartenballiauw Set user:1:followers 10 42 99 85 List user:1:timeline [0] -> { ... } [1] -> { ... } [2] -> { ... } Sorted set trending:no:tags Field #ndcoslo #redis Score 84632 82120 + Bitmap, Hyperloglog
- 16. Keys Good practice: use a pattern for keys E.g. users:1:maarten Get by id: KEYS users:1:* GET ...... Get by name: KEYS users:*:maarten GET ...... O(N) operation – use with caution!
- 17. But... .NET? Lots of options! http://www.redis.io/clients NuGet: ServiceStack.Redis StackExchange.Redis RedisAspNetProviders ...
- 18. Connecting to Redis from .NET demo
- 19. But... Azure? “It’s just Redis!” Main differences: auth mechanism and SSL support Use local for development from GitHub or NuGet (Redis-32 or Redis-64)
- 21. Azure Redis Cache Redis + Auth + SSL Monitoring Alerts Awesome tool: www.redsmin.com
- 22. Transactions MULTI, EXEC, DISCARD, WATCH No rollbacks (just discard queue) Failures are because of you, not Redis Optimistic locking with WATCH Only execute transaction queue if watched keys did not change
- 24. Pub/Sub (P)SUBSCRIBE, UNSUBSCRIBE, PUBLISH Subscribe to a queue SUBSCRIBE news (or PSUBSCRIBE news:*) Send data to a queue PUBLISH news “This just in!” (optional) Keyspace notifications http://www.redis.io/topics/notifications CONFIG SET notify-keyspace-events KEA PSUBSCRIBE '__key*__:*' (or __keyspace@0__:foo)
- 25. Pub/Sub demo
- 26. Scripting Run Lua scripts on Redis http://www.redis.io/commands/eval EVAL ‘return “Hello, World!”’ 0 Scripts are cached (SCRIPT FLUSH) Scripts can be used as functions Although we must pass the body all the time (or use SCRIPT LOAD + EVALSHA) Helper functions available! Base, table, string, math, debug, struct, cjson, cmsgpack, redis.sha1hex
- 27. Scripting demo
- 28. What if I need more memory? Sharding! On client (consistent hashing) On server (Redis Cluster, not on Azure) Using a proxy (Twemproxy - https://github.com/twitter/twemproxy) var options = new ConfigurationOptions { EndPoints = { "my-server" }, Proxy = Proxy.Twemproxy };
- 29. Patterns
- 30. When can I use Redis? ASP.NET Output Caching (RedisAspNetProviders) ASP.NET Session State General-purpose cache Table Storage secondary index “Cache with benefits” Pub/sub Generally: when use case maps and data fits in RAM
- 31. When should I avoid Redis? More data than can fit in RAM Can use multiple, but even then: SUM(RAM) == max. data size Data and query model fits well in a relational model Materialize certain queries in Redis if needed Avoid Redis for large objects... ...with lots of concurrent read/write operations
- 32. Counting stuff How would you count “likes” if you were Facebook? Table Storage 1 entity per likeable entity + per server instance for writes, sum entities for read I/O and CPU time... SQL UPDATE ... Locking... Redis INCR post:12345 (Every once in a while, check keys to persist)
- 33. Counting stuff (2) How would you count “likes” if you were Facebook? And how would you get the 5 most popular posts? Use a sorted set Elements are scored (= # of likes) We can use ZREVRANGE to get the top X posts ZREVRANGE posts:likes 0 5 withscores
- 34. Get the latest 5 product reviews No need to go to the database Use a Redis List, truncated at 5 items Need more? Query the database
- 35. Rate limiting API should only allows 5 requests per 60 seconds Redis List Record 5 latest requests If we’re at 5, check the leftmost item versus current time 1 2 3 4 5 12:20:25 12:20:22 12:20:21 12:20:18 12:19:50
- 36. What’sUp? Sending short messages between parties, how to implement? Pub/Sub: Every user gets a subscription, others can send to that subscription Server listens on all subscriptions to archive/persist messages When user comes online, server can publish messages since last visit from archive/a Redis list
- 37. Autocompletion How to provide autocompletion over our million- records product catalog? SQL “LIKE” Lucene.NET / ElasticSearch Redis Sorted Set
- 38. Intersecting collections “Which friends do we have in common?” Algorithm could be: Find friends that we have in common Find friends they have in common Score the # of connections
- 39. Sort We have a series of bugs Priority, Status, Title, ... A user follows a few of these bugs How to store? How to query? (give me my top 5 bugs by priority)
- 40. Conclusion
- 41. Conclusion Azure Cache Redis is not just a cache Data types Transactions Pub/sub Scripting Sharding/partitioning Patterns 1 thing to remember: http://openmymind.net/redis.pdf
Editor's Notes
- #13: Start server Show config file and some of the options in there Connect client, mention on localhost it autoconnects as I’m using default ports and such Run some commands: get name set name “Maarten” get name getrange name 2 4 expire name 5 get name Explain simple commands, yet powerful (e.g. substring) Check config with “info” (also cache misses/hits etc.)
- #16: Demo some of the data types. Here’s a bunch to choose from: String: get cache:/home set cache:/home "Home page" get cache:/home getrange cache:/home 2 5 expire cache:/home 5 get cache:/home incr site:visits incrby site:visits 6 Hash: hmset categories:1 name Books numproducts 4 hgetall categories:1 hincrby categories:1 numproducts 1 hget categories:1 name hset categories:1 numproducts 500 hgetall categories:1 Set: sadd categories:1:products 20 11 56 89 32 4 sadd categories:1:products 99 scard categories:1:products sadd categories:2:products 20 11 sinter categories:1:products categories:2:products sdiff categories:1:products categories:2:products List: rpush products:20:comments "Awesome product!" rpush products:20:comments "This sucks." rpush products:20:comments "Yarr!" lindex products:20:comments 1 llen products:20:comments brpop products:20:comments queue 0 #blocking at the end! pub/sub? Sorted Set: zadd products:bestsellers 100 "Eye Patch" 10 "Parrot" 1 "Ship" zrevrange products:bestsellers 0 1 zrevrank products:bestsellers "Ship" zscore products:bestsellers "Ship"
- #20: Create a quick console app Install StackExchange.Redis ConnectionMultiplexer redis = ConnectionMultiplexer.Connect("localhost:6379"); // explain: keep this one! var database = redis.GetDatabase(); // explain: can be disposed andall database.StringSet("test", "Just a quick test"); Console.WriteLine(database.StringGet("test")); Console.ReadLine();
- #22: Create a Redis cache through portal.azure.com Explain Standard vs. Basic (replica, but data loss still possible) Update previous sample ConnectionMultiplexer.Connect("maartenba.redis.cache.windows.net:6379,password=/tvkIbDBTyhu8vJueOt150iP3kzIXTb/u2dTB5PxJsM="); In the portal, show configuration of maxmemory In the portal, show alerts and notifications In the portal, show stats / hits / misses etc Show this connecting using the redis-cli, too (redis-cli –h host –a pass) And there is also... www.redsmin.com (login, explain some of the things we can see here)
- #25: Connect to localhost, perhaps flushall first SET where “Sweden” SET conference “Unknown” SET attendees 0 Connect a second client MULTI INCRBY attendees 100 SET conference “CloudBurst” EXEC Worked fine! Now let’s do this on client 1: WATCH attendees MULTI INCRBY attendees 100 And on client 2: SET attendees 99 And on client 1: EXEC .NET: var transaction = redis.GetDatabase().CreateTransaction(); transaction.AddCondition(Condition.KeyExists("foo")); transaction.StringSetAsync("foo", "bar"); bool committed = transaction.Execute();
- #27: Use two clients Client 1: SUBSCRIBE news Client 2: PUBLISH news “This just in” Let’s do that in .NET: ConnectionMultiplexer redis = ConnectionMultiplexer.Connect("localhost:6379"); ISubscriber subscriber = redis.GetSubscriber(); subscriber.Subscribe("news", (channel, message) => { Console.WriteLine("News: {0}", (string)message); }); ISubscriber publisher = redis.GetSubscriber(); publisher.Publish("news", "This just in!"); Console.ReadLine(); Send a message from one of the clients, too Keyspace notifications: CONFIG SET notify-keyspace-events KEA PSUBSCRIBE '__key*__:*'
- #29: Do some scripts in the sample project