Refactoring a Solr based API application
3 likes397 views
This document summarizes architectural lessons learned from refactoring a Solr-based API application at Shopping24. Key strategies discussed include sharding the index to improve performance as the data size grew from 500k to 7m documents, using caching and separate Solr cores to optimize access for different clients, and automating infrastructure to dynamically scale hardware resources. The document stresses examining business requirements to simplify queries and indexes by removing unsearchable data and duplicating frequently accessed content.
Refactoring a Solr based API application
- 1. Architectural lessons learnedfrom refactoring a Solr based API application. Torsten Bøgh Köster (Shopping24) Apache Lucene Eurocon, 19.10.2011
- 2. Contents Shopping24 and it‘sAPI Technical scaling solutions Sharding Caching Solr Cores „Elastic“ infrastructure business requirements as key factor
- 3. @tboeghk Software- and systems-architect 2 years experience with Solr 3 years experience with Lucene Team of 7 Java developers currently at Shopping24
- 4.
- 5. 1 portal became n portals
- 6. 30 partner shopsbecame 700
- 7. 500k to 7m documents
- 8. index fact time •16 Gig Data •Single-Core-Layout •Up to 17s response time •Machine size limited •Stalled at solr version 1.4 •API designed for small tools
- 9.
- 10. ask the nerds „Shard!“ That‘ll be fun! „Use spare compute cores at Amazon?“ breathe load into the cloud „Reduce that index size“ „Get rid of those long running queries!“
- 11.
- 12. ... is highlyeffective. 500ms 1shard 2shard 3shard 4shard 6shard 8shard 375ms 250ms 125ms 1 4 8 12 16 20 concurrent requests
- 13. Sharding: size matters the bigger your index gets, the more complex your queries are, the more concurrent requests, the more sharding you need
- 14.
- 15. Why do wehave such a big index?
- 16. 7m documents vs. 2m active poducts
- 17. fashion product lifecycle meets SEO Bastografie / photocase.com
- 18. Separation of duties!Remove unsearchable data from your index.
- 19. Why do wehave complex queries?
- 20. A Solr index designedfor 1 portal
- 21. Grown into a multi-portal index
- 22. Let “sharding“ followyour data ...
- 23. ... and buildseparate cores for every client.
- 24. Duplicate data aslong as access is fast. andybahn / photocase.com
- 25. Streamline your index provisioning process.
- 26. A thousand splendidcores at your fingertips.
- 27. Throwing hardware at problems.Automated.
- 28.
- 29. mirror your completesystem – solve load balancer problems froodmat / photocase.com
- 30.
- 31. use a cachelayer like Varnish.
- 32. What about thosecomplex queries? Why do we have them? And how do we get rid of them?
- 33. Lost in encapsulation: SolrAPI exposed to world.
- 34. What‘s the keyfactor?
- 35. look at your businessrequirements
- 36.
- 37. Questions? Comments? Ideas? Twitter:@tboeghk Github: @tboeghk Email: [email protected] Web: http://www.s24.com Images: sxc.hu (unless noted otherwise)