Modern SSO Using the MEAN Stack

Editor's Notes

• #2: Hi everybody my name is Dan Harabagiu and I am Head of Platform development within ProSiebenSat.1 Digital and Adjacent Today I will have the pleasure in presenting you one of our success projects and this is as the title mentions it, Modern SSO using MEAN stack.
• #3: What will I talk about today. Based on the agenda in front of you, I will begin with a short introduction of the company the team and the project. Will continue with describing our technology stack and architecture. After this will present our test scope and results that we acchieved and will close the presentation with conclusions that we had based on this project
• #6: So who we are? We are a central technology team also known as Central Technology Initiatives under the holding Chief Information Officer We are a team of very skilled persons, with various technology backgrounds that wish, like and do work with modern and scalable technologies (One of wich of course is MongoDB) So What would be our mission you might ask. We were formed with the purpose of providing central technology solutions for all or any Busineess Unit under the holding umbrela. From this solutions one will be presented today and to be more exact it is a Single Sign On solution that is consumed by most of the web assets of ProSiebenSat.1
• #8: SSO – is a web service that provides the user / consumer the possibility to easily login and be recognised on multiple web platforms and digital assets. Based on the intention of the project to unify the users of all assets and provide seamless integration across multiple domains and digital assets, we need to ensure that the service can be integrated on mutiple technology platforms as web, ott and mobile devices On top of the already dificult requirments that I‘ve told you until now, we were asked to be able to serve at least 300.000 logins or registrations per minute. After analysing our requirments we came to the conclusion that we will need a service that is based on Node.JS in order to be able to handle a higher load at a lower costs We have search for existing solutions on the market or open source. Unfotrunatly we hadn‘t find any to fulfill our requirments so we decided to buil it internally.
• #9: So what technology we used?
• #10: MongoDB provides a replicated and sharded structure, this in support of a system that will have a high load expected on a day to day activity Express is a framework that is on top of Node.js and it provides the developer a fast and easy way to handle web requests, views and in general web development Angular is a frontend framework that enables developers to create structured and modern web applications. We have used it to build our login / register forms, customer portals and self care. Node.JS as hopefully most of you heard of it, is a Javascript based backend technology with one of the main features being its asyncronus nature. So by this providing a high load handling capabilities compered to other „blocking“ backend technologies / languages
• #11: So how would our landscape look like? We based our deploy on AWS infrastructure The entry point will be the Amazon ELB, which could easily scale to handle high load and supports failover The Node.JS application servers are connected to ELB and distributed accross multiple availability zones Each time a Node.JS node is started or stoped is automatically added to the ELB balancing pool. Having this in place, provided us an easy and automatic scalability and failover handling The type of instances used for nodejs are CPU intensive with stateless disks The mongoDB setup is one of the standards approach with 3 Mongos, 3 Config servers, 3 shards and each shard with 2 replicas and one arbiter (to handle voting balancing in case of failover). Also, all the mongodb servers, are distributed accross multiple availability zones for failover handling (Each replica set of the same shard is within a different AZ) The type of instances used for mongodb are memory intensive with dedicated IOPS
• #13: So once the design and development was done, how would we test it?
• #14: So we have built a small DDOS mechanism to be able to generate as much as load is needed from multiple locations The script was a simple behaviour driven headless browser using Node.JS. We automated most of the steps of testing cycle like: Included the script into an AMI Created a starup script to run multiple instances of our fake users once the machine has finished booting Using an internal orchestration tool we were able to spawn a fine controled amount of instances in multiple AWS regions and availability zones. We needed all the automation an granularity in order to be able to measure gradually the evolution of the system load and response time The results we measured were provided from two perspectives Internal: the time that the application needed to respond to a request External: the time that the user took to finish his action (register / login)
• #15: So how did we tested? We prepared our infrastructure based on the layout that I described earlier with the amount of 16 Backend Nodes. We started slowly to enable load generators machine in incremental order over of a duration of 10 – 15 minutes and mainly in order to allow ELB to scale internally. After our scale up period we continued in starting load balancers until we reached either 90% cpu on nodejs or mongodb or IO load would reach 100%. In this case we stoped when the table lock and waiting of mongo started to increase the response time. So the results we had at this point were 150.000 requests / minute with an average response time from user perspecitve of 220 ms. Also would like to mention that these results are based on a sustainable runing environment, so was not peak times or peak values.
• #16: After further investigations we have realised that the bottleneck we had in our data communications was a rather simple scatter gather that was eating most of the processing time. This was based on a collection that was having multiple unique indexes that were not part of the sharding key. So the data insertion worked pretty well, until we needed to read the data. So the solution in this case was not that simple, as we realised that we would need to change our data model. Thanks to MongoDB flexibility that was done by extractng the respective unique indexes in a dedicated collection. This increased the read spead and also increased the distribution of the data on shards, now that the data model was smaller and granular.
• #17: So we started with the same server deployment as within the first test set and reached our previous numbers, but we wanted more. We have hit the limit on CPU and IO on data nodes, so went and increased the sharding, the disck IOPS. After this point Node.JS became a bottle neck, so we doubled the size of your infrastructure on the application nodes. We have stoped to investigate when we reached the point of adding more resources / servers will not help anymore. So we got the following results.
• #18: - Sharding is a very good mechanism to support distribution of read and writes operations, but under initial heavy data input, unless the mongo knows where to send the data segments, everything will hit one one. So a good solution for this topic is prespliting the collections before expecting heavy traffic - Closily monitoring the connections count and load on MongoS showed us that the mongodb driver we used within our nodeJS project was not distributing evenly the connections. So we decieded to tackle on our own this issue in randomising the order of mongoS server entries within the mongodb uri
• #19: As previously we started with our latest deployment and reached our previous values, but we still did not reached our target as the CPU of data nodes increased to almost 90%. So we increased the MongoDB clusted deployment (to total of 8 shards) and restarted the testing process. We finaly reached our target number (above 500.000) requests per minute and also hit again the CPU level on Node.JS and MongoDB data nodes. As we have reached our internal goal we decided to conclude the tests here and consider this numbers of current running limits of our platform. But if you wander yourself if the system could take more, we presume yes, but this might be a story for other times 
• #20: So what have we concluded during this project?
• #21: We reached out target, so pragmatically we could register or login the whole population of Germany in less then 3 hours. You should know very well your data model that you are planing to use, so you would know where to look for possible bottlenecks. Creating your data model in a way that could be extended or restructured could be a very good advantage. MongoDB provides this flexibility with ist own schemaless approach, but also the design should be flexibil. Please don‘t reflect SQL based logic and structures into a noSQL platform. It is always good to stress test your projects so you will always know your limits and have no unpleasent surprises in the future. Node.Js with its asyncronus design together with the data distribution (sharding) supported by MongoDB is a very good foundation for delivering good results, acceptable response time unde heavy traffic
• #23: With these last thoughts I would like to tank you for listening to our story and I hope you have also learned something new from us. Thank you very much and enjoy this day.