Grafana and MySQL - Benefits and Challenges
Download as PPTX, PDF6 likes17,363 views
The document presents a case study on integrating Grafana with MySQL for tracking and visualizing metrics in Ingram Content Group's automated print-on-demand division. It discusses the challenges faced while using InfluxDB, such as incompatibility with SQL and performance issues, which led to the decision to utilize MySQL instead. The benefits of this integration include leveraging existing knowledge of MySQL, handling large data volumes efficiently, and using full SQL capabilities for queries.
More Related Content
![[Pgday.Seoul 2021] 2. Porting Oracle UDF and Optimization](https://cdn.slidesharecdn.com/ss_thumbnails/oracleudfmigration20211203-211227052428-thumbnail.jpg?width=560&fit=bounds)
Similar to Grafana and MySQL - Benefits and Challenges (20)
![[4DEV][ĹĂłdĹş] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...](https://cdn.slidesharecdn.com/ss_thumbnails/influxdb-171113135140-thumbnail.jpg?width=560&fit=bounds)
Grafana and MySQL - Benefits and Challenges
- 1. Grafana and MySQL Benefits and Challenges Aemelia Presentation Template from http://www.slidescarnival.com
- 2. About me Philip Wernersbach Software Engineer Ingram Content Group https://github.com/philip-wernersbach https://www.linkedin.com/in/pwernersbach 2
- 3. ⢠I work in Ingram Content Groupâs Automated Print On Demand division ⢠We have an automated process in which publishers (independent or corporate) request books via a website, and we automatically print, bind, and ship those books to them ⢠This process involves lots of hardware devices and software components 3
- 4. We collect: ⢠Device Status (Online/Offline) ⢠Tracking data for every piece of material for every book ⢠Throughput metrics ⢠Etc. This produces gigabytes of data an hour. 4
- 6. The Problem âHow do we aggregate and track metrics from our hardware and software sources, and display those data points in a graph format to the end user?â ď Grafana! 6
- 7. Which data store should we use with Grafana? ⸠Out of the box, Grafana supports Elasticsearch, Graphite, InfluxDB, KairosDB, OpenTSDB 7
- 8. Which data store should we use with Grafana? ⸠We compared the options and tried InfluxDB ⸠There were several sticking points with InfluxDB, both technical and organizational, that caused us to rule it out 8
- 9. Which data store should we use with Grafana? ⸠We already have a MySQL cluster deployed, System Administrators and Operations know how to manage it ⸠Decided to go with MySQL as a data store for Grafana 9
- 10. The Solution: Ingram Contentâs Grafana-MySQL Integration 10
- 11. The Integration â¸Written in Nim â¸Emulates an InfluxDB server â¸Connects to an existing MySQL server â¸Protocol compatible with InfluxDB 0.9.3 â¸Acts as a proxy that converts the InfluxDB protocol to the MySQL 11
- 14. InfluxDB ⸠Time Series database ⸠âtimeâ is a special value, has special meaning ⸠Applies special logic to âtimeâ SMART! The Challenges! Database Engine Comparison MySQL ⸠Generic relational database ⸠âtimeâ is a generic data value ⸠Doesnât apply special logic DUMB! 14
- 15. The Challenges! Query Comparison InfluxDB âSELECT count(bar1) FROM foo WHERE bar1 > 0 AND bar2 > 0 AND time > now() - 7d GROUP BY time(1h)â 15
- 16. The Challenges! Query Comparison MySQL âSELECT time, count(bar1) FROM foo WHERE bar1 > 0 AND bar2 > 0 AND time > NOW(6) - INTERVAL 7 DAY GROUP BY YEAR(time), MONTH(time), DAY(time), HOUR(time) ORDER BY time ASCâ 16
- 17. The Challenges! InfluxQL != SQL InfluxQL != SQL ⸠InfluxQL is SQL-like, but different enough that it canât be passed through to MySQL 17
- 18. The Challenges! InfluxQL != SQL ⸠âtimeâ is SELECTâd automatically implicitly in InfluxDB âš âSELECT bar FROM fooâ ď âSELECT time, bar FROM FOOâ ⸠GROUPâing on âtimeâ is smart in InfluxDB, and dumb in MySQL âš See slides 16 and 17 ⸠âtimeâ in epoch format with millisecond precision is a float in MySQL âš âFROM_UNIXTIME(1444667802.145)â 18
- 19. The Challenges! InfluxQL != SQL ⸠InfluxDB series are roughly equivalent to MySQL tables, but the management syntax is sometimes different âš âDROP SERIES FROM fooâ ď âDELETE FROM fooâ ⸠InfluxDB allows quoted identifiers, MySQL is more restrictive âš âSELECT * FROM foo WHERE bar > â0ââ ď Silently returns garbage in MySQL âš Grafana quotes identifiers aggressively, so we had to patch it 19
- 20. The Challenges! InfluxQL != SQL Thereâs probably more InfluxQL- SQL incompatibilities, but these are the ones that weâve run into with Grafana at Ingram Content Group. 20
- 21. The Benefits! 21
- 22. The Benefits! ⸠Allows us to leverage System Administratorâs and Operationâs existing knowledge of MySQL ⸠Allows us to process large amounts of data âš InfluxDB had issues when we tried to insert about a million points at a time, MySQL handles this with no problem 22
- 23. The Benefits! ⸠We can use full, regular SQL for our Grafana queries in addition to InfluxQL âš Think JOINs, etc. ⸠Weâre doing massive batch inserts, and MySQLâs query cache makes Grafana queries significantly faster âš This is obviously use case specific, but shows how you can tune MySQL to your specific use case 23