Mastering D3.js

Table of Contents

Mastering D3.js

Credits

About the Author

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why subscribe?

Free access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Downloading the color images of this book

Errata

Piracy

Questions

1. Data Visualization

Defining data visualization

Some kinds of data visualizations

Infographics

Exploratory visualizations

Dashboards

Learning about data visualization

Introducing the D3 library

Summary

2. Reusable Charts

Creating reusable charts

Creating elements with D3

Binding data

Encapsulating the creation of elements

Creating the svg element

The barcode chart

Accessor methods

Chart initialization

Adding data

Adding the date accessor function

Updating the dataset

Fixing the enter and exit transitions

Using the barcode chart

Creating a layout algorithm

The radial layout

Computing the angles

Using the layout

Summary

3. Creating Visualizations without SVG

SVG support in the browser market

Visualizations without SVG

Loading and sorting the data

The force layout method

Setting the color and size

Creating a legend

Polyfilling

Feature detection

The canvg example

Using canvas and D3

Creating figures with canvas

Creating shapes

Integrating canvas and D3

Summary

4. Creating a Color Picker with D3

Creating a slider control

The drag behavior

Creating the slider

Using the slider

Creating a color picker

The color picker selector

Adding the color picker window

The color picker window

Summary

5. Creating User Interface Elements

Highlighting chart elements

Creating tooltips

Using the tooltip

Selecting a range with brushing

Creating the area chart

Adding brushing

The brush listener

Summary

6. Interaction between Charts

Learning the basics of Backbone

Events

Models

Collections

Views

Routers

The stock explorer application

Creating the stock charts

The stock title chart

The stock area chart

Preparing the application structure

The index page

Creating the models and collections

The stock model

The stock collection

The application model

Implementing the views

The title view

The stock selector view

The stock context view

The stock detail view

The application view

Defining the routes

Initializing the application

Summary

7. Creating a Charting Package

The development workflow

Writing the code

Creating a release

Semantic Versioning

Creating the package contents

The heat map chart

The matrix layout

The project setup

Installing the Node modules

Building with Grunt

Concatenating our source files

Minifying the library

Checking our code with JSHint

Testing our package

Writing a simple test

Testing the heat map chart

Testing the matrix layout

Running the tests with Grunt

Registering the sequences of tasks

Managing the frontend dependencies

Using the package in other projects

Summary

8. Data-driven Applications

Creating the application

The project setup

Generating a static site with Jekyll

Creating the application components

Creating the models and collections

Creating the views

The application setup

Hosting the visualization with GitHub Pages

Hosting the visualization in Amazon S3

Configuring Jekyll to deploy files to S3

Uploading the site to the S3 bucket

Summary

9. Creating a Dashboard

Defining a dashboard

Good practices in dashboard design

Making a dashboard

Defining the purpose of the dashboard

Obtaining the data

Organizing the information

Creating the dashboard sections

The students section

The courses section

The class section

Gathering the dashboard sections

Summary

10. Creating Maps

Obtaining geographic data

Understanding the GeoJSON and TopoJSON formats

Transforming and manipulating the files

Creating maps with D3

Creating a choropleth map

Mapping topology

Using Mapbox and D3

Creating a Mapbox project

Integrating Mapbox and D3

Summary

11. Creating Advanced Maps

Using cartographic projections

Using the Equirectangular projection

The Conic Equidistant projection

The Orthographic projection

Creating a rotating globe

Creating an interactive star map

Choosing our star catalog

Drawing the stars

Changing the projection and adding rotation

Adding colors and labels to the stars

Projecting raster images with D3

Rendering the raster image with canvas

Computing the geographic coordinates of each pixel

Reprojecting the image using the Orthographic projection

Summary

12. Creating a Real-time Application

Collaborating in real time with Firebase

Configuring Firebase

Integrating the application with Firebase

Creating a Twitter explorer application

Creating the streaming server

Using the Twitter-streaming API

Using Twit to access the Twitter-streaming API

Using Socket.IO

Implementing the streaming server

Creating the client application

The application structure

Models and collections

Implementing the topics views

The input view

The bar chart view

The topics map view

Creating the application view

The application setup

Summary

Index

Mastering D3.js

Mastering D3.js

Copyright © 2014 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: August 2014

Production reference: 1180814

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78328-627-0

www.packtpub.com

Cover image by Artie Ng (<[email protected]>)

Credits

Author

Pablo Navarro Castillo

Reviewers

Andrew Berls

Simon Heimler

Lars Kotthoff

Nathan Vander Wilt

Commissioning Editor

Edward Gordon

Acquisition Editors

Nikhil Chinnari

Mohammad Rizvi

Content Development Editor

Sankalp Pawar

Technical Editors

Indrajit A. Das

Humera Shaikh

Copy Editors

Dipti Kapadia

Deepa Nambiar

Stuti Srivastava

Project Coordinator

Harshal Ved

Proofreaders

Simran Bhogal

Maria Gould

Ameesha Green

Paul Hindle

Indexers

Hemangini Bari

Mariammal Chettiyar

Rekha Nair

Priya Subramani

Production Coordinator

Arvindkumar Gupta

Cover Work

Arvindkumar Gupta

About the Author

Pablo Navarro Castillo is a mathematical engineer and developer. He earned his Master's degree in Applied Mathematics from École des Mines de Saint-Etienne in France. After working for a few years in operations research and data analysis, he began to work as a data visualization consultant and developer.

He has collaborated with Packt Publishing as a technical reviewer for Data Visualization with D3.js and Data Visualization with D3.js Cookbook. In 2014, he founded Masega, which is a data visualization agency based in Santiago, Chile, where he currently works.

I wish to thank the Packt Publishing team for their collaboration in the inception and development of this book. I am also grateful to the technical reviewers, whose insightful comments and kind suggestions have been essential to improve the content and examples of every chapter.

To Miriam, for her patience and continuous support.

About the Reviewers

Andrew Berls is a Ruby and JavaScript developer who lives in Santa Barbara, CA. He has developed dashboards for www.causes.com using D3.js to visualize social networks and recently acted as a reviewer for Data Visualization with D3.js Cookbook, Packt Publishing. Andrew recently completed his degree in Computer Science at the University of California, Santa Barbara. When he's not programming, you can find him attempting to cook or hiking up a mountain.

Andrew regularly blogs about web technologies at http://www.andrewberls.com.

Simon Heimler is currently studying and working as a research assistant at the University of Applied Research in Augsburg in the field of Semantic Content Management. He has a degree in Interactive Media and over a decade of experience with web design and development.

Lars Kotthoff is a postdoctoral researcher at University College Cork, Ireland, where he uses artificial intelligence methods to make software faster and better. When he is not researching ways to make computers more intelligent, he plays around with JavaScript visualizations. He has extensive experience with D3.js.

Nathan Vander Wilt is a freelance software developer. He offers clients a wide range of expertise, including everything from creating HTML5 and native application interfaces to developing low-level control software for embedded and wireless systems. He especially enjoys solving problems such as peer-to-peer syncing or the many challenges of digital cartography. In order to stay sane in the suburbs, Nate also enjoys raising plants, fish, snails, honeybees, chickens, and rabbits with his family.

www.PacktPub.com

Support files, eBooks, discount offers, and more

You might want to visit www.PacktPub.com for support files and downloads related to your book.

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

http://PacktLib.PacktPub.com

Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can access, read and search across Packt's entire library of books.

Why subscribe?

Fully searchable across every book published by Packt

Copy and paste, print and bookmark content

On demand and accessible via web browser

Free access for Packt account holders

If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view nine entirely free books. Simply use your login credentials for immediate access.

Preface

D3 is an amazing library. On its website, there are hundreds of beautiful examples, visualizations, and charts created mainly with D3. Looking at the examples, we soon realize that D3 allows us to create an uncanny variety of visuals. We can find everything from simple bar charts to interactive maps.

The ability to create almost anything with D3 comes at a price; we must think about our charts at a more abstract level and learn how to bind data elements with elements in our page. This association between properties of our data items and visual attributes of the elements in our chart will allow us to create complex charts and visualizations.

In real-life projects, we will have to integrate components and charts created with D3 with other components and libraries. In most of the examples in this book, we will cover how to integrate D3 with other libraries and tools, creating complete applications that leverage the best of each library.

Through the examples of this book, we will cover reusable charts using external data sources, thereby creating user interface elements and interactive maps with D3. At the end, we will implement an application to visualize topics mentioned on Twitter in real time.

D3 is a great tool to experiment with visuals and data. I hope you will have fun following the examples in this book and creating your own visualizations.

What this book covers

Chapter 1, Data Visualization, provides us with examples of interesting visualization projects and references that help us learn more about data visualization. We also review some examples of historical relevance and discuss what makes D3 a good tool to create data-visualization projects.

Chapter 2, Reusable Charts, focuses on how to create configurable charts that can be used in several projects. In this chapter, we discuss how to use selections to manipulate elements in a web page and use this to create a reusable barcode chart from scratch. We also create a custom layout algorithm and use it to create a radial bar chart.

Chapter 3, Creating Visualizations without SVG, discusses the current state of SVG support in the browser market and provides some strategies to create visualizations that work in browsers that don't have SVG support. We create an animated bubble chart using div elements, learn how to detect whether the browser supports SVG, and use polyfills to render SVG figures using the HTML5 canvas element. We also learn how to create visualizations using D3 and canvas.

Chapter 4, Creating a Color Picker with D3, introduces concepts that allow us to create user interaction elements and controls. In this chapter, we use the D3 drag behavior and the reusable chart pattern to create a slider control. We use this control to create a color picker based on the CIE Lab color model, which is also a reusable chart.

Chapter 5, Creating User Interface Elements, discusses how to use event listeners to highlight elements in a chart. We also discuss how to create tooltips and how to integrate these tooltips with existing charts. We create an area chart and use the brush behavior to select a range in the chart.

Chapter 6, Interaction between Charts, discusses how to use Backbone to create structured web applications, separating data from its visual representation, and how to integrate D3 charts in this architecture. We will learn how to implement models, views, collections, and routes in order to keep a consistent application state. We will use this to create an application to explore the time series of stock prices using the area chart implemented in Chapter 5, Creating User Interface Elements.

Chapter 7, Creating a Charting Package, introduces the development workflow to create a charting package using D3. We introduce tools and best practices to implement, organize, and distribute the package. We will also create a sample project that uses the charting package as an external dependency.

Chapter 8, Data-driven Applications, provides us with an example of a web application and introduces tools to deploy visualization projects. We create an application that uses the World Bank data API to create a visualization of the evolution of indicators of human development. We will learn how to use GitHub pages to host our project and how to host a static website using Amazon S3.

Chapter 9, Creating a Dashboard, introduces concepts and best practices to create dashboards. We implement an example dashboard to monitor the performance of students in a class using D3 and custom charts.

Chapter 10, Creating Maps, discusses how to create vector maps using the geographic functions of D3. We will learn how to obtain geographic data and how to convert it to GeoJSON and TopoJSON formats, which are more suitable to be used with D3. We will create a choropleth map with D3 and use the TopoJSON library to visualize neighbors and boundaries between countries. We will also learn how to create a custom D3 layer to be used with Mapbox.

Chapter 11, Creating Advanced Maps, introduces some geographic projections and discusses how to configure projections to center and scale maps at specific locations. We also use the Orthographic projection to create a rotating globe. We also use a star catalog and the Stereographic projection to create a fullscreen star map. We will also learn how to use canvas to project raster images from Earth using the Orthographic projection.

Chapter 12, Creating a Real-time Application, introduces the concepts and tools that are used to create real-time applications. We will learn how to use Firebase to update the state of our applications in real time. We will also create a real-time application to explore the geographic distribution of geotagged tweets that match user-defined topics using Node, Socket.IO, and D3.

What you need for this book

The code bundle of this book was created using Jekyll, which is a static website generator. To run most of the examples in the code bundle, you will need a static web server and a modern web browser. The following list summarizes the main dependencies:

A modern web browser

D3 3.4

Jekyll or other static web servers

Text editor

Some chapters require you to install additional frontend libraries, such as Backbone, TopoJSON, Typeahead, and Bootstrap. Additional instructions on installing these libraries can be found in the corresponding chapters. In other chapters, we will use additional software to compile assets or process files. In those cases, installing the software is optional (the compiled files will be present as well), but it might be useful for you to install them for your own projects:

Node and Node packages

Git

Make

TopoJSON

GDAL

Instructions to install these applications can also be found in the corresponding chapters.

Who this book is for

This book is for frontend programmers who want to learn how to create charts, visualizations, and interactive maps with D3. We will cover everything from creating basic charts to complex real-time applications, integrating other libraries and components to create real-life applications.

We assume that you know the fundamentals of HTML, CSS, and JavaScript, but we review the main concepts as needed.

Conventions

In this book, you will find a number of styles of text that distinguish between different kinds of information. Here are some examples of these styles, and an explanation of their meaning.

Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows:

In the example file, we have a div element classed as chart-example and with the ID chart.

A block of code is set as follows:

divItems.enter()

    .append('div')

    .attr('class', 'data-item');

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

chart.onClick = function(d) {

    // ...

    // Invoke the user callback.

    onColorChange(color);

};

Any command-line input or output is written as follows:

$ grunt vows Running vows:all (vows) task (additional output not shown) Done, without errors.

New terms and important words are shown in bold. Words that you see on the screen, in menus or dialog boxes for example, appear in the text as follows:

By clicking on Create a Project, we can access the map editor, where we can customize the colors of land, buildings, and other features; select the base layer (street, terrain, or satellite) and select the primary language for the features and locations in the map.

Note

Warnings or important notes appear in a box like this.

Tip

Tips and tricks appear like this.

Reader feedback

Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or may have disliked. Reader feedback is important for us to develop titles that you really get the most out of.

To send us general feedback, simply send an e-mail to <[email protected]>, and mention the book title via the subject of your message.

If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide on www.packtpub.com/authors.

Customer support

Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.

Downloading the example code

You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

Downloading the color images of this book

We also provide you a PDF file that has color images of the screenshots/diagrams used in this book. The color images will help you better understand the changes in the output. You can download this file from: https://www.packtpub.com/sites/default/files/downloads/6270OS_Graphics.pdf.

Errata

Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you would report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the errata submission form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded on our website, or added to any list of existing errata, under the Errata section of that title. Any existing errata can be viewed by selecting your title from http://www.packtpub.com/support.

Piracy

Piracy of copyright material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works, in any form, on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.

Please contact us at <[email protected]> with a link to the suspected pirated material.

We appreciate your help in protecting our authors, and our ability to bring you valuable content.

Questions

You can contact us at <[email protected]> if you are having a problem with any aspect of the book, and we will do our best to address it.

Chapter 1. Data Visualization

Humans began to record things long before writing systems were created. When the number and diversity of things to remember outgrew the capacity of human memory, we began to use external devices to register quantitative information. Clay tokens were used as early as 8000-7500 BC to represent commodities like measures of wheat, livestock, and even units of man labor. These objects were handy to perform operations that would have been difficult to do with the real-life counterparts of the tokens; distribution and allocation of goods became easier to perform. With time, the tokens became increasingly complex, and soon, the limitations of the complex token system were identified and the system began to be replaced with simpler yet more abstract representations of quantities, thereby originating the earlier systems of writing.

Keeping records has always had a strong economic and practical drive. Having precise accounts of grains and pastures for the livestock allowed people to plan rations for the winter, and knowing about seasons and climate cycles allowed people to determine when to plant and when to harvest. As we became better at counting and registering quantitative information, trading with other nations and managing larger administrative units became possible, thereby providing us with access to goods and knowledge from other latitudes. We keep records because we think it's useful. Knowing what we have allows us to better distribute our assets, and knowing the past allows us to prepare for the future.

Today, we register and store more data than ever. Imagine that you want to go out for a morning cup of coffee. If you pay in cash, the date, price of the coffee, and the kind of coffee will be recorded before your coffee was actually prepared. These records will feed the accounting and stock systems of the store, being aggregated and transformed to financial statements, staff performance reports, and taxes to be paid by the store. Paying with credit card will generate a cascade of records in the accounting system of your bank. We measure things hoping that having the information will help us to make better decisions and to improve in the future.

History demonstrates that gathering and understanding data can help to solve relevant problems. An example of this is the famous report of John Snow about the Broad Street cholera outbreak. On August 31, 1854, a major outbreak of cholera was declared in the Soho district of London. Three days later, 127 people died from the disease. At the time, the mechanism of transmission of the cholera was not understood. The germ theory was yet to exist, and the mainstream theory was that the disease spread by a form of bad air. The physician, John Snow, began to investigate the case, collecting and classifying facts, recording deaths and their circumstances as well as a great number of testimonials. Refer to the following screenshot:

Details of the original map made for Snow, displaying the deaths by cholera in the Soho district

He gave special attention to the exceptions in the map and noticed that neither the workhouse inmates nor the brewery workers had been affected. The exceptions became further proof as he discovered that about 70 employees who worked in the brewery drank only beer made with water from a pump inside the walls of the brewery. In the workhouse, which also had its own water pump, only 5 out of 500 died, and further investigation revealed that the deceased were admitted when the outbreak had already begun. Although the map is convincing enough, Snow's original report contains more than 150 pages filled with tables and testimonials that support or raise questions about his theory. The local council decided to disable the pump by removing its handle, when the outbreak had already began to decline.

The report from John Snow is a great triumph of detective work and data visualization. He gathered information about the deaths and their circumstances and displayed them as data points in their geographic context, which made the pattern behind the causalities visible. He didn't stop at studying the data points; he also investigated the absence of the disease in certain places, faced the exceptions instead of quietly dismissing them, and eventually formed stronger evidence to support his case.

In this chapter, we will discuss what makes visual information so effective and discuss what data visualization is. We will comment about the different kinds of data visualization works, which gives a list of references to learn more about it. We will also discuss D3 and its differences with other tools to create visualizations.

Defining data visualization

Our brains are specially adapted to gather and analyze visual information. Images are easier to understand and recall. We tend to analyze and detect patterns in what we see even when we are not paying attention. The relation between visual perception and cognition can be used to our advantage if we can provide information that we want to communicate in a visual form.

Data visualization is the discipline that studies how to use visual perception to communicate and analyze data. Being a relatively young discipline, there are several working definitions of data visualization. One of the most accepted definitions states:

Data visualization is the representation and presentation of data that exploits our visual perception in order to amplify cognition.

The preceding quote is taken from Data Visualization: A successful design process, Andy Kirk, Packt Publishing.

There are several variants for this definition, but the essence remains the same—data visualization is a visual representation of data that aims to help us better understand the data and its relevant context. The capacity for visual processing of our brains can also play against us. Data visualization made without proper care can misrepresent the underlying data and fail to communicate the truth, or worse, succeed in communicating lies.

The kind of works that fall under this definition are also diverse; infographics, exploratory tools, and dashboards are data visualization subsets. In the next section, we will describe them and give some notable examples of each one.

Some kinds of data visualizations

There are countless ways to say things, and there are even more ways to communicate using visual means. We can create visualizations for the screen or for printed media, display the data in traditional charts, or try something new. The choice of colors alone can be overwhelming. When creating a project, a great number of decisions have to be made, and the emphasis given by the author