Week 2: Data Visualization Fundamentals¶
Sep 13, 2021
Housekeeping¶
- Piazza website: https://piazza.com/upenn/fall2021/musa550
- HW #1 due one week from today (9/20)
- Office hours:
- Nick: TBD
- Stella: Monday from 12:30 pm - 2 pm, remote
- Sign-up for time slots on Canvas calendar
Office hours survey: https://www.surveymonkey.com/r/TCKNWTX
Questions / concerns?
- Email: nhand@design.upenn.edu
- Post questions on Piazza
Guides¶
Guides to installing Python, using conda for managing packages, and working with Jupyter notebook on course website:
Git & GitHub Resources¶
File paths and working directories¶
Piazza post walking through somes tips for managing the folder structure on your laptop:
Harder option: downloading Github repository contents¶
Today's agenda¶
Part 1
- Wrapping up last week's pandas introduction
Part 2
- A brief overview of data visualization
- Practical tips on color in data vizualization
- The Python landscape:
Continuing with pandas: Zillow rental and home value data¶
import pandas as pd
from matplotlib import pyplot as plt
Load citywide Zillow Rent Index (ZRI) and Zillow Home Value Index (ZHVI) data.
Files were downloaded from https://www.zillow.com/research/data/
home_values = pd.read_csv("data/zillow/Metro_zhvi_uc_sfrcondo_tier_0.33_0.67_sm_sa_month.csv")
rent_values = pd.read_csv("data/zillow/Metro_ZORI_AllHomesPlusMultifamily_SSA.csv")
Peek at the first few rows of the ZRI data:
rent_values.head()
And do the same for the ZHVI data:
home_values.head()
Selecting the cities we want¶
valid_cities = [
"New York, NY",
"Chicago, IL",
"Los Angeles-Long Beach-Anaheim, CA",
"Philadelphia, PA",
"Houston, TX",
"Phoenix, AZ",
]
selection = home_values['RegionName'].isin(valid_cities)
home_values_trimmed = home_values.loc[selection]
selection = rent_values['RegionName'].isin(valid_cities)
rent_values_trimmed = rent_values.loc[selection]
rent_values_trimmed
Removing unwanted columns¶
Unwanted columns can be dropped from the data frame using the drop() function.
Note that the column axis is the second axis (axis=1), and if you wanted to remove rows, you could use the first axis (axis=0).
x = ['SizeRank', 'RegionID', "RegionType", "StateName"]
home_values_final = home_values_trimmed.drop(x, axis=1)
columns = ['SizeRank', 'RegionID']
rent_values_final = rent_values_trimmed.drop(columns, axis=1)
rent_values_final
Wide vs long format for datasets¶
Currently, our data is in wide format $\rightarrow$ each observation has its own column. This usually results in many columns but few rows.
home_values_final
Usually it's better to have data in tidy (also known as long) format.
Tidy datasets are arranged such that each variable is a column and each observation is a row.
In our case, we want to have a column called ZRI and one called ZHVI and a row for each month that the indices were measured.
pandas provides the melt() function for converting from wide formats to tidy formats.
melt() doesn’t aggregate or summarize the data. It transforms it into a different shape, but it contains the exact same information as before.
Imagine you have 6 rows of data (each row is a unique city) with 10 columns of home values (each column is a different month). That is wide data and is the format usually seen in spreadsheets or tables in a report.
If you melt() that wide data, you would get a table with 60 rows and 3 columns. Each row would contain the city name, the month, and the home value that city and month. This tidy-formatted data contains the same info as the wide data, but in a different form.
This animation shows the transformation from wide to long / long to wide. You can ignore gather() and spread() - those are the R versions of the pandas functions.
pd.melt?
Now, let's melt our datasets:
ZHVI = pd.melt(
home_values_final,
id_vars=["RegionName"],
value_name="ZHVI",
var_name="Date"
)
ZRI = pd.melt(
rent_values_final,
id_vars=["RegionName"],
value_name="ZRI",
var_name="Date"
)
and take a look:
ZRI.tail()
ZHVI.head()
Merging data frames¶
Another common operation is merging, also known as joining, two datasets.
We can use the merge() function to merge observations that have the same Date and RegionName values.
But first! Our date string formats don't match!¶
ZVHIhas the Date column in the format of YYYY-MM-DDZRIhas the Date column in the format of YYYY-MM
We need to put them into the same format before merging the data!
We can fix this by create Datetime objects and formatting the dates into the same format.
Datetime objects¶
Currently our Date column is stored as a string.
pandas includes additional functionality for dates, but first we must convert the strings using the to_datetime() function.
# Convert the Date column to Datetime objects
ZHVI["Date"] = pd.to_datetime(ZHVI["Date"])
The strftime function¶
We can use the ".dt" property of the Date column to access datetime functions of the new Datetime column.
For converting to strings in a certain format, we can use the "strftime" function (docs). This uses a special syntax to convert the date object to a string with a specific format.
Important reference: Use the this strftime guide to look up the syntax!
# Extract YYYY-MM string
date_strings = ZHVI["Date"].dt.strftime("%Y-%m")
# First entry is a string!
date_strings.iloc[0]
# Add the strings back as a column
ZHVI["Date"] = date_strings
ZHVI.head()
Now we can merge!¶
# Left dataframe is ZRI
# Right dataframe is ZHVI
zillow_data = pd.merge(ZRI, ZHVI, on=['Date', 'RegionName'], how='outer')
# Let's sort the data by Date
zillow_data = zillow_data.sort_values("Date", ascending=True)
zillow_data
Merging is very powerful and the merge can be done in a number of ways. In this case, we did a outer merge in order to keep all parts of each dataframe. By contrast, the inner merge only keeps the overlapping intersection of the merge.
See the infographic on joining in this repository.
# Convert the Date column back to a Datetime
zillow_data["Date"] = pd.to_datetime(zillow_data["Date"])
Quick trick: Series that hold Datetime objects have a dt attribute that let's you grab parts of the date easily.
For example, we can easily add new columns for the month and year using:
# Note the the dtype is now datetime64[ns]
zillow_data['Date'].head()
# Extract out the month and year of each date
# Add them to the data frame as new columns!
zillow_data['Month'] = zillow_data['Date'].dt.month
zillow_data['Year'] = zillow_data['Date'].dt.year
zillow_data.head()
Annual trends: grouping by Year¶
pandas is especially useful for grouping and aggregating data via the groupby() function.
From the pandas documentation, groupby means:
- Splitting the data into groups based on some criteria.
- Applying a function to each group independently.
- Combining the results into a data structure.
The documentation is available here.
We can calculate annual averages for each year by grouping by the RegionName and Year columns and taking the mean of our desired column. For example:
# calculate mean values for each Year and City (RegionName)
annual_ZHVI = zillow_data.groupby(['RegionName', 'Year'])['ZHVI'].mean()
annual_ZRI = zillow_data.groupby(['RegionName', 'Year'])['ZRI'].mean()
print(type(annual_ZHVI))
annual_ZHVI.head()
Imporant: The result of the groupby operation is always indexed by the group keys!
In this case, the result is indexed by the columns we grouped by (RegionName and Year).
We can reset the index so that the index values are listed as columns in the data frame again.
annual_ZHVI = annual_ZHVI.reset_index()
annual_ZRI = annual_ZRI.reset_index()
annual_ZHVI.head(n=50)
Plotting our results: ZHVI¶
with plt.style.context("ggplot"):
# Create figure and axes
fig, ax = plt.subplots(figsize=(10, 6))
# Plot for each unique city
for city in annual_ZHVI["RegionName"].unique():
# select the data for this city
selection = annual_ZHVI["RegionName"] == city
df = annual_ZHVI.loc[selection]
# plot
ax.plot(df["Year"], df["ZHVI"] / 1e3, label=city, linewidth=4)
# Format the axes
ax.set_ylim(50, 800)
ax.legend(loc=0, ncol=2, fontsize=12)
ax.set_ylabel("Zillow Home Value Index\n(in thousands of dollars)")
Home values in Philadelphia have only recently recovered to pre-2008 levels¶
Plotting the results: Zillow Rent Index¶
with plt.style.context('ggplot'):
# Create the figure and axes
fig, ax = plt.subplots(figsize=(10,6))
# Loop over the cities to plot each one
for city in annual_ZRI['RegionName'].unique():
# Select the city data
selection = annual_ZRI['RegionName'] == city
df = annual_ZRI.loc[selection]
# Plot
ax.plot(df['Year'], df['ZRI'], label=city, linewidth=4)
# Format
ax.set_ylim(1000, 3300)
ax.legend(loc=0, ncol=2)
ax.set_ylabel('Zillow Rent Index (in dollars)')
Rent prices in Philadelphia have remained relatively flat, relative to other large cities.¶
Week #2: Data Visualization Fundamentals¶
- Week #2 repository: https://github.com/MUSA-550-Fall-2021/week-2
- Recommended readings for the week listed here
A brief history¶
Starting with two of my favorite historical examples, and their modern renditions...
Example 1: the pioneering work of W. E. B. Du Bois¶
Re-making the Du Bois Spiral with census data¶
Example 2: the Statistical Atlas of the United States¶
- First census: 1790
- First map for the census: 1850
- First Statistical Atlas: 1870
- Largely discontinued after 1890, except for the 2000 Census Atlas
Using modern data¶
See http://projects.flowingdata.com/atlas, by Nathan Yau
Industry and Earnings by Sex¶
Median Household Income¶
Many more examples...¶
More recently...¶
Two main movements:¶
- 1st wave: clarity
- 2nd wave: the grammar of visualization
Wave 1: Clarity¶
- Pioneered by Edward Tufte and his release of The Visual Display of Quantitative Information in 1983
- Focuses on clarity, simplicity, and plain color schemes
- Charts should be immediately accessible and readable
The idea of "Chartjunk"¶
- Coined by Tufte in Visual Display
- Any unnecessary information on a chart
An extreme example¶
Wave 2: the grammar of visualization¶
- Influenced by The Grammar of Graphics by Leland Wilkinson in 1999
- Focuses on encoding data via channels onto geometry
- Mapping data attributes on to graphical channels, e.g., length, angle, color, or position (or any other graphical character)
- Less focus on clarity, more on the encoding system
- Leads to many, many (perhaps confusing) ways of visualizing data
ggplot2provides an R implementation of The Grammar of Graphics- A few different Python libraries available
Where are we now?¶
- Both movements converging together
- More visualization libraries available now than ever
A survey of common tools¶
- From a 2017 survey by Elijah Meeks
- Data visualization engineer: Apple, Netflix
- Excellent data viz resource
- Find him on Twitter or Medium: @Elijah_Meeks
- Executive director of the Data Visualization Society
- Community-based data viz organization
- Great resources for beginners
- Check out the Nightingale: The Data Visualization Society's Blog
The 7 kinds of data viz people¶
- From this blog post
- Illustrations by Susie Lu
See, e.g. Data Sketches
Data visualization as communication¶
- Data visualization is primarily a communication and design problem, not a technical one
- Two main modes:
- Fast: quickly understood or quickly made (or both!)
- Slow: more advanced, focus on design, takes longer to understand and/or longer to make
Fast visualization¶
- Classic trope: a report for busy executives created by subject experts $\rightarrow$ as clear and simplified as possible
- Leads readers to think that if the chart is not immediately understood then it must be a failure
- The dominant method of data visualization
Moving beyond fast visualizations¶
- Thinking about what charts say, beyond what is immediately clear
- Focusing on colors, design choices
Example: Fatalities in the Iraq War¶
by Simon Scarr in 2011
What design choices drive home the implicit message?¶
Data Visualization as Storytelling¶
The same data, but different design choices...
A negative portrayal¶
A positive portrayal¶
Design choices matter & data viz has never been more important¶
Some recent examples...
- Data Viz's Breakthrough Moment in the COVID-19 Crisis
- Interview with John Burn-Murdoch About his COVID Data Viz
- John Burn-Murdoch's Twitter
- COVID-19 Data Viz from the Financial Times
Data Viz Style Guides¶
Lots of companies, cities, institutions, etc. have started design guidelines to improve and standardize their data visualizations.
One I particularly like: City of London Data Design Guidelines
First few pages are listed in the "Recommended Reading" portion of this week's README.
London's style guide includes some basic data viz principles that everyone should know and includes the following example:
Good rules¶
- Less is more — minimize "chartjunk"
- Don't use legends if you can label directly
- Use color / line weight to focus the reader on the data you want to emphasize
- Don't make the viewer tilt their head — Use titles/subtitles to explain what is being plotted
Now onto colors...¶
Choose your colors carefully:
- Sequential schemes: for continuous data that progresses from low to high
- Diverging schemes: for continuous data that emphasizes positive or negative deviations from a central value
- Qualitative schemes: for data that has no inherent ordering, where color is used only to distinguish categories
ColorBrewer 2.0¶
- The classic tool for color selection
- Handles all three types of color schemes and provides a map-based visualization
- Provides explanations from Cynthia Brewer's published works on color theory
- Tests whether colors are colorblind safe, printer friendly, and photocopy safe
- ColorBrewer palettes are included by default in
matplotlib
Perceptually uniform color maps¶
- Created for
matplotliband available by default - perceptually uniform: equal steps in data are perceived as equal steps in the color space
- robust to color blindness
- colorful and beautiful
For quantitative data, these color maps are very strong options
Need more colors?¶
Almost too many tools available...
Some of my favorites¶
- Adobe Color CC: allows you to explore other people's color palettes and create new ones
- Paletton: similar to Adobe Color, but slightly more advanced
- Chroma.js Color Scale Helper: create color palettes by interpolating between named HTML colors
- Colorpicker for data: automatically generate new color palettes, but they aren't always useful
Making sure your colors work: Viz Palette¶
Wrapping up: some good rules to live by¶
- Optimize your color map for your dataset
- Think about who your audience is
- Avoid palettes with too many colors: ColorBrewer stops at ~9 for a reason
- Maintain a theme and make it pretty
- Think about how color interacts with the other parts of the visualization
Now onto the Python data viz landscape¶
So many tools...so little time
Which one is the best?¶
There isn't one...¶
You'll use different packages to achieve different goals, and they each have different things they are good at.
Today, we'll focus on:
- matplotlib: the classic
- pandas: built on matplotlib, quick plotting built in to DataFrames
- seaborn: built on matplotlib, adds functionality for fancy statistical plots
- altair: interactive, relying on javascript plotting library Vega
And next week for geospatial data:
- holoviews/geoviews
- matplotlib/cartopy
- geopandas/geopy
Goal: introduce you to the most common tools and enable you to know the best package for the job in the future
The classic: matplotlib¶
- Very well tested, robust plotting library
- Can reproduce just about any plot (sometimes with a lot of effort)
With some downsides...¶
- Imperative, overly verbose syntax
- Little support for interactive/web graphics
Available functionality¶
- Don't need to memorize syntax for all of the plotting functions
- For sample plots: https://matplotlib.org/tutorials/introductory/sample_plots.html
- See the cheat sheet available in this repository
Working with matplotlib¶
We'll use the object-oriented interface to matplotlib
- Create
FigureandAxesobjects - Add plots to the
Axesobject - Customize any and all aspects of the
FigureorAxesobjects
- Pro: Matplotlib is extraordinarily general — you can do pretty much anything with it
- Con: There's a steep learning curve, with a lot of matplotlib-specific terms to learn
Recommended Reading¶
- Introduction to the object-oriented interface
- A good walk through on using matplotlib to customize plots
- Listed in the README for this week's repository too
Let's explore colormaps in matplotlib¶
import numpy as np
from matplotlib import pyplot as plt
# Generate some random data using numpy (numbers between -1 and 1)
# Shape is (100, 100)
data = 2 * np.random.random(size=(100,100)) - 1
print(data.min(), data.max(), data.mean())
The new default color map: viridis¶
plt.pcolormesh(data, cmap='viridis')
The old default: jet¶
plt.pcolormesh(data, cmap='jet')
Better suited for a diverging palette...¶
plt.pcolormesh(data, cmap='coolwarm')
Important bookmark: Choosing Color Maps in Matplotlib
# print out all available color map names
print(len(plt.colormaps()))
Let's load some data to plot...¶
We'll use the Palmer penguins data set, data collected for three species of penguins at Palmer station in Antartica
Artwork by @allison_horst
# Load data on Palmer penguins
penguins = pd.read_csv("./data/penguins.csv")
penguins.head(n=10)
Data is already in tidy format
A simple visualization¶
I want to scatter flipper length vs. bill length, colored by the penguin species
Using matplotlib¶
# Initialize the figure and axes
fig, ax = plt.subplots(figsize=(10, 6))
# Color for each species
color_map = {"Adelie": "#1f77b4", "Gentoo": "#ff7f0e", "Chinstrap": "#D62728"}
# Group the data frame by species and loop over each group
# NOTE: "group" will be the dataframe holding the data for "species"
for species, group in penguins.groupby("species"):
print(f"Plotting {species}...")
# Plot flipper length vs bill length for this group
ax.scatter(
group["flipper_length_mm"],
group["bill_length_mm"],
marker="o",
label=species,
color=color_map[species],
alpha=0.75,
)
# Format the axes
ax.legend(loc="best")
ax.set_xlabel("Flipper Length (mm)")
ax.set_ylabel("Bill Length (mm)")
ax.grid(True)
How about in pandas?¶
# Tab complete on the plot attribute of a dataframe to see the available functions
#penguins.plot.scatter?
# Initialize the figure and axes
fig, ax = plt.subplots(figsize=(10, 6))
# Calculate a list of colors
color_map = {"Adelie": "#1f77b4", "Gentoo": "#ff7f0e", "Chinstrap": "#D62728"}
colors = [color_map[species] for species in penguins["species"]]
# Scatter plot two columns, colored by third
penguins.plot.scatter(
x="flipper_length_mm",
y="bill_length_mm",
c=colors,
alpha=0.75,
ax=ax, # Plot on the axes object we created already!
)
# Format
ax.set_xlabel("Flipper Length (mm)")
ax.set_ylabel("Bill Length (mm)")
ax.grid(True)
Note: no easy way to get legend added to the plot in this case...
Disclaimer¶
- In my experience, I have found the
pandasplotting capabilities are good for quick and unpolished plots during the data exploration phase - Most of the pandas plotting functions serve as shorcuts, removing some biolerplate matplotlib code
- If I'm trying to make polished, clean data visualization, I'll usually opt to use matplotlib from the beginning
That's it!¶
- See you on Wedndesday when we wrap up Data Viz Fundamentals