Monthly Archives: March 2012

Tidbits of geography (and of cake)

Futility Closet has plenty of great trivia. I want to share some of my favorite geographical tidbits from there, since I have maps on the mind lately.

Source: Wikimedia Commons

While we’re talking about shapes and areas, here’s a more mathematical-geometrical question: What’s the most efficient way to carve up a circle to fit inside a square of slightly-larger surface area?

Round peg into a square hole, er, that is, cake into pan

Round peg into a square hole, er, that is, cake into pan

I baked a cake in a round 9″ pan, so the surface area is \pi*r^2 = \pi*4.5^2 = 63.6 \text{ in}^2.  I wanted to transport it in a pan with a lid, and I have such a 8″ square  pan with surface area 8^2 = 64 \text{ in}^2. What’s the best way to fit it in, with the fewest cuts and least wasted scraps? (Well, not really wasted, I’ll eat them gladly :) )

Localized Comparisons: Idiopleth Maps?

In which we propose a unifying theme, name, and some new prototypes for visualizations that allow “localization of comparisons,” aka “How do I relate to others?”

When Nathan Yau visited the Bureau a few months ago, he compared two world maps of gasoline prices by country. The first one was your typical choropleth: various color shades correspond to different gas prices. Fair enough, but (say) an American viewing the map is most likely interested in how US gas prices compare to the rest of the world. So instead, present a map with America in a neutral color (grey) and recolor the other countries relative to the US, to show whether their prices are higher or lower than here (for instance, red for pricier and green for cheaper gas).

I liked this idea but wanted to take it further: Instead of a one-off map just for Americans, why not make an interactive map that recolors automatically when you select a new country?
As a statistician, I’m also interested in how to communicate uncertainty: is your local area’s estimate statistically significantly different from your neighbors’ estimates? Continue reading

Stats 101 resources

A few friends have asked for self-study resources on learning (or brushing up on) basic statistics. I plan to keep updating this post as I find more good suggestions.

Of course the ideal case is to have a good teacher in a nice classroom environment:

The best classroom setting

For self-study, however, you might try an open course online. MIT has some OpenCourseWare for mathematics (including course 18.433, “Statistics for Applications”), and Carnegie Mellon offers free online courses in statistics. I have not tried them myself yet but hear good things so far.

As for textbooks: Freedman, Pisani, and Purves’ Statistics is a classic intro to the key concepts and seems a great way to get up to speed.
Two other good “gentle” conceptual intros are The Cartoon Guide to
Statistics
and How to Lie with Statistics.
But I believe they all try to avoid equations, so you might need another source to show you how to actually crunch the numbers.
My undergrad statistics class used Devore and Farnum’s Applied Statistics for Engineers and Scientists. I haven’t touched it in years, so I ought to browse it again, but I remember it demonstrated the steps of each analysis quite clearly.
If you end up using the Devore and Farnum book, Jonathan Godfrey has converted the 2nd edtion’s examples into R.
[Edit: John Cook's blog and his commenters have some good advice about textbooks. They also cite a great article by George Cobb about how to choose a stats textbook.]

Speaking of R, I would suggest it if you don’t already have a particular statistical software package in mind. It is open source and a free download, and I find that working in R is similar to the way I think about math while working it out on paper (unlike SPSS or SAS or Stata, all of which are expensive and require a very different mindset).
I list plenty of R resources in my R101 post. In particular, John Verzani’s simpleR seems to be a good introduction to using R, and reviews on a lot of basic statistics along the way (though not in detail).
People have also recommended some books on intro stats with R, especially Dalgaard’s Introductory Statistics with R or Maindonald & Braun’s Data Analysis and Graphics Using R.

For a very different approach to introductory stats, my former professor Allen Downey wrote a book called Think Stats aimed at programmers and using Python. I’ve only just read it, and I have a few minor quibbles that I want to discuss with him, but it’s a great alternative to the classic approach. As Allen points out, “standard statistical techniques are really computational shortcuts, which is less important when computation is cheap.” Both mindsets are good to have under your belt, but Allen’s is one of the few intro books so far for the computational route. It’s published by O’Reilly but Allen also makes a free version available online, as well as a related blog full of good resources.
Speaking of O’Reilly, apparently their book Statistics Hacks contains major conceptual errors, so I would have to advise against it unless they fix them in a future edition.

DC Datadive

This weekend I had an absolute blast taking part in the DC Datadive hosted by the NYC-based Data Without Borders (DWB). It was somewhat like a hackathon, but rather than competing to develop an app with commercial potential, we were tasked with exploring data to produce insights for social good. (Perhaps it’s more like the appropriate-technology flash conferences for engineers that my classmates organized back at Olin.) In any case, we mingled on Friday night, chose one of three projects to focus on for Saturday (10am to 5am, in our case!), and presented Sunday morning.

The author, eager to point out a dotplot

The author, eager to point out a dotplot

There were three organizations acting as project sponsors (presentations here):

  • The National Environmental Education Foundation wondered how to evaluate their own efforts to increase environmental literacy among the US public. Their volunteers came up with great advice and even found some data NEEF didn’t realize they already had.
  • GuideStar, a major database of financial information on nonprofit organizations, wanted early-warning prediction of nonprofits that are at risk of failing, as well as ways to highlight high-performing organizations that are currently under the radar. This group of datadivers essentially ran their own Netflix prize contest, assembling an amazing range of machine learning approaches that each gave a new insight into the data.
  • DC Action for Children tasked us with creating a visualization to clearly express how children’s well-being, health, school performance, etc. are related to the neighborhood where they live. I chose to work on this project and am really pleased with the map we produced: screenshot and details below.

Click above and try it out. Mousing over each area gives its neighborhood-level information; hovering over a school gives school details.
In short, our map situates school performance (percent of children with Proficient or Advanced scores on reading and math tests) in the context of their DC neighborhood. Forgive me if I’m leaving out important nuances, but as I understood it the idea was to change the conversation from “The schools on this list are failing so they must have poor adminstration, bad teachers, etc.” towards “The children attending the schools in this neighborhood have it rough: socioeconomic conditions, few resources like libraries and swimming pools, no dentists or grocery stores, etc. Maybe there are other factors that public policy should address before putting full responsibility on the school.” I think our map is a good start on conveying this more effectively than a bunch of separated tables.

It was so exciting to have a tangible “product” to show off. There may be a few minor technical glitches, and we did not have time to show all of the data that the other subteams collected, but it’s a good first draft.
Planning and coordinating our giant group was a bit tough at first but our DWB coordinator, Zac, gamely kept us moving and communicating across the several sub-teams that we formed. The data sub-team found, organized, and cleaned a bunch more variables than we could put in, so that whoever continues this work will have lots of great data to use. And the GIS sub-team aggregated it all to several levels (Census tract, neighborhood, and ward); again we only had time to implement one level on map, but all is ready to add the other levels when time allows.
As for myself, I worked mostly with the visualization sub-team: Nick who set up the core map in TileMill; Jason who kept pushing it forward until 5am; Sisi who styled it and cobbled together the info boxes out of HTML and the Google Charts API and who knows what else; and a ton of other fantastic people whose handles I can’t place at the moment. I learned A TON from everybody and was just happy that my R skills let me contribute to this great effort.
[Edit: It was amiss not to mention Nick's coworkers Troy and Andy who provided massive help with the GIS prep and the TileMill hosting. Andy has a great writeup of the tools, which they also use for their maps of the week.]
I absolutely loved the collaborative spirit: people brought so many different skills and backgrounds to the team, and we made new connections that I hope will continue with future work on this or similar projects. Perhaps some more of us will join the Data Science DC Meetup group, for example.
I do wish I had spent more time talking to people on the other projects — I was so engrossed in my own team’s work that I didn’t get to see what other groups were doing until the Sunday presentations. Thank goodness for catching up later via Twitter and #dcdatadive.

A huge thanks to New America Foundation for hosting us (physically as well as with a temporary TileMill account), to the Independent Sector NGEN Fellows for facilitating, to whoever brought all the delicious food, and of course to DWB for putting it all together. I hope this is just the start of much more such awesomeness!

PS — my one and only concern: The wifi clogged up early on Saturday, when everyone was trying to get data from the shared Dropboxes at once. If you plan to attend a future datadive, I’d suggest bringing a USB stick to ease sharing of big files if the wifi collapses.

[Edit: I also recommend DC Action for Children's blog posts on their hopes before the datadive and their reactions afterwards. They have also shared a good article with more open questions about how kids are impacted by inequality in and among DC neighborhoods.]

R101

I’m preparing “R101,” an introductory workshop on the statistical software R. Perhaps other beginners might find some use in the following summary and resources. (See also the post on resources for teaching yourself introductory statistics.)

Do you have obligatory screenshots of nifty graphics that R can produce? Yes, we do.

Nice. So what exactly is R? It is an open-source software tool for statistics, data processing, data visualization, etc. (Technically there’s a programming language called S, and R is just one open-source software tool that implements the S language. But you’ll often hear people just say “the R language.” Beginners can worry about the nuances later.)
Open source means it is free to download and use; this is great for academics and others with low budgets. It also means you can inspect the code of any algorithm if you want to double-check it or just to see how it’s done; this is great for validating and building on each others’ ideas. And it is easy to share code in user-defined “packages,” of which there are thousands, all helping people use cutting-edge statistical tools as soon as they are invented.

How do I get started? Download and install R from CRAN, the Comprehensive R Archive Network. There are Windows, Mac, and Linux versions.
In Windows at least, when you open the program there is a big window containing a smaller window, the R Console. You can type and submit commands in the Console window at the prompts (the “>” signs). Try typing 3+5 and hit Enter, and you should see the output [1] 8 which is good. The output of 3+5 is a 1-item vector (hence the [1]) with the value 8 as it should be.
Great, now you know how to use R as a desktop calculator!
Or you can type your commands in a script, so that you can save your code easily. Go to “File -> New script” and it will open the R Editor window. Type 3+5 in there, highlight it, and then either click the “Run line or selection” icon on the top menu bar or just hit Ctrl+R on the keyboard. It should copy the command into the Console window and run it, with the same result as before.
Sweet, now you can save the code you used to do your calculations.
Quick-R has more details on using the R interface.
Next, try A Sample Session from the R manual to see examples of other things R can do.

What are the key concepts? Basically, everything is a function or an object. Objects are where your data and results are stored: data frames, matrices, vectors, lists, etc. Functions take objects in, think about them, and spit new objects out. Functions sometimes also have side effects (like displaying a table of output or a graph, or changing a display setting).
If you want to save the results or output of a function, use <- which is the assignment operator (think of an arrow pointing left). For example, to save the natural log of 10 into a variable called x, type the command x <- log(10). Then you can use x as the input to another function.
Note that functions create new output rather than affecting the input variable. If you have a vector called y that you need sorted, sort(y) will print out a sorted copy of y but will not changed y itself. If you actually want y to be sorted, you have to reassign it: y <- sort(y).
Functions always take their input in parentheses: (). So if you see a word followed by parentheses, you know it’s a function in R. You will also see square brackets: []. These are used for locating or extracting data in objects. For example, if you have a vector called y, then y[3] gives you the 3rd element of that vector. If y is a matrix, then y[4,7] is the element in the 4th row, 7th column.

How do I get help? If you know you want to use a function named foo, you can learn more about it by typing ?foo which will bring up the help file for that function. The “Usage” section tells you the arguments, their default order, and their default values. (If no default value is given, it is a required argument.) “Arguments” gives more details about each argument. “Value” gives the structure of the output. “Examples” shows an example of the function in use.
If you know what you want to do but don’t know what the function is called, I suggest looking through the R Reference Card. If that does not answer your question, you can try searching using RSeek.org or search.r-project.org, search engine tuned to the R sites and mailing lists… since just typing the letter R into Google is not always helpful :)

Where do I read more?
Online resources for general beginners:
R for Beginners
Simple R
Official Introduction to R
R Fundamentals
Kickstarting R
Let’s Use R Now
UCLA R Class Notes
A Quick and (Very) Dirty Intro to Doing Your Statistics in R
Hints for the R Beginner
R Tutorials from Universities Around the World (88 as of last count)

For statisticians used to other packages:
Quick-R
R for SAS and SPSS Users

For programmers:
R’s unconventional features
Google’s R code style guide

Good books (as suggested by Cosma Shalizi):
Paul Teetor, The R Cookbook: “explains how to use R to do many, many common tasks”
Norman Matloff, The Art of R Programming: “Good introduction to programming for complete novices using R.”