Bayesian statistics and applied computing at CMU?

A reader asks:

I wanted to pick your brain about stats and machine learning at CMU … I’m considering a Ph.D. in a finance or a related discipline.

Here’s the thing, I’m very much attracted to schools with established inter-disciplinary programs, like CMU’s additional masters in machine learning, and Duke’s supplemental masters in statistics. Duke bills itself as the best Bayesian shop under the sun, which is also attractive. I’m not dogmatically fixed on Bayesian methods, but I do find it a much more natural way of thinking, and more naturally applied to practical problems.

One person I spoke to suggested that Duke was the better program based on my interests, but I read on your blog that Bayesian methods and applied computing are pretty well represented at CMU, so I figured I’d get your thoughts. Leaving aside the reality that one can’t choose where they’re admitted, and that one should focus their choice on the strength of their primary department, I’d like to know which would be the better option.

My response:

I admit I don’t know much about the finance program here, nor about the supplemental masters in ML. And of course I know even less about Duke’s equivalent programs.

That said, CMU is absolutely a strong place for machine learning and statistics, including applied computing and Bayesian statistics.

Bayes:

The core courses for the ML masters (10-701, 10-702, and 10-705) do cover Bayesian methods and inference. We study the basic theory and plenty of applications, including less-often-taught methods like Bayesian nonparametrics. Parts of 702 and 705 are especially helpful for clarifying how Frequentist and Bayesian inferences differ. Although I’m a fan of the Bayesian approach, I really appreciate how Larry Wasserman challenges us to understand its weaknesses thoroughly, using plenty of examples where Classical methods have an advantage over Bayesian ones (such as Sec 12.6 here).
Beka Steorts also offers a pair of courses that go into more depth on Bayesian theory and applications.
There’s also a close link between the Statistics and Philosophy departments: particularly Kadane and Schervish here in Stats, and Seidenfeld in Phil, work together regularly on the foundations of statistical inference, incl. Bayesian.

ML and applied computing:

CMU is also rare (maybe unique?) in that we have a whole college of Computer Science with a whole department of Machine Learning (…rather than just a department of CS with just a ML research group). That means we have a ton of faculty and students with both applied and theoretical interests. Also, our theory courses account for how to optimize an estimator or algorithm, and how long it will take to converge, not merely whether it’s statistically a good estimator.
The Statistics department also encourages applications, not just theory. The undergrads get courses like Statistical Computing and Undergraduate Advanced Data Analysis. There’s a whole MS in Statistical Practice. And for the PhD students like myself, our first major research project (the Advanced Data Analysis) is very much intended to be an applied project.
There’s a constant stream of interesting applied talks on campus, more of them than you can hope to attend 🙂 including Stat Bytes, Machine Learning Lunch, Machine Learning & Social Sciences group, SIAM student chapter, Statistical Machine Learning Reading Group, Statistics in Education Research Group, …

I’m sure that you’d find CMU worthwhile if you end up coming here.

With loss of generality

Public service announcement: Dear math and statistics students, “WLOG” means you’re about to prove something “without loss of generality.”

So please don’t copy your friend’s homework and write it as “with log” or “using log”. It’s just too easy for your grader to catch you.

♪ The more you know! ♫

Reproducible research, training wheels, and knitr

Last week I gave a short talk at CMU’s statistical computing seminar, Stat Bytes. I summarized why reproducible research (RR) and literate programming are worthwhile, not just for serious research but also for homework reports or statistical blog posts. I demonstrated how to get started with a range of RR document formats in R: from the “training wheels” R Notebook in RStudio, through the more flexible but still simple R Markdown format, to R Sweave for $\LaTeX$ articles and Beamer slides.

If you’ve wanted to get on the RR bandwagon, but found Sweave too overwhelming, these other tools are a great way to start—and useful in their own right, not just for training.

My materials are here:

Overview and links (html output, Rmd source)
R Notebook example (html output, R source)
R Markdown example (html output, Rmd source)
R Sweave / Beamer example (pdf output, Rnw source)

Extra details below.

Reproducible research story time

First, story time! I was once asked to step in and take over the statistical analysis for an article, after the primary statistician became unavailable. It sounded like a pretty straightforward analysis of survey data, with clear scientific questions, and they told me they had the previous statistician’s R code, so I thought it sounded reasonable. Hah…

Continue reading “Reproducible research, training wheels, and knitr” →

After 1st semester of Statistics PhD program

Have you ever wondered whether the first semester of a PhD is really all that busy? My complete lack of posts last fall should prove it 🙂

Some thoughts on the Fall term, now that Spring is well under way [edit: added a few more points]:

RMarkdown and knitr are amazing. When I next teach a course using R, my students will be turning in homeworks using these tools: The output immediately shows whether the code runs and what its results are. This is much better than students copying and pasting possibly-broken code and unconnected output into a text file or (gasp) Word document.
I’m glad my cohort socializes outside the office, taking each other out for birthday lunches or going to see a Pirates game. Some of the older PhD students are so focused on their thesis work that they don’t take time for a social break, and I’d like to avoid getting stuck in that rut.
However! Our lunches always lead us back to the age old question: How many statisticians does it take to split a bill? Answer: too long. I threw together a Shiny app, DinneR, to help us answer this question 🙂

The first-year PhD courses in Statistics and in Machine Learning have rather different approaches.
- Statistics professor: Just assume we can compute this estimator. In class we’ll prove that the estimates are reasonably good (e.g. we’ll bound the probability that an estimate is far from the true value).
- Machine Learning professor: Just trust me that this algorithm gets useful estimates. In class we’ll prove that we can compute it in a reasonable amount of time (e.g. we’ll bound the number of steps until the algorithm converges).
- Somewhere between these ideas, I ran into the sensible concept of optimizing only until your solution is within statistical error. For example, say you only have enough data to publish an estimate with a confidence interval of +\- 0.1 units. If your optimization algorithm is computer-intensive, then running it until it converges to +\- 0.00001 units is just a waste of time. For instance, see Bottou & Bousquet’s “The Tradeoffs of Large Scale Learning.”
My ML professor, midway through a classification-focused semester, finally discussing regression for 10 minutes: “…And that’s all you need to know about regression.”
My Regression professor, at end of semester, finally discussing classification for 20 minutes: “…And that’s all you need to know about classification.” 🙂
In any class that covers proofs or other long detailed arguments, handouts+chalkboards are seriously better than slideshows. With a chalkboard, you can show the whole proof at once—so if students get lost halfway through, they can still see the claim we’re proving and all the steps we’ve made so far. But when you cram a proof onto slides, either you oversimplify to get it onto one slide; or you split it across slides, so that we lose the continuity (and may even forget what we’re trying to prove).
Good homeworks and quick feedback are critical. One of my classes had weekly homeworks, each directly tied to the material we just covered, each problem expanding on a good question or illustrating an interesting principle from class. Homeworks were graded within a week, every single time.
In another class, we had just a few homeworks, very loosely tied to the lecture contents and usually at a very different level (way too easy or too hard relative to what the lecture covered). Although this class had the same number of students and TAs as the other one, we never got our homeworks back in less than 2 weeks—and one of them took a full 2 months to return!
TAing is a mixed bag. I enjoy holding office hours and being there during lab sessions to help students understand something they were missing. I do not so much enjoy grading homeworks and labs by those students who don’t ask questions, don’t come to office hours, and clearly don’t read the comments I leave on their assignments since I see them make the same mistakes over and over. I especially don’t like finding instances of cheating. Urgh.
I was a bit worried about coming back to grad school as an “older” student (the youngest guy in our 1st-year PhD cohort is almost a decade younger than me!). But it’s been great, actually:
- My schedule seems much saner than some of my classmates’. Quite a few seem to stay in the office until late most nights, then may sleep through a morning class. For me, after years of waking at 6:30 to spend an hour on the crowded metro to work… it’s been luxurious to sleep in until 7:30 or 8, walk to school in half an hour in the fresh air, have a focused workday of reasonable length, and come home for dinner with my wife, actually relaxing in the evening instead of studying until 3am. Yes, there’s the occasional late night, but occasional is the key word there.
- The income’s lower than my old job, of course, but Pittsburgh is much cheaper than DC, especially for housing. Besides: my previous school loans are all paid off, I have a fair chunk of retirement savings already earning interest, and my wife and I are used to budgeting. (YNAB is an excellent tool for this—I will blog about it at some point. If you’re interested, here’s a slight discount referral code, or you can wait for the big sale they seem to have every 3-4 months.)
  [My point is: despite the drop in income, we’re still more financially secure (thanks to savings and paid-off loans) than if I’d gone straight into the PhD from my MSc.]
- As Cosma Shalizi points out: “Note to graduate students: It is important that you internalize that you are, in fact, a badass…” With age and experience, I’m far more able to speak confidently when it’s called for (e.g. giving a talk), and far less intimidated about tackling new topics, talking to professors, writing papers, speaking at conferences, etc.
On the other hand, despite longer experience as a statistician than my classmates, I appreciate and admire that they are much better at many things. I’m really impressed by my various classmates’ command of topics like real analysis and measure theory, scientific computing, or practical knowledge about fields like physics or economics.
Pittsburgh is a great town. Affordable housing, decent bus system, beautiful scenic views from the inclines, friendly people, livable walkable neighborhoods, tons of good food, extensive and well-run library system… It has a lot of what I liked about Portland, without as much of the “Portlandia” over-the-top hipsters. There are also beautiful old buildings, like the Carnegie Natural History Museum (with its sweet dinosaur exhibit) and UPitt’s Cathedral of Learning. The weather right now is pretty snowy/icy, but I don’t mind—I’m honestly impressed by how well Pittsburgh just goes ahead and deals with winter weather, in comparison to DC’s city-wide shutdown every time a snowflake is sighted.

Edit: Here’s another good post on the first semester of a PhD program, from several mathematics students. I agree with most of the responses, especially the ones that conflict each other 🙂

Next up: the 2nd, 3rd, 4th, 5th, 6th, 7th, 8th, 9th, and 10th semesters of my Statistics PhD program.

Is a Master’s degree in Statistics worthwhile?

A student who is considering a Master’s degree in Statistics asks, “I’m interested in finding a job in data analysis and have been looking around, but I’m not sure if a masters is necessary to break into the field.”

Without much info about her background or job goals, here’s what I replied. Readers, do you have any additional or contradictory advice?

Continue reading “Is a Master’s degree in Statistics worthwhile?” →

History of CMU’s Department of Statistics

As I’m about to begin my studies at CMU’s Department of Statistics, I have been curious about the department’s history. There is a nice writeup in Strength in Numbers: The Rising of Academic Statistics Departments in the U. S.. Luckily, the “Carnegie Mellon University Statistics Department” chapter happens to be the free sample chapter on the publisher’s website.

Some fun facts from the chapter:

I hadn’t known that Frederick Mosteller went here (back when it was Carnegie Tech). I enjoy his Fifty Challenging Problems in Probability, and I’ve also been meaning to read The Pleasures of Statistics: The Autobiography of Frederick Mosteller. One of his early students at Harvard (whose stats department Mosteller founded before CMU had one) was Steve Fienberg, still at CMU.

Although the department was formed in 1966, it didn’t have a permanent college to call home until it joined the humanities college in 1980.
StatLib, the department’s online collection of downloadable datasets, started in 1989 and is still in use today.
CMU’s stats department was one of the first anywhere to focus on Bayesian stats, applications, and statistical computing. All of these are areas of interest for me—good to know I’m in the right place!
Early on, they also agreed to evaluate applied research on whether it benefits the applied area, not necessarily statistics as a field. I saw this still in effect at a thesis defense this week: the focus was on a very practical contribution to improving a neurological-data processing pipeline, even if the statistical theory was not highly novel. I’m glad to know that applied thesis topics are appreciated here.
The department also chose not to run a drop-in consulting center like many others do. Instead, they form long-term joint research collaborations with other departments’ scholars.
Journal editorship is also valued at the department. Hopefully I can pick the many experienced editors’ brains in tailoring my publication submissions to the right journals.
Finally, there’s a strong focus not only on research but also on teaching, and today CMU has the largest group of undergrad stats majors in the US.

I’m looking forward to working with great colleagues in such an excellent environment!

Census-related links from June

A few links to share since I’ve been away:

Article on “An American Tradition: The U.S. Census Bureau Continues to Innovate in Data Visualization” (free access after watching an ad): A colleague summarizes some of the many mapping and datavis resources provided by the Census Bureau.

Two interactive web apps by data users (these use Census Bureau data but the datavises are someone else’s products):

Web app “Point Context”: a data user calls the Census Bureau’s API to find the distributions of age, race, income, and education for residents of the “average” neighborhood containing an arbitrary set of latitude-longitude coordinates.
Interactive map of “Is the United States spending less on public education?”: A Census data user practices with D3 and tries out the lessons from datavis classes — show comparisons, allow color-blind-safe color palettes, “catchy” headlines and informative annotations help guide readers, etc. I particularly like the arrow indicating where the selected state falls on the colorbar.

Several tools for making maps from Excel or spreadsheet-like tools:

Esri, the makers of ArcGIS software, have created a Microsoft Office add-on that lets you create maps of your data in Excel. A live demo looked promising, especially if your organization is already an Esri client… but otherwise ArcGIS is not cheap!
If you do have ArcGIS Online access, you can try using Esri’s “Story Maps” templates. Their published examples include this simple one based on Census Bureau data.
JMP, a SAS product, also has mapping tools that should be fairly simple for people used to spreadsheets. But again, SAS’s products tend to be expensive too.

Upcoming DataKind datadive with the World Bank in DC

DataKind (formerly Data Without Borders) is teaming up with the World Bank to host a datadive on monitoring poverty and corruption.

If you’ve never been to one of their datadives, here’s my writeup of last year’s DC event (which I thoroughly enjoyed), and DataKind’s writeup of our project results. These datadives are a great way for statisticians and other data scientists to put our skills to good use, and to connect with other good folks in the field.

The World Bank events will take place in Washington DC on two days: preliminary prep work on 2/23 (Open Data Day), and the main datadive on 3/15 to 3/17. Please consider attending if you’re around! If not, keep an eye out for future DataKind events or other related data science volunteer opportunities.

Small Area Estimation resources

Small Area Estimation is a field of statistics that seeks to improve the precision of your estimates when standard methods are not enough.

Say your organization has taken a large national survey of people’s income, and you are happy with the precision of the national estimate: The estimated national average income has a tight confidence interval around it. But then you try to use this data to estimate regional (state, county, province, etc.) average incomes, and some of the estimates are not as precise as you’d like: their standard errors are too high and the confidence intervals are too wide to be useful.

Unlike usual survey-sampling methods that treat each region’s data independently, a Small Area Estimation model makes some assumptions that let areas “borrow strength” from each other. This can lead to more precise and more stable estimates for the various regions (if the assumptions are reasonable).

Also note that it is sometimes called Small Domain Estimation because the “areas” do not have to be geographic: they can be other sub-domains of the data, such as finely cross-classified demographic categories of race by age by sex.

If you are interested in learning about the statistical techniques involved in Small Area Estimation, it can be difficult to get started. This field does not have as many textbooks yet as many other statistical topics do, and there are a few competing philosophies whose proponents do not cross-pollinate so much. (For example, the U.S. Census Bureau and the World Bank both use model-based small area estimation but in quite different ways.)

Recently I gave a couple of short tutorials on getting started with SAE, and I’m polishing those slides into something stand-alone I can post. [Edit: I still haven’t polished them, but I’ve posted my old slides and code.] Meanwhile, below is a list of resources I recommend if you would like to be more knowledgeable about this field. Continue reading “Small Area Estimation resources” →

Evil Queen of Numbers

Would there be any demand for a statistics class taught by M from the James Bond films?

M: You don’t like me, Bond. You don’t like my methods. You think I’m an accountant, a bean counter, more interested in my numbers than your instincts.
JB: The thought had occurred to me.
[…]
M: I’ve no compunction about sending you to your death, but I won’t do it on a whim.

Continue reading “Evil Queen of Numbers” →