After 3rd semester of Statistics PhD program

It’s time for another braindump of reflections on statistics grad school.
See also the previous two posts: After 1st semester of Statistics PhD program and  After 2nd semester of Statistics PhD program.

This was my last semester of required coursework. Having passed the Data Analysis Exam in May, and with all the courses under my belt, I am pretty much ready to focus on the thesis topic search and proposal. Exciting!

Classes:

• Let me elaborate on Cosma’s post: “Note to graduate students: It is important that you internalize that you are, in fact, a badass…”
  Ideally you should really internalize that you’re a badass before you come to grad school, because this is not the place to prove to yourself that you’re a badass. There are too many opportunities to feel bad about yourself at every stumble, when you’re surrounded by high-performing classmates and faculty who seem to do everything faster and more smoothly… It can be demoralizing when, say, you learn that you had the lowest score on an exam in a required class.
• On the other hand, now that the Advanced Statistical Theory course is over, I do feel much more badass about reading and doing statistical theory. I used to see a paper with a ton of unfamiliar math and my eyes would glaze over. Now I see it as: “Well, it may take a while, but I’m capable of learning to parse that, use it, and even contribute to the field.” It feels no more daunting than other things I’ve done. Thank you, Advanced Prob and Advanced Stat Theory!
  For example, I finally internalized that “hard math” is no worse than learning a new coding language. If I do an applied project and have to learn a new topic like Python, or parallel programming, or version control, it’s not an impossible task: it’s just a lot of work, like learning a foreign language. And I finally feel the same about math again: I may not have known what a Frobenius norm is, or my intuition about the difference between o(1) and O(1) may still be underdeveloped—but it’s not substantively different to get there than it is to keep track of the differences between for-loops in R vs Python vs MATLAB (like I had to do all year).
  Also, if I get stuck on a theory problem, it’s my own concern. I can read previous work on it and find a solution; or if there is none, I can write one and thus make a contribution to the literature. But if I’m stuck on an applied problem because I don’t have a codebook for the variables or don’t know what preprocessing was done to the dataset, I really am stuck waiting until the data owner responds (if he/she even knows or remembers what was done, which is not a safe bet…)
• I was a bit surprised by the choice of topics in Advanced Stat Theory. We covered several historically important topics in great detail, but then the professor told us that most of them are not especially popular directions or practically useful tools in modern statistical research. (For example, Neyman-Pearson hypothesis testing in exponential families seems to be a solved problem, tackled by tools specific to that scenario alone… So why spend so much course time on it?) Maybe the course could be better focused if it were split into two parts: one on historically-important foundations, vs. one on modern techniques.
• My TA assignment this semester was for Discrete Multivariate Analysis: advanced methods for contingency tables and log-linear models. I came away with a bigger appreciation for the rich and interesting questions that can arise about what looks, on the surface, to be a simple and small corner of statistics.

Journal Club:

• My favorite course this fall was the Statistical Journal Club, led by CMU’s Ryan Tibshirani jointly with his father Rob Tibshirani (on sabbatical here from Stanford). The Tibshiranis chose a great selection of papers for us to read and discuss. Each week a pair or trio of students would present that week’s paper. It was helpful to give practice “chalk talks” as well as to see simulations illustrating each paper. (On day 1, Rob Tibshirani told us he likes to implement a small simulation whenever he reads a new paper or attends a talk: it helps gain intuition, see how well the method really works in practice, and see how sensitive it is to the authors’ particular setup and assumptions.)
• I mentioned in Journal Club that we’d benefit from a MS/PhD level course on experimental design and sampling design for advanced stats & ML. Beyond just simple data collection for a basic psych experiment, how should one collect “big data” well, what to watch out for, how does the data collection affects analysis, etc.? Someone asked if I’m volunteering to teach it—maybe not a bad idea someday 🙂
• The papers on “A kernel two-sample test” and “Brownian distance covariance” reminded me of a few moments when I saw an abstract definition in AdvProb class and thought, “Too bad this is just a technical tool for proofs and not something you can check in practice on real data…” As it turns out, the authors of these papers DID find a way to use them with real data. (For instance, there’s a very abstract definition of equality of distributions that cannot be checked directly: “for any function, the mean of that function on X is the same as the mean of that function on Y.” You can’t take a real dataset and check this for ALL functions—but the authors figured out that you can use kernel methods to get pretty close, by checking a vast infinite space of functions. So they took the abstract impractical definition and developed a nice practical test you can run on real data.) Impressive, and a good reminder to watch out for that thought again in the future—maybe a second look could turn into something useful.
• Similarly, a few papers (like “Stability selection”) take an idea that seems reasonable to try in practice but without any theoretical grounding… (What if we just take random half-samples of the data, refit our lasso regression on each one, and see which variables are kept in the model on most of the half-samples?)… and develop proofs that give theoretical guarantees about how good this procedure can be.
• Still other papers (like my own team’s assigned paper, on Deep Learning) were unable to find a solid theoretical grounding for why the model does so well or any guarantees on how well it should be expected to do. But it seems like it should be tractable, if only we could hit on the right framework for looking at this problem. The Dropout paper had a nice way to look at the very top layer of a neural network, but not directly helpful for deeper networks.
• I got really excited about the “Post-selection inference paper” which discussed conditional hypothesis testing for regression coefficients. I thought we could apply it to the simplest OLS case to do some nifty new test that would let you make inferences such as: “Beta is estimated to be positive, and our conditional one-sided test says it’s significant, so it’s significantly positive.” You’re usually told not to do this: you’re supposed to decide ahead of time if you want a two-sided test or one-sided; and if it’s one-sided, then decide which side you want to check before looking at the data. However… after some scratch work, in the Normal case it looks like the correction you do (for deciding on the direction of the one-sided test after observing the sign of the estimate) is exactly equivalent to doing a two-sided test instead. (Basically you double the one-sided test’s p-value, which is the same as computing the two-sided p-value for a Normal statistic). So on the one hand, we don’t get a new better test out of this: it’s just what people do in practice anyway. On the other hand, it shows that the thing that people do, even though they’re told it’s wrong, is actually not wrong after all 🙂
  This made me wonder: Apart from this simple case of one coefficient in OLS, are there other aspects of sequential/adaptive/conditional hypothesis testing that could be simplified and spread to a wider audience? Are there common use-cases where these tools would help less-statistically-savvy users to get rigorous inference out of the missteps they normally do?
• A few of the papers were less technical, such as “Why most published research findings are false.” We discussed how to incentivize scientists to publish well-powered interesting null findings and avoid the file-drawer problem. Rob Tibshirani suggested the idea of a “PLoS Zero” 🙂 (vs. the existing PLoS ONE) He also told us how he encouraged PubMed to add a comment system, the PubMed Commons. Now you can point out issues or mistakes in a paper in this public space and get the authors’ responses right there, instead of having to go back & forth through the journal editors’ gatekeeping to publish letters slowly.

Research:

• Besides the year-long Advanced Data Analysis (ADA) project, I also got back in to research on Small Area Estimation with Beka Steorts, which led me to attend the SAE2014 conference in Poznań, Poland (near my hometown—the first time that business travel has ever taken me anywhere near family!). Beka also got me involved in the MIDAS (Models of Infectious Disease Agent Study) project: we are developing “synthetic ecosystems,” aka artificial populations that epidemiologists can plug into agent-based models to study the spread of disease. The current version is an EXTREMELY rudimentary first pass: I’ll write a bit more about the project once we have a version we’re happier with.
• I finally sat down and learned version control (via Git), and it turned out to be a good friend. For the MIDAS project we had three of us working on Dropbox, which led to: clogging all our Dropboxes, overwriting each other’s files, trying to coordinate by email, renaming things from “blahblah” to “blahblah_temp” and “blahblah_temp_2_tmp_recent” and so on… So it became clear it’s time for a better approach. Git lets you exclude files (so you don’t need to sync everything like Dropbox does); check differences between file versions; and use branching to try out temporary versions without renaming or breaking everything. I used the helpful tutorials by Bitbucket and Karl Broman.
• MIDAS also sponsored me to attend the North American Cartographic Information Society (NACIS) 2014 conference here in Pittsburgh. That deserves its own post, but I found it nifty that the conference was co-organized by Amy Griffin… whom I met (when she came to do some research on spatial visualization of uncertainty with the Census Bureau) via Nicholas Nagle… who first reached out to me through a comment on this blog. It all comes back around!
• As for the yearlong ADA project itself: it’s almost wrapped up, but quite differently from what we expected. There turned out to be major issues in getting and combining all the required dataset pieces: We needed (1) MEG brain scans, (2) MRI brain imagery, and (3) personal covariates about the medical/neuropsychological outcomes of each patient. Each of these three datasets had a different owner, and was de-identified for privacy/security… and we were never able to get a set of patient IDS that we could use to merge the different datasets together. In the end I had to switch topics entirely, to a similar neuroscientific dataset (which had been successfully combined and pre-processed) but for studying Autism instead of Epilepsy. This switch finally happened in the last few months of the semester, so I had just a short time in which to address the scientific questions in appropriate statistical ways, while also learning about a new disorder, and also refreshing my knowledge of MATLAB (since this data was in that format, not Python as the previous one had been)…
  Lessons learned: I should have been more proactive with collaborators about either pushing harder to get data quickly or just switching topics sooner. And for those stats students who are about to start a new applied project like this one, make sure your collaborators already have the full dataset in hand. (Of course, in general if you’re able to get in early and help to plan the data collection for optimal statistical efficiency, so much the better. But if you’re just a student whose goal is to practice data analysis, you’d better be sure the data has been compiled before you start.)

Life:

• Before coming to CMU, I always knew it as a strong technical school but didn’t realize how great the drama department was. We finally made it to a stage performance—actually Britten’s The Beggar’s Opera. I was wearing a sleep monitor watch that week, and the readout later claimed I was asleep during the show… It just noticed my low movement and the dim lighting, but I promise I was awake! 😛 Really, a great performance and I look forward to seeing more theater here.
• For a while I’ve been disappointed that Deschutes Brewery beers from Oregon hadn’t made it out to Pennsylvania yet. But no longer! I can finally buy my favorite Obsidian Stout down the street!
• Though I haven’t been posting much this fall, there’s been plenty of good stuff by first-year CMU student Lee Richardson. I especially like his recent post‘s comments about institutional knowledge—it’s far more important than we usually give it credit for.
• Nathan Yau is many steps ahead of me again, with great posts like how to improve government data websites, as well as one on a major life event. My own household size is also expected to increase from N to N+1 shortly, and everyone tells us “Your life is about to change!”—so I thank Nathan for a data-driven view of how exactly that change may look.

Next up: the 4th, 5th, 6th, 7th, 8th, 9th, and 10th semesters of my Statistics PhD program.