[Image via ICT4Accountability]

Yesterday and today, I attended the first ever Big Data class at MIT Sloan . The lecturers were Erik Brynjolfsson and Alex 'Sandy' Pentland. I'd previously heard Erik speak at MIT in October (when I first heard in-depth about the components of Big Data), and I've since read his book Race Against the Machine: How the Digital Revolution is Accelerating Innovation, Driving Productivity, and Irreversibly Transforming Employment and the Economy (highly recommended). I had high expectations, and they were ultimately exceeded.

It will take me a bit of time to catch up on everything I wrote down in 50 pages of notes. Due to a combination of no hotel wifi, Amex fraud false positives and Verizon order complexity, I had only my T-Mobile BlackBerry for connectivity on the first morning of the conference. For the first time in a long time, I took notes on paper throughout the class. This gave me mixed feelings. It was certainly nice to create diagrams easily, use different fonts and means of emphasis, create my own notation for action items and areas to research. However, now I'm left with a fragile notebook that I'm paranoid about losing and hours of transcription ahead.

Following a quick and dirty data mine of my own notes, here are some of the most interesting topics, insights, theories, and quotes from the day:


  • Balancing experienced gut vs data
  • Learning to discriminate correlation vs. causality
  • Effectiveness of different communication media for communicating and learning
  • Social graph patterns for creative vs cohesive groups
  • How to continuously run experiments and use Overall Evaluation Criteria
  • Stages of Organizational Evolution: Hubris, Measurement, Semmelweis Reflex, Fundamental Understanding.
  • Big Data Out in the Wild
  • Techniques for Building Viral Adoption
  • Email Analytics: Productivity and Information Diffusion
  • Privacy Legislative Issues in the US and Europe
  • Personal Data as Asset Class
  • The Matrix of Change

Insights and theories: 

  • Companies born on the web, such as Amazon, Facebook, and Google perform hundreds of experiments per day.
  • The Hawthorne Effect: letting people know that they are being experimented on changes their behavior.
  • Social metrics: between-ness, centrality, constraints, geodesic distance
  • Behavioral demographics (where you go, who you hang out with) are a more precise form of defining identity than iris scans or fingerprints.
  • Researchers are able to diagnose depression just by observing cell phone usage. 
  • The Panopticon: who needs a physical surveillance tower in the smart phone age?
  • Tuyman's Law: Any statistic that appears interesting is almost certainly a mistake.

Vox populi:

  • "A wealth of information creates a poverty of attention." -Herbert Simon
  • "When physicists have data that is too noisy, they build a better tool for finer resolution." - Erik Brynjolfsson
  • "Big data is a mental prosthetic." - Erik Brynjolfsson
  • "People are bundles of habits formed by the people around them." - Pentland (more so than a person's friends or peers).
  • "Go get the data! Don't argue about designs."
  • "To have a great idea, have a lot of them" -Edison
  • Lord Lever's Quandry: "Half of my marketing budget is wasted. I just don't know which half."
  • "Where you spend your time is who you are." - Pentland
  • "In Hong Kong, you'll buy everything but your house on your phone." -Pentland referring to the all knowing Octopus card.
  • "70% of all workers are information workers." unattributed.
  • "People care about privacy, but if you offer them an Amazon Gift Card, they will turn it over." -Pentland
  • "Gender predicts information diffusion, but not productivity" -Erik Brynjolfsson, from data on email analytics
  • "People being rational is an abominable model, but all economics is based upon it." -Pentland
  • "We are not concerned about data privacy. We don't give data to anyone except the government." Manager from China Mobile

I also learned that, as CEO, mine is the HiPPO (http://exp-platform.com/whatsahippo.aspx), ( term coined by Ronny Kohavi of Microsoft) which is fraught with danger for the organization.