Big Data: Government Surveillance

The big news at the end of this last week was that the US government has been collecting internet and telephone data in order to combat terrorism.  Is anyone really surprised? 

I have been involved in what can be done with big data since 1987.  I never worked with government but only commercial businesses that had large amounts of data in industries that were undergoing tremendous change.  Our biggest customers were telephone companies, large retailers, airlines, and banks.  As you may recall, the 1980’s began a period of deregulation.  AT&T agreed to be broken up into separate units in order to foster competition.   The Airline Deregulation Act of 1978 took the federal government out of setting fares and determining routes.  The financial sector saw the consolidation of banking and insurance and the repeal of Glass-Steagall.  The change in retail can be attributed to one firm: Wal-Mart dominated its category and influenced all of the industry with a big-box format, sourcing from low-cost manufacturers across the world, every day low prices and aggressive expansion.

What did all the firms in these industries do with all this data?  Typically they figured out who their customers were.  Prior to these changes either they didn’t care because they were operating in a monopoly environment and didn’t need to cater to customers.  Or, they operated at a small enough scope that the managers of the stores knew who their customers were from personal interaction.  However, once deregulation and consolidation occurred they couldn’t rely on personal knowledge; these businesses had to rely on data. 

The volume of data and the speed with which they processed it kept increasing.  They modified the capturing of the data at the customer touchpoint so that it could be collected in more detail.  They also introduced loyalty cards to induce customers to continue shopping or flying with them; but, more importantly, the loyalty cards allowed them to know when a customer came back and what they were buying; and, they got contact information for these individuals.

In other words, they could identify their customers; they knew where they lived, what their demographic are, who lived with whom.  Companies knew what they bought and when they bought it.  With telephone customers, they knew who was calling whom, how long they communicated, how often, and at what time of day these communications occurred.  The financial institutions knew your assets and how you spent your money.  The transportation firms knew where you traveled, when, and how long you stayed.

Then in the 1990’s the mobile phone became ubiquitous and the internet became consumer oriented with the changes derived from telecommunications deregulation.  We could communicate anywhere with the phone, web, email, text and tweet.  We could find out about anything from anywhere.  We could buy anything from anywhere.  The amount of data exploded and the amount of change to our lives was massive.  As Thomas Friedman stated so well in The World is Flat, the world became globalized.

The second thing these firms evolved to doing with all this data was predicting and then influencing what a customer was going to do.  They know what products and services you will likely buy.  They know when you’ll make this transaction. They predict how much bandwidth, airplane seats, teller machines, etc. will be needed.  Retailers know how much inventory they will likely need for a product that has never been sold.  On Black Friday they know within a few hours of stores opening on the East coast whether their predictions about retail customer behavior is accurate or not and then modify pricing before Denver stores even open.  They can instantaneously customize the webpage and the products offered to you based on what they know about you and your past behavior. Online advertisements are presented instantly to you because of where you’ve browsed in the past, who you’ve friended, and what you’ve liked.

Which leads me to my first question to you.  We have given very personal information about ourselves to the businesses we choose to interact with.  They know us intimately and seem to adapt to our whims instantaneously.  Now, place yourself in the federal government as someone responsible for defending us against terrorism.  Since the Oklahoma City bombing almost two decades ago, we have had a torrent of people who want to fly airplanes into iconic buildings, bomb public events and disable our digital economy.  Would you not want to use this information available about individuals to determine their nefarious whims?  Why should you be surprised that it has occurred?

When this kind of information is available and used openly in the commercial sector for twenty-six years, I can only assume that a government would use it, as well.  I admit I don’t like the thought of it with my libertarian ideal of privacy.  However, I like my frequent flyer miles and the discounts from using my loyalty card at supermarkets; I sell stuff on Craig’s List and I buy from Amazon; I use LinkedIn to market myself.  I use these things because I perceive them to be in my interest.  But, if I knowingly relinquish my privacy to an internet search engine, I am relinquishing this information to government, too. 

In some ways, its similar to the argument I made in my post on GMOs; in that case, if genetic material is made available to the environment, it will spread and you cannot predict how or where.  Genetic material is a code; it is information.  Likewise, any digital information, once disclosed on the internet, is going to spread; you cannot predict how its going to be used nor where it will be used.  Maybe it’s a law of nature that information, genetic or digital, cannot be contained.