Author: Christopher Surdak

Is Big Data a Big Bummer?

contributed by:Big data is why I'm sad.
Christopher Surdak, President & CEO

As an engineer, science fiction movies are usually a hit with me. Nothing entertains me like a space opera with lasers, robots and Homeric heroes.  But when I’m asked which movie is my favorite some that make my short list might be a surprise.  The Shawshank Redemption is one such movie. Set in the mid 1900’s and based on a novella written by Stephen King, this movie is about a man who, by a tragic turn of events, found himself serving two life terms in prison for murders he didn’t commit.

Rather than letting this hopeless situation crush his spirit, he sets himself on a forty-year-long mission to get out.  It’s a story that shows how your situation is not your fate. Rather how you choose to deal with your situation determines your fate.  If you are honest with yourself about your situation, and deal with it with hope instead of despair, you might generate surprising results in the end.

A Prison of Habit?

I recently had a discussion with two executives from a large financial firm.  We covered a range of topics, both internal and external to their company, from negative interest rates to ISIL, and from re-engagement with Cuba to their troubles with connecting with Generation Z.  As hither-and-yon as these topics may be, they are interrelated, or at least that’s what all of my research over the last decade tells me. These disruptions to the world most of us grew accustomed to are appearing in all aspects of our lives.  To me, this synchronicity is no accident.

Over the last three years I’ve given over three hundred presentations on Big Data, Analytics and Organizational Change so I’ve seen a lot of reactions from a lot of audiences.  The reactions almost always follow the Seven Stages of Dealing with Loss. According to psychologists these seven stages are:



From my hundreds of presentations on Big Data, disruption, and so on, shock is the most typical response I have seen.  Those who are shocked usually just leave the room, mouths open, eyes wide as saucers, a slight stagger to their gait. I have noticed that it is not unusual for me to finish my presentation, ask if there are any questions and no one will volunteer one, at least not at first.

However, when I stick around after a presentation, people will invariably approach me about half an hour later fairly bursting with questions.  That’s a good indication of shock: it takes people a while to process what they just heard, digest it a bit, and only then can they respond with a million questions on what they just heard. I’m generally sure that they’ll survive their initial shock when they’re asking questions faster than I can answer them.

Shock comes in Big Data once people realize how pervasive these tools and techniques already are, how far behind most organizations really are, and how difficult it will be to catch up to those with a head start. If you are here, there is much work for you to do.


The bulk of executives I meet with go with Denial: Maybe all of that change is going on, but we’re immune to it because INSERT SOME RATIONALE OR OTHER HERE.  They’re convinced that because of some specialness in what they do, change won’t happen to them.  And over the years, I’ve heard them all:  We’re too big, we’ve been doing this too long, we’re too regulated, we’re too smart, we’re too simple, we’re too complex, we’re too capital-intensive, our customers love us too much, and so on and so on.  Denial is easy because it costs nothing, at least for a while.  And, it is remarkably self-assuring to tell yourself that through your hard work you’re too-something for the outside world to affect you. Your competitors love it when you buy into the notion that you’re some kind of special snowflake, because you’re making it easy for them to annihilate you while you sit in your comfortable bubble.

Denial is common because it is easy, it doesn’t require much thought or effort, and it’s really cheap, at least for a while.


A small percent in the audience responds with anger.  However, those who are angered by what I present rarely approach me to discuss their anger directly.  Instead they let it fly on their surveys after the event.  After all, it’s easy to get in someone’s face when you’re not actually in their face.  Anyone can be courageous through anonymity.  When this happens they’re usually attacking me on style, rather than content.  I was too brash, I wasn’t PC enough, the color of my Starbuck’s cup was offensive, whatever.  I generally feel that everyone is entitled to their own opinion.  Variety makes us stronger.  So, when I get this sort of response I usually write it off as another example of people who believe in being ‘open minded,’ as long as you do it their way.

Anger in Big Data comes from the notion that everything people have worked so hard to achieve for so long is somehow wrong.  That somehow, everyone that has worked at business or technology for the last fifty years is somehow incompetent or ignorant, and that the changes we now face should have been knowable decades ago.  Anger with this viewpoint is fair, because this assessment is anything but fair.  People weren’t incompetent for the last 60 years.  Rather, they were so successful that their entire game has now changed.


Bargaining is the next phase of grief management, where the mind tries to make a deal with the universe, attempting to get a better outcome through karmic barter.  This almost always manifests as someone saying, “I retire in X years, I just want things to stay the same until then so I don’t have to deal with it.”  I hear that one A LOT!  Bargaining maintains your subconscious’ need for the illusion of control, rather than accepting that in this instance you have none.  Loss of control is very disquieting, so trying to trade a smaller loss for the one that you’re facing is an obvious psychic ploy.

Bargaining is the typical strategy for technical people. They frequently skip past all of the emotional mumbo-jumbo, and want to get right to solving the problem.  Or at least, what they believe to be the problem. You see bargaining when technical people start to offer less-offensive solutions to the problem at hand.  Sometimes these might actually work.  But, more often, they are palliative detours designed to make people feel like they can choose the degree of change required by the situation at hand.

In Big Data, you see bargaining every time a CIO or CTO claims that a new platform is the solution to everything.  You see it every time a CMO or COO gets a technology budget of their own.  You see it every time a CEO introduces their company to their new “Chief Data Officer,” who has no budget, staff or mandate other than to make this ‘data stuff’ go away.


A relatively few go with Depression; we are doomed and there’s nothing we can do about it. I try to console them with, “There’s plenty that you can do, it just might not be what you WANT to do.” Generally, when someone is at this point they’ve already found clear indications that their world has changed, and the results of that change are becoming clear.  Customers may be leaving in increasing numbers, revenues are down, profitability is tanking, your best employees are leaving, and so on.  Or worse, some upstart has already shown up in your market and is eating your lunch, Uber-style.

Depression sets in once you realize that not only have things around you changed, but you yourself must also change if you’re going to survive.  Passively accepting change is hard enough.  Full-blow depression sets in when you realize that you must respond to outside change by actively changing yourself.  Your task just got at least twice as hard, and it’s a bummer, isn’t it?

In Big Data, depression comes when you realize that your hyper-expensive, 23 wonkabyte, 10,000 node Hadoop cluster didn’t magically solve every problem your business has ever had.  It comes when you realize that the “data scientist” you hired off LinkedIn didn’t have a PhD in statistics, they had a PhD in the Appreciation of Statistics, and a minor in creative writing. It comes when you realize that your new Blockchain marketplace is just as hackable as every other “hack-proof” technology that has ever come before, in the six millennia that people have been trying to make information “Hack-Proof.” And depression comes when you realize that everything that your own compliance department said that you couldn’t possibly do was just done by six college drop-outs living on a boat anchored 12.01 nautical miles off the coast of the United States.

As we enter what Gartner Group calls the “trough of disillusionment” with Big Data, expect to see a lot more depressed people wondering about wondering what to do next.


Testing is where we begin to explore the possibility that what has happened is not going to undo itself, and what might that reality entail. We attempt to see what our new world might look like, and try to see ourselves in it.  Our minds do this all the time.  When we chose our preferred narrative, our preferred response and our desired outcome we call it daydreaming. Day dreaming is fun, because we are in control.  Testing is not as much fun, because we don’t control the narrative, our response is what we hope is the best compromise, and the outcome is very much in doubt.  Nonetheless, reaching the point of testing is a good sign because you’re returning to your present self, and you’re preparing to deal with change rationally and productively.

In the arena of Big Data disruption, Testing starts to show itself when business leaders start to brainstorm how they could change, what those changes would entail, and what benefits might accrue.  If these thoughts are joined with a healthy dose of “and here’s the pain of not changing,” then there’s a much greater chance of success in whatever steps you choose.


Acceptance feels like a release, because it is.  It is a release of all of the doubt, hurt, fear and other negative energies that cause us to freeze in the first place.  There’s a freedom in finally acknowledging change.  There’s a clarity that comes from finally accepting that you must change, and that you have no alternative.  Making a choice, even if it is one you don’t want to make, frees up enormous psychological energy.  Rather than focusing on methods of prevention or escape, you focus on how to succeed with your new reality.

When I wrote this I thought to myself, “Is this true? What about the Shawshank Redemption” Andy Dufresne seemed to never succumb to his reality in prison.  Andy never accepted that he was a convict, and struggled to regain his freedom. Superficially, he never gave up hope that he would get out. But, look closer and you will find that Steven King had a deeper thesis than this. In the story, Andy does indeed go through the seven phases of dealing with loss (of his freedom).

At the end of the movie, when he is finally looking for the strength to follow his escape plan he only gets there through acceptance.  His last statement to his friend Red is, “Get busy living or get busy dying.”  This scared Red to no end, as he interpreted this to mean Andy would kill himself.  But, this was not the case.  Andy wasn’t ending his life, he was ending his denial and indecision over his situation.  He came to accept that if he didn’t make a risky choice, then he was doomed by the world around him.  What appeared in the movie to be defeat and resignation was actually acceptance.  And this acceptance finally gave Andy the courage to take control of his situation, and reclaim his life.

This movie, The Shawshank Redemption, was not a story about how people suffer from injustice, it was a story about Andy chose to fix injustice by acceptance and action.

Holistic Medicine for Your Data

Wherever you are in your process of dealing with Big Data-Induced Grief, we are here to help.  We know what you’re going through because we’ve gone through it, too, and we took good notes along the way.  Surdak & Company won’t keep you from getting scars.  After all, you are crossing a chasm filled with digital thorns and sharing-economy dragons.  But, most of us have already earned our first aid merit badges, and we are well-stocked with Bactine.





Dubai: A City of Jerks

originally published at The European Business ReviewDubai

contributed by:
Christopher Surdak, President & CEO

In my new book, Jerk: Twelve Steps to Rule the World, I discuss how companies such as Uber, Airbnb and Simple Bank are using new business models to disrupt existing industries.  These new innovators cause disruption, or “jerk”, by breaking with the traditional rules of capitalism and embracing Information as the new source of wealth and power in our world.  Combining information with new consumption models such as sharing and bartering economies, Jerks create enormous value for end users by better utilizing present investments and creating increasing information wealth along the way.

Recently, my team and I spent several days with a delegation from the Department of Economic Development of the Government of Dubai. This group, led by H.E. Mohammed Shael Al Saadi, and Mr. Wael Osman, is tasked with driving Dubai to the forefront of the global information economy. Their mission was to visit with experts on Big Data, Security, Blockchain, Smart Cities and the Sharing Economy in several cities across the United States and explore the latest innovations in these areas. Through a comprehensive group of initiatives, Dubai is seeking to jerk or disrupt the way that information is created, stored and utilized around the globe, and to do so in a way that simultaneously emphasizes speed, openness, security and value.

Dubai’s goal is not to become merely a smart city, but rather to become the smartest city, and a model for all other cities around the world.  This is an enormous undertaking that would take any organization decades to implement.  For a government to undertake this effort, with the goal of completing this transformation ahead of Dubai’s 2020 World Expo is truly monumental.  But, as stated by Al Saadi, “We have a God-given ability to fix the challenges that present themselves.  The key is to choose to change and then to do it.  Only then can you find and fix the issues that arise.”

An Open Road to Success

The effort, driven by the Smart Dubai Office, began in March 2014 when His Highness Sheikh Mohammed Bin Rashid Al Maktoum, Vice President and Prime Minister of the UAE and Ruler of Dubai, launched the Smart Dubai Initiative within The Executive Office.  A small initiative within a government office was then turned into the Smart Dubai Office, lead by Her Excellency Dr. Aisha Bin Bishr as Director General, with the governance structure and authority to unite public and private sector leaders to deliver an efficient, seamless, safe and impactful city experience for residents and visitors.

To achieve its strategic pillars, Smart Dubai has introduced strategic initiatives and developed partnerships to contribute to six key dimensions: Smart Economy, Smart Living, Smart Governance, Smart Environment, Smart People and Smart Mobility.

This effort is firmly rooted in the very latest in technology advancement.  However, Smart Dubai is taking its city transformation mandate beyond technology and into measures of actual happiness. As discussed in Jerk, as part of the Six New Normals, the Smart Dubai leadership team is guided by the conviction that “technology is only a means to an end; our end goal is people’s happiness.” To fuel this city transformation to happiness, the Smart Dubai Office recently launched its Happiness Agenda, adopting a globally unique, science-based and methodical approach to impacting happiness for the whole city. Under the Happiness Agenda, Smart Dubai is introducing a framework to ensure that individuals’ happiness, satisfaction or well-being is factored into leadership’s decision making on important city projects.

One initiative within the Happiness Agenda is the so-called ‘Happiness Meter,’ a customer satisfaction metric that was launched by Smart Dubai in October 2014, and is now in use by upwards of 40 government departments at service centers, websites and other ‘interaction touchpoints.’ Results from the meters feed into the ‘Happiness Index,’ a real-time happiness score for the city. City leadership and heads of government departments are able to call up the ‘Happiness Index’ for the city, or their department, through a mobile app.

This May, the Happiness Meter was opened to the private sector as part of the city’s Open Data initiative. Emirates NBD, one of the largest banks in the city, has already adopted the system for its mobile banking services. Upgrades to the Happiness Meter functionality and analytical capabilities are currently underway, with new features scheduled to launch later this year. This is a prime example of how government Open Data initiatives can have a direct bearing on private enterprise, leading to entirely new solutions for residents and consumers.

‘Do, Then Learn’

As discussed in Jerk, “Do Then Learn” is one of the principles followed by successful innovators.  Dubai is actively embracing this approach, as demonstrated by their enactment of their new Dubai Data Law, late in 2015. This law required all government departments to open all of their data to each other and to the public. With the force of this decision by His Highness Sheikh Mohammed Bin Rashid Al Maktoum, Dubai has implemented its Open Data plan with a speed and certainty of purpose that is often elusive for government entities. As shared by Al Saadi, “We were certain of the need for smart, decisive action.  Anything less would be unacceptable to us.” “We knew there would be challenges, but we could not know and address them until we actually began to implement.”

The Dubai Data Law called for the establishment of a governing body to oversee the implementation of the law. In accordance with the law, the Dubai Data Establishment was announced in early 2016, under the Smart Dubai Office. Data is the fuel of the city’s smart transformation, and the two organizations are closely aligned. Assistant Director General of the Smart Dubai Office, Younus Al Nasser, is also the CEO of the Dubai Data Establishment.

This is not the city’s first digital transformation. In 1999, His Highness Sheikh Mohammed launched Dubai e-Government, a new department charged with digitizing all government operations. Setting the stage for Dubai’s current ambitions, Dubai e-Government was tasked by His Highness to “make Dubai a digital government in 18 months” – a remarkably short time table that was met.

The Dubai e-Government department continues to operate today, although it has evolved into the Smart Dubai Government Establishment. “SDG” operates within the Smart Dubai Office, under the leadership the CEO, Wesam Lootah. SDG is the ‘technology arm’ of Smart Dubai, bringing more than a decade’s experience developing, implementing and enabling smart services for government departments.  SDG recently published a report quantifying its impact on the government – over 1 billion USD saved over the past 13 years through the use of shared smart services and infrastructure.

SDG are the creators of DubaiNow, a mobile app which aggregates all government services into a single customer interface. Over twenty government departments are already participating in DubaiNow, in addition to several private sector companies, such as telco’s and health providers, and NGOs.

Smart Dubai Government is also the custodian of the Smart Dubai Platform, the ‘digital backbone’ of the city. Unlike most other smart city platforms on the market today, Dubai’s platform was co-created by Dubai leadership and a local ICT provider to fit the exacting requirements of the Dubai’s smart city mandate. The Smart Dubai Platform will ‘orchestrate’ Dubai’s data, and will become the public face of Open Data for the city. Current timelines suggest that the first open data sets will be available to access through the platform in early 2017.

By opening the city’s data for all to use, Dubai creates a tremendous engine of innovation. Before Open Data, it would be difficult for anyone to analyze how the city operated, where there were issues and how they could be improved.  With Open Data anyone can look at the city’s vital signs, its rhythms and moods and look for new ways to improve all aspects of life in Dubai. Open Data allows anyone to participate in this process, creating an environment of competition and innovation unlike any before.

As radical as this change may have been to Dubai’s departments, it is just the beginning.  Dubai’s leadership, including Abdulla Al Madani, chairman of the Dubai Open Data Committee, recognize that it is not enough to make the city’s data open.  This data must be shared by both public and private organizations in order to produce valuable outcomes for the people of the city.  Shared data is the real goal, allowing everyone to participate in the information economy. Creating an infrastructure, architecture and standards for such sharing are the key next steps to realizing this vision.

Securing a Public Resource

Shared Data is a great step towards Dubai’s digital future. But even this is not enough.  Shared data can be both a vulnerability and an asset, hence securing this data is imperative.  Users must be certain of the validity and accuracy of the data that they intend to use.  Hence, a state of the art security architecture is being implemented to protect this public asset.

As challenging as this may be, it is far easier to secure open, shared data than to secure how we operate today.  As explained by Al Saadi, “With the old way of managing data, anyone in the government could be an inadvertent security leak.  A photocopy of a license, application or photo could be left on a desk or thrown away, and a moment later be in the hands of someone wanting to use that information to do harm.  With Open Data, attempts at falsifying or stealing data are known to all, and errors or breaches can be prevented from occurring. With Open Data securing people’s information is much easier, rather than harder.”

Open, Yet Private

This leads to the natural question of privacy. Will Open Data still protect the privacy of individuals? In today’s world of smartphones, apps and social media, the concept of privacy that has existed for the last half-century is largely gone.  Being an invisible resident in a city of millions may have been a reality for many decades, but the Internet and the mobile revolution has brought such anonymity to an end. In this digital era we are returning to a time before the telephone or the computer or the car, where everyone knew each other and were neighbors. People still had their privacy, but much of their lives were spent as part of the community.

With Open and Shared Data we are reviving this sense of oneness and community, only now with millions or even billions of friends, family and neighbors. In this new world, the parts of our lives we wish to keep private will still be protected and respected, while the parts that we share will be used to benefit the whole community. The critical factor is choice, and the leadership of Dubai is ensuring that its people can choose what they wish to keep private, and what they choose to share. This will include anonymizing data, selective participation or opting out where this choice is deemed most appropriate.

Much of this privacy framework will build upon Dubai’s privacy law enacted in 2007, and another update to this law is expected late in 2016.

Welcoming the World

As deployed, this combined solution will enable innovative business models such as resource sharing, capital, labor and expertise bartering, and talent arbitrage, all enabled through the effective use of data.  As these capabilities are launched in Dubai it is hoped that the combined solution is so advanced and its utility so compelling that people worldwide will choose Dubai as the hub of all of their online activities. As Dubai has set itself as a global trade and travel hub, it will soon also be the center of the world of online commerce and living.

Backwards Compatibility

Dubai’s vision is so advanced that it leads to a problem of a different sort; how can others keep up? Indeed the online community and marketplace envisioned by Dubai is so far in advance of what is currently available that many people around the world may find it difficult to bridge the gap.  Again, Al Saadi and his team have anticipated this challenge, “We must anticipate the difficulties people from other cities may have in catching up to where we are heading.  We must provide a sort of backwards compatibility, as you see with computer operating systems, so that people who want to join us are able to make the leap ahead without too much difficulty.”

This goes beyond defining technical gateways such as APIs or standards.  It may include defining regulatory and legal frameworks  that allow for the free flow of open data from and between societies that have not yet caught up to Dubai.  Providing for these bridges now ensures that all cities may benefit from what Dubai learns as it moves forward with its vision, ensuring that it stays at the forefront of this new economy.

The Will to Change

Without doubt, the vision created by His Highness Sheikh Mohammed Bin Rashid Al Maktoum is a great leap forward.  many others talk of bold steps, but Dubai has the will to act. There is no illusion that this journey will be easy or that there will not be challenges along the way. But, as I have discussed in my writing elsewhere if you are not challenged by the unexpected or unanticipated you are likely not really innovating at all. To reach a bold new future you must first decide to make the trip and then face each unexpected challenge with the best problem solving you can muster.  This is the path towards new worlds, and by all appearances the people of Dubai may be there waiting for the rest of us once we arrive.