Ethan Blogs on BI Archive

I know that I shouldn’t be amused by these sorts of things, but I can’t help myself. Recently our good friends at HP announced that they are writing off $8.8 billion (yes, B, $8,800,000,000) worth of a company that they bought a year ago as the value of the company was severely overstated and there was, according to HP, “serious accounting improprieties” and “outright misrepresentations”, before they bought Autonomy for over $10 billion in October 2011.

Now my heart breaks for the shareholders of HP and if there was fraud involved I trust that the legal authorities will make it all right…but I suspect that the very large $8.8 sum is very much gone. I have some observations to share and point out for everyone else who is considering financing someone else’s retirement to a tropical island.

First of all, way back in 2011, everyone said that HP was paying way too much for the company. Other large tech companies such as Oracle went out of their way to point out that the price HP was paying was way too high and the products were way too green. The dream of magically printing money using their technology was just too good for HP to walk away from…but it was also just too good to be true.

Secondly, does anyone remember what Autonomy does? They were a BI (Big Data to be more specific) company based in the UK whose claim to fame was that they had mastered some very complex data extraction technologies of unstructured data streams (pictures, Twitter, Blogs, you name it) and then applied some fancy statistics to find bad guys for the government and good customers for businesses.

They had a healthy roster of very good clients that most companies in this sector would die for and they were at the top of a buzz word industry. What could go wrong? Well, I suspect that there are a lot of people asking themselves that very question right now. The part that I find most amusing in a schadenfreudeian way is that one of the advertised use cases of the Autonomy toolset is to stop fraud and unexpected write-offs from happening. How’d that work out for you HP? Sounds like it is time to review the marketing material!

Ultimately the courts and lots of lawyers will try to sort this all out and all of the involved consultancies and auditing firms will write big checks to HP but those of us not on the hook for this should still learn the most important lesson in this mess. If something is too good to be true, it probably is.

Sounds like something my Grandmother would say…wait, she did say that. So, to put it in more relevant terms, no matter how much your models suggest that something is a good deal, don’t assume that it is. Good executives put just as much energy into NOT doing something (i.e. showing that the new investment is a bad idea) as they put into doing it (i.e. showing the shareholders and board how good of an idea it is).

Good executives look not for reasons why projects and acquisitions will succeed, but rather why they might fail and what that post mortem will look like in advance to make sure that they actively address issues before they come up and develop mitigation plans and strategies while they have the ability to change things.

Momentum is a part of the problem in this case as well, it appears that HP tried to get out of the deal late in the game but couldn’t as they had been contractually bound and were unable to find sufficient reasons for exiting without paying big penalties. They obviously needed a better bail-out mechanism to save the company, oh I don’t know, a few billion dollars.

Let’s not pretend that anyone is above making these mistakes, but that doesn’t mean we should ignore them either and pretend that they are a cost of doing business. So for all of you movers and shakers out there, here’s to making sure that we not only source the right data, but also use our good management and decision making strategies with it.

Today’s goal is to find out who reads my blog posts.  So here it goes: Self-Service Business Intelligence is an outdated, over-stated and fundamentally flawed concept.  Those using this term should be mocked appropriately.

Ok, now let’s see if I get any comments!

I was going to end the blog there but my better half suggested that I at least explain myself before heading off smugly to my next philosophical rant.  So here it goes.

When I first started working in the corporate world many moons ago, I was in the accounting department of a fortune 500 company doing billing, AR and AP.  I was a data consumer at the lowest level of the proverbial food chain; so I was given piles and piles of reports every day and they were monsters.  These reports were spit out of an old, massive printer on wide sheets of alternating green and white background paper affectionately called (I think it was affectionately anyway) green-bar.

The green-bar reports were written by the IT professionals in the mainframe system and the reports ran every morning regardless of the day’s prior activity and even on holidays.  Monday morning was always fun as we had a huge pile to dig through and we ended up throwing away 50 pounds of blank paper before starting the day.  Ultimately these reports were processed/analyzed and the resulting values typed into Excel (or Lotus, we had both for some reason) and then dutifully filed away as they could not be reproduced.  Errors with the data were identified, corrected in the system and we went on our merry way.  Anyone want to guess why I eventually left accounting?

Anyway, we got a new system that was client server based and we actually had the ability to run our own reports.  These reports were also prompted so that we could just get the data we were looking for based on date, cost center, account number etc.  Life was good!  The term they used for this was…self-service.

Now this term made sense for me.  I had worked at McDonalds for almost 6 years during high school and college and self-service meant that people could go up and get the drink that they wanted from the soda machine and they didn’t require my help to get it.  This made them happy that they didn’t have to wait for me and it made me happy that they didn’t complain about too much ice.

Back to accounting though, we still had one issue that didn’t get any better and that was that changes to the reports still pretty much took an act of Congress. This one little issue, the painfully slow process to get new reports or to make changes to existing reports, caused us to still use Excel just as much as we did before.  The one positive was that fewer trees died with that seed we started to get a vision…a vision of control!  Pretty soon the idea of self-service morphed into one of the business having control over the entire process and being able to write and create their own data extracts… to put into Excel of course.

In reality though, at that point I wouldn’t be getting self-service anymore, instead I would be an IT person working in accounting.  So we have a dilemma now, the IT people see self-service as a chance to get the business out of their hair and just be admin and support people, the business see’s self-service as a model for getting rid of unhelpful IT resources.  Guess what?  Both teams need each other!  Trying to design a system to exclude your business partner is silly.

In the 1990’s, a restaurant chain tried the concept of having their customers grill their own steaks and guess what, it went out of business.  It may have sounded like a good idea, but people want their meals made by a professional who knows what they are doing.  Sometimes getting their own drinks can be nice, but when was the last time your waiter at Ruth’s Chris told you to get your own refills?  What kind of BI shop do you have?  McDonald’s or Ruth’s Chris…what kind are you aiming for?

Certainly technologies and techniques have come a long way over the years and now customers have  access to environments which allow them to truly interrogate and play with the data to get what they want in tools other than Excel… but this doesn’t happen without dedicated, knowledgeable IT professionals working with knowledgeable and dedicated business users.  In the SAP BusinessObjects suite of tools for example, Universes insulate consumers from the complexities and gyrations that must be dealt with to get their data in a reportable format.  But even with a “perfect” Universe, work still must be done in the reports.  There simply is no such thing as a completely self-service environment.

The goal of every IT group should therefore not be to give their customers “self-service”, which is now defined as a model where the IT department just creates the ecosystem and then lets the customer go from there.  That model is sort of like the customer getting fed up with their food cooked incorrectly and then going back into the kitchen to cook it themselves.  “Sure, we’ll provide the stove, but the rest is up to you, so I don’t have to hear complaints”, says the manager.   No, the correct solution is to take the chef to the customer’s table, get everyone in agreement on exactly what they want and then she goes back and cooks it.  Unless you think that either your customer is too dumb to tell the chef what he wants or the chef isn’t smart enough to cook the meal, then you should never hand the customer a sauce pan.  Hand the customer salt and pepper, sure, but be realistic!

Ultimately, the customer wants access to good clean data presented in a way that makes sense and allows for straight-forward manipulation and analysis.  Professional BI developers need to sit down, side by side with the business and determine what is needed and how best to get it…simply trying to pass the buck to the customer doesn’t end up working for anyone.

The bottom line is that I’m not advocating a return to green-bar.  I’m just suggesting that the tools have already created a self-service BI environment and pushing that as a goal will lead to failure for everyone, so just don’t go overboard; the goal now, as it always should have been, is to make our customers happy not to make them IT people.

Comments?

For those of you not fortunate enough to live in the beautiful desert southwest you might not have had the honor of knowing of the tree that is the bane of my existence.  It is the Palo Verde.  I believe that it is Spanish for “Cursed Tree From the 7^th Level of Hell” that or green stick…one of the two.  Yes, it is actually the official state tree of Arizona for some strange reason.

Anyway, this particular tree has some very specific adaptations to allow it to thrive in the desert.  First, it requires very little water and actually thrives in extreme temperatures.  As a matter of fact the one in my yard, or El Diablo as I like to call it, doesn’t seem to die no matter how often I “forget” to water it.  In addition to being very resilient to the heat and drought it also has thorns which grow approximately 25 feet long and are needle sharp to ward off predators or bad gardeners. The speed at which it grows is nothing short of amazing and the long thin limbs it grows have such a springy strength to them that they often slap back and will hit you in the face while trying to cut them.  Don’t worry though, the wood does dry out when cut from the tree and is no longer springy…in fact, when it dries out it, it becomes as hard as steel and can easily puncture a shoe or a car tire.  You can see this small branch that I didn’t pick up the last time I trimmed the tree and can imagine how painful it is to step on.

You might wonder why I have this particular tree in front of my house.  I wonder the same thing.  The tree was planted by the builders who did so as it also grows very quickly and looks green from a distance year round.  Needless to say they like these trees so much that no neighborhood in Arizona built within in the last 10 years is missing this particular delightful vegetation.

This brings me to my bi-summer blood ritual in which I have to prune El Diablo back so he stops clawing at the kids as they ride their bikes past the tree on the sidewalk.  I know that I should clean it up more often than I do and I’m sure that it would make the experience more pleasant if I did but what can I say, it isn’t my favorite bit of outside work.  In fact, the occurrence I’m talking about today actually took place in late July but I have been too injured and traumatized to write about it.

This past clearing was particularly painful and it was only partially my fault.  We were having friends over and I continued to put off the trimming until it was just a few days before they were going to be here and I had to trim it.  So I went outside after the sun went down (it was July remember) and before I went out I checked the temperature…114.  Yes, sun down, 60% humidity, 114 degrees.  Me and El Diablo.

You might be wondering what the final score was but you’ll just have to take my word for it, I was too embarrassed to take a picture, but it was better than I found it.  El Diablo was cut down until the kids could walk down the sidewalk without being attacked but I did run out of space in the garbage for branches.  He probably won the war though as I was full of holes in my hands and arms, one in the bottom of my right foot; in addition to the two broken-off pieces of the thorns which took a week to work their way out of my hands.  I only had one major scratch on my face which was an improvement from the spring trimming.

I won…I think.  Don’t worry; I’m going to trim it up before it gets so out of hand next time.  I promise.

So in case you were wondering why I am going on about El Diablo it is simple.  I was trimming it up and taking these stabs to my hands and arms and thinking about how much I’d rather be working.  As I was contemplating my current projects I was forced to recognize that I was not taking my own good advice.

Any time I go into a customer project I remind everyone involved that they should plan for maintenance on what they are building.  It is so hard to pull off though.  It isn’t just me of course.  We are all bad at planning beyond the current emergency.  It is so easy to focus on the immediate needs and just put in “temporary solutions” knowing that they will be cleaned up later.  Of course the problem is that the temporary solutions become permanent and they end up causing more work, more blood, more pain than if we had just built into our plan a little time at the end of the project to do clean up and maintenance.

In teaching BI for Executives, I always give the advice to put aside a few weeks every year to perform cleaning and maintenance in the systems you own.  I think it is good advice, but it is difficult to make happen.  For instance, how many reports do you have in your environment that aren’t being used?  Universes?  How about tables and ETL?

If the plan is just to wait until a project comes along which will replace these tables/ETL/reports/universes how much additional work are you going to create as the people who originally wrote these and know how (or even if) they are being used are long gone or certainly the memories are.  The normal prescription for these problems is to document everything and that will leave you with a map of how to deal with the eventual maintenance.  The problem with this is that the documentation is out of date before it is finished and just like my tree, knowing exactly which branch you should cut and where is a lot easier to know when you go after it on a regular basis and not at night when you have guests coming over the next night and the garbage truck coming to get your trimmings in 10 hours.

My experience has been typically that project teams end up even making the problem worse when confronting these issues because rather than doing something with the old universe/tables/reports/ETL they just build everything from scratch because of the unknown impact of touching the old stuff and there is never enough budget or time to deal with it.  Even with technology upgrades the default behavior is to import everything and figure it out later.  So now instead of one bad situation, we have two.  Don’t worry…we’ll get to the old stuff…in the next project.

My suggestion is the same I have for myself and El Diablo.  We all have schedules and it is incumbent upon us to put our highest priorities on there to make sure that they get done.  We must schedule, plan and prioritize maintenance outside of projects to clean things up.  That requires some sales to management and some hard decisions but this still must be done regardless of your company size or situation.  Don’t we make time to brush our teeth in the morning no matter how tight our schedules are?  If we put this maintenance off we might save a few minutes in the morning but we’ll pay for it in hours, dollars and pain in the dentist’s chair.

There are tools which can help but even those require a person to run the usage statistics and make phone calls and find out if what you are looking at is really not needed or just not needed often.  Call up your local friendly consultant if you need some advice on how to do this in your particular environment or at least sit down and figure out what maintenance is needed and can be scheduled on a regular basis in your organization.  If not for the sake of your developers do it for benefit of your poor customers who really aren’t sure why there are 10 shipping reports by region either.

I’m preparing for an upcoming business trip that I’d like to share with everyone. I will be at The Minerals, Metals and Materials (TMS2012) conference in beautiful Orlando, FL next week. The conference topics are exciting for me personally as well as professionally. My first job out of college (where I spent 10 years of my career) was with a metals refinery and extrusion plant that is a part of a major copper mining company based in the US. This is not just where I cut my teeth in business and IT but also where I was able to develop an understanding of the complexities and difficulties of running a business. These skills served me well, and in the ensuing years while I’ve been an employee at InfoSol I’ve had multiple opportunities to work in manufacturing and mining across the globe.

Even though this space is very familiar to me, I have been worried about how I can best deliver a short, easily consumed pitch to tell people at the conference what Business Intelligence is and how it can help their businesses. Those of us who have been in BI or IT for years might think the question a little blasé, but many of the people at this conference will be scientists, engineers and technicians who view IT as either a maintenance function or even worse as part of the problem keeping things from getting done. They all understand statistics and the art of analytical research so they know part of the BI world, but what about the rest of it? What can I tell them when they ask what I do for a living? I do a whole lot more than pour over data sets and try to extract meaning, much more than put together pretty reports. How can I explain all of this to people who don’t care about the tools that sometimes define my field?

When you do the typical Wikipedia search on Business Intelligence you see a few general definitions but they are quite frankly contradictory. One of the biggest problems is with the limiting word “Business” which tends to bring to mind the ideas and concerns of accounting, purchasing and other “Business” areas.

This is obviously an incomplete definition and I prefer the description from Forrester research group: “Business Intelligence is a set of methodologies, processes, architectures, and technologies that transform raw data into meaningful and useful information used to enable more effective strategic, tactical, and operational insights and decision-making.”

I like this definition better for a number of reasons. One, it more accurately describes what I do every day, and it also helps us describe a very complex world where we have a lot of heterogeneous data streams which need to be synthesized, cleaned and ultimately presented in a way and time that will help people make good decisions.

I think I’ll have to focus on 3 major misconceptions and biases which will keep people from seeing what we have to offer.

The first misconception that I will surely run into is that people will describe us as, “those people who write reports”. While report development is an important part of what we do it is ultimately a small, albeit visible, part of the proverbial puzzle.

Dashboards and reports are the icing on the cake of our entire processes. The final part, but still the icing! And, it often outshines the complex foundation that is in place to allow this final presentation layer to be useful. Our real focus is on decision support and all of the work that is required to allow for good decisions to be made with data is our domain. Our tools support technical activities which allow for data movement/management, cleansing, aggregation, consolidation, presentation and let’s not forget distribution! The youngest and greenest person at the conference will surely know that having a great plan and great science is only valuable if the final output is something that can be understood, consumed and ultimately implemented by the users and operators.

I remember vividly in an earlier part of my career desperately trying to build a project justification for C level management to show the value of implementing a laboratory information management system. We were presenting the cost justification based on the errors on the margin which would affect business decisions. Back then I was simply focused on the data acquisition and management which the LIMS system would provide. The part I was missing was the next step which went to the real crux of the problem, how do we get operators and geologists to implement decisions based on our chemical analysis? How can we get them to trust the output of our processes and system? Ultimately this is the problem that BI solves. While we put a very expensive solution in place to manage our chemistry data, the project was incomplete. We still had to go back and implement the consolidation of that data with other geological and operational data to get a complete picture which could be integrated with the mine plan and actually executed. This is BI, the whole chain of data regardless of the source application all the way to someone deciding whether a particular scoop or rock is waste or should be processed.

The second misconception is that it’s all about the tool. We often talk in terms of the technical tools in BI but really that is even more short-sighted, even if it simplifies the conversation. Unfortunately this bias tends to affect the BI practitioner as much as the consumer. For instance, one of most popular tools is SAP Dashboard Design more commonly known as Xcelsius. This is an amazing visualization tool but if we peel back the flash (little inside joke there, ask me about it at the conference!) and look at the reason behind changing data tables into charts and dynamic analysis we realize that this is a tool that allows us to change people’s understanding and perspective of their own data and their own business. Indeed it allows people to quickly go through tons of data to see trends, correlating data and ultimately make better decisions. It’s not just about the tool, but also the opportunity the tool provides to inspire action.

Take for example a manufacturing environment using Xcelsius along with some distribution technology that we have (yes, read distribution technology as a ‘shameless plug’ for InfoSol’s InfoBurst® platform) to help plant supervisors manage their daily operations. The key point to me is that the users simply didn’t have access to the data in a consumable way to make decisions. The incredible investment in technology and engineering that this plant represented was being hamstrung by the very complexity that made it great. A good BI solution changed that, and helped the people running the plant wrap their arms around everything that was going on.

The final misconception is that BI is just a fancy word for analytics. While it is true that analytics is an important component of the BI landscape, it is not the pinnacle of the field; we must remember that the various pieces of the BI landscape work together to help people make better decisions. I like the example of the humble control chart. Everyone knows that control charts help us quickly understand how various processes are working against controls and statistical analysis. Unfortunately just having a control chart doesn’t do you very much good unless it is in the hands of the right person, at the right time with the relevant data to help the user understand if this variation is a true issue or just a flier. The drill-down capability and ad hoc analysis of data that the BI tools can provide compliments and extends the raw analytical power that our statistical vision focuses on.

I don’t know if I’ve put together a clear vision of BI, but at least I hope that I’ve put together a plan to address some of the ideas which will help us present the perspective that every project involving data (i.e. all of them) needs BI. Not just as an afterthought or as an aside, but rather as a key component to the successful implementation of every effort. We need to help those engineers worried about the technical hydrometallurgy and the impact of process changes in their plants to also take a second and think: What can I do to help present this data to management and operators to help them follow my vision and plans and then to validate the end results?