the analytics edge

I was pleased to hear that Susan Cramm is putting out a new book , and once again, she delves into dealing with that perceived gap between IT and “the Business”.  I like her focus and insight on an organizational challenge that continues to hound many businesses, even though I have never liked the underlying us-versus-them mindset – IT groups should see themselves as part of the business, not a separate entity. Not surprisingly, none of the things she identifies were news to me, but then I’ve been in the business of IT a long time. As I reviewed her list, though, it struck me that there is a whole other perspective that should be discussed – how other people have learned to exploit these gaps. Specifically, how the IT supplier ecosystem has learned to survive and thrive.

Let’s take them one at a time, starting with Service or Control. If business leaders feel IT is overly bureaucratic and control oriented, they are more likely to accept a solution that “doesn’t require IT involvement”. That is the sales pitch, anyways, and it gets the sales cycle started without IT’s involvement.

When we look at Results or Relationships, if the IT group is seen as a bunch of condescending techies who won’t listen, it is really easy for any good salesperson to sink a second hook into the account. All they have to do is listen to the customer. That has got to be the Golden Rule in every sales technique I have heard of.

They also know that a profitable account is one that you establish a strategic relationship with, so the sale process always includes conversations about long term direction in which the first sale is merely a step in the right direction. If the IT group has been too reactive and not proactive enough, this “big picture” discussion will be music to the business leaders’ ears, and the third hook is firmly planted.

By taking that long-term approach to the account with a small first step, the vendor (who is simply looking for the easy money) actually responds to the fourth business leader sensitivity – IT proposing ‘deluxe’ when ‘good enough’ will do.

Of course, a big part of that first sale is to make sure it goes through without a hitch or the relationship is dead before it gets started. That is why vendor sales management focuses heavily on new accounts and delivery/resource priorities can often shift to lock in a lucrative account before they get a chance to have second thoughts. Pity the poor IT group that pushed for the ‘better’ solution and couldn’t deliver it fast enough, giving the business time to re-think what they might want.

And while they are thinking, they collect ideas and form opinions about what their business needs to succeed. It is the job of the suppliers’ marketing groups to work the latest buzz words and phrases into the ecosystem, to make it appear like their offering was custom-made for the target’s needs. If the IT group is trying to build a business case to justify the latest widget that people ‘need’, they find themselves quickly trying to fight emotion with logic. Good luck.

And of course, those sexy new widgets are ‘everywhere’ now, so when IT doesn’t quickly embrace them as well, the business leaders see this as suppressing innovation and lacking the responsiveness needed to survive. Suppliers know they have to go out of their way to announce (it’s not always necessary to deliver) inclusion of the latest fad in their solutions. People buy the kids meal, but they really wanted the toy.

And finally, vendors know that when there is a change in management or business direction, money is spent. Get in there and see how you can “help”. An IT group worried about their installed base and the new project workload will be seen as an impediment, and that makes the “help” all that more welcome.So it is easy to see how the IT supplier ecosystem has adapted to the timeless weaknesses of business-IT leader relationships. The challenge is to respond to that reality. I would be the last person to recommend putting a sales manager in charge of the IT group, or that they establish a customer-supplier relationship. But if the CIO identified IT suppliers as “the enemy” and worked to counter these “8 steps to establishing a beachhead”, they would be making the necessary behavioural changes to becoming a contributing member of the business they are a part of. Isn’t that the ideal end-state for an organization anyways? Business leaders that are hardened to the latest sales techniques, buying things they need rather than being talked into a quick fix that will cost them more in the long run.

As you review the interdepartmental challenges in your organization, remember that outside parties are likely taking advantage of any weakness. That might give you the extra incentive to close the gap a little.

I like to analyse stuff. Dig and poke around the bits of data that the business generates, looking for new insights. For our various web sites, we use Google Analytics, and I have been generally happy with the first-level analysis available through the web interface, but I keep running into the same brick wall – when I see something of specific interest, I can’t drill down far enough to understand what happened.

When the Google Analytics Data API was released, I saw this as an opportunity to get at the data underlying those high level reports, and see the actual pages a specific visitor looked at. There are several tools available to do this, some of which are free, but they all seemed to make the process more difficult than it needed to be or suffer from the same aggregation-only reporting paradigm. Enter Pivot Express [now called Nextanalytics for Excel].

Available as a free web site [no longer available, sorry] or as an inexpensive Excel add-in, this product can take your data and perform quick ‘pivot’ and other analytic functions, and we’ve added a Google Analytics query to both products that allows you to grab the detailed data from Google, pivot it by date or other selected dimension and dump it straight into Excel where you can do whatever you want with it.

The Excel add-in allows you to save your queries and re-run them on demand, picking the most recent time periods automatically (Past 7 days, for example). Enough fiddling with dropdowns and selection boxes, picking dates and then trying to format the results by date or country or referring site or what-have-you. No more getting aggregates when I want details. Gotta love it!

Try it yourself at www.nextanalytics.com

I’m really excited about a new product combination coming to market - the powerful analytic engine from Nextanalytics is being embedded inside an Excel add-in! The guys decided that since their code was so small, it could be packaged for individual use in the #1 BI client - Microsoft Excel.  With a strong analytic transformation capability front-ending Excel, it becomes an ideal solution for small businesses that don’t have (and don’t want) an expensive BI system.  Give it a test drive at www.nextanalytics.com.

I was reading a blog by Steve Miller this week, and was struck by what he said in his opening paragraph:

“I’ve assembled my BI-Searchers toolkit. It consists of an open source technology stack that includes the MySQL database for storage, either Pentaho’s Kettle or one of the Python/Ruby agile languages with database connectivity for ETL/programming, one of the Pentaho or Jaspersoft dialects of Mondrian OLAP server for slice and dice/drilling, the “other” Mondrian for for interactive statistical visualization of large data sets, and the R Project for Statistical Computing for advanced graphics and predictive analytics. Tying together the data delivered from all this wonderful technology are designs that provide context for analytic interpretation.”

From his image, he’s got as much gray hair as I have, so he really must remember the days of Lotus 1-2-3, Harvard Graphics and WordPerfect - the core desktop tools of the late 80’s. Each had a very different user interface, unique terminology, and special focus. They were very good at what they did individually, but they were never intended to work together. Users could become proficient at one with a bit of effort, but knowing one did nothing to prepare you for the others. Finding someone that could handle them all was a treat - they were the stars of the organization.

Somehow Steve’s proposition makes him look like one of those stars stuck in the old days. There aren’t too many people that can actually handle the mental gymnastics needed to cobble together all the pieces he’s identified. Big BI vendors know that - they’ve lived through the life lessons of failed deployments and dwindling support renewals. Analysts know it - that why they say BI ’suites’ are in demand. The sad state of the industry is that the tools still require the technical stars to make them all work.

Analytics is a services business (has anyone actually wondered why IBM is pushing an analytics agenda?). People use Excel spreadsheets for everything because it is a tool they know. They rely on the stars to do some preparation so they can load it into Excel and play.

It’s time to change that model…

Twice in the past weeks I have come across blog entries and comments with a blunt display of elitism from IT professionals. I find this appalling because I have known a lot of IT people, and most of them are usually completely ignorant of how to run a business. In the first case, Susan Cramm was commenting on the problem with building trust between IT and the business, but she used an unfortunate example of a CIO that forced his techs to teach their business counterparts how to use reporting tools. The way the story was told, it implied that the business counterparts might be smart enough to learn this stuff, and that the learning challenge was one-way.

In the second instance, LucidEra’s veterans admitted a lesson they learned was that they overestimated the ability of their customers to do their own analytics. OK, I’d wager that means the product did not fit the needs of the customers, but the comments went down that elitist path, some going so far as to question the customer’s IQ. Hey! Who’s business failed in this story?

Most IT people have very little experience in real business. Their lives are mostly insolated from the day to day challenges of making a buck to pay the bills, dealing with real customers and their ridiculous demands. Just because they can create something so complicated that other people can’t use it?

You’ve invested heavily in the development of your application but your customers want more than the simple reports you have provided. They want a dashboard with some analytics that make them more comfortable that their money was well spent. It shouldn’t be complicated – just a few charts of trends and stuff like that. That’s how it starts: the journey of discovery and disappointment. You’d think this stuff was a commodity by now, but technology companies don’t like to give up their high prices, and the experts believe the complexity and specialist skill sets are required, so the products follow suit.

I was on a few sales calls this week, and this sad state of the industry was driven home by a couple of reactions to our new product offering. As with any vendor, we have tried a number of demo scripts to try to hook people’s interest, and we recently cut the demo way back to just a couple of simple activities. In each case, I was about 5 minutes into the new demo – open a transaction file, create a pie chart of sales by product, then create a table showing top 10 salespeople’s performance over the past 12 months with some colored highlighting. I refresh the dashboard and show these two reports, then pause for acknowledgement. In the first call of the week, I was surprised by a “way cool!” response. The second call was a slightly more reserved “cool!”, and I found the common response a bit amusing.

It was the third demo that has stirred up some deeper thinking, though. A large organization that sees analytics as a competitive necessity, they started by telling us that they own many different packages, from large and small vendors. They have bought analytics companies, and have built their own products from scratch. They have processes to follow, and asked for copies of any presentation materials we would be using up front. Very professional and detached, clearly setting the bar high before we started talking.

Knowing our demo script well, I apologized for what I said will appear to be a tedious demonstration of me clicking and typing, I assured them it would be short, and it was faster to show them than to explain our unique value. Working slowly through the dozen-or-so simple steps and pausing with the new dashboard reports on screen, we sat through a long silent pause. When they did speak, they apologized for the delay – they busy were taking notes, and were excited by what they just saw!

So what did these professional and experienced individuals see in our product that elicited such a response? They saw that we have significantly lowered the bar for adding analytics to applications. The discussion revolved not around the big analytic projects underway, but opportunities throughout the organization where groups couldn’t justify the effort and investment because it was so high. We joked that we create “disposable analytics” – it is so easy to add a new analysis to the dashboard that you can throw them away when they have served their purpose. It has an extremely small footprint that doesn’t try to be an application all unto itself. It is easy to use without specialist skills. And it is available at a cost that makes sense.

And it dawned on me — we have turned business analytics into a commodity, and it is long overdue.

Number one Hot IT Job for 2009, according to Baseline, is Business Intelligence Experts. That is bad news for business managers. It means there is a shortage of the skills needed by businesses. These people help “in a lot of decisions having to do with how the customer base is changing, market differentiation, pricing and services”.

To BI Experts, this is received as good news - salaries will go up because of demand; jobs will become more interesting with less mundane report writing and more strategic analysis work; and career paths will open up as you gain more exposure to things outside IT.

Business executives, though, may not have the reaction you expect. They believe they are paid the big money because they make the tough decisions, and when others try to move into that space, they react. This is not necessarily political in nature - it is just good business sense. They are always looking for better ways to do things, and if the BI guru suddenly becomes a cog in the gears of success, alternatives should be considered.

Fortunately (or unfortunately for the BI experts), there are a number of new BI solutions appearing on the market that are geared to providing BI-like answers without the need for ‘expertise’. They require less technology infrastructure at a lower cost using a cheaper skillset. I am proud to be associated with one. The pendulum is swinging back from a technology-centric, expert-driven ’business intelligence solution’ to embedding easy to use business analytics into applications. They do not require experts to run them.

So I caution the BI workers out there — your position might be hotter than you think.

A long time ago, I graduated from an engineering program with the concepts of effectiveness and efficiency held in high regard. Truly elegant solutions could produce phenomenal results with very little resource or effort. By contrast, overly large and complex designs that relied on brute force to get the job done have always seemed to be the scourge of civilization. As the computer industry developed, I saw a number of elegant solutions emerge, packaged as lean, purpose-built objects that delivered results as no other product before it. These alternatives usually took the market by storm simply because they offered a better way to get the job done; the Return-On-Effort ratio was very high.  This is what attracted me to Nextanalytics.

In the business intelligence and reporting space, the various players have been battling it out for decades. Innovation has dwindled to feature creep as products have matured and grown bloated. One fact has held over the years – only 15% of the population actually have the desire and ability to conceptually analyse the data available. This has nothing to do with the ‘tools’ or making them easier to use; most people’s brains simply do not work that way (http://lsn.curtin.edu.au/tlf/tlf1995/whitefield.html). The big B.I. players have known this for a long time, and with many of the analytical types working in other roles, the few that actually use the BI tools to create new reports, dashboards and scorecards are in very high demand.

As we all know, high demand means high priced, so when it comes to generating new reports, those in control ‘do the math’ and decide where the effort ‘is worth it’ and where it isn’t. Because their effort is so expensive, a lot of potentially insightful reports never make the cut – if you don’t know the answer will be worthwhile, you’ll never do the work to prove it. That explains why there is a growing investment in analytics (http://www.tdwi.org/News/display.aspx?ID=9269). But is spending more the right answer?

The market has shown, time and again, that when demand for complex and expensive solutions climb steadily, something disruptive usually occurs. People don’t like spending lots of money on big, complicated things they don’t understand. I feel it is time for solutions like Nextanalytics that change the rules, change how things are done, change who can do them. It’s time to reduce the effort, reduce the investment, but still get the returns.

We’ve seen the cycle before — product life cycles have matured and the products bloat in size, complexity and price until people start to question their value. To those that have grown up with the products, this seems almost blasphemy. How could we question years of product refinement and adaptation? Don’t you remember all the lessons we learned along the way? There is NO WAY you can live without all those features!

Maybe it’s just us baby boomers getting older and less tolerant of complexity, but I am finding more and more telltales of a groundswell in search of simplicity. Being part of a software startup, we are constantly looking for new and innovative approaches (our potential competition), and I have been really surprised — there isn’t a lot of innovation in the Business Intelligence space, and it seems no one wants to simplify the status quo model of SQL databases and OLAP cubes feeding Excel spreadsheets.

First of all, Excel now sports even more features and can handle even bigger spreadsheets than ever before. That sounds great until you read about how many spreadsheets have errors and what the costs are. Then there are the increasingly scarce BI skills needed to build all the queries to feed all those spreadsheets.

Things like that make me believe we’re on to something with nextanalytics. Simple analytics, not complicated statistical analysis. A powerful script language that works on pages of data, instead of complex query statements, tedious cube construction, or a nightmare of formulas copied over thousands of cells. And it’s soooooo easy to build reports that would be a major challenge with all those other tools, and with a lot less to learn and a lot less to do. An order of magnitude faster to accomplish the task - and we’re not talking about compute speed; far less to learn, less to do, and much easier to reuse previous work.

And the whole thing is available to developers for free in a 300k download (on request), production servers get licensed for under $600. Why wouldn’t you consider it?

I read an article in DM Review today that included the question “So why are CIOs spending so much time and energy on BI?“.  It seemed like a strange question to be asking.  The article was discussing the trends in BI towards a new generation of performance management, but the question made me stop. It seemed like such a silly question.

Why do CIOs spend time and energy on BI? 

Isn’t that their job?

A looong time ago, I wrote up a one line description of what I wanted to do with my career. This was around the time “CIO” was a new idea in only very large organizations. Coming from an engineering background, I enjoyed technology. I also valued efficiency and effectiveness. I carved out a place for myself when I termed my career objective at the time to champion ‘improvements in business effectiveness and success through innovative use of computer technology.’ I found myself in I.T. because that was where it made sense for me to be.  I defined the role of the group based on my personal objectives.

To this day, I still see the I.T. organization as a middleman between the possibilities available with technology, and the challenges of the business. It is our job to continuously scan the technology landscape help the organization take advantage of opportunities where the business can gain a competitive advantage (or at least to cost-effectively not fall behind).

So why spend time and energy on BI? Because information is the lifeblood of any organization, and computer technology today offers so many possibilities for improvements. So many, in fact, that organizations hire specialists to manage it. They put those specialists in the Information Technology group, and the CIO heads up that group.

So the CIO should spend a lot of time and energy on BI. It’s their job.