Business Intelligence Unleashed, Expectations Exceeded ?
Introduction
Early November 2010, IBM Cognos 10 was finally released. "Business Intelligence Unleashed” was the –as usual- catchy and (overly ?) promising marketing slogan. Expectations were rather high as this was the first major release since Cognos had been acquired by IBM in 2007. Currently part of the IBM Business Analytics software offering, IBM Cognos 10 offers a lot of enhancements compared to the previous version. This insight tries to shed light on the highlights of this new release and investigates whether expectations have been met or even exceeded.
User interface targeted at business users
Since the early days of business intelligence, much has changed. Not only has the amount of available data exploded, but technology matured and business users have become far more proficient at using BI technology. This leads to new opportunities where users can be provided with the freedom on how to present the data they need, whilst IT is maintains the standards in security, performance and reliability centrally. When looking at the renewed product offering with IBM Cognos 10, the theme of self-servicing BI is considered to be the main plotline. IBM Cognos 10 features a completely new user experience by offering a unified BI workspace: Business Insight. Although marketed as a new product, Business Insight was already known as Cognos Go! Dashboard.
With this release it is no longer a separate product but now an integral part of IBM Cognos 10. The Business Insight workspace is built by dragging/dropping complete reports or parts of a report onto the canvas. Charts, crosstabs, metric objects, TM1 objects… everything can be included. Additionally filters can be added that will filter seamlessly all associated objects. Two types of filters are available: value filters using checkboxes and slider filters to filter on date.
Figure 1 : The Business Insight canvas
Importing objects is done by selecting reports or by selecting a part of a report from the content pane, which shows the content of the content store. Drilling and drill through definitions of the original report are kept in the imported version. By means of an on demand toolbar, changes can be made to the content or layout of the report. Lists can be switched to charts, columns can be added, highlights can be set …
Each report on the canvas is represented as a widget that is kept internally in the content store. This allows a user to thoroughly modify the content or layout of an object without modifying the original object. These modifications can be done in 2 modes. The simple modifications such as switching from table to chart can be done from within the on-demand toolbar. For more advanced modifications a new studio has been developed: Business Insight Advanced. This studio is aimed specifically at business users and is positioned between Query Studio and Report Studio. Business Insight Advanced specifically replaces the Report Studio Express mode. As with Query Studio, data is previewed while developing a report. However more formatting/data presentation options are available then in Query studio. Business Insight offers a common workspace for all supported analytics. This moves the user experience from a role centered approach to a progressively interactive and dynamic self-servicing Business Intelligence experience by leveraging existing content and easily creating new content.
Enhanced searching and collaboration
IBM Cognos 10 offers integrated search capabilities that will search both the content of a report as well as the metadata behind the report. By doing a refined search, even selections on specific object types are possible. In Cognos 8, these search capabilities were part of Cognos Go! Search. This product has now been added to the base product, requiring no extra license cost. Also added by IBM are a number of collaboration capabilities. According to IBM, this will offer faster and better decisions. To achieve this, Lotus Connections fully has become part of IBM Cognos 10 and allows users to consume BI content and discuss it with others through a common interface. This social software allows users to set up due dates, activity owners, discussion threads, wikis, blogs, communities …. By using an Inbox a user is notified of new tasks, approval requests and notification requests which the user can act upon. More importantly, users can finally create annotations in reports and dashboards. Annotations can be entered on 2 levels: cell and object level. The entire thread, including who posted what annotation when, is available to the user. When exporting a document to Excel or PDF, the annotations are exported as end notes.
IBM Cognos 10 offers improved data lineage functionalities. Report and model information can be viewed with IBM Cognos Lineage Viewer. With the integration of IBM Infosphere Metadata Workbench it is now possible to look at ETL, source data and impact analysis information.
Smartphones
Cognos Mobile features the ability to access Cognos from a mobile device. While Cognos Mobile 8 used to be limited to Blackberry, IBM Cognos 10 allows for any Windows Mobile system, Nokia Symbian, iPad, iPhone or Blackberry device to be used. The product allows users to fully view and interact with all relevant BI content such as Business Insight dashboards, reports, metrics and analysis in a secure environment. Users can get a real-time view of Cognos BI data, including real-time alerts or perform drill up and down and use improved prompting capabilities while running a report on their mobile device. IBM Cognos 10 Mobile also is location aware, so values can be filtered based on GPS coordinates.
click to enlarge
Offline reports
Viewing offline content used to be a feature within Cognos but was dropped when launching the new ReportNet architecture back in 2004. For years, existing and long-standing Cognos customers and end-users have requested this feature to become available again, especially as the main competing products offer such capability. Active Reports offer interactive dashboards that business users can use offline. Active Reports are an extension of normal Report Studio reports and can be scheduled or bursted through Cognos Connection. Existing Report Studio reports can easily be converted to Active Reports by adding active report controls and active report variables. A report can be run as HTML or be saved as MHT file. This file is viewable in Internet Explorer or Mozilla Firefox. Due to the very flexible nature and the ability to burst and apply row-level security, it is not unlikely that this format will soon replace a lot of the PDF burst reports that were traditionally sent to for example customers or business partners.
Enhanced reporting capabilities
With the acquisition of SPSS by IBM in 2009, it was only a matter of time before these capabilities would be integrated in Cognos. Report Studio now offers a new report type: Statistics. By means of a wizard driven interface, it is possible to create reports with descriptive statistics, histograms, boxplots and Q-Q plots (for more details, see below). At the backend, the Statistics service is running the SPSS Statistics engine. By integrating these features, statistical process control can now be embraced and for example, Six-Sigma projects can leverage the existing Cognos-platform.
New features include:
- Descriptive statistics: will quantitatively summarize a dataset such as mean, standard deviation, minimum, maximum…
- Histograms: display a range of variable values in intervals of equal length such as a frequency of incidents over a range of months
- Boxplot: is also known as a box and whisker chart and shows groups of numeric data. In the chart, minimum, maximum, median, upper and lower quartiles, outlying and extreme values.
- Q-Q Plot: a quartile-quartile plot charts the quartiles of a variable’s distribution against a distribution of your choice, including the normal distribution.
Report Studio is now using a new chart engine for any new chart created. Legacy charts can be converted to the new charting engine, or they can be used in legacy mode. The new charting engine will allow setting more properties to modify the appearance, and will give a preview of the chart at design time. Pie Charts for example can now group the smallest slices into a single category, preventing pie charts to become overly busy with small slices. Only one new chart type is available in Cognos 10: the Bullet chart. A bullet chart will compare the actual to the targeted measure.
It should be noted that Cognos 8.4 already included Marimekko and Micro-charts. Marimekko charts will show a 100% stacked chart, with the width of column being proportional to the total of the column value. Microcharts are a miniature version of line and bar charts and are ideal to be used in a table. Another small but convenient change in Report Studio is the naming of output files. When a report was run in Excel or PDF, the report name used to be a random string. IBM Cognos 10 will now use the report name as the default name when the report is saved. Other user interface related improvements include:
- Insertable Objects and Properties can now be positioned to the right of the work area
- Reports can be previewed when opening or saving a report
- When drilling up / down: the column title can now show the level label value
- Calculations have been made easier
- The default report style can be used as a new color scheme
- Alternative text can be added to non-text objects. In case the reader would not be able to show the object, this alternative text is shown.
- Using Map Manager, new region layer can now be created. Each new region is made up of one or more regions from the original map.
Another small improvement is the scheduling of reports during a certain period of the day. You can now schedule a report to run hourly between eg. 9 am and 5 pm. It is also possible to change the owner credentials for a schedule or suspend a scheduled activity.
Using external data
The use of external data within a Report Studio report allows users to leverage data not present in the data warehouse or centrally managed data sources. This allows users to act upon data much faster than before when external data files first had to be made available in the data warehouse or in a Transformer cube.
Excel, tab-delimited text files, csv-files and XML-data files can be imported as external data. The external data is not report specific. When external data is imported, a new package is created, containing both data and relationships between the objects. To prevent some users to tamper with data, the administrators can grant or deny the ‘Allow External Data’ grant. When the data is imported into the package, it is shareable with other users in a secured manner.
Business Warehouse
SAP BW packages can now be directly created in Cognos Connection. It remains possible however to create and publish SAP BW packages using Framework Manager. When creating a new SAP BW data source, a user friendly and easy wizard is available, that is capable of leveraging the SAP BW security. Also new, is the support for Microsoft Analysis Services 2008 using a similar mechanism.
Microsoft Office
IBM Cognos 10 heavily integrates with the Microsoft Office product family. Within Microsoft Excel, the user can connect to the Content Store and select a package on which data analysis will be performed. The actual analysis is created by dragging and dropping package items from the folders into Microsoft Excel. The use of the framework semantic layer ensures stability and security over direct access to the data source. A new feature is that the analysis can be saved in the content store, directly in Excel-format. While it is possible to create your own analysis in Microsoft Excel, you are not required to do so. Within Word/PowerPoint and Excel, the Cognos Content Store can be viewed and objects such as reports or analysis can be imported directly using the Cognos security model. When a report is imported, the data link is saved so the report content can be refreshed when reconnecting to Cognos.
Metadata
Framework Manager largely remains the same. Noteworthy is the Model Design Accelerator. This is a graphical tool that allows you to map tables on facts/dimensions and from there on easily generate a framework model. When the model has been generated, a 3-layered framework model has been built, containing a Physical, Business and Presentation View. In older Framework Manager versions, it was a best practice to use a design language that was not used by business users. This design language was usually a dialect of a main language such as English Zimbabwe. This ensured durable models by preventing compromising the report integrity when attribute names were changed in the model. With this new release, a new property: Use Design Locale for Reference ID has formalized this best practice. What is no longer available is the native support for repository control such as Visual SourceSafe though it remains possible to use an external repository control.
Under the hood
IBM Cognos 10 presents some remarkable performance gains: MDX queries could run up to three times faster, PDF generation is 40% faster … These gains are realized through the use of the dynamic query mode. With ever increasing query complexity and growing data volumes, Cognos created a JAVA-based optimization application that will optimize queries by using 2 separate mechanisms: key optimization and in-memory caching.
Key optimization is used for OLAP data providers such as IBM Cognos TM1, Oracle Hyperion Essbase and SAP BW. It uses a number of optimizations:
- Direct Java Connectivity is supported for SAP BW and Oracle Hyperion Essbase
- Null Suppression optimization: when levels are added to a report, the number of cells needed grow exponentially. A lot of these cells contain null values, slowing down the report evaluation process. By leaving out those cells, the report performance is enhanced. This feature is available in Business Insight Advanced, Query Studio and Report Studio.
- Master-detail optimization: master queries are pushed to the detail level, leaving only one query that is executed in the database.
- 64 bit optimization: allows more caching to occur as the Java Virtual machine is no longer limited to the 2GB memory boundary in the memory address space
When a report is built or run, several requests are made to the report service, returning metadata and data to the report. In-memory caching will cache all that data for future reuse. When a user requests the same report once more, the data no longer needs to be retrieved from the data source. There are however a few "buts": the cache is both data-source and user specific, diminishing the effect of the caching. As data is cached, changed data at the database is not captured, so care is required in setting cache refresh rates.
In order to be able to easily search for bottlenecks in queries, a new client based tool is delivered: Dynamic Query Analyzer. This tool provides a tree based view on the query and indicates possible bottlenecks using color coding. The Dynamic Query Mode must be configured to log the run trees. This is done by modifying the appropriate Query Service Setting. The resulting logs are written in the /XQELogs/ and can be viewed directly or remotely (HTTP) with the Dynamic Query Analyzer tool.
Interesting to know is that IBM Cognos 10 can be installed with or without a preinstalled content database. When you choose to install a preconfigured database, IBM DB2 is installed.
Several small improvements were finally made to the administration interface. One of them is the ability to graphically look at the schedule load by timeslot as seen in the image above. Another improvement includes the scorecard type view of the server topology indicating availability of servers and services.
Migrating from older versions
The last topic of this insight, discusses the inevitable migration process from previous releases. The Cognos 8 Upgrade Manager was face lifted and renamed to Cognos Lifecycle Manager. This tool enables the automated migration of Cognos 8 reports to IBM Cognos 10 in a visual and easy manner. Migration to IBM Cognos 10 can be done from any Cognos 8 release and this is the last release that will support ReportNet as a source system.
The Lifecycle Manager tool is a tool that automates the validation, executing side by side and finally comparing the report output. When creating a project in Lifecycle manager, the source and target environment are defined. Next, the application credentials to log on both content stores are defined and the list of content to be included in the project is added. The first step in the actual migration process is to manually deploy all the content from the source to the target environment with an identical folder structure. When this is done, Lifecycle Manager can be used to validate all the objects in the target environment. Before Lifecycle Manager can programmatically execute the reports, prompts must be supplied. If prompts are defined and saved in Cognos Connection, Lifecycle Manager will automatically include these in the migration project. If no prompts are defined, Lifecycle Manager allows for automatic generation. When all prompts are entered, the reports are run in both the source and target environment. Next the reports can be compared by Lifecycle Manager. When the outputs do not match, a blinking option is available, so the developer can quickly visually asses where the differences are located. If there is a real difference, the report can be rejected by the user.
Conclusion
In the past 5 years, IBM has invested US$ 14 billion in the acquisition of 24 software companies. With Cognos being one of them, the overall offering starts to clearly benefit from some of these new technologies. Following the consolidation in the Business Intelligence market a few years back, when IBM acquired Cognos and SAP acquired Business Objects, Cognos clearly seems to digest well the transition into the IBM family. Not only were integration issues tackled, but this release also delivers a series of enhancements, even if some of them originated from innovative, original approaches of smaller challengers in the BI space. However, all in all, one can say that IBM Cognos 10 offers an extremely complete, mature, enterprise BI solution, built from the ground up on a SOA Architecture, and clearly sets a new standard for end-to-end performance management solutions.
