Have you taken a look at TIBCO Spotfire and TIBCO’s growing ecosystem of offerings lately? My initial impression is…shock. I missed what has been happening outside of TIBCO Spotfire the past few years. In this series, I’ll start by sharing what is new in TIBCO Spotfire. Then I’ll explore other areas of the growing TIBCO Industry 4.0 portfolio. From a fantastic smart data datalog to inline data preparation, deeper analytics integrations with TERR/R, H2O.ai, ML/MLlib, SAS, MATLAB and the Statistica acquisition, StreamBase, live datamart, solution templates and accelerators, MDM, IoT, BPM, and on and on. TIBCO’s depth and breadth has grown tremendously.

To supplement this article, please join me in an upcoming webinar with TIBCO on August 24 at 10AM PDT. We will show you what is cool, new and exciting in live demos.

TIBCO Spotfire Webinar

What is New in TIBCO Spotfire

From connecting to hundreds of data sources to anticipating what’s next, TIBCO Spotfire has simple yet compelling sophisticated visual analytics. A skim through the publicly shared case studies is revealing. Companies in engineering, energy, life sciences, financial services, pharmaceutical, oil and gas, and other industries requiring savvy, complex analytics use TIBCO Spotfire more often than other solutions. Although TIBCO doesn’t make much noise in my channels, they are a strong vendor not to overlook or underestimate.

Last August I shared a quick update on TIBCO Spotfire 7.6. Since then, there have been several releases to catch up on. Let’s dive into the highlights. For details on each Spotfire release, check out the official What is New summary.

Data Source Additions

TIBCO Spotfire already has an extensive library of connectors. The following is a summarized list of new and updated data sources in the past few releases:

  • Salesforce, Google Analytics, OData
  • Amazon Redshift
  • Apache Spark SQL 2.0 and Apache KNOX
  • SAP BW 7.5, SAP SSO with SAP BW, SAP Hana and messaging server
  • SQL Server 2016, Analysis Services 2016, Azure SQL, Azure SQL Data Warehouse and Azure HDInsight
  • PostgreSQL 9.5 and 9.6, MySQL 5.7

Smart Data Catalog

Smart Data Catalog powered by Attivio technology is a stand-out addition to the TIBCO Spotfire offering. Data catalogs are critical to have in place to prepare for new GDPR data privacy laws in Europe that will be enforced in May 2018. I recently write an InformationWeek article called “What Analytics Pros Need to Know about GDPR”. If you don’t know about GDPR, you’d better up to speed on it.

With Attivio data marts and text analytics, TIBCO Spotfire analysts can transform structured or unstructured information for analysis. Attivio catalogs, identifies, and unifies information. It automatically generates data models and correlates structured data and unstructured content. Using natural language and full text search, Attivio recommends the most relevant data with context.

For example, if you wanted to run sentiment analytics on customer reviews for the words “fun” and “boring” you could use this new feature. The following illustration shows how to build a simple full text search based on Attivio data within TIBCO Spotfire.

TIBCO Spotfire Full Text Search

Full-text search

 

The results only return relevant records for review in tables or visualizations.

TIBCO Spotfire Full Text Search

Full-text search results

Data Preparation

A TIBCO Spotfire fan favorite addition has been easy, visual, inline data preparation for shaping, enriching, and transforming data. Data preparation typically involves the most level of effort. Thus, this is a valuable time saver for TIBCO Spotfire customers. They also added data wrangling API operations for app developers that integrate TIBCO Spotfire into products. Let’s check it out.

TIBCO Spotfire Data Prep

Inline data preparation

In Spotfire 7.6, Source View was added to visualize an overview of data transformations, calculations and how data tables are derived from rows, columns and combined from multiple data sources. In Spotfire 7.7, add rows (unions) became optionally editable and also got “smart”. Intelligence powered by machine learning reviews data usage and makes recommendations.

Then in Spotfire 7.9 one of the most anticipated features was made available – the ability to easily change data transformation settings more elegantly to adapt to changes in the underlying data sources. TIBCO Spotfire’s Source View provides an overview of all your data wrangling steps. It lets you navigate and edit transformation in any order without losing your work.

Another useful capability in TIBCO Spotfire is in-place, visual data cleansing. Everyone has dirty data that sticks out when you visualize it. In the latest TIBCO Spotfire release, you can now easily clean up issues in your data when you see them. A new Replace Value feature lets you change incorrect data values by double clicking in a table, in the details on demand panel or in the extended data panel. There are two flavors to the Replace Value feature, 1) the ability to replace the single value only, 2) or replace all occurrences of that value in the column.

Here is a peek at what Replace Value looks like in action.

TIBCO Spotfire Visual Prep

Visual data cleansing

A crucial, related capability is the ability to quickly see any Replace Value changes. You do know why that is important, right? I hope so. If not, I’ll explain it to you in our webinar.

The visual data source view lets you inspect, and if needed, remove the Replace Value transformations.

TIBCO Spotfire Data Cleaning

Inspecting changed data

Data Visualization Enhancements

TIBCO Spotfire is well-known as one of the first data discovery vendors in the market. I starting developing with this solution back in 2010 and loved the drag-and-drop interactive visualizations. In the past few releases, histograms, sparklines in KPI charts, color rules, and more granular control of tables was added.

TIBCO Spotfire Viz

KPI Sparklines

For KPIs with time dimensionality, there is now an option to add a sparkline for visual context that nicely showcases high and low in that period. Another nice visual improvement is color rules. You can add color rules to bar chart, line chart, scatter plot, map chart, KPI chart and treemap visualizations.

TIBCO Spotfire Viz

Color rules

Revealing my secrets… when I evaluate visual analytics solutions I often pay for additional testers by persona. Personas being information consumers and knowledge workers the use reports (executives, non-technical), Excel analysts (Mr. Underwood types) that can’t live without pivot tables, power data analysts (folks that use self-service ETL, data discovery tools and script languages), BI pros (ETL/ELT, data modeling, data warehouse, reporting techies), and data science talent (statisticians, Masters/PhD level folks building predictive and prescriptive models). In my TIBCO Spotfire tests, several personas frequently complimented tables. These same users mentioned table frustration or disappointment when testing other data discovery tools. You’d be amazed by how many people – across multiple persona types – love and need sophisticated tables for analysis.

TIBCO Spotfire’s tables were already fantastic. Now those tables have more fine grained controls. TIBCO Spotfire’s tables now have a contextual menu that allows you to easily control header and cell alignment, as well as header orientation.

TIBCO Spotfire Tables

Table controls

Sorting a TIBCO Spotfire cross table, in absolute terms, as opposed to nested at the lowest level of the hierarchy, allows an analyst to easily see what the top or bottom categories are, disregarding their position in the hierarchy. This makes it easier to answer questions such as “Who are my top X customers across all regions and states?”. Tables also have individual configurable settings to show a defined maximum number of rows in the table visualization. When a hierarchy in a cross table is extremely large, it is now possible to horizontally scroll the hierarchy or the displayed values. You freeze, scroll or adjust automatically. This is a super usability feature for working with big tables.

TIBCO Spotfire Tables

More table controls

Numerous visual enhancements were made to TIBCO Spotfire mapping. Recently multi-layer maps using location was added. Analysts can now create maps with multiple layers of data, mix feature layers and add marker layers to better understand geographic relations and trends between them. TIBCO Spotfire geospatial data functions include but are not to the following:

  • Coordinate system transformation
  • Contour analysis
  • Kriging
  • Hexbin analysis
  • Points in Polygon analysis
  • Polygon Area analysis
  • Trade Area analysis
  • Route optimization
  • Route charts
  • Drive distance, walk distance
Map layers and geospatial data functions

Map layers and geospatial data functions

For water or ocean operations, Nautical Miles was added as new unit of measurement in addition to existing imperial and metrics units when using radius and rectangle selection. Another cool new mapping perk is Get Coordinates. You can right-click anywhere on a map and get geographic coordinates (latitude and longitude) for a location. You can also see maps even if there is no internet connection.

TIBCO Spotfire Map

Get Coordinates from map

Smarter and Faster Visual Analytics

As one of the most mature data discovery offerings, TIBCO Spotfire got a head start in integrating sophisticated analytics, R, and other advanced analytics techniques. TIBCO Enterprise Runtime for R (TERR) is a high-performance statistical engine that is compatible with open-source R and has an artificial intelligence-driven recommendation engine.

TIBCO improved TERR aggregation functions and performance to efficiently distribute computational loads across multiple machines for larger data environments. Today TERR supports 99% of R packages on CRAN along with almost 10,000 community packages. Full details on R compatibility are available on the TERR Documentation site.

For my RStudio fans out there, TERR can be used in RStudio to create interactive R Markdown Notebooks. R Notebooks allow for direct interaction with R while producing a reproducible document with publication-quality output. A Guide to Graphics in TERR provides tips and examples on using Javascript-enabled packages, open-source R packages, and the TERR RinR package.

TIBCO Spotfire Optimization

Optimization and simulation

TIBCO Spotfire today has deep advanced analytics capabilities with TERR Machine Learning Data Functions such as Gradient Boosting, Random Forests, Deep Learning, Anomaly Detection, Segmentation, Propensity, Affinity, Non-Linear Regression, Decline Curves, Predsciptive Modeling and Simulation, Optimization and Genetic Algorithms. It also integrates with popular advanced analytics, data science, machine learning and real-time streaming analytics solutions including IronPython scripting, H2O.ai, ML/MLlib, SAS, MATLAB, StreamBase, and Statistica.

The recent Statistica acquisition adds superior data science, machine learning, IoT, and real-time streaming analytics along with better cross-platform integration across numerous advanced analytics offerings. Another TIBCO strength today is the depth of regulatory compliance features for groups that have mandatory government and legal reporting.

Summary

If you are interested in seeing these new TIBCO Spotfire capabilities or have questions that you’d like to ask, please attend the live webinar on August 24. Don’t miss this rare opportunity to investigate TIBCO Spotfire with me personally to get a different perspective than you’d usually get.

For additional online information, check out the following resources.