Infor’s integration

Prepare: making data transformation approachable without hiding the power

Prepare, or ‘Prep’ internally, was where business users ran into territory traditionally reserved for data engineers. Users needed to be able to filter, handle nulls, merge, split, pivot, summarize, rename, and convert data types across tables that could contain millions of rows. We needed non-technical users to feel confident here, without making the tool feel dumbed-down to power users.

Two design choices shaped Prep:

A column-first selection model. Users started by choosing one of the connections they'd already established in Connect from a side panel, then drilled into a specific table from that connection. The selected table's columns and rows displayed in the main area of the screen. From there, users could select one or more columns and apply transforms to them. Relevant transform tools surfaced in a side panel with clear icons for each operation, so the interaction stayed direct: pick the columns you want to change, then pick what to do with them. No hunting through menus.

An interactive transform pipeline. Every transform applied to a table was recorded as a step in a visual pipeline that showed the full lineage of the data. Users could insert new transforms between existing ones, delete steps from the middle of the pipeline, or modify earlier steps without losing the work that came after. The pipeline made it possible to audit, modify, and understand the state of any table at a glance, which mattered as much for one person revisiting their own work weeks later as it did for someone inheriting a teammate's prep work.

Among the transforms I designed directly were null handling, ignore, filter, and merge. Each one needed its own interaction model to feel appropriate to what it did.

Visual indicators: a small fight that mattered for clarity

Engineering initially didn't want to invest in the visual pipeline indicators across every transform. Their proposal was to apply transforms in place and update the table view, with no persistent visualization of what had happened. My co-designer and I pushed for a persistent indicator system: highlighting columns being imported, adding filter icons on columns with active transforms, and showing a visible pipeline of steps at the bottom of the screen.

Without these indicators, users would only see the resulting transformed table. They had no way to know what was being done to the data or in what order, and that ambiguity created exactly the kind of mistrust we were trying to eliminate. We argued that the cost of building the indicator system was small compared to the cost of users losing confidence in what the tool was actually doing to their data. The visual pipeline was eventually shipped and became one of the most-used and most-loved features of Prep.

Interactive pipeline: a design fight against engineering's first proposal

Other BI tools at the time had visual transform pipelines, but they were largely read-only. You could see the steps that had been applied to a table, but you couldn't reach back into the pipeline and modify them. We knew that wasn't enough. Real data work is iterative. Users needed to be able to delete steps, edit existing steps, insert new steps between existing ones, and click any step to see the table's state before that transform was applied.

Engineering initially pushed back on the level of interactivity I was proposing. The complexity of allowing arbitrary insertions and deletions across a pipeline meant managing dependencies between steps. If a user inserted a transform that broke a downstream step, what happened? I made the case alongside my co-designer that read-only pipelines were not actually useful, just decorative, and that the real value was letting users explore and iterate.

We landed on a design philosophy I've used since. We always let the user proceed, with a warning if something downstream might break. If a user inserted a transform that risked breaking the next step (a common case was renaming a column that a downstream step depended on), the system flagged it with a warning, but let them through. If they knew what they were doing, they could proceed and fix the downstream step. If they didn't, they could explore and recover. Forcing users to delete downstream steps to insert something new was a roadblock, and roadblocks were what we were trying to remove.

Relate: a visual canvas instead of a join builder

Relate was the module where users connected their sources into a unified data model. Sources could be either prepared sources (those that had been through the Prep module to apply transforms) or raw sources (those imported through Connect that didn't need any transforming and could go straight into Relate). The typical BI pattern for this at the time was a SQL-style join builder: a form or query writer that let you specify joins between tables with dropdowns. A few BI competitors had begun moving toward more visual approaches to connecting sources, but those implementations still came with significant usability limitations.

The visual canvas vs. the SQL join builder

The SQL approach was the painful incumbent at Birst. It was the main pain point in the product and was, in fact, the last part of the platform that received a design update during this era of redesigns. The original join experience had been built by developers in the company's earliest days, with no UX input, and while it technically worked, it required users to think in SQL to use it. The audience we were targeting (business users who avoided SQL like the plague) couldn't get past it.

A form-based join builder was discussed early on as a faster path to ship. Even some of our PMs got lost testing the form approach during prototype reviews, which told us everything we needed to know. The visual canvas was significantly more expensive to build, but the cost was paid back by every prospect who could imagine themselves successfully linking their own data without hiring a data scientist or learning SQL. The form-based version would have shipped faster and lost us the customers we were trying to win.

The canvas started empty. As users built relationships between their sources, those sources appeared as nodes on the canvas, connected by single lines. Each line represented a relationship between two sources, regardless of how many column-to-column joins actually existed underneath. This kept the high-level star schema legible at any scale, even when the underlying joins were complex.

To inspect or build the actual joins between two sources, users clicked the first source to open a panel on the right, then clicked the second source to open a panel on the left. A transparent white overlay settled over the rest of the canvas, focusing attention on the two panels. Each panel showed the columns of its source. Lines drawn between the panels represented the column-level joins, with Venn-diagram icons in the middle of each line showing the join type (inner, outer, left, right). Users could click any join icon to change the type directly. Each panel could be dismissed independently, returning the user to the canvas view.

The decision to keep join-type icons off the high-level star schema was deliberate. Two sources can have multiple joins between them with different types, and surfacing all of that complexity on the canvas would have created visual chaos and ambiguity. Collapsing to one line per source pair kept the schema readable at a glance, while the panel view preserved the full detail for anyone who needed it.

For users who already knew exactly what they wanted, a SQL join builder is more efficient. But for users who were still figuring out how their data fit together, a visual canvas made the relationships discoverable. You could see what was connected to what, which sources were underutilized, and where gaps existed.

The Infor integration

Infor acquired Birst specifically to bring our data integration approach into Infor's broader product portfolio. Infor had a rudimentary, product-siloed analytics tool, where data lived inside each individual product with no way to view it across products. Our work on Modeler was one of the main reasons Infor was interested in Birst in the first place. Centralizing Infor's data across all of its products was the end goal of the acquisition.

I spearheaded the integration work alongside my original Modeler co-designer, starting roughly three years before my departure from Infor. Infor already had a connection layer that linked their products to each other, but they were missing the transform and lineage experience that Modeler had built at Birst. Their existing tool was essentially a Prep and Relate in one module but with no star schema canvas and no transform pipeline. Just tables with results applied.

Infor leadership asked us to keep their connection flow intact but adapt the Modeler transform and pipeline patterns into their product. That meant:

Designing the pipeline within Infor's design system. We couldn't bring over the Birst component library wholesale. Infor's UX team had a defined design style guide, but it didn't include the custom components we needed (fan menus on pipeline nodes, styled connectors between steps, draggable panels for transform configurations, layouts that varied based on transform type). We worked alongside Infor's design and engineering teams to expand the style guide to include the components needed to communicate data flow clearly.

Bringing over 21 transform types into Infor's product. I designed and adapted many of these, including filters and joins specifically (the screens I'm including in this section show the Infor filter and join transforms). Each transform needed its own interaction model that fit Infor's design language while preserving what made Modeler work.

Designing the pipeline node interaction. I specifically designed the pipeline visualization within the Infor product, including the nodes representing each transform step, the lines and arrows connecting them, and the fan menu that opens when a user clicks a node to edit, delete, or view the table at that step. The pipeline pattern was central to letting Infor customers actually understand how their data was being parsed during this centralization push.

Adapting panel layouts to fit complex transforms. Infor's existing layouts couldn't accommodate the configuration depth that some transforms required. We designed new popouts, side panels, and bottom panels depending on the transform type and the space its configuration required. Some transforms needed wide horizontal space to show before/after data states. Some needed deep vertical space for complex parameter inputs. We codified those layout decisions into Infor's expanded style guide.

This work mattered because it took a pattern that had succeeded inside one product and proved it across Infor's broader portfolio. The architectural decisions my co-designer and I made on Modeler in 2017 to 2019 became the basis for how Infor's customers experience data integration across the entire suite.

Post-launch iteration

Modeler shipped and customers accelerated. They went from one of the clunkiest experiences in the industry to an industry-leading data integration flow, and they noticed. Adoption analytics from the production product showed customers adding source types they had historically avoided.

A few specific changes happened after launch:

• Module navigation got better. Customers wanted faster ways to move between Connect, Prep, and Relate, so we added shortcut buttons to each module in the header and built coach marks and help screens to introduce the three-module flow to first-time users.

• Mid-pipeline transform insertion came from customers. The original interactive pipeline supported delete and edit, but users started asking for the ability to insert new transforms between existing ones. We added it.

• The Publish button moved. Initially, Publish lived only in Relate, on the theory that Relate was the final step. We missed the use case where customers had clean raw data that didn't need transforming or joining. We rolled Publish out to all three modules and turned the button into a status indicator that showed the last publish date, current progress, and let users revert to a previous publish if needed.

• Two entry points stayed. Customers split on how they added a new connection in Connect, with about half clicking the icon in the main window and half using the plus icon in the side panel. Both worked. We kept both. Trying to consolidate them would have broken half of our customer base's muscle memory.