“What we want to do is have something that’s streamlined so that we can take all of the systems of record, whether that be our Colleague student information system, our CRM, or Canvas–any sort of data that could exist in our spider web technology stack–and then bring it into a staging area where we can transform it using logic before it gets loaded into a nice, clean enterprise data warehouse that would then be reliable for business intelligence and analytic reporting.”

To transport the data and make changes along the way, SRU is leveraging Flows.

How flows solve merging data sources & much more

Flow is more than classical middleware. What “classical middleware” sits somewhere in the center of a number of interacting applications and it has the responsibility of keeping any two systems from being dependent on one another’s exact configuration. Sometimes middleware strictly busses information from one location to another, while at other times it caches the data and retains it. 

We all have felt the pain of hearing the marketing team come up with a great idea (which involves another cloud application) and realizing we have to change everything upstream and downstream to make sure the systems continue to play nicely together. 

When you have a spider web of dozens and dozens of enterprise applications, the permutations of change mean that your entire ecosystem as a whole is always in flux. Middleware makes it so when one system’s configuration changes or a system is added, the only thing that is altered is its relationship to the middleware.

We should mention that for every rule, there are exceptions. There are integration protocols, such as LTI, which means that two systems are natively very closely coupled together at a presentation and view layer. There are systems, such as OAuth protected systems, where the user is actually going to make a call from the browser to bring data in and there won’t be a system in the way. So your Microsoft calendars, your Google documents, your Canvas or Moodle or Blackboard may very well have an LTI integration with OAuth in play. 

A conventional middleware has no relationship to these exceptions (for example, LTI Integrations or OAuth). Pathify and Flow operate a bit differently while remaining entirely cognizant of the necessity of airtight data security.

We used to block users based on whether or not they were physically connecting to routers on campus. Post global pandemic, we can’t say that anymore. The student portal often finds itself occupying a position of gating access to other systems. Although Pathify is not an identity provider and Flow is not an identity provider, we have several contexts in which we perform an identity mediation function, which is to say that a log-in occurs directly in Flow, not the gated system. Flow actually enriches the payload and adds information about the user. 

This is a vastly more efficient pattern. Rather than two systems back behind a wall exchanging database information, now it can just be done on demand. Authentication is treated as if it were just another integration step to be enriched and to be enhanced. In this manner, Flow is an application server, and it serves the function of standing as a web listener. It enables requests to be served. It enables resources to be loaded onto it. 

Flow sits in a position in your ecosystem that:

Flow safeguards your precious database credentials and API keys and then provides trusted access to back systems. 

We mentioned that endemic to our industry are things like proprietary systems (and data structure and “magical string codes”). We know that human interpretation problems are usually the most fallible part of a processing system, but we also know that they’re one of the most flexible. They’re there for a reason because you cannot anticipate everything upfront, and all of the data can’t be naturally represented. 

Flow was designed with the expectation that this was how it was going to be in higher education. Pathify knew that it wouldn’t be possible to just take a plug and connect it into a socket. For that reason, at many, many inflection points during a Flow process, you will find a processing function that enables you to do arbitrary scripting against a payload and transform it into whatever you need it to be for the downstream layout. How we express this scripting is JavaScript. (We chose it because it’s dominant in Microsoft ecosystems, in Linux-based ecosystems, in web-based design, and in gaming platforms all over the world.)

At this point, Flow has, like arrows in its quiver, some 220 distinct integrations across educational systems. The knowledge and patterns gained from those are available to Flow partners as a playbook. They’re not necessarily something that you would grab and just plugin, but they are something that you will adopt and adapt. We think that the right way to approach the flexibility required for a tertiary ecosystem is to present a pro forma use case and say, “This is how it tends to be,” and then tweak and refine for your particular use case.

Does this sound overwhelming? This is a complex problem, and the answer is, by necessity, comprehensive. However, as we always say, you don’t need to do any of this alone. At Pathify, we pride ourselves on our excellent partner support.