Integration process monitoring - possibility for business?

Many times integration process monitoring is implemented once something goes wrong for the first time. Usually it is seen as a mandatory cost for being able to operate integration environment and not a driver for better visibility for business stakeholders. Traditionally process monitoring has also been a good cash cow for integration solution providers as they have have been able to sell custom monitoring solutions and update existing integration solutions to support monitoring. This is the first part in a series of blog posts which aims to define how process monitoring should be implemented.

Typical monitoring story

Over the years I have been involved in numerous system integration projects where process monitoring has been implemented only after operating the developed solutions has become too cumbersome. There are few reasons for this as I see it:

Customers don’t fully understand the benefits that good monitoring can provide (or the cost of NOT having monitoring). Monitoring is easily seen as purely technical compontent that only benefits IT-people, not the business itself.
Integration platforms don’t have out-of-the-box solutions for business process monitoring. There are various levels of technical monitoring capabilities, but most of the products don’t include tools for monitoring the business processes. At least not in sensible way.
Integration solution providers want to be able to set the project prices as low as possible. Many times operating the solutions is not responsibility of the developing party so leaving monitoring out does not affect them. Therefore it is the implementation of monitoring that sadly gets cut out from the tenders to lower the prices. And it also gives an opportunity for aftersales once the customer is in trouble. Obviously this is not the way business should be done.

Obviously there are also customers that are very well aware of the benefits of good monitoring solution and naturally there are solution providers that have at least tried to include monitoring as part of their default delivery. The fact still remains: there is lots of room for improvement.

How can we improve the situation?

Customers buying integration solutions should be educated about the benefits of good process monitoring. In my opinion process monitoring should be seen as an enabler of better business the same way as automated processes. It is not only for techies, good monitoring tool is both for IT and business. For example you can have automated invoicing process, but how can you tell which invoices have been delivered (or paid) if you have no monitoring solution? In this case monitoring can be seen as part of your risk management.

Customers should not accept integration solutions that have no monitoring (logging files to disk is not monitoring!). On the other hand it should be accepted that this all comes with a price tag. Retrofitting monitoring solution usually is much more expensive than doing things right in the first place.

Integration solution providers should be proud enough of their profession so that they would not sell solutions that they know will lead the customer into problems. Even if the consultant developing the solution does not have to deal with the results once in production, it should be realized that the customer most likely will have to live with the solution for years.

It is the responsibility of the solution providers to make sure that their customers really understand what it means in terms of lost business benefits and increased amounts for IT-support manhours, if no real monitoring is implemented.

In the long run I think it is better business also for the solution providers to think long term effects of the provided customer service - sometimes fulfilling the specification requirements is just not enough. In my opinion this does not mean giving out work for free, it just means that solution providers should have courage to include also the “hard selling parts” to the tenders and to be honest towards their customers.

There will always be someone that can go lower in prices, but it is much harder to become the best in town in customer service and quality. Therefore it is better to differentiate yourself from the competition with excellence rather than with cheapest prices.

Qualities of good monitoring solution

What should we look for in a good monitoring solution? I will go through these qualities in more detail in future posts, but here are the main characteristics that I would look for:

Suppports different user groups and ways of presenting data

Many times monitoring solutions have only one way of presenting the process data. And most likely it contains all the possible technical details that has been collected. For the business user who is only interested whether the process that she is interested has been executed or not, this can be rather overwhelming. In fact there should be possibility to show “traffic light” type views for more business oriented people and then all the bells and whistles for the technically oriented gurus. Filtering processes based on the area of responsibility is also essential. It doesn’t make any sense to show invoicing data to HR-people. And vice versa.
“Everything” can be parameterized

This is probably the biggest issue that I have seen with most of the custom monitoring solutions. Many times what gets logged from the processes is defined in the code level (for example process name, event type, description, system name, etc.). This leads to a situation where even the smallest of changes to the monitoring data can take ages. Good monitoring system only needs an ID for an event to be able to add all the metadata related to the event. For example changing event type from error to warning with parameterized event metadata can be done in seconds without any code changes.
Lightweight to add to implementations

Adding monitoring to technical implementations should be as easy as possible. If adding monitoring to the solutions is painless experience for the developers, most likely it gets done better. Obviously this has link to the costs also.
Supports different sources of data

We are moving towards world where also integration solutions are scattered around cloud and on-premise worlds. Even if you have set up integration middleware to do your EAI workloads, it is not said that the whole process flows through your platform. To build a real monitoring view of the whole business process, it is essential to be able to combine data from different sources. For example you might have a process that starts with the end user usign a web-portal. Or the public endpoint for your process could be hosted in cloud and the rest of the process is running in your on-prem datacenter.
Supports alerts

Having good monitoring data available when solving problems is essential. Getting alert the moment there is critical problem is priceless. One of the main characteristics of monitoring solutions is that they are looking backwards in time. Usually it is easier to get hold of the source of the problems the sooner you get notified of the problem. Having solid alerting system as part of the monitoring solution can save lots of time and money.
Supports reporting

Monitoring solution itself does not need to become the center of your business reporting, but it can be used to collect the data for your reporting needs. Good monitoring system at least enables collecting business data from the processes and exposing it to reporting purposes.
Easy to use and easily accessible

UX has not been a key selling point when talking about integration solutions. Or at least it normally doesn’t show. You might need to use remote desktop connections to certain servers to even see the monitoring data. To really get all stakeholders on board, your monitoring solution needs to be easy to use. And you should be able to access the data from anywhere you want.
Evolving and future proof

Many times monitoring solutions are custom projects and therefore can be either very expensive or limited in functionality. It also means that usually monitoring systems are not actively developed as it would just mean more costs. The worst case being of course that some commercial solution which in the end does not meet the requirements is chosen.

In many industries companies end up using “de-facto” solutions which are used by several competitors also. This way the risk of ending up as the only user of the solution gets smaller. One option of cource is to choose an open source solution so that vendor lock-in can be avoided.

I haven’t really found any commercial or open source solution that would meet all the previous charasteristics. If you have, I would be more than happy to hear about them!