Can BPMN and Rules identify Complex Events?
I changed the blog title for more clarity. (Thanks to Paul Vincent).
Why are ‘complex events’ interesting for Adaptive Case Management? What is a complex event in difference to a simple one? Have you heard about edBPM, CEP, or CBE? One of the reasons to move on from flowcharting to Adaptive Case Management is the necessity to identify and deal with complex events. Clearly, the inability of BPMS deal with complexity led to the idea of edBPM.
‘Event-Driven Business Process Management’ or edBPM was suggested at the first CEP Complex Event Processing Symposium in 2006. The CEP Symposium takes place each year with current plans to expand the edBPM focus to Ubiquitous Complex Event Processing (U-CEP) that deals with events that are received from many different sources on the Internet in a kind of a publish/subscribe mechanism.
edBPM proponents suggest that it is BPM that leverages SOA, Event Driven Architecture, Business Activity Monitoring with Complex Event Processing (CEP) running independently to analyze processes and events. In fact, this is no more than a patchwork approach to squeeze some more usability out of flowcharts, but it doesn’t get rid of flowchart limitations.
Some see CEP as a means of finding repeatable patterns in massive data streams, while others see aggregate events defined by logic rules also as CEP. If they are linked into the context of a process flow then they become a Complex Business Event or CBE. (Don’t you just love TLAs?) Truly complex events do not present themselves in the form of simple, plausible logic and they will manifest themselves in several patterns over time. The orthodox school of BPM thinking that assumes that business operations can be encoded into flowcharts seems to believe that complex events can be predefined as well. They fall into the trap of oversimplified cause and effect chains. There is no such thing in the real world. Even an apparently simple cause is always mapped into a predisposed, complex context that itself has a causal network. The reality is that COMPLEX event patterns are difficult to recognize and define!
A Complex Business Event (CBE) is a complex construct of trigger(s) and context pattern(s) linked to action(s)! Complex means a trigger with a repeatibly recognizable signature that contains a certain chaotic element in that the signature and patterns leading to a common solution can vary substantially because of tiny differences in initial conditions. Using rules, the definition of complex events is no more than a trial and error approach with no reusability for similar future events. Boolean logic rules don’t do fuzzy patterns! I define the purpose of CEP as the recognition of event trigger signature patterns. But to identify the trigger is not yet an identified event that links to action! I propose that the CBE recognition has to be triggered by the event through the data and content used in the context of the processes and its business rules. To turn a CEP event trigger into a Complex Business Event the data and process context at the time of event has to be discovered. The external event, while apparently the primary cause, is in reality just the temporal trigger.
We have performed intensive research in this arena in the last five years with the result of announcing the first version of the User-Trained Agent in 2007 that discovers data patterns linked to events and thus discovers Complex Business Events automatically. The UTA has gone through a number of versions and can today not only be triggered by events to search patterns in current process data and content, but is also capable of identifying related patterns in already completed process activity. That means that the UTA considers the already executed parts of the process, all related previous events and also other case history. The state space for pattern discovery can also span multiple processes or cases or even a complete customer record. If the UTA finds multiple similar patterns that point to alternaitve solutions then it proposes a probability weighted list of possible actions.
Health insurance or patient care is an area where due to the very nature of the individual treatment, random and unexpected events must be linked to processes. Yes, it would be nice if there would be some simple way to infer general logic rules for all possible cases, but that is a fallacy. It ends up to be a completely manual procedure. If all events would be easily classifiable and represented as rule sets then CEP is no longer needed. Event-driven processes may use some rules, but they are not just rule-driven processes.
The key of CEP must be the ability to deal with utterly unexpected events that are only recognized by a person. It is that person’s knowledge that identifies the situation and moment of the event. The UTA is triggered by the user action to identify the related data patterns automatically.
Real-world CEP cannot be just rule- or flow-driven, as it must use some kind of machine-learning, pattern-matching technology to deal with unpredictability. Rules need PREDICTABLE definitions of expected conditions and cannot recognize fuzzy pattern similarities that can appear in many different ways! That does not mean that well-defined sub-processes and rules can not be included or linked to such auto-discovered complex events. GOAL-ORIENTED processes are the optimal way to react to unpredictability. Based on discovered event data, the UTA will actually suggest new goals. It means that the person is training the pattern matching software in recognizing the event patterns and which optional solutions can apply. The user is transferring his ability to recognize an event to the software without the need to infer rules. At the same time the user is not controlled by the software but he receives recommendations and his actions are again fed back into the training.
Obviously, the identification of the Complex Business Event works the better the more of relevant data that the user uses for his decisions are avaialble in the UTA-observed state space. Business Intelligence with all its implausible predictive analytics from random past data is not capable of performing such a feat!
Further, the BPMN events capability has nothing to do with CEP. Yes, one can just hang a BPMN task into midair and drive it from some event. Looks like it is an event driven task but what is that event? What is its signature? How does it happen if it is not created by another piece of BPMN flow someplace else? BPMN even lacks the data models to do it with rules. So BPMN/XPDL provides no benefit whatsoever for CEP or CBE as it lacks the necessary expressive capability.
Why do I propose complex events and rules as embedded ACM functions?
In ‘Mastering the Unpredictable’ I wrote about the problem of having to integrate standalone rule engines (BRE) with BPM. I see that as a substantial technological and management issue that drives up cost and complexity. Rules must be tightly integrated with the data content, process, user interface AND events to enable the automated discovery and human interaction. The same is true for CEP and CBE functionality. The current integration madness is not so because it is so great, but because it is sold this way! Current buyers need to decouple BREs and CEPs, because they run into substantial limitations with their BPM(N) flowcharts. Best-of-breed simply means best-of-integration projects and substantially higher maintenance costs.
There is this constant suggestions that at some time standards will solve the integration problem. That is many years in the future! But clearly, standards, adjectives and TLAs are simply smoke and mirrors. Some BPM vendors claim now to be adaptive, goal-oriented, rules-driven, and event-driven. Others propose that processes should be seen as a case-like organization of events, decisions and actions. Nice try, but there are no consolidated tools to do such a thing for the business users in BPM systems.
All the while, ACM Adaptive Case Management is portrayed by BPM pundits as the ugly duckling who is chasing his BPM siblings. But ACM is not a BPM enhancement and can’t be built by adding-on stuff to BPM fragments. ADAPTIVE already leapfroged all these TLA silos as a holistic, defragmented approach for a substantial, real-world business problem. The complexity of using CEP, CBE and BRE concepts disappears in a holistic solution and enables use in the real business world.