XSD Schema Architecture

MessageModelNamespaces

This subject matter here is architecture of XSD schema for an enterprise SOA, and an important side-theme is the exploration of automation potential in this area. The content is quite detailed, and perhaps not best suited for viewing in a web page; therefore PDF and Micorsoft Word versions are provided in the links below:

XSD Schema Architecture (PDF)
XSD Schema Architecture (MS Word)

This is an “insight” post; it aims to stimulate discussion around the subject it discusses; these articles may appear prescriptive and perhaps dogmatic, but the objective really is for debate that leads to reusable knowledge and that is accessible to the wider community of software engineers in the area of focus.

Please be aware that the content of the document is the output of work done in one SOA adoption context. The majority of the design choices were driven by standards, known best practise and some original thinking, however, a few decisions and choices were informed by the context in which the solution was being articulated.
The authors have moved on from some of these positions, and so feedback and criticisms will be very much appreciated, in order that all may get wiser.
Many thanks in anticipation.


Oyewole, Olanrewaju J (Mr.)
Internet Technologies Ltd.
lanre@net-technologies.com
www.net-technologies.com
Mobile: +44 [0] 793 920 3120

ESB Service Meta Design

OSB Service Meta Design

This post explores the idea of a meta model for the design of services in an Enterprise Service Bus (ESB), by identifying key competencies that services should realise. While there are exceptions to every case, the majority of services will find the realisation of these core competencies beneficial.

This is an “insight” post; it aims to stimulate discussion around the subject it discusses; these articles may appear prescriptive and perhaps dogmatic, but the objective really is for debate that leads to reusable knowledge and that is accessible to the wider community of software engineers in the area of focus.

This however is a meta design, so there is no code or example to show, neither is it advised to translate this directly into an implementation, without empathy for the usage context.

In my mind the core competencies of any enterprise service bus can be expressed in the acronym (REM); these are:
1). Routing
2). Enrichment
3). Mediation*

It is the complement of these core competencies that distinguish the ESB from other technologies within a SOA stack, and no business should countenance purchasing an ESB implementation if it does not provide these core competencies. Now, these core competencies are foundational and key, but there is a need for other ancillary comptencies to make for a compelling service offering, which will be introduced later. So, starting from the core, let us peruse the competencies.

Routing:
Routing is all about delivery and making sure that requests are delivered to the target most likely to return value to the client of the enterprise; often, logic is required to implement routing intelligently, including dynamic destinations. Routing by the ESB decouples the client from the provider/implementation of the service, so that a change of service provider might be limited in impact to a change of the rule(s) or computation of routing.

Enrichment:
Enrichment, as the name implies is adding information to an incoming or outgoing message. The simple case is where simple/atomic data item(s) are added from the context, but in some cases, the required data may need to be sourced from other services, in which scenario, enrichment takes the form of service composition.

Mediation:
I have marked out the third competence, Mediation, because it really is a composite of: security, protocol, message-pattern, data-format, etc. Mediation is the bridging of difference between two clients of the ESB; to wit, the client, and the provider. It is comparable to standing between the Taliban and US forces 🙂 In any conversation, there will be many differences. Mediation aims to bridge the gap, despite the many dimensions of difference, it may take various forms in one message exchange, even where canonicalisation is widespread.

In an implementation context, the service bus will sit on the boundary of the SOA, be it departmental or enterprise, and would be expected to provide certain value-add to applications, sub-systems, and services within the boundary, among which are:
1). Auditing
2). Security
3). Validation
4). Error management

Auditing:
The ESB is the best place for auditing of access to services since it is the first port of call of “clients” (external, and in some cases internal as well), therefore it is the most convenient location for keeping tabs on service usage parameters. One strategy I have witnessed is to log key-identifiers for each request; this makes it easy to trace problems through the rest of the enterprise, using only WS-Addressing headers.

Security:
If the ESB is fully delegated to man the boundary of the enterprise, it makes sense to delegate a significant chunk of the respobsibility for security to this layer. It is also a lot easier to implement a system in which all requests inside the boundary provided by the ESB, are to be trusted. The ESB becomes the frontier between the green zone and the red zone, said frontier being within the zone already protected by Firewalls and other security infrastructure that operate at the organisational/infrastructure level. This role (authentication and authorisation) may be realised by the ESB itslef, or further delegated to another component/layer within the SOA stack.

Validation:
Many years ago, the acronym GIGO was quite popular; it is not often heard these days, but it is still relevant. Spelled out it means Garbage in, Garbage out. For the enterprise, Garbage in is bad news. It means valuable resource is being given over to service requests that will eventually return no value to the client. The deeper into the enterprise this junk travels, the more expensive it is to the business and its ability to scale and be available to valid requests. The ESB, should filter out bad data, just as it filters out bad guys, leaving other services free to focus on their core competencies.

Error Management:
The service bus is an intelligent, real-time post office that has visibility of most of the traffic moving in and out of the enterprise. For this reason, it is able to apprehend requests whose signature is known to provide no value to clients of the enterprise, and for which there is no known owner/manager. The ESB should therefore provide a problem-handler of last resort within the enterprise, so that such maligned messages are handled predictably and by the most likely target that can transform the problem into a solution or a relationship.

Design:
Having said much about the core and ancillary competences, it is pertinent at this juncture to return to our meta-design, and to say, all that has passed before is a buffet, wisdom though is needed to discern where desert ends and the main course begins, and whether to have the cheese and jam, after lamb, or to go for a coffee instead 🙂

Design must therefore weigh the value of all these comptencies in light of the context of use, and determine if there is value in implementing the competency in the ESB. For example, security must be examined in light of the value/confidentiality of messages, as well as the completeness of existing security provisions. Where existing facilities are adequate, or the value of the data in the messages is limited, security may or not be a value-add in the ESB; there are other considerations as well, such as performance costs of security for high demand services.

Where beneficial, the ESB service should facilitate auditing of all service requests to the enterprise, irrespecitive of the validity of said requests. Validation should be enforced as the next step for incoming messages, to ensure that the request conforms to the service contract, validation should also occur as one of the last tasks performed on outgoing messages, to ensure that the enterprise honours its contract with clients. Requests that fail validation must be prevented from going any further into the enterprise and an indicative response must be returned to the client.
Most ESB services will be required to undertake some mediation, especially if the service is at the boundary of the enterprise; such mediation will be one of the first step for incoming requests, and the one of the last steps for outgoing responses. Mediation will be influenced by the extent of canonicalisation and centralisation patterns in use in the enterprise, and the degree of overlap between the enterprise, partner systems in its value chain, and clients of the enterprise.

Enrichment is an optional task, the need for which may decrease with canonicalisation, and in scenarios where services are deployed as a wrapper, or for decoupling in one of the following scenarios: client to service provider; SOA layer to SOA layer. However, every ESB service must perform some kind of routing, because every request to the enterprise will have a destination. Most scenarios will be static, where the destination is pre-determined and unchanging; or static-calculated, where the full complement of possible destinations are known and unchanging. In exceptional cases, and in very sophisticated SOA, dynamic routing comes into play where the destination is rule-based, or a lookup, in these scenarios, the routing takes a form of service composition where another service (rule engine, or database) is relied upon to determine the destination of a request. Typically, routing will be the last step for an incoming request. It is useful to make a note on error management; this is an event that has no specific sequence of incidence, however each service implementation should provide this competence, even if it is delegated. In whatever form this implementation takes though, it is important that it does not interfere with the context, especially where transactions are involved.

To conclude, it is advised that for each competence, a value should be attached against the cost of SOA-wide implementation against localised implementations, and provisions of the same competence by other technologies in the SOA stack; a useful tool would be a matrix that lists each competency against alternative technologies, and for each, to identify the cost and value of a SOA-wide or localised implementation; notes should be appended for any context-related issues. The outcome of such an exercise will go a long way to help an organisation identify a meta-design for how best to use the various competencies of an ESB in the SOA, be it departmental or enterprise.
The flexibility, efficacy and efficiency of the configuration arrived from a meta-desing will determine the success of the ESB implementation and the services therein.


Oyewole, Olanrewaju J (Mr.)
Internet Technologies Ltd.
lanre@net-technologies.com
www.net-technologies.com
Mobile: +44 [0] 793 920 3120

Test Automation With Plain SoapUI

Test Automation mit SOAP UI

Automation Illustration

Have you ever wanted to run some features that are only available in SOAP UI Pro?
Well, a little grasp of Groovy scripting can help you to simulate some of those high-falutin features
that would have cost you $$$ per year in licenses.  Here is one neat feature that I really like.

Running All Projects in a Workspace

This feature is not available in the free (community version) of SOAP UI, but the Groovy script
described in this post will do the same thing, without hurting your wallet.

Here are the steps for using the project file in SOAP UI

  1. Open or create a new Workspace in SOAP UI
  2. Add/import all the projects you want to run into the workspace
  3. Import the test harness SOAP UI project.  You can download the project file from > here <
  4. Remove any projects that are to be excluded
  5. Expand the “ExecutionHarnessExample” project and run the only TestStep – Groovy script “RunProject”

The script will loop through all the test Projects, TestSuites, TestCases and TestSteps, while also printing a summary of the progress to screen, and where indicated, logging the complete test logs to a file.  The script prompts the user for the path to the log file.  Accepting the default value “DO-NOT-LOG”, turns off logging.

soapUIDoNotLog

Enter a valid path to enable logging.  The name of the log file is saved to a global property, so it is available even after the test run (in case you forget)

soapUILog

A useful feature of the script is that assertions are exercised, so you can see TestSteps turning green or red as they are being run!

soapUI2

The script is quite simple, and you can open it in SOAP UI for a detailed walk-through, but below are the salient things:

// Save the name of this automation project
//it will be needed later to prevent infinite looping
def executionProjectName = testRunner.testCase.testSuite.project.name// Get a handle on the SOAP UI workspace
def testWorkspace = testRunner.getTestCase().getTestSuite().getProject().getWorkspace();// Ask user to confirm if test output should be logged to default file
filePath = ui.prompt(…);// First, a loop is created for all the Projects in the in the SOAP UI Workspace
for (int projectCounter = 0; projectCounter < testWorkspace.getProjectCount(); projectCounter++) {// Exclude the automation project else you get an infinite loop!
if (testProject.name != executionProjectName) {// Exclude disabled TestSuites from the loop
for (… (!testProject.getTestSuiteAt(suiteCounter).isDisabled())); suiteCounter++) {// Create a TestRunner for running the TestSteps in this TestCase
testCaseRunner = new com.eviware.soapui.impl.wsdl.testcase.WsdlTestCaseRunner(testCase, null);// Run each TestStep in the TestCase
testStepResult = … testStep.run(testCaseRunner, testStepContext);// Log the output if requested
if ((filePath != null) && (filePath != ”) && (filePath != ‘DO-NOT-LOG’)) { testStepResult.writeTo(fileWriter); }// Close the output file when all Projects have been processed
fileWriter.close()

 

At the end of the test run, you can view the log file in a text editor like TextPad and view the messages exchanged (including headers) as well as timings.

soapUILogFile

And that’s all folks; I hope you found the posting useful.
It has certainly helped me put off the purchase of a SOAP UI licence for another day 🙂
God bless!


Oyewole, Olanrewaju J (Mr.)
Internet Technologies Ltd.
lanre@net-technologies.com
www.net-technologies.com
Mobile: +44 [0] 793 920 3120