This document describes the message format for TermWeb Business Integration Message Adapter (BIMA). BIMA is used for integration of TermWeb with other enterprise systems via a message broker middleware, such as WebSphere MQ, ActiveMQ, RabbitMQ, Apache Kafka.
BIMA uses the Java Message Service (JMS) standard and Kafka Streams.

Introduction

There are two TermWeb operations that causes BIMA to generate messages: initial transfer (export of term database) and update of term data.

When such an operation occurs, BIMA generates one or more messages and publishes them to the message broker's publish/subscribe topics defined in BIMA's configuration file.
Each message consists of several properties and a message body, which is a XML structure. The properties are useful for applications when filtering messages to avoid unnecessary XML parsing from the body.

Installation

• Copy the sample configuration file BimaAdapter.properties to [TERMWEB_HOME] and edit it as necessary

• Edit [TERMWEB_HOME]/termweb.properties and add the following lines:

  adapter.class.1=com.termweb4.bima.BimaAdapter
  adapter.config.1=[TERMWEB_HOME]/BimaAdapter.properties

• If you’re going to use JMS you may replace activemq libraries with another: RabbitMQ, IBM MQ. If you’re going use another MQ implementation copy any message broker specific jar files to [TOMCAT_HOME]/webapps/ROOT/WEB-INF/lib that BIMA needs to create instances of TopicConnectionFactory and Topic. By default, TermWeb 4 comes with ActiveMQ implementation files, don’t forget to delete to avoid any conflicts:

  • activemq-client-*.jar

    • // And its transitive dependencies
      org.apache.geronimo.specs:geronimo-jms_1.1_spec:1.1.1
      org.apache.geronimo.specs:geronimo-j2ee-management_1.1_spec:1.0.1
      org.fusesource.hawtbuf:hawtbuf:1.11

Configuration

Name pattern

The name pattern for the different topics is a text string with variables that will be replaced with values depending on the dictionary, section and language that are affected by the update message. The variables are:

• <DICTIONARYID>

Is replaced by the id of the dictionary.

• <SECTIONID>

Is replaced by the id of the section.

• <LANGUAGE>

Is replaced by the language iso code. Only applicable for term update messages.

• <TAG>

Is replaced by the tag the user can enter when selecting an initial transmission. Thus, only applicable for initial transmission messages.

Adapter administration

When the adapter is installed in TermWeb an BIMA administration panel is available in TermWeb. Log in as administrator and go to the AdminView → System → BIMA. The administration panel opens.

General panel

The general panel allows the adapter to be enabled and disabled. It also allows temporary disable one of the parts JMS or Kafka. Please note that the setting in BimaAdapter.properties is not affected by this control. When restarting the adapter will be enabled or disabled according to the value in the properties file.

Information about current configuration is also displayed here.

Export panel

The export panel allows the administrator to do an initial transfer of all or a part of the term data in a dictionary. The selection of concept, terms and fields are based on regular Export settings used for normal exports.

Select settings and topic

Start by select one of the available export settings. If no setting is appropriate for your needs, you can click the “Create export setting” link which opens the Export settings panel and lets you define a new setting. Save the setting and then select the BIMA administration icon again in the AdminView → System and select the newly created setting.

Continue by entering a name of the topic to which the data should be published. Choose type of message broker configured in adapter (JMS or Kafka)

Click on the “Next >” button.

Confirm your selection

Your selected export setting is now displayed in detail with languages, fields and total number of concepts that will be published when using this setting. By moving your mouse pointer over the language and field summary you will see a list of all languages and fields included.

If this is the data you expected, click the “Start export” button to start publishing the messages. Otherwise, click the “< Back” button to go back and select another export setting and/or topic name.

Export progress

The selected concepts are now retrieved from the database, converted to XML segments, and published to the selected topic. For details about the message format, see section Message structure below.

A progress bar indicates how much of the export is completed. The elapsed time and an estimation of the total time is also given.

You can stop the export by pressing the Stop button. Please note that this may cause receiving systems to only get parts of the data intended.

Export summary

When the export is done, or if the export was stopped by the user, a summary is displayed with the total number of concepts published, together with elapsed time.

Message structure

Text

Message properties

The following specific properties are defined in a JMS message:

Initial transmission

During an initial transfer (described in Adapter administration above) BIMA will create a message (or several messages if the message is larger than the maximum message size specified in the config file) for each section in the dictionary being exported. The TransmissionType in the message is set to INITIAL. DictionaryId and SectionId are set to their respective values. All other properties are set to null.

The body of the message is an XML structure containing all the concepts being exported from the section, which could be none if the export setting's filter does not include any concept from this section.

Update transmissions

An update transmission occurs when some term data is updated in TermWeb. This is done by either renaming or deleting a section, creation, modification, or deletion of concepts, or by importing a file. BIMA then creates one or more messages and publishes them to the broker.

Section updates

Messages caused by section updates are published to the section update topic(s) defined by topic.update.section in the config file. All messages have DataType set to SECTION.

If a section is renamed a message is published to the topic with DataAction set to UPDATED. The XML in the body consists only of the dictionary and section. The new name is found in the “name” attribute of the section tag.

If a section is deleted a message with DataAction set to DELETED is published. Only dictionary and section in the XML.

Term data updates

For every change there will be a message for the entire concept together with a message for each affected term. There is therefore a redundancy in the messages since the data in the term messages also are included in the concept message.

Concept messages are published to the concept topic(s) defined by topic.update.concept in the config file. Term update messages are published to the topic(s) defined by topic.update.term. The subscribing client applications can thus select the most appropriate data structure.

In the group of messages, the concept message has DataType set to CONCEPT, an appropriate value set for DataAction and the body is the XML for the entire concept. Each term message has DataType set to TERM, a DataAction value as appropriate and the XML structure for the term.

When importing a file there will be a series of messages, just like if every concept was created or modified manually. The messages will be sent to the broker when the import of the file is completed.

Description of term data update transmissions

The messages generated from the different kinds of updates are as follows.

Creation of a new concept

One message for the entire concept, and one message for each term in the concept. All have DataAction set to CREATED.

Deletion of a concept

One message for the entire concept, and one message for each term in the concept. All have DataAction set to DELETED.

Modification of a concept

One message for the entire concept with DataAction set to UPDATED.

For every term that has been created, there is a message with the term data and DataAction set to CREATED.

For every term that has been deleted, there is a message with the term data and DataAction set to DELETED.

If any term field except the term name or language has been modified, there is a message with the term data and DataAction set to UPDATED.

If the term name or language has been changed, the old term is considered to have been deleted and a new one created. Therefore, this will create two messages: one DELETED and one CREATED.

If any field on the concept level has been modified, a message with DataAction UNMODIFIED is generated for every term that has not already generated a message with DataAction UPDATED, CREATED or DELETED. Subscribing systems can then determine if some action needs to be taken with the data.

If the section has been changed, the concept is considered to have been deleted and new one created. The messages for the deletion are published to the old section's topic and the creation messages are published to the new section's topic.

XML structure

The XML structure for the messages is like the structure of a TermWeb XML export file and is described in TermWeb-BIMA.dtd.
Message example:

<dictionary id=”8” name=”Automotive terms”>
  <section id=”16” name="Headings">
    <concept id="4116633" termno=”430”>
      <metadata>
        <createdOn>1999-05-12 03:12:54</createdOn>
        <createdBy>admin</createdBy>
        <changedOn>2004-10-04 13:53:27</changedOn>
        <changedBy>admin</changedBy>
      </metadata>
      <field name="Domain">All domains</field>
      <field name="Origin">Termlex</field>
      <field name="Term No.">430</field>
      ...
      <term>
	<metadata>
	  <createdOn />
	  <createdBy />
	  <changedOn>2004-10-04 13:53:22</changedOn>
	  <changedBy>admin</changedBy>
	</metadata>
        <field name="Term">Lubrication, injection pump</field>
        <field name="Language">en-GB</field>
        <field name="Region"/>
        ...
      </term>
      <term>
      ...
      </term>
      ...
    </concept> 
    ...
  </section>
  ....
</dictionary>

As previously stated, there are one message containing the entire concept and one for each term affected for every term data update. The XML structure in a concept message contains exactly one <concept> tag with corresponding content. For a term message, the body contains exactly one <term> tag with content. Since 2005-01-31 the Status field on the concept level is also included in every term message before the <term> tag.

Kafka integration

Integration for Kafka is using the same message structure. By default, all messages serialized as JSON strings. XML part of message will be stored in a field called body.

Keys for messages will be like topic name with additional suffix like part number or entry id.

For initial transmission key will be <topic-name>_p1-p100, where p1 and p100 sequential position number of first and last element in message.
Numbering starts with 1.

For update transmission key will be <topic-name>_C1234 or <topic-name>_T1234 depending on type of message. C1234 is and id for Concept and T1234 id for Term. In case of Section change BIMA will create two messages one with ending suffix _deleted and another one with _created.