TeamForge Webhooks-based Event Broker, which is also referred to as the integration broker, is a webhooks-based message broker that pushes the messages of specific events received from a Publisher to a Subscriber.

TeamForge Webhooks-based Event Broker - An Overview

TeamForge Webhooks-based Event Broker

is a light-weight webhooks-based message broker integrated with TeamForge to validate, transform, and route the messages that are received from a publisher to a subscriber.
acts both as an event broker and as a repository to store the messages that publishers intend to send to the subscribers.
is a robust message delivery middleware with minimal failure points.
can be used in the place of TeamForge EventQ, but is typically not a replacement for TeamForge EventQ.
can handle hundreds of messages over HTTPS per second.
provides a guaranteed once & once-only, in-order delivery of messages.
defines the Events, Publishers, and Subscribers.

Architecture

TeamForge Webhooks-based Event Broker is all about the events, publishers, subscribers and event messages that the publishers send to the subscribers. It also provides a mechanism to store the messages in the Inbox, and to push them to the Outbox, from which the messages are sent to the subscribers.

With TeamForge Webhooks-based Event Broker, Publishers/Subscribers can publish/subscribe to multiple events. It uses in-built TeamForge plugin to authenticate and send messages to TeamForge provided that the subscriber name starts with TeamForge.

Events

When an event occurs, the publisher sends the messages for that specific event to TeamForge Webhooks-based Event Broker which in turn sends it to the subscriber. Events must be registered in TeamForge Webhooks-based Event Broker. Each event is identified by a unique name. Registering an event is required before you register the publisher or subscriber.

Publishers

Whenever an event occurs, Publishers send messages to subscribers via TeamForge Webhooks-based Event Broker. Each publisher gets a unique REST endpoint for each event to publish. When you register a publisher, you will create the publisher name and select the event on which the publisher need to send the message.

Subscribers

Subscribers receive the messages sent by the Publishers via TeamForge Webhooks-based Event Broker. Subscribers can provide a Web endpoint where messages are to be pushed. Each subscriber gets a dedicated sender that sends messages on a First-in-First-out (FIFO) basis to the endpoint.

Inbox

TeamForge Webhooks-based Event Broker is designed using a box model. All messages published to TeamForge Webhooks-based Event Broker are stored in the Inbox and a unique reference number is generated and sent back to the publisher for storage and future retrieval.

Outbox

When a message is received by TeamForge Webhooks-based Event Broker, the subscriptions for the event represented by that message is checked. Multiple records or one separate record for each subscription, are then inserted into the Outbox, along with the Subscriber details. The status is set to PENDING.

TeamForge Webhooks-based Event Broker then sends the messages from the Outbox to the Subscribers. Once successfully delivered, the message in the Outbox is marked as DELIVERED.

Message Receiver

Message Receiver receives event messages from multiple publishers, stores in the Inbox and returns the unique Inbox message reference number (InboxId). This is completely a concurrent process.

Replicator

The Replicator is a single process that periodically replicates Inbox messages to the Outbox for the subscriptions active at that point in time.

Sender

Each Subscriber gets a unique Sender process. Each of these processes run in parallel and can process Outbox messages concurrently. However, each Sender sends the messages to the Subscriber in sequence.

Install the TeamForge Webhooks-based Event Broker

You can install the Webhooks-based Event Broker after installing TeamForge. As the Webhooks-based Event Broker requires relatively less RAM, you can install the Webhooks-based Event Broker on the TeamForge Application Server itself.

You should have the TeamForge installation respository configured as part of the TeamForge installation.

Do this on the TeamForge Application Server to install the Webhooks-based Event Broker.

  1. Run this command:

    yum install teamforge-webr
    
  2. Add the Webhooks-based Event Broker services (webr and webr-database) to the host:SERVICES token in site-options.conf file.

    For example:

    host:SERVICES=ctfcore ctfcore-database ctfcore-datamart search mail etl binary reviewboard reviewboard-database reviewboard-adapter cliserver webr webr-database
    
  3. Provision services.

    teamforge provision
    

TeamForge-Jenkins Integration Using TeamForge Webhooks-based Event Broker

Configure Jenkins Integration Plugin to Notify the Webhooks-based Event Broker

A new version of Jenkins integration plugin, v2.0.6, is released with TeamForge 18.3, which if configured can notify either the native TeamForge Webhooks-based Event Broker or EventQ about the build data.

CollabNet Plugin Features

  • Notify EventQ when builds complete. The CollabNet Plugin must be installed once on each Jenkins server you wish to connect to TeamForge/EventQ.
  • Notify TeamForge Webhooks-based Event Broker when builds complete.
  • Authenticate users from TeamForge. If setup as the “Build & Test” application, it can even use Single Sign-On.
  • Authorization from TeamForge, including the ability to set permissions in Jenkins based on roles in your TeamForge project.
  • Upload the build log or workspace artifacts to the TeamForge Documents.
  • Upload workspace artifacts to the TeamForge File Release System, as a post-build publishing task or as a build promotion task.
  • Open/update/close TeamForge Tracker artifacts based on the Jenkins build status.
  • Upload workspace artifacts to the Lab Management Project Build Library. (Requires CollabNet Lab Management).

    Click here to know more about the requirements for installing the latest CollabNet plugin.

It is highly recommended that you configure Jenkins to notify the TeamForge Webhooks-based Event Broker for TeamForge-Jenkins integration.

Use the following instructions to have the Jenkins integration plugin notify the native Webhooks-based Event Broker.

  1. If you are integrating TeamForge and Jenkins for the first time:
    1. Log on to the Jenkins Server as a previliged Jenkins user, navigate to Manage Jenkins > Manage Plugins > Available.
    2. Select the latest CollabNet Plugin and install the plugin.
    3. Restart your Jenkins server.
  2. Existing TeamForge-Jenkins integrations that use CollabNet Plugin v2.0.4 (or earlier):
    1. Log on to the Jenkins Server as a privileged Jenkins user, navigate to Manage Jenkins > Manage Plugins > Updates.
    2. Select the latest CollabNet Plugin and install the plugin.
  3. Existing TeamForge-Jenkins integrations that use EventQ Jenkins Adapter v2.0 (or earlier) plugin:
    1. Log on to the Jenkins Server as a privileged Jenkins user, navigate to Manage Jenkins > Manage Plugins > Installed.
    2. Select the EventQ Jenkins Adapter v2.0 and click Uninstall.
    3. Select the Available tab.
    4. Select the latest CollabNet Plugin and install the plugin.
    5. Download the migrate_jenkins_plugin.sh script and save it to <JENKINS_HOME_DIRECTORY>/jobs/.
    6. Change ownership of the migrate_jenkins_plugin.sh file.
      chmod 755 migrate_jenkins_plugin.sh
      
    7. Run the migrate_jenkins_plugin.sh script.
      ./migrate_jenkins_plugin.sh
      
  4. Configure an Individual Jenkins Job to notify the TeamForge Webhooks-based Event Broker.

    1. Create a build source and have the configuration details handy for the following steps.
    2. As a privileged Jenkins user, locate the job you wish to report build data to TeamForge Webhooks-based Event Broker and navigate to its configuration page.
    3. Add a post-build action to Notify TeamForge/EventQ when a build completes.
    4. Select the Notify TeamForge check box.
    5. Enter the TeamForge WebHook URL to which the build data will be sent, Username, and Password in the respective fields.

    6. By default, the Optional TeamForge Association View check box is selected. If required, you can override the global configuration by entering the TeamForge URL and user credentials.
    7. Save the job configuration.

    8. Run a build to test the new configuration and verify configuration. Information and errors will be reported to your Jenkins log and to the build console.

Migrate Existing Jenkins Data from EventQ to TeamForge

If you have configured the new Jenkins integration plugin v2.0.6 to notify the TeamForge-based Webhooks Event Broker, you need to migrate the Jenkins data from EventQ’s MongoDB database to TeamForge database, after upgrading to TeamForge 18.3.

To migrate the Jenkins data from EventQ’s MongoDB to TeamForge database:

This migration process is two-fold:

  1. Extract the existing Jenkins data from EventQ’s MongoDB database and generate an SQL file based on the database option chosen.
  2. Execute the generated SQL file on the TeamForge database (Postgres/Oracle).

Extract Jenkins Data from EventQ MongoDB

  1. Download the Jenkins plugin jenkins-1.0.jar.

  2. Run this command to execute the migration script.

    java -jar jenkins-1.0.jar -migrate
    
  3. Enter the MongoDB hostname as localhost or just press ENTER for localhost to be taken as default host name.
  4. Enter the MongoDB port number or just press ENTER for 27017 to be taken as default port number.

  5. Enter the MongoDB database name for EventQ.

  6. Enter the MongoDB username and password.

  7. For Does TeamForge use Oracle Database [y/n]:, press y if you use Oracle. Press n if you use PostgreSQL.

    The migration script is executed and generates the jenkins_data_migration.sql file.

Execute the SQL File on PostgreSQL/Oracle Database

  • To execute the SQL file on PostgreSQL Database:

    1. Log on to your PostgreSQL database.

    2. Run this command to import the migrated data.

      sudo /opt/collabnet/teamforge/runtime/scripts/psql-wrapper <filepath of `jenkins_data_migration.sql`> 
      

      OR

      cat <filepath of `jenkins_data_migration.sql`> | sudo /opt/collabnet/teamforge/runtime/scripts/psql-wrapper 
      
  • To execute the SQL file on Oracle Database:

    1. Log on to your Oracle database.

    2. Run this command to import the migrated data.

      @<filepath of `jenkins-data-migration.sql`>
      

TeamForge-JIRA Integration Using TeamForge Webhooks-based Event Broker

A new JIRA integration plugin 1.0 has been implemented in TeamForge 19.0 to integrate TeamForge with JIRA using TeamForge Webhooks-based Event Broker.

JIRA Version Information

The following table provides the JIRA product version and the version of the new TeamForge-JIRA integration plugin 1.0.

ProductVersion(s) SupportedIntegrations Supported
JIRA Software7.6TeamForge-JIRA-adapter-1.0

Configure JIRA Integration Plugin to Notify the TeamForge Webhooks-based Event Broker

This method of integration is based on the new TeamForge-JIRA integration plugin 1.0 and is recommended for on-premises installation of JIRA. The TeamForge-JIRA integration via TeamForge Webhooks-based Event Broker brings associations and traceability to JIRA. Traceability is therefore possible between JIRA, TeamForge, and various tools supported by TeamForge Webhooks-based Event Broker.

Within JIRA, all the SCM related activities for the particular JIRA issue are tracked and displayed under the TeamForge tab on the JIRA issue page.

The TeamForge tab provides a summary of associations and also details of a full listing of associations. This is a listing of immediate associations, activities with direct relationships to the JIRA issue at hand. To explore the chain of associations, click the Trace button. The traceability chain that includes commits, builds, reviews, deploys, and other XDS-based integration is displayed. On the Trace Associations page, clicking any node lights up all the association paths to that node. In addition, a small pop-up appears with details about the node in question. You can use SET AS TARGET button to expand the associations from the selected point. Another function exists to SEE more details about the node, opening a new browser tab.

The TeamForge-JIRA plugin 1.0 is packaged as a JIRA “add-on”. Once installed and configured, the add-on supplies issue tracker “work item” metadata to TeamForge. The TeamForge-JIRA plugin 1.0 needs configuration on a per project basis in JIRA, such that JIRA (many) to TeamForge (one) project mappings are established. It is therefore required to configure a TeamForge URL, Webhook URL, and a set of credentials each for TeamForge and Webhooks for each JIRA project. Note that once configured with the aforementioned URLs and credentials, the JIRA integration plugin will create and manage the needed sources.

Install the JIRA Integration Plugin

Use the TeamForge-JIRA plugin 1.0 to notify TeamForge of updates to JIRA issues and to vizualize the associations between JIRA and other tools. The TeamForge-JIRA plugin 1.0 must be installed once on each JIRA server you wish to connect to TeamForge. Install the TeamForge-JIRA plugin 1.0 using the JIRA add-on Manager. Refer to the Atlassian Marketplace for supported versions.

Install the TeamForge-JIRA Associations Add-on

  1. As a privileged JIRA user, navigate to Administration > Add-ons > Find new add-ons.
  2. Search the marketplace for the TeamForge Associations add-on for JIRA. Type ‘TeamForge Associations’ and search.
  3. Click Install.

Upgrade the TeamForge-JIRA Associations Add-on

New versions of the TeamForge-JIRA associations add-on will be visible in the Manage Add-ons section (Administration > Add-ons > Manage Add-ons).

Configure the JIRA Integration Plugin

Configure “TeamForge Associations” to notify TeamForge Webhooks-based Event Broker about work item activities.

Before you start configuration, make sure that you have installed the TeamForge Associations add-on: verify that TeamForge Associations is present in the User-installed Add-ons list.

  1. Configure the add-on from the Project settings page to notify TeamForge about the work item activities.
    1. Select Projects from JIRA Administration menu.
    2. Select your project from the list of project on the Administration page.
    3. Select TeamForge on the Project settings page.
    4. Enter TeamForge URL in the TeamForge URL field.

    5. Enter valid TeamForge login credentials in Username and Password fields.

    6. Enter the Webhook URL to which the JIRA work item activities are notified, Webhook Username, and Webhook Password in the respective fields.

    7. Click Test Configuration.

    8. Click Save to save the configuration.

Sync Existing JIRA Issues into TeamForge

To associate TeamForge objects to existing JIRA issues, click Sync Issues.

This step bootstraps JIRA issue data from the current project into the TeamForge. Note that this process may take several minutes (even hours) to complete, depending on the number of issues in the JIRA project.


Edit TeamForge-JIRA Mapping Configuration

Click Edit to modify current configuration, if required.


Delete Configured TeamForge Mapping from JIRA Project

Click Delete if you wish to completely dissociate the JIRA project from the configured TeamForge mapping.

Associate JIRA Issues to Version Control Commits

Associations between JIRA® issues and TeamForge-aware version control commits can be created via commit message references. Commits to TeamForge project repositories and external repositories that have been configured in TeamForge are also supported.

Creating Associations

  1. Using your desired terminal or IDE, instantiate a version control commit to a repository of your choice.
  2. In the commit message, make reference to JIRA ID(s) to which you wish to associate the commit surrounded by square brackets.

    Sample Commit Messages
    “[DEMO-123] This commit message will result in an association between
    JIRA issue DEMO-123 and this commit.”
    
    “[DEMO-123, DEMO-124] Here I’m associating two JIRA issues with project
    identifiers DEMO.” 
    

Viewing Associations inside JIRA

  1. Navigate to the desired JIRA issue details page.
  2. Click the TeamForge tab.
  3. A list of immediate associations appears. In other words, these objects are directly associated to the JIRA issue in question.
  4. Click Trace to view the first three levels of the traceability chain.

    All activities are mapped chronologically.
    Lines indicate direct associations.
    Use the “+” icon to explore further levels of traceability.

Remove TeamForge Mappings

As a JIRA site administrator, you can disable active TeamForge mappings for one or more or all the JIRA projects (Administration > Add-ons > TeamForge Associations), if you want to prevent a JIRA server that’s mirrored into a staging/testing environment from triggering events back to TeamForge (when there are changes to issues in JIRA projects that are mapped to TeamForge) thereby polluting the production TeamForge event data store.

Use this feature to disable all production JIRA-TeamForge mappings in stage environments and then test the TeamForge Associations add-on by creating a new mapping between a staging JIRA server and TeamForge server.

  1. Log on to the JIRA server as a site administrator.
  2. Select JIRA Administration > Add-ons.
  3. Select TeamForge Associations > Manage.
  4. Select one or more projects from the list to remove the TeamForge-JIRA mapping.
  5. Click Remove. A confirmation message is displayed.
  6. Click Remove to delete the mapping.

Migrate Existing JIRA Data from EventQ to TeamForge

If you configure the new JIRA integration plugin 1.0 to notify TeamForge, you must migrate the JIRA data from EventQ’s MongoDB database to TeamForge database, after upgrading to TeamForge 19.0.

To migrate the JIRA data from EventQ’s MongoDB to TeamForge database:

This migration process is two-fold:

  1. Extract the existing JIRA data from EventQ’s MongoDB database and generate an SQL file based on the database option chosen.
  2. Execute the generated SQL file on the TeamForge database (Postgres/Oracle).

Extract JIRA Data from EventQ MongoDB

  1. Download the JIRA plugin jira-1.0.jar.

  2. Run this command to execute the migration script.

    java -jar jira-1.0.jar -migrate
    
  3. Enter the MongoDB hostname as localhost or just press ENTER for localhost to be taken as default host name.
  4. Enter the MongoDB port number or just press ENTER for 27017 to be taken as default port number.

  5. Enter the MongoDB database name for EventQ.

  6. Enter the MongoDB username and password.

  7. For Does TeamForge use Oracle Database [y/n]:, press y if you use Oracle. Press n if you use PostgreSQL.

    The migration script is executed and generates the jira_data_migration.sql file.

Execute the SQL File on PostgreSQL/Oracle Database

  • To execute the SQL file on PostgreSQL Database:

    1. Log on to your PostgreSQL database.

    2. Run this command to import the migrated data.

      sudo /opt/collabnet/teamforge/runtime/scripts/psql-wrapper <filepath of `jira_data_migration.sql`> 
      

      OR

      cat <filepath of `jira_data_migration.sql`> | sudo /opt/collabnet/teamforge/runtime/scripts/psql-wrapper 
      
  • To execute the SQL file on Oracle Database:

    1. Log on to your Oracle database.

    2. Run this command to import the migrated data.

      @<filepath of `jira_data_migration.sql`>