TeamForge Webhooks-based Event Broker, which is also referred to as the integration broker, is a webhooks-based message broker that pushes the messages of specific events received from a Publisher to a Subscriber.

TeamForge Webhooks-based Event Broker - An Overview

TeamForge Webhooks-based Event Broker

is a light-weight webhooks-based message broker integrated with TeamForge to validate, transform, and route the messages that are received from a publisher to a subscriber.
acts both as an event broker and as a repository to store the messages that publishers intend to send to the subscribers.
is a robust message delivery middleware with minimal failure points.
can be used in the place of TeamForge EventQ, but is typically not a replacement for TeamForge EventQ.
can handle hundreds of messages over HTTPS per second.
provides a guaranteed once & once-only, in-order delivery of messages.
defines the Events, Publishers, and Subscribers.

Architecture

TeamForge Webhooks-based Event Broker is all about the events, publishers, subscribers and event messages that the publishers send to the subscribers. It also provides a mechanism to store the messages in the Inbox, and to push them to the Outbox, from which the messages are sent to the subscribers.

With TeamForge Webhooks-based Event Broker, Publishers/Subscribers can publish/subscribe to multiple events. It uses in-built TeamForge plugin to authenticate and send messages to TeamForge provided that the subscriber name starts with TeamForge.

Events

When an event occurs, the publisher sends the messages for that specific event to TeamForge Webhooks-based Event Broker which in turn sends it to the subscriber. Events must be registered in TeamForge Webhooks-based Event Broker. Each event is identified by a unique name. Registering an event is required before you register the publisher or subscriber.

Publishers

Whenever an event occurs, Publishers send messages to subscribers via TeamForge Webhooks-based Event Broker. Each publisher gets a unique REST endpoint for each event to publish. When you register a publisher, you will create the publisher name and select the event on which the publisher need to send the message.

Subscribers

Subscribers receive the messages sent by the Publishers via TeamForge Webhooks-based Event Broker. Subscribers can provide a Web endpoint where messages are to be pushed. Each subscriber gets a dedicated sender that sends messages on a First-in-First-out (FIFO) basis to the endpoint.

Inbox

TeamForge Webhooks-based Event Broker is designed using a box model. All messages published to TeamForge Webhooks-based Event Broker are stored in the Inbox and a unique reference number is generated and sent back to the publisher for storage and future retrieval.

Outbox

When a message is received by TeamForge Webhooks-based Event Broker, the subscriptions for the event represented by that message is checked. Multiple records or one separate record for each subscription, are then inserted into the Outbox, along with the Subscriber details. The status is set to PENDING.

TeamForge Webhooks-based Event Broker then sends the messages from the Outbox to the Subscribers. Once successfully delivered, the message in the Outbox is marked as DELIVERED.

Message Receiver

Message Receiver receives event messages from multiple publishers, stores in the Inbox and returns the unique Inbox message reference number (InboxId). This is completely a concurrent process.

Replicator

The Replicator is a single process that periodically replicates Inbox messages to the Outbox for the subscriptions active at that point in time.

Sender

Each Subscriber gets a unique Sender process. Each of these processes run in parallel and can process Outbox messages concurrently. However, each Sender sends the messages to the Subscriber in sequence.

Install the TeamForge Webhooks-based Event Broker

You can install the Webhooks-based Event Broker after installing TeamForge. As the Webhooks-based Event Broker requires relatively less RAM, you can install the Webhooks-based Event Broker on the TeamForge Application Server itself.

You should have the TeamForge installation respository configured as part of the TeamForge installation.

Do this on the TeamForge Application Server to install the Webhooks-based Event Broker.

  1. Run this command:

    yum install teamforge-webr
    
  2. Add the Webhooks-based Event Broker services (webr and webr-database) to the host:SERVICES token in site-options.conf file.

    For example:

    host:SERVICES=ctfcore ctfcore-database ctfcore-datamart search mail etl binary reviewboard reviewboard-database reviewboard-adapter cliserver webr webr-database
    
  3. Provision services.

    teamforge provision
    

Configure Jenkins Integration Plugin to Notify the Webhooks-based Event Broker

A new version of Jenkins integration plugin, v2.0.6, is released with TeamForge 18.3, which if configured can notify either the native TeamForge Webhooks-based Event Broker or EventQ about the build data.

CollabNet Plugin Features

  • Notify EventQ when builds complete. The CollabNet Plugin must be installed once on each Jenkins server you wish to connect to TeamForge/EventQ.
  • Notify TeamForge Webhooks-based Event Broker when builds complete.
  • Authenticate users from TeamForge. If setup as the “Build & Test” application, it can even use Single Sign-On.
  • Authorization from TeamForge, including the ability to set permissions in Jenkins based on roles in your TeamForge project.
  • Upload the build log or workspace artifacts to the TeamForge Documents.
  • Upload workspace artifacts to the TeamForge File Release System, as a post-build publishing task or as a build promotion task.
  • Open/update/close TeamForge Tracker artifacts based on the Jenkins build status.
  • Upload workspace artifacts to the Lab Management Project Build Library. (Requires CollabNet Lab Management).

    Click here to know more about the requirements for installing the latest CollabNet plugin.

It is highly recommended that you notify the TeamForge Webhooks-based Event Broker for TeamForge-Jenkins integration.

Use the following instructions to have the Jenkins integration plugin notify the native Webhooks-based Event Broker.

  1. If you are integrating TeamForge and Jenkins for the first time:
    1. Log on to the Jenkins Server as a previliged Jenkins user, navigate to Manage Jenkins > Manage Plugins > Available.
    2. Select the latest CollabNet Plugin and install the plugin.
    3. Restart your Jenkins server.
  2. Existing TeamForge-Jenkins integrations that use CollabNet Plugin v2.0.4 (or earlier):
    1. Log on to the Jenkins Server as a privileged Jenkins user, navigate to Manage Jenkins > Manage Plugins > Updates.
    2. Select the latest CollabNet Plugin and install the plugin.
  3. Existing TeamForge-Jenkins integrations that use EventQ Jenkins Adapter v2.0 (or earlier) plugin:
    1. Log on to the Jenkins Server as a privileged Jenkins user, navigate to Manage Jenkins > Manage Plugins > Installed.
    2. Select the EventQ Jenkins Adapter v2.0 and click Uninstall.
    3. Select the Available tab.
    4. Select the latest CollabNet Plugin and install the plugin.
    5. Download the migrate_jenkins_plugin.sh script and save it to <JENKINS_HOME_DIRECTORY>/jobs/.
    6. Change ownership of the migrate_jenkins_plugin.sh file.
      chmod 755 migrate_jenkins_plugin.sh
      
    7. Run the migrate_jenkins_plugin.sh script.
      ./migrate_jenkins_plugin.sh
      
  4. Configure an Individual Jenkins Job to notify TeamForge Webhooks-based Event Broker.

    1. Create a build source and have the configuration details handy for the following steps.
    2. As a privileged Jenkins user, locate the job you wish to report build data to TeamForge Webhooks-based Event Broker and navigate to its configuration page.
    3. Add a post-build action to Notify TeamForge/EventQ when a build completes.
    4. Select the Notify TeamForge check box.
    5. Enter the TeamForge WebHook URL to which the build data will be sent.

    6. By default, the Optional TeamForge Association View check box is selected. If required, you can override the global configuration by entering the TeamForge URL and user credentials.
    7. Save the job configuration.

    8. Run a build to test the new configuration and verify configuration. Information and errors will be reported to your Jenkins log and to the build console.

Migrate Existing Jenkins Data from EventQ to TeamForge

If you have configured the new Jenkins integration plugin v2.0.6 to notify the TeamForge-based Webhooks Event Broker, you need to migrate the Jenkins data from EventQ’s MongoDB database to TeamForge database, after upgrading to TeamForge 18.3.

To migrate the Jenkins data from EventQ’s MongoDB to TeamForge database:

This migration process is two-fold:

  1. Extract the existing Jenkins data from EventQ’s MongoDB database and generate an SQL file based on the database option chosen.
  2. Execute the generated SQL file on the TeamForge database (Postgres/Oracle).

Extract Jenkins Data from EventQ MongoDB

  1. Download the Jenkins plugin jenkins-1.0.jar.

  2. Run this command to execute the migration script.

    java -jar jenkins-1.0.jar -migrate
    
  3. Enter the MongoDB hostname as localhost or just press ENTER for localhost to be taken as default host name.
  4. Enter the MongoDB port number or just press ENTER for 27017 to be taken as default port number.

  5. Enter the MongoDB database name for EventQ.

  6. Enter the MongoDB username and password.

  7. For Does TeamForge use Oracle Database [y/n]:, press y if you use Oracle. Press n if you use PostgreSQL.

    The migration script is executed and generates the jenkins_data_migration.sql file.

Execute the SQL File on PostgreSQL/Oracle Database

  • To execute the SQL file on PostgreSQL Database:

    1. Log on to your PostgreSQL database.

    2. Run this command to import the migrated data.

      sudo /opt/collabnet/teamforge/runtime/scripts/psql-wrapper <filepath of `jenkins_data_migration.sql`> 
      

      OR

      cat <filepath of `jenkins_data_migration.sql`> | sudo /opt/collabnet/teamforge/runtime/scripts/psql-wrapper 
      
  • To execute the SQL file on Oracle Database:

    1. Log on to your Oracle database.

    2. Run this command to import the migrated data.

      @<filepath of `jenkins-data-migration.sql`>