You can upgrade TeamForge on the same hardware in a distributed multi-host setup.

In this distributed setup, TeamForge services are distributed across multiple servers as illustrated in the following table.

server-01 server-02 server-03 server-04 server-05 server-06
TeamForge Application Server TeamForge Database Server Review Board Server SCM Server Code Search Server Baseline Server
ctfcore ctfcore-database reviewboard 1 subversion codesearch baseline2
mail ctfcore-datamart   gerrit   baseline-post-install3
etl gerrit-database       baseline-database
search binary-database        
reviewboard-adapter4 reviewboard-database        
binary webr-database        
cliserver          
webr          
service-monitor          

Dos and Don’ts

Here’s a list of dos, don’ts and points to remember when you install or upgrade TeamForge.

Dos

  • Understand TeamForge installation requirements and plan your installation or upgrade.
  • Get your TeamForge license key and keep it handy.
  • Verify your basic networking setup before installing or upgrading TeamForge. See Set Up Networking for TeamForge.
  • Look for new or modified site-options.conf tokens and update your site-options.conf file as required during the upgrade process. See Site Options Change Log.
  • Set up a TeamForge Stage Server before you upgrade your Production Server.
  • Stop TeamForge services on all servers in a distributed setup while upgrading to TeamForge 20.2.
  • Uninstall hot fixes and add-ons, if any, before you start the TeamForge 20.2 upgrade procedure.
  • As a result of changes to the logging framework in Java 9, the PrintGCDetails and PrintGCTimeStamps logging options are no longer supported. Remove these options from the following tokens while upgrading to TeamForge 18.1 or later. TeamForge provision fails otherwise.

    • JBOSS_JAVA_OPTS
    • PHOENIX_JAVA_OPTS
    • INTEGRATION_JAVA_OPTS
    • ETL_JAVA_OPTS
    • ELASTICSEARCH_JAVA_OPTS

Don'ts

  • Do not customize your operating system installation. Select only the default packages list.
  • While upgrading TeamForge, whether in place or on new hardware, always reuse the old site-options.conf file and make changes as necessary. Do not try to start with a new site-options.conf file. Reusing the old site-options.conf avoids many potential problems, particularly around the management of usernames and passwords.
  • Do not manually modify TeamForge-managed site option tokens such as the AUTO_DATA token. See AUTO_DATA for more information.
  • If you are creating symlinks, note that you must create symlinks only to the TeamForge data directory (/opt/collabnet/teamforge/var). You should not create symlinks to TeamForge application directories (such as /opt/collabnet).

Points to Remember

  • Installing or upgrading TeamForge needs root privileges. You must log on as root or use a root shell to install or upgrade TeamForge.
  • SSL is enabled by default and a self-signed certificate is auto-generated. However, you can use a few site-options.conf tokens to adjust this behavior. To generate the SSL certificates, see Generate SSL Certificates.
  • For the ETL service to run as expected in a distributed TeamForge installation, all servers must have the same time zone.
  • If you have Git integration on a separate server, both TeamForge and Git servers must have their time and date synchronized. Similarly, if Subversion is on a separate server, both TeamForge and Subversion servers must have their time and date synchronized.
  • It’s highly recommended that you install the TeamForge Baseline services on a separate server as the baselining process can consume considerable CPU and database resources. For more information, see Install TeamForge in a Distributed Setup.
  • No backup is required for same hardware upgrades. However, you can create a backup as a measure of caution. See Back up and Restore TeamForge for more information.
  • Always use compatible JDBC drivers meant for specific database versions. See JDBC Drivers Reference for more information. Also see: Why do ETL jobs fail post TeamForge upgrade?
  • You can run the initial load job any time after the installation of TeamForge. We recommend that you run it before you hand over the site to the users. For more information, see ETL Initial Load Jobs.
  • SOAP50 APIs and event handlers are no longer supported in TeamForge 16.10 and later. Use the latest TeamForge SOAP/REST APIs.
  • TeamForge 20.2 installer expects the system locale to be LANG=en_US.UTF-8. TeamForge create runtime (teamforge provision) fails otherwise.
  • Installing TeamForge with service-specific FQDNs (instead of machine-specific host/domain names) is highly recommended so that you will be able to change the system landscape at a later point in time without having any impact on the URLs (in other words, end users do not have to notice or change anything). For example, you can create FQDNs specifically for services such as Subversion, Git, mail, Codesearch and so on. For more information, see Service-specific FQDNs.
  • All such service-specific FQDNs must belong to a single sub domain and it is recommended to create a new sub domain for TeamForge.
  • If you are using service-specific FQDNs
    • A wildcard SSL cert is required. SNI SSL cert cannot be used.
    • When SSL is enabled and no custom SSL certificates are provided, a self-signed wildcard cert is generated for the sub domain.
    • When SSL is enabled and a custom SSL certificate is provided, the CN of the certificate is verified to be a wildcard CN.
  • The ability to run separate PostgreSQL instances for TeamForge database and datamart on the same server is being deprecated in TeamForge 17.11. If you have TeamForge database and datamart on separate PostgreSQL instances on the same server and if you are upgrading on a new hardware, you must Create a Single Cluster for Both Database and Datamart while upgrading to TeamForge 17.11 or later.
  • While upgrading TeamForge-Git integration servers, it is important that Git master and slave servers are upgraded to the same version of TeamForge-Git integration. On sites with Git Replica Servers, you must upgrade the Git Replica Servers first and then upgrade the master Git servers.
  • From TeamForge 19.3, TeamForge Webhooks-based Event Broker is installed automatically when you install/upgrade TeamForge. In other words, you don’t have to run the command yum install teamforge-webr separately.
  • Call back URLs registered with WEBR are lost when you restart WEBR. This means, a TeamForge/Jboss restart must follow immediately after you stop or restart WEBR.
  • TeamForge supports Monit for monitoring services and recovering failed services. Monit is installed on the TeamForge Application server to monitor the health of services and restart the services when they fail. Monit log file is located at /opt/collabnet/teamforge/log/monit/monit.log.

One-hop Upgrade Compatibility

Though the TeamForge 20.2 installer supports one-hop upgrade from TeamForge 19.1 or later versions, TeamForge 20.2 upgrade instructions, in general, are for upgrading from TeamForge 20.1 (including update releases, if any) to TeamForge 20.2.

There is no support for one-hop upgrade from TeamForge 19.0 or earlier to TeamForge 20.2. You must upgrade your site to TeamForge 19.1 or later and then upgrade to TeamForge 20.2.

Before You Begin—CVS End-of-Life

CVS is no longer supported by TeamForge 20.2 and later. You must migrate your CVS repositories to any of the other supported SCM tool (Git/SVN for example) when you upgrade to TeamForge 20.2 or later.

  1. Undeploy CVS on the TeamForge SCM server that runs CVS. Do this after you stop the TeamForge services while upgrading to TeamForge 20.2 or later versions on the same hardware. Skip this step in case of new hardware upgrades.

    teamforge undeploy -s cvs
    
  2. Remove cvs from the host:SERVICES token of the site-options.conf file (on all the TeamForge servers), failing which the teamforge provision command aborts with an error.

Before You Begin—EventQ End-of-Life

EventQ as a TeamForge service is no longer supported and is completely removed from TeamForge 20.0 (and later). There are a few things to consider in case you have been using EventQ and are upgrading to TeamForge 20.0 or later. For more information, see EventQ End of Life.

Before You Begin—chmod /svnroot

Do this before you stop TeamForge while upgrading to TeamForge 18.2 or later versions.

Get value of SUBVERSION_REPOSITORY_BASE from the /opt/collabnet/teamforge/runtime/conf/runtime-options.conf file of your existing TeamForge server and run the following command:

chmod -R 775 $SUBVERSION_REPOSITORY_BASE

Where $SUBVERSION_REPOSITORY_BASE is the path to the /svnroot directory.

This is required to work around the unusually long time taken to migrate the Subversion data during the first run of the teamforge provision command.

The following instructions are valid for both RHEL/CentOS 6.10/7.8 platforms. Specific steps, if applicable only for a particular RHEL/CentOS platform, are called out explicitly.

No backup is required for same hardware upgrades. However, you can create a backup as a precaution. See Back up and Restore TeamForge Database, Data Directories and site-options.conf.

Uninstall Custom Event Handlers, Hot Fixes and Add-ons

Log on to the TeamForge Application Server.

  1. SOAP 50 is no longer supported. Back up all your custom event handlers and remove all the event handler JAR files before starting your TeamForge upgrade process.
    1. Go to My Workspace > Admin.
    2. Click System Tools from the Projects menu.
    3. Click Customizations.
    4. Select the custom event handler and click Delete.
  2. Uninstall hotfixes and add-ons, if any, installed on your site.

yum upgrade

  1. Stop TeamForge.

    teamforge stop
    
  2. Upgrade the operating system packages.

    yum upgrade
    

Configure the TeamForge Installation Repository

  1. TeamForge Installation Repository Configuration for Sites with Internet Access

    1. Contact the CollabNet Support and download the TeamForge 20.2 installation repository package to /tmp.
    2. Install the repository package.
      yum install -y /tmp/collabnet-teamforge-repo-20.2-0-noarch.rpm
      
    3. Refresh your repository cache.
      yum clean all
      

    TeamForge Installation Repository Configuration for Sites without Internet Access

    1. Contact the CollabNet Support to get the auxiliary installer package for TeamForge 20.2 disconnected installation and save it in /tmp.
      • RHEL/CentOS 6.10 64 bit: CTF-Disconnected-media-20.2.556-842.rhel6.x86_64.rpm
      • RHEL/CentOS 7.8 64 bit: CTF-Disconnected-media-20.2.556-842.rhel7.x86_64.rpm
      • In addition to the above CentOS 7.8 64 bit RPM package, you must get the following CentOS 7.8 compatibility RPM, which is required for TeamForge 20.2 disconnected media installation on CentOS 7.8 profile: compat-ctf-dc-media-1.2-1.el7.noarch.rpm.
    2. Unpack the disconnected installation package.
      rpm -Uvh <package-name>
      
    3. Unpack the compat-ctf-dc-media-1.2-1.el7.noarch.rpm package if you are installing TeamForge 20.2 on CentOS 7.8.
      rpm -ivh compat-ctf-dc-media-1.2-1.el7.noarch.rpm
      
    4. If you are installing TeamForge 20.2 on RHEL/CentOS 6.10, contact the CollabNet Support to get the python-modules-sources-el6.zip file and unzip it to /opt/collabnet/teamforge/service/reviewboard/resources/SOURCES/python-modules-sources.
         unzip python-modules-sources-el6.zip -d /opt/collabnet/teamforge/service/reviewboard/resources/SOURCES/python-modules-sources
      

      If you are installing TeamForge 20.2 on RHEL/CentOS 7.8, contact the CollabNet Support to get the python-modules-sources-el7.zip file and unzip it to /opt/collabnet/teamforge/service/reviewboard/resources/SOURCES/python-modules-sources.

         unzip python-modules-sources-el7.zip -d /opt/collabnet/teamforge/service/reviewboard/resources/SOURCES/python-modules-sources
      
    5. If not mounted already, mount the RHEL/CentOS installation DVD.

      The DVD contains the necessary software and utilities required for installing TeamForge without internet access. In the following commands, replace “cdrom” with the identifier for your server’s CD/DVD drive, if necessary.

      cd /media/
      mkdir cdrom
      mount /dev/cdrom ./cdrom/
      

      If there are any spaces in the automount, unmount it first and mount it as a filepath, with no spaces.

    6. Create a yum configuration file that points to the RHEL/CentOS installation DVD.
      vi /etc/yum.repos.d/cdrom.repo
      

      Here’s a sample yum configuration file.

      [RHEL-CDROM]
      name=RHEL CDRom 			
      baseurl=file:///media/cdrom/Server/
      gpgfile=file:///media/cdrom/RPM-GPG-KEY-redhat-release 
      enabled=1
      gpgcheck=0
      
    7. Verify your yum configuration files.
      yum list httpd
      yum list apr
      

Upgrade the TeamForge Services

  1. Install the TeamForge application services on the TeamForge Application Server (server-01).
    yum install teamforge
    

    Install Monit.

    1. Download Monit for

      • RHEL/CentOS 7.x from the EPEL repository.

        wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
        rpm -ivh epel-release-latest-7.noarch.rpm
        
      • RHEL/CentOS 6.x from the EPEL repository.

        wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm
        rpm -ivh epel-release-latest-6.noarch.rpm
        
    2. Install Monit.

       yum install monit
      

    Install the Baseline packages on the TeamForge Application Server (server-01) if you are installing TeamForge Baseline.

    yum install teamforge-baseline
    
  2. Install the database and baseline packages on the TeamForge Database Server (server-02).

    yum install teamforge-baseline
    
  3. Install the Review Board services on the Review Board Server (server-03).

    Before You Begin

    If you are installing TeamForge 20.2 on RHEL/CentOS 6.10, contact CollabNet Support to get the python-modules-sources-el6.zip file and unzip it to /opt/collabnet/teamforge/service/reviewboard/resources/SOURCES/python-modules-sources.

    unzip python-modules-sources-el6.zip -d /opt/collabnet/teamforge/service/reviewboard/resources/SOURCES/python-modules-sources
    
    yum install teamforge
    
  4. Install the SCM services on the SCM Server (server-04).
    yum install teamforge-scm teamforge-git
    
  5. Install the Code Search service on the Code Search Server (server-05).
    yum install teamforge-codesearch
    
  6. Install the Baseline packages on the Baseline Server (server-06).
    yum install teamforge-baseline
    

Set up the site-options.conf File

  1. Log on to the TeamForge Database Server (server-02) and set up the site-options.conf file.
    vi /opt/collabnet/teamforge/etc/site-options.conf
    

    host:SERVICES Token

    server-01:SERVICES=ctfcore service-monitor search mail etl binary reviewboard-adapter cliserver webr
    server-02:SERVICES=ctfcore-database ctfcore-datamart gerrit-database binary-database reviewboard-database webr-database
    server-03:SERVICES=reviewboard
    server-04:SERVICES=subversion gerrit
    server-05:SERVICES=codesearch
    server-06:SERVICES=baseline baseline-post-install baseline-database
    

    host:PUBLIC_FQDN Token

    server-01:PUBLIC_FQDN=my.app.domain.com
    

    Save the site-options.conf file.

    For further customization of your site configuration (SSL settings, password policy settings, PostgreSQL settings, LDAP settings and so on):

    SSL Tokens

    SSL is enabled by default and a self-signed certificate is auto-generated. Use the following tokens to adjust this behavior.

    SSL_CERT_FILE=
    SSL_KEY_FILE=
    SSL_CHAIN_FILE=
    
    • To generate the SSL certificates, see Generate SSL certificates.
    • Have the custom SSL certificate and private key for custom SSL certificate in place and provide their absolute paths in these tokens. SSL_CHAIN_FILE (intermediate certificate) is optional.
    • You can also encrypt the data traffic between the application and database servers and between the ETL and datamart servers in a distributed setup. Use the DATABASE_SSL token to do that. See Encrypt Database Network Traffic.

    Password Tokens

    Prevent Cross-site Scripting

    An attacker could potentially upload an HTML page to TeamForge that contains active code, such as JavaScript. This active code would then be executed by clients’ browsers when they view the page, which can harm the system.

    To prevent an attack of this sort, you can specify whether or not HTML code is displayed in TeamForge. This flag applies to all documents, tracker, task, and forum attachments, and files in the file release system.

    Set the SAFE_DOWNLOAD_MODE token according to your requirements. For more information, see SAFE_DOWNLOAD_MODE.

    PostgreSQL Tokens and Settings

    Make sure the PostgreSQL tokens in the site-options.conf file are set as recommended in the following topic: What are the right PostgreSQL settings for my site?

    JAVA_OPTS

    Configure the JBOSS_JAVA_OPTS site-options.conf token. See JBOSS_JAVA_OPTS.

    TeamForge 18.1 (and later) supports Java 9. As a result of changes to the logging framework in Java 9, the PrintGCDetails and PrintGCTimeStamps logging options are no longer supported. Remove these options from the following tokens while upgrading to TeamForge 18.1 or later.

    • JBOSS_JAVA_OPTS
    • PHOENIX_JAVA_OPTS
    • INTEGRATION_JAVA_OPTS
    • ETL_JAVA_OPTS
    • ELASTICSEARCH_JAVA_OPTS

    TeamForge provision fails on sites that use these options post upgrade to TeamForge 18.1.

    Save the site-options.conf file.

  2. Provision services.
    teamforge provision
    
    TeamForge 20.2 installer expects the system locale to be LANG=en_US.UTF-8. TeamForge create runtime (teamforge provision) fails otherwise.
  3. Copy the /opt/collabnet/teamforge/etc/site-options.conf file from the TeamForge Database Server (server-02) to the /opt/collabnet/teamforge/etc/ directory of all other servers.

Provision Services on All the Servers

TeamForge 16.10 and earlier versions use Oracle JDK. As TeamForge 19.2 and later use OpenJDK, the TeamForge installer checks if Oracle JDK is present when you upgrade to TeamForge 19.2 or later—and if found—would error out when you provision TeamForge. You must uninstall Oracle JDK and proceed.

Run the following command to uninstall Oracle JDK:

rpm -e jdk1.8.0_74-1.8.0_74-fcs.x86_64
  1. Provision services.
    teamforge provision
    
    TeamForge 20.2 installer expects the system locale to be LANG=en_US.UTF-8. TeamForge create runtime (teamforge provision) fails otherwise.

You must provision services in a particluar sequence. Usually you start with the Database Server, followed by the Application Server and then by other servers such as SCM, Review Board and Code Search servers.

The TeamForge installer derives this sequence from your site-options.conf file and shows you the order of provisioning servers when you try to provision one of the distributed servers. Follow the exact sequence as instructed.

Provisioning Sequence without Baseline

  1. Provision the Application Server (server-01)
  2. Provision the SCM server (server-04)
  3. Provision the Review Board Server (server-03)
  4. Provision the Code Search Server (server-05)

Provisioning Sequence with Baseline

  1. Provision the Application Server (server-01)
  2. Provision the Baseline Server (server-06)
  3. Copy the /opt/collabnet/teamforge/etc/site-options.conf file from the TeamForge Baseline Server (server-06) to the /opt/collabnet/teamforge/etc/ directory of all other servers.
  4. Provision the Database Server (server-02) again
  5. Provision the Application Server (server-01) again
  6. Provision the SCM server (server-04)
  7. Provision the Review Board Server (server-03)
  8. Provision the Code Search Server (server-05)

Reinitialize TeamForge

  1. Reinitialize TeamForge on the Review Board Server.
    teamforge reinitialize
    
  2. During teamforge provision, the Register SCM integration process fails on sites that use self-signed certificates. Perform these steps in such cases.
    1. Restart JBoss on the TeamForge Application Server.
      teamforge restart -s jboss
      
    2. Reinitialize TeamForge on the SCM Server.
      teamforge reinitialize
      
    Do you have Git and other SCM tools (SVN) on two separate servers?
    Git and other SCM tools (SVN) are typically installed on a server dedicated for SCM. However, if you have Git and SCM (SVN) installed on two separate servers, restart Jboss on the TeamForge Application Server and reinitialize TeamForge on the SCM Server (SVN server) as discussed earlier. In addition, you must also restart TeamForge on the Git Server.

    Restart TeamForge on the Git Server: teamforge restart

Finishing Tasks

  • Verify TeamForge upgrade.
    1. Reboot the server and make sure all services come up automatically at startup.
    2. Log on to the TeamForge web application using the default Admin credentials.
      • Username: admin
      • Password: admin
    3. If your site has custom branding, verify that your branding changes still work as intended. See Customize TeamForge.
    4. Let your site’s users know they’ve been upgraded. See Create a Site-wide Broadcast.

The Review Board initialization fails on RHEL/CentOS 6.x when I do teamforge reload after setting up both JBOSS_JAVA_OPTS and ETL_JAVA_OTS tokens. What should I do?

Reinitialize TeamForge.

teamforge reinitialize

SSH host key generation fails on RHEL/CentOS 6.x during data migration. What should I do?

An outdated version of OpenSSH could be the cause. Upgrade your OpenSSH and then try again.


Post Upgrade Tasks

Also See…


  1. TeamForge 20.2 supports Review Board 3.0.15 on RHEL/CentOS 7.8 and Review Board 2.5.6.1 on RHEL/CentOS 6.10. 

  2. It’s highly recommended that you install the TeamForge Baseline services on a separate server as the baselining process can consume considerable CPU and database resources. 

  3. Synchronizes the user information between the baseline database and TeamForge database. 

  4. reviewboard-adapter must always be installed on the TeamForge Application Server.