In this setup, TeamForge, Oracle database and other services are distributed across three servers, server-01 through server-03 as illustrated in the following table.
You can install TeamForge on both RHEL/CentOS 7.9 and 6.10. In this distributed setup, all the following services are installed on RHEL/CentOS 7.9 servers.
server-01 | server-02 |
---|---|
TeamForge Application Server | Oracle Database Server |
ctfcore | ctfcore-database |
ctfcore-datamart | |
etl | |
search | |
codesearch | |
gerrit | |
gerrit-database | |
subversion | |
reviewboard1 | |
reviewboard-database | |
reviewboard-adapter2 | |
binary | |
binary-database | |
cliserver | |
service-monitor |
Dos and Don’ts
Here’s a list of dos, don’ts and points to remember when you install or upgrade TeamForge.
Dos
- Understand TeamForge installation requirements and plan your installation or upgrade.
- Get your TeamForge license key and keep it handy.
- Verify your basic networking setup before installing or upgrading TeamForge. See Set Up Networking for TeamForge.
- Look for new or modified
site-options.conf
tokens and update yoursite-options.conf
file as required during the upgrade process. See Site Options Change Log. - Set up a TeamForge Stage Server before you upgrade your Production Server.
- Stop TeamForge services on all servers in a distributed setup while upgrading to TeamForge 20.3.
- Uninstall hot fixes and add-ons, if any, before you start the TeamForge 20.3 upgrade procedure.
-
As a result of changes to the logging framework in Java 9, the
PrintGCDetails
andPrintGCTimeStamps
logging options are no longer supported. Remove these options from the following tokens while upgrading to TeamForge 18.1 or later. TeamForge provision fails otherwise.- JBOSS_JAVA_OPTS
- PHOENIX_JAVA_OPTS
- INTEGRATION_JAVA_OPTS
- ETL_JAVA_OPTS
- ELASTICSEARCH_JAVA_OPTS
Don'ts
- Do not customize your operating system installation. Select only the default packages list.
- While upgrading TeamForge, whether in place or on new hardware, always reuse the old
site-options.conf
file and make changes as necessary. Do not try to start with a newsite-options.conf
file. Reusing the oldsite-options.conf
avoids many potential problems, particularly around the management of usernames and passwords. - Do not manually modify TeamForge-managed site option tokens such as the
AUTO_DATA
token. See AUTO_DATA for more information. -
If you are creating symlinks, note that you must create symlinks only to the TeamForge data directory (
/opt/collabnet/teamforge/var
). You should not create symlinks to TeamForge application directories (such as/opt/collabnet
).
Points to Remember
- Installing or upgrading TeamForge needs root privileges. You must log on as root or use a root shell to install or upgrade TeamForge.
- SSL is enabled by default and a self-signed certificate is auto-generated. However, you can use a few
site-options.conf
tokens to adjust this behavior. To generate the SSL certificates, see Generate SSL Certificates. - For the ETL service to run as expected in a distributed TeamForge installation, all servers must have the same time zone.
- If you have Git integration on a separate server, both TeamForge and Git servers must have their time and date synchronized. Similarly, if Subversion is on a separate server, both TeamForge and Subversion servers must have their time and date synchronized.
- It’s highly recommended that you install the TeamForge Baseline services on a separate server as the baselining process can consume considerable CPU and database resources. For more information, see Install TeamForge in a Distributed Setup.
- No backup is required for same hardware upgrades. However, you can create a backup as a measure of caution. See Back up and Restore TeamForge for more information.
- Always use compatible JDBC drivers meant for specific database versions. See JDBC Drivers Reference for more information. Also see: Why do ETL jobs fail post TeamForge upgrade?
- You can run the initial load job any time after the installation of TeamForge. We recommend that you run it before you hand over the site to the users. For more information, see ETL Initial Load Jobs.
- SOAP50 APIs and event handlers are no longer supported in TeamForge 16.10 and later. Use the latest TeamForge SOAP/REST APIs.
- TeamForge 20.3 installer expects the system locale to be
LANG=en_US.UTF-8
. TeamForge create runtime (teamforge provision
) fails otherwise. - Installing TeamForge with service-specific FQDNs (instead of machine-specific host/domain names) is highly recommended so that you will be able to change the system landscape at a later point in time without having any impact on the URLs (in other words, end users do not have to notice or change anything). For example, you can create FQDNs specifically for services such as Subversion, Git, mail, Codesearch and so on. For more information, see Service-specific FQDNs.
- All such service-specific FQDNs must belong to a single sub domain and it is recommended to create a new sub domain for TeamForge.
- If you are using service-specific FQDNs
- A wildcard SSL cert is required. SNI SSL cert cannot be used.
- When SSL is enabled and no custom SSL certificates are provided, a self-signed wildcard cert is generated for the sub domain.
- When SSL is enabled and a custom SSL certificate is provided, the CN of the certificate is verified to be a wildcard CN.
- The ability to run separate PostgreSQL instances for TeamForge database and datamart on the same server is being deprecated in TeamForge 17.11. If you have TeamForge database and datamart on separate PostgreSQL instances on the same server and if you are upgrading on a new hardware, you must Create a Single Cluster for Both Database and Datamart while upgrading to TeamForge 17.11 or later.
- While upgrading TeamForge-Git integration servers, it is important that Git master and slave servers are upgraded to the same version of TeamForge-Git integration. On sites with Git Replica Servers, you must upgrade the Git Replica Servers first and then upgrade the master Git servers.
- From TeamForge 19.3, TeamForge Webhooks-based Event Broker is installed automatically when you install/upgrade TeamForge. In other words, you don’t have to run the command
yum install teamforge-webr
separately. - Call back URLs registered with WEBR are lost when you restart WEBR. This means, a TeamForge/Jboss restart must follow immediately after you stop or restart WEBR.
- TeamForge supports Monit for monitoring services and recovering failed services. Monit is installed on the TeamForge Application server to monitor the health of services and restart the services when they fail. Monit log file is located at
/opt/collabnet/teamforge/log/monit/monit.log
.
CVS is no longer supported by TeamForge 20.2 and later. You must migrate your CVS repositories to any of the other supported SCM tool (Git/SVN for example) when you upgrade to TeamForge 20.2 or later.
-
Undeploy CVS on the TeamForge SCM server that runs CVS. Do this after you stop the TeamForge services while upgrading to TeamForge 20.2 or later versions on the same hardware. Skip this step in case of new hardware upgrades.
teamforge undeploy -s cvs
-
Remove
cvs
from thehost:SERVICES
token of thesite-options.conf
file (on all the TeamForge servers), failing which theteamforge provision
command aborts with an error.
EventQ as a TeamForge service is no longer supported and is completely removed from TeamForge 20.0 (and later). There are a few things to consider in case you have been using EventQ and are upgrading to TeamForge 20.0 or later. For more information, see EventQ End of Life.
Do this before you stop TeamForge while upgrading to TeamForge 18.2 or later versions.
Get value of SUBVERSION_REPOSITORY_BASE
from the /opt/collabnet/teamforge/runtime/conf/runtime-options.conf
file of your existing TeamForge server and run the following command:
chmod -R 775 $SUBVERSION_REPOSITORY_BASE
Where $SUBVERSION_REPOSITORY_BASE
is the path to the /svnroot
directory.
This is required to work around the unusually long time taken to migrate the Subversion data during the first run of the teamforge provision
command.
Back up Your Oracle Database
Uninstall Custom Event Handlers, Hot Fixes and Add-ons
Log on to the TeamForge Application Server.
- SOAP 50 is no longer supported. Back up all your custom event handlers and remove all the event handler JAR files before starting your TeamForge upgrade process.
- Go to My Workspace > Admin.
- Click System Tools from the Projects menu.
- Click Customizations.
- Select the custom event handler and click Delete.
Tip: Post upgrade, you can add custom event handlers again from the backup while making sure that you don’t have SOAP50 (deprecated) library used.
- Uninstall hotfixes and add-ons, if any, installed on your site.
yum upgrade
-
Stop TeamForge.
Important: Stop TeamForge on all the servers in a distributed setup.teamforge stop
-
Upgrade the operating system packages.
yum upgrade
Note: Runyum upgrade
on all the servers.
Configure the TeamForge Installation Repository
-
TeamForge Installation Repository Configuration for Sites with Internet Access
- Contact the CollabNet Support and download the TeamForge 20.3 installation repository package to
/tmp
. - Install the repository package.
yum install -y /tmp/collabnet-teamforge-repo-20.3-0-noarch.rpm
- Refresh your repository cache.
yum clean all
TeamForge Installation Repository Configuration for Sites without Internet Access
- Contact the CollabNet Support to get the auxiliary installer package for TeamForge 20.3 disconnected installation and save it in
/tmp
.- RHEL/CentOS 6.10 64 bit:
CTF-Disconnected-media-20.3.398-677.rhel6.x86_64.rpm
- RHEL/CentOS 7.9 64 bit:
CTF-Disconnected-media-20.3.398-677.rhel7.x86_64.rpm
- In addition to the above CentOS 7.9 64 bit RPM package, you must get the following CentOS 7.9 compatibility RPM, which is required for TeamForge 20.3 disconnected media installation on CentOS 7.9 profile:
compat-ctf-dc-media-1.2-1.el7.noarch.rpm
.
- RHEL/CentOS 6.10 64 bit:
- Unpack the disconnected installation package.
rpm -Uvh <package-name>
- Unpack the
compat-ctf-dc-media-1.2-1.el7.noarch.rpm
package if you are installing TeamForge 20.3 on CentOS 7.9.rpm -ivh compat-ctf-dc-media-1.2-1.el7.noarch.rpm
- If you are installing TeamForge 20.3 on RHEL/CentOS 6.10, contact the CollabNet Support to get the
python-modules-sources-el6.zip
file and unzip it to/opt/collabnet/teamforge/service/reviewboard/resources/SOURCES/python-modules-sources
.unzip python-modules-sources-el6.zip -d /opt/collabnet/teamforge/service/reviewboard/resources/SOURCES/python-modules-sources
If you are installing TeamForge 20.3 on RHEL/CentOS 7.9, contact the CollabNet Support to get the
python-modules-sources-el7.zip
file and unzip it to/opt/collabnet/teamforge/service/reviewboard/resources/SOURCES/python-modules-sources
.unzip python-modules-sources-el7.zip -d /opt/collabnet/teamforge/service/reviewboard/resources/SOURCES/python-modules-sources
-
If not mounted already, mount the RHEL/CentOS installation DVD.
The DVD contains the necessary software and utilities required for installing TeamForge without internet access. In the following commands, replace “cdrom” with the identifier for your server’s CD/DVD drive, if necessary.
cd /media/ mkdir cdrom mount /dev/cdrom ./cdrom/
If there are any spaces in the automount, unmount it first and mount it as a filepath, with no spaces.
- Create a yum configuration file that points to the RHEL/CentOS installation DVD.
vi /etc/yum.repos.d/cdrom.repo
Here’s a sample yum configuration file.
[RHEL-CDROM] name=RHEL CDRom baseurl=file:///media/cdrom/Server/ gpgfile=file:///media/cdrom/RPM-GPG-KEY-redhat-release enabled=1 gpgcheck=0
- Verify your yum configuration files.
yum list httpd yum list apr
- Contact the CollabNet Support and download the TeamForge 20.3 installation repository package to
Upgrade the TeamForge Services
-
Upgrade the TeamForge and Review Board application services on the TeamForge Application Server (server-01).
Before You BeginIf you are installing TeamForge 20.3 on RHEL/CentOS 6.10, contact CollabNet Support to get the
python-modules-sources-el6.zip
file and unzip it to/opt/collabnet/teamforge/service/reviewboard/resources/SOURCES/python-modules-sources
.unzip python-modules-sources-el6.zip -d /opt/collabnet/teamforge/service/reviewboard/resources/SOURCES/python-modules-sources
yum install teamforge
Install Monit.
Important: If you haven’t already installed the latest version of the Monit application, download it here.-
Download Monit for
-
RHEL/CentOS 7.x from the EPEL repository.
wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm rpm -ivh epel-release-latest-7.noarch.rpm
-
RHEL/CentOS 6.x from the EPEL repository.
wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm rpm -ivh epel-release-latest-6.noarch.rpm
-
-
Install Monit.
yum install monit
-
Back up the TeamForge Data Directories
On sites running TeamForge 16.7 or earlier versions:
- Back up the following data directories.
Tip: In a distributed setup, you must backup specific directories such as
/svnroot
and/cvsroot
from the server that hosts those SCM services.Note: CVS is no longer supported by TeamForge 20.2 (and later). Backing up and restoring the/cvsroot
is recommended, but optional though.Directory Contents /opt/collabnet/teamforge/var User-created data, such as artifact attachments /opt/collabnet/reviewboard Review Board data /svnroot Subversion source code repositories /sf-svnroot Subversion repository for branding data /cvsroot CVS source code repositories (required only if you have CVS) /gitroot Git source code repositories cp -Rpf /svnroot /sf-svnroot /cvsroot /gitroot /opt/collabnet/teamforge/var /opt/collabnet/reviewboard /tmp/backup_dir
- Back up the
/opt/collabnet/gerrit
directory if you have Git integration.Tip: Do this on the server that hosts the TeamForge-Git integration services.mkdir /tmp/backup_dir/gerrit cp -Rpfv /gitroot /tmp/backup_dir cp -Rpfv /opt/collabnet/gerrit/ /tmp/backup_dir/gerrit
On sites running TeamForge 16.10 or later versions:
- Back up the
/opt/collabnet/teamforge/var
directory.Tip: Do this on both the TeamForge Application and Database servers in case you have them running on two separate servers.mkdir -p /tmp/backup_dir cp -Rpfv /opt/collabnet/teamforge/var /tmp/backup_dir
- Back up the
/opt/collabnet/gerrit
directory if you have Git integration.Tip: Do this on the server that hosts the TeamForge-Git integration services.mkdir /tmp/backup_dir/gerrit cp -Rpfv /opt/collabnet/gerrit/ /tmp/backup_dir/gerrit
Back up and Restore Review Board Database and Data Directories
See Back up and Restore Review Board Database and Data Directories
Set up the site-options.conf File and Provision Services
- Log on to the TeamForge Application Server (server-01), set up the
site-options.conf
file, and provision the services.vi /opt/collabnet/teamforge/etc/site-options.conf
host:SERVICES Token
server-01:SERVICES=ctfcore mail etl service-monitor search subversion codesearch cliserver gerrit gerrit-database binary binary-database reviewboard reviewboard-database reviewboard-adapter cliserver server-02:SERVICES=ctfcore-database ctfcore-datamart
host:PUBLIC_FQDN Token
server-01:PUBLIC_FQDN=my.app.domain.com
Configure the Oracle Database Tokens
Configure the Oracle database name, usernames and passwords as configured on the Oracle Database Server.
- Database type is
oracle
(DATABASE_TYPE=oracle
) - Database service name is the host name of the Oracle Database Server (for example,
DATABASE_SERVICE_NAME=cu349.maa.collab.net
) - Reports database service name is the host name of the server where the datamart is (for example,
REPORTS_DATABASE_SERVICE_NAME=cu349.maa.collab.net
)
DATABASE_TYPE=oracle # Adjust usernames/passwords to match what has been configured on the database server. DATABASE_USERNAME=ctfuser DATABASE_PASSWORD=ctfpwd DATABASE_READ_ONLY_USER=ctfrouser DATABASE_READ_ONLY_PASSWORD=ctfropwd DATABASE_NAME=orcl DATABASE_SERVICE_NAME= # Adjust usernames/passwords to match what has been configured on the database server. REPORTS_DATABASE_USERNAME=ctfrptuser REPORTS_DATABASE_PASSWORD=ctfrptpwd REPORTS_DATABASE_NAME=orcl REPORTS_DATABASE_READ_ONLY_USER=ctfrptrouser REPORTS_DATABASE_READ_ONLY_PASSWORD=ctfrptropwd REPORTS_DATABASE_SERVICE_NAME=
Save the
site-options.conf
file.For further customization of your site configuration (SSL settings, password policy settings, PostgreSQL settings, LDAP settings and so on):
SSL Tokens
SSL is enabled by default and a self-signed certificate is auto-generated. Use the following tokens to adjust this behavior.
Note: TeamForge runs only with SSL from TeamForge 19.2. Hence thesite-options.conf
token optionSSL=off
is not supported any more. TeamForge provision fails and throws an error, ifSSL
is set tooff
.SSL_CERT_FILE= SSL_KEY_FILE= SSL_CHAIN_FILE=
- To generate the SSL certificates, see Generate SSL certificates.
- Have the custom SSL certificate and private key for custom SSL certificate in place and provide their absolute paths in these tokens. SSL_CHAIN_FILE (intermediate certificate) is optional.
Password Tokens
- TeamForge 7.1 and later support automatic password creation. See AUTO_DATA for more information.
-
Set the REQUIRE_PASSWORD_SECURITY token to
true
to enforce password security policy for the site.If the token REQUIRE_PASSWORD_SECURITY is enabled, then set a value for the token, PASSWORD_CONTROL_EFFECTIVE_DATE.
Warning: The Password Control Kit (PCK) disables, deletes or expires user accounts that don’t meet the password security requirements starting from the date set for thePASSWORD_CONTROL_EFFECTIVE_DATE
token. If a date is not set, the PCK disables, deletes or expires user accounts immediately. See PASSWORD_CONTROL_EFFECTIVE_DATE for more information.You can also set the following tokens to enforce a more stricter password policy:
- If the token REQUIRE_RANDOM_ADMIN_PASSWORD is already set to
true
, then set the token ADMIN_EMAIL with a valid email address.ADMIN_EMAIL=root@{__APPLICATION_HOST__}
- If you have LDAP set up for external authentication, you must set the REQUIRE_USER_PASSWORD_CHANGE site options token to
false
.
Prevent Cross-site Scripting
An attacker could potentially upload an HTML page to TeamForge that contains active code, such as JavaScript. This active code would then be executed by clients’ browsers when they view the page, which can harm the system.
To prevent an attack of this sort, you can specify whether or not HTML code is displayed in TeamForge. This flag applies to all documents, tracker, task, and forum attachments, and files in the file release system.
Set the SAFE_DOWNLOAD_MODE token according to your requirements. For more information, see SAFE_DOWNLOAD_MODE.
JAVA_OPTS
Configure the JBOSS_JAVA_OPTS site-options.conf token. See JBOSS_JAVA_OPTS.
Note: All JVM parameters but-Xms1024m
and-Xmx2048m
have been hard-coded in the TeamForge core application. You need not manually configure any other parameter (such as-XX:MaxMetaspaceSize=512m
-XX:ReservedCodeCacheSize=128M
-server -XX:+HeapDumpOnOutOfMemoryError
-Djsse.enableSNIExtension=false
-Dsun.rmi.dgc.client.gcInterval=600000
-Dsun.rmi.dgc.server.gcInterval=600000
) in the site-options.conf file.TeamForge 18.1 (and later) supports Java 9. As a result of changes to the logging framework in Java 9, the
PrintGCDetails
andPrintGCTimeStamps
logging options are no longer supported. Remove these options from the following tokens while upgrading to TeamForge 18.1 or later.- JBOSS_JAVA_OPTS
- PHOENIX_JAVA_OPTS
- INTEGRATION_JAVA_OPTS
- ETL_JAVA_OPTS
- ELASTICSEARCH_JAVA_OPTS
TeamForge provision fails on sites that use these options post upgrade to TeamForge 18.1.
Save the
site-options.conf
file. - Database type is
- Provision services.
teamforge provision
TeamForge 20.3 installer expects the system locale to beLANG=en_US.UTF-8
. TeamForge create runtime (teamforge provision
) fails otherwise.
Update the Webhook Events and Create New Webhooks
While TeamForge 19.2 (and earlier) offered simple webhooks support, TeamForge 19.3 (and later) offer pre-submit and post-submit webhooks. New webhook events have been added and webhook event names have been updated as well.
If you are upgrading from TeamForge 19.2 (or earlier) to TeamForge 19.3 (or later), you must truncate the webhook
and webhook_event
tables on the Oracle database server, insert the new webhook events into the webhook_event
table and create new webhooks.
- Note down the list of webhooks you have before truncating the
webhooks
table. You may take a screenshot of the webhook configuration just in case. - Run the following queries to back up the
webhook
andwebhook_event
TeamForge tables.create table webhookbackup as select * from webhook; create table webhookeventsbackup as select * from webhook_event;
- Truncate the
webhook
andwebhook_event
TeamForge tables.truncate table webhook; truncate table webhook_event;
- Insert the new TeamForge webhook events into the
webhook_event
table. Run the following queries one-by-one in order on the Oracle database server.insert into webhook_event (surrogate_id,webhook_surrogate_id,event_type, event_type_name) select webhook_event_key_seq.nextval,0,'Teamforge.Artifact.Create','TOPIC' from dual where Not exists (select 1 from webhook_event where event_type='Teamforge.Artifact.Create'); insert into webhook_event (surrogate_id,webhook_surrogate_id,event_type, event_type_name) select webhook_event_key_seq.nextval,0,'Teamforge.Artifact.Update','TOPIC' from dual where Not exists (select 1 from webhook_event where event_type='Teamforge.Artifact.Update'); insert into webhook_event (surrogate_id,webhook_surrogate_id,event_type, event_type_name) select webhook_event_key_seq.nextval,0,'Teamforge.Artifact.Move','TOPIC' from dual where Not exists (select 1 from webhook_event where event_type='Teamforge.Artifact.Move'); insert into webhook_event (surrogate_id,webhook_surrogate_id,event_type, event_type_name) select webhook_event_key_seq.nextval,0,'Teamforge.Artifact.Clone','TOPIC' from dual where Not exists (select 1 from webhook_event where event_type='Teamforge.Artifact.Clone'); insert into webhook_event (surrogate_id,webhook_surrogate_id,event_type, event_type_name) select webhook_event_key_seq.nextval,0,'Teamforge.Artifact.Delete','TOPIC' from dual where Not exists (select 1 from webhook_event where event_type='Teamforge.Artifact.Delete'); insert into webhook_event (surrogate_id,webhook_surrogate_id,event_type, event_type_name) select webhook_event_key_seq.nextval,0,'Teamforge.Artifact.Create.Presubmit','SYNC' from dual where Not exists (select 1 from webhook_event where event_type='Teamforge.Artifact.Create.Presubmit'); insert into webhook_event (surrogate_id,webhook_surrogate_id,event_type, event_type_name) select webhook_event_key_seq.nextval,0,'Teamforge.Artifact.Update.Presubmit','SYNC' from dual where Not exists (select 1 from webhook_event where event_type='Teamforge.Artifact.Update.Presubmit'); insert into webhook_event (surrogate_id,webhook_surrogate_id,event_type, event_type_name) select webhook_event_key_seq.nextval,0,'Teamforge.Artifact.Move.Presubmit','SYNC' from dual where Not exists (select 1 from webhook_event where event_type='Teamforge.Artifact.Move.Presubmit'); insert into webhook_event (surrogate_id,webhook_surrogate_id,event_type, event_type_name) select webhook_event_key_seq.nextval,0,'Teamforge.Artifact.Clone.Presubmit','SYNC' from dual where Not exists (select 1 from webhook_event where event_type='Teamforge.Artifact.Clone.Presubmit'); insert into webhook_event (surrogate_id,webhook_surrogate_id,event_type, event_type_name) select webhook_event_key_seq.nextval,0,'Teamforge.Artifact.Delete.Presubmit','SYNC' from dual where Not exists (select 1 from webhook_event where event_type='Teamforge.Artifact.Delete.Presubmit');
- Run the following query to verify the events inserted into the
webhook_event
table.select event_type, event_type_name from webhook_event;
The output lists the events:
Webhook events in TeamForge
Verify TeamForge Upgrade
- Verify TeamForge upgrade.
- Reboot the server and make sure all services come up automatically at startup.
- Log on to the TeamForge web application using the default Admin credentials.
- Username:
admin
- Password:
admin
- Username:
- If your site has custom branding, verify that your branding changes still work as intended. See Customize TeamForge.
- Let your site’s users know they’ve been upgraded. See Create a Site-wide Broadcast.
Post Upgrade Tasks
- Run TeamForge in SELinux
enabled
Mode - Users are not getting email notifications for review requests and reviews. What should I do?
- Integrate Jenkins, JIRA, and TestLink using the TeamForge Webhooks-based Event Broker