Migrate TermWeb 3.x to TermWeb 4.0 (Linux)

Prepare for migration

• Prior to start migration of TermWeb 3 it should be updated to latest version of 3.18.

• With migration to TermWeb 4 it keeps everything from previous version you won’t loose any history or metadata. Only old Grid View data will be dropped. Before starting migration if some users using Grid View, they must publish or discard their changes.

• Install Oracle JDK 8 (latest free version is 8u202) or OpenJDK 8 (we're using and testing with Amazon Corretto 8). Don’t install version higher than 8 TermWeb 4 won’t work with it.

• Install Tomcat 9.0.x version.
  You may get an idea how to do this by reading this article for Tomcat 8, but process is the same https://www.digitalocean.com/community/tutorials/how-to-install-apache-tomcat-8-on-centos-7.
  You may use some rpm/deb repositories with prepackaged Tomcat. But we don't know any repository which is updated regularly and may be trusted.

  One thing to note about configuring tomcat: much better to configure java options of Tomcat in setenv.sh file located in <tomcattw4_location>/bin folder (or in systemd/init.d service file) instead of modifying catalina.sh file.
  Probably you need to create file setenv.sh file if it missing. It will allow to you to update Tomcat easier by copying only this file to new installation.
  About required memory for tomcat: 750 megabytes of RAM is enough for testing of small termbases about 5k of concepts, recommended setup 2GB for 100k concepts and 50 concurrent users.

  setenv.sh

  export CATALINA_OPTS="-Xms750m -Xmx750m \ -Dspring.config.name=termweb \ -Dspring.profiles.active=mysql,termweb3-search \ -Dfile.encoding=UTF-8 \ -Duser.timezone=UTC \ -Dtermweb.home=/opt/tw4home \ -verbose:gc"

   

• Unpack termweb4.war into webapps/ROOT of tomcat.

• Download and install mysql-connector-j into /opt/tomcat/webapps/ROOT/WEB-INF/lib
  Go to https://dev.mysql.com/downloads/connector/j/ and select platform independent version of connector compatible with MySQL version 8.0
  
  

  Open

  

  Unpack and move it to lib folder mentioned before:
  tar xf mysql-connector-j-8.0.33.tar.gz && cp mysql-connector-j-8.0.33/mysql-connector-j-8.0.33.jar <tomcattw4_location>/webapps/ROOT/WEB-INF/lib

• Make all files accessible by tomcat
  sudo chown -R tomcat:tomcat <tomcattw4_location>/webapps/ROOT

• Stop tomcat with TermWeb 3.

• Create new TermWeb 4 home folder with permissions to write for Tomcat user (like /opt/tw4home, the same as value for tomcat settings in
  -Dtermweb.home=/opt/tw4home)

• Make copy of contents TermWeb 3 home folder into new TermWeb 4 home folder (old TermWeb 3 home folder usually been created right in Tomcat installation folder, it also was possible to specify it in environment variable TERMWEB_HOME)

• Make copy of <tomcat3_location>/webapps/termweb/WEB-INF/work folder into new TermWeb 4 home folder

• Make backup of TermWeb 3 database

• Start tomcat with TermWeb 3 it can continue working as usual

Update and start migration

• You may use MySQL, Tomcat and Elasticsearch on same or different hosts.

• Update or Install MySQL to version 8.0.x

• Create new scheme and user with next script on MySQL 8.0.x instance:

  create_scheme_and_user.sql

  CREATE DATABASE `termweb4db` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci; CREATE USER 'termwebuser'@'%' IDENTIFIED BY '<your_password_goes_here>'; GRANT ALL PRIVILEGES ON `termweb4db`.* TO 'termwebuser'@'%';

   

• Import old backup into new scheme on MySQL 8.0.x instance

• Install Elasticsearch 7.16.x (On linux much better to install it via rpm repository)
  https://www.elastic.co/guide/en/elasticsearch/reference/7.x/rpm.html

  In most cases for Elasticsearch process will be enough 1GB of RAM.
  Only staging areas rely on elasticsearch, so there is no need to create failover setup with two nodes.

  Install plugin for it bin/elasticsearch-plugin install analysis-icu
  
  

• Configure new termweb.properties in TermWeb 4 home folder (it should have been copied with TermWeb 3 folder), add here connection properties to new MySQL 8.0.x instance and Elasticsearch
  Disable use of SSL if you're sure about security of connection between Tomcat and MySQL by adding useSSL=false property

  termweb.properties

  // DB connection settings spring.datasource.driverClassName=com.mysql.jdbc.Driver spring.datasource.url=jdbc:mysql://mysql80host/termweb4db?autoReconnect=true&useUnicode=true&character_set_server=utf8mb4&connectionCollation=utf8mb4_general_ci&useSSL=false spring.datasource.username=dbuser spring.datasource.password=dbpassword // old TermWeb3 properties termweb.home = comment this property, now it should be specified in java options for tomcat work.dir = comment this property, work directory now is located in termweb.home folder base.url = set new value if server address will be different # cluster name used mostly in logs for identification termweb.elasticsearch.cluster.name=termwebes-cluster termweb.elasticsearch.index-prefix=termweb4 termweb.elasticsearch.address=172.1.1.1 termweb.elasticsearch.port=9200

   

• Update Apache httpd if it's too old, configure serving of angular application (termweb-ui.zip file from distributive), deploy termweb-ui (contents of dist folder) application into your httpd web directory, i.e /var/www/termweb4
  

  As example how to deploy you can check out our script for deployment. If your linux distribution is using SELinux security mechanism, then you need to run restorecon command too, otherwise Apache httpd won't be able to serve new files from /var/www/termweb4 folder.

  Deploy web static fiels

   

• Configure SSL certificate for your VirtualHost or do it later.

• Prepare database for first run by executing next sql script:

  schema_version.sql

   

• Some of old databases may have problems with migration scripts, you may encounter them or no. Check values in tw_userproperties table:

  If some of the values length of column properties bigger than 65535 then you need to update it. Safest way is to delete xml elements visitedTermOids from properties value.
  
  Sometimes migration may fail with wrong constraints for table or incompatible values, to be sure in successful result execute next script:

   

• Start tomcat as service and check the logs. TermWeb 4 should run migration scripts automatically. It can take a while (for 2GB database it can take about 1 hour)

  Run and wait

   

• Wait for message:    [           main] com.nescit.term.log.CustomLogger         : TermWeb started

• If log file contains errors check Known Issues or contact Tech support.

Known issues during migration:

• Warning about missing tables can be ignored.

  
  
  

• Problem with creation of Elasticsearch index:

  
  Occurs because of similar field names in dictionary, like 'Usage note' and 'usage note'. Can be solved by giving new name to one of the fields. And later by pressing 'Reindex' in dictionary list in AdminView.