Page Comparison

Prepare for migration

• Prior to start migration of TermWeb 3 it should be updated to latest version of 3.18.

• With migration to TermWeb 4 it keeps everything from previous version you won’t loose any history or metadata. Only old Grid View data will be dropped. Before starting migration if some users using Grid View, they must publish or discard their changes.

• Install Oracle JDK 8 (latest free version is 8u202) or OpenJDK 8 (we're using and testing with Amazon Corretto 8). Don’t install version higher than 8 TermWeb 4 won’t work with it.

• Install Tomcat 9.0.x version.
  You may get an idea how to do this by reading this article for Tomcat 8, but process is the same https://www.digitalocean.com/community/tutorials/how-to-install-apache-tomcat-8-on-centos-7.
  You may use some rpm/deb repositories with prepackaged Tomcat. But we don't know any repository which is updated regularly and may be trusted.

  One thing to note about configuring tomcat: much better to configure java options of Tomcat in setenv.sh file located in <tomcattw4_location>/bin folder (or in systemd/init.d service file) instead of modifying catalina.sh file.
  Probably you need to create file setenv.sh file if it missing. It will allow to you to update Tomcat easier by copying only this file to new installation.
  About required memory for tomcat: 750 megabytes of RAM is enough for testing of small termbases about 5k of concepts, recommended setup 2GB for 100k concepts and 50 concurrent users.

  setenv.sh

  Code Block
  language bash
  export CATALINA_OPTS="-Xms750m -Xmx750m \ -Dspring.config.name=termweb \ -Dspring.profiles.active=mysql,termweb3-search \ -Dfile.encoding=UTF-8 \ -Duser.timezone=UTC \ -Dtermweb.home=/opt/tw4home \ -verbose:gc"

• Unpack termweb4.war into webapps/ROOT of tomcat.

• Anchor
  mysql-connector-j mysql-connector-j

  Download and install mysql-connector-j into /opt/tomcat/webapps/ROOT/WEB-INF/lib
  Go to https://dev.mysql.com/downloads/connector/j/ and select platform independent version of connector compatible with MySQL version 8.0
  
  Image Added

  Unpack and move it to lib folder mentioned before:
  tar xf mysql-connector-j-8.0.33.tar.gz && cp mysql-connector-j-8.0.33/mysql-connector-j-8.0.33.jar <tomcattw4_location>/webapps/ROOT/WEB-INF/lib

• Make all files accessible by tomcat
  sudo chown -R tomcat:tomcat <tomcattw4_location>/webapps/ROOT

• Stop tomcat with TermWeb 3.

• Create new TermWeb 4 home folder with permissions to write for Tomcat user (like /opt/tw4home, the same as value for tomcat settings in
  -Dtermweb.home=/opt/tw4home)

• Make copy of contents TermWeb 3 home folder into new TermWeb 4 home folder (old TermWeb 3 home folder usually been created right in Tomcat installation folder, it also was possible to specify it in environment variable TERMWEB_HOME)

• Make copy of <tomcat3_location>/webapps/termweb/WEB-INF/work folder into new TermWeb 4 home folder

• Make backup of TermWeb 3 database

• Start tomcat with TermWeb 3 it can continue working as usual

Update and start migration

• You may use MySQL, Tomcat and Elasticsearch on same or different hosts.

• Update or Install MySQL to version 8.0.x

• Create new scheme and user with next script on MySQL 8.0.x instance:

  create_scheme_and_user.sql

  Code Block
  language sql
  CREATE DATABASE `termweb4db` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci; CREATE USER 'termwebuser'@'%' IDENTIFIED BY '<your_password_goes_here>'; GRANT ALL PRIVILEGES ON `termweb4db`.* TO 'termwebuser'@'%';

• Import old backup into new scheme on MySQL 8.0.x instance

• Install Elasticsearch 7.16.x (On linux much better to install it via rpm repository)
  https://www.elastic.co/guide/en/elasticsearch/reference/7.x/rpm.html

  In most cases for Elasticsearch process will be enough 1GB of RAM.
  Only staging areas rely on elasticsearch, so there is no need to create failover setup with two nodes.

  Install plugin for it bin/elasticsearch-plugin install analysis-icu
  

• Configure new termweb.properties in TermWeb 4 home folder (it should have been copied with TermWeb 3 folder), add here connection properties to new MySQL 8.0.x instance and Elasticsearch
  Disable use of SSL if you're sure about security of connection between Tomcat and MySQL by adding useSSL=false property

  termweb.properties

  Code Block
  language java
  // DB connection settings spring.datasource.driverClassName=com.mysql.jdbc.Driver spring.datasource.url=jdbc:mysql://mysql80host/termweb4db?autoReconnect=true&useUnicode=true&character_set_server=utf8mb4&connectionCollation=utf8mb4_general_ci&useSSL=false spring.datasource.username=dbuser spring.datasource.password=dbpassword // old TermWeb3 properties termweb.home = comment this property, now it should be specified in java options for tomcat work.dir = comment this property, work directory now is located in termweb.home folder base.url = set new value if server address will be different # cluster name used mostly in logs for identification termweb.elasticsearch.cluster.name=termwebes-cluster termweb.elasticsearch.index-prefix=termweb4 termweb.elasticsearch.address=172.1.1.1 termweb.elasticsearch.port=9200

• Update Apache httpd if it's too old, configure serving of angular application (termweb-ui.zip file from distributive), deploy termweb-ui (contents of dist folder) application into your httpd web directory, i.e /var/www/termweb4
  

  Insert excerpt
  Apache httpd configuration Apache httpd configuration nopanel true

  As example how to deploy you can check out our script for deployment. If your linux distribution is using SELinux security mechanism, then you need to run restorecon command too, otherwise Apache httpd won't be able to serve new files from /var/www/termweb4 folder.

  Deploy web static fiels

  Code Block
  language bash
  mkdir termweb-update rm -rf termweb-update/dist sudo apachectl stop unzip termweb-ui.zip -d termweb-update/ sudo rm -rfv /var/www/termweb4 sudo mkdir /var/www/termweb4 sudo cp -av /home/centos/termweb-update/dist/. /var/www/termweb4/ sudo chown -R apache: /var/www/termweb4/ sudo chmod -R 400 /var/www/termweb4 sudo find /var/www/termweb4 -type d -exec chmod u+x {} \; sudo restorecon -R -v /var/www/termweb4 sudo apachectl start

• Configure SSL certificate for your VirtualHost or do it later.

• Prepare database for first run by executing next sql script:

  schema_version.sql

  Code Block
  language sql
  CREATE TABLE `schema_version` ( `installed_rank` int NOT NULL, `version` varchar(50) DEFAULT NULL, `description` varchar(200) NOT NULL, `type` varchar(20) NOT NULL, `script` varchar(1000) NOT NULL, `checksum` int DEFAULT NULL, `installed_by` varchar(100) NOT NULL, `installed_on` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, `execution_time` int NOT NULL, `success` tinyint(1) NOT NULL, PRIMARY KEY (`installed_rank`), KEY `schema_version_s_idx` (`success`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; INSERT INTO `schema_version` (`installed_rank`, `version`, `description`, `type`, `script`, `checksum`, `installed_by`, `installed_on`, `execution_time`, `success`) VALUES ('1', '3.18', 'baseline', 'SQL', 'V3.18__baseline.sql', '-1645915861', 'term2', NOW(), '465', '1'); -- Before version 4.0.4 here must be this line --('1', '3.18', 'baseline', 'SQL', 'V3.18__baseline.sql', '942246576', 'term2', NOW(), '465', '1');

• Some of old databases may have problems with migration scripts, you may encounter them or no. Check values in tw_userproperties table:

  Code Block
  language sql
  SELECT *, LENGTH(properties) AS c1 FROM tw_userproperties ORDER BY c1 DESC;

  If some of the values length of column properties bigger than 65535 then you need to update it. Safest way is to delete xml elements visitedTermOids from properties value.
  
  Sometimes migration may fail with wrong constraints for table or incompatible values, to be sure in successful result execute next script:

  Code Block
  language sql
  UPDATE tw_client SET api_key = '' WHERE api_key IS NULL; UPDATE tw_filter SET group_oid = '' WHERE group_oid IS NULL; ALTER TABLE `tw_client` CHANGE COLUMN `oid` `oid` varchar(14) NOT NULL DEFAULT '', CHANGE COLUMN `created` `created` datetime NOT NULL, CHANGE COLUMN `changed` `changed` datetime NOT NULL, CHANGE COLUMN `created_by` `created_by` varchar(14) NOT NULL DEFAULT '', CHANGE COLUMN `changed_by` `changed_by` varchar(14) NOT NULL DEFAULT '', CHANGE COLUMN `name` `name` varchar(255) NOT NULL, CHANGE COLUMN `login` `login` varchar(255) NOT NULL, CHANGE COLUMN `max_concurrent_users` `max_concurrent_users` int NOT NULL, CHANGE COLUMN `max_user_accounts` `max_user_accounts` int NOT NULL, CHANGE COLUMN `active` `active` tinyint NOT NULL, CHANGE COLUMN `authkey` `authkey` varchar(255) NOT NULL, CHANGE COLUMN `api_key` `api_key` varchar(255) NOT NULL, CHANGE COLUMN `xtm_customer` `xtm_customer` VARCHAR(256) CHARACTER SET 'utf8' COLLATE 'utf8_unicode_ci' NULL; ALTER TABLE tw_concept CHANGE COLUMN `created` `created` datetime NOT NULL, CHANGE COLUMN `changed` `changed` datetime NOT NULL; ALTER TABLE tw_dictionaries CHANGE COLUMN `created` `created` datetime NOT NULL, CHANGE COLUMN `changed` `changed` datetime NOT NULL; ALTER TABLE tw_domain CHANGE COLUMN `created` `created` datetime NOT NULL, CHANGE COLUMN `changed` `changed` datetime NOT NULL; ALTER TABLE tw_exportsettings CHANGE COLUMN `created` `created` datetime NOT NULL, CHANGE COLUMN `changed` `changed` datetime NOT NULL; ALTER TABLE tw_field CHANGE COLUMN `created` `created` datetime NOT NULL, CHANGE COLUMN `changed` `changed` datetime NOT NULL; ALTER TABLE tw_filter CHANGE COLUMN `created` `created` datetime NOT NULL, CHANGE COLUMN `changed` `changed` datetime NOT NULL; ALTER TABLE tw_filter_baddata CHANGE COLUMN `created` `created` datetime NOT NULL, CHANGE COLUMN `changed` `changed` datetime NOT NULL; ALTER TABLE tw_groups CHANGE COLUMN `created` `created` datetime NOT NULL, CHANGE COLUMN `changed` `changed` datetime NOT NULL; ALTER TABLE tw_historyitem CHANGE COLUMN `date` `date` datetime NOT NULL; ALTER TABLE tw_importsettings CHANGE COLUMN `created` `created` datetime NOT NULL, CHANGE COLUMN `changed` `changed` datetime NOT NULL; ALTER TABLE tw_scheduledjobresult CHANGE COLUMN `start_date` `start_date` datetime NOT NULL, CHANGE COLUMN `end_date` `end_date` datetime NOT NULL; ALTER TABLE tw_sections CHANGE COLUMN `created` `created` datetime NOT NULL, CHANGE COLUMN `changed` `changed` datetime NOT NULL; ALTER TABLE tw_synchronization CHANGE COLUMN `elements_changed` `elements_changed` datetime NOT NULL, CHANGE COLUMN `contents_changed` `contents_changed` datetime NOT NULL; ALTER TABLE tw_term CHANGE COLUMN `created` `created` datetime NOT NULL, CHANGE COLUMN `changed` `changed` datetime NOT NULL; ALTER TABLE tw_user CHANGE COLUMN `created` `created` datetime NOT NULL, CHANGE COLUMN `changed` `changed` datetime NOT NULL; ALTER TABLE tw_userlog CHANGE COLUMN `login` `login` datetime NOT NULL, CHANGE COLUMN `logout` `logout` datetime DEFAULT NULL; ALTER TABLE tw_view CHANGE COLUMN `created` `created` datetime NOT NULL, CHANGE COLUMN `changed` `changed` datetime NOT NULL; ALTER TABLE tw_virtualfile CHANGE COLUMN `created` `created` datetime NOT NULL, CHANGE COLUMN `changed` `changed` datetime NOT NULL; UPDATE tw_exportsettings d JOIN tw_exportsettings s ON s.oid = d.oid SET d.file_name = s.name WHERE d.file_name IS NULL; UPDATE tw_groups g SET g.lang_code3 = '' WHERE g.lang_code3 IS NULL; UPDATE tw_term t SET t.config = '' WHERE t.config IS NULL; UPDATE tw_term t SET t.src_term_oid = '' WHERE t.src_term_oid IS NULL;

• Start tomcat as service and check the logs. TermWeb 4 should run migration scripts automatically. It can take a while (for 2GB database it can take about 1 hour)

  Run and wait

  Code Block
  language bash
  sudo service tomcat start && sudo tail -f <tomcattw4_location>/logs/catalina.out

• Wait for message:    [           main] com.nescit.term.log.CustomLogger         : TermWeb started

• If log file contains errors check Known Issues or contact Tech support.

Anchor
knownissues knownissues

Known issues during migration:

• Warning about missing tables can be ignored.

  Code Block
  WARN 14003 --- [ main] o.f.c.internal.dbsupport.JdbcTemplate : DB: Unknown table 'termweb4.tw4_dictionary_filter' (SQL State: 42S02 - Error Code: 1051) WARN 14003 --- [ main] o.f.c.internal.dbsupport.JdbcTemplate : DB: Unknown table 'termweb4.tw4_filter' (SQL State: 42S02 - Error Code: 1051)

  
  

• Problem with creation of Elasticsearch index:

  Code Block

  INFO 15658 --- [ main] com.nescit.term.log.CustomLogger : Starting creation of Elasticsearch index for Dictionary[oid=d2j0t]. ERROR 15658 --- [ main] c.t.c.m.search.service.IndexServiceImpl : Can't create ES index. org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=parse_exception, reason=Failed to parse content to map] at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177) at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1727) at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1704)

  
  Occurs because of similar field names in dictionary, like 'Usage note' and 'usage note'. Can be solved by giving new name to one of the fields. And later by pressing 'Reindex' in dictionary list in AdminView.