Project

General

Profile

Actions

Task #10294

open

Deploy treqs on test instance (ccsvli10) for test by PEB

Added by Chambon Bernard over 6 years ago. Updated over 6 years ago.

Status:
New
Priority:
Normal
Category:
Tests and feebacks
Start date:
07/03/2015
Due date:
% Done:

50%

Estimated time:
Actions #1

Updated by Brinette Pierre-Emmanuel over 6 years ago

  • % Done changed from 0 to 50

Déploiement de la dernière version disponible sur le repository :

http://maven.in2p3.fr/fr/in2p3/cc/storage/treqs/treqs-delivery/1.0-SNAPSHOT/treqs-delivery-1.0-20150703.143750-35.rpm

Arret de treqs :

# /etc/init.d/treqsd stop
Stopping treqs-delivery...
Stopped treqs-delivery.

Sauvegarde le version précédente :

# cp -rp treqs-1.7.0 treqs-20150626

Installation du package :


# rpm -Uvh --force treqs-delivery-1.0-20150703.143750-35.rpm
Preparing...                ########################################### [100%]
   1:treqs                  warning: /opt/treqs-1.7.0/etc/logback.xml created as /opt/treqs-1.7.0/etc/logback.xml.rpmnew
########################################### [100%]

Configuration

Modification du fichier de log du wrapper

# diff wrapper.conf.orig wrapper.conf
64c64
< wrapper.logfile=logs/wrapper.log
---
> wrapper.logfile=/var/log/jtreqs/wrapper.log
121c121,122

Fichiers de logs :

Activation du logfile et du tracefile dans logback.xml;


# diff logback.xml logback.xml.rpmnew  
58a59
> <!--
76a78
> -->
94c96
<     <appender-ref ref="LogFile" /> 
---
>   <!--  <appender-ref ref="LogFile" /> -->

Configuration de la BDD :

Installation des triggers et des procédures stockées :

# mysql -uroot -pxxxxxx treqsjobs  < DBManagement.sql 
# mysql -uroot -pxxxxxx treqsjobs  < archive.sql `

Démarrage de treqs

/etc/init.d/treqsd start
Starting treqs-delivery...
wrapper  | Working directory set to: ..
wrapper  | Spawning intermediate process...
ccsvli10_root# ls -ltr | tail -3
-rw-r--r-- 1 root root   4580 Jul  6 11:39 jtreqs.log
-rw-r--r-- 1 root root 218327 Jul  6 11:39 jtreqs-trace.log
-rw-r--r-- 1 root root  29813 Jul  6 11:39 wrapper.log

test

Depuis une ccdevli :

$ export TreqsDBHost=ccsvli10
$ /usr/local/products/treqsc/bin/trcp.pl -d  /hpss/in2p3.fr/home/p/pbrinett/GOTCHA
2015/07/06 11:49:11 [15461] INFO> Treqs Client v0.2 
2015/07/06 11:49:11 [15461] DEBUG> File to stage: /hpss/in2p3.fr/home/p/pbrinett/GOTCHA 
2015/07/06 11:49:11 [15461] DEBUG> Resolved IP for hostname: ccosvms0007 -> 134.158.239.7
2015/07/06 11:49:11 [15461] DEBUG> Client IP adress is: 134.158.239.7 
2015/07/06 11:49:11 [15461] DEBUG> Username is: pbrinett 
2015/07/06 11:49:11 [15461] INFO> Connect to TREQS SERVER (ccsvli10) on DB (treqsjobs) as treqsc
2015/07/06 11:49:11 [15461] INFO> Timout set to (21600 s) 
2015/07/06 11:49:11 [15461] INFO> First pool interval is 30s , Pool interval is 60 
2015/07/06 11:49:11 [15461] DEBUG> Connection to Database (DBI:mysql:database=treqsjobs;host=ccsvli10) succeed
2015/07/06 11:49:11 [15461] DEBUG> Resolved IP for hostname: ccosvms0007 -> 134.158.239.7
2015/07/06 11:49:11 [15461] DEBUG> Insert file request:         
  INSERT INTO treqsjobs.jrequests(file,creation_time,user,client,email,version,status)
        VALUES            
  ('/hpss/in2p3.fr/home/p/pbrinett/GOTCHA' , NOW(), 'pbrinett', '134.158.239.7', 'pbrinett', 'treqs perl v0.2', 100)          ;    
2015/07/06 11:49:11 [15461] INFO> Request for file /hpss/in2p3.fr/home/p/pbrinett/GOTCHA succefully inserted, request id is (87867)
2015/07/06 11:49:11 [15461] DEBUG> Database disconnected
2015/07/06 11:49:11 [15461] DEBUG> Sleep 30 s before first pool
2015/07/06 11:49:41 [15461] DEBUG> Connection to Database (DBI:mysql:database=treqsjobs;host=ccsvli10) succeed
2015/07/06 11:49:41 [15461] DEBUG> Pool file request: select * from treqsjobs.jrequests where id = 87867
2015/07/06 11:49:41 [15461] INFO> Request id (87867) status : 150 (ON_DISK)
2015/07/06 11:49:41 [15461] DEBUG> Database disconnected
2015/07/06 11:49:41 [15461] DEBUG> Loop ended, status is : 150 (ON_DISK) Timout exceed : 0 
2015/07/06 11:49:41 [15461] INFO> File /hpss/in2p3.fr/home/p/pbrinett/GOTCHA allready on disk
2015/07/06 11:49:41 [15461] INFO> Program terminating: return code is (0)...

/var/log/jtreqs/jtreqs-trace.log

2015-07-06 11:49:36,411 [Dispatcher] INFO  Dispatcher - Looking for new requests
2015-07-06 11:49:36,412 [Dispatcher] INFO  MySQLBroker - Mysql access duration (MySQLBroker.executeSelect method) took 0 ms (0 s) 
2015-07-06 11:49:36,415 [Dispatcher] INFO  Dispatcher - Beginning MetaData fishing on HSM for 1 files
2015-07-06 11:49:36,423 [Dispatcher_req_87867] INFO  AbstractHSMBridge - Testing keytab: /var/hpss/etc/keytab.root
2015-07-06 11:49:36,442 [Observer] INFO  MySQLBroker - Mysql access duration (MySQLBroker.executeSelect method) took 1 ms (0 s) 
2015-07-06 11:49:36,571 [Dispatcher_req_87867] INFO  HPSSJNIBridge - Hpss query attributes duration for file /hpss/in2p3.fr/home/p/pbrinett/GOTCHA was 136 ms (0 s) 
2015-07-06 11:49:36,571 [Dispatcher_req_87867] INFO  HPSSJNIBridge - filename '/hpss/in2p3.fr/home/p/pbrinett/GOTCHA' '{ position: 0, offset: 0 }' storageName 'DISK' size '9'
2015-07-06 11:49:36,572 [Dispatcher_req_87867] INFO  Dispatcher - File /hpss/in2p3.fr/home/p/pbrinett/GOTCHA is on disk, set the request as done
2015-07-06 11:49:36,579 [Dispatcher_req_87867] INFO  MySQLReadingDAO - Mysql access duration (MySQLReadingDAO.setRequestStatusById method) took 4 ms (0 s) 
2015-07-06 11:49:36,580 [Dispatcher_req_87867] INFO  Dispatcher - Dispatching duration (innerProcess method) took 164 ms (0 s) 
2015-07-06 11:49:36,580 [Dispatcher_req_87867] INFO  Dispatcher - Processing 1 request(s)

Actions #2

Updated by Brinette Pierre-Emmanuel over 6 years ago

Staging de 1620 fichiers sur une bande T10K-D insérée dans l'ordre :

$  query_volume.ksh KT279300

Volume KT279300 found on Subsystem 3 , Storage class 114

           VV           Storage
Volume    Cond    Family  Class                 Space Used            Max Volume Size
========  ======  ======  =====       ====================       ==================== 
KT279300   EOM      0    114          6,466,516,418,247          8,571,050,393,287  Repack output tape
$ lsvol -i3 -v KT279300 | sort -n | process_ls_vol.pl - > KT279300.treqs.test
$ export TreqsDBHost=ccsvli10
$ trcp.pl -p KT279300.treqs.test -u test01
2015/07/06 12:14:48 WARN> 1620 prestage requests has been inserted for user test01
2015/07/06 12:14:48 WARN> Program succefully ended

Démarrage des stagers :

2015-07-06 12:17:06,415 [Stager_QnKT279300_QId151_stagerNo_1] INFO  AbstractProcess - Setting status  to value STARTED 
2015-07-06 12:17:06,415 [Stager_QnKT279300_QId151_stagerNo_1] INFO  Stager - Stager Stager_QnKT279300_QId151_stagerNo_1: starting.
2015-07-06 12:17:06,420 [Stager_QnKT279300_QId151_stagerNo_1] INFO  MySQLQueueDAO - Mysql access duration (MySQLQueueDAO.processUpdate method) took 0 ms (0 s) 
2015-07-06 12:17:06,420 [Stager_QnKT279300_QId151_stagerNo_1] INFO  MySQLQueueDAO - Updated queue 151
2015-07-06 12:17:06,423 [Stager_QnKT279300_QId151_stagerNo_1] INFO  MySQLReadingDAO - Mysql access duration (MySQLReadingDAO.processUpdate method) took 2 ms (0 s) 
2015-07-06 12:17:06,423 [Stager_QnKT279300_QId151_stagerNo_1] INFO  Reading - File /hpss3/dcache/cms/hpssdata/000098214CB7343149D1A701C344ED3C7497 in tape KT279300 at, position { position: 1634, offset: 0 }: Started.
2015-07-06 12:17:06,674 [Observer] INFO  MySQLBroker - Mysql access duration (MySQLBroker.executeSelect method) took 1 ms (0 s) 
2015-07-06 12:17:07,415 [Activator] INFO  Activator - Starting stager 2 of 2
2015-07-06 12:17:07,415 [Activator] INFO  AbstractProcess - Setting status  to value STARTING 
2015-07-06 12:17:07,415 [Activator] INFO  Activator - Sleeping between stagers, 1000 millis
2015-07-06 12:17:07,416 [Stager_QnKT279300_QId151_stagerNo_2] INFO  AbstractProcess - Setting status  to value STARTED 
2015-07-06 12:17:07,416 [Stager_QnKT279300_QId151_stagerNo_2] INFO  Stager - Stager Stager_QnKT279300_QId151_stagerNo_2: starting.
2015-07-06 12:17:07,423 [Stager_QnKT279300_QId151_stagerNo_2] INFO  MySQLQueueDAO - Mysql access duration (MySQLQueueDAO.processUpdate method) took 2 ms (0 s) 
2015-07-06 12:17:07,423 [Stager_QnKT279300_QId151_stagerNo_2] INFO  MySQLQueueDAO - Updated queue 151
2015-07-06 12:17:07,427 [Stager_QnKT279300_QId151_stagerNo_2] INFO  MySQLReadingDAO - Mysql access duration (MySQLReadingDAO.processUpdate method) took 3 ms (0 s) 
2015-07-06 12:17:07,427 [Stager_QnKT279300_QId151_stagerNo_2] INFO  Reading - File /hpss3/dcache/cms/hpssdata/0000F1BBDAC3130F4D2B87D58B58099BB7BD in tape KT279300 at, position { position: 1199, offset: 0 }: Started.

PROBLEMES

Les stagers ont démarrés avec des fichier en position 1199 et 1634 alors qu'il n'y a qu'1 seule queue contenant tous les fichiers de la bande !
Les stages auraient du démarraer en * position 1 et 2*

Actions #3

Updated by Brinette Pierre-Emmanuel over 6 years ago

Nouveaux tests

Utilisation du package RPM 'treqs-delivery-1.0-20150707.132153-37.rpm'

Installation

pm -Uvh treqs-delivery-1.0-20150707.132153-37.rpm 
Preparing...                ########################################### [100%]
    package treqs-1.7-0.x86_64 is already installed
    file /opt/treqs-1.7.0/etc/version.txt from install of treqs-1.7-0.x86_64 conflicts with file from package treqs-1.7-0.x86_64
    file /opt/treqs-1.7.0/lib/treqs-java-1.0-SNAPSHOT.jar from install of treqs-1.7-0.x86_64 conflicts with file from package treqs-1.7-0.x86_64
ccsvli10_root# rpm -Uvh treqs-delivery-1.0-20150707.132153-37.rpm  --force 
Preparing...                ########################################### [100%]
   1:treqs                  ########################################### [100%]

Test avec les fichiers de la bande KT279300

$ trcp.pl -p KT279300.treqs.test -u test01
2015/07/07 16:10:53 WARN> 1620 prestage requests has been inserted for user test01
2015/07/07 16:10:53 WARN> Program succefully ended
Actions #4

Updated by Brinette Pierre-Emmanuel over 6 years ago

Installation du RPM

VERSION: treqs-delivery-1.0-20150713.130035-41.rpm

rpm -Uvh treqs-delivery-1.0-20150713.130035-41.rpm  
Preparing...                ########################################### [100%]
   1:treqs                  ########################################### [100%]

Configuration

Mise a jour du schema de la base :

# mysql -uroot -pxxxxxxx treqsjobs < CreateDBTables.sql 
ERROR 1217 (23000) at line 25: Cannot delete or update a parent row: a foreign key constraint fails
Actions

Also available in: Atom PDF