TReqS

Let's start with ConcurrentModificationException

• ConcurrentModificationException
  Found on both servers !, found 3 use cases of such error

use case 1

java.util.ConcurrentModificationException: null
    at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922) ~[na:1.7.0_65]
    at java.util.HashMap$KeyIterator.next(HashMap.java:956) ~[na:1.7.0_65]
    at fr.in2p3.cc.storage.treqs.control.controller.QueuesController.getWaitingQueues(QueuesController.java:626) ~[jtreqs-server-jav
a-1.6.1-SNAPSHOT.jar:na]
    at fr.in2p3.cc.storage.treqs.control.activator.Activator.process(Activator.java:466) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
    at fr.in2p3.cc.storage.treqs.control.activator.Activator.action(Activator.java:251) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
    at fr.in2p3.cc.storage.treqs.control.activator.Activator.toStart(Activator.java:670) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
    at fr.in2p3.cc.storage.treqs.control.process.AbstractProcess.run(AbstractProcess.java:204) [jtreqs-server-java-1.6.1-SNAPSHOT.ja
r:na]
2014-09-21 22:00:25,577 [Activator] WARN  Activator Stopped
JNIException created
2014-09-21 22:00:25,578 [tape-1411329623542-54-JTC61400-27061] WARN  Error processing this file: /hpss/in2p3.fr/group/dchooz/DATA/raw/DC
Run0017577_RecoP_v17_FD_prof_5.root File locked or HSM is currently unavailable.
2014-09-21 22:00:25,578 [tape-1411329623542-54-JTC61400-27061] WARN  General error. Retrying /hpss/in2p3.fr/group/dchooz/DATA/raw/DCRun0
017577_RecoP_v17_FD_prof_5.root
2014-09-21 22:01:21,871 [Dispatcher] WARN  Dispatcher Stopped
2014-09-21 22:02:04,347 [main] ERROR No heartbeat, the application is dying.
2014-09-21 22:02:04,347 [main] WARN  Dispatcher and Activator are stopped
jTReqS-Server started

2015-03-11 07:17:23,872 [Dispatcher] ERROR f.i.c.s.t.c.dispatcher.Dispatcher - Stopping
fr.in2p3.cc.storage.treqs.control.dispatcher.DispatcherException: java.util.ConcurrentModificationException
        at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.action(Dispatcher.java:237) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
        at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.toStart(Dispatcher.java:887) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
        at fr.in2p3.cc.storage.treqs.control.process.AbstractProcess.run(AbstractProcess.java:204) [jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
Caused by: java.util.ConcurrentModificationException: null
        at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922) ~[na:1.7.0_75]
        at java.util.HashMap$ValueIterator.next(HashMap.java:950) ~[na:1.7.0_75]
        at fr.in2p3.cc.storage.treqs.control.controller.FilePositionOnTapesController.exists(FilePositionOnTapesController.java:276) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
        at fr.in2p3.cc.storage.treqs.control.controller.TapesController.cleanup(TapesController.java:166) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
        at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.cleaningReferences(Dispatcher.java:342) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
        at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.action(Dispatcher.java:232) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]

2015-04-04 06:31:12,608 [Dispatcher] ERROR f.i.c.s.t.c.dispatcher.Dispatcher - Stopping
fr.in2p3.cc.storage.treqs.control.dispatcher.DispatcherException: java.util.ConcurrentModificationException
        at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.action(Dispatcher.java:255) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
        at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.toStart(Dispatcher.java:887) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
        at fr.in2p3.cc.storage.treqs.control.process.AbstractProcess.run(AbstractProcess.java:204) [jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
Caused by: java.util.ConcurrentModificationException: null
        at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859) ~[na:1.7.0_75]
        at java.util.ArrayList$Itr.next(ArrayList.java:831) ~[na:1.7.0_75]
        at fr.in2p3.cc.storage.treqs.control.controller.StagersController.getActiveStagersForQueue(StagersController.java:219) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
        at fr.in2p3.cc.storage.treqs.control.controller.QueuesController.addFilePositionOnTape(QueuesController.java:200) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
        at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.submitRequest(Dispatcher.java:866) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
        at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.innerProcess(Dispatcher.java:610) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
        at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.process(Dispatcher.java:701) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
        at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.retrieveNewRequests(Dispatcher.java:772) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
        at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.action(Dispatcher.java:242) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
        ... 2 common frames omitted
2015-04-04 06:31:12,609 [Dispatcher] WARN  f.i.c.s.t.c.dispatcher.Dispatcher - Dispatcher Stopped