Bug #9695
Updated by Chambon Bernard over 9 years ago
Let's start with ConcurrentModificationException
* ConcurrentModificationException
Found on both hosts !, found 3 use cases of such error
*To be investiguated*
** Solution ? :
Provide sync access for :
QueuesController on queuesMap
FilePositionsOnTape on objectMap
TapeController idem
ResourceController on resources (List<Resource> resources)
StagerController on stagers
AbstractController : provide all sync methods (add, exists, )
...
use case 1
<pre>
java.util.ConcurrentModificationException: null
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922) ~[na:1.7.0_65]
at java.util.HashMap$KeyIterator.next(HashMap.java:956) ~[na:1.7.0_65]
at fr.in2p3.cc.storage.treqs.control.controller.QueuesController.getWaitingQueues(QueuesController.java:626) ~[jtreqs-server-jav
a-1.6.1-SNAPSHOT.jar:na]
at fr.in2p3.cc.storage.treqs.control.activator.Activator.process(Activator.java:466) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
at fr.in2p3.cc.storage.treqs.control.activator.Activator.action(Activator.java:251) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
at fr.in2p3.cc.storage.treqs.control.activator.Activator.toStart(Activator.java:670) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
at fr.in2p3.cc.storage.treqs.control.process.AbstractProcess.run(AbstractProcess.java:204) [jtreqs-server-java-1.6.1-SNAPSHOT.ja
r:na]
2014-09-21 22:00:25,577 [Activator] WARN Activator Stopped
JNIException created
2014-09-21 22:00:25,578 [tape-1411329623542-54-JTC61400-27061] WARN Error processing this file: /hpss/in2p3.fr/group/dchooz/DATA/raw/DC
Run0017577_RecoP_v17_FD_prof_5.root File locked or HSM is currently unavailable.
2014-09-21 22:00:25,578 [tape-1411329623542-54-JTC61400-27061] WARN General error. Retrying /hpss/in2p3.fr/group/dchooz/DATA/raw/DCRun0
017577_RecoP_v17_FD_prof_5.root
2014-09-21 22:01:21,871 [Dispatcher] WARN Dispatcher Stopped
2014-09-21 22:02:04,347 [main] ERROR No heartbeat, the application is dying.
2014-09-21 22:02:04,347 [main] WARN Dispatcher and Activator are stopped
jTReqS-Server started
</pre>
use case 2
<pre>
2015-03-11 07:17:23,872 [Dispatcher] ERROR f.i.c.s.t.c.dispatcher.Dispatcher - Stopping
fr.in2p3.cc.storage.treqs.control.dispatcher.DispatcherException: java.util.ConcurrentModificationException
at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.action(Dispatcher.java:237) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.toStart(Dispatcher.java:887) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
at fr.in2p3.cc.storage.treqs.control.process.AbstractProcess.run(AbstractProcess.java:204) [jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
Caused by: java.util.ConcurrentModificationException: null
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922) ~[na:1.7.0_75]
at java.util.HashMap$ValueIterator.next(HashMap.java:950) ~[na:1.7.0_75]
at fr.in2p3.cc.storage.treqs.control.controller.FilePositionOnTapesController.exists(FilePositionOnTapesController.java:276) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
at fr.in2p3.cc.storage.treqs.control.controller.TapesController.cleanup(TapesController.java:166) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.cleaningReferences(Dispatcher.java:342) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.action(Dispatcher.java:232) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
</pre>
use case 3
<pre>
2015-04-04 06:31:12,608 [Dispatcher] ERROR f.i.c.s.t.c.dispatcher.Dispatcher - Stopping
fr.in2p3.cc.storage.treqs.control.dispatcher.DispatcherException: java.util.ConcurrentModificationException
at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.action(Dispatcher.java:255) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.toStart(Dispatcher.java:887) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
at fr.in2p3.cc.storage.treqs.control.process.AbstractProcess.run(AbstractProcess.java:204) [jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
Caused by: java.util.ConcurrentModificationException: null
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859) ~[na:1.7.0_75]
at java.util.ArrayList$Itr.next(ArrayList.java:831) ~[na:1.7.0_75]
at fr.in2p3.cc.storage.treqs.control.controller.StagersController.getActiveStagersForQueue(StagersController.java:219) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
at fr.in2p3.cc.storage.treqs.control.controller.QueuesController.addFilePositionOnTape(QueuesController.java:200) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.submitRequest(Dispatcher.java:866) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.innerProcess(Dispatcher.java:610) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.process(Dispatcher.java:701) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.retrieveNewRequests(Dispatcher.java:772) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.action(Dispatcher.java:242) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na]
... 2 common frames omitted
2015-04-04 06:31:12,609 [Dispatcher] WARN f.i.c.s.t.c.dispatcher.Dispatcher - Dispatcher Stopped
</pre>