Bug #9695: Facing ConcurrentModificationException ? - TReqS - IN2P3-Forge

Bug #9695

Updated by Chambon Bernard over 9 years ago

Let's start with ConcurrentModificationException 

 * ConcurrentModificationException 
 Found on both hosts !, found 3 use cases of such error 

 
 *To be investiguated*  
 ** Solution ? :  
 Provide sync access for :  
  QueuesController on queuesMap 
  FilePositionsOnTape on objectMap 
  TapeController     idem 
  ResourceController on resources (List<Resource> resources) 
  StagerController on stagers 
  AbstractController : provide all sync methods (add, exists, ) 
  ... 
  
  


 use case 1 
 <pre> 
 java.util.ConcurrentModificationException: null 
	 at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922) ~[na:1.7.0_65] 
	 at java.util.HashMap$KeyIterator.next(HashMap.java:956) ~[na:1.7.0_65] 
	 at fr.in2p3.cc.storage.treqs.control.controller.QueuesController.getWaitingQueues(QueuesController.java:626) ~[jtreqs-server-jav 
 a-1.6.1-SNAPSHOT.jar:na] 
	 at fr.in2p3.cc.storage.treqs.control.activator.Activator.process(Activator.java:466) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na] 
	 at fr.in2p3.cc.storage.treqs.control.activator.Activator.action(Activator.java:251) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na] 
	 at fr.in2p3.cc.storage.treqs.control.activator.Activator.toStart(Activator.java:670) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na] 
	 at fr.in2p3.cc.storage.treqs.control.process.AbstractProcess.run(AbstractProcess.java:204) [jtreqs-server-java-1.6.1-SNAPSHOT.ja 
 r:na] 
 2014-09-21 22:00:25,577 [Activator] WARN    Activator Stopped 
 JNIException created 
 2014-09-21 22:00:25,578 [tape-1411329623542-54-JTC61400-27061] WARN    Error processing this file: /hpss/in2p3.fr/group/dchooz/DATA/raw/DC 
 Run0017577_RecoP_v17_FD_prof_5.root File locked or HSM is currently unavailable. 
 2014-09-21 22:00:25,578 [tape-1411329623542-54-JTC61400-27061] WARN    General error. Retrying /hpss/in2p3.fr/group/dchooz/DATA/raw/DCRun0 
 017577_RecoP_v17_FD_prof_5.root 
 2014-09-21 22:01:21,871 [Dispatcher] WARN    Dispatcher Stopped 
 2014-09-21 22:02:04,347 [main] ERROR No heartbeat, the application is dying. 
 2014-09-21 22:02:04,347 [main] WARN    Dispatcher and Activator are stopped 
 jTReqS-Server started 
 </pre> 
 use case 2 
 <pre> 
 2015-03-11 07:17:23,872 [Dispatcher] ERROR f.i.c.s.t.c.dispatcher.Dispatcher - Stopping 
 fr.in2p3.cc.storage.treqs.control.dispatcher.DispatcherException: java.util.ConcurrentModificationException 
         at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.action(Dispatcher.java:237) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na] 
         at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.toStart(Dispatcher.java:887) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na] 
         at fr.in2p3.cc.storage.treqs.control.process.AbstractProcess.run(AbstractProcess.java:204) [jtreqs-server-java-1.6.1-SNAPSHOT.jar:na] 
 Caused by: java.util.ConcurrentModificationException: null 
         at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922) ~[na:1.7.0_75] 
         at java.util.HashMap$ValueIterator.next(HashMap.java:950) ~[na:1.7.0_75] 
         at fr.in2p3.cc.storage.treqs.control.controller.FilePositionOnTapesController.exists(FilePositionOnTapesController.java:276) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na] 
         at fr.in2p3.cc.storage.treqs.control.controller.TapesController.cleanup(TapesController.java:166) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na] 
         at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.cleaningReferences(Dispatcher.java:342) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na] 
         at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.action(Dispatcher.java:232) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na] 
 </pre> 
 use case 3 
 <pre> 
 2015-04-04 06:31:12,608 [Dispatcher] ERROR f.i.c.s.t.c.dispatcher.Dispatcher - Stopping 
 fr.in2p3.cc.storage.treqs.control.dispatcher.DispatcherException: java.util.ConcurrentModificationException 
         at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.action(Dispatcher.java:255) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na] 
         at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.toStart(Dispatcher.java:887) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na] 
         at fr.in2p3.cc.storage.treqs.control.process.AbstractProcess.run(AbstractProcess.java:204) [jtreqs-server-java-1.6.1-SNAPSHOT.jar:na] 
 Caused by: java.util.ConcurrentModificationException: null 
         at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859) ~[na:1.7.0_75] 
         at java.util.ArrayList$Itr.next(ArrayList.java:831) ~[na:1.7.0_75] 
         at fr.in2p3.cc.storage.treqs.control.controller.StagersController.getActiveStagersForQueue(StagersController.java:219) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na] 
         at fr.in2p3.cc.storage.treqs.control.controller.QueuesController.addFilePositionOnTape(QueuesController.java:200) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na] 
         at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.submitRequest(Dispatcher.java:866) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na] 
         at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.innerProcess(Dispatcher.java:610) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na] 
         at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.process(Dispatcher.java:701) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na] 
         at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.retrieveNewRequests(Dispatcher.java:772) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na] 
         at fr.in2p3.cc.storage.treqs.control.dispatcher.Dispatcher.action(Dispatcher.java:242) ~[jtreqs-server-java-1.6.1-SNAPSHOT.jar:na] 
         ... 2 common frames omitted 
 2015-04-04 06:31:12,609 [Dispatcher] WARN    f.i.c.s.t.c.dispatcher.Dispatcher - Dispatcher Stopped 

 </pre>

Back

Project

General

Profile

TReqS

Bug #9695