00:00:00 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOQ834 0,7,1,14 00:00:00 stkenmvr195a.fnal.gov 021479 root E LTO4_95MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:00 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:00 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ825 0,6,1,8 00:00:00 stkenmvr206a.fnal.gov 026766 root E LTO4_106MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:00 stkenmvr206a.fnal.gov 026766 root I LTO4_106MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:00 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:00 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ859 0,7,1,3 00:00:00 stkenmvr141a.fnal.gov 013963 root E LTO4_41MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:00 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:00 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:01 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC PROCESSING listDrives 00:00:01 stkensrv4n.fnal.gov 005843 enstore I SL8500F1MC PROCESSING listDrives 00:00:01 stkensrv4n.fnal.gov 005852 enstore I STKMC PROCESSING listDrives 00:00:01 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:01 last message repeated 1 times 00:00:01 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:01 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:02 cmsstor132.fnal.gov 000490 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Scientific Linux SL release 5.2 (Boron) Rocks release 5.1 (V.I) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms2 FF Wrapper: cpio_odc FF Width: 7 Current working directory: cmsstor132.fnal.gov:/root Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --age-time 30 --delpri 1 --priority 25 --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/PATLayer1_RECO_Output_QCDDiJet_Pt1400to1800_7.root --put-cache 0010000000000000094DDAA0 /storage/data3/write-pool-3/data/0010000000000000094DDAA0 00:00:02 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:02 cmsstor132.fnal.gov 000490 root I ENCP Creating zero length output files. 00:00:02 cmsstor132.fnal.gov 000490 root I ENCP Sending /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/PATLayer1_RECO_Output_QCDDiJet_Pt1400to1800_7.root write request to LM: unique_id: cmsstor132.fnal.gov-1257051602-490-0 inputfile: /storage/data3/write-pool-3/data/0010000000000000094DDAA0 outputfile: /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/.(access)(0010000000000000094DDAA0) 00:00:02 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:02 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data3/write-pool-3/data/0010000000000000094DDAA0 (cmsstor132.fnal.gov-1257051602-490-0) -> /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/PATLayer1_RECO_Output_QCDDiJet_Pt1400to1800_7.root : library=CD-LTO4G1 family=cms2 requester:root volume_family:cms.cms2.cpio_odc 00:00:02 cmsstor132.fnal.gov 000490 root I ENCP Listening for control socket at: ('131.225.189.103', 42145) 00:00:02 cmsstor132.fnal.gov 000490 root I ENCP Waiting for mover to connect control socket at address: ('131.225.189.103', 42145) elapsed=0.790sec 00:00:02 stkensrv4n.fnal.gov 019416 enstore I SL8500G1MC dismount VOLUME 0,7,1,0 force => 0,Dismount: Forced dismount of VOQ856 from 0, 7, 1, 0 00:00:02 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC FINISHED dismount VOQ856 0,7,1,0 returned ('ok', 0, 'dismount VOLUME 0,7,1,0 force => 0,Dismount: Forced dismount of VOQ856 from 0, 7, 1, 0', '', '') 00:00:02 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV dismounted VOQ856 ULTRIUM-TD4 00:00:02 Thread media_thread 00:00:02 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC PROCESSING dismount VOU664 0,6,1,0 00:00:03 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Assigning volume VOS175 from storage group cms.cms2.cpio_odc to library CD-LTO4G1, volume family cms.cms2.cpio_odc 00:00:03 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm work on vol=VOS175 mover=LTO4_98.mover requester:root 00:00:03 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:03 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM IDLE:sending {'vc': {'storage_group': 'cms', 'library': 'CD-LTO4G1', 'file_family': 'cms2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms2.cpio_odc', 'file_family_width': 7}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/PATLayer1_RECO_Output_QCDDiJet_Pt1400to1800_7.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 416687072L, 'mover': 'LTO4_98.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/PATLayer1_RECO_Output_QCDDiJet_Pt1400to1800_7.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257051602, 'rmajor': 0, 'machine': ('Linux', 'cmsstor132.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16893, 273340320, 32L, 1, 12124, 5063, 512, 1257050578, 1257050578, 1257036570), 'mode_octal': '0100036', 'mode': 33188, 'gname': 'root', 'size_bytes': 416687072L, 'fullname': '/storage/data3/write-pool-3/data/0010000000000000094DDAA0', 'inode': 273537696L, 'minor': 32}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/0010000000000000094DDAA0', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/.(access)(0010000000000000094DDAA0)', 'fc': {'size': 416687072L, 'external_label': 'VOS175', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data3/write-pool-3/data/0010000000000000094DDAA0', 'at_the_top': 2, 'local_inode': 3877, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.189.103', 42145), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051603.039181, 'encp_start_time': 1257051601.739692, 'job_queued': 1257051602.5255649, 't0': 1257051601, 'in_queue': 0.51357007026672363}, 'user_level': 'admin', 'unique_id': 'cmsstor132.fnal.gov-1257051602-490-0'} to mover 00:00:03 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM can not remove from sg LTO4_98.mover VOS175 00:00:03 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM SG: {'exp-db': [('LTO4_102.mover', 'VON089')], 'cms': [('LTO4_42.mover', 'VOS303'), ('LTO4_109.mover', 'VOQ826'), ('LTO4_40.mover', 'VOQ839'), ('LTO4_44.mover', 'VOQ819'), ('LTO4_43.mover', 'VOQ856'), ('LTO4_95.mover', 'VOQ834'), ('LTO4_93.mover', 'VOQ827'), ('LTO4_51.mover', 'VOQ870'), ('LTO4_108.mover', 'VOQ873'), ('LTO4_97.mover', 'VOQ844'), ('LTO4_41.mover', 'VOQ859'), ('LTO4_47.mover', 'VOU664'), ('LTO4_106.mover', 'VOQ825'), ('LTO4_110.mover', 'VOQ862'), ('LTO4_94.mover', 'VOQ828'), ('LTO4_104.mover', 'VOU867'), ('LTO4_96.mover', 'VOQ922'), ('LTO4_107.mover', 'VOU517')], 'minos': [('LTO4_101.mover', 'VON584')]} 00:00:03 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM can not remove from vf LTO4_98.mover VOS175 00:00:03 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM VF: {'minos.reco_far_dogwood1_bntp.cpio_odc': [('LTO4_101.mover', 'VON584')], 'cms.cms2.cpio_odc': [('LTO4_42.mover', 'VOS303'), ('LTO4_47.mover', 'VOU664'), ('LTO4_104.mover', 'VOU867'), ('LTO4_107.mover', 'VOU517')], 'cms.BACKFILL2.cpio_odc': [('LTO4_109.mover', 'VOQ826'), ('LTO4_40.mover', 'VOQ839'), ('LTO4_44.mover', 'VOQ819'), ('LTO4_43.mover', 'VOQ856'), ('LTO4_95.mover', 'VOQ834'), ('LTO4_93.mover', 'VOQ827'), ('LTO4_51.mover', 'VOQ870'), ('LTO4_108.mover', 'VOQ873'), ('LTO4_97.mover', 'VOQ844'), ('LTO4_41.mover', 'VOQ859'), ('LTO4_106.mover', 'VOQ825'), ('LTO4_110.mover', 'VOQ862'), ('LTO4_94.mover', 'VOQ828'), ('LTO4_96.mover', 'VOQ922')], 'exp-db.archive-d0-online.cpio_odc': [('LTO4_102.mover', 'VON089')]} 00:00:03 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV WRITE_TO_HSM Thread MainThread 00:00:03 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:03 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV SENDING {'vc': {'storage_group': 'cms', 'library': 'CD-LTO4G1', 'file_family': 'cms2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms2.cpio_odc', 'file_family_width': 7}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/PATLayer1_RECO_Output_QCDDiJet_Pt1400to1800_7.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 416687072L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7859, 'mc_device': '0,7,1,1', 'compression': 0, 'serial_num': '9310069991', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr198a', 'max_time_in_state': 1200, 'device': 'stkenmvr198a:/dev/rmt/tps0d0n', 'name': 'LTO4_98.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_98.stat', 'mover_address': ('131.225.13.193', 7859), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.193', 59812), 'data_ip': 'stkenmvr198a', 'logname': 'LTO4_98MV', 'hostip': '131.225.13.193', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/PATLayer1_RECO_Output_QCDDiJet_Pt1400to1800_7.root', 'uid': 0, 'uname': 'root', 'mtime': 1257051602, 'rmajor': 0, 'machine': ('Linux', 'cmsstor132.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 273340320, 32L, 1, 12124, 5063, 512, 1257050578, 1257050578, 1257036570), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 416687072L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data3/write-pool-3/data/0010000000000000094DDAA0', 'inode': 273537696L, 'minor': 32}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/0010000000000000094DDAA0', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/.(access)(0010000000000000094DDAA0)', 'fc': {'size': 416687072L, 'external_label': 'VOS175', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data3/write-pool-3/data/0010000000000000094DDAA0', 'at_the_top': 2, 'local_inode': 3877, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.189.103', 42145), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051603.039181, 'encp_start_time': 1257051601.739692, 'job_queued': 1257051602.5255649, 't0': 1257051601, 'in_queue': 0.51357007026672363}, 'user_level': 'admin', 'unique_id': 'cmsstor132.fnal.gov-1257051602-490-0'} Thread client_connect_thread 00:00:03 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:03 cmsstor132.fnal.gov 000490 root I ENCP Control socket ('131.225.189.103', 42145) is connected to ('131.225.13.193', 59813) for cmsstor132.fnal.gov-1257051602-490-0. elapsed=1.320sec 00:00:03 cmsstor132.fnal.gov 000490 root I ENCP Received callback ticket from mover LTO4_98.mover for transfer cmsstor132.fnal.gov-1257051602-490-0. elapsed=1.330sec 00:00:03 cmsstor132.fnal.gov 000490 root I ENCP Atempting to connect data socket to mover at ('131.225.13.193', 59812). 00:00:03 cmsstor132.fnal.gov 000490 root I ENCP Connecting data socket to mover (('131.225.13.193', 59812)) with interface 131.225.189.103. elapsed=1.330sec 00:00:03 cmsstor132.fnal.gov 000490 root I ENCP Data socket ('131.225.189.103', 53562) is connected to ('131.225.13.193', 59812). elapsed=1.330sec 00:00:03 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:03 cmsstor132.fnal.gov 000490 root I ENCP Opening local file. 00:00:03 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV MSG_TYPE=MC_LOAD_REQ mounting VOS175 ULTRIUM-TD4 00:00:03 Thread media_thread 00:00:03 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOS175 0,7,1,1 00:00:03 stkenmvr198a.fnal.gov 015662 root E LTO4_98MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:03 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:03 cmsstor132.fnal.gov 000490 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:03 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:03 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:03 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:04 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:04 stkensrv4n.fnal.gov 019392 enstore I SL8500G1MC mount VOQ844 0,7,1,13 => 0,Mount: VOQ844 mounted on 0, 7, 1,13 00:00:04 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC FINISHED mount VOQ844 0,7,1,13 returned ('ok', 0, 'mount VOQ844 0,7,1,13 => 0,Mount: VOQ844 mounted on 0, 7, 1,13', '', '') 00:00:04 last message repeated 1 times 00:00:04 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_mounts = '93', last_access = '2009-11-01 00:00:04' where label = 'VOQ844'; 00:00:04 stkenmvr197a.fnal.gov 025501 root I LTO4_97MV MSG_TYPE=MC_LOAD_DONE mounted VOQ844 ULTRIUM-TD4 00:00:04 Thread media_thread 00:00:05 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:05 last message repeated 1 times 00:00:05 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:05 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:06 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:07 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:07 last message repeated 1 times 00:00:07 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:07 stkensrv4n.fnal.gov 019432 enstore I SL8500G1MC query vol VOQ826 => 0,VOQ826 home 0, 7,28, 1, 0 LTO-800G 00:00:07 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:08 stkensrv0n.fnal.gov 023312 enstore I INFSRV start listing all volumes (3) 00:00:08 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:08 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:08 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC PROCESSING dismount VOS303 0,7,1,12 00:00:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:09 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:00:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:09 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:09 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VON584 0,6,1,7 00:00:09 stkenmvr201a.fnal.gov 018563 root E LTO4_101MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:09 stkenmvr201a.fnal.gov 018563 root I LTO4_101MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:09 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:10 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOQ834 0,7,1,14 00:00:10 stkenmvr195a.fnal.gov 021479 root E LTO4_95MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:10 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:10 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC FINISHED listDrives returned ('ok', 0, None, '', '') 00:00:10 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ825 0,6,1,8 00:00:10 stkenmvr206a.fnal.gov 026766 root E LTO4_106MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:10 stkenmvr206a.fnal.gov 026766 root I LTO4_106MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:10 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:10 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ859 0,7,1,3 00:00:10 stkenmvr141a.fnal.gov 013963 root E LTO4_41MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:10 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:10 last message repeated 1 times 00:00:10 cmsstor215.fnal.gov 000746 root I ENCP File /storage/data3/write-pool-3/data/000700000000000061806230 transfered for cmsstor215.fnal.gov-1257051576-746-0 in 33.597509861 seconds. 00:00:10 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV written bytes 3960039936/3960039936, blocks 30213 header 285 trailer 241 Thread tape_thread 00:00:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:11 last message repeated 1 times 00:00:11 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:11 stkensrv4n.fnal.gov 019877 enstore I SL8500G1MC query drive 0,6,1,0 => 0,0,6,1,0 online in use VOU664 IBM-LTO4 00:00:11 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV filemarks written. Tape VOQ828 absolute location in blocks 3129674 Thread tape_thread 00:00:11 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV write_tape timing:pull 0.322819232941 write 7.97989058495 crc_check 18.1713438034 freespace 0.348395824432 block_write 26.8224494457 idle 0.598745584488 Thread tape_thread 00:00:11 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV drive stats after write. Tape VOQ828 position 136 block 0 block_size 0 bloc_loc 3129674 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 3960039936 block_write_tot 26.8224494457 tape_rate 147639012.015 Thread tape_thread 00:00:11 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV remaining bytes info in DB 397292544000 estimated 393332504064 reported from drive 393611264000 position 137 Thread tape_thread 00:00:11 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV new bitfile request {'uid': 0, 'external_label': 'VOQ828', 'gid': 0, 'location_cookie': '0000_000000000_0000136', 'complete_crc': 2157243369L, 'sanity_cookie': (65536L, 1235365070L), 'size': 3960039410L} Thread tape_thread 00:00:11 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV set remaining: VOQ828 393611264000 0000_000000000_0000137 Thread tape_thread 00:00:11 last message repeated 1 times 00:00:11 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-01 00:00:11', eod_cookie = '0000_000000000_0000137', non_del_files = '136', remaining_bytes = 393611264000 where label = 'VOQ828'; 00:00:11 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV transfer complete volume=VOQ828 location=137 Thread tape_thread 00:00:11 last message repeated 1 times 00:00:11 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '136', modification_time = '2009-11-01 00:00:11' where label = 'VOQ828'; 00:00:11 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1255358447.0, 'library': 'CD-LTO4G1', 'si_time': [1241450288.0, 1241507765.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOQ828', 'wrapper': 'cpio_odc', 'remaining_bytes': 397292544000L, 'sum_mounts': 103, 'sum_rd_access': 0, 'file_family_width': 15, 'last_access': 1257051554.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000136', 'capacity_bytes': 858993459200L, 'non_del_files': 135, 'sum_wr_err': 0, 'sum_wr_access': 135, 'file_family': 'BACKFILL2', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.BACKFILL2.cpio_odc', 'modification_time': 1257051554.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1256991855.0}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Jet/RAW/v10_31_pre20replaythingy_v15/000/000/005/987A3F3A-EBC5-DE11-B974-0019B9F581C9.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 3960039410L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7855, 'mc_device': '0,7,1,7', 'compression': 0, 'serial_num': '9310800990', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr194a', 'max_time_in_state': 1200, 'device': 'stkenmvr194a:/dev/rmt/tps0d0n', 'name': 'LTO4_94.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_94.stat', 'mover_address': ('131.225.13.189', 7855), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.189', 54721), 'data_ip': 'stkenmvr194a', 'logname': 'LTO4_94MV', 'hostip': '131.225.13.189', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Jet/RAW/v10_31_pre20replaythingy_v15/000/000/005/987A3F3A-EBC5-DE11-B974-0019B9F581C9.root', 'uid': 0, 'uname': 'root', 'mtime': 1257051576, 'rmajor': 0, 'machine': ('Linux', 'cmsstor215.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 125634776, 21L, 1, 9811, 5063, 512, 1257051083, 1257051083, 1257038589), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 3960039410L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data3/write-pool-3/data/000700000000000061806230', 'inode': 125854256L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/000700000000000061806230', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Jet/RAW/v10_31_pre20replaythingy_v15/000/000/005/.(access)(000700000000000061806230)', 'fc': {'uid': 0, 'external_label': 'VOQ828', 'gid': 0, 'location_cookie': '0000_000000000_0000136', 'complete_crc': 2157243369L, 'bfid': 'CDMS125705161100000', 'sanity_cookie': (65536L, 1235365070L), 'size': 3960039410L}, 'infilepath': '/storage/data3/write-pool-3/data/000700000000000061806230', 'at_the_top': 1, 'local_inode': 8279, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.191.87', 59765), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257051575.538007, 'seek_time': 0.02968907356262207, 'transfer_time': 34.848037004470825, 't0': 1257051575, 'mount_time(trunc. 4363) Thread tape_thread 00:00:11 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV Done is sent Thread tape_thread 00:00:11 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV maybe_clean Thread tape_thread 00:00:11 cmsstor215.fnal.gov 000746 root I ENCP Received final dialog for cmsstor215.fnal.gov-1257051576-746-0. elapsed=36.420sec 00:00:11 cmsstor215.fnal.gov 000746 root I ENCP leaving transfer_file(). 00:00:11 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV returned from maybe_clean Thread tape_thread 00:00:11 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:12 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:12 cmsstor215.fnal.gov 000746 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:12 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Jet/RAW/v10_31_pre20replaythingy_v15/000/000/005/987A3F3A-EBC5-DE11-B974-0019B9F581C9.root', drive = 'stkenmvr194a:/dev/rmt/tps0d0n:9310800990', pnfs_id = '000700000000000061806230' where bfid = 'CDMS125705161100000'; 00:00:12 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:12 cmsstor215.fnal.gov 000746 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /storage/data3/write-pool-3/data/000700000000000061806230 -> /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Jet/RAW/v10_31_pre20replaythingy_v15/000/000/005/987A3F3A-EBC5-DE11-B974-0019B9F581C9.root: 3960039410 bytes copied to VOQ828 at 108 MB/S (96.7 MB/S network) (130 MB/S drive) (112 MB/S disk) mover=LTO4_94.mover drive_id=ULTRIUM-TD4 drive_sn=9310800990 drive_vendor=IBM elapsed=36.887 {'storage_group': 'cms', 'encp_crc': 2157243369L, 'disk_rate': 112.42442666064805, 'drive_rate': 130.10689546787901, 'driver': 'FTTDriver', 'transfer_rate': 108.32055990887268, 'mover_interface': 'stkenmvr194a', 'overall_rate': 106.6827033060634, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 108.32055990887268, 'encp_ip': '131.225.191.87', 'unique_id': 'cmsstor215.fnal.gov-1257051576-746-0'} 00:00:12 cmsstor215.fnal.gov 000746 root I ENCP File /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Jet/RAW/v10_31_pre20replaythingy_v15/000/000/005/987A3F3A-EBC5-DE11-B974-0019B9F581C9.root copied successfully. 00:00:12 cmsstor215.fnal.gov 000746 root I ENCP Completed transferring 3960039410 bytes in 1 files in 36.8879189491 sec. Overall rate = 107 MB/sec. Transfer rate = 108 MB/sec. Network rate = 96.7 MB/sec. Drive rate = 130 MB/sec. Disk rate = 112 MB/sec. Exit status = 0. 00:00:13 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:13 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOS175 0,7,1,1 00:00:13 stkenmvr198a.fnal.gov 015662 root E LTO4_98MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:13 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:13 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:13 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:13 cmsstor154.fnal.gov 005860 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms2 FF Wrapper: cpio_odc FF Width: 7 Current working directory: cmsstor154.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --age-time 30 --delpri 1 --priority 25 --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/PATLayer1_RECO_Output_QCDDiJet_Pt1800to2200_20.root --put-cache 0010000000000000094DD9B0 /storage/data1/write-pool-1/data/0010000000000000094DD9B0 00:00:13 cmsstor154.fnal.gov 005860 root I ENCP Creating zero length output files. 00:00:13 cmsstor154.fnal.gov 005860 root I ENCP Sending /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/PATLayer1_RECO_Output_QCDDiJet_Pt1800to2200_20.root write request to LM: unique_id: cmsstor154.fnal.gov-1257051613-5860-0 inputfile: /storage/data1/write-pool-1/data/0010000000000000094DD9B0 outputfile: /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/.(access)(0010000000000000094DD9B0) 00:00:13 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:13 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data1/write-pool-1/data/0010000000000000094DD9B0 (cmsstor154.fnal.gov-1257051613-5860-0) -> /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/PATLayer1_RECO_Output_QCDDiJet_Pt1800to2200_20.root : library=CD-LTO4G1 family=cms2 requester:root volume_family:cms.cms2.cpio_odc 00:00:13 cmsstor154.fnal.gov 005860 root I ENCP Listening for control socket at: ('131.225.189.125', 57100) 00:00:13 cmsstor154.fnal.gov 005860 root I ENCP Waiting for mover to connect control socket at address: ('131.225.189.125', 57100) elapsed=0.740sec 00:00:13 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:14 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Assigning volume VOS188 from storage group cms.cms2.cpio_odc to library CD-LTO4G1, volume family cms.cms2.cpio_odc 00:00:14 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm work on vol=VOS188 mover=LTO4_46.mover requester:root 00:00:14 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:14 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM IDLE:sending {'vc': {'storage_group': 'cms', 'library': 'CD-LTO4G1', 'file_family': 'cms2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms2.cpio_odc', 'file_family_width': 7}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/PATLayer1_RECO_Output_QCDDiJet_Pt1800to2200_20.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 433741153L, 'mover': 'LTO4_46.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/PATLayer1_RECO_Output_QCDDiJet_Pt1800to2200_20.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257051613, 'rmajor': 0, 'machine': ('Linux', 'cmsstor154.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16893, 273340400, 22L, 1, 12124, 5063, 512, 1257050553, 1257050553, 1257036571), 'mode_octal': '0100036', 'mode': 33188, 'gname': 'root', 'size_bytes': 433741153L, 'fullname': '/storage/data1/write-pool-1/data/0010000000000000094DD9B0', 'inode': 273537456L, 'minor': 22}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/0010000000000000094DD9B0', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/.(access)(0010000000000000094DD9B0)', 'fc': {'size': 433741153L, 'external_label': 'VOS188', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data1/write-pool-1/data/0010000000000000094DD9B0', 'at_the_top': 2, 'local_inode': 859, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.189.125', 57100), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051614.0757589, 'encp_start_time': 1257051613.139801, 'job_queued': 1257051613.8600559, 't0': 1257051613, 'in_queue': 0.21565914154052734}, 'user_level': 'admin', 'unique_id': 'cmsstor154.fnal.gov-1257051613-5860-0'} to mover 00:00:14 stkenmvr146a.fnal.gov 005990 root I LTO4_46MV WRITE_TO_HSM Thread MainThread 00:00:14 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM can not remove from sg LTO4_46.mover VOS188 00:00:14 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM SG: {'exp-db': [('LTO4_102.mover', 'VON089')], 'cms': [('LTO4_42.mover', 'VOS303'), ('LTO4_109.mover', 'VOQ826'), ('LTO4_44.mover', 'VOQ819'), ('LTO4_95.mover', 'VOQ834'), ('LTO4_93.mover', 'VOQ827'), ('LTO4_51.mover', 'VOQ870'), ('LTO4_108.mover', 'VOQ873'), ('LTO4_97.mover', 'VOQ844'), ('LTO4_41.mover', 'VOQ859'), ('LTO4_47.mover', 'VOU664'), ('LTO4_106.mover', 'VOQ825'), ('LTO4_104.mover', 'VOU867'), ('LTO4_98.mover', 'VOS175'), ('LTO4_110.mover', 'VOQ862'), ('LTO4_40.mover', 'VOQ839'), ('LTO4_94.mover', 'VOQ828'), ('LTO4_107.mover', 'VOU517'), ('LTO4_96.mover', 'VOQ922')], 'minos': [('LTO4_101.mover', 'VON584')]} 00:00:14 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM can not remove from vf LTO4_46.mover VOS188 00:00:14 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM VF: {'minos.reco_far_dogwood1_bntp.cpio_odc': [('LTO4_101.mover', 'VON584')], 'cms.cms2.cpio_odc': [('LTO4_42.mover', 'VOS303'), ('LTO4_47.mover', 'VOU664'), ('LTO4_104.mover', 'VOU867'), ('LTO4_98.mover', 'VOS175'), ('LTO4_107.mover', 'VOU517')], 'cms.BACKFILL2.cpio_odc': [('LTO4_109.mover', 'VOQ826'), ('LTO4_44.mover', 'VOQ819'), ('LTO4_95.mover', 'VOQ834'), ('LTO4_93.mover', 'VOQ827'), ('LTO4_51.mover', 'VOQ870'), ('LTO4_108.mover', 'VOQ873'), ('LTO4_97.mover', 'VOQ844'), ('LTO4_41.mover', 'VOQ859'), ('LTO4_106.mover', 'VOQ825'), ('LTO4_110.mover', 'VOQ862'), ('LTO4_40.mover', 'VOQ839'), ('LTO4_94.mover', 'VOQ828'), ('LTO4_96.mover', 'VOQ922')], 'exp-db.archive-d0-online.cpio_odc': [('LTO4_102.mover', 'VON089')]} 00:00:14 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:14 stkenmvr146a.fnal.gov 005990 root I LTO4_46MV SENDING {'vc': {'storage_group': 'cms', 'library': 'CD-LTO4G1', 'file_family': 'cms2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms2.cpio_odc', 'file_family_width': 7}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/PATLayer1_RECO_Output_QCDDiJet_Pt1800to2200_20.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 433741153L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7846, 'mc_device': '0,6,1,12', 'compression': 0, 'serial_num': '9311032672', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr146a', 'max_time_in_state': 1200, 'device': 'stkenmvr146a:/dev/rmt/tps0d0n', 'name': 'LTO4_46.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_46.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.174', 7846), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.174', 53256), 'data_ip': 'stkenmvr146a', 'logname': 'LTO4_46MV', 'hostip': '131.225.13.174', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/PATLayer1_RECO_Output_QCDDiJet_Pt1800to2200_20.root', 'uid': 0, 'uname': 'root', 'mtime': 1257051613, 'rmajor': 0, 'machine': ('Linux', 'cmsstor154.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 273340400, 22L, 1, 12124, 5063, 512, 1257050553, 1257050553, 1257036571), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 433741153L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data1/write-pool-1/data/0010000000000000094DD9B0', 'inode': 273537456L, 'minor': 22}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/0010000000000000094DD9B0', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/.(access)(0010000000000000094DD9B0)', 'fc': {'size': 433741153L, 'external_label': 'VOS188', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data1/write-pool-1/data/0010000000000000094DD9B0', 'at_the_top': 2, 'local_inode': 859, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.189.125', 57100), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051614.0757589, 'encp_start_time': 1257051613.139801, 'job_queued': 1257051613.8600559, 't0': 1257051613, 'in_queue': 0.21565914154052734}, 'user_level': 'admin', 'unique_id': 'cmsstor154.fnal.gov-1257051613-5860-0'} Thread client_connect_thread 00:00:14 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:14 cmsstor154.fnal.gov 005860 root I ENCP Control socket ('131.225.189.125', 57100) is connected to ('131.225.13.174', 53257) for cmsstor154.fnal.gov-1257051613-5860-0. elapsed=0.980sec 00:00:14 cmsstor154.fnal.gov 005860 root I ENCP Received callback ticket from mover LTO4_46.mover for transfer cmsstor154.fnal.gov-1257051613-5860-0. elapsed=0.980sec 00:00:14 cmsstor154.fnal.gov 005860 root I ENCP Atempting to connect data socket to mover at ('131.225.13.174', 53256). 00:00:14 cmsstor154.fnal.gov 005860 root I ENCP Connecting data socket to mover (('131.225.13.174', 53256)) with interface 131.225.189.125. elapsed=0.980sec 00:00:14 cmsstor154.fnal.gov 005860 root I ENCP Data socket ('131.225.189.125', 53561) is connected to ('131.225.13.174', 53256). elapsed=0.980sec 00:00:14 cmsstor154.fnal.gov 005860 root I ENCP Opening local file. 00:00:14 stkenmvr146a.fnal.gov 005990 root I LTO4_46MV MSG_TYPE=MC_LOAD_REQ mounting VOS188 ULTRIUM-TD4 00:00:14 Thread media_thread 00:00:14 cmsstor154.fnal.gov 005860 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:14 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOS188 0,6,1,12 00:00:14 stkenmvr146a.fnal.gov 005990 root E LTO4_46MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:14 stkenmvr146a.fnal.gov 005990 root I LTO4_46MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:14 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:14 last message repeated 1 times 00:00:14 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:15 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:15 stkenmvr196a.fnal.gov 027637 root I LTO4_96MV Updating stats Thread media_thread 00:00:15 stkenmvr196a.fnal.gov 027637 root I LTO4_96MV writing a tape mark before dismount Thread media_thread 00:00:15 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:15 cmsstor63.fnal.gov 023538 root I ENCP File /storage/data3/write-pool-3/data/000700000000000061806710 transfered for cmsstor63.fnal.gov-1257051591-23538-0 in 22.7622990608 seconds. 00:00:15 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV written bytes 2663687168/2663687168, blocks 20323 header 288 trailer 555 Thread tape_thread 00:00:15 cmsstor224.fnal.gov 013968 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: BACKFILL2 FF Wrapper: cpio_odc FF Width: 15 Current working directory: cmsstor224.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --age-time 30 --delpri 1 --priority 25 --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/346F128B-0EC6-DE11-AA76-0019B9F72BFF.root --put-cache 000700000000000061806490 /storage/data1/write-pool-1/data/000700000000000061806490 00:00:15 stkensrv4n.fnal.gov 019432 enstore I SL8500G1MC query drive 0,5,1,7 => 0,0,5,1,7 online available IBM-LTO4 00:00:15 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:15 cmsstor224.fnal.gov 013968 root I ENCP Creating zero length output files. 00:00:15 cmsstor224.fnal.gov 013968 root I ENCP Sending /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/346F128B-0EC6-DE11-AA76-0019B9F72BFF.root write request to LM: unique_id: cmsstor224.fnal.gov-1257051615-13968-0 inputfile: /storage/data1/write-pool-1/data/000700000000000061806490 outputfile: /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(000700000000000061806490) 00:00:15 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:15 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data1/write-pool-1/data/000700000000000061806490 (cmsstor224.fnal.gov-1257051615-13968-0) -> /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/346F128B-0EC6-DE11-AA76-0019B9F72BFF.root : library=CD-LTO4G1 family=BACKFILL2 requester:root volume_family:cms.BACKFILL2.cpio_odc 00:00:15 cmsstor224.fnal.gov 013968 root I ENCP Listening for control socket at: ('131.225.191.96', 50854) 00:00:15 cmsstor224.fnal.gov 013968 root I ENCP Waiting for mover to connect control socket at address: ('131.225.191.96', 50854) elapsed=0.800sec 00:00:15 stkensrv4n.fnal.gov 005852 enstore I STKMC FINISHED listDrives returned ('ok', 0, None, '', '') 00:00:15 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:15 stkensrv4n.fnal.gov 019886 enstore I SL8500G1MC query drive 0,7,1,12 => 0,0,7,1,12 online in use VOS303 IBM-LTO4 00:00:16 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:16 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VOQ828 mover=LTO4_94.mover requester:root 00:00:16 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/346F128B-0EC6-DE11-AA76-0019B9F72BFF.root to mover LTO4_94.mover ('131.225.13.189', 7855) DEL_DISM None 00:00:16 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:16 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'cms', 'external_label': 'VOQ828', 'library': 'CD-LTO4G1', 'file_family': 'BACKFILL2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.BACKFILL2.cpio_odc', 'file_family_width': 15}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/346F128B-0EC6-DE11-AA76-0019B9F72BFF.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 3933110695L, 'mover': 'LTO4_94.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/346F128B-0EC6-DE11-AA76-0019B9F72BFF.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257051615, 'rmajor': 0, 'machine': ('Linux', 'cmsstor224.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16877, 125853680, 24L, 1, 9811, 5063, 512, 1257051251, 1257051251, 1257051006), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 3933110695L, 'fullname': '/storage/data1/write-pool-1/data/000700000000000061806490', 'inode': 125854864L, 'minor': 24}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/000700000000000061806490', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(000700000000000061806490)', 'fc': {'size': 3933110695L, 'external_label': 'VOQ828', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data1/write-pool-1/data/000700000000000061806490', 'at_the_top': 1, 'local_inode': 9571, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.191.96', 50854), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051616.0810161, 'encp_start_time': 1257051614.880209, 'job_queued': 1257051615.6564751, 't0': 1257051614, 'in_queue': 0.42449784278869629}, 'user_level': 'admin', 'unique_id': 'cmsstor224.fnal.gov-1257051615-13968-0'} to mover 00:00:16 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:16 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV WRITE_TO_HSM Thread MainThread 00:00:16 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV SENDING {'vc': {'storage_group': 'cms', 'external_label': 'VOQ828', 'library': 'CD-LTO4G1', 'file_family': 'BACKFILL2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.BACKFILL2.cpio_odc', 'file_family_width': 15}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/346F128B-0EC6-DE11-AA76-0019B9F72BFF.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 3933110695L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7855, 'mc_device': '0,7,1,7', 'compression': 0, 'serial_num': '9310800990', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr194a', 'max_time_in_state': 1200, 'device': 'stkenmvr194a:/dev/rmt/tps0d0n', 'name': 'LTO4_94.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_94.stat', 'mover_address': ('131.225.13.189', 7855), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.189', 54723), 'data_ip': 'stkenmvr194a', 'logname': 'LTO4_94MV', 'hostip': '131.225.13.189', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/346F128B-0EC6-DE11-AA76-0019B9F72BFF.root', 'uid': 0, 'uname': 'root', 'mtime': 1257051615, 'rmajor': 0, 'machine': ('Linux', 'cmsstor224.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 125853680, 24L, 1, 9811, 5063, 512, 1257051251, 1257051251, 1257051006), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 3933110695L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data1/write-pool-1/data/000700000000000061806490', 'inode': 125854864L, 'minor': 24}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/000700000000000061806490', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(000700000000000061806490)', 'fc': {'size': 3933110695L, 'external_label': 'VOQ828', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data1/write-pool-1/data/000700000000000061806490', 'at_the_top': 1, 'local_inode': 9571, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.191.96', 50854), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051616.0810161, 'encp_start_time': 1257051614.880209, 'job_queued': 1257051615.6564751, 't0': 1257051614, 'in_queue': 0.42449784278869629}, 'user_level': 'admin', 'unique_id': 'cmsstor224.fnal.gov-1257051615-13968-0'} Thread client_connect_thread 00:00:16 cmsstor224.fnal.gov 013968 root I ENCP Control socket ('131.225.191.96', 50854) is connected to ('131.225.13.189', 54724) for cmsstor224.fnal.gov-1257051615-13968-0. elapsed=1.230sec 00:00:16 cmsstor224.fnal.gov 013968 root I ENCP Received callback ticket from mover LTO4_94.mover for transfer cmsstor224.fnal.gov-1257051615-13968-0. elapsed=1.240sec 00:00:16 cmsstor224.fnal.gov 013968 root I ENCP Atempting to connect data socket to mover at ('131.225.13.189', 54723). 00:00:16 cmsstor224.fnal.gov 013968 root I ENCP Connecting data socket to mover (('131.225.13.189', 54723)) with interface 131.225.191.96. elapsed=1.240sec 00:00:16 cmsstor224.fnal.gov 013968 root I ENCP Data socket ('131.225.191.96', 34114) is connected to ('131.225.13.189', 54723). elapsed=1.240sec 00:00:16 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:16 cmsstor224.fnal.gov 013968 root I ENCP Opening local file. 00:00:16 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV tape position: target 137 current 137 0 Thread seek_thread 00:00:16 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV write_tape starting, bytes_to_write=3933111296 Thread tape_thread 00:00:16 cmsstor224.fnal.gov 013968 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:16 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV Write starting. Tape VOQ828 absolute location in blocks 3129674 Thread tape_thread 00:00:16 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:16 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV filemarks written. Tape VOQ862 absolute location in blocks 1288013 Thread tape_thread 00:00:16 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV write_tape timing:pull 0.211153268814 write 6.10074472427 crc_check 12.0996110439 freespace 0.223010778427 block_write 18.6345198154 idle 0.389652729034 Thread tape_thread 00:00:16 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV drive stats after write. Tape VOQ862 position 56 block 0 block_size 0 bloc_loc 1288013 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 2663687168 block_write_tot 18.6345198154 tape_rate 142943697.738 Thread tape_thread 00:00:16 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV remaining bytes info in DB 624556032000 estimated 621892344832 reported from drive 622087168000 position 57 Thread tape_thread 00:00:16 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV new bitfile request {'uid': 0, 'external_label': 'VOQ862', 'gid': 0, 'location_cookie': '0000_000000000_0000056', 'complete_crc': 4092551843L, 'sanity_cookie': (65536L, 3902465857L), 'size': 2663686325L} Thread tape_thread 00:00:16 cmsstor79.fnal.gov 012785 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: BACKFILL2 FF Wrapper: cpio_odc FF Width: 15 Current working directory: cmsstor79.fnal.gov:/root Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --age-time 30 --delpri 1 --priority 25 --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/CE53548A-0DC6-DE11-A322-001D09F2437B.root --put-cache 000700000000000061806770 /storage/data3/write-pool-3/data/000700000000000061806770 00:00:16 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV set remaining: VOQ862 622087168000 0000_000000000_0000057 Thread tape_thread 00:00:16 last message repeated 1 times 00:00:16 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-01 00:00:16', eod_cookie = '0000_000000000_0000057', non_del_files = '56', remaining_bytes = 622087168000 where label = 'VOQ862'; 00:00:16 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV transfer complete volume=VOQ862 location=57 Thread tape_thread 00:00:16 last message repeated 1 times 00:00:16 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '56', modification_time = '2009-11-01 00:00:16' where label = 'VOQ862'; 00:00:16 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1255358534.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 1242983135.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOQ862', 'wrapper': 'cpio_odc', 'remaining_bytes': 624556032000L, 'sum_mounts': 294, 'sum_rd_access': 0, 'file_family_width': 15, 'last_access': 1257051580.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000056', 'capacity_bytes': 858993459200L, 'non_del_files': 55, 'sum_wr_err': 0, 'sum_wr_access': 55, 'file_family': 'BACKFILL2', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.BACKFILL2.cpio_odc', 'modification_time': 1257051580.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257039541.0}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/DA6A1760-F7C5-DE11-AE81-000423D992A4.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2663686325L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7870, 'mc_device': '0,5,1,14', 'compression': 0, 'serial_num': '1310049971', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr210a', 'max_time_in_state': 1200, 'device': 'stkenmvr210a:/dev/rmt/tps0d0n', 'name': 'LTO4_110.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_110.stat', 'mover_address': ('131.225.13.206', 7870), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.206', 53936), 'data_ip': 'stkenmvr210a', 'logname': 'LTO4_110MV', 'hostip': '131.225.13.206', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/DA6A1760-F7C5-DE11-AE81-000423D992A4.root', 'uid': 0, 'uname': 'root', 'mtime': 1257051591, 'rmajor': 0, 'machine': ('Linux', 'cmsstor63.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 125854968, 23L, 1, 9811, 5063, 512, 1257051362, 1257051362, 1257051186), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 2663686325L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data3/write-pool-3/data/000700000000000061806710', 'inode': 125855504L, 'minor': 23}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/000700000000000061806710', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/.(access)(000700000000000061806710)', 'fc': {'uid': 0, 'external_label': 'VOQ862', 'gid': 0, 'location_cookie': '0000_000000000_0000056', 'complete_crc': 4092551843L, 'bfid': 'CDMS125705161600000', 'sanity_cookie': (65536L, 3902465857L), 'size': 2663686325L}, 'infilepath': '/storage/data3/write-pool-3/data/000700000000000061806710', 'at_the_top': 1, 'local_inode': 1073743012, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.204.239', 44626), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257051590.937578, 'seek_time': 0.029681921005249023, 'transfer_time': 24.21983003616333, 't0': 1257051590, 'mo(trunc. 4374) Thread tape_thread 00:00:16 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV Done is sent Thread tape_thread 00:00:16 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV maybe_clean Thread tape_thread 00:00:16 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV returned from maybe_clean Thread tape_thread 00:00:16 cmsstor63.fnal.gov 023538 root I ENCP Received final dialog for cmsstor63.fnal.gov-1257051591-23538-0. elapsed=25.830sec 00:00:16 cmsstor63.fnal.gov 023538 root I ENCP leaving transfer_file(). 00:00:16 cmsstor79.fnal.gov 012785 root I ENCP Creating zero length output files. 00:00:16 cmsstor79.fnal.gov 012785 root I ENCP Sending /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/CE53548A-0DC6-DE11-A322-001D09F2437B.root write request to LM: unique_id: cmsstor79.fnal.gov-1257051616-12785-0 inputfile: /storage/data3/write-pool-3/data/000700000000000061806770 outputfile: /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(000700000000000061806770) 00:00:16 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:16 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data3/write-pool-3/data/000700000000000061806770 (cmsstor79.fnal.gov-1257051616-12785-0) -> /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/CE53548A-0DC6-DE11-A322-001D09F2437B.root : library=CD-LTO4G1 family=BACKFILL2 requester:root volume_family:cms.BACKFILL2.cpio_odc 00:00:16 cmsstor63.fnal.gov 023538 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:16 cmsstor79.fnal.gov 012785 root I ENCP Listening for control socket at: ('131.225.205.201', 56272) 00:00:16 cmsstor79.fnal.gov 012785 root I ENCP Waiting for mover to connect control socket at address: ('131.225.205.201', 56272) elapsed=1.040sec 00:00:17 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/DA6A1760-F7C5-DE11-AE81-000423D992A4.root', drive = 'stkenmvr210a:/dev/rmt/tps0d0n:1310049971', pnfs_id = '000700000000000061806710' where bfid = 'CDMS125705161600000'; 00:00:17 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:17 cmsstor63.fnal.gov 023538 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /storage/data3/write-pool-3/data/000700000000000061806710 -> /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/DA6A1760-F7C5-DE11-AE81-000423D992A4.root: 2663686325 bytes copied to VOQ862 at 105 MB/S (84.3 MB/S network) (123 MB/S drive) (112 MB/S disk) mover=LTO4_110.mover drive_id=ULTRIUM-TD4 drive_sn=1310049971 drive_vendor=IBM elapsed=26.258 {'storage_group': 'cms', 'encp_crc': 4092551843L, 'disk_rate': 111.62842036791825, 'drive_rate': 123.42322396164126, 'driver': 'FTTDriver', 'transfer_rate': 104.68393861615471, 'mover_interface': 'stkenmvr210a', 'overall_rate': 102.65172359687828, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 104.68393861615471, 'encp_ip': '131.225.204.239', 'unique_id': 'cmsstor63.fnal.gov-1257051591-23538-0'} 00:00:17 cmsstor63.fnal.gov 023538 root I ENCP File /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/DA6A1760-F7C5-DE11-AE81-000423D992A4.root copied successfully. 00:00:17 cmsstor63.fnal.gov 023538 root I ENCP Completed transferring 2663686325 bytes in 1 files in 26.2608411312 sec. Overall rate = 103 MB/sec. Transfer rate = 105 MB/sec. Network rate = 84.3 MB/sec. Drive rate = 123 MB/sec. Disk rate = 112 MB/sec. Exit status = 0. 00:00:17 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:17 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VOQ862 mover=LTO4_110.mover requester:root 00:00:17 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/CE53548A-0DC6-DE11-A322-001D09F2437B.root to mover LTO4_110.mover ('131.225.13.206', 7870) DEL_DISM None 00:00:17 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:17 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'cms', 'external_label': 'VOQ862', 'library': 'CD-LTO4G1', 'file_family': 'BACKFILL2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.BACKFILL2.cpio_odc', 'file_family_width': 15}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/CE53548A-0DC6-DE11-A322-001D09F2437B.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2648935928L, 'mover': 'LTO4_110.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/CE53548A-0DC6-DE11-A322-001D09F2437B.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257051616, 'rmajor': 0, 'machine': ('Linux', 'cmsstor79.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16877, 125853680, 27L, 1, 9811, 5063, 512, 1257051251, 1257051251, 1257051006), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 2648935928L, 'fullname': '/storage/data3/write-pool-3/data/000700000000000061806770', 'inode': 125855600L, 'minor': 27}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/000700000000000061806770', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(000700000000000061806770)', 'fc': {'size': 2648935928L, 'external_label': 'VOQ862', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data3/write-pool-3/data/000700000000000061806770', 'at_the_top': 1, 'local_inode': 1073742676, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.205.201', 56272), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051617.4523311, 'encp_start_time': 1257051615.852011, 'job_queued': 1257051616.8547571, 't0': 1257051615, 'in_queue': 0.59752988815307617}, 'user_level': 'admin', 'unique_id': 'cmsstor79.fnal.gov-1257051616-12785-0'} to mover 00:00:17 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:17 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV WRITE_TO_HSM Thread MainThread 00:00:17 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV SENDING {'vc': {'storage_group': 'cms', 'external_label': 'VOQ862', 'library': 'CD-LTO4G1', 'file_family': 'BACKFILL2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.BACKFILL2.cpio_odc', 'file_family_width': 15}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/CE53548A-0DC6-DE11-A322-001D09F2437B.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2648935928L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7870, 'mc_device': '0,5,1,14', 'compression': 0, 'serial_num': '1310049971', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr210a', 'max_time_in_state': 1200, 'device': 'stkenmvr210a:/dev/rmt/tps0d0n', 'name': 'LTO4_110.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_110.stat', 'mover_address': ('131.225.13.206', 7870), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.206', 53938), 'data_ip': 'stkenmvr210a', 'logname': 'LTO4_110MV', 'hostip': '131.225.13.206', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/CE53548A-0DC6-DE11-A322-001D09F2437B.root', 'uid': 0, 'uname': 'root', 'mtime': 1257051616, 'rmajor': 0, 'machine': ('Linux', 'cmsstor79.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 125853680, 27L, 1, 9811, 5063, 512, 1257051251, 1257051251, 1257051006), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 2648935928L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data3/write-pool-3/data/000700000000000061806770', 'inode': 125855600L, 'minor': 27}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/000700000000000061806770', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(000700000000000061806770)', 'fc': {'size': 2648935928L, 'external_label': 'VOQ862', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data3/write-pool-3/data/000700000000000061806770', 'at_the_top': 1, 'local_inode': 1073742676, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.205.201', 56272), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051617.4523311, 'encp_start_time': 1257051615.852011, 'job_queued': 1257051616.8547571, 't0': 1257051615, 'in_queue': 0.59752988815307617}, 'user_level': 'admin', 'unique_id': 'cmsstor79.fnal.gov-1257051616-12785-0'} Thread client_connect_thread 00:00:17 cmsstor79.fnal.gov 012785 root I ENCP Control socket ('131.225.205.201', 56272) is connected to ('131.225.13.206', 53939) for cmsstor79.fnal.gov-1257051616-12785-0. elapsed=1.640sec 00:00:17 cmsstor79.fnal.gov 012785 root I ENCP Received callback ticket from mover LTO4_110.mover for transfer cmsstor79.fnal.gov-1257051616-12785-0. elapsed=1.650sec 00:00:17 cmsstor79.fnal.gov 012785 root I ENCP Atempting to connect data socket to mover at ('131.225.13.206', 53938). 00:00:17 cmsstor79.fnal.gov 012785 root I ENCP Connecting data socket to mover (('131.225.13.206', 53938)) with interface 131.225.205.201. elapsed=1.650sec 00:00:17 cmsstor79.fnal.gov 012785 root I ENCP Data socket ('131.225.205.201', 37763) is connected to ('131.225.13.206', 53938). elapsed=1.650sec 00:00:17 cmsstor79.fnal.gov 012785 root I ENCP Opening local file. 00:00:17 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV tape position: target 57 current 57 0 Thread seek_thread 00:00:17 cmsstor79.fnal.gov 012785 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:17 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV write_tape starting, bytes_to_write=2648936448 Thread tape_thread 00:00:17 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:17 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV Write starting. Tape VOQ862 absolute location in blocks 1288013 Thread tape_thread 00:00:17 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:18 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:18 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:18 stkenmvr196a.fnal.gov 027637 root I LTO4_96MV Ejecting tape Thread media_thread 00:00:18 stkenmvr151a.fnal.gov 013228 root I LTO4_51MV skip_fm 37 Thread seek_thread 00:00:18 stkenmvr151a.fnal.gov 013228 root I LTO4_51MV write_tape starting, bytes_to_write=3929902080 Thread tape_thread 00:00:18 stkenmvr151a.fnal.gov 013228 root I LTO4_51MV Write starting. Tape VOQ870 absolute location in blocks 773184 Thread tape_thread 00:00:19 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:19 last message repeated 1 times 00:00:19 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:19 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VON584 0,6,1,7 00:00:19 stkenmvr201a.fnal.gov 018563 root E LTO4_101MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:19 stkenmvr201a.fnal.gov 018563 root I LTO4_101MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:19 cmsstor193.fnal.gov 017873 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Scientific Linux SL release 5.2 (Boron) Rocks release 5.1 (V.I) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: BACKFILL2 FF Wrapper: cpio_odc FF Width: 15 Current working directory: cmsstor193.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --age-time 30 --delpri 1 --priority 25 --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/3478C9BA-14C6-DE11-AB41-001D09F248F8.root --put-cache 0007000000000000618063C8 /storage/data2/write-pool-2/data/0007000000000000618063C8 00:00:19 stkensrv0n.fnal.gov 023312 enstore I INFSRV stop listing all volumes (3) 00:00:19 cmsstor193.fnal.gov 017873 root I ENCP Creating zero length output files. 00:00:19 cmsstor193.fnal.gov 017873 root I ENCP Sending /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/3478C9BA-14C6-DE11-AB41-001D09F248F8.root write request to LM: unique_id: cmsstor193.fnal.gov-1257051619-17873-0 inputfile: /storage/data2/write-pool-2/data/0007000000000000618063C8 outputfile: /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(0007000000000000618063C8) 00:00:19 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:19 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data2/write-pool-2/data/0007000000000000618063C8 (cmsstor193.fnal.gov-1257051619-17873-0) -> /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/3478C9BA-14C6-DE11-AB41-001D09F248F8.root : library=CD-LTO4G1 family=BACKFILL2 requester:root volume_family:cms.BACKFILL2.cpio_odc 00:00:19 cmsstor193.fnal.gov 017873 root I ENCP Listening for control socket at: ('131.225.207.18', 57223) 00:00:19 cmsstor193.fnal.gov 017873 root I ENCP Waiting for mover to connect control socket at address: ('131.225.207.18', 57223) elapsed=0.710sec 00:00:19 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:20 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Assigning volume VOQ856 from storage group cms.BACKFILL2.cpio_odc to library CD-LTO4G1, volume family cms.BACKFILL2.cpio_odc 00:00:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm work on vol=VOQ856 mover=LTO4_105.mover requester:root 00:00:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM IDLE:sending {'vc': {'storage_group': 'cms', 'library': 'CD-LTO4G1', 'file_family': 'BACKFILL2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.BACKFILL2.cpio_odc', 'file_family_width': 15}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/3478C9BA-14C6-DE11-AB41-001D09F248F8.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 3951272194L, 'mover': 'LTO4_105.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/3478C9BA-14C6-DE11-AB41-001D09F248F8.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257051619, 'rmajor': 0, 'machine': ('Linux', 'cmsstor193.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16877, 125853680, 28L, 1, 9811, 5063, 512, 1257051251, 1257051251, 1257051006), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 3951272194L, 'fullname': '/storage/data2/write-pool-2/data/0007000000000000618063C8', 'inode': 125854664L, 'minor': 28}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/0007000000000000618063C8', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(0007000000000000618063C8)', 'fc': {'size': 3951272194L, 'external_label': 'VOQ856', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data2/write-pool-2/data/0007000000000000618063C8', 'at_the_top': 2, 'local_inode': 8047, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.207.18', 57223), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051620.102401, 'encp_start_time': 1257051619.1430669, 'job_queued': 1257051619.836942, 't0': 1257051619, 'in_queue': 0.26541614532470703}, 'user_level': 'admin', 'unique_id': 'cmsstor193.fnal.gov-1257051619-17873-0'} to mover 00:00:20 stkenmvr205a.fnal.gov 011191 root I LTO4_105MV WRITE_TO_HSM Thread MainThread 00:00:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM can not remove from sg LTO4_105.mover VOQ856 00:00:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM SG: {'exp-db': [('LTO4_102.mover', 'VON089')], 'cms': [('LTO4_42.mover', 'VOS303'), ('LTO4_109.mover', 'VOQ826'), ('LTO4_44.mover', 'VOQ819'), ('LTO4_95.mover', 'VOQ834'), ('LTO4_93.mover', 'VOQ827'), ('LTO4_51.mover', 'VOQ870'), ('LTO4_108.mover', 'VOQ873'), ('LTO4_97.mover', 'VOQ844'), ('LTO4_41.mover', 'VOQ859'), ('LTO4_47.mover', 'VOU664'), ('LTO4_106.mover', 'VOQ825'), ('LTO4_104.mover', 'VOU867'), ('LTO4_40.mover', 'VOQ839'), ('LTO4_46.mover', 'VOS188'), ('LTO4_96.mover', 'VOQ922'), ('LTO4_94.mover', 'VOQ828'), ('LTO4_107.mover', 'VOU517'), ('LTO4_110.mover', 'VOQ862'), ('LTO4_98.mover', 'VOS175')], 'minos': [('LTO4_101.mover', 'VON584')]} 00:00:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM can not remove from vf LTO4_105.mover VOQ856 00:00:20 stkenmvr205a.fnal.gov 011191 root I LTO4_105MV SENDING {'vc': {'storage_group': 'cms', 'library': 'CD-LTO4G1', 'file_family': 'BACKFILL2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.BACKFILL2.cpio_odc', 'file_family_width': 15}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/3478C9BA-14C6-DE11-AB41-001D09F248F8.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 3951272194L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7865, 'mc_device': '0,6,1,1', 'compression': 0, 'serial_num': '9310801322', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr205a', 'max_time_in_state': 1200, 'device': 'stkenmvr205a:/dev/rmt/tps0d0n', 'name': 'LTO4_105.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_105.stat', 'mover_address': ('131.225.13.201', 7865), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.201', 55182), 'data_ip': 'stkenmvr205a', 'logname': 'LTO4_105MV', 'hostip': '131.225.13.201', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/3478C9BA-14C6-DE11-AB41-001D09F248F8.root', 'uid': 0, 'uname': 'root', 'mtime': 1257051619, 'rmajor': 0, 'machine': ('Linux', 'cmsstor193.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 125853680, 28L, 1, 9811, 5063, 512, 1257051251, 1257051251, 1257051006), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 3951272194L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data2/write-pool-2/data/0007000000000000618063C8', 'inode': 125854664L, 'minor': 28}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/0007000000000000618063C8', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(0007000000000000618063C8)', 'fc': {'size': 3951272194L, 'external_label': 'VOQ856', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data2/write-pool-2/data/0007000000000000618063C8', 'at_the_top': 2, 'local_inode': 8047, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.207.18', 57223), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051620.102401, 'encp_start_time': 1257051619.1430669, 'job_queued': 1257051619.836942, 't0': 1257051619, 'in_queue': 0.26541614532470703}, 'user_level': 'admin', 'unique_id': 'cmsstor193.fnal.gov-1257051619-17873-0'} Thread client_connect_thread 00:00:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM VF: {'minos.reco_far_dogwood1_bntp.cpio_odc': [('LTO4_101.mover', 'VON584')], 'cms.cms2.cpio_odc': [('LTO4_42.mover', 'VOS303'), ('LTO4_47.mover', 'VOU664'), ('LTO4_104.mover', 'VOU867'), ('LTO4_46.mover', 'VOS188'), ('LTO4_107.mover', 'VOU517'), ('LTO4_98.mover', 'VOS175')], 'cms.BACKFILL2.cpio_odc': [('LTO4_109.mover', 'VOQ826'), ('LTO4_44.mover', 'VOQ819'), ('LTO4_95.mover', 'VOQ834'), ('LTO4_93.mover', 'VOQ827'), ('LTO4_51.mover', 'VOQ870'), ('LTO4_108.mover', 'VOQ873'), ('LTO4_97.mover', 'VOQ844'), ('LTO4_41.mover', 'VOQ859'), ('LTO4_106.mover', 'VOQ825'), ('LTO4_40.mover', 'VOQ839'), ('LTO4_96.mover', 'VOQ922'), ('LTO4_94.mover', 'VOQ828'), ('LTO4_110.mover', 'VOQ862')], 'exp-db.archive-d0-online.cpio_odc': [('LTO4_102.mover', 'VON089')]} 00:00:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:20 cmsstor193.fnal.gov 017873 root I ENCP Control socket ('131.225.207.18', 57223) is connected to ('131.225.13.201', 55183) for cmsstor193.fnal.gov-1257051619-17873-0. elapsed=0.990sec 00:00:20 cmsstor193.fnal.gov 017873 root I ENCP Received callback ticket from mover LTO4_105.mover for transfer cmsstor193.fnal.gov-1257051619-17873-0. elapsed=0.990sec 00:00:20 cmsstor193.fnal.gov 017873 root I ENCP Atempting to connect data socket to mover at ('131.225.13.201', 55182). 00:00:20 cmsstor193.fnal.gov 017873 root I ENCP Connecting data socket to mover (('131.225.13.201', 55182)) with interface 131.225.207.18. elapsed=0.990sec 00:00:20 cmsstor193.fnal.gov 017873 root I ENCP Data socket ('131.225.207.18', 39186) is connected to ('131.225.13.201', 55182). elapsed=1.000sec 00:00:20 cmsstor193.fnal.gov 017873 root I ENCP Opening local file. 00:00:20 stkenmvr205a.fnal.gov 011191 root I LTO4_105MV MSG_TYPE=MC_LOAD_REQ mounting VOQ856 ULTRIUM-TD4 00:00:20 Thread media_thread 00:00:20 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOQ856 0,6,1,1 00:00:20 stkenmvr205a.fnal.gov 011191 root E LTO4_105MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:20 stkenmvr205a.fnal.gov 011191 root I LTO4_105MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:20 cmsstor193.fnal.gov 017873 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:20 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOQ834 0,7,1,14 00:00:20 stkenmvr195a.fnal.gov 021479 root E LTO4_95MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:20 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:20 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ825 0,6,1,8 00:00:20 stkenmvr206a.fnal.gov 026766 root E LTO4_106MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:20 stkenmvr206a.fnal.gov 026766 root I LTO4_106MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:20 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ859 0,7,1,3 00:00:20 stkenmvr141a.fnal.gov 013963 root E LTO4_41MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:20 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:21 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:21 cmsstor68.fnal.gov 022209 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: BACKFILL2 FF Wrapper: cpio_odc FF Width: 15 Current working directory: cmsstor68.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --age-time 30 --delpri 1 --priority 25 --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/A0B1C030-11C6-DE11-957D-001D09F2AF96.root --put-cache 000700000000000061806468 /storage/data1/write-pool-1/data/000700000000000061806468 00:00:21 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:21 cmsstor68.fnal.gov 022209 root I ENCP Creating zero length output files. 00:00:21 cmsstor68.fnal.gov 022209 root I ENCP Sending /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/A0B1C030-11C6-DE11-957D-001D09F2AF96.root write request to LM: unique_id: cmsstor68.fnal.gov-1257051621-22209-0 inputfile: /storage/data1/write-pool-1/data/000700000000000061806468 outputfile: /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(000700000000000061806468) 00:00:21 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:21 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data1/write-pool-1/data/000700000000000061806468 (cmsstor68.fnal.gov-1257051621-22209-0) -> /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/A0B1C030-11C6-DE11-957D-001D09F2AF96.root : library=CD-LTO4G1 family=BACKFILL2 requester:root volume_family:cms.BACKFILL2.cpio_odc 00:00:21 cmsstor68.fnal.gov 022209 root I ENCP Listening for control socket at: ('131.225.189.68', 43851) 00:00:21 cmsstor68.fnal.gov 022209 root I ENCP Waiting for mover to connect control socket at address: ('131.225.189.68', 43851) elapsed=1.030sec 00:00:21 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:22 stkenmvr193a.fnal.gov 031647 root I LTO4_93MV Tape is ejected Thread media_thread 00:00:22 stkenmvr193a.fnal.gov 031647 root I LTO4_93MV dismounting VOQ827 Thread media_thread 00:00:22 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ827 0,7,1,11 00:00:22 stkenmvr193a.fnal.gov 031647 root E LTO4_93MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:22 stkenmvr193a.fnal.gov 031647 root I LTO4_93MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:22 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:22 stkenmvr207a.fnal.gov 006260 root I LTO4_107MV Updating stats Thread media_thread 00:00:22 stkenmvr207a.fnal.gov 006260 root I LTO4_107MV writing a tape mark before dismount Thread media_thread 00:00:22 stkenmvr197a.fnal.gov 025501 root I LTO4_97MV tape position: target 95 current 0 1 Thread seek_thread 00:00:22 stkensrv4n.fnal.gov 005843 enstore I SL8500F1MC FINISHED listDrives returned ('ok', 0, None, '', '') 00:00:22 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Assigning volume VOQ857 from storage group cms.BACKFILL2.cpio_odc to library CD-LTO4G1, volume family cms.BACKFILL2.cpio_odc 00:00:22 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm work on vol=VOQ857 mover=LTO4_43.mover requester:root 00:00:22 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:22 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM IDLE:sending {'vc': {'storage_group': 'cms', 'library': 'CD-LTO4G1', 'file_family': 'BACKFILL2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.BACKFILL2.cpio_odc', 'file_family_width': 15}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/A0B1C030-11C6-DE11-957D-001D09F2AF96.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 3953416277L, 'mover': 'LTO4_43.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/A0B1C030-11C6-DE11-957D-001D09F2AF96.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257051621, 'rmajor': 0, 'machine': ('Linux', 'cmsstor68.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16877, 125853680, 25L, 1, 9811, 5063, 512, 1257051251, 1257051251, 1257051006), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 3953416277L, 'fullname': '/storage/data1/write-pool-1/data/000700000000000061806468', 'inode': 125854824L, 'minor': 25}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/000700000000000061806468', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(000700000000000061806468)', 'fc': {'size': 3953416277L, 'external_label': 'VOQ857', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data1/write-pool-1/data/000700000000000061806468', 'at_the_top': 2, 'local_inode': 2147484944L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.189.68', 43851), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051622.9033101, 'encp_start_time': 1257051620.783365, 'job_queued': 1257051621.7740841, 't0': 1257051620, 'in_queue': 1.1291828155517578}, 'user_level': 'admin', 'unique_id': 'cmsstor68.fnal.gov-1257051621-22209-0'} to mover 00:00:22 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV WRITE_TO_HSM Thread MainThread 00:00:23 last message repeated 1 times 00:00:23 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV SENDING {'vc': {'storage_group': 'cms', 'library': 'CD-LTO4G1', 'file_family': 'BACKFILL2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.BACKFILL2.cpio_odc', 'file_family_width': 15}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/A0B1C030-11C6-DE11-957D-001D09F2AF96.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 3953416277L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7843, 'mc_device': '0,7,1,0', 'compression': 0, 'serial_num': '9310055909', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr143a', 'max_time_in_state': 1200, 'device': 'stkenmvr143a:/dev/rmt/tps0d0n', 'name': 'LTO4_43.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_43.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.171', 7843), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.171', 38353), 'data_ip': 'stkenmvr143a', 'logname': 'LTO4_43MV', 'hostip': '131.225.13.171', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/A0B1C030-11C6-DE11-957D-001D09F2AF96.root', 'uid': 0, 'uname': 'root', 'mtime': 1257051621, 'rmajor': 0, 'machine': ('Linux', 'cmsstor68.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 125853680, 25L, 1, 9811, 5063, 512, 1257051251, 1257051251, 1257051006), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 3953416277L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data1/write-pool-1/data/000700000000000061806468', 'inode': 125854824L, 'minor': 25}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/000700000000000061806468', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(000700000000000061806468)', 'fc': {'size': 3953416277L, 'external_label': 'VOQ857', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data1/write-pool-1/data/000700000000000061806468', 'at_the_top': 2, 'local_inode': 2147484944L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.189.68', 43851), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051622.9033101, 'encp_start_time': 1257051620.783365, 'job_queued': 1257051621.7740841, 't0': 1257051620, 'in_queue': 1.1291828155517578}, 'user_level': 'admin', 'unique_id': 'cmsstor68.fnal.gov-1257051621-22209-0'} Thread client_connect_thread 00:00:23 cmsstor68.fnal.gov 022209 root I ENCP Control socket ('131.225.189.68', 43851) is connected to ('131.225.13.171', 38354) for cmsstor68.fnal.gov-1257051621-22209-0. elapsed=2.180sec 00:00:23 cmsstor68.fnal.gov 022209 root I ENCP Received callback ticket from mover LTO4_43.mover for transfer cmsstor68.fnal.gov-1257051621-22209-0. elapsed=2.180sec 00:00:23 cmsstor68.fnal.gov 022209 root I ENCP Atempting to connect data socket to mover at ('131.225.13.171', 38353). 00:00:23 cmsstor68.fnal.gov 022209 root I ENCP Connecting data socket to mover (('131.225.13.171', 38353)) with interface 131.225.189.68. elapsed=2.180sec 00:00:23 cmsstor68.fnal.gov 022209 root I ENCP Data socket ('131.225.189.68', 58746) is connected to ('131.225.13.171', 38353). elapsed=2.180sec 00:00:23 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM can not remove from sg LTO4_43.mover VOQ857 00:00:23 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM SG: {'exp-db': [('LTO4_102.mover', 'VON089')], 'cms': [('LTO4_42.mover', 'VOS303'), ('LTO4_109.mover', 'VOQ826'), ('LTO4_44.mover', 'VOQ819'), ('LTO4_95.mover', 'VOQ834'), ('LTO4_93.mover', 'VOQ827'), ('LTO4_51.mover', 'VOQ870'), ('LTO4_108.mover', 'VOQ873'), ('LTO4_97.mover', 'VOQ844'), ('LTO4_41.mover', 'VOQ859'), ('LTO4_47.mover', 'VOU664'), ('LTO4_106.mover', 'VOQ825'), ('LTO4_104.mover', 'VOU867'), ('LTO4_40.mover', 'VOQ839'), ('LTO4_46.mover', 'VOS188'), ('LTO4_96.mover', 'VOQ922'), ('LTO4_94.mover', 'VOQ828'), ('LTO4_110.mover', 'VOQ862'), ('LTO4_98.mover', 'VOS175'), ('LTO4_105.mover', 'VOQ856'), ('LTO4_107.mover', 'VOU517')], 'minos': [('LTO4_101.mover', 'VON584')]} 00:00:23 cmsstor68.fnal.gov 022209 root I ENCP Opening local file. 00:00:23 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM can not remove from vf LTO4_43.mover VOQ857 00:00:23 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM VF: {'minos.reco_far_dogwood1_bntp.cpio_odc': [('LTO4_101.mover', 'VON584')], 'cms.cms2.cpio_odc': [('LTO4_42.mover', 'VOS303'), ('LTO4_47.mover', 'VOU664'), ('LTO4_104.mover', 'VOU867'), ('LTO4_46.mover', 'VOS188'), ('LTO4_98.mover', 'VOS175'), ('LTO4_107.mover', 'VOU517')], 'cms.BACKFILL2.cpio_odc': [('LTO4_109.mover', 'VOQ826'), ('LTO4_44.mover', 'VOQ819'), ('LTO4_95.mover', 'VOQ834'), ('LTO4_93.mover', 'VOQ827'), ('LTO4_51.mover', 'VOQ870'), ('LTO4_108.mover', 'VOQ873'), ('LTO4_97.mover', 'VOQ844'), ('LTO4_41.mover', 'VOQ859'), ('LTO4_106.mover', 'VOQ825'), ('LTO4_40.mover', 'VOQ839'), ('LTO4_96.mover', 'VOQ922'), ('LTO4_94.mover', 'VOQ828'), ('LTO4_110.mover', 'VOQ862'), ('LTO4_105.mover', 'VOQ856')], 'exp-db.archive-d0-online.cpio_odc': [('LTO4_102.mover', 'VON089')]} 00:00:23 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:23 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV MSG_TYPE=MC_LOAD_REQ mounting VOQ857 ULTRIUM-TD4 00:00:22 Thread media_thread 00:00:23 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOQ857 0,7,1,0 00:00:23 stkenmvr143a.fnal.gov 028425 root E LTO4_43MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:23 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:23 cmsstor68.fnal.gov 022209 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:23 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOS175 0,7,1,1 00:00:23 stkenmvr198a.fnal.gov 015662 root E LTO4_98MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:23 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:23 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:23 stkenmvr202a.fnal.gov 006121 root I LTO4_102MV Updating stats Thread media_thread 00:00:23 stkenmvr202a.fnal.gov 006121 root I LTO4_102MV writing a tape mark before dismount Thread media_thread 00:00:23 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:23 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:23 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:24 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOS188 0,6,1,12 00:00:24 stkenmvr146a.fnal.gov 005990 root E LTO4_46MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:24 stkenmvr146a.fnal.gov 005990 root I LTO4_46MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:24 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:25 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:25 stkenmvr207a.fnal.gov 006260 root I LTO4_107MV Ejecting tape Thread media_thread 00:00:25 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:26 stkenmvr202a.fnal.gov 006121 root I LTO4_102MV Ejecting tape Thread media_thread 00:00:26 cmsstor196.fnal.gov 029300 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Scientific Linux SL release 5.2 (Boron) Rocks release 5.1 (V.I) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: BACKFILL2 FF Wrapper: cpio_odc FF Width: 15 Current working directory: cmsstor196.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --age-time 30 --delpri 1 --priority 25 --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/F4DA130B-16C6-DE11-BC20-000423D992A4.root --put-cache 0007000000000000618064A0 /storage/data3/write-pool-3/data/0007000000000000618064A0 00:00:26 cmsstor196.fnal.gov 029300 root I ENCP Creating zero length output files. 00:00:26 cmsstor196.fnal.gov 029300 root I ENCP Sending /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/F4DA130B-16C6-DE11-BC20-000423D992A4.root write request to LM: unique_id: cmsstor196.fnal.gov-1257051626-29300-0 inputfile: /storage/data3/write-pool-3/data/0007000000000000618064A0 outputfile: /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(0007000000000000618064A0) 00:00:27 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:27 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data3/write-pool-3/data/0007000000000000618064A0 (cmsstor196.fnal.gov-1257051626-29300-0) -> /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/F4DA130B-16C6-DE11-BC20-000423D992A4.root : library=CD-LTO4G1 family=BACKFILL2 requester:root volume_family:cms.BACKFILL2.cpio_odc 00:00:27 cmsstor196.fnal.gov 029300 root I ENCP Listening for control socket at: ('131.225.207.21', 38842) 00:00:27 cmsstor196.fnal.gov 029300 root I ENCP Waiting for mover to connect control socket at address: ('131.225.207.21', 38842) elapsed=0.660sec 00:00:27 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:27 stkenmvr212a.fnal.gov 006265 root I LTO4_112MV Updating stats Thread media_thread 00:00:27 stkenmvr212a.fnal.gov 006265 root I LTO4_112MV writing a tape mark before dismount Thread media_thread 00:00:27 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:28 cmsstor199.fnal.gov 003605 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Scientific Linux SL release 5.2 (Boron) Rocks release 5.1 (V.I) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms2 FF Wrapper: cpio_odc FF Width: 7 Current working directory: cmsstor199.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --age-time 30 --delpri 1 --priority 25 --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt3000to3500/PATLayer1_RECO_Output_QCDDiJet_Pt3000to3500_22.root --put-cache 0010000000000000094DDB60 /storage/data1/write-pool-1/data/0010000000000000094DDB60 00:00:28 cmsstor199.fnal.gov 003605 root I ENCP Creating zero length output files. 00:00:28 cmsstor199.fnal.gov 003605 root I ENCP Sending /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt3000to3500/PATLayer1_RECO_Output_QCDDiJet_Pt3000to3500_22.root write request to LM: unique_id: cmsstor199.fnal.gov-1257051628-3605-0 inputfile: /storage/data1/write-pool-1/data/0010000000000000094DDB60 outputfile: /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt3000to3500/.(access)(0010000000000000094DDB60) 00:00:28 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:28 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data1/write-pool-1/data/0010000000000000094DDB60 (cmsstor199.fnal.gov-1257051628-3605-0) -> /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt3000to3500/PATLayer1_RECO_Output_QCDDiJet_Pt3000to3500_22.root : library=CD-LTO4G1 family=cms2 requester:root volume_family:cms.cms2.cpio_odc 00:00:28 cmsstor199.fnal.gov 003605 root I ENCP Listening for control socket at: ('131.225.207.24', 46713) 00:00:28 cmsstor199.fnal.gov 003605 root I ENCP Waiting for mover to connect control socket at address: ('131.225.207.24', 46713) elapsed=0.710sec 00:00:29 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Assigning volume VOU507 from storage group cms.cms2.cpio_odc to library CD-LTO4G1, volume family cms.cms2.cpio_odc 00:00:29 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm work on vol=VOU507 mover=LTO4_50.mover requester:root 00:00:29 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:29 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM IDLE:sending {'vc': {'storage_group': 'cms', 'library': 'CD-LTO4G1', 'file_family': 'cms2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms2.cpio_odc', 'file_family_width': 7}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt3000to3500/PATLayer1_RECO_Output_QCDDiJet_Pt3000to3500_22.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 418131495L, 'mover': 'LTO4_50.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt3000to3500/PATLayer1_RECO_Output_QCDDiJet_Pt3000to3500_22.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257051628, 'rmajor': 0, 'machine': ('Linux', 'cmsstor199.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16893, 273340616, 21L, 1, 12124, 5063, 512, 1257050989, 1257050989, 1257036572), 'mode_octal': '0100036', 'mode': 33188, 'gname': 'root', 'size_bytes': 418131495L, 'fullname': '/storage/data1/write-pool-1/data/0010000000000000094DDB60', 'inode': 273537888L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/0010000000000000094DDB60', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt3000to3500/.(access)(0010000000000000094DDB60)', 'fc': {'size': 418131495L, 'external_label': 'VOU507', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data1/write-pool-1/data/0010000000000000094DDB60', 'at_the_top': 1, 'local_inode': 1794, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.207.24', 46713), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051629.491936, 'encp_start_time': 1257051627.999048, 'job_queued': 1257051628.685257, 't0': 1257051627, 'in_queue': 0.80662107467651367}, 'user_level': 'admin', 'unique_id': 'cmsstor199.fnal.gov-1257051628-3605-0'} to mover 00:00:29 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV WRITE_TO_HSM Thread MainThread 00:00:29 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM can not remove from sg LTO4_50.mover VOU507 00:00:29 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM SG: {'exp-db': [('LTO4_102.mover', 'VON089')], 'cms': [('LTO4_109.mover', 'VOQ826'), ('LTO4_44.mover', 'VOQ819'), ('LTO4_95.mover', 'VOQ834'), ('LTO4_93.mover', 'VOQ827'), ('LTO4_51.mover', 'VOQ870'), ('LTO4_108.mover', 'VOQ873'), ('LTO4_97.mover', 'VOQ844'), ('LTO4_41.mover', 'VOQ859'), ('LTO4_47.mover', 'VOU664'), ('LTO4_106.mover', 'VOQ825'), ('LTO4_104.mover', 'VOU867'), ('LTO4_40.mover', 'VOQ839'), ('LTO4_46.mover', 'VOS188'), ('LTO4_96.mover', 'VOQ922'), ('LTO4_94.mover', 'VOQ828'), ('LTO4_110.mover', 'VOQ862'), ('LTO4_98.mover', 'VOS175'), ('LTO4_105.mover', 'VOQ856'), ('LTO4_107.mover', 'VOU517'), ('LTO4_43.mover', 'VOQ857'), ('LTO4_42.mover', 'VOS303')], 'minos': [('LTO4_101.mover', 'VON584')]} 00:00:29 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM can not remove from vf LTO4_50.mover VOU507 00:00:29 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM VF: {'minos.reco_far_dogwood1_bntp.cpio_odc': [('LTO4_101.mover', 'VON584')], 'cms.cms2.cpio_odc': [('LTO4_47.mover', 'VOU664'), ('LTO4_104.mover', 'VOU867'), ('LTO4_46.mover', 'VOS188'), ('LTO4_98.mover', 'VOS175'), ('LTO4_107.mover', 'VOU517'), ('LTO4_42.mover', 'VOS303')], 'cms.BACKFILL2.cpio_odc': [('LTO4_109.mover', 'VOQ826'), ('LTO4_44.mover', 'VOQ819'), ('LTO4_95.mover', 'VOQ834'), ('LTO4_93.mover', 'VOQ827'), ('LTO4_51.mover', 'VOQ870'), ('LTO4_108.mover', 'VOQ873'), ('LTO4_97.mover', 'VOQ844'), ('LTO4_41.mover', 'VOQ859'), ('LTO4_106.mover', 'VOQ825'), ('LTO4_40.mover', 'VOQ839'), ('LTO4_96.mover', 'VOQ922'), ('LTO4_94.mover', 'VOQ828'), ('LTO4_110.mover', 'VOQ862'), ('LTO4_105.mover', 'VOQ856'), ('LTO4_43.mover', 'VOQ857')], 'exp-db.archive-d0-online.cpio_odc': [('LTO4_102.mover', 'VON089')]} 00:00:29 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:29 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV SENDING {'vc': {'storage_group': 'cms', 'library': 'CD-LTO4G1', 'file_family': 'cms2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms2.cpio_odc', 'file_family_width': 7}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt3000to3500/PATLayer1_RECO_Output_QCDDiJet_Pt3000to3500_22.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 418131495L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7850, 'mc_device': '0,5,1,2', 'compression': 0, 'serial_num': '9310801229', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr150a', 'max_time_in_state': 1200, 'device': 'stkenmvr150a:/dev/rmt/tps0d0n', 'name': 'LTO4_50.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_50.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.178', 7850), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.178', 54190), 'data_ip': 'stkenmvr150a', 'logname': 'LTO4_50MV', 'hostip': '131.225.13.178', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt3000to3500/PATLayer1_RECO_Output_QCDDiJet_Pt3000to3500_22.root', 'uid': 0, 'uname': 'root', 'mtime': 1257051628, 'rmajor': 0, 'machine': ('Linux', 'cmsstor199.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 273340616, 21L, 1, 12124, 5063, 512, 1257050989, 1257050989, 1257036572), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 418131495L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data1/write-pool-1/data/0010000000000000094DDB60', 'inode': 273537888L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/0010000000000000094DDB60', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt3000to3500/.(access)(0010000000000000094DDB60)', 'fc': {'size': 418131495L, 'external_label': 'VOU507', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data1/write-pool-1/data/0010000000000000094DDB60', 'at_the_top': 1, 'local_inode': 1794, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.207.24', 46713), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051629.491936, 'encp_start_time': 1257051627.999048, 'job_queued': 1257051628.685257, 't0': 1257051627, 'in_queue': 0.80662107467651367}, 'user_level': 'admin', 'unique_id': 'cmsstor199.fnal.gov-1257051628-3605-0'} Thread client_connect_thread 00:00:29 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:29 cmsstor199.fnal.gov 003605 root I ENCP Control socket ('131.225.207.24', 46713) is connected to ('131.225.13.178', 54191) for cmsstor199.fnal.gov-1257051628-3605-0. elapsed=1.540sec 00:00:29 cmsstor199.fnal.gov 003605 root I ENCP Received callback ticket from mover LTO4_50.mover for transfer cmsstor199.fnal.gov-1257051628-3605-0. elapsed=1.540sec 00:00:29 cmsstor199.fnal.gov 003605 root I ENCP Atempting to connect data socket to mover at ('131.225.13.178', 54190). 00:00:29 cmsstor199.fnal.gov 003605 root I ENCP Connecting data socket to mover (('131.225.13.178', 54190)) with interface 131.225.207.24. elapsed=1.540sec 00:00:29 cmsstor199.fnal.gov 003605 root I ENCP Data socket ('131.225.207.24', 55325) is connected to ('131.225.13.178', 54190). elapsed=1.540sec 00:00:29 cmsstor199.fnal.gov 003605 root I ENCP Opening local file. 00:00:29 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VON584 0,6,1,7 00:00:29 stkenmvr201a.fnal.gov 018563 root E LTO4_101MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:29 stkenmvr201a.fnal.gov 018563 root I LTO4_101MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:29 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV MSG_TYPE=MC_LOAD_REQ mounting VOU507 ULTRIUM-TD4 00:00:29 Thread media_thread 00:00:29 cmsstor199.fnal.gov 003605 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:29 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOU507 0,5,1,2 00:00:29 stkenmvr150a.fnal.gov 006006 root E LTO4_50MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:29 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:29 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:30 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOQ856 0,6,1,1 00:00:30 stkenmvr205a.fnal.gov 011191 root E LTO4_105MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:30 stkenmvr205a.fnal.gov 011191 root I LTO4_105MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:30 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOQ834 0,7,1,14 00:00:30 stkenmvr195a.fnal.gov 021479 root E LTO4_95MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:30 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:30 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ825 0,6,1,8 00:00:30 stkenmvr206a.fnal.gov 026766 root E LTO4_106MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:30 stkenmvr206a.fnal.gov 026766 root I LTO4_106MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:30 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ859 0,7,1,3 00:00:30 stkenmvr141a.fnal.gov 013963 root E LTO4_41MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:30 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:31 stkenmvr212a.fnal.gov 006265 root I LTO4_112MV Ejecting tape Thread media_thread 00:00:31 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:32 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:32 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ827 0,7,1,11 00:00:32 stkenmvr193a.fnal.gov 031647 root E LTO4_93MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:32 stkenmvr193a.fnal.gov 031647 root I LTO4_93MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:33 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOQ857 0,7,1,0 00:00:33 stkenmvr143a.fnal.gov 028425 root E LTO4_43MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:33 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:33 cmsstor184.fnal.gov 005655 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Scientific Linux SL release 5.2 (Boron) Rocks release 5.1 (V.I) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: BACKFILL2 FF Wrapper: cpio_odc FF Width: 15 Current working directory: cmsstor184.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --age-time 30 --delpri 1 --priority 25 --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/121A5A58-F8C5-DE11-AA61-001D09F29849.root --put-cache 000700000000000061806760 /storage/data2/write-pool-2/data/000700000000000061806760 00:00:33 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOS175 0,7,1,1 00:00:33 stkenmvr198a.fnal.gov 015662 root E LTO4_98MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:33 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:33 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:33 cmsstor184.fnal.gov 005655 root I ENCP Creating zero length output files. 00:00:33 cmsstor184.fnal.gov 005655 root I ENCP Sending /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/121A5A58-F8C5-DE11-AA61-001D09F29849.root write request to LM: unique_id: cmsstor184.fnal.gov-1257051633-5655-0 inputfile: /storage/data2/write-pool-2/data/000700000000000061806760 outputfile: /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/.(access)(000700000000000061806760) 00:00:33 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:33 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data2/write-pool-2/data/000700000000000061806760 (cmsstor184.fnal.gov-1257051633-5655-0) -> /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/121A5A58-F8C5-DE11-AA61-001D09F29849.root : library=CD-LTO4G1 family=BACKFILL2 requester:root volume_family:cms.BACKFILL2.cpio_odc 00:00:33 cmsstor184.fnal.gov 005655 root I ENCP Listening for control socket at: ('131.225.207.5', 45635) 00:00:33 cmsstor184.fnal.gov 005655 root I ENCP Waiting for mover to connect control socket at address: ('131.225.207.5', 45635) elapsed=0.710sec 00:00:33 stkenmvr113a.fnal.gov 005961 root I LTO13MV skip_fm -9 Thread seek_thread 00:00:33 stkenmvr113a.fnal.gov 005961 root I LTO13MV skip_fm 1 Thread seek_thread 00:00:33 stkenmvr113a.fnal.gov 005961 root I LTO13MV read_tape starting, bytes_to_read=17878595 Thread tape_thread 00:00:33 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:34 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOS188 0,6,1,12 00:00:34 stkenmvr146a.fnal.gov 005990 root E LTO4_46MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:34 stkenmvr146a.fnal.gov 005990 root I LTO4_46MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:34 stkenmvr113a.fnal.gov 005961 root I LTO13MV read bytes 17878743/17878743, blocks 137 header 148 Thread tape_thread 00:00:34 stkenmvr113a.fnal.gov 005961 root I LTO13MV write_client: wrote 17878595/17878595 bytes Thread net_thread 00:00:34 stkensrv3n.fnal.gov 013625 root I ENCP File /pnfs/cms/WAX/11/store/TAC/TIF/edm_2007_07_11/EDM0013281_001.root transfered for stkensrv3n.fnal.gov-1257051591-13625-0 in 0.233088970184 seconds. 00:00:34 stkenmvr113a.fnal.gov 005961 root I LTO13MV write_client waits for tape thread to exit Thread net_thread 00:00:34 stkenmvr113a.fnal.gov 005961 root I LTO13MV drive stats after read. Tape VOE575 position 1 block 137 block_size 0 bloc_loc 139 tot_blocks 381469000 BOT 0 read_err 0 write_err 0 bytes 17878743 block_read_tot 0.275540351868 tape_rate 64886115.1509 Thread tape_thread 00:00:34 stkenmvr113a.fnal.gov 005961 root I LTO13MV read_tape exiting, read 17878743/17878743 bytes Thread tape_thread 00:00:34 stkenmvr113a.fnal.gov 005961 root I LTO13MV read_tape timing:get_space 0.00342750549316 read 0.203475236893 crc_check 0.0662772655487 push 0.00236034393311 block_read 0.275540351868 idle 0.00501394271851 Thread tape_thread 00:00:34 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:34 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:35 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV Tape is ejected Thread media_thread 00:00:35 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV dismounting VOQ873 Thread media_thread 00:00:35 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ873 0,5,1,11 00:00:35 stkenmvr208a.fnal.gov 006111 root E LTO4_108MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:35 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:35 stkenmvr113a.fnal.gov 005961 root I LTO13MV write_client exits Thread net_thread 00:00:35 stkenmvr113a.fnal.gov 005961 root I LTO13MV transfer complete volume=VOE575 location=1 Thread net_thread 00:00:35 last message repeated 1 times 00:00:35 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_rd_access = '42', last_access = '2009-11-01 00:00:35' where label = 'VOE575'; 00:00:35 stkenmvr113a.fnal.gov 005961 root I LTO13MV Sending done to client: {'vc': {'comment': '', 'declared': 1183670260.0, 'blocksize': 131072, 'storage_group': 'cms', 'library': 'CD-LTO3', 'eod_cookie': '0000_000000000_0000268', 'current_location': '0000_000000000_0000001', 'user_inhibit': ['none', 'none'], 'sum_rd_access': 41, 'system_inhibit': ['none', 'full'], 'external_label': 'VOE575', 'wrapper': 'cpio_odc', 'remaining_bytes': 180380160L, 'sum_mounts': 182, 'capacity_bytes': 429496729600L, 'media_type': 'LTO3', 'last_access': 1257051589.0, 'status': ('ok', None), 'si_time': [0.0, 1184275501.0], 'non_del_files': 267, 'sum_wr_err': 0, 'sum_wr_access': 267, 'file_family': 'cms11', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'modification_time': 1220711104.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1184212262.0}, 'outfilepath': '/dev/null', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': 1, 'curpri': 1}, 'file_size': 17878595L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 300, 'max_failures': 5, 'library': 'CD-LTO3.library_manager', 'local_mover': 0, 'dismount_delay': 20, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7813, 'mc_device': '0,3,1,12', 'compression': 0, 'serial_num': '1210116540', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 57671680.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr113a', 'max_time_in_state': 1200, 'device': 'stkenmvr113a:/dev/rmt/tps2d0n', 'name': 'LTO3_13.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD3', 'statistics_path': '/tmp/enstore/enstore/LTO3_13.stat', 'mover_address': ('131.225.13.143', 7813), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.143', 59037), 'data_ip': 'stkenmvr113a', 'logname': 'LTO13MV', 'hostip': '131.225.13.143', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'lm': {'address': ('131.225.13.4', 7506)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/cms/WAX/11/store/TAC/TIF/edm_2007_07_11/EDM0013281_001.root', 'uid': 0, 'uname': 'root', 'rmajor': 0, 'machine': ('Linux', 'stkensrv3n.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 08:50:41 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (33188, 117913896L, 20L, 1, 9811, 5063, 17878595L, 1256499005, 1184212286, 1184212183), 'mode_octal': '0100744', 'mode': 33252, 'size_bytes': 17878595L, 'gname': 'root', 'gid': 0, 'fullname': '/dev/null', 'inode': 117913896L, 'minor': 20}, 'version': 'v3_7d CVS $Revision: 1.929 $ encp', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/cms/WAX/11/store/TAC/TIF/edm_2007_07_11/.(access)(000700000000000005073928)', 'status': ('ok', None), 'volume': 'VOE575', 'outfile': '/dev/null', 'fc': {'status': ('ok', None), 'pnfsid': '000700000000000005073928', 'pnfs_name0': '/pnfs/fs/usr/cms/WAX/11/store/TAC/TIF/edm_2007_07_11/EDM0013281_001.root', 'deleted': 'no', 'external_label': 'VOE575', 'drive': 'stkenmvr112a:/dev/rmt/tps2d0n:1210116641', 'update': '2007-07-11 22:51:27.082114', 'gid': 0, 'location_cookie': '0000_000000000_0000001', 'complete_crc': 796964744L, 'address': ('131.225.13.59', 7501), 'uid': 0, 'bfid': 'CDMS118421228600000', 'sanity_cookie': (65536L, 2965334931L), 'size': 17878595L}, 'infilepath': '/pnfs/cms/WAX/11/store/TAC/TIF/edm_2007_07_11/EDM0013281_001.root', 'at_the_top': 11, 'bfid': 'CDMS118421228600000', 'local_inode': 2469L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': 7, 'retry': 0}, 'callback_addr': ('131.225.13.51', 50464), 'work': 'read_from_hsm', 'override_ro_mount': 1, 'times': {'encp_start_time': 1257051590.162286, 'seek_time': 40.892081022262573, 'transfer_time': 1.3000009059906006, 't0': 1257051590, 'mount_time': 0.019365072250366211, 'lm_dequeued': 1257051593.0143061, 'job_queued': 1257051591.4075489, 'drive_call_time': 0.2000889778137207, 'in_queue': 1.6067130565643311, 'drive_transfer_time': 0.27783036231994629}, 'user_level': 'admin', 'unique_id': 'stkensrv3n.fnal.gov-1257051591-13625-0'} Thread net_thread 00:00:35 stkenmvr113a.fnal.gov 005961 root I LTO13MV Done is sent Thread net_thread 00:00:35 stkenmvr113a.fnal.gov 005961 root I LTO13MV maybe_clean Thread net_thread 00:00:35 stkensrv3n.fnal.gov 013625 root I ENCP Received final dialog for stkensrv3n.fnal.gov-1257051591-13625-0. elapsed=45.280sec 00:00:35 stkensrv3n.fnal.gov 013625 root I ENCP leaving transfer_file(). 00:00:35 stkenmvr113a.fnal.gov 005961 root I LTO13MV returned from maybe_clean Thread net_thread 00:00:35 stkensrv3n.fnal.gov 013625 root I ENCP MSG_TYPE=ENCP_XFER read_from_hsm /pnfs/cms/WAX/11/store/TAC/TIF/edm_2007_07_11/EDM0013281_001.root -> /dev/null: 17878595 bytes copied from VOE575 at 12.3 MB/S (73.2 MB/S network) (61.4 MB/S drive) (73.2 MB/S disk) mover=LTO3_13.mover drive_id=ULTRIUM-TD3 drive_sn=1210116540 drive_vendor=IBM elapsed=45.306 {'storage_group': 'cms', 'encp_crc': 796964744L, 'disk_rate': 73.19259053605667, 'drive_rate': 61.369667168681161, 'driver': 'FTTDriver', 'transfer_rate': 12.250737765815392, 'mover_interface': 'stkenmvr113a', 'overall_rate': 0.40227002071628409, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 12.250737765815392, 'encp_ip': '131.225.13.51', 'unique_id': 'stkensrv3n.fnal.gov-1257051591-13625-0'} 00:00:35 stkensrv3n.fnal.gov 013625 root I ENCP File /dev/null copied successfully. 00:00:35 stkensrv3n.fnal.gov 013625 root I ENCP Completed transferring 17878595 bytes in 1 files in 45.3127698898 sec. Overall rate = 0.402 MB/sec. Transfer rate = 12.3 MB/sec. Network rate = 73.2 MB/sec. Drive rate = 61.4 MB/sec. Disk rate = 73.2 MB/sec. Exit status = 0. 00:00:35 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:35 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:36 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:00:36 stkensrv3n.fnal.gov 013824 root I ENCP Version: v3_7d CVS $Revision: 1.929 $ encp OS: Linux 2.6.9-89.0.9.ELsmp x86_64 Release: Scientific Linux SL release 4.4 (Beryllium), User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Current working directory: stkensrv3n.fnal.gov:/root Command line: /opt/enstore/bin/encp --verbose 10 --bypass-filesystem-max-filesize-check --max-resubmit 7 --delayed-dismount 1 --override-ro-mount --get-bfid CDMS118423631600000 /dev/null 00:00:36 stkensrv4n.fnal.gov 005357 enstore E LTO4G1LM MOVER ERROR RQ {'status': ('ok', None), 'returned_work': None, 'library': None, 'mover_type': '', 'current_location': '0000_000000000_0000000', 'ip_map': '', 'operation': None, 'read_only': 0, 'current_time': 1257051636.4123681, 'error_source': None, 'external_label': None, 'state': 'OFFLINE', 'r_a': (('131.225.13.194', 54880), 9295L, '131.225.13.194-54880-1256586929.509013-4866'), 'library_list': [('CD-LTO4G1.library_manager', ('131.225.13.4', 8541))], 'current_priority': (None, None), 'transfer_deficiency': 1, 'time_in_state': 464706.05230998993, 'address': ('131.225.13.194', 7860), 'volume_family': None, 'work': 'mover_error', 'mover': 'LTO4_99.mover', 'volume_status': (['none', 'none'], ['none', 'none']), 'unique_id': None} 00:00:36 stkensrv3n.fnal.gov 013824 root I ENCP Creating zero length output files. 00:00:36 stkensrv3n.fnal.gov 013824 root I ENCP Sending /pnfs/cms/WAX/11/store/TAC/TIF/edm_2007_07_11/EDM0013290_109.root read request to LM: unique_id: stkensrv3n.fnal.gov-1257051636-13824-0 inputfile: /pnfs/cms/WAX/11/store/TAC/TIF/edm_2007_07_11/.(access)(0007000000000000050A70B8) outputfile: /dev/null 00:00:36 stkensrv4n.fnal.gov 005806 enstore I LTO3LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:36 stkensrv4n.fnal.gov 005806 enstore I LTO3LM read Q'd /pnfs/cms/WAX/11/store/TAC/TIF/edm_2007_07_11/EDM0013290_109.root (stkensrv3n.fnal.gov-1257051636-13824-0) -> /dev/null : library=CD-LTO3 family=cms.cms11.cpio_odc requester:root 00:00:37 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:38 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:00:38 stkensrv4n.fnal.gov 005806 enstore I LTO3LM read_from_hsm next work on vol=VOE575 mover=LTO3_13.mover requester:root 00:00:38 stkensrv4n.fnal.gov 005806 enstore I LTO3LM HAVE_BOUND:sending read_from_hsm /pnfs/cms/WAX/11/store/TAC/TIF/edm_2007_07_11/EDM0013290_109.root to mover LTO3_13.mover ('131.225.13.143', 7813) DEL_DISM 1 00:00:38 stkensrv4n.fnal.gov 005806 enstore I LTO3LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:38 stkensrv4n.fnal.gov 005806 enstore I LTO3LM HAVE_BOUND:sending {'vc': {'comment': '', 'declared': 1183670260.0, 'blocksize': 131072, 'storage_group': 'cms', 'library': 'CD-LTO3', 'eod_cookie': '0000_000000000_0000268', 'current_location': '0000_000000000_0000010', 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'full'], 'external_label': 'VOE575', 'wrapper': 'cpio_odc', 'remaining_bytes': 180380160L, 'sum_mounts': 182, 'sum_rd_access': 42, 'media_type': 'LTO3', 'last_access': 1257051635.0, 'status': ('ok', None), 'si_time': [0.0, 1184275501.0], 'capacity_bytes': 429496729600L, 'non_del_files': 267, 'sum_wr_err': 0, 'sum_wr_access': 267, 'file_family': 'cms11', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'modification_time': 1220711104.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1184212262.0}, 'outfilepath': '/dev/null', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': 1, 'curpri': 1}, 'file_size': 17439631L, 'mover': 'LTO3_13.mover', 'lm': {'address': ('131.225.13.4', 7506)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/cms/WAX/11/store/TAC/TIF/edm_2007_07_11/EDM0013290_109.root', 'uid': 0, 'uname': 'root', 'gid': 0, 'rmajor': 0, 'machine': ('Linux', 'stkensrv3n.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 08:50:41 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (33188, 118124728L, 20L, 1, 9811, 5063, 17439631L, 1256499047, 1184236316, 1184236135), 'mode_octal': '0100744', 'mode': 33252, 'gname': 'root', 'size_bytes': 17439631L, 'fullname': '/dev/null', 'inode': 118124728L, 'minor': 20}, 'version': 'v3_7d CVS $Revision: 1.929 $ encp', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/cms/WAX/11/store/TAC/TIF/edm_2007_07_11/.(access)(0007000000000000050A70B8)', 'status': ('ok', None), 'volume': 'VOE575', 'outfile': '/dev/null', 'fc': {'status': ('ok', None), 'pnfsid': '0007000000000000050A70B8', 'pnfs_name0': '/pnfs/fs/usr/cms/WAX/11/store/TAC/TIF/edm_2007_07_11/EDM0013290_109.root', 'deleted': 'no', 'external_label': 'VOE575', 'drive': 'stkenmvr115a:/dev/rmt/tps2d0n:1210114342', 'update': '2007-07-12 05:31:56.870138', 'gid': 0, 'location_cookie': '0000_000000000_0000010', 'complete_crc': 497667335L, 'address': ('131.225.13.59', 7501), 'uid': 0, 'bfid': 'CDMS118423631600000', 'sanity_cookie': (65536L, 3662479943L), 'size': 17439631L}, 'infilepath': '/pnfs/cms/WAX/11/store/TAC/TIF/edm_2007_07_11/EDM0013290_109.root', 'at_the_top': 11, 'bfid': 'CDMS118423631600000', 'local_inode': 2469L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': 7, 'retry': 0}, 'callback_addr': ('131.225.13.51', 50468), 'work': 'read_from_hsm', 'override_ro_mount': 1, 'times': {'lm_dequeued': 1257051638.1235771, 'encp_start_time': 1257051635.6477201, 'job_queued': 1257051636.7958441, 't0': 1257051635, 'in_queue': 1.3276588916778564}, 'user_level': 'admin', 'unique_id': 'stkensrv3n.fnal.gov-1257051636-13824-0'} to mover 00:00:38 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:00:38 stkenmvr113a.fnal.gov 005961 root I LTO13MV READ FROM HSM Thread MainThread 00:00:38 stkensrv3n.fnal.gov 013824 root I ENCP Listening for control socket at: ('131.225.13.51', 50468) 00:00:38 stkensrv3n.fnal.gov 013824 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.51', 50468) elapsed=2.510sec 00:00:38 stkenmvr113a.fnal.gov 005961 root I LTO13MV SENDING {'vc': {'comment': '', 'declared': 1183670260.0, 'blocksize': 131072, 'storage_group': 'cms', 'library': 'CD-LTO3', 'eod_cookie': '0000_000000000_0000268', 'current_location': '0000_000000000_0000010', 'user_inhibit': ['none', 'none'], 'sum_rd_access': 42, 'system_inhibit': ['none', 'full'], 'external_label': 'VOE575', 'wrapper': 'cpio_odc', 'remaining_bytes': 180380160L, 'sum_mounts': 182, 'capacity_bytes': 429496729600L, 'media_type': 'LTO3', 'last_access': 1257051635.0, 'status': ('ok', None), 'si_time': [0.0, 1184275501.0], 'non_del_files': 267, 'sum_wr_err': 0, 'sum_wr_access': 267, 'file_family': 'cms11', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'modification_time': 1220711104.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1184212262.0}, 'outfilepath': '/dev/null', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': 1, 'curpri': 1}, 'file_size': 17439631L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 300, 'max_failures': 5, 'library': 'CD-LTO3.library_manager', 'local_mover': 0, 'dismount_delay': 20, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7813, 'mc_device': '0,3,1,12', 'compression': 0, 'serial_num': '1210116540', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 57671680.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr113a', 'max_time_in_state': 1200, 'device': 'stkenmvr113a:/dev/rmt/tps2d0n', 'name': 'LTO3_13.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD3', 'statistics_path': '/tmp/enstore/enstore/LTO3_13.stat', 'mover_address': ('131.225.13.143', 7813), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.143', 59044), 'data_ip': 'stkenmvr113a', 'logname': 'LTO13MV', 'hostip': '131.225.13.143', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'lm': {'address': ('131.225.13.4', 7506)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/cms/WAX/11/store/TAC/TIF/edm_2007_07_11/EDM0013290_109.root', 'uid': 0, 'uname': 'root', 'rmajor': 0, 'machine': ('Linux', 'stkensrv3n.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 08:50:41 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (33188, 118124728L, 20L, 1, 9811, 5063, 17439631L, 1256499047, 1184236316, 1184236135), 'mode_octal': '0100744', 'mode': 33252, 'size_bytes': 17439631L, 'gname': 'root', 'gid': 0, 'fullname': '/dev/null', 'inode': 118124728L, 'minor': 20}, 'version': 'v3_7d CVS $Revision: 1.929 $ encp', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/cms/WAX/11/store/TAC/TIF/edm_2007_07_11/.(access)(0007000000000000050A70B8)', 'status': ('ok', None), 'volume': 'VOE575', 'outfile': '/dev/null', 'fc': {'status': ('ok', None), 'pnfsid': '0007000000000000050A70B8', 'pnfs_name0': '/pnfs/fs/usr/cms/WAX/11/store/TAC/TIF/edm_2007_07_11/EDM0013290_109.root', 'deleted': 'no', 'external_label': 'VOE575', 'drive': 'stkenmvr115a:/dev/rmt/tps2d0n:1210114342', 'update': '2007-07-12 05:31:56.870138', 'gid': 0, 'location_cookie': '0000_000000000_0000010', 'complete_crc': 497667335L, 'address': ('131.225.13.59', 7501), 'uid': 0, 'bfid': 'CDMS118423631600000', 'sanity_cookie': (65536L, 3662479943L), 'size': 17439631L}, 'infilepath': '/pnfs/cms/WAX/11/store/TAC/TIF/edm_2007_07_11/EDM0013290_109.root', 'at_the_top': 11, 'bfid': 'CDMS118423631600000', 'local_inode': 2469L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': 7, 'retry': 0}, 'callback_addr': ('131.225.13.51', 50468), 'work': 'read_from_hsm', 'override_ro_mount': 1, 'times': {'lm_dequeued': 1257051638.1235771, 'encp_start_time': 1257051635.6477201, 'job_queued': 1257051636.7958441, 't0': 1257051635, 'in_queue': 1.3276588916778564}, 'user_level': 'admin', 'unique_id': 'stkensrv3n.fnal.gov-1257051636-13824-0'} Thread client_connect_thread 00:00:38 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:00:38 stkensrv3n.fnal.gov 013824 root I ENCP Control socket ('131.225.13.51', 50468) is connected to ('131.225.13.143', 59045) for stkensrv3n.fnal.gov-1257051636-13824-0. elapsed=2.530sec 00:00:38 stkensrv3n.fnal.gov 013824 root I ENCP Received callback ticket from mover LTO3_13.mover for transfer stkensrv3n.fnal.gov-1257051636-13824-0. elapsed=2.540sec 00:00:38 stkensrv3n.fnal.gov 013824 root I ENCP Attempting to connect data socket to mover at ('131.225.13.143', 59044). 00:00:38 stkensrv3n.fnal.gov 013824 root I ENCP Connecting data socket to mover (('131.225.13.143', 59044)) with interface 131.225.13.51. elapsed=2.540sec 00:00:38 stkensrv3n.fnal.gov 013824 root I ENCP Data socket ('131.225.13.51', 50469) is connected to ('131.225.13.143', 59044). elapsed=2.540sec 00:00:38 last message repeated 1 times 00:00:38 stkensrv3n.fnal.gov 013824 root I ENCP Opening local file. 00:00:38 stkenmvr113a.fnal.gov 005961 root I LTO13MV tape position: target 10 current 1 137 Thread seek_thread 00:00:38 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV skip_fm 591 Thread seek_thread 00:00:38 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV write_tape starting, bytes_to_write=388103168 Thread tape_thread 00:00:38 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV Write starting. Tape VOU867 absolute location in blocks 728231 Thread tape_thread 00:00:39 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VON584 0,6,1,7 00:00:39 stkenmvr201a.fnal.gov 018563 root E LTO4_101MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:39 stkenmvr201a.fnal.gov 018563 root I LTO4_101MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:39 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOU507 0,5,1,2 00:00:39 stkenmvr150a.fnal.gov 006006 root E LTO4_50MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:39 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:39 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:40 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOQ856 0,6,1,1 00:00:40 stkenmvr205a.fnal.gov 011191 root E LTO4_105MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:40 stkenmvr205a.fnal.gov 011191 root I LTO4_105MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:40 cmsstor79.fnal.gov 012785 root I ENCP File /storage/data3/write-pool-3/data/000700000000000061806770 transfered for cmsstor79.fnal.gov-1257051616-12785-0 in 22.6233019829 seconds. 00:00:40 stkensrv4n.fnal.gov 019389 enstore I SL8500F1MC mount VOO268 1,2,1,4 => 0,Mount: VOO268 mounted on 1, 2, 1, 4 00:00:40 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOQ834 0,7,1,14 00:00:40 stkensrv4n.fnal.gov 005843 enstore I SL8500F1MC FINISHED mount VOO268 1,2,1,4 returned ('ok', 0, 'mount VOO268 1,2,1,4 => 0,Mount: VOO268 mounted on 1, 2, 1, 4', '', '') 00:00:40 stkenmvr195a.fnal.gov 021479 root E LTO4_95MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:40 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:40 last message repeated 1 times 00:00:40 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV written bytes 2648936448/2648936448, blocks 20210 header 285 trailer 235 Thread tape_thread 00:00:40 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_mounts = '4850', last_access = '2009-11-01 00:00:40' where label = 'VOO268'; 00:00:40 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ825 0,6,1,8 00:00:40 stkenmvr206a.fnal.gov 026766 root E LTO4_106MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:40 stkenmvr206a.fnal.gov 026766 root I LTO4_106MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:40 stkenmvr216a.fnal.gov 007455 root I LTO4_116MV MSG_TYPE=MC_LOAD_DONE mounted VOO268 ULTRIUM-TD4 00:00:40 Thread media_thread 00:00:40 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ859 0,7,1,3 00:00:40 stkenmvr141a.fnal.gov 013963 root E LTO4_41MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:40 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:41 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV filemarks written. Tape VOQ862 absolute location in blocks 1308224 Thread tape_thread 00:00:41 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV write_tape timing:pull 0.219658136368 write 7.89999461174 crc_check 12.2167830467 freespace 0.22917675972 block_write 20.5656125546 idle 0.395574092865 Thread tape_thread 00:00:41 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV drive stats after write. Tape VOQ862 position 57 block 0 block_size 0 bloc_loc 1308224 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 2648936448 block_write_tot 20.5656125546 tape_rate 128804159.904 Thread tape_thread 00:00:41 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV remaining bytes info in DB 622087168000 estimated 619438231552 reported from drive 619633664000 position 58 Thread tape_thread 00:00:41 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV new bitfile request {'uid': 0, 'external_label': 'VOQ862', 'gid': 0, 'location_cookie': '0000_000000000_0000057', 'complete_crc': 4182564546L, 'sanity_cookie': (65536L, 47320582L), 'size': 2648935928L} Thread tape_thread 00:00:41 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV set remaining: VOQ862 619633664000 0000_000000000_0000058 Thread tape_thread 00:00:41 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:41 last message repeated 1 times 00:00:41 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-01 00:00:41', eod_cookie = '0000_000000000_0000058', non_del_files = '57', remaining_bytes = 619633664000 where label = 'VOQ862'; 00:00:41 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV transfer complete volume=VOQ862 location=58 Thread tape_thread 00:00:41 last message repeated 1 times 00:00:41 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '57', modification_time = '2009-11-01 00:00:41' where label = 'VOQ862'; 00:00:41 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1255358534.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 1242983135.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOQ862', 'wrapper': 'cpio_odc', 'remaining_bytes': 622087168000L, 'sum_mounts': 294, 'sum_rd_access': 0, 'file_family_width': 15, 'last_access': 1257051616.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000057', 'capacity_bytes': 858993459200L, 'non_del_files': 56, 'sum_wr_err': 0, 'sum_wr_access': 56, 'file_family': 'BACKFILL2', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.BACKFILL2.cpio_odc', 'modification_time': 1257051616.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257039541.0}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/CE53548A-0DC6-DE11-A322-001D09F2437B.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2648935928L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7870, 'mc_device': '0,5,1,14', 'compression': 0, 'serial_num': '1310049971', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr210a', 'max_time_in_state': 1200, 'device': 'stkenmvr210a:/dev/rmt/tps0d0n', 'name': 'LTO4_110.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_110.stat', 'mover_address': ('131.225.13.206', 7870), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.206', 53938), 'data_ip': 'stkenmvr210a', 'logname': 'LTO4_110MV', 'hostip': '131.225.13.206', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/CE53548A-0DC6-DE11-A322-001D09F2437B.root', 'uid': 0, 'uname': 'root', 'mtime': 1257051616, 'rmajor': 0, 'machine': ('Linux', 'cmsstor79.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 125853680, 27L, 1, 9811, 5063, 512, 1257051251, 1257051251, 1257051006), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 2648935928L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data3/write-pool-3/data/000700000000000061806770', 'inode': 125855600L, 'minor': 27}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/000700000000000061806770', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(000700000000000061806770)', 'fc': {'uid': 0, 'external_label': 'VOQ862', 'gid': 0, 'location_cookie': '0000_000000000_0000057', 'complete_crc': 4182564546L, 'bfid': 'CDMS125705164100000', 'sanity_cookie': (65536L, 47320582L), 'size': 2648935928L}, 'infilepath': '/storage/data3/write-pool-3/data/000700000000000061806770', 'at_the_top': 1, 'local_inode': 1073742676, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.205.201', 56272), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257051615.852011, 'seek_time': 0.030022859573364258, 'transfer_time': 24.370850086212158, 't0': 1257051615, 'mount_time':(trunc. 4364) Thread tape_thread 00:00:41 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV Done is sent Thread tape_thread 00:00:41 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV maybe_clean Thread tape_thread 00:00:41 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV returned from maybe_clean Thread tape_thread 00:00:41 cmsstor79.fnal.gov 012785 root I ENCP Received final dialog for cmsstor79.fnal.gov-1257051616-12785-0. elapsed=26.110sec 00:00:41 cmsstor79.fnal.gov 012785 root I ENCP leaving transfer_file(). 00:00:41 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:42 stkenmvr196a.fnal.gov 027637 root I LTO4_96MV Tape is ejected Thread media_thread 00:00:42 stkenmvr196a.fnal.gov 027637 root I LTO4_96MV dismounting VOQ922 Thread media_thread 00:00:42 cmsstor172.fnal.gov 007097 root I ENCP File /storage/data1/write-pool-1/data/0010000000000000094DD638 transfered for cmsstor172.fnal.gov-1257051261-7097-0 in 3.30473208427 seconds. 00:00:42 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ922 0,7,1,2 00:00:42 stkenmvr196a.fnal.gov 027637 root E LTO4_96MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:42 stkenmvr196a.fnal.gov 027637 root I LTO4_96MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:42 cmsstor79.fnal.gov 012785 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:42 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/CE53548A-0DC6-DE11-A322-001D09F2437B.root', drive = 'stkenmvr210a:/dev/rmt/tps0d0n:1310049971', pnfs_id = '000700000000000061806770' where bfid = 'CDMS125705164100000'; 00:00:42 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ827 0,7,1,11 00:00:42 stkenmvr193a.fnal.gov 031647 root E LTO4_93MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:42 stkenmvr193a.fnal.gov 031647 root I LTO4_93MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:42 cmsstor79.fnal.gov 012785 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /storage/data3/write-pool-3/data/000700000000000061806770 -> /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/CE53548A-0DC6-DE11-A322-001D09F2437B.root: 2648935928 bytes copied to VOQ862 at 104 MB/S (88.4 MB/S network) (111 MB/S drive) (112 MB/S disk) mover=LTO4_110.mover drive_id=ULTRIUM-TD4 drive_sn=1310049971 drive_vendor=IBM elapsed=26.473 {'storage_group': 'cms', 'encp_crc': 4182564546L, 'disk_rate': 111.68663185879205, 'drive_rate': 110.97623758622311, 'driver': 'FTTDriver', 'transfer_rate': 103.56186477636605, 'mover_interface': 'stkenmvr210a', 'overall_rate': 101.756763262254, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 103.56186477636605, 'encp_ip': '131.225.205.201', 'unique_id': 'cmsstor79.fnal.gov-1257051616-12785-0'} 00:00:42 cmsstor79.fnal.gov 012785 root I ENCP File /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/CE53548A-0DC6-DE11-A322-001D09F2437B.root copied successfully. 00:00:42 cmsstor79.fnal.gov 012785 root I ENCP Completed transferring 2648935928 bytes in 1 files in 26.4746501446 sec. Overall rate = 102 MB/sec. Transfer rate = 104 MB/sec. Network rate = 88.4 MB/sec. Drive rate = 111 MB/sec. Disk rate = 112 MB/sec. Exit status = 0. 00:00:42 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:42 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:42 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VOQ862 mover=LTO4_110.mover requester:root 00:00:42 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/F4DA130B-16C6-DE11-BC20-000423D992A4.root to mover LTO4_110.mover ('131.225.13.206', 7870) DEL_DISM None 00:00:42 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:42 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'cms', 'external_label': 'VOQ862', 'library': 'CD-LTO4G1', 'file_family': 'BACKFILL2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.BACKFILL2.cpio_odc', 'file_family_width': 15}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/F4DA130B-16C6-DE11-BC20-000423D992A4.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 3937101471L, 'mover': 'LTO4_110.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/F4DA130B-16C6-DE11-BC20-000423D992A4.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257051626, 'rmajor': 0, 'machine': ('Linux', 'cmsstor196.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16877, 125853680, 21L, 1, 9811, 5063, 512, 1257051251, 1257051251, 1257051006), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 3937101471L, 'fullname': '/storage/data3/write-pool-3/data/0007000000000000618064A0', 'inode': 125854880L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/0007000000000000618064A0', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(0007000000000000618064A0)', 'fc': {'size': 3937101471L, 'external_label': 'VOQ862', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data3/write-pool-3/data/0007000000000000618064A0', 'at_the_top': 48, 'local_inode': 5922, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.207.21', 38842), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051642.5053689, 'encp_start_time': 1257051626.3732641, 'job_queued': 1257051627.012378, 't0': 1257051626, 'in_queue': 15.492713928222656}, 'user_level': 'admin', 'unique_id': 'cmsstor196.fnal.gov-1257051626-29300-0'} to mover 00:00:42 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:42 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV WRITE_TO_HSM Thread MainThread 00:00:42 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV SENDING {'vc': {'storage_group': 'cms', 'external_label': 'VOQ862', 'library': 'CD-LTO4G1', 'file_family': 'BACKFILL2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.BACKFILL2.cpio_odc', 'file_family_width': 15}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/F4DA130B-16C6-DE11-BC20-000423D992A4.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 3937101471L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7870, 'mc_device': '0,5,1,14', 'compression': 0, 'serial_num': '1310049971', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr210a', 'max_time_in_state': 1200, 'device': 'stkenmvr210a:/dev/rmt/tps0d0n', 'name': 'LTO4_110.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_110.stat', 'mover_address': ('131.225.13.206', 7870), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.206', 53944), 'data_ip': 'stkenmvr210a', 'logname': 'LTO4_110MV', 'hostip': '131.225.13.206', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/F4DA130B-16C6-DE11-BC20-000423D992A4.root', 'uid': 0, 'uname': 'root', 'mtime': 1257051626, 'rmajor': 0, 'machine': ('Linux', 'cmsstor196.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 125853680, 21L, 1, 9811, 5063, 512, 1257051251, 1257051251, 1257051006), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 3937101471L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data3/write-pool-3/data/0007000000000000618064A0', 'inode': 125854880L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/0007000000000000618064A0', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(0007000000000000618064A0)', 'fc': {'size': 3937101471L, 'external_label': 'VOQ862', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data3/write-pool-3/data/0007000000000000618064A0', 'at_the_top': 48, 'local_inode': 5922, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.207.21', 38842), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051642.5053689, 'encp_start_time': 1257051626.3732641, 'job_queued': 1257051627.012378, 't0': 1257051626, 'in_queue': 15.492713928222656}, 'user_level': 'admin', 'unique_id': 'cmsstor196.fnal.gov-1257051626-29300-0'} Thread client_connect_thread 00:00:42 cmsstor196.fnal.gov 029300 root I ENCP Control socket ('131.225.207.21', 38842) is connected to ('131.225.13.206', 53945) for cmsstor196.fnal.gov-1257051626-29300-0. elapsed=16.160sec 00:00:42 cmsstor196.fnal.gov 029300 root I ENCP Received callback ticket from mover LTO4_110.mover for transfer cmsstor196.fnal.gov-1257051626-29300-0. elapsed=16.170sec 00:00:42 cmsstor196.fnal.gov 029300 root I ENCP Atempting to connect data socket to mover at ('131.225.13.206', 53944). 00:00:42 cmsstor196.fnal.gov 029300 root I ENCP Connecting data socket to mover (('131.225.13.206', 53944)) with interface 131.225.207.21. elapsed=16.170sec 00:00:42 cmsstor196.fnal.gov 029300 root I ENCP Data socket ('131.225.207.21', 51848) is connected to ('131.225.13.206', 53944). elapsed=16.170sec 00:00:42 cmsstor196.fnal.gov 029300 root I ENCP Opening local file. 00:00:42 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV tape position: target 58 current 58 0 Thread seek_thread 00:00:42 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV write_tape starting, bytes_to_write=3937102336 Thread tape_thread 00:00:42 cmsstor196.fnal.gov 029300 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:42 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV Write starting. Tape VOQ862 absolute location in blocks 1308224 Thread tape_thread 00:00:42 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:43 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOQ857 0,7,1,0 00:00:43 stkenmvr143a.fnal.gov 028425 root E LTO4_43MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:43 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:43 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOS175 0,7,1,1 00:00:43 stkenmvr198a.fnal.gov 015662 root E LTO4_98MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:43 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:43 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:43 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:44 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOS188 0,6,1,12 00:00:44 stkenmvr146a.fnal.gov 005990 root E LTO4_46MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:44 stkenmvr146a.fnal.gov 005990 root I LTO4_46MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:44 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV written bytes 388103168/388103168, blocks 2961 header 234 trailer 126 Thread tape_thread 00:00:44 cmsstor229.fnal.gov 032233 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: BACKFILL2 FF Wrapper: cpio_odc FF Width: 15 Current working directory: cmsstor229.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --age-time 30 --delpri 1 --priority 25 --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/D2772D1F-0EC6-DE11-A0FE-0016177CA7A0.root --put-cache 000700000000000061806428 /storage/data1/write-pool-1/data/000700000000000061806428 00:00:44 cmsstor229.fnal.gov 032233 root I ENCP Creating zero length output files. 00:00:44 cmsstor229.fnal.gov 032233 root I ENCP Sending /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/D2772D1F-0EC6-DE11-A0FE-0016177CA7A0.root write request to LM: unique_id: cmsstor229.fnal.gov-1257051644-32233-0 inputfile: /storage/data1/write-pool-1/data/000700000000000061806428 outputfile: /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(000700000000000061806428) 00:00:44 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:44 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data1/write-pool-1/data/000700000000000061806428 (cmsstor229.fnal.gov-1257051644-32233-0) -> /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/D2772D1F-0EC6-DE11-A0FE-0016177CA7A0.root : library=CD-LTO4G1 family=BACKFILL2 requester:root volume_family:cms.BACKFILL2.cpio_odc 00:00:44 cmsstor229.fnal.gov 032233 root I ENCP Listening for control socket at: ('131.225.191.101', 39791) 00:00:44 cmsstor229.fnal.gov 032233 root I ENCP Waiting for mover to connect control socket at address: ('131.225.191.101', 39791) elapsed=0.790sec 00:00:45 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ873 0,5,1,11 00:00:45 stkenmvr208a.fnal.gov 006111 root E LTO4_108MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:45 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:45 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:45 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV filemarks written. Tape VOU867 absolute location in blocks 731193 Thread tape_thread 00:00:45 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV write_tape timing:pull 0.0296213626862 write 3.06766462326 crc_check 1.79387688637 freespace 0.0330338478088 block_write 4.92419672012 idle 0.0551183223724 Thread tape_thread 00:00:45 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV drive stats after write. Tape VOU867 position 591 block 0 block_size 0 bloc_loc 731193 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 388103168 block_write_tot 4.92419672012 tape_rate 78815528.7164 Thread tape_thread 00:00:45 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV remaining bytes info in DB 688552960000 estimated 688164856832 reported from drive 688189440000 position 592 Thread tape_thread 00:00:45 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV new bitfile request {'uid': 0, 'external_label': 'VOU867', 'gid': 0, 'location_cookie': '0000_000000000_0000591', 'complete_crc': 847613053L, 'sanity_cookie': (65536L, 4133886158L), 'size': 388102808L} Thread tape_thread 00:00:45 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV set remaining: VOU867 688189440000 0000_000000000_0000592 Thread tape_thread 00:00:45 last message repeated 1 times 00:00:45 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-01 00:00:45', eod_cookie = '0000_000000000_0000592', non_del_files = '593', remaining_bytes = 688189440000 where label = 'VOU867'; 00:00:45 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV transfer complete volume=VOU867 location=592 Thread tape_thread 00:00:45 last message repeated 1 times 00:00:45 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '591', modification_time = '2009-11-01 00:00:45' where label = 'VOU867'; 00:00:45 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1244479824.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOU867', 'wrapper': 'cpio_odc', 'remaining_bytes': 688552960000L, 'sum_mounts': 21, 'sum_rd_access': 0, 'file_family_width': 7, 'last_access': 1257050244.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000591', 'capacity_bytes': 858993459200L, 'non_del_files': 592, 'sum_wr_err': 0, 'sum_wr_access': 590, 'file_family': 'cms2', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms2.cpio_odc', 'modification_time': 1257050244.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1256259225.0}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt600to800/PATLayer1_RECO_Output_QCDDiJet_Pt600to800_17.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 388102808L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7864, 'mc_device': '0,6,1,13', 'compression': 0, 'serial_num': '1310050819', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr204a', 'max_time_in_state': 1200, 'device': 'stkenmvr204a:/dev/rmt/tps0d0n', 'name': 'LTO4_104.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_104.stat', 'mover_address': ('131.225.13.199', 7864), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.199', 53993), 'data_ip': 'stkenmvr204a', 'logname': 'LTO4_104MV', 'hostip': '131.225.13.199', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt600to800/PATLayer1_RECO_Output_QCDDiJet_Pt600to800_17.root', 'uid': 0, 'uname': 'root', 'mtime': 1257051261, 'rmajor': 0, 'machine': ('Linux', 'cmsstor172.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 273340088, 21L, 1, 12124, 5063, 512, 1257049983, 1257049983, 1257036569), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 388102808L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data1/write-pool-1/data/0010000000000000094DD638', 'inode': 273536568L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/0010000000000000094DD638', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt600to800/.(access)(0010000000000000094DD638)', 'fc': {'uid': 0, 'external_label': 'VOU867', 'gid': 0, 'location_cookie': '0000_000000000_0000591', 'complete_crc': 847613053L, 'bfid': 'CDMS125705164500000', 'sanity_cookie': (65536L, 4133886158L), 'size': 388102808L}, 'infilepath': '/storage/data1/write-pool-1/data/0010000000000000094DD638', 'at_the_top': 3, 'local_inode': 7341, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.206.133', 34321), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257051260.99983, 'seek_time': 349.51250004768372, 'transfer_time': 7.2531299591064453, 't0': 1257051260, 'lm_dequeued': 1257051289.1252229, 'job_queued': 1257051261.6517961, 'drive_call_time': 3.0373704433441162, 'in_queue': 27.473379850387573, 'drive_transfer_time': 6.5404088497161865}, 'user_level': 'adm(trunc. 4138) Thread tape_thread 00:00:46 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV Done is sent Thread tape_thread 00:00:46 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV maybe_clean Thread tape_thread 00:00:46 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:46 cmsstor172.fnal.gov 007097 root I ENCP Received final dialog for cmsstor172.fnal.gov-1257051261-7097-0. elapsed=384.990sec 00:00:46 cmsstor172.fnal.gov 007097 root I ENCP leaving transfer_file(). 00:00:46 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV returned from maybe_clean Thread tape_thread 00:00:46 cmsstor172.fnal.gov 007097 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:46 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt600to800/PATLayer1_RECO_Output_QCDDiJet_Pt600to800_17.root', drive = 'stkenmvr204a:/dev/rmt/tps0d0n:1310050819', pnfs_id = '0010000000000000094DD638' where bfid = 'CDMS125705164500000'; 00:00:46 cmsstor172.fnal.gov 007097 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /storage/data1/write-pool-1/data/0010000000000000094DD638 -> /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt600to800/PATLayer1_RECO_Output_QCDDiJet_Pt600to800_17.root: 388102808 bytes copied to VOU867 at 50.7 MB/S (99.8 MB/S network) (56.6 MB/S drive) (112 MB/S disk) mover=LTO4_104.mover drive_id=ULTRIUM-TD4 drive_sn=1310050819 drive_vendor=IBM elapsed=385.25 {'storage_group': 'cms', 'encp_crc': 847613053L, 'disk_rate': 112.17456907328175, 'drive_rate': 56.59029712352114, 'driver': 'FTTDriver', 'transfer_rate': 50.669944979248285, 'mover_interface': 'stkenmvr204a', 'overall_rate': 1.0364870305738201, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 50.669944979248285, 'encp_ip': '131.225.206.133', 'unique_id': 'cmsstor172.fnal.gov-1257051261-7097-0'} 00:00:46 cmsstor172.fnal.gov 007097 root I ENCP File /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt600to800/PATLayer1_RECO_Output_QCDDiJet_Pt600to800_17.root copied successfully. 00:00:46 cmsstor172.fnal.gov 007097 root I ENCP Completed transferring 388102808 bytes in 1 files in 385.251415968 sec. Overall rate = 1.04 MB/sec. Transfer rate = 50.7 MB/sec. Network rate = 99.8 MB/sec. Drive rate = 56.6 MB/sec. Disk rate = 112 MB/sec. Exit status = 0. 00:00:46 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:47 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:47 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:48 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:49 stkensrv2n.fnal.gov 006421 enstore E INQSRV Could not create outage file lock file (/srv2/enstore/www/web-pages//enstore_outage.py.lock) 00:00:49 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:49 cmsstor224.fnal.gov 013968 root I ENCP File /storage/data1/write-pool-1/data/000700000000000061806490 transfered for cmsstor224.fnal.gov-1257051615-13968-0 in 33.3483989239 seconds. 00:00:49 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV written bytes 3933111296/3933111296, blocks 30008 header 285 trailer 316 Thread tape_thread 00:00:49 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VON584 0,6,1,7 00:00:49 stkenmvr201a.fnal.gov 018563 root E LTO4_101MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:49 stkenmvr201a.fnal.gov 018563 root I LTO4_101MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:49 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOU507 0,5,1,2 00:00:49 stkenmvr150a.fnal.gov 006006 root E LTO4_50MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:49 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:49 stkensrv2n.fnal.gov 006421 enstore E INQSRV Could not read file /srv2/enstore/www/web-pages//enstore_outage.py 00:00:49 cmsstor49.fnal.gov 001396 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Scientific Linux SL release 5.2 (Boron) Rocks release 5.1 (V.I) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: NotWrittenToTape Storage Group: cms File Family: CCRCDELETE FF Wrapper: cpio_odc FF Width: 3 Current working directory: cmsstor49.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --priority 25 --pnfs-mount /pnfs/fs --put-cache 0007000000000000617ED318 /storage/data4/write-pool-4/data/0007000000000000617ED318 00:00:49 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:50 cmsstor49.fnal.gov 001396 root I ENCP Configuration Server: no such name: 'NotWrittenToTape.library_manager' 00:00:50 cmsstor49.fnal.gov 001396 root E ENCP INFILE=/storage/data4/write-pool-4/data/0007000000000000617ED318 OUTFILE=/pnfs/fs/usr/cms/WAX/11/store/PhEDEx_LoadTest07/LoadTest07_Debug_PIC/FNAL/584/LoadTest07_PIC_2A_amrM3i4bWVBe0f1w_584 FILESIZE=973217792 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=1.13 CRC= STATUS=KEYERROR Configuration Server: no such name: 'NotWrittenToTape.library_manager' 00:00:50 stkendca22a.fnal.gov 007962 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: garbage Storage Group: test File Family: dcache FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0003 --put-cache 0005000000000000005FA5F8 /diskb/write-pool-2/data/0005000000000000005FA5F8 00:00:50 stkendca22a.fnal.gov 008002 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: garbage Storage Group: test File Family: dcache FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0001 --put-cache 0005000000000000005FA590 /diskb/write-pool-2/data/0005000000000000005FA590 00:00:50 stkendca22a.fnal.gov 008019 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: garbage Storage Group: test File Family: dcache FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0002 --put-cache 0005000000000000005FA5C8 /diskb/write-pool-2/data/0005000000000000005FA5C8 00:00:50 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOQ856 0,6,1,1 00:00:50 stkenmvr205a.fnal.gov 011191 root E LTO4_105MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:50 stkenmvr205a.fnal.gov 011191 root I LTO4_105MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:50 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOQ834 0,7,1,14 00:00:50 stkenmvr195a.fnal.gov 021479 root E LTO4_95MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:50 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:50 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:50 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ825 0,6,1,8 00:00:50 stkenmvr206a.fnal.gov 026766 root E LTO4_106MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:50 stkenmvr206a.fnal.gov 026766 root I LTO4_106MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:50 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ859 0,7,1,3 00:00:50 stkenmvr141a.fnal.gov 013963 root E LTO4_41MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:50 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:50 stkendca22a.fnal.gov 007962 root I ENCP Configuration Server: no such name: 'garbage.library_manager' 00:00:50 stkendca22a.fnal.gov 007962 root E ENCP INFILE=/diskb/write-pool-2/data/0005000000000000005FA5F8 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0003 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=0.93 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager' 00:00:50 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV filemarks written. Tape VOQ828 absolute location in blocks 3159683 Thread tape_thread 00:00:50 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV write_tape timing:pull 0.316775560379 write 6.87404632568 crc_check 18.0954883099 freespace 0.345195770264 block_write 25.6315059662 idle 0.58901309967 Thread tape_thread 00:00:50 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV drive stats after write. Tape VOQ828 position 137 block 0 block_size 0 bloc_loc 3159683 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 3933111296 block_write_tot 25.6315059662 tape_rate 153448310.887 Thread tape_thread 00:00:50 stkendca22a.fnal.gov 008002 root I ENCP Configuration Server: no such name: 'garbage.library_manager' 00:00:50 stkendca22a.fnal.gov 008002 root E ENCP INFILE=/diskb/write-pool-2/data/0005000000000000005FA590 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0001 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=0.98 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager' 00:00:50 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV remaining bytes info in DB 393611264000 estimated 389678152704 reported from drive 389957632000 position 138 Thread tape_thread 00:00:50 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV new bitfile request {'uid': 0, 'external_label': 'VOQ828', 'gid': 0, 'location_cookie': '0000_000000000_0000137', 'complete_crc': 486118760L, 'sanity_cookie': (65536L, 12545026L), 'size': 3933110695L} Thread tape_thread 00:00:50 stkendca22a.fnal.gov 008019 root I ENCP Configuration Server: no such name: 'garbage.library_manager' 00:00:50 stkendca22a.fnal.gov 008019 root E ENCP INFILE=/diskb/write-pool-2/data/0005000000000000005FA5C8 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0002 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=0.98 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager' 00:00:50 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV set remaining: VOQ828 389957632000 0000_000000000_0000138 Thread tape_thread 00:00:50 last message repeated 1 times 00:00:50 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-01 00:00:50', eod_cookie = '0000_000000000_0000138', non_del_files = '137', remaining_bytes = 389957632000 where label = 'VOQ828'; 00:00:50 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV transfer complete volume=VOQ828 location=138 Thread tape_thread 00:00:50 last message repeated 1 times 00:00:50 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '137', modification_time = '2009-11-01 00:00:50' where label = 'VOQ828'; 00:00:50 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1255358447.0, 'library': 'CD-LTO4G1', 'si_time': [1241450288.0, 1241507765.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOQ828', 'wrapper': 'cpio_odc', 'remaining_bytes': 393611264000L, 'sum_mounts': 103, 'sum_rd_access': 0, 'file_family_width': 15, 'last_access': 1257051611.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000137', 'capacity_bytes': 858993459200L, 'non_del_files': 136, 'sum_wr_err': 0, 'sum_wr_access': 136, 'file_family': 'BACKFILL2', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.BACKFILL2.cpio_odc', 'modification_time': 1257051611.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1256991855.0}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/346F128B-0EC6-DE11-AA76-0019B9F72BFF.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 3933110695L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7855, 'mc_device': '0,7,1,7', 'compression': 0, 'serial_num': '9310800990', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr194a', 'max_time_in_state': 1200, 'device': 'stkenmvr194a:/dev/rmt/tps0d0n', 'name': 'LTO4_94.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_94.stat', 'mover_address': ('131.225.13.189', 7855), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.189', 54723), 'data_ip': 'stkenmvr194a', 'logname': 'LTO4_94MV', 'hostip': '131.225.13.189', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/346F128B-0EC6-DE11-AA76-0019B9F72BFF.root', 'uid': 0, 'uname': 'root', 'mtime': 1257051615, 'rmajor': 0, 'machine': ('Linux', 'cmsstor224.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 125853680, 24L, 1, 9811, 5063, 512, 1257051251, 1257051251, 1257051006), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 3933110695L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data1/write-pool-1/data/000700000000000061806490', 'inode': 125854864L, 'minor': 24}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/000700000000000061806490', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(000700000000000061806490)', 'fc': {'uid': 0, 'external_label': 'VOQ828', 'gid': 0, 'location_cookie': '0000_000000000_0000137', 'complete_crc': 486118760L, 'bfid': 'CDMS125705165000000', 'sanity_cookie': (65536L, 12545026L), 'size': 3933110695L}, 'infilepath': '/storage/data1/write-pool-1/data/000700000000000061806490', 'at_the_top': 1, 'local_inode': 9571, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.191.96', 50854), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257051614.880209, 'seek_time': 0.029710054397583008, 'transfer_time': 34.533056974411011, 't0': 1257051614, 'mount_time':(trunc. 4365) Thread tape_thread 00:00:50 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV Done is sent Thread tape_thread 00:00:50 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV maybe_clean Thread tape_thread 00:00:50 cmsstor224.fnal.gov 013968 root I ENCP Received final dialog for cmsstor224.fnal.gov-1257051615-13968-0. elapsed=35.850sec 00:00:50 cmsstor224.fnal.gov 013968 root I ENCP leaving transfer_file(). 00:00:50 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV returned from maybe_clean Thread tape_thread 00:00:50 cmsstor224.fnal.gov 013968 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:50 stkensrv2n.fnal.gov 006421 enstore E INQSRV Could not create seen-down file lock file (/srv2/enstore/www/web-pages//enstore_seen_down.py.lock) 00:00:51 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/346F128B-0EC6-DE11-AA76-0019B9F72BFF.root', drive = 'stkenmvr194a:/dev/rmt/tps0d0n:9310800990', pnfs_id = '000700000000000061806490' where bfid = 'CDMS125705165000000'; 00:00:51 stkensrv2n.fnal.gov 006421 enstore E INQSRV Could not read file /srv2/enstore/www/web-pages//enstore_seen_down.py 00:00:51 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:51 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VOQ828 mover=LTO4_94.mover requester:root 00:00:51 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/121A5A58-F8C5-DE11-AA61-001D09F29849.root to mover LTO4_94.mover ('131.225.13.189', 7855) DEL_DISM None 00:00:51 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:51 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'cms', 'external_label': 'VOQ828', 'library': 'CD-LTO4G1', 'file_family': 'BACKFILL2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.BACKFILL2.cpio_odc', 'file_family_width': 15}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/121A5A58-F8C5-DE11-AA61-001D09F29849.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2670926470L, 'mover': 'LTO4_94.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/121A5A58-F8C5-DE11-AA61-001D09F29849.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257051633, 'rmajor': 0, 'machine': ('Linux', 'cmsstor184.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16877, 125854968, 21L, 1, 9811, 5063, 512, 1257051362, 1257051362, 1257051186), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 2670926470L, 'fullname': '/storage/data2/write-pool-2/data/000700000000000061806760', 'inode': 125855584L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/000700000000000061806760', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/.(access)(000700000000000061806760)', 'fc': {'size': 2670926470L, 'external_label': 'VOQ828', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data2/write-pool-2/data/000700000000000061806760', 'at_the_top': 26, 'local_inode': 2788, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.207.5', 45635), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051651.189909, 'encp_start_time': 1257051632.6476951, 'job_queued': 1257051633.345377, 't0': 1257051632, 'in_queue': 17.844249963760376}, 'user_level': 'admin', 'unique_id': 'cmsstor184.fnal.gov-1257051633-5655-0'} to mover 00:00:51 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:51 cmsstor224.fnal.gov 013968 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /storage/data1/write-pool-1/data/000700000000000061806490 -> /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/346F128B-0EC6-DE11-AA76-0019B9F72BFF.root: 3933110695 bytes copied to VOQ828 at 109 MB/S (97.4 MB/S network) (135 MB/S drive) (112 MB/S disk) mover=LTO4_94.mover drive_id=ULTRIUM-TD4 drive_sn=9310800990 drive_vendor=IBM elapsed=36.319 {'storage_group': 'cms', 'encp_crc': 486118760L, 'disk_rate': 112.49354356573131, 'drive_rate': 135.47237708399919, 'driver': 'FTTDriver', 'transfer_rate': 108.57173744658745, 'mover_interface': 'stkenmvr194a', 'overall_rate': 106.92706371240915, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 108.57173744658745, 'encp_ip': '131.225.191.96', 'unique_id': 'cmsstor224.fnal.gov-1257051615-13968-0'} 00:00:51 cmsstor224.fnal.gov 013968 root I ENCP File /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/346F128B-0EC6-DE11-AA76-0019B9F72BFF.root copied successfully. 00:00:51 cmsstor224.fnal.gov 013968 root I ENCP Completed transferring 3933110695 bytes in 1 files in 36.3204290867 sec. Overall rate = 107 MB/sec. Transfer rate = 109 MB/sec. Network rate = 97.4 MB/sec. Drive rate = 135 MB/sec. Disk rate = 112 MB/sec. Exit status = 0. 00:00:51 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV WRITE_TO_HSM Thread MainThread 00:00:51 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV SENDING {'vc': {'storage_group': 'cms', 'external_label': 'VOQ828', 'library': 'CD-LTO4G1', 'file_family': 'BACKFILL2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.BACKFILL2.cpio_odc', 'file_family_width': 15}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/121A5A58-F8C5-DE11-AA61-001D09F29849.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2670926470L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7855, 'mc_device': '0,7,1,7', 'compression': 0, 'serial_num': '9310800990', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr194a', 'max_time_in_state': 1200, 'device': 'stkenmvr194a:/dev/rmt/tps0d0n', 'name': 'LTO4_94.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_94.stat', 'mover_address': ('131.225.13.189', 7855), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.189', 54730), 'data_ip': 'stkenmvr194a', 'logname': 'LTO4_94MV', 'hostip': '131.225.13.189', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/121A5A58-F8C5-DE11-AA61-001D09F29849.root', 'uid': 0, 'uname': 'root', 'mtime': 1257051633, 'rmajor': 0, 'machine': ('Linux', 'cmsstor184.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 125854968, 21L, 1, 9811, 5063, 512, 1257051362, 1257051362, 1257051186), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 2670926470L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data2/write-pool-2/data/000700000000000061806760', 'inode': 125855584L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/000700000000000061806760', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/.(access)(000700000000000061806760)', 'fc': {'size': 2670926470L, 'external_label': 'VOQ828', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data2/write-pool-2/data/000700000000000061806760', 'at_the_top': 26, 'local_inode': 2788, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.207.5', 45635), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051651.189909, 'encp_start_time': 1257051632.6476951, 'job_queued': 1257051633.345377, 't0': 1257051632, 'in_queue': 17.844249963760376}, 'user_level': 'admin', 'unique_id': 'cmsstor184.fnal.gov-1257051633-5655-0'} Thread client_connect_thread 00:00:51 cmsstor184.fnal.gov 005655 root I ENCP Control socket ('131.225.207.5', 45635) is connected to ('131.225.13.189', 54731) for cmsstor184.fnal.gov-1257051633-5655-0. elapsed=18.560sec 00:00:51 cmsstor184.fnal.gov 005655 root I ENCP Received callback ticket from mover LTO4_94.mover for transfer cmsstor184.fnal.gov-1257051633-5655-0. elapsed=18.570sec 00:00:51 cmsstor184.fnal.gov 005655 root I ENCP Atempting to connect data socket to mover at ('131.225.13.189', 54730). 00:00:51 cmsstor184.fnal.gov 005655 root I ENCP Connecting data socket to mover (('131.225.13.189', 54730)) with interface 131.225.207.5. elapsed=18.570sec 00:00:51 cmsstor184.fnal.gov 005655 root I ENCP Data socket ('131.225.207.5', 56747) is connected to ('131.225.13.189', 54730). elapsed=18.570sec 00:00:51 cmsstor184.fnal.gov 005655 root I ENCP Opening local file. 00:00:51 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV tape position: target 138 current 138 0 Thread seek_thread 00:00:51 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV write_tape starting, bytes_to_write=2670926848 Thread tape_thread 00:00:51 cmsstor184.fnal.gov 005655 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:51 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV Write starting. Tape VOQ828 absolute location in blocks 3159683 Thread tape_thread 00:00:51 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:51 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:52 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ922 0,7,1,2 00:00:52 stkenmvr196a.fnal.gov 027637 root E LTO4_96MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:52 stkenmvr196a.fnal.gov 027637 root I LTO4_96MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:52 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:52 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:52 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ827 0,7,1,11 00:00:52 stkenmvr193a.fnal.gov 031647 root E LTO4_93MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:52 stkenmvr193a.fnal.gov 031647 root I LTO4_93MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:52 stkensrv4n.fnal.gov 019877 enstore I SL8500G1MC dismount VOLUME 0,6,1,0 force => 0,Dismount: Forced dismount of VOU664 from 0, 6, 1, 0 00:00:52 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC FINISHED dismount VOU664 0,6,1,0 returned ('ok', 0, 'dismount VOLUME 0,6,1,0 force => 0,Dismount: Forced dismount of VOU664 from 0, 6, 1, 0', '', '') 00:00:52 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV dismounted VOU664 ULTRIUM-TD4 00:00:52 Thread media_thread 00:00:53 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC PROCESSING mount VOQ857 0,7,1,0 00:00:53 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOS175 0,7,1,1 00:00:53 stkenmvr198a.fnal.gov 015662 root E LTO4_98MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:53 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:53 cmsstor60.fnal.gov 028299 root I ENCP File /storage/data3/write-pool-3/data/0007000000000000618028A0 transfered for cmsstor60.fnal.gov-1257050914-28299-0 in 34.1886777878 seconds. 00:00:53 stkensrv2n.fnal.gov 007713 root E MNTR_SRV Error extablishing connection: (, MonitorError: Connection timed out) 00:00:53 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:54 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOS188 0,6,1,12 00:00:54 stkenmvr146a.fnal.gov 005990 root E LTO4_46MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:54 stkenmvr146a.fnal.gov 005990 root I LTO4_46MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:54 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:00:54 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:54 stkensrv2n.fnal.gov 006421 enstore E INQSRV Could not create seen-down file lock file (/srv2/enstore/www/web-pages//enstore_seen_down.py.lock) 00:00:54 stkensrv2n.fnal.gov 006421 enstore E INQSRV Could not read file /srv2/enstore/www/web-pages//enstore_seen_down.py 00:00:55 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ873 0,5,1,11 00:00:55 stkenmvr208a.fnal.gov 006111 root E LTO4_108MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:55 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:55 stkenmvr151a.fnal.gov 013228 root I LTO4_51MV written bytes 3929902080/3929902080, blocks 29983 header 285 trailer 405 Thread tape_thread 00:00:55 stkensrv2n.fnal.gov 006421 enstore E INQSRV Could not create outage file lock file (/srv2/enstore/www/web-pages//enstore_outage.py.lock) 00:00:55 stkensrv4n.fnal.gov 020283 enstore I SL8500G1MC query drive 0,7,1,0 => 0,0,7,1,0 online available IBM-LTO4 00:00:55 stkensrv4n.fnal.gov 020283 enstore I SL8500G1MC Dismount 14 ignored: Drive 0,7,1,0 is empty. Thought VOQ857 was there =>. dismount VOLUME 0,7,1,0 force => ok,0,7,1,0 online available IBM-LTO4 00:00:55 cmsstor103.fnal.gov 014386 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms2 FF Wrapper: cpio_odc FF Width: 7 Current working directory: cmsstor103.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --age-time 30 --delpri 1 --priority 25 --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/PATLayer1_RECO_Output_QCDDiJet_Pt1400to1800_20.root --put-cache 0010000000000000094DDBA8 /storage/data3/write-pool-3/data/0010000000000000094DDBA8 00:00:55 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:56 cmsstor103.fnal.gov 014386 root I ENCP Creating zero length output files. 00:00:56 cmsstor103.fnal.gov 014386 root I ENCP Sending /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/PATLayer1_RECO_Output_QCDDiJet_Pt1400to1800_20.root write request to LM: unique_id: cmsstor103.fnal.gov-1257051656-14386-0 inputfile: /storage/data3/write-pool-3/data/0010000000000000094DDBA8 outputfile: /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/.(access)(0010000000000000094DDBA8) 00:00:56 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:56 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data3/write-pool-3/data/0010000000000000094DDBA8 (cmsstor103.fnal.gov-1257051656-14386-0) -> /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/PATLayer1_RECO_Output_QCDDiJet_Pt1400to1800_20.root : library=CD-LTO4G1 family=cms2 requester:root volume_family:cms.cms2.cpio_odc 00:00:56 cmsstor103.fnal.gov 014386 root I ENCP Listening for control socket at: ('131.225.205.225', 51888) 00:00:56 cmsstor103.fnal.gov 014386 root I ENCP Waiting for mover to connect control socket at address: ('131.225.205.225', 51888) elapsed=0.950sec 00:00:56 stkensrv2n.fnal.gov 006421 enstore E INQSRV Could not read file /srv2/enstore/www/web-pages//enstore_outage.py 00:00:56 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:56 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VOU867 mover=LTO4_104.mover requester:root 00:00:56 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/PATLayer1_RECO_Output_QCDDiJet_Pt1400to1800_20.root to mover LTO4_104.mover ('131.225.13.199', 7864) DEL_DISM None 00:00:56 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:56 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'cms', 'external_label': 'VOU867', 'library': 'CD-LTO4G1', 'file_family': 'cms2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms2.cpio_odc', 'file_family_width': 7}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/PATLayer1_RECO_Output_QCDDiJet_Pt1400to1800_20.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 410026139L, 'mover': 'LTO4_104.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/PATLayer1_RECO_Output_QCDDiJet_Pt1400to1800_20.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257051656, 'rmajor': 0, 'machine': ('Linux', 'cmsstor103.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16893, 273340320, 21L, 1, 12124, 5063, 512, 1257050578, 1257050578, 1257036570), 'mode_octal': '0100036', 'mode': 33188, 'gname': 'root', 'size_bytes': 410026139L, 'fullname': '/storage/data3/write-pool-3/data/0010000000000000094DDBA8', 'inode': 273537960L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/0010000000000000094DDBA8', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/.(access)(0010000000000000094DDBA8)', 'fc': {'size': 410026139L, 'external_label': 'VOU867', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data3/write-pool-3/data/0010000000000000094DDBA8', 'at_the_top': 1, 'local_inode': 1073743330, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.205.225', 51888), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051656.30022, 'encp_start_time': 1257051655.249567, 'job_queued': 1257051656.15483, 't0': 1257051655, 'in_queue': 0.14533305168151855}, 'user_level': 'admin', 'unique_id': 'cmsstor103.fnal.gov-1257051656-14386-0'} to mover 00:00:56 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:56 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV WRITE_TO_HSM Thread MainThread 00:00:56 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV SENDING {'vc': {'storage_group': 'cms', 'external_label': 'VOU867', 'library': 'CD-LTO4G1', 'file_family': 'cms2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms2.cpio_odc', 'file_family_width': 7}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/PATLayer1_RECO_Output_QCDDiJet_Pt1400to1800_20.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 410026139L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7864, 'mc_device': '0,6,1,13', 'compression': 0, 'serial_num': '1310050819', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr204a', 'max_time_in_state': 1200, 'device': 'stkenmvr204a:/dev/rmt/tps0d0n', 'name': 'LTO4_104.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_104.stat', 'mover_address': ('131.225.13.199', 7864), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.199', 54009), 'data_ip': 'stkenmvr204a', 'logname': 'LTO4_104MV', 'hostip': '131.225.13.199', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/PATLayer1_RECO_Output_QCDDiJet_Pt1400to1800_20.root', 'uid': 0, 'uname': 'root', 'mtime': 1257051656, 'rmajor': 0, 'machine': ('Linux', 'cmsstor103.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 273340320, 21L, 1, 12124, 5063, 512, 1257050578, 1257050578, 1257036570), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 410026139L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data3/write-pool-3/data/0010000000000000094DDBA8', 'inode': 273537960L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/0010000000000000094DDBA8', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/.(access)(0010000000000000094DDBA8)', 'fc': {'size': 410026139L, 'external_label': 'VOU867', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data3/write-pool-3/data/0010000000000000094DDBA8', 'at_the_top': 1, 'local_inode': 1073743330, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.205.225', 51888), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051656.30022, 'encp_start_time': 1257051655.249567, 'job_queued': 1257051656.15483, 't0': 1257051655, 'in_queue': 0.14533305168151855}, 'user_level': 'admin', 'unique_id': 'cmsstor103.fnal.gov-1257051656-14386-0'} Thread client_connect_thread 00:00:56 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:56 cmsstor103.fnal.gov 014386 root I ENCP Control socket ('131.225.205.225', 51888) is connected to ('131.225.13.199', 54010) for cmsstor103.fnal.gov-1257051656-14386-0. elapsed=1.090sec 00:00:56 cmsstor103.fnal.gov 014386 root I ENCP Received callback ticket from mover LTO4_104.mover for transfer cmsstor103.fnal.gov-1257051656-14386-0. elapsed=1.100sec 00:00:56 cmsstor103.fnal.gov 014386 root I ENCP Atempting to connect data socket to mover at ('131.225.13.199', 54009). 00:00:56 cmsstor103.fnal.gov 014386 root I ENCP Connecting data socket to mover (('131.225.13.199', 54009)) with interface 131.225.205.225. elapsed=1.100sec 00:00:56 cmsstor103.fnal.gov 014386 root I ENCP Data socket ('131.225.205.225', 56678) is connected to ('131.225.13.199', 54009). elapsed=1.100sec 00:00:56 cmsstor103.fnal.gov 014386 root I ENCP Opening local file. 00:00:56 last message repeated 1 times 00:00:56 cmsstor103.fnal.gov 014386 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:56 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV tape position: target 592 current 592 0 Thread seek_thread 00:00:56 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV write_tape starting, bytes_to_write=410026496 Thread tape_thread 00:00:56 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV Write starting. Tape VOU867 absolute location in blocks 731193 Thread tape_thread 00:00:56 stkensrv2n.fnal.gov 006421 enstore E INQSRV Could not create seen-down file lock file (/srv2/enstore/www/web-pages//enstore_seen_down.py.lock) 00:00:56 stkensrv2n.fnal.gov 006421 enstore E INQSRV Could not read file /srv2/enstore/www/web-pages//enstore_seen_down.py 00:00:56 stkenmvr151a.fnal.gov 013228 root I LTO4_51MV filemarks written. Tape VOQ870 absolute location in blocks 803168 Thread tape_thread 00:00:56 stkenmvr151a.fnal.gov 013228 root I LTO4_51MV write_tape timing:pull 0.404633760452 write 15.9800014496 crc_check 15.8100817204 freespace 0.436969280243 block_write 32.6316862106 idle 0.66366314888 Thread tape_thread 00:00:56 stkenmvr151a.fnal.gov 013228 root I LTO4_51MV drive stats after write. Tape VOQ870 position 37 block 0 block_size 0 bloc_loc 803168 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 3929902080 block_write_tot 32.6316862106 tape_rate 120432087.224 Thread tape_thread 00:00:56 stkenmvr151a.fnal.gov 013228 root I LTO4_51MV remaining bytes info in DB 685006848000 estimated 681076945920 reported from drive 681372672000 position 38 Thread tape_thread 00:00:56 stkenmvr151a.fnal.gov 013228 root I LTO4_51MV new bitfile request {'uid': 0, 'external_label': 'VOQ870', 'gid': 0, 'location_cookie': '0000_000000000_0000037', 'complete_crc': 3228128717L, 'sanity_cookie': (65536L, 1856294363L), 'size': 3929901390L} Thread tape_thread 00:00:56 stkenmvr151a.fnal.gov 013228 root I LTO4_51MV set remaining: VOQ870 681372672000 0000_000000000_0000038 Thread tape_thread 00:00:56 last message repeated 1 times 00:00:56 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-01 00:00:56', eod_cookie = '0000_000000000_0000038', non_del_files = '37', remaining_bytes = 681372672000 where label = 'VOQ870'; 00:00:57 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:57 stkenmvr151a.fnal.gov 013228 root I LTO4_51MV transfer complete volume=VOQ870 location=38 Thread tape_thread 00:00:57 last message repeated 1 times 00:00:57 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '37', last_access = '2009-11-01 00:00:57', modification_time = '2009-11-01 00:00:57' where label = 'VOQ870'; 00:00:57 stkenmvr151a.fnal.gov 013228 root I LTO4_51MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1255358549.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 1244673789.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOQ870', 'wrapper': 'cpio_odc', 'remaining_bytes': 685006848000L, 'sum_mounts': 276, 'sum_rd_access': 0, 'file_family_width': 15, 'last_access': 1257050111.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000037', 'capacity_bytes': 858993459200L, 'non_del_files': 36, 'sum_wr_err': 0, 'sum_wr_access': 36, 'file_family': 'BACKFILL2', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.BACKFILL2.cpio_odc', 'modification_time': 1257050111.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257039578.0}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/005/94D9FF44-EDC5-DE11-9DE6-000423D944FC.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 3929901390L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7851, 'mc_device': '0,5,1,12', 'compression': 0, 'serial_num': '1310154445', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr151a', 'max_time_in_state': 1200, 'device': 'stkenmvr151a:/dev/rmt/tps0d0n', 'name': 'LTO4_51.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_51.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.179', 7851), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.179', 54523), 'data_ip': 'stkenmvr151a', 'logname': 'LTO4_51MV', 'hostip': '131.225.13.179', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/005/94D9FF44-EDC5-DE11-9DE6-000423D944FC.root', 'uid': 0, 'uname': 'root', 'mtime': 1257050914, 'rmajor': 0, 'machine': ('Linux', 'cmsstor60.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 125621296, 23L, 1, 9811, 5063, 512, 1257050632, 1257050632, 1257038274), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 3929901390L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data3/write-pool-3/data/0007000000000000618028A0', 'inode': 125839520L, 'minor': 23}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/0007000000000000618028A0', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/005/.(access)(0007000000000000618028A0)', 'fc': {'uid': 0, 'external_label': 'VOQ870', 'gid': 0, 'location_cookie': '0000_000000000_0000037', 'complete_crc': 3228128717L, 'bfid': 'CDMS125705165600000', 'sanity_cookie': (65536L, 1856294363L), 'size': 3929901390L}, 'infilepath': '/storage/data3/write-pool-3/data/0007000000000000618028A0', 'at_the_top': 3, 'local_inode': 1181, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.189.60', 34270), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257050913.7947941, 'seek_time': 493.08798885345459, 'transfer_time': 38.0576791763(trunc. 4364) Thread tape_thread 00:00:57 stkenmvr151a.fnal.gov 013228 root I LTO4_51MV Done is sent Thread tape_thread 00:00:57 stkenmvr151a.fnal.gov 013228 root I LTO4_51MV maybe_clean Thread tape_thread 00:00:57 stkenmvr151a.fnal.gov 013228 root I LTO4_51MV returned from maybe_clean Thread tape_thread 00:00:57 cmsstor60.fnal.gov 028299 root I ENCP Received final dialog for cmsstor60.fnal.gov-1257050914-28299-0. elapsed=743.370sec 00:00:57 cmsstor60.fnal.gov 028299 root I ENCP leaving transfer_file(). 00:00:57 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:57 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VOQ870 mover=LTO4_51.mover requester:root 00:00:57 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/D2772D1F-0EC6-DE11-A0FE-0016177CA7A0.root to mover LTO4_51.mover ('131.225.13.179', 7851) DEL_DISM None 00:00:57 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:57 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'cms', 'external_label': 'VOQ870', 'library': 'CD-LTO4G1', 'file_family': 'BACKFILL2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.BACKFILL2.cpio_odc', 'file_family_width': 15}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/D2772D1F-0EC6-DE11-A0FE-0016177CA7A0.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 3996954709L, 'mover': 'LTO4_51.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/D2772D1F-0EC6-DE11-A0FE-0016177CA7A0.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257051644, 'rmajor': 0, 'machine': ('Linux', 'cmsstor229.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16877, 125853680, 23L, 1, 9811, 5063, 512, 1257051251, 1257051251, 1257051006), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 3996954709L, 'fullname': '/storage/data1/write-pool-1/data/000700000000000061806428', 'inode': 125854760L, 'minor': 23}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/000700000000000061806428', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(000700000000000061806428)', 'fc': {'size': 3996954709L, 'external_label': 'VOQ870', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data1/write-pool-1/data/000700000000000061806428', 'at_the_top': 26, 'local_inode': 3922, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.191.101', 39791), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051657.2062719, 'encp_start_time': 1257051644.183953, 'job_queued': 1257051644.9537511, 't0': 1257051644, 'in_queue': 12.252477884292603}, 'user_level': 'admin', 'unique_id': 'cmsstor229.fnal.gov-1257051644-32233-0'} to mover 00:00:57 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:57 stkenmvr151a.fnal.gov 013228 root I LTO4_51MV WRITE_TO_HSM Thread MainThread 00:00:57 stkenmvr151a.fnal.gov 013228 root I LTO4_51MV SENDING {'vc': {'storage_group': 'cms', 'external_label': 'VOQ870', 'library': 'CD-LTO4G1', 'file_family': 'BACKFILL2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.BACKFILL2.cpio_odc', 'file_family_width': 15}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/D2772D1F-0EC6-DE11-A0FE-0016177CA7A0.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 3996954709L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7851, 'mc_device': '0,5,1,12', 'compression': 0, 'serial_num': '1310154445', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr151a', 'max_time_in_state': 1200, 'device': 'stkenmvr151a:/dev/rmt/tps0d0n', 'name': 'LTO4_51.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_51.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.179', 7851), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.179', 54543), 'data_ip': 'stkenmvr151a', 'logname': 'LTO4_51MV', 'hostip': '131.225.13.179', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/D2772D1F-0EC6-DE11-A0FE-0016177CA7A0.root', 'uid': 0, 'uname': 'root', 'mtime': 1257051644, 'rmajor': 0, 'machine': ('Linux', 'cmsstor229.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 125853680, 23L, 1, 9811, 5063, 512, 1257051251, 1257051251, 1257051006), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 3996954709L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data1/write-pool-1/data/000700000000000061806428', 'inode': 125854760L, 'minor': 23}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/000700000000000061806428', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(000700000000000061806428)', 'fc': {'size': 3996954709L, 'external_label': 'VOQ870', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data1/write-pool-1/data/000700000000000061806428', 'at_the_top': 26, 'local_inode': 3922, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.191.101', 39791), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051657.2062719, 'encp_start_time': 1257051644.183953, 'job_queued': 1257051644.9537511, 't0': 1257051644, 'in_queue': 12.252477884292603}, 'user_level': 'admin', 'unique_id': 'cmsstor229.fnal.gov-1257051644-32233-0'} Thread client_connect_thread 00:00:57 cmsstor229.fnal.gov 032233 root I ENCP Control socket ('131.225.191.101', 39791) is connected to ('131.225.13.179', 54544) for cmsstor229.fnal.gov-1257051644-32233-0. elapsed=13.060sec 00:00:57 cmsstor229.fnal.gov 032233 root I ENCP Received callback ticket from mover LTO4_51.mover for transfer cmsstor229.fnal.gov-1257051644-32233-0. elapsed=13.070sec 00:00:57 cmsstor229.fnal.gov 032233 root I ENCP Atempting to connect data socket to mover at ('131.225.13.179', 54543). 00:00:57 cmsstor229.fnal.gov 032233 root I ENCP Connecting data socket to mover (('131.225.13.179', 54543)) with interface 131.225.191.101. elapsed=13.070sec 00:00:57 cmsstor229.fnal.gov 032233 root I ENCP Data socket ('131.225.191.101', 38493) is connected to ('131.225.13.179', 54543). elapsed=13.070sec 00:00:57 cmsstor229.fnal.gov 032233 root I ENCP Opening local file. 00:00:57 cmsstor60.fnal.gov 028299 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:57 cmsstor229.fnal.gov 032233 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:57 stkenmvr151a.fnal.gov 013228 root I LTO4_51MV tape position: target 38 current 38 0 Thread seek_thread 00:00:57 stkenmvr151a.fnal.gov 013228 root I LTO4_51MV write_tape starting, bytes_to_write=3996955136 Thread tape_thread 00:00:57 stkenmvr151a.fnal.gov 013228 root I LTO4_51MV Write starting. Tape VOQ870 absolute location in blocks 803168 Thread tape_thread 00:00:57 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/005/94D9FF44-EDC5-DE11-9DE6-000423D944FC.root', drive = 'stkenmvr151a:/dev/rmt/tps0d0n:1310154445', pnfs_id = '0007000000000000618028A0' where bfid = 'CDMS125705165600000'; 00:00:57 cmsstor60.fnal.gov 028299 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /storage/data3/write-pool-3/data/0007000000000000618028A0 -> /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/005/94D9FF44-EDC5-DE11-9DE6-000423D944FC.root: 3929901390 bytes copied to VOQ870 at 98.4 MB/S (90.7 MB/S network) (106 MB/S drive) (110 MB/S disk) mover=LTO4_51.mover drive_id=ULTRIUM-TD4 drive_sn=1310154445 drive_vendor=IBM elapsed=743.91 {'storage_group': 'cms', 'encp_crc': 3228128717L, 'disk_rate': 109.76029711657003, 'drive_rate': 106.43854634611068, 'driver': 'FTTDriver', 'transfer_rate': 98.353705115829911, 'mover_interface': 'stkenmvr151a', 'overall_rate': 7.0473950261201193, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 98.353705115829911, 'encp_ip': '131.225.189.60', 'unique_id': 'cmsstor60.fnal.gov-1257050914-28299-0'} 00:00:57 cmsstor60.fnal.gov 028299 root I ENCP File /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/005/94D9FF44-EDC5-DE11-9DE6-000423D944FC.root copied successfully. 00:00:57 cmsstor60.fnal.gov 028299 root I ENCP Completed transferring 3929901390 bytes in 1 files in 743.912475824 sec. Overall rate = 7.05 MB/sec. Transfer rate = 98.4 MB/sec. Network rate = 90.7 MB/sec. Drive rate = 106 MB/sec. Disk rate = 110 MB/sec. Exit status = 0. 00:00:57 stkenmvr216a.fnal.gov 007455 root I LTO4_116MV tape position: target 4846 current 0 1 Thread seek_thread 00:00:57 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:58 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:59 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VON584 0,6,1,7 00:00:59 stkenmvr201a.fnal.gov 018563 root E LTO4_101MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:59 stkenmvr201a.fnal.gov 018563 root I LTO4_101MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:59 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOU507 0,5,1,2 00:00:59 stkenmvr150a.fnal.gov 006006 root E LTO4_50MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:00:59 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:00:59 cmsstor103.fnal.gov 014386 root I ENCP File /storage/data3/write-pool-3/data/0010000000000000094DDBA8 transfered for cmsstor103.fnal.gov-1257051656-14386-0 in 3.45974898338 seconds. 00:00:59 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:00 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOQ856 0,6,1,1 00:01:00 stkenmvr205a.fnal.gov 011191 root E LTO4_105MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:00 stkenmvr205a.fnal.gov 011191 root I LTO4_105MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:00 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOQ834 0,7,1,14 00:01:00 stkenmvr195a.fnal.gov 021479 root E LTO4_95MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:00 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:00 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ825 0,6,1,8 00:01:00 stkenmvr206a.fnal.gov 026766 root E LTO4_106MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:00 stkenmvr206a.fnal.gov 026766 root I LTO4_106MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:00 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV written bytes 410026496/410026496, blocks 3129 header 238 trailer 119 Thread tape_thread 00:01:00 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ859 0,7,1,3 00:01:00 stkenmvr141a.fnal.gov 013963 root E LTO4_41MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:00 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:00 stkensrv4n.fnal.gov 020283 enstore I SL8500G1MC query vol VOQ857 => 0,VOQ857 home 0, 5,18, 5, 0 LTO-800G 00:01:01 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV filemarks written. Tape VOU867 absolute location in blocks 734323 Thread tape_thread 00:01:01 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV write_tape timing:pull 0.0341439247131 write 1.37372994423 crc_check 1.91356182098 freespace 0.0378272533417 block_write 3.35926294327 idle 0.0631442070007 Thread tape_thread 00:01:01 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV drive stats after write. Tape VOU867 position 592 block 0 block_size 0 bloc_loc 734323 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 410026496 block_write_tot 3.35926294327 tape_rate 122058470.243 Thread tape_thread 00:01:01 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV remaining bytes info in DB 688189440000 estimated 687779413504 reported from drive 687804416000 position 593 Thread tape_thread 00:01:01 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV new bitfile request {'uid': 0, 'external_label': 'VOU867', 'gid': 0, 'location_cookie': '0000_000000000_0000592', 'complete_crc': 14197294L, 'sanity_cookie': (65536L, 4222365876L), 'size': 410026139L} Thread tape_thread 00:01:01 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:02 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ922 0,7,1,2 00:01:02 stkenmvr196a.fnal.gov 027637 root E LTO4_96MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:02 stkenmvr196a.fnal.gov 027637 root I LTO4_96MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:02 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV set remaining: VOU867 687804416000 0000_000000000_0000593 Thread tape_thread 00:01:02 last message repeated 1 times 00:01:02 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-01 00:01:02', eod_cookie = '0000_000000000_0000593', non_del_files = '594', remaining_bytes = 687804416000 where label = 'VOU867'; 00:01:02 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV transfer complete volume=VOU867 location=593 Thread tape_thread 00:01:02 last message repeated 1 times 00:01:02 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '592', modification_time = '2009-11-01 00:01:02' where label = 'VOU867'; 00:01:02 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1244479824.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOU867', 'wrapper': 'cpio_odc', 'remaining_bytes': 688189440000L, 'sum_mounts': 22, 'sum_rd_access': 0, 'file_family_width': 7, 'last_access': 1257051645.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000592', 'capacity_bytes': 858993459200L, 'non_del_files': 593, 'sum_wr_err': 0, 'sum_wr_access': 591, 'file_family': 'cms2', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms2.cpio_odc', 'modification_time': 1257051645.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1256259225.0}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/PATLayer1_RECO_Output_QCDDiJet_Pt1400to1800_20.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 410026139L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7864, 'mc_device': '0,6,1,13', 'compression': 0, 'serial_num': '1310050819', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr204a', 'max_time_in_state': 1200, 'device': 'stkenmvr204a:/dev/rmt/tps0d0n', 'name': 'LTO4_104.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_104.stat', 'mover_address': ('131.225.13.199', 7864), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.199', 54009), 'data_ip': 'stkenmvr204a', 'logname': 'LTO4_104MV', 'hostip': '131.225.13.199', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/PATLayer1_RECO_Output_QCDDiJet_Pt1400to1800_20.root', 'uid': 0, 'uname': 'root', 'mtime': 1257051656, 'rmajor': 0, 'machine': ('Linux', 'cmsstor103.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 273340320, 21L, 1, 12124, 5063, 512, 1257050578, 1257050578, 1257036570), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 410026139L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data3/write-pool-3/data/0010000000000000094DDBA8', 'inode': 273537960L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/0010000000000000094DDBA8', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/.(access)(0010000000000000094DDBA8)', 'fc': {'uid': 0, 'external_label': 'VOU867', 'gid': 0, 'location_cookie': '0000_000000000_0000592', 'complete_crc': 14197294L, 'bfid': 'CDMS125705166100000', 'sanity_cookie': (65536L, 4222365876L), 'size': 410026139L}, 'infilepath': '/storage/data3/write-pool-3/data/0010000000000000094DDBA8', 'at_the_top': 1, 'local_inode': 1073743330, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.205.225', 51888), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257051655.249567, 'seek_time': 0.029726982116699219, 'transfer_time': 5.7533299922943115, 't0': 1257051655, 'mount_time': 0.049715995788574219, 'lm_dequeued': 1257051656.30022, 'job_queued': 1257051656.15483, 'drive_call_time': 1.3288929462432861, 'in_queue': 0.14533305168151855, 'drive_tran(trunc. 4190) Thread tape_thread 00:01:02 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV Done is sent Thread tape_thread 00:01:02 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV maybe_clean Thread tape_thread 00:01:02 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV returned from maybe_clean Thread tape_thread 00:01:02 cmsstor103.fnal.gov 014386 root I ENCP Received final dialog for cmsstor103.fnal.gov-1257051656-14386-0. elapsed=6.990sec 00:01:02 cmsstor103.fnal.gov 014386 root I ENCP leaving transfer_file(). 00:01:02 cmsstor103.fnal.gov 014386 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:01:02 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ827 0,7,1,11 00:01:02 stkenmvr193a.fnal.gov 031647 root E LTO4_93MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:02 stkenmvr193a.fnal.gov 031647 root I LTO4_93MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:02 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:02 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/PATLayer1_RECO_Output_QCDDiJet_Pt1400to1800_20.root', drive = 'stkenmvr204a:/dev/rmt/tps0d0n:1310050819', pnfs_id = '0010000000000000094DDBA8' where bfid = 'CDMS125705166100000'; 00:01:02 cmsstor103.fnal.gov 014386 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /storage/data3/write-pool-3/data/0010000000000000094DDBA8 -> /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/PATLayer1_RECO_Output_QCDDiJet_Pt1400to1800_20.root: 410026139 bytes copied to VOU867 at 67.5 MB/S (94.9 MB/S network) (78.6 MB/S drive) (113 MB/S disk) mover=LTO4_104.mover drive_id=ULTRIUM-TD4 drive_sn=1310050819 drive_vendor=IBM elapsed=7.2241 {'storage_group': 'cms', 'encp_crc': 14197294L, 'disk_rate': 113.1984307556542, 'drive_rate': 78.555374272379339, 'driver': 'FTTDriver', 'transfer_rate': 67.472181984899294, 'mover_interface': 'stkenmvr204a', 'overall_rate': 63.845320280652366, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 67.472181984899294, 'encp_ip': '131.225.205.225', 'unique_id': 'cmsstor103.fnal.gov-1257051656-14386-0'} 00:01:02 cmsstor103.fnal.gov 014386 root I ENCP File /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1400to1800/PATLayer1_RECO_Output_QCDDiJet_Pt1400to1800_20.root copied successfully. 00:01:02 cmsstor103.fnal.gov 014386 root I ENCP Completed transferring 410026139 bytes in 1 files in 7.22547793388 sec. Overall rate = 63.8 MB/sec. Transfer rate = 67.5 MB/sec. Network rate = 94.9 MB/sec. Drive rate = 78.6 MB/sec. Disk rate = 113 MB/sec. Exit status = 0. 00:01:03 stkensrv4n.fnal.gov 019886 enstore I SL8500G1MC dismount VOLUME 0,7,1,12 force => 0,Dismount: Forced dismount of VOS303 from 0, 7, 1,12 00:01:03 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC FINISHED dismount VOS303 0,7,1,12 returned ('ok', 0, 'dismount VOLUME 0,7,1,12 force => 0,Dismount: Forced dismount of VOS303 from 0, 7, 1,12', '', '') 00:01:03 stkenmvr142a.fnal.gov 009485 root I LTO4_42MV dismounted VOS303 ULTRIUM-TD4 00:01:03 Thread media_thread 00:01:03 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC PROCESSING mount VOS175 0,7,1,1 00:01:03 stkensrv2n.fnal.gov 007713 root E MNTR_SRV Error extablishing connection: (, MonitorError: Connection timed out) 00:01:04 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:04 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOS188 0,6,1,12 00:01:04 stkenmvr146a.fnal.gov 005990 root E LTO4_46MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:04 stkenmvr146a.fnal.gov 005990 root I LTO4_46MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:04 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:04 stkensrv4n.fnal.gov 019423 enstore I SL8500G1MC mount VOQ819 0,6,1,15 => 0,Mount: VOQ819 mounted on 0, 6, 1,15 00:01:04 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC FINISHED mount VOQ819 0,6,1,15 returned ('ok', 0, 'mount VOQ819 0,6,1,15 => 0,Mount: VOQ819 mounted on 0, 6, 1,15', '', '') 00:01:04 last message repeated 1 times 00:01:04 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_mounts = '138', last_access = '2009-11-01 00:01:04' where label = 'VOQ819'; 00:01:04 stkenmvr144a.fnal.gov 006329 root I LTO4_44MV MSG_TYPE=MC_LOAD_DONE mounted VOQ819 ULTRIUM-TD4 00:01:04 Thread media_thread 00:01:05 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:05 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC PROCESSING dismount VOQ873 0,5,1,11 00:01:05 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:06 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:07 stkensrv4n.fnal.gov 020283 enstore I SL8500G1MC query drive 0,7,1,0 => 0,0,7,1,0 online available IBM-LTO4 00:01:07 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:08 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:08 stkensrv4n.fnal.gov 019432 enstore I SL8500G1MC mount VOQ826 0,5,1,7 => 0,Mount: VOQ826 mounted on 0, 5, 1, 7 00:01:08 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC FINISHED mount VOQ826 0,5,1,7 returned ('ok', 0, 'mount VOQ826 0,5,1,7 => 0,Mount: VOQ826 mounted on 0, 5, 1, 7', '', '') 00:01:08 last message repeated 1 times 00:01:08 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_mounts = '102', last_access = '2009-11-01 00:01:08' where label = 'VOQ826'; 00:01:08 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV MSG_TYPE=MC_LOAD_DONE mounted VOQ826 ULTRIUM-TD4 00:01:08 Thread media_thread 00:01:09 stkensrv4n.fnal.gov 020296 enstore I SL8500G1MC query drive 0,7,1,1 => 0,0,7,1,1 online available IBM-LTO4 00:01:09 stkensrv4n.fnal.gov 020296 enstore I SL8500G1MC Dismount 14 ignored: Drive 0,7,1,1 is empty. Thought VOS175 was there =>. dismount VOLUME 0,7,1,1 force => ok,0,7,1,1 online available IBM-LTO4 00:01:09 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC PROCESSING mount VON584 0,6,1,7 00:01:09 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOU507 0,5,1,2 00:01:09 stkenmvr150a.fnal.gov 006006 root E LTO4_50MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:09 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:09 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:10 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOQ856 0,6,1,1 00:01:10 stkenmvr205a.fnal.gov 011191 root E LTO4_105MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:10 stkenmvr205a.fnal.gov 011191 root I LTO4_105MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:10 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOQ834 0,7,1,14 00:01:10 stkenmvr195a.fnal.gov 021479 root E LTO4_95MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:10 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:10 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:10 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ825 0,6,1,8 00:01:10 stkenmvr206a.fnal.gov 026766 root E LTO4_106MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:10 stkenmvr206a.fnal.gov 026766 root I LTO4_106MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:10 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ859 0,7,1,3 00:01:10 stkenmvr141a.fnal.gov 013963 root E LTO4_41MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:10 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:11 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:11 stkensrv4n.fnal.gov 020299 enstore I SL8500G1MC query drive 0,5,1,11 => 0,0,5,1,11 online in use VOQ873 IBM-LTO4 00:01:12 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:12 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ922 0,7,1,2 00:01:12 stkenmvr196a.fnal.gov 027637 root E LTO4_96MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:12 stkenmvr196a.fnal.gov 027637 root I LTO4_96MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:12 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ827 0,7,1,11 00:01:12 stkenmvr193a.fnal.gov 031647 root E LTO4_93MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:12 stkenmvr193a.fnal.gov 031647 root I LTO4_93MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:12 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:13 cmsstor184.fnal.gov 005655 root I ENCP File /storage/data2/write-pool-2/data/000700000000000061806760 transfered for cmsstor184.fnal.gov-1257051633-5655-0 in 22.7022509575 seconds. 00:01:14 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:14 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOS188 0,6,1,12 00:01:14 stkenmvr146a.fnal.gov 005990 root E LTO4_46MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:14 stkenmvr146a.fnal.gov 005990 root I LTO4_46MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:14 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:14 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV written bytes 2670926848/2670926848, blocks 20378 header 288 trailer 90 Thread tape_thread 00:01:15 cmsstor196.fnal.gov 029300 root I ENCP File /storage/data3/write-pool-3/data/0007000000000000618064A0 transfered for cmsstor196.fnal.gov-1257051626-29300-0 in 33.2841989994 seconds. 00:01:16 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:16 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV filemarks written. Tape VOQ828 absolute location in blocks 3180062 Thread tape_thread 00:01:16 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV write_tape timing:pull 0.231234073639 write 8.13455080986 crc_check 12.4364054203 freespace 0.249274969101 block_write 21.0514652729 idle 0.412665843964 Thread tape_thread 00:01:16 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV drive stats after write. Tape VOQ828 position 138 block 0 block_size 0 bloc_loc 3180062 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 2670926848 block_write_tot 21.0514652729 tape_rate 126876054.155 Thread tape_thread 00:01:16 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV remaining bytes info in DB 389957632000 estimated 387286705152 reported from drive 387477504000 position 139 Thread tape_thread 00:01:16 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV new bitfile request {'uid': 0, 'external_label': 'VOQ828', 'gid': 0, 'location_cookie': '0000_000000000_0000138', 'complete_crc': 631530098L, 'sanity_cookie': (65536L, 2080603967L), 'size': 2670926470L} Thread tape_thread 00:01:16 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV written bytes 3937102336/3937102336, blocks 30038 header 285 trailer 580 Thread tape_thread 00:01:16 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV set remaining: VOQ828 387477504000 0000_000000000_0000139 Thread tape_thread 00:01:16 last message repeated 1 times 00:01:16 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-01 00:01:16', eod_cookie = '0000_000000000_0000139', non_del_files = '138', remaining_bytes = 387477504000 where label = 'VOQ828'; 00:01:16 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV transfer complete volume=VOQ828 location=139 Thread tape_thread 00:01:16 last message repeated 1 times 00:01:16 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '138', modification_time = '2009-11-01 00:01:16' where label = 'VOQ828'; 00:01:16 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1255358447.0, 'library': 'CD-LTO4G1', 'si_time': [1241450288.0, 1241507765.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOQ828', 'wrapper': 'cpio_odc', 'remaining_bytes': 389957632000L, 'sum_mounts': 103, 'sum_rd_access': 0, 'file_family_width': 15, 'last_access': 1257051650.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000138', 'capacity_bytes': 858993459200L, 'non_del_files': 137, 'sum_wr_err': 0, 'sum_wr_access': 137, 'file_family': 'BACKFILL2', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.BACKFILL2.cpio_odc', 'modification_time': 1257051650.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1256991855.0}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/121A5A58-F8C5-DE11-AA61-001D09F29849.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2670926470L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7855, 'mc_device': '0,7,1,7', 'compression': 0, 'serial_num': '9310800990', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr194a', 'max_time_in_state': 1200, 'device': 'stkenmvr194a:/dev/rmt/tps0d0n', 'name': 'LTO4_94.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_94.stat', 'mover_address': ('131.225.13.189', 7855), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.189', 54730), 'data_ip': 'stkenmvr194a', 'logname': 'LTO4_94MV', 'hostip': '131.225.13.189', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/121A5A58-F8C5-DE11-AA61-001D09F29849.root', 'uid': 0, 'uname': 'root', 'mtime': 1257051633, 'rmajor': 0, 'machine': ('Linux', 'cmsstor184.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 125854968, 21L, 1, 9811, 5063, 512, 1257051362, 1257051362, 1257051186), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 2670926470L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data2/write-pool-2/data/000700000000000061806760', 'inode': 125855584L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/000700000000000061806760', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/.(access)(000700000000000061806760)', 'fc': {'uid': 0, 'external_label': 'VOQ828', 'gid': 0, 'location_cookie': '0000_000000000_0000138', 'complete_crc': 631530098L, 'bfid': 'CDMS125705167600000', 'sanity_cookie': (65536L, 2080603967L), 'size': 2670926470L}, 'infilepath': '/storage/data2/write-pool-2/data/000700000000000061806760', 'at_the_top': 26, 'local_inode': 2788, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.207.5', 45635), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257051632.6476951, 'seek_time': 0.029911041259765625, 'transfer_time': 24.910209894180298, 't0': 1257051632, '(trunc. 4373) Thread tape_thread 00:01:16 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV Done is sent Thread tape_thread 00:01:16 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV maybe_clean Thread tape_thread 00:01:16 cmsstor184.fnal.gov 005655 root I ENCP Received final dialog for cmsstor184.fnal.gov-1257051633-5655-0. elapsed=43.560sec 00:01:16 cmsstor184.fnal.gov 005655 root I ENCP leaving transfer_file(). 00:01:16 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV returned from maybe_clean Thread tape_thread 00:01:16 cmsstor184.fnal.gov 005655 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:01:16 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:16 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/121A5A58-F8C5-DE11-AA61-001D09F29849.root', drive = 'stkenmvr194a:/dev/rmt/tps0d0n:9310800990', pnfs_id = '000700000000000061806760' where bfid = 'CDMS125705167600000'; 00:01:16 cmsstor184.fnal.gov 005655 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /storage/data2/write-pool-2/data/000700000000000061806760 -> /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/121A5A58-F8C5-DE11-AA61-001D09F29849.root: 2670926470 bytes copied to VOQ828 at 102 MB/S (96.4 MB/S network) (109 MB/S drive) (112 MB/S disk) mover=LTO4_94.mover drive_id=ULTRIUM-TD4 drive_sn=9310800990 drive_vendor=IBM elapsed=43.936 {'storage_group': 'cms', 'encp_crc': 631530098L, 'disk_rate': 112.22693937093072, 'drive_rate': 109.46374872513333, 'driver': 'FTTDriver', 'transfer_rate': 102.1920333569183, 'mover_interface': 'stkenmvr194a', 'overall_rate': 100.44356097306354, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 102.1920333569183, 'encp_ip': '131.225.207.5', 'unique_id': 'cmsstor184.fnal.gov-1257051633-5655-0'} 00:01:16 cmsstor184.fnal.gov 005655 root I ENCP File /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/121A5A58-F8C5-DE11-AA61-001D09F29849.root copied successfully. 00:01:16 cmsstor184.fnal.gov 005655 root I ENCP Completed transferring 2670926470 bytes in 1 files in 43.9369850159 sec. Overall rate = 100 MB/sec. Transfer rate = 102 MB/sec. Network rate = 96.4 MB/sec. Drive rate = 109 MB/sec. Disk rate = 112 MB/sec. Exit status = 0. 00:01:16 last message repeated 1 times 00:01:16 stkenmvr113a.fnal.gov 005961 root I LTO13MV skip_fm 9 Thread seek_thread 00:01:16 stkenmvr113a.fnal.gov 005961 root I LTO13MV read_tape starting, bytes_to_read=17439631 Thread tape_thread 00:01:16 stkenmvr113a.fnal.gov 005961 root I LTO13MV read bytes 17439779/17439779, blocks 134 header 148 Thread tape_thread 00:01:16 stkenmvr113a.fnal.gov 005961 root I LTO13MV write_client: wrote 17439631/17439631 bytes Thread net_thread 00:01:16 stkensrv3n.fnal.gov 013824 root I ENCP File /pnfs/cms/WAX/11/store/TAC/TIF/edm_2007_07_11/EDM0013290_109.root transfered for stkensrv3n.fnal.gov-1257051636-13824-0 in 0.201434850693 seconds. 00:01:16 stkenmvr113a.fnal.gov 005961 root I LTO13MV write_client waits for tape thread to exit Thread net_thread 00:01:16 stkenmvr113a.fnal.gov 005961 root I LTO13MV drive stats after read. Tape VOE575 position 10 block 134 block_size 0 bloc_loc 33071 tot_blocks 381469000 BOT 0 read_err 0 write_err 0 bytes 17439779 block_read_tot 0.291779518127 tape_rate 59770401.6784 Thread tape_thread 00:01:16 stkenmvr113a.fnal.gov 005961 root I LTO13MV read_tape exiting, read 17439779/17439779 bytes Thread tape_thread 00:01:16 stkenmvr113a.fnal.gov 005961 root I LTO13MV read_tape timing:get_space 0.00455498695374 read 0.22025680542 crc_check 0.0647504329681 push 0.00221729278564 block_read 0.291779518127 idle 0.00482654571533 Thread tape_thread 00:01:17 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:17 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV filemarks written. Tape VOQ862 absolute location in blocks 1338263 Thread tape_thread 00:01:17 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV write_tape timing:pull 0.317083597183 write 11.6316709518 crc_check 18.1441907883 freespace 0.327921152115 block_write 30.4208664894 idle 0.571264982224 Thread tape_thread 00:01:17 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV drive stats after write. Tape VOQ862 position 58 block 0 block_size 0 bloc_loc 1338263 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 3937102336 block_write_tot 30.4208664894 tape_rate 129421110.913 Thread tape_thread 00:01:17 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV remaining bytes info in DB 619633664000 estimated 615696561664 reported from drive 615989248000 position 59 Thread tape_thread 00:01:17 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV new bitfile request {'uid': 0, 'external_label': 'VOQ862', 'gid': 0, 'location_cookie': '0000_000000000_0000058', 'complete_crc': 222820099L, 'sanity_cookie': (65536L, 830426854L), 'size': 3937101471L} Thread tape_thread 00:01:17 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:17 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV set remaining: VOQ862 615989248000 0000_000000000_0000059 Thread tape_thread 00:01:17 last message repeated 1 times 00:01:17 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-01 00:01:17', eod_cookie = '0000_000000000_0000059', non_del_files = '58', remaining_bytes = 615989248000 where label = 'VOQ862'; 00:01:17 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV transfer complete volume=VOQ862 location=59 Thread tape_thread 00:01:17 last message repeated 1 times 00:01:17 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '58', modification_time = '2009-11-01 00:01:17' where label = 'VOQ862'; 00:01:17 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1255358534.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 1242983135.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOQ862', 'wrapper': 'cpio_odc', 'remaining_bytes': 619633664000L, 'sum_mounts': 294, 'sum_rd_access': 0, 'file_family_width': 15, 'last_access': 1257051641.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000058', 'capacity_bytes': 858993459200L, 'non_del_files': 57, 'sum_wr_err': 0, 'sum_wr_access': 57, 'file_family': 'BACKFILL2', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.BACKFILL2.cpio_odc', 'modification_time': 1257051641.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257039541.0}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/F4DA130B-16C6-DE11-BC20-000423D992A4.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 3937101471L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7870, 'mc_device': '0,5,1,14', 'compression': 0, 'serial_num': '1310049971', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr210a', 'max_time_in_state': 1200, 'device': 'stkenmvr210a:/dev/rmt/tps0d0n', 'name': 'LTO4_110.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_110.stat', 'mover_address': ('131.225.13.206', 7870), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.206', 53944), 'data_ip': 'stkenmvr210a', 'logname': 'LTO4_110MV', 'hostip': '131.225.13.206', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/F4DA130B-16C6-DE11-BC20-000423D992A4.root', 'uid': 0, 'uname': 'root', 'mtime': 1257051626, 'rmajor': 0, 'machine': ('Linux', 'cmsstor196.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 125853680, 21L, 1, 9811, 5063, 512, 1257051251, 1257051251, 1257051006), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 3937101471L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data3/write-pool-3/data/0007000000000000618064A0', 'inode': 125854880L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/0007000000000000618064A0', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(0007000000000000618064A0)', 'fc': {'uid': 0, 'external_label': 'VOQ862', 'gid': 0, 'location_cookie': '0000_000000000_0000058', 'complete_crc': 222820099L, 'bfid': 'CDMS125705167700000', 'sanity_cookie': (65536L, 830426854L), 'size': 3937101471L}, 'infilepath': '/storage/data3/write-pool-3/data/0007000000000000618064A0', 'at_the_top': 48, 'local_inode': 5922, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.207.21', 38842), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257051626.3732641, 'seek_time': 0.029961824417114258, 'transfer_time': 35.133284091949463, 't0': 1257051626, 'mount_time': 0.0(trunc. 4359) Thread tape_thread 00:01:17 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV Done is sent Thread tape_thread 00:01:17 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV maybe_clean Thread tape_thread 00:01:17 cmsstor196.fnal.gov 029300 root I ENCP Received final dialog for cmsstor196.fnal.gov-1257051626-29300-0. elapsed=51.380sec 00:01:17 cmsstor196.fnal.gov 029300 root I ENCP leaving transfer_file(). 00:01:17 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV returned from maybe_clean Thread tape_thread 00:01:17 cmsstor48.fnal.gov 003713 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: BACKFILL2 FF Wrapper: cpio_odc FF Width: 15 Current working directory: cmsstor48.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --age-time 30 --delpri 1 --priority 25 --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/327EE30A-10C6-DE11-A60B-0019B9F707D8.root --put-cache 0007000000000000618063E8 /storage/data3/write-pool-3/data/0007000000000000618063E8 00:01:17 cmsstor196.fnal.gov 029300 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:01:17 stkenmvr113a.fnal.gov 005961 root I LTO13MV write_client exits Thread net_thread 00:01:17 stkenmvr113a.fnal.gov 005961 root I LTO13MV transfer complete volume=VOE575 location=10 Thread net_thread 00:01:17 last message repeated 1 times 00:01:17 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_rd_access = '43', last_access = '2009-11-01 00:01:17' where label = 'VOE575'; 00:01:17 stkenmvr113a.fnal.gov 005961 root I LTO13MV Sending done to client: {'vc': {'comment': '', 'declared': 1183670260.0, 'blocksize': 131072, 'storage_group': 'cms', 'library': 'CD-LTO3', 'eod_cookie': '0000_000000000_0000268', 'current_location': '0000_000000000_0000010', 'user_inhibit': ['none', 'none'], 'sum_rd_access': 42, 'system_inhibit': ['none', 'full'], 'external_label': 'VOE575', 'wrapper': 'cpio_odc', 'remaining_bytes': 180380160L, 'sum_mounts': 182, 'capacity_bytes': 429496729600L, 'media_type': 'LTO3', 'last_access': 1257051635.0, 'status': ('ok', None), 'si_time': [0.0, 1184275501.0], 'non_del_files': 267, 'sum_wr_err': 0, 'sum_wr_access': 267, 'file_family': 'cms11', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'modification_time': 1220711104.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1184212262.0}, 'outfilepath': '/dev/null', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': 1, 'curpri': 1}, 'file_size': 17439631L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 300, 'max_failures': 5, 'library': 'CD-LTO3.library_manager', 'local_mover': 0, 'dismount_delay': 20, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7813, 'mc_device': '0,3,1,12', 'compression': 0, 'serial_num': '1210116540', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 57671680.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr113a', 'max_time_in_state': 1200, 'device': 'stkenmvr113a:/dev/rmt/tps2d0n', 'name': 'LTO3_13.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD3', 'statistics_path': '/tmp/enstore/enstore/LTO3_13.stat', 'mover_address': ('131.225.13.143', 7813), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.143', 59044), 'data_ip': 'stkenmvr113a', 'logname': 'LTO13MV', 'hostip': '131.225.13.143', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'lm': {'address': ('131.225.13.4', 7506)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/cms/WAX/11/store/TAC/TIF/edm_2007_07_11/EDM0013290_109.root', 'uid': 0, 'uname': 'root', 'rmajor': 0, 'machine': ('Linux', 'stkensrv3n.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 08:50:41 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (33188, 118124728L, 20L, 1, 9811, 5063, 17439631L, 1256499047, 1184236316, 1184236135), 'mode_octal': '0100744', 'mode': 33252, 'size_bytes': 17439631L, 'gname': 'root', 'gid': 0, 'fullname': '/dev/null', 'inode': 118124728L, 'minor': 20}, 'version': 'v3_7d CVS $Revision: 1.929 $ encp', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/cms/WAX/11/store/TAC/TIF/edm_2007_07_11/.(access)(0007000000000000050A70B8)', 'status': ('ok', None), 'volume': 'VOE575', 'outfile': '/dev/null', 'fc': {'status': ('ok', None), 'pnfsid': '0007000000000000050A70B8', 'pnfs_name0': '/pnfs/fs/usr/cms/WAX/11/store/TAC/TIF/edm_2007_07_11/EDM0013290_109.root', 'deleted': 'no', 'external_label': 'VOE575', 'drive': 'stkenmvr115a:/dev/rmt/tps2d0n:1210114342', 'update': '2007-07-12 05:31:56.870138', 'gid': 0, 'location_cookie': '0000_000000000_0000010', 'complete_crc': 497667335L, 'address': ('131.225.13.59', 7501), 'uid': 0, 'bfid': 'CDMS118423631600000', 'sanity_cookie': (65536L, 3662479943L), 'size': 17439631L}, 'infilepath': '/pnfs/cms/WAX/11/store/TAC/TIF/edm_2007_07_11/EDM0013290_109.root', 'at_the_top': 11, 'bfid': 'CDMS118423631600000', 'local_inode': 2469L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': 7, 'retry': 0}, 'callback_addr': ('131.225.13.51', 50468), 'work': 'read_from_hsm', 'override_ro_mount': 1, 'times': {'encp_start_time': 1257051635.6477201, 'seek_time': 38.398486852645874, 'transfer_time': 1.3182330131530762, 't0': 1257051635, 'mount_time': 0.021585941314697266, 'lm_dequeued': 1257051638.1235771, 'job_queued': 1257051636.7958441, 'drive_call_time': 0.21702909469604492, 'in_queue': 1.3276588916778564, 'drive_transfer_time': 0.29403901100158691}, 'user_level': 'admin', 'unique_id': 'stkensrv3n.fnal.gov-1257051636-13824-0'} Thread net_thread 00:01:17 stkenmvr113a.fnal.gov 005961 root I LTO13MV Done is sent Thread net_thread 00:01:17 stkenmvr113a.fnal.gov 005961 root I LTO13MV maybe_clean Thread net_thread 00:01:17 stkensrv3n.fnal.gov 013824 root I ENCP Received final dialog for stkensrv3n.fnal.gov-1257051636-13824-0. elapsed=42.330sec 00:01:17 stkensrv3n.fnal.gov 013824 root I ENCP leaving transfer_file(). 00:01:17 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/F4DA130B-16C6-DE11-BC20-000423D992A4.root', drive = 'stkenmvr210a:/dev/rmt/tps0d0n:1310049971', pnfs_id = '0007000000000000618064A0' where bfid = 'CDMS125705167700000'; 00:01:18 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:18 stkensrv3n.fnal.gov 013824 root I ENCP MSG_TYPE=ENCP_XFER read_from_hsm /pnfs/cms/WAX/11/store/TAC/TIF/edm_2007_07_11/EDM0013290_109.root -> /dev/null: 17439631 bytes copied from VOE575 at 13.3 MB/S (82.6 MB/S network) (56.6 MB/S drive) (82.6 MB/S disk) mover=LTO3_13.mover drive_id=ULTRIUM-TD3 drive_sn=1210116540 drive_vendor=IBM elapsed=42.352 {'storage_group': 'cms', 'encp_crc': 497667335L, 'disk_rate': 82.61833841719347, 'drive_rate': 56.562998615896191, 'driver': 'FTTDriver', 'transfer_rate': 13.254084039054776, 'mover_interface': 'stkenmvr113a', 'overall_rate': 0.41784473902695896, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 13.254084039054776, 'encp_ip': '131.225.13.51', 'unique_id': 'stkensrv3n.fnal.gov-1257051636-13824-0'} 00:01:18 stkenmvr113a.fnal.gov 005961 root I LTO13MV returned from maybe_clean Thread net_thread 00:01:18 stkensrv3n.fnal.gov 013824 root I ENCP File /dev/null copied successfully. 00:01:18 stkensrv3n.fnal.gov 013824 root I ENCP Completed transferring 17439631 bytes in 1 files in 42.3590857983 sec. Overall rate = 0.418 MB/sec. Transfer rate = 13.3 MB/sec. Network rate = 82.6 MB/sec. Drive rate = 56.6 MB/sec. Disk rate = 82.6 MB/sec. Exit status = 0. 00:01:18 cmsstor48.fnal.gov 003713 root I ENCP Creating zero length output files. 00:01:18 cmsstor48.fnal.gov 003713 root I ENCP Sending /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/327EE30A-10C6-DE11-A60B-0019B9F707D8.root write request to LM: unique_id: cmsstor48.fnal.gov-1257051677-3713-0 inputfile: /storage/data3/write-pool-3/data/0007000000000000618063E8 outputfile: /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(0007000000000000618063E8) 00:01:18 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:01:18 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data3/write-pool-3/data/0007000000000000618063E8 (cmsstor48.fnal.gov-1257051677-3713-0) -> /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/327EE30A-10C6-DE11-A60B-0019B9F707D8.root : library=CD-LTO4G1 family=BACKFILL2 requester:root volume_family:cms.BACKFILL2.cpio_odc 00:01:18 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV skip_fm 120 Thread seek_thread 00:01:18 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV write_tape starting, bytes_to_write=2675564032 Thread tape_thread 00:01:18 cmsstor48.fnal.gov 003713 root I ENCP Listening for control socket at: ('131.225.189.6', 50952) 00:01:18 cmsstor48.fnal.gov 003713 root I ENCP Waiting for mover to connect control socket at address: ('131.225.189.6', 50952) elapsed=1.030sec 00:01:18 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV Write starting. Tape VOQ839 absolute location in blocks 2741030 Thread tape_thread 00:01:18 cmsstor196.fnal.gov 029300 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /storage/data3/write-pool-3/data/0007000000000000618064A0 -> /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/F4DA130B-16C6-DE11-BC20-000423D992A4.root: 3937101471 bytes copied to VOQ862 at 107 MB/S (97.5 MB/S network) (114 MB/S drive) (113 MB/S disk) mover=LTO4_110.mover drive_id=ULTRIUM-TD4 drive_sn=1310049971 drive_vendor=IBM elapsed=51.774 {'storage_group': 'cms', 'encp_crc': 222820099L, 'disk_rate': 112.83068304486342, 'drive_rate': 113.77550462214079, 'driver': 'FTTDriver', 'transfer_rate': 106.82037780745416, 'mover_interface': 'stkenmvr210a', 'overall_rate': 105.46922589706386, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 106.82037780745416, 'encp_ip': '131.225.207.21', 'unique_id': 'cmsstor196.fnal.gov-1257051626-29300-0'} 00:01:18 cmsstor196.fnal.gov 029300 root I ENCP File /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/F4DA130B-16C6-DE11-BC20-000423D992A4.root copied successfully. 00:01:18 cmsstor196.fnal.gov 029300 root I ENCP Completed transferring 3937101471 bytes in 1 files in 51.7747578621 sec. Overall rate = 105 MB/sec. Transfer rate = 107 MB/sec. Network rate = 97.5 MB/sec. Drive rate = 114 MB/sec. Disk rate = 113 MB/sec. Exit status = 0. 00:01:18 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:01:18 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:18 cmsstor187.fnal.gov 019936 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Scientific Linux SL release 5.2 (Boron) Rocks release 5.1 (V.I) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms2 FF Wrapper: cpio_odc FF Width: 7 Current working directory: cmsstor187.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --age-time 30 --delpri 1 --priority 25 --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/PATLayer1_RECO_Output_QCDDiJet_Pt1800to2200_24.root --put-cache 0010000000000000094DDB18 /storage/data3/write-pool-3/data/0010000000000000094DDB18 00:01:18 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:18 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VOQ862 mover=LTO4_110.mover requester:root 00:01:18 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/327EE30A-10C6-DE11-A60B-0019B9F707D8.root to mover LTO4_110.mover ('131.225.13.206', 7870) DEL_DISM None 00:01:18 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:01:18 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'cms', 'external_label': 'VOQ862', 'library': 'CD-LTO4G1', 'file_family': 'BACKFILL2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.BACKFILL2.cpio_odc', 'file_family_width': 15}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/327EE30A-10C6-DE11-A60B-0019B9F707D8.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 3996632888L, 'mover': 'LTO4_110.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/327EE30A-10C6-DE11-A60B-0019B9F707D8.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257051677, 'rmajor': 0, 'machine': ('Linux', 'cmsstor48.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16877, 125853680, 21L, 1, 9811, 5063, 512, 1257051251, 1257051251, 1257051006), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 3996632888L, 'fullname': '/storage/data3/write-pool-3/data/0007000000000000618063E8', 'inode': 125854696L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/0007000000000000618063E8', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(0007000000000000618063E8)', 'fc': {'size': 3996632888L, 'external_label': 'VOQ862', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data3/write-pool-3/data/0007000000000000618063E8', 'at_the_top': 4, 'local_inode': 6022, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.189.6', 50952), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051678.5192921, 'encp_start_time': 1257051677.0900431, 'job_queued': 1257051678.0931721, 't0': 1257051677, 'in_queue': 0.42607688903808594}, 'user_level': 'admin', 'unique_id': 'cmsstor48.fnal.gov-1257051677-3713-0'} to mover 00:01:18 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:18 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV WRITE_TO_HSM Thread MainThread 00:01:18 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV SENDING {'vc': {'storage_group': 'cms', 'external_label': 'VOQ862', 'library': 'CD-LTO4G1', 'file_family': 'BACKFILL2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.BACKFILL2.cpio_odc', 'file_family_width': 15}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/327EE30A-10C6-DE11-A60B-0019B9F707D8.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 3996632888L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7870, 'mc_device': '0,5,1,14', 'compression': 0, 'serial_num': '1310049971', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr210a', 'max_time_in_state': 1200, 'device': 'stkenmvr210a:/dev/rmt/tps0d0n', 'name': 'LTO4_110.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_110.stat', 'mover_address': ('131.225.13.206', 7870), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.206', 53947), 'data_ip': 'stkenmvr210a', 'logname': 'LTO4_110MV', 'hostip': '131.225.13.206', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/327EE30A-10C6-DE11-A60B-0019B9F707D8.root', 'uid': 0, 'uname': 'root', 'mtime': 1257051677, 'rmajor': 0, 'machine': ('Linux', 'cmsstor48.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 125853680, 21L, 1, 9811, 5063, 512, 1257051251, 1257051251, 1257051006), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 3996632888L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data3/write-pool-3/data/0007000000000000618063E8', 'inode': 125854696L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/0007000000000000618063E8', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(0007000000000000618063E8)', 'fc': {'size': 3996632888L, 'external_label': 'VOQ862', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data3/write-pool-3/data/0007000000000000618063E8', 'at_the_top': 4, 'local_inode': 6022, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.189.6', 50952), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051678.5192921, 'encp_start_time': 1257051677.0900431, 'job_queued': 1257051678.0931721, 't0': 1257051677, 'in_queue': 0.42607688903808594}, 'user_level': 'admin', 'unique_id': 'cmsstor48.fnal.gov-1257051677-3713-0'} Thread client_connect_thread 00:01:18 cmsstor48.fnal.gov 003713 root I ENCP Control socket ('131.225.189.6', 50952) is connected to ('131.225.13.206', 53948) for cmsstor48.fnal.gov-1257051677-3713-0. elapsed=1.460sec 00:01:18 cmsstor48.fnal.gov 003713 root I ENCP Received callback ticket from mover LTO4_110.mover for transfer cmsstor48.fnal.gov-1257051677-3713-0. elapsed=1.460sec 00:01:18 cmsstor48.fnal.gov 003713 root I ENCP Atempting to connect data socket to mover at ('131.225.13.206', 53947). 00:01:18 cmsstor48.fnal.gov 003713 root I ENCP Connecting data socket to mover (('131.225.13.206', 53947)) with interface 131.225.189.6. elapsed=1.470sec 00:01:18 cmsstor48.fnal.gov 003713 root I ENCP Data socket ('131.225.189.6', 55219) is connected to ('131.225.13.206', 53947). elapsed=1.470sec 00:01:18 cmsstor48.fnal.gov 003713 root I ENCP Opening local file. 00:01:18 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV tape position: target 59 current 59 0 Thread seek_thread 00:01:18 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV write_tape starting, bytes_to_write=3996633600 Thread tape_thread 00:01:18 cmsstor48.fnal.gov 003713 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:01:18 cmsstor187.fnal.gov 019936 root I ENCP Creating zero length output files. 00:01:18 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV Write starting. Tape VOQ862 absolute location in blocks 1338263 Thread tape_thread 00:01:18 cmsstor187.fnal.gov 019936 root I ENCP Sending /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/PATLayer1_RECO_Output_QCDDiJet_Pt1800to2200_24.root write request to LM: unique_id: cmsstor187.fnal.gov-1257051678-19936-0 inputfile: /storage/data3/write-pool-3/data/0010000000000000094DDB18 outputfile: /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/.(access)(0010000000000000094DDB18) 00:01:18 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:01:18 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data3/write-pool-3/data/0010000000000000094DDB18 (cmsstor187.fnal.gov-1257051678-19936-0) -> /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/PATLayer1_RECO_Output_QCDDiJet_Pt1800to2200_24.root : library=CD-LTO4G1 family=cms2 requester:root volume_family:cms.cms2.cpio_odc 00:01:18 cmsstor187.fnal.gov 019936 root I ENCP Listening for control socket at: ('131.225.207.10', 42819) 00:01:18 cmsstor187.fnal.gov 019936 root I ENCP Waiting for mover to connect control socket at address: ('131.225.207.10', 42819) elapsed=0.660sec 00:01:18 stkensrv4n.fnal.gov 020296 enstore I SL8500G1MC query vol VOS175 => 0,VOS175 home 0, 6,15,11, 1 LTO-800G 00:01:19 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:19 stkensrv4n.fnal.gov 020306 enstore I SL8500G1MC query drive 0,6,1,7 => 0,0,6,1,7 online available IBM-LTO4 00:01:19 stkensrv4n.fnal.gov 020306 enstore I SL8500G1MC Dismount 14 ignored: Drive 0,6,1,7 is empty. Thought VON584 was there =>. dismount VOLUME 0,6,1,7 force => ok,0,6,1,7 online available IBM-LTO4 00:01:19 cmsstor79.fnal.gov 012910 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: BACKFILL2 FF Wrapper: cpio_odc FF Width: 15 Current working directory: cmsstor79.fnal.gov:/root Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --age-time 30 --delpri 1 --priority 25 --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/E60A8A9F-16C6-DE11-83F0-001D09F291D2.root --put-cache 000700000000000061806408 /storage/data1/write-pool-1/data/000700000000000061806408 00:01:19 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:19 cmsstor79.fnal.gov 012910 root I ENCP Creating zero length output files. 00:01:19 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOU507 0,5,1,2 00:01:19 stkenmvr150a.fnal.gov 006006 root E LTO4_50MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:19 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:19 cmsstor79.fnal.gov 012910 root I ENCP Sending /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/E60A8A9F-16C6-DE11-83F0-001D09F291D2.root write request to LM: unique_id: cmsstor79.fnal.gov-1257051679-12910-0 inputfile: /storage/data1/write-pool-1/data/000700000000000061806408 outputfile: /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(000700000000000061806408) 00:01:19 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:01:19 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data1/write-pool-1/data/000700000000000061806408 (cmsstor79.fnal.gov-1257051679-12910-0) -> /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/E60A8A9F-16C6-DE11-83F0-001D09F291D2.root : library=CD-LTO4G1 family=BACKFILL2 requester:root volume_family:cms.BACKFILL2.cpio_odc 00:01:19 cmsstor79.fnal.gov 012910 root I ENCP Listening for control socket at: ('131.225.205.201', 60545) 00:01:19 cmsstor79.fnal.gov 012910 root I ENCP Waiting for mover to connect control socket at address: ('131.225.205.201', 60545) elapsed=1.010sec 00:01:20 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:20 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:01:20 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOQ856 0,6,1,1 00:01:20 stkenmvr205a.fnal.gov 011191 root E LTO4_105MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:20 stkenmvr205a.fnal.gov 011191 root I LTO4_105MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:20 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOQ834 0,7,1,14 00:01:20 stkenmvr195a.fnal.gov 021479 root E LTO4_95MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:20 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VOU867 mover=LTO4_104.mover requester:root 00:01:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/PATLayer1_RECO_Output_QCDDiJet_Pt1800to2200_24.root to mover LTO4_104.mover ('131.225.13.199', 7864) DEL_DISM None 00:01:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:01:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'cms', 'external_label': 'VOU867', 'library': 'CD-LTO4G1', 'file_family': 'cms2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms2.cpio_odc', 'file_family_width': 7}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/PATLayer1_RECO_Output_QCDDiJet_Pt1800to2200_24.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 428897126L, 'mover': 'LTO4_104.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/PATLayer1_RECO_Output_QCDDiJet_Pt1800to2200_24.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257051678, 'rmajor': 0, 'machine': ('Linux', 'cmsstor187.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16893, 273340400, 24L, 1, 12124, 5063, 512, 1257050553, 1257050553, 1257036571), 'mode_octal': '0100036', 'mode': 33188, 'gname': 'root', 'size_bytes': 428897126L, 'fullname': '/storage/data3/write-pool-3/data/0010000000000000094DDB18', 'inode': 273537816L, 'minor': 24}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/0010000000000000094DDB18', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/.(access)(0010000000000000094DDB18)', 'fc': {'size': 428897126L, 'external_label': 'VOU867', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data3/write-pool-3/data/0010000000000000094DDB18', 'at_the_top': 1, 'local_inode': 2544, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.207.10', 42819), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051680.3608, 'encp_start_time': 1257051678.006125, 'job_queued': 1257051678.6470239, 't0': 1257051678, 'in_queue': 1.7137141227722168}, 'user_level': 'admin', 'unique_id': 'cmsstor187.fnal.gov-1257051678-19936-0'} to mover 00:01:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:20 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV WRITE_TO_HSM Thread MainThread 00:01:20 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV SENDING {'vc': {'storage_group': 'cms', 'external_label': 'VOU867', 'library': 'CD-LTO4G1', 'file_family': 'cms2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms2.cpio_odc', 'file_family_width': 7}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/PATLayer1_RECO_Output_QCDDiJet_Pt1800to2200_24.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 428897126L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7864, 'mc_device': '0,6,1,13', 'compression': 0, 'serial_num': '1310050819', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr204a', 'max_time_in_state': 1200, 'device': 'stkenmvr204a:/dev/rmt/tps0d0n', 'name': 'LTO4_104.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_104.stat', 'mover_address': ('131.225.13.199', 7864), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.199', 54011), 'data_ip': 'stkenmvr204a', 'logname': 'LTO4_104MV', 'hostip': '131.225.13.199', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/PATLayer1_RECO_Output_QCDDiJet_Pt1800to2200_24.root', 'uid': 0, 'uname': 'root', 'mtime': 1257051678, 'rmajor': 0, 'machine': ('Linux', 'cmsstor187.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 273340400, 24L, 1, 12124, 5063, 512, 1257050553, 1257050553, 1257036571), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 428897126L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data3/write-pool-3/data/0010000000000000094DDB18', 'inode': 273537816L, 'minor': 24}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/0010000000000000094DDB18', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/.(access)(0010000000000000094DDB18)', 'fc': {'size': 428897126L, 'external_label': 'VOU867', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data3/write-pool-3/data/0010000000000000094DDB18', 'at_the_top': 1, 'local_inode': 2544, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.207.10', 42819), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051680.3608, 'encp_start_time': 1257051678.006125, 'job_queued': 1257051678.6470239, 't0': 1257051678, 'in_queue': 1.7137141227722168}, 'user_level': 'admin', 'unique_id': 'cmsstor187.fnal.gov-1257051678-19936-0'} Thread client_connect_thread 00:01:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:20 cmsstor187.fnal.gov 019936 root I ENCP Control socket ('131.225.207.10', 42819) is connected to ('131.225.13.199', 54012) for cmsstor187.fnal.gov-1257051678-19936-0. elapsed=2.380sec 00:01:20 cmsstor187.fnal.gov 019936 root I ENCP Received callback ticket from mover LTO4_104.mover for transfer cmsstor187.fnal.gov-1257051678-19936-0. elapsed=2.390sec 00:01:20 cmsstor187.fnal.gov 019936 root I ENCP Atempting to connect data socket to mover at ('131.225.13.199', 54011). 00:01:20 cmsstor187.fnal.gov 019936 root I ENCP Connecting data socket to mover (('131.225.13.199', 54011)) with interface 131.225.207.10. elapsed=2.390sec 00:01:20 cmsstor187.fnal.gov 019936 root I ENCP Data socket ('131.225.207.10', 55003) is connected to ('131.225.13.199', 54011). elapsed=2.390sec 00:01:20 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ825 0,6,1,8 00:01:20 stkenmvr206a.fnal.gov 026766 root E LTO4_106MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:20 stkenmvr206a.fnal.gov 026766 root I LTO4_106MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:20 cmsstor187.fnal.gov 019936 root I ENCP Opening local file. 00:01:20 cmsstor187.fnal.gov 019936 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:01:20 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV tape position: target 593 current 593 0 Thread seek_thread 00:01:20 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV write_tape starting, bytes_to_write=428897792 Thread tape_thread 00:01:20 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV Write starting. Tape VOU867 absolute location in blocks 734323 Thread tape_thread 00:01:20 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ859 0,7,1,3 00:01:20 stkenmvr141a.fnal.gov 013963 root E LTO4_41MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:20 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:21 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:21 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VOQ828 mover=LTO4_94.mover requester:root 00:01:21 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/E60A8A9F-16C6-DE11-83F0-001D09F291D2.root to mover LTO4_94.mover ('131.225.13.189', 7855) DEL_DISM None 00:01:21 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:01:21 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'cms', 'external_label': 'VOQ828', 'library': 'CD-LTO4G1', 'file_family': 'BACKFILL2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.BACKFILL2.cpio_odc', 'file_family_width': 15}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/E60A8A9F-16C6-DE11-83F0-001D09F291D2.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 3947781132L, 'mover': 'LTO4_94.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/E60A8A9F-16C6-DE11-83F0-001D09F291D2.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257051679, 'rmajor': 0, 'machine': ('Linux', 'cmsstor79.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16877, 125853680, 27L, 1, 9811, 5063, 512, 1257051251, 1257051251, 1257051006), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 3947781132L, 'fullname': '/storage/data1/write-pool-1/data/000700000000000061806408', 'inode': 125854728L, 'minor': 27}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/000700000000000061806408', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(000700000000000061806408)', 'fc': {'size': 3947781132L, 'external_label': 'VOQ828', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data1/write-pool-1/data/000700000000000061806408', 'at_the_top': 10, 'local_inode': 1073742569, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.205.201', 60545), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051681.2435961, 'encp_start_time': 1257051678.69889, 'job_queued': 1257051679.6657131, 't0': 1257051678, 'in_queue': 1.5778398513793945}, 'user_level': 'admin', 'unique_id': 'cmsstor79.fnal.gov-1257051679-12910-0'} to mover 00:01:21 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:21 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV WRITE_TO_HSM Thread MainThread 00:01:21 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV SENDING {'vc': {'storage_group': 'cms', 'external_label': 'VOQ828', 'library': 'CD-LTO4G1', 'file_family': 'BACKFILL2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.BACKFILL2.cpio_odc', 'file_family_width': 15}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/E60A8A9F-16C6-DE11-83F0-001D09F291D2.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 3947781132L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7855, 'mc_device': '0,7,1,7', 'compression': 0, 'serial_num': '9310800990', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr194a', 'max_time_in_state': 1200, 'device': 'stkenmvr194a:/dev/rmt/tps0d0n', 'name': 'LTO4_94.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_94.stat', 'mover_address': ('131.225.13.189', 7855), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.189', 54733), 'data_ip': 'stkenmvr194a', 'logname': 'LTO4_94MV', 'hostip': '131.225.13.189', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/E60A8A9F-16C6-DE11-83F0-001D09F291D2.root', 'uid': 0, 'uname': 'root', 'mtime': 1257051679, 'rmajor': 0, 'machine': ('Linux', 'cmsstor79.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 125853680, 27L, 1, 9811, 5063, 512, 1257051251, 1257051251, 1257051006), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 3947781132L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data1/write-pool-1/data/000700000000000061806408', 'inode': 125854728L, 'minor': 27}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/000700000000000061806408', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(000700000000000061806408)', 'fc': {'size': 3947781132L, 'external_label': 'VOQ828', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data1/write-pool-1/data/000700000000000061806408', 'at_the_top': 10, 'local_inode': 1073742569, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.205.201', 60545), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257051681.2435961, 'encp_start_time': 1257051678.69889, 'job_queued': 1257051679.6657131, 't0': 1257051678, 'in_queue': 1.5778398513793945}, 'user_level': 'admin', 'unique_id': 'cmsstor79.fnal.gov-1257051679-12910-0'} Thread client_connect_thread 00:01:21 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:21 cmsstor79.fnal.gov 012910 root I ENCP Control socket ('131.225.205.201', 60545) is connected to ('131.225.13.189', 54734) for cmsstor79.fnal.gov-1257051679-12910-0. elapsed=2.590sec 00:01:21 cmsstor79.fnal.gov 012910 root I ENCP Received callback ticket from mover LTO4_94.mover for transfer cmsstor79.fnal.gov-1257051679-12910-0. elapsed=2.590sec 00:01:21 cmsstor79.fnal.gov 012910 root I ENCP Atempting to connect data socket to mover at ('131.225.13.189', 54733). 00:01:21 cmsstor79.fnal.gov 012910 root I ENCP Connecting data socket to mover (('131.225.13.189', 54733)) with interface 131.225.205.201. elapsed=2.590sec 00:01:21 cmsstor79.fnal.gov 012910 root I ENCP Data socket ('131.225.205.201', 60466) is connected to ('131.225.13.189', 54733). elapsed=2.600sec 00:01:21 cmsstor79.fnal.gov 012910 root I ENCP Opening local file. 00:01:21 cmsstor79.fnal.gov 012910 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:01:21 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV tape position: target 139 current 139 0 Thread seek_thread 00:01:21 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV write_tape starting, bytes_to_write=3947781632 Thread tape_thread 00:01:21 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV Write starting. Tape VOQ828 absolute location in blocks 3180062 Thread tape_thread 00:01:22 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:22 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ922 0,7,1,2 00:01:22 stkenmvr196a.fnal.gov 027637 root E LTO4_96MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:22 stkenmvr196a.fnal.gov 027637 root I LTO4_96MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:22 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:01:22 stkenmvr144a.fnal.gov 006329 root I LTO4_44MV tape position: target 251 current 0 1 Thread seek_thread 00:01:22 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ827 0,7,1,11 00:01:22 stkenmvr193a.fnal.gov 031647 root E LTO4_93MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:22 stkenmvr193a.fnal.gov 031647 root I LTO4_93MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:24 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:24 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:01:24 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOS188 0,6,1,12 00:01:24 stkenmvr146a.fnal.gov 005990 root E LTO4_46MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:24 stkenmvr146a.fnal.gov 005990 root I LTO4_46MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:24 cmsstor187.fnal.gov 019936 root I ENCP File /storage/data3/write-pool-3/data/0010000000000000094DDB18 transfered for cmsstor187.fnal.gov-1257051678-19936-0 in 3.85522818565 seconds. 00:01:24 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV written bytes 428897792/428897792, blocks 3273 header 238 trailer 428 Thread tape_thread 00:01:26 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:26 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV filemarks written. Tape VOU867 absolute location in blocks 737597 Thread tape_thread 00:01:26 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV write_tape timing:pull 0.0383965969086 write 1.23438692093 crc_check 2.00556302071 freespace 0.0426123142242 block_write 3.32095885277 idle 0.0689387321472 Thread tape_thread 00:01:26 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV drive stats after write. Tape VOU867 position 593 block 0 block_size 0 bloc_loc 737597 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 428897792 block_write_tot 3.32095885277 tape_rate 129148782.329 Thread tape_thread 00:01:26 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV remaining bytes info in DB 687804416000 estimated 687375518208 reported from drive 687401984000 position 594 Thread tape_thread 00:01:26 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV new bitfile request {'uid': 0, 'external_label': 'VOU867', 'gid': 0, 'location_cookie': '0000_000000000_0000593', 'complete_crc': 2931169465L, 'sanity_cookie': (65536L, 1488608144L), 'size': 428897126L} Thread tape_thread 00:01:26 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV set remaining: VOU867 687401984000 0000_000000000_0000594 Thread tape_thread 00:01:26 last message repeated 1 times 00:01:26 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:01:26 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-01 00:01:26', eod_cookie = '0000_000000000_0000594', non_del_files = '595', remaining_bytes = 687401984000 where label = 'VOU867'; 00:01:26 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV tape position: target 190 current 0 1 Thread seek_thread 00:01:26 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV transfer complete volume=VOU867 location=594 Thread tape_thread 00:01:26 last message repeated 1 times 00:01:26 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '593', modification_time = '2009-11-01 00:01:26' where label = 'VOU867'; 00:01:26 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1244479824.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOU867', 'wrapper': 'cpio_odc', 'remaining_bytes': 687804416000L, 'sum_mounts': 22, 'sum_rd_access': 0, 'file_family_width': 7, 'last_access': 1257051662.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000593', 'capacity_bytes': 858993459200L, 'non_del_files': 594, 'sum_wr_err': 0, 'sum_wr_access': 592, 'file_family': 'cms2', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms2.cpio_odc', 'modification_time': 1257051662.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1256259225.0}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/PATLayer1_RECO_Output_QCDDiJet_Pt1800to2200_24.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 428897126L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7864, 'mc_device': '0,6,1,13', 'compression': 0, 'serial_num': '1310050819', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr204a', 'max_time_in_state': 1200, 'device': 'stkenmvr204a:/dev/rmt/tps0d0n', 'name': 'LTO4_104.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_104.stat', 'mover_address': ('131.225.13.199', 7864), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.199', 54011), 'data_ip': 'stkenmvr204a', 'logname': 'LTO4_104MV', 'hostip': '131.225.13.199', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/PATLayer1_RECO_Output_QCDDiJet_Pt1800to2200_24.root', 'uid': 0, 'uname': 'root', 'mtime': 1257051678, 'rmajor': 0, 'machine': ('Linux', 'cmsstor187.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 273340400, 24L, 1, 12124, 5063, 512, 1257050553, 1257050553, 1257036571), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 428897126L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data3/write-pool-3/data/0010000000000000094DDB18', 'inode': 273537816L, 'minor': 24}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/0010000000000000094DDB18', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/.(access)(0010000000000000094DDB18)', 'fc': {'uid': 0, 'external_label': 'VOU867', 'gid': 0, 'location_cookie': '0000_000000000_0000593', 'complete_crc': 2931169465L, 'bfid': 'CDMS125705168600000', 'sanity_cookie': (65536L, 1488608144L), 'size': 428897126L}, 'infilepath': '/storage/data3/write-pool-3/data/0010000000000000094DDB18', 'at_the_top': 1, 'local_inode': 2544, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.207.10', 42819), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257051678.006125, 'seek_time': 0.029865026473999023, 'transfer_time': 5.7665348052978516, 't0': 1257051678, 'mount_time': 0.018490076065063477, 'lm_dequeued': 1257051680.3608, 'job_queued': 1257051678.6470239, 'drive_call_time': 1.1864054203033447, 'in_queue': 1.7137141227722168, 'drive_transfer_(trunc. 4185) Thread tape_thread 00:01:26 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV Done is sent Thread tape_thread 00:01:26 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV maybe_clean Thread tape_thread 00:01:26 cmsstor187.fnal.gov 019936 root I ENCP Received final dialog for cmsstor187.fnal.gov-1257051678-19936-0. elapsed=8.230sec 00:01:26 cmsstor187.fnal.gov 019936 root I ENCP leaving transfer_file(). 00:01:26 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV returned from maybe_clean Thread tape_thread 00:01:26 cmsstor187.fnal.gov 019936 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:01:26 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:26 stkensrv4n.fnal.gov 005357 enstore E LTO4G1LM MOVER ERROR RQ {'status': ('ok', None), 'returned_work': None, 'library': None, 'mover_type': '', 'current_location': '0000_000000000_0000000', 'ip_map': '', 'operation': None, 'read_only': 0, 'current_time': 1257051686.4176919, 'error_source': None, 'external_label': None, 'state': 'OFFLINE', 'r_a': (('131.225.13.194', 54880), 9296L, '131.225.13.194-54880-1256586929.509013-4866'), 'library_list': [('CD-LTO4G1.library_manager', ('131.225.13.4', 8541))], 'current_priority': (None, None), 'transfer_deficiency': 1, 'time_in_state': 464756.0576338768, 'address': ('131.225.13.194', 7860), 'volume_family': None, 'work': 'mover_error', 'mover': 'LTO4_99.mover', 'volume_status': (['none', 'none'], ['none', 'none']), 'unique_id': None} 00:01:26 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/PATLayer1_RECO_Output_QCDDiJet_Pt1800to2200_24.root', drive = 'stkenmvr204a:/dev/rmt/tps0d0n:1310050819', pnfs_id = '0010000000000000094DDB18' where bfid = 'CDMS125705168600000'; 00:01:26 cmsstor187.fnal.gov 019936 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /storage/data3/write-pool-3/data/0010000000000000094DDB18 -> /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/PATLayer1_RECO_Output_QCDDiJet_Pt1800to2200_24.root: 428897126 bytes copied to VOU867 at 70.8 MB/S (93.2 MB/S network) (82.7 MB/S drive) (106 MB/S disk) mover=LTO4_104.mover drive_id=ULTRIUM-TD4 drive_sn=1310050819 drive_vendor=IBM elapsed=8.5091 {'storage_group': 'cms', 'encp_crc': 2931169465L, 'disk_rate': 106.26471115824108, 'drive_rate': 82.738029032484732, 'driver': 'FTTDriver', 'transfer_rate': 70.76525046533331, 'mover_interface': 'stkenmvr204a', 'overall_rate': 66.911990329912328, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 70.76525046533331, 'encp_ip': '131.225.207.10', 'unique_id': 'cmsstor187.fnal.gov-1257051678-19936-0'} 00:01:26 cmsstor187.fnal.gov 019936 root I ENCP File /pnfs/fnal.gov/usr/cms/WAX/2/sushil/test/QSTAR/Backgrounds/QCDdijet_Added_Akt5/PATTuples_QCDDiJet_Pt1800to2200/PATLayer1_RECO_Output_QCDDiJet_Pt1800to2200_24.root copied successfully. 00:01:26 cmsstor187.fnal.gov 019936 root I ENCP Completed transferring 428897126 bytes in 1 files in 8.51010298729 sec. Overall rate = 66.9 MB/sec. Transfer rate = 70.8 MB/sec. Network rate = 93.2 MB/sec. Drive rate = 82.7 MB/sec. Disk rate = 106 MB/sec. Exit status = 0. 00:01:27 cmsstor94.fnal.gov 027036 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: BACKFILL2 FF Wrapper: cpio_odc FF Width: 15 Current working directory: cmsstor94.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --age-time 30 --delpri 1 --priority 25 --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/010/680219F9-22C6-DE11-9DFC-0030487C5CFA.root --put-cache 000700000000000061806D20 /storage/data2/write-pool-2/data/000700000000000061806D20 00:01:27 cmsstor94.fnal.gov 027036 root I ENCP Creating zero length output files. 00:01:27 cmsstor94.fnal.gov 027036 root I ENCP Sending /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/010/680219F9-22C6-DE11-9DFC-0030487C5CFA.root write request to LM: unique_id: cmsstor94.fnal.gov-1257051687-27036-0 inputfile: /storage/data2/write-pool-2/data/000700000000000061806D20 outputfile: /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/010/.(access)(000700000000000061806D20) 00:01:27 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:01:27 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data2/write-pool-2/data/000700000000000061806D20 (cmsstor94.fnal.gov-1257051687-27036-0) -> /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/010/680219F9-22C6-DE11-9DFC-0030487C5CFA.root : library=CD-LTO4G1 family=BACKFILL2 requester:root volume_family:cms.BACKFILL2.cpio_odc 00:01:27 cmsstor94.fnal.gov 027036 root I ENCP Listening for control socket at: ('131.225.189.94', 51569) 00:01:27 cmsstor94.fnal.gov 027036 root I ENCP Waiting for mover to connect control socket at address: ('131.225.189.94', 51569) elapsed=1.210sec 00:01:28 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:28 cmsstor91.fnal.gov 025432 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: BACKFILL2 FF Wrapper: cpio_odc FF Width: 15 Current working directory: cmsstor91.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --age-time 30 --delpri 1 --priority 25 --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/48D7ADAB-12C6-DE11-843C-000423D60FF6.root --put-cache 0007000000000000618063D8 /storage/data3/write-pool-3/data/0007000000000000618063D8 00:01:28 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:01:28 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:28 cmsstor91.fnal.gov 025432 root I ENCP Creating zero length output files. 00:01:28 cmsstor91.fnal.gov 025432 root I ENCP Sending /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/48D7ADAB-12C6-DE11-843C-000423D60FF6.root write request to LM: unique_id: cmsstor91.fnal.gov-1257051688-25432-0 inputfile: /storage/data3/write-pool-3/data/0007000000000000618063D8 outputfile: /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/.(access)(0007000000000000618063D8) 00:01:28 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:01:28 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data3/write-pool-3/data/0007000000000000618063D8 (cmsstor91.fnal.gov-1257051688-25432-0) -> /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/009/48D7ADAB-12C6-DE11-843C-000423D60FF6.root : library=CD-LTO4G1 family=BACKFILL2 requester:root volume_family:cms.BACKFILL2.cpio_odc 00:01:28 cmsstor91.fnal.gov 025432 root I ENCP Listening for control socket at: ('131.225.205.213', 58823) 00:01:28 cmsstor91.fnal.gov 025432 root I ENCP Waiting for mover to connect control socket at address: ('131.225.205.213', 58823) elapsed=1.190sec 00:01:29 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOU507 0,5,1,2 00:01:29 stkenmvr150a.fnal.gov 006006 root E LTO4_50MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:29 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:29 stkensrv4n.fnal.gov 020306 enstore I SL8500G1MC query vol VON584 => 0,VON584 home 0,10,32, 2, 1 LTO-800G 00:01:29 cmsstor154.fnal.gov 006316 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: BACKFILL2 FF Wrapper: cpio_odc FF Width: 15 Current working directory: cmsstor154.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --age-time 30 --delpri 1 --priority 25 --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/010/7CB34B42-1EC6-DE11-A88B-000423D6C8EE.root --put-cache 000700000000000061806A28 /storage/data3/write-pool-3/data/000700000000000061806A28 00:01:29 stkensrv4n.fnal.gov 020296 enstore I SL8500G1MC query drive 0,7,1,1 => 0,0,7,1,1 online available IBM-LTO4 00:01:30 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:30 cmsstor154.fnal.gov 006316 root I ENCP Creating zero length output files. 00:01:30 cmsstor154.fnal.gov 006316 root I ENCP Sending /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/010/7CB34B42-1EC6-DE11-A88B-000423D6C8EE.root write request to LM: unique_id: cmsstor154.fnal.gov-1257051689-6316-0 inputfile: /storage/data3/write-pool-3/data/000700000000000061806A28 outputfile: /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/010/.(access)(000700000000000061806A28) 00:01:30 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:01:30 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data3/write-pool-3/data/000700000000000061806A28 (cmsstor154.fnal.gov-1257051689-6316-0) -> /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Ele/RAW/v10_31_pre20replaythingy_v15/000/000/010/7CB34B42-1EC6-DE11-A88B-000423D6C8EE.root : library=CD-LTO4G1 family=BACKFILL2 requester:root volume_family:cms.BACKFILL2.cpio_odc 00:01:30 cmsstor154.fnal.gov 006316 root I ENCP Listening for control socket at: ('131.225.189.125', 58431) 00:01:30 cmsstor154.fnal.gov 006316 root I ENCP Waiting for mover to connect control socket at address: ('131.225.189.125', 58431) elapsed=0.790sec 00:01:30 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:01:30 cmsstor116.fnal.gov 031489 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Scientific Linux SL release 5.2 (Boron) Rocks release 5.1 (V.I) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: BACKFILL2 FF Wrapper: cpio_odc FF Width: 15 Current working directory: cmsstor116.fnal.gov:/root Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --age-time 30 --delpri 1 --priority 25 --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/BA921D51-F8C5-DE11-B38D-001D09F25401.root --put-cache 000700000000000061806788 /storage/data4/write-pool-4/data/000700000000000061806788 00:01:30 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOQ856 0,6,1,1 00:01:30 stkenmvr205a.fnal.gov 011191 root E LTO4_105MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:30 stkenmvr205a.fnal.gov 011191 root I LTO4_105MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:30 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOQ834 0,7,1,14 00:01:30 stkenmvr195a.fnal.gov 021479 root E LTO4_95MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:30 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:30 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:30 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ825 0,6,1,8 00:01:30 stkenmvr206a.fnal.gov 026766 root E LTO4_106MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:30 stkenmvr206a.fnal.gov 026766 root I LTO4_106MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:30 cmsstor116.fnal.gov 031489 root I ENCP Creating zero length output files. 00:01:30 cmsstor116.fnal.gov 031489 root I ENCP Sending /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/BA921D51-F8C5-DE11-B38D-001D09F25401.root write request to LM: unique_id: cmsstor116.fnal.gov-1257051690-31489-0 inputfile: /storage/data4/write-pool-4/data/000700000000000061806788 outputfile: /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/.(access)(000700000000000061806788) 00:01:30 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:01:30 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data4/write-pool-4/data/000700000000000061806788 (cmsstor116.fnal.gov-1257051690-31489-0) -> /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/BA921D51-F8C5-DE11-B38D-001D09F25401.root : library=CD-LTO4G1 family=BACKFILL2 requester:root volume_family:cms.BACKFILL2.cpio_odc 00:01:30 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ859 0,7,1,3 00:01:30 stkenmvr141a.fnal.gov 013963 root E LTO4_41MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:30 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:30 cmsstor116.fnal.gov 031489 root I ENCP Listening for control socket at: ('131.225.189.65', 60678) 00:01:30 cmsstor116.fnal.gov 031489 root I ENCP Waiting for mover to connect control socket at address: ('131.225.189.65', 60678) elapsed=1.090sec 00:01:31 cmsstor229.fnal.gov 032233 root I ENCP File /storage/data1/write-pool-1/data/000700000000000061806428 transfered for cmsstor229.fnal.gov-1257051644-32233-0 in 33.7065150738 seconds. 00:01:32 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:32 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ922 0,7,1,2 00:01:32 stkenmvr196a.fnal.gov 027637 root E LTO4_96MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:32 stkenmvr196a.fnal.gov 027637 root I LTO4_96MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:32 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:01:32 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 dismount VOQ827 0,7,1,11 00:01:32 stkenmvr193a.fnal.gov 031647 root E LTO4_93MV unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:32 stkenmvr193a.fnal.gov 031647 root I LTO4_93MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:32 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:32 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:33 cmsstor188.fnal.gov 008129 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Scientific Linux SL release 5.2 (Boron) Rocks release 5.1 (V.I) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: BACKFILL2 FF Wrapper: cpio_odc FF Width: 15 Current working directory: cmsstor188.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --age-time 30 --delpri 1 --priority 25 --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/782E3525-F4C5-DE11-B3B6-0019B9F705A3.root --put-cache 000700000000000061806720 /storage/data1/write-pool-1/data/000700000000000061806720 00:01:33 cmsstor188.fnal.gov 008129 root I ENCP Creating zero length output files. 00:01:33 cmsstor188.fnal.gov 008129 root I ENCP Sending /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/782E3525-F4C5-DE11-B3B6-0019B9F705A3.root write request to LM: unique_id: cmsstor188.fnal.gov-1257051693-8129-0 inputfile: /storage/data1/write-pool-1/data/000700000000000061806720 outputfile: /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/.(access)(000700000000000061806720) 00:01:33 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:01:33 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data1/write-pool-1/data/000700000000000061806720 (cmsstor188.fnal.gov-1257051693-8129-0) -> /pnfs/fnal.gov/usr/cms/WAX//11/store/backfill/2/data/IansMagicMushroomSoup-T0Test-AnalyzeThisAndGetAFreeBigFatHairyPhD/Photon/RAW/v10_31_pre20replaythingy_v15/000/000/006/782E3525-F4C5-DE11-B3B6-0019B9F705A3.root : library=CD-LTO4G1 family=BACKFILL2 requester:root volume_family:cms.BACKFILL2.cpio_odc 00:01:33 cmsstor188.fnal.gov 008129 root I ENCP Listening for control socket at: ('131.225.207.11', 56414) 00:01:33 cmsstor188.fnal.gov 008129 root I ENCP Waiting for mover to connect control socket at address: ('131.225.207.11', 56414) elapsed=0.690sec 00:01:33 stkenmvr151a.fnal.gov 013228 root I LTO4_51MV written bytes 3996955136/3996955136, blocks 30495 header 285 trailer 142 Thread tape_thread 00:01:34 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:34 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:34 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:01:34 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VOS188 0,6,1,12 00:01:34 stkenmvr146a.fnal.gov 005990 root E LTO4_46MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:34 stkenmvr146a.fnal.gov 005990 root I LTO4_46MV Media Changer returned MEDIA_CHANGER_QUEUE_FU