00:00:00 cmsstor187.fnal.gov 000870 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Scientific Linux SL release 5.2 (Boron) Rocks release 5.1 (V.I) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms11 FF Wrapper: cpio_odc FF Width: 10 Current working directory: cmsstor187.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --priority 25 --pnfs-mount /pnfs/fs --put-cache 000700000000000061D69938 /storage/data3/write-pool-3/data/000700000000000061D69938 00:00:00 cmsstor187.fnal.gov 000870 root I ENCP Creating zero length output files. 00:00:00 cmsstor187.fnal.gov 000870 root I ENCP Sending /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/6673876A-3BC8-DE11-B93C-001A92971B12.root write request to LM: unique_id: cmsstor187.fnal.gov-1257228000-870-0 inputfile: /storage/data3/write-pool-3/data/000700000000000061D69938 outputfile: /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D69938) 00:00:00 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:00 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data3/write-pool-3/data/000700000000000061D69938 (cmsstor187.fnal.gov-1257228000-870-0) -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/6673876A-3BC8-DE11-B93C-001A92971B12.root : library=CD-LTO4G1 family=cms11 requester:root volume_family:cms.cms11.cpio_odc 00:00:00 cmsstor187.fnal.gov 000870 root I ENCP Listening for control socket at: ('131.225.207.10', 37602) 00:00:00 cmsstor187.fnal.gov 000870 root I ENCP Waiting for mover to connect control socket at address: ('131.225.207.10', 37602) elapsed=0.840sec 00:00:01 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC PROCESSING listDrives 00:00:01 stkensrv4n.fnal.gov 005843 enstore I SL8500F1MC PROCESSING listDrives 00:00:01 stkensrv4n.fnal.gov 005852 enstore I STKMC PROCESSING listDrives 00:00:01 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:01 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:02 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:03 cmsstor167.fnal.gov 007701 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms11 FF Wrapper: cpio_odc FF Width: 10 Current working directory: cmsstor167.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --priority 25 --pnfs-mount /pnfs/fs --put-cache 000700000000000061D74878 /storage/data3/write-pool-3/data/000700000000000061D74878 00:00:03 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:03 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:03 cmsstor81.fnal.gov 008503 root I ENCP File /storage/data4/write-pool-4/data/000700000000000061D19F70 transfered for cmsstor81.fnal.gov-1257226911-8503-0 in 29.320182085 seconds. 00:00:03 stkensrv4n.fnal.gov 005357 enstore E LTO4G1LM MOVER ERROR RQ {'status': ('ok', None), 'returned_work': None, 'library': None, 'mover_type': '', 'current_location': '0000_000000000_0000000', 'ip_map': '', 'operation': None, 'read_only': 0, 'current_time': 1257228003.8300829, 'error_source': None, 'external_label': None, 'state': 'OFFLINE', 'r_a': (('131.225.13.194', 54880), 12822L, '131.225.13.194-54880-1256586929.509013-4866'), 'library_list': [('CD-LTO4G1.library_manager', ('131.225.13.4', 8541))], 'current_priority': (None, None), 'transfer_deficiency': 1, 'time_in_state': 641073.47002482414, 'address': ('131.225.13.194', 7860), 'volume_family': None, 'work': 'mover_error', 'mover': 'LTO4_99.mover', 'volume_status': (['none', 'none'], ['none', 'none']), 'unique_id': None} 00:00:03 cmsstor167.fnal.gov 007701 root I ENCP Creating zero length output files. 00:00:03 cmsstor167.fnal.gov 007701 root I ENCP Sending /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/B24817FE-3CC8-DE11-9FBF-002618943811.root write request to LM: unique_id: cmsstor167.fnal.gov-1257228003-7701-0 inputfile: /storage/data3/write-pool-3/data/000700000000000061D74878 outputfile: /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D74878) 00:00:03 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:03 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data3/write-pool-3/data/000700000000000061D74878 (cmsstor167.fnal.gov-1257228003-7701-0) -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/B24817FE-3CC8-DE11-9FBF-002618943811.root : library=CD-LTO4G1 family=cms11 requester:root volume_family:cms.cms11.cpio_odc 00:00:03 cmsstor167.fnal.gov 007701 root I ENCP Listening for control socket at: ('131.225.204.214', 52731) 00:00:03 cmsstor167.fnal.gov 007701 root I ENCP Waiting for mover to connect control socket at address: ('131.225.204.214', 52731) elapsed=1.020sec 00:00:04 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:05 stkensrv0n.fnal.gov 023312 enstore I INFSRV start listing all volumes (3) 00:00:05 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:05 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:06 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:07 stkendca22a.fnal.gov 031572 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: garbage Storage Group: test File Family: dcache FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0001 --put-cache 0005000000000000005FA590 /diskb/write-pool-2/data/0005000000000000005FA590 00:00:07 stkendca22a.fnal.gov 031605 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: garbage Storage Group: test File Family: dcache FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0002 --put-cache 0005000000000000005FA5C8 /diskb/write-pool-2/data/0005000000000000005FA5C8 00:00:07 stkendca22a.fnal.gov 031607 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: garbage Storage Group: test File Family: dcache FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0003 --put-cache 0005000000000000005FA5F8 /diskb/write-pool-2/data/0005000000000000005FA5F8 00:00:07 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC FINISHED listDrives returned ('ok', 0, None, '', '') 00:00:07 stkendca22a.fnal.gov 031572 root I ENCP Configuration Server: no such name: 'garbage.library_manager' 00:00:07 stkendca22a.fnal.gov 031572 root E ENCP INFILE=/diskb/write-pool-2/data/0005000000000000005FA590 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0001 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=1.02 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager' 00:00:07 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:07 last message repeated 1 times 00:00:07 stkendca22a.fnal.gov 031605 root I ENCP Configuration Server: no such name: 'garbage.library_manager' 00:00:07 stkendca22a.fnal.gov 031605 root E ENCP INFILE=/diskb/write-pool-2/data/0005000000000000005FA5C8 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0002 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=1.06 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager' 00:00:07 last message repeated 1 times 00:00:07 stkendca22a.fnal.gov 031607 root I ENCP Configuration Server: no such name: 'garbage.library_manager' 00:00:07 stkendca22a.fnal.gov 031607 root E ENCP INFILE=/diskb/write-pool-2/data/0005000000000000005FA5F8 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0003 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=1.09 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager' 00:00:07 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:08 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:08 cmsstor163.fnal.gov 011141 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: Commissioning09 FF Wrapper: cpio_odc FF Width: 15 Current working directory: cmsstor163.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --age-time 30 --delpri 1 --priority 25 --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/62EDF3BB-2FC8-DE11-A620-0019B9F705A3.root --put-cache 000700000000000061D6F420 /storage/data1/write-pool-1/data/000700000000000061D6F420 00:00:08 cmsstor163.fnal.gov 011141 root I ENCP Creating zero length output files. 00:00:08 cmsstor163.fnal.gov 011141 root I ENCP Sending /pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/62EDF3BB-2FC8-DE11-A620-0019B9F705A3.root write request to LM: unique_id: cmsstor163.fnal.gov-1257228008-11141-0 inputfile: /storage/data1/write-pool-1/data/000700000000000061D6F420 outputfile: /pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/.(access)(000700000000000061D6F420) 00:00:08 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:08 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data1/write-pool-1/data/000700000000000061D6F420 (cmsstor163.fnal.gov-1257228008-11141-0) -> /pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/62EDF3BB-2FC8-DE11-A620-0019B9F705A3.root : library=CD-LTO4G1 family=Commissioning09 requester:root volume_family:cms.Commissioning09.cpio_odc 00:00:08 cmsstor163.fnal.gov 011141 root I ENCP Listening for control socket at: ('131.225.204.234', 57121) 00:00:08 cmsstor163.fnal.gov 011141 root I ENCP Waiting for mover to connect control socket at address: ('131.225.204.234', 57121) elapsed=0.830sec 00:00:08 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:08 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV written bytes 2418608640/2418608640, blocks 18453 header 192 trailer 557 Thread tape_thread 00:00:09 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Assigning volume VOQ181 from storage group cms.Commissioning09.cpio_odc to library CD-LTO4G1, volume family cms.Commissioning09.cpio_odc 00:00:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm work on vol=VOQ181 mover=LTO4_106.mover requester:root 00:00:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM IDLE:sending {'vc': {'storage_group': 'cms', 'library': 'CD-LTO4G1', 'file_family': 'Commissioning09', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.Commissioning09.cpio_odc', 'file_family_width': 15}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/62EDF3BB-2FC8-DE11-A620-0019B9F705A3.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2233147480L, 'mover': 'LTO4_106.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/62EDF3BB-2FC8-DE11-A620-0019B9F705A3.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257228008, 'rmajor': 0, 'machine': ('Linux', 'cmsstor163.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16877, 131522040, 21L, 1, 9811, 5063, 512, 1257227735, 1257227735, 1257227606), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 2233147480L, 'fullname': '/storage/data1/write-pool-1/data/000700000000000061D6F420', 'inode': 131527712L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/000700000000000061D6F420', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/.(access)(000700000000000061D6F420)', 'fc': {'size': 2233147480L, 'external_label': 'VOQ181', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data1/write-pool-1/data/000700000000000061D6F420', 'at_the_top': 1, 'local_inode': 6585, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.204.234', 57121), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228009.2884891, 'encp_start_time': 1257228007.696888, 'job_queued': 1257228008.507684, 't0': 1257228007, 'in_queue': 0.78074908256530762}, 'user_level': 'admin', 'unique_id': 'cmsstor163.fnal.gov-1257228008-11141-0'} to mover 00:00:09 stkenmvr206a.fnal.gov 026766 root I LTO4_106MV WRITE_TO_HSM Thread MainThread 00:00:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM can not remove from sg LTO4_106.mover VOQ181 00:00:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM SG: {'cms': [('LTO4_46.mover', 'VOR470'), ('LTO4_40.mover', 'VOR468'), ('LTO4_45.mover', 'VOQ780'), ('LTO4_102.mover', 'VOR469'), ('LTO4_110.mover', 'VOR463'), ('LTO4_94.mover', 'VOQ174'), ('LTO4_95.mover', 'VOR467'), ('LTO4_104.mover', 'VOR466'), ('LTO4_50.mover', 'VOR464'), ('LTO4_48.mover', 'VOR460'), ('LTO4_49.mover', 'VOR473'), ('LTO4_42.mover', 'VOQ806'), ('LTO4_98.mover', 'VOR465')]} 00:00:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM can not remove from vf LTO4_106.mover VOQ181 00:00:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM VF: {'cms.NonCustodial.cpio_odc': [('LTO4_45.mover', 'VOQ780'), ('LTO4_42.mover', 'VOQ806')], 'cms.cms11.cpio_odc': [('LTO4_46.mover', 'VOR470'), ('LTO4_40.mover', 'VOR468'), ('LTO4_102.mover', 'VOR469'), ('LTO4_110.mover', 'VOR463'), ('LTO4_95.mover', 'VOR467'), ('LTO4_104.mover', 'VOR466'), ('LTO4_50.mover', 'VOR464'), ('LTO4_48.mover', 'VOR460'), ('LTO4_49.mover', 'VOR473'), ('LTO4_98.mover', 'VOR465')], 'cms.Commissioning09.cpio_odc': [('LTO4_94.mover', 'VOQ174')]} 00:00:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:09 stkenmvr206a.fnal.gov 026766 root I LTO4_106MV SENDING {'vc': {'storage_group': 'cms', 'library': 'CD-LTO4G1', 'file_family': 'Commissioning09', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.Commissioning09.cpio_odc', 'file_family_width': 15}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/62EDF3BB-2FC8-DE11-A620-0019B9F705A3.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2233147480L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7866, 'mc_device': '0,6,1,8', 'compression': 0, 'serial_num': '1310152619', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr206a', 'max_time_in_state': 1200, 'device': 'stkenmvr206a:/dev/rmt/tps0d0n', 'name': 'LTO4_106.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_106.stat', 'mover_address': ('131.225.13.202', 7866), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.202', 43709), 'data_ip': 'stkenmvr206a', 'logname': 'LTO4_106MV', 'hostip': '131.225.13.202', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/62EDF3BB-2FC8-DE11-A620-0019B9F705A3.root', 'uid': 0, 'uname': 'root', 'mtime': 1257228008, 'rmajor': 0, 'machine': ('Linux', 'cmsstor163.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 131522040, 21L, 1, 9811, 5063, 512, 1257227735, 1257227735, 1257227606), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 2233147480L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data1/write-pool-1/data/000700000000000061D6F420', 'inode': 131527712L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/000700000000000061D6F420', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/.(access)(000700000000000061D6F420)', 'fc': {'size': 2233147480L, 'external_label': 'VOQ181', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data1/write-pool-1/data/000700000000000061D6F420', 'at_the_top': 1, 'local_inode': 6585, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.204.234', 57121), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228009.2884891, 'encp_start_time': 1257228007.696888, 'job_queued': 1257228008.507684, 't0': 1257228007, 'in_queue': 0.78074908256530762}, 'user_level': 'admin', 'unique_id': 'cmsstor163.fnal.gov-1257228008-11141-0'} Thread client_connect_thread 00:00:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:09 cmsstor163.fnal.gov 011141 root I ENCP Control socket ('131.225.204.234', 57121) is connected to ('131.225.13.202', 43710) for cmsstor163.fnal.gov-1257228008-11141-0. elapsed=1.620sec 00:00:09 cmsstor163.fnal.gov 011141 root I ENCP Received callback ticket from mover LTO4_106.mover for transfer cmsstor163.fnal.gov-1257228008-11141-0. elapsed=1.630sec 00:00:09 cmsstor163.fnal.gov 011141 root I ENCP Atempting to connect data socket to mover at ('131.225.13.202', 43709). 00:00:09 cmsstor163.fnal.gov 011141 root I ENCP Connecting data socket to mover (('131.225.13.202', 43709)) with interface 131.225.204.234. elapsed=1.630sec 00:00:09 cmsstor163.fnal.gov 011141 root I ENCP Data socket ('131.225.204.234', 54597) is connected to ('131.225.13.202', 43709). elapsed=1.630sec 00:00:09 cmsstor163.fnal.gov 011141 root I ENCP Opening local file. 00:00:09 stkenmvr206a.fnal.gov 026766 root I LTO4_106MV MSG_TYPE=MC_LOAD_REQ mounting VOQ181 ULTRIUM-TD4 00:00:09 Thread media_thread 00:00:09 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC PROCESSING mount VOQ181 0,6,1,8 00:00:09 cmsstor163.fnal.gov 011141 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:09 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:09 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:10 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:10 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV filemarks written. Tape VOR463 absolute location in blocks 5063822 Thread tape_thread 00:00:10 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV write_tape timing:pull 0.241574048996 write 10.4246163368 crc_check 11.3929617405 freespace 0.200864076614 block_write 22.2600162029 idle 0.356110811234 Thread tape_thread 00:00:10 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV drive stats after write. Tape VOR463 position 224 block 0 block_size 0 bloc_loc 5063822 tot_blocks 762939000 BOT 0 read_err 0 write_err 11 bytes 2418608640 block_write_tot 22.2600162029 tape_rate 108652600.158 Thread tape_thread 00:00:10 cmsstor199.fnal.gov 002911 root I ENCP File /storage/data2/write-pool-2/data/000700000000000061D58B68 transfered for cmsstor199.fnal.gov-1257226937-2911-0 in 24.3901619911 seconds. 00:00:10 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV remaining bytes info in DB 159156224000 estimated 156737615360 reported from drive 156900352000 position 225 Thread tape_thread 00:00:10 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV new bitfile request {'uid': 0, 'external_label': 'VOR463', 'gid': 0, 'location_cookie': '0000_000000000_0000224', 'complete_crc': 3694192912L, 'sanity_cookie': (65536L, 4242598549L), 'size': 2418607891L} Thread tape_thread 00:00:10 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV written bytes 2701029376/2701029376, blocks 20608 header 192 trailer 440 Thread tape_thread 00:00:10 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV set remaining: VOR463 156900352000 0000_000000000_0000225 Thread tape_thread 00:00:10 last message repeated 1 times 00:00:10 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-03 00:00:10', eod_cookie = '0000_000000000_0000225', non_del_files = '225', remaining_bytes = 156900352000 where label = 'VOR463'; 00:00:10 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV transfer complete volume=VOR463 location=225 Thread tape_thread 00:00:10 last message repeated 1 times 00:00:10 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '224', modification_time = '2009-11-03 00:00:10' where label = 'VOR463'; 00:00:10 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1255359021.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOR463', 'wrapper': 'cpio_odc', 'remaining_bytes': 159156224000L, 'sum_mounts': 2, 'sum_rd_access': 0, 'file_family_width': 10, 'last_access': 1257227974.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000224', 'capacity_bytes': 858993459200L, 'non_del_files': 224, 'sum_wr_err': 0, 'sum_wr_access': 223, 'file_family': 'cms11', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'modification_time': 1257227974.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257219432.0}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/8252A61B-34C8-DE11-B48E-001731AF6B79.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2418607891L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7870, 'mc_device': '0,5,1,14', 'compression': 0, 'serial_num': '1310049971', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr210a', 'max_time_in_state': 1200, 'device': 'stkenmvr210a:/dev/rmt/tps0d0n', 'name': 'LTO4_110.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_110.stat', 'mover_address': ('131.225.13.206', 7870), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.206', 33474), 'data_ip': 'stkenmvr210a', 'logname': 'LTO4_110MV', 'hostip': '131.225.13.206', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/8252A61B-34C8-DE11-B48E-001731AF6B79.root', 'uid': 0, 'uname': 'root', 'mtime': 1257226911, 'rmajor': 0, 'machine': ('Linux', 'cmsstor81.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 130813352, 21L, 1, 9811, 5063, 512, 1257226372, 1257226372, 1257220971), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2418607891L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data4/write-pool-4/data/000700000000000061D19F70', 'inode': 131178352L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data4/write-pool-4/data/000700000000000061D19F70', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/.(access)(000700000000000061D19F70)', 'fc': {'uid': 0, 'external_label': 'VOR463', 'gid': 0, 'location_cookie': '0000_000000000_0000224', 'complete_crc': 3694192912L, 'bfid': 'CDMS125722801000000', 'sanity_cookie': (65536L, 4242598549L), 'size': 2418607891L}, 'infilepath': '/storage/data4/write-pool-4/data/000700000000000061D19F70', 'at_the_top': 53, 'local_inode': 1073744313, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.205.203', 47576), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257226910.332824, 'seek_time': 0.029920816421508789, 'transfer_time': 36.049283027648926, 't0': 1257226910, 'mount_time': 0.014466047286987305, 'lm_dequeued': 1257227974.3475471, 'job_queued': 1257226911.853802, 'drive_call_time': 10.165457725524902, 'in_queue': 1062.4934561252594, 'drive_transfer_time': 24.402584075927734}, 'user_level': 'admin', 'unique_id': 'cmsstor81.fnal.gov-1257226911-8503-0'} Thread tape_thread 00:00:10 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV Done is sent Thread tape_thread 00:00:10 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV maybe_clean Thread tape_thread 00:00:10 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV returned from maybe_clean Thread tape_thread 00:00:10 cmsstor81.fnal.gov 008503 root I ENCP Received final dialog for cmsstor81.fnal.gov-1257226911-8503-0. elapsed=1100.300sec 00:00:10 cmsstor81.fnal.gov 008503 root I ENCP leaving transfer_file(). 00:00:10 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV filemarks written. Tape VOR466 absolute location in blocks 3587360 Thread tape_thread 00:00:10 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV write_tape timing:pull 0.21931552887 write 3.95973014832 crc_check 12.3466498852 freespace 0.23988699913 block_write 16.7655825615 idle 0.404032230377 Thread tape_thread 00:00:10 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV drive stats after write. Tape VOR466 position 151 block 0 block_size 0 bloc_loc 3587360 tot_blocks 762939000 BOT 0 read_err 0 write_err 15 bytes 2701029376 block_write_tot 16.7655825615 tape_rate 161105608.236 Thread tape_thread 00:00:10 cmsstor81.fnal.gov 008503 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:10 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV remaining bytes info in DB 337926144000 estimated 335225114624 reported from drive 335404032000 position 152 Thread tape_thread 00:00:10 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV new bitfile request {'uid': 0, 'external_label': 'VOR466', 'gid': 0, 'location_cookie': '0000_000000000_0000151', 'complete_crc': 1643037122L, 'sanity_cookie': (65536L, 3899111030L), 'size': 2701028744L} Thread tape_thread 00:00:10 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV set remaining: VOR466 335404032000 0000_000000000_0000152 Thread tape_thread 00:00:10 last message repeated 1 times 00:00:10 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-03 00:00:10', eod_cookie = '0000_000000000_0000152', non_del_files = '151', remaining_bytes = 335404032000 where label = 'VOR466'; 00:00:10 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/8252A61B-34C8-DE11-B48E-001731AF6B79.root', drive = 'stkenmvr210a:/dev/rmt/tps0d0n:1310049971', pnfs_id = '000700000000000061D19F70' where bfid = 'CDMS125722801000000'; 00:00:10 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV transfer complete volume=VOR466 location=152 Thread tape_thread 00:00:10 last message repeated 1 times 00:00:10 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '151', modification_time = '2009-11-03 00:00:10' where label = 'VOR466'; 00:00:10 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1255359031.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOR466', 'wrapper': 'cpio_odc', 'remaining_bytes': 337926144000L, 'sum_mounts': 2, 'sum_rd_access': 0, 'file_family_width': 10, 'last_access': 1257227985.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000151', 'capacity_bytes': 858993459200L, 'non_del_files': 150, 'sum_wr_err': 0, 'sum_wr_access': 150, 'file_family': 'cms11', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'modification_time': 1257227985.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257221826.0}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/5E9725A7-3AC8-DE11-AEAD-002618943971.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2701028744L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7864, 'mc_device': '0,6,1,13', 'compression': 0, 'serial_num': '1310050819', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr204a', 'max_time_in_state': 1200, 'device': 'stkenmvr204a:/dev/rmt/tps0d0n', 'name': 'LTO4_104.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_104.stat', 'mover_address': ('131.225.13.199', 7864), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.199', 34091), 'data_ip': 'stkenmvr204a', 'logname': 'LTO4_104MV', 'hostip': '131.225.13.199', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/5E9725A7-3AC8-DE11-AEAD-002618943971.root', 'uid': 0, 'uname': 'root', 'mtime': 1257226937, 'rmajor': 0, 'machine': ('Linux', 'cmsstor199.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 131219952, 21L, 1, 9811, 5063, 512, 1257226936, 1257226936, 1257224768), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2701028744L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data2/write-pool-2/data/000700000000000061D58B68', 'inode': 131435368L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/000700000000000061D58B68', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D58B68)', 'fc': {'uid': 0, 'external_label': 'VOR466', 'gid': 0, 'location_cookie': '0000_000000000_0000151', 'complete_crc': 1643037122L, 'bfid': 'CDMS125722801000001', 'sanity_cookie': (65536L, 3899111030L), 'size': 2701028744L}, 'infilepath': '/storage/data2/write-pool-2/data/000700000000000061D58B68', 'at_the_top': 2, 'local_inode': 3794, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.207.24', 45090), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257226936.5530779, 'seek_time': 0.030001163482666016, 'transfer_time': 25.036484956741333, 't0': 1257226936, 'mount_time': 0.014443874359130859, 'lm_dequeued': 1257227985.693408, 'job_queued': 1257226937.532114, 'drive_call_time': 3.66892409324646, 'in_queue': 1048.1610169410706, 'drive_transfer_time': 17.943971872329712}, 'user_level': 'admin', 'unique_id': 'cmsstor199.fnal.gov-1257226937-2911-0'} Thread tape_thread 00:00:10 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV Done is sent Thread tape_thread 00:00:10 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV maybe_clean Thread tape_thread 00:00:10 cmsstor199.fnal.gov 002911 root I ENCP Received final dialog for cmsstor199.fnal.gov-1257226937-2911-0. elapsed=1074.310sec 00:00:10 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV returned from maybe_clean Thread tape_thread 00:00:10 cmsstor199.fnal.gov 002911 root I ENCP leaving transfer_file(). 00:00:10 cmsstor81.fnal.gov 008503 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /storage/data4/write-pool-4/data/000700000000000061D19F70 -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/8252A61B-34C8-DE11-B48E-001731AF6B79.root: 2418607891 bytes copied to VOR463 at 64 MB/S (96.9 MB/S network) (94.5 MB/S drive) (78.8 MB/S disk) mover=LTO4_110.mover drive_id=ULTRIUM-TD4 drive_sn=1310049971 drive_vendor=IBM elapsed=1100.6 {'storage_group': 'cms', 'encp_crc': 3694192912L, 'disk_rate': 78.793697736653044, 'drive_rate': 94.521310527090009, 'driver': 'FTTDriver', 'transfer_rate': 64.023255698065398, 'mover_interface': 'stkenmvr210a', 'overall_rate': 63.188038574152166, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 64.023255698065398, 'encp_ip': '131.225.205.203', 'unique_id': 'cmsstor81.fnal.gov-1257226911-8503-0'} 00:00:10 cmsstor81.fnal.gov 008503 root I ENCP File /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/8252A61B-34C8-DE11-B48E-001731AF6B79.root copied successfully. 00:00:10 cmsstor81.fnal.gov 008503 root I ENCP Completed transferring 2418607891 bytes in 1 files in 1100.62827301 sec. Overall rate = 63.2 MB/sec. Transfer rate = 64 MB/sec. Network rate = 96.9 MB/sec. Drive rate = 94.5 MB/sec. Disk rate = 78.8 MB/sec. Exit status = 0. 00:00:11 cmsstor199.fnal.gov 002911 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:11 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/5E9725A7-3AC8-DE11-AEAD-002618943971.root', drive = 'stkenmvr204a:/dev/rmt/tps0d0n:1310050819', pnfs_id = '000700000000000061D58B68' where bfid = 'CDMS125722801000001'; 00:00:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VOR463 mover=LTO4_110.mover requester:root 00:00:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/4224F208-32C8-DE11-9276-00261894385D.root to mover LTO4_110.mover ('131.225.13.206', 7870) DEL_DISM None 00:00:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'cms', 'external_label': 'VOR463', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/4224F208-32C8-DE11-9276-00261894385D.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2706915438L, 'mover': 'LTO4_110.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/4224F208-32C8-DE11-9276-00261894385D.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257226956, 'rmajor': 0, 'machine': ('Linux', 'cmsstor224.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16893, 130813352, 21L, 1, 9811, 5063, 512, 1257226372, 1257226372, 1257220971), 'mode_octal': '0100036', 'mode': 33188, 'gname': 'root', 'size_bytes': 2706915438L, 'fullname': '/storage/data2/write-pool-2/data/000700000000000061CF8C48', 'inode': 131042376L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/000700000000000061CF8C48', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/.(access)(000700000000000061CF8C48)', 'fc': {'size': 2706915438L, 'external_label': 'VOR463', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data2/write-pool-2/data/000700000000000061CF8C48', 'at_the_top': 184, 'local_inode': 5333, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.191.96', 32826), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228011.3968539, 'encp_start_time': 1257226954.4518671, 'job_queued': 1257226956.5584309, 't0': 1257226954, 'in_queue': 1054.838142156601}, 'user_level': 'admin', 'unique_id': 'cmsstor224.fnal.gov-1257226956-20055-0'} to mover 00:00:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:11 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV WRITE_TO_HSM Thread MainThread 00:00:11 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV SENDING {'vc': {'storage_group': 'cms', 'external_label': 'VOR463', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/4224F208-32C8-DE11-9276-00261894385D.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2706915438L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7870, 'mc_device': '0,5,1,14', 'compression': 0, 'serial_num': '1310049971', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr210a', 'max_time_in_state': 1200, 'device': 'stkenmvr210a:/dev/rmt/tps0d0n', 'name': 'LTO4_110.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_110.stat', 'mover_address': ('131.225.13.206', 7870), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.206', 33477), 'data_ip': 'stkenmvr210a', 'logname': 'LTO4_110MV', 'hostip': '131.225.13.206', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/4224F208-32C8-DE11-9276-00261894385D.root', 'uid': 0, 'uname': 'root', 'mtime': 1257226956, 'rmajor': 0, 'machine': ('Linux', 'cmsstor224.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 130813352, 21L, 1, 9811, 5063, 512, 1257226372, 1257226372, 1257220971), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2706915438L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data2/write-pool-2/data/000700000000000061CF8C48', 'inode': 131042376L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/000700000000000061CF8C48', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/.(access)(000700000000000061CF8C48)', 'fc': {'size': 2706915438L, 'external_label': 'VOR463', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data2/write-pool-2/data/000700000000000061CF8C48', 'at_the_top': 184, 'local_inode': 5333, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.191.96', 32826), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228011.3968539, 'encp_start_time': 1257226954.4518671, 'job_queued': 1257226956.5584309, 't0': 1257226954, 'in_queue': 1054.838142156601}, 'user_level': 'admin', 'unique_id': 'cmsstor224.fnal.gov-1257226956-20055-0'} Thread client_connect_thread 00:00:11 cmsstor224.fnal.gov 020055 root I ENCP Control socket ('131.225.191.96', 32826) is connected to ('131.225.13.206', 33478) for cmsstor224.fnal.gov-1257226956-20055-0. elapsed=1056.980sec 00:00:11 cmsstor224.fnal.gov 020055 root I ENCP Received callback ticket from mover LTO4_110.mover for transfer cmsstor224.fnal.gov-1257226956-20055-0. elapsed=1056.980sec 00:00:11 cmsstor224.fnal.gov 020055 root I ENCP Atempting to connect data socket to mover at ('131.225.13.206', 33477). 00:00:11 cmsstor224.fnal.gov 020055 root I ENCP Connecting data socket to mover (('131.225.13.206', 33477)) with interface 131.225.191.96. elapsed=1056.980sec 00:00:11 cmsstor224.fnal.gov 020055 root I ENCP Data socket ('131.225.191.96', 35763) is connected to ('131.225.13.206', 33477). elapsed=1056.980sec 00:00:11 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV tape position: target 225 current 225 0 Thread seek_thread 00:00:11 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV write_tape starting, bytes_to_write=2706915840 Thread tape_thread 00:00:11 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV Write starting. Tape VOR463 absolute location in blocks 5063822 Thread tape_thread 00:00:11 cmsstor224.fnal.gov 020055 root I ENCP Opening local file. 00:00:11 cmsstor199.fnal.gov 002911 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /storage/data2/write-pool-2/data/000700000000000061D58B68 -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/5E9725A7-3AC8-DE11-AEAD-002618943971.root: 2701028744 bytes copied to VOR466 at 103 MB/S (90.7 MB/S network) (144 MB/S drive) (106 MB/S disk) mover=LTO4_104.mover drive_id=ULTRIUM-TD4 drive_sn=1310050819 drive_vendor=IBM elapsed=1075 {'storage_group': 'cms', 'encp_crc': 1643037122L, 'disk_rate': 105.69724362249049, 'drive_rate': 143.55248433040461, 'driver': 'FTTDriver', 'transfer_rate': 102.87112673953567, 'mover_interface': 'stkenmvr204a', 'overall_rate': 99.674167249369333, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 102.87112673953567, 'encp_ip': '131.225.207.24', 'unique_id': 'cmsstor199.fnal.gov-1257226937-2911-0'} 00:00:11 cmsstor199.fnal.gov 002911 root I ENCP File /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/5E9725A7-3AC8-DE11-AEAD-002618943971.root copied successfully. 00:00:11 cmsstor199.fnal.gov 002911 root I ENCP Completed transferring 2701028744 bytes in 1 files in 1075.04187107 sec. Overall rate = 99.7 MB/sec. Transfer rate = 103 MB/sec. Network rate = 90.7 MB/sec. Drive rate = 144 MB/sec. Disk rate = 106 MB/sec. Exit status = 0. 00:00:11 cmsstor224.fnal.gov 020055 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:11 stkensrv4n.fnal.gov 005174 enstore I SL8500G1MC query drive 0,6,1,8 => 0,0,6,1,8 online available IBM-LTO4 00:00:11 stkensrv4n.fnal.gov 005174 enstore I SL8500G1MC Dismount 14 ignored: Drive 0,6,1,8 is empty. Thought VOQ181 was there =>. dismount VOLUME 0,6,1,8 force => ok,0,6,1,8 online available IBM-LTO4 00:00:11 last message repeated 1 times 00:00:11 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VOR466 mover=LTO4_104.mover requester:root 00:00:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/D0DC4BDC-36C8-DE11-A678-003048678D52.root to mover LTO4_104.mover ('131.225.13.199', 7864) DEL_DISM None 00:00:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'cms', 'external_label': 'VOR466', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/D0DC4BDC-36C8-DE11-A678-003048678D52.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2994385853L, 'mover': 'LTO4_104.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/D0DC4BDC-36C8-DE11-A678-003048678D52.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257226957, 'rmajor': 0, 'machine': ('Linux', 'cmsstor88.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16893, 130813352, 21L, 1, 9811, 5063, 512, 1257226372, 1257226372, 1257220971), 'mode_octal': '0100036', 'mode': 33188, 'gname': 'root', 'size_bytes': 2994385853L, 'fullname': '/storage/data1/write-pool-1/data/000700000000000061D337D0', 'inode': 131282896L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/000700000000000061D337D0', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/.(access)(000700000000000061D337D0)', 'fc': {'size': 2994385853L, 'external_label': 'VOR466', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data1/write-pool-1/data/000700000000000061D337D0', 'at_the_top': 2, 'local_inode': 3583, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.189.88', 38263), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228011.726547, 'encp_start_time': 1257226955.6078351, 'job_queued': 1257226957.494972, 't0': 1257226955, 'in_queue': 1054.2313029766083}, 'user_level': 'admin', 'unique_id': 'cmsstor88.fnal.gov-1257226957-6559-0'} to mover 00:00:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:11 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV WRITE_TO_HSM Thread MainThread 00:00:11 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV SENDING {'vc': {'storage_group': 'cms', 'external_label': 'VOR466', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/D0DC4BDC-36C8-DE11-A678-003048678D52.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2994385853L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7864, 'mc_device': '0,6,1,13', 'compression': 0, 'serial_num': '1310050819', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr204a', 'max_time_in_state': 1200, 'device': 'stkenmvr204a:/dev/rmt/tps0d0n', 'name': 'LTO4_104.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_104.stat', 'mover_address': ('131.225.13.199', 7864), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.199', 34093), 'data_ip': 'stkenmvr204a', 'logname': 'LTO4_104MV', 'hostip': '131.225.13.199', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/D0DC4BDC-36C8-DE11-A678-003048678D52.root', 'uid': 0, 'uname': 'root', 'mtime': 1257226957, 'rmajor': 0, 'machine': ('Linux', 'cmsstor88.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 130813352, 21L, 1, 9811, 5063, 512, 1257226372, 1257226372, 1257220971), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2994385853L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data1/write-pool-1/data/000700000000000061D337D0', 'inode': 131282896L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/000700000000000061D337D0', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/.(access)(000700000000000061D337D0)', 'fc': {'size': 2994385853L, 'external_label': 'VOR466', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data1/write-pool-1/data/000700000000000061D337D0', 'at_the_top': 2, 'local_inode': 3583, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.189.88', 38263), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228011.726547, 'encp_start_time': 1257226955.6078351, 'job_queued': 1257226957.494972, 't0': 1257226955, 'in_queue': 1054.2313029766083}, 'user_level': 'admin', 'unique_id': 'cmsstor88.fnal.gov-1257226957-6559-0'} Thread client_connect_thread 00:00:11 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:11 cmsstor88.fnal.gov 006559 root I ENCP Control socket ('131.225.189.88', 38263) is connected to ('131.225.13.199', 34094) for cmsstor88.fnal.gov-1257226957-6559-0. elapsed=1056.400sec 00:00:11 cmsstor88.fnal.gov 006559 root I ENCP Received callback ticket from mover LTO4_104.mover for transfer cmsstor88.fnal.gov-1257226957-6559-0. elapsed=1056.410sec 00:00:11 cmsstor88.fnal.gov 006559 root I ENCP Atempting to connect data socket to mover at ('131.225.13.199', 34093). 00:00:11 cmsstor88.fnal.gov 006559 root I ENCP Connecting data socket to mover (('131.225.13.199', 34093)) with interface 131.225.189.88. elapsed=1056.410sec 00:00:11 cmsstor88.fnal.gov 006559 root I ENCP Data socket ('131.225.189.88', 59186) is connected to ('131.225.13.199', 34093). elapsed=1056.410sec 00:00:11 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV tape position: target 152 current 152 0 Thread seek_thread 00:00:11 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV write_tape starting, bytes_to_write=2994386432 Thread tape_thread 00:00:11 cmsstor88.fnal.gov 006559 root I ENCP Opening local file. 00:00:11 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV Write starting. Tape VOR466 absolute location in blocks 3587360 Thread tape_thread 00:00:12 cmsstor88.fnal.gov 006559 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:12 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:12 stkensrv4n.fnal.gov 005852 enstore I STKMC FINISHED listDrives returned ('ok', 0, None, '', '') 00:00:12 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:12 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:12 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:00:13 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:13 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:14 stkensrv4n.fnal.gov 005174 enstore I SL8500G1MC query vol VOQ181 => 0,VOQ181 home 0, 5,25, 4, 0 LTO-800G 00:00:14 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:14 cmsstor81.fnal.gov 021774 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms11 FF Wrapper: cpio_odc FF Width: 10 Current working directory: cmsstor81.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --priority 25 --pnfs-mount /pnfs/fs --put-cache 000700000000000061D37458 /storage/data2/write-pool-2/data/000700000000000061D37458 00:00:15 cmsstor81.fnal.gov 021774 root I ENCP Creating zero length output files. 00:00:15 cmsstor81.fnal.gov 021774 root I ENCP Sending /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/FA0F69B7-36C8-DE11-8809-001731AF6AC5.root write request to LM: unique_id: cmsstor81.fnal.gov-1257228015-21774-0 inputfile: /storage/data2/write-pool-2/data/000700000000000061D37458 outputfile: /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D37458) 00:00:15 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:15 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data2/write-pool-2/data/000700000000000061D37458 (cmsstor81.fnal.gov-1257228015-21774-0) -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/FA0F69B7-36C8-DE11-8809-001731AF6AC5.root : library=CD-LTO4G1 family=cms11 requester:root volume_family:cms.cms11.cpio_odc 00:00:15 cmsstor81.fnal.gov 021774 root I ENCP Listening for control socket at: ('131.225.205.203', 51290) 00:00:15 cmsstor81.fnal.gov 021774 root I ENCP Waiting for mover to connect control socket at address: ('131.225.205.203', 51290) elapsed=1.440sec 00:00:15 stkensrv4n.fnal.gov 005843 enstore I SL8500F1MC FINISHED listDrives returned ('ok', 0, None, '', '') 00:00:15 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:15 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:16 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:16 stkenmvr142a.fnal.gov 009485 root I LTO4_42MV Updating stats Thread media_thread 00:00:16 stkensrv4n.fnal.gov 005174 enstore I SL8500G1MC query drive 0,6,1,8 => 0,0,6,1,8 online available IBM-LTO4 00:00:16 stkenmvr142a.fnal.gov 009485 root I LTO4_42MV writing a tape mark before dismount Thread media_thread 00:00:17 cmsstor69.fnal.gov 010776 root I ENCP File /storage/data2/write-pool-2/data/000700000000000061D54CA8 transfered for cmsstor69.fnal.gov-1257226942-10776-0 in 25.3074150085 seconds. 00:00:17 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:17 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:17 cmsstor82.fnal.gov 006961 root I ENCP File /storage/data4/write-pool-4/data/000700000000000061D343D8 transfered for cmsstor82.fnal.gov-1257226928-6961-0 in 33.4056181908 seconds. 00:00:18 cmsstor75.fnal.gov 002762 root I ENCP File /storage/data4/write-pool-4/data/000700000000000061D455C8 transfered for cmsstor75.fnal.gov-1257226892-2762-0 in 55.5774548054 seconds. 00:00:18 stkensrv0n.fnal.gov 023312 enstore I INFSRV stop listing all volumes (3) 00:00:18 cmsstor237.fnal.gov 012370 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Scientific Linux SL release 5.2 (Boron) Rocks release 5.1 (V.I) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms11 FF Wrapper: cpio_odc FF Width: 10 Current working directory: cmsstor237.fnal.gov:/root Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --priority 25 --pnfs-mount /pnfs/fs --put-cache 000700000000000061D6DE90 /storage/data3/write-pool-3/data/000700000000000061D6DE90 00:00:18 cmsstor237.fnal.gov 012370 root I ENCP Creating zero length output files. 00:00:18 cmsstor237.fnal.gov 012370 root I ENCP Sending /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/D234D9BE-3AC8-DE11-AAC3-003048678BC6.root write request to LM: unique_id: cmsstor237.fnal.gov-1257228018-12370-0 inputfile: /storage/data3/write-pool-3/data/000700000000000061D6DE90 outputfile: /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D6DE90) 00:00:18 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:18 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data3/write-pool-3/data/000700000000000061D6DE90 (cmsstor237.fnal.gov-1257228018-12370-0) -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/D234D9BE-3AC8-DE11-AAC3-003048678BC6.root : library=CD-LTO4G1 family=cms11 requester:root volume_family:cms.cms11.cpio_odc 00:00:18 cmsstor237.fnal.gov 012370 root I ENCP Listening for control socket at: ('131.225.191.109', 49119) 00:00:18 cmsstor237.fnal.gov 012370 root I ENCP Waiting for mover to connect control socket at address: ('131.225.191.109', 49119) elapsed=1.000sec 00:00:18 cmsstor59.fnal.gov 015330 root I ENCP File /storage/data1/write-pool-1/data/000700000000000061D57A58 transfered for cmsstor59.fnal.gov-1257226949-15330-0 in 21.3856680393 seconds. 00:00:19 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV written bytes 3129242112/3129242112, blocks 23875 header 192 trailer 226 Thread tape_thread 00:00:19 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:19 cmsstor178.fnal.gov 026913 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Scientific Linux SL release 5.2 (Boron) Rocks release 5.1 (V.I) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms11 FF Wrapper: cpio_odc FF Width: 10 Current working directory: cmsstor178.fnal.gov:/root Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --priority 25 --pnfs-mount /pnfs/fs --put-cache 000700000000000061D72730 /storage/data2/write-pool-2/data/000700000000000061D72730 00:00:19 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:19 stkenmvr142a.fnal.gov 009485 root I LTO4_42MV Ejecting tape Thread media_thread 00:00:20 cmsstor178.fnal.gov 026913 root I ENCP Creating zero length output files. 00:00:20 cmsstor178.fnal.gov 026913 root I ENCP Sending /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/940A11F7-3CC8-DE11-94BB-00304867BFB0.root write request to LM: unique_id: cmsstor178.fnal.gov-1257228019-26913-0 inputfile: /storage/data2/write-pool-2/data/000700000000000061D72730 outputfile: /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D72730) 00:00:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data2/write-pool-2/data/000700000000000061D72730 (cmsstor178.fnal.gov-1257228019-26913-0) -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/940A11F7-3CC8-DE11-94BB-00304867BFB0.root : library=CD-LTO4G1 family=cms11 requester:root volume_family:cms.cms11.cpio_odc 00:00:20 cmsstor178.fnal.gov 026913 root I ENCP Listening for control socket at: ('131.225.206.210', 41574) 00:00:20 cmsstor178.fnal.gov 026913 root I ENCP Waiting for mover to connect control socket at address: ('131.225.206.210', 41574) elapsed=0.770sec 00:00:20 cmsstor107.fnal.gov 017860 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Scientific Linux SL release 5.2 (Boron) Rocks release 5.1 (V.I) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms11 FF Wrapper: cpio_odc FF Width: 10 Current working directory: cmsstor107.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --priority 25 --pnfs-mount /pnfs/fs --put-cache 000700000000000061D75100 /storage/data1/write-pool-1/data/000700000000000061D75100 00:00:20 cmsstor107.fnal.gov 017860 root I ENCP Creating zero length output files. 00:00:20 cmsstor107.fnal.gov 017860 root I ENCP Sending /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/34C7E722-3DC8-DE11-A550-003048678B88.root write request to LM: unique_id: cmsstor107.fnal.gov-1257228020-17860-0 inputfile: /storage/data1/write-pool-1/data/000700000000000061D75100 outputfile: /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D75100) 00:00:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data1/write-pool-1/data/000700000000000061D75100 (cmsstor107.fnal.gov-1257228020-17860-0) -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/34C7E722-3DC8-DE11-A550-003048678B88.root : library=CD-LTO4G1 family=cms11 requester:root volume_family:cms.cms11.cpio_odc 00:00:20 cmsstor107.fnal.gov 017860 root I ENCP Listening for control socket at: ('131.225.205.229', 45732) 00:00:20 cmsstor107.fnal.gov 017860 root I ENCP Waiting for mover to connect control socket at address: ('131.225.205.229', 45732) elapsed=1.220sec 00:00:21 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV filemarks written. Tape VOR467 absolute location in blocks 3334172 Thread tape_thread 00:00:21 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV write_tape timing:pull 0.324285268784 write 9.38025021553 crc_check 14.247679472 freespace 0.280846118927 block_write 24.2330610752 idle 0.476657629013 Thread tape_thread 00:00:21 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV drive stats after write. Tape VOR467 position 140 block 0 block_size 0 bloc_loc 3334172 tot_blocks 762939000 BOT 0 read_err 0 write_err 3 bytes 3129242112 block_write_tot 24.2330610752 tape_rate 129131111.513 Thread tape_thread 00:00:21 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV remaining bytes info in DB 374365184000 estimated 371235941888 reported from drive 371446784000 position 141 Thread tape_thread 00:00:21 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV new bitfile request {'uid': 0, 'external_label': 'VOR467', 'gid': 0, 'location_cookie': '0000_000000000_0000140', 'complete_crc': 1888665990L, 'sanity_cookie': (65536L, 675393142L), 'size': 3129241694L} Thread tape_thread 00:00:21 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV set remaining: VOR467 371446784000 0000_000000000_0000141 Thread tape_thread 00:00:21 last message repeated 1 times 00:00:21 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-03 00:00:21', eod_cookie = '0000_000000000_0000141', non_del_files = '141', remaining_bytes = 371446784000 where label = 'VOR467'; 00:00:21 cmsstor144.fnal.gov 010329 root I ENCP File /storage/data2/write-pool-2/data/000700000000000061D58CC8 transfered for cmsstor144.fnal.gov-1257226948-10329-0 in 24.4229221344 seconds. 00:00:21 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV transfer complete volume=VOR467 location=141 Thread tape_thread 00:00:21 last message repeated 1 times 00:00:21 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '140', modification_time = '2009-11-03 00:00:21' where label = 'VOR467'; 00:00:21 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1255359034.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOR467', 'wrapper': 'cpio_odc', 'remaining_bytes': 374365184000L, 'sum_mounts': 2, 'sum_rd_access': 0, 'file_family_width': 10, 'last_access': 1257227983.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000140', 'capacity_bytes': 858993459200L, 'non_del_files': 140, 'sum_wr_err': 0, 'sum_wr_access': 139, 'file_family': 'cms11', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'modification_time': 1257227983.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257222369.0}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/9C9AB5E3-36C8-DE11-A7D1-003048678B06.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 3129241694L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7856, 'mc_device': '0,7,1,14', 'compression': 0, 'serial_num': '1310155266', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr195a', 'max_time_in_state': 1200, 'device': 'stkenmvr195a:/dev/rmt/tps0d0n', 'name': 'LTO4_95.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_95.stat', 'mover_address': ('131.225.13.190', 7856), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.190', 34718), 'data_ip': 'stkenmvr195a', 'logname': 'LTO4_95MV', 'hostip': '131.225.13.190', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/9C9AB5E3-36C8-DE11-A7D1-003048678B06.root', 'uid': 0, 'uname': 'root', 'mtime': 1257226928, 'rmajor': 0, 'machine': ('Linux', 'cmsstor82.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 130813352, 21L, 1, 9811, 5063, 512, 1257226372, 1257226372, 1257220971), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 3129241694L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data4/write-pool-4/data/000700000000000061D343D8', 'inode': 131285976L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data4/write-pool-4/data/000700000000000061D343D8', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/.(access)(000700000000000061D343D8)', 'fc': {'uid': 0, 'external_label': 'VOR467', 'gid': 0, 'location_cookie': '0000_000000000_0000140', 'complete_crc': 1888665990L, 'bfid': 'CDMS125722802100000', 'sanity_cookie': (65536L, 675393142L), 'size': 3129241694L}, 'infilepath': '/storage/data4/write-pool-4/data/000700000000000061D343D8', 'at_the_top': 100, 'local_inode': 2147485025L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.189.82', 56768), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257226927.6721511, 'seek_time': 0.030250072479248047, 'transfer_time': 36.916579008102417, 't0': 1257226927, 'mount_time': 0.013916969299316406, 'lm_dequeued': 1257227984.3536439, 'job_queued': 1257226928.9336381, 'drive_call_time': 9.0412893295288086, 'in_queue': 1055.4197268486023, 'drive_transfer_time': 26.572129487991333}, 'user_level': 'admin', 'unique_id': 'cmsstor82.fnal.gov-1257226928-6961-0'} Thread tape_thread 00:00:21 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV Done is sent Thread tape_thread 00:00:21 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV maybe_clean Thread tape_thread 00:00:21 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV returned from maybe_clean Thread tape_thread 00:00:21 cmsstor82.fnal.gov 006961 root I ENCP Received final dialog for cmsstor82.fnal.gov-1257226928-6961-0. elapsed=1093.820sec 00:00:21 cmsstor82.fnal.gov 006961 root I ENCP leaving transfer_file(). 00:00:21 cmsstor82.fnal.gov 006961 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:21 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:21 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/9C9AB5E3-36C8-DE11-A7D1-003048678B06.root', drive = 'stkenmvr195a:/dev/rmt/tps0d0n:1310155266', pnfs_id = '000700000000000061D343D8' where bfid = 'CDMS125722802100000'; 00:00:21 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:22 cmsstor82.fnal.gov 006961 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /storage/data4/write-pool-4/data/000700000000000061D343D8 -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/9C9AB5E3-36C8-DE11-A7D1-003048678B06.root: 3129241694 bytes copied to VOR467 at 80.9 MB/S (79.1 MB/S network) (112 MB/S drive) (89.4 MB/S disk) mover=LTO4_95.mover drive_id=ULTRIUM-TD4 drive_sn=1310155266 drive_vendor=IBM elapsed=1094.3 {'storage_group': 'cms', 'encp_crc': 1888665990L, 'disk_rate': 89.368048953515355, 'drive_rate': 112.3085537703729, 'driver': 'FTTDriver', 'transfer_rate': 80.850268142043831, 'mover_interface': 'stkenmvr195a', 'overall_rate': 79.436314153870811, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 80.850268142043831, 'encp_ip': '131.225.189.82', 'unique_id': 'cmsstor82.fnal.gov-1257226928-6961-0'} 00:00:22 cmsstor82.fnal.gov 006961 root I ENCP File /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/9C9AB5E3-36C8-DE11-A7D1-003048678B06.root copied successfully. 00:00:22 cmsstor82.fnal.gov 006961 root I ENCP Completed transferring 3129241694 bytes in 1 files in 1094.33267188 sec. Overall rate = 79.4 MB/sec. Transfer rate = 80.9 MB/sec. Network rate = 79.1 MB/sec. Drive rate = 112 MB/sec. Disk rate = 89.4 MB/sec. Exit status = 0. 00:00:22 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:22 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VOR467 mover=LTO4_95.mover requester:root 00:00:22 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/ACF4ABCC-38C8-DE11-8FD4-003048678FF4.root to mover LTO4_95.mover ('131.225.13.190', 7856) DEL_DISM None 00:00:22 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:22 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'cms', 'external_label': 'VOR467', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/ACF4ABCC-38C8-DE11-8FD4-003048678FF4.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2755906760L, 'mover': 'LTO4_95.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/ACF4ABCC-38C8-DE11-8FD4-003048678FF4.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257226958, 'rmajor': 0, 'machine': ('Linux', 'cmsstor245.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16893, 131219952, 21L, 1, 9811, 5063, 512, 1257226957, 1257226957, 1257224768), 'mode_octal': '0100036', 'mode': 33188, 'gname': 'root', 'size_bytes': 2755906760L, 'fullname': '/storage/data2/write-pool-2/data/000700000000000061D475B8', 'inode': 131364280L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/000700000000000061D475B8', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D475B8)', 'fc': {'size': 2755906760L, 'external_label': 'VOR467', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data2/write-pool-2/data/000700000000000061D475B8', 'at_the_top': 92, 'local_inode': 5081, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.191.117', 57486), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228022.4328079, 'encp_start_time': 1257226957.522341, 'job_queued': 1257226958.9714041, 't0': 1257226957, 'in_queue': 1063.4611349105835}, 'user_level': 'admin', 'unique_id': 'cmsstor245.fnal.gov-1257226958-15098-0'} to mover 00:00:22 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:22 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV WRITE_TO_HSM Thread MainThread 00:00:22 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV SENDING {'vc': {'storage_group': 'cms', 'external_label': 'VOR467', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/ACF4ABCC-38C8-DE11-8FD4-003048678FF4.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2755906760L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7856, 'mc_device': '0,7,1,14', 'compression': 0, 'serial_num': '1310155266', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr195a', 'max_time_in_state': 1200, 'device': 'stkenmvr195a:/dev/rmt/tps0d0n', 'name': 'LTO4_95.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_95.stat', 'mover_address': ('131.225.13.190', 7856), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.190', 34720), 'data_ip': 'stkenmvr195a', 'logname': 'LTO4_95MV', 'hostip': '131.225.13.190', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/ACF4ABCC-38C8-DE11-8FD4-003048678FF4.root', 'uid': 0, 'uname': 'root', 'mtime': 1257226958, 'rmajor': 0, 'machine': ('Linux', 'cmsstor245.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 131219952, 21L, 1, 9811, 5063, 512, 1257226957, 1257226957, 1257224768), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2755906760L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data2/write-pool-2/data/000700000000000061D475B8', 'inode': 131364280L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/000700000000000061D475B8', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D475B8)', 'fc': {'size': 2755906760L, 'external_label': 'VOR467', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data2/write-pool-2/data/000700000000000061D475B8', 'at_the_top': 92, 'local_inode': 5081, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.191.117', 57486), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228022.4328079, 'encp_start_time': 1257226957.522341, 'job_queued': 1257226958.9714041, 't0': 1257226957, 'in_queue': 1063.4611349105835}, 'user_level': 'admin', 'unique_id': 'cmsstor245.fnal.gov-1257226958-15098-0'} Thread client_connect_thread 00:00:22 cmsstor245.fnal.gov 015098 root I ENCP Control socket ('131.225.191.117', 57486) is connected to ('131.225.13.190', 34721) for cmsstor245.fnal.gov-1257226958-15098-0. elapsed=1064.920sec 00:00:22 cmsstor245.fnal.gov 015098 root I ENCP Received callback ticket from mover LTO4_95.mover for transfer cmsstor245.fnal.gov-1257226958-15098-0. elapsed=1064.930sec 00:00:22 cmsstor245.fnal.gov 015098 root I ENCP Atempting to connect data socket to mover at ('131.225.13.190', 34720). 00:00:22 cmsstor245.fnal.gov 015098 root I ENCP Connecting data socket to mover (('131.225.13.190', 34720)) with interface 131.225.191.117. elapsed=1064.930sec 00:00:22 cmsstor245.fnal.gov 015098 root I ENCP Data socket ('131.225.191.117', 52863) is connected to ('131.225.13.190', 34720). elapsed=1064.930sec 00:00:22 cmsstor245.fnal.gov 015098 root I ENCP Opening local file. 00:00:22 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV tape position: target 141 current 141 0 Thread seek_thread 00:00:22 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV write_tape starting, bytes_to_write=2755907072 Thread tape_thread 00:00:22 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV Write starting. Tape VOR467 absolute location in blocks 3334172 Thread tape_thread 00:00:22 cmsstor245.fnal.gov 015098 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:23 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:23 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:23 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:24 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV written bytes 2342144000/2342144000, blocks 17870 header 192 trailer 536 Thread tape_thread 00:00:24 cmsstor53.fnal.gov 014455 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms11 FF Wrapper: cpio_odc FF Width: 10 Current working directory: cmsstor53.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --priority 25 --pnfs-mount /pnfs/fs --put-cache 000700000000000061D58B98 /storage/data4/write-pool-4/data/000700000000000061D58B98 00:00:25 cmsstor53.fnal.gov 014455 root I ENCP Creating zero length output files. 00:00:25 cmsstor53.fnal.gov 014455 root I ENCP Sending /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/EA1353D5-3AC8-DE11-BD1B-002618943843.root write request to LM: unique_id: cmsstor53.fnal.gov-1257228025-14455-0 inputfile: /storage/data4/write-pool-4/data/000700000000000061D58B98 outputfile: /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D58B98) 00:00:25 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:25 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data4/write-pool-4/data/000700000000000061D58B98 (cmsstor53.fnal.gov-1257228025-14455-0) -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/EA1353D5-3AC8-DE11-BD1B-002618943843.root : library=CD-LTO4G1 family=cms11 requester:root volume_family:cms.cms11.cpio_odc 00:00:25 cmsstor53.fnal.gov 014455 root I ENCP Listening for control socket at: ('131.225.204.229', 47043) 00:00:25 cmsstor53.fnal.gov 014455 root I ENCP Waiting for mover to connect control socket at address: ('131.225.204.229', 47043) elapsed=1.300sec 00:00:25 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:25 cmsstor191.fnal.gov 026649 root I ENCP File /storage/data2/write-pool-2/data/000700000000000061D2DF50 transfered for cmsstor191.fnal.gov-1257226938-26649-0 in 35.1460399628 seconds. 00:00:25 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:25 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:26 cmsstor65.fnal.gov 023627 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Scientific Linux SL release 5.2 (Boron) Rocks release 5.1 (V.I) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms11 FF Wrapper: cpio_odc FF Width: 10 Current working directory: cmsstor65.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --priority 25 --pnfs-mount /pnfs/fs --put-cache 000700000000000061D72EF8 /storage/data4/write-pool-4/data/000700000000000061D72EF8 00:00:26 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV filemarks written. Tape VOR468 absolute location in blocks 1387452 Thread tape_thread 00:00:26 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV write_tape timing:pull 0.207036733627 write 9.23860836029 crc_check 9.16845607758 freespace 0.173515796661 block_write 18.7876169682 idle 0.312617540359 Thread tape_thread 00:00:26 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV drive stats after write. Tape VOR468 position 65 block 0 block_size 0 bloc_loc 1387452 tot_blocks 762939000 BOT 0 read_err 0 write_err 24 bytes 2342144000 block_write_tot 18.7876169682 tape_rate 124664240.493 Thread tape_thread 00:00:26 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV remaining bytes info in DB 608172032000 estimated 605829888000 reported from drive 605984768000 position 66 Thread tape_thread 00:00:26 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV new bitfile request {'uid': 0, 'external_label': 'VOR468', 'gid': 0, 'location_cookie': '0000_000000000_0000065', 'complete_crc': 3477730448L, 'sanity_cookie': (65536L, 3107264507L), 'size': 2342143272L} Thread tape_thread 00:00:26 cmsstor65.fnal.gov 023627 root I ENCP Creating zero length output files. 00:00:26 cmsstor65.fnal.gov 023627 root I ENCP Sending /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/F6B02007-3DC8-DE11-BD75-003048678F92.root write request to LM: unique_id: cmsstor65.fnal.gov-1257228026-23627-0 inputfile: /storage/data4/write-pool-4/data/000700000000000061D72EF8 outputfile: /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D72EF8) 00:00:26 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:26 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data4/write-pool-4/data/000700000000000061D72EF8 (cmsstor65.fnal.gov-1257228026-23627-0) -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/F6B02007-3DC8-DE11-BD75-003048678F92.root : library=CD-LTO4G1 family=cms11 requester:root volume_family:cms.cms11.cpio_odc 00:00:26 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV set remaining: VOR468 605984768000 0000_000000000_0000066 Thread tape_thread 00:00:26 cmsstor65.fnal.gov 023627 root I ENCP Listening for control socket at: ('131.225.204.241', 45400) 00:00:26 cmsstor65.fnal.gov 023627 root I ENCP Waiting for mover to connect control socket at address: ('131.225.204.241', 45400) elapsed=1.460sec 00:00:26 last message repeated 1 times 00:00:26 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-03 00:00:26', eod_cookie = '0000_000000000_0000066', non_del_files = '67', remaining_bytes = 605984768000 where label = 'VOR468'; 00:00:26 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV transfer complete volume=VOR468 location=66 Thread tape_thread 00:00:26 last message repeated 1 times 00:00:26 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '65', modification_time = '2009-11-03 00:00:26' where label = 'VOR468'; 00:00:26 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1255359037.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOR468', 'wrapper': 'cpio_odc', 'remaining_bytes': 608172032000L, 'sum_mounts': 2, 'sum_rd_access': 0, 'file_family_width': 10, 'last_access': 1257227961.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000065', 'capacity_bytes': 858993459200L, 'non_del_files': 66, 'sum_wr_err': 0, 'sum_wr_access': 64, 'file_family': 'cms11', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'modification_time': 1257227961.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257225233.0}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0062/C27CD785-38C8-DE11-A6B3-003048678B08.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2342143272L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7840, 'mc_device': '0,7,1,15', 'compression': 0, 'serial_num': '9310801386', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr140a', 'max_time_in_state': 1200, 'device': 'stkenmvr140a:/dev/rmt/tps0d0n', 'name': 'LTO4_40.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_40.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.168', 7840), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.168', 46379), 'data_ip': 'stkenmvr140a', 'logname': 'LTO4_40MV', 'hostip': '131.225.13.168', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0062/C27CD785-38C8-DE11-A6B3-003048678B08.root', 'uid': 0, 'uname': 'root', 'mtime': 1257226892, 'rmajor': 0, 'machine': ('Linux', 'cmsstor75.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 130433880, 21L, 1, 9811, 5063, 512, 1257226874, 1257226874, 1257217225), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2342143272L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data4/write-pool-4/data/000700000000000061D455C8', 'inode': 131356104L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data4/write-pool-4/data/000700000000000061D455C8', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0062/.(access)(000700000000000061D455C8)', 'fc': {'uid': 0, 'external_label': 'VOR468', 'gid': 0, 'location_cookie': '0000_000000000_0000065', 'complete_crc': 3477730448L, 'bfid': 'CDMS125722802600000', 'sanity_cookie': (65536L, 3107264507L), 'size': 2342143272L}, 'infilepath': '/storage/data4/write-pool-4/data/000700000000000061D455C8', 'at_the_top': 44, 'local_inode': 1073744104, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.204.251', 48289), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257226891.2045579, 'seek_time': 0.034985065460205078, 'transfer_time': 64.198529958724976, 't0': 1257226891, 'mount_time': 0.038763999938964844, 'lm_dequeued': 1257227962.3432529, 'job_queued': 1257226892.346678, 'drive_call_time': 9.0149369239807129, 'in_queue': 1069.9962909221649, 'drive_transfer_time': 20.850041389465332}, 'user_level': 'admin', 'unique_id': 'cmsstor(trunc. 4111) Thread tape_thread 00:00:26 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV Done is sent Thread tape_thread 00:00:26 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV maybe_clean Thread tape_thread 00:00:26 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV returned from maybe_clean Thread tape_thread 00:00:26 cmsstor75.fnal.gov 002762 root I ENCP Received final dialog for cmsstor75.fnal.gov-1257226892-2762-0. elapsed=1135.590sec 00:00:26 cmsstor75.fnal.gov 002762 root I ENCP leaving transfer_file(). 00:00:27 cmsstor82.fnal.gov 013752 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms11 FF Wrapper: cpio_odc FF Width: 10 Current working directory: cmsstor82.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --priority 25 --pnfs-mount /pnfs/fs --put-cache 000700000000000061D749F0 /storage/data2/write-pool-2/data/000700000000000061D749F0 00:00:27 cmsstor75.fnal.gov 002762 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:27 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:27 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VOR468 mover=LTO4_40.mover requester:root 00:00:27 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/466ED5C9-3AC8-DE11-81DA-002618FDA208.root to mover LTO4_40.mover ('131.225.13.168', 7840) DEL_DISM None 00:00:27 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:27 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'cms', 'external_label': 'VOR468', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/466ED5C9-3AC8-DE11-81DA-002618FDA208.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2382122093L, 'mover': 'LTO4_40.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/466ED5C9-3AC8-DE11-81DA-002618FDA208.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257226960, 'rmajor': 0, 'machine': ('Linux', 'cmsstor139.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16893, 131219952, 21L, 1, 9811, 5063, 512, 1257226957, 1257226957, 1257224768), 'mode_octal': '0100036', 'mode': 33188, 'gname': 'root', 'size_bytes': 2382122093L, 'fullname': '/storage/data3/write-pool-3/data/000700000000000061D592E8', 'inode': 131437288L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/000700000000000061D592E8', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D592E8)', 'fc': {'size': 2382122093L, 'external_label': 'VOR468', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data3/write-pool-3/data/000700000000000061D592E8', 'at_the_top': 47, 'local_inode': 3618, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.189.110', 45033), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228027.4073009, 'encp_start_time': 1257226959.9829371, 'job_queued': 1257226960.8563781, 't0': 1257226959, 'in_queue': 1066.5506448745728}, 'user_level': 'admin', 'unique_id': 'cmsstor139.fnal.gov-1257226960-6705-0'} to mover 00:00:27 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:27 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0062/C27CD785-38C8-DE11-A6B3-003048678B08.root', drive = 'stkenmvr140a:/dev/rmt/tps0d0n:9310801386', pnfs_id = '000700000000000061D455C8' where bfid = 'CDMS125722802600000'; 00:00:27 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV WRITE_TO_HSM Thread MainThread 00:00:27 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV SENDING {'vc': {'storage_group': 'cms', 'external_label': 'VOR468', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/466ED5C9-3AC8-DE11-81DA-002618FDA208.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2382122093L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7840, 'mc_device': '0,7,1,15', 'compression': 0, 'serial_num': '9310801386', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr140a', 'max_time_in_state': 1200, 'device': 'stkenmvr140a:/dev/rmt/tps0d0n', 'name': 'LTO4_40.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_40.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.168', 7840), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.168', 46386), 'data_ip': 'stkenmvr140a', 'logname': 'LTO4_40MV', 'hostip': '131.225.13.168', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/466ED5C9-3AC8-DE11-81DA-002618FDA208.root', 'uid': 0, 'uname': 'root', 'mtime': 1257226960, 'rmajor': 0, 'machine': ('Linux', 'cmsstor139.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 131219952, 21L, 1, 9811, 5063, 512, 1257226957, 1257226957, 1257224768), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2382122093L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data3/write-pool-3/data/000700000000000061D592E8', 'inode': 131437288L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/000700000000000061D592E8', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D592E8)', 'fc': {'size': 2382122093L, 'external_label': 'VOR468', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data3/write-pool-3/data/000700000000000061D592E8', 'at_the_top': 47, 'local_inode': 3618, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.189.110', 45033), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228027.4073009, 'encp_start_time': 1257226959.9829371, 'job_queued': 1257226960.8563781, 't0': 1257226959, 'in_queue': 1066.5506448745728}, 'user_level': 'admin', 'unique_id': 'cmsstor139.fnal.gov-1257226960-6705-0'} Thread client_connect_thread 00:00:27 cmsstor139.fnal.gov 006705 root I ENCP Control socket ('131.225.189.110', 45033) is connected to ('131.225.13.168', 46387) for cmsstor139.fnal.gov-1257226960-6705-0. elapsed=1067.310sec 00:00:27 cmsstor139.fnal.gov 006705 root I ENCP Received callback ticket from mover LTO4_40.mover for transfer cmsstor139.fnal.gov-1257226960-6705-0. elapsed=1067.310sec 00:00:27 cmsstor139.fnal.gov 006705 root I ENCP Atempting to connect data socket to mover at ('131.225.13.168', 46386). 00:00:27 cmsstor139.fnal.gov 006705 root I ENCP Connecting data socket to mover (('131.225.13.168', 46386)) with interface 131.225.189.110. elapsed=1067.310sec 00:00:27 cmsstor139.fnal.gov 006705 root I ENCP Data socket ('131.225.189.110', 43771) is connected to ('131.225.13.168', 46386). elapsed=1067.330sec 00:00:27 cmsstor139.fnal.gov 006705 root I ENCP Opening local file. 00:00:27 cmsstor82.fnal.gov 013752 root I ENCP Creating zero length output files. 00:00:27 cmsstor82.fnal.gov 013752 root I ENCP Sending /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/2204F404-3DC8-DE11-AB92-002618943946.root write request to LM: unique_id: cmsstor82.fnal.gov-1257228027-13752-0 inputfile: /storage/data2/write-pool-2/data/000700000000000061D749F0 outputfile: /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D749F0) 00:00:27 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:27 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data2/write-pool-2/data/000700000000000061D749F0 (cmsstor82.fnal.gov-1257228027-13752-0) -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/2204F404-3DC8-DE11-AB92-002618943946.root : library=CD-LTO4G1 family=cms11 requester:root volume_family:cms.cms11.cpio_odc 00:00:27 cmsstor139.fnal.gov 006705 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:27 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV tape position: target 66 current 66 0 Thread seek_thread 00:00:27 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV write_tape starting, bytes_to_write=2382122496 Thread tape_thread 00:00:27 cmsstor82.fnal.gov 013752 root I ENCP Listening for control socket at: ('131.225.189.82', 34948) 00:00:27 cmsstor82.fnal.gov 013752 root I ENCP Waiting for mover to connect control socket at address: ('131.225.189.82', 34948) elapsed=1.410sec 00:00:27 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV Write starting. Tape VOR468 absolute location in blocks 1387452 Thread tape_thread 00:00:27 cmsstor75.fnal.gov 002762 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /storage/data4/write-pool-4/data/000700000000000061D455C8 -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0062/C27CD785-38C8-DE11-A6B3-003048678B08.root: 2342143272 bytes copied to VOR468 at 34.8 MB/S (97.5 MB/S network) (107 MB/S drive) (40.2 MB/S disk) mover=LTO4_40.mover drive_id=ULTRIUM-TD4 drive_sn=9310801386 drive_vendor=IBM elapsed=1136.5 {'storage_group': 'cms', 'encp_crc': 3477730448L, 'disk_rate': 40.193747789355754, 'drive_rate': 107.12889447685534, 'driver': 'FTTDriver', 'transfer_rate': 34.824691499075691, 'mover_interface': 'stkenmvr140a', 'overall_rate': 34.238123665617593, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 34.824691499075691, 'encp_ip': '131.225.204.251', 'unique_id': 'cmsstor75.fnal.gov-1257226892-2762-0'} 00:00:27 cmsstor75.fnal.gov 002762 root I ENCP File /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0062/C27CD785-38C8-DE11-A6B3-003048678B08.root copied successfully. 00:00:27 cmsstor75.fnal.gov 002762 root I ENCP Completed transferring 2342143272 bytes in 1 files in 1136.48022914 sec. Overall rate = 34.2 MB/sec. Transfer rate = 34.8 MB/sec. Network rate = 97.5 MB/sec. Drive rate = 107 MB/sec. Disk rate = 40.2 MB/sec. Exit status = 0. 00:00:27 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:27 stkenmvr212a.fnal.gov 006265 root I LTO4_112MV Updating stats Thread media_thread 00:00:27 stkenmvr212a.fnal.gov 006265 root I LTO4_112MV writing a tape mark before dismount Thread media_thread 00:00:27 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:28 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:28 stkensrv4n.fnal.gov 004918 enstore I SL8500G1MC dismount VOLUME 0,6,1,3 force => 0,Dismount: Forced dismount of VOQ780 from 0, 6, 1, 3 00:00:28 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC FINISHED dismount VOQ780 0,6,1,3 returned ('ok', 0, 'dismount VOLUME 0,6,1,3 force => 0,Dismount: Forced dismount of VOQ780 from 0, 6, 1, 3', '', '') 00:00:28 stkenmvr145a.fnal.gov 022974 root I LTO4_45MV dismounted VOQ780 ULTRIUM-TD4 00:00:28 Thread media_thread 00:00:29 stkenmvr149a.fnal.gov 006318 root I LTO4_49MV written bytes 2167039488/2167039488, blocks 16534 header 192 trailer 179 Thread tape_thread 00:00:29 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:30 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV written bytes 2717346304/2717346304, blocks 20732 header 192 trailer 446 Thread tape_thread 00:00:30 stkensrv4n.fnal.gov 004888 enstore I SL8500G1MC mount VOR469 0,6,1,14 => 0,Mount: VOR469 mounted on 0, 6, 1,14 00:00:30 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC FINISHED mount VOR469 0,6,1,14 returned ('ok', 0, 'mount VOR469 0,6,1,14 => 0,Mount: VOR469 mounted on 0, 6, 1,14', '', '') 00:00:30 last message repeated 1 times 00:00:30 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_mounts = '2', last_access = '2009-11-03 00:00:30', first_access = '2009-11-03 00:00:30' where label = 'VOR469'; 00:00:30 stkenmvr202a.fnal.gov 006121 root I LTO4_102MV MSG_TYPE=MC_LOAD_DONE mounted VOR469 ULTRIUM-TD4 00:00:30 Thread media_thread 00:00:30 stkensrv4n.fnal.gov 005795 enstore I 9940BLM 00:00:30 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV written bytes 3910281728/3910281728, blocks 29834 header 192 trailer 95 Thread tape_thread 00:00:30 stkenmvr149a.fnal.gov 006318 root I LTO4_49MV filemarks written. Tape VOR473 absolute location in blocks 1509948 Thread tape_thread 00:00:30 stkenmvr149a.fnal.gov 006318 root I LTO4_49MV write_tape timing:pull 0.212467432022 write 8.52052307129 crc_check 8.56460809708 freespace 0.187907457352 block_write 17.4855060577 idle 0.310869455338 Thread tape_thread 00:00:30 stkenmvr149a.fnal.gov 006318 root I LTO4_49MV drive stats after write. Tape VOR473 position 64 block 0 block_size 0 bloc_loc 1509948 tot_blocks 762939000 BOT 0 read_err 0 write_err 1 bytes 2167039488 block_write_tot 17.4855060577 tape_rate 123933472.72 Thread tape_thread 00:00:31 stkenmvr149a.fnal.gov 006318 root I LTO4_49MV remaining bytes info in DB 593134592000 estimated 590967552512 reported from drive 591117312000 position 65 Thread tape_thread 00:00:31 stkenmvr149a.fnal.gov 006318 root I LTO4_49MV new bitfile request {'uid': 0, 'external_label': 'VOR473', 'gid': 0, 'location_cookie': '0000_000000000_0000064', 'complete_crc': 3093115817L, 'sanity_cookie': (65536L, 2762201813L), 'size': 2167039117L} Thread tape_thread 00:00:31 stkenmvr149a.fnal.gov 006318 root I LTO4_49MV set remaining: VOR473 591117312000 0000_000000000_0000065 Thread tape_thread 00:00:31 last message repeated 1 times 00:00:31 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-03 00:00:31', eod_cookie = '0000_000000000_0000065', non_del_files = '64', remaining_bytes = 591117312000 where label = 'VOR473'; 00:00:31 stkenmvr149a.fnal.gov 006318 root I LTO4_49MV transfer complete volume=VOR473 location=65 Thread tape_thread 00:00:31 last message repeated 1 times 00:00:31 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '64', modification_time = '2009-11-03 00:00:31' where label = 'VOR473'; 00:00:31 stkenmvr149a.fnal.gov 006318 root I LTO4_49MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1255359054.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 1246822737.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOR473', 'wrapper': 'cpio_odc', 'remaining_bytes': 593134592000L, 'sum_mounts': 67, 'sum_rd_access': 0, 'file_family_width': 10, 'last_access': 1257227995.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000064', 'capacity_bytes': 858993459200L, 'non_del_files': 63, 'sum_wr_err': 0, 'sum_wr_access': 63, 'file_family': 'cms11', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'modification_time': 1257227995.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257225495.0}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/700DDD85-3AC8-DE11-BDFF-003048678FC6.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2167039117L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 50, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7849, 'mc_device': '0,5,1,3', 'compression': 0, 'serial_num': '1310154355', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr149a', 'max_time_in_state': 1200, 'device': 'stkenmvr149a:/dev/rmt/tps0d0n', 'name': 'LTO4_49.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_49.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.177', 7849), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.177', 34540), 'data_ip': 'stkenmvr149a', 'logname': 'LTO4_49MV', 'hostip': '131.225.13.177', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/700DDD85-3AC8-DE11-BDFF-003048678FC6.root', 'uid': 0, 'uname': 'root', 'mtime': 1257226949, 'rmajor': 0, 'machine': ('Linux', 'cmsstor59.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 131219952, 21L, 1, 9811, 5063, 512, 1257226946, 1257226946, 1257224768), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2167039117L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data1/write-pool-1/data/000700000000000061D57A58', 'inode': 131431000L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/000700000000000061D57A58', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D57A58)', 'fc': {'uid': 0, 'external_label': 'VOR473', 'gid': 0, 'location_cookie': '0000_000000000_0000064', 'complete_crc': 3093115817L, 'bfid': 'CDMS125722803100000', 'sanity_cookie': (65536L, 2762201813L), 'size': 2167039117L}, 'infilepath': '/storage/data1/write-pool-1/data/000700000000000061D57A58', 'at_the_top': 5, 'local_inode': 1073744444, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.204.235', 51586), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257226948.589783, 'seek_time': 0.033584117889404297, 'transfer_time': 35.563602924346924, 't0': 1257226948, 'mount_time': 0.039058923721313477, 'lm_dequeued': 1257227995.4894471, 'job_queued': 1257226949.8278761, 'drive_call_time': 8.3022029399871826, 'in_queue': 1045.6612908840179, 'drive_transfer_time': 19.5204918384552}, 'user_level': 'admin', 'unique_id': 'cm(trunc. 4117) Thread tape_thread 00:00:31 stkenmvr149a.fnal.gov 006318 root I LTO4_49MV Done is sent Thread tape_thread 00:00:31 stkenmvr149a.fnal.gov 006318 root I LTO4_49MV maybe_clean Thread tape_thread 00:00:31 stkenmvr149a.fnal.gov 006318 root I LTO4_49MV returned from maybe_clean Thread tape_thread 00:00:31 cmsstor59.fnal.gov 015330 root I ENCP Received final dialog for cmsstor59.fnal.gov-1257226949-15330-0. elapsed=1082.930sec 00:00:31 cmsstor59.fnal.gov 015330 root I ENCP leaving transfer_file(). 00:00:31 stkenmvr212a.fnal.gov 006265 root I LTO4_112MV Ejecting tape Thread media_thread 00:00:31 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:31 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VOR473 mover=LTO4_49.mover requester:root 00:00:31 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/14474B7F-36C8-DE11-A2BC-0026189438AE.root to mover LTO4_49.mover ('131.225.13.177', 7849) DEL_DISM None 00:00:31 cmsstor59.fnal.gov 015330 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:31 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:31 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'cms', 'external_label': 'VOR473', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/14474B7F-36C8-DE11-A2BC-0026189438AE.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 3941440637L, 'mover': 'LTO4_49.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/14474B7F-36C8-DE11-A2BC-0026189438AE.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257226966, 'rmajor': 0, 'machine': ('Linux', 'cmsstor95.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16893, 130813352, 21L, 1, 9811, 5063, 512, 1257226372, 1257226372, 1257220971), 'mode_octal': '0100036', 'mode': 33188, 'gname': 'root', 'size_bytes': 3941440637L, 'fullname': '/storage/data3/write-pool-3/data/000700000000000061D2A8C0', 'inode': 131246272L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/000700000000000061D2A8C0', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/.(access)(000700000000000061D2A8C0)', 'fc': {'size': 3941440637L, 'external_label': 'VOR473', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data3/write-pool-3/data/000700000000000061D2A8C0', 'at_the_top': 50, 'local_inode': 1073744024, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.205.217', 40513), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228031.5469439, 'encp_start_time': 1257226965.417119, 'job_queued': 1257226966.562278, 't0': 1257226965, 'in_queue': 1064.984365940094}, 'user_level': 'admin', 'unique_id': 'cmsstor95.fnal.gov-1257226966-22242-0'} to mover 00:00:31 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:31 stkenmvr149a.fnal.gov 006318 root I LTO4_49MV WRITE_TO_HSM Thread MainThread 00:00:31 cmsstor232.fnal.gov 013690 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms11 FF Wrapper: cpio_odc FF Width: 10 Current working directory: cmsstor232.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --priority 25 --pnfs-mount /pnfs/fs --put-cache 000700000000000061D74020 /storage/data2/write-pool-2/data/000700000000000061D74020 00:00:31 stkenmvr149a.fnal.gov 006318 root I LTO4_49MV SENDING {'vc': {'storage_group': 'cms', 'external_label': 'VOR473', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/14474B7F-36C8-DE11-A2BC-0026189438AE.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 3941440637L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 50, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7849, 'mc_device': '0,5,1,3', 'compression': 0, 'serial_num': '1310154355', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr149a', 'max_time_in_state': 1200, 'device': 'stkenmvr149a:/dev/rmt/tps0d0n', 'name': 'LTO4_49.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_49.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.177', 7849), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.177', 34547), 'data_ip': 'stkenmvr149a', 'logname': 'LTO4_49MV', 'hostip': '131.225.13.177', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/14474B7F-36C8-DE11-A2BC-0026189438AE.root', 'uid': 0, 'uname': 'root', 'mtime': 1257226966, 'rmajor': 0, 'machine': ('Linux', 'cmsstor95.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 130813352, 21L, 1, 9811, 5063, 512, 1257226372, 1257226372, 1257220971), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 3941440637L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data3/write-pool-3/data/000700000000000061D2A8C0', 'inode': 131246272L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/000700000000000061D2A8C0', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/.(access)(000700000000000061D2A8C0)', 'fc': {'size': 3941440637L, 'external_label': 'VOR473', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data3/write-pool-3/data/000700000000000061D2A8C0', 'at_the_top': 50, 'local_inode': 1073744024, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.205.217', 40513), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228031.5469439, 'encp_start_time': 1257226965.417119, 'job_queued': 1257226966.562278, 't0': 1257226965, 'in_queue': 1064.984365940094}, 'user_level': 'admin', 'unique_id': 'cmsstor95.fnal.gov-1257226966-22242-0'} Thread client_connect_thread 00:00:31 cmsstor95.fnal.gov 022242 root I ENCP Control socket ('131.225.205.217', 40513) is connected to ('131.225.13.177', 34548) for cmsstor95.fnal.gov-1257226966-22242-0. elapsed=1066.280sec 00:00:31 cmsstor95.fnal.gov 022242 root I ENCP Received callback ticket from mover LTO4_49.mover for transfer cmsstor95.fnal.gov-1257226966-22242-0. elapsed=1066.280sec 00:00:31 cmsstor95.fnal.gov 022242 root I ENCP Atempting to connect data socket to mover at ('131.225.13.177', 34547). 00:00:31 cmsstor95.fnal.gov 022242 root I ENCP Connecting data socket to mover (('131.225.13.177', 34547)) with interface 131.225.205.217. elapsed=1066.280sec 00:00:31 cmsstor95.fnal.gov 022242 root I ENCP Data socket ('131.225.205.217', 36660) is connected to ('131.225.13.177', 34547). elapsed=1066.300sec 00:00:31 stkenmvr149a.fnal.gov 006318 root I LTO4_49MV tape position: target 65 current 65 0 Thread seek_thread 00:00:31 cmsstor95.fnal.gov 022242 root I ENCP Opening local file. 00:00:31 stkenmvr149a.fnal.gov 006318 root I LTO4_49MV write_tape starting, bytes_to_write=3941441024 Thread tape_thread 00:00:31 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV filemarks written. Tape VOR465 absolute location in blocks 3875431 Thread tape_thread 00:00:31 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV write_tape timing:pull 0.331876516342 write 10.5042772293 crc_check 12.3950629234 freespace 0.204426527023 block_write 23.4356431961 idle 0.372652292252 Thread tape_thread 00:00:31 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV drive stats after write. Tape VOR465 position 151 block 0 block_size 0 bloc_loc 3875431 tot_blocks 762939000 BOT 0 read_err 0 write_err 5 bytes 2717346304 block_write_tot 23.4356431961 tape_rate 115949294.895 Thread tape_thread 00:00:31 stkenmvr149a.fnal.gov 006318 root I LTO4_49MV Write starting. Tape VOR473 absolute location in blocks 1509948 Thread tape_thread 00:00:31 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV remaining bytes info in DB 308188160000 estimated 305470813696 reported from drive 303965184000 position 152 Thread tape_thread 00:00:31 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV new bitfile request {'uid': 0, 'external_label': 'VOR465', 'gid': 0, 'location_cookie': '0000_000000000_0000151', 'complete_crc': 2783870329L, 'sanity_cookie': (65536L, 75241336L), 'size': 2717345666L} Thread tape_thread 00:00:31 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/700DDD85-3AC8-DE11-BDFF-003048678FC6.root', drive = 'stkenmvr149a:/dev/rmt/tps0d0n:1310154355', pnfs_id = '000700000000000061D57A58' where bfid = 'CDMS125722803100000'; 00:00:31 cmsstor95.fnal.gov 022242 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:31 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:31 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV set remaining: VOR465 303965184000 0000_000000000_0000152 Thread tape_thread 00:00:31 last message repeated 1 times 00:00:31 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-03 00:00:31', eod_cookie = '0000_000000000_0000152', non_del_files = '152', remaining_bytes = 303965184000 where label = 'VOR465'; 00:00:31 cmsstor232.fnal.gov 013690 root I ENCP Creating zero length output files. 00:00:31 cmsstor232.fnal.gov 013690 root I ENCP Sending /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/D433840B-3DC8-DE11-80FA-00261894397E.root write request to LM: unique_id: cmsstor232.fnal.gov-1257228031-13690-0 inputfile: /storage/data2/write-pool-2/data/000700000000000061D74020 outputfile: /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D74020) 00:00:31 cmsstor59.fnal.gov 015330 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /storage/data1/write-pool-1/data/000700000000000061D57A58 -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/700DDD85-3AC8-DE11-BDFF-003048678FC6.root: 2167039117 bytes copied to VOR473 at 61 MB/S (84.7 MB/S network) (106 MB/S drive) (97.3 MB/S disk) mover=LTO4_49.mover drive_id=ULTRIUM-TD4 drive_sn=1310154355 drive_vendor=IBM elapsed=1083.4 {'storage_group': 'cms', 'encp_crc': 3093115817L, 'disk_rate': 97.29846908202201, 'drive_rate': 105.87077239822906, 'driver': 'FTTDriver', 'transfer_rate': 61.022189733021683, 'mover_interface': 'stkenmvr149a', 'overall_rate': 57.001436156430188, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 61.022189733021683, 'encp_ip': '131.225.204.235', 'unique_id': 'cmsstor59.fnal.gov-1257226949-15330-0'} 00:00:31 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:31 cmsstor59.fnal.gov 015330 root I ENCP File /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/700DDD85-3AC8-DE11-BDFF-003048678FC6.root copied successfully. 00:00:31 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data2/write-pool-2/data/000700000000000061D74020 (cmsstor232.fnal.gov-1257228031-13690-0) -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/D433840B-3DC8-DE11-80FA-00261894397E.root : library=CD-LTO4G1 family=cms11 requester:root volume_family:cms.cms11.cpio_odc 00:00:31 cmsstor59.fnal.gov 015330 root I ENCP Completed transferring 2167039117 bytes in 1 files in 1083.39676595 sec. Overall rate = 57 MB/sec. Transfer rate = 61 MB/sec. Network rate = 84.7 MB/sec. Drive rate = 106 MB/sec. Disk rate = 97.3 MB/sec. Exit status = 0. 00:00:32 cmsstor232.fnal.gov 013690 root I ENCP Listening for control socket at: ('131.225.191.104', 55754) 00:00:32 cmsstor232.fnal.gov 013690 root I ENCP Waiting for mover to connect control socket at address: ('131.225.191.104', 55754) elapsed=0.980sec 00:00:32 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV transfer complete volume=VOR465 location=152 Thread tape_thread 00:00:32 last message repeated 1 times 00:00:32 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '151', last_access = '2009-11-03 00:00:32', modification_time = '2009-11-03 00:00:32' where label = 'VOR465'; 00:00:32 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1255359028.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOR465', 'wrapper': 'cpio_odc', 'remaining_bytes': 308188160000L, 'sum_mounts': 2, 'sum_rd_access': 0, 'file_family_width': 10, 'last_access': 1257227991.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000151', 'capacity_bytes': 858993459200L, 'non_del_files': 151, 'sum_wr_err': 0, 'sum_wr_access': 150, 'file_family': 'cms11', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'modification_time': 1257227991.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257221677.0}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/40BC3528-39C8-DE11-AB6E-001731AF6845.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2717345666L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7859, 'mc_device': '0,7,1,1', 'compression': 0, 'serial_num': '9310069991', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr198a', 'max_time_in_state': 1200, 'device': 'stkenmvr198a:/dev/rmt/tps0d0n', 'name': 'LTO4_98.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_98.stat', 'mover_address': ('131.225.13.193', 7859), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.193', 39717), 'data_ip': 'stkenmvr198a', 'logname': 'LTO4_98MV', 'hostip': '131.225.13.193', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/40BC3528-39C8-DE11-AB6E-001731AF6845.root', 'uid': 0, 'uname': 'root', 'mtime': 1257226942, 'rmajor': 0, 'machine': ('Linux', 'cmsstor69.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 131219952, 21L, 1, 9811, 5063, 512, 1257226940, 1257226940, 1257224768), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2717345666L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data2/write-pool-2/data/000700000000000061D54CA8', 'inode': 131419304L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/000700000000000061D54CA8', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D54CA8)', 'fc': {'uid': 0, 'external_label': 'VOR465', 'gid': 0, 'location_cookie': '0000_000000000_0000151', 'complete_crc': 2783870329L, 'bfid': 'CDMS125722803100001', 'sanity_cookie': (65536L, 75241336L), 'size': 2717345666L}, 'infilepath': '/storage/data2/write-pool-2/data/000700000000000061D54CA8', 'at_the_top': 41, 'local_inode': 1073743639, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.204.245', 52584), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257226940.770072, 'seek_time': 0.030030965805053711, 'transfer_time': 40.32497501373291, 't0': 1257226940, 'mount_time': 0.014431953430175781, 'lm_dequeued': 1257227991.5935559, 'job_queued': 1257226942.16523, 'drive_call_time': 10.225377321243286, 'in_queue': 1049.4280428886414, 'drive_transfer_time': 25.644637584686279}, 'user_level': 'admin', 'unique_id': 'cmsstor69.fnal.gov-1257226942-10776-0'} Thread tape_thread 00:00:32 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV Done is sent Thread tape_thread 00:00:32 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV maybe_clean Thread tape_thread 00:00:32 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV returned from maybe_clean Thread tape_thread 00:00:32 cmsstor69.fnal.gov 010776 root I ENCP Received final dialog for cmsstor69.fnal.gov-1257226942-10776-0. elapsed=1091.480sec 00:00:32 cmsstor69.fnal.gov 010776 root I ENCP leaving transfer_file(). 00:00:32 cmsstor75.fnal.gov 017998 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms11 FF Wrapper: cpio_odc FF Width: 10 Current working directory: cmsstor75.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --priority 25 --pnfs-mount /pnfs/fs --put-cache 000700000000000061D5D3A8 /storage/data1/write-pool-1/data/000700000000000061D5D3A8 00:00:32 cmsstor69.fnal.gov 010776 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:32 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV filemarks written. Tape VOR464 absolute location in blocks 4536909 Thread tape_thread 00:00:32 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV write_tape timing:pull 0.346265554428 write 16.4324603081 crc_check 15.4729485512 freespace 0.38512134552 block_write 32.6367957592 idle 0.5994181633 Thread tape_thread 00:00:32 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV drive stats after write. Tape VOR464 position 196 block 0 block_size 0 bloc_loc 4536909 tot_blocks 762939000 BOT 0 read_err 0 write_err 1 bytes 3910281728 block_write_tot 32.6367957592 tape_rate 119812059.886 Thread tape_thread 00:00:32 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV remaining bytes info in DB 227433472000 estimated 223523190272 reported from drive 223783936000 position 197 Thread tape_thread 00:00:32 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV new bitfile request {'uid': 0, 'external_label': 'VOR464', 'gid': 0, 'location_cookie': '0000_000000000_0000196', 'complete_crc': 3602264151L, 'sanity_cookie': (65536L, 889057972L), 'size': 3910281441L} Thread tape_thread 00:00:32 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV set remaining: VOR464 223783936000 0000_000000000_0000197 Thread tape_thread 00:00:32 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/40BC3528-39C8-DE11-AB6E-001731AF6845.root', drive = 'stkenmvr198a:/dev/rmt/tps0d0n:9310069991', pnfs_id = '000700000000000061D54CA8' where bfid = 'CDMS125722803100001'; 00:00:32 last message repeated 1 times 00:00:32 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:32 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-03 00:00:32', eod_cookie = '0000_000000000_0000197', non_del_files = '198', remaining_bytes = 223783936000 where label = 'VOR464'; 00:00:32 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV transfer complete volume=VOR464 location=197 Thread tape_thread 00:00:32 last message repeated 1 times 00:00:32 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '196', modification_time = '2009-11-03 00:00:32' where label = 'VOR464'; 00:00:32 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1255359024.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOR464', 'wrapper': 'cpio_odc', 'remaining_bytes': 227433472000L, 'sum_mounts': 2, 'sum_rd_access': 0, 'file_family_width': 10, 'last_access': 1257227990.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000196', 'capacity_bytes': 858993459200L, 'non_del_files': 197, 'sum_wr_err': 0, 'sum_wr_access': 195, 'file_family': 'cms11', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'modification_time': 1257227990.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257220197.0}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/D0C99E56-36C8-DE11-A842-003048678F1C.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 3910281441L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7850, 'mc_device': '0,5,1,2', 'compression': 0, 'serial_num': '9310801229', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr150a', 'max_time_in_state': 1200, 'device': 'stkenmvr150a:/dev/rmt/tps0d0n', 'name': 'LTO4_50.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_50.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.178', 7850), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.178', 34187), 'data_ip': 'stkenmvr150a', 'logname': 'LTO4_50MV', 'hostip': '131.225.13.178', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/D0C99E56-36C8-DE11-A842-003048678F1C.root', 'uid': 0, 'uname': 'root', 'mtime': 1257226938, 'rmajor': 0, 'machine': ('Linux', 'cmsstor191.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 131219952, 21L, 1, 9811, 5063, 512, 1257226936, 1257226936, 1257224768), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 3910281441L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data2/write-pool-2/data/000700000000000061D2DF50', 'inode': 131260240L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/000700000000000061D2DF50', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D2DF50)', 'fc': {'uid': 0, 'external_label': 'VOR464', 'gid': 0, 'location_cookie': '0000_000000000_0000196', 'complete_crc': 3602264151L, 'bfid': 'CDMS125722803200000', 'sanity_cookie': (65536L, 889057972L), 'size': 3910281441L}, 'infilepath': '/storage/data2/write-pool-2/data/000700000000000061D2DF50', 'at_the_top': 50, 'local_inode': 6256, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.207.16', 47842), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257226937.853683, 'seek_time': 0.034972190856933594, 'transfer_time': 42.183003902435303, 't0': 1257226937, 'mount_time': 0.039960861206054688, 'lm_dequeued': 1257227990.2025909, 'job_queued': 1257226938.6652229, 'drive_call_time': 16.011427879333496, 'in_queue': 1051.5370562076569, 'drive_transfer_time': 35.199234008789062}, 'user_level': 'admin', 'unique_id': 'cmsstor191.fn(trunc. 4107) Thread tape_thread 00:00:32 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV Done is sent Thread tape_thread 00:00:32 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV maybe_clean Thread tape_thread 00:00:32 cmsstor191.fnal.gov 026649 root I ENCP Received final dialog for cmsstor191.fnal.gov-1257226938-26649-0. elapsed=1094.690sec 00:00:32 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV returned from maybe_clean Thread tape_thread 00:00:32 cmsstor191.fnal.gov 026649 root I ENCP leaving transfer_file(). 00:00:32 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:32 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VOR465 mover=LTO4_98.mover requester:root 00:00:32 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/4C524A4A-38C8-DE11-8B2A-001A92971B20.root to mover LTO4_98.mover ('131.225.13.193', 7859) DEL_DISM None 00:00:32 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:32 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'cms', 'external_label': 'VOR465', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/4C524A4A-38C8-DE11-8B2A-001A92971B20.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2253800099L, 'mover': 'LTO4_98.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/4C524A4A-38C8-DE11-8B2A-001A92971B20.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257226972, 'rmajor': 0, 'machine': ('Linux', 'cmsstor238.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16893, 131219952, 21L, 1, 9811, 5063, 512, 1257226968, 1257226968, 1257224768), 'mode_octal': '0100036', 'mode': 33188, 'gname': 'root', 'size_bytes': 2253800099L, 'fullname': '/storage/data1/write-pool-1/data/000700000000000061D40EF8', 'inode': 131337976L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/000700000000000061D40EF8', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D40EF8)', 'fc': {'size': 2253800099L, 'external_label': 'VOR465', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data1/write-pool-1/data/000700000000000061D40EF8', 'at_the_top': 2, 'local_inode': 5472, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.191.110', 37756), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228032.6350739, 'encp_start_time': 1257226967.1891639, 'job_queued': 1257226972.9696579, 't0': 1257226967, 'in_queue': 1059.6651310920715}, 'user_level': 'admin', 'unique_id': 'cmsstor238.fnal.gov-1257226972-7478-0'} to mover 00:00:32 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:32 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV WRITE_TO_HSM Thread MainThread 00:00:32 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV SENDING {'vc': {'storage_group': 'cms', 'external_label': 'VOR465', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/4C524A4A-38C8-DE11-8B2A-001A92971B20.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2253800099L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7859, 'mc_device': '0,7,1,1', 'compression': 0, 'serial_num': '9310069991', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr198a', 'max_time_in_state': 1200, 'device': 'stkenmvr198a:/dev/rmt/tps0d0n', 'name': 'LTO4_98.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_98.stat', 'mover_address': ('131.225.13.193', 7859), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.193', 39724), 'data_ip': 'stkenmvr198a', 'logname': 'LTO4_98MV', 'hostip': '131.225.13.193', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/4C524A4A-38C8-DE11-8B2A-001A92971B20.root', 'uid': 0, 'uname': 'root', 'mtime': 1257226972, 'rmajor': 0, 'machine': ('Linux', 'cmsstor238.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 131219952, 21L, 1, 9811, 5063, 512, 1257226968, 1257226968, 1257224768), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2253800099L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data1/write-pool-1/data/000700000000000061D40EF8', 'inode': 131337976L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/000700000000000061D40EF8', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D40EF8)', 'fc': {'size': 2253800099L, 'external_label': 'VOR465', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data1/write-pool-1/data/000700000000000061D40EF8', 'at_the_top': 2, 'local_inode': 5472, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.191.110', 37756), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228032.6350739, 'encp_start_time': 1257226967.1891639, 'job_queued': 1257226972.9696579, 't0': 1257226967, 'in_queue': 1059.6651310920715}, 'user_level': 'admin', 'unique_id': 'cmsstor238.fnal.gov-1257226972-7478-0'} Thread client_connect_thread 00:00:32 cmsstor238.fnal.gov 007478 root I ENCP Control socket ('131.225.191.110', 37756) is connected to ('131.225.13.193', 39725) for cmsstor238.fnal.gov-1257226972-7478-0. elapsed=1065.480sec 00:00:32 cmsstor238.fnal.gov 007478 root I ENCP Received callback ticket from mover LTO4_98.mover for transfer cmsstor238.fnal.gov-1257226972-7478-0. elapsed=1065.490sec 00:00:32 cmsstor238.fnal.gov 007478 root I ENCP Atempting to connect data socket to mover at ('131.225.13.193', 39724). 00:00:32 cmsstor238.fnal.gov 007478 root I ENCP Connecting data socket to mover (('131.225.13.193', 39724)) with interface 131.225.191.110. elapsed=1065.490sec 00:00:32 cmsstor191.fnal.gov 026649 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:32 cmsstor238.fnal.gov 007478 root I ENCP Data socket ('131.225.191.110', 41610) is connected to ('131.225.13.193', 39724). elapsed=1065.500sec 00:00:32 cmsstor238.fnal.gov 007478 root I ENCP Opening local file. 00:00:32 cmsstor69.fnal.gov 010776 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /storage/data2/write-pool-2/data/000700000000000061D54CA8 -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/40BC3528-39C8-DE11-AB6E-001731AF6845.root: 2717345666 bytes copied to VOR465 at 64.3 MB/S (83.1 MB/S network) (101 MB/S drive) (102 MB/S disk) mover=LTO4_98.mover drive_id=ULTRIUM-TD4 drive_sn=9310069991 drive_vendor=IBM elapsed=1091.9 {'storage_group': 'cms', 'encp_crc': 2783870329L, 'disk_rate': 102.49894806719854, 'drive_rate': 101.05281316237163, 'driver': 'FTTDriver', 'transfer_rate': 64.286294828404962, 'mover_interface': 'stkenmvr198a', 'overall_rate': 63.375187961542757, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 64.286294828404962, 'encp_ip': '131.225.204.245', 'unique_id': 'cmsstor69.fnal.gov-1257226942-10776-0'} 00:00:32 cmsstor69.fnal.gov 010776 root I ENCP File /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/40BC3528-39C8-DE11-AB6E-001731AF6845.root copied successfully. 00:00:32 cmsstor69.fnal.gov 010776 root I ENCP Completed transferring 2717345666 bytes in 1 files in 1091.91488004 sec. Overall rate = 63.4 MB/sec. Transfer rate = 64.3 MB/sec. Network rate = 83.1 MB/sec. Drive rate = 101 MB/sec. Disk rate = 102 MB/sec. Exit status = 0. 00:00:32 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV tape position: target 152 current 152 0 Thread seek_thread 00:00:32 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV write_tape starting, bytes_to_write=2253800448 Thread tape_thread 00:00:32 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV Write starting. Tape VOR465 absolute location in blocks 3875431 Thread tape_thread 00:00:32 cmsstor238.fnal.gov 007478 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:32 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/D0C99E56-36C8-DE11-A842-003048678F1C.root', drive = 'stkenmvr150a:/dev/rmt/tps0d0n:9310801229', pnfs_id = '000700000000000061D2DF50' where bfid = 'CDMS125722803200000'; 00:00:32 cmsstor75.fnal.gov 017998 root I ENCP Creating zero length output files. 00:00:32 cmsstor75.fnal.gov 017998 root I ENCP Sending /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/DC949791-3AC8-DE11-B353-002618943937.root write request to LM: unique_id: cmsstor75.fnal.gov-1257228032-17998-0 inputfile: /storage/data1/write-pool-1/data/000700000000000061D5D3A8 outputfile: /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D5D3A8) 00:00:32 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:32 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data1/write-pool-1/data/000700000000000061D5D3A8 (cmsstor75.fnal.gov-1257228032-17998-0) -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/DC949791-3AC8-DE11-B353-002618943937.root : library=CD-LTO4G1 family=cms11 requester:root volume_family:cms.cms11.cpio_odc 00:00:32 cmsstor75.fnal.gov 017998 root I ENCP Listening for control socket at: ('131.225.204.251', 34308) 00:00:32 cmsstor75.fnal.gov 017998 root I ENCP Waiting for mover to connect control socket at address: ('131.225.204.251', 34308) elapsed=1.750sec 00:00:33 cmsstor191.fnal.gov 026649 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /storage/data2/write-pool-2/data/000700000000000061D2DF50 -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/D0C99E56-36C8-DE11-A842-003048678F1C.root: 3910281441 bytes copied to VOR464 at 88.4 MB/S (92.6 MB/S network) (106 MB/S drive) (106 MB/S disk) mover=LTO4_50.mover drive_id=ULTRIUM-TD4 drive_sn=9310801229 drive_vendor=IBM elapsed=1095.2 {'storage_group': 'cms', 'encp_crc': 3602264151L, 'disk_rate': 106.11870948176038, 'drive_rate': 105.94364011644616, 'driver': 'FTTDriver', 'transfer_rate': 88.3772836983091, 'mover_interface': 'stkenmvr150a', 'overall_rate': 87.223661848744541, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 88.3772836983091, 'encp_ip': '131.225.207.16', 'unique_id': 'cmsstor191.fnal.gov-1257226938-26649-0'} 00:00:33 cmsstor191.fnal.gov 026649 root I ENCP File /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/D0C99E56-36C8-DE11-A842-003048678F1C.root copied successfully. 00:00:33 cmsstor191.fnal.gov 026649 root I ENCP Completed transferring 3910281441 bytes in 1 files in 1095.15483499 sec. Overall rate = 87.2 MB/sec. Transfer rate = 88.4 MB/sec. Network rate = 92.6 MB/sec. Drive rate = 106 MB/sec. Disk rate = 106 MB/sec. Exit status = 0. 00:00:33 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:33 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VOR464 mover=LTO4_50.mover requester:root 00:00:33 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/4C2FD603-34C8-DE11-9451-00304867918E.root to mover LTO4_50.mover ('131.225.13.178', 7850) DEL_DISM None 00:00:33 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:33 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'cms', 'external_label': 'VOR464', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/4C2FD603-34C8-DE11-9451-00304867918E.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2707845004L, 'mover': 'LTO4_50.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/4C2FD603-34C8-DE11-9451-00304867918E.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257226973, 'rmajor': 0, 'machine': ('Linux', 'cmsstor49.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16893, 130813352, 21L, 1, 9811, 5063, 512, 1257226372, 1257226372, 1257220971), 'mode_octal': '0100036', 'mode': 33188, 'gname': 'root', 'size_bytes': 2707845004L, 'fullname': '/storage/data2/write-pool-2/data/000700000000000061D0EA98', 'inode': 131132056L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/000700000000000061D0EA98', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/.(access)(000700000000000061D0EA98)', 'fc': {'size': 2707845004L, 'external_label': 'VOR464', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data2/write-pool-2/data/000700000000000061D0EA98', 'at_the_top': 8, 'local_inode': 1073743311, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.204.225', 38381), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228033.2479081, 'encp_start_time': 1257226972.471864, 'job_queued': 1257226973.7516301, 't0': 1257226972, 'in_queue': 1059.495992898941}, 'user_level': 'admin', 'unique_id': 'cmsstor49.fnal.gov-1257226973-6338-0'} to mover 00:00:33 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:33 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV WRITE_TO_HSM Thread MainThread 00:00:33 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV SENDING {'vc': {'storage_group': 'cms', 'external_label': 'VOR464', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/4C2FD603-34C8-DE11-9451-00304867918E.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2707845004L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7850, 'mc_device': '0,5,1,2', 'compression': 0, 'serial_num': '9310801229', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr150a', 'max_time_in_state': 1200, 'device': 'stkenmvr150a:/dev/rmt/tps0d0n', 'name': 'LTO4_50.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_50.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.178', 7850), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.178', 34193), 'data_ip': 'stkenmvr150a', 'logname': 'LTO4_50MV', 'hostip': '131.225.13.178', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/4C2FD603-34C8-DE11-9451-00304867918E.root', 'uid': 0, 'uname': 'root', 'mtime': 1257226973, 'rmajor': 0, 'machine': ('Linux', 'cmsstor49.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 130813352, 21L, 1, 9811, 5063, 512, 1257226372, 1257226372, 1257220971), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2707845004L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data2/write-pool-2/data/000700000000000061D0EA98', 'inode': 131132056L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/000700000000000061D0EA98', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/.(access)(000700000000000061D0EA98)', 'fc': {'size': 2707845004L, 'external_label': 'VOR464', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data2/write-pool-2/data/000700000000000061D0EA98', 'at_the_top': 8, 'local_inode': 1073743311, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.204.225', 38381), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228033.2479081, 'encp_start_time': 1257226972.471864, 'job_queued': 1257226973.7516301, 't0': 1257226972, 'in_queue': 1059.495992898941}, 'user_level': 'admin', 'unique_id': 'cmsstor49.fnal.gov-1257226973-6338-0'} Thread client_connect_thread 00:00:33 cmsstor49.fnal.gov 006338 root I ENCP Control socket ('131.225.204.225', 38381) is connected to ('131.225.13.178', 34194) for cmsstor49.fnal.gov-1257226973-6338-0. elapsed=1060.920sec 00:00:33 cmsstor49.fnal.gov 006338 root I ENCP Received callback ticket from mover LTO4_50.mover for transfer cmsstor49.fnal.gov-1257226973-6338-0. elapsed=1060.920sec 00:00:33 cmsstor49.fnal.gov 006338 root I ENCP Atempting to connect data socket to mover at ('131.225.13.178', 34193). 00:00:33 cmsstor49.fnal.gov 006338 root I ENCP Connecting data socket to mover (('131.225.13.178', 34193)) with interface 131.225.204.225. elapsed=1060.920sec 00:00:33 cmsstor49.fnal.gov 006338 root I ENCP Data socket ('131.225.204.225', 35437) is connected to ('131.225.13.178', 34193). elapsed=1060.930sec 00:00:33 cmsstor49.fnal.gov 006338 root I ENCP Opening local file. 00:00:33 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV tape position: target 197 current 197 0 Thread seek_thread 00:00:33 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV write_tape starting, bytes_to_write=2707845632 Thread tape_thread 00:00:33 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV Write starting. Tape VOR464 absolute location in blocks 4536909 Thread tape_thread 00:00:33 cmsstor49.fnal.gov 006338 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:33 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:33 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:34 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV written bytes 2114616832/2114616832, blocks 16134 header 192 trailer 100 Thread tape_thread 00:00:34 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:34 cmsstor191.fnal.gov 017814 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Scientific Linux SL release 5.2 (Boron) Rocks release 5.1 (V.I) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms11 FF Wrapper: cpio_odc FF Width: 10 Current working directory: cmsstor191.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --priority 25 --pnfs-mount /pnfs/fs --put-cache 000700000000000061D5B208 /storage/data3/write-pool-3/data/000700000000000061D5B208 00:00:34 cmsstor191.fnal.gov 017814 root I ENCP Creating zero length output files. 00:00:34 cmsstor191.fnal.gov 017814 root I ENCP Sending /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0062/9EE37FA7-3AC8-DE11-8ABC-003048678B20.root write request to LM: unique_id: cmsstor191.fnal.gov-1257228034-17814-0 inputfile: /storage/data3/write-pool-3/data/000700000000000061D5B208 outputfile: /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0062/.(access)(000700000000000061D5B208) 00:00:34 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:34 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data3/write-pool-3/data/000700000000000061D5B208 (cmsstor191.fnal.gov-1257228034-17814-0) -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0062/9EE37FA7-3AC8-DE11-8ABC-003048678B20.root : library=CD-LTO4G1 family=cms11 requester:root volume_family:cms.cms11.cpio_odc 00:00:34 cmsstor191.fnal.gov 017814 root I ENCP Listening for control socket at: ('131.225.207.16', 58306) 00:00:34 cmsstor191.fnal.gov 017814 root I ENCP Waiting for mover to connect control socket at address: ('131.225.207.16', 58306) elapsed=0.900sec 00:00:35 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV filemarks written. Tape VOR460 absolute location in blocks 5300122 Thread tape_thread 00:00:35 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV write_tape timing:pull 0.227476119995 write 10.1398701668 crc_check 8.34397244453 freespace 0.166632175446 block_write 18.8779509068 idle 0.288079738617 Thread tape_thread 00:00:35 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV drive stats after write. Tape VOR460 position 231 block 0 block_size 0 bloc_loc 5300122 tot_blocks 762939000 BOT 0 read_err 0 write_err 1 bytes 2114616832 block_write_tot 18.8779509068 tape_rate 112015167.454 Thread tape_thread 00:00:35 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV remaining bytes info in DB 120305664000 estimated 118191047168 reported from drive 124443648000 position 232 Thread tape_thread 00:00:35 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV new bitfile request {'uid': 0, 'external_label': 'VOR460', 'gid': 0, 'location_cookie': '0000_000000000_0000231', 'complete_crc': 1707028065L, 'sanity_cookie': (65536L, 2455724287L), 'size': 2114616540L} Thread tape_thread 00:00:35 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV set remaining: VOR460 124443648000 0000_000000000_0000232 Thread tape_thread 00:00:35 last message repeated 1 times 00:00:35 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-03 00:00:35', eod_cookie = '0000_000000000_0000232', non_del_files = '233', remaining_bytes = 124443648000 where label = 'VOR460'; 00:00:35 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:35 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:35 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV transfer complete volume=VOR460 location=232 Thread tape_thread 00:00:36 last message repeated 1 times 00:00:36 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '231', modification_time = '2009-11-03 00:00:35' where label = 'VOR460'; 00:00:36 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1255359010.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOR460', 'wrapper': 'cpio_odc', 'remaining_bytes': 120305664000L, 'sum_mounts': 2, 'sum_rd_access': 0, 'file_family_width': 10, 'last_access': 1257227993.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000231', 'capacity_bytes': 858993459200L, 'non_del_files': 232, 'sum_wr_err': 0, 'sum_wr_access': 230, 'file_family': 'cms11', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'modification_time': 1257227993.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257218947.0}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/A894E5C2-3AC8-DE11-9EB7-00261894391F.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2114616540L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7848, 'mc_device': '0,5,1,15', 'compression': 0, 'serial_num': '9310036898', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr148a', 'max_time_in_state': 1200, 'device': 'stkenmvr148a:/dev/rmt/tps0d0n', 'name': 'LTO4_48.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_48.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.176', 7848), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.176', 35545), 'data_ip': 'stkenmvr148a', 'logname': 'LTO4_48MV', 'hostip': '131.225.13.176', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/A894E5C2-3AC8-DE11-9EB7-00261894391F.root', 'uid': 0, 'uname': 'root', 'mtime': 1257226948, 'rmajor': 0, 'machine': ('Linux', 'cmsstor144.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 131219952, 21L, 1, 9811, 5063, 512, 1257226946, 1257226946, 1257224768), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2114616540L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data2/write-pool-2/data/000700000000000061D58CC8', 'inode': 131435720L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/000700000000000061D58CC8', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D58CC8)', 'fc': {'uid': 0, 'external_label': 'VOR460', 'gid': 0, 'location_cookie': '0000_000000000_0000231', 'complete_crc': 1707028065L, 'bfid': 'CDMS125722803500000', 'sanity_cookie': (65536L, 2455724287L), 'size': 2114616540L}, 'infilepath': '/storage/data2/write-pool-2/data/000700000000000061D58CC8', 'at_the_top': 8, 'local_inode': 4224, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.189.115', 53694), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257226947.9778409, 'seek_time': 0.034959077835083008, 'transfer_time': 41.64684796333313, 't0': 1257226947, 'mount_time': 0.039777994155883789, 'lm_dequeued': 1257227994.2172, 'job_queued': 1257226948.9161589, 'drive_call_time': 9.9351139068603516, 'in_queue': 1045.3005721569061, 'drive_transfer_time': 20.884227991104126}, 'user_level': 'admin', 'unique_id': 'cmsstor144.fna(trunc. 4106) Thread tape_thread 00:00:36 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV Done is sent Thread tape_thread 00:00:36 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV maybe_clean Thread tape_thread 00:00:36 cmsstor144.fnal.gov 010329 root I ENCP Received final dialog for cmsstor144.fnal.gov-1257226948-10329-0. elapsed=1087.880sec 00:00:36 cmsstor144.fnal.gov 010329 root I ENCP leaving transfer_file(). 00:00:36 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV returned from maybe_clean Thread tape_thread 00:00:36 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:36 cmsstor144.fnal.gov 010329 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:36 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VOR460 mover=LTO4_48.mover requester:root 00:00:36 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/E406E6EA-31C8-DE11-A8AD-001A92971B88.root to mover LTO4_48.mover ('131.225.13.176', 7848) DEL_DISM None 00:00:36 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:36 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'cms', 'external_label': 'VOR460', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/E406E6EA-31C8-DE11-A8AD-001A92971B88.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 5822106756L, 'mover': 'LTO4_48.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/E406E6EA-31C8-DE11-A8AD-001A92971B88.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257226986, 'rmajor': 0, 'machine': ('Linux', 'cmsstor231.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16893, 130813352, 21L, 1, 9811, 5063, 512, 1257226372, 1257226372, 1257220971), 'mode_octal': '0100036', 'mode': 33188, 'gname': 'root', 'size_bytes': 5822106756L, 'fullname': '/storage/data1/write-pool-1/data/000700000000000061D1A540', 'inode': 131179840L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/000700000000000061D1A540', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/.(access)(000700000000000061D1A540)', 'fc': {'size': 5822106756L, 'external_label': 'VOR460', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data1/write-pool-1/data/000700000000000061D1A540', 'at_the_top': 44, 'local_inode': 8038, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.191.103', 40516), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228036.2973011, 'encp_start_time': 1257226985.1531971, 'job_queued': 1257226986.816463, 't0': 1257226985, 'in_queue': 1049.4805610179901}, 'user_level': 'admin', 'unique_id': 'cmsstor231.fnal.gov-1257226986-8981-0'} to mover 00:00:36 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:36 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV WRITE_TO_HSM Thread MainThread 00:00:36 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV SENDING {'vc': {'storage_group': 'cms', 'external_label': 'VOR460', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/E406E6EA-31C8-DE11-A8AD-001A92971B88.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 5822106756L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7848, 'mc_device': '0,5,1,15', 'compression': 0, 'serial_num': '9310036898', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr148a', 'max_time_in_state': 1200, 'device': 'stkenmvr148a:/dev/rmt/tps0d0n', 'name': 'LTO4_48.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_48.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.176', 7848), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.176', 35552), 'data_ip': 'stkenmvr148a', 'logname': 'LTO4_48MV', 'hostip': '131.225.13.176', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/E406E6EA-31C8-DE11-A8AD-001A92971B88.root', 'uid': 0, 'uname': 'root', 'mtime': 1257226986, 'rmajor': 0, 'machine': ('Linux', 'cmsstor231.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 130813352, 21L, 1, 9811, 5063, 512, 1257226372, 1257226372, 1257220971), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 5822106756L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data1/write-pool-1/data/000700000000000061D1A540', 'inode': 131179840L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/000700000000000061D1A540', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/.(access)(000700000000000061D1A540)', 'fc': {'size': 5822106756L, 'external_label': 'VOR460', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data1/write-pool-1/data/000700000000000061D1A540', 'at_the_top': 44, 'local_inode': 8038, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.191.103', 40516), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228036.2973011, 'encp_start_time': 1257226985.1531971, 'job_queued': 1257226986.816463, 't0': 1257226985, 'in_queue': 1049.4805610179901}, 'user_level': 'admin', 'unique_id': 'cmsstor231.fnal.gov-1257226986-8981-0'} Thread client_connect_thread 00:00:36 cmsstor231.fnal.gov 008981 root I ENCP Control socket ('131.225.191.103', 40516) is connected to ('131.225.13.176', 35553) for cmsstor231.fnal.gov-1257226986-8981-0. elapsed=1051.200sec 00:00:36 cmsstor231.fnal.gov 008981 root I ENCP Received callback ticket from mover LTO4_48.mover for transfer cmsstor231.fnal.gov-1257226986-8981-0. elapsed=1051.210sec 00:00:36 cmsstor231.fnal.gov 008981 root I ENCP Atempting to connect data socket to mover at ('131.225.13.176', 35552). 00:00:36 cmsstor231.fnal.gov 008981 root I ENCP Connecting data socket to mover (('131.225.13.176', 35552)) with interface 131.225.191.103. elapsed=1051.210sec 00:00:36 cmsstor231.fnal.gov 008981 root I ENCP Data socket ('131.225.191.103', 39173) is connected to ('131.225.13.176', 35552). elapsed=1051.210sec 00:00:36 cmsstor231.fnal.gov 008981 root I ENCP Opening local file. 00:00:36 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV tape position: target 232 current 232 0 Thread seek_thread 00:00:36 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV write_tape starting, bytes_to_write=5822107136 Thread tape_thread 00:00:36 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV Write starting. Tape VOR460 absolute location in blocks 5300122 Thread tape_thread 00:00:36 cmsstor231.fnal.gov 008981 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:36 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/A894E5C2-3AC8-DE11-9EB7-00261894391F.root', drive = 'stkenmvr148a:/dev/rmt/tps0d0n:9310036898', pnfs_id = '000700000000000061D58CC8' where bfid = 'CDMS125722803500000'; 00:00:36 cmsstor144.fnal.gov 010329 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /storage/data2/write-pool-2/data/000700000000000061D58CC8 -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/A894E5C2-3AC8-DE11-9EB7-00261894391F.root: 2114616540 bytes copied to VOR460 at 51.4 MB/S (96.2 MB/S network) (96.6 MB/S drive) (82.6 MB/S disk) mover=LTO4_48.mover drive_id=ULTRIUM-TD4 drive_sn=9310036898 drive_vendor=IBM elapsed=1088.8 {'storage_group': 'cms', 'encp_crc': 1707028065L, 'disk_rate': 82.585851917356095, 'drive_rate': 96.563563858087875, 'driver': 'FTTDriver', 'transfer_rate': 51.438243989050598, 'mover_interface': 'stkenmvr148a', 'overall_rate': 47.484952315732805, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 51.438243989050598, 'encp_ip': '131.225.189.115', 'unique_id': 'cmsstor144.fnal.gov-1257226948-10329-0'} 00:00:36 cmsstor144.fnal.gov 010329 root I ENCP File /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/A894E5C2-3AC8-DE11-9EB7-00261894391F.root copied successfully. 00:00:36 cmsstor144.fnal.gov 010329 root I ENCP Completed transferring 2114616540 bytes in 1 files in 1088.7528162 sec. Overall rate = 47.5 MB/sec. Transfer rate = 51.4 MB/sec. Network rate = 96.2 MB/sec. Drive rate = 96.6 MB/sec. Disk rate = 82.6 MB/sec. Exit status = 0. 00:00:37 cmsstor145.fnal.gov 030171 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Scientific Linux SL release 5.2 (Boron) Rocks release 5.1 (V.I) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms11 FF Wrapper: cpio_odc FF Width: 10 Current working directory: cmsstor145.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --priority 25 --pnfs-mount /pnfs/fs --put-cache 000700000000000061D5A088 /storage/data2/write-pool-2/data/000700000000000061D5A088 00:00:37 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:37 cmsstor145.fnal.gov 030171 root I ENCP Creating zero length output files. 00:00:37 cmsstor145.fnal.gov 030171 root I ENCP Sending /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/42215C8F-3AC8-DE11-9CC4-0026189438F2.root write request to LM: unique_id: cmsstor145.fnal.gov-1257228037-30171-0 inputfile: /storage/data2/write-pool-2/data/000700000000000061D5A088 outputfile: /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D5A088) 00:00:37 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:37 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data2/write-pool-2/data/000700000000000061D5A088 (cmsstor145.fnal.gov-1257228037-30171-0) -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/42215C8F-3AC8-DE11-9CC4-0026189438F2.root : library=CD-LTO4G1 family=cms11 requester:root volume_family:cms.cms11.cpio_odc 00:00:37 cmsstor145.fnal.gov 030171 root I ENCP Listening for control socket at: ('131.225.189.116', 45966) 00:00:37 cmsstor145.fnal.gov 030171 root I ENCP Waiting for mover to connect control socket at address: ('131.225.189.116', 45966) elapsed=0.980sec 00:00:37 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:37 stkensrv4n.fnal.gov 004885 enstore I SL8500F1MC mount VOO268 1,1,1,4 => 0,Mount: VOO268 mounted on 1, 1, 1, 4 00:00:37 stkensrv4n.fnal.gov 005843 enstore I SL8500F1MC FINISHED mount VOO268 1,1,1,4 returned ('ok', 0, 'mount VOO268 1,1,1,4 => 0,Mount: VOO268 mounted on 1, 1, 1, 4', '', '') 00:00:37 last message repeated 1 times 00:00:37 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_mounts = '4991', last_access = '2009-11-03 00:00:37' where label = 'VOO268'; 00:00:38 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV MSG_TYPE=MC_LOAD_DONE mounted VOO268 ULTRIUM-TD4 00:00:38 Thread media_thread 00:00:39 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:40 cmsstor224.fnal.gov 020055 root I ENCP File /storage/data2/write-pool-2/data/000700000000000061CF8C48 transfered for cmsstor224.fnal.gov-1257226956-20055-0 in 28.5485949516 seconds. 00:00:41 cmsstor230.fnal.gov 010924 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms11 FF Wrapper: cpio_odc FF Width: 10 Current working directory: cmsstor230.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --priority 25 --pnfs-mount /pnfs/fs --put-cache 000700000000000061D70928 /storage/data3/write-pool-3/data/000700000000000061D70928 00:00:41 cmsstor194.fnal.gov 007680 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Scientific Linux SL release 5.2 (Boron) Rocks release 5.1 (V.I) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms11 FF Wrapper: cpio_odc FF Width: 10 Current working directory: cmsstor194.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --priority 25 --pnfs-mount /pnfs/fs --put-cache 000700000000000061D76CC0 /storage/data3/write-pool-3/data/000700000000000061D76CC0 00:00:41 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:41 cmsstor230.fnal.gov 010924 root I ENCP Creating zero length output files. 00:00:41 cmsstor230.fnal.gov 010924 root I ENCP Sending /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/F0D71543-3BC8-DE11-B2C5-0018F3D095EE.root write request to LM: unique_id: cmsstor230.fnal.gov-1257228041-10924-0 inputfile: /storage/data3/write-pool-3/data/000700000000000061D70928 outputfile: /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D70928) 00:00:41 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:41 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data3/write-pool-3/data/000700000000000061D70928 (cmsstor230.fnal.gov-1257228041-10924-0) -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/F0D71543-3BC8-DE11-B2C5-0018F3D095EE.root : library=CD-LTO4G1 family=cms11 requester:root volume_family:cms.cms11.cpio_odc 00:00:41 cmsstor230.fnal.gov 010924 root I ENCP Listening for control socket at: ('131.225.191.102', 52855) 00:00:41 cmsstor230.fnal.gov 010924 root I ENCP Waiting for mover to connect control socket at address: ('131.225.191.102', 52855) elapsed=1.100sec 00:00:42 cmsstor194.fnal.gov 007680 root I ENCP Creating zero length output files. 00:00:42 cmsstor194.fnal.gov 007680 root I ENCP Sending /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/2670F10A-3DC8-DE11-B777-001731AF6993.root write request to LM: unique_id: cmsstor194.fnal.gov-1257228042-7680-0 inputfile: /storage/data3/write-pool-3/data/000700000000000061D76CC0 outputfile: /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D76CC0) 00:00:42 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:42 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data3/write-pool-3/data/000700000000000061D76CC0 (cmsstor194.fnal.gov-1257228042-7680-0) -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/2670F10A-3DC8-DE11-B777-001731AF6993.root : library=CD-LTO4G1 family=cms11 requester:root volume_family:cms.cms11.cpio_odc 00:00:42 cmsstor194.fnal.gov 007680 root I ENCP Listening for control socket at: ('131.225.207.19', 54382) 00:00:42 cmsstor194.fnal.gov 007680 root I ENCP Waiting for mover to connect control socket at address: ('131.225.207.19', 54382) elapsed=1.140sec 00:00:43 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:44 cmsstor88.fnal.gov 006559 root I ENCP File /storage/data1/write-pool-1/data/000700000000000061D337D0 transfered for cmsstor88.fnal.gov-1257226957-6559-0 in 32.8547790051 seconds. 00:00:45 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:47 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV written bytes 2994386432/2994386432, blocks 22846 header 192 trailer 387 Thread tape_thread 00:00:47 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:47 cmsstor245.fnal.gov 015098 root I ENCP File /storage/data2/write-pool-2/data/000700000000000061D475B8 transfered for cmsstor245.fnal.gov-1257226958-15098-0 in 25.1999349594 seconds. 00:00:47 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:48 stkenmvr202a.fnal.gov 006121 root I LTO4_102MV tape position: target 1 current 0 1 Thread seek_thread 00:00:48 stkenmvr202a.fnal.gov 006121 root I LTO4_102MV skip_fm 1 Thread seek_thread 00:00:48 stkenmvr202a.fnal.gov 006121 root I LTO4_102MV write_tape starting, bytes_to_write=2089199616 Thread tape_thread 00:00:48 stkenmvr202a.fnal.gov 006121 root I LTO4_102MV Write starting. Tape VOR469 absolute location in blocks 2 Thread tape_thread 00:00:48 cmsstor153.fnal.gov 024127 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms11 FF Wrapper: cpio_odc FF Width: 10 Current working directory: cmsstor153.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --priority 25 --pnfs-mount /pnfs/fs --put-cache 000700000000000061D75468 /storage/data1/write-pool-1/data/000700000000000061D75468 00:00:48 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV filemarks written. Tape VOR466 absolute location in blocks 3610207 Thread tape_thread 00:00:48 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV write_tape timing:pull 0.270149707794 write 10.8826804161 crc_check 13.3585636616 freespace 0.248740196228 block_write 24.7601339817 idle 0.427793741226 Thread tape_thread 00:00:48 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV drive stats after write. Tape VOR466 position 152 block 0 block_size 0 bloc_loc 3610207 tot_blocks 762939000 BOT 0 read_err 0 write_err 15 bytes 2994386432 block_write_tot 24.7601339817 tape_rate 120935792.763 Thread tape_thread 00:00:48 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV remaining bytes info in DB 335404032000 estimated 332409645568 reported from drive 332606464000 position 153 Thread tape_thread 00:00:48 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV new bitfile request {'uid': 0, 'external_label': 'VOR466', 'gid': 0, 'location_cookie': '0000_000000000_0000152', 'complete_crc': 1045879869L, 'sanity_cookie': (65536L, 2568557746L), 'size': 2994385853L} Thread tape_thread 00:00:48 stkenmvr115a.fnal.gov 005896 root I LTO15MV skip_fm -323 Thread seek_thread 00:00:48 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV set remaining: VOR466 332606464000 0000_000000000_0000153 Thread tape_thread 00:00:48 last message repeated 1 times 00:00:48 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-03 00:00:48', eod_cookie = '0000_000000000_0000153', non_del_files = '152', remaining_bytes = 332606464000 where label = 'VOR466'; 00:00:48 stkenmvr115a.fnal.gov 005896 root I LTO15MV skip_fm 1 Thread seek_thread 00:00:48 stkenmvr115a.fnal.gov 005896 root I LTO15MV read_tape starting, bytes_to_read=881593643 Thread tape_thread 00:00:48 cmsstor153.fnal.gov 024127 root I ENCP Creating zero length output files. 00:00:48 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV transfer complete volume=VOR466 location=153 Thread tape_thread 00:00:48 cmsstor153.fnal.gov 024127 root I ENCP Sending /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/2ED35FFF-3CC8-DE11-A93F-001A92971AA8.root write request to LM: unique_id: cmsstor153.fnal.gov-1257228048-24127-0 inputfile: /storage/data1/write-pool-1/data/000700000000000061D75468 outputfile: /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D75468) 00:00:48 last message repeated 1 times 00:00:48 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '152', modification_time = '2009-11-03 00:00:48' where label = 'VOR466'; 00:00:48 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1255359031.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOR466', 'wrapper': 'cpio_odc', 'remaining_bytes': 335404032000L, 'sum_mounts': 2, 'sum_rd_access': 0, 'file_family_width': 10, 'last_access': 1257228010.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000152', 'capacity_bytes': 858993459200L, 'non_del_files': 151, 'sum_wr_err': 0, 'sum_wr_access': 151, 'file_family': 'cms11', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'modification_time': 1257228010.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257221826.0}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/D0DC4BDC-36C8-DE11-A678-003048678D52.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2994385853L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7864, 'mc_device': '0,6,1,13', 'compression': 0, 'serial_num': '1310050819', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr204a', 'max_time_in_state': 1200, 'device': 'stkenmvr204a:/dev/rmt/tps0d0n', 'name': 'LTO4_104.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_104.stat', 'mover_address': ('131.225.13.199', 7864), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.199', 34093), 'data_ip': 'stkenmvr204a', 'logname': 'LTO4_104MV', 'hostip': '131.225.13.199', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/D0DC4BDC-36C8-DE11-A678-003048678D52.root', 'uid': 0, 'uname': 'root', 'mtime': 1257226957, 'rmajor': 0, 'machine': ('Linux', 'cmsstor88.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 130813352, 21L, 1, 9811, 5063, 512, 1257226372, 1257226372, 1257220971), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2994385853L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data1/write-pool-1/data/000700000000000061D337D0', 'inode': 131282896L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/000700000000000061D337D0', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/.(access)(000700000000000061D337D0)', 'fc': {'uid': 0, 'external_label': 'VOR466', 'gid': 0, 'location_cookie': '0000_000000000_0000152', 'complete_crc': 1045879869L, 'bfid': 'CDMS125722804800000', 'sanity_cookie': (65536L, 2568557746L), 'size': 2994385853L}, 'infilepath': '/storage/data1/write-pool-1/data/000700000000000061D337D0', 'at_the_top': 2, 'local_inode': 3583, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.189.88', 38263), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257226955.6078351, 'seek_time': 0.031024932861328125, 'transfer_time': 36.941407918930054, 't0': 1257226955, 'mount_time': 0.016840934753417969, 'lm_dequeued': 1257228011.726547, 'job_queued': 1257226957.494972, 'drive_call_time': 10.569391489028931, 'in_queue': 1054.2313029766083, 'drive_transfer_time': 27.057755947113037}, 'user_level': 'admin', 'unique_id': 'cmsstor88.fnal.gov-1257226957-6559-0'} Thread tape_thread 00:00:48 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV Done is sent Thread tape_thread 00:00:48 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV maybe_clean Thread tape_thread 00:00:48 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:48 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data1/write-pool-1/data/000700000000000061D75468 (cmsstor153.fnal.gov-1257228048-24127-0) -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/2ED35FFF-3CC8-DE11-A93F-001A92971AA8.root : library=CD-LTO4G1 family=cms11 requester:root volume_family:cms.cms11.cpio_odc 00:00:48 cmsstor153.fnal.gov 024127 root I ENCP Listening for control socket at: ('131.225.189.124', 56546) 00:00:48 cmsstor153.fnal.gov 024127 root I ENCP Waiting for mover to connect control socket at address: ('131.225.189.124', 56546) elapsed=0.930sec 00:00:48 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV returned from maybe_clean Thread tape_thread 00:00:48 cmsstor88.fnal.gov 006559 root I ENCP Received final dialog for cmsstor88.fnal.gov-1257226957-6559-0. elapsed=1093.440sec 00:00:48 cmsstor88.fnal.gov 006559 root I ENCP leaving transfer_file(). 00:00:49 cmsstor88.fnal.gov 006559 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:49 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:49 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/D0DC4BDC-36C8-DE11-A678-003048678D52.root', drive = 'stkenmvr204a:/dev/rmt/tps0d0n:1310050819', pnfs_id = '000700000000000061D337D0' where bfid = 'CDMS125722804800000'; 00:00:49 cmsstor88.fnal.gov 006559 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /storage/data1/write-pool-1/data/000700000000000061D337D0 -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/D0DC4BDC-36C8-DE11-A678-003048678D52.root: 2994385853 bytes copied to VOR466 at 77.4 MB/S (90.3 MB/S network) (106 MB/S drive) (87.2 MB/S disk) mover=LTO4_104.mover drive_id=ULTRIUM-TD4 drive_sn=1310050819 drive_vendor=IBM elapsed=1094 {'storage_group': 'cms', 'encp_crc': 1045879869L, 'disk_rate': 87.167051630340353, 'drive_rate': 105.53975307479297, 'driver': 'FTTDriver', 'transfer_rate': 77.403022568190352, 'mover_interface': 'stkenmvr204a', 'overall_rate': 75.921353306037616, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 77.403022568190352, 'encp_ip': '131.225.189.88', 'unique_id': 'cmsstor88.fnal.gov-1257226957-6559-0'} 00:00:49 cmsstor88.fnal.gov 006559 root I ENCP File /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/D0DC4BDC-36C8-DE11-A678-003048678D52.root copied successfully. 00:00:49 cmsstor88.fnal.gov 006559 root I ENCP Completed transferring 2994385853 bytes in 1 files in 1093.97535706 sec. Overall rate = 75.9 MB/sec. Transfer rate = 77.4 MB/sec. Network rate = 90.3 MB/sec. Drive rate = 106 MB/sec. Disk rate = 87.2 MB/sec. Exit status = 0. 00:00:49 cmsstor139.fnal.gov 006705 root I ENCP File /storage/data3/write-pool-3/data/000700000000000061D592E8 transfered for cmsstor139.fnal.gov-1257226960-6705-0 in 22.0360019207 seconds. 00:00:49 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:49 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VOR466 mover=LTO4_104.mover requester:root 00:00:49 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/288F67DA-3AC8-DE11-92F5-002618943950.root to mover LTO4_104.mover ('131.225.13.199', 7864) DEL_DISM None 00:00:49 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:49 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'cms', 'external_label': 'VOR466', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/288F67DA-3AC8-DE11-92F5-002618943950.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2119291361L, 'mover': 'LTO4_104.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/288F67DA-3AC8-DE11-92F5-002618943950.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257226986, 'rmajor': 0, 'machine': ('Linux', 'cmsstor91.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16893, 131219952, 21L, 1, 9811, 5063, 512, 1257226981, 1257226981, 1257224768), 'mode_octal': '0100036', 'mode': 33188, 'gname': 'root', 'size_bytes': 2119291361L, 'fullname': '/storage/data4/write-pool-4/data/000700000000000061D58DA8', 'inode': 131435944L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data4/write-pool-4/data/000700000000000061D58DA8', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D58DA8)', 'fc': {'size': 2119291361L, 'external_label': 'VOR466', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data4/write-pool-4/data/000700000000000061D58DA8', 'at_the_top': 107, 'local_inode': 1073745000, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.205.213', 40810), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228049.752449, 'encp_start_time': 1257226985.2599549, 'job_queued': 1257226987.0520489, 't0': 1257226985, 'in_queue': 1062.7001240253448}, 'user_level': 'admin', 'unique_id': 'cmsstor91.fnal.gov-1257226986-22583-0'} to mover 00:00:49 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:49 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV WRITE_TO_HSM Thread MainThread 00:00:49 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV SENDING {'vc': {'storage_group': 'cms', 'external_label': 'VOR466', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/288F67DA-3AC8-DE11-92F5-002618943950.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2119291361L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7864, 'mc_device': '0,6,1,13', 'compression': 0, 'serial_num': '1310050819', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr204a', 'max_time_in_state': 1200, 'device': 'stkenmvr204a:/dev/rmt/tps0d0n', 'name': 'LTO4_104.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_104.stat', 'mover_address': ('131.225.13.199', 7864), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.199', 34100), 'data_ip': 'stkenmvr204a', 'logname': 'LTO4_104MV', 'hostip': '131.225.13.199', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/288F67DA-3AC8-DE11-92F5-002618943950.root', 'uid': 0, 'uname': 'root', 'mtime': 1257226986, 'rmajor': 0, 'machine': ('Linux', 'cmsstor91.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 131219952, 21L, 1, 9811, 5063, 512, 1257226981, 1257226981, 1257224768), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2119291361L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data4/write-pool-4/data/000700000000000061D58DA8', 'inode': 131435944L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data4/write-pool-4/data/000700000000000061D58DA8', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D58DA8)', 'fc': {'size': 2119291361L, 'external_label': 'VOR466', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data4/write-pool-4/data/000700000000000061D58DA8', 'at_the_top': 107, 'local_inode': 1073745000, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.205.213', 40810), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228049.752449, 'encp_start_time': 1257226985.2599549, 'job_queued': 1257226987.0520489, 't0': 1257226985, 'in_queue': 1062.7001240253448}, 'user_level': 'admin', 'unique_id': 'cmsstor91.fnal.gov-1257226986-22583-0'} Thread client_connect_thread 00:00:49 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:49 cmsstor91.fnal.gov 022583 root I ENCP Control socket ('131.225.205.213', 40810) is connected to ('131.225.13.199', 34101) for cmsstor91.fnal.gov-1257226986-22583-0. elapsed=1064.690sec 00:00:49 cmsstor91.fnal.gov 022583 root I ENCP Received callback ticket from mover LTO4_104.mover for transfer cmsstor91.fnal.gov-1257226986-22583-0. elapsed=1064.700sec 00:00:49 cmsstor91.fnal.gov 022583 root I ENCP Atempting to connect data socket to mover at ('131.225.13.199', 34100). 00:00:49 cmsstor91.fnal.gov 022583 root I ENCP Connecting data socket to mover (('131.225.13.199', 34100)) with interface 131.225.205.213. elapsed=1064.700sec 00:00:49 cmsstor91.fnal.gov 022583 root I ENCP Data socket ('131.225.205.213', 58752) is connected to ('131.225.13.199', 34100). elapsed=1064.700sec 00:00:49 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV tape position: target 153 current 153 0 Thread seek_thread 00:00:49 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV write_tape starting, bytes_to_write=2119291904 Thread tape_thread 00:00:49 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV Write starting. Tape VOR466 absolute location in blocks 3610207 Thread tape_thread 00:00:49 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV written bytes 2382122496/2382122496, blocks 18175 header 192 trailer 211 Thread tape_thread 00:00:49 cmsstor91.fnal.gov 022583 root I ENCP Opening local file. 00:00:49 stkenmvr212a.fnal.gov 006265 root I LTO4_112MV Tape is ejected Thread media_thread 00:00:50 stkenmvr212a.fnal.gov 006265 root I LTO4_112MV dismounting VOO096 Thread media_thread 00:00:50 cmsstor91.fnal.gov 022583 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:50 stkensrv4n.fnal.gov 005843 enstore I SL8500F1MC PROCESSING dismount VOO096 1,0,1,11 00:00:50 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV filemarks written. Tape VOR468 absolute location in blocks 1405628 Thread tape_thread 00:00:50 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV write_tape timing:pull 0.20902967453 write 6.26980113983 crc_check 9.4083404541 freespace 0.229799509048 block_write 16.1169707775 idle 0.35854268074 Thread tape_thread 00:00:50 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV drive stats after write. Tape VOR468 position 66 block 0 block_size 0 bloc_loc 1405628 tot_blocks 762939000 BOT 0 read_err 0 write_err 24 bytes 2382122496 block_write_tot 16.1169707775 tape_rate 147802122.923 Thread tape_thread 00:00:50 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV remaining bytes info in DB 605984768000 estimated 603602645504 reported from drive 603759616000 position 67 Thread tape_thread 00:00:50 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV new bitfile request {'uid': 0, 'external_label': 'VOR468', 'gid': 0, 'location_cookie': '0000_000000000_0000066', 'complete_crc': 2629810170L, 'sanity_cookie': (65536L, 2812451317L), 'size': 2382122093L} Thread tape_thread 00:00:50 cmsstor140.fnal.gov 027668 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms11 FF Wrapper: cpio_odc FF Width: 10 Current working directory: cmsstor140.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --priority 25 --pnfs-mount /pnfs/fs --put-cache 000700000000000061D772C0 /storage/data3/write-pool-3/data/000700000000000061D772C0 00:00:50 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV set remaining: VOR468 603759616000 0000_000000000_0000067 Thread tape_thread 00:00:50 last message repeated 1 times 00:00:50 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-03 00:00:50', eod_cookie = '0000_000000000_0000067', non_del_files = '68', remaining_bytes = 603759616000 where label = 'VOR468'; 00:00:50 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV transfer complete volume=VOR468 location=67 Thread tape_thread 00:00:50 last message repeated 1 times 00:00:50 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '66', modification_time = '2009-11-03 00:00:50' where label = 'VOR468'; 00:00:50 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1255359037.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOR468', 'wrapper': 'cpio_odc', 'remaining_bytes': 605984768000L, 'sum_mounts': 2, 'sum_rd_access': 0, 'file_family_width': 10, 'last_access': 1257228026.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000066', 'capacity_bytes': 858993459200L, 'non_del_files': 67, 'sum_wr_err': 0, 'sum_wr_access': 65, 'file_family': 'cms11', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'modification_time': 1257228026.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257225233.0}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/466ED5C9-3AC8-DE11-81DA-002618FDA208.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2382122093L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7840, 'mc_device': '0,7,1,15', 'compression': 0, 'serial_num': '9310801386', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr140a', 'max_time_in_state': 1200, 'device': 'stkenmvr140a:/dev/rmt/tps0d0n', 'name': 'LTO4_40.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_40.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.168', 7840), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.168', 46386), 'data_ip': 'stkenmvr140a', 'logname': 'LTO4_40MV', 'hostip': '131.225.13.168', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/466ED5C9-3AC8-DE11-81DA-002618FDA208.root', 'uid': 0, 'uname': 'root', 'mtime': 1257226960, 'rmajor': 0, 'machine': ('Linux', 'cmsstor139.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 131219952, 21L, 1, 9811, 5063, 512, 1257226957, 1257226957, 1257224768), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2382122093L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data3/write-pool-3/data/000700000000000061D592E8', 'inode': 131437288L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/000700000000000061D592E8', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D592E8)', 'fc': {'uid': 0, 'external_label': 'VOR468', 'gid': 0, 'location_cookie': '0000_000000000_0000066', 'complete_crc': 2629810170L, 'bfid': 'CDMS125722805000000', 'sanity_cookie': (65536L, 2812451317L), 'size': 2382122093L}, 'infilepath': '/storage/data3/write-pool-3/data/000700000000000061D592E8', 'at_the_top': 47, 'local_inode': 3618, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.189.110', 45033), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257226959.9829371, 'seek_time': 0.035223007202148438, 'transfer_time': 22.930948972702026, 't0': 1257226959, 'mount_time': 0.042315006256103516, 'lm_dequeued': 1257228027.4073009, 'job_queued': 1257226960.8563781, 'drive_call_time': 6.0158476829528809, 'in_queue': 1066.5506448745728, 'drive_transfer_time': 17.104048013687134}, 'user_level': 'admin', 'unique_id': 'cmsstor139.(trunc. 4108) Thread tape_thread 00:00:50 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV Done is sent Thread tape_thread 00:00:50 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV maybe_clean Thread tape_thread 00:00:50 cmsstor139.fnal.gov 006705 root I ENCP Received final dialog for cmsstor139.fnal.gov-1257226960-6705-0. elapsed=1090.360sec 00:00:50 cmsstor139.fnal.gov 006705 root I ENCP leaving transfer_file(). 00:00:50 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV returned from maybe_clean Thread tape_thread 00:00:50 cmsstor69.fnal.gov 022775 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms11 FF Wrapper: cpio_odc FF Width: 10 Current working directory: cmsstor69.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --priority 25 --pnfs-mount /pnfs/fs --put-cache 000700000000000061D1D450 /storage/data4/write-pool-4/data/000700000000000061D1D450 00:00:50 cmsstor139.fnal.gov 006705 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:50 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:50 cmsstor140.fnal.gov 027668 root I ENCP Creating zero length output files. 00:00:50 cmsstor140.fnal.gov 027668 root I ENCP Sending /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/04ECF7FA-3CC8-DE11-941B-002618943951.root write request to LM: unique_id: cmsstor140.fnal.gov-1257228050-27668-0 inputfile: /storage/data3/write-pool-3/data/000700000000000061D772C0 outputfile: /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D772C0) 00:00:50 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:50 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data3/write-pool-3/data/000700000000000061D772C0 (cmsstor140.fnal.gov-1257228050-27668-0) -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/04ECF7FA-3CC8-DE11-941B-002618943951.root : library=CD-LTO4G1 family=cms11 requester:root volume_family:cms.cms11.cpio_odc 00:00:50 cmsstor140.fnal.gov 027668 root I ENCP Listening for control socket at: ('131.225.189.111', 45311) 00:00:50 cmsstor140.fnal.gov 027668 root I ENCP Waiting for mover to connect control socket at address: ('131.225.189.111', 45311) elapsed=1.020sec 00:00:50 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/466ED5C9-3AC8-DE11-81DA-002618FDA208.root', drive = 'stkenmvr140a:/dev/rmt/tps0d0n:9310801386', pnfs_id = '000700000000000061D592E8' where bfid = 'CDMS125722805000000'; 00:00:50 cmsstor69.fnal.gov 022775 root I ENCP Creating zero length output files. 00:00:50 cmsstor69.fnal.gov 022775 root I ENCP Sending /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/D488CF23-34C8-DE11-A38A-0017312310E7.root write request to LM: unique_id: cmsstor69.fnal.gov-1257228050-22775-0 inputfile: /storage/data4/write-pool-4/data/000700000000000061D1D450 outputfile: /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/.(access)(000700000000000061D1D450) 00:00:50 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:50 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data4/write-pool-4/data/000700000000000061D1D450 (cmsstor69.fnal.gov-1257228050-22775-0) -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/D488CF23-34C8-DE11-A38A-0017312310E7.root : library=CD-LTO4G1 family=cms11 requester:root volume_family:cms.cms11.cpio_odc 00:00:51 cmsstor69.fnal.gov 022775 root I ENCP Listening for control socket at: ('131.225.204.245', 45848) 00:00:51 cmsstor69.fnal.gov 022775 root I ENCP Waiting for mover to connect control socket at address: ('131.225.204.245', 45848) elapsed=1.250sec 00:00:51 cmsstor139.fnal.gov 006705 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /storage/data3/write-pool-3/data/000700000000000061D592E8 -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/466ED5C9-3AC8-DE11-81DA-002618FDA208.root: 2382122093 bytes copied to VOR468 at 99 MB/S (93.4 MB/S network) (133 MB/S drive) (103 MB/S disk) mover=LTO4_40.mover drive_id=ULTRIUM-TD4 drive_sn=9310801386 drive_vendor=IBM elapsed=1091.1 {'storage_group': 'cms', 'encp_crc': 2629810170L, 'disk_rate': 103.14672503958248, 'drive_rate': 132.82052627659067, 'driver': 'FTTDriver', 'transfer_rate': 99.045282475286669, 'mover_interface': 'stkenmvr140a', 'overall_rate': 96.319530782146273, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 99.045282475286669, 'encp_ip': '131.225.189.110', 'unique_id': 'cmsstor139.fnal.gov-1257226960-6705-0'} 00:00:51 cmsstor139.fnal.gov 006705 root I ENCP File /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/466ED5C9-3AC8-DE11-81DA-002618FDA208.root copied successfully. 00:00:51 cmsstor139.fnal.gov 006705 root I ENCP Completed transferring 2382122093 bytes in 1 files in 1091.07684183 sec. Overall rate = 96.3 MB/sec. Transfer rate = 99 MB/sec. Network rate = 93.4 MB/sec. Drive rate = 133 MB/sec. Disk rate = 103 MB/sec. Exit status = 0. 00:00:51 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV written bytes 2706915840/2706915840, blocks 20653 header 192 trailer 210 Thread tape_thread 00:00:51 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:51 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VOR468 mover=LTO4_40.mover requester:root 00:00:51 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/62FD4D7D-36C8-DE11-BCCC-003048678FEA.root to mover LTO4_40.mover ('131.225.13.168', 7840) DEL_DISM None 00:00:51 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:51 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'cms', 'external_label': 'VOR468', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/62FD4D7D-36C8-DE11-BCCC-003048678FEA.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2001871474L, 'mover': 'LTO4_40.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/62FD4D7D-36C8-DE11-BCCC-003048678FEA.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257226998, 'rmajor': 0, 'machine': ('Linux', 'cmsstor109.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16893, 130813352, 21L, 1, 9811, 5063, 512, 1257226372, 1257226372, 1257220971), 'mode_octal': '0100036', 'mode': 33188, 'gname': 'root', 'size_bytes': 2001871474L, 'fullname': '/storage/data4/write-pool-4/data/000700000000000061D2C3A0', 'inode': 131253152L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data4/write-pool-4/data/000700000000000061D2C3A0', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/.(access)(000700000000000061D2C3A0)', 'fc': {'size': 2001871474L, 'external_label': 'VOR468', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data4/write-pool-4/data/000700000000000061D2C3A0', 'at_the_top': 41, 'local_inode': 1073743984, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.205.232', 39145), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228051.4222419, 'encp_start_time': 1257226996.8017869, 'job_queued': 1257226998.8393509, 't0': 1257226996, 'in_queue': 1052.5825929641724}, 'user_level': 'admin', 'unique_id': 'cmsstor109.fnal.gov-1257226998-1465-0'} to mover 00:00:51 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:51 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV WRITE_TO_HSM Thread MainThread 00:00:51 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV SENDING {'vc': {'storage_group': 'cms', 'external_label': 'VOR468', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/62FD4D7D-36C8-DE11-BCCC-003048678FEA.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2001871474L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7840, 'mc_device': '0,7,1,15', 'compression': 0, 'serial_num': '9310801386', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr140a', 'max_time_in_state': 1200, 'device': 'stkenmvr140a:/dev/rmt/tps0d0n', 'name': 'LTO4_40.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_40.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.168', 7840), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.168', 46389), 'data_ip': 'stkenmvr140a', 'logname': 'LTO4_40MV', 'hostip': '131.225.13.168', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/62FD4D7D-36C8-DE11-BCCC-003048678FEA.root', 'uid': 0, 'uname': 'root', 'mtime': 1257226998, 'rmajor': 0, 'machine': ('Linux', 'cmsstor109.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 130813352, 21L, 1, 9811, 5063, 512, 1257226372, 1257226372, 1257220971), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2001871474L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data4/write-pool-4/data/000700000000000061D2C3A0', 'inode': 131253152L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data4/write-pool-4/data/000700000000000061D2C3A0', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/.(access)(000700000000000061D2C3A0)', 'fc': {'size': 2001871474L, 'external_label': 'VOR468', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data4/write-pool-4/data/000700000000000061D2C3A0', 'at_the_top': 41, 'local_inode': 1073743984, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.205.232', 39145), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228051.4222419, 'encp_start_time': 1257226996.8017869, 'job_queued': 1257226998.8393509, 't0': 1257226996, 'in_queue': 1052.5825929641724}, 'user_level': 'admin', 'unique_id': 'cmsstor109.fnal.gov-1257226998-1465-0'} Thread client_connect_thread 00:00:51 cmsstor109.fnal.gov 001465 root I ENCP Control socket ('131.225.205.232', 39145) is connected to ('131.225.13.168', 46390) for cmsstor109.fnal.gov-1257226998-1465-0. elapsed=1054.760sec 00:00:51 cmsstor109.fnal.gov 001465 root I ENCP Received callback ticket from mover LTO4_40.mover for transfer cmsstor109.fnal.gov-1257226998-1465-0. elapsed=1054.770sec 00:00:51 cmsstor109.fnal.gov 001465 root I ENCP Atempting to connect data socket to mover at ('131.225.13.168', 46389). 00:00:51 cmsstor109.fnal.gov 001465 root I ENCP Connecting data socket to mover (('131.225.13.168', 46389)) with interface 131.225.205.232. elapsed=1054.770sec 00:00:51 cmsstor109.fnal.gov 001465 root I ENCP Data socket ('131.225.205.232', 53455) is connected to ('131.225.13.168', 46389). elapsed=1054.770sec 00:00:51 cmsstor109.fnal.gov 001465 root I ENCP Opening local file. 00:00:51 cmsstor109.fnal.gov 001465 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:51 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV tape position: target 67 current 67 0 Thread seek_thread 00:00:51 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV write_tape starting, bytes_to_write=2001871872 Thread tape_thread 00:00:51 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV Write starting. Tape VOR468 absolute location in blocks 1405628 Thread tape_thread 00:00:51 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:52 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:52 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV filemarks written. Tape VOR463 absolute location in blocks 5084476 Thread tape_thread 00:00:52 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV write_tape timing:pull 0.32720375061 write 8.24625945091 crc_check 12.3125767708 freespace 0.205993652344 block_write 21.0920336246 idle 0.368387460709 Thread tape_thread 00:00:52 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV drive stats after write. Tape VOR463 position 225 block 0 block_size 0 bloc_loc 5084476 tot_blocks 762939000 BOT 0 read_err 0 write_err 11 bytes 2706915840 block_write_tot 21.0920336246 tape_rate 128338304.792 Thread tape_thread 00:00:52 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV remaining bytes info in DB 156900352000 estimated 154193436160 reported from drive 154380288000 position 226 Thread tape_thread 00:00:52 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV new bitfile request {'uid': 0, 'external_label': 'VOR463', 'gid': 0, 'location_cookie': '0000_000000000_0000225', 'complete_crc': 3745019343L, 'sanity_cookie': (65536L, 31095670L), 'size': 2706915438L} Thread tape_thread 00:00:52 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV written bytes 2755907072/2755907072, blocks 21026 header 192 trailer 120 Thread tape_thread 00:00:52 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV set remaining: VOR463 154380288000 0000_000000000_0000226 Thread tape_thread 00:00:52 last message repeated 1 times 00:00:52 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-03 00:00:52', eod_cookie = '0000_000000000_0000226', non_del_files = '226', remaining_bytes = 154380288000 where label = 'VOR463'; 00:00:52 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV transfer complete volume=VOR463 location=226 Thread tape_thread 00:00:52 last message repeated 1 times 00:00:52 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '225', modification_time = '2009-11-03 00:00:52' where label = 'VOR463'; 00:00:52 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1255359021.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOR463', 'wrapper': 'cpio_odc', 'remaining_bytes': 156900352000L, 'sum_mounts': 2, 'sum_rd_access': 0, 'file_family_width': 10, 'last_access': 1257228010.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000225', 'capacity_bytes': 858993459200L, 'non_del_files': 225, 'sum_wr_err': 0, 'sum_wr_access': 224, 'file_family': 'cms11', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'modification_time': 1257228010.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257219432.0}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/4224F208-32C8-DE11-9276-00261894385D.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2706915438L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7870, 'mc_device': '0,5,1,14', 'compression': 0, 'serial_num': '1310049971', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr210a', 'max_time_in_state': 1200, 'device': 'stkenmvr210a:/dev/rmt/tps0d0n', 'name': 'LTO4_110.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_110.stat', 'mover_address': ('131.225.13.206', 7870), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.206', 33477), 'data_ip': 'stkenmvr210a', 'logname': 'LTO4_110MV', 'hostip': '131.225.13.206', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/4224F208-32C8-DE11-9276-00261894385D.root', 'uid': 0, 'uname': 'root', 'mtime': 1257226956, 'rmajor': 0, 'machine': ('Linux', 'cmsstor224.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 130813352, 21L, 1, 9811, 5063, 512, 1257226372, 1257226372, 1257220971), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2706915438L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data2/write-pool-2/data/000700000000000061CF8C48', 'inode': 131042376L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/000700000000000061CF8C48', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/.(access)(000700000000000061CF8C48)', 'fc': {'uid': 0, 'external_label': 'VOR463', 'gid': 0, 'location_cookie': '0000_000000000_0000225', 'complete_crc': 3745019343L, 'bfid': 'CDMS125722805200000', 'sanity_cookie': (65536L, 31095670L), 'size': 2706915438L}, 'infilepath': '/storage/data2/write-pool-2/data/000700000000000061CF8C48', 'at_the_top': 184, 'local_inode': 5333, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.191.96', 32826), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257226954.4518671, 'seek_time': 0.030906915664672852, 'transfer_time': 41.373388051986694, 't0': 1257226954, 'mount_time': 0.017039060592651367, 'lm_dequeued': 1257228011.3968539, 'job_queued': 1257226956.5584309, 'drive_call_time': 7.9687714576721191, 'in_queue': 1054.838142156601, 'drive_transfer_time': 23.298122882843018}, 'user_level': 'admin', 'unique_id': 'cmsstor224.fnal.gov-1257226956-20055-0'} Thread tape_thread 00:00:52 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV Done is sent Thread tape_thread 00:00:52 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV maybe_clean Thread tape_thread 00:00:52 cmsstor224.fnal.gov 020055 root I ENCP Received final dialog for cmsstor224.fnal.gov-1257226956-20055-0. elapsed=1098.440sec 00:00:52 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV returned from maybe_clean Thread tape_thread 00:00:52 cmsstor224.fnal.gov 020055 root I ENCP leaving transfer_file(). 00:00:53 cmsstor224.fnal.gov 020055 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:53 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/4224F208-32C8-DE11-9276-00261894385D.root', drive = 'stkenmvr210a:/dev/rmt/tps0d0n:1310049971', pnfs_id = '000700000000000061CF8C48' where bfid = 'CDMS125722805200000'; 00:00:53 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:53 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VOR463 mover=LTO4_110.mover requester:root 00:00:53 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/F271E9F6-3AC8-DE11-B931-0026189438DA.root to mover LTO4_110.mover ('131.225.13.206', 7870) DEL_DISM None 00:00:53 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:53 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'cms', 'external_label': 'VOR463', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/F271E9F6-3AC8-DE11-B931-0026189438DA.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2288270007L, 'mover': 'LTO4_110.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/F271E9F6-3AC8-DE11-B931-0026189438DA.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257227006, 'rmajor': 0, 'machine': ('Linux', 'cmsstor239.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16893, 131219952, 21L, 1, 9811, 5063, 512, 1257227005, 1257227005, 1257224768), 'mode_octal': '0100036', 'mode': 33188, 'gname': 'root', 'size_bytes': 2288270007L, 'fullname': '/storage/data3/write-pool-3/data/000700000000000061D5A058', 'inode': 131440728L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/000700000000000061D5A058', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D5A058)', 'fc': {'size': 2288270007L, 'external_label': 'VOR463', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data3/write-pool-3/data/000700000000000061D5A058', 'at_the_top': 8, 'local_inode': 9163, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.191.111', 34381), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228053.4204049, 'encp_start_time': 1257227005.170119, 'job_queued': 1257227006.1859829, 't0': 1257227005, 'in_queue': 1047.2341251373291}, 'user_level': 'admin', 'unique_id': 'cmsstor239.fnal.gov-1257227006-11356-0'} to mover 00:00:53 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:53 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV WRITE_TO_HSM Thread MainThread 00:00:53 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV SENDING {'vc': {'storage_group': 'cms', 'external_label': 'VOR463', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/F271E9F6-3AC8-DE11-B931-0026189438DA.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2288270007L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7870, 'mc_device': '0,5,1,14', 'compression': 0, 'serial_num': '1310049971', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr210a', 'max_time_in_state': 1200, 'device': 'stkenmvr210a:/dev/rmt/tps0d0n', 'name': 'LTO4_110.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_110.stat', 'mover_address': ('131.225.13.206', 7870), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.206', 33483), 'data_ip': 'stkenmvr210a', 'logname': 'LTO4_110MV', 'hostip': '131.225.13.206', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/F271E9F6-3AC8-DE11-B931-0026189438DA.root', 'uid': 0, 'uname': 'root', 'mtime': 1257227006, 'rmajor': 0, 'machine': ('Linux', 'cmsstor239.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 131219952, 21L, 1, 9811, 5063, 512, 1257227005, 1257227005, 1257224768), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2288270007L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data3/write-pool-3/data/000700000000000061D5A058', 'inode': 131440728L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/000700000000000061D5A058', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D5A058)', 'fc': {'size': 2288270007L, 'external_label': 'VOR463', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data3/write-pool-3/data/000700000000000061D5A058', 'at_the_top': 8, 'local_inode': 9163, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.191.111', 34381), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228053.4204049, 'encp_start_time': 1257227005.170119, 'job_queued': 1257227006.1859829, 't0': 1257227005, 'in_queue': 1047.2341251373291}, 'user_level': 'admin', 'unique_id': 'cmsstor239.fnal.gov-1257227006-11356-0'} Thread client_connect_thread 00:00:53 cmsstor239.fnal.gov 011356 root I ENCP Control socket ('131.225.191.111', 34381) is connected to ('131.225.13.206', 33484) for cmsstor239.fnal.gov-1257227006-11356-0. elapsed=1048.280sec 00:00:53 cmsstor239.fnal.gov 011356 root I ENCP Received callback ticket from mover LTO4_110.mover for transfer cmsstor239.fnal.gov-1257227006-11356-0. elapsed=1048.280sec 00:00:53 cmsstor239.fnal.gov 011356 root I ENCP Atempting to connect data socket to mover at ('131.225.13.206', 33483). 00:00:53 cmsstor239.fnal.gov 011356 root I ENCP Connecting data socket to mover (('131.225.13.206', 33483)) with interface 131.225.191.111. elapsed=1048.280sec 00:00:53 cmsstor239.fnal.gov 011356 root I ENCP Data socket ('131.225.191.111', 50839) is connected to ('131.225.13.206', 33483). elapsed=1048.280sec 00:00:53 cmsstor239.fnal.gov 011356 root I ENCP Opening local file. 00:00:53 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV tape position: target 226 current 226 0 Thread seek_thread 00:00:53 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV write_tape starting, bytes_to_write=2288270336 Thread tape_thread 00:00:53 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV Write starting. Tape VOR463 absolute location in blocks 5084476 Thread tape_thread 00:00:53 cmsstor239.fnal.gov 011356 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:53 cmsstor224.fnal.gov 020055 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /storage/data2/write-pool-2/data/000700000000000061CF8C48 -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/4224F208-32C8-DE11-9276-00261894385D.root: 2706915438 bytes copied to VOR463 at 62.6 MB/S (93.7 MB/S network) (111 MB/S drive) (90.4 MB/S disk) mover=LTO4_110.mover drive_id=ULTRIUM-TD4 drive_sn=1310049971 drive_vendor=IBM elapsed=1099.2 {'storage_group': 'cms', 'encp_crc': 3745019343L, 'disk_rate': 90.440820408062379, 'drive_rate': 110.80359318583687, 'driver': 'FTTDriver', 'transfer_rate': 62.58168045557511, 'mover_interface': 'stkenmvr210a', 'overall_rate': 61.29777276631507, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 62.58168045557511, 'encp_ip': '131.225.191.96', 'unique_id': 'cmsstor224.fnal.gov-1257226956-20055-0'} 00:00:53 cmsstor224.fnal.gov 020055 root I ENCP File /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/4224F208-32C8-DE11-9276-00261894385D.root copied successfully. 00:00:53 cmsstor224.fnal.gov 020055 root I ENCP Completed transferring 2706915438 bytes in 1 files in 1099.23598099 sec. Overall rate = 61.3 MB/sec. Transfer rate = 62.6 MB/sec. Network rate = 93.7 MB/sec. Drive rate = 111 MB/sec. Disk rate = 90.4 MB/sec. Exit status = 0. 00:00:53 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:53 stkensrv4n.fnal.gov 005357 enstore E LTO4G1LM MOVER ERROR RQ {'status': ('ok', None), 'returned_work': None, 'library': None, 'mover_type': '', 'current_location': '0000_000000000_0000000', 'ip_map': '', 'operation': None, 'read_only': 0, 'current_time': 1257228053.8354089, 'error_source': None, 'external_label': None, 'state': 'OFFLINE', 'r_a': (('131.225.13.194', 54880), 12823L, '131.225.13.194-54880-1256586929.509013-4866'), 'library_list': [('CD-LTO4G1.library_manager', ('131.225.13.4', 8541))], 'current_priority': (None, None), 'transfer_deficiency': 1, 'time_in_state': 641123.47535085678, 'address': ('131.225.13.194', 7860), 'volume_family': None, 'work': 'mover_error', 'mover': 'LTO4_99.mover', 'volume_status': (['none', 'none'], ['none', 'none']), 'unique_id': None} 00:00:54 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV filemarks written. Tape VOR467 absolute location in blocks 3355199 Thread tape_thread 00:00:54 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV write_tape timing:pull 0.216860294342 write 12.0840232372 crc_check 12.7382712364 freespace 0.222018718719 block_write 25.2611734867 idle 0.393649101257 Thread tape_thread 00:00:54 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV drive stats after write. Tape VOR467 position 141 block 0 block_size 0 bloc_loc 3355199 tot_blocks 762939000 BOT 0 read_err 0 write_err 3 bytes 2755907072 block_write_tot 25.2611734867 tape_rate 109096557.745 Thread tape_thread 00:00:54 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV remaining bytes info in DB 371446784000 estimated 368690876928 reported from drive 368874496000 position 142 Thread tape_thread 00:00:54 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV new bitfile request {'uid': 0, 'external_label': 'VOR467', 'gid': 0, 'location_cookie': '0000_000000000_0000141', 'complete_crc': 3369524612L, 'sanity_cookie': (65536L, 1400859234L), 'size': 2755906760L} Thread tape_thread 00:00:54 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV set remaining: VOR467 368874496000 0000_000000000_0000142 Thread tape_thread 00:00:54 last message repeated 1 times 00:00:54 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-03 00:00:54', eod_cookie = '0000_000000000_0000142', non_del_files = '142', remaining_bytes = 368874496000 where label = 'VOR467'; 00:00:54 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV transfer complete volume=VOR467 location=142 Thread tape_thread 00:00:54 last message repeated 1 times 00:00:54 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '141', modification_time = '2009-11-03 00:00:54' where label = 'VOR467'; 00:00:54 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1255359034.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOR467', 'wrapper': 'cpio_odc', 'remaining_bytes': 371446784000L, 'sum_mounts': 2, 'sum_rd_access': 0, 'file_family_width': 10, 'last_access': 1257228021.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000141', 'capacity_bytes': 858993459200L, 'non_del_files': 141, 'sum_wr_err': 0, 'sum_wr_access': 140, 'file_family': 'cms11', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'modification_time': 1257228021.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257222369.0}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/ACF4ABCC-38C8-DE11-8FD4-003048678FF4.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2755906760L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7856, 'mc_device': '0,7,1,14', 'compression': 0, 'serial_num': '1310155266', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr195a', 'max_time_in_state': 1200, 'device': 'stkenmvr195a:/dev/rmt/tps0d0n', 'name': 'LTO4_95.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_95.stat', 'mover_address': ('131.225.13.190', 7856), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.190', 34720), 'data_ip': 'stkenmvr195a', 'logname': 'LTO4_95MV', 'hostip': '131.225.13.190', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/ACF4ABCC-38C8-DE11-8FD4-003048678FF4.root', 'uid': 0, 'uname': 'root', 'mtime': 1257226958, 'rmajor': 0, 'machine': ('Linux', 'cmsstor245.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 131219952, 21L, 1, 9811, 5063, 512, 1257226957, 1257226957, 1257224768), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2755906760L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data2/write-pool-2/data/000700000000000061D475B8', 'inode': 131364280L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/000700000000000061D475B8', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D475B8)', 'fc': {'uid': 0, 'external_label': 'VOR467', 'gid': 0, 'location_cookie': '0000_000000000_0000141', 'complete_crc': 3369524612L, 'bfid': 'CDMS125722805400000', 'sanity_cookie': (65536L, 1400859234L), 'size': 2755906760L}, 'infilepath': '/storage/data2/write-pool-2/data/000700000000000061D475B8', 'at_the_top': 92, 'local_inode': 5081, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.191.117', 57486), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257226957.522341, 'seek_time': 0.030194997787475586, 'transfer_time': 31.721252918243408, 't0': 1257226957, 'mount_time': 0.016824007034301758, 'lm_dequeued': 1257228022.4328079, 'job_queued': 1257226958.9714041, 'drive_call_time': 11.793040990829468, 'in_queue': 1063.4611349105835, 'drive_transfer_time': 27.503332853317261}, 'user_level': 'admin', 'unique_id': 'cmsstor245.fnal.gov-1257226958-15098-0'} Thread tape_thread 00:00:54 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV Done is sent Thread tape_thread 00:00:54 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV maybe_clean Thread tape_thread 00:00:54 cmsstor245.fnal.gov 015098 root I ENCP Received final dialog for cmsstor245.fnal.gov-1257226958-15098-0. elapsed=1096.740sec 00:00:54 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV returned from maybe_clean Thread tape_thread 00:00:54 cmsstor245.fnal.gov 015098 root I ENCP leaving transfer_file(). 00:00:54 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:54 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:54 cmsstor245.fnal.gov 015098 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:54 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VOR467 mover=LTO4_95.mover requester:root 00:00:54 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/0450A596-3AC8-DE11-B718-00304867BEC0.root to mover LTO4_95.mover ('131.225.13.190', 7856) DEL_DISM None 00:00:54 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:54 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'cms', 'external_label': 'VOR467', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/0450A596-3AC8-DE11-B718-00304867BEC0.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2372786731L, 'mover': 'LTO4_95.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/0450A596-3AC8-DE11-B718-00304867BEC0.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257227007, 'rmajor': 0, 'machine': ('Linux', 'cmsstor60.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16893, 131219952, 21L, 1, 9811, 5063, 512, 1257227005, 1257227005, 1257224768), 'mode_octal': '0100036', 'mode': 33188, 'gname': 'root', 'size_bytes': 2372786731L, 'fullname': '/storage/data4/write-pool-4/data/000700000000000061D58330', 'inode': 131433264L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data4/write-pool-4/data/000700000000000061D58330', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D58330)', 'fc': {'size': 2372786731L, 'external_label': 'VOR467', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data4/write-pool-4/data/000700000000000061D58330', 'at_the_top': 5, 'local_inode': 2608, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.189.60', 47221), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228054.440516, 'encp_start_time': 1257227006.5399599, 'job_queued': 1257227008.2494781, 't0': 1257227006, 'in_queue': 1046.1907649040222}, 'user_level': 'admin', 'unique_id': 'cmsstor60.fnal.gov-1257227007-5464-0'} to mover 00:00:54 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:54 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV WRITE_TO_HSM Thread MainThread 00:00:54 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV SENDING {'vc': {'storage_group': 'cms', 'external_label': 'VOR467', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/0450A596-3AC8-DE11-B718-00304867BEC0.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2372786731L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7856, 'mc_device': '0,7,1,14', 'compression': 0, 'serial_num': '1310155266', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr195a', 'max_time_in_state': 1200, 'device': 'stkenmvr195a:/dev/rmt/tps0d0n', 'name': 'LTO4_95.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_95.stat', 'mover_address': ('131.225.13.190', 7856), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.190', 34727), 'data_ip': 'stkenmvr195a', 'logname': 'LTO4_95MV', 'hostip': '131.225.13.190', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/0450A596-3AC8-DE11-B718-00304867BEC0.root', 'uid': 0, 'uname': 'root', 'mtime': 1257227007, 'rmajor': 0, 'machine': ('Linux', 'cmsstor60.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 131219952, 21L, 1, 9811, 5063, 512, 1257227005, 1257227005, 1257224768), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2372786731L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data4/write-pool-4/data/000700000000000061D58330', 'inode': 131433264L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data4/write-pool-4/data/000700000000000061D58330', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D58330)', 'fc': {'size': 2372786731L, 'external_label': 'VOR467', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data4/write-pool-4/data/000700000000000061D58330', 'at_the_top': 5, 'local_inode': 2608, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.189.60', 47221), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228054.440516, 'encp_start_time': 1257227006.5399599, 'job_queued': 1257227008.2494781, 't0': 1257227006, 'in_queue': 1046.1907649040222}, 'user_level': 'admin', 'unique_id': 'cmsstor60.fnal.gov-1257227007-5464-0'} Thread client_connect_thread 00:00:54 cmsstor60.fnal.gov 005464 root I ENCP Control socket ('131.225.189.60', 47221) is connected to ('131.225.13.190', 34728) for cmsstor60.fnal.gov-1257227007-5464-0. elapsed=1048.030sec 00:00:54 cmsstor60.fnal.gov 005464 root I ENCP Received callback ticket from mover LTO4_95.mover for transfer cmsstor60.fnal.gov-1257227007-5464-0. elapsed=1048.040sec 00:00:54 cmsstor60.fnal.gov 005464 root I ENCP Atempting to connect data socket to mover at ('131.225.13.190', 34727). 00:00:54 cmsstor60.fnal.gov 005464 root I ENCP Connecting data socket to mover (('131.225.13.190', 34727)) with interface 131.225.189.60. elapsed=1048.040sec 00:00:54 cmsstor60.fnal.gov 005464 root I ENCP Data socket ('131.225.189.60', 36626) is connected to ('131.225.13.190', 34727). elapsed=1048.040sec 00:00:54 cmsstor60.fnal.gov 005464 root I ENCP Opening local file. 00:00:54 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV tape position: target 142 current 142 0 Thread seek_thread 00:00:54 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV write_tape starting, bytes_to_write=2372787200 Thread tape_thread 00:00:54 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV Write starting. Tape VOR467 absolute location in blocks 3355199 Thread tape_thread 00:00:54 cmsstor60.fnal.gov 005464 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:54 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/ACF4ABCC-38C8-DE11-8FD4-003048678FF4.root', drive = 'stkenmvr195a:/dev/rmt/tps0d0n:1310155266', pnfs_id = '000700000000000061D475B8' where bfid = 'CDMS125722805400000'; 00:00:54 cmsstor146.fnal.gov 020630 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: Commissioning09 FF Wrapper: cpio_odc FF Width: 15 Current working directory: cmsstor146.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --age-time 30 --delpri 1 --priority 25 --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/966D61ED-31C8-DE11-8C27-001617E30D40.root --put-cache 000700000000000061D70190 /storage/data2/write-pool-2/data/000700000000000061D70190 00:00:54 cmsstor245.fnal.gov 015098 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /storage/data2/write-pool-2/data/000700000000000061D475B8 -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/ACF4ABCC-38C8-DE11-8FD4-003048678FF4.root: 2755906760 bytes copied to VOR467 at 82.8 MB/S (98.5 MB/S network) (95.6 MB/S drive) (104 MB/S disk) mover=LTO4_95.mover drive_id=ULTRIUM-TD4 drive_sn=1310155266 drive_vendor=IBM elapsed=1097.3 {'storage_group': 'cms', 'encp_crc': 3369524612L, 'disk_rate': 104.32113972202757, 'drive_rate': 95.560691114762975, 'driver': 'FTTDriver', 'transfer_rate': 82.835780511870155, 'mover_interface': 'stkenmvr195a', 'overall_rate': 81.238594431026797, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 82.835780511870155, 'encp_ip': '131.225.191.117', 'unique_id': 'cmsstor245.fnal.gov-1257226958-15098-0'} 00:00:54 cmsstor245.fnal.gov 015098 root I ENCP File /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/ACF4ABCC-38C8-DE11-8FD4-003048678FF4.root copied successfully. 00:00:54 cmsstor245.fnal.gov 015098 root I ENCP Completed transferring 2755906760 bytes in 1 files in 1097.30420804 sec. Overall rate = 81.2 MB/sec. Transfer rate = 82.8 MB/sec. Network rate = 98.5 MB/sec. Drive rate = 95.6 MB/sec. Disk rate = 104 MB/sec. Exit status = 0. 00:00:54 cmsstor146.fnal.gov 020630 root I ENCP Creating zero length output files. 00:00:54 cmsstor146.fnal.gov 020630 root I ENCP Sending /pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/966D61ED-31C8-DE11-8C27-001617E30D40.root write request to LM: unique_id: cmsstor146.fnal.gov-1257228054-20630-0 inputfile: /storage/data2/write-pool-2/data/000700000000000061D70190 outputfile: /pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/.(access)(000700000000000061D70190) 00:00:55 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:55 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data2/write-pool-2/data/000700000000000061D70190 (cmsstor146.fnal.gov-1257228054-20630-0) -> /pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/966D61ED-31C8-DE11-8C27-001617E30D40.root : library=CD-LTO4G1 family=Commissioning09 requester:root volume_family:cms.Commissioning09.cpio_odc 00:00:55 cmsstor146.fnal.gov 020630 root I ENCP Listening for control socket at: ('131.225.189.117', 47731) 00:00:55 cmsstor146.fnal.gov 020630 root I ENCP Waiting for mover to connect control socket at address: ('131.225.189.117', 47731) elapsed=0.930sec 00:00:55 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:55 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 4987 current 0 1 Thread seek_thread 00:00:55 cmsstor238.fnal.gov 007478 root I ENCP File /storage/data1/write-pool-1/data/000700000000000061D40EF8 transfered for cmsstor238.fnal.gov-1257226972-7478-0 in 22.9892778397 seconds. 00:00:55 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:55 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Assigning volume VOQ182 from storage group cms.Commissioning09.cpio_odc to library CD-LTO4G1, volume family cms.Commissioning09.cpio_odc 00:00:55 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm work on vol=VOQ182 mover=LTO4_109.mover requester:root 00:00:55 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:55 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM IDLE:sending {'vc': {'storage_group': 'cms', 'library': 'CD-LTO4G1', 'file_family': 'Commissioning09', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.Commissioning09.cpio_odc', 'file_family_width': 15}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/966D61ED-31C8-DE11-8C27-001617E30D40.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2227468075L, 'mover': 'LTO4_109.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/966D61ED-31C8-DE11-8C27-001617E30D40.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257228054, 'rmajor': 0, 'machine': ('Linux', 'cmsstor146.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16877, 131522040, 21L, 1, 9811, 5063, 512, 1257227735, 1257227735, 1257227606), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 2227468075L, 'fullname': '/storage/data2/write-pool-2/data/000700000000000061D70190', 'inode': 131531152L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/000700000000000061D70190', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/.(access)(000700000000000061D70190)', 'fc': {'size': 2227468075L, 'external_label': 'VOQ182', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data2/write-pool-2/data/000700000000000061D70190', 'at_the_top': 1, 'local_inode': 4134, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.189.117', 47731), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228055.917841, 'encp_start_time': 1257228054.099031, 'job_queued': 1257228055.004138, 't0': 1257228054, 'in_queue': 0.91364693641662598}, 'user_level': 'admin', 'unique_id': 'cmsstor146.fnal.gov-1257228054-20630-0'} to mover 00:00:55 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV WRITE_TO_HSM Thread MainThread 00:00:55 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV SENDING {'vc': {'storage_group': 'cms', 'library': 'CD-LTO4G1', 'file_family': 'Commissioning09', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.Commissioning09.cpio_odc', 'file_family_width': 15}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/966D61ED-31C8-DE11-8C27-001617E30D40.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2227468075L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7869, 'mc_device': '0,5,1,7', 'compression': 0, 'serial_num': '9310025229', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr209a', 'max_time_in_state': 1200, 'device': 'stkenmvr209a:/dev/rmt/tps0d0n', 'name': 'LTO4_109.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_109.stat', 'mover_address': ('131.225.13.205', 7869), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.205', 33312), 'data_ip': 'stkenmvr209a', 'logname': 'LTO4_109MV', 'hostip': '131.225.13.205', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/966D61ED-31C8-DE11-8C27-001617E30D40.root', 'uid': 0, 'uname': 'root', 'mtime': 1257228054, 'rmajor': 0, 'machine': ('Linux', 'cmsstor146.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 131522040, 21L, 1, 9811, 5063, 512, 1257227735, 1257227735, 1257227606), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 2227468075L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data2/write-pool-2/data/000700000000000061D70190', 'inode': 131531152L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/000700000000000061D70190', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/.(access)(000700000000000061D70190)', 'fc': {'size': 2227468075L, 'external_label': 'VOQ182', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data2/write-pool-2/data/000700000000000061D70190', 'at_the_top': 1, 'local_inode': 4134, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.189.117', 47731), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228055.917841, 'encp_start_time': 1257228054.099031, 'job_queued': 1257228055.004138, 't0': 1257228054, 'in_queue': 0.91364693641662598}, 'user_level': 'admin', 'unique_id': 'cmsstor146.fnal.gov-1257228054-20630-0'} Thread client_connect_thread 00:00:55 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM can not remove from sg LTO4_109.mover VOQ182 00:00:55 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM SG: {'cms': [('LTO4_102.mover', 'VOR469'), ('LTO4_94.mover', 'VOQ174'), ('LTO4_106.mover', 'VOQ181'), ('LTO4_49.mover', 'VOR473'), ('LTO4_98.mover', 'VOR465'), ('LTO4_50.mover', 'VOR464'), ('LTO4_42.mover', 'VOQ806'), ('LTO4_48.mover', 'VOR460'), ('LTO4_46.mover', 'VOR470'), ('LTO4_104.mover', 'VOR466'), ('LTO4_40.mover', 'VOR468'), ('LTO4_110.mover', 'VOR463'), ('LTO4_95.mover', 'VOR467')]} 00:00:55 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM can not remove from vf LTO4_109.mover VOQ182 00:00:55 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM VF: {'cms.NonCustodial.cpio_odc': [('LTO4_42.mover', 'VOQ806')], 'cms.cms11.cpio_odc': [('LTO4_102.mover', 'VOR469'), ('LTO4_49.mover', 'VOR473'), ('LTO4_98.mover', 'VOR465'), ('LTO4_50.mover', 'VOR464'), ('LTO4_48.mover', 'VOR460'), ('LTO4_46.mover', 'VOR470'), ('LTO4_104.mover', 'VOR466'), ('LTO4_40.mover', 'VOR468'), ('LTO4_110.mover', 'VOR463'), ('LTO4_95.mover', 'VOR467')], 'cms.Commissioning09.cpio_odc': [('LTO4_94.mover', 'VOQ174'), ('LTO4_106.mover', 'VOQ181')]} 00:00:55 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:55 cmsstor146.fnal.gov 020630 root I ENCP Control socket ('131.225.189.117', 47731) is connected to ('131.225.13.205', 33313) for cmsstor146.fnal.gov-1257228054-20630-0. elapsed=1.850sec 00:00:55 cmsstor146.fnal.gov 020630 root I ENCP Received callback ticket from mover LTO4_109.mover for transfer cmsstor146.fnal.gov-1257228054-20630-0. elapsed=1.860sec 00:00:55 cmsstor146.fnal.gov 020630 root I ENCP Atempting to connect data socket to mover at ('131.225.13.205', 33312). 00:00:55 cmsstor146.fnal.gov 020630 root I ENCP Connecting data socket to mover (('131.225.13.205', 33312)) with interface 131.225.189.117. elapsed=1.860sec 00:00:55 cmsstor146.fnal.gov 020630 root I ENCP Data socket ('131.225.189.117', 39668) is connected to ('131.225.13.205', 33312). elapsed=1.870sec 00:00:55 cmsstor146.fnal.gov 020630 root I ENCP Opening local file. 00:00:56 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV MSG_TYPE=MC_LOAD_REQ mounting VOQ182 ULTRIUM-TD4 00:00:55 Thread media_thread 00:00:56 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC PROCESSING mount VOQ182 0,5,1,7 00:00:56 cmsstor146.fnal.gov 020630 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:56 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:56 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:57 stkensrv4n.fnal.gov 005389 enstore I SL8500F1MC query drive 1,0,1,11 => 0,1,0,1,11 online in use VOO096 IBM-LTO4 00:00:57 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:58 cmsstor224.fnal.gov 006799 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms11 FF Wrapper: cpio_odc FF Width: 10 Current working directory: cmsstor224.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --priority 25 --pnfs-mount /pnfs/fs --put-cache 000700000000000061D2CA00 /storage/data3/write-pool-3/data/000700000000000061D2CA00 00:00:58 cmsstor131.fnal.gov 011680 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms11 FF Wrapper: cpio_odc FF Width: 10 Current working directory: cmsstor131.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --priority 25 --pnfs-mount /pnfs/fs --put-cache 000700000000000061D763F0 /storage/data3/write-pool-3/data/000700000000000061D763F0 00:00:58 cmsstor49.fnal.gov 006338 root I ENCP File /storage/data2/write-pool-2/data/000700000000000061D0EA98 transfered for cmsstor49.fnal.gov-1257226973-6338-0 in 24.8287250996 seconds. 00:00:58 stkensrv4n.fnal.gov 005397 enstore I SL8500G1MC query drive 0,5,1,7 => 0,0,5,1,7 online available IBM-LTO4 00:00:58 stkensrv4n.fnal.gov 005397 enstore I SL8500G1MC Dismount 14 ignored: Drive 0,5,1,7 is empty. Thought VOQ182 was there =>. dismount VOLUME 0,5,1,7 force => ok,0,5,1,7 online available IBM-LTO4 00:00:58 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV written bytes 2707845632/2707845632, blocks 20660 header 192 trailer 436 Thread tape_thread 00:00:58 cmsstor224.fnal.gov 006799 root I ENCP Creating zero length output files. 00:00:58 cmsstor224.fnal.gov 006799 root I ENCP Sending /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/D624BD86-36C8-DE11-B1D7-002618FDA210.root write request to LM: unique_id: cmsstor224.fnal.gov-1257228058-6799-0 inputfile: /storage/data3/write-pool-3/data/000700000000000061D2CA00 outputfile: /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D2CA00) 00:00:58 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:58 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data3/write-pool-3/data/000700000000000061D2CA00 (cmsstor224.fnal.gov-1257228058-6799-0) -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/D624BD86-36C8-DE11-B1D7-002618FDA210.root : library=CD-LTO4G1 family=cms11 requester:root volume_family:cms.cms11.cpio_odc 00:00:58 cmsstor224.fnal.gov 006799 root I ENCP Listening for control socket at: ('131.225.191.96', 38582) 00:00:58 cmsstor224.fnal.gov 006799 root I ENCP Waiting for mover to connect control socket at address: ('131.225.191.96', 38582) elapsed=1.000sec 00:00:58 cmsstor131.fnal.gov 011680 root I ENCP Creating zero length output files. 00:00:58 cmsstor131.fnal.gov 011680 root I ENCP Sending /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/8EB65D2C-3DC8-DE11-B653-003048678DA2.root write request to LM: unique_id: cmsstor131.fnal.gov-1257228058-11680-0 inputfile: /storage/data3/write-pool-3/data/000700000000000061D763F0 outputfile: /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D763F0) 00:00:58 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:58 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data3/write-pool-3/data/000700000000000061D763F0 (cmsstor131.fnal.gov-1257228058-11680-0) -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/8EB65D2C-3DC8-DE11-B653-003048678DA2.root : library=CD-LTO4G1 family=cms11 requester:root volume_family:cms.cms11.cpio_odc 00:00:58 cmsstor131.fnal.gov 011680 root I ENCP Listening for control socket at: ('131.225.189.102', 41007) 00:00:58 cmsstor131.fnal.gov 011680 root I ENCP Waiting for mover to connect control socket at address: ('131.225.189.102', 41007) elapsed=1.060sec 00:00:58 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV filemarks written. Tape VOR464 absolute location in blocks 4557570 Thread tape_thread 00:00:58 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV write_tape timing:pull 0.298470020294 write 7.45738649368 crc_check 10.9641475677 freespace 0.338772773743 block_write 19.0587768555 idle 0.510458946228 Thread tape_thread 00:00:58 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV drive stats after write. Tape VOR464 position 197 block 0 block_size 0 bloc_loc 4557570 tot_blocks 762939000 BOT 0 read_err 0 write_err 1 bytes 2707845632 block_write_tot 19.0587768555 tape_rate 142078668.14 Thread tape_thread 00:00:58 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV remaining bytes info in DB 223783936000 estimated 221076090368 reported from drive 221255680000 position 198 Thread tape_thread 00:00:58 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV new bitfile request {'uid': 0, 'external_label': 'VOR464', 'gid': 0, 'location_cookie': '0000_000000000_0000197', 'complete_crc': 1487663617L, 'sanity_cookie': (65536L, 1833443954L), 'size': 2707845004L} Thread tape_thread 00:00:59 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV set remaining: VOR464 221255680000 0000_000000000_0000198 Thread tape_thread 00:00:59 last message repeated 1 times 00:00:59 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-03 00:00:59', eod_cookie = '0000_000000000_0000198', non_del_files = '199', remaining_bytes = 221255680000 where label = 'VOR464'; 00:00:59 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV transfer complete volume=VOR464 location=198 Thread tape_thread 00:00:59 last message repeated 1 times 00:00:59 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '197', modification_time = '2009-11-03 00:00:59' where label = 'VOR464'; 00:00:59 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1255359024.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOR464', 'wrapper': 'cpio_odc', 'remaining_bytes': 223783936000L, 'sum_mounts': 2, 'sum_rd_access': 0, 'file_family_width': 10, 'last_access': 1257228032.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000197', 'capacity_bytes': 858993459200L, 'non_del_files': 198, 'sum_wr_err': 0, 'sum_wr_access': 196, 'file_family': 'cms11', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'modification_time': 1257228032.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257220197.0}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/4C2FD603-34C8-DE11-9451-00304867918E.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2707845004L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7850, 'mc_device': '0,5,1,2', 'compression': 0, 'serial_num': '9310801229', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr150a', 'max_time_in_state': 1200, 'device': 'stkenmvr150a:/dev/rmt/tps0d0n', 'name': 'LTO4_50.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_50.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.178', 7850), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.178', 34193), 'data_ip': 'stkenmvr150a', 'logname': 'LTO4_50MV', 'hostip': '131.225.13.178', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/4C2FD603-34C8-DE11-9451-00304867918E.root', 'uid': 0, 'uname': 'root', 'mtime': 1257226973, 'rmajor': 0, 'machine': ('Linux', 'cmsstor49.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 130813352, 21L, 1, 9811, 5063, 512, 1257226372, 1257226372, 1257220971), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2707845004L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data2/write-pool-2/data/000700000000000061D0EA98', 'inode': 131132056L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/000700000000000061D0EA98', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/.(access)(000700000000000061D0EA98)', 'fc': {'uid': 0, 'external_label': 'VOR464', 'gid': 0, 'location_cookie': '0000_000000000_0000197', 'complete_crc': 1487663617L, 'bfid': 'CDMS125722805800000', 'sanity_cookie': (65536L, 1833443954L), 'size': 2707845004L}, 'infilepath': '/storage/data2/write-pool-2/data/000700000000000061D0EA98', 'at_the_top': 8, 'local_inode': 1073743311, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.204.225', 38381), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257226972.471864, 'seek_time': 0.0343170166015625, 'transfer_time': 25.692440986633301, 't0': 1257226972, 'mount_time': 0.04143214225769043, 'lm_dequeued': 1257228033.2479081, 'job_queued': 1257226973.7516301, 'drive_call_time': 7.156538724899292, 'in_queue': 1059.495992898941, 'drive_transfer_time': 20.19468355178833}, 'user_level': 'admin', 'unique_id': 'cmsstor49.fna(trunc. 4105) Thread tape_thread 00:00:59 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV Done is sent Thread tape_thread 00:00:59 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV maybe_clean Thread tape_thread 00:00:59 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV returned from maybe_clean Thread tape_thread 00:00:59 cmsstor49.fnal.gov 006338 root I ENCP Received final dialog for cmsstor49.fnal.gov-1257226973-6338-0. elapsed=1086.740sec 00:00:59 cmsstor49.fnal.gov 006338 root I ENCP leaving transfer_file(). 00:00:59 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:59 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VOR464 mover=LTO4_50.mover requester:root 00:00:59 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/6E24B99E-3AC8-DE11-9645-0030486792F0.root to mover LTO4_50.mover ('131.225.13.178', 7850) DEL_DISM None 00:00:59 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:59 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'cms', 'external_label': 'VOR464', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/6E24B99E-3AC8-DE11-9645-0030486792F0.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 3971741532L, 'mover': 'LTO4_50.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/6E24B99E-3AC8-DE11-9645-0030486792F0.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257227020, 'rmajor': 0, 'machine': ('Linux', 'cmsstor42.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16893, 131219952, 21L, 1, 9811, 5063, 512, 1257227015, 1257227015, 1257224768), 'mode_octal': '0100036', 'mode': 33188, 'gname': 'root', 'size_bytes': 3971741532L, 'fullname': '/storage/data2/write-pool-2/data/000700000000000061D58CC0', 'inode': 131435712L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/000700000000000061D58CC0', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D58CC0)', 'fc': {'size': 3971741532L, 'external_label': 'VOR464', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data2/write-pool-2/data/000700000000000061D58CC0', 'at_the_top': 49, 'local_inode': 5850, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.188.6', 46016), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228059.2684219, 'encp_start_time': 1257227018.7842219, 'job_queued': 1257227020.2016411, 't0': 1257227018, 'in_queue': 1039.0665099620819}, 'user_level': 'admin', 'unique_id': 'cmsstor42.fnal.gov-1257227020-348-0'} to mover 00:00:59 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:59 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV WRITE_TO_HSM Thread MainThread 00:00:59 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV SENDING {'vc': {'storage_group': 'cms', 'external_label': 'VOR464', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/6E24B99E-3AC8-DE11-9645-0030486792F0.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 3971741532L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7850, 'mc_device': '0,5,1,2', 'compression': 0, 'serial_num': '9310801229', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr150a', 'max_time_in_state': 1200, 'device': 'stkenmvr150a:/dev/rmt/tps0d0n', 'name': 'LTO4_50.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_50.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.178', 7850), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.178', 34196), 'data_ip': 'stkenmvr150a', 'logname': 'LTO4_50MV', 'hostip': '131.225.13.178', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/6E24B99E-3AC8-DE11-9645-0030486792F0.root', 'uid': 0, 'uname': 'root', 'mtime': 1257227020, 'rmajor': 0, 'machine': ('Linux', 'cmsstor42.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 131219952, 21L, 1, 9811, 5063, 512, 1257227015, 1257227015, 1257224768), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 3971741532L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data2/write-pool-2/data/000700000000000061D58CC0', 'inode': 131435712L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/000700000000000061D58CC0', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D58CC0)', 'fc': {'size': 3971741532L, 'external_label': 'VOR464', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data2/write-pool-2/data/000700000000000061D58CC0', 'at_the_top': 49, 'local_inode': 5850, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.188.6', 46016), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228059.2684219, 'encp_start_time': 1257227018.7842219, 'job_queued': 1257227020.2016411, 't0': 1257227018, 'in_queue': 1039.0665099620819}, 'user_level': 'admin', 'unique_id': 'cmsstor42.fnal.gov-1257227020-348-0'} Thread client_connect_thread 00:00:59 cmsstor42.fnal.gov 000348 root I ENCP Control socket ('131.225.188.6', 46016) is connected to ('131.225.13.178', 34197) for cmsstor42.fnal.gov-1257227020-348-0. elapsed=1040.610sec 00:00:59 cmsstor42.fnal.gov 000348 root I ENCP Received callback ticket from mover LTO4_50.mover for transfer cmsstor42.fnal.gov-1257227020-348-0. elapsed=1040.620sec 00:00:59 cmsstor42.fnal.gov 000348 root I ENCP Atempting to connect data socket to mover at ('131.225.13.178', 34196). 00:00:59 cmsstor42.fnal.gov 000348 root I ENCP Connecting data socket to mover (('131.225.13.178', 34196)) with interface 131.225.188.6. elapsed=1040.620sec 00:00:59 cmsstor42.fnal.gov 000348 root I ENCP Data socket ('131.225.188.6', 35529) is connected to ('131.225.13.178', 34196). elapsed=1040.620sec 00:00:59 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV written bytes 2253800448/2253800448, blocks 17196 header 192 trailer 157 Thread tape_thread 00:00:59 cmsstor42.fnal.gov 000348 root I ENCP Opening local file. 00:00:59 cmsstor96.fnal.gov 022537 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Scientific Linux SL release 5.2 (Boron) Rocks release 5.1 (V.I) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms11 FF Wrapper: cpio_odc FF Width: 10 Current working directory: cmsstor96.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --priority 25 --pnfs-mount /pnfs/fs --put-cache 000700000000000061D76B08 /storage/data1/write-pool-1/data/000700000000000061D76B08 00:00:59 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV tape position: target 198 current 198 0 Thread seek_thread 00:00:59 cmsstor42.fnal.gov 000348 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:59 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV write_tape starting, bytes_to_write=3971742208 Thread tape_thread 00:00:59 stkenmvr150a.fnal.gov 006006 root I LTO4_50MV Write starting. Tape VOR464 absolute location in blocks 4557570 Thread tape_thread 00:00:59 cmsstor96.fnal.gov 022537 root I ENCP Creating zero length output files. 00:00:59 cmsstor96.fnal.gov 022537 root I ENCP Sending /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/1860E64C-3DC8-DE11-9E5F-00304867924E.root write request to LM: unique_id: cmsstor96.fnal.gov-1257228059-22537-0 inputfile: /storage/data1/write-pool-1/data/000700000000000061D76B08 outputfile: /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D76B08) 00:00:59 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:59 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data1/write-pool-1/data/000700000000000061D76B08 (cmsstor96.fnal.gov-1257228059-22537-0) -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/1860E64C-3DC8-DE11-9E5F-00304867924E.root : library=CD-LTO4G1 family=cms11 requester:root volume_family:cms.cms11.cpio_odc 00:00:59 cmsstor96.fnal.gov 022537 root I ENCP Listening for control socket at: ('131.225.189.96', 38524) 00:00:59 cmsstor96.fnal.gov 022537 root I ENCP Waiting for mover to connect control socket at address: ('131.225.189.96', 38524) elapsed=1.260sec 00:00:59 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:59 stkenmvr115a.fnal.gov 005896 root I LTO15MV read bytes 881593858/881593858, blocks 6727 header 215 Thread tape_thread 00:00:59 stkenmvr115a.fnal.gov 005896 root I LTO15MV write_client: wrote 881593643/881593643 bytes Thread net_thread 00:00:59 stkenmvr115a.fnal.gov 005896 root I LTO15MV write_client waits for tape thread to exit Thread net_thread 00:00:59 stkensrv3n.fnal.gov 031426 root I ENCP File /pnfs/cms/WAX/11/store/mc/2008/4/7/CSA08-Cosmic_Underground_BOFF_corr_1BX-4228/0008/E6FD5CB8-A405-DD11-916A-001C23C0DFDD.root transfered for stkensrv3n.fnal.gov-1257227979-31426-0 in 11.0136010647 seconds. 00:00:59 cmsstor49.fnal.gov 006338 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:59 stkenmvr115a.fnal.gov 005896 root I LTO15MV drive stats after read. Tape VOI827 position 1 block 6727 block_size 0 bloc_loc 6729 tot_blocks 381469000 BOT 0 read_err 0 write_err 0 bytes 881593858 block_read_tot 10.2358012199 tape_rate 86128466.0631 Thread tape_thread 00:00:59 stkenmvr115a.fnal.gov 005896 root I LTO15MV read_tape exiting, read 881593858/881593858 bytes Thread tape_thread 00:00:59 stkenmvr115a.fnal.gov 005896 root I LTO15MV read_tape timing:get_space 0.0928585529327 read 6.54926276207 crc_check 3.49076032639 push 0.102919578552 block_read 10.2358012199 idle 0.265989542007 Thread tape_thread 00:01:00 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:00 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/4C2FD603-34C8-DE11-9451-00304867918E.root', drive = 'stkenmvr150a:/dev/rmt/tps0d0n:9310801229', pnfs_id = '000700000000000061D0EA98' where bfid = 'CDMS125722805800000'; 00:01:00 cmsstor49.fnal.gov 006338 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /storage/data2/write-pool-2/data/000700000000000061D0EA98 -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/4C2FD603-34C8-DE11-9451-00304867918E.root: 2707845004 bytes copied to VOR464 at 101 MB/S (87.9 MB/S network) (128 MB/S drive) (104 MB/S disk) mover=LTO4_50.mover drive_id=ULTRIUM-TD4 drive_sn=9310801229 drive_vendor=IBM elapsed=1088 {'storage_group': 'cms', 'encp_crc': 1487663617L, 'disk_rate': 104.16048818830156, 'drive_rate': 127.87535028718467, 'driver': 'FTTDriver', 'transfer_rate': 100.53930950472858, 'mover_interface': 'stkenmvr150a', 'overall_rate': 95.036543133556293, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 100.53930950472858, 'encp_ip': '131.225.204.225', 'unique_id': 'cmsstor49.fnal.gov-1257226973-6338-0'} 00:01:00 cmsstor49.fnal.gov 006338 root I ENCP File /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/4C2FD603-34C8-DE11-9451-00304867918E.root copied successfully. 00:01:00 cmsstor49.fnal.gov 006338 root I ENCP Completed transferring 2707845004 bytes in 1 files in 1088.04102612 sec. Overall rate = 95 MB/sec. Transfer rate = 101 MB/sec. Network rate = 87.9 MB/sec. Drive rate = 128 MB/sec. Disk rate = 104 MB/sec. Exit status = 0. 00:01:00 stkensrv4n.fnal.gov 005397 enstore I SL8500G1MC query vol VOQ182 => 0,VOQ182 home 0, 4, 3,19, 0 LTO-800G 00:01:00 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV filemarks written. Tape VOR465 absolute location in blocks 3892628 Thread tape_thread 00:01:00 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV write_tape timing:pull 0.193810224533 write 9.90328788757 crc_check 10.3109126091 freespace 0.188539743423 block_write 20.5965504646 idle 0.321922779083 Thread tape_thread 00:01:00 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV drive stats after write. Tape VOR465 position 152 block 0 block_size 0 bloc_loc 3892628 tot_blocks 762939000 BOT 0 read_err 0 write_err 5 bytes 2253800448 block_write_tot 20.5965504646 tape_rate 109426112.488 Thread tape_thread 00:01:00 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV remaining bytes info in DB 303965184000 estimated 301711383552 reported from drive 301863936000 position 153 Thread tape_thread 00:01:00 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV new bitfile request {'uid': 0, 'external_label': 'VOR465', 'gid': 0, 'location_cookie': '0000_000000000_0000152', 'complete_crc': 1164033056L, 'sanity_cookie': (65536L, 1649089936L), 'size': 2253800099L} Thread tape_thread 00:01:00 stkenmvr115a.fnal.gov 005896 root I LTO15MV write_client exits Thread net_thread 00:01:00 stkenmvr115a.fnal.gov 005896 root I LTO15MV transfer complete volume=VOI827 location=1 Thread net_thread 00:01:00 last message repeated 1 times 00:01:00 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV set remaining: VOR465 301863936000 0000_000000000_0000153 Thread tape_thread 00:01:00 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_rd_access = '3', last_access = '2009-11-03 00:01:00' where label = 'VOI827'; 00:01:00 last message repeated 1 times 00:01:00 stkenmvr115a.fnal.gov 005896 root I LTO15MV Sending done to client: {'vc': {'comment': '', 'declared': 1202427005.0, 'blocksize': 131072, 'storage_group': 'cms', 'library': 'CD-LTO3', 'eod_cookie': '0000_000000000_0000360', 'current_location': '0000_000000000_0000001', 'user_inhibit': ['none', 'none'], 'sum_rd_access': 2, 'system_inhibit': ['none', 'full'], 'external_label': 'VOI827', 'wrapper': 'cpio_odc', 'remaining_bytes': 1371494912L, 'sum_mounts': 35, 'capacity_bytes': 429496729600L, 'media_type': 'LTO3', 'last_access': 1257227977.0, 'status': ('ok', None), 'si_time': [0.0, 1209569077.0], 'non_del_files': 359, 'sum_wr_err': 0, 'sum_wr_access': 359, 'file_family': 'cms11', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'modification_time': 1253277095.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1209514668.0}, 'outfilepath': '/dev/null', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': 1, 'curpri': 1}, 'file_size': 881593643L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 300, 'max_failures': 5, 'library': 'CD-LTO3.library_manager', 'local_mover': 0, 'dismount_delay': 20, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7815, 'mc_device': '0,1,1,13', 'compression': 0, 'serial_num': '9210806209', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 57671680.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr115a', 'max_time_in_state': 1200, 'device': 'stkenmvr115a:/dev/rmt/tps2d0n', 'name': 'LTO3_15.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD3', 'statistics_path': '/tmp/enstore/enstore/LTO3_15.stat', 'mover_address': ('131.225.13.145', 7815), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.145', 36780), 'data_ip': 'stkenmvr115a', 'logname': 'LTO15MV', 'hostip': '131.225.13.145', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'lm': {'address': ('131.225.13.4', 7506)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/cms/WAX/11/store/mc/2008/4/7/CSA08-Cosmic_Underground_BOFF_corr_1BX-4228/0008/E6FD5CB8-A405-DD11-916A-001C23C0DFDD.root', 'uid': 0, 'uname': 'root', 'rmajor': 0, 'machine': ('Linux', 'stkensrv3n.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 08:50:41 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (33188, 133235560L, 20L, 1, 9811, 5063, 881593643L, 1209514707, 1209514707, 1209513323), 'mode_octal': '0100744', 'mode': 33252, 'size_bytes': 881593643L, 'gname': 'root', 'gid': 0, 'fullname': '/dev/null', 'inode': 133235560L, 'minor': 20}, 'version': 'v3_7d CVS $Revision: 1.929 $ encp', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/cms/WAX/11/store/mc/2008/4/7/CSA08-Cosmic_Underground_BOFF_corr_1BX-4228/0008/.(access)(000700000000000015F10368)', 'status': ('ok', None), 'volume': 'VOI827', 'outfile': '/dev/null', 'fc': {'status': ('ok', None), 'pnfsid': '000700000000000015F10368', 'pnfs_name0': '/pnfs/fnal.gov/usr/cms/WAX//11/store/mc/2008/4/7/CSA08-Cosmic_Underground_BOFF_corr_1BX-4228/0008/E6FD5CB8-A405-DD11-916A-001C23C0DFDD.root', 'deleted': 'no', 'external_label': 'VOI827', 'drive': 'stkenmvr107a:/dev/rmt/tps2d0n:1210116552', 'update': '2008-04-29 19:18:27.862166', 'gid': 0, 'location_cookie': '0000_000000000_0000001', 'complete_crc': 1795180056L, 'address': ('131.225.13.59', 7501), 'uid': 0, 'bfid': 'CDMS120951470700001', 'sanity_cookie': (65536L, 3859470780L), 'size': 881593643L}, 'infilepath': '/pnfs/cms/WAX/11/store/mc/2008/4/7/CSA08-Cosmic_Underground_BOFF_corr_1BX-4228/0008/E6FD5CB8-A405-DD11-916A-001C23C0DFDD.root', 'at_the_top': 8, 'bfid': 'CDMS120951470700001', 'local_inode': 2469L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': 7, 'retry': 0}, 'callback_addr': ('131.225.13.51', 57651), 'work': 'read_from_hsm', 'override_ro_mount': 1, 'times': {'encp_start_time': 1257227977.984293, 'seek_time': 67.940690994262695, 'transfer_time': 12.080810070037842, 't0': 1257227977, 'mount_time': 0.019160985946655273, 'lm_dequeued': 12572279(trunc. 4317) Thread net_thread 00:01:00 stkenmvr115a.fnal.gov 005896 root I LTO15MV Done is sent Thread net_thread 00:01:00 stkenmvr115a.fnal.gov 005896 root I LTO15MV maybe_clean Thread net_thread 00:01:00 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-03 00:01:00', eod_cookie = '0000_000000000_0000153', non_del_files = '153', remaining_bytes = 301863936000 where label = 'VOR465'; 00:01:00 stkensrv3n.fnal.gov 031426 root I ENCP Received final dialog for stkensrv3n.fnal.gov-1257227979-31426-0. elapsed=82.960sec 00:01:00 stkensrv3n.fnal.gov 031426 root I ENCP leaving transfer_file(). 00:01:00 stkenmvr115a.fnal.gov 005896 root I LTO15MV returned from maybe_clean Thread net_thread 00:01:00 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV transfer complete volume=VOR465 location=153 Thread tape_thread 00:01:00 stkensrv3n.fnal.gov 031426 root I ENCP MSG_TYPE=ENCP_XFER read_from_hsm /pnfs/cms/WAX/11/store/mc/2008/4/7/CSA08-Cosmic_Underground_BOFF_corr_1BX-4228/0008/E6FD5CB8-A405-DD11-916A-001C23C0DFDD.root -> /dev/null: 881593643 bytes copied from VOI827 at 69.7 MB/S (76.3 MB/S network) (81.2 MB/S drive) (76.3 MB/S disk) mover=LTO3_15.mover drive_id=ULTRIUM-TD3 drive_sn=9210806209 drive_vendor=IBM elapsed=83.001 {'storage_group': 'cms', 'encp_crc': 1795180056L, 'disk_rate': 76.338796176332394, 'drive_rate': 81.232133667997516, 'driver': 'FTTDriver', 'transfer_rate': 69.699361267079638, 'mover_interface': 'stkenmvr115a', 'overall_rate': 10.493327333588924, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 69.699361267079638, 'encp_ip': '131.225.13.51', 'unique_id': 'stkensrv3n.fnal.gov-1257227979-31426-0'} 00:01:01 stkensrv3n.fnal.gov 031426 root I ENCP File /dev/null copied successfully. 00:01:01 stkensrv3n.fnal.gov 031426 root I ENCP Completed transferring 881593643 bytes in 1 files in 83.0076289177 sec. Overall rate = 10.5 MB/sec. Transfer rate = 69.7 MB/sec. Network rate = 76.3 MB/sec. Drive rate = 81.2 MB/sec. Disk rate = 76.3 MB/sec. Exit status = 0. 00:01:01 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '152', modification_time = '2009-11-03 00:01:00' where label = 'VOR465'; 00:01:01 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1255359028.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOR465', 'wrapper': 'cpio_odc', 'remaining_bytes': 303965184000L, 'sum_mounts': 2, 'sum_rd_access': 0, 'file_family_width': 10, 'last_access': 1257228032.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000152', 'capacity_bytes': 858993459200L, 'non_del_files': 152, 'sum_wr_err': 0, 'sum_wr_access': 151, 'file_family': 'cms11', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'modification_time': 1257228032.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257221677.0}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/4C524A4A-38C8-DE11-8B2A-001A92971B20.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2253800099L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7859, 'mc_device': '0,7,1,1', 'compression': 0, 'serial_num': '9310069991', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr198a', 'max_time_in_state': 1200, 'device': 'stkenmvr198a:/dev/rmt/tps0d0n', 'name': 'LTO4_98.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_98.stat', 'mover_address': ('131.225.13.193', 7859), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.193', 39724), 'data_ip': 'stkenmvr198a', 'logname': 'LTO4_98MV', 'hostip': '131.225.13.193', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/4C524A4A-38C8-DE11-8B2A-001A92971B20.root', 'uid': 0, 'uname': 'root', 'mtime': 1257226972, 'rmajor': 0, 'machine': ('Linux', 'cmsstor238.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 131219952, 21L, 1, 9811, 5063, 512, 1257226968, 1257226968, 1257224768), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2253800099L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data1/write-pool-1/data/000700000000000061D40EF8', 'inode': 131337976L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/000700000000000061D40EF8', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D40EF8)', 'fc': {'uid': 0, 'external_label': 'VOR465', 'gid': 0, 'location_cookie': '0000_000000000_0000152', 'complete_crc': 1164033056L, 'bfid': 'CDMS125722806000000', 'sanity_cookie': (65536L, 1649089936L), 'size': 2253800099L}, 'infilepath': '/storage/data1/write-pool-1/data/000700000000000061D40EF8', 'at_the_top': 2, 'local_inode': 5472, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.191.110', 37756), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257226967.1891639, 'seek_time': 0.029913902282714844, 'transfer_time': 28.254045963287354, 't0': 1257226967, 'mount_time': 0.022926092147827148, 'lm_dequeued': 1257228032.6350739, 'job_queued': 1257226972.9696579, 'drive_call_time': 9.6679770946502686, 'in_queue': 1059.6651310920715, 'drive_transfer_time': 22.689756393432617}, 'user_level': 'admin', 'unique_id': 'cmsstor238.fnal.gov-1257226972-7478-0'} Thread tape_thread 00:01:01 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV Done is sent Thread tape_thread 00:01:01 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV maybe_clean Thread tape_thread 00:01:01 cmsstor238.fnal.gov 007478 root I ENCP Received final dialog for cmsstor238.fnal.gov-1257226972-7478-0. elapsed=1093.840sec 00:01:01 cmsstor238.fnal.gov 007478 root I ENCP leaving transfer_file(). 00:01:01 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV returned from maybe_clean Thread tape_thread 00:01:01 cmsstor238.fnal.gov 007478 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:01:01 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/4C524A4A-38C8-DE11-8B2A-001A92971B20.root', drive = 'stkenmvr198a:/dev/rmt/tps0d0n:9310069991', pnfs_id = '000700000000000061D40EF8' where bfid = 'CDMS125722806000000'; 00:01:01 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:01 last message repeated 1 times 00:01:01 stkensrv3n.fnal.gov 032007 root I ENCP Version: v3_7d CVS $Revision: 1.929 $ encp OS: Linux 2.6.9-89.0.9.ELsmp x86_64 Release: Scientific Linux SL release 4.4 (Beryllium), User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Current working directory: stkensrv3n.fnal.gov:/root Command line: /opt/enstore/bin/encp --verbose 10 --bypass-filesystem-max-filesize-check --max-resubmit 7 --delayed-dismount 1 --override-ro-mount --get-bfid CDMS120956907700000 /dev/null 00:01:01 cmsstor238.fnal.gov 007478 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /storage/data1/write-pool-1/data/000700000000000061D40EF8 -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/4C524A4A-38C8-DE11-8B2A-001A92971B20.root: 2253800099 bytes copied to VOR465 at 76.1 MB/S (98.9 MB/S network) (94.7 MB/S drive) (93.6 MB/S disk) mover=LTO4_98.mover drive_id=ULTRIUM-TD4 drive_sn=9310069991 drive_vendor=IBM elapsed=1094.5 {'storage_group': 'cms', 'encp_crc': 1164033056L, 'disk_rate': 93.555649227890399, 'drive_rate': 94.729587724982764, 'driver': 'FTTDriver', 'transfer_rate': 76.060116478480211, 'mover_interface': 'stkenmvr198a', 'overall_rate': 74.19226354099483, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 76.060116478480211, 'encp_ip': '131.225.191.110', 'unique_id': 'cmsstor238.fnal.gov-1257226972-7478-0'} 00:01:01 cmsstor238.fnal.gov 007478 root I ENCP File /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/4C524A4A-38C8-DE11-8B2A-001A92971B20.root copied successfully. 00:01:01 cmsstor238.fnal.gov 007478 root I ENCP Completed transferring 2253800099 bytes in 1 files in 1094.46519399 sec. Overall rate = 74.2 MB/sec. Transfer rate = 76.1 MB/sec. Network rate = 98.9 MB/sec. Drive rate = 94.7 MB/sec. Disk rate = 93.6 MB/sec. Exit status = 0. 00:01:01 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VOR465 mover=LTO4_98.mover requester:root 00:01:01 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/5EA57FFA-33C8-DE11-8363-00304867BFB0.root to mover LTO4_98.mover ('131.225.13.193', 7859) DEL_DISM None 00:01:01 last message repeated 1 times 00:01:01 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:01:01 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'cms', 'external_label': 'VOR465', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/5EA57FFA-33C8-DE11-8363-00304867BFB0.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2263516708L, 'mover': 'LTO4_98.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/5EA57FFA-33C8-DE11-8363-00304867BFB0.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257227021, 'rmajor': 0, 'machine': ('Linux', 'cmsstor101.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16893, 130813352, 21L, 1, 9811, 5063, 512, 1257226372, 1257226372, 1257220971), 'mode_octal': '0100036', 'mode': 33188, 'gname': 'root', 'size_bytes': 2263516708L, 'fullname': '/storage/data2/write-pool-2/data/000700000000000061D0D638', 'inode': 131126840L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/000700000000000061D0D638', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/.(access)(000700000000000061D0D638)', 'fc': {'size': 2263516708L, 'external_label': 'VOR465', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data2/write-pool-2/data/000700000000000061D0D638', 'at_the_top': 38, 'local_inode': 1073742838, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.205.223', 36067), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228061.689873, 'encp_start_time': 1257227020.0774419, 'job_queued': 1257227021.603734, 't0': 1257227020, 'in_queue': 1040.0858619213104}, 'user_level': 'admin', 'unique_id': 'cmsstor101.fnal.gov-1257227021-20849-0'} to mover 00:01:01 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:01 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV WRITE_TO_HSM Thread MainThread 00:01:01 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV SENDING {'vc': {'storage_group': 'cms', 'external_label': 'VOR465', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/5EA57FFA-33C8-DE11-8363-00304867BFB0.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2263516708L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7859, 'mc_device': '0,7,1,1', 'compression': 0, 'serial_num': '9310069991', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr198a', 'max_time_in_state': 1200, 'device': 'stkenmvr198a:/dev/rmt/tps0d0n', 'name': 'LTO4_98.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_98.stat', 'mover_address': ('131.225.13.193', 7859), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.193', 39726), 'data_ip': 'stkenmvr198a', 'logname': 'LTO4_98MV', 'hostip': '131.225.13.193', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/5EA57FFA-33C8-DE11-8363-00304867BFB0.root', 'uid': 0, 'uname': 'root', 'mtime': 1257227021, 'rmajor': 0, 'machine': ('Linux', 'cmsstor101.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 130813352, 21L, 1, 9811, 5063, 512, 1257226372, 1257226372, 1257220971), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2263516708L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data2/write-pool-2/data/000700000000000061D0D638', 'inode': 131126840L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/000700000000000061D0D638', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/.(access)(000700000000000061D0D638)', 'fc': {'size': 2263516708L, 'external_label': 'VOR465', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data2/write-pool-2/data/000700000000000061D0D638', 'at_the_top': 38, 'local_inode': 1073742838, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.205.223', 36067), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228061.689873, 'encp_start_time': 1257227020.0774419, 'job_queued': 1257227021.603734, 't0': 1257227020, 'in_queue': 1040.0858619213104}, 'user_level': 'admin', 'unique_id': 'cmsstor101.fnal.gov-1257227021-20849-0'} Thread client_connect_thread 00:01:01 cmsstor101.fnal.gov 020849 root I ENCP Control socket ('131.225.205.223', 36067) is connected to ('131.225.13.193', 39727) for cmsstor101.fnal.gov-1257227021-20849-0. elapsed=1041.800sec 00:01:01 cmsstor101.fnal.gov 020849 root I ENCP Received callback ticket from mover LTO4_98.mover for transfer cmsstor101.fnal.gov-1257227021-20849-0. elapsed=1041.820sec 00:01:01 cmsstor101.fnal.gov 020849 root I ENCP Atempting to connect data socket to mover at ('131.225.13.193', 39726). 00:01:01 cmsstor101.fnal.gov 020849 root I ENCP Connecting data socket to mover (('131.225.13.193', 39726)) with interface 131.225.205.223. elapsed=1041.820sec 00:01:01 cmsstor101.fnal.gov 020849 root I ENCP Data socket ('131.225.205.223', 39217) is connected to ('131.225.13.193', 39726). elapsed=1041.820sec 00:01:01 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:01 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV tape position: target 153 current 153 0 Thread seek_thread 00:01:01 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV write_tape starting, bytes_to_write=2263517184 Thread tape_thread 00:01:01 cmsstor101.fnal.gov 020849 root I ENCP Opening local file. 00:01:01 stkenmvr198a.fnal.gov 015662 root I LTO4_98MV Write starting. Tape VOR465 absolute location in blocks 3892628 Thread tape_thread 00:01:01 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:01:01 cmsstor101.fnal.gov 020849 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:01:02 stkensrv3n.fnal.gov 032007 root I ENCP Creating zero length output files. 00:01:02 stkensrv3n.fnal.gov 032007 root I ENCP Sending /pnfs/cms/WAX/11/store/data/2008/4/30/T1ReReco-B2-GlobalMar08-A-Pass4/0000/7881368C-C616-DD11-866E-001731AF685D.root read request to LM: unique_id: stkensrv3n.fnal.gov-1257228062-32007-0 inputfile: /pnfs/cms/WAX/11/store/data/2008/4/30/T1ReReco-B2-GlobalMar08-A-Pass4/0000/.(access)(000700000000000015FA1C50) outputfile: /dev/null 00:01:02 stkensrv4n.fnal.gov 005806 enstore I LTO3LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:01:02 stkensrv4n.fnal.gov 005806 enstore I LTO3LM read Q'd /pnfs/cms/WAX/11/store/data/2008/4/30/T1ReReco-B2-GlobalMar08-A-Pass4/0000/7881368C-C616-DD11-866E-001731AF685D.root (stkensrv3n.fnal.gov-1257228062-32007-0) -> /dev/null : library=CD-LTO3 family=cms.cms11.cpio_odc requester:root 00:01:02 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:03 stkensrv4n.fnal.gov 005397 enstore I SL8500G1MC query drive 0,5,1,7 => 0,0,5,1,7 online available IBM-LTO4 00:01:03 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:03 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:01:03 stkensrv4n.fnal.gov 005806 enstore I LTO3LM read_from_hsm next work on vol=VOI827 mover=LTO3_15.mover requester:root 00:01:03 stkensrv4n.fnal.gov 005806 enstore I LTO3LM HAVE_BOUND:sending read_from_hsm /pnfs/cms/WAX/11/store/data/2008/4/30/T1ReReco-B2-GlobalMar08-A-Pass4/0000/7881368C-C616-DD11-866E-001731AF685D.root to mover LTO3_15.mover ('131.225.13.145', 7815) DEL_DISM 1 00:01:03 stkensrv4n.fnal.gov 005806 enstore I LTO3LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:01:03 stkensrv4n.fnal.gov 005806 enstore I LTO3LM HAVE_BOUND:sending {'vc': {'comment': '', 'declared': 1202427005.0, 'blocksize': 131072, 'storage_group': 'cms', 'library': 'CD-LTO3', 'eod_cookie': '0000_000000000_0000360', 'current_location': '0000_000000000_0000359', 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'full'], 'external_label': 'VOI827', 'wrapper': 'cpio_odc', 'remaining_bytes': 1371494912L, 'sum_mounts': 35, 'sum_rd_access': 3, 'media_type': 'LTO3', 'last_access': 1257228060.0, 'status': ('ok', None), 'si_time': [0.0, 1209569077.0], 'capacity_bytes': 429496729600L, 'non_del_files': 359, 'sum_wr_err': 0, 'sum_wr_access': 359, 'file_family': 'cms11', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'modification_time': 1253277095.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1209514668.0}, 'outfilepath': '/dev/null', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': 1, 'curpri': 1}, 'file_size': 1666637142L, 'mover': 'LTO3_15.mover', 'lm': {'address': ('131.225.13.4', 7506)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/cms/WAX/11/store/data/2008/4/30/T1ReReco-B2-GlobalMar08-A-Pass4/0000/7881368C-C616-DD11-866E-001731AF685D.root', 'uid': 0, 'uname': 'root', 'gid': 0, 'rmajor': 0, 'machine': ('Linux', 'stkensrv3n.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 08:50:41 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (33188, 133831760L, 20L, 1, 9811, 5063, 1666637142L, 1209569077, 1209569077, 1209567917), 'mode_octal': '0100744', 'mode': 33252, 'gname': 'root', 'size_bytes': 1666637142L, 'fullname': '/dev/null', 'inode': 133831760L, 'minor': 20}, 'version': 'v3_7d CVS $Revision: 1.929 $ encp', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/cms/WAX/11/store/data/2008/4/30/T1ReReco-B2-GlobalMar08-A-Pass4/0000/.(access)(000700000000000015FA1C50)', 'status': ('ok', None), 'volume': 'VOI827', 'outfile': '/dev/null', 'fc': {'status': ('ok', None), 'pnfsid': '000700000000000015FA1C50', 'pnfs_name0': '/pnfs/fnal.gov/usr/cms/WAX/11/store/data/2008/4/30/T1ReReco-B2-GlobalMar08-A-Pass4/0000/7881368C-C616-DD11-866E-001731AF685D.root', 'deleted': 'no', 'external_label': 'VOI827', 'drive': 'stkenmvr106a:/dev/rmt/tps2d0n:1210116666', 'update': '2008-04-30 10:24:37.439132', 'gid': 0, 'location_cookie': '0000_000000000_0000359', 'complete_crc': 2472083105L, 'address': ('131.225.13.59', 7501), 'uid': 0, 'bfid': 'CDMS120956907700000', 'sanity_cookie': (65536L, 3436853835L), 'size': 1666637142L}, 'infilepath': '/pnfs/cms/WAX/11/store/data/2008/4/30/T1ReReco-B2-GlobalMar08-A-Pass4/0000/7881368C-C616-DD11-866E-001731AF685D.root', 'at_the_top': 20, 'bfid': 'CDMS120956907700000', 'local_inode': 2469L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': 7, 'retry': 0}, 'callback_addr': ('131.225.13.51', 57660), 'work': 'read_from_hsm', 'override_ro_mount': 1, 'times': {'lm_dequeued': 1257228063.928705, 'encp_start_time': 1257228061.1645861, 'job_queued': 1257228062.3536999, 't0': 1257228061, 'in_queue': 1.5749599933624268}, 'user_level': 'admin', 'unique_id': 'stkensrv3n.fnal.gov-1257228062-32007-0'} to mover 00:01:03 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:01:03 stkenmvr115a.fnal.gov 005896 root I LTO15MV READ FROM HSM Thread MainThread 00:01:03 stkensrv3n.fnal.gov 032007 root I ENCP Listening for control socket at: ('131.225.13.51', 57660) 00:01:03 stkensrv3n.fnal.gov 032007 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.51', 57660) elapsed=2.800sec 00:01:04 stkenmvr115a.fnal.gov 005896 root I LTO15MV SENDING {'vc': {'comment': '', 'declared': 1202427005.0, 'blocksize': 131072, 'storage_group': 'cms', 'library': 'CD-LTO3', 'eod_cookie': '0000_000000000_0000360', 'current_location': '0000_000000000_0000359', 'user_inhibit': ['none', 'none'], 'sum_rd_access': 3, 'system_inhibit': ['none', 'full'], 'external_label': 'VOI827', 'wrapper': 'cpio_odc', 'remaining_bytes': 1371494912L, 'sum_mounts': 35, 'capacity_bytes': 429496729600L, 'media_type': 'LTO3', 'last_access': 1257228060.0, 'status': ('ok', None), 'si_time': [0.0, 1209569077.0], 'non_del_files': 359, 'sum_wr_err': 0, 'sum_wr_access': 359, 'file_family': 'cms11', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'modification_time': 1253277095.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1209514668.0}, 'outfilepath': '/dev/null', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': 1, 'curpri': 1}, 'file_size': 1666637142L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 300, 'max_failures': 5, 'library': 'CD-LTO3.library_manager', 'local_mover': 0, 'dismount_delay': 20, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7815, 'mc_device': '0,1,1,13', 'compression': 0, 'serial_num': '9210806209', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 57671680.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr115a', 'max_time_in_state': 1200, 'device': 'stkenmvr115a:/dev/rmt/tps2d0n', 'name': 'LTO3_15.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD3', 'statistics_path': '/tmp/enstore/enstore/LTO3_15.stat', 'mover_address': ('131.225.13.145', 7815), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.145', 36787), 'data_ip': 'stkenmvr115a', 'logname': 'LTO15MV', 'hostip': '131.225.13.145', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'lm': {'address': ('131.225.13.4', 7506)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/cms/WAX/11/store/data/2008/4/30/T1ReReco-B2-GlobalMar08-A-Pass4/0000/7881368C-C616-DD11-866E-001731AF685D.root', 'uid': 0, 'uname': 'root', 'rmajor': 0, 'machine': ('Linux', 'stkensrv3n.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 08:50:41 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (33188, 133831760L, 20L, 1, 9811, 5063, 1666637142L, 1209569077, 1209569077, 1209567917), 'mode_octal': '0100744', 'mode': 33252, 'size_bytes': 1666637142L, 'gname': 'root', 'gid': 0, 'fullname': '/dev/null', 'inode': 133831760L, 'minor': 20}, 'version': 'v3_7d CVS $Revision: 1.929 $ encp', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/cms/WAX/11/store/data/2008/4/30/T1ReReco-B2-GlobalMar08-A-Pass4/0000/.(access)(000700000000000015FA1C50)', 'status': ('ok', None), 'volume': 'VOI827', 'outfile': '/dev/null', 'fc': {'status': ('ok', None), 'pnfsid': '000700000000000015FA1C50', 'pnfs_name0': '/pnfs/fnal.gov/usr/cms/WAX/11/store/data/2008/4/30/T1ReReco-B2-GlobalMar08-A-Pass4/0000/7881368C-C616-DD11-866E-001731AF685D.root', 'deleted': 'no', 'external_label': 'VOI827', 'drive': 'stkenmvr106a:/dev/rmt/tps2d0n:1210116666', 'update': '2008-04-30 10:24:37.439132', 'gid': 0, 'location_cookie': '0000_000000000_0000359', 'complete_crc': 2472083105L, 'address': ('131.225.13.59', 7501), 'uid': 0, 'bfid': 'CDMS120956907700000', 'sanity_cookie': (65536L, 3436853835L), 'size': 1666637142L}, 'infilepath': '/pnfs/cms/WAX/11/store/data/2008/4/30/T1ReReco-B2-GlobalMar08-A-Pass4/0000/7881368C-C616-DD11-866E-001731AF685D.root', 'at_the_top': 20, 'bfid': 'CDMS120956907700000', 'local_inode': 2469L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': 7, 'retry': 0}, 'callback_addr': ('131.225.13.51', 57660), 'work': 'read_from_hsm', 'override_ro_mount': 1, 'times': {'lm_dequeued': 1257228063.928705, 'encp_start_time': 1257228061.1645861, 'job_queued': 1257228062.3536999, 't0': 1257228061, 'in_queue': 1.5749599933624268}, 'user_level': 'admin', 'unique_id': 'stkensrv3n.fnal.gov-1257228062-32007-0'} Thread client_connect_thread 00:01:04 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:01:04 stkensrv3n.fnal.gov 032007 root I ENCP Control socket ('131.225.13.51', 57660) is connected to ('131.225.13.145', 36788) for stkensrv3n.fnal.gov-1257228062-32007-0. elapsed=2.810sec 00:01:04 stkensrv3n.fnal.gov 032007 root I ENCP Received callback ticket from mover LTO3_15.mover for transfer stkensrv3n.fnal.gov-1257228062-32007-0. elapsed=2.820sec 00:01:04 stkensrv3n.fnal.gov 032007 root I ENCP Attempting to connect data socket to mover at ('131.225.13.145', 36787). 00:01:04 stkensrv3n.fnal.gov 032007 root I ENCP Connecting data socket to mover (('131.225.13.145', 36787)) with interface 131.225.13.51. elapsed=2.820sec 00:01:04 stkensrv3n.fnal.gov 032007 root I ENCP Data socket ('131.225.13.51', 57661) is connected to ('131.225.13.145', 36787). elapsed=2.820sec 00:01:04 stkensrv3n.fnal.gov 032007 root I ENCP Opening local file. 00:01:04 stkenmvr115a.fnal.gov 005896 root I LTO15MV tape position: target 359 current 1 6727 Thread seek_thread 00:01:04 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:04 cmsstor186.fnal.gov 025087 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Scientific Linux SL release 5.2 (Boron) Rocks release 5.1 (V.I) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: Commissioning09 FF Wrapper: cpio_odc FF Width: 15 Current working directory: cmsstor186.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --age-time 30 --delpri 1 --priority 25 --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/FE8AB576-33C8-DE11-AEF9-001D09F25456.root --put-cache 000700000000000061D70260 /storage/data2/write-pool-2/data/000700000000000061D70260 00:01:04 stkendm7a.fnal.gov 025396 root I ENCP File /data/data1/Migration_Spool/VO6012:0000_000000000_0000416 transfered for stkendm7a.fnal.gov-1257227965-25396-280 in 83.1150538921 seconds. Thread WRITE 00:01:04 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV written bytes 4800565248/4800565248, blocks 36626 header 0 trailer 324 Thread tape_thread 00:01:05 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:06 cmsstor186.fnal.gov 025087 root I ENCP Creating zero length output files. 00:01:06 cmsstor186.fnal.gov 025087 root I ENCP Sending /pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/FE8AB576-33C8-DE11-AEF9-001D09F25456.root write request to LM: unique_id: cmsstor186.fnal.gov-1257228066-25087-0 inputfile: /storage/data2/write-pool-2/data/000700000000000061D70260 outputfile: /pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/.(access)(000700000000000061D70260) 00:01:06 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:01:06 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data2/write-pool-2/data/000700000000000061D70260 (cmsstor186.fnal.gov-1257228066-25087-0) -> /pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/FE8AB576-33C8-DE11-AEF9-001D09F25456.root : library=CD-LTO4G1 family=Commissioning09 requester:root volume_family:cms.Commissioning09.cpio_odc 00:01:06 cmsstor186.fnal.gov 025087 root I ENCP Listening for control socket at: ('131.225.207.7', 34785) 00:01:06 cmsstor186.fnal.gov 025087 root I ENCP Waiting for mover to connect control socket at address: ('131.225.207.7', 34785) elapsed=3.320sec 00:01:07 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Assigning volume VOQ193 from storage group cms.Commissioning09.cpio_odc to library CD-LTO4G1, volume family cms.Commissioning09.cpio_odc 00:01:07 stkendca22a.fnal.gov 000605 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: garbage Storage Group: test File Family: dcache FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0003 --put-cache 0005000000000000005FA5F8 /diskb/write-pool-2/data/0005000000000000005FA5F8 00:01:07 stkendca22a.fnal.gov 000594 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: garbage Storage Group: test File Family: dcache FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0002 --put-cache 0005000000000000005FA5C8 /diskb/write-pool-2/data/0005000000000000005FA5C8 00:01:07 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm work on vol=VOQ193 mover=LTO4_51.mover requester:root 00:01:07 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:01:07 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM IDLE:sending {'vc': {'storage_group': 'cms', 'library': 'CD-LTO4G1', 'file_family': 'Commissioning09', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.Commissioning09.cpio_odc', 'file_family_width': 15}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/FE8AB576-33C8-DE11-AEF9-001D09F25456.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2226932284L, 'mover': 'LTO4_51.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/FE8AB576-33C8-DE11-AEF9-001D09F25456.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257228066, 'rmajor': 0, 'machine': ('Linux', 'cmsstor186.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16877, 131522040, 21L, 1, 9811, 5063, 512, 1257227735, 1257227735, 1257227606), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 2226932284L, 'fullname': '/storage/data2/write-pool-2/data/000700000000000061D70260', 'inode': 131531360L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/000700000000000061D70260', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/.(access)(000700000000000061D70260)', 'fc': {'size': 2226932284L, 'external_label': 'VOQ193', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data2/write-pool-2/data/000700000000000061D70260', 'at_the_top': 1, 'local_inode': 5869, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.207.7', 34785), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228067.7119429, 'encp_start_time': 1257228063.4360819, 'job_queued': 1257228066.7241759, 't0': 1257228063, 'in_queue': 0.98770999908447266}, 'user_level': 'admin', 'unique_id': 'cmsstor186.fnal.gov-1257228066-25087-0'} to mover 00:01:07 last message repeated 1 times 00:01:07 stkendca22a.fnal.gov 000615 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: garbage Storage Group: test File Family: dcache FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0001 --put-cache 0005000000000000005FA590 /diskb/write-pool-2/data/0005000000000000005FA590 00:01:07 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM can not remove from sg LTO4_51.mover VOQ193 00:01:07 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM SG: {'cms': [('LTO4_106.mover', 'VOQ181'), ('LTO4_49.mover', 'VOR473'), ('LTO4_42.mover', 'VOQ806'), ('LTO4_48.mover', 'VOR460'), ('LTO4_46.mover', 'VOR470'), ('LTO4_104.mover', 'VOR466'), ('LTO4_40.mover', 'VOR468'), ('LTO4_110.mover', 'VOR463'), ('LTO4_95.mover', 'VOR467'), ('LTO4_94.mover', 'VOQ174'), ('LTO4_109.mover', 'VOQ182'), ('LTO4_50.mover', 'VOR464'), ('LTO4_98.mover', 'VOR465'), ('LTO4_102.mover', 'VOR469')]} 00:01:07 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM can not remove from vf LTO4_51.mover VOQ193 00:01:07 stkenmvr151a.fnal.gov 013228 root I LTO4_51MV WRITE_TO_HSM Thread MainThread 00:01:07 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM VF: {'cms.NonCustodial.cpio_odc': [('LTO4_42.mover', 'VOQ806')], 'cms.cms11.cpio_odc': [('LTO4_49.mover', 'VOR473'), ('LTO4_48.mover', 'VOR460'), ('LTO4_46.mover', 'VOR470'), ('LTO4_104.mover', 'VOR466'), ('LTO4_40.mover', 'VOR468'), ('LTO4_110.mover', 'VOR463'), ('LTO4_95.mover', 'VOR467'), ('LTO4_50.mover', 'VOR464'), ('LTO4_98.mover', 'VOR465'), ('LTO4_102.mover', 'VOR469')], 'cms.Commissioning09.cpio_odc': [('LTO4_106.mover', 'VOQ181'), ('LTO4_94.mover', 'VOQ174'), ('LTO4_109.mover', 'VOQ182')]} 00:01:07 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:07 stkenmvr151a.fnal.gov 013228 root I LTO4_51MV SENDING {'vc': {'storage_group': 'cms', 'library': 'CD-LTO4G1', 'file_family': 'Commissioning09', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.Commissioning09.cpio_odc', 'file_family_width': 15}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/FE8AB576-33C8-DE11-AEF9-001D09F25456.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2226932284L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7851, 'mc_device': '0,5,1,12', 'compression': 0, 'serial_num': '1310154445', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr151a', 'max_time_in_state': 1200, 'device': 'stkenmvr151a:/dev/rmt/tps0d0n', 'name': 'LTO4_51.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_51.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.179', 7851), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.179', 33571), 'data_ip': 'stkenmvr151a', 'logname': 'LTO4_51MV', 'hostip': '131.225.13.179', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/FE8AB576-33C8-DE11-AEF9-001D09F25456.root', 'uid': 0, 'uname': 'root', 'mtime': 1257228066, 'rmajor': 0, 'machine': ('Linux', 'cmsstor186.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 131522040, 21L, 1, 9811, 5063, 512, 1257227735, 1257227735, 1257227606), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 2226932284L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data2/write-pool-2/data/000700000000000061D70260', 'inode': 131531360L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/000700000000000061D70260', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX//11/store/data/Commissioning09/HcalNZS/RAW/v3/000/119/231/.(access)(000700000000000061D70260)', 'fc': {'size': 2226932284L, 'external_label': 'VOQ193', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data2/write-pool-2/data/000700000000000061D70260', 'at_the_top': 1, 'local_inode': 5869, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.207.7', 34785), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228067.7119429, 'encp_start_time': 1257228063.4360819, 'job_queued': 1257228066.7241759, 't0': 1257228063, 'in_queue': 0.98770999908447266}, 'user_level': 'admin', 'unique_id': 'cmsstor186.fnal.gov-1257228066-25087-0'} Thread client_connect_thread 00:01:07 cmsstor186.fnal.gov 025087 root I ENCP Control socket ('131.225.207.7', 34785) is connected to ('131.225.13.179', 33572) for cmsstor186.fnal.gov-1257228066-25087-0. elapsed=4.330sec 00:01:07 cmsstor186.fnal.gov 025087 root I ENCP Received callback ticket from mover LTO4_51.mover for transfer cmsstor186.fnal.gov-1257228066-25087-0. elapsed=4.330sec 00:01:07 cmsstor186.fnal.gov 025087 root I ENCP Atempting to connect data socket to mover at ('131.225.13.179', 33571). 00:01:07 cmsstor186.fnal.gov 025087 root I ENCP Connecting data socket to mover (('131.225.13.179', 33571)) with interface 131.225.207.7. elapsed=4.330sec 00:01:07 cmsstor183.fnal.gov 012759 root I ENCP File /storage/data2/write-pool-2/data/000700000000000061CE4DB0 transfered for cmsstor183.fnal.gov-1257226898-12759-0 in 19.4039411545 seconds. 00:01:07 cmsstor186.fnal.gov 025087 root I ENCP Data socket ('131.225.207.7', 41644) is connected to ('131.225.13.179', 33571). elapsed=4.330sec 00:01:07 cmsstor186.fnal.gov 025087 root I ENCP Opening local file. 00:01:07 stkenmvr151a.fnal.gov 013228 root I LTO4_51MV MSG_TYPE=MC_LOAD_REQ mounting VOQ193 ULTRIUM-TD4 00:01:07 Thread media_thread 00:01:07 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC PROCESSING mount VOQ193 0,5,1,12 00:01:07 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:07 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:07 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:08 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV filemarks written. Tape VOV134 absolute location in blocks 1730066 Thread tape_thread 00:01:08 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV write_tape timing:pull 0.470349788666 write 16.4230430126 crc_check 22.3699395657 freespace 0.425440073013 block_write 39.68877244 idle 0.711123466492 Thread tape_thread 00:01:08 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV drive stats after write. Tape VOV134 position 1402 block 0 block_size 0 bloc_loc 1730066 tot_blocks 762939000 BOT 0 read_err 0 write_err 7 bytes 4800565248 block_write_tot 39.68877244 tape_rate 120955246.355 Thread tape_thread 00:01:08 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV remaining bytes info in DB 568387584000 estimated 563587018752 reported from drive 563850240000 position 1405 Thread tape_thread 00:01:08 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV new bitfile request {'uid': 0, 'external_label': 'VOV134', 'gid': 0, 'location_cookie': '0000_000000000_0001403', 'complete_crc': 1797820836L, 'sanity_cookie': (65536L, 633114557L), 'size': 4800564924L} Thread tape_thread 00:01:08 stkendca22a.fnal.gov 000605 root I ENCP Configuration Server: no such name: 'garbage.library_manager' 00:01:08 stkendca22a.fnal.gov 000605 root E ENCP INFILE=/diskb/write-pool-2/data/0005000000000000005FA5F8 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0003 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=1.42 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager' 00:01:08 stkendca22a.fnal.gov 000594 root I ENCP Configuration Server: no such name: 'garbage.library_manager' 00:01:08 stkendca22a.fnal.gov 000594 root E ENCP INFILE=/diskb/write-pool-2/data/0005000000000000005FA5C8 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0002 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=1.44 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager' 00:01:08 stkendca22a.fnal.gov 000615 root I ENCP Configuration Server: no such name: 'garbage.library_manager' 00:01:08 stkendca22a.fnal.gov 000615 root E ENCP INFILE=/diskb/write-pool-2/data/0005000000000000005FA590 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0001 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=1.42 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager' 00:01:08 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV set remaining: VOV134 563850240000 0000_000000000_0001405 Thread tape_thread 00:01:08 last message repeated 1 times 00:01:08 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-03 00:01:08', eod_cookie = '0000_000000000_0001405', non_del_files = '469', remaining_bytes = 563850240000 where label = 'VOV134'; 00:01:08 cmsstor186.fnal.gov 025087 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:01:08 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV transfer complete volume=VOV134 location=1405 Thread tape_thread 00:01:08 last message repeated 1 times 00:01:08 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '468', modification_time = '2009-11-03 00:01:08' where label = 'VOV134'; 00:01:08 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'ktev', 'blocksize': 131072, 'declared': 1254165874.0, 'library': 'CD-LTO4F1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOV134', 'wrapper': 'cern', 'remaining_bytes': 568387584000L, 'sum_mounts': 1, 'sum_rd_access': 0, 'file_family_width': 1, 'last_access': 1257227981.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0001402', 'capacity_bytes': 858993459200L, 'non_del_files': 468, 'sum_wr_err': 0, 'sum_wr_access': 467, 'file_family': 'ktev_migration-MIGRATION', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'modification_time': 1257227981.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257221152.0}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZS1/KZS109/f3', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 4800564924L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4F1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7853, 'mc_device': '1,0,1,1', 'compression': 0, 'serial_num': '1310060800', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'cdfenmvr192a', 'max_time_in_state': 1200, 'device': 'cdfenmvr192a:/dev/rmt/tps0d0n', 'name': 'LTO4_92.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_92.stat', 'mover_address': ('131.225.215.76', 7853), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.215.76', 39699), 'data_ip': 'cdfenmvr192a', 'logname': 'LTO4_92MV', 'hostip': '131.225.215.76', 'do_cleaning': 'No', 'media_changer': 'SL8500F1.media_changer'}, 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZS1/KZS109/f3', 'uid': 0, 'uname': 'root', 'mtime': 1257227965, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cern', 'pstat': (16877, 672157744L, 20L, 1, 0, 0, 512L, 1257227487, 1257227487, 1256019929), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 4800564924L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data1/Migration_Spool/VO6012:0000_000000000_0000416', 'inode': 675060072L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data1/Migration_Spool/VO6012:0000_000000000_0000416', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZS1/KZS109/.(access)(0028000000000000083C9968)', 'fc': {'uid': 0, 'external_label': 'VOV134', 'gid': 0, 'location_cookie': '0000_000000000_0001403', 'complete_crc': 1797820836L, 'bfid': 'CDMS125722806800000', 'sanity_cookie': (65536L, 633114557L), 'size': 4800564924L}, 'infilepath': '/data/data1/Migration_Spool/VO6012:0000_000000000_0000416', 'at_the_top': 36, 'local_inode': 39503173L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 48608), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257227965.497185, 'seek_time': 0.026781797409057617, 'transfer_time': 86.598263025283813, 't0': 1257227965, 'mount_time': 0.015999078750610352, 'lm_dequeued': 1257227981.6887, 'job_queued': 1257227965.777462, 'drive_call_time': 15.926268815994263, 'in_queue': 15.910967111587524, 'drive_transfer_time': 45.530725955963135}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257227965-25396-280'} Thread tape_thread 00:01:08 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV Done is sent Thread tape_thread 00:01:08 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV maybe_clean Thread tape_thread 00:01:08 stkendm7a.fnal.gov 025396 root I ENCP Received final dialog for stkendm7a.fnal.gov-1257227965-25396-280. elapsed=6085.400sec Thread WRITE 00:01:08 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:08 stkendm7a.fnal.gov 025396 root I ENCP leaving transfer_file(). Thread WRITE 00:01:08 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV returned from maybe_clean Thread tape_thread 00:01:08 stkendca21a.fnal.gov 003526 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: minos File Family: mcin_near FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca21a.fnal.gov:/ Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047028_0000_L010185R_D07_r4.reroot.root --put-cache 000F0000000000000B1BAC60 /diskb/write-pool-2/data/000F0000000000000B1BAC60 00:01:08 stkendca21a.fnal.gov 003357 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: minos File Family: mcin_near FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca21a.fnal.gov:/ Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0016_L010185R_D07_r4.reroot.root --put-cache 000F0000000000000B1BA9C0 /diskb/write-pool-2/data/000F0000000000000B1BA9C0 00:01:08 stkendca21a.fnal.gov 002844 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: minos File Family: mcin_near FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca21a.fnal.gov:/ Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0014_L010185R_D07_r4.reroot.root --put-cache 000F0000000000000B1BA938 /diskb/write-pool-2/data/000F0000000000000B1BA938 00:01:08 stkendca21a.fnal.gov 002796 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: minos File Family: mcin_near FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca21a.fnal.gov:/ Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0023_L010185R_D07_r4.reroot.root --put-cache 000F0000000000000B1BAAD8 /diskb/write-pool-2/data/000F0000000000000B1BAAD8 00:01:08 stkendca21a.fnal.gov 003442 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: minos File Family: mcin_near FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca21a.fnal.gov:/ Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047028_0005_L010185R_D07_r4.reroot.root --put-cache 000F0000000000000B1BAD38 /diskb/write-pool-2/data/000F0000000000000B1BAD38 00:01:08 stkendca21a.fnal.gov 003534 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: minos File Family: mcin_near FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca21a.fnal.gov:/ Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0030_L010185R_D07_r4.reroot.root --put-cache 000F0000000000000B1BAC00 /diskb/write-pool-2/data/000F0000000000000B1BAC00 00:01:08 stkendca21a.fnal.gov 003478 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: minos File Family: mcin_near FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca21a.fnal.gov:/ Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0011_L010185R_D07_r4.reroot.root --put-cache 000F0000000000000B1BA8B8 /diskb/write-pool-2/data/000F0000000000000B1BA8B8 00:01:08 stkenmvr202a.fnal.gov 006121 root I LTO4_102MV written bytes 2089199616/2089199616, blocks 15940 header 192 trailer 456 Thread tape_thread 00:01:08 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZS1/KZS109/f3', drive = 'cdfenmvr192a:/dev/rmt/tps0d0n:1310060800', pnfs_id = '0028000000000000083C9968' where bfid = 'CDMS125722806800000'; 00:01:08 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:08 last message repeated 1 times 00:01:08 stkendca21a.fnal.gov 003497 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: minos File Family: mcin_near FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca21a.fnal.gov:/ Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0028_L010185R_D07_r4.reroot.root --put-cache 000F0000000000000B1BABB0 /diskb/write-pool-2/data/000F0000000000000B1BABB0 00:01:08 last message repeated 1 times 00:01:08 stkendca21a.fnal.gov 003566 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: minos File Family: mcin_near FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca21a.fnal.gov:/ Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0025_L010185R_D07_r4.reroot.root --put-cache 000F0000000000000B1BAB28 /diskb/write-pool-2/data/000F0000000000000B1BAB28 00:01:08 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM write_to_hsm next work on vol=VOV134 mover=LTO4_92.mover requester:root 00:01:08 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM HAVE_BOUND:sending write_to_hsm /pnfs/fs/usr/ktev/migrated_from_dlt/KQ/KQRC/KQRC39/f5402 to mover LTO4_92.mover ('131.225.215.76', 7853) DEL_DISM 2 00:01:08 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : ktev 00:01:08 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM HAVE_BOUND:sending {'vc': {'storage_group': 'ktev', 'external_label': 'VOV134', 'library': 'CD-LTO4F1', 'file_family': 'ktev_migration-MIGRATION', 'wrapper': 'cern', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'file_family_width': 1}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KQ/KQRC/KQRC39/f5402', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 2888212L, 'mover': 'LTO4_92.mover', 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/KQ/KQRC/KQRC39/f5402', 'uid': 0, 'uname': 'root', 'type': 'cern', 'gid': 0, 'mtime': 1257227968, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (16877, 678573792L, 20L, 1, 0, 0, 512L, 1257227956, 1257227956, 1252942437), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 2888212L, 'fullname': '/data/data2/Migration_Spool/VO6153:0000_000000000_0013034', 'inode': 675060120L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data2/Migration_Spool/VO6153:0000_000000000_0013034', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KQ/KQRC/KQRC39/.(access)(0028000000000000083C9998)', 'fc': {'size': 2888212L, 'external_label': 'VOV134', 'pnfsid': '0028000000000000083C9998', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/data/data2/Migration_Spool/VO6153:0000_000000000_0013034', 'at_the_top': 468, 'local_inode': 4490943L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 48611), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228068.775569, 'encp_start_time': 1257227968.5021839, 'job_queued': 1257227968.7410481, 't0': 1257227968, 'in_queue': 100.03409481048584}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257227968-18643-18229'} to mover 00:01:08 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:08 stkendca21a.fnal.gov 003592 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: minos File Family: mcin_near FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca21a.fnal.gov:/ Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0019_L010185R_D07_r4.reroot.root --put-cache 000F0000000000000B1BAA38 /diskb/write-pool-2/data/000F0000000000000B1BAA38 00:01:08 stkendca21a.fnal.gov 003579 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: minos File Family: mcin_near FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca21a.fnal.gov:/ Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0021_L010185R_D07_r4.reroot.root --put-cache 000F0000000000000B1BAA88 /diskb/write-pool-2/data/000F0000000000000B1BAA88 00:01:08 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV WRITE_TO_HSM Thread MainThread 00:01:08 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV SENDING {'vc': {'storage_group': 'ktev', 'external_label': 'VOV134', 'library': 'CD-LTO4F1', 'file_family': 'ktev_migration-MIGRATION', 'wrapper': 'cern', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'file_family_width': 1}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KQ/KQRC/KQRC39/f5402', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 2888212L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4F1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7853, 'mc_device': '1,0,1,1', 'compression': 0, 'serial_num': '1310060800', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'cdfenmvr192a', 'max_time_in_state': 1200, 'device': 'cdfenmvr192a:/dev/rmt/tps0d0n', 'name': 'LTO4_92.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_92.stat', 'mover_address': ('131.225.215.76', 7853), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.215.76', 39708), 'data_ip': 'cdfenmvr192a', 'logname': 'LTO4_92MV', 'hostip': '131.225.215.76', 'do_cleaning': 'No', 'media_changer': 'SL8500F1.media_changer'}, 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/KQ/KQRC/KQRC39/f5402', 'uid': 0, 'uname': 'root', 'mtime': 1257227968, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cern', 'pstat': (16877, 678573792L, 20L, 1, 0, 0, 512L, 1257227956, 1257227956, 1252942437), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 2888212L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data2/Migration_Spool/VO6153:0000_000000000_0013034', 'inode': 675060120L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data2/Migration_Spool/VO6153:0000_000000000_0013034', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KQ/KQRC/KQRC39/.(access)(0028000000000000083C9998)', 'fc': {'size': 2888212L, 'external_label': 'VOV134', 'pnfsid': '0028000000000000083C9998', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/data/data2/Migration_Spool/VO6153:0000_000000000_0013034', 'at_the_top': 468, 'local_inode': 4490943L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 48611), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228068.775569, 'encp_start_time': 1257227968.5021839, 'job_queued': 1257227968.7410481, 't0': 1257227968, 'in_queue': 100.03409481048584}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257227968-18643-18229'} Thread client_connect_thread 00:01:08 stkendm7a.fnal.gov 018643 root I ENCP Listening for control socket at: ('131.225.13.19', 48611) Thread WRITE 00:01:08 stkendm7a.fnal.gov 018643 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.19', 48611) elapsed=68316.040sec Thread WRITE 00:01:08 stkendm7a.fnal.gov 018643 root I ENCP Control socket ('131.225.13.19', 48611) is connected to ('131.225.215.76', 39709) for stkendm7a.fnal.gov-1257227968-18643-18229. elapsed=68316.050sec Thread WRITE 00:01:08 stkendm7a.fnal.gov 018643 root I ENCP Received callback ticket from mover LTO4_92.mover for transfer stkendm7a.fnal.gov-1257227968-18643-18229. elapsed=68316.050sec Thread WRITE 00:01:08 stkendm7a.fnal.gov 018643 root I ENCP Attempting to connect data socket to mover at ('131.225.215.76', 39708). Thread WRITE 00:01:08 stkendm7a.fnal.gov 018643 root I ENCP Connecting data socket to mover (('131.225.215.76', 39708)) with interface 131.225.13.19. elapsed=68316.050sec Thread WRITE 00:01:08 stkendm7a.fnal.gov 018643 root I ENCP Data socket ('131.225.13.19', 48621) is connected to ('131.225.215.76', 39708). elapsed=68316.050sec Thread WRITE 00:01:08 stkendm7a.fnal.gov 018643 root I ENCP Opening local file. Thread WRITE 00:01:08 stkendm7a.fnal.gov 018643 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False Thread WRITE 00:01:08 stkendca21a.fnal.gov 003606 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: minos File Family: mcin_near FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca21a.fnal.gov:/ Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047028_0009_L010185R_D07_r4.reroot.root --put-cache 000F0000000000000B1BADE0 /diskb/write-pool-2/data/000F0000000000000B1BADE0 00:01:08 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV tape position: target 1405 current 1405 0 Thread seek_thread 00:01:08 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV write_tape starting, bytes_to_write=2888704 Thread tape_thread 00:01:08 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV Write starting. Tape VOV134 absolute location in blocks 1730066 Thread tape_thread 00:01:08 stkendm7a.fnal.gov 018643 root I ENCP File /data/data2/Migration_Spool/VO6153:0000_000000000_0013034 transfered for stkendm7a.fnal.gov-1257227968-18643-18229 in 0.0434119701385 seconds. Thread WRITE 00:01:08 stkendca21a.fnal.gov 003526 root I ENCP Creating zero length output files. 00:01:08 stkendca21a.fnal.gov 002796 root I ENCP Creating zero length output files. 00:01:08 stkendca21a.fnal.gov 003526 root I ENCP Sending /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047028_0000_L010185R_D07_r4.reroot.root write request to LM: unique_id: stkendca21a.fnal.gov-1257228068-3526-0 inputfile: /diskb/write-pool-2/data/000F0000000000000B1BAC60 outputfile: /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/.(access)(000F0000000000000B1BAC60) 00:01:08 stkendca21a.fnal.gov 003442 root I ENCP Creating zero length output files. 00:01:08 stkendca21a.fnal.gov 003478 root I ENCP Creating zero length output files. 00:01:08 stkendca21a.fnal.gov 002844 root I ENCP Creating zero length output files. 00:01:08 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : minos 00:01:09 stkendca21a.fnal.gov 003442 root I ENCP Sending /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047028_0005_L010185R_D07_r4.reroot.root write request to LM: unique_id: stkendca21a.fnal.gov-1257228068-3442-0 inputfile: /diskb/write-pool-2/data/000F0000000000000B1BAD38 outputfile: /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/.(access)(000F0000000000000B1BAD38) 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /diskb/write-pool-2/data/000F0000000000000B1BAC60 (stkendca21a.fnal.gov-1257228068-3526-0) -> /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047028_0000_L010185R_D07_r4.reroot.root : library=CD-LTO4G1 family=mcin_near requester:root volume_family:minos.mcin_near.cpio_odc 00:01:09 stkendca21a.fnal.gov 002796 root I ENCP Sending /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0023_L010185R_D07_r4.reroot.root write request to LM: unique_id: stkendca21a.fnal.gov-1257228068-2796-0 inputfile: /diskb/write-pool-2/data/000F0000000000000B1BAAD8 outputfile: /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/.(access)(000F0000000000000B1BAAD8) 00:01:09 stkendca21a.fnal.gov 003621 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: minos File Family: mcin_near FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca21a.fnal.gov:/ Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047028_0002_L010185R_D07_r4.reroot.root --put-cache 000F0000000000000B1BACB0 /diskb/write-pool-2/data/000F0000000000000B1BACB0 00:01:09 stkendca21a.fnal.gov 003478 root I ENCP Sending /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0011_L010185R_D07_r4.reroot.root write request to LM: unique_id: stkendca21a.fnal.gov-1257228068-3478-0 inputfile: /diskb/write-pool-2/data/000F0000000000000B1BA8B8 outputfile: /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/.(access)(000F0000000000000B1BA8B8) 00:01:09 stkendca21a.fnal.gov 003526 root I ENCP Listening for control socket at: ('131.225.13.21', 49478) 00:01:09 stkendca21a.fnal.gov 003526 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.21', 49478) elapsed=2.230sec 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : minos 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /diskb/write-pool-2/data/000F0000000000000B1BAD38 (stkendca21a.fnal.gov-1257228068-3442-0) -> /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047028_0005_L010185R_D07_r4.reroot.root : library=CD-LTO4G1 family=mcin_near requester:root volume_family:minos.mcin_near.cpio_odc 00:01:09 stkendca21a.fnal.gov 003357 root I ENCP Creating zero length output files. 00:01:09 stkendca21a.fnal.gov 002844 root I ENCP Sending /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0014_L010185R_D07_r4.reroot.root write request to LM: unique_id: stkendca21a.fnal.gov-1257228068-2844-0 inputfile: /diskb/write-pool-2/data/000F0000000000000B1BA938 outputfile: /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/.(access)(000F0000000000000B1BA938) 00:01:09 stkendca21a.fnal.gov 003534 root I ENCP Creating zero length output files. 00:01:09 stkendca21a.fnal.gov 003442 root I ENCP Listening for control socket at: ('131.225.13.21', 49482) 00:01:09 stkendca21a.fnal.gov 003442 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.21', 49482) elapsed=2.330sec 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : minos 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /diskb/write-pool-2/data/000F0000000000000B1BAAD8 (stkendca21a.fnal.gov-1257228068-2796-0) -> /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0023_L010185R_D07_r4.reroot.root : library=CD-LTO4G1 family=mcin_near requester:root volume_family:minos.mcin_near.cpio_odc 00:01:09 stkendca21a.fnal.gov 003357 root I ENCP Sending /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0016_L010185R_D07_r4.reroot.root write request to LM: unique_id: stkendca21a.fnal.gov-1257228068-3357-0 inputfile: /diskb/write-pool-2/data/000F0000000000000B1BA9C0 outputfile: /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/.(access)(000F0000000000000B1BA9C0) 00:01:09 stkendca21a.fnal.gov 002796 root I ENCP Listening for control socket at: ('131.225.13.21', 49481) 00:01:09 stkendca21a.fnal.gov 002796 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.21', 49481) elapsed=2.350sec 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : minos 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /diskb/write-pool-2/data/000F0000000000000B1BA8B8 (stkendca21a.fnal.gov-1257228068-3478-0) -> /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0011_L010185R_D07_r4.reroot.root : library=CD-LTO4G1 family=mcin_near requester:root volume_family:minos.mcin_near.cpio_odc 00:01:09 stkendca21a.fnal.gov 003534 root I ENCP Sending /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0030_L010185R_D07_r4.reroot.root write request to LM: unique_id: stkendca21a.fnal.gov-1257228068-3534-0 inputfile: /diskb/write-pool-2/data/000F0000000000000B1BAC00 outputfile: /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/.(access)(000F0000000000000B1BAC00) 00:01:09 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : minos 00:01:09 stkendca21a.fnal.gov 003478 root I ENCP Listening for control socket at: ('131.225.13.21', 49484) 00:01:09 stkendca21a.fnal.gov 003478 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.21', 49484) elapsed=2.370sec 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /diskb/write-pool-2/data/000F0000000000000B1BA938 (stkendca21a.fnal.gov-1257228068-2844-0) -> /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0014_L010185R_D07_r4.reroot.root : library=CD-LTO4G1 family=mcin_near requester:root volume_family:minos.mcin_near.cpio_odc 00:01:09 stkendm7a.fnal.gov 025396 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /data/data1/Migration_Spool/VO6012:0000_000000000_0000416 -> /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZS1/KZS109/f3: 4800564924 bytes copied to VOV134 at 52.9 MB/S (90.2 MB/S network) (101 MB/S drive) (61.3 MB/S disk) mover=LTO4_92.mover drive_id=ULTRIUM-TD4 drive_sn=1310060800 drive_vendor=IBM elapsed=103.57 {'storage_group': 'ktev', 'encp_crc': 1797820836L, 'disk_rate': 61.273880552805132, 'drive_rate': 100.55133925796586, 'driver': 'FTTDriver', 'transfer_rate': 52.863132387625733, 'mover_interface': 'cdfenmvr192a', 'overall_rate': 52.418654548281147, 'media_changer': 'SL8500F1.media_changer', 'network_rate': 52.863132387625733, 'encp_ip': '131.225.13.19', 'unique_id': 'stkendm7a.fnal.gov-1257227965-25396-280'} Thread WRITE 00:01:09 stkendm7a.fnal.gov 025396 root I ENCP File /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZS1/KZS109/f3 copied successfully. Thread WRITE 00:01:09 stkendm7a.fnal.gov 025396 root I ENCP Completed transferring 4800564924 bytes in 1 files in 103.568392038 sec. Overall rate = 52.4 MB/sec. Transfer rate = 52.9 MB/sec. Network rate = 90.2 MB/sec. Drive rate = 101 MB/sec. Disk rate = 61.3 MB/sec. Exit status = 0. Thread WRITE 00:01:09 stkendca21a.fnal.gov 002844 root I ENCP Listening for control socket at: ('131.225.13.21', 49480) 00:01:09 stkendca21a.fnal.gov 002844 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.21', 49480) elapsed=2.390sec 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : minos 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /diskb/write-pool-2/data/000F0000000000000B1BA9C0 (stkendca21a.fnal.gov-1257228068-3357-0) -> /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0016_L010185R_D07_r4.reroot.root : library=CD-LTO4G1 family=mcin_near requester:root volume_family:minos.mcin_near.cpio_odc 00:01:09 stkendca21a.fnal.gov 003357 root I ENCP Listening for control socket at: ('131.225.13.21', 49479) 00:01:09 stkendca21a.fnal.gov 003357 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.21', 49479) elapsed=2.390sec 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : minos 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /diskb/write-pool-2/data/000F0000000000000B1BAC00 (stkendca21a.fnal.gov-1257228068-3534-0) -> /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0030_L010185R_D07_r4.reroot.root : library=CD-LTO4G1 family=mcin_near requester:root volume_family:minos.mcin_near.cpio_odc 00:01:09 stkendca21a.fnal.gov 003534 root I ENCP Listening for control socket at: ('131.225.13.21', 49483) 00:01:09 stkendca21a.fnal.gov 003534 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.21', 49483) elapsed=2.340sec 00:01:09 stkendca21a.fnal.gov 003566 root I ENCP Creating zero length output files. 00:01:09 stkendca21a.fnal.gov 003497 root I ENCP Creating zero length output files. 00:01:09 stkensrv0n.fnal.gov 028796 enstore I FILSRV start modifying {'pnfsid': '0028000000000000083C9968', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZS1/KZS109/f3', 'deleted': 'no', 'external_label': 'VOV134', 'drive': 'cdfenmvr192a:/dev/rmt/tps0d0n:1310060800', 'update': '2009-11-03 00:01:08.596283', 'gid': 0, 'location_cookie': '0000_000000000_0001403', 'complete_crc': 1797820836L, 'uid': 0, 'bfid': 'CDMS125722806800000', 'sanity_cookie': (65536L, 633114557L), 'size': 4800564924L} 00:01:09 stkendca21a.fnal.gov 003566 root I ENCP Sending /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0025_L010185R_D07_r4.reroot.root write request to LM: unique_id: stkendca21a.fnal.gov-1257228069-3566-0 inputfile: /diskb/write-pool-2/data/000F0000000000000B1BAB28 outputfile: /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/.(access)(000F0000000000000B1BAB28) 00:01:09 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set pnfs_id = '001F00000000000000762CF0' where bfid = 'CDMS125722806800000'; 00:01:09 stkensrv0n.fnal.gov 028796 enstore I FILSRV modified to {'pnfsid': '001F00000000000000762CF0', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZS1/KZS109/f3', 'deleted': 'no', 'external_label': 'VOV134', 'drive': 'cdfenmvr192a:/dev/rmt/tps0d0n:1310060800', 'update': '2009-11-03 00:01:08.596283', 'gid': 0, 'location_cookie': '0000_000000000_0001403', 'complete_crc': 1797820836L, 'uid': 0, 'bfid': 'CDMS125722806800000', 'sanity_cookie': (65536L, 633114557L), 'size': 4800564924L} 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : minos 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /diskb/write-pool-2/data/000F0000000000000B1BAB28 (stkendca21a.fnal.gov-1257228069-3566-0) -> /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0025_L010185R_D07_r4.reroot.root : library=CD-LTO4G1 family=mcin_near requester:root volume_family:minos.mcin_near.cpio_odc 00:01:09 stkendca21a.fnal.gov 003497 root I ENCP Sending /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0028_L010185R_D07_r4.reroot.root write request to LM: unique_id: stkendca21a.fnal.gov-1257228069-3497-0 inputfile: /diskb/write-pool-2/data/000F0000000000000B1BABB0 outputfile: /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/.(access)(000F0000000000000B1BABB0) 00:01:09 stkendca21a.fnal.gov 003566 root I ENCP Listening for control socket at: ('131.225.13.21', 49486) 00:01:09 stkendca21a.fnal.gov 003566 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.21', 49486) elapsed=1.580sec 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : minos 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /diskb/write-pool-2/data/000F0000000000000B1BABB0 (stkendca21a.fnal.gov-1257228069-3497-0) -> /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0028_L010185R_D07_r4.reroot.root : library=CD-LTO4G1 family=mcin_near requester:root volume_family:minos.mcin_near.cpio_odc 00:01:09 stkendca21a.fnal.gov 003497 root I ENCP Listening for control socket at: ('131.225.13.21', 49485) 00:01:09 stkendca21a.fnal.gov 003497 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.21', 49485) elapsed=2.460sec 00:01:09 stkendca21a.fnal.gov 003579 root I ENCP Creating zero length output files. 00:01:09 stkendca21a.fnal.gov 003592 root I ENCP Creating zero length output files. 00:01:09 stkendca21a.fnal.gov 003579 root I ENCP Sending /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0021_L010185R_D07_r4.reroot.root write request to LM: unique_id: stkendca21a.fnal.gov-1257228069-3579-0 inputfile: /diskb/write-pool-2/data/000F0000000000000B1BAA88 outputfile: /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/.(access)(000F0000000000000B1BAA88) 00:01:09 stkendca21a.fnal.gov 003592 root I ENCP Sending /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0019_L010185R_D07_r4.reroot.root write request to LM: unique_id: stkendca21a.fnal.gov-1257228069-3592-0 inputfile: /diskb/write-pool-2/data/000F0000000000000B1BAA38 outputfile: /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/.(access)(000F0000000000000B1BAA38) 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : minos 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /diskb/write-pool-2/data/000F0000000000000B1BAA88 (stkendca21a.fnal.gov-1257228069-3579-0) -> /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0021_L010185R_D07_r4.reroot.root : library=CD-LTO4G1 family=mcin_near requester:root volume_family:minos.mcin_near.cpio_odc 00:01:09 stkendca21a.fnal.gov 003579 root I ENCP Listening for control socket at: ('131.225.13.21', 49488) 00:01:09 stkendca21a.fnal.gov 003579 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.21', 49488) elapsed=1.710sec 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : minos 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /diskb/write-pool-2/data/000F0000000000000B1BAA38 (stkendca21a.fnal.gov-1257228069-3592-0) -> /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047027_0019_L010185R_D07_r4.reroot.root : library=CD-LTO4G1 family=mcin_near requester:root volume_family:minos.mcin_near.cpio_odc 00:01:09 stkendca21a.fnal.gov 003592 root I ENCP Listening for control socket at: ('131.225.13.21', 49487) 00:01:09 stkendca21a.fnal.gov 003592 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.21', 49487) elapsed=1.510sec 00:01:09 stkendca21a.fnal.gov 003605 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: minos File Family: mcin_near FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca21a.fnal.gov:/ Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047028_0007_L010185R_D07_r4.reroot.root --put-cache 000F0000000000000B1BAD88 /diskb/write-pool-2/data/000F0000000000000B1BAD88 00:01:09 stkendca21a.fnal.gov 003606 root I ENCP Creating zero length output files. 00:01:09 stkendca21a.fnal.gov 003606 root I ENCP Sending /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047028_0009_L010185R_D07_r4.reroot.root write request to LM: unique_id: stkendca21a.fnal.gov-1257228069-3606-0 inputfile: /diskb/write-pool-2/data/000F0000000000000B1BADE0 outputfile: /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/.(access)(000F0000000000000B1BADE0) 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : minos 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /diskb/write-pool-2/data/000F0000000000000B1BADE0 (stkendca21a.fnal.gov-1257228069-3606-0) -> /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047028_0009_L010185R_D07_r4.reroot.root : library=CD-LTO4G1 family=mcin_near requester:root volume_family:minos.mcin_near.cpio_odc 00:01:09 stkendca21a.fnal.gov 003606 root I ENCP Listening for control socket at: ('131.225.13.21', 49489) 00:01:09 stkendca21a.fnal.gov 003606 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.21', 49489) elapsed=1.470sec 00:01:09 stkendca21a.fnal.gov 003621 root I ENCP Creating zero length output files. 00:01:09 stkendca21a.fnal.gov 003621 root I ENCP Sending /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047028_0002_L010185R_D07_r4.reroot.root write request to LM: unique_id: stkendca21a.fnal.gov-1257228069-3621-0 inputfile: /diskb/write-pool-2/data/000F0000000000000B1BACB0 outputfile: /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/.(access)(000F0000000000000B1BACB0) 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : minos 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /diskb/write-pool-2/data/000F0000000000000B1BACB0 (stkendca21a.fnal.gov-1257228069-3621-0) -> /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047028_0002_L010185R_D07_r4.reroot.root : library=CD-LTO4G1 family=mcin_near requester:root volume_family:minos.mcin_near.cpio_odc 00:01:09 stkendca21a.fnal.gov 003621 root I ENCP Listening for control socket at: ('131.225.13.21', 49490) 00:01:09 stkendca21a.fnal.gov 003621 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.21', 49490) elapsed=1.240sec 00:01:09 stkendca21a.fnal.gov 003627 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: minos File Family: mcin_near FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca21a.fnal.gov:/ Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047025_0028_L010185R_D07_r4.reroot.root --put-cache 000F0000000000000B1B9768 /diskb/write-pool-2/data/000F0000000000000B1B9768 00:01:09 stkendca21a.fnal.gov 003605 root I ENCP Creating zero length output files. 00:01:09 stkendca21a.fnal.gov 003605 root I ENCP Sending /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047028_0007_L010185R_D07_r4.reroot.root write request to LM: unique_id: stkendca21a.fnal.gov-1257228069-3605-0 inputfile: /diskb/write-pool-2/data/000F0000000000000B1BAD88 outputfile: /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/.(access)(000F0000000000000B1BAD88) 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : minos 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /diskb/write-pool-2/data/000F0000000000000B1BAD88 (stkendca21a.fnal.gov-1257228069-3605-0) -> /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047028_0007_L010185R_D07_r4.reroot.root : library=CD-LTO4G1 family=mcin_near requester:root volume_family:minos.mcin_near.cpio_odc 00:01:09 stkendca21a.fnal.gov 003605 root I ENCP Listening for control socket at: ('131.225.13.21', 49492) 00:01:09 stkendca21a.fnal.gov 003605 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.21', 49492) elapsed=1.330sec 00:01:09 stkendca21a.fnal.gov 003618 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: minos File Family: mcin_near FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca21a.fnal.gov:/ Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047028_0012_L010185R_D07_r4.reroot.root --put-cache 000F0000000000000B1BAE38 /diskb/write-pool-2/data/000F0000000000000B1BAE38 00:01:09 stkendca21a.fnal.gov 003627 root I ENCP Creating zero length output files. 00:01:09 stkendca21a.fnal.gov 003627 root I ENCP Sending /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047025_0028_L010185R_D07_r4.reroot.root write request to LM: unique_id: stkendca21a.fnal.gov-1257228069-3627-0 inputfile: /diskb/write-pool-2/data/000F0000000000000B1B9768 outputfile: /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/.(access)(000F0000000000000B1B9768) 00:01:09 cmsstor91.fnal.gov 022583 root I ENCP File /storage/data4/write-pool-4/data/000700000000000061D58DA8 transfered for cmsstor91.fnal.gov-1257226986-22583-0 in 19.673017025 seconds. 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : minos 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /diskb/write-pool-2/data/000F0000000000000B1B9768 (stkendca21a.fnal.gov-1257228069-3627-0) -> /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047025_0028_L010185R_D07_r4.reroot.root : library=CD-LTO4G1 family=mcin_near requester:root volume_family:minos.mcin_near.cpio_odc 00:01:09 stkendca21a.fnal.gov 003627 root I ENCP Listening for control socket at: ('131.225.13.21', 49493) 00:01:09 stkendca21a.fnal.gov 003627 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.21', 49493) elapsed=0.710sec 00:01:09 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:09 stkendca21a.fnal.gov 003618 root I ENCP Creating zero length output files. 00:01:09 stkendca21a.fnal.gov 003618 root I ENCP Sending /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047028_0012_L010185R_D07_r4.reroot.root write request to LM: unique_id: stkendca21a.fnal.gov-1257228069-3618-0 inputfile: /diskb/write-pool-2/data/000F0000000000000B1BAE38 outputfile: /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/.(access)(000F0000000000000B1BAE38) 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : minos 00:01:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /diskb/write-pool-2/data/000F0000000000000B1BAE38 (stkendca21a.fnal.gov-1257228069-3618-0) -> /pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047028_0012_L010185R_D07_r4.reroot.root : library=CD-LTO4G1 family=mcin_near requester:root volume_family:minos.mcin_near.cpio_odc 00:01:09 stkendca21a.fnal.gov 003618 root I ENCP Listening for control socket at: ('131.225.13.21', 49494) 00:01:09 stkendca21a.fnal.gov 003618 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.21', 49494) elapsed=1.810sec 00:01:09 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:10 stkenmvr202a.fnal.gov 006121 root I LTO4_102MV filemarks written. Tape VOR469 absolute location in blocks 15943 Thread tape_thread 00:01:10 stkenmvr202a.fnal.gov 006121 root I LTO4_102MV write_tape timing:pull 0.189755916595 write 8.16742801666 crc_check 9.63536190987 freespace 0.19853925705 block_write 18.1910851002 idle 0.32485127449 Thread tape_thread 00:01:10 stkenmvr202a.fnal.gov 006121 root I LTO4_102MV drive stats after write. Tape VOR469 position 1 block 0 block_size 0 bloc_loc 15943 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 2089199616 block_write_tot 18.1910851002 tape_rate 114847443.377 Thread tape_thread 00:01:10 stkenmvr202a.fnal.gov 006121 root I LTO4_102MV remaining bytes info in DB 858993459200 estimated 856904259584 reported from drive 779253760000 position 2 Thread tape_thread 00:01:10 stkenmvr202a.fnal.gov 006121 root I LTO4_102MV new bitfile request {'uid': 0, 'external_label': 'VOR469', 'gid': 0, 'location_cookie': '0000_000000000_0000001', 'complete_crc': 2633941196L, 'sanity_cookie': (65536L, 2538428661L), 'size': 2089198968L} Thread tape_thread 00:01:10 stkensrv4n.fnal.gov 005418 enstore I SL8500G1MC query drive 0,5,1,12 => 0,0,5,1,12 online available IBM-LTO4 00:01:10 stkensrv4n.fnal.gov 005418 enstore I SL8500G1MC Dismount 14 ignored: Drive 0,5,1,12 is empty. Thought VOQ193 was there =>. dismount VOLUME 0,5,1,12 force => ok,0,5,1,12 online available IBM-LTO4 00:01:10 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV written bytes 2888704/2888704, blocks 23 header 0 trailer 492 Thread tape_thread 00:01:10 stkenmvr202a.fnal.gov 006121 root I LTO4_102MV set remaining: VOR469 779253760000 0000_000000000_0000002 Thread tape_thread 00:01:10 last message repeated 1 times 00:01:10 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set modification_time = '2009-11-03 00:01:10', last_access = '2009-11-03 00:01:10', eod_cookie = '0000_000000000_0000002', non_del_files = '1', remaining_bytes = 779253760000 where label = 'VOR469'; 00:01:10 stkenmvr202a.fnal.gov 006121 root I LTO4_102MV transfer complete volume=VOR469 location=2 Thread tape_thread 00:01:10 last message repeated 1 times 00:01:10 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '1' where label = 'VOR469'; 00:01:10 stkenmvr202a.fnal.gov 006121 root I LTO4_102MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1255359040.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOR469', 'wrapper': 'cpio_odc', 'remaining_bytes': 858993459200L, 'sum_mounts': 1, 'sum_rd_access': 0, 'file_family_width': 10, 'last_access': -1, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000001', 'capacity_bytes': 858993459200L, 'non_del_files': 0, 'sum_wr_err': 0, 'sum_wr_access': 0, 'file_family': 'cms11', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'modification_time': -1, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': -1}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/50AB94D3-2FC8-DE11-91E4-00304867916E.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2089198968L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7862, 'mc_device': '0,6,1,14', 'compression': 0, 'serial_num': '1310051194', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr202a', 'max_time_in_state': 1200, 'device': 'stkenmvr202a:/dev/rmt/tps0d0n', 'name': 'LTO4_102.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_102.stat', 'mover_address': ('131.225.13.197', 7862), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.197', 33321), 'data_ip': 'stkenmvr202a', 'logname': 'LTO4_102MV', 'hostip': '131.225.13.197', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/50AB94D3-2FC8-DE11-91E4-00304867916E.root', 'uid': 0, 'uname': 'root', 'mtime': 1257226898, 'rmajor': 0, 'machine': ('Linux', 'cmsstor183.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 130813352, 21L, 1, 9811, 5063, 512, 1257226372, 1257226372, 1257220971), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2089198968L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data2/write-pool-2/data/000700000000000061CE4DB0', 'inode': 130960816L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data2/write-pool-2/data/000700000000000061CE4DB0', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/.(access)(000700000000000061CE4DB0)', 'fc': {'uid': 0, 'external_label': 'VOR469', 'gid': 0, 'location_cookie': '0000_000000000_0000001', 'complete_crc': 2633941196L, 'bfid': 'CDMS125722807000000', 'sanity_cookie': (65536L, 2538428661L), 'size': 2089198968L}, 'infilepath': '/storage/data2/write-pool-2/data/000700000000000061CE4DB0', 'at_the_top': 6, 'local_inode': 6062, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.207.3', 48584), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257226897.7600019, 'seek_time': 77.704575061798096, 'transfer_time': 21.888175964355469, 't0': 1257226897, 'lm_dequeued': 1257227970.5865691, 'job_queued': 1257226898.6934609, 'drive_call_time': 7.9461574554443359, 'in_queue': 1071.8928210735321, 'drive_transfer_time': 20.276193618774414}, 'user_level': 'admin', 'unique_id': 'cmsstor183.fnal.gov-1257226898-12759-0'} Thread tape_thread 00:01:10 stkenmvr202a.fnal.gov 006121 root I LTO4_102MV Done is sent Thread tape_thread 00:01:10 stkenmvr202a.fnal.gov 006121 root I LTO4_102MV maybe_clean Thread tape_thread 00:01:10 cmsstor183.fnal.gov 012759 root I ENCP Received final dialog for cmsstor183.fnal.gov-1257226898-12759-0. elapsed=1172.500sec 00:01:10 cmsstor183.fnal.gov 012759 root I ENCP leaving transfer_file(). 00:01:10 stkenmvr202a.fnal.gov 006121 root I LTO4_102MV returned from maybe_clean Thread tape_thread 00:01:10 cmsstor183.fnal.gov 012759 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:01:10 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:10 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VOR469 mover=LTO4_102.mover requester:root 00:01:10 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0062/82287180-38C8-DE11-9418-003048678FFE.root to mover LTO4_102.mover ('131.225.13.197', 7862) DEL_DISM None 00:01:10 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:01:10 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'cms', 'external_label': 'VOR469', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0062/82287180-38C8-DE11-9418-003048678FFE.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2277838090L, 'mover': 'LTO4_102.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0062/82287180-38C8-DE11-9418-003048678FFE.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257227034, 'rmajor': 0, 'machine': ('Linux', 'cmsstor238.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16893, 130433880, 21L, 1, 9811, 5063, 512, 1257226953, 1257226953, 1257217225), 'mode_octal': '0100036', 'mode': 33188, 'gname': 'root', 'size_bytes': 2277838090L, 'fullname': '/storage/data1/write-pool-1/data/000700000000000061D41C48', 'inode': 131341384L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/000700000000000061D41C48', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0062/.(access)(000700000000000061D41C48)', 'fc': {'size': 2277838090L, 'external_label': 'VOR469', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data1/write-pool-1/data/000700000000000061D41C48', 'at_the_top': 55, 'local_inode': 5473, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.191.110', 43253), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228070.478301, 'encp_start_time': 1257227033.5496249, 'job_queued': 1257227034.568567, 't0': 1257227033, 'in_queue': 1035.909451007843}, 'user_level': 'admin', 'unique_id': 'cmsstor238.fnal.gov-1257227034-10874-0'} to mover 00:01:10 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:10 stkenmvr202a.fnal.gov 006121 root I LTO4_102MV WRITE_TO_HSM Thread MainThread 00:01:10 stkenmvr202a.fnal.gov 006121 root I LTO4_102MV SENDING {'vc': {'storage_group': 'cms', 'external_label': 'VOR469', 'library': 'CD-LTO4G1', 'file_family': 'cms11', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms11.cpio_odc', 'file_family_width': 10}, 'outfilepath': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0062/82287180-38C8-DE11-9418-003048678FFE.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2277838090L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7862, 'mc_device': '0,6,1,14', 'compression': 0, 'serial_num': '1310051194', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr202a', 'max_time_in_state': 1200, 'device': 'stkenmvr202a:/dev/rmt/tps0d0n', 'name': 'LTO4_102.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_102.stat', 'mover_address': ('131.225.13.197', 7862), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.197', 33328), 'data_ip': 'stkenmvr202a', 'logname': 'LTO4_102MV', 'hostip': '131.225.13.197', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0062/82287180-38C8-DE11-9418-003048678FFE.root', 'uid': 0, 'uname': 'root', 'mtime': 1257227034, 'rmajor': 0, 'machine': ('Linux', 'cmsstor238.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 130433880, 21L, 1, 9811, 5063, 512, 1257226953, 1257226953, 1257217225), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2277838090L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data1/write-pool-1/data/000700000000000061D41C48', 'inode': 131341384L, 'minor': 21}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/000700000000000061D41C48', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0062/.(access)(000700000000000061D41C48)', 'fc': {'size': 2277838090L, 'external_label': 'VOR469', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data1/write-pool-1/data/000700000000000061D41C48', 'at_the_top': 55, 'local_inode': 5473, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.191.110', 43253), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228070.478301, 'encp_start_time': 1257227033.5496249, 'job_queued': 1257227034.568567, 't0': 1257227033, 'in_queue': 1035.909451007843}, 'user_level': 'admin', 'unique_id': 'cmsstor238.fnal.gov-1257227034-10874-0'} Thread client_connect_thread 00:01:10 cmsstor238.fnal.gov 010874 root I ENCP Control socket ('131.225.191.110', 43253) is connected to ('131.225.13.197', 33329) for cmsstor238.fnal.gov-1257227034-10874-0. elapsed=1036.960sec 00:01:10 cmsstor238.fnal.gov 010874 root I ENCP Received callback ticket from mover LTO4_102.mover for transfer cmsstor238.fnal.gov-1257227034-10874-0. elapsed=1036.960sec 00:01:10 cmsstor238.fnal.gov 010874 root I ENCP Atempting to connect data socket to mover at ('131.225.13.197', 33328). 00:01:10 cmsstor238.fnal.gov 010874 root I ENCP Connecting data socket to mover (('131.225.13.197', 33328)) with interface 131.225.191.110. elapsed=1036.960sec 00:01:10 cmsstor238.fnal.gov 010874 root I ENCP Data socket ('131.225.191.110', 44561) is connected to ('131.225.13.197', 33328). elapsed=1036.960sec 00:01:10 cmsstor238.fnal.gov 010874 root I ENCP Opening local file. 00:01:10 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/50AB94D3-2FC8-DE11-91E4-00304867916E.root', drive = 'stkenmvr202a:/dev/rmt/tps0d0n:1310051194', pnfs_id = '000700000000000061CE4DB0' where bfid = 'CDMS125722807000000'; 00:01:10 stkenmvr202a.fnal.gov 006121 root I LTO4_102MV tape position: target 2 current 2 0 Thread seek_thread 00:01:10 stkenmvr202a.fnal.gov 006121 root I LTO4_102MV write_tape starting, bytes_to_write=2277838848 Thread tape_thread 00:01:10 stkenmvr202a.fnal.gov 006121 root I LTO4_102MV Write starting. Tape VOR469 absolute location in blocks 15943 Thread tape_thread 00:01:10 cmsstor238.fnal.gov 010874 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:01:10 cmsstor183.fnal.gov 012759 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /storage/data2/write-pool-2/data/000700000000000061CE4DB0 -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/50AB94D3-2FC8-DE11-91E4-00304867916E.root: 2089198968 bytes copied to VOR469 at 91 MB/S (97.2 MB/S network) (98.3 MB/S drive) (103 MB/S disk) mover=LTO4_102.mover drive_id=ULTRIUM-TD4 drive_sn=1310051194 drive_vendor=IBM elapsed=1172.9 {'storage_group': 'cms', 'encp_crc': 2633941196L, 'disk_rate': 102.70554006748276, 'drive_rate': 98.263778453920366, 'driver': 'FTTDriver', 'transfer_rate': 90.997134984801633, 'mover_interface': 'stkenmvr202a', 'overall_rate': 19.916219975082406, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 90.997134984801633, 'encp_ip': '131.225.207.3', 'unique_id': 'cmsstor183.fnal.gov-1257226898-12759-0'} 00:01:10 cmsstor183.fnal.gov 012759 root I ENCP File /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0063/50AB94D3-2FC8-DE11-91E4-00304867916E.root copied successfully. 00:01:10 cmsstor183.fnal.gov 012759 root I ENCP Completed transferring 2089198968 bytes in 1 files in 1172.90813303 sec. Overall rate = 19.9 MB/sec. Transfer rate = 91 MB/sec. Network rate = 97.2 MB/sec. Drive rate = 98.3 MB/sec. Disk rate = 103 MB/sec. Exit status = 0. 00:01:10 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Assigning volume VON685 from storage group minos.mcin_near.cpio_odc to library CD-LTO4G1, volume family minos.mcin_near.cpio_odc 00:01:10 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm work on vol=VON685 mover=LTO4_108.mover requester:root 00:01:10 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : minos 00:01:10 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM IDLE:sending {'vc': {'storage_group': 'minos', 'library': 'CD-LTO4G1', 'file_family': 'mcin_near', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'minos.mcin_near.cpio_odc', 'file_family_width': 1}, 'outfilepath': '/pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047028_0000_L010185R_D07_r4.reroot.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 242267279L, 'mover': 'LTO4_108.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047028_0000_L010185R_D07_r4.reroot.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257228068, 'rmajor': 0, 'machine': ('Linux', 'stkendca21a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (16893, 253455592L, 20L, 1, 1060, 5111, 512L, 1257226544, 1257226544, 1257219771), 'mode_octal': '0100036', 'mode': 33188, 'gname': 'root', 'size_bytes': 242267279L, 'fullname': '/diskb/write-pool-2/data/000F0000000000000B1BAC60', 'inode': 253471840L, 'minor': 20}, 'version': 'v3_8b CVS $Revision: 1.922 $ ', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskb/write-pool-2/data/000F0000000000000B1BAC60', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/.(access)(000F0000000000000B1BAC60)', 'fc': {'size': 242267279L, 'external_label': 'VON685', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diskb/write-pool-2/data/000F0000000000000B1BAC60', 'at_the_top': 1, 'local_inode': 1073745011L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.21', 49478), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228070.823643, 'encp_start_time': 1257228066.7822449, 'job_queued': 1257228068.9912779, 't0': 1257228066, 'in_queue': 1.8320930004119873}, 'user_level': 'user2', 'unique_id': 'stkendca21a.fnal.gov-1257228068-3526-0'} to mover 00:01:10 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV WRITE_TO_HSM Thread MainThread 00:01:10 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV SENDING {'vc': {'storage_group': 'minos', 'library': 'CD-LTO4G1', 'file_family': 'mcin_near', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'minos.mcin_near.cpio_odc', 'file_family_width': 1}, 'outfilepath': '/pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047028_0000_L010185R_D07_r4.reroot.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 242267279L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7868, 'mc_device': '0,5,1,11', 'compression': 0, 'serial_num': '1310154353', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr208a', 'max_time_in_state': 1200, 'device': 'stkenmvr208a:/dev/rmt/tps0d0n', 'name': 'LTO4_108.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_108.stat', 'mover_address': ('131.225.13.204', 7868), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.204', 33741), 'data_ip': 'stkenmvr208a', 'logname': 'LTO4_108MV', 'hostip': '131.225.13.204', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/n13047028_0000_L010185R_D07_r4.reroot.root', 'uid': 0, 'uname': 'root', 'mtime': 1257228068, 'rmajor': 0, 'machine': ('Linux', 'stkendca21a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 253455592L, 20L, 1, 1060, 5111, 512L, 1257226544, 1257226544, 1257219771), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 242267279L, 'gname': 'root', 'gid': 0, 'fullname': '/diskb/write-pool-2/data/000F0000000000000B1BAC60', 'inode': 253471840L, 'minor': 20}, 'version': 'v3_8b CVS $Revision: 1.922 $ ', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskb/write-pool-2/data/000F0000000000000B1BAC60', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/minos/mcin_data/near/daikon_07/L010185R_r4/702/.(access)(000F0000000000000B1BAC60)', 'fc': {'size': 242267279L, 'external_label': 'VON685', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diskb/write-pool-2/data/000F0000000000000B1BAC60', 'at_the_top': 1, 'local_inode': 1073745011L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.21', 49478), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228070.823643, 'encp_start_time': 1257228066.7822449, 'job_queued': 1257228068.9912779, 't0': 1257228066, 'in_queue': 1.8320930004119873}, 'user_level': 'user2', 'unique_id': 'stkendca21a.fnal.gov-1257228068-3526-0'} Thread client_connect_thread 00:01:10 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM can not remove from sg LTO4_108.mover VON685 00:01:10 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM SG: {'cms': [('LTO4_106.mover', 'VOQ181'), ('LTO4_49.mover', 'VOR473'), ('LTO4_48.mover', 'VOR460'), ('LTO4_46.mover', 'VOR470'), ('LTO4_104.mover', 'VOR466'), ('LTO4_110.mover', 'VOR463'), ('LTO4_95.mover', 'VOR467'), ('LTO4_94.mover', 'VOQ174'), ('LTO4_109.mover', 'VOQ182'), ('LTO4_50.mover', 'VOR464'), ('LTO4_98.mover', 'VOR465'), ('LTO4_42.mover', 'VOQ806'), ('LTO4_51.mover', 'VOQ193'), ('LTO4_40.mover', 'VOR468'), ('LTO4_102.mover', 'VOR469')]} 00:01:10 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM can not remove from vf LTO4_108.mover VON685 00:01:10 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM VF: {'cms.NonCustodial.cpio_odc': [('LTO4_42.mover', 'VOQ806')], 'cms.cms11.cpio_odc': [('LTO4_49.mover', 'VOR473'), ('LTO4_48.mover', 'VOR460'), ('LTO4_46.mover', 'VOR470'), ('LTO4_104.mover', 'VOR466'), ('LTO4_110.mover', 'VOR463'), ('LTO4_95.mover', 'VOR467'), ('LTO4_50.mover', 'VOR464'), ('LTO4_98.mover', 'VOR465'), ('LTO4_40.mover', 'VOR468'), ('LTO4_102.mover', 'VOR469')], 'cms.Commissioning09.cpio_odc': [('LTO4_106.mover', 'VOQ181'), ('LTO4_94.mover', 'VOQ174'), ('LTO4_109.mover', 'VOQ182'), ('LTO4_51.mover', 'VOQ193')]} 00:01:10 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:10 stkendca21a.fnal.gov 003526 root I ENCP Control socket ('131.225.13.21', 49478) is connected to ('131.225.13.204', 33742) for stkendca21a.fnal.gov-1257228068-3526-0. elapsed=4.080sec 00:01:10 stkendca21a.fnal.gov 003526 root I ENCP Received callback ticket from mover LTO4_108.mover for transfer stkendca21a.fnal.gov-1257228068-3526-0. elapsed=4.080sec 00:01:10 stkendca21a.fnal.gov 003526 root I ENCP Atempting to connect data socket to mover at ('131.225.13.204', 33741). 00:01:10 stkendca21a.fnal.gov 003526 root I ENCP Connecting data socket to mover (('131.225.13.204', 33741)) with interface 131.225.13.21. elapsed=4.080sec 00:01:10 stkendca21a.fnal.gov 003526 root I ENCP Data socket ('131.225.13.21', 49496) is connected to ('131.225.13.204', 33741). elapsed=4.080sec 00:01:10 stkendca21a.fnal.gov 003526 root I ENCP Opening local file. 00:01:10 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV MSG_TYPE=MC_LOAD_REQ mounting VON685 ULTRIUM-TD4 00:01:10 Thread media_thread 00:01:10 stkendca21a.fnal.gov 003526 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:01:10 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC MC Overflow: 4 mount VON685 0,5,1,11 00:01:10 stkenmvr208a.fnal.gov 006111 root E LTO4_108MV loadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread 00:01:10 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread 00:01:10 cmsstor214.fnal.gov 011706 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms11 FF Wrapper: cpio_odc FF Width: 10 Current working directory: cmsstor214.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --priority 25 --pnfs-mount /pnfs/fs --put-cache 000700000000000061D772D0 /storage/data3/write-pool-3/data/000700000000000061D772D0 00:01:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:11 cmsstor214.fnal.gov 011706 root I ENCP Creating zero length output files. 00:01:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:11 cmsstor214.fnal.gov 011706 root I ENCP Sending /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/EEE5DD5D-3DC8-DE11-82B5-002618943950.root write request to LM: unique_id: cmsstor214.fnal.gov-1257228071-11706-0 inputfile: /storage/data3/write-pool-3/data/000700000000000061D772D0 outputfile: /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D772D0) 00:01:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:01:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data3/write-pool-3/data/000700000000000061D772D0 (cmsstor214.fnal.gov-1257228071-11706-0) -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/EEE5DD5D-3DC8-DE11-82B5-002618943950.root : library=CD-LTO4G1 family=cms11 requester:root volume_family:cms.cms11.cpio_odc 00:01:11 cmsstor214.fnal.gov 011706 root I ENCP Listening for control socket at: ('131.225.191.86', 45801) 00:01:11 cmsstor214.fnal.gov 011706 root I ENCP Waiting for mover to connect control socket at address: ('131.225.191.86', 45801) elapsed=1.120sec 00:01:11 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:12 cmsstor238.fnal.gov 024075 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cms11 FF Wrapper: cpio_odc FF Width: 10 Current working directory: cmsstor238.fnal.gov:/root Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --priority 25 --pnfs-mount /pnfs/fs --put-cache 000700000000000061D47AD8 /storage/data1/write-pool-1/data/000700000000000061D47AD8 00:01:13 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:13 stkensrv4n.fnal.gov 005418 enstore I SL8500G1MC query vol VOQ193 => 0,VOQ193 home 0,11,21, 1, 1 LTO-800G 00:01:13 cmsstor238.fnal.gov 024075 root I ENCP Creating zero length output files. 00:01:13 cmsstor238.fnal.gov 024075 root I ENCP Sending /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/F0732CCA-38C8-DE11-8CB8-001A92971B32.root write request to LM: unique_id: cmsstor238.fnal.gov-1257228073-24075-0 inputfile: /storage/data1/write-pool-1/data/000700000000000061D47AD8 outputfile: /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/.(access)(000700000000000061D47AD8) 00:01:13 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:01:13 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data1/write-pool-1/data/000700000000000061D47AD8 (cmsstor238.fnal.gov-1257228073-24075-0) -> /pnfs/fs/usr/cms/WAX/11/store/data/1/Calo/RECO/backfill_CosmicsSeq_v1/0064/F0732CCA-38C8-DE11-8CB8-001A92971B32.root : library=CD-LTO4G1 family=cms11 requester:root volume_family:cms.cms11.cpio_odc 00:01:13 cmsstor238.fnal.gov 024075 root I ENCP Listening for control socket at: ('131.225.191.110', 40645) 00:01:13 cmsstor238.fnal.gov 024075 root I ENCP Waiting for mover to connect control socket at address: ('131.225.191.110', 40645) elapsed=2.060sec 00:01:13 cmsstor239.fnal.gov 011356 root I ENCP File /storage/data3/write-pool-3/data/000700000000000061D5A058 transfered for cmsstor239.fnal.gov-1257227006-11356-0 in 20.1420509815 seconds. 00:01:13 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:14 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV filemarks written. Tape VOV134 absolute location in blocks 1730094 Thread tape_thread 00:01:14 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV write_tape timing:pull 0.000183343887329 write 0.00680041313171 crc_check 0.013343334198 freespace 0.000180721282959 block_write 0.0205078125 idle 0.000352144241333 Thread tape_thread 00:01:14 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV drive stats after write. Tape VOV134 position 1405 block 0 block_size 0 bloc_loc 1730094 tot_blocks 762939000 BOT 0 read_err 0 write_err 7 bytes 2888704 block_write_tot 0.0205078125 tape_rate 140858709.333 Thread tape_thread 00:01:14 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV remaining bytes info in DB 563850240000 estimated 563847351296 reported from drive 563844096000 position 1408 Thread tape_thread 00:01:14 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV new bitfile request {'uid': 0, 'external_label': 'VOV134', 'gid': 0, 'location_cookie': '0000_000000000_0001406', 'complete_crc': 1036903317L, 'sanity_cookie': (65536L, 943266205L), 'size': 2888212L} Thread tape_thread 00:01:14 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV set remaining: VOV134 563844096000 0000_000000000_0001408 Thread tape_thread 00:01:14 last message repeated 1 times 00:01:14 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-03 00:01:14', eod_cookie = '0000_000000000_0001408', non_del_files = '470', remaining_bytes = 563844096000 where label = 'VOV134'; 00:01:14 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV transfer complete volume=VOV134 location=1408 Thread tape_thread 00:01:14 last message repeated 1 times 00:01:14 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '469', modification_time = '2009-11-03 00:01:14' where label = 'VOV134'; 00:01:14 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'ktev', 'blocksize': 131072, 'declared': 1254165874.0, 'library': 'CD-LTO4F1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOV134', 'wrapper': 'cern', 'remaining_bytes': 563850240000L, 'sum_mounts': 1, 'sum_rd_access': 0, 'file_family_width': 1, 'last_access': 1257228068.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0001405', 'capacity_bytes': 858993459200L, 'non_del_files': 469, 'sum_wr_err': 0, 'sum_wr_access': 468, 'file_family': 'ktev_migration-MIGRATION', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'modification_time': 1257228068.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257221152.0}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KQ/KQRC/KQRC39/f5402', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 2888212L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4F1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7853, 'mc_device': '1,0,1,1', 'compression': 0, 'serial_num': '1310060800', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'cdfenmvr192a', 'max_time_in_state': 1200, 'device': 'cdfenmvr192a:/dev/rmt/tps0d0n', 'name': 'LTO4_92.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_92.stat', 'mover_address': ('131.225.215.76', 7853), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.215.76', 39708), 'data_ip': 'cdfenmvr192a', 'logname': 'LTO4_92MV', 'hostip': '131.225.215.76', 'do_cleaning': 'No', 'media_changer': 'SL8500F1.media_changer'}, 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/KQ/KQRC/KQRC39/f5402', 'uid': 0, 'uname': 'root', 'mtime': 1257227968, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cern', 'pstat': (16877, 678573792L, 20L, 1, 0, 0, 512L, 1257227956, 1257227956, 1252942437), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 2888212L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data2/Migration_Spool/VO6153:0000_000000000_0013034', 'inode': 675060120L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data2/Migration_Spool/VO6153:0000_000000000_0013034', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KQ/KQRC/KQRC39/.(access)(0028000000000000083C9998)', 'fc': {'uid': 0, 'external_label': 'VOV134', 'gid': 0, 'location_cookie': '0000_000000000_0001406', 'complete_crc': 1036903317L, 'bfid': 'CDMS125722807400000', 'sanity_cookie': (65536L, 943266205L), 'size': 2888212L}, 'infilepath': '/data/data2/Migration_Spool/VO6153:0000_000000000_0013034', 'at_the_top': 468, 'local_inode': 4490943L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 48611), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257227968.5021839, 'seek_time': 0.026943206787109375, 'transfer_time': 5.4523317813873291, 't0': 1257227968, 'mount_time': 0.018481016159057617, 'lm_dequeued': 1257228068.775569, 'job_queued': 1257227968.7410481, 'drive_call_time': 0.017557382583618164, 'in_queue': 100.03409481048584, 'drive_transfer_time': 5.3348925113677979}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257227968-18643-18229'} Thread tape_thread 00:01:14 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV Done is sent Thread tape_thread 00:01:14 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV maybe_clean Thread tape_thread 00:01:14 stkendm7a.fnal.gov 018643 root I ENCP Received final dialog for stkendm7a.fnal.gov-1257227968-18643-18229. elapsed=68321.570sec Thread WRITE 00:01:14 stkendm7a.fnal.gov 018643 root I ENCP leaving transfer_file(). Thread WRITE 00:01:14 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV returned from maybe_clean Thread tape_thread 00:01:14 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fs/usr/ktev/migrated_from_dlt/KQ/KQRC/KQRC39/f5402', drive = 'cdfenmvr192a:/dev/rmt/tps0d0n:1310060800', pnfs_id = '0028000000000000083C9998' where bfid = 'CDMS125722807400000'; 00:01:14 stkendm7a.fnal.gov 018643 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /data/data2/Migration_Spool/VO6153:0000_000000000_0013034 -> /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KQ/KQRC/KQRC39/f5402: 2888212 bytes copied to VOV134 at 0.504 MB/S (87.5 MB/S network) (0.516 MB/S drive) (71.1 MB/S disk) mover=LTO4_92.mover drive_id=ULTRIUM-TD4 drive_sn=1310060800 drive_vendor=IBM elapsed=106.21 {'storage_group': 'ktev', 'encp_crc': 1036903317L, 'disk_rate': 71.059635987710394, 'drive_rate': 0.51630161223813142, 'driver': 'FTTDriver', 'transfer_rate': 0.50427147962653995, 'mover_interface': 'cdfenmvr192a', 'overall_rate': 0.46666347824906756, 'media_changer': 'SL8500F1.media_changer', 'network_rate': 0.50427147962653995, 'encp_ip': '131.225.13.19', 'unique_id': 'stkendm7a.fnal.gov-1257227968-18643-18229'} Thread WRITE 00:01:14 stkendm7a.fnal.gov 018643 root I ENCP File /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KQ/KQRC/KQRC39/f5402 copied successfully. Thread WRITE 00:01:14 stkendm7a.fnal.gov 018643 root I ENCP Completed transferring 2888212 bytes in 1 files in 106.214245081 sec. Overall rate = 0.467 MB/sec. Transfer rate = 0.504 MB/sec. Network rate = 87.5 MB/sec. Drive rate = 0.516 MB/sec. Disk rate = 71.1 MB/sec. Exit status = 0. Thread WRITE 00:01:14 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:14 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM write_to_hsm next work on vol=VOV134 mover=LTO4_92.mover requester:root 00:01:14 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM HAVE_BOUND:sending write_to_hsm /pnfs/fs/usr/ktev/migrated_from_dlt/KQ/KQRC/KQRC42/f809 to mover LTO4_92.mover ('131.225.215.76', 7853) DEL_DISM 2 00:01:14 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : ktev 00:01:14 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM HAVE_BOUND:sending {'vc': {'storage_group': 'ktev', 'external_label': 'VOV134', 'library': 'CD-LTO4F1', 'file_family': 'ktev_migration-MIGRATION', 'wrapper': 'cern', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'file_family_width': 1}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KQ/KQRC/KQRC42/f809', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 364000L, 'mover': 'LTO4_92.mover', 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/KQ/KQRC/KQRC42/f809', 'uid': 0, 'uname': 'root', 'type': 'cern', 'gid': 0, 'mtime': 1257227976, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (16877, 678574128L, 20L, 1, 0, 0, 512L, 1257226645, 1257226645, 1252942651), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 364000L, 'fullname': '/data/data2/Migration_Spool/VO6153:0000_000000000_0013049', 'inode': 675060152L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data2/Migration_Spool/VO6153:0000_000000000_0013049', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KQ/KQRC/KQRC42/.(access)(0028000000000000083C99B8)', 'fc': {'size': 364000L, 'external_label': 'VOV134', 'pnfsid': '0028000000000000083C99B8', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/data/data2/Migration_Spool/VO6153:0000_000000000_0013049', 'at_the_top': 33, 'local_inode': 4490948L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 48615), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257228074.797718, 'encp_start_time': 1257227975.8543341, 'job_queued': 1257227976.087235, 't0': 1257227975, 'in_queue': 98.710210084915161}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257227976-18643-18230'} to mover 00:01:14 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:14 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV WRITE_TO_HSM Thread MainThread 00:01:14 stkendm7a.fnal.gov 018643 root I ENCP Listening for control socket at: ('131.225.13.19', 48615) Thread WRITE_2 00:01:14 cdfenmvr192a.fnal.gov 006151 root I LTO4_92MV SENDING {'vc': {'storage_group': 'ktev', 'external_label': 'VOV134', 'library': 'CD-LTO4F1', 'file_family': 'ktev_migration-MIGRATION', 'wrapper': 'cern', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'file_family_width': 1}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KQ/KQRC/KQRC42/f809', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 364000L, 'mover': {'max_buffer'