00:00:01 stkensrv4n.fnal.gov 008130 enstore I SL8500G1MC  query vol VON102 => 0,VON102      home              0, 5,21, 4, 0         LTO-800G
00:00:01 stkensrv4n.fnal.gov 008137 enstore I SL8500G1MC  query drive 0,5,1,15 => 0,0,5,1,15 online          available              IBM-LTO4
00:00:01 stkensrv4n.fnal.gov 008137 enstore I SL8500G1MC  Dismount 14 ignored: Drive 0,5,1,15 is empty. Thought VOS188 was there =>. dismount VOLUME 0,5,1,15 force => ok,0,5,1,15 online          available              IBM-LTO4
00:00:01 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:00:01 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  PROCESSING listDrives
00:00:01 stkensrv4n.fnal.gov 005852 enstore I STKMC  PROCESSING listDrives
00:00:01 stkensrv4n.fnal.gov 005843 enstore I SL8500F1MC  PROCESSING listDrives
00:00:03 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:00:05 last message repeated 1 times
00:00:05 stkensrv0n.fnal.gov 023312 enstore I INFSRV  start listing all volumes (3)
00:00:07 southport.fnal.gov 015491 e898 I ENCP  Version: v3_6c  CVS $Revision: 1.822 $ <frozen>  OS: Linux 2.4.21-52.ELsmp i686 Release:  Scientific Linux release 3.0.5 (Fermi) LSB_VERSION="1.3" User: e898(3557)  Group: e898(5468)  Euser: e898(3557)  Egroup: e898(5468)  Library: CD-LTO4F1  Storage Group: miniboone  File Family: OpenRootTree  FF Wrapper: cpio_odc  FF Width: 1  Current working directory: southport.fnal.gov:/home/Nearline/admin  Command line: encp --delayed-dismount 10 /Nearline/data01/boone_0022365_0012.root.other /pnfs/miniboone/OpenRootTree/Other/run0022000_0022999/run0022360_0022369/
00:00:07 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:00:07 southport.fnal.gov 015491 e898 I ENCP  Sending /pnfs/miniboone/OpenRootTree/Other/run0022000_0022999/run0022360_0022369/boone_0022365_0012.root.other write request to LM: uninque_id: southport.fnal.gov-1257487207-15491-0 inputfile: /Nearline/data01/boone_0022365_0012.root.other outputfile: /pnfs/miniboone/OpenRootTree/Other/run0022000_0022999/run0022360_0022369/boone_0022365_0012.root.other
00:00:07 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  MSG_TYPE=ADD_TO_LMQ  request added to pending queue for storage group : miniboone
00:00:07 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  write Q'd /Nearline/data01/boone_0022365_0012.root.other (southport.fnal.gov-1257487207-15491-0) -> /pnfs/miniboone/OpenRootTree/Other/run0022000_0022999/run0022360_0022369/boone_0022365_0012.root.other : library=CD-LTO4F1 family=OpenRootTree requester:e898 volume_family:miniboone.OpenRootTree.cpio_odc
00:00:07 southport.fnal.gov 015491 e898 I ENCP  Waiting for mover to connect control socket at address: ('131.225.194.144', 45750)  elapsed=0.940sec
00:00:07 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV  Tape is ejected Thread media_thread
00:00:07 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV  dismounting VOS175 Thread media_thread
00:00:07 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  MC Overflow: 4 dismount VOS175 0,7,1,14
00:00:07 stkenmvr195a.fnal.gov 021479 root E LTO4_95MV  unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread
00:00:07 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV  Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread
00:00:08 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_109.mover', 'VON245'), ('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_95.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_95.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:00:09 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:00:09 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  write_to_hsm next work on vol=VOO436 mover=LTO4_111.mover requester:e898
00:00:09 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  HAVE_BOUND:sending write_to_hsm /pnfs/miniboone/OpenRootTree/Other/run0022000_0022999/run0022360_0022369/boone_0022365_0012.root.other to mover LTO4_111.mover ('131.225.13.106', 7871) DEL_DISM 10
00:00:09 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  MSG_TYPE=ADD_TO_LMQ  request added to work at movers queue for storage group : miniboone
00:00:09 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  HAVE_BOUND:sending {'vc': {'storage_group': 'miniboone', 'external_label': 'VOO436', 'wrapper': 'cpio_odc', 'file_family': 'OpenRootTree', 'library': 'CD-LTO4F1', 'address': ('131.225.13.59', 7502), 'volume_family': 'miniboone.OpenRootTree.cpio_odc', 'file_family_width': 1}, 'encp': {'agetime': 30, 'adminpri': 1, 'delpri': 1, 'basepri': 1000003, 'delayed_dismount': 10, 'curpri': 1000003}, 'file_size': 10610741L, 'mover': 'LTO4_111.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'mode_octal': '0100011', 'pnfsFilename': '/pnfs/miniboone/OpenRootTree/Other/run0022000_0022999/run0022360_0022369/boone_0022365_0012.root.other', 'uid': 3557, 'uname': 'e898', 'type': 'cpio_odc', 'mtime': 1257487207, 'rmajor': 0, 'machine': ('Linux', 'southport.fnal.gov', '2.4.21-52.ELsmp', '#1 SMP Fri Sep 28 13:31:57 CDT 2007', 'i686'), 'sanity_size': 65536, 'pstat': (16872, 488647240L, 10L, 1, 3557, 5468, 512L, 1257485908, 1257485908, 1257355692), 'rminor': 0, 'mode': 33188, 'size_bytes': 10610741L, 'gname': 'e898', 'gid': 5468, 'fullname': '/Nearline/data01/boone_0022365_0012.root.other', 'inode': 488673448L, 'minor': 10}, 'version': 'v3_6c  CVS $Revision: 1.822 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'infile': '/Nearline/data01/boone_0022365_0012.root.other', 'status': ('ok', None), 'outfile': '/pnfs/miniboone/OpenRootTree/Other/run0022000_0022999/run0022360_0022369/boone_0022365_0012.root.other', 'fc': {'size': 10610741L, 'external_label': 'VOO436', 'pnfsid': '001D000000000000052090A8', 'address': ('131.225.13.59', 7501)}, 'at_the_top': 1, 'local_inode': 3712L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.194.144', 45750), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487209.430197, 'encp_start_time': 1257487206.6692209, 'job_queued': 1257487207.5719731, 't0': 1257487206, 'in_queue': 1.8581688404083252}, 'unique_id': 'southport.fnal.gov-1257487207-15491-0'} to mover
00:00:09 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:00:09 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  WRITE_TO_HSM Thread MainThread
00:00:09 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:00:09 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  SENDING {'vc': {'storage_group': 'miniboone', 'external_label': 'VOO436', 'wrapper': 'cpio_odc', 'file_family': 'OpenRootTree', 'library': 'CD-LTO4F1', 'address': ('131.225.13.59', 7502), 'volume_family': 'miniboone.OpenRootTree.cpio_odc', 'file_family_width': 1}, 'encp': {'agetime': 30, 'adminpri': 1, 'delpri': 1, 'basepri': 1000003, 'delayed_dismount': 10, 'curpri': 1000003}, 'file_size': 10610741L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4F1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7871, 'mc_device': '1,0,1,8', 'compression': 0, 'serial_num': '1310051081', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr211a', 'max_time_in_state': 1200, 'device': 'stkenmvr211a:/dev/rmt/tps0d0n', 'name': 'LTO4_111.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_111.stat', 'mover_address': ('131.225.13.106', 7871), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.106', 43729), 'data_ip': 'stkenmvr211a', 'logname': 'LTO4_111MV', 'hostip': '131.225.13.106', 'do_cleaning': 'No', 'media_changer': 'SL8500F1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'mode_octal': '0100011', 'pnfsFilename': '/pnfs/miniboone/OpenRootTree/Other/run0022000_0022999/run0022360_0022369/boone_0022365_0012.root.other', 'uid': 3557, 'uname': 'e898', 'type': 'cpio_odc', 'rmajor': 0, 'rminor': 0, 'machine': ('Linux', 'southport.fnal.gov', '2.4.21-52.ELsmp', '#1 SMP Fri Sep 28 13:31:57 CDT 2007', 'i686'), 'sanity_size': 65536, 'pstat': (16872, 488647240L, 10L, 1, 3557, 5468, 512L, 1257485908, 1257485908, 1257355692), 'gid': 5468, 'mtime': 1257487207, 'mode': 33188, 'gname': 'e898', 'size_bytes': 10610741L, 'fullname': '/Nearline/data01/boone_0022365_0012.root.other', 'inode': 488673448L, 'minor': 10}, 'version': 'v3_6c  CVS $Revision: 1.822 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'infile': '/Nearline/data01/boone_0022365_0012.root.other', 'status': ('ok', None), 'outfile': '/pnfs/miniboone/OpenRootTree/Other/run0022000_0022999/run0022360_0022369/boone_0022365_0012.root.other', 'fc': {'size': 10610741L, 'external_label': 'VOO436', 'pnfsid': '001D000000000000052090A8', 'address': ('131.225.13.59', 7501)}, 'at_the_top': 1, 'local_inode': 3712L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.194.144', 45750), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487209.430197, 'encp_start_time': 1257487206.6692209, 'job_queued': 1257487207.5719731, 't0': 1257487206, 'in_queue': 1.8581688404083252}, 'unique_id': 'southport.fnal.gov-1257487207-15491-0'} Thread client_connect_thread
00:00:09 southport.fnal.gov 015491 e898 I ENCP  Control socket ('131.225.194.144', 45750) is connected to ('131.225.13.106', 43730) for southport.fnal.gov-1257487207-15491-0.  elapsed=2.810sec
00:00:09 southport.fnal.gov 015491 e898 I ENCP  Received callback ticket from mover LTO4_111.mover for transfer southport.fnal.gov-1257487207-15491-0.  elapsed=2.810sec
00:00:09 southport.fnal.gov 015491 e898 I ENCP  Atempting to connect data socket to mover at ('131.225.13.106', 43729).
00:00:09 southport.fnal.gov 015491 e898 I ENCP  Connecting data socket to mover (('131.225.13.106', 43729)) with interface 131.225.194.144.  elapsed=2.810sec
00:00:09 southport.fnal.gov 015491 e898 I ENCP  Data socket ('131.225.194.144', 45751) is connected to ('131.225.13.106', 43729).  elapsed=2.810sec
00:00:09 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  tape position: target 26223 current 26223 0 Thread seek_thread
00:00:09 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  write_tape starting, bytes_to_write=10611200 Thread tape_thread
00:00:09 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  Write starting. Tape VOO436 absolute location in blocks 2830050 Thread tape_thread
00:00:09 southport.fnal.gov 015491 e898 I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = 0
00:00:09 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  MC Overflow: 4 dismount VON245 0,5,1,7
00:00:09 stkenmvr209a.fnal.gov 006118 root E LTO4_109MV  unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread
00:00:09 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread
00:00:10 stkensrv4n.fnal.gov 008130 enstore I SL8500G1MC  query drive 0,6,1,13 => 0,0,6,1,13 online          available              IBM-LTO4
00:00:10 southport.fnal.gov 015491 e898 I ENCP  File /Nearline/data01/boone_0022365_0012.root.other transfered for southport.fnal.gov-1257487207-15491-0 in 0.921363830566 seconds.
00:00:10 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  written bytes 10611200/10611200, blocks 81 header 178 trailer 281 Thread tape_thread
00:00:10 stkensrv4n.fnal.gov 008114 enstore I SL8500G1MC  dismount VOLUME 0,6,1,0 force => 0,Dismount: Forced dismount of VON277 from   0, 6, 1, 0
00:00:10 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  FINISHED dismount VON277 0,6,1,0 returned ('ok', 0, 'dismount VOLUME 0,6,1,0 force => 0,Dismount: Forced dismount of VON277 from   0, 6, 1, 0', '', '')
00:00:10 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  dismounted VON277 ULTRIUM-TD4 00:00:10 Thread media_thread
00:00:11 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Assigning  volume VON277 from storage group exp-db.daily-d0-offline.cpio_odc to library CD-LTO4G1, volume family exp-db.daily-d0-offline.cpio_odc
00:00:11 stkensrv4n.fnal.gov 008137 enstore I SL8500G1MC  query vol VOS188 => 0,VOS188      home              0, 7,38, 9, 1         LTO-800G
00:00:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  write_to_hsm work on vol=VON277 mover=LTO4_47.mover requester:root
00:00:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  MSG_TYPE=ADD_TO_LMQ  request added to work at movers queue for storage group : exp-db
00:00:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  IDLE:sending {'vc': {'storage_group': 'exp-db', 'library': 'CD-LTO4G1', 'file_family': 'daily-d0-offline', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'file_family_width': 2}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702152110_s103297_p4', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': 'LTO4_47.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702152110_s103297_p4', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257487130, 'rmajor': 0, 'machine': ('Linux', 'stkendca20a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (16877, 820619448L, 20L, 1, 1602, 2752, 512L, 1257487118, 1257487118, 1254398418), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 1966080000L, 'fullname': '/diskb/write-pool-2/data/003000000000000006208498', 'inode': 807437464L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskb/write-pool-2/data/003000000000000006208498', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/.(access)(003000000000000006208498)', 'fc': {'size': 1966080000L, 'external_label': 'VON277', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diskb/write-pool-2/data/003000000000000006208498', 'at_the_top': 275, 'local_inode': 268812423L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.12', 44946), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487211.4352219, 'encp_start_time': 1257487129.2778189, 'job_queued': 1257487130.281184, 't0': 1257487129, 'in_queue': 81.153573036193848}, 'user_level': 'user2', 'unique_id': 'stkendca20a.fnal.gov-1257487130-25712-0'} to mover
00:00:11 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  WRITE_TO_HSM Thread MainThread
00:00:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  can not remove from sg LTO4_47.mover VON277
00:00:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  SG: {'exp-db': [('LTO4_109.mover', 'VON245'), ('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_95.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]}
00:00:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  can not remove from vf LTO4_47.mover VON277
00:00:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  VF: {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_95.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]}
00:00:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_109.mover', 'VON245'), ('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_95.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_95.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:00:11 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  SENDING {'vc': {'storage_group': 'exp-db', 'library': 'CD-LTO4G1', 'file_family': 'daily-d0-offline', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'file_family_width': 2}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702152110_s103297_p4', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7847, 'mc_device': '0,6,1,0', 'compression': 0, 'serial_num': '1310154418', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr147a', 'max_time_in_state': 1200, 'device': 'stkenmvr147a:/dev/rmt/tps0d0n', 'name': 'LTO4_47.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_47.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.175', 7847), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.175', 59772), 'data_ip': 'stkenmvr147a', 'logname': 'LTO4_47MV', 'hostip': '131.225.13.175', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702152110_s103297_p4', 'uid': 0, 'uname': 'root', 'mtime': 1257487130, 'rmajor': 0, 'machine': ('Linux', 'stkendca20a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 820619448L, 20L, 1, 1602, 2752, 512L, 1257487118, 1257487118, 1254398418), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 1966080000L, 'gname': 'root', 'gid': 0, 'fullname': '/diskb/write-pool-2/data/003000000000000006208498', 'inode': 807437464L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskb/write-pool-2/data/003000000000000006208498', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/.(access)(003000000000000006208498)', 'fc': {'size': 1966080000L, 'external_label': 'VON277', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diskb/write-pool-2/data/003000000000000006208498', 'at_the_top': 275, 'local_inode': 268812423L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.12', 44946), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487211.4352219, 'encp_start_time': 1257487129.2778189, 'job_queued': 1257487130.281184, 't0': 1257487129, 'in_queue': 81.153573036193848}, 'user_level': 'user2', 'unique_id': 'stkendca20a.fnal.gov-1257487130-25712-0'} Thread client_connect_thread
00:00:11 stkendca20a.fnal.gov 025712 root I ENCP  Control socket ('131.225.13.12', 44946) is connected to ('131.225.13.175', 59773) for stkendca20a.fnal.gov-1257487130-25712-0.  elapsed=82.210sec
00:00:11 stkendca20a.fnal.gov 025712 root I ENCP  Received callback ticket from mover LTO4_47.mover for transfer stkendca20a.fnal.gov-1257487130-25712-0.  elapsed=82.220sec
00:00:11 stkendca20a.fnal.gov 025712 root I ENCP  Atempting to connect data socket to mover at ('131.225.13.175', 59772).
00:00:11 stkendca20a.fnal.gov 025712 root I ENCP  Connecting data socket to mover (('131.225.13.175', 59772)) with interface 131.225.13.12.  elapsed=82.220sec
00:00:11 stkendca20a.fnal.gov 025712 root I ENCP  Data socket ('131.225.13.12', 44948) is connected to ('131.225.13.175', 59772).  elapsed=82.220sec
00:00:11 stkendca20a.fnal.gov 025712 root I ENCP  Opening local file.
00:00:11 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  FINISHED listDrives returned ('ok', 0, None, '', '')
00:00:11 stkensrv4n.fnal.gov 003689 enstore E SL8500G1MC  dispatching_worker: no ticket!!!
00:00:11 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  MSG_TYPE=MC_LOAD_REQ  mounting VON277 ULTRIUM-TD4 00:00:11 Thread media_thread
00:00:11 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  PROCESSING mount VON277 0,6,1,0
00:00:11 stkendca20a.fnal.gov 025712 root I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = False
00:00:12 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_109.mover', 'VON245'), ('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_95.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_95.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:00:12 stkensrv4n.fnal.gov 005852 enstore I STKMC  FINISHED listDrives returned ('ok', 0, None, '', '')
00:00:12 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  filemarks written. Tape VOO436 absolute location in blocks 2830132 Thread tape_thread
00:00:12 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  write_tape timing:pull 0.000860452651978 write 0.0220279693604 crc_check 0.0493023395538 freespace 0.000986576080322 block_write 0.0731773376465 idle 0.00150895118713 Thread tape_thread
00:00:12 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  drive stats after write. Tape VOO436 position 26223 block 0 block_size 0 bloc_loc 2830132 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 10611200 block_write_tot 0.0731773376465 tape_rate 145006641.964 Thread tape_thread
00:00:12 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  remaining bytes info in DB 407336960000 estimated 407326348800 reported from drive 407325696000 position 26224 Thread tape_thread
00:00:12 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  new bitfile request {'uid': 3557, 'external_label': 'VOO436', 'gid': 5468, 'location_cookie': '0000_000000000_0026223', 'complete_crc': 147077490L, 'sanity_cookie': (65536L, 2780654618L), 'size': 10610741L} Thread tape_thread
00:00:12 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  set remaining: VOO436 407325696000 0000_000000000_0026224 Thread tape_thread
00:00:12 last message repeated 1 times
00:00:12 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set last_access = '2009-11-06 00:00:12', eod_cookie = '0000_000000000_0026224', non_del_files = '26907', remaining_bytes = 407325696000 where label = 'VOO436';
00:00:12 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  transfer complete volume=VOO436 location=26224 Thread tape_thread
00:00:12 last message repeated 1 times
00:00:12 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_wr_access = '26224', modification_time = '2009-11-06 00:00:12' where label = 'VOO436';
00:00:12 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  Sending done to client: {'vc': {'comment': '', 'storage_group': 'miniboone', 'blocksize': 131072, 'declared': 1237240176.0, 'library': 'CD-LTO4F1', 'si_time': [1251869136.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOO436', 'wrapper': 'cpio_odc', 'remaining_bytes': 407336960000L, 'sum_mounts': 2982, 'sum_rd_access': 0, 'file_family_width': 1, 'last_access': 1257487158.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0026223', 'capacity_bytes': 858993459200L, 'non_del_files': 26906, 'sum_wr_err': 1, 'sum_wr_access': 26223, 'file_family': 'OpenRootTree', 'address': ('131.225.13.59', 7502), 'volume_family': 'miniboone.OpenRootTree.cpio_odc', 'modification_time': 1257487158.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1241646225.0}, 'encp': {'agetime': 30, 'adminpri': 1, 'delpri': 1, 'basepri': 1000003, 'delayed_dismount': 10, 'curpri': 1000003}, 'file_size': 10610741L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4F1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7871, 'mc_device': '1,0,1,8', 'compression': 0, 'serial_num': '1310051081', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr211a', 'max_time_in_state': 1200, 'device': 'stkenmvr211a:/dev/rmt/tps0d0n', 'name': 'LTO4_111.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_111.stat', 'mover_address': ('131.225.13.106', 7871), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.106', 43729), 'data_ip': 'stkenmvr211a', 'logname': 'LTO4_111MV', 'hostip': '131.225.13.106', 'do_cleaning': 'No', 'media_changer': 'SL8500F1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'mode_octal': '0100011', 'pnfsFilename': '/pnfs/miniboone/OpenRootTree/Other/run0022000_0022999/run0022360_0022369/boone_0022365_0012.root.other', 'uid': 3557, 'uname': 'e898', 'type': 'cpio_odc', 'rmajor': 0, 'rminor': 0, 'machine': ('Linux', 'southport.fnal.gov', '2.4.21-52.ELsmp', '#1 SMP Fri Sep 28 13:31:57 CDT 2007', 'i686'), 'sanity_size': 65536, 'pstat': (16872, 488647240L, 10L, 1, 3557, 5468, 512L, 1257485908, 1257485908, 1257355692), 'gid': 5468, 'mtime': 1257487207, 'mode': 33188, 'gname': 'e898', 'size_bytes': 10610741L, 'fullname': '/Nearline/data01/boone_0022365_0012.root.other', 'inode': 488673448L, 'minor': 10}, 'version': 'v3_6c  CVS $Revision: 1.822 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'infile': '/Nearline/data01/boone_0022365_0012.root.other', 'status': ('ok', None), 'outfile': '/pnfs/miniboone/OpenRootTree/Other/run0022000_0022999/run0022360_0022369/boone_0022365_0012.root.other', 'fc': {'uid': 3557, 'external_label': 'VOO436', 'gid': 5468, 'location_cookie': '0000_000000000_0026223', 'complete_crc': 147077490L, 'bfid': 'CDMS125748721200000', 'sanity_cookie': (65536L, 2780654618L), 'size': 10610741L}, 'at_the_top': 1, 'local_inode': 3712L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.194.144', 45750), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257487206.6692209, 'seek_time': 0.034826993942260742, 'transfer_time': 3.2116708755493164, 't0': 1257487206, 'mount_time': 0.046323060989379883, 'lm_dequeued': 1257487209.430197, 'job_queued': 1257487207.5719731, 'drive_call_time': 0.020891427993774414, 'in_queue': 1.8581688404083252, 'drive_transfer_time': 2.0946140289306641}, 'unique_id': 'southport.fnal.gov-1257487207-15491-0'} Thread tape_thread
00:00:12 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  Done is sent Thread tape_thread
00:00:12 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  maybe_clean Thread tape_thread
00:00:12 southport.fnal.gov 015491 e898 I ENCP  Received final dialog for southport.fnal.gov-1257487207-15491-0.  elapsed=6.200sec
00:00:12 southport.fnal.gov 015491 e898 I ENCP  Mover CRC: 147077490  ENCP CRC: 147077490
00:00:12 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  returned from maybe_clean Thread tape_thread
00:00:12 southport.fnal.gov 015491 e898 I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = 0
00:00:13 stkensrv0n.fnal.gov 028796 enstore I FILSRV  Updating  			update file set deleted = 'n', pnfs_path = '/pnfs/miniboone/OpenRootTree/Other/run0022000_0022999/run0022360_0022369/boone_0022365_0012.root.other', drive = 'stkenmvr211a:/dev/rmt/tps0d0n:1310051081', pnfs_id = '001D000000000000052090A8' where bfid = 'CDMS125748721200000';
00:00:13 southport.fnal.gov 015491 e898 I ENCP  MSG_TYPE=ENCP_XFER    write_to_hsm /Nearline/data01/boone_0022365_0012.root.other -> /pnfs/miniboone/OpenRootTree/Other/run0022000_0022999/run0022360_0022369/boone_0022365_0012.root.other: 10610741 bytes copied to VOO436 at 3.11 MB/S (11 MB/S network) (4.83 MB/S drive) (11 MB/S disk) mover=LTO4_111.mover drive_id=ULTRIUM-TD4 drive_sn=1310051081 drive_vendor=IBM elapsed=6.526 {'storage_group': 'miniboone', 'encp_crc': 147077490L, 'disk_rate': 10.985961564708935, 'drive_rate': 4.8310528956519914, 'driver': 'FTTDriver', 'transfer_rate': 3.106903468794088, 'mover_interface': 'stkenmvr211a', 'overall_rate': 2.7218437266749183, 'unique_id': 'southport.fnal.gov-1257487207-15491-0', 'network_rate': 3.106903468794088, 'encp_ip': '131.225.194.144', 'media_changer': 'SL8500F1.media_changer'}
00:00:13 southport.fnal.gov 015491 e898 I ENCP  Completed transferring 10610741 bytes in 1 files in 6.52676415443 sec.  Overall rate = 2.72 MB/sec.  Transfer rate = 3.11 MB/sec.  Network rate = 11 MB/sec.  Drive rate = 4.83 MB/sec.  Disk rate = 11 MB/sec.  Exit status = 0.
00:00:13 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:00:15 last message repeated 1 times
00:00:15 stkensrv4n.fnal.gov 008137 enstore I SL8500G1MC  query drive 0,5,1,15 => 0,0,5,1,15 online          available              IBM-LTO4
00:00:15 stkensrv4n.fnal.gov 008483 enstore I SL8500G1MC  query drive 0,6,1,0 => 0,0,6,1,0 online          available              IBM-LTO4
00:00:15 stkensrv4n.fnal.gov 008483 enstore I SL8500G1MC  Dismount 14 ignored: Drive 0,6,1,0 is empty. Thought VON277 was there =>. dismount VOLUME 0,6,1,0 force => ok,0,6,1,0 online          available              IBM-LTO4
00:00:15 stkensrv0n.fnal.gov 023312 enstore I INFSRV  stop listing all volumes (3)
00:00:17 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:00:17 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245')], 'cms': [('LTO4_95.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_95.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:00:17 stkensrv4n.fnal.gov 005843 enstore I SL8500F1MC  FINISHED listDrives returned ('ok', 0, None, '', '')
00:00:17 stkendca20a.fnal.gov 026341 root I ENCP  Version: v3_8b  CVS $Revision: 1.922 $ <frozen>  OS: Linux 2.6.9-89.0.9.ELsmp i686 Release:  Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)  Library: CD-LTO4G1  Storage Group: exp-db  File Family: daily-d0-offline  FF Wrapper: cpio_odc  FF Width: 2  Current working directory: stkendca20a.fnal.gov:/home/enstore/dcache-code/dcache-fermi-config  Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702154603_s103300_p15 --put-cache 003000000000000006208558 /diska/write-pool-1/data/003000000000000006208558
00:00:17 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  MC Overflow: 4 dismount VOS175 0,7,1,14
00:00:17 stkenmvr195a.fnal.gov 021479 root E LTO4_95MV  unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread
00:00:17 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV  Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread
00:00:18 stkendca20a.fnal.gov 026341 root I ENCP  Creating zero length output files.
00:00:18 stkendca20a.fnal.gov 026341 root I ENCP  Sending /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702154603_s103300_p15 write request to LM: unique_id: stkendca20a.fnal.gov-1257487218-26341-0 inputfile: /diska/write-pool-1/data/003000000000000006208558 outputfile: /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/.(access)(003000000000000006208558)
00:00:18 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  MSG_TYPE=ADD_TO_LMQ  request added to pending queue for storage group : exp-db
00:00:18 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  write Q'd /diska/write-pool-1/data/003000000000000006208558 (stkendca20a.fnal.gov-1257487218-26341-0) -> /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702154603_s103300_p15 : library=CD-LTO4G1 family=daily-d0-offline requester:root volume_family:exp-db.daily-d0-offline.cpio_odc
00:00:18 stkendca20a.fnal.gov 026341 root I ENCP  Listening for control socket at: ('131.225.13.12', 44951)
00:00:18 stkendca20a.fnal.gov 026341 root I ENCP  Waiting for mover to connect control socket at address: ('131.225.13.12', 44951)  elapsed=1.050sec
00:00:18 stkensrv4n.fnal.gov 008483 enstore I SL8500G1MC  query vol VON277 => 0,VON277      home              0, 7, 3,24, 0         LTO-800G
00:00:19 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:00:19 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  MC Overflow: 4 dismount VON245 0,5,1,7
00:00:19 stkenmvr209a.fnal.gov 006118 root E LTO4_109MV  unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread
00:00:19 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread
00:00:21 stkensrv4n.fnal.gov 008483 enstore I SL8500G1MC  query drive 0,6,1,0 => 0,0,6,1,0 online          available              IBM-LTO4
00:00:21 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:00:25 last message repeated 1 times
00:00:25 stkendm7a.fnal.gov 009044 root I MIGRATION  File /data/data1/Migration_Spool/VO5178:0000_000000000_0000113 transfered for stkendm7a.fnal.gov-1257486991-9044-78 in 120.730132103 seconds. Thread WRITE
00:00:25 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:00:27 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  MC Overflow: 4 dismount VOS175 0,7,1,14
00:00:27 stkenmvr195a.fnal.gov 021479 root E LTO4_95MV  unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread
00:00:27 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV  Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread
00:00:29 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:00:29 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  MC Overflow: 4 dismount VON245 0,5,1,7
00:00:29 stkenmvr209a.fnal.gov 006118 root E LTO4_109MV  unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread
00:00:29 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread
00:00:30 stkendca22a.fnal.gov 019783 root I ENCP  Version: v3_8b  CVS $Revision: 1.922 $ <frozen>  OS: Linux 2.6.9-89.0.9.ELsmp i686 Release:  Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)  Library: garbage  Storage Group: test  File Family: dcache  FF Wrapper: cpio_odc  FF Width: 1  Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log  Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0003 --put-cache 0005000000000000005FA5F8 /diskb/write-pool-2/data/0005000000000000005FA5F8
00:00:30 stkendca22a.fnal.gov 019857 root I ENCP  Version: v3_8b  CVS $Revision: 1.922 $ <frozen>  OS: Linux 2.6.9-89.0.9.ELsmp i686 Release:  Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)  Library: garbage  Storage Group: test  File Family: dcache  FF Wrapper: cpio_odc  FF Width: 1  Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log  Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0002 --put-cache 0005000000000000005FA5C8 /diskb/write-pool-2/data/0005000000000000005FA5C8
00:00:30 stkendca22a.fnal.gov 019868 root I ENCP  Version: v3_8b  CVS $Revision: 1.922 $ <frozen>  OS: Linux 2.6.9-89.0.9.ELsmp i686 Release:  Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)  Library: garbage  Storage Group: test  File Family: dcache  FF Wrapper: cpio_odc  FF Width: 1  Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log  Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0001 --put-cache 0005000000000000005FA590 /diskb/write-pool-2/data/0005000000000000005FA590
00:00:30 stkendca22a.fnal.gov 019783 root I ENCP  Configuration Server: no such name: 'garbage.library_manager'
00:00:30 stkendca22a.fnal.gov 019783 root E ENCP  INFILE=/diskb/write-pool-2/data/0005000000000000005FA5F8 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0003 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=0.87 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager'
00:00:30 stkendca22a.fnal.gov 019857 root I ENCP  Configuration Server: no such name: 'garbage.library_manager'
00:00:30 stkendca22a.fnal.gov 019857 root E ENCP  INFILE=/diskb/write-pool-2/data/0005000000000000005FA5C8 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0002 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=0.91 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager'
00:00:30 stkendca22a.fnal.gov 019868 root I ENCP  Configuration Server: no such name: 'garbage.library_manager'
00:00:30 stkendca22a.fnal.gov 019868 root E ENCP  INFILE=/diskb/write-pool-2/data/0005000000000000005FA590 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0001 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=0.97 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager'
00:00:31 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:00:32 stkensrv4n.fnal.gov 005357 enstore E LTO4G1LM  MOVER ERROR RQ {'status': ('ok', None), 'returned_work': None, 'library': None, 'mover_type': '', 'current_location': '0000_000000000_0000000', 'ip_map': '', 'operation': None, 'read_only': 0, 'current_time': 1257487232.3667221, 'error_source': None, 'external_label': None, 'state': 'OFFLINE', 'r_a': (('131.225.13.194', 54880), 18006L, '131.225.13.194-54880-1256586929.509013-4866'), 'library_list': [('CD-LTO4G1.library_manager', ('131.225.13.4', 8541))], 'current_priority': (None, None), 'transfer_deficiency': 1, 'time_in_state': 900302.0066640377, 'address': ('131.225.13.194', 7860), 'volume_family': None, 'work': 'mover_error', 'mover': 'LTO4_99.mover', 'volume_status': (['none', 'none'], ['none', 'none']), 'unique_id': None}
00:00:33 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:00:35 last message repeated 1 times
00:00:35 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  written bytes 5826241024/5826241024, blocks 44451 header 0 trailer 384 Thread tape_thread
00:00:36 stkensrv4n.fnal.gov 008125 enstore I SL8500G1MC  mount VON268 0,7,1,0 => 0,Mount: VON268 mounted on   0, 7, 1, 0
00:00:37 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  FINISHED mount VON268 0,7,1,0 returned ('ok', 0, 'mount VON268 0,7,1,0 => 0,Mount: VON268 mounted on   0, 7, 1, 0', '', '')
00:00:37 last message repeated 1 times
00:00:37 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_mounts = '150', last_access = '2009-11-06 00:00:37' where label = 'VON268';
00:00:37 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  MSG_TYPE=MC_LOAD_DONE  mounted VON268 ULTRIUM-TD4 00:00:37 Thread media_thread
00:00:37 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:00:37 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  PROCESSING dismount VOS175 0,7,1,14
00:00:39 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:00:39 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  MC Overflow: 4 dismount VON245 0,5,1,7
00:00:39 stkenmvr209a.fnal.gov 006118 root E LTO4_109MV  unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread
00:00:39 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread
00:00:40 stkensrv4n.fnal.gov 008793 enstore I SL8500G1MC  query drive 0,7,1,14 => 0,0,7,1,14 online          in use      VOS175     IBM-LTO4
00:00:40 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  filemarks written. Tape VOV147 absolute location in blocks 3446765 Thread tape_thread
00:00:40 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  write_tape timing:pull 0.601916074753 write 19.0309467316 crc_check 26.8396091461 freespace 0.46405339241 block_write 46.9365253448 idle 0.806867361069 Thread tape_thread
00:00:40 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  drive stats after write. Tape VOV147 position 7414 block 0 block_size 0 bloc_loc 3446765 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 5826241024 block_write_tot 46.9365253448 tape_rate 124130215.886 Thread tape_thread
00:00:40 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  remaining bytes info in DB 343602176000 estimated 337775934976 reported from drive 338154496000 position 7417 Thread tape_thread
00:00:40 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  new bitfile request {'uid': 0, 'external_label': 'VOV147', 'gid': 0, 'location_cookie': '0000_000000000_0007415', 'complete_crc': 1772614942L, 'sanity_cookie': (65536L, 848733646L), 'size': 5826240640L} Thread tape_thread
00:00:40 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  set remaining: VOV147 338154496000 0000_000000000_0007417 Thread tape_thread
00:00:40 last message repeated 1 times
00:00:40 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set last_access = '2009-11-06 00:00:40', eod_cookie = '0000_000000000_0007417', non_del_files = '2474', remaining_bytes = 338154496000 where label = 'VOV147';
00:00:40 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  transfer complete volume=VOV147 location=7417 Thread tape_thread
00:00:40 last message repeated 1 times
00:00:40 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_wr_access = '2472', modification_time = '2009-11-06 00:00:40' where label = 'VOV147';
00:00:40 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  Sending done to client: {'vc': {'comment': '', 'storage_group': 'ktev', 'blocksize': 131072, 'declared': 1254165878.0, 'library': 'CD-LTO4F1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOV147', 'wrapper': 'cern', 'remaining_bytes': 343602176000L, 'sum_mounts': 5, 'sum_rd_access': 0, 'file_family_width': 1, 'last_access': 1257487104.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0007414', 'capacity_bytes': 858993459200L, 'non_del_files': 2473, 'sum_wr_err': 0, 'sum_wr_access': 2471, 'file_family': 'ktev_migration-MIGRATION', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'modification_time': 1257487104.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257458755.0}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC2/KZC267/f1', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 5826240640L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4F1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7873, 'mc_device': '1,1,1,4', 'compression': 0, 'serial_num': '1310156224', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr213a', 'max_time_in_state': 1200, 'device': 'stkenmvr213a:/dev/rmt/tps0d0n', 'name': 'LTO4_113.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_113.stat', 'mover_address': ('131.225.13.108', 7873), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.108', 43827), 'data_ip': 'stkenmvr213a', 'logname': 'LTO4_113MV', 'hostip': '131.225.13.108', 'do_cleaning': 'No', 'media_changer': 'SL8500F1.media_changer'}, 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC2/KZC267/f1', 'uid': 0, 'uname': 'root', 'mtime': 1257486991, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cern', 'pstat': (16877, 674895104L, 20L, 1, 0, 0, 512L, 1257175805, 1257175805, 1257175652), 'mode_octal': '0100016', 'mode': 33204, 'size_bytes': 5826240640L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000113', 'inode': 675737800L, 'minor': 20}, 'version': 'v3_8  CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000113', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC2/KZC267/.(access)(00280000000000000846F0C8)', 'fc': {'uid': 0, 'external_label': 'VOV147', 'gid': 0, 'location_cookie': '0000_000000000_0007415', 'complete_crc': 1772614942L, 'bfid': 'CDMS125748724000000', 'sanity_cookie': (65536L, 848733646L), 'size': 5826240640L}, 'infilepath': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000113', 'at_the_top': 317, 'local_inode': 39503173L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 44355), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257486991.7283399, 'seek_time': 0.034382104873657227, 'transfer_time': 136.14157104492188, 't0': 1257486991, 'mount_time': 0.043227910995483398, 'lm_dequeued': 1257487104.383944, 'job_queued': 1257486992.03965, 'drive_call_time': 18.438944578170776, 'in_queue': 112.34403014183044, 'drive_transfer_time': 54.447163105010986}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257486991-9044-78'} Thread tape_thread
00:00:40 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  Done is sent Thread tape_thread
00:00:40 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  maybe_clean Thread tape_thread
00:00:40 stkendm7a.fnal.gov 009044 root I MIGRATION  Received final dialog for stkendm7a.fnal.gov-1257486991-9044-78.  elapsed=5678.020sec Thread WRITE
00:00:40 stkendm7a.fnal.gov 009044 root I MIGRATION  leaving transfer_file(). Thread WRITE
00:00:40 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  returned from maybe_clean Thread tape_thread
00:00:40 stkensrv0n.fnal.gov 028796 enstore I FILSRV  Updating  			update file set deleted = 'n', pnfs_path = '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC2/KZC267/f1', drive = 'stkenmvr213a:/dev/rmt/tps0d0n:1310156224', pnfs_id = '00280000000000000846F0C8' where bfid = 'CDMS125748724000000';
00:00:41 stkendm7a.fnal.gov 009044 root I MIGRATION  MSG_TYPE=ENCP_XFER    write_to_hsm /data/data1/Migration_Spool/VO5178:0000_000000000_0000113 -> /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC2/KZC267/f1: 5826240640 bytes copied to VOV147 at 40.8 MB/S (90.8 MB/S network) (102 MB/S drive) (50 MB/S disk) mover=LTO4_113.mover drive_id=ULTRIUM-TD4 drive_sn=1310156224 drive_vendor=IBM elapsed=249.39 {'storage_group': 'ktev', 'encp_crc': 1772614942L, 'disk_rate': 50.009311460939372, 'drive_rate': 102.05005628766986, 'driver': 'FTTDriver', 'transfer_rate': 40.811119581691536, 'mover_interface': 'stkenmvr213a', 'overall_rate': 40.650558737551009, 'media_changer': 'SL8500F1.media_changer', 'network_rate': 40.811119581691536, 'encp_ip': '131.225.13.19', 'unique_id': 'stkendm7a.fnal.gov-1257486991-9044-78'} Thread WRITE
00:00:41 stkendm7a.fnal.gov 009044 root I MIGRATION  File /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC2/KZC267/f1 copied successfully. Thread WRITE
00:00:41 stkendm7a.fnal.gov 009044 root I MIGRATION  Completed transferring 5826240640 bytes in 1 files in 249.390737057 sec.  Overall rate = 40.7 MB/sec.  Transfer rate = 40.8 MB/sec.  Network rate = 90.8 MB/sec.  Drive rate = 102 MB/sec.  Disk rate = 50 MB/sec.  Exit status = 0. Thread WRITE
00:00:41 stkensrv0n.fnal.gov 028796 enstore I FILSRV  start modifying {'pnfsid': '00280000000000000846F0C8', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC2/KZC267/f1', 'deleted': 'no', 'external_label': 'VOV147', 'drive': 'stkenmvr213a:/dev/rmt/tps0d0n:1310156224', 'update': '2009-11-06 00:00:40.801301', 'gid': 0, 'location_cookie': '0000_000000000_0007415', 'complete_crc': 1772614942L, 'uid': 0, 'bfid': 'CDMS125748724000000', 'sanity_cookie': (65536L, 848733646L), 'size': 5826240640L}
00:00:41 stkensrv0n.fnal.gov 028796 enstore I FILSRV  Updating  			update file set pnfs_id = '001F000000000000005665B8' where bfid = 'CDMS125748724000000';
00:00:41 stkensrv0n.fnal.gov 028796 enstore I FILSRV  modified to {'pnfsid': '001F000000000000005665B8', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC2/KZC267/f1', 'deleted': 'no', 'external_label': 'VOV147', 'drive': 'stkenmvr213a:/dev/rmt/tps0d0n:1310156224', 'update': '2009-11-06 00:00:40.801301', 'gid': 0, 'location_cookie': '0000_000000000_0007415', 'complete_crc': 1772614942L, 'uid': 0, 'bfid': 'CDMS125748724000000', 'sanity_cookie': (65536L, 848733646L), 'size': 5826240640L}
00:00:41 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:00:41 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  write_to_hsm next work on vol=VOV147 mover=LTO4_113.mover requester:root
00:00:41 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  HAVE_BOUND:sending write_to_hsm /pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f0 to mover LTO4_113.mover ('131.225.13.108', 7873) DEL_DISM 2
00:00:41 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  MSG_TYPE=ADD_TO_LMQ  request added to work at movers queue for storage group : ktev
00:00:41 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  HAVE_BOUND:sending {'vc': {'storage_group': 'ktev', 'external_label': 'VOV147', 'library': 'CD-LTO4F1', 'file_family': 'ktev_migration-MIGRATION', 'wrapper': 'cern', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'file_family_width': 1}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f0', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 1668757988L, 'mover': 'LTO4_113.mover', 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f0', 'uid': 0, 'uname': 'root', 'type': 'cern', 'gid': 0, 'mtime': 1257487050, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (16877, 683683784L, 20L, 1, 0, 0, 512L, 1255728736, 1255728736, 1255127721), 'mode_octal': '0100016', 'mode': 33204, 'gname': 'root', 'size_bytes': 1668757988L, 'fullname': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000092', 'inode': 675737848L, 'minor': 20}, 'version': 'v3_8  CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000092', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC099/.(access)(00280000000000000846F0F8)', 'fc': {'size': 1668757988L, 'external_label': 'VOV147', 'pnfsid': '00280000000000000846F0F8', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000092', 'at_the_top': 781, 'local_inode': 39503171L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 44361), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487241.696538, 'encp_start_time': 1257487050.553689, 'job_queued': 1257487050.866416, 't0': 1257487050, 'in_queue': 190.82983899116516}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257487050-9044-79'} to mover
00:00:41 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:00:41 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  WRITE_TO_HSM Thread MainThread
00:00:41 stkendm7a.fnal.gov 009044 root I ENCP  Listening for control socket at: ('131.225.13.19', 44361) Thread WRITE_2
00:00:41 stkendm7a.fnal.gov 009044 root I ENCP  Waiting for mover to connect control socket at address: ('131.225.13.19', 44361)  elapsed=5679.080sec Thread WRITE_2
00:00:41 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  SENDING {'vc': {'storage_group': 'ktev', 'external_label': 'VOV147', 'library': 'CD-LTO4F1', 'file_family': 'ktev_migration-MIGRATION', 'wrapper': 'cern', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'file_family_width': 1}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f0', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 1668757988L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4F1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7873, 'mc_device': '1,1,1,4', 'compression': 0, 'serial_num': '1310156224', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr213a', 'max_time_in_state': 1200, 'device': 'stkenmvr213a:/dev/rmt/tps0d0n', 'name': 'LTO4_113.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_113.stat', 'mover_address': ('131.225.13.108', 7873), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.108', 43835), 'data_ip': 'stkenmvr213a', 'logname': 'LTO4_113MV', 'hostip': '131.225.13.108', 'do_cleaning': 'No', 'media_changer': 'SL8500F1.media_changer'}, 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f0', 'uid': 0, 'uname': 'root', 'mtime': 1257487050, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cern', 'pstat': (16877, 683683784L, 20L, 1, 0, 0, 512L, 1255728736, 1255728736, 1255127721), 'mode_octal': '0100016', 'mode': 33204, 'size_bytes': 1668757988L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000092', 'inode': 675737848L, 'minor': 20}, 'version': 'v3_8  CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000092', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC099/.(access)(00280000000000000846F0F8)', 'fc': {'size': 1668757988L, 'external_label': 'VOV147', 'pnfsid': '00280000000000000846F0F8', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000092', 'at_the_top': 781, 'local_inode': 39503171L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 44361), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487241.696538, 'encp_start_time': 1257487050.553689, 'job_queued': 1257487050.866416, 't0': 1257487050, 'in_queue': 190.82983899116516}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257487050-9044-79'} Thread client_connect_thread
00:00:41 stkendm7a.fnal.gov 009044 root I ENCP  Control socket ('131.225.13.19', 44361) is connected to ('131.225.13.108', 43836) for stkendm7a.fnal.gov-1257487050-9044-79.  elapsed=5679.090sec Thread WRITE_2
00:00:41 stkendm7a.fnal.gov 009044 root I ENCP  Received callback ticket from mover LTO4_113.mover for transfer stkendm7a.fnal.gov-1257487050-9044-79.  elapsed=5679.090sec Thread WRITE_2
00:00:41 stkendm7a.fnal.gov 009044 root I ENCP  Attempting to connect data socket to mover at ('131.225.13.108', 43835). Thread WRITE_2
00:00:41 stkendm7a.fnal.gov 009044 root I ENCP  Connecting data socket to mover (('131.225.13.108', 43835)) with interface 131.225.13.19.  elapsed=5679.090sec Thread WRITE_2
00:00:41 stkendm7a.fnal.gov 009044 root I ENCP  Data socket ('131.225.13.19', 44369) is connected to ('131.225.13.108', 43835).  elapsed=5679.100sec Thread WRITE_2
00:00:41 stkendm7a.fnal.gov 009044 root I ENCP  Opening local file. Thread WRITE_2
00:00:41 stkendm7a.fnal.gov 009044 root I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = False Thread WRITE_2
00:00:41 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  tape position: target 7417 current 7417 0 Thread seek_thread
00:00:41 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  write_tape starting, bytes_to_write=1668758016 Thread tape_thread
00:00:41 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  Write starting. Tape VOV147 absolute location in blocks 3446765 Thread tape_thread
00:00:42 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:00:46 last message repeated 2 times
00:00:46 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245')], 'cms': [('LTO4_95.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_95.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} >
00:00:47 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:00:49 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  MC Overflow: 4 dismount VON245 0,5,1,7
00:00:49 stkenmvr209a.fnal.gov 006118 root E LTO4_109MV  unloadvol ('MEDIA_CHANGER_QUEUE_FULL', 0, 'retry later') Thread media_thread
00:00:49 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  Media Changer returned MEDIA_CHANGER_QUEUE_FULL Thread media_thread
00:00:51 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:00:53 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:00:54 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  tape position: target 283 current 0 1 Thread seek_thread
00:00:54 stkensrv4n.fnal.gov 008130 enstore I SL8500G1MC  mount VON102 0,6,1,13 => 0,Mount: VON102 mounted on   0, 6, 1,13
00:00:54 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  FINISHED mount VON102 0,6,1,13 returned ('ok', 0, 'mount VON102 0,6,1,13 => 0,Mount: VON102 mounted on   0, 6, 1,13', '', '')
00:00:54 last message repeated 1 times
00:00:54 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_mounts = '180', last_access = '2009-11-06 00:00:54' where label = 'VON102';
00:00:54 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV  MSG_TYPE=MC_LOAD_DONE  mounted VON102 ULTRIUM-TD4 00:00:54 Thread media_thread
00:00:55 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:00:56 stkendm7a.fnal.gov 009044 root I ENCP  Version: v3_8  CVS $Revision: 1.963 $ encp.pyc  OS: Linux 2.6.9-89.0.9.ELsmp i686 Release:  Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)  Library: CD-LTO4F1  Storage Group: ktev  File Family: ktev_migration-MIGRATION  FF Wrapper: cern  FF Width: 1  Current working directory: stkendm7a.fnal.gov:/opt/enstore/src  Command line: encp --delayed-dismount 2 --ignore-fair-share --threaded --priority 0 --storage-group ktev --file-family ktev_migration-MIGRATION --file-family-wrapper cern --library CD-LTO4F1 --override-path /pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f2 --threaded /data/data1/Migration_Spool/VO5178:0000_000000000_0000122 /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f2 Thread WRITE
00:00:56 stkendm7a.fnal.gov 009044 root I ENCP  Creating zero length output files. Thread WRITE
00:00:56 stkendm7a.fnal.gov 009044 root I ENCP  Sending /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f2 write request to LM: unique_id: stkendm7a.fnal.gov-1257487256-9044-81 inputfile: /data/data1/Migration_Spool/VO5178:0000_000000000_0000122 outputfile: /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC099/.(access)(00280000000000000846F1A0) Thread WRITE
00:00:56 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  MSG_TYPE=ADD_TO_LMQ  request added to pending queue for storage group : ktev
00:00:56 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  write Q'd /data/data1/Migration_Spool/VO5178:0000_000000000_0000122 (stkendm7a.fnal.gov-1257487256-9044-81) -> /pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f2 : library=CD-LTO4F1 family=ktev_migration-MIGRATION requester:root volume_family:ktev.ktev_migration-MIGRATION.cern
00:00:57 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:00:59 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  PROCESSING dismount VON245 0,5,1,7
00:01:01 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:02 stkensrv4n.fnal.gov 009007 enstore I SL8500G1MC  query drive 0,5,1,7 => 0,0,5,1,7 online          in use      VON245     IBM-LTO4
00:01:03 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:08 last message repeated 2 times
00:01:08 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_104.mover', 'VON102'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_43.mover', 'VON268')], 'cms': [('LTO4_95.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_95.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} >
00:01:09 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:09 southport.fnal.gov 016471 e898 I ENCP  Version: v3_6c  CVS $Revision: 1.822 $ <frozen>  OS: Linux 2.4.21-52.ELsmp i686 Release:  Scientific Linux release 3.0.5 (Fermi) LSB_VERSION="1.3" User: e898(3557)  Group: e898(5468)  Euser: e898(3557)  Egroup: e898(5468)  Library: CD-LTO4F1  Storage Group: miniboone  File Family: OpenRootTree  FF Wrapper: cpio_odc  FF Width: 1  Current working directory: southport.fnal.gov:/home/Nearline/admin  Command line: encp --delayed-dismount 10 /Nearline/data01/boone_0022365_0012.root.big_nu /pnfs/miniboone/OpenRootTree/Big_Nu/run0022000_0022999/run0022360_0022369/
00:01:10 southport.fnal.gov 016471 e898 I ENCP  Sending /pnfs/miniboone/OpenRootTree/Big_Nu/run0022000_0022999/run0022360_0022369/boone_0022365_0012.root.big_nu write request to LM: uninque_id: southport.fnal.gov-1257487270-16471-0 inputfile: /Nearline/data01/boone_0022365_0012.root.big_nu outputfile: /pnfs/miniboone/OpenRootTree/Big_Nu/run0022000_0022999/run0022360_0022369/boone_0022365_0012.root.big_nu
00:01:10 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  MSG_TYPE=ADD_TO_LMQ  request added to pending queue for storage group : miniboone
00:01:10 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  write Q'd /Nearline/data01/boone_0022365_0012.root.big_nu (southport.fnal.gov-1257487270-16471-0) -> /pnfs/miniboone/OpenRootTree/Big_Nu/run0022000_0022999/run0022360_0022369/boone_0022365_0012.root.big_nu : library=CD-LTO4F1 family=OpenRootTree requester:e898 volume_family:miniboone.OpenRootTree.cpio_odc
00:01:10 southport.fnal.gov 016471 e898 I ENCP  Waiting for mover to connect control socket at address: ('131.225.194.144', 45787)  elapsed=0.900sec
00:01:11 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:11 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  write_to_hsm next work on vol=VOO436 mover=LTO4_111.mover requester:e898
00:01:11 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  HAVE_BOUND:sending write_to_hsm /pnfs/miniboone/OpenRootTree/Big_Nu/run0022000_0022999/run0022360_0022369/boone_0022365_0012.root.big_nu to mover LTO4_111.mover ('131.225.13.106', 7871) DEL_DISM 10
00:01:11 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  MSG_TYPE=ADD_TO_LMQ  request added to work at movers queue for storage group : miniboone
00:01:11 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  HAVE_BOUND:sending {'vc': {'storage_group': 'miniboone', 'external_label': 'VOO436', 'wrapper': 'cpio_odc', 'file_family': 'OpenRootTree', 'library': 'CD-LTO4F1', 'address': ('131.225.13.59', 7502), 'volume_family': 'miniboone.OpenRootTree.cpio_odc', 'file_family_width': 1}, 'encp': {'agetime': 30, 'adminpri': 1, 'delpri': 1, 'basepri': 1000003, 'delayed_dismount': 10, 'curpri': 1000003}, 'file_size': 8869590L, 'mover': 'LTO4_111.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'mode_octal': '0100011', 'pnfsFilename': '/pnfs/miniboone/OpenRootTree/Big_Nu/run0022000_0022999/run0022360_0022369/boone_0022365_0012.root.big_nu', 'uid': 3557, 'uname': 'e898', 'type': 'cpio_odc', 'mtime': 1257487270, 'rmajor': 0, 'machine': ('Linux', 'southport.fnal.gov', '2.4.21-52.ELsmp', '#1 SMP Fri Sep 28 13:31:57 CDT 2007', 'i686'), 'sanity_size': 65536, 'pstat': (16872, 488647360L, 10L, 1, 3557, 5468, 512L, 1257485963, 1257485963, 1257355749), 'rminor': 0, 'mode': 33188, 'size_bytes': 8869590L, 'gname': 'e898', 'gid': 5468, 'fullname': '/Nearline/data01/boone_0022365_0012.root.big_nu', 'inode': 488673480L, 'minor': 10}, 'version': 'v3_6c  CVS $Revision: 1.822 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'infile': '/Nearline/data01/boone_0022365_0012.root.big_nu', 'status': ('ok', None), 'outfile': '/pnfs/miniboone/OpenRootTree/Big_Nu/run0022000_0022999/run0022360_0022369/boone_0022365_0012.root.big_nu', 'fc': {'size': 8869590L, 'external_label': 'VOO436', 'pnfsid': '001D000000000000052090C8', 'address': ('131.225.13.59', 7501)}, 'at_the_top': 4, 'local_inode': 3714L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.194.144', 45787), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487271.5347641, 'encp_start_time': 1257487269.646157, 'job_queued': 1257487270.504926, 't0': 1257487269, 'in_queue': 1.0297489166259766}, 'unique_id': 'southport.fnal.gov-1257487270-16471-0'} to mover
00:01:11 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:11 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  WRITE_TO_HSM Thread MainThread
00:01:11 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:11 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  SENDING {'vc': {'storage_group': 'miniboone', 'external_label': 'VOO436', 'wrapper': 'cpio_odc', 'file_family': 'OpenRootTree', 'library': 'CD-LTO4F1', 'address': ('131.225.13.59', 7502), 'volume_family': 'miniboone.OpenRootTree.cpio_odc', 'file_family_width': 1}, 'encp': {'agetime': 30, 'adminpri': 1, 'delpri': 1, 'basepri': 1000003, 'delayed_dismount': 10, 'curpri': 1000003}, 'file_size': 8869590L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4F1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7871, 'mc_device': '1,0,1,8', 'compression': 0, 'serial_num': '1310051081', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr211a', 'max_time_in_state': 1200, 'device': 'stkenmvr211a:/dev/rmt/tps0d0n', 'name': 'LTO4_111.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_111.stat', 'mover_address': ('131.225.13.106', 7871), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.106', 43736), 'data_ip': 'stkenmvr211a', 'logname': 'LTO4_111MV', 'hostip': '131.225.13.106', 'do_cleaning': 'No', 'media_changer': 'SL8500F1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'mode_octal': '0100011', 'pnfsFilename': '/pnfs/miniboone/OpenRootTree/Big_Nu/run0022000_0022999/run0022360_0022369/boone_0022365_0012.root.big_nu', 'uid': 3557, 'uname': 'e898', 'type': 'cpio_odc', 'rmajor': 0, 'rminor': 0, 'machine': ('Linux', 'southport.fnal.gov', '2.4.21-52.ELsmp', '#1 SMP Fri Sep 28 13:31:57 CDT 2007', 'i686'), 'sanity_size': 65536, 'pstat': (16872, 488647360L, 10L, 1, 3557, 5468, 512L, 1257485963, 1257485963, 1257355749), 'gid': 5468, 'mtime': 1257487270, 'mode': 33188, 'gname': 'e898', 'size_bytes': 8869590L, 'fullname': '/Nearline/data01/boone_0022365_0012.root.big_nu', 'inode': 488673480L, 'minor': 10}, 'version': 'v3_6c  CVS $Revision: 1.822 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'infile': '/Nearline/data01/boone_0022365_0012.root.big_nu', 'status': ('ok', None), 'outfile': '/pnfs/miniboone/OpenRootTree/Big_Nu/run0022000_0022999/run0022360_0022369/boone_0022365_0012.root.big_nu', 'fc': {'size': 8869590L, 'external_label': 'VOO436', 'pnfsid': '001D000000000000052090C8', 'address': ('131.225.13.59', 7501)}, 'at_the_top': 4, 'local_inode': 3714L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.194.144', 45787), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487271.5347641, 'encp_start_time': 1257487269.646157, 'job_queued': 1257487270.504926, 't0': 1257487269, 'in_queue': 1.0297489166259766}, 'unique_id': 'southport.fnal.gov-1257487270-16471-0'} Thread client_connect_thread
00:01:11 southport.fnal.gov 016471 e898 I ENCP  Control socket ('131.225.194.144', 45787) is connected to ('131.225.13.106', 43737) for southport.fnal.gov-1257487270-16471-0.  elapsed=1.940sec
00:01:11 southport.fnal.gov 016471 e898 I ENCP  Received callback ticket from mover LTO4_111.mover for transfer southport.fnal.gov-1257487270-16471-0.  elapsed=1.950sec
00:01:11 southport.fnal.gov 016471 e898 I ENCP  Atempting to connect data socket to mover at ('131.225.13.106', 43736).
00:01:11 southport.fnal.gov 016471 e898 I ENCP  Connecting data socket to mover (('131.225.13.106', 43736)) with interface 131.225.194.144.  elapsed=1.950sec
00:01:11 southport.fnal.gov 016471 e898 I ENCP  Data socket ('131.225.194.144', 45792) is connected to ('131.225.13.106', 43736).  elapsed=1.950sec
00:01:11 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  tape position: target 26224 current 26224 0 Thread seek_thread
00:01:11 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  write_tape starting, bytes_to_write=8869888 Thread tape_thread
00:01:11 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  Write starting. Tape VOO436 absolute location in blocks 2830132 Thread tape_thread
00:01:11 southport.fnal.gov 016471 e898 I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = 0
00:01:12 southport.fnal.gov 016471 e898 I ENCP  File /Nearline/data01/boone_0022365_0012.root.big_nu transfered for southport.fnal.gov-1257487270-16471-0 in 0.750121116638 seconds.
00:01:12 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  written bytes 8869888/8869888, blocks 68 header 180 trailer 118 Thread tape_thread
00:01:12 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV  tape position: target 394 current 0 1 Thread seek_thread
00:01:14 stkensrv4n.fnal.gov 008137 enstore I SL8500G1MC  mount VOS188 0,5,1,15 => 0,Mount: VOS188 mounted on   0, 5, 1,15
00:01:14 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  FINISHED mount VOS188 0,5,1,15 returned ('ok', 0, 'mount VOS188 0,5,1,15 => 0,Mount: VOS188 mounted on   0, 5, 1,15', '', '')
00:01:14 last message repeated 1 times
00:01:14 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_mounts = '435', last_access = '2009-11-06 00:01:14' where label = 'VOS188';
00:01:14 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV  MSG_TYPE=MC_LOAD_DONE  mounted VOS188 ULTRIUM-TD4 00:01:14 Thread media_thread
00:01:14 stkensrv4n.fnal.gov 009007 enstore I SL8500G1MC  dismount VOLUME 0,5,1,7 force => 0,Dismount: Forced dismount of VON245 from   0, 5, 1, 7
00:01:14 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  FINISHED dismount VON245 0,5,1,7 returned ('ok', 0, 'dismount VOLUME 0,5,1,7 force => 0,Dismount: Forced dismount of VON245 from   0, 5, 1, 7', '', '')
00:01:14 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  dismounted VON245 ULTRIUM-TD4 00:01:14 Thread media_thread
00:01:15 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  filemarks written. Tape VOO436 absolute location in blocks 2830201 Thread tape_thread
00:01:15 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  write_tape timing:pull 0.000541925430298 write 0.0195095539093 crc_check 0.0403571128845 freespace 0.000532150268555 block_write 0.0609407424927 idle 0.00106430053711 Thread tape_thread
00:01:15 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  drive stats after write. Tape VOO436 position 26224 block 0 block_size 0 bloc_loc 2830201 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 8869888 block_write_tot 0.0609407424927 tape_rate 145549391.707 Thread tape_thread
00:01:15 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  remaining bytes info in DB 407325696000 estimated 407316826112 reported from drive 407315456000 position 26225 Thread tape_thread
00:01:15 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  new bitfile request {'uid': 3557, 'external_label': 'VOO436', 'gid': 5468, 'location_cookie': '0000_000000000_0026224', 'complete_crc': 3452424606L, 'sanity_cookie': (65536L, 819419212L), 'size': 8869590L} Thread tape_thread
00:01:15 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  set remaining: VOO436 407315456000 0000_000000000_0026225 Thread tape_thread
00:01:15 last message repeated 1 times
00:01:15 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set last_access = '2009-11-06 00:01:15', eod_cookie = '0000_000000000_0026225', non_del_files = '26908', remaining_bytes = 407315456000 where label = 'VOO436';
00:01:15 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  transfer complete volume=VOO436 location=26225 Thread tape_thread
00:01:15 last message repeated 1 times
00:01:15 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_wr_access = '26225', modification_time = '2009-11-06 00:01:15' where label = 'VOO436';
00:01:15 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  Sending done to client: {'vc': {'comment': '', 'storage_group': 'miniboone', 'blocksize': 131072, 'declared': 1237240176.0, 'library': 'CD-LTO4F1', 'si_time': [1251869136.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOO436', 'wrapper': 'cpio_odc', 'remaining_bytes': 407325696000L, 'sum_mounts': 2982, 'sum_rd_access': 0, 'file_family_width': 1, 'last_access': 1257487212.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0026224', 'capacity_bytes': 858993459200L, 'non_del_files': 26907, 'sum_wr_err': 1, 'sum_wr_access': 26224, 'file_family': 'OpenRootTree', 'address': ('131.225.13.59', 7502), 'volume_family': 'miniboone.OpenRootTree.cpio_odc', 'modification_time': 1257487212.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1241646225.0}, 'encp': {'agetime': 30, 'adminpri': 1, 'delpri': 1, 'basepri': 1000003, 'delayed_dismount': 10, 'curpri': 1000003}, 'file_size': 8869590L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4F1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7871, 'mc_device': '1,0,1,8', 'compression': 0, 'serial_num': '1310051081', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr211a', 'max_time_in_state': 1200, 'device': 'stkenmvr211a:/dev/rmt/tps0d0n', 'name': 'LTO4_111.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_111.stat', 'mover_address': ('131.225.13.106', 7871), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.106', 43736), 'data_ip': 'stkenmvr211a', 'logname': 'LTO4_111MV', 'hostip': '131.225.13.106', 'do_cleaning': 'No', 'media_changer': 'SL8500F1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'mode_octal': '0100011', 'pnfsFilename': '/pnfs/miniboone/OpenRootTree/Big_Nu/run0022000_0022999/run0022360_0022369/boone_0022365_0012.root.big_nu', 'uid': 3557, 'uname': 'e898', 'type': 'cpio_odc', 'rmajor': 0, 'rminor': 0, 'machine': ('Linux', 'southport.fnal.gov', '2.4.21-52.ELsmp', '#1 SMP Fri Sep 28 13:31:57 CDT 2007', 'i686'), 'sanity_size': 65536, 'pstat': (16872, 488647360L, 10L, 1, 3557, 5468, 512L, 1257485963, 1257485963, 1257355749), 'gid': 5468, 'mtime': 1257487270, 'mode': 33188, 'gname': 'e898', 'size_bytes': 8869590L, 'fullname': '/Nearline/data01/boone_0022365_0012.root.big_nu', 'inode': 488673480L, 'minor': 10}, 'version': 'v3_6c  CVS $Revision: 1.822 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'infile': '/Nearline/data01/boone_0022365_0012.root.big_nu', 'status': ('ok', None), 'outfile': '/pnfs/miniboone/OpenRootTree/Big_Nu/run0022000_0022999/run0022360_0022369/boone_0022365_0012.root.big_nu', 'fc': {'uid': 3557, 'external_label': 'VOO436', 'gid': 5468, 'location_cookie': '0000_000000000_0026224', 'complete_crc': 3452424606L, 'bfid': 'CDMS125748727500000', 'sanity_cookie': (65536L, 819419212L), 'size': 8869590L}, 'at_the_top': 4, 'local_inode': 3714L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.194.144', 45787), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257487269.646157, 'seek_time': 0.034556865692138672, 'transfer_time': 4.2555489540100098, 't0': 1257487269, 'mount_time': 0.045350074768066406, 'lm_dequeued': 1257487271.5347641, 'job_queued': 1257487270.504926, 'drive_call_time': 0.01865077018737793, 'in_queue': 1.0297489166259766, 'drive_transfer_time': 3.286555290222168}, 'unique_id': 'southport.fnal.gov-1257487270-16471-0'} Thread tape_thread
00:01:15 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  Done is sent Thread tape_thread
00:01:15 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  maybe_clean Thread tape_thread
00:01:15 southport.fnal.gov 016471 e898 I ENCP  Received final dialog for southport.fnal.gov-1257487270-16471-0.  elapsed=6.310sec
00:01:15 southport.fnal.gov 016471 e898 I ENCP  Mover CRC: 3452424606  ENCP CRC: 3452424606
00:01:15 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV  returned from maybe_clean Thread tape_thread
00:01:16 southport.fnal.gov 016471 e898 I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = 0
00:01:16 stkensrv0n.fnal.gov 028796 enstore I FILSRV  Updating  			update file set deleted = 'n', pnfs_path = '/pnfs/miniboone/OpenRootTree/Big_Nu/run0022000_0022999/run0022360_0022369/boone_0022365_0012.root.big_nu', drive = 'stkenmvr211a:/dev/rmt/tps0d0n:1310051081', pnfs_id = '001D000000000000052090C8' where bfid = 'CDMS125748727500000';
00:01:16 southport.fnal.gov 016471 e898 I ENCP  MSG_TYPE=ENCP_XFER    write_to_hsm /Nearline/data01/boone_0022365_0012.root.big_nu -> /pnfs/miniboone/OpenRootTree/Big_Nu/run0022000_0022999/run0022360_0022369/boone_0022365_0012.root.big_nu: 8869590 bytes copied to VOO436 at 1.99 MB/S (11.3 MB/S network) (2.57 MB/S drive) (11.3 MB/S disk) mover=LTO4_111.mover drive_id=ULTRIUM-TD4 drive_sn=1310051081 drive_vendor=IBM elapsed=6.6227 {'storage_group': 'miniboone', 'encp_crc': 3452424606L, 'disk_rate': 11.280598230614663, 'drive_rate': 2.5737282452600732, 'driver': 'FTTDriver', 'transfer_rate': 1.9900660375718053, 'mover_interface': 'stkenmvr211a', 'overall_rate': 1.8051731285829102, 'unique_id': 'southport.fnal.gov-1257487270-16471-0', 'network_rate': 1.9900660375718053, 'encp_ip': '131.225.194.144', 'media_changer': 'SL8500F1.media_changer'}
00:01:16 southport.fnal.gov 016471 e898 I ENCP  Completed transferring 8869590 bytes in 1 files in 6.62348294258 sec.  Overall rate = 1.81 MB/sec.  Transfer rate = 1.99 MB/sec.  Network rate = 11.3 MB/sec.  Drive rate = 2.57 MB/sec.  Disk rate = 11.3 MB/sec.  Exit status = 0.
00:01:16 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_104.mover', 'VON102'), ('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_95.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_95.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} >
00:01:16 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:18 last message repeated 1 times
00:01:18 stkendca19a.fnal.gov 005688 root I ENCP  Version: v3_8b  CVS $Revision: 1.922 $ <frozen>  OS: Linux 2.6.9-89.0.9.ELsmp i686 Release:  Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)  Library: CD-LTO4G1  Storage Group: exp-db  File Family: daily-d0-online-readonly  FF Wrapper: cpio_odc  FF Width: 2  Current working directory: stkendca19a.fnal.gov:/home/enstore/dcache-code/dcache-fermi-config  Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/exp-db/daily/d0-online-readonly/d0onlprd/2009/11-November/05/channel01/D0ONLPRD_t702168009_s2963_p22 --put-cache 003000000000000006208600 /diskc/write-pool-3/data/003000000000000006208600
00:01:18 stkendca19a.fnal.gov 005688 root I ENCP  Creating zero length output files.
00:01:18 stkendca19a.fnal.gov 005688 root I ENCP  Sending /pnfs/fnal.gov/usr/exp-db/daily/d0-online-readonly/d0onlprd/2009/11-November/05/channel01/D0ONLPRD_t702168009_s2963_p22 write request to LM: unique_id: stkendca19a.fnal.gov-1257487278-5688-0 inputfile: /diskc/write-pool-3/data/003000000000000006208600 outputfile: /pnfs/fnal.gov/usr/exp-db/daily/d0-online-readonly/d0onlprd/2009/11-November/05/channel01/.(access)(003000000000000006208600)
00:01:18 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  MSG_TYPE=ADD_TO_LMQ  request added to pending queue for storage group : exp-db
00:01:18 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  write Q'd /diskc/write-pool-3/data/003000000000000006208600 (stkendca19a.fnal.gov-1257487278-5688-0) -> /pnfs/fnal.gov/usr/exp-db/daily/d0-online-readonly/d0onlprd/2009/11-November/05/channel01/D0ONLPRD_t702168009_s2963_p22 : library=CD-LTO4G1 family=daily-d0-online-readonly requester:root volume_family:exp-db.daily-d0-online-readonly.cpio_odc
00:01:18 stkendca19a.fnal.gov 005688 root I ENCP  Listening for control socket at: ('131.225.13.11', 51935)
00:01:18 stkendca19a.fnal.gov 005688 root I ENCP  Waiting for mover to connect control socket at address: ('131.225.13.11', 51935)  elapsed=1.030sec
00:01:19 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Assigning  volume VON245 from storage group exp-db.daily-d0-online-readonly.cpio_odc to library CD-LTO4G1, volume family exp-db.daily-d0-online-readonly.cpio_odc
00:01:19 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  write_to_hsm work on vol=VON245 mover=LTO4_109.mover requester:root
00:01:19 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  MSG_TYPE=ADD_TO_LMQ  request added to work at movers queue for storage group : exp-db
00:01:19 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  IDLE:sending {'vc': {'storage_group': 'exp-db', 'library': 'CD-LTO4G1', 'file_family': 'daily-d0-online-readonly', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-online-readonly.cpio_odc', 'file_family_width': 2}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-online-readonly/d0onlprd/2009/11-November/05/channel01/D0ONLPRD_t702168009_s2963_p22', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 2147467264L, 'mover': 'LTO4_109.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-online-readonly/d0onlprd/2009/11-November/05/channel01/D0ONLPRD_t702168009_s2963_p22', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257487278, 'rmajor': 0, 'machine': ('Linux', 'stkendca19a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (16877, 820608480L, 20L, 1, 1602, 2752, 512L, 1257487268, 1257487268, 1254398413), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 2147467264L, 'fullname': '/diskc/write-pool-3/data/003000000000000006208600', 'inode': 807437824L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskc/write-pool-3/data/003000000000000006208600', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-online-readonly/d0onlprd/2009/11-November/05/channel01/.(access)(003000000000000006208600)', 'fc': {'size': 2147467264L, 'external_label': 'VON245', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diskc/write-pool-3/data/003000000000000006208600', 'at_the_top': 1, 'local_inode': 539180112L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.11', 51935), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487279.746479, 'encp_start_time': 1257487277.8918359, 'job_queued': 1257487278.881531, 't0': 1257487277, 'in_queue': 0.86488604545593262}, 'user_level': 'user2', 'unique_id': 'stkendca19a.fnal.gov-1257487278-5688-0'} to mover
00:01:19 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  WRITE_TO_HSM Thread MainThread
00:01:19 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  can not remove from sg LTO4_109.mover VON245
00:01:19 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  SENDING {'vc': {'storage_group': 'exp-db', 'library': 'CD-LTO4G1', 'file_family': 'daily-d0-online-readonly', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-online-readonly.cpio_odc', 'file_family_width': 2}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-online-readonly/d0onlprd/2009/11-November/05/channel01/D0ONLPRD_t702168009_s2963_p22', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 2147467264L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7869, 'mc_device': '0,5,1,7', 'compression': 0, 'serial_num': '9310025229', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr209a', 'max_time_in_state': 1200, 'device': 'stkenmvr209a:/dev/rmt/tps0d0n', 'name': 'LTO4_109.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_109.stat', 'mover_address': ('131.225.13.205', 7869), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.205', 46288), 'data_ip': 'stkenmvr209a', 'logname': 'LTO4_109MV', 'hostip': '131.225.13.205', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-online-readonly/d0onlprd/2009/11-November/05/channel01/D0ONLPRD_t702168009_s2963_p22', 'uid': 0, 'uname': 'root', 'mtime': 1257487278, 'rmajor': 0, 'machine': ('Linux', 'stkendca19a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 820608480L, 20L, 1, 1602, 2752, 512L, 1257487268, 1257487268, 1254398413), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 2147467264L, 'gname': 'root', 'gid': 0, 'fullname': '/diskc/write-pool-3/data/003000000000000006208600', 'inode': 807437824L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskc/write-pool-3/data/003000000000000006208600', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-online-readonly/d0onlprd/2009/11-November/05/channel01/.(access)(003000000000000006208600)', 'fc': {'size': 2147467264L, 'external_label': 'VON245', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diskc/write-pool-3/data/003000000000000006208600', 'at_the_top': 1, 'local_inode': 539180112L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.11', 51935), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487279.746479, 'encp_start_time': 1257487277.8918359, 'job_queued': 1257487278.881531, 't0': 1257487277, 'in_queue': 0.86488604545593262}, 'user_level': 'user2', 'unique_id': 'stkendca19a.fnal.gov-1257487278-5688-0'} Thread client_connect_thread
00:01:19 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  SG: {'exp-db': [('LTO4_104.mover', 'VON102'), ('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_95.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]}
00:01:19 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  can not remove from vf LTO4_109.mover VON245
00:01:19 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  VF: {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_95.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]}
00:01:19 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_104.mover', 'VON102'), ('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245')], 'cms': [('LTO4_95.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_95.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} >
00:01:19 stkendca19a.fnal.gov 005688 root I ENCP  Control socket ('131.225.13.11', 51935) is connected to ('131.225.13.205', 46289) for stkendca19a.fnal.gov-1257487278-5688-0.  elapsed=1.910sec
00:01:19 stkendca19a.fnal.gov 005688 root I ENCP  Received callback ticket from mover LTO4_109.mover for transfer stkendca19a.fnal.gov-1257487278-5688-0.  elapsed=1.920sec
00:01:19 stkendca19a.fnal.gov 005688 root I ENCP  Atempting to connect data socket to mover at ('131.225.13.205', 46288).
00:01:19 stkendca19a.fnal.gov 005688 root I ENCP  Connecting data socket to mover (('131.225.13.205', 46288)) with interface 131.225.13.11.  elapsed=1.920sec
00:01:19 stkendca19a.fnal.gov 005688 root I ENCP  Data socket ('131.225.13.11', 51936) is connected to ('131.225.13.205', 46288).  elapsed=1.920sec
00:01:19 stkendca19a.fnal.gov 005688 root I ENCP  Opening local file.
00:01:19 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  MSG_TYPE=MC_LOAD_REQ  mounting VON245 ULTRIUM-TD4 00:01:19 Thread media_thread
00:01:19 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  PROCESSING mount VON245 0,5,1,7
00:01:19 stkendca19a.fnal.gov 005688 root I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = False
00:01:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_104.mover', 'VON102'), ('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245')], 'cms': [('LTO4_95.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_95.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} >
00:01:20 stkendca18a.fnal.gov 022627 root I ENCP  Version: v3_8b  CVS $Revision: 1.922 $ <frozen>  OS: Linux 2.6.9-89.0.9.ELsmp i686 Release:  Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)  Library: CD-LTO4G1  Storage Group: exp-db  File Family: daily-d0-offline  FF Wrapper: cpio_odc  FF Width: 2  Current working directory: stkendca18a.fnal.gov:/home/enstore/dcache-code/dcache-fermi-config  Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702158466_s103306_p17 --put-cache 0030000000000000062085D0 /diskc/write-pool-3/data/0030000000000000062085D0
00:01:20 stkendca18a.fnal.gov 022627 root I ENCP  Creating zero length output files.
00:01:20 stkendca18a.fnal.gov 022627 root I ENCP  Sending /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702158466_s103306_p17 write request to LM: unique_id: stkendca18a.fnal.gov-1257487280-22627-0 inputfile: /diskc/write-pool-3/data/0030000000000000062085D0 outputfile: /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/.(access)(0030000000000000062085D0)
00:01:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  MSG_TYPE=ADD_TO_LMQ  request added to pending queue for storage group : exp-db
00:01:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  write Q'd /diskc/write-pool-3/data/0030000000000000062085D0 (stkendca18a.fnal.gov-1257487280-22627-0) -> /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702158466_s103306_p17 : library=CD-LTO4G1 family=daily-d0-offline requester:root volume_family:exp-db.daily-d0-offline.cpio_odc
00:01:20 stkensrv4n.fnal.gov 008483 enstore I SL8500G1MC  mount VON277 0,6,1,0 => 0,Mount: VON277 mounted on   0, 6, 1, 0
00:01:20 stkendca18a.fnal.gov 022627 root I ENCP  Listening for control socket at: ('131.225.13.6', 57799)
00:01:20 stkendca18a.fnal.gov 022627 root I ENCP  Waiting for mover to connect control socket at address: ('131.225.13.6', 57799)  elapsed=1.120sec
00:01:20 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  FINISHED mount VON277 0,6,1,0 returned ('ok', 0, 'mount VON277 0,6,1,0 => 0,Mount: VON277 mounted on   0, 6, 1, 0', '', '')
00:01:20 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:20 last message repeated 1 times
00:01:20 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_mounts = '113', last_access = '2009-11-06 00:01:20' where label = 'VON277';
00:01:20 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  MSG_TYPE=MC_LOAD_DONE  mounted VON277 ULTRIUM-TD4 00:01:20 Thread media_thread
00:01:21 stkensrv4n.fnal.gov 008793 enstore I SL8500G1MC  dismount VOLUME 0,7,1,14 force => 0,Dismount: Forced dismount of VOS175 from   0, 7, 1,14
00:01:21 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  FINISHED dismount VOS175 0,7,1,14 returned ('ok', 0, 'dismount VOLUME 0,7,1,14 force => 0,Dismount: Forced dismount of VOS175 from   0, 7, 1,14', '', '')
00:01:21 stkenmvr195a.fnal.gov 021479 root I LTO4_95MV  dismounted VOS175 ULTRIUM-TD4 00:01:21 Thread media_thread
00:01:22 stkensrv4n.fnal.gov 005357 enstore E LTO4G1LM  MOVER ERROR RQ {'status': ('ok', None), 'returned_work': None, 'library': None, 'mover_type': '', 'current_location': '0000_000000000_0000000', 'ip_map': '', 'operation': None, 'read_only': 0, 'current_time': 1257487282.3710661, 'error_source': None, 'external_label': None, 'state': 'OFFLINE', 'r_a': (('131.225.13.194', 54880), 18007L, '131.225.13.194-54880-1256586929.509013-4866'), 'library_list': [('CD-LTO4G1.library_manager', ('131.225.13.4', 8541))], 'current_priority': (None, None), 'transfer_deficiency': 1, 'time_in_state': 900352.01100802422, 'address': ('131.225.13.194', 7860), 'volume_family': None, 'work': 'mover_error', 'mover': 'LTO4_99.mover', 'volume_status': (['none', 'none'], ['none', 'none']), 'unique_id': None}
00:01:22 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:22 stkensrv4n.fnal.gov 009024 enstore I SL8500G1MC  query drive 0,5,1,7 => 0,0,5,1,7 online          available              IBM-LTO4
00:01:22 stkensrv4n.fnal.gov 009024 enstore I SL8500G1MC  Dismount 14 ignored: Drive 0,5,1,7 is empty. Thought VON245 was there =>. dismount VOLUME 0,5,1,7 force => ok,0,5,1,7 online          available              IBM-LTO4
00:01:24 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:24 stkensrv4n.fnal.gov 009024 enstore I SL8500G1MC  query vol VON245 => 0,VON245      home              0, 5,32,10, 1         LTO-800G
00:01:25 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} >
00:01:26 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:26 stkensrv4n.fnal.gov 009024 enstore I SL8500G1MC  query drive 0,5,1,7 => 0,0,5,1,7 online          available              IBM-LTO4
00:01:28 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:30 stkendca22a.fnal.gov 021230 root I ENCP  Version: v3_8b  CVS $Revision: 1.922 $ <frozen>  OS: Linux 2.6.9-89.0.9.ELsmp i686 Release:  Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)  Library: garbage  Storage Group: test  File Family: dcache  FF Wrapper: cpio_odc  FF Width: 1  Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log  Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0003 --put-cache 0005000000000000005FA5F8 /diskb/write-pool-2/data/0005000000000000005FA5F8
00:01:30 stkendca22a.fnal.gov 021299 root I ENCP  Version: v3_8b  CVS $Revision: 1.922 $ <frozen>  OS: Linux 2.6.9-89.0.9.ELsmp i686 Release:  Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)  Library: garbage  Storage Group: test  File Family: dcache  FF Wrapper: cpio_odc  FF Width: 1  Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log  Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0001 --put-cache 0005000000000000005FA590 /diskb/write-pool-2/data/0005000000000000005FA590
00:01:30 stkendca22a.fnal.gov 021341 root I ENCP  Version: v3_8b  CVS $Revision: 1.922 $ <frozen>  OS: Linux 2.6.9-89.0.9.ELsmp i686 Release:  Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)  Library: garbage  Storage Group: test  File Family: dcache  FF Wrapper: cpio_odc  FF Width: 1  Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log  Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0002 --put-cache 0005000000000000005FA5C8 /diskb/write-pool-2/data/0005000000000000005FA5C8
00:01:30 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:30 stkendca22a.fnal.gov 021230 root I ENCP  Configuration Server: no such name: 'garbage.library_manager'
00:01:30 stkendca22a.fnal.gov 021230 root E ENCP  INFILE=/diskb/write-pool-2/data/0005000000000000005FA5F8 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0003 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=0.86 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager'
00:01:30 stkendca22a.fnal.gov 021299 root I ENCP  Configuration Server: no such name: 'garbage.library_manager'
00:01:30 stkendca22a.fnal.gov 021299 root E ENCP  INFILE=/diskb/write-pool-2/data/0005000000000000005FA590 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0001 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=0.89 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager'
00:01:30 stkendm7a.fnal.gov 009044 root I ENCP  File /data/data1/Migration_Spool/VO5178:0000_000000000_0000092 transfered for stkendm7a.fnal.gov-1257487050-9044-79 in 49.0059971809 seconds. Thread WRITE_2
00:01:30 stkendca22a.fnal.gov 021341 root I ENCP  Configuration Server: no such name: 'garbage.library_manager'
00:01:30 stkendca22a.fnal.gov 021341 root E ENCP  INFILE=/diskb/write-pool-2/data/0005000000000000005FA5C8 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0002 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=0.91 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager'
00:01:31 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV  skip_fm 3409 Thread seek_thread
00:01:31 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV  read_tape starting, bytes_to_read=1376065438 Thread tape_thread
00:01:32 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV  tape position: target 1856 current 0 1 Thread seek_thread
00:01:32 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  written bytes 1668758016/1668758016, blocks 12732 header 0 trailer 28 Thread tape_thread
00:01:32 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:36 last message repeated 2 times
00:01:36 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  filemarks written. Tape VOV147 absolute location in blocks 3459502 Thread tape_thread
00:01:36 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  write_tape timing:pull 0.164526700974 write 6.5849802494 crc_check 7.73324680328 freespace 0.139710187912 block_write 14.6224639416 idle 0.236247301102 Thread tape_thread
00:01:36 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  drive stats after write. Tape VOV147 position 7417 block 0 block_size 0 bloc_loc 3459502 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 1668758016 block_write_tot 14.6224639416 tape_rate 114122901.767 Thread tape_thread
00:01:36 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  remaining bytes info in DB 338154496000 estimated 336485737984 reported from drive 336587776000 position 7420 Thread tape_thread
00:01:36 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  new bitfile request {'uid': 0, 'external_label': 'VOV147', 'gid': 0, 'location_cookie': '0000_000000000_0007418', 'complete_crc': 3093558071L, 'sanity_cookie': (65536L, 4170562793L), 'size': 1668757988L} Thread tape_thread
00:01:36 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  set remaining: VOV147 336587776000 0000_000000000_0007420 Thread tape_thread
00:01:37 last message repeated 1 times
00:01:37 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set last_access = '2009-11-06 00:01:36', eod_cookie = '0000_000000000_0007420', non_del_files = '2475', remaining_bytes = 336587776000 where label = 'VOV147';
00:01:37 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  transfer complete volume=VOV147 location=7420 Thread tape_thread
00:01:37 last message repeated 1 times
00:01:37 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_wr_access = '2473', last_access = '2009-11-06 00:01:37', modification_time = '2009-11-06 00:01:37' where label = 'VOV147';
00:01:37 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  Sending done to client: {'vc': {'comment': '', 'storage_group': 'ktev', 'blocksize': 131072, 'declared': 1254165878.0, 'library': 'CD-LTO4F1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOV147', 'wrapper': 'cern', 'remaining_bytes': 338154496000L, 'sum_mounts': 5, 'sum_rd_access': 0, 'file_family_width': 1, 'last_access': 1257487240.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0007417', 'capacity_bytes': 858993459200L, 'non_del_files': 2474, 'sum_wr_err': 0, 'sum_wr_access': 2472, 'file_family': 'ktev_migration-MIGRATION', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'modification_time': 1257487240.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257458755.0}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f0', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 1668757988L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4F1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7873, 'mc_device': '1,1,1,4', 'compression': 0, 'serial_num': '1310156224', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr213a', 'max_time_in_state': 1200, 'device': 'stkenmvr213a:/dev/rmt/tps0d0n', 'name': 'LTO4_113.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_113.stat', 'mover_address': ('131.225.13.108', 7873), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.108', 43835), 'data_ip': 'stkenmvr213a', 'logname': 'LTO4_113MV', 'hostip': '131.225.13.108', 'do_cleaning': 'No', 'media_changer': 'SL8500F1.media_changer'}, 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f0', 'uid': 0, 'uname': 'root', 'mtime': 1257487050, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cern', 'pstat': (16877, 683683784L, 20L, 1, 0, 0, 512L, 1255728736, 1255728736, 1255127721), 'mode_octal': '0100016', 'mode': 33204, 'size_bytes': 1668757988L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000092', 'inode': 675737848L, 'minor': 20}, 'version': 'v3_8  CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000092', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC099/.(access)(00280000000000000846F0F8)', 'fc': {'uid': 0, 'external_label': 'VOV147', 'gid': 0, 'location_cookie': '0000_000000000_0007418', 'complete_crc': 3093558071L, 'bfid': 'CDMS125748729600000', 'sanity_cookie': (65536L, 4170562793L), 'size': 1668757988L}, 'infilepath': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000092', 'at_the_top': 781, 'local_inode': 39503171L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 44361), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257487050.553689, 'seek_time': 0.033666849136352539, 'transfer_time': 55.327911138534546, 't0': 1257487050, 'mount_time': 0.04389500617980957, 'lm_dequeued': 1257487241.696538, 'job_queued': 1257487050.866416, 'drive_call_time': 6.4207019805908203, 'in_queue': 190.82983899116516, 'drive_transfer_time': 21.191303014755249}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257487050-9044-79'} Thread tape_thread
00:01:37 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  Done is sent Thread tape_thread
00:01:37 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  maybe_clean Thread tape_thread
00:01:37 stkendm7a.fnal.gov 009044 root I ENCP  Received final dialog for stkendm7a.fnal.gov-1257487050-9044-79.  elapsed=5734.520sec Thread WRITE_2
00:01:37 stkendm7a.fnal.gov 009044 root I ENCP  leaving transfer_file(). Thread WRITE_2
00:01:37 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  returned from maybe_clean Thread tape_thread
00:01:37 stkensrv0n.fnal.gov 028796 enstore I FILSRV  Updating  			update file set deleted = 'n', pnfs_path = '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f0', drive = 'stkenmvr213a:/dev/rmt/tps0d0n:1310156224', pnfs_id = '00280000000000000846F0F8' where bfid = 'CDMS125748729600000';
00:01:37 stkendm7a.fnal.gov 009044 root I ENCP  MSG_TYPE=ENCP_XFER    write_to_hsm /data/data1/Migration_Spool/VO5178:0000_000000000_0000092 -> /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f0: 1668757988 bytes copied to VOV147 at 28.8 MB/S (90.3 MB/S network) (75.1 MB/S drive) (34.1 MB/S disk) mover=LTO4_113.mover drive_id=ULTRIUM-TD4 drive_sn=1310156224 drive_vendor=IBM elapsed=247.03 {'storage_group': 'ktev', 'encp_crc': 3093558071L, 'disk_rate': 34.122844675853948, 'drive_rate': 75.099281641400751, 'driver': 'FTTDriver', 'transfer_rate': 28.761867642396687, 'mover_interface': 'stkenmvr213a', 'overall_rate': 28.500237949162241, 'media_changer': 'SL8500F1.media_changer', 'network_rate': 28.761867642396687, 'encp_ip': '131.225.13.19', 'unique_id': 'stkendm7a.fnal.gov-1257487050-9044-79'} Thread WRITE_2
00:01:37 stkendm7a.fnal.gov 009044 root I ENCP  File /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f0 copied successfully. Thread WRITE_2
00:01:37 stkendm7a.fnal.gov 009044 root I ENCP  Completed transferring 1668757988 bytes in 1 files in 247.028927088 sec.  Overall rate = 28.5 MB/sec.  Transfer rate = 28.8 MB/sec.  Network rate = 90.3 MB/sec.  Drive rate = 75.1 MB/sec.  Disk rate = 34.1 MB/sec.  Exit status = 0. Thread WRITE_2
00:01:37 stkensrv0n.fnal.gov 028796 enstore I FILSRV  start modifying {'pnfsid': '00280000000000000846F0F8', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f0', 'deleted': 'no', 'external_label': 'VOV147', 'drive': 'stkenmvr213a:/dev/rmt/tps0d0n:1310156224', 'update': '2009-11-06 00:01:37.295075', 'gid': 0, 'location_cookie': '0000_000000000_0007418', 'complete_crc': 3093558071L, 'uid': 0, 'bfid': 'CDMS125748729600000', 'sanity_cookie': (65536L, 4170562793L), 'size': 1668757988L}
00:01:37 stkensrv0n.fnal.gov 028796 enstore I FILSRV  Updating  			update file set pnfs_id = '001F00000000000000566788' where bfid = 'CDMS125748729600000';
00:01:37 stkensrv0n.fnal.gov 028796 enstore I FILSRV  modified to {'pnfsid': '001F00000000000000566788', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f0', 'deleted': 'no', 'external_label': 'VOV147', 'drive': 'stkenmvr213a:/dev/rmt/tps0d0n:1310156224', 'update': '2009-11-06 00:01:37.295075', 'gid': 0, 'location_cookie': '0000_000000000_0007418', 'complete_crc': 3093558071L, 'uid': 0, 'bfid': 'CDMS125748729600000', 'sanity_cookie': (65536L, 4170562793L), 'size': 1668757988L}
00:01:37 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:37 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  write_to_hsm next work on vol=VOV147 mover=LTO4_113.mover requester:root
00:01:37 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  HAVE_BOUND:sending write_to_hsm /pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC092/f0 to mover LTO4_113.mover ('131.225.13.108', 7873) DEL_DISM 2
00:01:37 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  MSG_TYPE=ADD_TO_LMQ  request added to work at movers queue for storage group : ktev
00:01:37 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  HAVE_BOUND:sending {'vc': {'storage_group': 'ktev', 'external_label': 'VOV147', 'library': 'CD-LTO4F1', 'file_family': 'ktev_migration-MIGRATION', 'wrapper': 'cern', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'file_family_width': 1}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC092/f0', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 11981447772L, 'mover': 'LTO4_113.mover', 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC092/f0', 'uid': 0, 'uname': 'root', 'type': 'cern', 'gid': 0, 'mtime': 1257487106, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (16877, 675737896L, 20L, 1, 0, 0, 512L, 1257487106, 1257487106, 1257487106), 'mode_octal': '0100016', 'mode': 33204, 'gname': 'root', 'size_bytes': 11981447772L, 'fullname': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000116', 'inode': 675737968L, 'minor': 20}, 'version': 'v3_8  CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000116', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC092/.(access)(00280000000000000846F170)', 'fc': {'size': 11981447772L, 'external_label': 'VOV147', 'pnfsid': '00280000000000000846F170', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000116', 'at_the_top': 314, 'local_inode': 39503182L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 44365), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487297.744405, 'encp_start_time': 1257487106.4977331, 'job_queued': 1257487106.742687, 't0': 1257487106, 'in_queue': 191.00144791603088}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257487106-9044-80'} to mover
00:01:37 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:37 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  WRITE_TO_HSM Thread MainThread
00:01:37 stkendm7a.fnal.gov 009044 root I MIGRATION  Listening for control socket at: ('131.225.13.19', 44365) Thread WRITE_1
00:01:37 stkendm7a.fnal.gov 009044 root I MIGRATION  Waiting for mover to connect control socket at address: ('131.225.13.19', 44365)  elapsed=5735.130sec Thread WRITE_1
00:01:37 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  SENDING {'vc': {'storage_group': 'ktev', 'external_label': 'VOV147', 'library': 'CD-LTO4F1', 'file_family': 'ktev_migration-MIGRATION', 'wrapper': 'cern', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'file_family_width': 1}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC092/f0', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 11981447772L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4F1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7873, 'mc_device': '1,1,1,4', 'compression': 0, 'serial_num': '1310156224', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr213a', 'max_time_in_state': 1200, 'device': 'stkenmvr213a:/dev/rmt/tps0d0n', 'name': 'LTO4_113.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_113.stat', 'mover_address': ('131.225.13.108', 7873), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.108', 43838), 'data_ip': 'stkenmvr213a', 'logname': 'LTO4_113MV', 'hostip': '131.225.13.108', 'do_cleaning': 'No', 'media_changer': 'SL8500F1.media_changer'}, 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC092/f0', 'uid': 0, 'uname': 'root', 'mtime': 1257487106, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cern', 'pstat': (16877, 675737896L, 20L, 1, 0, 0, 512L, 1257487106, 1257487106, 1257487106), 'mode_octal': '0100016', 'mode': 33204, 'size_bytes': 11981447772L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000116', 'inode': 675737968L, 'minor': 20}, 'version': 'v3_8  CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000116', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC092/.(access)(00280000000000000846F170)', 'fc': {'size': 11981447772L, 'external_label': 'VOV147', 'pnfsid': '00280000000000000846F170', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000116', 'at_the_top': 314, 'local_inode': 39503182L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 44365), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487297.744405, 'encp_start_time': 1257487106.4977331, 'job_queued': 1257487106.742687, 't0': 1257487106, 'in_queue': 191.00144791603088}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257487106-9044-80'} Thread client_connect_thread
00:01:37 stkendm7a.fnal.gov 009044 root I MIGRATION  Control socket ('131.225.13.19', 44365) is connected to ('131.225.13.108', 43839) for stkendm7a.fnal.gov-1257487106-9044-80.  elapsed=5735.140sec Thread WRITE_1
00:01:37 stkendm7a.fnal.gov 009044 root I MIGRATION  Received callback ticket from mover LTO4_113.mover for transfer stkendm7a.fnal.gov-1257487106-9044-80.  elapsed=5735.140sec Thread WRITE_1
00:01:37 stkendm7a.fnal.gov 009044 root I MIGRATION  Attempting to connect data socket to mover at ('131.225.13.108', 43838). Thread WRITE_1
00:01:37 stkendm7a.fnal.gov 009044 root I MIGRATION  Connecting data socket to mover (('131.225.13.108', 43838)) with interface 131.225.13.19.  elapsed=5735.140sec Thread WRITE_1
00:01:37 stkendm7a.fnal.gov 009044 root I MIGRATION  Data socket ('131.225.13.19', 44373) is connected to ('131.225.13.108', 43838).  elapsed=5735.140sec Thread WRITE_1
00:01:37 stkendm7a.fnal.gov 009044 root I MIGRATION  Opening local file. Thread WRITE_1
00:01:37 stkendm7a.fnal.gov 009044 root I MIGRATION  pf_status = ('ok', None)  skip_layer_cleanup = False Thread WRITE_1
00:01:37 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  tape position: target 7420 current 7420 0 Thread seek_thread
00:01:37 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  write_tape starting, bytes_to_write=11981448192 Thread tape_thread
00:01:37 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  Write starting. Tape VOV147 absolute location in blocks 3459502 Thread tape_thread
00:01:38 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  tape position: target 91 current 0 1 Thread seek_thread
00:01:38 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:41 last message repeated 2 times
00:01:41 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} >
00:01:42 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:42 stkendm7a.fnal.gov 009044 root I ENCP  Version: v3_8  CVS $Revision: 1.963 $ encp.pyc  OS: Linux 2.6.9-89.0.9.ELsmp i686 Release:  Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)  Library: CD-LTO4F1  Storage Group: ktev  File Family: ktev_migration-MIGRATION  FF Wrapper: cern  FF Width: 1  Current working directory: stkendm7a.fnal.gov:/opt/enstore/src  Command line: encp --delayed-dismount 2 --ignore-fair-share --threaded --priority 0 --storage-group ktev --file-family ktev_migration-MIGRATION --file-family-wrapper cern --library CD-LTO4F1 --override-path /pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f3 --threaded /data/data1/Migration_Spool/VO5178:0000_000000000_0000101 /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f3 Thread WRITE_2
00:01:43 stkendm7a.fnal.gov 009044 root I ENCP  Creating zero length output files. Thread WRITE_2
00:01:43 stkendm7a.fnal.gov 009044 root I ENCP  Sending /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f3 write request to LM: unique_id: stkendm7a.fnal.gov-1257487303-9044-82 inputfile: /data/data1/Migration_Spool/VO5178:0000_000000000_0000101 outputfile: /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC099/.(access)(00280000000000000846F1D0) Thread WRITE_2
00:01:43 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  MSG_TYPE=ADD_TO_LMQ  request added to pending queue for storage group : ktev
00:01:43 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  write Q'd /data/data1/Migration_Spool/VO5178:0000_000000000_0000101 (stkendm7a.fnal.gov-1257487303-9044-82) -> /pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f3 : library=CD-LTO4F1 family=ktev_migration-MIGRATION requester:root volume_family:ktev.ktev_migration-MIGRATION.cern
00:01:43 cmsstor68.fnal.gov 010129 root I ENCP  Version: v3_7d  CVS $Revision: 1.922 $ encp  OS: Linux 2.6.31 x86_64 Release:  Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)  Library: CD-LTO4G1  Storage Group: cms  File Family: cms2  FF Wrapper: cpio_odc  FF Width: 7  Current working directory: cmsstor68.fnal.gov:/  Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --age-time 30 --delpri 1 --priority 25 --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/cms/WAX/2/olga/JPT_AKT05_170_230/JetAnalysis_6.root --put-cache 0010000000000000094F6470 /storage/data1/write-pool-1/data/0010000000000000094F6470
00:01:43 cmsstor68.fnal.gov 010129 root I ENCP  Creating zero length output files.
00:01:43 cmsstor68.fnal.gov 010129 root I ENCP  Sending /pnfs/fnal.gov/usr/cms/WAX/2/olga/JPT_AKT05_170_230/JetAnalysis_6.root write request to LM: unique_id: cmsstor68.fnal.gov-1257487303-10129-0 inputfile: /storage/data1/write-pool-1/data/0010000000000000094F6470 outputfile: /pnfs/fnal.gov/usr/cms/WAX/2/olga/JPT_AKT05_170_230/.(access)(0010000000000000094F6470)
00:01:43 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  MSG_TYPE=ADD_TO_LMQ  request added to pending queue for storage group : cms
00:01:43 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  write Q'd /storage/data1/write-pool-1/data/0010000000000000094F6470 (cmsstor68.fnal.gov-1257487303-10129-0) -> /pnfs/fnal.gov/usr/cms/WAX/2/olga/JPT_AKT05_170_230/JetAnalysis_6.root : library=CD-LTO4G1 family=cms2 requester:root volume_family:cms.cms2.cpio_odc
00:01:43 cmsstor68.fnal.gov 010129 root I ENCP  Listening for control socket at: ('131.225.189.68', 33740)
00:01:43 cmsstor68.fnal.gov 010129 root I ENCP  Waiting for mover to connect control socket at address: ('131.225.189.68', 33740)  elapsed=0.920sec
00:01:44 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:44 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV  read bytes 1376065606/1376065606, blocks 10499 header 168 Thread tape_thread
00:01:44 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV  drive stats after read. Tape VOS642 position 3410 block 10499 block_size 0 bloc_loc 6226187 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 1376065606 block_read_tot 12.0416705608 tape_rate 114275307.487 Thread tape_thread
00:01:44 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV  read_tape exiting, read 1376065606/1376065606 bytes Thread tape_thread
00:01:44 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV  read_tape timing:get_space 0.425273656845 read 6.0216460228 crc_check 5.46746325493 push 0.127287626266 block_read 12.0416705608 idle 0.212054252625 Thread tape_thread
00:01:44 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Assigning  volume VOS175 from storage group cms.cms2.cpio_odc to library CD-LTO4G1, volume family cms.cms2.cpio_odc
00:01:44 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  write_to_hsm work on vol=VOS175 mover=LTO4_41.mover requester:root
00:01:44 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  MSG_TYPE=ADD_TO_LMQ  request added to work at movers queue for storage group : cms
00:01:44 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  IDLE:sending {'vc': {'storage_group': 'cms', 'library': 'CD-LTO4G1', 'file_family': 'cms2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms2.cpio_odc', 'file_family_width': 7}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX/2/olga/JPT_AKT05_170_230/JetAnalysis_6.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 21428160L, 'mover': 'LTO4_41.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX/2/olga/JPT_AKT05_170_230/JetAnalysis_6.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257487303, 'rmajor': 0, 'machine': ('Linux', 'cmsstor68.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16895, 273637888, 23L, 1, 9811, 5063, 512, 1257487250, 1257487250, 1257485909), 'mode_octal': '0100040', 'mode': 33188, 'gname': 'root', 'size_bytes': 21428160L, 'fullname': '/storage/data1/write-pool-1/data/0010000000000000094F6470', 'inode': 273638512L, 'minor': 23}, 'version': 'v3_7d  CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/0010000000000000094F6470', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX/2/olga/JPT_AKT05_170_230/.(access)(0010000000000000094F6470)', 'fc': {'size': 21428160L, 'external_label': 'VOS175', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data1/write-pool-1/data/0010000000000000094F6470', 'at_the_top': 2, 'local_inode': 2147484328L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.189.68', 33740), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487304.8881299, 'encp_start_time': 1257487302.974324, 'job_queued': 1257487303.845396, 't0': 1257487302, 'in_queue': 1.042672872543335}, 'user_level': 'admin', 'unique_id': 'cmsstor68.fnal.gov-1257487303-10129-0'} to mover
00:01:44 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV  WRITE_TO_HSM Thread MainThread
00:01:44 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  can not remove from sg LTO4_41.mover VOS175
00:01:44 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  SG: {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]}
00:01:44 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  can not remove from vf LTO4_41.mover VOS175
00:01:44 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  VF: {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]}
00:01:44 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175')]} >
00:01:44 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV  SENDING {'vc': {'storage_group': 'cms', 'library': 'CD-LTO4G1', 'file_family': 'cms2', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms2.cpio_odc', 'file_family_width': 7}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX/2/olga/JPT_AKT05_170_230/JetAnalysis_6.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 21428160L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7841, 'mc_device': '0,7,1,3', 'compression': 0, 'serial_num': '1310154471', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr141a', 'max_time_in_state': 1200, 'device': 'stkenmvr141a:/dev/rmt/tps1d0n', 'name': 'LTO4_41.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_41.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.169', 7841), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.169', 55114), 'data_ip': 'stkenmvr141a', 'logname': 'LTO4_41MV', 'hostip': '131.225.13.169', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX/2/olga/JPT_AKT05_170_230/JetAnalysis_6.root', 'uid': 0, 'uname': 'root', 'mtime': 1257487303, 'rmajor': 0, 'machine': ('Linux', 'cmsstor68.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16895, 273637888, 23L, 1, 9811, 5063, 512, 1257487250, 1257487250, 1257485909), 'mode_octal': '0100040', 'mode': 33188, 'size_bytes': 21428160L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data1/write-pool-1/data/0010000000000000094F6470', 'inode': 273638512L, 'minor': 23}, 'version': 'v3_7d  CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/0010000000000000094F6470', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX/2/olga/JPT_AKT05_170_230/.(access)(0010000000000000094F6470)', 'fc': {'size': 21428160L, 'external_label': 'VOS175', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data1/write-pool-1/data/0010000000000000094F6470', 'at_the_top': 2, 'local_inode': 2147484328L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.189.68', 33740), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487304.8881299, 'encp_start_time': 1257487302.974324, 'job_queued': 1257487303.845396, 't0': 1257487302, 'in_queue': 1.042672872543335}, 'user_level': 'admin', 'unique_id': 'cmsstor68.fnal.gov-1257487303-10129-0'} Thread client_connect_thread
00:01:44 cmsstor68.fnal.gov 010129 root I ENCP  Control socket ('131.225.189.68', 33740) is connected to ('131.225.13.169', 55115) for cmsstor68.fnal.gov-1257487303-10129-0.  elapsed=1.970sec
00:01:44 cmsstor68.fnal.gov 010129 root I ENCP  Received callback ticket from mover LTO4_41.mover for transfer cmsstor68.fnal.gov-1257487303-10129-0.  elapsed=1.970sec
00:01:44 cmsstor68.fnal.gov 010129 root I ENCP  Atempting to connect data socket to mover at ('131.225.13.169', 55114).
00:01:44 cmsstor68.fnal.gov 010129 root I ENCP  Connecting data socket to mover (('131.225.13.169', 55114)) with interface 131.225.189.68.  elapsed=1.980sec
00:01:44 cmsstor68.fnal.gov 010129 root I ENCP  Data socket ('131.225.189.68', 35732) is connected to ('131.225.13.169', 55114).  elapsed=1.980sec
00:01:44 cmsstor68.fnal.gov 010129 root I ENCP  Opening local file.
00:01:44 cmsstor68.fnal.gov 010129 root I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = False
00:01:45 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV  MSG_TYPE=MC_LOAD_REQ  mounting VOS175 ULTRIUM-TD4 00:01:44 Thread media_thread
00:01:45 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  PROCESSING mount VOS175 0,7,1,3
00:01:45 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175')]} >
00:01:45 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188')]} >
00:01:46 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:01:46 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:47 stkensrv4n.fnal.gov 009037 enstore I SL8500G1MC  query drive 0,7,1,3 => 0,0,7,1,3 online          available              IBM-LTO4
00:01:47 stkensrv4n.fnal.gov 009037 enstore I SL8500G1MC  Dismount 14 ignored: Drive 0,7,1,3 is empty. Thought VOS175 was there =>. dismount VOLUME 0,7,1,3 force => ok,0,7,1,3 online          available              IBM-LTO4
00:01:47 stkensrv3n.fnal.gov 029976 root I ENCP  File /pnfs/cms/WAX/2/jrsmith/work/technicolor/CMSSW_1_6_7/higgs/200/higgs_200_200_462.root transfered for stkensrv3n.fnal.gov-1257487175-29976-0 in 15.8152649403 seconds.
00:01:47 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV  write_client: wrote 1376065438/1376065438 bytes Thread net_thread
00:01:47 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV  write_client exits Thread net_thread
00:01:47 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV  transfer complete volume=VOS642 location=3410 Thread net_thread
00:01:47 last message repeated 1 times
00:01:47 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_rd_access = '3416', last_access = '2009-11-06 00:01:47' where label = 'VOS642';
00:01:47 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV  Sending done to client: {'vc': {'comment': '<= VOJ007 VOJ516 VOJ521 VOJ636 VOJ643', 'declared': 1244477818.0, 'blocksize': 131072, 'storage_group': 'cms', 'library': 'CD-LTO4G1', 'eod_cookie': '0000_000000000_0003411', 'current_location': '0000_000000000_0003410', 'user_inhibit': ['none', 'none'], 'sum_rd_access': 3415, 'system_inhibit': ['none', 'full'], 'external_label': 'VOS642', 'wrapper': 'cpio_odc', 'remaining_bytes': 133770752L, 'sum_mounts': 12, 'capacity_bytes': 858993459200L, 'media_type': 'LTO4', 'last_access': 1257487174.0, 'status': ('ok', None), 'si_time': [0.0, 1251799992.0], 'non_del_files': 3560, 'sum_wr_err': 0, 'sum_wr_access': 3410, 'file_family': 'cms2', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms2.cpio_odc', 'modification_time': 1251799991.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1251759219.0}, 'outfilepath': '/dev/null', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': 1, 'curpri': 1}, 'file_size': 1376065438L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7840, 'mc_device': '0,7,1,15', 'compression': 0, 'serial_num': '9310801386', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr140a', 'max_time_in_state': 1200, 'device': 'stkenmvr140a:/dev/rmt/tps0d0n', 'name': 'LTO4_40.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_40.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.168', 7840), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.168', 34508), 'data_ip': 'stkenmvr140a', 'logname': 'LTO4_40MV', 'hostip': '131.225.13.168', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/cms/WAX/2/jrsmith/work/technicolor/CMSSW_1_6_7/higgs/200/higgs_200_200_462.root', 'uid': 0, 'uname': 'root', 'rmajor': 0, 'machine': ('Linux', 'stkensrv3n.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 08:50:41 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (33188, 275259056L, 20L, 1, 9811, 5063, 1376065438L, 1256204076, 1212932029, 1212931815), 'mode_octal': '0100744', 'mode': 33252, 'size_bytes': 1376065438L, 'gname': 'root', 'gid': 0, 'fullname': '/dev/null', 'inode': 275259056L, 'minor': 20}, 'version': 'v3_7d  CVS $Revision: 1.929 $ encp', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/cms/WAX/2/jrsmith/work/technicolor/CMSSW_1_6_7/higgs/200/.(access)(001000000000000007681EB0)', 'status': ('ok', None), 'volume': 'VOS642', 'outfile': '/dev/null', 'fc': {'status': ('ok', None), 'pnfsid': '001000000000000007681EB0', 'pnfs_name0': '/pnfs/fnal.gov/usr/cms/WAX/2/jrsmith/work/technicolor/CMSSW_1_6_7/higgs/200/higgs_200_200_462.root', 'deleted': 'no', 'external_label': 'VOS642', 'drive': 'stkenmvr149a:/dev/rmt/tps0d0n:1310154355', 'update': '2009-09-01 05:13:13.169788', 'gid': 0, 'location_cookie': '0000_000000000_0003410', 'complete_crc': 2773655400L, 'address': ('131.225.13.59', 7501), 'uid': 0, 'bfid': 'CDMS125179999000002', 'sanity_cookie': (65536L, 1310704558L), 'size': 1376065438L}, 'infilepath': '/pnfs/cms/WAX/2/jrsmith/work/technicolor/CMSSW_1_6_7/higgs/200/higgs_200_200_462.root', 'at_the_top': 3, 'bfid': 'CDMS125179999000002', 'local_inode': 2469L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': 7, 'retry': 0}, 'callback_addr': ('131.225.13.51', 39961), 'work': 'read_from_hsm', 'override_ro_mount': 1, 'times': {'encp_start_time': 1257487174.270777, 'seek_time': 115.45905995368958, 'transfer_time': 15.969874143600464, 't0': 1257487174, 'mount_time': 0.038440942764282227, 'lm_dequeued': 1257487176.400099, 'job_queued': 1257487175.347573, 'drive_call_time': 5.855789(trunc. 4245) Thread net_thread
00:01:47 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV  Done is sent Thread net_thread
00:01:47 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV  maybe_clean Thread net_thread
00:01:47 stkensrv3n.fnal.gov 029976 root I ENCP  Received final dialog for stkensrv3n.fnal.gov-1257487175-29976-0.  elapsed=133.720sec
00:01:47 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV  returned from maybe_clean Thread net_thread
00:01:47 stkensrv3n.fnal.gov 029976 root I ENCP  leaving transfer_file().
00:01:48 stkensrv3n.fnal.gov 029976 root I ENCP  MSG_TYPE=ENCP_XFER    read_from_hsm /pnfs/cms/WAX/2/jrsmith/work/technicolor/CMSSW_1_6_7/higgs/200/higgs_200_200_462.root -> /dev/null: 1376065438 bytes copied from VOS642 at 82.5 MB/S (83 MB/S network) (108 MB/S drive) (83 MB/S disk) mover=LTO4_40.mover drive_id=ULTRIUM-TD4 drive_sn=9310801386 drive_vendor=IBM elapsed=133.74 {'storage_group': 'cms', 'encp_crc': 2773655400L, 'disk_rate': 82.9788947190697, 'drive_rate': 107.90625384240916, 'driver': 'FTTDriver', 'transfer_rate': 82.529033417316327, 'mover_interface': 'stkenmvr140a', 'overall_rate': 9.9770587537707307, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 82.529033417316327, 'encp_ip': '131.225.13.51', 'unique_id': 'stkensrv3n.fnal.gov-1257487175-29976-0'}
00:01:48 stkensrv3n.fnal.gov 029976 root I ENCP  File /dev/null copied successfully.
00:01:48 stkensrv3n.fnal.gov 029976 root I ENCP  Completed transferring 1376065438 bytes in 1 files in 133.742175102 sec.  Overall rate = 9.98 MB/sec.  Transfer rate = 82.5 MB/sec.  Network rate = 83 MB/sec.  Drive rate = 108 MB/sec.  Disk rate = 83 MB/sec.  Exit status = 0.
00:01:48 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:01:48 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:49 stkensrv4n.fnal.gov 009037 enstore I SL8500G1MC  query vol VOS175 => 0,VOS175      home              0,11,42, 0, 1         LTO-800G
00:01:49 stkensrv4n.fnal.gov 009024 enstore I SL8500G1MC  mount VON245 0,5,1,7 => 0,Mount: VON245 mounted on   0, 5, 1, 7
00:01:49 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  FINISHED mount VON245 0,5,1,7 returned ('ok', 0, 'mount VON245 0,5,1,7 => 0,Mount: VON245 mounted on   0, 5, 1, 7', '', '')
00:01:49 last message repeated 1 times
00:01:49 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_mounts = '25', last_access = '2009-11-06 00:01:49' where label = 'VON245';
00:01:49 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  MSG_TYPE=MC_LOAD_DONE  mounted VON245 ULTRIUM-TD4 00:01:49 Thread media_thread
00:01:49 stkendca20a.fnal.gov 026781 root I ENCP  Version: v3_8b  CVS $Revision: 1.922 $ <frozen>  OS: Linux 2.6.9-89.0.9.ELsmp i686 Release:  Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)  Library: CD-LTO4G1  Storage Group: exp-db  File Family: archive-d0-offline  FF Wrapper: cpio_odc  FF Width: 1  Current working directory: stkendca20a.fnal.gov:/home/enstore/dcache-code/dcache-fermi-config  Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/exp-db/archive/d0-offline/d0ofprd1/Nov05/arc_819772_819838.tar --put-cache 0030000000000000061FF670 /diskb/write-pool-2/data/0030000000000000061FF670
00:01:50 stkendca20a.fnal.gov 026781 root I ENCP  Creating zero length output files.
00:01:50 stkendca20a.fnal.gov 026781 root I ENCP  Sending /pnfs/fnal.gov/usr/exp-db/archive/d0-offline/d0ofprd1/Nov05/arc_819772_819838.tar write request to LM: unique_id: stkendca20a.fnal.gov-1257487310-26781-0 inputfile: /diskb/write-pool-2/data/0030000000000000061FF670 outputfile: /pnfs/fnal.gov/usr/exp-db/archive/d0-offline/d0ofprd1/Nov05/.(access)(0030000000000000061FF670)
00:01:50 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  MSG_TYPE=ADD_TO_LMQ  request added to pending queue for storage group : exp-db
00:01:50 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  write Q'd /diskb/write-pool-2/data/0030000000000000061FF670 (stkendca20a.fnal.gov-1257487310-26781-0) -> /pnfs/fnal.gov/usr/exp-db/archive/d0-offline/d0ofprd1/Nov05/arc_819772_819838.tar : library=CD-LTO4G1 family=archive-d0-offline requester:root volume_family:exp-db.archive-d0-offline.cpio_odc
00:01:50 stkendca20a.fnal.gov 026781 root I ENCP  Listening for control socket at: ('131.225.13.12', 44956)
00:01:50 stkendca20a.fnal.gov 026781 root I ENCP  Waiting for mover to connect control socket at address: ('131.225.13.12', 44956)  elapsed=0.970sec
00:01:50 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Assigning  volume VON399 from storage group exp-db.archive-d0-offline.cpio_odc to library CD-LTO4G1, volume family exp-db.archive-d0-offline.cpio_odc
00:01:50 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  write_to_hsm work on vol=VON399 mover=LTO4_108.mover requester:root
00:01:50 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  MSG_TYPE=ADD_TO_LMQ  request added to work at movers queue for storage group : exp-db
00:01:50 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  IDLE:sending {'vc': {'storage_group': 'exp-db', 'library': 'CD-LTO4G1', 'file_family': 'archive-d0-offline', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.archive-d0-offline.cpio_odc', 'file_family_width': 1}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/archive/d0-offline/d0ofprd1/Nov05/arc_819772_819838.tar', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 916990976L, 'mover': 'LTO4_108.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/archive/d0-offline/d0ofprd1/Nov05/arc_819772_819838.tar', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257487310, 'rmajor': 0, 'machine': ('Linux', 'stkendca20a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (16877, 820612056L, 20L, 1, 1602, 2752, 512L, 1257481891, 1257481891, 1254398415), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 916990976L, 'fullname': '/diskb/write-pool-2/data/0030000000000000061FF670', 'inode': 807401072L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskb/write-pool-2/data/0030000000000000061FF670', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/archive/d0-offline/d0ofprd1/Nov05/.(access)(0030000000000000061FF670)', 'fc': {'size': 916990976L, 'external_label': 'VON399', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diskb/write-pool-2/data/0030000000000000061FF670', 'at_the_top': 1, 'local_inode': 271274660L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.12', 44956), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487310.355309, 'encp_start_time': 1257487309.1597381, 'job_queued': 1257487310.0729289, 't0': 1257487309, 'in_queue': 0.28232598304748535}, 'user_level': 'user2', 'unique_id': 'stkendca20a.fnal.gov-1257487310-26781-0'} to mover
00:01:50 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV  WRITE_TO_HSM Thread MainThread
00:01:50 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  can not remove from sg LTO4_108.mover VON399
00:01:50 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  SG: {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]}
00:01:50 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  can not remove from vf LTO4_108.mover VON399
00:01:50 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV  SENDING {'vc': {'storage_group': 'exp-db', 'library': 'CD-LTO4G1', 'file_family': 'archive-d0-offline', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.archive-d0-offline.cpio_odc', 'file_family_width': 1}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/archive/d0-offline/d0ofprd1/Nov05/arc_819772_819838.tar', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 916990976L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7868, 'mc_device': '0,5,1,11', 'compression': 0, 'serial_num': '1310154353', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr208a', 'max_time_in_state': 1200, 'device': 'stkenmvr208a:/dev/rmt/tps0d0n', 'name': 'LTO4_108.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_108.stat', 'mover_address': ('131.225.13.204', 7868), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.204', 46995), 'data_ip': 'stkenmvr208a', 'logname': 'LTO4_108MV', 'hostip': '131.225.13.204', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/archive/d0-offline/d0ofprd1/Nov05/arc_819772_819838.tar', 'uid': 0, 'uname': 'root', 'mtime': 1257487310, 'rmajor': 0, 'machine': ('Linux', 'stkendca20a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 820612056L, 20L, 1, 1602, 2752, 512L, 1257481891, 1257481891, 1254398415), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 916990976L, 'gname': 'root', 'gid': 0, 'fullname': '/diskb/write-pool-2/data/0030000000000000061FF670', 'inode': 807401072L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskb/write-pool-2/data/0030000000000000061FF670', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/archive/d0-offline/d0ofprd1/Nov05/.(access)(0030000000000000061FF670)', 'fc': {'size': 916990976L, 'external_label': 'VON399', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diskb/write-pool-2/data/0030000000000000061FF670', 'at_the_top': 1, 'local_inode': 271274660L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.12', 44956), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487310.355309, 'encp_start_time': 1257487309.1597381, 'job_queued': 1257487310.0729289, 't0': 1257487309, 'in_queue': 0.28232598304748535}, 'user_level': 'user2', 'unique_id': 'stkendca20a.fnal.gov-1257487310-26781-0'} Thread client_connect_thread
00:01:50 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  VF: {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]}
00:01:50 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:01:50 stkendca20a.fnal.gov 026781 root I ENCP  Control socket ('131.225.13.12', 44956) is connected to ('131.225.13.204', 46996) for stkendca20a.fnal.gov-1257487310-26781-0.  elapsed=1.250sec
00:01:50 stkendca20a.fnal.gov 026781 root I ENCP  Received callback ticket from mover LTO4_108.mover for transfer stkendca20a.fnal.gov-1257487310-26781-0.  elapsed=1.260sec
00:01:50 stkendca20a.fnal.gov 026781 root I ENCP  Atempting to connect data socket to mover at ('131.225.13.204', 46995).
00:01:50 stkendca20a.fnal.gov 026781 root I ENCP  Connecting data socket to mover (('131.225.13.204', 46995)) with interface 131.225.13.12.  elapsed=1.260sec
00:01:50 stkendca20a.fnal.gov 026781 root I ENCP  Data socket ('131.225.13.12', 44957) is connected to ('131.225.13.204', 46995).  elapsed=1.260sec
00:01:50 stkendca20a.fnal.gov 026781 root I ENCP  Opening local file.
00:01:50 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV  MSG_TYPE=MC_LOAD_REQ  mounting VON399 ULTRIUM-TD4 00:01:50 Thread media_thread
00:01:50 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  PROCESSING mount VON399 0,5,1,11
00:01:50 stkendca20a.fnal.gov 026781 root I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = False
00:01:50 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:01:50 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:50 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:01:51 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:01:52 last message repeated 1 times
00:01:52 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:52 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV  skip_fm 1856 Thread seek_thread
00:01:52 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV  write_tape starting, bytes_to_write=24394240 Thread tape_thread
00:01:52 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV  Write starting. Tape VOS188 absolute location in blocks 4104813 Thread tape_thread
00:01:52 cmsstor88.fnal.gov 031105 root I ENCP  File /storage/data1/write-pool-1/data/0010000000000000094F6448 transfered for cmsstor88.fnal.gov-1257487197-31105-0 in 0.226191997528 seconds.
00:01:52 stkensrv4n.fnal.gov 009037 enstore I SL8500G1MC  query drive 0,7,1,3 => 0,0,7,1,3 online          available              IBM-LTO4
00:01:53 stkensrv4n.fnal.gov 009044 enstore I SL8500G1MC  query drive 0,5,1,11 => 0,0,5,1,11 online          available              IBM-LTO4
00:01:53 stkensrv4n.fnal.gov 009044 enstore I SL8500G1MC  Dismount 14 ignored: Drive 0,5,1,11 is empty. Thought VON399 was there =>. dismount VOLUME 0,5,1,11 force => ok,0,5,1,11 online          available              IBM-LTO4
00:01:53 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV  written bytes 24394240/24394240, blocks 187 header 146 trailer 422 Thread tape_thread
00:01:54 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:01:54 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:56 stkensrv4n.fnal.gov 009044 enstore I SL8500G1MC  query vol VON399 => 0,VON399      home              0, 5,42, 9, 1         LTO-800G
00:01:56 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:01:56 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV  filemarks written. Tape VOS188 absolute location in blocks 4105001 Thread tape_thread
00:01:56 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV  write_tape timing:pull 0.00137948989868 write 0.103654146194 crc_check 0.0946969985962 freespace 0.00135540962219 block_write 0.201086044312 idle 0.0024266242981 Thread tape_thread
00:01:56 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV  drive stats after write. Tape VOS188 position 1856 block 0 block_size 0 bloc_loc 4105001 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 24394240 block_write_tot 0.201086044312 tape_rate 121312446.538 Thread tape_thread
00:01:56 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV  remaining bytes info in DB 271936512000 estimated 271912117760 reported from drive 271912960000 position 1857 Thread tape_thread
00:01:56 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV  new bitfile request {'uid': 0, 'external_label': 'VOS188', 'gid': 0, 'location_cookie': '0000_000000000_0001856', 'complete_crc': 4136823898L, 'sanity_cookie': (65536L, 3155346150L), 'size': 24393672L} Thread tape_thread
00:01:56 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:56 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV  set remaining: VOS188 271912960000 0000_000000000_0001857 Thread tape_thread
00:01:56 last message repeated 1 times
00:01:56 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set last_access = '2009-11-06 00:01:56', eod_cookie = '0000_000000000_0001857', non_del_files = '1871', remaining_bytes = 271912960000 where label = 'VOS188';
00:01:56 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV  transfer complete volume=VOS188 location=1857 Thread tape_thread
00:01:56 last message repeated 1 times
00:01:56 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_wr_access = '1857', modification_time = '2009-11-06 00:01:56' where label = 'VOS188';
00:01:56 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV  Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1244477615.0, 'library': 'CD-LTO4G1', 'si_time': [1256668425.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOS188', 'wrapper': 'cpio_odc', 'remaining_bytes': 271936512000L, 'sum_mounts': 434, 'sum_rd_access': 1, 'file_family_width': 7, 'last_access': 1257486904.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0001856', 'capacity_bytes': 858993459200L, 'non_del_files': 1870, 'sum_wr_err': 1, 'sum_wr_access': 1856, 'file_family': 'cms2', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms2.cpio_odc', 'modification_time': 1257486904.0, 'write_protected': 'n', 'sum_rd_err': 1, 'first_access': 1250512534.0}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX/2/olga/JPT_AKT05_170_230/JetAnalysis_1.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 24393672L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7848, 'mc_device': '0,5,1,15', 'compression': 0, 'serial_num': '9310036898', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr148a', 'max_time_in_state': 1200, 'device': 'stkenmvr148a:/dev/rmt/tps0d0n', 'name': 'LTO4_48.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_48.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.176', 7848), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.176', 47647), 'data_ip': 'stkenmvr148a', 'logname': 'LTO4_48MV', 'hostip': '131.225.13.176', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX/2/olga/JPT_AKT05_170_230/JetAnalysis_1.root', 'uid': 0, 'uname': 'root', 'mtime': 1257487197, 'rmajor': 0, 'machine': ('Linux', 'cmsstor88.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16895, 273637888, 25L, 1, 9811, 5063, 512, 1257487130, 1257487130, 1257485909), 'mode_octal': '0100040', 'mode': 33188, 'size_bytes': 24393672L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data1/write-pool-1/data/0010000000000000094F6448', 'inode': 273638472L, 'minor': 25}, 'version': 'v3_7d  CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/0010000000000000094F6448', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX/2/olga/JPT_AKT05_170_230/.(access)(0010000000000000094F6448)', 'fc': {'uid': 0, 'external_label': 'VOS188', 'gid': 0, 'location_cookie': '0000_000000000_0001856', 'complete_crc': 4136823898L, 'bfid': 'CDMS125748731600000', 'sanity_cookie': (65536L, 3155346150L), 'size': 24393672L}, 'infilepath': '/storage/data1/write-pool-1/data/0010000000000000094F6448', 'at_the_top': 2, 'local_inode': 229, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.189.88', 46415), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257487196.6543159, 'seek_time': 114.93135213851929, 'transfer_time': 4.029094934463501, 't0': 1257487196, 'lm_dequeued': 1257487197.6688409, 'job_queued': 1257487197.575644, 'drive_call_time': 0.11188244819641113, 'in_queue': 0.093137025833129883, 'drive_transfer_time': 3.5749497413635254}, 'user_level': 'admin', 'unique_id': 'cmsstor88.fnal.gov-1257487197-31105-0'} Thread tape_thread
00:01:56 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV  Done is sent Thread tape_thread
00:01:56 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV  maybe_clean Thread tape_thread
00:01:56 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV  returned from maybe_clean Thread tape_thread
00:01:56 cmsstor88.fnal.gov 031105 root I ENCP  Received final dialog for cmsstor88.fnal.gov-1257487197-31105-0.  elapsed=120.090sec
00:01:56 cmsstor88.fnal.gov 031105 root I ENCP  leaving transfer_file().
00:01:56 cmsstor88.fnal.gov 031105 root I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = False
00:01:56 stkensrv0n.fnal.gov 028796 enstore I FILSRV  Updating  			update file set deleted = 'n', pnfs_path = '/pnfs/fnal.gov/usr/cms/WAX/2/olga/JPT_AKT05_170_230/JetAnalysis_1.root', drive = 'stkenmvr148a:/dev/rmt/tps0d0n:9310036898', pnfs_id = '0010000000000000094F6448' where bfid = 'CDMS125748731600000';
00:01:56 cmsstor88.fnal.gov 031105 root I ENCP  MSG_TYPE=ENCP_XFER    write_to_hsm /storage/data1/write-pool-1/data/0010000000000000094F6448 -> /pnfs/fnal.gov/usr/cms/WAX/2/olga/JPT_AKT05_170_230/JetAnalysis_1.root: 24393672 bytes copied to VOS188 at 5.74 MB/S (95.5 MB/S network) (6.51 MB/S drive) (112 MB/S disk) mover=LTO4_48.mover drive_id=ULTRIUM-TD4 drive_sn=9310036898 drive_vendor=IBM elapsed=120.33 {'storage_group': 'cms', 'encp_crc': 4136823898L, 'disk_rate': 112.4443256199871, 'drive_rate': 6.5073973488548349, 'driver': 'FTTDriver', 'transfer_rate': 5.7443915102478922, 'mover_interface': 'stkenmvr148a', 'overall_rate': 0.19507414994570474, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 5.7443915102478922, 'encp_ip': '131.225.189.88', 'unique_id': 'cmsstor88.fnal.gov-1257487197-31105-0'}
00:01:56 cmsstor88.fnal.gov 031105 root I ENCP  File /pnfs/fnal.gov/usr/cms/WAX/2/olga/JPT_AKT05_170_230/JetAnalysis_1.root copied successfully.
00:01:56 cmsstor88.fnal.gov 031105 root I ENCP  Completed transferring 24393672 bytes in 1 files in 120.331575155 sec.  Overall rate = 0.195 MB/sec.  Transfer rate = 5.74 MB/sec.  Network rate = 95.5 MB/sec.  Drive rate = 6.51 MB/sec.  Disk rate = 112 MB/sec.  Exit status = 0.
00:01:57 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} >
00:01:58 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:01:58 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:01:58 stkensrv4n.fnal.gov 009044 enstore I SL8500G1MC  query drive 0,5,1,11 => 0,0,5,1,11 online          available              IBM-LTO4
00:01:59 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} >
00:02:00 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:00 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:01 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} >
00:02:02 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:02 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:03 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} >
00:02:04 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:04 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:05 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} >
00:02:06 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:06 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:07 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} >
00:02:07 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  tape position: target 110 current 0 1 Thread seek_thread
00:02:08 stkendca19a.fnal.gov 006120 root I ENCP  Version: v3_8b  CVS $Revision: 1.922 $ <frozen>  OS: Linux 2.6.9-89.0.9.ELsmp i686 Release:  Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)  Library: CD-LTO4G1  Storage Group: exp-db  File Family: daily-d0-offline  FF Wrapper: cpio_odc  FF Width: 2  Current working directory: stkendca19a.fnal.gov:/home/enstore/dcache-code/dcache-fermi-config  Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702152110_s103297_p27 --put-cache 003000000000000006208660 /diska/write-pool-1/data/003000000000000006208660
00:02:08 stkendca19a.fnal.gov 006120 root I ENCP  Creating zero length output files.
00:02:08 stkendca19a.fnal.gov 006120 root I ENCP  Sending /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702152110_s103297_p27 write request to LM: unique_id: stkendca19a.fnal.gov-1257487328-6120-0 inputfile: /diska/write-pool-1/data/003000000000000006208660 outputfile: /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/.(access)(003000000000000006208660)
00:02:08 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  MSG_TYPE=ADD_TO_LMQ  request added to pending queue for storage group : exp-db
00:02:08 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  write Q'd /diska/write-pool-1/data/003000000000000006208660 (stkendca19a.fnal.gov-1257487328-6120-0) -> /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702152110_s103297_p27 : library=CD-LTO4G1 family=daily-d0-offline requester:root volume_family:exp-db.daily-d0-offline.cpio_odc
00:02:08 stkendca19a.fnal.gov 006120 root I ENCP  Listening for control socket at: ('131.225.13.11', 51940)
00:02:08 stkendca19a.fnal.gov 006120 root I ENCP  Waiting for mover to connect control socket at address: ('131.225.13.11', 51940)  elapsed=1.100sec
00:02:08 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:08 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} >
00:02:10 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:10 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} >
00:02:12 stkensrv4n.fnal.gov 005357 enstore E LTO4G1LM  MOVER ERROR RQ {'status': ('ok', None), 'returned_work': None, 'library': None, 'mover_type': '', 'current_location': '0000_000000000_0000000', 'ip_map': '', 'operation': None, 'read_only': 0, 'current_time': 1257487332.376435, 'error_source': None, 'external_label': None, 'state': 'OFFLINE', 'r_a': (('131.225.13.194', 54880), 18008L, '131.225.13.194-54880-1256586929.509013-4866'), 'library_list': [('CD-LTO4G1.library_manager', ('131.225.13.4', 8541))], 'current_priority': (None, None), 'transfer_deficiency': 1, 'time_in_state': 900402.0163769722, 'address': ('131.225.13.194', 7860), 'volume_family': None, 'work': 'mover_error', 'mover': 'LTO4_99.mover', 'volume_status': (['none', 'none'], ['none', 'none']), 'unique_id': None}
00:02:12 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:12 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:13 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} >
00:02:14 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:14 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:15 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} >
00:02:16 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:16 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:17 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} >
00:02:18 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:18 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:19 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} >
00:02:19 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV  skip_fm 394 Thread seek_thread
00:02:19 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV  write_tape starting, bytes_to_write=2147467776 Thread tape_thread
00:02:19 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV  Write starting. Tape VON102 absolute location in blocks 6090662 Thread tape_thread
00:02:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:20 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:20 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:21 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} >
00:02:22 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:22 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:23 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} >
00:02:24 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:24 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:25 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} >
00:02:26 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:26 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:27 stkenmvr23a.fnal.gov 005810 root I DBT23MV  read bytes 6015745268/6015745268, blocks 45897 header 0 Thread tape_thread
00:02:27 stkenmvr23a.fnal.gov 005810 root I DBT23MV  write_client: wrote 6015745268/6015745268 bytes Thread net_thread
00:02:27 stkenmvr23a.fnal.gov 005810 root I DBT23MV  write_client waits for tape thread to exit Thread net_thread
00:02:27 stkenmvr23a.fnal.gov 005810 root I DBT23MV  drive stats after read. Tape VO5178 position 146 block 45897 block_size 0 bloc_loc 1553708 tot_blocks 0 BOT 0 read_err 0 write_err 0 bytes 6015745268 block_read_tot 163.28999567 tape_rate 36840868.5621 Thread tape_thread
00:02:27 stkenmvr23a.fnal.gov 005810 root I DBT23MV  read_tape exiting, read 6015745268/6015745268 bytes Thread tape_thread
00:02:27 stkenmvr23a.fnal.gov 005810 root I DBT23MV  read_tape timing:get_space 1.47773766518 read 133.168499231 crc_check 27.7483925819 push 0.895366191864 block_read 163.28999567 idle 1.79660272598 Thread tape_thread
00:02:27 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} >
00:02:27 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV  Updating stats Thread media_thread
00:02:27 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV  writing a tape mark before dismount Thread media_thread
00:02:27 stkendm7a.fnal.gov 009044 root I ENCP  File /pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZE0/KZE047/f1 transfered for stkendm7a.fnal.gov-1257486813-9044-75 in 170.547472 seconds. Thread READ_1
00:02:28 stkenmvr23a.fnal.gov 005810 root I DBT23MV  write_client exits Thread net_thread
00:02:28 stkenmvr23a.fnal.gov 005810 root I DBT23MV  transfer complete volume=VO5178 location=146 Thread net_thread
00:02:28 last message repeated 1 times
00:02:28 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_rd_access = '88', last_access = '2009-11-06 00:02:28' where label = 'VO5178';
00:02:28 stkenmvr23a.fnal.gov 005810 root I DBT23MV  Sending done to client: {'vc': {'comment': 'copied new media 04/24/06, bad file 86 and 125', 'declared': 1077641864.0, 'blocksize': 131072, 'storage_group': 'ktev', 'library': 'CD-9940B', 'eod_cookie': '0000_000000000_0000148', 'current_location': '0000_000000000_0000146', 'user_inhibit': ['none', 'none'], 'sum_rd_access': 87, 'system_inhibit': ['none', 'migrating'], 'external_label': 'VO5178', 'wrapper': 'cern', 'remaining_bytes': 1269815808L, 'sum_mounts': 21, 'capacity_bytes': 214748364800L, 'media_type': '9940B', 'last_access': 1257487177.0, 'status': ('ok', None), 'si_time': [1145990018.0, 1257481562.0], 'non_del_files': 49, 'sum_wr_err': 0, 'sum_wr_access': 49, 'file_family': 'ktev_migration', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration.cern', 'modification_time': 1218307054.0, 'write_protected': 'y', 'sum_rd_err': 16, 'first_access': 1078413242.0}, 'outfilepath': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000146', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 6015745268L, 'mover': {'max_buffer': 1048576000L, 'check_written_file': 10, 'max_failures': 5, 'library': 'CD-9940B.library_manager', 'local_mover': 0, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7583, 'mc_device': '0,0,10,4', 'compression': 0, 'serial_num': '479002042523', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 28311552.0, 'vendor_id': 'STK', 'driver': 'FTTDriver', 'host': 'stkenmvr23a', 'max_time_in_state': 1200, 'device': 'stkenmvr23a:/dev/rmt/tps0d0n', 'name': '9940B23.mover', 'send_stats': 1, 'product_id': 'T9940B', 'statistics_path': '/tmp/enstore/enstore/DBT23MV.stat', 'mover_address': ('131.225.13.32', 7583), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.32', 50946), 'data_ip': 'stkenmvr23a', 'logname': 'DBT23MV', 'hostip': '131.225.13.32', 'do_cleaning': 'No', 'media_changer': 'stk.media_changer'}, 'lm': {'address': ('131.225.13.4', 7522)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZE0/KZE047/f1', 'uid': 0, 'uname': 'root', 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (33204, 525757152L, 20L, 1, 5406, 2539, 1L, 1150922484, 1078424509, 1078362565), 'mode_octal': '0100764', 'mode': 33268, 'size_bytes': 6015745268L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000146', 'inode': 525757152L, 'minor': 20}, 'version': 'v3_8  CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZE0/KZE047/.(access)(001F00000000000000566AE0)', 'status': ('ok', None), 'volume': 'VO5178', 'outfile': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000146', 'fc': {'status': ('ok', None), 'pnfsid': '001F00000000000000566AE0', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZE0/KZE047/f1', 'deleted': 'no', 'external_label': 'VO5178', 'drive': 'stkenmvr11a:/dev/rmt/tps0d0n:4790000019', 'update': None, 'gid': -1, 'location_cookie': '0000_000000000_0000146', 'complete_crc': 383801200L, 'address': ('131.225.13.59', 7501), 'uid': -1, 'bfid': 'CDMS107842450700000', 'sanity_cookie': (65536L, 2029974703L), 'size': 6015745268L}, 'infilepath': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZE0/KZE047/f1', 'at_the_top': 1683, 'bfid': 'CDMS107842450700000', 'local_inode': 39503170L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 44344), 'work': 'read_from_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257486813.230767, 'seek_time': 0.023787021636962891, 'transfer_time': 170.7384979724884, 't0': 1257486813, 'mount_time': 0.019741058349609375, 'lm_dequeued': 1257487177.322459, 'job_queued': 1257486813.4214711, 'drive_call_time': 131.72889757156372, 'in_queue': 363.900930881500(trunc. 4204) Thread net_thread
00:02:28 stkenmvr23a.fnal.gov 005810 root I DBT23MV  Done is sent Thread net_thread
00:02:28 stkenmvr23a.fnal.gov 005810 root I DBT23MV  maybe_clean Thread net_thread
00:02:28 stkendm7a.fnal.gov 009044 root I ENCP  Received final dialog for stkendm7a.fnal.gov-1257486813-9044-75.  elapsed=5785.580sec Thread READ_1
00:02:28 stkendm7a.fnal.gov 009044 root I ENCP  leaving transfer_file(). Thread READ_1
00:02:28 stkenmvr23a.fnal.gov 005810 root I DBT23MV  returned from maybe_clean Thread net_thread
00:02:28 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:02:28 stkendm7a.fnal.gov 009044 root I ENCP  MSG_TYPE=ENCP_XFER    read_from_hsm /pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZE0/KZE047/f1 -> /data/data1/Migration_Spool/VO5178:0000_000000000_0000146: 6015745268 bytes copied from VO5178 at 33.6 MB/S (50.3 MB/S network) (34.9 MB/S drive) (60.4 MB/S disk) mover=9940B23.mover drive_id=T9940B drive_sn=479002042523 drive_vendor=STK elapsed=535.23 {'storage_group': 'ktev', 'encp_crc': 383801200L, 'disk_rate': 60.40163121721973, 'drive_rate': 34.948160328947225, 'driver': 'FTTDriver', 'transfer_rate': 33.589865678149209, 'mover_interface': 'stkenmvr23a', 'overall_rate': 33.533790348403421, 'media_changer': 'stk.media_changer', 'network_rate': 33.589865678149209, 'encp_ip': '131.225.13.19', 'unique_id': 'stkendm7a.fnal.gov-1257486813-9044-75'} Thread READ_1
00:02:28 stkendm7a.fnal.gov 009044 root I ENCP  File /data/data1/Migration_Spool/VO5178:0000_000000000_0000146 copied successfully. Thread READ_1
00:02:28 stkendm7a.fnal.gov 009044 root I ENCP  Completed transferring 6015745268 bytes in 1 files in 535.245552063 sec.  Overall rate = 33.5 MB/sec.  Transfer rate = 33.6 MB/sec.  Network rate = 50.3 MB/sec.  Drive rate = 34.9 MB/sec.  Disk rate = 60.4 MB/sec.  Exit status = 0. Thread READ_1
00:02:28 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:28 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:29 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV  Ejecting tape Thread media_thread
00:02:29 stkensrv4n.fnal.gov 009044 enstore I SL8500G1MC  mount VON399 0,5,1,11 => 0,Mount: VON399 mounted on   0, 5, 1,11
00:02:29 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  FINISHED mount VON399 0,5,1,11 returned ('ok', 0, 'mount VON399 0,5,1,11 => 0,Mount: VON399 mounted on   0, 5, 1,11', '', '')
00:02:29 last message repeated 1 times
00:02:29 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_mounts = '273', last_access = '2009-11-06 00:02:29' where label = 'VON399';
00:02:29 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV  MSG_TYPE=MC_LOAD_DONE  mounted VON399 ULTRIUM-TD4 00:02:29 Thread media_thread
00:02:30 stkendca22a.fnal.gov 022603 root I ENCP  Version: v3_8b  CVS $Revision: 1.922 $ <frozen>  OS: Linux 2.6.9-89.0.9.ELsmp i686 Release:  Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)  Library: garbage  Storage Group: test  File Family: dcache  FF Wrapper: cpio_odc  FF Width: 1  Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log  Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0003 --put-cache 0005000000000000005FA5F8 /diskb/write-pool-2/data/0005000000000000005FA5F8
00:02:30 stkendca22a.fnal.gov 022632 root I ENCP  Version: v3_8b  CVS $Revision: 1.922 $ <frozen>  OS: Linux 2.6.9-89.0.9.ELsmp i686 Release:  Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)  Library: garbage  Storage Group: test  File Family: dcache  FF Wrapper: cpio_odc  FF Width: 1  Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log  Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0001 --put-cache 0005000000000000005FA590 /diskb/write-pool-2/data/0005000000000000005FA590
00:02:30 stkendca22a.fnal.gov 022644 root I ENCP  Version: v3_8b  CVS $Revision: 1.922 $ <frozen>  OS: Linux 2.6.9-89.0.9.ELsmp i686 Release:  Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)  Library: garbage  Storage Group: test  File Family: dcache  FF Wrapper: cpio_odc  FF Width: 1  Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log  Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0002 --put-cache 0005000000000000005FA5C8 /diskb/write-pool-2/data/0005000000000000005FA5C8
00:02:30 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:02:30 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:30 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:30 stkendca22a.fnal.gov 022603 root I ENCP  Configuration Server: no such name: 'garbage.library_manager'
00:02:30 stkendca22a.fnal.gov 022603 root E ENCP  INFILE=/diskb/write-pool-2/data/0005000000000000005FA5F8 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0003 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=0.90 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager'
00:02:30 stkendca22a.fnal.gov 022632 root I ENCP  Configuration Server: no such name: 'garbage.library_manager'
00:02:30 stkendca22a.fnal.gov 022632 root E ENCP  INFILE=/diskb/write-pool-2/data/0005000000000000005FA590 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0001 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=0.89 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager'
00:02:30 stkendca22a.fnal.gov 022644 root I ENCP  Configuration Server: no such name: 'garbage.library_manager'
00:02:30 stkendca22a.fnal.gov 022644 root E ENCP  INFILE=/diskb/write-pool-2/data/0005000000000000005FA5C8 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0002 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=0.91 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager'
00:02:30 stkensrv4n.fnal.gov 009037 enstore I SL8500G1MC  mount VOS175 0,7,1,3 => 0,Mount: VOS175 mounted on   0, 7, 1, 3
00:02:30 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  FINISHED mount VOS175 0,7,1,3 returned ('ok', 0, 'mount VOS175 0,7,1,3 => 0,Mount: VOS175 mounted on   0, 7, 1, 3', '', '')
00:02:30 last message repeated 1 times
00:02:30 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_mounts = '688', last_access = '2009-11-06 00:02:30' where label = 'VOS175';
00:02:30 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV  MSG_TYPE=MC_LOAD_DONE  mounted VOS175 ULTRIUM-TD4 00:02:30 Thread media_thread
00:02:32 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:02:32 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:32 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:32 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  skip_fm 283 Thread seek_thread
00:02:32 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  write_tape starting, bytes_to_write=1966080512 Thread tape_thread
00:02:32 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  Write starting. Tape VON268 absolute location in blocks 4265959 Thread tape_thread
00:02:34 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:02:34 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:34 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:35 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:36 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:02:36 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:36 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:38 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:02:38 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:38 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:40 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:02:40 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:40 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:42 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:02:42 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:42 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:44 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:02:44 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:44 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:45 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642'), ('LTO4_48.mover', 'VOS188')]} >
00:02:46 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:02:46 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:46 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:47 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV  tape position: target 366 current 0 1 Thread seek_thread
00:02:48 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:02:48 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:48 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV  Updating stats Thread media_thread
00:02:48 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV  volume VOS642 write protection 1  override_ro_mount 1 Thread media_thread
00:02:48 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV  Ejecting tape Thread media_thread
00:02:48 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:48 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV  tape position: target 2918 current 0 1 Thread seek_thread
00:02:48 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_43.mover', 'VON268')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:50 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:02:50 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:51 stkendca17a.fnal.gov 027861 root I ENCP  File /diska/write-pool-1/data/003000000000000006208408 transfered for stkendca17a.fnal.gov-1257487098-27861-0 in 18.7733681202 seconds.
00:02:52 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:02:52 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:53 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  written bytes 1966080512/1966080512, blocks 15001 header 189 trailer 323 Thread tape_thread
00:02:54 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:02:54 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  filemarks written. Tape VON268 absolute location in blocks 4280961 Thread tape_thread
00:02:54 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  write_tape timing:pull 0.209099769592 write 9.67178463936 crc_check 7.95908045769 freespace 0.227706432343 block_write 18.067671299 idle 0.350021600723 Thread tape_thread
00:02:54 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  drive stats after write. Tape VON268 position 283 block 0 block_size 0 bloc_loc 4280961 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 1966080512 block_write_tot 18.067671299 tape_rate 108817593.561 Thread tape_thread
00:02:54 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:54 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  remaining bytes info in DB 251846656000 estimated 249880575488 reported from drive 250011648000 position 284 Thread tape_thread
00:02:54 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  new bitfile request {'uid': 0, 'external_label': 'VON268', 'gid': 0, 'location_cookie': '0000_000000000_0000283', 'complete_crc': 3199621942L, 'sanity_cookie': (65536L, 963967056L), 'size': 1966080000L} Thread tape_thread
00:02:54 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  set remaining: VON268 250011648000 0000_000000000_0000284 Thread tape_thread
00:02:54 last message repeated 1 times
00:02:54 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set last_access = '2009-11-06 00:02:54', eod_cookie = '0000_000000000_0000284', non_del_files = '284', remaining_bytes = 250011648000 where label = 'VON268';
00:02:54 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  transfer complete volume=VON268 location=284 Thread tape_thread
00:02:54 last message repeated 1 times
00:02:54 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_wr_access = '283', modification_time = '2009-11-06 00:02:54' where label = 'VON268';
00:02:54 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  Sending done to client: {'vc': {'comment': '', 'storage_group': 'exp-db', 'blocksize': 131072, 'declared': 1248994058.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 1242753248.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VON268', 'wrapper': 'cpio_odc', 'remaining_bytes': 251846656000L, 'sum_mounts': 149, 'sum_rd_access': 0, 'file_family_width': 2, 'last_access': 1257487062.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000283', 'capacity_bytes': 858993459200L, 'non_del_files': 283, 'sum_wr_err': 0, 'sum_wr_access': 282, 'file_family': 'daily-d0-offline', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'modification_time': 1257487062.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257406785.0}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702152110_s103298_p33', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7843, 'mc_device': '0,7,1,0', 'compression': 0, 'serial_num': '9310055909', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr143a', 'max_time_in_state': 1200, 'device': 'stkenmvr143a:/dev/rmt/tps0d0n', 'name': 'LTO4_43.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_43.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.171', 7843), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.171', 56239), 'data_ip': 'stkenmvr143a', 'logname': 'LTO4_43MV', 'hostip': '131.225.13.171', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702152110_s103298_p33', 'uid': 0, 'uname': 'root', 'mtime': 1257487098, 'rmajor': 0, 'machine': ('Linux', 'stkendca17a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 820619664L, 20L, 1, 1602, 2752, 512L, 1257487037, 1257487037, 1254398418), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 1966080000L, 'gname': 'root', 'gid': 0, 'fullname': '/diska/write-pool-1/data/003000000000000006208408', 'inode': 807437320L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diska/write-pool-1/data/003000000000000006208408', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/.(access)(003000000000000006208408)', 'fc': {'uid': 0, 'external_label': 'VON268', 'gid': 0, 'location_cookie': '0000_000000000_0000283', 'complete_crc': 3199621942L, 'bfid': 'CDMS125748737400000', 'sanity_cookie': (65536L, 963967056L), 'size': 1966080000L}, 'infilepath': '/diska/write-pool-1/data/003000000000000006208408', 'at_the_top': 1263, 'local_inode': 268435947L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.5', 46697), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257487097.963901, 'seek_time': 161.77867388725281, 'transfer_time': 22.018096923828125, 't0': 1257487097, 'lm_dequeued': 1257487190.9960899, 'job_queued': 1257487098.90344, 'drive_call_time': 9.467130184173584, 'in_queue': 92.092375040054321, 'drive_transfer_time': 20.10367751121521}, 'user_level': 'user2', 'unique_id': 'stkendca17a.fnal.go(trunc. 4102) Thread tape_thread
00:02:54 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  Done is sent Thread tape_thread
00:02:54 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  maybe_clean Thread tape_thread
00:02:54 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  returned from maybe_clean Thread tape_thread
00:02:54 stkendca17a.fnal.gov 027861 root I ENCP  Received final dialog for stkendca17a.fnal.gov-1257487098-27861-0.  elapsed=276.940sec
00:02:54 stkendca17a.fnal.gov 027861 root I ENCP  leaving transfer_file().
00:02:55 stkendca17a.fnal.gov 027861 root I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = False
00:02:55 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_43.mover', 'VON268')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:55 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  write_to_hsm next work on vol=VON268 mover=LTO4_43.mover requester:root
00:02:55 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  HAVE_BOUND:sending write_to_hsm /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702155247_s103301_p16 to mover LTO4_43.mover ('131.225.13.171', 7843) DEL_DISM None
00:02:55 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  MSG_TYPE=ADD_TO_LMQ  request added to work at movers queue for storage group : exp-db
00:02:55 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  HAVE_BOUND:sending {'vc': {'storage_group': 'exp-db', 'external_label': 'VON268', 'library': 'CD-LTO4G1', 'file_family': 'daily-d0-offline', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'file_family_width': 2}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702155247_s103301_p16', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': 'LTO4_43.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702155247_s103301_p16', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257487158, 'rmajor': 0, 'machine': ('Linux', 'stkendca19a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (16877, 820619664L, 20L, 1, 1602, 2752, 512L, 1257487155, 1257487155, 1254398418), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 1966080000L, 'fullname': '/diskc/write-pool-3/data/003000000000000006208530', 'inode': 807437616L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskc/write-pool-3/data/003000000000000006208530', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/.(access)(003000000000000006208530)', 'fc': {'size': 1966080000L, 'external_label': 'VON268', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diskc/write-pool-3/data/003000000000000006208530', 'at_the_top': 2151, 'local_inode': 575188415L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.11', 51925), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487375.116276, 'encp_start_time': 1257487157.9395051, 'job_queued': 1257487158.9967749, 't0': 1257487157, 'in_queue': 216.11900115013123}, 'user_level': 'user2', 'unique_id': 'stkendca19a.fnal.gov-1257487158-4614-0'} to mover
00:02:55 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_43.mover', 'VON268')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:55 stkensrv0n.fnal.gov 028796 enstore I FILSRV  Updating  			update file set deleted = 'n', pnfs_path = '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702152110_s103298_p33', drive = 'stkenmvr143a:/dev/rmt/tps0d0n:9310055909', pnfs_id = '003000000000000006208408' where bfid = 'CDMS125748737400000';
00:02:55 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  WRITE_TO_HSM Thread MainThread
00:02:55 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  SENDING {'vc': {'storage_group': 'exp-db', 'external_label': 'VON268', 'library': 'CD-LTO4G1', 'file_family': 'daily-d0-offline', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'file_family_width': 2}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702155247_s103301_p16', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7843, 'mc_device': '0,7,1,0', 'compression': 0, 'serial_num': '9310055909', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr143a', 'max_time_in_state': 1200, 'device': 'stkenmvr143a:/dev/rmt/tps0d0n', 'name': 'LTO4_43.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_43.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.171', 7843), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.171', 56251), 'data_ip': 'stkenmvr143a', 'logname': 'LTO4_43MV', 'hostip': '131.225.13.171', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702155247_s103301_p16', 'uid': 0, 'uname': 'root', 'mtime': 1257487158, 'rmajor': 0, 'machine': ('Linux', 'stkendca19a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 820619664L, 20L, 1, 1602, 2752, 512L, 1257487155, 1257487155, 1254398418), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 1966080000L, 'gname': 'root', 'gid': 0, 'fullname': '/diskc/write-pool-3/data/003000000000000006208530', 'inode': 807437616L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskc/write-pool-3/data/003000000000000006208530', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/.(access)(003000000000000006208530)', 'fc': {'size': 1966080000L, 'external_label': 'VON268', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diskc/write-pool-3/data/003000000000000006208530', 'at_the_top': 2151, 'local_inode': 575188415L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.11', 51925), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487375.116276, 'encp_start_time': 1257487157.9395051, 'job_queued': 1257487158.9967749, 't0': 1257487157, 'in_queue': 216.11900115013123}, 'user_level': 'user2', 'unique_id': 'stkendca19a.fnal.gov-1257487158-4614-0'} Thread client_connect_thread
00:02:55 stkendca19a.fnal.gov 004614 root I ENCP  Control socket ('131.225.13.11', 51925) is connected to ('131.225.13.171', 56252) for stkendca19a.fnal.gov-1257487158-4614-0.  elapsed=217.230sec
00:02:55 stkendca19a.fnal.gov 004614 root I ENCP  Received callback ticket from mover LTO4_43.mover for transfer stkendca19a.fnal.gov-1257487158-4614-0.  elapsed=217.230sec
00:02:55 stkendca19a.fnal.gov 004614 root I ENCP  Atempting to connect data socket to mover at ('131.225.13.171', 56251).
00:02:55 stkendca19a.fnal.gov 004614 root I ENCP  Connecting data socket to mover (('131.225.13.171', 56251)) with interface 131.225.13.11.  elapsed=217.230sec
00:02:55 stkendca19a.fnal.gov 004614 root I ENCP  Data socket ('131.225.13.11', 51942) is connected to ('131.225.13.171', 56251).  elapsed=217.230sec
00:02:55 stkendca19a.fnal.gov 004614 root I ENCP  Opening local file.
00:02:55 stkendca17a.fnal.gov 027861 root I ENCP  MSG_TYPE=ENCP_XFER    write_to_hsm /diska/write-pool-1/data/003000000000000006208408 -> /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702152110_s103298_p33: 1966080000 bytes copied to VON268 at 85.1 MB/S (81.1 MB/S network) (93.3 MB/S drive) (100 MB/S disk) mover=LTO4_43.mover drive_id=ULTRIUM-TD4 drive_sn=9310055909 drive_vendor=IBM elapsed=277.27 {'storage_group': 'exp-db', 'encp_crc': 3199621942L, 'disk_rate': 100.09063139977049, 'drive_rate': 93.266517976822726, 'driver': 'FTTDriver', 'transfer_rate': 85.090779370718479, 'mover_interface': 'stkenmvr143a', 'overall_rate': 10.180253312816591, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 85.090779370718479, 'encp_ip': '131.225.13.5', 'unique_id': 'stkendca17a.fnal.gov-1257487098-27861-0'}
00:02:55 stkendca17a.fnal.gov 027861 root I ENCP  File /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702152110_s103298_p33 copied successfully.
00:02:55 stkendca17a.fnal.gov 027861 root I ENCP  Completed transferring 1966080000 bytes in 1 files in 277.267912865 sec.  Overall rate = 10.2 MB/sec.  Transfer rate = 85.1 MB/sec.  Network rate = 81.1 MB/sec.  Drive rate = 93.3 MB/sec.  Disk rate = 100 MB/sec.  Exit status = 0.
00:02:55 stkendca19a.fnal.gov 004614 root I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = False
00:02:55 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  tape position: target 284 current 284 0 Thread seek_thread
00:02:55 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  write_tape starting, bytes_to_write=1966080512 Thread tape_thread
00:02:55 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  Write starting. Tape VON268 absolute location in blocks 4280961 Thread tape_thread
00:02:56 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_43.mover', 'VON268')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:02:56 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:02:56 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:58 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:02:58 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:02:58 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV  Tape is ejected Thread media_thread
00:02:58 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV  dismounting VOS188 Thread media_thread
00:02:58 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  PROCESSING dismount VOS188 0,5,1,15
00:03:00 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:00 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:01 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_43.mover', 'VON268'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642')]} >
00:03:01 stkensrv4n.fnal.gov 009077 enstore I SL8500G1MC  query drive 0,5,1,15 => 0,0,5,1,15 online          in use      VOS188     IBM-LTO4
00:03:01 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_43.mover', 'VON268'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_48.mover', 'VOS188'), ('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:03:02 stkensrv4n.fnal.gov 005357 enstore E LTO4G1LM  MOVER ERROR RQ {'status': ('ok', None), 'returned_work': None, 'library': None, 'mover_type': '', 'current_location': '0000_000000000_0000000', 'ip_map': '', 'operation': None, 'read_only': 0, 'current_time': 1257487382.3817611, 'error_source': None, 'external_label': None, 'state': 'OFFLINE', 'r_a': (('131.225.13.194', 54880), 18009L, '131.225.13.194-54880-1256586929.509013-4866'), 'library_list': [('CD-LTO4G1.library_manager', ('131.225.13.4', 8541))], 'current_priority': (None, None), 'transfer_deficiency': 1, 'time_in_state': 900452.02170300484, 'address': ('131.225.13.194', 7860), 'volume_family': None, 'work': 'mover_error', 'mover': 'LTO4_99.mover', 'volume_status': (['none', 'none'], ['none', 'none']), 'unique_id': None}
00:03:02 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:02 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:04 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:04 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_43.mover', 'VON268'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:03:04 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:06 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:06 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:06 stkendca20a.fnal.gov 027216 root I ENCP  Version: v3_8b  CVS $Revision: 1.922 $ <frozen>  OS: Linux 2.6.9-89.0.9.ELsmp i686 Release:  Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)  Library: CD-LTO4G1  Storage Group: exp-db  File Family: daily-d0-offline  FF Wrapper: cpio_odc  FF Width: 2  Current working directory: stkendca20a.fnal.gov:/home/enstore/dcache-code/dcache-fermi-config  Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702152110_s103297_p50 --put-cache 0030000000000000062086E8 /diskc/write-pool-3/data/0030000000000000062086E8
00:03:07 stkendca20a.fnal.gov 027216 root I ENCP  Creating zero length output files.
00:03:07 stkendca20a.fnal.gov 027216 root I ENCP  Sending /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702152110_s103297_p50 write request to LM: unique_id: stkendca20a.fnal.gov-1257487386-27216-0 inputfile: /diskc/write-pool-3/data/0030000000000000062086E8 outputfile: /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/.(access)(0030000000000000062086E8)
00:03:07 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  MSG_TYPE=ADD_TO_LMQ  request added to pending queue for storage group : exp-db
00:03:07 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  write Q'd /diskc/write-pool-3/data/0030000000000000062086E8 (stkendca20a.fnal.gov-1257487386-27216-0) -> /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702152110_s103297_p50 : library=CD-LTO4G1 family=daily-d0-offline requester:root volume_family:exp-db.daily-d0-offline.cpio_odc
00:03:07 stkendca20a.fnal.gov 027216 root I ENCP  Listening for control socket at: ('131.225.13.12', 44962)
00:03:07 stkendca20a.fnal.gov 027216 root I ENCP  Waiting for mover to connect control socket at address: ('131.225.13.12', 44962)  elapsed=1.170sec
00:03:08 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:08 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:10 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:10 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:11 stkendca19a.fnal.gov 005050 root I ENCP  File /diskb/write-pool-2/data/003000000000000006208580 transfered for stkendca19a.fnal.gov-1257487194-5050-0 in 51.406441927 seconds.
00:03:12 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_43.mover', 'VON268')], 'cms': [('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:03:12 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:12 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:13 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV  written bytes 2147467776/2147467776, blocks 16384 header 195 trailer 317 Thread tape_thread
00:03:14 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:14 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:14 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV  filemarks written. Tape VON102 absolute location in blocks 6107047 Thread tape_thread
00:03:14 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV  write_tape timing:pull 0.212910175323 write 6.99778962135 crc_check 9.87317228317 freespace 0.156232595444 block_write 17.2401046753 idle 0.287122488022 Thread tape_thread
00:03:14 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV  drive stats after write. Tape VON102 position 394 block 0 block_size 0 bloc_loc 6107047 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 2147467776 block_write_tot 17.2401046753 tape_rate 124562339.756 Thread tape_thread
00:03:14 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV  remaining bytes info in DB 25657015808 estimated 23509548032 reported from drive 30673920000 position 395 Thread tape_thread
00:03:14 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV  new bitfile request {'uid': 0, 'external_label': 'VON102', 'gid': 0, 'location_cookie': '0000_000000000_0000394', 'complete_crc': 2333110552L, 'sanity_cookie': (65536L, 3482994919L), 'size': 2147467264L} Thread tape_thread
00:03:14 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV  set remaining: VON102 23509548032 0000_000000000_0000395 Thread tape_thread
00:03:14 last message repeated 1 times
00:03:14 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set last_access = '2009-11-06 00:03:14', eod_cookie = '0000_000000000_0000395', non_del_files = '446', remaining_bytes = 23509548032 where label = 'VON102';
00:03:15 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV  transfer complete volume=VON102 location=395 Thread tape_thread
00:03:15 last message repeated 1 times
00:03:15 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_wr_access = '394', last_access = '2009-11-06 00:03:15', modification_time = '2009-11-06 00:03:15' where label = 'VON102';
00:03:15 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV  Sending done to client: {'vc': {'comment': '', 'storage_group': 'exp-db', 'blocksize': 131072, 'declared': 1225477331.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VON102', 'wrapper': 'cpio_odc', 'remaining_bytes': 25657015808L, 'sum_mounts': 179, 'sum_rd_access': 0, 'file_family_width': 2, 'last_access': 1257486795.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000394', 'capacity_bytes': 858993459200L, 'non_del_files': 445, 'sum_wr_err': 0, 'sum_wr_access': 393, 'file_family': 'daily-d0-online-readonly', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-online-readonly.cpio_odc', 'modification_time': 1257486795.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1239342136.0}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-online-readonly/d0onlprd/2009/11-November/05/channel01/D0ONLPRD_t702168009_s2963_p21', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 2147467264L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7864, 'mc_device': '0,6,1,13', 'compression': 0, 'serial_num': '1310050819', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr204a', 'max_time_in_state': 1200, 'device': 'stkenmvr204a:/dev/rmt/tps0d0n', 'name': 'LTO4_104.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_104.stat', 'mover_address': ('131.225.13.199', 7864), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.199', 44921), 'data_ip': 'stkenmvr204a', 'logname': 'LTO4_104MV', 'hostip': '131.225.13.199', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-online-readonly/d0onlprd/2009/11-November/05/channel01/D0ONLPRD_t702168009_s2963_p21', 'uid': 0, 'uname': 'root', 'mtime': 1257487194, 'rmajor': 0, 'machine': ('Linux', 'stkendca19a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 820608480L, 20L, 1, 1602, 2752, 512L, 1257487190, 1257487190, 1254398413), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 2147467264L, 'gname': 'root', 'gid': 0, 'fullname': '/diskb/write-pool-2/data/003000000000000006208580', 'inode': 807437696L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskb/write-pool-2/data/003000000000000006208580', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-online-readonly/d0onlprd/2009/11-November/05/channel01/.(access)(003000000000000006208580)', 'fc': {'uid': 0, 'external_label': 'VON102', 'gid': 0, 'location_cookie': '0000_000000000_0000394', 'complete_crc': 2333110552L, 'bfid': 'CDMS125748739400000', 'sanity_cookie': (65536L, 3482994919L), 'size': 2147467264L}, 'infilepath': '/diskb/write-pool-2/data/003000000000000006208580', 'at_the_top': 1, 'local_inode': 628435375L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.11', 51929), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257487193.1633611, 'seek_time': 145.4759521484375, 'transfer_time': 55.430405855178833, 't0': 1257487193, 'lm_dequeued': 1257487194.232517, 'job_queued': 1257487194.140867, 'drive_call_time': 6.7916271686553955, 'in_queue': 0.09159398078918457, 'drive_transfer_time': 19.271548986434937}, 'user_level': 'user2', 'unique_id': 'stkendca19a.fnal.gov-1257487194-5050-0'} Thread tape_thread
00:03:15 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV  Done is sent Thread tape_thread
00:03:15 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV  maybe_clean Thread tape_thread
00:03:15 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV  returned from maybe_clean Thread tape_thread
00:03:15 stkendca19a.fnal.gov 005050 root I ENCP  Received final dialog for stkendca19a.fnal.gov-1257487194-5050-0.  elapsed=202.070sec
00:03:15 stkendca19a.fnal.gov 005050 root I ENCP  leaving transfer_file().
00:03:15 stkendca19a.fnal.gov 005050 root I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = False
00:03:15 stkensrv0n.fnal.gov 028796 enstore I FILSRV  Updating  			update file set deleted = 'n', pnfs_path = '/pnfs/fnal.gov/usr/exp-db/daily/d0-online-readonly/d0onlprd/2009/11-November/05/channel01/D0ONLPRD_t702168009_s2963_p21', drive = 'stkenmvr204a:/dev/rmt/tps0d0n:1310050819', pnfs_id = '003000000000000006208580' where bfid = 'CDMS125748739400000';
00:03:15 stkendca19a.fnal.gov 005050 root I ENCP  MSG_TYPE=ENCP_XFER    write_to_hsm /diskb/write-pool-2/data/003000000000000006208580 -> /pnfs/fnal.gov/usr/exp-db/daily/d0-online-readonly/d0onlprd/2009/11-November/05/channel01/D0ONLPRD_t702168009_s2963_p21: 2147467264 bytes copied to VON102 at 36.9 MB/S (68 MB/S network) (106 MB/S drive) (39.9 MB/S disk) mover=LTO4_104.mover drive_id=ULTRIUM-TD4 drive_sn=1310050819 drive_vendor=IBM elapsed=202.41 {'storage_group': 'exp-db', 'encp_crc': 2333110552L, 'disk_rate': 39.898416745546101, 'drive_rate': 106.26983728404795, 'driver': 'FTTDriver', 'transfer_rate': 36.930422603589513, 'mover_interface': 'stkenmvr204a', 'overall_rate': 10.174114328810345, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 36.930422603589513, 'encp_ip': '131.225.13.11', 'unique_id': 'stkendca19a.fnal.gov-1257487194-5050-0'}
00:03:15 stkendca19a.fnal.gov 005050 root I ENCP  File /pnfs/fnal.gov/usr/exp-db/daily/d0-online-readonly/d0onlprd/2009/11-November/05/channel01/D0ONLPRD_t702168009_s2963_p21 copied successfully.
00:03:15 stkendca19a.fnal.gov 005050 root I ENCP  Completed transferring 2147467264 bytes in 1 files in 202.417135 sec.  Overall rate = 10.2 MB/sec.  Transfer rate = 36.9 MB/sec.  Network rate = 68 MB/sec.  Drive rate = 106 MB/sec.  Disk rate = 39.9 MB/sec.  Exit status = 0.
00:03:15 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:03:16 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:16 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:17 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:03:17 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV  skip_fm 366 Thread seek_thread
00:03:17 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV  write_tape starting, bytes_to_write=916991488 Thread tape_thread
00:03:17 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV  Write starting. Tape VON399 absolute location in blocks 2063323 Thread tape_thread
00:03:18 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:18 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:19 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  skip_fm 91 Thread seek_thread
00:03:19 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  write_tape starting, bytes_to_write=1966080512 Thread tape_thread
00:03:19 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  Write starting. Tape VON277 absolute location in blocks 1245092 Thread tape_thread
00:03:19 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:03:20 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:20 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:21 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:03:22 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:22 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:23 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:03:24 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:24 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:25 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:03:26 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:26 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:27 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:03:27 stkendca19a.fnal.gov 004614 root I ENCP  File /diskc/write-pool-3/data/003000000000000006208530 transfered for stkendca19a.fnal.gov-1257487158-4614-0 in 32.4915988445 seconds.
00:03:28 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:28 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:29 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:03:30 stkendca22a.fnal.gov 023884 root I ENCP  Version: v3_8b  CVS $Revision: 1.922 $ <frozen>  OS: Linux 2.6.9-89.0.9.ELsmp i686 Release:  Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)  Library: garbage  Storage Group: test  File Family: dcache  FF Wrapper: cpio_odc  FF Width: 1  Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log  Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0003 --put-cache 0005000000000000005FA5F8 /diskb/write-pool-2/data/0005000000000000005FA5F8
00:03:30 stkendca22a.fnal.gov 023934 root I ENCP  Version: v3_8b  CVS $Revision: 1.922 $ <frozen>  OS: Linux 2.6.9-89.0.9.ELsmp i686 Release:  Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)  Library: garbage  Storage Group: test  File Family: dcache  FF Wrapper: cpio_odc  FF Width: 1  Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log  Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0001 --put-cache 0005000000000000005FA590 /diskb/write-pool-2/data/0005000000000000005FA590
00:03:30 stkendca22a.fnal.gov 023955 root I ENCP  Version: v3_8b  CVS $Revision: 1.922 $ <frozen>  OS: Linux 2.6.9-89.0.9.ELsmp i686 Release:  Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)  Library: garbage  Storage Group: test  File Family: dcache  FF Wrapper: cpio_odc  FF Width: 1  Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log  Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0002 --put-cache 0005000000000000005FA5C8 /diskb/write-pool-2/data/0005000000000000005FA5C8
00:03:30 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:30 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:30 stkendca22a.fnal.gov 023884 root I ENCP  Configuration Server: no such name: 'garbage.library_manager'
00:03:30 stkendca22a.fnal.gov 023884 root E ENCP  INFILE=/diskb/write-pool-2/data/0005000000000000005FA5F8 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0003 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=0.89 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager'
00:03:30 stkendca22a.fnal.gov 023934 root I ENCP  Configuration Server: no such name: 'garbage.library_manager'
00:03:30 stkendca22a.fnal.gov 023934 root E ENCP  INFILE=/diskb/write-pool-2/data/0005000000000000005FA590 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0001 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=0.91 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager'
00:03:30 stkendca22a.fnal.gov 023955 root I ENCP  Configuration Server: no such name: 'garbage.library_manager'
00:03:30 stkendca22a.fnal.gov 023955 root E ENCP  INFILE=/diskb/write-pool-2/data/0005000000000000005FA5C8 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0002 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=0.89 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager'
00:03:31 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  written bytes 1966080512/1966080512, blocks 15001 header 189 trailer 323 Thread tape_thread
00:03:31 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_48.mover', 'VOS188'), ('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:03:32 stkensrv4n.fnal.gov 009077 enstore I SL8500G1MC  dismount VOLUME 0,5,1,15 force => 0,Dismount: Forced dismount of VOS188 from   0, 5, 1,15
00:03:32 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  FINISHED dismount VOS188 0,5,1,15 returned ('ok', 0, 'dismount VOLUME 0,5,1,15 force => 0,Dismount: Forced dismount of VOS188 from   0, 5, 1,15', '', '')
00:03:32 stkenmvr148a.fnal.gov 006010 root I LTO4_48MV  dismounted VOS188 ULTRIUM-TD4 00:03:32 Thread media_thread
00:03:32 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:32 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  filemarks written. Tape VON268 absolute location in blocks 4295963 Thread tape_thread
00:03:32 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  write_tape timing:pull 0.179538488388 write 7.49983072281 crc_check 7.79196619987 freespace 0.168742656708 block_write 15.6400780678 idle 0.28443312645 Thread tape_thread
00:03:32 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  drive stats after write. Tape VON268 position 284 block 0 block_size 0 bloc_loc 4295963 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 1966080512 block_write_tot 15.6400780678 tape_rate 125707845.158 Thread tape_thread
00:03:32 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  remaining bytes info in DB 250011648000 estimated 248045567488 reported from drive 248175616000 position 285 Thread tape_thread
00:03:32 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  new bitfile request {'uid': 0, 'external_label': 'VON268', 'gid': 0, 'location_cookie': '0000_000000000_0000284', 'complete_crc': 1332105296L, 'sanity_cookie': (65536L, 523324373L), 'size': 1966080000L} Thread tape_thread
00:03:32 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:32 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  set remaining: VON268 248175616000 0000_000000000_0000285 Thread tape_thread
00:03:32 last message repeated 1 times
00:03:32 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set last_access = '2009-11-06 00:03:32', eod_cookie = '0000_000000000_0000285', non_del_files = '285', remaining_bytes = 248175616000 where label = 'VON268';
00:03:32 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  transfer complete volume=VON268 location=285 Thread tape_thread
00:03:32 last message repeated 1 times
00:03:32 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_wr_access = '284', modification_time = '2009-11-06 00:03:32' where label = 'VON268';
00:03:32 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  Sending done to client: {'vc': {'comment': '', 'storage_group': 'exp-db', 'blocksize': 131072, 'declared': 1248994058.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 1242753248.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VON268', 'wrapper': 'cpio_odc', 'remaining_bytes': 250011648000L, 'sum_mounts': 150, 'sum_rd_access': 0, 'file_family_width': 2, 'last_access': 1257487374.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000284', 'capacity_bytes': 858993459200L, 'non_del_files': 284, 'sum_wr_err': 0, 'sum_wr_access': 283, 'file_family': 'daily-d0-offline', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'modification_time': 1257487374.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257406785.0}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702155247_s103301_p16', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7843, 'mc_device': '0,7,1,0', 'compression': 0, 'serial_num': '9310055909', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr143a', 'max_time_in_state': 1200, 'device': 'stkenmvr143a:/dev/rmt/tps0d0n', 'name': 'LTO4_43.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_43.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.171', 7843), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.171', 56251), 'data_ip': 'stkenmvr143a', 'logname': 'LTO4_43MV', 'hostip': '131.225.13.171', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702155247_s103301_p16', 'uid': 0, 'uname': 'root', 'mtime': 1257487158, 'rmajor': 0, 'machine': ('Linux', 'stkendca19a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 820619664L, 20L, 1, 1602, 2752, 512L, 1257487155, 1257487155, 1254398418), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 1966080000L, 'gname': 'root', 'gid': 0, 'fullname': '/diskc/write-pool-3/data/003000000000000006208530', 'inode': 807437616L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskc/write-pool-3/data/003000000000000006208530', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/.(access)(003000000000000006208530)', 'fc': {'uid': 0, 'external_label': 'VON268', 'gid': 0, 'location_cookie': '0000_000000000_0000284', 'complete_crc': 1332105296L, 'bfid': 'CDMS125748741200000', 'sanity_cookie': (65536L, 523324373L), 'size': 1966080000L}, 'infilepath': '/diskc/write-pool-3/data/003000000000000006208530', 'at_the_top': 2151, 'local_inode': 575188415L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.11', 51925), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257487157.9395051, 'seek_time': 0.035042047500610352, 'transfer_time': 37.584661960601807, 't0': 1257487157, 'mount_time': 0.041553020477294922, 'lm_dequeued': 1257487375.116276, 'job_queued': 1257487158.9967749, 'drive_call_time': 7.3055734634399414, 'in_queue': 216.11900115013123, 'drive_transfer_time': 17.631802558898926}, 'user_level':(trunc. 4144) Thread tape_thread
00:03:32 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  Done is sent Thread tape_thread
00:03:32 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  maybe_clean Thread tape_thread
00:03:32 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  returned from maybe_clean Thread tape_thread
00:03:32 stkendca19a.fnal.gov 004614 root I ENCP  Received final dialog for stkendca19a.fnal.gov-1257487158-4614-0.  elapsed=254.930sec
00:03:32 stkendca19a.fnal.gov 004614 root I ENCP  leaving transfer_file().
00:03:32 stkendca19a.fnal.gov 004614 root I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = False
00:03:33 stkensrv0n.fnal.gov 028796 enstore I FILSRV  Updating  			update file set deleted = 'n', pnfs_path = '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702155247_s103301_p16', drive = 'stkenmvr143a:/dev/rmt/tps0d0n:9310055909', pnfs_id = '003000000000000006208530' where bfid = 'CDMS125748741200000';
00:03:33 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_104.mover', 'VON102'), ('LTO4_43.mover', 'VON268')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:03:33 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  write_to_hsm next work on vol=VON268 mover=LTO4_43.mover requester:root
00:03:33 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  HAVE_BOUND:sending write_to_hsm /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702153886_s103299_p3 to mover LTO4_43.mover ('131.225.13.171', 7843) DEL_DISM None
00:03:33 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  MSG_TYPE=ADD_TO_LMQ  request added to work at movers queue for storage group : exp-db
00:03:33 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  HAVE_BOUND:sending {'vc': {'storage_group': 'exp-db', 'external_label': 'VON268', 'library': 'CD-LTO4G1', 'file_family': 'daily-d0-offline', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'file_family_width': 2}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702153886_s103299_p3', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': 'LTO4_43.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702153886_s103299_p3', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257487161, 'rmajor': 0, 'machine': ('Linux', 'stkendca18a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (16877, 820619664L, 20L, 1, 1602, 2752, 512L, 1257487155, 1257487155, 1254398418), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 1966080000L, 'fullname': '/diska/write-pool-1/data/0030000000000000062084C0', 'inode': 807437504L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diska/write-pool-1/data/0030000000000000062084C0', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/.(access)(0030000000000000062084C0)', 'fc': {'size': 1966080000L, 'external_label': 'VON268', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diska/write-pool-1/data/0030000000000000062084C0', 'at_the_top': 506, 'local_inode': 536955568L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.6', 57792), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487413.1500881, 'encp_start_time': 1257487160.3017311, 'job_queued': 1257487161.2976379, 't0': 1257487160, 'in_queue': 251.85218405723572}, 'user_level': 'user2', 'unique_id': 'stkendca18a.fnal.gov-1257487161-21984-0'} to mover
00:03:33 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_104.mover', 'VON102'), ('LTO4_43.mover', 'VON268')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:03:33 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  WRITE_TO_HSM Thread MainThread
00:03:33 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  SENDING {'vc': {'storage_group': 'exp-db', 'external_label': 'VON268', 'library': 'CD-LTO4G1', 'file_family': 'daily-d0-offline', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'file_family_width': 2}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702153886_s103299_p3', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7843, 'mc_device': '0,7,1,0', 'compression': 0, 'serial_num': '9310055909', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr143a', 'max_time_in_state': 1200, 'device': 'stkenmvr143a:/dev/rmt/tps0d0n', 'name': 'LTO4_43.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_43.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.171', 7843), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.171', 56254), 'data_ip': 'stkenmvr143a', 'logname': 'LTO4_43MV', 'hostip': '131.225.13.171', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702153886_s103299_p3', 'uid': 0, 'uname': 'root', 'mtime': 1257487161, 'rmajor': 0, 'machine': ('Linux', 'stkendca18a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 820619664L, 20L, 1, 1602, 2752, 512L, 1257487155, 1257487155, 1254398418), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 1966080000L, 'gname': 'root', 'gid': 0, 'fullname': '/diska/write-pool-1/data/0030000000000000062084C0', 'inode': 807437504L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diska/write-pool-1/data/0030000000000000062084C0', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/.(access)(0030000000000000062084C0)', 'fc': {'size': 1966080000L, 'external_label': 'VON268', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diska/write-pool-1/data/0030000000000000062084C0', 'at_the_top': 506, 'local_inode': 536955568L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.6', 57792), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487413.1500881, 'encp_start_time': 1257487160.3017311, 'job_queued': 1257487161.2976379, 't0': 1257487160, 'in_queue': 251.85218405723572}, 'user_level': 'user2', 'unique_id': 'stkendca18a.fnal.gov-1257487161-21984-0'} Thread client_connect_thread
00:03:33 stkendca19a.fnal.gov 004614 root I ENCP  MSG_TYPE=ENCP_XFER    write_to_hsm /diskc/write-pool-3/data/003000000000000006208530 -> /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702155247_s103301_p16: 1966080000 bytes copied to VON268 at 49.9 MB/S (68.1 MB/S network) (106 MB/S drive) (57.8 MB/S disk) mover=LTO4_43.mover drive_id=ULTRIUM-TD4 drive_sn=9310055909 drive_vendor=IBM elapsed=255.24 {'storage_group': 'exp-db', 'encp_crc': 1332105296L, 'disk_rate': 57.76589255233646, 'drive_rate': 106.34193490634745, 'driver': 'FTTDriver', 'transfer_rate': 49.864520048383099, 'mover_interface': 'stkenmvr143a', 'overall_rate': 49.33782474969162, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 49.864520048383099, 'encp_ip': '131.225.13.11', 'unique_id': 'stkendca19a.fnal.gov-1257487158-4614-0'}
00:03:33 stkendca19a.fnal.gov 004614 root I ENCP  File /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702155247_s103301_p16 copied successfully.
00:03:33 stkendca19a.fnal.gov 004614 root I ENCP  Completed transferring 1966080000 bytes in 1 files in 255.23842001 sec.  Overall rate = 49.3 MB/sec.  Transfer rate = 49.9 MB/sec.  Network rate = 68.1 MB/sec.  Drive rate = 106 MB/sec.  Disk rate = 57.8 MB/sec.  Exit status = 0.
00:03:33 stkendca18a.fnal.gov 021984 root I ENCP  Control socket ('131.225.13.6', 57792) is connected to ('131.225.13.171', 56255) for stkendca18a.fnal.gov-1257487161-21984-0.  elapsed=252.900sec
00:03:33 stkendca18a.fnal.gov 021984 root I ENCP  Received callback ticket from mover LTO4_43.mover for transfer stkendca18a.fnal.gov-1257487161-21984-0.  elapsed=252.900sec
00:03:33 stkendca18a.fnal.gov 021984 root I ENCP  Atempting to connect data socket to mover at ('131.225.13.171', 56254).
00:03:33 stkendca18a.fnal.gov 021984 root I ENCP  Connecting data socket to mover (('131.225.13.171', 56254)) with interface 131.225.13.6.  elapsed=252.910sec
00:03:33 stkendca18a.fnal.gov 021984 root I ENCP  Data socket ('131.225.13.6', 57802) is connected to ('131.225.13.171', 56254).  elapsed=252.910sec
00:03:33 stkendca18a.fnal.gov 021984 root I ENCP  Opening local file.
00:03:33 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  tape position: target 285 current 285 0 Thread seek_thread
00:03:33 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  write_tape starting, bytes_to_write=1966080512 Thread tape_thread
00:03:33 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  Write starting. Tape VON268 absolute location in blocks 4295963 Thread tape_thread
00:03:33 stkendca18a.fnal.gov 021984 root I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = False
00:03:33 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:03:33 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:03:34 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_43.mover', 'VON268')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:03:34 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:34 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:35 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:03:35 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:03:36 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:36 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:37 stkendca20a.fnal.gov 026781 root I ENCP  File /diskb/write-pool-2/data/0030000000000000061FF670 transfered for stkendca20a.fnal.gov-1257487310-26781-0 in 19.4297339916 seconds.
00:03:37 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV  skip_fm 2918 Thread seek_thread
00:03:37 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV  write_tape starting, bytes_to_write=21428736 Thread tape_thread
00:03:37 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV  Write starting. Tape VOS175 absolute location in blocks 5198770 Thread tape_thread
00:03:37 cmsstor68.fnal.gov 010129 root I ENCP  File /storage/data1/write-pool-1/data/0010000000000000094F6470 transfered for cmsstor68.fnal.gov-1257487303-10129-0 in 0.194604158401 seconds.
00:03:37 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:03:37 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV  written bytes 21428736/21428736, blocks 164 header 146 trailer 430 Thread tape_thread
00:03:37 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV  written bytes 916991488/916991488, blocks 6997 header 157 trailer 355 Thread tape_thread
00:03:38 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:38 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:39 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:03:40 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:40 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV  filemarks written. Tape VON399 absolute location in blocks 2070321 Thread tape_thread
00:03:40 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV  write_tape timing:pull 0.0760741233826 write 3.11182785034 crc_check 4.24315643311 freespace 0.0796234607697 block_write 7.5106818676 idle 0.13153886795 Thread tape_thread
00:03:40 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV  drive stats after write. Tape VON399 position 366 block 0 block_size 0 bloc_loc 2070321 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 916991488 block_write_tot 7.5106818676 tape_rate 122091642.832 Thread tape_thread
00:03:40 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV  remaining bytes info in DB 520690688000 estimated 519773696512 reported from drive 519832576000 position 367 Thread tape_thread
00:03:40 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV  new bitfile request {'uid': 0, 'external_label': 'VON399', 'gid': 0, 'location_cookie': '0000_000000000_0000366', 'complete_crc': 2852043954L, 'sanity_cookie': (65536L, 960068744L), 'size': 916990976L} Thread tape_thread
00:03:40 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:40 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV  set remaining: VON399 519832576000 0000_000000000_0000367 Thread tape_thread
00:03:40 last message repeated 1 times
00:03:40 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set last_access = '2009-11-06 00:03:40', eod_cookie = '0000_000000000_0000367', non_del_files = '373', remaining_bytes = 519832576000 where label = 'VON399';
00:03:40 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV  transfer complete volume=VON399 location=367 Thread tape_thread
00:03:40 last message repeated 1 times
00:03:40 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_wr_access = '367', modification_time = '2009-11-06 00:03:40' where label = 'VON399';
00:03:40 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV  Sending done to client: {'vc': {'comment': 'Used as an FTT test tape, returned to service 2009-09-14 -tj', 'storage_group': 'exp-db', 'blocksize': 131072, 'declared': 1240352675.0, 'library': 'CD-LTO4G1', 'si_time': [1252940184.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VON399', 'wrapper': 'cpio_odc', 'remaining_bytes': 520690688000L, 'sum_mounts': 272, 'sum_rd_access': 0, 'file_family_width': 1, 'last_access': 1257473097.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000366', 'capacity_bytes': 858993459200L, 'non_del_files': 372, 'sum_wr_err': 1, 'sum_wr_access': 366, 'file_family': 'archive-d0-offline', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.archive-d0-offline.cpio_odc', 'modification_time': 1257473097.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1245637123.0}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/archive/d0-offline/d0ofprd1/Nov05/arc_819772_819838.tar', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 916990976L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7868, 'mc_device': '0,5,1,11', 'compression': 0, 'serial_num': '1310154353', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr208a', 'max_time_in_state': 1200, 'device': 'stkenmvr208a:/dev/rmt/tps0d0n', 'name': 'LTO4_108.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_108.stat', 'mover_address': ('131.225.13.204', 7868), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.204', 46995), 'data_ip': 'stkenmvr208a', 'logname': 'LTO4_108MV', 'hostip': '131.225.13.204', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/archive/d0-offline/d0ofprd1/Nov05/arc_819772_819838.tar', 'uid': 0, 'uname': 'root', 'mtime': 1257487310, 'rmajor': 0, 'machine': ('Linux', 'stkendca20a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 820612056L, 20L, 1, 1602, 2752, 512L, 1257481891, 1257481891, 1254398415), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 916990976L, 'gname': 'root', 'gid': 0, 'fullname': '/diskb/write-pool-2/data/0030000000000000061FF670', 'inode': 807401072L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskb/write-pool-2/data/0030000000000000061FF670', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/archive/d0-offline/d0ofprd1/Nov05/.(access)(0030000000000000061FF670)', 'fc': {'uid': 0, 'external_label': 'VON399', 'gid': 0, 'location_cookie': '0000_000000000_0000366', 'complete_crc': 2852043954L, 'bfid': 'CDMS125748742000000', 'sanity_cookie': (65536L, 960068744L), 'size': 916990976L}, 'infilepath': '/diskb/write-pool-2/data/0030000000000000061FF670', 'at_the_top': 1, 'local_inode': 271274660L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.12', 44956), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257487309.1597381, 'seek_time': 87.293232917785645, 'transfer_time': 23.075768947601318, 't0': 1257487309, 'lm_dequeued': 1257487310.355309, 'job_queued': 1257487310.0729289, 'drive_call_time': 3.0256941318511963, 'in_queue': 0.28232598304748535, 'drive_transfer_time': 10.482874631881714}, 'user_level': 'user2', 'unique_id': 'stkendca20a.fnal.gov-1257487310-26781-0'} Thread tape_thread
00:03:40 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV  Done is sent Thread tape_thread
00:03:40 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV  maybe_clean Thread tape_thread
00:03:40 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV  returned from maybe_clean Thread tape_thread
00:03:40 stkendca20a.fnal.gov 026781 root I ENCP  Received final dialog for stkendca20a.fnal.gov-1257487310-26781-0.  elapsed=111.670sec
00:03:40 stkendca20a.fnal.gov 026781 root I ENCP  leaving transfer_file().
00:03:40 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_108.mover', 'VON399'), ('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:03:40 stkendca20a.fnal.gov 026781 root I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = False
00:03:40 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:03:40 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV  filemarks written. Tape VOS175 absolute location in blocks 5198935 Thread tape_thread
00:03:41 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV  write_tape timing:pull 0.00121760368347 write 0.12221121788 crc_check 0.0825095176697 freespace 0.00116181373596 block_write 0.207100152969 idle 0.00212383270264 Thread tape_thread
00:03:41 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV  drive stats after write. Tape VOS175 position 2918 block 0 block_size 0 bloc_loc 5198935 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 21428736 block_write_tot 0.207100152969 tape_rate 103470401.604 Thread tape_thread
00:03:41 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV  remaining bytes info in DB 139442176000 estimated 139420747264 reported from drive 139422720000 position 2919 Thread tape_thread
00:03:41 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV  new bitfile request {'uid': 0, 'external_label': 'VOS175', 'gid': 0, 'location_cookie': '0000_000000000_0002918', 'complete_crc': 2570631670L, 'sanity_cookie': (65536L, 3956295089L), 'size': 21428160L} Thread tape_thread
00:03:41 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV  set remaining: VOS175 139422720000 0000_000000000_0002919 Thread tape_thread
00:03:41 last message repeated 1 times
00:03:41 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set last_access = '2009-11-06 00:03:41', eod_cookie = '0000_000000000_0002919', non_del_files = '2938', remaining_bytes = 139422720000 where label = 'VOS175';
00:03:41 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV  transfer complete volume=VOS175 location=2919 Thread tape_thread
00:03:41 last message repeated 1 times
00:03:41 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_wr_access = '2918', modification_time = '2009-11-06 00:03:41' where label = 'VOS175';
00:03:41 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV  Sending done to client: {'vc': {'comment': '', 'storage_group': 'cms', 'blocksize': 131072, 'declared': 1244477608.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOS175', 'wrapper': 'cpio_odc', 'remaining_bytes': 139442176000L, 'sum_mounts': 687, 'sum_rd_access': 4, 'file_family_width': 7, 'last_access': 1257487089.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0002918', 'capacity_bytes': 858993459200L, 'non_del_files': 2937, 'sum_wr_err': 0, 'sum_wr_access': 2917, 'file_family': 'cms2', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cms2.cpio_odc', 'modification_time': 1257487089.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1250490198.0}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX/2/olga/JPT_AKT05_170_230/JetAnalysis_6.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 21428160L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7841, 'mc_device': '0,7,1,3', 'compression': 0, 'serial_num': '1310154471', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr141a', 'max_time_in_state': 1200, 'device': 'stkenmvr141a:/dev/rmt/tps1d0n', 'name': 'LTO4_41.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_41.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.169', 7841), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.169', 55114), 'data_ip': 'stkenmvr141a', 'logname': 'LTO4_41MV', 'hostip': '131.225.13.169', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX/2/olga/JPT_AKT05_170_230/JetAnalysis_6.root', 'uid': 0, 'uname': 'root', 'mtime': 1257487303, 'rmajor': 0, 'machine': ('Linux', 'cmsstor68.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16895, 273637888, 23L, 1, 9811, 5063, 512, 1257487250, 1257487250, 1257485909), 'mode_octal': '0100040', 'mode': 33188, 'size_bytes': 21428160L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data1/write-pool-1/data/0010000000000000094F6470', 'inode': 273638512L, 'minor': 23}, 'version': 'v3_7d  CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data1/write-pool-1/data/0010000000000000094F6470', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX/2/olga/JPT_AKT05_170_230/.(access)(0010000000000000094F6470)', 'fc': {'uid': 0, 'external_label': 'VOS175', 'gid': 0, 'location_cookie': '0000_000000000_0002918', 'complete_crc': 2570631670L, 'bfid': 'CDMS125748742100000', 'sanity_cookie': (65536L, 3956295089L), 'size': 21428160L}, 'infilepath': '/storage/data1/write-pool-1/data/0010000000000000094F6470', 'at_the_top': 2, 'local_inode': 2147484328L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.189.68', 33740), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257487302.974324, 'seek_time': 112.37899112701416, 'transfer_time': 3.8204660415649414, 't0': 1257487302, 'lm_dequeued': 1257487304.8881299, 'job_queued': 1257487303.845396, 'drive_call_time': 0.13099789619445801, 'in_queue': 1.042672872543335, 'drive_transfer_time': 3.430168628692627}, 'user_level': 'admin', 'unique_id': 'cmsstor68.fnal.gov-1257487303-10129-0'} Thread tape_thread
00:03:41 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV  Done is sent Thread tape_thread
00:03:41 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV  maybe_clean Thread tape_thread
00:03:41 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV  returned from maybe_clean Thread tape_thread
00:03:41 cmsstor68.fnal.gov 010129 root I ENCP  Received final dialog for cmsstor68.fnal.gov-1257487303-10129-0.  elapsed=118.220sec
00:03:41 cmsstor68.fnal.gov 010129 root I ENCP  leaving transfer_file().
00:03:41 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:03:41 cmsstor68.fnal.gov 010129 root I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = False
00:03:41 stkensrv0n.fnal.gov 028796 enstore I FILSRV  Updating  			update file set deleted = 'n', pnfs_path = '/pnfs/fnal.gov/usr/exp-db/archive/d0-offline/d0ofprd1/Nov05/arc_819772_819838.tar', drive = 'stkenmvr208a:/dev/rmt/tps0d0n:1310154353', pnfs_id = '0030000000000000061FF670' where bfid = 'CDMS125748742000000';
00:03:41 stkensrv0n.fnal.gov 028796 enstore I FILSRV  Updating  			update file set deleted = 'n', pnfs_path = '/pnfs/fnal.gov/usr/cms/WAX/2/olga/JPT_AKT05_170_230/JetAnalysis_6.root', drive = 'stkenmvr141a:/dev/rmt/tps1d0n:1310154471', pnfs_id = '0010000000000000094F6470' where bfid = 'CDMS125748742100000';
00:03:41 cmsstor68.fnal.gov 010129 root I ENCP  MSG_TYPE=ENCP_XFER    write_to_hsm /storage/data1/write-pool-1/data/0010000000000000094F6470 -> /pnfs/fnal.gov/usr/cms/WAX/2/olga/JPT_AKT05_170_230/JetAnalysis_6.root: 21428160 bytes copied to VOS175 at 5.33 MB/S (86.7 MB/S network) (5.96 MB/S drive) (109 MB/S disk) mover=LTO4_41.mover drive_id=ULTRIUM-TD4 drive_sn=1310154471 drive_vendor=IBM elapsed=118.47 {'storage_group': 'cms', 'encp_crc': 2570631670L, 'disk_rate': 108.86038587030379, 'drive_rate': 5.9575747002363908, 'driver': 'FTTDriver', 'transfer_rate': 5.3336877814707755, 'mover_interface': 'stkenmvr141a', 'overall_rate': 0.17541617048811642, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 5.3336877814707755, 'encp_ip': '131.225.189.68', 'unique_id': 'cmsstor68.fnal.gov-1257487303-10129-0'}
00:03:41 cmsstor68.fnal.gov 010129 root I ENCP  File /pnfs/fnal.gov/usr/cms/WAX/2/olga/JPT_AKT05_170_230/JetAnalysis_6.root copied successfully.
00:03:41 cmsstor68.fnal.gov 010129 root I ENCP  Completed transferring 21428160 bytes in 1 files in 118.47004509 sec.  Overall rate = 0.175 MB/sec.  Transfer rate = 5.33 MB/sec.  Network rate = 86.7 MB/sec.  Drive rate = 5.96 MB/sec.  Disk rate = 109 MB/sec.  Exit status = 0.
00:03:41 stkendca20a.fnal.gov 026781 root I ENCP  MSG_TYPE=ENCP_XFER    write_to_hsm /diskb/write-pool-2/data/0030000000000000061FF670 -> /pnfs/fnal.gov/usr/exp-db/archive/d0-offline/d0ofprd1/Nov05/arc_819772_819838.tar: 916990976 bytes copied to VON399 at 37.9 MB/S (50.4 MB/S network) (83.4 MB/S drive) (45.2 MB/S disk) mover=LTO4_108.mover drive_id=ULTRIUM-TD4 drive_sn=1310154353 drive_vendor=IBM elapsed=112.48 {'storage_group': 'exp-db', 'encp_crc': 2852043954L, 'disk_rate': 45.175578304277536, 'drive_rate': 83.422798888373435, 'driver': 'FTTDriver', 'transfer_rate': 37.856958654303568, 'mover_interface': 'stkenmvr208a', 'overall_rate': 7.8615755308376967, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 37.856958654303568, 'encp_ip': '131.225.13.12', 'unique_id': 'stkendca20a.fnal.gov-1257487310-26781-0'}
00:03:41 stkendca20a.fnal.gov 026781 root I ENCP  File /pnfs/fnal.gov/usr/exp-db/archive/d0-offline/d0ofprd1/Nov05/arc_819772_819838.tar copied successfully.
00:03:41 stkendca20a.fnal.gov 026781 root I ENCP  Completed transferring 916990976 bytes in 1 files in 112.494158983 sec.  Overall rate = 7.86 MB/sec.  Transfer rate = 37.9 MB/sec.  Network rate = 50.4 MB/sec.  Drive rate = 83.4 MB/sec.  Disk rate = 45.2 MB/sec.  Exit status = 0.
00:03:41 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:03:42 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:42 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:42 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:03:43 last message repeated 1 times
00:03:43 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:03:44 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:44 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:44 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:03:45 last message repeated 1 times
00:03:45 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:03:46 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:46 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:46 last message repeated 1 times
00:03:46 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:03:47 last message repeated 1 times
00:03:47 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:03:48 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  skip_fm 110 Thread seek_thread
00:03:48 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  write_tape starting, bytes_to_write=2147467776 Thread tape_thread
00:03:48 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  Write starting. Tape VON245 absolute location in blocks 1702995 Thread tape_thread
00:03:48 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:48 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:48 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:03:49 last message repeated 1 times
00:03:49 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:03:50 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:50 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:50 stkendca20a.fnal.gov 025712 root I ENCP  File /diskb/write-pool-2/data/003000000000000006208498 transfered for stkendca20a.fnal.gov-1257487130-25712-0 in 31.1803910732 seconds.
00:03:50 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:03:51 last message repeated 1 times
00:03:51 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:03:52 stkensrv4n.fnal.gov 005357 enstore E LTO4G1LM  MOVER ERROR RQ {'status': ('ok', None), 'returned_work': None, 'library': None, 'mover_type': '', 'current_location': '0000_000000000_0000000', 'ip_map': '', 'operation': None, 'read_only': 0, 'current_time': 1257487432.386107, 'error_source': None, 'external_label': None, 'state': 'OFFLINE', 'r_a': (('131.225.13.194', 54880), 18010L, '131.225.13.194-54880-1256586929.509013-4866'), 'library_list': [('CD-LTO4G1.library_manager', ('131.225.13.4', 8541))], 'current_priority': (None, None), 'transfer_deficiency': 1, 'time_in_state': 900502.0260488987, 'address': ('131.225.13.194', 7860), 'volume_family': None, 'work': 'mover_error', 'mover': 'LTO4_99.mover', 'volume_status': (['none', 'none'], ['none', 'none']), 'unique_id': None}
00:03:52 stkendca18a.fnal.gov 021984 root I ENCP  File /diska/write-pool-1/data/0030000000000000062084C0 transfered for stkendca18a.fnal.gov-1257487161-21984-0 in 19.0502109528 seconds.
00:03:52 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:52 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:52 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:03:53 last message repeated 1 times
00:03:53 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:03:54 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:54 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:54 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:03:55 last message repeated 1 times
00:03:55 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:03:56 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:56 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:56 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:03:57 last message repeated 1 times
00:03:57 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  written bytes 1966080512/1966080512, blocks 15001 header 188 trailer 324 Thread tape_thread
00:03:57 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:03:58 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:03:58 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:03:59 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:03:59 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  filemarks written. Tape VON268 absolute location in blocks 4310965 Thread tape_thread
00:03:59 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  write_tape timing:pull 0.181966543198 write 11.1500082016 crc_check 7.82408642769 freespace 0.198454141617 block_write 19.3545153141 idle 0.309754371643 Thread tape_thread
00:03:59 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  drive stats after write. Tape VON268 position 285 block 0 block_size 0 bloc_loc 4310965 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 1966080512 block_write_tot 19.3545153141 tape_rate 101582523.773 Thread tape_thread
00:03:59 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  remaining bytes info in DB 248175616000 estimated 246209535488 reported from drive 246338560000 position 286 Thread tape_thread
00:03:59 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  new bitfile request {'uid': 0, 'external_label': 'VON268', 'gid': 0, 'location_cookie': '0000_000000000_0000285', 'complete_crc': 548183610L, 'sanity_cookie': (65536L, 1580364549L), 'size': 1966080000L} Thread tape_thread
00:03:59 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  set remaining: VON268 246338560000 0000_000000000_0000286 Thread tape_thread
00:03:59 last message repeated 1 times
00:03:59 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set last_access = '2009-11-06 00:03:59', eod_cookie = '0000_000000000_0000286', non_del_files = '286', remaining_bytes = 246338560000 where label = 'VON268';
00:03:59 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  transfer complete volume=VON268 location=286 Thread tape_thread
00:03:59 last message repeated 1 times
00:03:59 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_wr_access = '285', modification_time = '2009-11-06 00:03:59' where label = 'VON268';
00:03:59 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  Sending done to client: {'vc': {'comment': '', 'storage_group': 'exp-db', 'blocksize': 131072, 'declared': 1248994058.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 1242753248.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VON268', 'wrapper': 'cpio_odc', 'remaining_bytes': 248175616000L, 'sum_mounts': 150, 'sum_rd_access': 0, 'file_family_width': 2, 'last_access': 1257487412.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000285', 'capacity_bytes': 858993459200L, 'non_del_files': 285, 'sum_wr_err': 0, 'sum_wr_access': 284, 'file_family': 'daily-d0-offline', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'modification_time': 1257487412.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257406785.0}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702153886_s103299_p3', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7843, 'mc_device': '0,7,1,0', 'compression': 0, 'serial_num': '9310055909', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr143a', 'max_time_in_state': 1200, 'device': 'stkenmvr143a:/dev/rmt/tps0d0n', 'name': 'LTO4_43.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_43.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.171', 7843), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.171', 56254), 'data_ip': 'stkenmvr143a', 'logname': 'LTO4_43MV', 'hostip': '131.225.13.171', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702153886_s103299_p3', 'uid': 0, 'uname': 'root', 'mtime': 1257487161, 'rmajor': 0, 'machine': ('Linux', 'stkendca18a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 820619664L, 20L, 1, 1602, 2752, 512L, 1257487155, 1257487155, 1254398418), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 1966080000L, 'gname': 'root', 'gid': 0, 'fullname': '/diska/write-pool-1/data/0030000000000000062084C0', 'inode': 807437504L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diska/write-pool-1/data/0030000000000000062084C0', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/.(access)(0030000000000000062084C0)', 'fc': {'uid': 0, 'external_label': 'VON268', 'gid': 0, 'location_cookie': '0000_000000000_0000285', 'complete_crc': 548183610L, 'bfid': 'CDMS125748743900000', 'sanity_cookie': (65536L, 1580364549L), 'size': 1966080000L}, 'infilepath': '/diska/write-pool-1/data/0030000000000000062084C0', 'at_the_top': 506, 'local_inode': 536955568L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.6', 57792), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257487160.3017311, 'seek_time': 0.034365177154541016, 'transfer_time': 25.918223857879639, 't0': 1257487160, 'mount_time': 0.043428897857666016, 'lm_dequeued': 1257487413.1500881, 'job_queued': 1257487161.2976379, 'drive_call_time': 10.939089059829712, 'in_queue': 251.85218405723572, 'drive_transfer_time': 21.361647129058838}, 'user_level': 'u(trunc. 4142) Thread tape_thread
00:03:59 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  Done is sent Thread tape_thread
00:03:59 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  maybe_clean Thread tape_thread
00:03:59 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:03:59 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  returned from maybe_clean Thread tape_thread
00:03:59 stkendca18a.fnal.gov 021984 root I ENCP  Received final dialog for stkendca18a.fnal.gov-1257487161-21984-0.  elapsed=278.940sec
00:03:59 stkendca18a.fnal.gov 021984 root I ENCP  leaving transfer_file().
00:03:59 stkendca18a.fnal.gov 021984 root I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = False
00:03:59 stkensrv0n.fnal.gov 028796 enstore I FILSRV  Updating  			update file set deleted = 'n', pnfs_path = '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702153886_s103299_p3', drive = 'stkenmvr143a:/dev/rmt/tps0d0n:9310055909', pnfs_id = '0030000000000000062084C0' where bfid = 'CDMS125748743900000';
00:03:59 stkendca18a.fnal.gov 021984 root I ENCP  MSG_TYPE=ENCP_XFER    write_to_hsm /diska/write-pool-1/data/0030000000000000062084C0 -> /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702153886_s103299_p3: 1966080000 bytes copied to VON268 at 72.4 MB/S (81.8 MB/S network) (87.8 MB/S drive) (98.6 MB/S disk) mover=LTO4_43.mover drive_id=ULTRIUM-TD4 drive_sn=9310055909 drive_vendor=IBM elapsed=279.26 {'storage_group': 'exp-db', 'encp_crc': 548183610L, 'disk_rate': 98.631297430983565, 'drive_rate': 87.774130368878986, 'driver': 'FTTDriver', 'transfer_rate': 72.438815456178332, 'mover_interface': 'stkenmvr143a', 'overall_rate': 71.138707335823696, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 72.438815456178332, 'encp_ip': '131.225.13.6', 'unique_id': 'stkendca18a.fnal.gov-1257487161-21984-0'}
00:03:59 stkendca18a.fnal.gov 021984 root I ENCP  File /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702153886_s103299_p3 copied successfully.
00:03:59 stkendca18a.fnal.gov 021984 root I ENCP  Completed transferring 1966080000 bytes in 1 files in 279.270310879 sec.  Overall rate = 71.1 MB/sec.  Transfer rate = 72.4 MB/sec.  Network rate = 81.8 MB/sec.  Drive rate = 87.8 MB/sec.  Disk rate = 98.6 MB/sec.  Exit status = 0.
00:03:59 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  written bytes 1966080512/1966080512, blocks 15001 header 188 trailer 324 Thread tape_thread
00:03:59 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:00 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_104.mover', 'VON102'), ('LTO4_43.mover', 'VON268')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:00 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  write_to_hsm next work on vol=VON268 mover=LTO4_43.mover requester:root
00:04:00 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  HAVE_BOUND:sending write_to_hsm /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702154603_s103300_p15 to mover LTO4_43.mover ('131.225.13.171', 7843) DEL_DISM None
00:04:00 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  MSG_TYPE=ADD_TO_LMQ  request added to work at movers queue for storage group : exp-db
00:04:00 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  HAVE_BOUND:sending {'vc': {'storage_group': 'exp-db', 'external_label': 'VON268', 'library': 'CD-LTO4G1', 'file_family': 'daily-d0-offline', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'file_family_width': 2}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702154603_s103300_p15', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': 'LTO4_43.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702154603_s103300_p15', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257487218, 'rmajor': 0, 'machine': ('Linux', 'stkendca20a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (16877, 820619448L, 20L, 1, 1602, 2752, 512L, 1257487175, 1257487175, 1254398418), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 1966080000L, 'fullname': '/diska/write-pool-1/data/003000000000000006208558', 'inode': 807437656L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diska/write-pool-1/data/003000000000000006208558', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/.(access)(003000000000000006208558)', 'fc': {'size': 1966080000L, 'external_label': 'VON268', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diska/write-pool-1/data/003000000000000006208558', 'at_the_top': 335, 'local_inode': 268435591L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.12', 44951), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487440.187356, 'encp_start_time': 1257487217.0995319, 'job_queued': 1257487218.1049941, 't0': 1257487217, 'in_queue': 222.08206987380981}, 'user_level': 'user2', 'unique_id': 'stkendca20a.fnal.gov-1257487218-26341-0'} to mover
00:04:00 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_104.mover', 'VON102'), ('LTO4_43.mover', 'VON268')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:00 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  WRITE_TO_HSM Thread MainThread
00:04:00 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  SENDING {'vc': {'storage_group': 'exp-db', 'external_label': 'VON268', 'library': 'CD-LTO4G1', 'file_family': 'daily-d0-offline', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'file_family_width': 2}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702154603_s103300_p15', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7843, 'mc_device': '0,7,1,0', 'compression': 0, 'serial_num': '9310055909', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr143a', 'max_time_in_state': 1200, 'device': 'stkenmvr143a:/dev/rmt/tps0d0n', 'name': 'LTO4_43.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_43.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.171', 7843), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.171', 56256), 'data_ip': 'stkenmvr143a', 'logname': 'LTO4_43MV', 'hostip': '131.225.13.171', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702154603_s103300_p15', 'uid': 0, 'uname': 'root', 'mtime': 1257487218, 'rmajor': 0, 'machine': ('Linux', 'stkendca20a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 820619448L, 20L, 1, 1602, 2752, 512L, 1257487175, 1257487175, 1254398418), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 1966080000L, 'gname': 'root', 'gid': 0, 'fullname': '/diska/write-pool-1/data/003000000000000006208558', 'inode': 807437656L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diska/write-pool-1/data/003000000000000006208558', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/.(access)(003000000000000006208558)', 'fc': {'size': 1966080000L, 'external_label': 'VON268', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diska/write-pool-1/data/003000000000000006208558', 'at_the_top': 335, 'local_inode': 268435591L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.12', 44951), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487440.187356, 'encp_start_time': 1257487217.0995319, 'job_queued': 1257487218.1049941, 't0': 1257487217, 'in_queue': 222.08206987380981}, 'user_level': 'user2', 'unique_id': 'stkendca20a.fnal.gov-1257487218-26341-0'} Thread client_connect_thread
00:04:00 stkendca20a.fnal.gov 026341 root I ENCP  Control socket ('131.225.13.12', 44951) is connected to ('131.225.13.171', 56257) for stkendca20a.fnal.gov-1257487218-26341-0.  elapsed=223.140sec
00:04:00 stkendca20a.fnal.gov 026341 root I ENCP  Received callback ticket from mover LTO4_43.mover for transfer stkendca20a.fnal.gov-1257487218-26341-0.  elapsed=223.150sec
00:04:00 stkendca20a.fnal.gov 026341 root I ENCP  Atempting to connect data socket to mover at ('131.225.13.171', 56256).
00:04:00 stkendca20a.fnal.gov 026341 root I ENCP  Connecting data socket to mover (('131.225.13.171', 56256)) with interface 131.225.13.12.  elapsed=223.150sec
00:04:00 stkendca20a.fnal.gov 026341 root I ENCP  Data socket ('131.225.13.12', 44964) is connected to ('131.225.13.171', 56256).  elapsed=223.150sec
00:04:00 stkendca20a.fnal.gov 026341 root I ENCP  Opening local file.
00:04:00 stkendca20a.fnal.gov 026341 root I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = False
00:04:00 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  tape position: target 286 current 286 0 Thread seek_thread
00:04:00 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  write_tape starting, bytes_to_write=1966080512 Thread tape_thread
00:04:00 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  Write starting. Tape VON268 absolute location in blocks 4310965 Thread tape_thread
00:04:00 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:04:00 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:01 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_43.mover', 'VON268'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:01 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  filemarks written. Tape VON277 absolute location in blocks 1260094 Thread tape_thread
00:04:01 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  write_tape timing:pull 0.210483312607 write 9.39158058167 crc_check 7.80828380585 freespace 0.172444105148 block_write 17.5827918053 idle 0.294243812561 Thread tape_thread
00:04:01 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  drive stats after write. Tape VON277 position 91 block 0 block_size 0 bloc_loc 1260094 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 1966080512 block_write_tot 17.5827918053 tape_rate 111818449.185 Thread tape_thread
00:04:01 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  remaining bytes info in DB 627623936000 estimated 625657855488 reported from drive 624101376000 position 92 Thread tape_thread
00:04:01 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  new bitfile request {'uid': 0, 'external_label': 'VON277', 'gid': 0, 'location_cookie': '0000_000000000_0000091', 'complete_crc': 3092837484L, 'sanity_cookie': (65536L, 265879707L), 'size': 1966080000L} Thread tape_thread
00:04:01 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_43.mover', 'VON268')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:01 last message repeated 1 times
00:04:01 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  set remaining: VON277 624101376000 0000_000000000_0000092 Thread tape_thread
00:04:01 last message repeated 1 times
00:04:01 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set last_access = '2009-11-06 00:04:01', eod_cookie = '0000_000000000_0000092', non_del_files = '91', remaining_bytes = 624101376000 where label = 'VON277';
00:04:01 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  transfer complete volume=VON277 location=92 Thread tape_thread
00:04:01 last message repeated 1 times
00:04:01 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_wr_access = '91', modification_time = '2009-11-06 00:04:01' where label = 'VON277';
00:04:01 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  Sending done to client: {'vc': {'comment': '', 'storage_group': 'exp-db', 'blocksize': 131072, 'declared': 1248994091.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 1243130430.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VON277', 'wrapper': 'cpio_odc', 'remaining_bytes': 627623936000L, 'sum_mounts': 112, 'sum_rd_access': 0, 'file_family_width': 2, 'last_access': 1257487028.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000091', 'capacity_bytes': 858993459200L, 'non_del_files': 90, 'sum_wr_err': 0, 'sum_wr_access': 90, 'file_family': 'daily-d0-offline', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'modification_time': 1257487028.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257481244.0}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702152110_s103297_p4', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7847, 'mc_device': '0,6,1,0', 'compression': 0, 'serial_num': '1310154418', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr147a', 'max_time_in_state': 1200, 'device': 'stkenmvr147a:/dev/rmt/tps0d0n', 'name': 'LTO4_47.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_47.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.175', 7847), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.175', 59772), 'data_ip': 'stkenmvr147a', 'logname': 'LTO4_47MV', 'hostip': '131.225.13.175', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702152110_s103297_p4', 'uid': 0, 'uname': 'root', 'mtime': 1257487130, 'rmajor': 0, 'machine': ('Linux', 'stkendca20a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 820619448L, 20L, 1, 1602, 2752, 512L, 1257487118, 1257487118, 1254398418), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 1966080000L, 'gname': 'root', 'gid': 0, 'fullname': '/diskb/write-pool-2/data/003000000000000006208498', 'inode': 807437464L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskb/write-pool-2/data/003000000000000006208498', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/.(access)(003000000000000006208498)', 'fc': {'uid': 0, 'external_label': 'VON277', 'gid': 0, 'location_cookie': '0000_000000000_0000091', 'complete_crc': 3092837484L, 'bfid': 'CDMS125748744100000', 'sanity_cookie': (65536L, 265879707L), 'size': 1966080000L}, 'infilepath': '/diskb/write-pool-2/data/003000000000000006208498', 'at_the_top': 275, 'local_inode': 268812423L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.12', 44946), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257487129.2778189, 'seek_time': 187.9321129322052, 'transfer_time': 42.005645036697388, 't0': 1257487129, 'lm_dequeued': 1257487211.4352219, 'job_queued': 1257487130.281184, 'drive_call_time': 9.2070984840393066, 'in_queue': 81.153573036193848, 'drive_transfer_time': 19.56718111038208}, 'user_level': 'user2', 'unique_id': 'stkendca20a.fnal.gov-(trunc. 4100) Thread tape_thread
00:04:01 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  Done is sent Thread tape_thread
00:04:01 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  maybe_clean Thread tape_thread
00:04:01 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  returned from maybe_clean Thread tape_thread
00:04:01 stkendca20a.fnal.gov 025712 root I ENCP  Received final dialog for stkendca20a.fnal.gov-1257487130-25712-0.  elapsed=312.190sec
00:04:01 stkendca20a.fnal.gov 025712 root I ENCP  leaving transfer_file().
00:04:01 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:01 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  write_to_hsm next work on vol=VON277 mover=LTO4_47.mover requester:root
00:04:01 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  HAVE_BOUND:sending write_to_hsm /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702158466_s103306_p17 to mover LTO4_47.mover ('131.225.13.175', 7847) DEL_DISM None
00:04:01 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  MSG_TYPE=ADD_TO_LMQ  request added to work at movers queue for storage group : exp-db
00:04:01 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  HAVE_BOUND:sending {'vc': {'storage_group': 'exp-db', 'external_label': 'VON277', 'library': 'CD-LTO4G1', 'file_family': 'daily-d0-offline', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'file_family_width': 2}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702158466_s103306_p17', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': 'LTO4_47.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702158466_s103306_p17', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257487280, 'rmajor': 0, 'machine': ('Linux', 'stkendca18a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (16877, 820619448L, 20L, 1, 1602, 2752, 512L, 1257487280, 1257487280, 1254398418), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 1966080000L, 'fullname': '/diskc/write-pool-3/data/0030000000000000062085D0', 'inode': 807437776L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskc/write-pool-3/data/0030000000000000062085D0', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/.(access)(0030000000000000062085D0)', 'fc': {'size': 1966080000L, 'external_label': 'VON277', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diskc/write-pool-3/data/0030000000000000062085D0', 'at_the_top': 11, 'local_inode': 584856936L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.6', 57799), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487441.5558889, 'encp_start_time': 1257487279.447506, 'job_queued': 1257487280.5250151, 't0': 1257487279, 'in_queue': 161.03060483932495}, 'user_level': 'user2', 'unique_id': 'stkendca18a.fnal.gov-1257487280-22627-0'} to mover
00:04:01 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:01 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  WRITE_TO_HSM Thread MainThread
00:04:01 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  SENDING {'vc': {'storage_group': 'exp-db', 'external_label': 'VON277', 'library': 'CD-LTO4G1', 'file_family': 'daily-d0-offline', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'file_family_width': 2}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702158466_s103306_p17', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7847, 'mc_device': '0,6,1,0', 'compression': 0, 'serial_num': '1310154418', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr147a', 'max_time_in_state': 1200, 'device': 'stkenmvr147a:/dev/rmt/tps0d0n', 'name': 'LTO4_47.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_47.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.175', 7847), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.175', 59781), 'data_ip': 'stkenmvr147a', 'logname': 'LTO4_47MV', 'hostip': '131.225.13.175', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702158466_s103306_p17', 'uid': 0, 'uname': 'root', 'mtime': 1257487280, 'rmajor': 0, 'machine': ('Linux', 'stkendca18a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 820619448L, 20L, 1, 1602, 2752, 512L, 1257487280, 1257487280, 1254398418), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 1966080000L, 'gname': 'root', 'gid': 0, 'fullname': '/diskc/write-pool-3/data/0030000000000000062085D0', 'inode': 807437776L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskc/write-pool-3/data/0030000000000000062085D0', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/.(access)(0030000000000000062085D0)', 'fc': {'size': 1966080000L, 'external_label': 'VON277', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diskc/write-pool-3/data/0030000000000000062085D0', 'at_the_top': 11, 'local_inode': 584856936L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.6', 57799), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487441.5558889, 'encp_start_time': 1257487279.447506, 'job_queued': 1257487280.5250151, 't0': 1257487279, 'in_queue': 161.03060483932495}, 'user_level': 'user2', 'unique_id': 'stkendca18a.fnal.gov-1257487280-22627-0'} Thread client_connect_thread
00:04:01 stkendca20a.fnal.gov 025712 root I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = False
00:04:01 stkendca18a.fnal.gov 022627 root I ENCP  Control socket ('131.225.13.6', 57799) is connected to ('131.225.13.175', 59782) for stkendca18a.fnal.gov-1257487280-22627-0.  elapsed=162.170sec
00:04:01 stkendca18a.fnal.gov 022627 root I ENCP  Received callback ticket from mover LTO4_47.mover for transfer stkendca18a.fnal.gov-1257487280-22627-0.  elapsed=162.170sec
00:04:01 stkendca18a.fnal.gov 022627 root I ENCP  Atempting to connect data socket to mover at ('131.225.13.175', 59781).
00:04:01 stkendca18a.fnal.gov 022627 root I ENCP  Connecting data socket to mover (('131.225.13.175', 59781)) with interface 131.225.13.6.  elapsed=162.170sec
00:04:01 stkendca18a.fnal.gov 022627 root I ENCP  Data socket ('131.225.13.6', 57804) is connected to ('131.225.13.175', 59781).  elapsed=162.170sec
00:04:01 stkendca18a.fnal.gov 022627 root I ENCP  Opening local file.
00:04:01 last message repeated 1 times
00:04:01 stkendca18a.fnal.gov 022627 root I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = False
00:04:01 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  tape position: target 92 current 92 0 Thread seek_thread
00:04:01 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:01 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  write_tape starting, bytes_to_write=1966080512 Thread tape_thread
00:04:01 stkensrv0n.fnal.gov 028796 enstore I FILSRV  Updating  			update file set deleted = 'n', pnfs_path = '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702152110_s103297_p4', drive = 'stkenmvr147a:/dev/rmt/tps0d0n:1310154418', pnfs_id = '003000000000000006208498' where bfid = 'CDMS125748744100000';
00:04:01 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  Write starting. Tape VON277 absolute location in blocks 1260094 Thread tape_thread
00:04:01 stkendca20a.fnal.gov 025712 root I ENCP  MSG_TYPE=ENCP_XFER    write_to_hsm /diskb/write-pool-2/data/003000000000000006208498 -> /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702152110_s103297_p4: 1966080000 bytes copied to VON277 at 44.6 MB/S (63.6 MB/S network) (95.8 MB/S drive) (60.2 MB/S disk) mover=LTO4_47.mover drive_id=ULTRIUM-TD4 drive_sn=1310154418 drive_vendor=IBM elapsed=312.54 {'storage_group': 'exp-db', 'encp_crc': 3092837484L, 'disk_rate': 60.159589433800896, 'drive_rate': 95.823715711669394, 'driver': 'FTTDriver', 'transfer_rate': 44.623844215556062, 'mover_interface': 'stkenmvr147a', 'overall_rate': 8.1407966953453421, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 44.623844215556062, 'encp_ip': '131.225.13.12', 'unique_id': 'stkendca20a.fnal.gov-1257487130-25712-0'}
00:04:01 stkendca20a.fnal.gov 025712 root I ENCP  File /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702152110_s103297_p4 copied successfully.
00:04:01 stkendca20a.fnal.gov 025712 root I ENCP  Completed transferring 1966080000 bytes in 1 files in 312.537683964 sec.  Overall rate = 8.14 MB/sec.  Transfer rate = 44.6 MB/sec.  Network rate = 63.6 MB/sec.  Drive rate = 95.8 MB/sec.  Disk rate = 60.2 MB/sec.  Exit status = 0.
00:04:02 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:04:02 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:02 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:03 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_109.mover', 'VON245'), ('LTO4_43.mover', 'VON268'), ('LTO4_104.mover', 'VON102'), ('LTO4_47.mover', 'VON277'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:03 last message repeated 1 times
00:04:03 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_109.mover', 'VON245'), ('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_108.mover', 'VON399'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:04 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_108.mover', 'VON399'), ('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:04 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:04:04 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:05 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:05 last message repeated 1 times
00:04:05 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:06 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:04:06 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:07 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:07 last message repeated 1 times
00:04:07 stkendm7a.fnal.gov 009044 root I MIGRATION  File /data/data1/Migration_Spool/VO5178:0000_000000000_0000116 transfered for stkendm7a.fnal.gov-1257487106-9044-80 in 149.47487998 seconds. Thread WRITE_1
00:04:07 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:08 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:04:08 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:09 last message repeated 1 times
00:04:09 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:10 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:04:10 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:10 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  written bytes 11981448192/11981448192, blocks 91412 header 0 trailer 420 Thread tape_thread
00:04:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:11 last message repeated 1 times
00:04:11 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV  Updating stats Thread media_thread
00:04:11 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV  writing a tape mark before dismount Thread media_thread
00:04:11 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:12 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:04:12 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:13 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:13 stkenmvr141a.fnal.gov 013963 root I LTO4_41MV  Ejecting tape Thread media_thread
00:04:13 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:14 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:04:14 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:15 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  filemarks written. Tape VOV147 absolute location in blocks 3550919 Thread tape_thread
00:04:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  write_tape timing:pull 1.30362939835 write 40.0726127625 crc_check 54.7612540722 freespace 1.12485051155 block_write 97.2623467445 idle 1.83021402359 Thread tape_thread
00:04:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  drive stats after write. Tape VOV147 position 7420 block 0 block_size 0 bloc_loc 3550919 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 11981448192 block_write_tot 97.2623467445 tape_rate 123186912.439 Thread tape_thread
00:04:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  remaining bytes info in DB 336587776000 estimated 324606327808 reported from drive 323701760000 position 7423 Thread tape_thread
00:04:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  new bitfile request {'uid': 0, 'external_label': 'VOV147', 'gid': 0, 'location_cookie': '0000_000000000_0007421', 'complete_crc': 3734097480L, 'sanity_cookie': (65536L, 3817899768L), 'size': 11981447772L} Thread tape_thread
00:04:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  set remaining: VOV147 323701760000 0000_000000000_0007423 Thread tape_thread
00:04:15 last message repeated 1 times
00:04:15 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set last_access = '2009-11-06 00:04:15', eod_cookie = '0000_000000000_0007423', non_del_files = '2476', remaining_bytes = 323701760000 where label = 'VOV147';
00:04:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  transfer complete volume=VOV147 location=7423 Thread tape_thread
00:04:15 last message repeated 1 times
00:04:15 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_wr_access = '2474', modification_time = '2009-11-06 00:04:15' where label = 'VOV147';
00:04:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  Sending done to client: {'vc': {'comment': '', 'storage_group': 'ktev', 'blocksize': 131072, 'declared': 1254165878.0, 'library': 'CD-LTO4F1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOV147', 'wrapper': 'cern', 'remaining_bytes': 336587776000L, 'sum_mounts': 5, 'sum_rd_access': 0, 'file_family_width': 1, 'last_access': 1257487297.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0007420', 'capacity_bytes': 858993459200L, 'non_del_files': 2475, 'sum_wr_err': 0, 'sum_wr_access': 2473, 'file_family': 'ktev_migration-MIGRATION', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'modification_time': 1257487297.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257458755.0}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC092/f0', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 11981447772L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4F1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7873, 'mc_device': '1,1,1,4', 'compression': 0, 'serial_num': '1310156224', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr213a', 'max_time_in_state': 1200, 'device': 'stkenmvr213a:/dev/rmt/tps0d0n', 'name': 'LTO4_113.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_113.stat', 'mover_address': ('131.225.13.108', 7873), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.108', 43838), 'data_ip': 'stkenmvr213a', 'logname': 'LTO4_113MV', 'hostip': '131.225.13.108', 'do_cleaning': 'No', 'media_changer': 'SL8500F1.media_changer'}, 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC092/f0', 'uid': 0, 'uname': 'root', 'mtime': 1257487106, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cern', 'pstat': (16877, 675737896L, 20L, 1, 0, 0, 512L, 1257487106, 1257487106, 1257487106), 'mode_octal': '0100016', 'mode': 33204, 'size_bytes': 11981447772L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000116', 'inode': 675737968L, 'minor': 20}, 'version': 'v3_8  CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000116', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC092/.(access)(00280000000000000846F170)', 'fc': {'uid': 0, 'external_label': 'VOV147', 'gid': 0, 'location_cookie': '0000_000000000_0007421', 'complete_crc': 3734097480L, 'bfid': 'CDMS125748745500000', 'sanity_cookie': (65536L, 3817899768L), 'size': 11981447772L}, 'infilepath': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000116', 'at_the_top': 314, 'local_inode': 39503182L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 44365), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257487106.4977331, 'seek_time': 0.033709049224853516, 'transfer_time': 157.73358607292175, 't0': 1257487106, 'mount_time': 0.045883893966674805, 'lm_dequeued': 1257487297.744405, 'job_queued': 1257487106.742687, 'drive_call_time': 38.775449275970459, 'in_queue': 191.00144791603088, 'drive_transfer_time': 106.16113257408142}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257487106-9044-80'} Thread tape_thread
00:04:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  Done is sent Thread tape_thread
00:04:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  maybe_clean Thread tape_thread
00:04:15 stkendm7a.fnal.gov 009044 root I MIGRATION  Received final dialog for stkendm7a.fnal.gov-1257487106-9044-80.  elapsed=5892.970sec Thread WRITE_1
00:04:15 stkendm7a.fnal.gov 009044 root I MIGRATION  leaving transfer_file(). Thread WRITE_1
00:04:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  returned from maybe_clean Thread tape_thread
00:04:15 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:15 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV  Updating stats Thread media_thread
00:04:15 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV  writing a tape mark before dismount Thread media_thread
00:04:15 stkensrv0n.fnal.gov 028796 enstore I FILSRV  Updating  			update file set deleted = 'n', pnfs_path = '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC092/f0', drive = 'stkenmvr213a:/dev/rmt/tps0d0n:1310156224', pnfs_id = '00280000000000000846F170' where bfid = 'CDMS125748745500000';
00:04:15 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:15 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  write_to_hsm next work on vol=VOV147 mover=LTO4_113.mover requester:root
00:04:15 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  HAVE_BOUND:sending write_to_hsm /pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f2 to mover LTO4_113.mover ('131.225.13.108', 7873) DEL_DISM 2
00:04:15 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  MSG_TYPE=ADD_TO_LMQ  request added to work at movers queue for storage group : ktev
00:04:15 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  HAVE_BOUND:sending {'vc': {'storage_group': 'ktev', 'external_label': 'VOV147', 'library': 'CD-LTO4F1', 'file_family': 'ktev_migration-MIGRATION', 'wrapper': 'cern', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'file_family_width': 1}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f2', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 1173211400L, 'mover': 'LTO4_113.mover', 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f2', 'uid': 0, 'uname': 'root', 'type': 'cern', 'gid': 0, 'mtime': 1257487256, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (16877, 683683784L, 20L, 1, 0, 0, 512L, 1257487050, 1257487050, 1255127721), 'mode_octal': '0100016', 'mode': 33204, 'gname': 'root', 'size_bytes': 1173211400L, 'fullname': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000122', 'inode': 675738016L, 'minor': 20}, 'version': 'v3_8  CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000122', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC099/.(access)(00280000000000000846F1A0)', 'fc': {'size': 1173211400L, 'external_label': 'VOV147', 'pnfsid': '00280000000000000846F1A0', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000122', 'at_the_top': 906, 'local_inode': 39503181L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 44372), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487455.876488, 'encp_start_time': 1257487256.0404711, 'job_queued': 1257487256.4334469, 't0': 1257487256, 'in_queue': 199.44274711608887}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257487256-9044-81'} to mover
00:04:15 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  WRITE_TO_HSM Thread MainThread
00:04:15 stkendm7a.fnal.gov 009044 root I MIGRATION  Listening for control socket at: ('131.225.13.19', 44372) Thread WRITE
00:04:15 stkendm7a.fnal.gov 009044 root I MIGRATION  Waiting for mover to connect control socket at address: ('131.225.13.19', 44372)  elapsed=5893.260sec Thread WRITE
00:04:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  SENDING {'vc': {'storage_group': 'ktev', 'external_label': 'VOV147', 'library': 'CD-LTO4F1', 'file_family': 'ktev_migration-MIGRATION', 'wrapper': 'cern', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'file_family_width': 1}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f2', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 1173211400L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4F1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7873, 'mc_device': '1,1,1,4', 'compression': 0, 'serial_num': '1310156224', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr213a', 'max_time_in_state': 1200, 'device': 'stkenmvr213a:/dev/rmt/tps0d0n', 'name': 'LTO4_113.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_113.stat', 'mover_address': ('131.225.13.108', 7873), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.108', 43842), 'data_ip': 'stkenmvr213a', 'logname': 'LTO4_113MV', 'hostip': '131.225.13.108', 'do_cleaning': 'No', 'media_changer': 'SL8500F1.media_changer'}, 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f2', 'uid': 0, 'uname': 'root', 'mtime': 1257487256, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cern', 'pstat': (16877, 683683784L, 20L, 1, 0, 0, 512L, 1257487050, 1257487050, 1255127721), 'mode_octal': '0100016', 'mode': 33204, 'size_bytes': 1173211400L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000122', 'inode': 675738016L, 'minor': 20}, 'version': 'v3_8  CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000122', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC099/.(access)(00280000000000000846F1A0)', 'fc': {'size': 1173211400L, 'external_label': 'VOV147', 'pnfsid': '00280000000000000846F1A0', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000122', 'at_the_top': 906, 'local_inode': 39503181L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 44372), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487455.876488, 'encp_start_time': 1257487256.0404711, 'job_queued': 1257487256.4334469, 't0': 1257487256, 'in_queue': 199.44274711608887}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257487256-9044-81'} Thread client_connect_thread
00:04:15 stkendm7a.fnal.gov 009044 root I MIGRATION  Control socket ('131.225.13.19', 44372) is connected to ('131.225.13.108', 43843) for stkendm7a.fnal.gov-1257487256-9044-81.  elapsed=5893.270sec Thread WRITE
00:04:15 stkendm7a.fnal.gov 009044 root I MIGRATION  Received callback ticket from mover LTO4_113.mover for transfer stkendm7a.fnal.gov-1257487256-9044-81.  elapsed=5893.270sec Thread WRITE
00:04:15 stkendm7a.fnal.gov 009044 root I MIGRATION  Attempting to connect data socket to mover at ('131.225.13.108', 43842). Thread WRITE
00:04:15 stkendm7a.fnal.gov 009044 root I MIGRATION  Connecting data socket to mover (('131.225.13.108', 43842)) with interface 131.225.13.19.  elapsed=5893.270sec Thread WRITE
00:04:15 stkendm7a.fnal.gov 009044 root I MIGRATION  Data socket ('131.225.13.19', 44379) is connected to ('131.225.13.108', 43842).  elapsed=5893.270sec Thread WRITE
00:04:15 stkendm7a.fnal.gov 009044 root I MIGRATION  Opening local file. Thread WRITE
00:04:15 stkendm7a.fnal.gov 009044 root I MIGRATION  pf_status = ('ok', None)  skip_layer_cleanup = False Thread WRITE
00:04:16 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  tape position: target 7423 current 7423 0 Thread seek_thread
00:04:16 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  write_tape starting, bytes_to_write=1173211648 Thread tape_thread
00:04:16 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  Write starting. Tape VOV147 absolute location in blocks 3550919 Thread tape_thread
00:04:16 stkendm7a.fnal.gov 009044 root I MIGRATION  MSG_TYPE=ENCP_XFER    write_to_hsm /data/data1/Migration_Spool/VO5178:0000_000000000_0000116 -> /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC092/f0: 11981447772 bytes copied to VOV147 at 72.4 MB/S (94.3 MB/S network) (108 MB/S drive) (88.1 MB/S disk) mover=LTO4_113.mover drive_id=ULTRIUM-TD4 drive_sn=1310156224 drive_vendor=IBM elapsed=349.55 {'storage_group': 'ktev', 'encp_crc': 3734097480L, 'disk_rate': 88.059213907859643, 'drive_rate': 107.63260278469346, 'driver': 'FTTDriver', 'transfer_rate': 72.443002156421386, 'mover_interface': 'stkenmvr213a', 'overall_rate': 72.197799815260737, 'media_changer': 'SL8500F1.media_changer', 'network_rate': 72.443002156421386, 'encp_ip': '131.225.13.19', 'unique_id': 'stkendm7a.fnal.gov-1257487106-9044-80'} Thread WRITE_1
00:04:16 stkendm7a.fnal.gov 009044 root I MIGRATION  File /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC092/f0 copied successfully. Thread WRITE_1
00:04:16 stkendm7a.fnal.gov 009044 root I MIGRATION  Completed transferring 11981447772 bytes in 1 files in 349.55633688 sec.  Overall rate = 72.2 MB/sec.  Transfer rate = 72.4 MB/sec.  Network rate = 94.3 MB/sec.  Drive rate = 108 MB/sec.  Disk rate = 88.1 MB/sec.  Exit status = 0. Thread WRITE_1
00:04:16 stkensrv0n.fnal.gov 028796 enstore I FILSRV  start modifying {'pnfsid': '00280000000000000846F170', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC092/f0', 'deleted': 'no', 'external_label': 'VOV147', 'drive': 'stkenmvr213a:/dev/rmt/tps0d0n:1310156224', 'update': '2009-11-06 00:04:15.732398', 'gid': 0, 'location_cookie': '0000_000000000_0007421', 'complete_crc': 3734097480L, 'uid': 0, 'bfid': 'CDMS125748745500000', 'sanity_cookie': (65536L, 3817899768L), 'size': 11981447772L}
00:04:16 stkensrv0n.fnal.gov 028796 enstore I FILSRV  Updating  			update file set pnfs_id = '001F00000000000000566828' where bfid = 'CDMS125748745500000';
00:04:16 stkensrv0n.fnal.gov 028796 enstore I FILSRV  modified to {'pnfsid': '001F00000000000000566828', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC092/f0', 'deleted': 'no', 'external_label': 'VOV147', 'drive': 'stkenmvr213a:/dev/rmt/tps0d0n:1310156224', 'update': '2009-11-06 00:04:15.732398', 'gid': 0, 'location_cookie': '0000_000000000_0007421', 'complete_crc': 3734097480L, 'uid': 0, 'bfid': 'CDMS125748745500000', 'sanity_cookie': (65536L, 3817899768L), 'size': 11981447772L}
00:04:16 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:04:16 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:17 last message repeated 1 times
00:04:17 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:18 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:04:18 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:19 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:19 stkenmvr204a.fnal.gov 006261 root I LTO4_104MV  Ejecting tape Thread media_thread
00:04:20 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:04:20 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:21 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:21 stkendca18a.fnal.gov 022627 root I ENCP  File /diskc/write-pool-3/data/0030000000000000062085D0 transfered for stkendca18a.fnal.gov-1257487280-22627-0 in 19.4377560616 seconds.
00:04:21 stkendca18a.fnal.gov 023081 root I ENCP  Version: v3_8b  CVS $Revision: 1.922 $ <frozen>  OS: Linux 2.6.9-89.0.9.ELsmp i686 Release:  Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)  Library: CD-LTO4G1  Storage Group: exp-db  File Family: daily-d0-offline  FF Wrapper: cpio_odc  FF Width: 2  Current working directory: stkendca18a.fnal.gov:/home/enstore/dcache-code/dcache-fermi-config  Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702157805_s103305_p22 --put-cache 0030000000000000062085A8 /diska/write-pool-1/data/0030000000000000062085A8
00:04:21 stkendca18a.fnal.gov 023081 root I ENCP  Creating zero length output files.
00:04:21 stkendca18a.fnal.gov 023081 root I ENCP  Sending /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702157805_s103305_p22 write request to LM: unique_id: stkendca18a.fnal.gov-1257487461-23081-0 inputfile: /diska/write-pool-1/data/0030000000000000062085A8 outputfile: /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/.(access)(0030000000000000062085A8)
00:04:21 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  MSG_TYPE=ADD_TO_LMQ  request added to pending queue for storage group : exp-db
00:04:21 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  write Q'd /diska/write-pool-1/data/0030000000000000062085A8 (stkendca18a.fnal.gov-1257487461-23081-0) -> /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel02/D0OFPRD1_t702157805_s103305_p22 : library=CD-LTO4G1 family=daily-d0-offline requester:root volume_family:exp-db.daily-d0-offline.cpio_odc
00:04:21 stkendca18a.fnal.gov 023081 root I ENCP  Listening for control socket at: ('131.225.13.6', 57806)
00:04:21 stkendca18a.fnal.gov 023081 root I ENCP  Waiting for mover to connect control socket at address: ('131.225.13.6', 57806)  elapsed=1.180sec
00:04:22 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:04:22 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:23 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:24 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:04:24 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:04:24 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:25 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:04:26 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  written bytes 1966080512/1966080512, blocks 15001 header 189 trailer 323 Thread tape_thread
00:04:26 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:04:26 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:27 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:04:27 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV  Tape is ejected Thread media_thread
00:04:27 stkenmvr140a.fnal.gov 030526 root I LTO4_40MV  dismounting VOS642 Thread media_thread
00:04:27 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC  PROCESSING dismount VOS642 0,7,1,15
00:04:27 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  filemarks written. Tape VON277 absolute location in blocks 1275096 Thread tape_thread
00:04:27 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  write_tape timing:pull 0.182055473328 write 10.2548508644 crc_check 7.81030082703 freespace 0.195642948151 block_write 18.4428501129 idle 0.319978952408 Thread tape_thread
00:04:27 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  drive stats after write. Tape VON277 position 92 block 0 block_size 0 bloc_loc 1275096 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 1966080512 block_write_tot 18.4428501129 tape_rate 106603941.363 Thread tape_thread
00:04:27 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  remaining bytes info in DB 624101376000 estimated 622135295488 reported from drive 622263296000 position 93 Thread tape_thread
00:04:27 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  new bitfile request {'uid': 0, 'external_label': 'VON277', 'gid': 0, 'location_cookie': '0000_000000000_0000092', 'complete_crc': 2834493324L, 'sanity_cookie': (65536L, 3787862729L), 'size': 1966080000L} Thread tape_thread
00:04:28 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  set remaining: VON277 622263296000 0000_000000000_0000093 Thread tape_thread
00:04:28 last message repeated 1 times
00:04:28 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set last_access = '2009-11-06 00:04:28', eod_cookie = '0000_000000000_0000093', non_del_files = '92', remaining_bytes = 622263296000 where label = 'VON277';
00:04:28 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  transfer complete volume=VON277 location=93 Thread tape_thread
00:04:28 last message repeated 1 times
00:04:28 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_wr_access = '92', modification_time = '2009-11-06 00:04:28' where label = 'VON277';
00:04:28 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  Sending done to client: {'vc': {'comment': '', 'storage_group': 'exp-db', 'blocksize': 131072, 'declared': 1248994091.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 1243130430.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VON277', 'wrapper': 'cpio_odc', 'remaining_bytes': 624101376000L, 'sum_mounts': 113, 'sum_rd_access': 0, 'file_family_width': 2, 'last_access': 1257487441.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000092', 'capacity_bytes': 858993459200L, 'non_del_files': 91, 'sum_wr_err': 0, 'sum_wr_access': 91, 'file_family': 'daily-d0-offline', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'modification_time': 1257487441.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257481244.0}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702158466_s103306_p17', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7847, 'mc_device': '0,6,1,0', 'compression': 0, 'serial_num': '1310154418', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr147a', 'max_time_in_state': 1200, 'device': 'stkenmvr147a:/dev/rmt/tps0d0n', 'name': 'LTO4_47.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_47.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.175', 7847), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.175', 59781), 'data_ip': 'stkenmvr147a', 'logname': 'LTO4_47MV', 'hostip': '131.225.13.175', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702158466_s103306_p17', 'uid': 0, 'uname': 'root', 'mtime': 1257487280, 'rmajor': 0, 'machine': ('Linux', 'stkendca18a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 820619448L, 20L, 1, 1602, 2752, 512L, 1257487280, 1257487280, 1254398418), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 1966080000L, 'gname': 'root', 'gid': 0, 'fullname': '/diskc/write-pool-3/data/0030000000000000062085D0', 'inode': 807437776L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskc/write-pool-3/data/0030000000000000062085D0', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/.(access)(0030000000000000062085D0)', 'fc': {'uid': 0, 'external_label': 'VON277', 'gid': 0, 'location_cookie': '0000_000000000_0000092', 'complete_crc': 2834493324L, 'bfid': 'CDMS125748746800000', 'sanity_cookie': (65536L, 3787862729L), 'size': 1966080000L}, 'infilepath': '/diskc/write-pool-3/data/0030000000000000062085D0', 'at_the_top': 11, 'local_inode': 584856936L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.6', 57799), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257487279.447506, 'seek_time': 0.034570932388305664, 'transfer_time': 26.451677083969116, 't0': 1257487279, 'mount_time': 0.047320127487182617, 'lm_dequeued': 1257487441.5558889, 'job_queued': 1257487280.5250151, 'drive_call_time': 10.045979261398315, 'in_queue': 161.03060483932495, 'drive_transfer_time': 20.460679769515991}, 'user_level': 'us(trunc. 4141) Thread tape_thread
00:04:28 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  Done is sent Thread tape_thread
00:04:28 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  maybe_clean Thread tape_thread
00:04:28 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  returned from maybe_clean Thread tape_thread
00:04:28 stkendca18a.fnal.gov 022627 root I ENCP  Received final dialog for stkendca18a.fnal.gov-1257487280-22627-0.  elapsed=188.740sec
00:04:28 stkendca18a.fnal.gov 022627 root I ENCP  leaving transfer_file().
00:04:28 stkendca18a.fnal.gov 022627 root I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = False
00:04:28 stkensrv0n.fnal.gov 028796 enstore I FILSRV  Updating  			update file set deleted = 'n', pnfs_path = '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702158466_s103306_p17', drive = 'stkenmvr147a:/dev/rmt/tps0d0n:1310154418', pnfs_id = '0030000000000000062085D0' where bfid = 'CDMS125748746800000';
00:04:28 stkendca18a.fnal.gov 022627 root I ENCP  MSG_TYPE=ENCP_XFER    write_to_hsm /diskc/write-pool-3/data/0030000000000000062085D0 -> /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702158466_s103306_p17: 1966080000 bytes copied to VON277 at 70.9 MB/S (79.1 MB/S network) (91.6 MB/S drive) (96.5 MB/S disk) mover=LTO4_47.mover drive_id=ULTRIUM-TD4 drive_sn=1310154418 drive_vendor=IBM elapsed=189.02 {'storage_group': 'exp-db', 'encp_crc': 2834493324L, 'disk_rate': 96.488497953329031, 'drive_rate': 91.639184089745143, 'driver': 'FTTDriver', 'transfer_rate': 70.850699937512658, 'mover_interface': 'stkenmvr147a', 'overall_rate': 69.8190886481418, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 70.850699937512658, 'encp_ip': '131.225.13.6', 'unique_id': 'stkendca18a.fnal.gov-1257487280-22627-0'}
00:04:28 stkendca18a.fnal.gov 022627 root I ENCP  File /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702158466_s103306_p17 copied successfully.
00:04:28 stkendca18a.fnal.gov 022627 root I ENCP  Completed transferring 1966080000 bytes in 1 files in 189.019863129 sec.  Overall rate = 69.8 MB/sec.  Transfer rate = 70.9 MB/sec.  Network rate = 79.1 MB/sec.  Drive rate = 91.6 MB/sec.  Disk rate = 96.5 MB/sec.  Exit status = 0.
00:04:28 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:04:28 stkenmvr23a.fnal.gov 005810 root I DBT23MV  Updating stats Thread media_thread
00:04:28 stkenmvr23a.fnal.gov 005810 root I DBT23MV  Ejecting tape Thread media_thread
00:04:28 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:04:28 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  write_to_hsm next work on vol=VON277 mover=LTO4_47.mover requester:root
00:04:28 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  HAVE_BOUND:sending write_to_hsm /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702152110_s103297_p27 to mover LTO4_47.mover ('131.225.13.175', 7847) DEL_DISM None
00:04:28 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  MSG_TYPE=ADD_TO_LMQ  request added to work at movers queue for storage group : exp-db
00:04:28 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  HAVE_BOUND:sending {'vc': {'storage_group': 'exp-db', 'external_label': 'VON277', 'library': 'CD-LTO4G1', 'file_family': 'daily-d0-offline', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'file_family_width': 2}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702152110_s103297_p27', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': 'LTO4_47.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702152110_s103297_p27', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257487328, 'rmajor': 0, 'machine': ('Linux', 'stkendca19a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (16877, 820619448L, 20L, 1, 1602, 2752, 512L, 1257487280, 1257487280, 1254398418), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 1966080000L, 'fullname': '/diska/write-pool-1/data/003000000000000006208660', 'inode': 807437920L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diska/write-pool-1/data/003000000000000006208660', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/.(access)(003000000000000006208660)', 'fc': {'size': 1966080000L, 'external_label': 'VON277', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diska/write-pool-1/data/003000000000000006208660', 'at_the_top': 386, 'local_inode': 628299043L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.11', 51940), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487468.563911, 'encp_start_time': 1257487327.348331, 'job_queued': 1257487328.3911309, 't0': 1257487327, 'in_queue': 140.17250299453735}, 'user_level': 'user2', 'unique_id': 'stkendca19a.fnal.gov-1257487328-6120-0'} to mover
00:04:28 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:04:28 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  WRITE_TO_HSM Thread MainThread
00:04:28 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  SENDING {'vc': {'storage_group': 'exp-db', 'external_label': 'VON277', 'library': 'CD-LTO4G1', 'file_family': 'daily-d0-offline', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'file_family_width': 2}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702152110_s103297_p27', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7847, 'mc_device': '0,6,1,0', 'compression': 0, 'serial_num': '1310154418', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr147a', 'max_time_in_state': 1200, 'device': 'stkenmvr147a:/dev/rmt/tps0d0n', 'name': 'LTO4_47.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_47.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.175', 7847), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.175', 59784), 'data_ip': 'stkenmvr147a', 'logname': 'LTO4_47MV', 'hostip': '131.225.13.175', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702152110_s103297_p27', 'uid': 0, 'uname': 'root', 'mtime': 1257487328, 'rmajor': 0, 'machine': ('Linux', 'stkendca19a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 820619448L, 20L, 1, 1602, 2752, 512L, 1257487280, 1257487280, 1254398418), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 1966080000L, 'gname': 'root', 'gid': 0, 'fullname': '/diska/write-pool-1/data/003000000000000006208660', 'inode': 807437920L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diska/write-pool-1/data/003000000000000006208660', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/.(access)(003000000000000006208660)', 'fc': {'size': 1966080000L, 'external_label': 'VON277', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diska/write-pool-1/data/003000000000000006208660', 'at_the_top': 386, 'local_inode': 628299043L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.11', 51940), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487468.563911, 'encp_start_time': 1257487327.348331, 'job_queued': 1257487328.3911309, 't0': 1257487327, 'in_queue': 140.17250299453735}, 'user_level': 'user2', 'unique_id': 'stkendca19a.fnal.gov-1257487328-6120-0'} Thread client_connect_thread
00:04:28 stkendca19a.fnal.gov 006120 root I ENCP  Control socket ('131.225.13.11', 51940) is connected to ('131.225.13.175', 59785) for stkendca19a.fnal.gov-1257487328-6120-0.  elapsed=141.280sec
00:04:28 stkendca19a.fnal.gov 006120 root I ENCP  Received callback ticket from mover LTO4_47.mover for transfer stkendca19a.fnal.gov-1257487328-6120-0.  elapsed=141.280sec
00:04:28 stkendca19a.fnal.gov 006120 root I ENCP  Atempting to connect data socket to mover at ('131.225.13.175', 59784).
00:04:28 stkendca19a.fnal.gov 006120 root I ENCP  Connecting data socket to mover (('131.225.13.175', 59784)) with interface 131.225.13.11.  elapsed=141.280sec
00:04:28 stkendca19a.fnal.gov 006120 root I ENCP  Data socket ('131.225.13.11', 51946) is connected to ('131.225.13.175', 59784).  elapsed=141.290sec
00:04:28 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:28 stkendca19a.fnal.gov 006120 root I ENCP  Opening local file.
00:04:28 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  tape position: target 93 current 93 0 Thread seek_thread
00:04:28 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  write_tape starting, bytes_to_write=1966080512 Thread tape_thread
00:04:28 stkenmvr147a.fnal.gov 009702 root I LTO4_47MV  Write starting. Tape VON277 absolute location in blocks 1275096 Thread tape_thread
00:04:28 stkendca19a.fnal.gov 006120 root I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = False
00:04:29 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_47.mover', 'VON277'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_41.mover', 'VOS175'), ('LTO4_40.mover', 'VOS642')]} >
00:04:29 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_47.mover', 'VON277'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:29 stkensrv4n.fnal.gov 009490 enstore I SL8500G1MC  query drive 0,7,1,15 => 0,0,7,1,15 online          in use      VOS642     IBM-LTO4
00:04:29 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_47.mover', 'VON277')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:30 stkendca22a.fnal.gov 025157 root I ENCP  Version: v3_8b  CVS $Revision: 1.922 $ <frozen>  OS: Linux 2.6.9-89.0.9.ELsmp i686 Release:  Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)  Library: garbage  Storage Group: test  File Family: dcache  FF Wrapper: cpio_odc  FF Width: 1  Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log  Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0003 --put-cache 0005000000000000005FA5F8 /diskb/write-pool-2/data/0005000000000000005FA5F8
00:04:30 stkendca22a.fnal.gov 025236 root I ENCP  Version: v3_8b  CVS $Revision: 1.922 $ <frozen>  OS: Linux 2.6.9-89.0.9.ELsmp i686 Release:  Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)  Library: garbage  Storage Group: test  File Family: dcache  FF Wrapper: cpio_odc  FF Width: 1  Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log  Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0001 --put-cache 0005000000000000005FA590 /diskb/write-pool-2/data/0005000000000000005FA590
00:04:30 stkendca22a.fnal.gov 025252 root I ENCP  Version: v3_8b  CVS $Revision: 1.922 $ <frozen>  OS: Linux 2.6.9-89.0.9.ELsmp i686 Release:  Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)  Library: garbage  Storage Group: test  File Family: dcache  FF Wrapper: cpio_odc  FF Width: 1  Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log  Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0002 --put-cache 0005000000000000005FA5C8 /diskb/write-pool-2/data/0005000000000000005FA5C8
00:04:30 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:30 stkendca19a.fnal.gov 005688 root I ENCP  File /diskc/write-pool-3/data/003000000000000006208600 transfered for stkendca19a.fnal.gov-1257487278-5688-0 in 42.3047959805 seconds.
00:04:30 stkendca22a.fnal.gov 025157 root I ENCP  Configuration Server: no such name: 'garbage.library_manager'
00:04:30 stkendca22a.fnal.gov 025157 root E ENCP  INFILE=/diskb/write-pool-2/data/0005000000000000005FA5F8 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0003 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=0.90 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager'
00:04:30 stkendca22a.fnal.gov 025236 root I ENCP  Configuration Server: no such name: 'garbage.library_manager'
00:04:30 stkendca22a.fnal.gov 025236 root E ENCP  INFILE=/diskb/write-pool-2/data/0005000000000000005FA590 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0001 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=0.89 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager'
00:04:30 stkendca22a.fnal.gov 025252 root I ENCP  Configuration Server: no such name: 'garbage.library_manager'
00:04:30 stkendca22a.fnal.gov 025252 root E ENCP  INFILE=/diskb/write-pool-2/data/0005000000000000005FA5C8 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0002 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=0.93 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager'
00:04:31 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_47.mover', 'VON277'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:32 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:33 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_47.mover', 'VON277'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:33 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  written bytes 2147467776/2147467776, blocks 16384 header 195 trailer 317 Thread tape_thread
00:04:33 stkendca20a.fnal.gov 026341 root I ENCP  File /diska/write-pool-1/data/003000000000000006208558 transfered for stkendca20a.fnal.gov-1257487218-26341-0 in 33.2486460209 seconds.
00:04:34 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  filemarks written. Tape VON245 absolute location in blocks 1719380 Thread tape_thread
00:04:34 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  write_tape timing:pull 0.191066026688 write 12.1602959633 crc_check 9.93850207329 freespace 0.178670167923 block_write 22.4685342312 idle 0.306344747543 Thread tape_thread
00:04:34 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  drive stats after write. Tape VON245 position 110 block 0 block_size 0 bloc_loc 1719380 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 2147467776 block_write_tot 22.4685342312 tape_rate 95576674.2015 Thread tape_thread
00:04:34 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  remaining bytes info in DB 572571648000 estimated 570424180224 reported from drive 568888320000 position 111 Thread tape_thread
00:04:34 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  new bitfile request {'uid': 0, 'external_label': 'VON245', 'gid': 0, 'location_cookie': '0000_000000000_0000110', 'complete_crc': 3845501279L, 'sanity_cookie': (65536L, 1954066076L), 'size': 2147467264L} Thread tape_thread
00:04:34 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:34 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  set remaining: VON245 568888320000 0000_000000000_0000111 Thread tape_thread
00:04:34 last message repeated 1 times
00:04:34 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set last_access = '2009-11-06 00:04:34', eod_cookie = '0000_000000000_0000111', non_del_files = '112', remaining_bytes = 568888320000 where label = 'VON245';
00:04:34 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  transfer complete volume=VON245 location=111 Thread tape_thread
00:04:34 last message repeated 1 times
00:04:34 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_wr_access = '110', modification_time = '2009-11-06 00:04:34' where label = 'VON245';
00:04:34 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  Sending done to client: {'vc': {'comment': '', 'storage_group': 'exp-db', 'blocksize': 131072, 'declared': 1240352612.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VON245', 'wrapper': 'cpio_odc', 'remaining_bytes': 572571648000L, 'sum_mounts': 24, 'sum_rd_access': 0, 'file_family_width': 2, 'last_access': 1257487025.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000110', 'capacity_bytes': 858993459200L, 'non_del_files': 111, 'sum_wr_err': 0, 'sum_wr_access': 109, 'file_family': 'daily-d0-online-readonly', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-online-readonly.cpio_odc', 'modification_time': 1257487025.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1241768910.0}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-online-readonly/d0onlprd/2009/11-November/05/channel01/D0ONLPRD_t702168009_s2963_p22', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 2147467264L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7869, 'mc_device': '0,5,1,7', 'compression': 0, 'serial_num': '9310025229', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr209a', 'max_time_in_state': 1200, 'device': 'stkenmvr209a:/dev/rmt/tps0d0n', 'name': 'LTO4_109.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_109.stat', 'mover_address': ('131.225.13.205', 7869), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.205', 46288), 'data_ip': 'stkenmvr209a', 'logname': 'LTO4_109MV', 'hostip': '131.225.13.205', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-online-readonly/d0onlprd/2009/11-November/05/channel01/D0ONLPRD_t702168009_s2963_p22', 'uid': 0, 'uname': 'root', 'mtime': 1257487278, 'rmajor': 0, 'machine': ('Linux', 'stkendca19a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 820608480L, 20L, 1, 1602, 2752, 512L, 1257487268, 1257487268, 1254398413), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 2147467264L, 'gname': 'root', 'gid': 0, 'fullname': '/diskc/write-pool-3/data/003000000000000006208600', 'inode': 807437824L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskc/write-pool-3/data/003000000000000006208600', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-online-readonly/d0onlprd/2009/11-November/05/channel01/.(access)(003000000000000006208600)', 'fc': {'uid': 0, 'external_label': 'VON245', 'gid': 0, 'location_cookie': '0000_000000000_0000110', 'complete_crc': 3845501279L, 'bfid': 'CDMS125748747400000', 'sanity_cookie': (65536L, 1954066076L), 'size': 2147467264L}, 'infilepath': '/diskc/write-pool-3/data/003000000000000006208600', 'at_the_top': 1, 'local_inode': 539180112L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.11', 51935), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257487277.8918359, 'seek_time': 148.54683208465576, 'transfer_time': 46.442465782165527, 't0': 1257487277, 'lm_dequeued': 1257487279.746479, 'job_queued': 1257487278.881531, 'drive_call_time': 11.945704936981201, 'in_queue': 0.86488604545593262, 'drive_transfer_time': 24.550339937210083}, 'user_level': 'user2', 'unique_id': 'stkendca19a.fnal.gov-1257487278-5688-0'} Thread tape_thread
00:04:34 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  Done is sent Thread tape_thread
00:04:34 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  maybe_clean Thread tape_thread
00:04:34 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV  returned from maybe_clean Thread tape_thread
00:04:34 stkendca19a.fnal.gov 005688 root I ENCP  Received final dialog for stkendca19a.fnal.gov-1257487278-5688-0.  elapsed=196.940sec
00:04:34 stkendca19a.fnal.gov 005688 root I ENCP  leaving transfer_file().
00:04:34 stkendca19a.fnal.gov 005688 root I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = False
00:04:35 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102'), ('LTO4_47.mover', 'VON277'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:35 stkensrv0n.fnal.gov 028796 enstore I FILSRV  Updating  			update file set deleted = 'n', pnfs_path = '/pnfs/fnal.gov/usr/exp-db/daily/d0-online-readonly/d0onlprd/2009/11-November/05/channel01/D0ONLPRD_t702168009_s2963_p22', drive = 'stkenmvr209a:/dev/rmt/tps0d0n:9310025229', pnfs_id = '003000000000000006208600' where bfid = 'CDMS125748747400000';
00:04:35 stkendca19a.fnal.gov 005688 root I ENCP  MSG_TYPE=ENCP_XFER    write_to_hsm /diskc/write-pool-3/data/003000000000000006208600 -> /pnfs/fnal.gov/usr/exp-db/daily/d0-online-readonly/d0onlprd/2009/11-November/05/channel01/D0ONLPRD_t702168009_s2963_p22: 2147467264 bytes copied to VON245 at 44.1 MB/S (79 MB/S network) (83.4 MB/S drive) (48.5 MB/S disk) mover=LTO4_109.mover drive_id=ULTRIUM-TD4 drive_sn=9310025229 drive_vendor=IBM elapsed=197.22 {'storage_group': 'exp-db', 'encp_crc': 3845501279L, 'disk_rate': 48.453545849532325, 'drive_rate': 83.4197970471253, 'driver': 'FTTDriver', 'transfer_rate': 44.083923973686744, 'mover_interface': 'stkenmvr209a', 'overall_rate': 10.485470149800896, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 44.083923973686744, 'encp_ip': '131.225.13.11', 'unique_id': 'stkendca19a.fnal.gov-1257487278-5688-0'}
00:04:35 stkendca19a.fnal.gov 005688 root I ENCP  File /pnfs/fnal.gov/usr/exp-db/daily/d0-online-readonly/d0onlprd/2009/11-November/05/channel01/D0ONLPRD_t702168009_s2963_p22 copied successfully.
00:04:35 stkendca19a.fnal.gov 005688 root I ENCP  Completed transferring 2147467264 bytes in 1 files in 197.222337008 sec.  Overall rate = 10.5 MB/sec.  Transfer rate = 44.1 MB/sec.  Network rate = 79 MB/sec.  Drive rate = 83.4 MB/sec.  Disk rate = 48.5 MB/sec.  Exit status = 0.
00:04:35 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_47.mover', 'VON277'), ('LTO4_108.mover', 'VON399'), ('LTO4_104.mover', 'VON102')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_109.mover', 'VON245'), ('LTO4_104.mover', 'VON102')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:35 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_108.mover', 'VON399'), ('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:36 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  written bytes 1966080512/1966080512, blocks 15001 header 189 trailer 323 Thread tape_thread
00:04:36 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:37 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:37 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_109.mover', 'VON245')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_43.mover', 'VON268'), ('LTO4_47.mover', 'VON277')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:37 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  filemarks written. Tape VON268 absolute location in blocks 4325967 Thread tape_thread
00:04:37 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  write_tape timing:pull 0.176265239716 write 7.5584757328 crc_check 7.79408431053 freespace 0.178921699524 block_write 15.7077469826 idle 0.293728351593 Thread tape_thread
00:04:37 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  drive stats after write. Tape VON268 position 286 block 0 block_size 0 bloc_loc 4325967 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 1966080512 block_write_tot 15.7077469826 tape_rate 125166296.235 Thread tape_thread
00:04:37 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  remaining bytes info in DB 246338560000 estimated 244372479488 reported from drive 244503552000 position 287 Thread tape_thread
00:04:37 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  new bitfile request {'uid': 0, 'external_label': 'VON268', 'gid': 0, 'location_cookie': '0000_000000000_0000286', 'complete_crc': 3319289056L, 'sanity_cookie': (65536L, 3839108348L), 'size': 1966080000L} Thread tape_thread
00:04:37 stkendm7a.fnal.gov 009044 root I ENCP  File /data/data1/Migration_Spool/VO5178:0000_000000000_0000122 transfered for stkendm7a.fnal.gov-1257487256-9044-81 in 21.6724259853 seconds. Thread WRITE
00:04:37 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  set remaining: VON268 244503552000 0000_000000000_0000287 Thread tape_thread
00:04:37 last message repeated 1 times
00:04:37 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set last_access = '2009-11-06 00:04:37', eod_cookie = '0000_000000000_0000287', non_del_files = '287', remaining_bytes = 244503552000 where label = 'VON268';
00:04:37 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  transfer complete volume=VON268 location=287 Thread tape_thread
00:04:37 last message repeated 1 times
00:04:37 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_wr_access = '286', modification_time = '2009-11-06 00:04:37' where label = 'VON268';
00:04:37 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  Sending done to client: {'vc': {'comment': '', 'storage_group': 'exp-db', 'blocksize': 131072, 'declared': 1248994058.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 1242753248.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VON268', 'wrapper': 'cpio_odc', 'remaining_bytes': 246338560000L, 'sum_mounts': 150, 'sum_rd_access': 0, 'file_family_width': 2, 'last_access': 1257487439.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000286', 'capacity_bytes': 858993459200L, 'non_del_files': 286, 'sum_wr_err': 0, 'sum_wr_access': 285, 'file_family': 'daily-d0-offline', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'modification_time': 1257487439.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257406785.0}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702154603_s103300_p15', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7843, 'mc_device': '0,7,1,0', 'compression': 0, 'serial_num': '9310055909', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr143a', 'max_time_in_state': 1200, 'device': 'stkenmvr143a:/dev/rmt/tps0d0n', 'name': 'LTO4_43.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_43.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.171', 7843), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.171', 56256), 'data_ip': 'stkenmvr143a', 'logname': 'LTO4_43MV', 'hostip': '131.225.13.171', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702154603_s103300_p15', 'uid': 0, 'uname': 'root', 'mtime': 1257487218, 'rmajor': 0, 'machine': ('Linux', 'stkendca20a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 820619448L, 20L, 1, 1602, 2752, 512L, 1257487175, 1257487175, 1254398418), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 1966080000L, 'gname': 'root', 'gid': 0, 'fullname': '/diska/write-pool-1/data/003000000000000006208558', 'inode': 807437656L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diska/write-pool-1/data/003000000000000006208558', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/.(access)(003000000000000006208558)', 'fc': {'uid': 0, 'external_label': 'VON268', 'gid': 0, 'location_cookie': '0000_000000000_0000286', 'complete_crc': 3319289056L, 'bfid': 'CDMS125748747700000', 'sanity_cookie': (65536L, 3839108348L), 'size': 1966080000L}, 'infilepath': '/diska/write-pool-1/data/003000000000000006208558', 'at_the_top': 335, 'local_inode': 268435591L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.12', 44951), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257487217.0995319, 'seek_time': 0.035616159439086914, 'transfer_time': 37.475091934204102, 't0': 1257487217, 'mount_time': 0.041234970092773438, 'lm_dequeued': 1257487440.187356, 'job_queued': 1257487218.1049941, 'drive_call_time': 7.3600192070007324, 'in_queue': 222.08206987380981, 'drive_transfer_time': 17.695683479309082}, 'user_level':(trunc. 4145) Thread tape_thread
00:04:37 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  Done is sent Thread tape_thread
00:04:37 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  maybe_clean Thread tape_thread
00:04:37 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  returned from maybe_clean Thread tape_thread
00:04:37 stkendca20a.fnal.gov 026341 root I ENCP  Received final dialog for stkendca20a.fnal.gov-1257487218-26341-0.  elapsed=260.740sec
00:04:37 stkendca20a.fnal.gov 026341 root I ENCP  leaving transfer_file().
00:04:37 stkendca20a.fnal.gov 026341 root I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = False
00:04:38 stkensrv0n.fnal.gov 028796 enstore I FILSRV  Updating  			update file set deleted = 'n', pnfs_path = '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702154603_s103300_p15', drive = 'stkenmvr143a:/dev/rmt/tps0d0n:9310055909', pnfs_id = '003000000000000006208558' where bfid = 'CDMS125748747700000';
00:04:38 stkendca20a.fnal.gov 026341 root I ENCP  MSG_TYPE=ENCP_XFER    write_to_hsm /diska/write-pool-1/data/003000000000000006208558 -> /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702154603_s103300_p15: 1966080000 bytes copied to VON268 at 50 MB/S (79.6 MB/S network) (106 MB/S drive) (56.4 MB/S disk) mover=LTO4_43.mover drive_id=ULTRIUM-TD4 drive_sn=9310055909 drive_vendor=IBM elapsed=261.06 {'storage_group': 'exp-db', 'encp_crc': 3319289056L, 'disk_rate': 56.421765910349443, 'drive_rate': 105.95804350774974, 'driver': 'FTTDriver', 'transfer_rate': 50.006944985111588, 'mover_interface': 'stkenmvr143a', 'overall_rate': 49.454127940207009, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 50.006944985111588, 'encp_ip': '131.225.13.12', 'unique_id': 'stkendca20a.fnal.gov-1257487218-26341-0'}
00:04:38 stkendca20a.fnal.gov 026341 root I ENCP  File /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702154603_s103300_p15 copied successfully.
00:04:38 stkendca20a.fnal.gov 026341 root I ENCP  Completed transferring 1966080000 bytes in 1 files in 261.061390162 sec.  Overall rate = 49.5 MB/sec.  Transfer rate = 50 MB/sec.  Network rate = 79.6 MB/sec.  Drive rate = 106 MB/sec.  Disk rate = 56.4 MB/sec.  Exit status = 0.
00:04:38 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_109.mover', 'VON245'), ('LTO4_43.mover', 'VON268')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:38 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  write_to_hsm next work on vol=VON268 mover=LTO4_43.mover requester:root
00:04:38 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  HAVE_BOUND:sending write_to_hsm /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702152110_s103297_p50 to mover LTO4_43.mover ('131.225.13.171', 7843) DEL_DISM None
00:04:38 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  MSG_TYPE=ADD_TO_LMQ  request added to work at movers queue for storage group : exp-db
00:04:38 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  HAVE_BOUND:sending {'vc': {'storage_group': 'exp-db', 'external_label': 'VON268', 'library': 'CD-LTO4G1', 'file_family': 'daily-d0-offline', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'file_family_width': 2}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702152110_s103297_p50', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': 'LTO4_43.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702152110_s103297_p50', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257487386, 'rmajor': 0, 'machine': ('Linux', 'stkendca20a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (16877, 820619448L, 20L, 1, 1602, 2752, 512L, 1257487337, 1257487337, 1254398418), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 1966080000L, 'fullname': '/diskc/write-pool-3/data/0030000000000000062086E8', 'inode': 807438056L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskc/write-pool-3/data/0030000000000000062086E8', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/.(access)(0030000000000000062086E8)', 'fc': {'size': 1966080000L, 'external_label': 'VON268', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diskc/write-pool-3/data/0030000000000000062086E8', 'at_the_top': 134, 'local_inode': 270525112L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.12', 44962), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487478.2069509, 'encp_start_time': 1257487385.9267659, 'job_queued': 1257487387.0482759, 't0': 1257487385, 'in_queue': 91.158414125442505}, 'user_level': 'user2', 'unique_id': 'stkendca20a.fnal.gov-1257487386-27216-0'} to mover
00:04:38 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_109.mover', 'VON245'), ('LTO4_43.mover', 'VON268')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:38 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  WRITE_TO_HSM Thread MainThread
00:04:38 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  SENDING {'vc': {'storage_group': 'exp-db', 'external_label': 'VON268', 'library': 'CD-LTO4G1', 'file_family': 'daily-d0-offline', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'file_family_width': 2}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702152110_s103297_p50', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'dismount_delay': 30, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7843, 'mc_device': '0,7,1,0', 'compression': 0, 'serial_num': '9310055909', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr143a', 'max_time_in_state': 1200, 'device': 'stkenmvr143a:/dev/rmt/tps0d0n', 'name': 'LTO4_43.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_43.stat', 'single_filemark': 1, 'mount_delay': 15, 'mover_address': ('131.225.13.171', 7843), 'check_first_written_file': 0, 'callback_addr': ('131.225.13.171', 56259), 'data_ip': 'stkenmvr143a', 'logname': 'LTO4_43MV', 'hostip': '131.225.13.171', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/D0OFPRD1_t702152110_s103297_p50', 'uid': 0, 'uname': 'root', 'mtime': 1257487386, 'rmajor': 0, 'machine': ('Linux', 'stkendca20a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 820619448L, 20L, 1, 1602, 2752, 512L, 1257487337, 1257487337, 1254398418), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 1966080000L, 'gname': 'root', 'gid': 0, 'fullname': '/diskc/write-pool-3/data/0030000000000000062086E8', 'inode': 807438056L, 'minor': 20}, 'version': 'v3_8b  CVS $Revision: 1.922 $ <frozen>', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskc/write-pool-3/data/0030000000000000062086E8', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/05/channel01/.(access)(0030000000000000062086E8)', 'fc': {'size': 1966080000L, 'external_label': 'VON268', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diskc/write-pool-3/data/0030000000000000062086E8', 'at_the_top': 134, 'local_inode': 270525112L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.12', 44962), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487478.2069509, 'encp_start_time': 1257487385.9267659, 'job_queued': 1257487387.0482759, 't0': 1257487385, 'in_queue': 91.158414125442505}, 'user_level': 'user2', 'unique_id': 'stkendca20a.fnal.gov-1257487386-27216-0'} Thread client_connect_thread
00:04:38 stkendca20a.fnal.gov 027216 root I ENCP  Control socket ('131.225.13.12', 44962) is connected to ('131.225.13.171', 56260) for stkendca20a.fnal.gov-1257487386-27216-0.  elapsed=92.350sec
00:04:38 stkendca20a.fnal.gov 027216 root I ENCP  Received callback ticket from mover LTO4_43.mover for transfer stkendca20a.fnal.gov-1257487386-27216-0.  elapsed=92.350sec
00:04:38 stkendca20a.fnal.gov 027216 root I ENCP  Atempting to connect data socket to mover at ('131.225.13.171', 56259).
00:04:38 stkendca20a.fnal.gov 027216 root I ENCP  Connecting data socket to mover (('131.225.13.171', 56259)) with interface 131.225.13.12.  elapsed=92.350sec
00:04:38 stkendca20a.fnal.gov 027216 root I ENCP  Data socket ('131.225.13.12', 44966) is connected to ('131.225.13.171', 56259).  elapsed=92.350sec
00:04:38 stkendca20a.fnal.gov 027216 root I ENCP  Opening local file.
00:04:38 stkendca20a.fnal.gov 027216 root I ENCP  pf_status = ('ok', None)  skip_layer_cleanup = False
00:04:38 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  tape position: target 287 current 287 0 Thread seek_thread
00:04:38 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  write_tape starting, bytes_to_write=1966080512 Thread tape_thread
00:04:38 stkenmvr143a.fnal.gov 028425 root I LTO4_43MV  Write starting. Tape VON268 absolute location in blocks 4325967 Thread tape_thread
00:04:38 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:39 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245'), ('LTO4_43.mover', 'VON268'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:39 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399'), ('LTO4_43.mover', 'VON268')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:39 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_104.mover', 'VON102'), ('LTO4_108.mover', 'VON399'), ('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:40 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:41 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_104.mover', 'VON102'), ('LTO4_43.mover', 'VON268'), ('LTO4_109.mover', 'VON245'), ('LTO4_108.mover', 'VON399')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:41 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV  Updating stats Thread media_thread
00:04:41 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV  writing a tape mark before dismount Thread media_thread
00:04:41 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_104.mover', 'VON102'), ('LTO4_43.mover', 'VON268'), ('LTO4_108.mover', 'VON399'), ('LTO4_109.mover', 'VON245')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:42 stkensrv4n.fnal.gov 005357 enstore E LTO4G1LM  MOVER ERROR RQ {'status': ('ok', None), 'returned_work': None, 'library': None, 'mover_type': '', 'current_location': '0000_000000000_0000000', 'ip_map': '', 'operation': None, 'read_only': 0, 'current_time': 1257487482.3904381, 'error_source': None, 'external_label': None, 'state': 'OFFLINE', 'r_a': (('131.225.13.194', 54880), 18011L, '131.225.13.194-54880-1256586929.509013-4866'), 'library_list': [('CD-LTO4G1.library_manager', ('131.225.13.4', 8541))], 'current_priority': (None, None), 'transfer_deficiency': 1, 'time_in_state': 900552.0303800106, 'address': ('131.225.13.194', 7860), 'volume_family': None, 'work': 'mover_error', 'mover': 'LTO4_99.mover', 'volume_status': (['none', 'none'], ['none', 'none']), 'unique_id': None}
00:04:42 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:42 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  written bytes 1173211648/1173211648, blocks 8951 header 0 trailer 248 Thread tape_thread
00:04:43 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_104.mover', 'VON102'), ('LTO4_43.mover', 'VON268'), ('LTO4_108.mover', 'VON399'), ('LTO4_109.mover', 'VON245')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:44 stkensrv4n.fnal.gov 005795 enstore I 9940BLM  <storage groups {'ktev': [('9940B23.mover', 'VO5178')]} volume_families {'ktev.ktev_migration.cern': [('9940B23.mover', 'VO5178')]} >
00:04:44 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:44 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV  Ejecting tape Thread media_thread
00:04:45 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_104.mover', 'VON102'), ('LTO4_43.mover', 'VON268'), ('LTO4_108.mover', 'VON399'), ('LTO4_109.mover', 'VON245')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:46 last message repeated 1 times
00:04:46 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  filemarks written. Tape VOV147 absolute location in blocks 3559875 Thread tape_thread
00:04:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  write_tape timing:pull 0.0980994701385 write 5.08798193932 crc_check 5.39181494713 freespace 0.0890729427338 block_write 10.6669692993 idle 0.188401460648 Thread tape_thread
00:04:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  drive stats after write. Tape VOV147 position 7423 block 0 block_size 0 bloc_loc 3559875 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 1173211648 block_write_tot 10.6669692993 tape_rate 109985471.513 Thread tape_thread
00:04:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  remaining bytes info in DB 323701760000 estimated 322528548352 reported from drive 322606080000 position 7426 Thread tape_thread
00:04:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  new bitfile request {'uid': 0, 'external_label': 'VOV147', 'gid': 0, 'location_cookie': '0000_000000000_0007424', 'complete_crc': 1430703843L, 'sanity_cookie': (65536L, 3692817132L), 'size': 1173211400L} Thread tape_thread
00:04:47 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM  <storage groups {'exp-db': [('LTO4_47.mover', 'VON277'), ('LTO4_104.mover', 'VON102'), ('LTO4_43.mover', 'VON268'), ('LTO4_108.mover', 'VON399'), ('LTO4_109.mover', 'VON245')], 'cms': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} volume_families {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_47.mover', 'VON277'), ('LTO4_43.mover', 'VON268')], 'exp-db.archive-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON399')], 'exp-db.daily-d0-online-readonly.cpio_odc': [('LTO4_104.mover', 'VON102'), ('LTO4_109.mover', 'VON245')], 'cms.cms2.cpio_odc': [('LTO4_40.mover', 'VOS642'), ('LTO4_41.mover', 'VOS175')]} >
00:04:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  set remaining: VOV147 322606080000 0000_000000000_0007426 Thread tape_thread
00:04:47 last message repeated 1 times
00:04:47 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set last_access = '2009-11-06 00:04:47', eod_cookie = '0000_000000000_0007426', non_del_files = '2477', remaining_bytes = 322606080000 where label = 'VOV147';
00:04:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  transfer complete volume=VOV147 location=7426 Thread tape_thread
00:04:47 last message repeated 1 times
00:04:47 stkensrv0n.fnal.gov 028762 enstore I VOLSRV  Updating  			update volume set sum_wr_access = '2475', modification_time = '2009-11-06 00:04:47' where label = 'VOV147';
00:04:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  Sending done to client: {'vc': {'comment': '', 'storage_group': 'ktev', 'blocksize': 131072, 'declared': 1254165878.0, 'library': 'CD-LTO4F1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOV147', 'wrapper': 'cern', 'remaining_bytes': 323701760000L, 'sum_mounts': 5, 'sum_rd_access': 0, 'file_family_width': 1, 'last_access': 1257487455.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0007423', 'capacity_bytes': 858993459200L, 'non_del_files': 2476, 'sum_wr_err': 0, 'sum_wr_access': 2474, 'file_family': 'ktev_migration-MIGRATION', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'modification_time': 1257487455.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257458755.0}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f2', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 1173211400L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4F1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7873, 'mc_device': '1,1,1,4', 'compression': 0, 'serial_num': '1310156224', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr213a', 'max_time_in_state': 1200, 'device': 'stkenmvr213a:/dev/rmt/tps0d0n', 'name': 'LTO4_113.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_113.stat', 'mover_address': ('131.225.13.108', 7873), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.108', 43842), 'data_ip': 'stkenmvr213a', 'logname': 'LTO4_113MV', 'hostip': '131.225.13.108', 'do_cleaning': 'No', 'media_changer': 'SL8500F1.media_changer'}, 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f2', 'uid': 0, 'uname': 'root', 'mtime': 1257487256, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cern', 'pstat': (16877, 683683784L, 20L, 1, 0, 0, 512L, 1257487050, 1257487050, 1255127721), 'mode_octal': '0100016', 'mode': 33204, 'size_bytes': 1173211400L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000122', 'inode': 675738016L, 'minor': 20}, 'version': 'v3_8  CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000122', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC099/.(access)(00280000000000000846F1A0)', 'fc': {'uid': 0, 'external_label': 'VOV147', 'gid': 0, 'location_cookie': '0000_000000000_0007424', 'complete_crc': 1430703843L, 'bfid': 'CDMS125748748700000', 'sanity_cookie': (65536L, 3692817132L), 'size': 1173211400L}, 'infilepath': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000122', 'at_the_top': 906, 'local_inode': 39503181L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 44372), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257487256.0404711, 'seek_time': 0.033707141876220703, 'transfer_time': 31.560661792755127, 't0': 1257487256, 'mount_time': 0.04339289665222168, 'lm_dequeued': 1257487455.876488, 'job_queued': 1257487256.4334469, 'drive_call_time': 4.9788992404937744, 'in_queue': 199.44274711608887, 'drive_transfer_time': 16.72136664390564}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257487256-9044-81'} Thread tape_thread
00:04:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  Done is sent Thread tape_thread
00:04:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  maybe_clean Thread tape_thread
00:04:47 stkendm7a.fnal.gov 009044 root I ENCP  Received final dialog for stkendm7a.fnal.gov-1257487256-9044-81.  elapsed=5924.940sec Thread WRITE
00:04:47 stkendm7a.fnal.gov 009044 root I ENCP  leaving transfer_file(). Thread WRITE
00:04:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  returned from maybe_clean Thread tape_thread
00:04:47 stkensrv0n.fnal.gov 028796 enstore I FILSRV  Updating  			update file set deleted = 'n', pnfs_path = '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f2', drive = 'stkenmvr213a:/dev/rmt/tps0d0n:1310156224', pnfs_id = '00280000000000000846F1A0' where bfid = 'CDMS125748748700000';
00:04:47 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:47 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  write_to_hsm next work on vol=VOV147 mover=LTO4_113.mover requester:root
00:04:47 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  HAVE_BOUND:sending write_to_hsm /pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f3 to mover LTO4_113.mover ('131.225.13.108', 7873) DEL_DISM 2
00:04:47 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  MSG_TYPE=ADD_TO_LMQ  request added to work at movers queue for storage group : ktev
00:04:47 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  HAVE_BOUND:sending {'vc': {'storage_group': 'ktev', 'external_label': 'VOV147', 'library': 'CD-LTO4F1', 'file_family': 'ktev_migration-MIGRATION', 'wrapper': 'cern', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'file_family_width': 1}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f3', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 6694632932L, 'mover': 'LTO4_113.mover', 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f3', 'uid': 0, 'uname': 'root', 'type': 'cern', 'gid': 0, 'mtime': 1257487303, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (16877, 683683784L, 20L, 1, 0, 0, 512L, 1257487297, 1257487297, 1255127721), 'mode_octal': '0100016', 'mode': 33204, 'gname': 'root', 'size_bytes': 6694632932L, 'fullname': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000101', 'inode': 675738064L, 'minor': 20}, 'version': 'v3_8  CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000101', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC099/.(access)(00280000000000000846F1D0)', 'fc': {'size': 6694632932L, 'external_label': 'VOV147', 'pnfsid': '00280000000000000846F1D0', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000101', 'at_the_top': 183, 'local_inode': 39503176L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 44375), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487487.903271, 'encp_start_time': 1257487302.8291211, 'job_queued': 1257487303.229677, 't0': 1257487302, 'in_queue': 184.67355012893677}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257487303-9044-82'} to mover
00:04:47 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM  <storage groups {'miniboone': [('LTO4_111.mover', 'VOO436')], 'ktev': [('LTO4_113.mover', 'VOV147')]} volume_families {'ktev.ktev_migration-MIGRATION.cern': [('LTO4_113.mover', 'VOV147')], 'miniboone.OpenRootTree.cpio_odc': [('LTO4_111.mover', 'VOO436')]} >
00:04:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  WRITE_TO_HSM Thread MainThread
00:04:47 stkendm7a.fnal.gov 009044 root I ENCP  Listening for control socket at: ('131.225.13.19', 44375) Thread WRITE_2
00:04:47 stkendm7a.fnal.gov 009044 root I ENCP  Waiting for mover to connect control socket at address: ('131.225.13.19', 44375)  elapsed=5925.280sec Thread WRITE_2
00:04:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV  SENDING {'vc': {'storage_group': 'ktev', 'external_label': 'VOV147', 'library': 'CD-LTO4F1', 'file_family': 'ktev_migration-MIGRATION', 'wrapper': 'cern', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'file_family_width': 1}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f3', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 6694632932L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4F1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7873, 'mc_device': '1,1,1,4', 'compression': 0, 'serial_num': '1310156224', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr213a', 'max_time_in_state': 1200, 'device': 'stkenmvr213a:/dev/rmt/tps0d0n', 'name': 'LTO4_113.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_113.stat', 'mover_address': ('131.225.13.108', 7873), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.108', 43845), 'data_ip': 'stkenmvr213a', 'logname': 'LTO4_113MV', 'hostip': '131.225.13.108', 'do_cleaning': 'No', 'media_changer': 'SL8500F1.media_changer'}, 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZC0/KZC099/f3', 'uid': 0, 'uname': 'root', 'mtime': 1257487303, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cern', 'pstat': (16877, 683683784L, 20L, 1, 0, 0, 512L, 1257487297, 1257487297, 1255127721), 'mode_octal': '0100016', 'mode': 33204, 'size_bytes': 6694632932L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000101', 'inode': 675738064L, 'minor': 20}, 'version': 'v3_8  CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000101', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/KZ/KZC0/KZC099/.(access)(00280000000000000846F1D0)', 'fc': {'size': 6694632932L, 'external_label': 'VOV147', 'pnfsid': '00280000000000000846F1D0', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/data/data1/Migration_Spool/VO5178:0000_000000000_0000101', 'at_the_top': 183, 'local_inode': 39503176L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 44375), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257487487.903271, 'encp_start_time': 1257487302.8291211, 'job_queued': 1257487303.229677, 't0': 1257487302, 'in_queue': 184.67355012893677}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257487303-9044-82'} Thread client_connect_thread
00:04:48 stkendm7a.fnal.gov 009044 root I ENCP  Control socket ('131.225.13.19', 44375) is connected to ('131.225.13.108', 43846) for stkendm7a.fnal.gov-1257487303-9044-82.  elapsed=5925.290sec Thread WRITE_2
00:04:48 stkendm7a.fnal.gov 009044 root I ENCP  Received callback ticket from mover LTO4_113.mover for transfer stkendm7a.fnal.gov-1257487303-9044-82.  elapsed=5925.300sec Thread WRITE_2
00:04:48 stkendm7a.fnal.gov 009044 root I ENCP  Attempting to connect data socket to mover at ('131.225.13.108', 43845). Thread WRITE_2
00:04:48 stkendm7a.fnal.gov 009044 root I ENCP  Connecting data socket to mover (('131.225.13.108', 43845)) with interface 131.225.13.19.  elapsed=5925.300sec Thread WRITE_2
00:04:48 stkendm7a.fnal.gov 009044 root I ENCP  Data socket ('131.225.13.19', 44380) is connected to ('131.225.13.108', 43845).  elapsed=5925.300sec Thread WRITE_2
00:04:48 stkendm7a.fnal.gov 009044 root I ENCP  Opening local