00:00:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30062 current 30061 41 Thread seek_thread 00:00:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=19965346 Thread tape_thread 00:00:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 19965485/19965485, blocks 153 header 139 Thread tape_thread 00:00:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 19965346/19965346 bytes Thread net_thread 00:00:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30062 block 153 block_size 0 bloc_loc 5644053 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 19965485 block_read_tot 0.192321777344 tape_rate 103812918.515 Thread tape_thread 00:00:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 19965485/19965485 bytes Thread tape_thread 00:00:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.0023455619812 read 0.0902459621429 crc_check 0.0984501838684 push 0.0012800693512 block_read 0.192321777344 idle 0.0033814907074 Thread tape_thread 00:00:00 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:01 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:01 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30063 current 30062 153 Thread seek_thread 00:00:01 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC PROCESSING listDrives 00:00:01 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:01 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=20003967 Thread tape_thread 00:00:01 stkensrv4n.fnal.gov 005843 enstore I SL8500F1MC PROCESSING listDrives 00:00:01 stkensrv4n.fnal.gov 005852 enstore I STKMC PROCESSING listDrives 00:00:01 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:01 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 20004106/20004106, blocks 153 header 139 Thread tape_thread 00:00:01 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 20003967/20003967 bytes Thread net_thread 00:00:01 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:01 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30063 block 153 block_size 0 bloc_loc 5644207 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 20004106 block_read_tot 0.208140850067 tape_rate 96108505.3393 Thread tape_thread 00:00:01 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 20004106/20004106 bytes Thread tape_thread 00:00:01 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00441765785217 read 0.102741003036 crc_check 0.0996606349945 push 0.00132155418396 block_read 0.208140850067 idle 0.00339221954346 Thread tape_thread 00:00:02 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:02 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:02 stkenmvr193a.fnal.gov 031647 root I LTO4_93MV Updating stats Thread media_thread 00:00:02 stkenmvr193a.fnal.gov 031647 root I LTO4_93MV writing a tape mark before dismount Thread media_thread 00:00:02 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30064 current 30063 153 Thread seek_thread 00:00:02 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:02 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=60787205 Thread tape_thread 00:00:02 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:03 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV filemarks written. Tape VOV152 absolute location in blocks 427979 Thread tape_thread 00:00:03 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV write_tape timing:pull 0.000182628631592 write 0.00699186325073 crc_check 0.0133230686188 freespace 0.000175952911377 block_write 0.0206735134125 idle 0.000351905822754 Thread tape_thread 00:00:03 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV drive stats after write. Tape VOV152 position 244 block 0 block_size 0 bloc_loc 427979 tot_blocks 762939000 BOT 0 read_err 0 write_err 6 bytes 2885120 block_write_tot 0.0206735134125 tape_rate 139556346.444 Thread tape_thread 00:00:03 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV remaining bytes info in DB 722391040000 estimated 722388154880 reported from drive 722384896000 position 247 Thread tape_thread 00:00:03 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV new bitfile request {'uid': 0, 'external_label': 'VOV152', 'gid': 0, 'location_cookie': '0000_000000000_0000245', 'complete_crc': 560721132L, 'sanity_cookie': (65536L, 2285160672L), 'size': 2884912L} Thread tape_thread 00:00:03 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV set remaining: VOV152 722384896000 0000_000000000_0000247 Thread tape_thread 00:00:03 last message repeated 1 times 00:00:03 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-07 00:00:03', eod_cookie = '0000_000000000_0000247', non_del_files = '82', remaining_bytes = 722384896000 where label = 'VOV152'; 00:00:03 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV transfer complete volume=VOV152 location=247 Thread tape_thread 00:00:03 last message repeated 1 times 00:00:03 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '82', modification_time = '2009-11-07 00:00:03' where label = 'VOV152'; 00:00:03 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'ktev', 'blocksize': 131072, 'declared': 1254165880.0, 'library': 'CD-LTO4F1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOV152', 'wrapper': 'cern', 'remaining_bytes': 722391040000L, 'sum_mounts': 1, 'sum_rd_access': 0, 'file_family_width': 1, 'last_access': 1257573597.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000244', 'capacity_bytes': 858993459200L, 'non_del_files': 81, 'sum_wr_err': 0, 'sum_wr_access': 81, 'file_family': 'ktev_migration-MIGRATION', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'modification_time': 1257573597.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257571652.0}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NQ/NQMM/NQMM06/f0', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 2884912L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4F1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7875, 'mc_device': '1,1,1,11', 'compression': 0, 'serial_num': '1310051338', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr215a', 'max_time_in_state': 1200, 'device': 'stkenmvr215a:/dev/rmt/tps0d0n', 'name': 'LTO4_115.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_115.stat', 'mover_address': ('131.225.13.111', 7875), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.111', 39673), 'data_ip': 'stkenmvr215a', 'logname': 'LTO4_115MV', 'hostip': '131.225.13.111', 'do_cleaning': 'No', 'media_changer': 'SL8500F1.media_changer'}, 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM06/f0', 'uid': 0, 'uname': 'root', 'mtime': 1257573530, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cern', 'pstat': (16877, 675903896L, 20L, 1, 0, 0, 512L, 1257573520, 1257573520, 1257572656), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 2884912L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000395', 'inode': 675905376L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000395', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NQ/NQMM/NQMM06/.(access)(002800000000000008497F60)', 'fc': {'uid': 0, 'external_label': 'VOV152', 'gid': 0, 'location_cookie': '0000_000000000_0000245', 'complete_crc': 560721132L, 'bfid': 'CDMS125757360300000', 'sanity_cookie': (65536L, 2285160672L), 'size': 2884912L}, 'infilepath': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000395', 'at_the_top': 90, 'local_inode': 39503270L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40490), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257573529.9130931, 'seek_time': 0.035004854202270508, 'transfer_time': 5.6878299713134766, 't0': 1257573529, 'mount_time': 0.042342185974121094, 'lm_dequeued': 1257573597.6021919, 'job_queued': 1257573530.3287611, 'drive_call_time': 0.019403934478759766, 'in_queue': 67.273151874542236, 'drive_transfer_time': 5.4755778312683105}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573530-642-427'} Thread tape_thread 00:00:03 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV Done is sent Thread tape_thread 00:00:03 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV maybe_clean Thread tape_thread 00:00:03 stkendm7a.fnal.gov 000642 root I ENCP Received final dialog for stkendm7a.fnal.gov-1257573530-642-427. elapsed=5383.130sec Thread WRITE_2 00:00:03 stkendm7a.fnal.gov 000642 root I ENCP leaving transfer_file(). Thread WRITE_2 00:00:03 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV returned from maybe_clean Thread tape_thread 00:00:03 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 60787344/60787344, blocks 464 header 139 Thread tape_thread 00:00:03 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 60787205/60787205 bytes Thread net_thread 00:00:03 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:03 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30064 block 464 block_size 0 bloc_loc 5644672 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 60787344 block_read_tot 0.548105478287 tape_rate 110904463.48 Thread tape_thread 00:00:03 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 60787344/60787344 bytes Thread tape_thread 00:00:03 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00517296791077 read 0.239266872406 crc_check 0.299747228622 push 0.00391840934753 block_read 0.548105478287 idle 0.0106062889099 Thread tape_thread 00:00:03 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:03 last message repeated 1 times 00:00:03 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM06/f0', drive = 'stkenmvr215a:/dev/rmt/tps0d0n:1310051338', pnfs_id = '002800000000000008497F60' where bfid = 'CDMS125757360300000'; 00:00:03 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM write_to_hsm next work on vol=VOV152 mover=LTO4_115.mover requester:root 00:00:03 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM HAVE_BOUND:sending write_to_hsm /pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM04/f0 to mover LTO4_115.mover ('131.225.13.111', 7875) DEL_DISM 2 00:00:03 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : ktev 00:00:03 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM HAVE_BOUND:sending {'vc': {'storage_group': 'ktev', 'external_label': 'VOV152', 'library': 'CD-LTO4F1', 'file_family': 'ktev_migration-MIGRATION', 'wrapper': 'cern', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'file_family_width': 1}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NQ/NQMM/NQMM04/f0', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 539908500L, 'mover': 'LTO4_115.mover', 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM04/f0', 'uid': 0, 'uname': 'root', 'type': 'cern', 'gid': 0, 'mtime': 1257573541, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (16877, 675903776L, 20L, 1, 0, 0, 512L, 1257573107, 1257573107, 1257572647), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 539908500L, 'fullname': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000398', 'inode': 675905424L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000398', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NQ/NQMM/NQMM04/.(access)(002800000000000008497F90)', 'fc': {'size': 539908500L, 'external_label': 'VOV152', 'pnfsid': '002800000000000008497F90', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000398', 'at_the_top': 27, 'local_inode': 39503211L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40502), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573603.620652, 'encp_start_time': 1257573541.0217879, 'job_queued': 1257573541.2625999, 't0': 1257573541, 'in_queue': 62.35779595375061}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573541-642-428'} to mover 00:00:03 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:03 last message repeated 1 times 00:00:03 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV WRITE_TO_HSM Thread MainThread 00:00:03 stkendm7a.fnal.gov 000642 root I ENCP Listening for control socket at: ('131.225.13.19', 40502) Thread WRITE 00:00:03 stkendm7a.fnal.gov 000642 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.19', 40502) elapsed=5383.350sec Thread WRITE 00:00:03 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV SENDING {'vc': {'storage_group': 'ktev', 'external_label': 'VOV152', 'library': 'CD-LTO4F1', 'file_family': 'ktev_migration-MIGRATION', 'wrapper': 'cern', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'file_family_width': 1}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NQ/NQMM/NQMM04/f0', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 539908500L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4F1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7875, 'mc_device': '1,1,1,11', 'compression': 0, 'serial_num': '1310051338', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr215a', 'max_time_in_state': 1200, 'device': 'stkenmvr215a:/dev/rmt/tps0d0n', 'name': 'LTO4_115.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_115.stat', 'mover_address': ('131.225.13.111', 7875), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.111', 39675), 'data_ip': 'stkenmvr215a', 'logname': 'LTO4_115MV', 'hostip': '131.225.13.111', 'do_cleaning': 'No', 'media_changer': 'SL8500F1.media_changer'}, 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM04/f0', 'uid': 0, 'uname': 'root', 'mtime': 1257573541, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cern', 'pstat': (16877, 675903776L, 20L, 1, 0, 0, 512L, 1257573107, 1257573107, 1257572647), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 539908500L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000398', 'inode': 675905424L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000398', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NQ/NQMM/NQMM04/.(access)(002800000000000008497F90)', 'fc': {'size': 539908500L, 'external_label': 'VOV152', 'pnfsid': '002800000000000008497F90', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000398', 'at_the_top': 27, 'local_inode': 39503211L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40502), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573603.620652, 'encp_start_time': 1257573541.0217879, 'job_queued': 1257573541.2625999, 't0': 1257573541, 'in_queue': 62.35779595375061}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573541-642-428'} Thread client_connect_thread 00:00:03 stkendm7a.fnal.gov 000642 root I ENCP Control socket ('131.225.13.19', 40502) is connected to ('131.225.13.111', 39676) for stkendm7a.fnal.gov-1257573541-642-428. elapsed=5383.360sec Thread WRITE 00:00:03 stkendm7a.fnal.gov 000642 root I ENCP Received callback ticket from mover LTO4_115.mover for transfer stkendm7a.fnal.gov-1257573541-642-428. elapsed=5383.360sec Thread WRITE 00:00:03 stkendm7a.fnal.gov 000642 root I ENCP Attempting to connect data socket to mover at ('131.225.13.111', 39675). Thread WRITE 00:00:03 stkendm7a.fnal.gov 000642 root I ENCP Connecting data socket to mover (('131.225.13.111', 39675)) with interface 131.225.13.19. elapsed=5383.360sec Thread WRITE 00:00:03 stkendm7a.fnal.gov 000642 root I ENCP Data socket ('131.225.13.19', 40519) is connected to ('131.225.13.111', 39675). elapsed=5383.360sec Thread WRITE 00:00:03 stkendm7a.fnal.gov 000642 root I ENCP Opening local file. Thread WRITE 00:00:03 stkendm7a.fnal.gov 000642 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False Thread WRITE 00:00:03 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV tape position: target 247 current 247 0 Thread seek_thread 00:00:03 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV write_tape starting, bytes_to_write=539908608 Thread tape_thread 00:00:03 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV Write starting. Tape VOV152 absolute location in blocks 427979 Thread tape_thread 00:00:03 stkendm7a.fnal.gov 000642 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /data/data1/Migration_Spool/VO6043:0000_000000000_0000395 -> /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NQ/NQMM/NQMM06/f0: 2884912 bytes copied to VOV152 at 0.483 MB/S (54.9 MB/S network) (0.502 MB/S drive) (5.67 MB/S disk) mover=LTO4_115.mover drive_id=ULTRIUM-TD4 drive_sn=1310051338 drive_vendor=IBM elapsed=73.977 {'storage_group': 'ktev', 'encp_crc': 560721132L, 'disk_rate': 5.6725740401760287, 'drive_rate': 0.5024613957235835, 'driver': 'FTTDriver', 'transfer_rate': 0.48348445895705533, 'mover_interface': 'stkenmvr215a', 'overall_rate': 0.44063622327344276, 'media_changer': 'SL8500F1.media_changer', 'network_rate': 0.48348445895705533, 'encp_ip': '131.225.13.19', 'unique_id': 'stkendm7a.fnal.gov-1257573530-642-427'} Thread WRITE_2 00:00:03 stkendm7a.fnal.gov 000642 root I ENCP File /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NQ/NQMM/NQMM06/f0 copied successfully. Thread WRITE_2 00:00:03 stkendm7a.fnal.gov 000642 root I ENCP Completed transferring 2884912 bytes in 1 files in 73.9779629707 sec. Overall rate = 0.441 MB/sec. Transfer rate = 0.483 MB/sec. Network rate = 54.9 MB/sec. Drive rate = 0.502 MB/sec. Disk rate = 5.67 MB/sec. Exit status = 0. Thread WRITE_2 00:00:04 stkensrv0n.fnal.gov 028796 enstore I FILSRV start modifying {'pnfsid': '002800000000000008497F60', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM06/f0', 'deleted': 'no', 'external_label': 'VOV152', 'drive': 'stkenmvr215a:/dev/rmt/tps0d0n:1310051338', 'update': '2009-11-07 00:00:03.611593', 'gid': 0, 'location_cookie': '0000_000000000_0000245', 'complete_crc': 560721132L, 'uid': 0, 'bfid': 'CDMS125757360300000', 'sanity_cookie': (65536L, 2285160672L), 'size': 2884912L} 00:00:04 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set pnfs_id = '001F000000000000007E6B30' where bfid = 'CDMS125757360300000'; 00:00:04 stkensrv0n.fnal.gov 028796 enstore I FILSRV modified to {'pnfsid': '001F000000000000007E6B30', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM06/f0', 'deleted': 'no', 'external_label': 'VOV152', 'drive': 'stkenmvr215a:/dev/rmt/tps0d0n:1310051338', 'update': '2009-11-07 00:00:03.611593', 'gid': 0, 'location_cookie': '0000_000000000_0000245', 'complete_crc': 560721132L, 'uid': 0, 'bfid': 'CDMS125757360300000', 'sanity_cookie': (65536L, 2285160672L), 'size': 2884912L} 00:00:04 stkendm7a.fnal.gov 000642 root I ENCP Version: v3_8 CVS $Revision: 1.963 $ encp.pyc OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4F1 Storage Group: ktev File Family: ktev_migration-MIGRATION FF Wrapper: cern FF Width: 1 Current working directory: stkendm7a.fnal.gov:/opt/enstore/src Command line: encp --delayed-dismount 2 --ignore-fair-share --threaded --priority 0 --storage-group ktev --file-family ktev_migration-MIGRATION --file-family-wrapper cern --library CD-LTO4F1 --override-path /pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM06/f13 --threaded /data/data1/Migration_Spool/VO6043:0000_000000000_0000404 /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NQ/NQMM/NQMM06/f13 Thread WRITE_2 00:00:04 stkendm7a.fnal.gov 000642 root I ENCP Creating zero length output files. Thread WRITE_2 00:00:04 stkendm7a.fnal.gov 000642 root I ENCP Sending /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NQ/NQMM/NQMM06/f13 write request to LM: unique_id: stkendm7a.fnal.gov-1257573604-642-430 inputfile: /data/data1/Migration_Spool/VO6043:0000_000000000_0000404 outputfile: /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NQ/NQMM/NQMM06/.(access)(0028000000000000084980C8) Thread WRITE_2 00:00:04 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : ktev 00:00:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:04 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM write Q'd /data/data1/Migration_Spool/VO6043:0000_000000000_0000404 (stkendm7a.fnal.gov-1257573604-642-430) -> /pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM06/f13 : library=CD-LTO4F1 family=ktev_migration-MIGRATION requester:root volume_family:ktev.ktev_migration-MIGRATION.cern 00:00:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30065 current 30064 464 Thread seek_thread 00:00:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=15017473 Thread tape_thread 00:00:04 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 15017612/15017612, blocks 115 header 139 Thread tape_thread 00:00:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 15017473/15017473 bytes Thread net_thread 00:00:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30065 block 115 block_size 0 bloc_loc 5644788 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 15017612 block_read_tot 0.149585485458 tape_rate 100394847.495 Thread tape_thread 00:00:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 15017612/15017612 bytes Thread tape_thread 00:00:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00206398963928 read 0.072238445282 crc_check 0.0742809772491 push 0.00100207328796 block_read 0.149585485458 idle 0.00260424613953 Thread tape_thread 00:00:05 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:05 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:05 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30066 current 30065 115 Thread seek_thread 00:00:05 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:05 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=14828853 Thread tape_thread 00:00:05 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 14828992/14828992, blocks 114 header 139 Thread tape_thread 00:00:05 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 14828853/14828853 bytes Thread net_thread 00:00:06 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:06 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30066 block 114 block_size 0 bloc_loc 5644903 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 14828992 block_read_tot 0.149569034576 tape_rate 99144799.8712 Thread tape_thread 00:00:06 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 14828992/14828992 bytes Thread tape_thread 00:00:06 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00156736373901 read 0.074107170105 crc_check 0.0729427337646 push 0.000951766967773 block_read 0.149569034576 idle 0.00271129608154 Thread tape_thread 00:00:06 stkenmvr193a.fnal.gov 031647 root I LTO4_93MV Ejecting tape Thread media_thread 00:00:06 stkensrv4n.fnal.gov 031653 enstore I SL8500F1MC dismount VOLUME 1,2,1,8 force => 0,Dismount: Forced dismount of VOV128 from 1, 2, 1, 8 00:00:06 stkensrv4n.fnal.gov 005843 enstore I SL8500F1MC FINISHED dismount VOV128 1,2,1,8 returned ('ok', 0, 'dismount VOLUME 1,2,1,8 force => 0,Dismount: Forced dismount of VOV128 from 1, 2, 1, 8', '', '') 00:00:06 stkenmvr217a.fnal.gov 007075 root I LTO4_117MV dismounted VOV128 ULTRIUM-TD4 00:00:06 Thread media_thread 00:00:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30067 current 30066 114 Thread seek_thread 00:00:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=14216215 Thread tape_thread 00:00:07 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC FINISHED listDrives returned ('ok', 0, None, '', '') 00:00:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 14216354/14216354, blocks 109 header 139 Thread tape_thread 00:00:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 14216215/14216215 bytes Thread net_thread 00:00:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30067 block 109 block_size 0 bloc_loc 5645013 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 14216354 block_read_tot 0.143262386322 tape_rate 99232983.3739 Thread tape_thread 00:00:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 14216354/14216354 bytes Thread tape_thread 00:00:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00176954269409 read 0.0705363750458 crc_check 0.0700261592865 push 0.000930309295654 block_read 0.143262386322 idle 0.00249409675598 Thread tape_thread 00:00:07 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:08 stkensrv0n.fnal.gov 023312 enstore I INFSRV start listing all volumes (3) 00:00:08 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:08 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30068 current 30067 109 Thread seek_thread 00:00:08 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:08 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=20245520 Thread tape_thread 00:00:08 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 20245659/20245659, blocks 155 header 139 Thread tape_thread 00:00:08 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 20245520/20245520 bytes Thread net_thread 00:00:08 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:08 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30068 block 155 block_size 0 bloc_loc 5645169 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 20245659 block_read_tot 0.196655511856 tape_rate 102949868.066 Thread tape_thread 00:00:08 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 20245659/20245659 bytes Thread tape_thread 00:00:08 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00197696685791 read 0.0934307575226 crc_check 0.0999362468719 push 0.00131154060364 block_read 0.196655511856 idle 0.00354599952698 Thread tape_thread 00:00:09 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:09 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:09 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30069 current 30068 155 Thread seek_thread 00:00:09 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:09 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=19950995 Thread tape_thread 00:00:09 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 19951134/19951134, blocks 153 header 139 Thread tape_thread 00:00:09 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 19950995/19950995 bytes Thread net_thread 00:00:09 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:09 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30069 block 153 block_size 0 bloc_loc 5645323 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 19951134 block_read_tot 0.195259094238 tape_rate 102177745.307 Thread tape_thread 00:00:09 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 19951134/19951134 bytes Thread tape_thread 00:00:09 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00186467170715 read 0.0936033725739 crc_check 0.098518371582 push 0.00127267837524 block_read 0.195259094238 idle 0.00338077545166 Thread tape_thread 00:00:10 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:11 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30070 current 30069 153 Thread seek_thread 00:00:11 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:11 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=60569585 Thread tape_thread 00:00:11 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 60569724/60569724, blocks 463 header 139 Thread tape_thread 00:00:11 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 60569585/60569585 bytes Thread net_thread 00:00:11 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:11 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30070 block 463 block_size 0 bloc_loc 5645787 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 60569724 block_read_tot 0.544750928879 tape_rate 111187922.386 Thread tape_thread 00:00:11 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 60569724/60569724 bytes Thread tape_thread 00:00:11 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00468754768372 read 0.23699760437 crc_check 0.299197912216 push 0.00386786460876 block_read 0.544750928879 idle 0.0110158920288 Thread tape_thread 00:00:11 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:12 stkenmvr109a.fnal.gov 031352 root I LTO09MV skip_fm 241 Thread seek_thread 00:00:12 stkenmvr109a.fnal.gov 031352 root I LTO09MV read_tape starting, bytes_to_read=1784268367 Thread tape_thread 00:00:12 stkensrv4n.fnal.gov 005852 enstore I STKMC FINISHED listDrives returned ('ok', 0, None, '', '') 00:00:12 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:12 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30071 current 30070 463 Thread seek_thread 00:00:12 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:12 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=9584376 Thread tape_thread 00:00:12 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:12 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 9584515/9584515, blocks 74 header 139 Thread tape_thread 00:00:12 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 9584376/9584376 bytes Thread net_thread 00:00:12 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:12 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30071 block 74 block_size 0 bloc_loc 5645862 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 9584515 block_read_tot 0.101683616638 tape_rate 94258203.2079 Thread tape_thread 00:00:12 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 9584515/9584515 bytes Thread tape_thread 00:00:12 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00165557861328 read 0.052122592926 crc_check 0.0472877025604 push 0.000617742538452 block_read 0.101683616638 idle 0.00175786018372 Thread tape_thread 00:00:13 stkendca20a.fnal.gov 023640 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: exp-db File Family: daily-d0-offline FF Wrapper: cpio_odc FF Width: 2 Current working directory: stkendca20a.fnal.gov:/home/enstore/dcache-code/dcache-fermi-config Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel01/D0OFPRD1_t702245888_s103346_p1 --put-cache 003000000000000006222A10 /diskc/write-pool-3/data/003000000000000006222A10 00:00:13 stkendca20a.fnal.gov 023649 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: exp-db File Family: daily-d0-offline FF Wrapper: Unknown FF Width: 2 Current working directory: stkendca20a.fnal.gov:/home/enstore/dcache-code/dcache-fermi-config Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel02/D0OFPRD1_t702243438_s103342_p12 --put-cache 003000000000000006222948 /diskc/write-pool-3/data/003000000000000006222948 00:00:13 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:13 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:13 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30072 current 30071 74 Thread seek_thread 00:00:13 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:13 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=19046571 Thread tape_thread 00:00:13 stkendca20a.fnal.gov 023640 root I ENCP Creating zero length output files. 00:00:13 stkendca20a.fnal.gov 023640 root I ENCP Sending /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel01/D0OFPRD1_t702245888_s103346_p1 write request to LM: unique_id: stkendca20a.fnal.gov-1257573613-23640-0 inputfile: /diskc/write-pool-3/data/003000000000000006222A10 outputfile: /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel01/.(access)(003000000000000006222A10) 00:00:13 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : exp-db 00:00:13 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /diskc/write-pool-3/data/003000000000000006222A10 (stkendca20a.fnal.gov-1257573613-23640-0) -> /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel01/D0OFPRD1_t702245888_s103346_p1 : library=CD-LTO4G1 family=daily-d0-offline requester:root volume_family:exp-db.daily-d0-offline.cpio_odc 00:00:13 stkendca20a.fnal.gov 023640 root I ENCP Listening for control socket at: ('131.225.13.12', 49618) 00:00:13 stkendca20a.fnal.gov 023640 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.12', 49618) elapsed=2.110sec 00:00:13 stkendca20a.fnal.gov 023649 root I ENCP Creating zero length output files. 00:00:14 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 19046710/19046710, blocks 146 header 139 Thread tape_thread 00:00:14 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 19046571/19046571 bytes Thread net_thread 00:00:14 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:14 stkendca20a.fnal.gov 023649 root I ENCP Sending /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel02/D0OFPRD1_t702243438_s103342_p12 write request to LM: unique_id: stkendca20a.fnal.gov-1257573613-23649-0 inputfile: /diskc/write-pool-3/data/003000000000000006222948 outputfile: /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel02/.(access)(003000000000000006222948) 00:00:14 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30072 block 146 block_size 0 bloc_loc 5646009 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 19046710 block_read_tot 0.18542265892 tape_rate 102720509.515 Thread tape_thread 00:00:14 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 19046710/19046710 bytes Thread tape_thread 00:00:14 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00240588188171 read 0.0877876281738 crc_check 0.0939786434174 push 0.00125050544739 block_read 0.18542265892 idle 0.00323748588562 Thread tape_thread 00:00:14 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : exp-db 00:00:14 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /diskc/write-pool-3/data/003000000000000006222948 (stkendca20a.fnal.gov-1257573613-23649-0) -> /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel02/D0OFPRD1_t702243438_s103342_p12 : library=CD-LTO4G1 family=daily-d0-offline requester:root volume_family:exp-db.daily-d0-offline.cpio_odc 00:00:14 stkendca20a.fnal.gov 023649 root I ENCP Listening for control socket at: ('131.225.13.12', 49619) 00:00:14 stkendca20a.fnal.gov 023649 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.12', 49619) elapsed=2.210sec 00:00:14 stkendca17a.fnal.gov 002253 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: exp-db File Family: daily-cdf-offline FF Wrapper: cpio_odc FF Width: 2 Current working directory: stkendca17a.fnal.gov:/opt/d-cache/config Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/exp-db/daily/cdf-offline/cdfofpr2/2009/11-November/06/channel01/CDFOFPR2_t702232504_s30426_p4 --put-cache 003000000000000006222860 /diskc/write-pool-3/data/003000000000000006222860 00:00:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30073 current 30072 146 Thread seek_thread 00:00:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=5318459 Thread tape_thread 00:00:15 stkendca17a.fnal.gov 002253 root I ENCP Creating zero length output files. 00:00:15 stkendca17a.fnal.gov 002253 root I ENCP Sending /pnfs/fnal.gov/usr/exp-db/daily/cdf-offline/cdfofpr2/2009/11-November/06/channel01/CDFOFPR2_t702232504_s30426_p4 write request to LM: unique_id: stkendca17a.fnal.gov-1257573615-2253-0 inputfile: /diskc/write-pool-3/data/003000000000000006222860 outputfile: /pnfs/fnal.gov/usr/exp-db/daily/cdf-offline/cdfofpr2/2009/11-November/06/channel01/.(access)(003000000000000006222860) 00:00:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 5318598/5318598, blocks 41 header 139 Thread tape_thread 00:00:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 5318459/5318459 bytes Thread net_thread 00:00:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:15 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : exp-db 00:00:15 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /diskc/write-pool-3/data/003000000000000006222860 (stkendca17a.fnal.gov-1257573615-2253-0) -> /pnfs/fnal.gov/usr/exp-db/daily/cdf-offline/cdfofpr2/2009/11-November/06/channel01/CDFOFPR2_t702232504_s30426_p4 : library=CD-LTO4G1 family=daily-cdf-offline requester:root volume_family:exp-db.daily-cdf-offline.cpio_odc 00:00:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30073 block 41 block_size 0 bloc_loc 5646051 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 5318598 block_read_tot 0.0677392482758 tape_rate 78515751.7301 Thread tape_thread 00:00:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 5318598/5318598 bytes Thread tape_thread 00:00:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.000919818878174 read 0.0402688980103 crc_check 0.0261926651001 push 0.000357866287231 block_read 0.0677392482758 idle 0.000851154327393 Thread tape_thread 00:00:15 stkendca17a.fnal.gov 002253 root I ENCP Listening for control socket at: ('131.225.13.5', 50143) 00:00:15 stkendca17a.fnal.gov 002253 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.5', 50143) elapsed=1.190sec 00:00:15 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Assigning volume VON267 from storage group exp-db.daily-cdf-offline.cpio_odc to library CD-LTO4G1, volume family exp-db.daily-cdf-offline.cpio_odc 00:00:15 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm work on vol=VON267 mover=LTO4_110.mover requester:root 00:00:15 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : exp-db 00:00:15 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM IDLE:sending {'vc': {'storage_group': 'exp-db', 'library': 'CD-LTO4G1', 'file_family': 'daily-cdf-offline', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-cdf-offline.cpio_odc', 'file_family_width': 2}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/cdf-offline/cdfofpr2/2009/11-November/06/channel01/CDFOFPR2_t702232504_s30426_p4', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 2147475456L, 'mover': 'LTO4_110.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/cdf-offline/cdfofpr2/2009/11-November/06/channel01/CDFOFPR2_t702232504_s30426_p4', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257573615, 'rmajor': 0, 'machine': ('Linux', 'stkendca17a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (16877, 820657904L, 20L, 1, 1602, 2752, 512L, 1257573591, 1257573591, 1254398467), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 2147475456L, 'fullname': '/diskc/write-pool-3/data/003000000000000006222860', 'inode': 807544928L, 'minor': 20}, 'version': 'v3_8b CVS $Revision: 1.922 $ ', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskc/write-pool-3/data/003000000000000006222860', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/cdf-offline/cdfofpr2/2009/11-November/06/channel01/.(access)(003000000000000006222860)', 'fc': {'size': 2147475456L, 'external_label': 'VON267', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diskc/write-pool-3/data/003000000000000006222860', 'at_the_top': 1, 'local_inode': 268645592L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.5', 50143), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573615.2926919, 'encp_start_time': 1257573614.037148, 'job_queued': 1257573615.1887839, 't0': 1257573614, 'in_queue': 0.10385012626647949}, 'user_level': 'user2', 'unique_id': 'stkendca17a.fnal.gov-1257573615-2253-0'} to mover 00:00:15 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV WRITE_TO_HSM Thread MainThread 00:00:15 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM can not remove from sg LTO4_110.mover VON267 00:00:15 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM SG: {'exp-db': [('LTO4_108.mover', 'VON279'), ('LTO4_93.mover', 'VON726'), ('LTO4_94.mover', 'VON278')]} 00:00:15 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM can not remove from vf LTO4_110.mover VON267 00:00:15 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM VF: {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON279'), ('LTO4_94.mover', 'VON278')], 'exp-db.daily-cdf-offline.cpio_odc': [('LTO4_93.mover', 'VON726')]} 00:00:15 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:15 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV SENDING {'vc': {'storage_group': 'exp-db', 'library': 'CD-LTO4G1', 'file_family': 'daily-cdf-offline', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-cdf-offline.cpio_odc', 'file_family_width': 2}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/cdf-offline/cdfofpr2/2009/11-November/06/channel01/CDFOFPR2_t702232504_s30426_p4', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 2147475456L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7870, 'mc_device': '0,5,1,14', 'compression': 0, 'serial_num': '1310049971', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr210a', 'max_time_in_state': 1200, 'device': 'stkenmvr210a:/dev/rmt/tps0d0n', 'name': 'LTO4_110.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_110.stat', 'mover_address': ('131.225.13.206', 7870), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.206', 46260), 'data_ip': 'stkenmvr210a', 'logname': 'LTO4_110MV', 'hostip': '131.225.13.206', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/cdf-offline/cdfofpr2/2009/11-November/06/channel01/CDFOFPR2_t702232504_s30426_p4', 'uid': 0, 'uname': 'root', 'mtime': 1257573615, 'rmajor': 0, 'machine': ('Linux', 'stkendca17a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 820657904L, 20L, 1, 1602, 2752, 512L, 1257573591, 1257573591, 1254398467), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 2147475456L, 'gname': 'root', 'gid': 0, 'fullname': '/diskc/write-pool-3/data/003000000000000006222860', 'inode': 807544928L, 'minor': 20}, 'version': 'v3_8b CVS $Revision: 1.922 $ ', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskc/write-pool-3/data/003000000000000006222860', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/cdf-offline/cdfofpr2/2009/11-November/06/channel01/.(access)(003000000000000006222860)', 'fc': {'size': 2147475456L, 'external_label': 'VON267', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diskc/write-pool-3/data/003000000000000006222860', 'at_the_top': 1, 'local_inode': 268645592L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.5', 50143), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573615.2926919, 'encp_start_time': 1257573614.037148, 'job_queued': 1257573615.1887839, 't0': 1257573614, 'in_queue': 0.10385012626647949}, 'user_level': 'user2', 'unique_id': 'stkendca17a.fnal.gov-1257573615-2253-0'} Thread client_connect_thread 00:00:15 stkendca17a.fnal.gov 002253 root I ENCP Control socket ('131.225.13.5', 50143) is connected to ('131.225.13.206', 46261) for stkendca17a.fnal.gov-1257573615-2253-0. elapsed=1.300sec 00:00:15 stkendca17a.fnal.gov 002253 root I ENCP Received callback ticket from mover LTO4_110.mover for transfer stkendca17a.fnal.gov-1257573615-2253-0. elapsed=1.310sec 00:00:15 stkendca17a.fnal.gov 002253 root I ENCP Atempting to connect data socket to mover at ('131.225.13.206', 46260). 00:00:15 stkendca17a.fnal.gov 002253 root I ENCP Connecting data socket to mover (('131.225.13.206', 46260)) with interface 131.225.13.5. elapsed=1.310sec 00:00:15 stkendca17a.fnal.gov 002253 root I ENCP Data socket ('131.225.13.5', 50144) is connected to ('131.225.13.206', 46260). elapsed=1.310sec 00:00:15 stkendca17a.fnal.gov 002253 root I ENCP Opening local file. 00:00:15 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV MSG_TYPE=MC_LOAD_REQ mounting VON267 ULTRIUM-TD4 00:00:15 Thread media_thread 00:00:15 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC PROCESSING mount VON267 0,5,1,14 00:00:15 stkendca17a.fnal.gov 002253 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:15 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:15 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:16 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:16 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30074 current 30073 41 Thread seek_thread 00:00:16 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:16 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=19941445 Thread tape_thread 00:00:16 stkendm7a.fnal.gov 000642 root I ENCP File /data/data1/Migration_Spool/VO6043:0000_000000000_0000398 transfered for stkendm7a.fnal.gov-1257573541-642-428 in 12.5992200375 seconds. Thread WRITE 00:00:16 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 19941584/19941584, blocks 153 header 139 Thread tape_thread 00:00:16 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 19941445/19941445 bytes Thread net_thread 00:00:16 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:16 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30074 block 153 block_size 0 bloc_loc 5646205 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 19941584 block_read_tot 0.191915988922 tape_rate 103907882.36 Thread tape_thread 00:00:16 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 19941584/19941584 bytes Thread tape_thread 00:00:16 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00260424613953 read 0.0897097587585 crc_check 0.0982775688171 push 0.00132441520691 block_read 0.191915988922 idle 0.00327563285828 Thread tape_thread 00:00:16 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:17 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:17 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30075 current 30074 153 Thread seek_thread 00:00:17 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:17 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=1344184 Thread tape_thread 00:00:17 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 1344323/1344323, blocks 11 header 139 Thread tape_thread 00:00:17 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 1344184/1344184 bytes Thread net_thread 00:00:17 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:17 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30075 block 11 block_size 0 bloc_loc 5646217 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 1344323 block_read_tot 0.0332741737366 tape_rate 40401393.9052 Thread tape_thread 00:00:17 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 1344323/1344323 bytes Thread tape_thread 00:00:17 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.000468254089355 read 0.0261857509613 crc_check 0.00652837753296 push 9.17911529541e-05 block_read 0.0332741737366 idle 0.00035572052002 Thread tape_thread 00:00:17 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:17 stkensrv4n.fnal.gov 032104 enstore I SL8500G1MC query drive 0,5,1,14 => 0,0,5,1,14 online available IBM-LTO4 00:00:17 stkensrv4n.fnal.gov 032104 enstore I SL8500G1MC Dismount 14 ignored: Drive 0,5,1,14 is empty. Thought VON267 was there =>. dismount VOLUME 0,5,1,14 force => ok,0,5,1,14 online available IBM-LTO4 00:00:18 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:18 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30076 current 30075 11 Thread seek_thread 00:00:18 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:18 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=6941521 Thread tape_thread 00:00:18 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 6941660/6941660, blocks 53 header 139 Thread tape_thread 00:00:18 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 6941521/6941521 bytes Thread net_thread 00:00:18 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:18 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30076 block 53 block_size 0 bloc_loc 5646271 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 6941660 block_read_tot 0.0816617012024 tape_rate 85005086.8129 Thread tape_thread 00:00:18 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 6941660/6941660 bytes Thread tape_thread 00:00:18 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.000910043716431 read 0.0459296703339 crc_check 0.0343499183655 push 0.000472068786621 block_read 0.0816617012024 idle 0.00120878219604 Thread tape_thread 00:00:19 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV written bytes 539908608/539908608, blocks 4120 header 0 trailer 108 Thread tape_thread 00:00:19 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:19 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:19 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30077 current 30076 53 Thread seek_thread 00:00:19 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:19 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=15097611 Thread tape_thread 00:00:20 stkensrv4n.fnal.gov 032104 enstore I SL8500G1MC query vol VON267 => 0,VON267 home 0, 5,42,24, 0 LTO-800G 00:00:20 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 15097750/15097750, blocks 116 header 139 Thread tape_thread 00:00:20 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 15097611/15097611 bytes Thread net_thread 00:00:20 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:20 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30077 block 116 block_size 0 bloc_loc 5646388 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 15097750 block_read_tot 0.148838758469 tape_rate 101436952.01 Thread tape_thread 00:00:20 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 15097750/15097750 bytes Thread tape_thread 00:00:20 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00155782699585 read 0.0735220909119 crc_check 0.0727732181549 push 0.000985622406006 block_read 0.148838758469 idle 0.00237512588501 Thread tape_thread 00:00:20 stkensrv4n.fnal.gov 005843 enstore I SL8500F1MC FINISHED listDrives returned ('ok', 0, None, '', '') 00:00:21 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:21 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30078 current 30077 116 Thread seek_thread 00:00:21 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:21 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=14822057 Thread tape_thread 00:00:21 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 14822196/14822196, blocks 114 header 139 Thread tape_thread 00:00:21 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 14822057/14822057 bytes Thread net_thread 00:00:21 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:21 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30078 block 114 block_size 0 bloc_loc 5646503 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 14822196 block_read_tot 0.148630619049 tape_rate 99725050.5638 Thread tape_thread 00:00:21 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 14822196/14822196 bytes Thread tape_thread 00:00:21 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00166940689087 read 0.0713322162628 crc_check 0.0744314193726 push 0.00119757652283 block_read 0.148630619049 idle 0.00231575965881 Thread tape_thread 00:00:21 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:22 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:22 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:22 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30079 current 30078 114 Thread seek_thread 00:00:22 stkensrv4n.fnal.gov 032104 enstore I SL8500G1MC query drive 0,5,1,14 => 0,0,5,1,14 online available IBM-LTO4 00:00:22 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:22 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=14211901 Thread tape_thread 00:00:22 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 14212040/14212040, blocks 109 header 139 Thread tape_thread 00:00:22 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 14211901/14211901 bytes Thread net_thread 00:00:22 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:22 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30079 block 109 block_size 0 bloc_loc 5646613 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 14212040 block_read_tot 0.142596244812 tape_rate 99666299.1984 Thread tape_thread 00:00:22 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 14212040/14212040 bytes Thread tape_thread 00:00:22 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00158739089966 read 0.0683917999268 crc_check 0.0715923309326 push 0.00102472305298 block_read 0.142596244812 idle 0.00223398208618 Thread tape_thread 00:00:23 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:23 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30080 current 30079 109 Thread seek_thread 00:00:23 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:23 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=3217837 Thread tape_thread 00:00:23 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV filemarks written. Tape VOV152 absolute location in blocks 432104 Thread tape_thread 00:00:23 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV write_tape timing:pull 0.0403928756714 write 3.14381575584 crc_check 2.45578956604 freespace 0.0344109535217 block_write 5.67440915108 idle 0.0672609806061 Thread tape_thread 00:00:23 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV drive stats after write. Tape VOV152 position 247 block 0 block_size 0 bloc_loc 432104 tot_blocks 762939000 BOT 0 read_err 0 write_err 6 bytes 539908608 block_write_tot 5.67440915108 tape_rate 95147986.9754 Thread tape_thread 00:00:23 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:23 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV remaining bytes info in DB 722384896000 estimated 721844987392 reported from drive 721863680000 position 250 Thread tape_thread 00:00:23 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV new bitfile request {'uid': 0, 'external_label': 'VOV152', 'gid': 0, 'location_cookie': '0000_000000000_0000248', 'complete_crc': 2940749554L, 'sanity_cookie': (65536L, 4107904059L), 'size': 539908500L} Thread tape_thread 00:00:23 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 3217976/3217976, blocks 25 header 139 Thread tape_thread 00:00:23 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 3217837/3217837 bytes Thread net_thread 00:00:23 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:23 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30080 block 25 block_size 0 bloc_loc 5646639 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 3217976 block_read_tot 0.0603437423706 tape_rate 53327418.4461 Thread tape_thread 00:00:23 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 3217976/3217976 bytes Thread tape_thread 00:00:23 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00300860404968 read 0.0418283939362 crc_check 0.0152959823608 push 0.000210762023926 block_read 0.0603437423706 idle 0.000523805618286 Thread tape_thread 00:00:23 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV set remaining: VOV152 721863680000 0000_000000000_0000250 Thread tape_thread 00:00:23 last message repeated 1 times 00:00:23 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-07 00:00:23', eod_cookie = '0000_000000000_0000250', non_del_files = '83', remaining_bytes = 721863680000 where label = 'VOV152'; 00:00:23 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV transfer complete volume=VOV152 location=250 Thread tape_thread 00:00:23 last message repeated 1 times 00:00:23 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '83', modification_time = '2009-11-07 00:00:23' where label = 'VOV152'; 00:00:23 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'ktev', 'blocksize': 131072, 'declared': 1254165880.0, 'library': 'CD-LTO4F1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOV152', 'wrapper': 'cern', 'remaining_bytes': 722384896000L, 'sum_mounts': 1, 'sum_rd_access': 0, 'file_family_width': 1, 'last_access': 1257573603.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000247', 'capacity_bytes': 858993459200L, 'non_del_files': 82, 'sum_wr_err': 0, 'sum_wr_access': 82, 'file_family': 'ktev_migration-MIGRATION', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'modification_time': 1257573603.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257571652.0}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NQ/NQMM/NQMM04/f0', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 539908500L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4F1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7875, 'mc_device': '1,1,1,11', 'compression': 0, 'serial_num': '1310051338', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr215a', 'max_time_in_state': 1200, 'device': 'stkenmvr215a:/dev/rmt/tps0d0n', 'name': 'LTO4_115.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_115.stat', 'mover_address': ('131.225.13.111', 7875), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.111', 39675), 'data_ip': 'stkenmvr215a', 'logname': 'LTO4_115MV', 'hostip': '131.225.13.111', 'do_cleaning': 'No', 'media_changer': 'SL8500F1.media_changer'}, 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM04/f0', 'uid': 0, 'uname': 'root', 'mtime': 1257573541, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cern', 'pstat': (16877, 675903776L, 20L, 1, 0, 0, 512L, 1257573107, 1257573107, 1257572647), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 539908500L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000398', 'inode': 675905424L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000398', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NQ/NQMM/NQMM04/.(access)(002800000000000008497F90)', 'fc': {'uid': 0, 'external_label': 'VOV152', 'gid': 0, 'location_cookie': '0000_000000000_0000248', 'complete_crc': 2940749554L, 'bfid': 'CDMS125757362300000', 'sanity_cookie': (65536L, 4107904059L), 'size': 539908500L}, 'infilepath': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000398', 'at_the_top': 27, 'local_inode': 39503211L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40502), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257573541.0217879, 'seek_time': 0.03514409065246582, 'transfer_time': 20.179939985275269, 't0': 1257573541, 'mount_time': 0.045815944671630859, 'lm_dequeued': 1257573603.620652, 'job_queued': 1257573541.2625999, 'drive_call_time': 3.1043074131011963, 'in_queue': 62.35779595375061, 'drive_transfer_time': 11.818505525588989}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573541-642-428'} Thread tape_thread 00:00:23 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV Done is sent Thread tape_thread 00:00:23 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV maybe_clean Thread tape_thread 00:00:23 stkendm7a.fnal.gov 000642 root I ENCP Received final dialog for stkendm7a.fnal.gov-1257573541-642-428. elapsed=5403.640sec Thread WRITE 00:00:23 stkendm7a.fnal.gov 000642 root I ENCP leaving transfer_file(). Thread WRITE 00:00:23 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV returned from maybe_clean Thread tape_thread 00:00:24 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM04/f0', drive = 'stkenmvr215a:/dev/rmt/tps0d0n:1310051338', pnfs_id = '002800000000000008497F90' where bfid = 'CDMS125757362300000'; 00:00:24 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:24 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM write_to_hsm next work on vol=VOV152 mover=LTO4_115.mover requester:root 00:00:24 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM HAVE_BOUND:sending write_to_hsm /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZN0/NZN088/f3 to mover LTO4_115.mover ('131.225.13.111', 7875) DEL_DISM 2 00:00:24 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : ktev 00:00:24 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM HAVE_BOUND:sending {'vc': {'storage_group': 'ktev', 'external_label': 'VOV152', 'library': 'CD-LTO4F1', 'file_family': 'ktev_migration-MIGRATION', 'wrapper': 'cern', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'file_family_width': 1}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NZ/NZN0/NZN088/f3', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 1259823748L, 'mover': 'LTO4_115.mover', 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZN0/NZN088/f3', 'uid': 0, 'uname': 'root', 'type': 'cern', 'gid': 0, 'mtime': 1257573558, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (16877, 684747552L, 20L, 1, 0, 0, 512L, 1255843636, 1255843636, 1255251459), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 1259823748L, 'fullname': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000143', 'inode': 675905472L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000143', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NZ/NZN0/NZN088/.(access)(002800000000000008497FC0)', 'fc': {'size': 1259823748L, 'external_label': 'VOV152', 'pnfsid': '002800000000000008497FC0', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000143', 'at_the_top': 99, 'local_inode': 31244313L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40508), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573624.656059, 'encp_start_time': 1257573558.4021201, 'job_queued': 1257573558.690475, 't0': 1257573558, 'in_queue': 65.965303897857666}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573558-4556-139'} to mover 00:00:24 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:24 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV WRITE_TO_HSM Thread MainThread 00:00:24 stkendm7a.fnal.gov 004556 root I ENCP Listening for control socket at: ('131.225.13.19', 40508) Thread WRITE_2 00:00:24 stkendm7a.fnal.gov 004556 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.19', 40508) elapsed=3885.020sec Thread WRITE_2 00:00:24 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV SENDING {'vc': {'storage_group': 'ktev', 'external_label': 'VOV152', 'library': 'CD-LTO4F1', 'file_family': 'ktev_migration-MIGRATION', 'wrapper': 'cern', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'file_family_width': 1}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NZ/NZN0/NZN088/f3', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 1259823748L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4F1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7875, 'mc_device': '1,1,1,11', 'compression': 0, 'serial_num': '1310051338', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr215a', 'max_time_in_state': 1200, 'device': 'stkenmvr215a:/dev/rmt/tps0d0n', 'name': 'LTO4_115.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_115.stat', 'mover_address': ('131.225.13.111', 7875), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.111', 39677), 'data_ip': 'stkenmvr215a', 'logname': 'LTO4_115MV', 'hostip': '131.225.13.111', 'do_cleaning': 'No', 'media_changer': 'SL8500F1.media_changer'}, 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZN0/NZN088/f3', 'uid': 0, 'uname': 'root', 'mtime': 1257573558, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cern', 'pstat': (16877, 684747552L, 20L, 1, 0, 0, 512L, 1255843636, 1255843636, 1255251459), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 1259823748L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000143', 'inode': 675905472L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000143', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NZ/NZN0/NZN088/.(access)(002800000000000008497FC0)', 'fc': {'size': 1259823748L, 'external_label': 'VOV152', 'pnfsid': '002800000000000008497FC0', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000143', 'at_the_top': 99, 'local_inode': 31244313L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40508), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573624.656059, 'encp_start_time': 1257573558.4021201, 'job_queued': 1257573558.690475, 't0': 1257573558, 'in_queue': 65.965303897857666}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573558-4556-139'} Thread client_connect_thread 00:00:24 stkendm7a.fnal.gov 004556 root I ENCP Control socket ('131.225.13.19', 40508) is connected to ('131.225.13.111', 39678) for stkendm7a.fnal.gov-1257573558-4556-139. elapsed=3885.020sec Thread WRITE_2 00:00:24 stkendm7a.fnal.gov 004556 root I ENCP Received callback ticket from mover LTO4_115.mover for transfer stkendm7a.fnal.gov-1257573558-4556-139. elapsed=3885.030sec Thread WRITE_2 00:00:24 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:24 stkendm7a.fnal.gov 004556 root I ENCP Attempting to connect data socket to mover at ('131.225.13.111', 39677). Thread WRITE_2 00:00:24 stkendm7a.fnal.gov 004556 root I ENCP Connecting data socket to mover (('131.225.13.111', 39677)) with interface 131.225.13.19. elapsed=3885.030sec Thread WRITE_2 00:00:24 stkendm7a.fnal.gov 004556 root I ENCP Data socket ('131.225.13.19', 40522) is connected to ('131.225.13.111', 39677). elapsed=3885.030sec Thread WRITE_2 00:00:24 stkendm7a.fnal.gov 004556 root I ENCP Opening local file. Thread WRITE_2 00:00:24 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30081 current 30080 25 Thread seek_thread 00:00:24 stkendm7a.fnal.gov 004556 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False Thread WRITE_2 00:00:24 stkendm7a.fnal.gov 000642 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /data/data1/Migration_Spool/VO6043:0000_000000000_0000398 -> /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NQ/NQMM/NQMM04/f0: 539908500 bytes copied to VOV152 at 25.5 MB/S (79.8 MB/S network) (43.6 MB/S drive) (45.5 MB/S disk) mover=LTO4_115.mover drive_id=ULTRIUM-TD4 drive_sn=1310051338 drive_vendor=IBM elapsed=83.74 {'storage_group': 'ktev', 'encp_crc': 2940749554L, 'disk_rate': 45.541602936277933, 'drive_rate': 43.567003335962255, 'driver': 'FTTDriver', 'transfer_rate': 25.515374750221515, 'mover_interface': 'stkenmvr215a', 'overall_rate': 24.40870735618665, 'media_changer': 'SL8500F1.media_changer', 'network_rate': 25.515374750221515, 'encp_ip': '131.225.13.19', 'unique_id': 'stkendm7a.fnal.gov-1257573541-642-428'} Thread WRITE 00:00:24 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:24 stkendm7a.fnal.gov 000642 root I ENCP File /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NQ/NQMM/NQMM04/f0 copied successfully. Thread WRITE 00:00:24 stkendm7a.fnal.gov 000642 root I ENCP Completed transferring 539908500 bytes in 1 files in 83.7424380779 sec. Overall rate = 24.4 MB/sec. Transfer rate = 25.5 MB/sec. Network rate = 79.8 MB/sec. Drive rate = 43.6 MB/sec. Disk rate = 45.5 MB/sec. Exit status = 0. Thread WRITE 00:00:24 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=54782958 Thread tape_thread 00:00:24 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV tape position: target 250 current 250 0 Thread seek_thread 00:00:24 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV write_tape starting, bytes_to_write=1259824128 Thread tape_thread 00:00:24 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV Write starting. Tape VOV152 absolute location in blocks 432104 Thread tape_thread 00:00:24 stkensrv0n.fnal.gov 028796 enstore I FILSRV start modifying {'pnfsid': '002800000000000008497F90', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM04/f0', 'deleted': 'no', 'external_label': 'VOV152', 'drive': 'stkenmvr215a:/dev/rmt/tps0d0n:1310051338', 'update': '2009-11-07 00:00:24.103413', 'gid': 0, 'location_cookie': '0000_000000000_0000248', 'complete_crc': 2940749554L, 'uid': 0, 'bfid': 'CDMS125757362300000', 'sanity_cookie': (65536L, 4107904059L), 'size': 539908500L} 00:00:24 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set pnfs_id = '001F000000000000007E7278' where bfid = 'CDMS125757362300000'; 00:00:24 stkensrv0n.fnal.gov 028796 enstore I FILSRV modified to {'pnfsid': '001F000000000000007E7278', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM04/f0', 'deleted': 'no', 'external_label': 'VOV152', 'drive': 'stkenmvr215a:/dev/rmt/tps0d0n:1310051338', 'update': '2009-11-07 00:00:24.103413', 'gid': 0, 'location_cookie': '0000_000000000_0000248', 'complete_crc': 2940749554L, 'uid': 0, 'bfid': 'CDMS125757362300000', 'sanity_cookie': (65536L, 4107904059L), 'size': 539908500L} 00:00:25 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 54783097/54783097, blocks 418 header 139 Thread tape_thread 00:00:25 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 54782958/54782958 bytes Thread net_thread 00:00:25 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:25 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30081 block 418 block_size 0 bloc_loc 5647058 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 54783097 block_read_tot 0.493680477142 tape_rate 110968732.888 Thread tape_thread 00:00:25 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 54783097/54783097 bytes Thread tape_thread 00:00:25 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00427198410034 read 0.209568977356 crc_check 0.274871826172 push 0.00496768951416 block_read 0.493680477142 idle 0.00852370262146 Thread tape_thread 00:00:25 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:25 stkendm7a.fnal.gov 000642 root I ENCP Version: v3_8 CVS $Revision: 1.963 $ encp.pyc OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4F1 Storage Group: ktev File Family: ktev_migration-MIGRATION FF Wrapper: cern FF Width: 1 Current working directory: stkendm7a.fnal.gov:/opt/enstore/src Command line: encp --delayed-dismount 2 --ignore-fair-share --threaded --priority 0 --storage-group ktev --file-family ktev_migration-MIGRATION --file-family-wrapper cern --library CD-LTO4F1 --override-path /pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM03/f31 --threaded /data/data1/Migration_Spool/VO6043:0000_000000000_0000407 /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NQ/NQMM/NQMM03/f31 Thread WRITE 00:00:25 stkendca20a.fnal.gov 022750 root I ENCP File /diskb/write-pool-2/data/003000000000000006222988 transfered for stkendca20a.fnal.gov-1257573536-22750-0 in 48.5757350922 seconds. 00:00:25 stkendm7a.fnal.gov 000642 root I ENCP Creating zero length output files. Thread WRITE 00:00:26 stkendm7a.fnal.gov 000642 root I ENCP Sending /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NQ/NQMM/NQMM03/f31 write request to LM: unique_id: stkendm7a.fnal.gov-1257573625-642-431 inputfile: /data/data1/Migration_Spool/VO6043:0000_000000000_0000407 outputfile: /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NQ/NQMM/NQMM03/.(access)(0028000000000000084980F8) Thread WRITE 00:00:26 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : ktev 00:00:26 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM write Q'd /data/data1/Migration_Spool/VO6043:0000_000000000_0000407 (stkendm7a.fnal.gov-1257573625-642-431) -> /pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM03/f31 : library=CD-LTO4F1 family=ktev_migration-MIGRATION requester:root volume_family:ktev.ktev_migration-MIGRATION.cern 00:00:26 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:26 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30082 current 30081 418 Thread seek_thread 00:00:26 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:26 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=25587016 Thread tape_thread 00:00:26 stkensrv0n.fnal.gov 023312 enstore I INFSRV stop listing all volumes (3) 00:00:26 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 25587155/25587155, blocks 196 header 139 Thread tape_thread 00:00:26 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 25587016/25587016 bytes Thread net_thread 00:00:26 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:26 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30082 block 196 block_size 0 bloc_loc 5647255 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 25587155 block_read_tot 0.243418216705 tape_rate 105116023.551 Thread tape_thread 00:00:26 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 25587155/25587155 bytes Thread tape_thread 00:00:26 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00226736068726 read 0.110061883926 crc_check 0.128970146179 push 0.00211882591248 block_read 0.243418216705 idle 0.00398254394531 Thread tape_thread 00:00:27 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV written bytes 1966080512/1966080512, blocks 15001 header 189 trailer 323 Thread tape_thread 00:00:27 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:27 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:27 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30083 current 30082 196 Thread seek_thread 00:00:27 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:27 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=10983325 Thread tape_thread 00:00:27 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 10983464/10983464, blocks 84 header 139 Thread tape_thread 00:00:27 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 10983325/10983325 bytes Thread net_thread 00:00:27 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:27 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30083 block 84 block_size 0 bloc_loc 5647340 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 10983464 block_read_tot 0.119772434235 tape_rate 91702770.0922 Thread tape_thread 00:00:27 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 10983464/10983464 bytes Thread tape_thread 00:00:27 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.000928163528442 read 0.0626754760742 crc_check 0.0554776191711 push 0.000691175460815 block_read 0.119772434235 idle 0.00173139572144 Thread tape_thread 00:00:28 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:00:28 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:28 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:28 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30084 current 30083 84 Thread seek_thread 00:00:28 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:28 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=14912409 Thread tape_thread 00:00:29 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV filemarks written. Tape VON279 absolute location in blocks 1191745 Thread tape_thread 00:00:29 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV write_tape timing:pull 0.210045099258 write 6.27891564369 crc_check 9.07596683502 freespace 0.157977819443 block_write 15.7229053974 idle 0.274476528168 Thread tape_thread 00:00:29 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV drive stats after write. Tape VON279 position 84 block 0 block_size 0 bloc_loc 1191745 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 1966080512 block_write_tot 15.7229053974 tape_rate 125045623.713 Thread tape_thread 00:00:29 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV remaining bytes info in DB 634706944000 estimated 632740863488 reported from drive 632858624000 position 85 Thread tape_thread 00:00:29 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV new bitfile request {'uid': 0, 'external_label': 'VON279', 'gid': 0, 'location_cookie': '0000_000000000_0000084', 'complete_crc': 3782180365L, 'sanity_cookie': (65536L, 305786856L), 'size': 1966080000L} Thread tape_thread 00:00:29 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 14912548/14912548, blocks 114 header 139 Thread tape_thread 00:00:29 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 14912409/14912409 bytes Thread net_thread 00:00:29 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:29 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30084 block 114 block_size 0 bloc_loc 5647455 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 14912548 block_read_tot 0.150928974152 tape_rate 98805070.9536 Thread tape_thread 00:00:29 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 14912548/14912548 bytes Thread tape_thread 00:00:29 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00179553031921 read 0.0729789733887 crc_check 0.0749101638794 push 0.00124430656433 block_read 0.150928974152 idle 0.00259041786194 Thread tape_thread 00:00:29 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV set remaining: VON279 632858624000 0000_000000000_0000085 Thread tape_thread 00:00:29 last message repeated 1 times 00:00:29 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-07 00:00:29', eod_cookie = '0000_000000000_0000085', non_del_files = '84', remaining_bytes = 632858624000 where label = 'VON279'; 00:00:29 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV transfer complete volume=VON279 location=85 Thread tape_thread 00:00:29 last message repeated 1 times 00:00:29 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '84', modification_time = '2009-11-07 00:00:29' where label = 'VON279'; 00:00:29 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'exp-db', 'blocksize': 131072, 'declared': 1248994100.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 1243106842.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VON279', 'wrapper': 'cpio_odc', 'remaining_bytes': 634706944000L, 'sum_mounts': 46, 'sum_rd_access': 0, 'file_family_width': 2, 'last_access': 1257573576.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000084', 'capacity_bytes': 858993459200L, 'non_del_files': 83, 'sum_wr_err': 0, 'sum_wr_access': 83, 'file_family': 'daily-d0-offline', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'modification_time': 1257573576.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257567920.0}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel01/D0OFPRD1_t702243578_s103343_p16', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7868, 'mc_device': '0,5,1,11', 'compression': 0, 'serial_num': '1310154353', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr208a', 'max_time_in_state': 1200, 'device': 'stkenmvr208a:/dev/rmt/tps0d0n', 'name': 'LTO4_108.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_108.stat', 'mover_address': ('131.225.13.204', 7868), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.204', 49952), 'data_ip': 'stkenmvr208a', 'logname': 'LTO4_108MV', 'hostip': '131.225.13.204', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel01/D0OFPRD1_t702243578_s103343_p16', 'uid': 0, 'uname': 'root', 'mtime': 1257573536, 'rmajor': 0, 'machine': ('Linux', 'stkendca20a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 820620024L, 20L, 1, 1602, 2752, 512L, 1257573497, 1257573497, 1254398419), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 1966080000L, 'gname': 'root', 'gid': 0, 'fullname': '/diskb/write-pool-2/data/003000000000000006222988', 'inode': 807545224L, 'minor': 20}, 'version': 'v3_8b CVS $Revision: 1.922 $ ', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskb/write-pool-2/data/003000000000000006222988', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel01/.(access)(003000000000000006222988)', 'fc': {'uid': 0, 'external_label': 'VON279', 'gid': 0, 'location_cookie': '0000_000000000_0000084', 'complete_crc': 3782180365L, 'bfid': 'CDMS125757362900000', 'sanity_cookie': (65536L, 305786856L), 'size': 1966080000L}, 'infilepath': '/diskb/write-pool-2/data/003000000000000006222988', 'at_the_top': 622, 'local_inode': 271282496L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.12', 49609), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257573534.7440391, 'seek_time': 0.029278039932250977, 'transfer_time': 52.15677285194397, 't0': 1257573534, 'mount_time': 0.014158964157104492, 'lm_dequeued': 1257573577.146349, 'job_queued': 1257573536.870615, 'drive_call_time': 6.0740351676940918, 'in_queue': 40.275671005249023, 'drive_transfer_time': 17.737414836883545}, 'user_level': 'user2', 'unique_id': 'stkendca20a.fnal.gov-1257(trunc. 4096) Thread tape_thread 00:00:29 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV Done is sent Thread tape_thread 00:00:29 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV maybe_clean Thread tape_thread 00:00:29 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV returned from maybe_clean Thread tape_thread 00:00:29 stkendca20a.fnal.gov 022750 root I ENCP Received final dialog for stkendca20a.fnal.gov-1257573536-22750-0. elapsed=94.710sec 00:00:29 stkendca20a.fnal.gov 022750 root I ENCP leaving transfer_file(). 00:00:29 stkendca20a.fnal.gov 022750 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:29 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:29 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel01/D0OFPRD1_t702243578_s103343_p16', drive = 'stkenmvr208a:/dev/rmt/tps0d0n:1310154353', pnfs_id = '003000000000000006222988' where bfid = 'CDMS125757362900000'; 00:00:29 stkendca20a.fnal.gov 022750 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /diskb/write-pool-2/data/003000000000000006222988 -> /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel01/D0OFPRD1_t702243578_s103343_p16: 1966080000 bytes copied to VON279 at 35.9 MB/S (57.6 MB/S network) (106 MB/S drive) (38.6 MB/S disk) mover=LTO4_108.mover drive_id=ULTRIUM-TD4 drive_sn=1310154353 drive_vendor=IBM elapsed=95.096 {'storage_group': 'exp-db', 'encp_crc': 3782180365L, 'disk_rate': 38.639323736448368, 'drive_rate': 105.7087527829076, 'driver': 'FTTDriver', 'transfer_rate': 35.933342213649546, 'mover_interface': 'stkenmvr208a', 'overall_rate': 35.617739254816904, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 35.933342213649546, 'encp_ip': '131.225.13.12', 'unique_id': 'stkendca20a.fnal.gov-1257573536-22750-0'} 00:00:29 stkendca20a.fnal.gov 022750 root I ENCP File /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel01/D0OFPRD1_t702243578_s103343_p16 copied successfully. 00:00:29 stkendca20a.fnal.gov 022750 root I ENCP Completed transferring 1966080000 bytes in 1 files in 95.1158719063 sec. Overall rate = 35.6 MB/sec. Transfer rate = 35.9 MB/sec. Network rate = 57.6 MB/sec. Drive rate = 106 MB/sec. Disk rate = 38.6 MB/sec. Exit status = 0. 00:00:30 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:30 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30085 current 30084 114 Thread seek_thread 00:00:30 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:30 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:30 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=5551259 Thread tape_thread 00:00:30 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VON279 mover=LTO4_108.mover requester:root 00:00:30 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel01/D0OFPRD1_t702245888_s103346_p1 to mover LTO4_108.mover ('131.225.13.204', 7868) DEL_DISM None 00:00:30 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : exp-db 00:00:30 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'exp-db', 'external_label': 'VON279', 'library': 'CD-LTO4G1', 'file_family': 'daily-d0-offline', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'file_family_width': 2}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel01/D0OFPRD1_t702245888_s103346_p1', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': 'LTO4_108.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel01/D0OFPRD1_t702245888_s103346_p1', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257573613, 'rmajor': 0, 'machine': ('Linux', 'stkendca20a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (16877, 820620024L, 20L, 1, 1602, 2752, 512L, 1257573594, 1257573594, 1254398419), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 1966080000L, 'fullname': '/diskc/write-pool-3/data/003000000000000006222A10', 'inode': 807545360L, 'minor': 20}, 'version': 'v3_8b CVS $Revision: 1.922 $ ', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskc/write-pool-3/data/003000000000000006222A10', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel01/.(access)(003000000000000006222A10)', 'fc': {'size': 1966080000L, 'external_label': 'VON279', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diskc/write-pool-3/data/003000000000000006222A10', 'at_the_top': 270, 'local_inode': 270526672L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.12', 49618), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573630.2049761, 'encp_start_time': 1257573611.889703, 'job_queued': 1257573613.9431551, 't0': 1257573611, 'in_queue': 16.261560916900635}, 'user_level': 'user2', 'unique_id': 'stkendca20a.fnal.gov-1257573613-23640-0'} to mover 00:00:30 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:30 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV WRITE_TO_HSM Thread MainThread 00:00:30 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV SENDING {'vc': {'storage_group': 'exp-db', 'external_label': 'VON279', 'library': 'CD-LTO4G1', 'file_family': 'daily-d0-offline', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'file_family_width': 2}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel01/D0OFPRD1_t702245888_s103346_p1', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7868, 'mc_device': '0,5,1,11', 'compression': 0, 'serial_num': '1310154353', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr208a', 'max_time_in_state': 1200, 'device': 'stkenmvr208a:/dev/rmt/tps0d0n', 'name': 'LTO4_108.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_108.stat', 'mover_address': ('131.225.13.204', 7868), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.204', 49959), 'data_ip': 'stkenmvr208a', 'logname': 'LTO4_108MV', 'hostip': '131.225.13.204', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel01/D0OFPRD1_t702245888_s103346_p1', 'uid': 0, 'uname': 'root', 'mtime': 1257573613, 'rmajor': 0, 'machine': ('Linux', 'stkendca20a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 820620024L, 20L, 1, 1602, 2752, 512L, 1257573594, 1257573594, 1254398419), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 1966080000L, 'gname': 'root', 'gid': 0, 'fullname': '/diskc/write-pool-3/data/003000000000000006222A10', 'inode': 807545360L, 'minor': 20}, 'version': 'v3_8b CVS $Revision: 1.922 $ ', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskc/write-pool-3/data/003000000000000006222A10', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel01/.(access)(003000000000000006222A10)', 'fc': {'size': 1966080000L, 'external_label': 'VON279', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diskc/write-pool-3/data/003000000000000006222A10', 'at_the_top': 270, 'local_inode': 270526672L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.12', 49618), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573630.2049761, 'encp_start_time': 1257573611.889703, 'job_queued': 1257573613.9431551, 't0': 1257573611, 'in_queue': 16.261560916900635}, 'user_level': 'user2', 'unique_id': 'stkendca20a.fnal.gov-1257573613-23640-0'} Thread client_connect_thread 00:00:30 stkendca20a.fnal.gov 023640 root I ENCP Control socket ('131.225.13.12', 49618) is connected to ('131.225.13.204', 49960) for stkendca20a.fnal.gov-1257573613-23640-0. elapsed=18.370sec 00:00:30 stkendca20a.fnal.gov 023640 root I ENCP Received callback ticket from mover LTO4_108.mover for transfer stkendca20a.fnal.gov-1257573613-23640-0. elapsed=18.370sec 00:00:30 stkendca20a.fnal.gov 023640 root I ENCP Atempting to connect data socket to mover at ('131.225.13.204', 49959). 00:00:30 stkendca20a.fnal.gov 023640 root I ENCP Connecting data socket to mover (('131.225.13.204', 49959)) with interface 131.225.13.12. elapsed=18.370sec 00:00:30 stkendca20a.fnal.gov 023640 root I ENCP Data socket ('131.225.13.12', 49620) is connected to ('131.225.13.204', 49959). elapsed=18.370sec 00:00:30 stkendca20a.fnal.gov 023640 root I ENCP Opening local file. 00:00:30 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 5551398/5551398, blocks 43 header 139 Thread tape_thread 00:00:30 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 5551259/5551259 bytes Thread net_thread 00:00:30 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:30 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30085 block 43 block_size 0 bloc_loc 5647499 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 5551398 block_read_tot 0.0673034191132 tape_rate 82483149.7887 Thread tape_thread 00:00:30 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 5551398/5551398 bytes Thread tape_thread 00:00:30 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00130105018616 read 0.0382635593414 crc_check 0.0273756980896 push 0.000363111495972 block_read 0.0673034191132 idle 0.00113344192505 Thread tape_thread 00:00:30 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV tape position: target 85 current 85 0 Thread seek_thread 00:00:30 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV write_tape starting, bytes_to_write=1966080512 Thread tape_thread 00:00:30 stkendca20a.fnal.gov 023640 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:00:30 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV Write starting. Tape VON279 absolute location in blocks 1191745 Thread tape_thread 00:00:31 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:31 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:31 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30086 current 30085 43 Thread seek_thread 00:00:31 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:31 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=19982103 Thread tape_thread 00:00:31 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 19982242/19982242, blocks 153 header 139 Thread tape_thread 00:00:31 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 19982103/19982103 bytes Thread net_thread 00:00:31 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:31 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30086 block 153 block_size 0 bloc_loc 5647653 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 19982242 block_read_tot 0.194000244141 tape_rate 103001117.8 Thread tape_thread 00:00:31 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 19982242/19982242 bytes Thread tape_thread 00:00:31 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00161409378052 read 0.0926508903503 crc_check 0.0984027385712 push 0.0013325214386 block_read 0.194000244141 idle 0.00399136543274 Thread tape_thread 00:00:31 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:32 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:32 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30087 current 30086 153 Thread seek_thread 00:00:32 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:32 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=55031169 Thread tape_thread 00:00:33 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 55031308/55031308, blocks 420 header 139 Thread tape_thread 00:00:33 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 55031169/55031169 bytes Thread net_thread 00:00:33 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:33 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30087 block 420 block_size 0 bloc_loc 5648074 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 55031308 block_read_tot 0.49690079689 tape_rate 110749083.81 Thread tape_thread 00:00:33 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 55031308/55031308 bytes Thread tape_thread 00:00:33 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00569009780884 read 0.216424703598 crc_check 0.271153211594 push 0.00363278388977 block_read 0.49690079689 idle 0.00989365577698 Thread tape_thread 00:00:33 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV read bytes 14999790496/14999790496, blocks 114440 header 0 Thread tape_thread 00:00:33 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV write_client: wrote 14999790496/14999790496 bytes Thread net_thread 00:00:33 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV write_client waits for tape thread to exit Thread net_thread 00:00:33 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV drive stats after read. Tape VOV146 position 380 block 114440 block_size 0 bloc_loc 3429987 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 14999790496 block_read_tot 138.381920099 tape_rate 108394149.216 Thread tape_thread 00:00:33 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV read_tape exiting, read 14999790496/14999790496 bytes Thread tape_thread 00:00:33 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV read_tape timing:get_space 1.05318164825 read 65.5892543793 crc_check 70.7524151802 push 0.987068891525 block_read 138.381920099 idle 2.57185864449 Thread tape_thread 00:00:33 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:34 stkenmvr109a.fnal.gov 031352 root I LTO09MV read bytes 1784268599/1784268599, blocks 13613 header 232 Thread tape_thread 00:00:34 stkenmvr109a.fnal.gov 031352 root I LTO09MV write_client: wrote 1784268367/1784268367 bytes Thread net_thread 00:00:34 stkenmvr109a.fnal.gov 031352 root I LTO09MV write_client waits for tape thread to exit Thread net_thread 00:00:34 stkensrv3n.fnal.gov 006919 root I ENCP File /pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/C09BA8A1-387F-DC11-B283-001731A28999.root transfered for stkensrv3n.fnal.gov-1257573434-6919-0 in 21.9917960167 seconds. 00:00:34 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:34 stkenmvr109a.fnal.gov 031352 root I LTO09MV drive stats after read. Tape VOF410 position 241 block 13613 block_size 0 bloc_loc 2983569 tot_blocks 381469000 BOT 0 read_err 0 write_err 0 bytes 1784268599 block_read_tot 20.4790332317 tape_rate 87126603.0388 Thread tape_thread 00:00:34 stkenmvr109a.fnal.gov 031352 root I LTO09MV read_tape exiting, read 1784268599/1784268599 bytes Thread tape_thread 00:00:34 stkenmvr109a.fnal.gov 031352 root I LTO09MV read_tape timing:get_space 0.171549320221 read 13.3052046299 crc_check 6.77130866051 push 0.230970621109 block_read 20.4790332317 idle 0.520879745483 Thread tape_thread 00:00:34 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30088 current 30087 420 Thread seek_thread 00:00:34 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:34 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=25443306 Thread tape_thread 00:00:34 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 25443445/25443445, blocks 195 header 139 Thread tape_thread 00:00:34 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 25443306/25443306 bytes Thread net_thread 00:00:34 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:34 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30088 block 195 block_size 0 bloc_loc 5648270 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 25443445 block_read_tot 0.240800857544 tape_rate 105661770.724 Thread tape_thread 00:00:34 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 25443445/25443445 bytes Thread tape_thread 00:00:34 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00287294387817 read 0.110894203186 crc_check 0.125368833542 push 0.00166487693787 block_read 0.240800857544 idle 0.00437521934509 Thread tape_thread 00:00:34 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV write_client exits Thread net_thread 00:00:34 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV tape position: target 383 current 380 114440 Thread seek_thread 00:00:34 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV skip_fm 3 Thread seek_thread 00:00:34 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV read_tape starting, bytes_to_read=6233969476 Thread tape_thread 00:00:35 stkenmvr109a.fnal.gov 031352 root I LTO09MV write_client exits Thread net_thread 00:00:35 stkenmvr109a.fnal.gov 031352 root I LTO09MV transfer complete volume=VOF410 location=241 Thread net_thread 00:00:35 last message repeated 1 times 00:00:35 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_rd_access = '25', last_access = '2009-11-07 00:00:35' where label = 'VOF410'; 00:00:35 stkenmvr109a.fnal.gov 031352 root I LTO09MV Sending done to client: {'vc': {'comment': '', 'declared': 1190942903.0, 'blocksize': 131072, 'storage_group': 'cms', 'library': 'CD-LTO3', 'eod_cookie': '0000_000000000_0000258', 'current_location': '0000_000000000_0000241', 'user_inhibit': ['none', 'none'], 'sum_rd_access': 24, 'system_inhibit': ['none', 'none'], 'external_label': 'VOF410', 'wrapper': 'cpio_odc', 'remaining_bytes': 1097543680L, 'sum_mounts': 121, 'capacity_bytes': 429496729600L, 'media_type': 'LTO3', 'last_access': 1247963682.0, 'status': ('ok', None), 'si_time': [0.0, 1192921340.0], 'non_del_files': 257, 'sum_wr_err': 0, 'sum_wr_access': 256, 'file_family': 'CSA07-Skims', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.CSA07-Skims.cpio_odc', 'modification_time': 1200327609.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1192731952.0}, 'outfilepath': '/dev/null', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': 1, 'curpri': 1}, 'file_size': 1784268367L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 300, 'max_failures': 5, 'library': ['CD-LTO3.library_manager', 'CD-LTO3_test.library_manager'], 'local_mover': 0, 'dismount_delay': 20, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7809, 'mc_device': '0,0,1,14', 'compression': 0, 'serial_num': '9210007592', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 57671680.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr109a', 'max_time_in_state': 1200, 'device': 'stkenmvr109a:/dev/rmt/tps0d0n', 'name': 'LTO3_09.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD3', 'statistics_path': '/tmp/enstore/enstore/LTO3_09.stat', 'mover_address': ('131.225.13.139', 7809), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.139', 46897), 'data_ip': 'stkenmvr109a', 'logname': 'LTO09MV', 'hostip': '131.225.13.139', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'lm': {'address': ('131.225.13.4', 7506)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/C09BA8A1-387F-DC11-B283-001731A28999.root', 'uid': 0, 'uname': 'root', 'rmajor': 0, 'machine': ('Linux', 'stkensrv3n.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 08:50:41 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (33188, 125223176L, 20L, 1, 9811, 5063, 1784268367L, 1192916973, 1192916973, 1192916735), 'mode_octal': '0100744', 'mode': 33252, 'size_bytes': 1784268367L, 'gname': 'root', 'gid': 0, 'fullname': '/dev/null', 'inode': 125223176L, 'minor': 20}, 'version': 'v3_7d CVS $Revision: 1.929 $ encp', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/.(access)(00070000000000000976C108)', 'status': ('ok', None), 'volume': 'VOF410', 'outfile': '/dev/null', 'fc': {'status': ('ok', None), 'pnfsid': '00070000000000000976C108', 'pnfs_name0': '/pnfs/fnal.gov/usr/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/C09BA8A1-387F-DC11-B283-001731A28999.root', 'deleted': 'no', 'external_label': 'VOF410', 'drive': 'stkenmvr114a:/dev/rmt/tps2d0n:1210116659', 'update': '2007-10-20 16:49:31.648828', 'gid': 0, 'location_cookie': '0000_000000000_0000241', 'complete_crc': 2491397348L, 'address': ('131.225.13.59', 7501), 'uid': 0, 'bfid': 'CDMS119291696000000', 'sanity_cookie': (65536L, 3116902659L), 'size': 1784268367L}, 'infilepath': '/pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/C09BA8A1-387F-DC11-B283-001731A28999.root', 'at_the_top': 3, 'bfid': 'CDMS119291696000000', 'local_inode': 2469L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': 7, 'retry': 0}, 'callback_addr': ('131.225.13.51', 43470), 'work': 'read_from_hsm', 'override_ro_mount': 1, 'times': {'encp_start_time': 1257573433.488004, 'seek_time': 176.2519190(trunc. 4403) Thread net_thread 00:00:35 stkenmvr109a.fnal.gov 031352 root I LTO09MV Done is sent Thread net_thread 00:00:35 stkenmvr109a.fnal.gov 031352 root I LTO09MV maybe_clean Thread net_thread 00:00:35 stkensrv3n.fnal.gov 006919 root I ENCP Received final dialog for stkensrv3n.fnal.gov-1257573434-6919-0. elapsed=201.720sec 00:00:35 stkensrv3n.fnal.gov 006919 root I ENCP leaving transfer_file(). 00:00:35 stkenmvr109a.fnal.gov 031352 root I LTO09MV returned from maybe_clean Thread net_thread 00:00:35 stkensrv3n.fnal.gov 006919 root I ENCP MSG_TYPE=ENCP_XFER read_from_hsm /pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/C09BA8A1-387F-DC11-B283-001731A28999.root -> /dev/null: 1784268367 bytes copied from VOF410 at 73.9 MB/S (77.4 MB/S network) (82.2 MB/S drive) (77.4 MB/S disk) mover=LTO3_09.mover drive_id=ULTRIUM-TD3 drive_sn=9210007592 drive_vendor=IBM elapsed=201.73 {'storage_group': 'cms', 'encp_crc': 2491397348L, 'disk_rate': 77.375324927532986, 'drive_rate': 82.174113629130161, 'driver': 'FTTDriver', 'transfer_rate': 73.855534721502309, 'mover_interface': 'stkenmvr109a', 'overall_rate': 8.5321360558832637, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 73.855534721502309, 'encp_ip': '131.225.13.51', 'unique_id': 'stkensrv3n.fnal.gov-1257573434-6919-0'} 00:00:35 stkensrv3n.fnal.gov 006919 root I ENCP File /dev/null copied successfully. 00:00:35 stkensrv3n.fnal.gov 006919 root I ENCP Completed transferring 1784268367 bytes in 1 files in 201.740987062 sec. Overall rate = 8.53 MB/sec. Transfer rate = 73.9 MB/sec. Network rate = 77.4 MB/sec. Drive rate = 82.2 MB/sec. Disk rate = 77.4 MB/sec. Exit status = 0. 00:00:35 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:35 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30089 current 30088 195 Thread seek_thread 00:00:35 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:35 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=15059997 Thread tape_thread 00:00:35 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:35 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 15060136/15060136, blocks 115 header 139 Thread tape_thread 00:00:35 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 15059997/15059997 bytes Thread net_thread 00:00:35 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:35 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30089 block 115 block_size 0 bloc_loc 5648386 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 15060136 block_read_tot 0.153668165207 tape_rate 98004267.7006 Thread tape_thread 00:00:35 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 15060136/15060136 bytes Thread tape_thread 00:00:35 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00122809410095 read 0.0771894454956 crc_check 0.0742480754852 push 0.00100255012512 block_read 0.153668165207 idle 0.00272393226624 Thread tape_thread 00:00:35 stkensrv3n.fnal.gov 007327 root I ENCP Version: v3_7d CVS $Revision: 1.929 $ encp OS: Linux 2.6.9-89.0.9.ELsmp x86_64 Release: Scientific Linux SL release 4.4 (Beryllium), User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Current working directory: stkensrv3n.fnal.gov:/root Command line: /opt/enstore/bin/encp --verbose 10 --bypass-filesystem-max-filesize-check --max-resubmit 7 --delayed-dismount 1 --override-ro-mount --get-bfid CDMS119284576600000 /dev/null 00:00:36 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:00:36 stkensrv3n.fnal.gov 007327 root I ENCP Creating zero length output files. 00:00:36 stkensrv3n.fnal.gov 007327 root I ENCP Sending /pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/F815C1A7-907E-DC11-86AA-003048726DD9.root read request to LM: unique_id: stkensrv3n.fnal.gov-1257573636-7327-0 inputfile: /pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/.(access)(0007000000000000096FA700) outputfile: /dev/null 00:00:36 stkensrv4n.fnal.gov 005806 enstore I LTO3LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:00:36 stkensrv4n.fnal.gov 005806 enstore I LTO3LM read Q'd /pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/F815C1A7-907E-DC11-86AA-003048726DD9.root (stkensrv3n.fnal.gov-1257573636-7327-0) -> /dev/null : library=CD-LTO3 family=cms.CSA07-Skims.cpio_odc requester:root 00:00:36 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:36 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30090 current 30089 115 Thread seek_thread 00:00:36 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:36 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=10630141 Thread tape_thread 00:00:36 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 10630280/10630280, blocks 82 header 139 Thread tape_thread 00:00:36 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 10630141/10630141 bytes Thread net_thread 00:00:36 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:37 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30090 block 82 block_size 0 bloc_loc 5648469 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 10630280 block_read_tot 0.112038135529 tape_rate 94880907.736 Thread tape_thread 00:00:37 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 10630280/10630280 bytes Thread tape_thread 00:00:37 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00159692764282 read 0.0572369098663 crc_check 0.0524878501892 push 0.0007164478302 block_read 0.112038135529 idle 0.00192284584045 Thread tape_thread 00:00:37 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:37 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:38 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30091 current 30090 82 Thread seek_thread 00:00:38 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:38 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=14257212 Thread tape_thread 00:00:38 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:00:38 stkensrv4n.fnal.gov 005806 enstore I LTO3LM read_from_hsm next work on vol=VOF410 mover=LTO3_09.mover requester:root 00:00:38 stkensrv4n.fnal.gov 005806 enstore I LTO3LM HAVE_BOUND:sending read_from_hsm /pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/F815C1A7-907E-DC11-86AA-003048726DD9.root to mover LTO3_09.mover ('131.225.13.139', 7809) DEL_DISM 1 00:00:38 stkensrv4n.fnal.gov 005806 enstore I LTO3LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:00:38 stkensrv4n.fnal.gov 005806 enstore I LTO3LM HAVE_BOUND:sending {'vc': {'comment': '', 'declared': 1190942903.0, 'blocksize': 131072, 'storage_group': 'cms', 'library': 'CD-LTO3', 'eod_cookie': '0000_000000000_0000258', 'current_location': '0000_000000000_0000083', 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'full'], 'external_label': 'VOF410', 'wrapper': 'cpio_odc', 'remaining_bytes': 1097543680L, 'sum_mounts': 122, 'sum_rd_access': 25, 'media_type': 'LTO3', 'last_access': 1257573635.0, 'status': ('ok', None), 'si_time': [0.0, 1192921340.0], 'capacity_bytes': 429496729600L, 'non_del_files': 257, 'sum_wr_err': 0, 'sum_wr_access': 256, 'file_family': 'CSA07-Skims', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.CSA07-Skims.cpio_odc', 'modification_time': 1200327609.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1192731952.0}, 'outfilepath': '/dev/null', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': 1, 'curpri': 1}, 'file_size': 1498719455L, 'mover': 'LTO3_09.mover', 'lm': {'address': ('131.225.13.4', 7506)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/F815C1A7-907E-DC11-86AA-003048726DD9.root', 'uid': 0, 'uname': 'root', 'gid': 0, 'rmajor': 0, 'machine': ('Linux', 'stkensrv3n.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 08:50:41 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (33188, 124757760L, 20L, 1, 9811, 5063, 1498719455L, 1247963569, 1192845783, 1192845374), 'mode_octal': '0100744', 'mode': 33252, 'gname': 'root', 'size_bytes': 1498719455L, 'fullname': '/dev/null', 'inode': 124757760L, 'minor': 20}, 'version': 'v3_7d CVS $Revision: 1.929 $ encp', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/.(access)(0007000000000000096FA700)', 'status': ('ok', None), 'volume': 'VOF410', 'outfile': '/dev/null', 'fc': {'status': ('ok', None), 'pnfsid': '0007000000000000096FA700', 'pnfs_name0': '/pnfs/fnal.gov/usr/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/F815C1A7-907E-DC11-86AA-003048726DD9.root', 'deleted': 'no', 'external_label': 'VOF410', 'drive': 'stkenmvr106a:/dev/rmt/tps2d0n:1210116666', 'update': '2007-10-19 21:02:57.784073', 'gid': 0, 'location_cookie': '0000_000000000_0000083', 'complete_crc': 3596548126L, 'address': ('131.225.13.59', 7501), 'uid': 0, 'bfid': 'CDMS119284576600000', 'sanity_cookie': (65536L, 1854190849L), 'size': 1498719455L}, 'infilepath': '/pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/F815C1A7-907E-DC11-86AA-003048726DD9.root', 'at_the_top': 14, 'bfid': 'CDMS119284576600000', 'local_inode': 2469L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': 7, 'retry': 0}, 'callback_addr': ('131.225.13.51', 43476), 'work': 'read_from_hsm', 'override_ro_mount': 1, 'times': {'lm_dequeued': 1257573638.1305771, 'encp_start_time': 1257573635.401355, 'job_queued': 1257573636.5777709, 't0': 1257573635, 'in_queue': 1.5527629852294922}, 'user_level': 'admin', 'unique_id': 'stkensrv3n.fnal.gov-1257573636-7327-0'} to mover 00:00:38 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:00:38 stkenmvr109a.fnal.gov 031352 root I LTO09MV READ FROM HSM Thread MainThread 00:00:38 stkensrv3n.fnal.gov 007327 root I ENCP Listening for control socket at: ('131.225.13.51', 43476) 00:00:38 stkensrv3n.fnal.gov 007327 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.51', 43476) elapsed=2.760sec 00:00:38 stkenmvr109a.fnal.gov 031352 root I LTO09MV SENDING {'vc': {'comment': '', 'declared': 1190942903.0, 'blocksize': 131072, 'storage_group': 'cms', 'library': 'CD-LTO3', 'eod_cookie': '0000_000000000_0000258', 'current_location': '0000_000000000_0000083', 'user_inhibit': ['none', 'none'], 'sum_rd_access': 25, 'system_inhibit': ['none', 'full'], 'external_label': 'VOF410', 'wrapper': 'cpio_odc', 'remaining_bytes': 1097543680L, 'sum_mounts': 122, 'capacity_bytes': 429496729600L, 'media_type': 'LTO3', 'last_access': 1257573635.0, 'status': ('ok', None), 'si_time': [0.0, 1192921340.0], 'non_del_files': 257, 'sum_wr_err': 0, 'sum_wr_access': 256, 'file_family': 'CSA07-Skims', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.CSA07-Skims.cpio_odc', 'modification_time': 1200327609.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1192731952.0}, 'outfilepath': '/dev/null', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': 1, 'curpri': 1}, 'file_size': 1498719455L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 300, 'max_failures': 5, 'library': ['CD-LTO3.library_manager', 'CD-LTO3_test.library_manager'], 'local_mover': 0, 'dismount_delay': 20, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7809, 'mc_device': '0,0,1,14', 'compression': 0, 'serial_num': '9210007592', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 57671680.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr109a', 'max_time_in_state': 1200, 'device': 'stkenmvr109a:/dev/rmt/tps0d0n', 'name': 'LTO3_09.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD3', 'statistics_path': '/tmp/enstore/enstore/LTO3_09.stat', 'mover_address': ('131.225.13.139', 7809), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.139', 46907), 'data_ip': 'stkenmvr109a', 'logname': 'LTO09MV', 'hostip': '131.225.13.139', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'lm': {'address': ('131.225.13.4', 7506)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/F815C1A7-907E-DC11-86AA-003048726DD9.root', 'uid': 0, 'uname': 'root', 'rmajor': 0, 'machine': ('Linux', 'stkensrv3n.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 08:50:41 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (33188, 124757760L, 20L, 1, 9811, 5063, 1498719455L, 1247963569, 1192845783, 1192845374), 'mode_octal': '0100744', 'mode': 33252, 'size_bytes': 1498719455L, 'gname': 'root', 'gid': 0, 'fullname': '/dev/null', 'inode': 124757760L, 'minor': 20}, 'version': 'v3_7d CVS $Revision: 1.929 $ encp', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/.(access)(0007000000000000096FA700)', 'status': ('ok', None), 'volume': 'VOF410', 'outfile': '/dev/null', 'fc': {'status': ('ok', None), 'pnfsid': '0007000000000000096FA700', 'pnfs_name0': '/pnfs/fnal.gov/usr/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/F815C1A7-907E-DC11-86AA-003048726DD9.root', 'deleted': 'no', 'external_label': 'VOF410', 'drive': 'stkenmvr106a:/dev/rmt/tps2d0n:1210116666', 'update': '2007-10-19 21:02:57.784073', 'gid': 0, 'location_cookie': '0000_000000000_0000083', 'complete_crc': 3596548126L, 'address': ('131.225.13.59', 7501), 'uid': 0, 'bfid': 'CDMS119284576600000', 'sanity_cookie': (65536L, 1854190849L), 'size': 1498719455L}, 'infilepath': '/pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/F815C1A7-907E-DC11-86AA-003048726DD9.root', 'at_the_top': 14, 'bfid': 'CDMS119284576600000', 'local_inode': 2469L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': 7, 'retry': 0}, 'callback_addr': ('131.225.13.51', 43476), 'work': 'read_from_hsm', 'override_ro_mount': 1, 'times': {'lm_dequeued': 1257573638.1305771, 'encp_start_time': 1257573635.401355, 'job(trunc. 4237) Thread client_connect_thread 00:00:38 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:00:38 stkensrv3n.fnal.gov 007327 root I ENCP Control socket ('131.225.13.51', 43476) is connected to ('131.225.13.139', 46908) for stkensrv3n.fnal.gov-1257573636-7327-0. elapsed=2.780sec 00:00:38 stkensrv3n.fnal.gov 007327 root I ENCP Received callback ticket from mover LTO3_09.mover for transfer stkensrv3n.fnal.gov-1257573636-7327-0. elapsed=2.780sec 00:00:38 stkensrv3n.fnal.gov 007327 root I ENCP Attempting to connect data socket to mover at ('131.225.13.139', 46907). 00:00:38 stkensrv3n.fnal.gov 007327 root I ENCP Connecting data socket to mover (('131.225.13.139', 46907)) with interface 131.225.13.51. elapsed=2.780sec 00:00:38 stkensrv3n.fnal.gov 007327 root I ENCP Data socket ('131.225.13.51', 43477) is connected to ('131.225.13.139', 46907). elapsed=2.780sec 00:00:38 stkensrv3n.fnal.gov 007327 root I ENCP Opening local file. 00:00:38 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 14257351/14257351, blocks 109 header 139 Thread tape_thread 00:00:38 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 14257212/14257212 bytes Thread net_thread 00:00:38 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:38 stkenmvr109a.fnal.gov 031352 root I LTO09MV tape position: target 83 current 241 13613 Thread seek_thread 00:00:38 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:00:38 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30091 block 109 block_size 0 bloc_loc 5648579 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 14257351 block_read_tot 0.145267009735 tape_rate 98145828.3336 Thread tape_thread 00:00:38 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 14257351/14257351 bytes Thread tape_thread 00:00:38 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00190806388855 read 0.0720524787903 crc_check 0.0704057216644 push 0.000900745391846 block_read 0.145267009735 idle 0.00236105918884 Thread tape_thread 00:00:38 stkendca22a.fnal.gov 004374 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: garbage Storage Group: test File Family: dcache FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0003 --put-cache 0005000000000000005FA5F8 /diskb/write-pool-2/data/0005000000000000005FA5F8 00:00:38 stkendca22a.fnal.gov 004441 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: garbage Storage Group: test File Family: dcache FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0001 --put-cache 0005000000000000005FA590 /diskb/write-pool-2/data/0005000000000000005FA590 00:00:38 stkendca22a.fnal.gov 004475 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: garbage Storage Group: test File Family: dcache FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0002 --put-cache 0005000000000000005FA5C8 /diskb/write-pool-2/data/0005000000000000005FA5C8 00:00:39 stkendca22a.fnal.gov 004374 root I ENCP Configuration Server: no such name: 'garbage.library_manager' 00:00:39 stkendca22a.fnal.gov 004374 root E ENCP INFILE=/diskb/write-pool-2/data/0005000000000000005FA5F8 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0003 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=0.88 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager' 00:00:39 stkendca22a.fnal.gov 004441 root I ENCP Configuration Server: no such name: 'garbage.library_manager' 00:00:39 stkendca22a.fnal.gov 004441 root E ENCP INFILE=/diskb/write-pool-2/data/0005000000000000005FA590 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0001 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=0.89 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager' 00:00:39 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:39 stkendca22a.fnal.gov 004475 root I ENCP Configuration Server: no such name: 'garbage.library_manager' 00:00:39 stkendca22a.fnal.gov 004475 root E ENCP INFILE=/diskb/write-pool-2/data/0005000000000000005FA5C8 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0002 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=0.91 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager' 00:00:39 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30092 current 30091 109 Thread seek_thread 00:00:39 stkenmvr24a.fnal.gov 005803 root I DBT24MV read bytes 2929187876/2929187876, blocks 22348 header 0 Thread tape_thread 00:00:39 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:39 stkenmvr24a.fnal.gov 005803 root I DBT24MV write_client: wrote 2929187876/2929187876 bytes Thread net_thread 00:00:39 stkenmvr24a.fnal.gov 005803 root I DBT24MV write_client waits for tape thread to exit Thread net_thread 00:00:39 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=19889812 Thread tape_thread 00:00:39 stkenmvr24a.fnal.gov 005803 root I DBT24MV drive stats after read. Tape VO6001 position 269 block 22348 block_size 0 bloc_loc 847128 tot_blocks 0 BOT 0 read_err 0 write_err 0 bytes 2929187876 block_read_tot 93.1581351757 tape_rate 31443178.5316 Thread tape_thread 00:00:39 stkenmvr24a.fnal.gov 005803 root I DBT24MV read_tape exiting, read 2929187876/2929187876 bytes Thread tape_thread 00:00:39 stkenmvr24a.fnal.gov 005803 root I DBT24MV read_tape timing:get_space 0.400324106216 read 78.6953315735 crc_check 13.6206333637 push 0.441846132278 block_read 93.1581351757 idle 0.856170415878 Thread tape_thread 00:00:39 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 19889951/19889951, blocks 152 header 139 Thread tape_thread 00:00:39 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 19889812/19889812 bytes Thread net_thread 00:00:39 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:39 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30092 block 152 block_size 0 bloc_loc 5648732 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 19889951 block_read_tot 0.195956468582 tape_rate 101501885.311 Thread tape_thread 00:00:39 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 19889951/19889951 bytes Thread tape_thread 00:00:39 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00202655792236 read 0.0949070453644 crc_check 0.0977523326874 push 0.00127053260803 block_read 0.195956468582 idle 0.00324249267578 Thread tape_thread 00:00:39 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:40 stkendm7a.fnal.gov 004556 root I ENCP File /pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZS0/KZS013/f6 transfered for stkendm7a.fnal.gov-1257573532-4556-136 in 96.9673900604 seconds. Thread READ_2 00:00:40 stkenmvr24a.fnal.gov 005803 root I DBT24MV write_client exits Thread net_thread 00:00:40 stkenmvr24a.fnal.gov 005803 root I DBT24MV transfer complete volume=VO6001 location=269 Thread net_thread 00:00:40 last message repeated 1 times 00:00:40 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_rd_access = '282', last_access = '2009-11-07 00:00:40' where label = 'VO6001'; 00:00:40 stkenmvr24a.fnal.gov 005803 root I DBT24MV Sending done to client: {'vc': {'comment': '', 'declared': 1080687077.0, 'blocksize': 131072, 'storage_group': 'ktev', 'library': 'CD-9940B', 'eod_cookie': '0000_000000000_0000505', 'current_location': '0000_000000000_0000269', 'user_inhibit': ['none', 'none'], 'sum_rd_access': 281, 'system_inhibit': ['none', 'migrating'], 'external_label': 'VO6001', 'wrapper': 'cern', 'remaining_bytes': 1167480832L, 'sum_mounts': 112, 'capacity_bytes': 214748364800L, 'media_type': '9940B', 'last_access': 1257573542.0, 'status': ('ok', None), 'si_time': [0.0, 1257569739.0], 'non_del_files': 168, 'sum_wr_err': 0, 'sum_wr_access': 168, 'file_family': 'ktev_migration', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration.cern', 'modification_time': 1224966315.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1081278425.0}, 'outfilepath': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000269', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 2929187876L, 'mover': {'max_buffer': 1048576000L, 'check_written_file': 10, 'max_failures': 5, 'library': 'CD-9940B.library_manager', 'local_mover': 0, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7584, 'mc_device': '0,0,10,5', 'compression': 0, 'serial_num': '479002020778', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 28311552.0, 'vendor_id': 'STK', 'driver': 'FTTDriver', 'host': 'stkenmvr24a', 'max_time_in_state': 1200, 'device': 'stkenmvr24a:/dev/rmt/tps0d0n', 'name': '9940B24.mover', 'send_stats': 1, 'product_id': 'T9940B', 'statistics_path': '/tmp/enstore/enstore/DBT24MV.stat', 'mover_address': ('131.225.13.43', 7584), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.43', 57420), 'data_ip': 'stkenmvr24a', 'logname': 'DBT24MV', 'hostip': '131.225.13.43', 'do_cleaning': 'No', 'media_changer': 'stk.media_changer'}, 'lm': {'address': ('131.225.13.4', 7522)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZS0/KZS013/f6', 'uid': 0, 'uname': 'root', 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (33188, 527672520L, 20L, 1, 5406, 2539, 1L, 1178549339, 1081285622, 1081145308), 'mode_octal': '0100744', 'mode': 33252, 'size_bytes': 2929187876L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000269', 'inode': 527672520L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZS0/KZS013/.(access)(001F0000000000000073A4C8)', 'status': ('ok', None), 'volume': 'VO6001', 'outfile': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000269', 'fc': {'status': ('ok', None), 'pnfsid': '001F0000000000000073A4C8', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZS0/KZS013/f6', 'deleted': 'no', 'external_label': 'VO6001', 'drive': 'stkenmvr16a:/dev/rmt/tps0d0n:4790000143', 'update': None, 'gid': -1, 'location_cookie': '0000_000000000_0000269', 'complete_crc': 1680417830L, 'address': ('131.225.13.59', 7501), 'uid': -1, 'bfid': 'CDMS108128561900000', 'sanity_cookie': (65536L, 66910523L), 'size': 2929187876L}, 'infilepath': '/pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZS0/KZS013/f6', 'at_the_top': 29, 'bfid': 'CDMS108128561900000', 'local_inode': 31244297L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40495), 'work': 'read_from_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257573532.700732, 'seek_time': 0.024815797805786133, 'transfer_time': 97.232318162918091, 't0': 1257573532, 'mount_time': 0.019973039627075195, 'lm_dequeued': 1257573543.0311389, 'job_queued': 1257573532.8832381, 'drive_call_time': 77.948771238327026, 'in_queue': 10.147464990615845, 'drive_transfer_time': 93.588562250137329}, 'user(trunc. 4152) Thread net_thread 00:00:40 stkenmvr24a.fnal.gov 005803 root I DBT24MV Done is sent Thread net_thread 00:00:40 stkenmvr24a.fnal.gov 005803 root I DBT24MV maybe_clean Thread net_thread 00:00:40 stkendm7a.fnal.gov 004556 root I ENCP Received final dialog for stkendm7a.fnal.gov-1257573532-4556-136. elapsed=3900.720sec Thread READ_2 00:00:40 stkendm7a.fnal.gov 004556 root I ENCP leaving transfer_file(). Thread READ_2 00:00:40 stkenmvr24a.fnal.gov 005803 root I DBT24MV returned from maybe_clean Thread net_thread 00:00:40 stkendm7a.fnal.gov 004556 root I ENCP MSG_TYPE=ENCP_XFER read_from_hsm /pnfs/fs/usr/ktev/migrated_from_dlt/KZ/KZS0/KZS013/f6 -> /data/data2/Migration_Spool/VO6001:0000_000000000_0000269: 2929187876 bytes copied from VO6001 at 28.7 MB/S (38.9 MB/S network) (29.8 MB/S drive) (61.7 MB/S disk) mover=9940B24.mover drive_id=T9940B drive_sn=479002020778 drive_vendor=STK elapsed=107.79 {'storage_group': 'ktev', 'encp_crc': 1680417830L, 'disk_rate': 61.663580806899979, 'drive_rate': 29.848639386119817, 'driver': 'FTTDriver', 'transfer_rate': 28.726708139543977, 'mover_interface': 'stkenmvr24a', 'overall_rate': 28.679634175026361, 'media_changer': 'stk.media_changer', 'network_rate': 28.726708139543977, 'encp_ip': '131.225.13.19', 'unique_id': 'stkendm7a.fnal.gov-1257573532-4556-136'} Thread READ_2 00:00:40 stkendm7a.fnal.gov 004556 root I ENCP File /data/data2/Migration_Spool/VO6001:0000_000000000_0000269 copied successfully. Thread READ_2 00:00:40 stkendm7a.fnal.gov 004556 root I ENCP Completed transferring 2929187876 bytes in 1 files in 107.800241947 sec. Overall rate = 28.7 MB/sec. Transfer rate = 28.7 MB/sec. Network rate = 38.9 MB/sec. Drive rate = 29.8 MB/sec. Disk rate = 61.7 MB/sec. Exit status = 0. Thread READ_2 00:00:40 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:40 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30093 current 30092 152 Thread seek_thread 00:00:40 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:40 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=54899818 Thread tape_thread 00:00:40 stkensrv4n.fnal.gov 005357 enstore E LTO4G1LM MOVER ERROR RQ {'status': ('ok', None), 'returned_work': None, 'library': None, 'mover_type': '', 'current_location': '0000_000000000_0000000', 'ip_map': '', 'operation': None, 'read_only': 0, 'current_time': 1257573640.883759, 'error_source': None, 'external_label': None, 'state': 'OFFLINE', 'r_a': (('131.225.13.194', 54880), 19734L, '131.225.13.194-54880-1256586929.509013-4866'), 'library_list': [('CD-LTO4G1.library_manager', ('131.225.13.4', 8541))], 'current_priority': (None, None), 'transfer_deficiency': 1, 'time_in_state': 986710.52370095253, 'address': ('131.225.13.194', 7860), 'volume_family': None, 'work': 'mover_error', 'mover': 'LTO4_99.mover', 'volume_status': (['none', 'none'], ['none', 'none']), 'unique_id': None} 00:00:41 stkensrv4n.fnal.gov 005795 enstore I 9940BLM 00:00:41 stkensrv4n.fnal.gov 005795 enstore I 9940BLM read_from_hsm next work on vol=VO6001 mover=9940B24.mover requester:root 00:00:41 stkensrv4n.fnal.gov 005795 enstore I 9940BLM HAVE_BOUND:sending read_from_hsm /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT06/f8 to mover 9940B24.mover ('131.225.13.43', 7584) DEL_DISM 2 00:00:41 stkensrv4n.fnal.gov 005795 enstore I 9940BLM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : ktev 00:00:41 stkensrv4n.fnal.gov 005795 enstore I 9940BLM HAVE_BOUND:sending {'vc': {'comment': '', 'declared': 1080687077.0, 'blocksize': 131072, 'storage_group': 'ktev', 'library': 'CD-9940B', 'eod_cookie': '0000_000000000_0000505', 'current_location': '0000_000000000_0000272', 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'migrating'], 'external_label': 'VO6001', 'wrapper': 'cern', 'remaining_bytes': 1167480832L, 'sum_mounts': 112, 'sum_rd_access': 280, 'media_type': '9940B', 'last_access': 1257573534.0, 'status': ('ok', None), 'si_time': [0.0, 1257569739.0], 'capacity_bytes': 214748364800L, 'non_del_files': 168, 'sum_wr_err': 0, 'sum_wr_access': 168, 'file_family': 'ktev_migration', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration.cern', 'modification_time': 1224966315.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1081278425.0}, 'outfilepath': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000272', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 19063304L, 'mover': '9940B24.mover', 'lm': {'address': ('131.225.13.4', 7522)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT06/f8', 'uid': 0, 'uname': 'root', 'gid': 0, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (33188, 527688960L, 20L, 1, 5406, 2539, 19063304L, 1102924806, 1081285634, 1081169780), 'mode_octal': '0100744', 'mode': 33252, 'gname': 'root', 'size_bytes': 19063304L, 'fullname': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000272', 'inode': 527688960L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT06/.(access)(001F0000000000000073E500)', 'status': ('ok', None), 'volume': 'VO6001', 'outfile': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000272', 'fc': {'status': ('ok', None), 'pnfsid': '001F0000000000000073E500', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT06/f8', 'deleted': 'no', 'external_label': 'VO6001', 'drive': 'stkenmvr16a:/dev/rmt/tps0d0n:4790000143', 'update': None, 'gid': -1, 'location_cookie': '0000_000000000_0000272', 'complete_crc': 459333098L, 'address': ('131.225.13.59', 7501), 'uid': -1, 'bfid': 'CDMS108128563200000', 'sanity_cookie': (65536L, 3107600827L), 'size': 19063304L}, 'infilepath': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT06/f8', 'at_the_top': 297, 'bfid': 'CDMS108128563200000', 'local_inode': 31244298L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40498), 'work': 'read_from_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573641.1113851, 'encp_start_time': 1257573534.744729, 'job_queued': 1257573534.889178, 't0': 1257573534, 'in_queue': 106.22175192832947}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573534-4556-137'} to mover 00:00:41 stkensrv4n.fnal.gov 005795 enstore I 9940BLM 00:00:41 stkenmvr24a.fnal.gov 005803 root I DBT24MV READ FROM HSM Thread MainThread 00:00:41 stkenmvr24a.fnal.gov 005803 root I DBT24MV SENDING {'vc': {'comment': '', 'declared': 1080687077.0, 'blocksize': 131072, 'storage_group': 'ktev', 'library': 'CD-9940B', 'eod_cookie': '0000_000000000_0000505', 'current_location': '0000_000000000_0000272', 'user_inhibit': ['none', 'none'], 'sum_rd_access': 280, 'system_inhibit': ['none', 'migrating'], 'external_label': 'VO6001', 'wrapper': 'cern', 'remaining_bytes': 1167480832L, 'sum_mounts': 112, 'capacity_bytes': 214748364800L, 'media_type': '9940B', 'last_access': 1257573534.0, 'status': ('ok', None), 'si_time': [0.0, 1257569739.0], 'non_del_files': 168, 'sum_wr_err': 0, 'sum_wr_access': 168, 'file_family': 'ktev_migration', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration.cern', 'modification_time': 1224966315.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1081278425.0}, 'outfilepath': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000272', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 19063304L, 'mover': {'max_buffer': 1048576000L, 'check_written_file': 10, 'max_failures': 5, 'library': 'CD-9940B.library_manager', 'local_mover': 0, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7584, 'mc_device': '0,0,10,5', 'compression': 0, 'serial_num': '479002020778', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 28311552.0, 'vendor_id': 'STK', 'driver': 'FTTDriver', 'host': 'stkenmvr24a', 'max_time_in_state': 1200, 'device': 'stkenmvr24a:/dev/rmt/tps0d0n', 'name': '9940B24.mover', 'send_stats': 1, 'product_id': 'T9940B', 'statistics_path': '/tmp/enstore/enstore/DBT24MV.stat', 'mover_address': ('131.225.13.43', 7584), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.43', 57429), 'data_ip': 'stkenmvr24a', 'logname': 'DBT24MV', 'hostip': '131.225.13.43', 'do_cleaning': 'No', 'media_changer': 'stk.media_changer'}, 'lm': {'address': ('131.225.13.4', 7522)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT06/f8', 'uid': 0, 'uname': 'root', 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (33188, 527688960L, 20L, 1, 5406, 2539, 19063304L, 1102924806, 1081285634, 1081169780), 'mode_octal': '0100744', 'mode': 33252, 'size_bytes': 19063304L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000272', 'inode': 527688960L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT06/.(access)(001F0000000000000073E500)', 'status': ('ok', None), 'volume': 'VO6001', 'outfile': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000272', 'fc': {'status': ('ok', None), 'pnfsid': '001F0000000000000073E500', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT06/f8', 'deleted': 'no', 'external_label': 'VO6001', 'drive': 'stkenmvr16a:/dev/rmt/tps0d0n:4790000143', 'update': None, 'gid': -1, 'location_cookie': '0000_000000000_0000272', 'complete_crc': 459333098L, 'address': ('131.225.13.59', 7501), 'uid': -1, 'bfid': 'CDMS108128563200000', 'sanity_cookie': (65536L, 3107600827L), 'size': 19063304L}, 'infilepath': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT06/f8', 'at_the_top': 297, 'bfid': 'CDMS108128563200000', 'local_inode': 31244298L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40498), 'work': 'read_from_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573641.1113851, 'encp_start_time': 1257573534.744729, 'job_queued': 1257573534.889178, 't0': 1257573534, 'in_queue': 106.22175192832947}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573534-4556-137'} Thread client_connect_thread 00:00:41 stkendm7a.fnal.gov 004556 root I MIGRATION Listening for control socket at: ('131.225.13.19', 40498) Thread READ 00:00:41 stkendm7a.fnal.gov 004556 root I MIGRATION Waiting for mover to connect control socket at address: ('131.225.13.19', 40498) elapsed=3901.480sec Thread READ 00:00:41 stkendm7a.fnal.gov 004556 root I MIGRATION Control socket ('131.225.13.19', 40498) is connected to ('131.225.13.43', 57430) for stkendm7a.fnal.gov-1257573534-4556-137. elapsed=3901.490sec Thread READ 00:00:41 stkendm7a.fnal.gov 004556 root I MIGRATION Received callback ticket from mover 9940B24.mover for transfer stkendm7a.fnal.gov-1257573534-4556-137. elapsed=3901.500sec Thread READ 00:00:41 stkendm7a.fnal.gov 004556 root I MIGRATION Attempting to connect data socket to mover at ('131.225.13.43', 57429). Thread READ 00:00:41 stkendm7a.fnal.gov 004556 root I MIGRATION Connecting data socket to mover (('131.225.13.43', 57429)) with interface 131.225.13.19. elapsed=3901.500sec Thread READ 00:00:41 stkendm7a.fnal.gov 004556 root I MIGRATION Data socket ('131.225.13.19', 40526) is connected to ('131.225.13.43', 57429). elapsed=3901.500sec Thread READ 00:00:41 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 54899957/54899957, blocks 419 header 139 Thread tape_thread 00:00:41 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 54899818/54899818 bytes Thread net_thread 00:00:41 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:41 last message repeated 1 times 00:00:41 stkendm7a.fnal.gov 004556 root I MIGRATION Opening local file. Thread READ 00:00:41 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30093 block 419 block_size 0 bloc_loc 5649152 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 54899957 block_read_tot 0.495603322983 tape_rate 110773988.902 Thread tape_thread 00:00:41 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 54899957/54899957 bytes Thread tape_thread 00:00:41 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00524663925171 read 0.215514659882 crc_check 0.271211862564 push 0.0036301612854 block_read 0.495603322983 idle 0.0100629329681 Thread tape_thread 00:00:41 stkenmvr24a.fnal.gov 005803 root I DBT24MV tape position: target 272 current 269 22348 Thread seek_thread 00:00:41 stkenmvr24a.fnal.gov 005803 root I DBT24MV skip_fm 3 Thread seek_thread 00:00:41 stkenmvr24a.fnal.gov 005803 root I DBT24MV read_tape starting, bytes_to_read=19063304 Thread tape_thread 00:00:41 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:41 last message repeated 1 times 00:00:41 stkenmvr24a.fnal.gov 005803 root I DBT24MV read bytes 19063304/19063304, blocks 146 header 0 Thread tape_thread 00:00:41 stkenmvr24a.fnal.gov 005803 root I DBT24MV write_client: wrote 19063304/19063304 bytes Thread net_thread 00:00:41 stkenmvr24a.fnal.gov 005803 root I DBT24MV write_client waits for tape thread to exit Thread net_thread 00:00:41 stkenmvr24a.fnal.gov 005803 root I DBT24MV drive stats after read. Tape VO6001 position 272 block 146 block_size 0 bloc_loc 847279 tot_blocks 0 BOT 0 read_err 0 write_err 0 bytes 19063304 block_read_tot 0.513081073761 tape_rate 37154564.7947 Thread tape_thread 00:00:41 stkenmvr24a.fnal.gov 005803 root I DBT24MV read_tape exiting, read 19063304/19063304 bytes Thread tape_thread 00:00:41 stkenmvr24a.fnal.gov 005803 root I DBT24MV read_tape timing:get_space 0.00282645225525 read 0.419941425323 crc_check 0.0874364376068 push 0.00287675857544 block_read 0.513081073761 idle 0.00532960891724 Thread tape_thread 00:00:42 stkendm7a.fnal.gov 004556 root I MIGRATION File /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT06/f8 transfered for stkendm7a.fnal.gov-1257573534-4556-137 in 0.858628988266 seconds. Thread READ 00:00:42 stkensrv4n.fnal.gov 005795 enstore I 9940BLM 00:00:42 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:42 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30094 current 30093 419 Thread seek_thread 00:00:42 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:42 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=25455114 Thread tape_thread 00:00:42 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 25455253/25455253, blocks 195 header 139 Thread tape_thread 00:00:42 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 25455114/25455114 bytes Thread net_thread 00:00:42 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:42 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30094 block 195 block_size 0 bloc_loc 5649348 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 25455253 block_read_tot 0.240310907364 tape_rate 105926332.18 Thread tape_thread 00:00:42 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 25455253/25455253 bytes Thread tape_thread 00:00:42 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00287508964539 read 0.110697746277 crc_check 0.125100851059 push 0.00163722038269 block_read 0.240310907364 idle 0.0045063495636 Thread tape_thread 00:00:42 stkenmvr24a.fnal.gov 005803 root I DBT24MV write_client exits Thread net_thread 00:00:42 stkenmvr24a.fnal.gov 005803 root I DBT24MV transfer complete volume=VO6001 location=272 Thread net_thread 00:00:42 last message repeated 1 times 00:00:42 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_rd_access = '283', last_access = '2009-11-07 00:00:42' where label = 'VO6001'; 00:00:42 stkenmvr24a.fnal.gov 005803 root I DBT24MV Sending done to client: {'vc': {'comment': '', 'declared': 1080687077.0, 'blocksize': 131072, 'storage_group': 'ktev', 'library': 'CD-9940B', 'eod_cookie': '0000_000000000_0000505', 'current_location': '0000_000000000_0000272', 'user_inhibit': ['none', 'none'], 'sum_rd_access': 282, 'system_inhibit': ['none', 'migrating'], 'external_label': 'VO6001', 'wrapper': 'cern', 'remaining_bytes': 1167480832L, 'sum_mounts': 112, 'capacity_bytes': 214748364800L, 'media_type': '9940B', 'last_access': 1257573640.0, 'status': ('ok', None), 'si_time': [0.0, 1257569739.0], 'non_del_files': 168, 'sum_wr_err': 0, 'sum_wr_access': 168, 'file_family': 'ktev_migration', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration.cern', 'modification_time': 1224966315.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1081278425.0}, 'outfilepath': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000272', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 19063304L, 'mover': {'max_buffer': 1048576000L, 'check_written_file': 10, 'max_failures': 5, 'library': 'CD-9940B.library_manager', 'local_mover': 0, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7584, 'mc_device': '0,0,10,5', 'compression': 0, 'serial_num': '479002020778', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 28311552.0, 'vendor_id': 'STK', 'driver': 'FTTDriver', 'host': 'stkenmvr24a', 'max_time_in_state': 1200, 'device': 'stkenmvr24a:/dev/rmt/tps0d0n', 'name': '9940B24.mover', 'send_stats': 1, 'product_id': 'T9940B', 'statistics_path': '/tmp/enstore/enstore/DBT24MV.stat', 'mover_address': ('131.225.13.43', 7584), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.43', 57429), 'data_ip': 'stkenmvr24a', 'logname': 'DBT24MV', 'hostip': '131.225.13.43', 'do_cleaning': 'No', 'media_changer': 'stk.media_changer'}, 'lm': {'address': ('131.225.13.4', 7522)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT06/f8', 'uid': 0, 'uname': 'root', 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (33188, 527688960L, 20L, 1, 5406, 2539, 19063304L, 1102924806, 1081285634, 1081169780), 'mode_octal': '0100744', 'mode': 33252, 'size_bytes': 19063304L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000272', 'inode': 527688960L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT06/.(access)(001F0000000000000073E500)', 'status': ('ok', None), 'volume': 'VO6001', 'outfile': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000272', 'fc': {'status': ('ok', None), 'pnfsid': '001F0000000000000073E500', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT06/f8', 'deleted': 'no', 'external_label': 'VO6001', 'drive': 'stkenmvr16a:/dev/rmt/tps0d0n:4790000143', 'update': None, 'gid': -1, 'location_cookie': '0000_000000000_0000272', 'complete_crc': 459333098L, 'address': ('131.225.13.59', 7501), 'uid': -1, 'bfid': 'CDMS108128563200000', 'sanity_cookie': (65536L, 3107600827L), 'size': 19063304L}, 'infilepath': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT06/f8', 'at_the_top': 297, 'bfid': 'CDMS108128563200000', 'local_inode': 31244298L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40498), 'work': 'read_from_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257573534.744729, 'seek_time': 0.024063825607299805, 'transfer_time': 1.5419051647186279, 't0': 1257573534, 'mount_time': 0.022266149520874023, 'lm_dequeued': 1257573641.1113851, 'job_queued': 1257573534.889178, 'drive_call_time': 0.41528058052062988, 'in_queue': 106.22175192832947, 'drive_transfer_time': 0.51578116416931152}, '(trunc. 4156) Thread net_thread 00:00:42 stkenmvr24a.fnal.gov 005803 root I DBT24MV Done is sent Thread net_thread 00:00:42 stkenmvr24a.fnal.gov 005803 root I DBT24MV maybe_clean Thread net_thread 00:00:42 stkendm7a.fnal.gov 004556 root I MIGRATION Received final dialog for stkendm7a.fnal.gov-1257573534-4556-137. elapsed=3903.120sec Thread READ 00:00:42 stkenmvr24a.fnal.gov 005803 root I DBT24MV returned from maybe_clean Thread net_thread 00:00:42 stkendm7a.fnal.gov 004556 root I MIGRATION leaving transfer_file(). Thread READ 00:00:42 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:42 stkendm7a.fnal.gov 004556 root I MIGRATION MSG_TYPE=ENCP_XFER read_from_hsm /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT06/f8 -> /data/data2/Migration_Spool/VO6001:0000_000000000_0000272: 19063304 bytes copied from VO6001 at 11.6 MB/S (34.8 MB/S network) (35.2 MB/S drive) (38.1 MB/S disk) mover=9940B24.mover drive_id=T9940B drive_sn=479002020778 drive_vendor=STK elapsed=108.19 {'storage_group': 'ktev', 'encp_crc': 459333098L, 'disk_rate': 38.086045571106936, 'drive_rate': 35.247862220646475, 'driver': 'FTTDriver', 'transfer_rate': 11.598091148786207, 'mover_interface': 'stkenmvr24a', 'overall_rate': 10.354912876419156, 'media_changer': 'stk.media_changer', 'network_rate': 11.598091148786207, 'encp_ip': '131.225.13.19', 'unique_id': 'stkendm7a.fnal.gov-1257573534-4556-137'} Thread READ 00:00:42 stkendm7a.fnal.gov 004556 root I MIGRATION File /data/data2/Migration_Spool/VO6001:0000_000000000_0000272 copied successfully. Thread READ 00:00:42 stkendm7a.fnal.gov 004556 root I MIGRATION Completed transferring 19063304 bytes in 1 files in 108.196960926 sec. Overall rate = 10.4 MB/sec. Transfer rate = 11.6 MB/sec. Network rate = 34.8 MB/sec. Drive rate = 35.2 MB/sec. Disk rate = 38.1 MB/sec. Exit status = 0. Thread READ 00:00:43 stkensrv4n.fnal.gov 005795 enstore I 9940BLM 00:00:43 stkensrv4n.fnal.gov 005795 enstore I 9940BLM read_from_hsm next work on vol=VO6001 mover=9940B24.mover requester:root 00:00:43 stkensrv4n.fnal.gov 005795 enstore I 9940BLM HAVE_BOUND:sending read_from_hsm /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT11/f19 to mover 9940B24.mover ('131.225.13.43', 7584) DEL_DISM 2 00:00:43 stkensrv4n.fnal.gov 005795 enstore I 9940BLM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : ktev 00:00:43 stkensrv4n.fnal.gov 005795 enstore I 9940BLM HAVE_BOUND:sending {'vc': {'comment': '', 'declared': 1080687077.0, 'blocksize': 131072, 'storage_group': 'ktev', 'library': 'CD-9940B', 'eod_cookie': '0000_000000000_0000505', 'current_location': '0000_000000000_0000275', 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'migrating'], 'external_label': 'VO6001', 'wrapper': 'cern', 'remaining_bytes': 1167480832L, 'sum_mounts': 112, 'sum_rd_access': 281, 'media_type': '9940B', 'last_access': 1257573542.0, 'status': ('ok', None), 'si_time': [0.0, 1257569739.0], 'capacity_bytes': 214748364800L, 'non_del_files': 168, 'sum_wr_err': 0, 'sum_wr_access': 168, 'file_family': 'ktev_migration', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration.cern', 'modification_time': 1224966315.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1081278425.0}, 'outfilepath': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000275', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 41105488L, 'mover': '9940B24.mover', 'lm': {'address': ('131.225.13.4', 7522)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT11/f19', 'uid': 0, 'uname': 'root', 'gid': 0, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (33188, 527694392L, 20L, 1, 5406, 2539, 41105488L, 1102928573, 1081285654, 1081174868), 'mode_octal': '0100744', 'mode': 33252, 'gname': 'root', 'size_bytes': 41105488L, 'fullname': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000275', 'inode': 527694392L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT11/.(access)(001F0000000000000073FA38)', 'status': ('ok', None), 'volume': 'VO6001', 'outfile': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000275', 'fc': {'status': ('ok', None), 'pnfsid': '001F0000000000000073FA38', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT11/f19', 'deleted': 'no', 'external_label': 'VO6001', 'drive': 'stkenmvr16a:/dev/rmt/tps0d0n:4790000143', 'update': None, 'gid': -1, 'location_cookie': '0000_000000000_0000275', 'complete_crc': 1279529005L, 'address': ('131.225.13.59', 7501), 'uid': -1, 'bfid': 'CDMS108128564600000', 'sanity_cookie': (65536L, 3990704794L), 'size': 41105488L}, 'infilepath': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT11/f19', 'at_the_top': 7, 'bfid': 'CDMS108128564600000', 'local_inode': 31244299L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40504), 'work': 'read_from_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573643.1141601, 'encp_start_time': 1257573542.6823599, 'job_queued': 1257573542.825985, 't0': 1257573542, 'in_queue': 100.2881190776825}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573542-4556-138'} to mover 00:00:43 stkensrv4n.fnal.gov 005795 enstore I 9940BLM 00:00:43 stkenmvr24a.fnal.gov 005803 root I DBT24MV READ FROM HSM Thread MainThread 00:00:43 stkenmvr24a.fnal.gov 005803 root I DBT24MV SENDING {'vc': {'comment': '', 'declared': 1080687077.0, 'blocksize': 131072, 'storage_group': 'ktev', 'library': 'CD-9940B', 'eod_cookie': '0000_000000000_0000505', 'current_location': '0000_000000000_0000275', 'user_inhibit': ['none', 'none'], 'sum_rd_access': 281, 'system_inhibit': ['none', 'migrating'], 'external_label': 'VO6001', 'wrapper': 'cern', 'remaining_bytes': 1167480832L, 'sum_mounts': 112, 'capacity_bytes': 214748364800L, 'media_type': '9940B', 'last_access': 1257573542.0, 'status': ('ok', None), 'si_time': [0.0, 1257569739.0], 'non_del_files': 168, 'sum_wr_err': 0, 'sum_wr_access': 168, 'file_family': 'ktev_migration', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration.cern', 'modification_time': 1224966315.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1081278425.0}, 'outfilepath': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000275', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 41105488L, 'mover': {'max_buffer': 1048576000L, 'check_written_file': 10, 'max_failures': 5, 'library': 'CD-9940B.library_manager', 'local_mover': 0, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7584, 'mc_device': '0,0,10,5', 'compression': 0, 'serial_num': '479002020778', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 28311552.0, 'vendor_id': 'STK', 'driver': 'FTTDriver', 'host': 'stkenmvr24a', 'max_time_in_state': 1200, 'device': 'stkenmvr24a:/dev/rmt/tps0d0n', 'name': '9940B24.mover', 'send_stats': 1, 'product_id': 'T9940B', 'statistics_path': '/tmp/enstore/enstore/DBT24MV.stat', 'mover_address': ('131.225.13.43', 7584), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.43', 57431), 'data_ip': 'stkenmvr24a', 'logname': 'DBT24MV', 'hostip': '131.225.13.43', 'do_cleaning': 'No', 'media_changer': 'stk.media_changer'}, 'lm': {'address': ('131.225.13.4', 7522)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT11/f19', 'uid': 0, 'uname': 'root', 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (33188, 527694392L, 20L, 1, 5406, 2539, 41105488L, 1102928573, 1081285654, 1081174868), 'mode_octal': '0100744', 'mode': 33252, 'size_bytes': 41105488L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000275', 'inode': 527694392L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT11/.(access)(001F0000000000000073FA38)', 'status': ('ok', None), 'volume': 'VO6001', 'outfile': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000275', 'fc': {'status': ('ok', None), 'pnfsid': '001F0000000000000073FA38', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT11/f19', 'deleted': 'no', 'external_label': 'VO6001', 'drive': 'stkenmvr16a:/dev/rmt/tps0d0n:4790000143', 'update': None, 'gid': -1, 'location_cookie': '0000_000000000_0000275', 'complete_crc': 1279529005L, 'address': ('131.225.13.59', 7501), 'uid': -1, 'bfid': 'CDMS108128564600000', 'sanity_cookie': (65536L, 3990704794L), 'size': 41105488L}, 'infilepath': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT11/f19', 'at_the_top': 7, 'bfid': 'CDMS108128564600000', 'local_inode': 31244299L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40504), 'work': 'read_from_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573643.1141601, 'encp_start_time': 1257573542.6823599, 'job_queued': 1257573542.825985, 't0': 1257573542, 'in_queue': 100.2881190776825}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573542-4556-138'} Thread client_connect_thread 00:00:43 stkendm7a.fnal.gov 004556 root I MIGRATION Listening for control socket at: ('131.225.13.19', 40504) Thread READ_1 00:00:43 stkendm7a.fnal.gov 004556 root I MIGRATION Waiting for mover to connect control socket at address: ('131.225.13.19', 40504) elapsed=3903.490sec Thread READ_1 00:00:43 stkendm7a.fnal.gov 004556 root I MIGRATION Control socket ('131.225.13.19', 40504) is connected to ('131.225.13.43', 57432) for stkendm7a.fnal.gov-1257573542-4556-138. elapsed=3903.490sec Thread READ_1 00:00:43 stkendm7a.fnal.gov 004556 root I MIGRATION Received callback ticket from mover 9940B24.mover for transfer stkendm7a.fnal.gov-1257573542-4556-138. elapsed=3903.500sec Thread READ_1 00:00:43 stkendm7a.fnal.gov 004556 root I MIGRATION Attempting to connect data socket to mover at ('131.225.13.43', 57431). Thread READ_1 00:00:43 stkendm7a.fnal.gov 004556 root I MIGRATION Connecting data socket to mover (('131.225.13.43', 57431)) with interface 131.225.13.19. elapsed=3903.500sec Thread READ_1 00:00:43 stkendm7a.fnal.gov 004556 root I MIGRATION Data socket ('131.225.13.19', 40527) is connected to ('131.225.13.43', 57431). elapsed=3903.500sec Thread READ_1 00:00:43 stkendm7a.fnal.gov 004556 root I MIGRATION Opening local file. Thread READ_1 00:00:43 stkenmvr24a.fnal.gov 005803 root I DBT24MV tape position: target 275 current 272 146 Thread seek_thread 00:00:43 stkenmvr24a.fnal.gov 005803 root I DBT24MV skip_fm 3 Thread seek_thread 00:00:43 stkenmvr24a.fnal.gov 005803 root I DBT24MV read_tape starting, bytes_to_read=41105488 Thread tape_thread 00:00:43 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:43 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30095 current 30094 195 Thread seek_thread 00:00:43 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:43 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=14324540 Thread tape_thread 00:00:43 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:43 stkendm7a.fnal.gov 004556 root I ENCP Version: v3_8 CVS $Revision: 1.963 $ encp.pyc OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson), User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Current working directory: stkendm7a.fnal.gov:/opt/enstore/src Command line: encp --delayed-dismount 2 --ignore-fair-share --bypass-filesystem-max-filesize-check --priority 0 --threaded /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZT0/NZT067/f2 /data/data2/Migration_Spool/VO6001:0000_000000000_0000278 Thread READ_2 00:00:43 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 14324679/14324679, blocks 110 header 139 Thread tape_thread 00:00:43 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 14324540/14324540 bytes Thread net_thread 00:00:43 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:43 stkendm7a.fnal.gov 004556 root I ENCP Creating zero length output files. Thread READ_2 00:00:43 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30095 block 110 block_size 0 bloc_loc 5649459 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 14324679 block_read_tot 0.149104356766 tape_rate 96071498.5847 Thread tape_thread 00:00:43 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 14324679/14324679 bytes Thread tape_thread 00:00:43 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00144672393799 read 0.0762705802917 crc_check 0.070451259613 push 0.000935792922974 block_read 0.149104356766 idle 0.00239396095276 Thread tape_thread 00:00:43 stkendm7a.fnal.gov 004556 root I ENCP Sending /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZT0/NZT067/f2 read request to LM: unique_id: stkendm7a.fnal.gov-1257573643-4556-142 inputfile: /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZT0/NZT067/.(access)(001F00000000000000735A68) outputfile: /data/data2/Migration_Spool/VO6001:0000_000000000_0000278 Thread READ_2 00:00:43 stkensrv4n.fnal.gov 005795 enstore I 9940BLM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : ktev 00:00:43 stkensrv4n.fnal.gov 005795 enstore I 9940BLM read Q'd /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZT0/NZT067/f2 (stkendm7a.fnal.gov-1257573643-4556-142) -> /data/data2/Migration_Spool/VO6001:0000_000000000_0000278 : library=CD-9940B family=ktev.ktev_migration.cern requester:root 00:00:43 stkendm7a.fnal.gov 004556 root I ENCP Version: v3_8 CVS $Revision: 1.963 $ encp.pyc OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson), User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Current working directory: stkendm7a.fnal.gov:/opt/enstore/src Command line: encp --delayed-dismount 2 --ignore-fair-share --bypass-filesystem-max-filesize-check --priority 0 --threaded /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZN0/NZN083/f4 /data/data2/Migration_Spool/VO6001:0000_000000000_0000281 Thread READ 00:00:43 stkendm7a.fnal.gov 004556 root I ENCP Creating zero length output files. Thread READ 00:00:44 stkendm7a.fnal.gov 004556 root I ENCP Sending /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZN0/NZN083/f4 read request to LM: unique_id: stkendm7a.fnal.gov-1257573643-4556-143 inputfile: /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZN0/NZN083/.(access)(001F00000000000000738078) outputfile: /data/data2/Migration_Spool/VO6001:0000_000000000_0000281 Thread READ 00:00:44 stkensrv4n.fnal.gov 005795 enstore I 9940BLM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : ktev 00:00:44 stkensrv4n.fnal.gov 005795 enstore I 9940BLM read Q'd /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZN0/NZN083/f4 (stkendm7a.fnal.gov-1257573643-4556-143) -> /data/data2/Migration_Spool/VO6001:0000_000000000_0000281 : library=CD-9940B family=ktev.ktev_migration.cern requester:root 00:00:44 stkensrv4n.fnal.gov 005795 enstore I 9940BLM 00:00:44 stkenmvr24a.fnal.gov 005803 root I DBT24MV read bytes 41105488/41105488, blocks 314 header 0 Thread tape_thread 00:00:44 stkenmvr24a.fnal.gov 005803 root I DBT24MV write_client: wrote 41105488/41105488 bytes Thread net_thread 00:00:44 stkenmvr24a.fnal.gov 005803 root I DBT24MV write_client waits for tape thread to exit Thread net_thread 00:00:44 stkenmvr24a.fnal.gov 005803 root I DBT24MV drive stats after read. Tape VO6001 position 275 block 314 block_size 0 bloc_loc 847598 tot_blocks 0 BOT 0 read_err 0 write_err 0 bytes 41105488 block_read_tot 1.12058091164 tape_rate 36682302.5211 Thread tape_thread 00:00:44 stkenmvr24a.fnal.gov 005803 root I DBT24MV read_tape exiting, read 41105488/41105488 bytes Thread tape_thread 00:00:44 stkenmvr24a.fnal.gov 005803 root I DBT24MV read_tape timing:get_space 0.0048680305481 read 0.920627593994 crc_check 0.188757896423 push 0.00632739067078 block_read 1.12058091164 idle 0.0114974975586 Thread tape_thread 00:00:44 stkendm7a.fnal.gov 004556 root I ENCP File /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT11/f19 transfered for stkendm7a.fnal.gov-1257573542-4556-138 in 1.37872719765 seconds. Thread READ_1 00:00:44 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:44 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30096 current 30095 110 Thread seek_thread 00:00:44 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:44 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=5581931 Thread tape_thread 00:00:44 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 5582070/5582070, blocks 43 header 139 Thread tape_thread 00:00:44 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 5581931/5581931 bytes Thread net_thread 00:00:44 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:44 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30096 block 43 block_size 0 bloc_loc 5649503 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 5582070 block_read_tot 0.0665528774261 tape_rate 83874209.7394 Thread tape_thread 00:00:44 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 5582070/5582070 bytes Thread tape_thread 00:00:44 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00142621994019 read 0.037220954895 crc_check 0.0275435447693 push 0.000362157821655 block_read 0.0665528774261 idle 0.00100159645081 Thread tape_thread 00:00:45 stkenmvr24a.fnal.gov 005803 root I DBT24MV write_client exits Thread net_thread 00:00:45 stkenmvr24a.fnal.gov 005803 root I DBT24MV transfer complete volume=VO6001 location=275 Thread net_thread 00:00:45 last message repeated 1 times 00:00:45 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_rd_access = '284', last_access = '2009-11-07 00:00:45' where label = 'VO6001'; 00:00:45 stkenmvr24a.fnal.gov 005803 root I DBT24MV Sending done to client: {'vc': {'comment': '', 'declared': 1080687077.0, 'blocksize': 131072, 'storage_group': 'ktev', 'library': 'CD-9940B', 'eod_cookie': '0000_000000000_0000505', 'current_location': '0000_000000000_0000275', 'user_inhibit': ['none', 'none'], 'sum_rd_access': 283, 'system_inhibit': ['none', 'migrating'], 'external_label': 'VO6001', 'wrapper': 'cern', 'remaining_bytes': 1167480832L, 'sum_mounts': 112, 'capacity_bytes': 214748364800L, 'media_type': '9940B', 'last_access': 1257573642.0, 'status': ('ok', None), 'si_time': [0.0, 1257569739.0], 'non_del_files': 168, 'sum_wr_err': 0, 'sum_wr_access': 168, 'file_family': 'ktev_migration', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration.cern', 'modification_time': 1224966315.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1081278425.0}, 'outfilepath': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000275', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 41105488L, 'mover': {'max_buffer': 1048576000L, 'check_written_file': 10, 'max_failures': 5, 'library': 'CD-9940B.library_manager', 'local_mover': 0, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7584, 'mc_device': '0,0,10,5', 'compression': 0, 'serial_num': '479002020778', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 28311552.0, 'vendor_id': 'STK', 'driver': 'FTTDriver', 'host': 'stkenmvr24a', 'max_time_in_state': 1200, 'device': 'stkenmvr24a:/dev/rmt/tps0d0n', 'name': '9940B24.mover', 'send_stats': 1, 'product_id': 'T9940B', 'statistics_path': '/tmp/enstore/enstore/DBT24MV.stat', 'mover_address': ('131.225.13.43', 7584), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.43', 57431), 'data_ip': 'stkenmvr24a', 'logname': 'DBT24MV', 'hostip': '131.225.13.43', 'do_cleaning': 'No', 'media_changer': 'stk.media_changer'}, 'lm': {'address': ('131.225.13.4', 7522)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT11/f19', 'uid': 0, 'uname': 'root', 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (33188, 527694392L, 20L, 1, 5406, 2539, 41105488L, 1102928573, 1081285654, 1081174868), 'mode_octal': '0100744', 'mode': 33252, 'size_bytes': 41105488L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000275', 'inode': 527694392L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT11/.(access)(001F0000000000000073FA38)', 'status': ('ok', None), 'volume': 'VO6001', 'outfile': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000275', 'fc': {'status': ('ok', None), 'pnfsid': '001F0000000000000073FA38', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT11/f19', 'deleted': 'no', 'external_label': 'VO6001', 'drive': 'stkenmvr16a:/dev/rmt/tps0d0n:4790000143', 'update': None, 'gid': -1, 'location_cookie': '0000_000000000_0000275', 'complete_crc': 1279529005L, 'address': ('131.225.13.59', 7501), 'uid': -1, 'bfid': 'CDMS108128564600000', 'sanity_cookie': (65536L, 3990704794L), 'size': 41105488L}, 'infilepath': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT11/f19', 'at_the_top': 7, 'bfid': 'CDMS108128564600000', 'local_inode': 31244299L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40504), 'work': 'read_from_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257573542.6823599, 'seek_time': 0.023931980133056641, 'transfer_time': 2.1706759929656982, 't0': 1257573542, 'mount_time': 0.022406101226806641, 'lm_dequeued': 1257573643.1141601, 'job_queued': 1257573542.825985, 'drive_call_time': 0.91053128242492676, 'in_queue': 100.2881190776825, 'drive_transfer_time': 1.126500129699707}, '(trunc. 4156) Thread net_thread 00:00:45 stkenmvr24a.fnal.gov 005803 root I DBT24MV Done is sent Thread net_thread 00:00:45 stkenmvr24a.fnal.gov 005803 root I DBT24MV maybe_clean Thread net_thread 00:00:45 stkendm7a.fnal.gov 004556 root I ENCP Received final dialog for stkendm7a.fnal.gov-1257573542-4556-138. elapsed=3905.750sec Thread READ_1 00:00:45 stkendm7a.fnal.gov 004556 root I ENCP leaving transfer_file(). Thread READ_1 00:00:45 stkenmvr24a.fnal.gov 005803 root I DBT24MV returned from maybe_clean Thread net_thread 00:00:45 stkendm7a.fnal.gov 004556 root I ENCP MSG_TYPE=ENCP_XFER read_from_hsm /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT11/f19 -> /data/data2/Migration_Spool/VO6001:0000_000000000_0000275: 41105488 bytes copied from VO6001 at 17.9 MB/S (33.8 MB/S network) (34.8 MB/S drive) (74.9 MB/S disk) mover=9940B24.mover drive_id=T9940B drive_sn=479002020778 drive_vendor=STK elapsed=102.92 {'storage_group': 'ktev', 'encp_crc': 1279529005L, 'disk_rate': 74.879705860015449, 'drive_rate': 34.799151047941074, 'driver': 'FTTDriver', 'transfer_rate': 17.905052296680648, 'mover_interface': 'stkenmvr24a', 'overall_rate': 16.157034844911699, 'media_changer': 'stk.media_changer', 'network_rate': 17.905052296680648, 'encp_ip': '131.225.13.19', 'unique_id': 'stkendm7a.fnal.gov-1257573542-4556-138'} Thread READ_1 00:00:45 stkendm7a.fnal.gov 004556 root I ENCP File /data/data2/Migration_Spool/VO6001:0000_000000000_0000275 copied successfully. Thread READ_1 00:00:45 stkendm7a.fnal.gov 004556 root I ENCP Completed transferring 41105488 bytes in 1 files in 102.935189962 sec. Overall rate = 16.2 MB/sec. Transfer rate = 17.9 MB/sec. Network rate = 33.8 MB/sec. Drive rate = 34.8 MB/sec. Disk rate = 74.9 MB/sec. Exit status = 0. Thread READ_1 00:00:45 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:45 stkendm7a.fnal.gov 004556 root I ENCP Version: v3_8 CVS $Revision: 1.963 $ encp.pyc OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson), User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Current working directory: stkendm7a.fnal.gov:/opt/enstore/src Command line: encp --delayed-dismount 2 --ignore-fair-share --bypass-filesystem-max-filesize-check --priority 0 --threaded /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT11/f21 /data/data2/Migration_Spool/VO6001:0000_000000000_0000284 Thread READ_1 00:00:45 stkendm7a.fnal.gov 004556 root I ENCP Creating zero length output files. Thread READ_1 00:00:45 stkendm7a.fnal.gov 004556 root I ENCP Sending /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT11/f21 read request to LM: unique_id: stkendm7a.fnal.gov-1257573645-4556-144 inputfile: /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT11/.(access)(001F0000000000000073FCC0) outputfile: /data/data2/Migration_Spool/VO6001:0000_000000000_0000284 Thread READ_1 00:00:45 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:45 stkensrv4n.fnal.gov 005795 enstore I 9940BLM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : ktev 00:00:45 stkensrv4n.fnal.gov 005795 enstore I 9940BLM read Q'd /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT11/f21 (stkendm7a.fnal.gov-1257573645-4556-144) -> /data/data2/Migration_Spool/VO6001:0000_000000000_0000284 : library=CD-9940B family=ktev.ktev_migration.cern requester:root 00:00:45 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30097 current 30096 43 Thread seek_thread 00:00:45 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:45 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=19921686 Thread tape_thread 00:00:46 stkensrv4n.fnal.gov 005795 enstore I 9940BLM 00:00:46 stkensrv4n.fnal.gov 005795 enstore I 9940BLM read_from_hsm next work on vol=VO6001 mover=9940B24.mover requester:root 00:00:46 stkensrv4n.fnal.gov 005795 enstore I 9940BLM HAVE_BOUND:sending read_from_hsm /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZT0/NZT067/f2 to mover 9940B24.mover ('131.225.13.43', 7584) DEL_DISM 2 00:00:46 stkensrv4n.fnal.gov 005795 enstore I 9940BLM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : ktev 00:00:46 stkensrv4n.fnal.gov 005795 enstore I 9940BLM HAVE_BOUND:sending {'vc': {'comment': '', 'declared': 1080687077.0, 'blocksize': 131072, 'storage_group': 'ktev', 'library': 'CD-9940B', 'eod_cookie': '0000_000000000_0000505', 'current_location': '0000_000000000_0000278', 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'migrating'], 'external_label': 'VO6001', 'wrapper': 'cern', 'remaining_bytes': 1167480832L, 'sum_mounts': 112, 'sum_rd_access': 283, 'media_type': '9940B', 'last_access': 1257573642.0, 'status': ('ok', None), 'si_time': [0.0, 1257569739.0], 'capacity_bytes': 214748364800L, 'non_del_files': 168, 'sum_wr_err': 0, 'sum_wr_access': 168, 'file_family': 'ktev_migration', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration.cern', 'modification_time': 1224966315.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1081278425.0}, 'outfilepath': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000278', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 31522036L, 'mover': '9940B24.mover', 'lm': {'address': ('131.225.13.4', 7522)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZT0/NZT067/f2', 'uid': 0, 'uname': 'root', 'gid': 0, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (33188, 527653480L, 20L, 1, 5406, 2539, 31522036L, 1099972193, 1081285666, 1081107960), 'mode_octal': '0100744', 'mode': 33252, 'gname': 'root', 'size_bytes': 31522036L, 'fullname': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000278', 'inode': 527653480L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZT0/NZT067/.(access)(001F00000000000000735A68)', 'status': ('ok', None), 'volume': 'VO6001', 'outfile': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000278', 'fc': {'status': ('ok', None), 'pnfsid': '001F00000000000000735A68', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZT0/NZT067/f2', 'deleted': 'no', 'external_label': 'VO6001', 'drive': 'stkenmvr16a:/dev/rmt/tps0d0n:4790000143', 'update': None, 'gid': -1, 'location_cookie': '0000_000000000_0000278', 'complete_crc': 1340496339L, 'address': ('131.225.13.59', 7501), 'uid': -1, 'bfid': 'CDMS108128566000000', 'sanity_cookie': (65536L, 3966311299L), 'size': 31522036L}, 'infilepath': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZT0/NZT067/f2', 'at_the_top': 10, 'bfid': 'CDMS108128566000000', 'local_inode': 31244300L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40529), 'work': 'read_from_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573646.117142, 'encp_start_time': 1257573643.6699891, 'job_queued': 1257573643.8793011, 't0': 1257573643, 'in_queue': 2.2374088764190674}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573643-4556-142'} to mover 00:00:46 stkensrv4n.fnal.gov 005795 enstore I 9940BLM 00:00:46 stkenmvr24a.fnal.gov 005803 root I DBT24MV READ FROM HSM Thread MainThread 00:00:46 stkenmvr24a.fnal.gov 005803 root I DBT24MV SENDING {'vc': {'comment': '', 'declared': 1080687077.0, 'blocksize': 131072, 'storage_group': 'ktev', 'library': 'CD-9940B', 'eod_cookie': '0000_000000000_0000505', 'current_location': '0000_000000000_0000278', 'user_inhibit': ['none', 'none'], 'sum_rd_access': 283, 'system_inhibit': ['none', 'migrating'], 'external_label': 'VO6001', 'wrapper': 'cern', 'remaining_bytes': 1167480832L, 'sum_mounts': 112, 'capacity_bytes': 214748364800L, 'media_type': '9940B', 'last_access': 1257573642.0, 'status': ('ok', None), 'si_time': [0.0, 1257569739.0], 'non_del_files': 168, 'sum_wr_err': 0, 'sum_wr_access': 168, 'file_family': 'ktev_migration', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration.cern', 'modification_time': 1224966315.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1081278425.0}, 'outfilepath': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000278', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 31522036L, 'mover': {'max_buffer': 1048576000L, 'check_written_file': 10, 'max_failures': 5, 'library': 'CD-9940B.library_manager', 'local_mover': 0, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7584, 'mc_device': '0,0,10,5', 'compression': 0, 'serial_num': '479002020778', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 28311552.0, 'vendor_id': 'STK', 'driver': 'FTTDriver', 'host': 'stkenmvr24a', 'max_time_in_state': 1200, 'device': 'stkenmvr24a:/dev/rmt/tps0d0n', 'name': '9940B24.mover', 'send_stats': 1, 'product_id': 'T9940B', 'statistics_path': '/tmp/enstore/enstore/DBT24MV.stat', 'mover_address': ('131.225.13.43', 7584), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.43', 57433), 'data_ip': 'stkenmvr24a', 'logname': 'DBT24MV', 'hostip': '131.225.13.43', 'do_cleaning': 'No', 'media_changer': 'stk.media_changer'}, 'lm': {'address': ('131.225.13.4', 7522)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZT0/NZT067/f2', 'uid': 0, 'uname': 'root', 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (33188, 527653480L, 20L, 1, 5406, 2539, 31522036L, 1099972193, 1081285666, 1081107960), 'mode_octal': '0100744', 'mode': 33252, 'size_bytes': 31522036L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000278', 'inode': 527653480L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZT0/NZT067/.(access)(001F00000000000000735A68)', 'status': ('ok', None), 'volume': 'VO6001', 'outfile': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000278', 'fc': {'status': ('ok', None), 'pnfsid': '001F00000000000000735A68', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZT0/NZT067/f2', 'deleted': 'no', 'external_label': 'VO6001', 'drive': 'stkenmvr16a:/dev/rmt/tps0d0n:4790000143', 'update': None, 'gid': -1, 'location_cookie': '0000_000000000_0000278', 'complete_crc': 1340496339L, 'address': ('131.225.13.59', 7501), 'uid': -1, 'bfid': 'CDMS108128566000000', 'sanity_cookie': (65536L, 3966311299L), 'size': 31522036L}, 'infilepath': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZT0/NZT067/f2', 'at_the_top': 10, 'bfid': 'CDMS108128566000000', 'local_inode': 31244300L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40529), 'work': 'read_from_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573646.117142, 'encp_start_time': 1257573643.6699891, 'job_queued': 1257573643.8793011, 't0': 1257573643, 'in_queue': 2.2374088764190674}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573643-4556-142'} Thread client_connect_thread 00:00:46 stkendm7a.fnal.gov 004556 root I ENCP Listening for control socket at: ('131.225.13.19', 40529) Thread READ_2 00:00:46 stkendm7a.fnal.gov 004556 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.19', 40529) elapsed=3906.490sec Thread READ_2 00:00:46 stkendm7a.fnal.gov 004556 root I ENCP Control socket ('131.225.13.19', 40529) is connected to ('131.225.13.43', 57434) for stkendm7a.fnal.gov-1257573643-4556-142. elapsed=3906.490sec Thread READ_2 00:00:46 stkendm7a.fnal.gov 004556 root I ENCP Received callback ticket from mover 9940B24.mover for transfer stkendm7a.fnal.gov-1257573643-4556-142. elapsed=3906.500sec Thread READ_2 00:00:46 stkendm7a.fnal.gov 004556 root I ENCP Attempting to connect data socket to mover at ('131.225.13.43', 57433). Thread READ_2 00:00:46 stkendm7a.fnal.gov 004556 root I ENCP Connecting data socket to mover (('131.225.13.43', 57433)) with interface 131.225.13.19. elapsed=3906.500sec Thread READ_2 00:00:46 stkendm7a.fnal.gov 004556 root I ENCP Data socket ('131.225.13.19', 40534) is connected to ('131.225.13.43', 57433). elapsed=3906.500sec Thread READ_2 00:00:46 stkendm7a.fnal.gov 004556 root I ENCP Opening local file. Thread READ_2 00:00:46 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 19921825/19921825, blocks 152 header 139 Thread tape_thread 00:00:46 stkenmvr24a.fnal.gov 005803 root I DBT24MV tape position: target 278 current 275 314 Thread seek_thread 00:00:46 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 19921686/19921686 bytes Thread net_thread 00:00:46 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:46 stkenmvr24a.fnal.gov 005803 root I DBT24MV skip_fm 3 Thread seek_thread 00:00:46 stkenmvr24a.fnal.gov 005803 root I DBT24MV read_tape starting, bytes_to_read=31522036 Thread tape_thread 00:00:46 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30097 block 152 block_size 0 bloc_loc 5649656 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 19921825 block_read_tot 0.193316221237 tape_rate 103053043.725 Thread tape_thread 00:00:46 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 19921825/19921825 bytes Thread tape_thread 00:00:46 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00188326835632 read 0.0920066833496 crc_check 0.0981073379517 push 0.00131893157959 block_read 0.193316221237 idle 0.00386476516724 Thread tape_thread 00:00:47 stkensrv4n.fnal.gov 005795 enstore I 9940BLM 00:00:47 stkenmvr24a.fnal.gov 005803 root I DBT24MV read bytes 31522036/31522036, blocks 241 header 0 Thread tape_thread 00:00:47 stkenmvr24a.fnal.gov 005803 root I DBT24MV write_client: wrote 31522036/31522036 bytes Thread net_thread 00:00:47 stkenmvr24a.fnal.gov 005803 root I DBT24MV write_client waits for tape thread to exit Thread net_thread 00:00:47 stkenmvr24a.fnal.gov 005803 root I DBT24MV drive stats after read. Tape VO6001 position 278 block 241 block_size 0 bloc_loc 847844 tot_blocks 0 BOT 0 read_err 0 write_err 0 bytes 31522036 block_read_tot 0.86668586731 tape_rate 36370774.2205 Thread tape_thread 00:00:47 stkenmvr24a.fnal.gov 005803 root I DBT24MV read_tape exiting, read 31522036/31522036 bytes Thread tape_thread 00:00:47 stkenmvr24a.fnal.gov 005803 root I DBT24MV read_tape timing:get_space 0.00371646881104 read 0.713885784149 crc_check 0.144418239594 push 0.00466537475586 block_read 0.86668586731 idle 0.00881719589233 Thread tape_thread 00:00:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30098 current 30097 152 Thread seek_thread 00:00:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=5476509 Thread tape_thread 00:00:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 5476648/5476648, blocks 42 header 139 Thread tape_thread 00:00:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 5476509/5476509 bytes Thread net_thread 00:00:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30098 block 42 block_size 0 bloc_loc 5649699 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 5476648 block_read_tot 0.0685024261475 tape_rate 79948234.07 Thread tape_thread 00:00:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 5476648/5476648 bytes Thread tape_thread 00:00:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00094199180603 read 0.0401237010956 crc_check 0.0270674228668 push 0.000369310379028 block_read 0.0685024261475 idle 0.00112128257751 Thread tape_thread 00:00:47 stkendm7a.fnal.gov 004556 root I ENCP File /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZT0/NZT067/f2 transfered for stkendm7a.fnal.gov-1257573643-4556-142 in 1.22150301933 seconds. Thread READ_2 00:00:47 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:48 stkenmvr24a.fnal.gov 005803 root I DBT24MV write_client exits Thread net_thread 00:00:48 stkenmvr24a.fnal.gov 005803 root I DBT24MV transfer complete volume=VO6001 location=278 Thread net_thread 00:00:48 last message repeated 1 times 00:00:48 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_rd_access = '285', last_access = '2009-11-07 00:00:48' where label = 'VO6001'; 00:00:48 stkenmvr24a.fnal.gov 005803 root I DBT24MV Sending done to client: {'vc': {'comment': '', 'declared': 1080687077.0, 'blocksize': 131072, 'storage_group': 'ktev', 'library': 'CD-9940B', 'eod_cookie': '0000_000000000_0000505', 'current_location': '0000_000000000_0000278', 'user_inhibit': ['none', 'none'], 'sum_rd_access': 284, 'system_inhibit': ['none', 'migrating'], 'external_label': 'VO6001', 'wrapper': 'cern', 'remaining_bytes': 1167480832L, 'sum_mounts': 112, 'capacity_bytes': 214748364800L, 'media_type': '9940B', 'last_access': 1257573645.0, 'status': ('ok', None), 'si_time': [0.0, 1257569739.0], 'non_del_files': 168, 'sum_wr_err': 0, 'sum_wr_access': 168, 'file_family': 'ktev_migration', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration.cern', 'modification_time': 1224966315.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1081278425.0}, 'outfilepath': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000278', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 31522036L, 'mover': {'max_buffer': 1048576000L, 'check_written_file': 10, 'max_failures': 5, 'library': 'CD-9940B.library_manager', 'local_mover': 0, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7584, 'mc_device': '0,0,10,5', 'compression': 0, 'serial_num': '479002020778', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 28311552.0, 'vendor_id': 'STK', 'driver': 'FTTDriver', 'host': 'stkenmvr24a', 'max_time_in_state': 1200, 'device': 'stkenmvr24a:/dev/rmt/tps0d0n', 'name': '9940B24.mover', 'send_stats': 1, 'product_id': 'T9940B', 'statistics_path': '/tmp/enstore/enstore/DBT24MV.stat', 'mover_address': ('131.225.13.43', 7584), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.43', 57433), 'data_ip': 'stkenmvr24a', 'logname': 'DBT24MV', 'hostip': '131.225.13.43', 'do_cleaning': 'No', 'media_changer': 'stk.media_changer'}, 'lm': {'address': ('131.225.13.4', 7522)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZT0/NZT067/f2', 'uid': 0, 'uname': 'root', 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (33188, 527653480L, 20L, 1, 5406, 2539, 31522036L, 1099972193, 1081285666, 1081107960), 'mode_octal': '0100744', 'mode': 33252, 'size_bytes': 31522036L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000278', 'inode': 527653480L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZT0/NZT067/.(access)(001F00000000000000735A68)', 'status': ('ok', None), 'volume': 'VO6001', 'outfile': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000278', 'fc': {'status': ('ok', None), 'pnfsid': '001F00000000000000735A68', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZT0/NZT067/f2', 'deleted': 'no', 'external_label': 'VO6001', 'drive': 'stkenmvr16a:/dev/rmt/tps0d0n:4790000143', 'update': None, 'gid': -1, 'location_cookie': '0000_000000000_0000278', 'complete_crc': 1340496339L, 'address': ('131.225.13.59', 7501), 'uid': -1, 'bfid': 'CDMS108128566000000', 'sanity_cookie': (65536L, 3966311299L), 'size': 31522036L}, 'infilepath': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZT0/NZT067/f2', 'at_the_top': 10, 'bfid': 'CDMS108128566000000', 'local_inode': 31244300L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40529), 'work': 'read_from_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257573643.6699891, 'seek_time': 0.023908853530883789, 'transfer_time': 1.9060611724853516, 't0': 1257573643, 'mount_time': 0.022130966186523438, 'lm_dequeued': 1257573646.117142, 'job_queued': 1257573643.8793011, 'drive_call_time': 0.70608067512512207, 'in_queue': 2.2374088764190674, 'drive_transfer_time': 0.87106060981750488}, (trunc. 4157) Thread net_thread 00:00:48 stkenmvr24a.fnal.gov 005803 root I DBT24MV Done is sent Thread net_thread 00:00:48 stkenmvr24a.fnal.gov 005803 root I DBT24MV maybe_clean Thread net_thread 00:00:48 stkendm7a.fnal.gov 004556 root I ENCP Received final dialog for stkendm7a.fnal.gov-1257573643-4556-142. elapsed=3908.490sec Thread READ_2 00:00:48 stkenmvr24a.fnal.gov 005803 root I DBT24MV returned from maybe_clean Thread net_thread 00:00:48 stkendm7a.fnal.gov 004556 root I ENCP leaving transfer_file(). Thread READ_2 00:00:48 stkendm7a.fnal.gov 004556 root I ENCP MSG_TYPE=ENCP_XFER read_from_hsm /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZT0/NZT067/f2 -> /data/data2/Migration_Spool/VO6001:0000_000000000_0000278: 31522036 bytes copied from VO6001 at 15.6 MB/S (34 MB/S network) (34.5 MB/S drive) (55.1 MB/S disk) mover=9940B24.mover drive_id=T9940B drive_sn=479002020778 drive_vendor=STK elapsed=4.6643 {'storage_group': 'ktev', 'encp_crc': 1340496339L, 'disk_rate': 55.084493090021873, 'drive_rate': 34.511669791073913, 'driver': 'FTTDriver', 'transfer_rate': 15.578494516311451, 'mover_interface': 'stkenmvr24a', 'overall_rate': 13.974613464747691, 'media_changer': 'stk.media_changer', 'network_rate': 15.578494516311451, 'encp_ip': '131.225.13.19', 'unique_id': 'stkendm7a.fnal.gov-1257573643-4556-142'} Thread READ_2 00:00:48 stkendm7a.fnal.gov 004556 root I ENCP File /data/data2/Migration_Spool/VO6001:0000_000000000_0000278 copied successfully. Thread READ_2 00:00:48 stkendm7a.fnal.gov 004556 root I ENCP Completed transferring 31522036 bytes in 1 files in 4.67716884613 sec. Overall rate = 14 MB/sec. Transfer rate = 15.6 MB/sec. Network rate = 34 MB/sec. Drive rate = 34.5 MB/sec. Disk rate = 55.1 MB/sec. Exit status = 0. Thread READ_2 00:00:48 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:48 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30099 current 30098 42 Thread seek_thread 00:00:48 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:48 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=55061456 Thread tape_thread 00:00:48 stkendm7a.fnal.gov 004556 root I ENCP Version: v3_8 CVS $Revision: 1.963 $ encp.pyc OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson), User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Current working directory: stkendm7a.fnal.gov:/opt/enstore/src Command line: encp --delayed-dismount 2 --ignore-fair-share --bypass-filesystem-max-filesize-check --priority 0 --threaded /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT02/f3 /data/data2/Migration_Spool/VO6001:0000_000000000_0000287 Thread READ_2 00:00:48 stkendm7a.fnal.gov 004556 root I ENCP Creating zero length output files. Thread READ_2 00:00:48 stkendm7a.fnal.gov 004556 root I ENCP Sending /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT02/f3 read request to LM: unique_id: stkendm7a.fnal.gov-1257573648-4556-145 inputfile: /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT02/.(access)(001F0000000000000073C348) outputfile: /data/data2/Migration_Spool/VO6001:0000_000000000_0000287 Thread READ_2 00:00:48 stkensrv4n.fnal.gov 005795 enstore I 9940BLM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : ktev 00:00:48 stkensrv4n.fnal.gov 005795 enstore I 9940BLM read Q'd /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT02/f3 (stkendm7a.fnal.gov-1257573648-4556-145) -> /data/data2/Migration_Spool/VO6001:0000_000000000_0000287 : library=CD-9940B family=ktev.ktev_migration.cern requester:root 00:00:48 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 55061595/55061595, blocks 421 header 139 Thread tape_thread 00:00:48 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 55061456/55061456 bytes Thread net_thread 00:00:48 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:49 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30099 block 421 block_size 0 bloc_loc 5650121 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 55061595 block_read_tot 0.496838092804 tape_rate 110824020.536 Thread tape_thread 00:00:49 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 55061595/55061595 bytes Thread tape_thread 00:00:49 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00543832778931 read 0.216226577759 crc_check 0.27162361145 push 0.00354957580566 block_read 0.496838092804 idle 0.0100162029266 Thread tape_thread 00:00:49 stkensrv4n.fnal.gov 005795 enstore I 9940BLM 00:00:49 stkensrv4n.fnal.gov 005795 enstore I 9940BLM read_from_hsm next work on vol=VO6001 mover=9940B24.mover requester:root 00:00:49 stkensrv4n.fnal.gov 005795 enstore I 9940BLM HAVE_BOUND:sending read_from_hsm /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZN0/NZN083/f4 to mover 9940B24.mover ('131.225.13.43', 7584) DEL_DISM 2 00:00:49 stkensrv4n.fnal.gov 005795 enstore I 9940BLM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : ktev 00:00:49 stkensrv4n.fnal.gov 005795 enstore I 9940BLM HAVE_BOUND:sending {'vc': {'comment': '', 'declared': 1080687077.0, 'blocksize': 131072, 'storage_group': 'ktev', 'library': 'CD-9940B', 'eod_cookie': '0000_000000000_0000505', 'current_location': '0000_000000000_0000281', 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'migrating'], 'external_label': 'VO6001', 'wrapper': 'cern', 'remaining_bytes': 1167480832L, 'sum_mounts': 112, 'sum_rd_access': 283, 'media_type': '9940B', 'last_access': 1257573642.0, 'status': ('ok', None), 'si_time': [0.0, 1257569739.0], 'capacity_bytes': 214748364800L, 'non_del_files': 168, 'sum_wr_err': 0, 'sum_wr_access': 168, 'file_family': 'ktev_migration', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration.cern', 'modification_time': 1224966315.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1081278425.0}, 'outfilepath': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000281', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 2684704932L, 'mover': '9940B24.mover', 'lm': {'address': ('131.225.13.4', 7522)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZN0/NZN083/f4', 'uid': 0, 'uname': 'root', 'gid': 0, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (33188, 527663224L, 20L, 1, 5406, 2539, 1L, 1224964201, 1081285810, 1081128910), 'mode_octal': '0100744', 'mode': 33252, 'gname': 'root', 'size_bytes': 2684704932L, 'fullname': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000281', 'inode': 527663224L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZN0/NZN083/.(access)(001F00000000000000738078)', 'status': ('ok', None), 'volume': 'VO6001', 'outfile': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000281', 'fc': {'status': ('ok', None), 'pnfsid': '001F00000000000000738078', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZN0/NZN083/f4', 'deleted': 'no', 'external_label': 'VO6001', 'drive': 'stkenmvr16a:/dev/rmt/tps0d0n:4790000143', 'update': None, 'gid': -1, 'location_cookie': '0000_000000000_0000281', 'complete_crc': 2461987128L, 'address': ('131.225.13.59', 7501), 'uid': -1, 'bfid': 'CDMS108128580800000', 'sanity_cookie': (65536L, 4272572212L), 'size': 2684704932L}, 'infilepath': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZN0/NZN083/f4', 'at_the_top': 8, 'bfid': 'CDMS108128580800000', 'local_inode': 31244301L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40531), 'work': 'read_from_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573649.121706, 'encp_start_time': 1257573643.834518, 'job_queued': 1257573644.0482581, 't0': 1257573643, 'in_queue': 5.0730140209197998}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573643-4556-143'} to mover 00:00:49 stkensrv4n.fnal.gov 005795 enstore I 9940BLM 00:00:49 stkenmvr24a.fnal.gov 005803 root I DBT24MV READ FROM HSM Thread MainThread 00:00:49 stkenmvr24a.fnal.gov 005803 root I DBT24MV SENDING {'vc': {'comment': '', 'declared': 1080687077.0, 'blocksize': 131072, 'storage_group': 'ktev', 'library': 'CD-9940B', 'eod_cookie': '0000_000000000_0000505', 'current_location': '0000_000000000_0000281', 'user_inhibit': ['none', 'none'], 'sum_rd_access': 283, 'system_inhibit': ['none', 'migrating'], 'external_label': 'VO6001', 'wrapper': 'cern', 'remaining_bytes': 1167480832L, 'sum_mounts': 112, 'capacity_bytes': 214748364800L, 'media_type': '9940B', 'last_access': 1257573642.0, 'status': ('ok', None), 'si_time': [0.0, 1257569739.0], 'non_del_files': 168, 'sum_wr_err': 0, 'sum_wr_access': 168, 'file_family': 'ktev_migration', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration.cern', 'modification_time': 1224966315.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1081278425.0}, 'outfilepath': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000281', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 2684704932L, 'mover': {'max_buffer': 1048576000L, 'check_written_file': 10, 'max_failures': 5, 'library': 'CD-9940B.library_manager', 'local_mover': 0, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7584, 'mc_device': '0,0,10,5', 'compression': 0, 'serial_num': '479002020778', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 28311552.0, 'vendor_id': 'STK', 'driver': 'FTTDriver', 'host': 'stkenmvr24a', 'max_time_in_state': 1200, 'device': 'stkenmvr24a:/dev/rmt/tps0d0n', 'name': '9940B24.mover', 'send_stats': 1, 'product_id': 'T9940B', 'statistics_path': '/tmp/enstore/enstore/DBT24MV.stat', 'mover_address': ('131.225.13.43', 7584), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.43', 57435), 'data_ip': 'stkenmvr24a', 'logname': 'DBT24MV', 'hostip': '131.225.13.43', 'do_cleaning': 'No', 'media_changer': 'stk.media_changer'}, 'lm': {'address': ('131.225.13.4', 7522)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZN0/NZN083/f4', 'uid': 0, 'uname': 'root', 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (33188, 527663224L, 20L, 1, 5406, 2539, 1L, 1224964201, 1081285810, 1081128910), 'mode_octal': '0100744', 'mode': 33252, 'size_bytes': 2684704932L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000281', 'inode': 527663224L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZN0/NZN083/.(access)(001F00000000000000738078)', 'status': ('ok', None), 'volume': 'VO6001', 'outfile': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000281', 'fc': {'status': ('ok', None), 'pnfsid': '001F00000000000000738078', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZN0/NZN083/f4', 'deleted': 'no', 'external_label': 'VO6001', 'drive': 'stkenmvr16a:/dev/rmt/tps0d0n:4790000143', 'update': None, 'gid': -1, 'location_cookie': '0000_000000000_0000281', 'complete_crc': 2461987128L, 'address': ('131.225.13.59', 7501), 'uid': -1, 'bfid': 'CDMS108128580800000', 'sanity_cookie': (65536L, 4272572212L), 'size': 2684704932L}, 'infilepath': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZN0/NZN083/f4', 'at_the_top': 8, 'bfid': 'CDMS108128580800000', 'local_inode': 31244301L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40531), 'work': 'read_from_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573649.121706, 'encp_start_time': 1257573643.834518, 'job_queued': 1257573644.0482581, 't0': 1257573643, 'in_queue': 5.0730140209197998}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573643-4556-143'} Thread client_connect_thread 00:00:49 stkendm7a.fnal.gov 004556 root I ENCP Listening for control socket at: ('131.225.13.19', 40531) Thread READ 00:00:49 stkendm7a.fnal.gov 004556 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.19', 40531) elapsed=3909.490sec Thread READ 00:00:49 stkendm7a.fnal.gov 004556 root I ENCP Control socket ('131.225.13.19', 40531) is connected to ('131.225.13.43', 57436) for stkendm7a.fnal.gov-1257573643-4556-143. elapsed=3909.500sec Thread READ 00:00:49 stkendm7a.fnal.gov 004556 root I ENCP Received callback ticket from mover 9940B24.mover for transfer stkendm7a.fnal.gov-1257573643-4556-143. elapsed=3909.510sec Thread READ 00:00:49 stkendm7a.fnal.gov 004556 root I ENCP Attempting to connect data socket to mover at ('131.225.13.43', 57435). Thread READ 00:00:49 stkendm7a.fnal.gov 004556 root I ENCP Connecting data socket to mover (('131.225.13.43', 57435)) with interface 131.225.13.19. elapsed=3909.510sec Thread READ 00:00:49 stkendm7a.fnal.gov 004556 root I ENCP Data socket ('131.225.13.19', 40537) is connected to ('131.225.13.43', 57435). elapsed=3909.510sec Thread READ 00:00:49 stkendm7a.fnal.gov 004556 root I ENCP Opening local file. Thread READ 00:00:49 stkenmvr24a.fnal.gov 005803 root I DBT24MV tape position: target 281 current 278 241 Thread seek_thread 00:00:49 stkenmvr24a.fnal.gov 005803 root I DBT24MV skip_fm 3 Thread seek_thread 00:00:49 stkenmvr24a.fnal.gov 005803 root I DBT24MV read_tape starting, bytes_to_read=2684704932 Thread tape_thread 00:00:49 stkendm7a.fnal.gov 004556 root I ENCP File /data/data2/Migration_Spool/VO6001:0000_000000000_0000143 transfered for stkendm7a.fnal.gov-1257573558-4556-139 in 24.8071889877 seconds. Thread WRITE_2 00:00:49 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:49 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:50 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30100 current 30099 421 Thread seek_thread 00:00:50 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:50 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=25585575 Thread tape_thread 00:00:50 stkensrv4n.fnal.gov 005795 enstore I 9940BLM 00:00:50 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 25585714/25585714, blocks 196 header 139 Thread tape_thread 00:00:50 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 25585575/25585575 bytes Thread net_thread 00:00:50 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:50 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30100 block 196 block_size 0 bloc_loc 5650318 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 25585714 block_read_tot 0.242220163345 tape_rate 105629992.345 Thread tape_thread 00:00:50 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 25585714/25585714 bytes Thread tape_thread 00:00:50 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00275921821594 read 0.111324548721 crc_check 0.126493215561 push 0.00164318084717 block_read 0.242220163345 idle 0.0049102306366 Thread tape_thread 00:00:50 stkensrv4n.fnal.gov 032104 enstore I SL8500G1MC mount VON267 0,5,1,14 => 0,Mount: VON267 mounted on 0, 5, 1,14 00:00:50 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC FINISHED mount VON267 0,5,1,14 returned ('ok', 0, 'mount VON267 0,5,1,14 => 0,Mount: VON267 mounted on 0, 5, 1,14', '', '') 00:00:50 last message repeated 1 times 00:00:50 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_mounts = '149', last_access = '2009-11-07 00:00:50' where label = 'VON267'; 00:00:50 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV MSG_TYPE=MC_LOAD_DONE mounted VON267 ULTRIUM-TD4 00:00:50 Thread media_thread 00:00:51 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:51 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30101 current 30100 196 Thread seek_thread 00:00:51 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:51 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=9460733 Thread tape_thread 00:00:51 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 9460872/9460872, blocks 73 header 139 Thread tape_thread 00:00:51 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 9460733/9460733 bytes Thread net_thread 00:00:51 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:51 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30101 block 73 block_size 0 bloc_loc 5650392 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 9460872 block_read_tot 0.102451086044 tape_rate 92345258.2622 Thread tape_thread 00:00:51 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 9460872/9460872 bytes Thread tape_thread 00:00:51 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00152206420898 read 0.0534560680389 crc_check 0.0468707084656 push 0.000602245330811 block_read 0.102451086044 idle 0.00163531303406 Thread tape_thread 00:00:51 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:52 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:52 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30102 current 30101 73 Thread seek_thread 00:00:52 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:52 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=14215299 Thread tape_thread 00:00:52 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 14215438/14215438, blocks 109 header 139 Thread tape_thread 00:00:52 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV written bytes 1259824128/1259824128, blocks 9612 header 0 trailer 380 Thread tape_thread 00:00:52 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 14215299/14215299 bytes Thread net_thread 00:00:52 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:52 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30102 block 109 block_size 0 bloc_loc 5650502 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 14215438 block_read_tot 0.143361330032 tape_rate 99158106.2815 Thread tape_thread 00:00:52 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 14215438/14215438 bytes Thread tape_thread 00:00:52 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00145316123962 read 0.0710518360138 crc_check 0.0699212551117 push 0.000935077667236 block_read 0.143361330032 idle 0.00261092185974 Thread tape_thread 00:00:53 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:53 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:53 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30103 current 30102 109 Thread seek_thread 00:00:53 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:53 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=19943479 Thread tape_thread 00:00:54 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 19943618/19943618, blocks 153 header 139 Thread tape_thread 00:00:54 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 19943479/19943479 bytes Thread net_thread 00:00:54 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:54 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30103 block 153 block_size 0 bloc_loc 5650656 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 19943618 block_read_tot 0.195976018906 tape_rate 101765604.34 Thread tape_thread 00:00:54 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 19943618/19943618 bytes Thread tape_thread 00:00:54 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00224328041077 read 0.0937511920929 crc_check 0.0986661911011 push 0.0013153553009 block_read 0.195976018906 idle 0.00361657142639 Thread tape_thread 00:00:54 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:00:55 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:55 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30104 current 30103 153 Thread seek_thread 00:00:55 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:55 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=15102794 Thread tape_thread 00:00:55 stkendca20a.fnal.gov 024294 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: exp-db File Family: daily-d0-offline FF Wrapper: cpio_odc FF Width: 2 Current working directory: stkendca20a.fnal.gov:/home/enstore/dcache-code/dcache-fermi-config Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel02/D0OFPRD1_t702242164_s103340_p26 --put-cache 003000000000000006222A38 /diskb/write-pool-2/data/003000000000000006222A38 00:00:55 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 15102933/15102933, blocks 116 header 139 Thread tape_thread 00:00:55 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 15102794/15102794 bytes Thread net_thread 00:00:55 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:55 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30104 block 116 block_size 0 bloc_loc 5650773 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 15102933 block_read_tot 0.151587963104 tape_rate 99631479.2462 Thread tape_thread 00:00:55 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 15102933/15102933 bytes Thread tape_thread 00:00:55 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00182461738586 read 0.0739657878876 crc_check 0.074832201004 push 0.000965356826782 block_read 0.151587963104 idle 0.00261807441711 Thread tape_thread 00:00:55 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:55 stkendca20a.fnal.gov 024294 root I ENCP Creating zero length output files. 00:00:55 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:55 stkendca20a.fnal.gov 024294 root I ENCP Sending /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel02/D0OFPRD1_t702242164_s103340_p26 write request to LM: unique_id: stkendca20a.fnal.gov-1257573655-24294-0 inputfile: /diskb/write-pool-2/data/003000000000000006222A38 outputfile: /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel02/.(access)(003000000000000006222A38) 00:00:55 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : exp-db 00:00:55 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /diskb/write-pool-2/data/003000000000000006222A38 (stkendca20a.fnal.gov-1257573655-24294-0) -> /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel02/D0OFPRD1_t702242164_s103340_p26 : library=CD-LTO4G1 family=daily-d0-offline requester:root volume_family:exp-db.daily-d0-offline.cpio_odc 00:00:55 stkendca20a.fnal.gov 024294 root I ENCP Listening for control socket at: ('131.225.13.12', 49623) 00:00:55 stkendca20a.fnal.gov 024294 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.12', 49623) elapsed=1.690sec 00:00:56 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:56 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30105 current 30104 116 Thread seek_thread 00:00:56 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:56 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=54618631 Thread tape_thread 00:00:56 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 54618770/54618770, blocks 417 header 139 Thread tape_thread 00:00:56 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 54618631/54618631 bytes Thread net_thread 00:00:56 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:56 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30105 block 417 block_size 0 bloc_loc 5651191 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 54618770 block_read_tot 0.492933034897 tape_rate 110803630.784 Thread tape_thread 00:00:56 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 54618770/54618770 bytes Thread tape_thread 00:00:56 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00465822219849 read 0.214948177338 crc_check 0.269856452942 push 0.00347018241882 block_read 0.492933034897 idle 0.00936007499695 Thread tape_thread 00:00:57 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV filemarks written. Tape VOV152 absolute location in blocks 441721 Thread tape_thread 00:00:57 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV write_tape timing:pull 0.103267908096 write 5.67481780052 crc_check 5.80626368523 freespace 0.100877046585 block_write 11.6852264404 idle 0.169587612152 Thread tape_thread 00:00:57 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV drive stats after write. Tape VOV152 position 250 block 0 block_size 0 bloc_loc 441721 tot_blocks 762939000 BOT 0 read_err 0 write_err 6 bytes 1259824128 block_write_tot 11.6852264404 tape_rate 107813411.612 Thread tape_thread 00:00:57 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV remaining bytes info in DB 721863680000 estimated 720603855872 reported from drive 720652288000 position 253 Thread tape_thread 00:00:57 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV new bitfile request {'uid': 0, 'external_label': 'VOV152', 'gid': 0, 'location_cookie': '0000_000000000_0000251', 'complete_crc': 56258062L, 'sanity_cookie': (65536L, 1949751648L), 'size': 1259823748L} Thread tape_thread 00:00:57 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV set remaining: VOV152 720652288000 0000_000000000_0000253 Thread tape_thread 00:00:57 last message repeated 1 times 00:00:57 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-07 00:00:57', eod_cookie = '0000_000000000_0000253', non_del_files = '84', remaining_bytes = 720652288000 where label = 'VOV152'; 00:00:57 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV transfer complete volume=VOV152 location=253 Thread tape_thread 00:00:57 last message repeated 1 times 00:00:57 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '84', modification_time = '2009-11-07 00:00:57' where label = 'VOV152'; 00:00:57 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'ktev', 'blocksize': 131072, 'declared': 1254165880.0, 'library': 'CD-LTO4F1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOV152', 'wrapper': 'cern', 'remaining_bytes': 721863680000L, 'sum_mounts': 1, 'sum_rd_access': 0, 'file_family_width': 1, 'last_access': 1257573623.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000250', 'capacity_bytes': 858993459200L, 'non_del_files': 83, 'sum_wr_err': 0, 'sum_wr_access': 83, 'file_family': 'ktev_migration-MIGRATION', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'modification_time': 1257573623.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257571652.0}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NZ/NZN0/NZN088/f3', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 1259823748L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4F1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7875, 'mc_device': '1,1,1,11', 'compression': 0, 'serial_num': '1310051338', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr215a', 'max_time_in_state': 1200, 'device': 'stkenmvr215a:/dev/rmt/tps0d0n', 'name': 'LTO4_115.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_115.stat', 'mover_address': ('131.225.13.111', 7875), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.111', 39677), 'data_ip': 'stkenmvr215a', 'logname': 'LTO4_115MV', 'hostip': '131.225.13.111', 'do_cleaning': 'No', 'media_changer': 'SL8500F1.media_changer'}, 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZN0/NZN088/f3', 'uid': 0, 'uname': 'root', 'mtime': 1257573558, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cern', 'pstat': (16877, 684747552L, 20L, 1, 0, 0, 512L, 1255843636, 1255843636, 1255251459), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 1259823748L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000143', 'inode': 675905472L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000143', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NZ/NZN0/NZN088/.(access)(002800000000000008497FC0)', 'fc': {'uid': 0, 'external_label': 'VOV152', 'gid': 0, 'location_cookie': '0000_000000000_0000251', 'complete_crc': 56258062L, 'bfid': 'CDMS125757365700000', 'sanity_cookie': (65536L, 1949751648L), 'size': 1259823748L}, 'infilepath': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000143', 'at_the_top': 99, 'local_inode': 31244313L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40508), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257573558.4021201, 'seek_time': 0.035459041595458984, 'transfer_time': 32.774789094924927, 't0': 1257573558, 'mount_time': 0.050221920013427734, 'lm_dequeued': 1257573624.656059, 'job_queued': 1257573558.690475, 'drive_call_time': 5.5574498176574707, 'in_queue': 65.965303897857666, 'drive_transfer_time': 18.584547996520996}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573558-4556-139'} Thread tape_thread 00:00:57 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV Done is sent Thread tape_thread 00:00:57 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV maybe_clean Thread tape_thread 00:00:57 stkendm7a.fnal.gov 004556 root I ENCP Received final dialog for stkendm7a.fnal.gov-1257573558-4556-139. elapsed=3917.930sec Thread WRITE_2 00:00:57 stkendm7a.fnal.gov 004556 root I ENCP leaving transfer_file(). Thread WRITE_2 00:00:57 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV returned from maybe_clean Thread tape_thread 00:00:57 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:57 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM write_to_hsm next work on vol=VOV152 mover=LTO4_115.mover requester:root 00:00:57 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM HAVE_BOUND:sending write_to_hsm /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT09/f6 to mover LTO4_115.mover ('131.225.13.111', 7875) DEL_DISM 2 00:00:57 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : ktev 00:00:57 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM HAVE_BOUND:sending {'vc': {'storage_group': 'ktev', 'external_label': 'VOV152', 'library': 'CD-LTO4F1', 'file_family': 'ktev_migration-MIGRATION', 'wrapper': 'cern', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'file_family_width': 1}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NZ/NZNT/NZNT09/f6', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 329475780L, 'mover': 'LTO4_115.mover', 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT09/f6', 'uid': 0, 'uname': 'root', 'type': 'cern', 'gid': 0, 'mtime': 1257573564, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (16877, 684350712L, 20L, 1, 0, 0, 512L, 1257573165, 1257573165, 1255198270), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 329475780L, 'fullname': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000146', 'inode': 675905520L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000146', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NZ/NZNT/NZNT09/.(access)(002800000000000008497FF0)', 'fc': {'size': 329475780L, 'external_label': 'VOV152', 'pnfsid': '002800000000000008497FF0', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000146', 'at_the_top': 150, 'local_inode': 31244314L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40511), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573657.6673989, 'encp_start_time': 1257573564.5066569, 'job_queued': 1257573564.7983661, 't0': 1257573564, 'in_queue': 92.86873984336853}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573564-4556-140'} to mover 00:00:57 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:57 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:57 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV WRITE_TO_HSM Thread MainThread 00:00:57 stkendm7a.fnal.gov 004556 root I ENCP Listening for control socket at: ('131.225.13.19', 40511) Thread WRITE 00:00:57 stkendm7a.fnal.gov 004556 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.19', 40511) elapsed=3918.030sec Thread WRITE 00:00:57 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV SENDING {'vc': {'storage_group': 'ktev', 'external_label': 'VOV152', 'library': 'CD-LTO4F1', 'file_family': 'ktev_migration-MIGRATION', 'wrapper': 'cern', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'file_family_width': 1}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NZ/NZNT/NZNT09/f6', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 329475780L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4F1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7875, 'mc_device': '1,1,1,11', 'compression': 0, 'serial_num': '1310051338', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr215a', 'max_time_in_state': 1200, 'device': 'stkenmvr215a:/dev/rmt/tps0d0n', 'name': 'LTO4_115.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_115.stat', 'mover_address': ('131.225.13.111', 7875), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.111', 39684), 'data_ip': 'stkenmvr215a', 'logname': 'LTO4_115MV', 'hostip': '131.225.13.111', 'do_cleaning': 'No', 'media_changer': 'SL8500F1.media_changer'}, 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT09/f6', 'uid': 0, 'uname': 'root', 'mtime': 1257573564, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cern', 'pstat': (16877, 684350712L, 20L, 1, 0, 0, 512L, 1257573165, 1257573165, 1255198270), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 329475780L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000146', 'inode': 675905520L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000146', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NZ/NZNT/NZNT09/.(access)(002800000000000008497FF0)', 'fc': {'size': 329475780L, 'external_label': 'VOV152', 'pnfsid': '002800000000000008497FF0', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000146', 'at_the_top': 150, 'local_inode': 31244314L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40511), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573657.6673989, 'encp_start_time': 1257573564.5066569, 'job_queued': 1257573564.7983661, 't0': 1257573564, 'in_queue': 92.86873984336853}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573564-4556-140'} Thread client_connect_thread 00:00:57 stkendm7a.fnal.gov 004556 root I ENCP Control socket ('131.225.13.19', 40511) is connected to ('131.225.13.111', 39685) for stkendm7a.fnal.gov-1257573564-4556-140. elapsed=3918.030sec Thread WRITE 00:00:57 stkendm7a.fnal.gov 004556 root I ENCP Received callback ticket from mover LTO4_115.mover for transfer stkendm7a.fnal.gov-1257573564-4556-140. elapsed=3918.040sec Thread WRITE 00:00:57 stkendm7a.fnal.gov 004556 root I ENCP Attempting to connect data socket to mover at ('131.225.13.111', 39684). Thread WRITE 00:00:57 stkendm7a.fnal.gov 004556 root I ENCP Connecting data socket to mover (('131.225.13.111', 39684)) with interface 131.225.13.19. elapsed=3918.040sec Thread WRITE 00:00:57 stkendm7a.fnal.gov 004556 root I ENCP Data socket ('131.225.13.19', 40538) is connected to ('131.225.13.111', 39684). elapsed=3918.040sec Thread WRITE 00:00:57 stkendm7a.fnal.gov 004556 root I ENCP Opening local file. Thread WRITE 00:00:57 stkendm7a.fnal.gov 004556 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False Thread WRITE 00:00:57 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV tape position: target 253 current 253 0 Thread seek_thread 00:00:57 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV write_tape starting, bytes_to_write=329476096 Thread tape_thread 00:00:57 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZN0/NZN088/f3', drive = 'stkenmvr215a:/dev/rmt/tps0d0n:1310051338', pnfs_id = '002800000000000008497FC0' where bfid = 'CDMS125757365700000'; 00:00:57 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV Write starting. Tape VOV152 absolute location in blocks 441721 Thread tape_thread 00:00:57 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:57 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30106 current 30105 417 Thread seek_thread 00:00:57 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:57 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=25495827 Thread tape_thread 00:00:58 stkendm7a.fnal.gov 004556 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /data/data2/Migration_Spool/VO6001:0000_000000000_0000143 -> /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NZ/NZN0/NZN088/f3: 1259823748 bytes copied to VOV152 at 36.6 MB/S (81.4 MB/S network) (64.6 MB/S drive) (53.9 MB/S disk) mover=LTO4_115.mover drive_id=ULTRIUM-TD4 drive_sn=1310051338 drive_vendor=IBM elapsed=99.728 {'storage_group': 'ktev', 'encp_crc': 56258062L, 'disk_rate': 53.937602487721556, 'drive_rate': 64.648413934585435, 'driver': 'FTTDriver', 'transfer_rate': 36.6353232491985, 'mover_interface': 'stkenmvr215a', 'overall_rate': 35.944234333444918, 'media_changer': 'SL8500F1.media_changer', 'network_rate': 36.6353232491985, 'encp_ip': '131.225.13.19', 'unique_id': 'stkendm7a.fnal.gov-1257573558-4556-139'} Thread WRITE_2 00:00:58 stkendm7a.fnal.gov 004556 root I ENCP File /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NZ/NZN0/NZN088/f3 copied successfully. Thread WRITE_2 00:00:58 stkendm7a.fnal.gov 004556 root I ENCP Completed transferring 1259823748 bytes in 1 files in 99.7294199467 sec. Overall rate = 35.9 MB/sec. Transfer rate = 36.6 MB/sec. Network rate = 81.4 MB/sec. Drive rate = 64.6 MB/sec. Disk rate = 53.9 MB/sec. Exit status = 0. Thread WRITE_2 00:00:58 stkensrv0n.fnal.gov 028796 enstore I FILSRV start modifying {'pnfsid': '002800000000000008497FC0', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZN0/NZN088/f3', 'deleted': 'no', 'external_label': 'VOV152', 'drive': 'stkenmvr215a:/dev/rmt/tps0d0n:1310051338', 'update': '2009-11-07 00:00:57.796824', 'gid': 0, 'location_cookie': '0000_000000000_0000251', 'complete_crc': 56258062L, 'uid': 0, 'bfid': 'CDMS125757365700000', 'sanity_cookie': (65536L, 1949751648L), 'size': 1259823748L} 00:00:58 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set pnfs_id = '001F000000000000007396F8' where bfid = 'CDMS125757365700000'; 00:00:58 stkensrv0n.fnal.gov 028796 enstore I FILSRV modified to {'pnfsid': '001F000000000000007396F8', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZN0/NZN088/f3', 'deleted': 'no', 'external_label': 'VOV152', 'drive': 'stkenmvr215a:/dev/rmt/tps0d0n:1310051338', 'update': '2009-11-07 00:00:57.796824', 'gid': 0, 'location_cookie': '0000_000000000_0000251', 'complete_crc': 56258062L, 'uid': 0, 'bfid': 'CDMS125757365700000', 'sanity_cookie': (65536L, 1949751648L), 'size': 1259823748L} 00:00:58 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 25495966/25495966, blocks 195 header 139 Thread tape_thread 00:00:58 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 25495827/25495827 bytes Thread net_thread 00:00:58 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:58 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30106 block 195 block_size 0 bloc_loc 5651387 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 25495966 block_read_tot 0.2418384552 tape_rate 105425607.267 Thread tape_thread 00:00:58 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 25495966/25495966 bytes Thread tape_thread 00:00:58 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00217318534851 read 0.112119674683 crc_check 0.125876665115 push 0.00166893005371 block_read 0.2418384552 idle 0.00482082366943 Thread tape_thread 00:00:58 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:59 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:00:59 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30107 current 30106 195 Thread seek_thread 00:00:59 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:00:59 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=14976043 Thread tape_thread 00:00:59 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 14976182/14976182, blocks 115 header 139 Thread tape_thread 00:00:59 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 14976043/14976043 bytes Thread net_thread 00:00:59 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:00:59 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30107 block 115 block_size 0 bloc_loc 5651503 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 14976182 block_read_tot 0.149776697159 tape_rate 99990067.1072 Thread tape_thread 00:00:59 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 14976182/14976182 bytes Thread tape_thread 00:00:59 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00214838981628 read 0.0726301670074 crc_check 0.0739974975586 push 0.00100064277649 block_read 0.149776697159 idle 0.00238013267517 Thread tape_thread 00:00:59 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:00:59 stkendm7a.fnal.gov 004556 root I ENCP Version: v3_8 CVS $Revision: 1.963 $ encp.pyc OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4F1 Storage Group: ktev File Family: ktev_migration-MIGRATION FF Wrapper: cern FF Width: 1 Current working directory: stkendm7a.fnal.gov:/opt/enstore/src Command line: encp --delayed-dismount 2 --ignore-fair-share --threaded --priority 0 --storage-group ktev --file-family ktev_migration-MIGRATION --file-family-wrapper cern --library CD-LTO4F1 --override-path /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT01/f14 --threaded /data/data2/Migration_Spool/VO6001:0000_000000000_0000152 /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NZ/NZNT/NZNT01/f14 Thread WRITE_2 00:00:59 stkendm7a.fnal.gov 004556 root I ENCP Creating zero length output files. Thread WRITE_2 00:00:59 stkendm7a.fnal.gov 004556 root I ENCP Sending /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NZ/NZNT/NZNT01/f14 write request to LM: unique_id: stkendm7a.fnal.gov-1257573659-4556-146 inputfile: /data/data2/Migration_Spool/VO6001:0000_000000000_0000152 outputfile: /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NZ/NZNT/NZNT01/.(access)(002800000000000008498118) Thread WRITE_2 00:00:59 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : ktev 00:00:59 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM write Q'd /data/data2/Migration_Spool/VO6001:0000_000000000_0000152 (stkendm7a.fnal.gov-1257573659-4556-146) -> /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT01/f14 : library=CD-LTO4F1 family=ktev_migration-MIGRATION requester:root volume_family:ktev.ktev_migration-MIGRATION.cern 00:01:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30108 current 30107 115 Thread seek_thread 00:01:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=10543000 Thread tape_thread 00:01:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 10543139/10543139, blocks 81 header 139 Thread tape_thread 00:01:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 10543000/10543000 bytes Thread net_thread 00:01:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30108 block 81 block_size 0 bloc_loc 5651585 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 10543139 block_read_tot 0.125532627106 tape_rate 83987240.9515 Thread tape_thread 00:01:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 10543139/10543139 bytes Thread tape_thread 00:01:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00376415252686 read 0.0679244995117 crc_check 0.0531520843506 push 0.000691890716553 block_read 0.125532627106 idle 0.00194501876831 Thread tape_thread 00:01:01 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:01 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:01 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30109 current 30108 81 Thread seek_thread 00:01:01 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:01 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=19962749 Thread tape_thread 00:01:02 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 19962888/19962888, blocks 153 header 139 Thread tape_thread 00:01:02 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 19962749/19962749 bytes Thread net_thread 00:01:02 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:02 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30109 block 153 block_size 0 bloc_loc 5651739 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 19962888 block_read_tot 0.196330070496 tape_rate 101680236.5 Thread tape_thread 00:01:02 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 19962888/19962888 bytes Thread tape_thread 00:01:02 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00236535072327 read 0.0947177410126 crc_check 0.0979378223419 push 0.00130915641785 block_read 0.196330070496 idle 0.00352239608765 Thread tape_thread 00:01:03 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:03 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30110 current 30109 153 Thread seek_thread 00:01:03 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:03 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=14953506 Thread tape_thread 00:01:03 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 14953645/14953645, blocks 115 header 139 Thread tape_thread 00:01:03 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 14953506/14953506 bytes Thread net_thread 00:01:03 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:03 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30110 block 115 block_size 0 bloc_loc 5651855 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 14953645 block_read_tot 0.150161266327 tape_rate 99583903.1315 Thread tape_thread 00:01:03 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 14953645/14953645 bytes Thread tape_thread 00:01:03 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00175476074219 read 0.073903799057 crc_check 0.0735268592834 push 0.000975847244263 block_read 0.150161266327 idle 0.00259304046631 Thread tape_thread 00:01:03 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:03 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30111 current 30110 115 Thread seek_thread 00:01:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=55243172 Thread tape_thread 00:01:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 55243311/55243311, blocks 422 header 139 Thread tape_thread 00:01:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 55243172/55243172 bytes Thread net_thread 00:01:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30111 block 422 block_size 0 bloc_loc 5652278 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 55243311 block_read_tot 0.50319647789 tape_rate 109784772.802 Thread tape_thread 00:01:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 55243311/55243311 bytes Thread tape_thread 00:01:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00501275062561 read 0.222830057144 crc_check 0.271735668182 push 0.00361800193787 block_read 0.50319647789 idle 0.00966262817383 Thread tape_thread 00:01:05 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:05 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:05 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30112 current 30111 422 Thread seek_thread 00:01:06 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:06 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=25400130 Thread tape_thread 00:01:06 stkendm7a.fnal.gov 004556 root I ENCP File /data/data2/Migration_Spool/VO6001:0000_000000000_0000146 transfered for stkendm7a.fnal.gov-1257573564-4556-140 in 8.33397912979 seconds. Thread WRITE 00:01:06 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 25400269/25400269, blocks 194 header 139 Thread tape_thread 00:01:06 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 25400130/25400130 bytes Thread net_thread 00:01:06 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:06 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30112 block 194 block_size 0 bloc_loc 5652473 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 25400269 block_read_tot 0.240477561951 tape_rate 105624278.598 Thread tape_thread 00:01:06 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 25400269/25400269 bytes Thread tape_thread 00:01:06 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00267314910889 read 0.110789060593 crc_check 0.125343561172 push 0.00167179107666 block_read 0.240477561951 idle 0.00495076179504 Thread tape_thread 00:01:06 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30113 current 30112 194 Thread seek_thread 00:01:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=9648008 Thread tape_thread 00:01:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 9648147/9648147, blocks 74 header 139 Thread tape_thread 00:01:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 9648008/9648008 bytes Thread net_thread 00:01:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30113 block 74 block_size 0 bloc_loc 5652548 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 9648147 block_read_tot 0.105141639709 tape_rate 91763330.1769 Thread tape_thread 00:01:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 9648147/9648147 bytes Thread tape_thread 00:01:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00120091438293 read 0.0559222698212 crc_check 0.0473840236664 push 0.000634431838989 block_read 0.105141639709 idle 0.00178670883179 Thread tape_thread 00:01:07 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:08 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV tape position: target 232 current 0 1 Thread seek_thread 00:01:08 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:08 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30114 current 30113 74 Thread seek_thread 00:01:08 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:08 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=10456243 Thread tape_thread 00:01:08 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 10456382/10456382, blocks 80 header 139 Thread tape_thread 00:01:08 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 10456243/10456243 bytes Thread net_thread 00:01:08 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:08 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30114 block 80 block_size 0 bloc_loc 5652629 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 10456382 block_read_tot 0.116045475006 tape_rate 90105900.29 Thread tape_thread 00:01:08 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 10456382/10456382 bytes Thread tape_thread 00:01:08 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00102281570435 read 0.0621190071106 crc_check 0.0521991252899 push 0.000704526901245 block_read 0.116045475006 idle 0.00189971923828 Thread tape_thread 00:01:08 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:09 last message repeated 1 times 00:01:09 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:09 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30115 current 30114 80 Thread seek_thread 00:01:09 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:09 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=19948748 Thread tape_thread 00:01:09 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV written bytes 329476096/329476096, blocks 2514 header 0 trailer 316 Thread tape_thread 00:01:09 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 19948887/19948887, blocks 153 header 139 Thread tape_thread 00:01:09 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 19948748/19948748 bytes Thread net_thread 00:01:09 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:10 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30115 block 153 block_size 0 bloc_loc 5652783 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 19948887 block_read_tot 0.19318819046 tape_rate 103261420.651 Thread tape_thread 00:01:10 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 19948887/19948887 bytes Thread tape_thread 00:01:10 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00261664390564 read 0.0908524990082 crc_check 0.0984230041504 push 0.001296043396 block_read 0.19318819046 idle 0.00337767601013 Thread tape_thread 00:01:10 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:01:10 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:11 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30116 current 30115 153 Thread seek_thread 00:01:11 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:11 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=47570452 Thread tape_thread 00:01:11 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 47570591/47570591, blocks 363 header 139 Thread tape_thread 00:01:11 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 47570452/47570452 bytes Thread net_thread 00:01:11 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:11 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30116 block 363 block_size 0 bloc_loc 5653147 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 47570591 block_read_tot 0.431526422501 tape_rate 110237956.518 Thread tape_thread 00:01:11 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 47570591/47570591 bytes Thread tape_thread 00:01:11 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.0044903755188 read 0.189161539078 crc_check 0.234755754471 push 0.00311875343323 block_read 0.431526422501 idle 0.00839042663574 Thread tape_thread 00:01:11 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:12 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:12 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30117 current 30116 363 Thread seek_thread 00:01:12 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:12 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=55035099 Thread tape_thread 00:01:13 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 55035238/55035238, blocks 420 header 139 Thread tape_thread 00:01:13 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 55035099/55035099 bytes Thread net_thread 00:01:13 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:13 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30117 block 420 block_size 0 bloc_loc 5653568 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 55035238 block_read_tot 0.512319803238 tape_rate 107423600.751 Thread tape_thread 00:01:13 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 55035238/55035238 bytes Thread tape_thread 00:01:13 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00785040855408 read 0.230001926422 crc_check 0.270962715149 push 0.00350475311279 block_read 0.512319803238 idle 0.0092670917511 Thread tape_thread 00:01:13 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:14 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:14 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30118 current 30117 420 Thread seek_thread 00:01:14 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:14 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=25288856 Thread tape_thread 00:01:14 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 25288995/25288995, blocks 193 header 139 Thread tape_thread 00:01:14 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 25288856/25288856 bytes Thread net_thread 00:01:14 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:14 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30118 block 193 block_size 0 bloc_loc 5653762 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 25288995 block_read_tot 0.239250421524 tape_rate 105700942.297 Thread tape_thread 00:01:14 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 25288995/25288995 bytes Thread tape_thread 00:01:14 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00250625610352 read 0.110138654709 crc_check 0.124949216843 push 0.00165629386902 block_read 0.239250421524 idle 0.00445890426636 Thread tape_thread 00:01:15 stkenmvr109a.fnal.gov 031352 root I LTO09MV skip_fm -159 Thread seek_thread 00:01:15 stkenmvr109a.fnal.gov 031352 root I LTO09MV skip_fm 1 Thread seek_thread 00:01:15 stkenmvr109a.fnal.gov 031352 root I LTO09MV read_tape starting, bytes_to_read=1498719455 Thread tape_thread 00:01:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30119 current 30118 193 Thread seek_thread 00:01:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=9602930 Thread tape_thread 00:01:15 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 9603069/9603069, blocks 74 header 139 Thread tape_thread 00:01:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 9602930/9602930 bytes Thread net_thread 00:01:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30119 block 74 block_size 0 bloc_loc 5653837 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 9603069 block_read_tot 0.104394197464 tape_rate 91988532.2488 Thread tape_thread 00:01:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 9603069/9603069 bytes Thread tape_thread 00:01:15 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00128746032715 read 0.0549685955048 crc_check 0.0474863052368 push 0.000651836395264 block_read 0.104394197464 idle 0.00176429748535 Thread tape_thread 00:01:16 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV filemarks written. Tape VOV152 absolute location in blocks 444240 Thread tape_thread 00:01:16 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV write_tape timing:pull 0.0237905979156 write 2.44688296318 crc_check 1.49847245216 freespace 0.0218834877014 block_write 3.99102950096 idle 0.0413336753845 Thread tape_thread 00:01:16 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV drive stats after write. Tape VOV152 position 253 block 0 block_size 0 bloc_loc 444240 tot_blocks 762939000 BOT 0 read_err 0 write_err 6 bytes 329476096 block_write_tot 3.99102950096 tape_rate 82554162.0077 Thread tape_thread 00:01:16 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV remaining bytes info in DB 720652288000 estimated 720322811904 reported from drive 718644224000 position 256 Thread tape_thread 00:01:16 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV new bitfile request {'uid': 0, 'external_label': 'VOV152', 'gid': 0, 'location_cookie': '0000_000000000_0000254', 'complete_crc': 2160213274L, 'sanity_cookie': (65536L, 1520671881L), 'size': 329475780L} Thread tape_thread 00:01:16 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV set remaining: VOV152 718644224000 0000_000000000_0000256 Thread tape_thread 00:01:16 last message repeated 1 times 00:01:16 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-07 00:01:16', eod_cookie = '0000_000000000_0000256', non_del_files = '85', remaining_bytes = 718644224000 where label = 'VOV152'; 00:01:16 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV transfer complete volume=VOV152 location=256 Thread tape_thread 00:01:16 last message repeated 1 times 00:01:16 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '85', modification_time = '2009-11-07 00:01:16' where label = 'VOV152'; 00:01:16 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'ktev', 'blocksize': 131072, 'declared': 1254165880.0, 'library': 'CD-LTO4F1', 'si_time': [0.0, 0.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VOV152', 'wrapper': 'cern', 'remaining_bytes': 720652288000L, 'sum_mounts': 1, 'sum_rd_access': 0, 'file_family_width': 1, 'last_access': 1257573657.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000253', 'capacity_bytes': 858993459200L, 'non_del_files': 84, 'sum_wr_err': 0, 'sum_wr_access': 84, 'file_family': 'ktev_migration-MIGRATION', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'modification_time': 1257573657.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257571652.0}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NZ/NZNT/NZNT09/f6', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 329475780L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4F1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7875, 'mc_device': '1,1,1,11', 'compression': 0, 'serial_num': '1310051338', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr215a', 'max_time_in_state': 1200, 'device': 'stkenmvr215a:/dev/rmt/tps0d0n', 'name': 'LTO4_115.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_115.stat', 'mover_address': ('131.225.13.111', 7875), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.111', 39684), 'data_ip': 'stkenmvr215a', 'logname': 'LTO4_115MV', 'hostip': '131.225.13.111', 'do_cleaning': 'No', 'media_changer': 'SL8500F1.media_changer'}, 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT09/f6', 'uid': 0, 'uname': 'root', 'mtime': 1257573564, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cern', 'pstat': (16877, 684350712L, 20L, 1, 0, 0, 512L, 1257573165, 1257573165, 1255198270), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 329475780L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000146', 'inode': 675905520L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000146', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NZ/NZNT/NZNT09/.(access)(002800000000000008497FF0)', 'fc': {'uid': 0, 'external_label': 'VOV152', 'gid': 0, 'location_cookie': '0000_000000000_0000254', 'complete_crc': 2160213274L, 'bfid': 'CDMS125757367600000', 'sanity_cookie': (65536L, 1520671881L), 'size': 329475780L}, 'infilepath': '/data/data2/Migration_Spool/VO6001:0000_000000000_0000146', 'at_the_top': 150, 'local_inode': 31244314L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40511), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257573564.5066569, 'seek_time': 0.034470081329345703, 'transfer_time': 18.462450981140137, 't0': 1257573564, 'mount_time': 0.045170068740844727, 'lm_dequeued': 1257573657.6673989, 'job_queued': 1257573564.7983661, 'drive_call_time': 2.427332878112793, 'in_queue': 92.86873984336853, 'drive_transfer_time': 11.738099098205566}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573564-4556-140'} Thread tape_thread 00:01:16 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV Done is sent Thread tape_thread 00:01:16 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV maybe_clean Thread tape_thread 00:01:16 stkendm7a.fnal.gov 004556 root I ENCP Received final dialog for stkendm7a.fnal.gov-1257573564-4556-140. elapsed=3936.600sec Thread WRITE 00:01:16 stkendm7a.fnal.gov 004556 root I ENCP leaving transfer_file(). Thread WRITE 00:01:16 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV returned from maybe_clean Thread tape_thread 00:01:16 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT09/f6', drive = 'stkenmvr215a:/dev/rmt/tps0d0n:1310051338', pnfs_id = '002800000000000008497FF0' where bfid = 'CDMS125757367600000'; 00:01:16 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:16 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM write_to_hsm next work on vol=VOV152 mover=LTO4_115.mover requester:root 00:01:16 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM HAVE_BOUND:sending write_to_hsm /pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQNB/NQNB06/f0 to mover LTO4_115.mover ('131.225.13.111', 7875) DEL_DISM 2 00:01:16 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : ktev 00:01:16 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM HAVE_BOUND:sending {'vc': {'storage_group': 'ktev', 'external_label': 'VOV152', 'library': 'CD-LTO4F1', 'file_family': 'ktev_migration-MIGRATION', 'wrapper': 'cern', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'file_family_width': 1}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NQ/NQNB/NQNB06/f0', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 17404147048L, 'mover': 'LTO4_115.mover', 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQNB/NQNB06/f0', 'uid': 0, 'uname': 'root', 'type': 'cern', 'gid': 0, 'mtime': 1257573575, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (16877, 675905568L, 20L, 1, 0, 0, 512L, 1257573575, 1257573575, 1257573575), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 17404147048L, 'fullname': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000401', 'inode': 675905640L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000401', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NQ/NQNB/NQNB06/.(access)(002800000000000008498068)', 'fc': {'size': 17404147048L, 'external_label': 'VOV152', 'pnfsid': '002800000000000008498068', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000401', 'at_the_top': 90, 'local_inode': 39503212L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40515), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573676.674314, 'encp_start_time': 1257573575.6435671, 'job_queued': 1257573576.046139, 't0': 1257573575, 'in_queue': 100.62791800498962}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573575-642-429'} to mover 00:01:16 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:16 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV WRITE_TO_HSM Thread MainThread 00:01:16 stkendm7a.fnal.gov 000642 root I ENCP Listening for control socket at: ('131.225.13.19', 40515) Thread WRITE_1 00:01:16 stkendm7a.fnal.gov 000642 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.19', 40515) elapsed=5456.400sec Thread WRITE_1 00:01:16 stkendm7a.fnal.gov 004556 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /data/data2/Migration_Spool/VO6001:0000_000000000_0000146 -> /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NZ/NZNT/NZNT09/f6: 329475780 bytes copied to VOV152 at 17 MB/S (80.4 MB/S network) (26.8 MB/S drive) (41.8 MB/S disk) mover=LTO4_115.mover drive_id=ULTRIUM-TD4 drive_sn=1310051338 drive_vendor=IBM elapsed=112.19 {'storage_group': 'ktev', 'encp_crc': 2160213274L, 'disk_rate': 41.758517247614833, 'drive_rate': 26.76860934935798, 'driver': 'FTTDriver', 'transfer_rate': 17.017565743120823, 'mover_interface': 'stkenmvr215a', 'overall_rate': 16.55298260703772, 'media_changer': 'SL8500F1.media_changer', 'network_rate': 17.017565743120823, 'encp_ip': '131.225.13.19', 'unique_id': 'stkendm7a.fnal.gov-1257573564-4556-140'} Thread WRITE 00:01:16 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV SENDING {'vc': {'storage_group': 'ktev', 'external_label': 'VOV152', 'library': 'CD-LTO4F1', 'file_family': 'ktev_migration-MIGRATION', 'wrapper': 'cern', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration-MIGRATION.cern', 'file_family_width': 1}, 'outfilepath': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NQ/NQNB/NQNB06/f0', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 17404147048L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4F1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7875, 'mc_device': '1,1,1,11', 'compression': 0, 'serial_num': '1310051338', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr215a', 'max_time_in_state': 1200, 'device': 'stkenmvr215a:/dev/rmt/tps0d0n', 'name': 'LTO4_115.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_115.stat', 'mover_address': ('131.225.13.111', 7875), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.111', 39686), 'data_ip': 'stkenmvr215a', 'logname': 'LTO4_115MV', 'hostip': '131.225.13.111', 'do_cleaning': 'No', 'media_changer': 'SL8500F1.media_changer'}, 'ignore_fair_share': 1, 'retry': None, 'lm': {'address': ('131.225.13.4', 8542)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQNB/NQNB06/f0', 'uid': 0, 'uname': 'root', 'mtime': 1257573575, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cern', 'pstat': (16877, 675905568L, 20L, 1, 0, 0, 512L, 1257573575, 1257573575, 1257573575), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 17404147048L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000401', 'inode': 675905640L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000401', 'status': ('ok', None), 'outfile': '/pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NQ/NQNB/NQNB06/.(access)(002800000000000008498068)', 'fc': {'size': 17404147048L, 'external_label': 'VOV152', 'pnfsid': '002800000000000008498068', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000401', 'at_the_top': 90, 'local_inode': 39503212L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40515), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573676.674314, 'encp_start_time': 1257573575.6435671, 'job_queued': 1257573576.046139, 't0': 1257573575, 'in_queue': 100.62791800498962}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573575-642-429'} Thread client_connect_thread 00:01:16 stkendm7a.fnal.gov 004556 root I ENCP File /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NZ/NZNT/NZNT09/f6 copied successfully. Thread WRITE 00:01:16 stkendm7a.fnal.gov 004556 root I ENCP Completed transferring 329475780 bytes in 1 files in 112.193089008 sec. Overall rate = 16.6 MB/sec. Transfer rate = 17 MB/sec. Network rate = 80.4 MB/sec. Drive rate = 26.8 MB/sec. Disk rate = 41.8 MB/sec. Exit status = 0. Thread WRITE 00:01:16 stkendm7a.fnal.gov 000642 root I ENCP Control socket ('131.225.13.19', 40515) is connected to ('131.225.13.111', 39687) for stkendm7a.fnal.gov-1257573575-642-429. elapsed=5456.410sec Thread WRITE_1 00:01:16 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:16 stkendm7a.fnal.gov 000642 root I ENCP Received callback ticket from mover LTO4_115.mover for transfer stkendm7a.fnal.gov-1257573575-642-429. elapsed=5456.420sec Thread WRITE_1 00:01:16 stkendm7a.fnal.gov 000642 root I ENCP Attempting to connect data socket to mover at ('131.225.13.111', 39686). Thread WRITE_1 00:01:16 stkendm7a.fnal.gov 000642 root I ENCP Connecting data socket to mover (('131.225.13.111', 39686)) with interface 131.225.13.19. elapsed=5456.420sec Thread WRITE_1 00:01:16 stkendm7a.fnal.gov 000642 root I ENCP Data socket ('131.225.13.19', 40541) is connected to ('131.225.13.111', 39686). elapsed=5456.420sec Thread WRITE_1 00:01:16 stkendm7a.fnal.gov 000642 root I ENCP Opening local file. Thread WRITE_1 00:01:16 stkendm7a.fnal.gov 000642 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False Thread WRITE_1 00:01:16 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30120 current 30119 74 Thread seek_thread 00:01:16 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:16 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=14955158 Thread tape_thread 00:01:16 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV tape position: target 256 current 256 0 Thread seek_thread 00:01:16 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV write_tape starting, bytes_to_write=17404147200 Thread tape_thread 00:01:16 stkensrv0n.fnal.gov 028796 enstore I FILSRV start modifying {'pnfsid': '002800000000000008497FF0', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT09/f6', 'deleted': 'no', 'external_label': 'VOV152', 'drive': 'stkenmvr215a:/dev/rmt/tps0d0n:1310051338', 'update': '2009-11-07 00:01:16.433776', 'gid': 0, 'location_cookie': '0000_000000000_0000254', 'complete_crc': 2160213274L, 'uid': 0, 'bfid': 'CDMS125757367600000', 'sanity_cookie': (65536L, 1520671881L), 'size': 329475780L} 00:01:16 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set pnfs_id = '001F0000000000000073F558' where bfid = 'CDMS125757367600000'; 00:01:16 stkenmvr215a.fnal.gov 006272 root I LTO4_115MV Write starting. Tape VOV152 absolute location in blocks 444240 Thread tape_thread 00:01:16 stkensrv0n.fnal.gov 028796 enstore I FILSRV modified to {'pnfsid': '001F0000000000000073F558', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT09/f6', 'deleted': 'no', 'external_label': 'VOV152', 'drive': 'stkenmvr215a:/dev/rmt/tps0d0n:1310051338', 'update': '2009-11-07 00:01:16.433776', 'gid': 0, 'location_cookie': '0000_000000000_0000254', 'complete_crc': 2160213274L, 'uid': 0, 'bfid': 'CDMS125757367600000', 'sanity_cookie': (65536L, 1520671881L), 'size': 329475780L} 00:01:16 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 14955297/14955297, blocks 115 header 139 Thread tape_thread 00:01:16 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 14955158/14955158 bytes Thread net_thread 00:01:16 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:16 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30120 block 115 block_size 0 bloc_loc 5653953 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 14955297 block_read_tot 0.150176286697 tape_rate 99584943.3282 Thread tape_thread 00:01:16 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 14955297/14955297 bytes Thread tape_thread 00:01:16 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00181555747986 read 0.0737373828888 crc_check 0.07364153862 push 0.00098180770874 block_read 0.150176286697 idle 0.00262117385864 Thread tape_thread 00:01:17 stkendm7a.fnal.gov 004556 root I ENCP Version: v3_8 CVS $Revision: 1.963 $ encp.pyc OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4F1 Storage Group: ktev File Family: ktev_migration-MIGRATION FF Wrapper: cern FF Width: 1 Current working directory: stkendm7a.fnal.gov:/opt/enstore/src Command line: encp --delayed-dismount 2 --ignore-fair-share --threaded --priority 0 --storage-group ktev --file-family ktev_migration-MIGRATION --file-family-wrapper cern --library CD-LTO4F1 --override-path /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT10/f13 --threaded /data/data2/Migration_Spool/VO6001:0000_000000000_0000155 /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NZ/NZNT/NZNT10/f13 Thread WRITE 00:01:17 stkendm7a.fnal.gov 004556 root I ENCP Creating zero length output files. Thread WRITE 00:01:17 stkendm7a.fnal.gov 004556 root I ENCP Sending /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NZ/NZNT/NZNT10/f13 write request to LM: unique_id: stkendm7a.fnal.gov-1257573677-4556-147 inputfile: /data/data2/Migration_Spool/VO6001:0000_000000000_0000155 outputfile: /pnfs/fs/usr/Migration/ktev/migrated_from_dlt/NZ/NZNT/NZNT10/.(access)(002800000000000008498148) Thread WRITE 00:01:17 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : ktev 00:01:17 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM write Q'd /data/data2/Migration_Spool/VO6001:0000_000000000_0000155 (stkendm7a.fnal.gov-1257573677-4556-147) -> /pnfs/fs/usr/ktev/migrated_from_dlt/NZ/NZNT/NZNT10/f13 : library=CD-LTO4F1 family=ktev_migration-MIGRATION requester:root volume_family:ktev.ktev_migration-MIGRATION.cern 00:01:17 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:17 last message repeated 1 times 00:01:17 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:18 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30121 current 30120 115 Thread seek_thread 00:01:18 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:18 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=19862239 Thread tape_thread 00:01:18 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 19862378/19862378, blocks 152 header 139 Thread tape_thread 00:01:18 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 19862239/19862239 bytes Thread net_thread 00:01:18 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:18 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30121 block 152 block_size 0 bloc_loc 5654106 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 19862378 block_read_tot 0.191586494446 tape_rate 103673163.693 Thread tape_thread 00:01:18 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 19862378/19862378 bytes Thread tape_thread 00:01:18 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00235557556152 read 0.0898797512054 crc_check 0.0980236530304 push 0.00132751464844 block_read 0.191586494446 idle 0.00323796272278 Thread tape_thread 00:01:19 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:19 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30122 current 30121 152 Thread seek_thread 00:01:19 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:19 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=15048163 Thread tape_thread 00:01:19 stkenmvr193a.fnal.gov 031647 root I LTO4_93MV Tape is ejected Thread media_thread 00:01:19 stkenmvr193a.fnal.gov 031647 root I LTO4_93MV dismounting VON726 Thread media_thread 00:01:19 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC PROCESSING dismount VON726 0,7,1,11 00:01:19 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 15048302/15048302, blocks 115 header 139 Thread tape_thread 00:01:19 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 15048163/15048163 bytes Thread net_thread 00:01:19 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:19 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30122 block 115 block_size 0 bloc_loc 5654222 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 15048302 block_read_tot 0.149838924408 tape_rate 100429858.66 Thread tape_thread 00:01:19 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 15048302/15048302 bytes Thread tape_thread 00:01:19 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00168752670288 read 0.072847366333 crc_check 0.0743193626404 push 0.000984668731689 block_read 0.149838924408 idle 0.00284194946289 Thread tape_thread 00:01:19 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:20 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:20 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30123 current 30122 115 Thread seek_thread 00:01:20 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:20 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=55047761 Thread tape_thread 00:01:21 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 55047900/55047900, blocks 420 header 139 Thread tape_thread 00:01:21 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 55047761/55047761 bytes Thread net_thread 00:01:21 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:21 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30123 block 420 block_size 0 bloc_loc 5654643 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 55047900 block_read_tot 0.498161554337 tape_rate 110502104.229 Thread tape_thread 00:01:21 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 55047900/55047900 bytes Thread tape_thread 00:01:21 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00466132164001 read 0.21840929985 crc_check 0.271541595459 push 0.00354933738708 block_read 0.498161554337 idle 0.00896883010864 Thread tape_thread 00:01:21 stkensrv4n.fnal.gov 032143 enstore I SL8500G1MC query drive 0,7,1,11 => 0,0,7,1,11 online in use VON726 IBM-LTO4 00:01:21 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:21 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:22 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:22 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30124 current 30123 420 Thread seek_thread 00:01:22 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:22 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=25558188 Thread tape_thread 00:01:22 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 25558327/25558327, blocks 195 header 139 Thread tape_thread 00:01:22 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 25558188/25558188 bytes Thread net_thread 00:01:22 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:22 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30124 block 195 block_size 0 bloc_loc 5654839 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 25558327 block_read_tot 0.241759777069 tape_rate 105717863.037 Thread tape_thread 00:01:22 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 25558327/25558327 bytes Thread tape_thread 00:01:22 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.0026261806488 read 0.111665725708 crc_check 0.125818490982 push 0.00164937973022 block_read 0.241759777069 idle 0.00425171852112 Thread tape_thread 00:01:22 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:23 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:23 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30125 current 30124 195 Thread seek_thread 00:01:23 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:23 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=9725574 Thread tape_thread 00:01:23 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 9725713/9725713, blocks 75 header 139 Thread tape_thread 00:01:23 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 9725574/9725574 bytes Thread net_thread 00:01:23 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:23 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:23 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30125 block 75 block_size 0 bloc_loc 5654915 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 9725713 block_read_tot 0.103099584579 tape_rate 94333192.8996 Thread tape_thread 00:01:23 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 9725713/9725713 bytes Thread tape_thread 00:01:23 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.0016496181488 read 0.0528707504272 crc_check 0.0479452610016 push 0.000633955001831 block_read 0.103099584579 idle 0.00166487693787 Thread tape_thread 00:01:24 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:24 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30126 current 30125 75 Thread seek_thread 00:01:24 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:24 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=14862933 Thread tape_thread 00:01:24 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 14863072/14863072, blocks 114 header 139 Thread tape_thread 00:01:24 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 14862933/14862933 bytes Thread net_thread 00:01:24 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:24 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30126 block 114 block_size 0 bloc_loc 5655030 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 14863072 block_read_tot 0.149033546448 tape_rate 99729707.5341 Thread tape_thread 00:01:24 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 14863072/14863072 bytes Thread tape_thread 00:01:24 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00192523002625 read 0.0728123188019 crc_check 0.0733382701874 push 0.000957727432251 block_read 0.149033546448 idle 0.00259160995483 Thread tape_thread 00:01:25 stkendca17a.fnal.gov 002905 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: exp-db File Family: daily-d0-offline FF Wrapper: cpio_odc FF Width: 2 Current working directory: stkendca17a.fnal.gov:/opt/d-cache/config Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel02/D0OFPRD1_t702240508_s103338_p20 --put-cache 003000000000000006222B38 /diska/write-pool-1/data/003000000000000006222B38 00:01:25 stkendca17a.fnal.gov 002905 root I ENCP Creating zero length output files. 00:01:25 stkendca17a.fnal.gov 002905 root I ENCP Sending /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel02/D0OFPRD1_t702240508_s103338_p20 write request to LM: unique_id: stkendca17a.fnal.gov-1257573685-2905-0 inputfile: /diska/write-pool-1/data/003000000000000006222B38 outputfile: /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel02/.(access)(003000000000000006222B38) 00:01:25 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : exp-db 00:01:25 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /diska/write-pool-1/data/003000000000000006222B38 (stkendca17a.fnal.gov-1257573685-2905-0) -> /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel02/D0OFPRD1_t702240508_s103338_p20 : library=CD-LTO4G1 family=daily-d0-offline requester:root volume_family:exp-db.daily-d0-offline.cpio_odc 00:01:25 stkendca17a.fnal.gov 002905 root I ENCP Listening for control socket at: ('131.225.13.5', 50148) 00:01:25 stkendca17a.fnal.gov 002905 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.5', 50148) elapsed=1.020sec 00:01:25 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:25 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:25 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30127 current 30126 114 Thread seek_thread 00:01:25 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:25 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=19958958 Thread tape_thread 00:01:26 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 19959097/19959097, blocks 153 header 139 Thread tape_thread 00:01:26 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 19958958/19958958 bytes Thread net_thread 00:01:26 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:26 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30127 block 153 block_size 0 bloc_loc 5655184 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 19959097 block_read_tot 0.192274332047 tape_rate 103805311.856 Thread tape_thread 00:01:26 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 19959097/19959097 bytes Thread tape_thread 00:01:26 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00232219696045 read 0.0902729034424 crc_check 0.0983779430389 push 0.00130128860474 block_read 0.192274332047 idle 0.00349926948547 Thread tape_thread 00:01:26 stkendca20a.fnal.gov 023640 root I ENCP File /diskc/write-pool-3/data/003000000000000006222A10 transfered for stkendca20a.fnal.gov-1257573613-23640-0 in 56.1355800629 seconds. 00:01:27 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:27 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30128 current 30127 153 Thread seek_thread 00:01:27 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:27 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=14693402 Thread tape_thread 00:01:27 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 14693541/14693541, blocks 113 header 139 Thread tape_thread 00:01:27 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 14693402/14693402 bytes Thread net_thread 00:01:27 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:27 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30128 block 113 block_size 0 bloc_loc 5655298 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 14693541 block_read_tot 0.148720741272 tape_rate 98799541.1691 Thread tape_thread 00:01:27 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 14693541/14693541 bytes Thread tape_thread 00:01:27 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00157618522644 read 0.0739014148712 crc_check 0.0722806453705 push 0.000962495803833 block_read 0.148720741272 idle 0.00270342826843 Thread tape_thread 00:01:27 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:28 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:28 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30129 current 30128 113 Thread seek_thread 00:01:28 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:28 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=17569072 Thread tape_thread 00:01:28 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 17569211/17569211, blocks 135 header 139 Thread tape_thread 00:01:28 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 17569072/17569072 bytes Thread net_thread 00:01:28 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:28 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30129 block 135 block_size 0 bloc_loc 5655434 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 17569211 block_read_tot 0.172053575516 tape_rate 102114768.306 Thread tape_thread 00:01:28 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 17569211/17569211 bytes Thread tape_thread 00:01:28 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00207662582397 read 0.0820860862732 crc_check 0.0867207050323 push 0.00117015838623 block_read 0.172053575516 idle 0.00302863121033 Thread tape_thread 00:01:29 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:29 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:29 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30130 current 30129 135 Thread seek_thread 00:01:29 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:29 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=19867103 Thread tape_thread 00:01:29 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 19867242/19867242, blocks 152 header 139 Thread tape_thread 00:01:29 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 19867103/19867103 bytes Thread net_thread 00:01:29 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:30 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30130 block 152 block_size 0 bloc_loc 5655587 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 19867242 block_read_tot 0.190653800964 tape_rate 104205853.225 Thread tape_thread 00:01:30 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 19867242/19867242 bytes Thread tape_thread 00:01:30 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00246334075928 read 0.0888500213623 crc_check 0.0980756282806 push 0.00126481056213 block_read 0.190653800964 idle 0.00347685813904 Thread tape_thread 00:01:30 stkensrv4n.fnal.gov 005357 enstore E LTO4G1LM MOVER ERROR RQ {'status': ('ok', None), 'returned_work': None, 'library': None, 'mover_type': '', 'current_location': '0000_000000000_0000000', 'ip_map': '', 'operation': None, 'read_only': 0, 'current_time': 1257573690.8890841, 'error_source': None, 'external_label': None, 'state': 'OFFLINE', 'r_a': (('131.225.13.194', 54880), 19735L, '131.225.13.194-54880-1256586929.509013-4866'), 'library_list': [('CD-LTO4G1.library_manager', ('131.225.13.4', 8541))], 'current_priority': (None, None), 'transfer_deficiency': 1, 'time_in_state': 986760.52902603149, 'address': ('131.225.13.194', 7860), 'volume_family': None, 'work': 'mover_error', 'mover': 'LTO4_99.mover', 'volume_status': (['none', 'none'], ['none', 'none']), 'unique_id': None} 00:01:30 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:31 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30131 current 30130 152 Thread seek_thread 00:01:31 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:31 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=15044805 Thread tape_thread 00:01:31 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 15044944/15044944, blocks 115 header 139 Thread tape_thread 00:01:31 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 15044805/15044805 bytes Thread net_thread 00:01:31 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:31 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30131 block 115 block_size 0 bloc_loc 5655703 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 15044944 block_read_tot 0.151847362518 tape_rate 99079389.6614 Thread tape_thread 00:01:31 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 15044944/15044944 bytes Thread tape_thread 00:01:31 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00133442878723 read 0.0752215385437 crc_check 0.0743112564087 push 0.000980138778687 block_read 0.151847362518 idle 0.00274705886841 Thread tape_thread 00:01:31 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:01:31 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:32 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:32 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30132 current 30131 115 Thread seek_thread 00:01:32 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:32 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=9503396 Thread tape_thread 00:01:32 stkensrv4n.fnal.gov 032143 enstore I SL8500G1MC dismount VOLUME 0,7,1,11 force => 0,Dismount: Forced dismount of VON726 from 0, 7, 1,11 00:01:32 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC FINISHED dismount VON726 0,7,1,11 returned ('ok', 0, 'dismount VOLUME 0,7,1,11 force => 0,Dismount: Forced dismount of VON726 from 0, 7, 1,11', '', '') 00:01:32 stkenmvr193a.fnal.gov 031647 root I LTO4_93MV dismounted VON726 ULTRIUM-TD4 00:01:32 Thread media_thread 00:01:32 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 9503535/9503535, blocks 73 header 139 Thread tape_thread 00:01:32 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 9503396/9503396 bytes Thread net_thread 00:01:32 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:32 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30132 block 73 block_size 0 bloc_loc 5655777 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 9503535 block_read_tot 0.102434873581 tape_rate 92776362.8516 Thread tape_thread 00:01:32 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 9503535/9503535 bytes Thread tape_thread 00:01:32 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00138592720032 read 0.0534043312073 crc_check 0.0470194816589 push 0.000625133514404 block_read 0.102434873581 idle 0.00185942649841 Thread tape_thread 00:01:33 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV written bytes 1966080512/1966080512, blocks 15001 header 188 trailer 324 Thread tape_thread 00:01:33 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:33 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30133 current 30132 73 Thread seek_thread 00:01:33 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:33 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=6936005 Thread tape_thread 00:01:33 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 6936144/6936144, blocks 53 header 139 Thread tape_thread 00:01:33 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 6936005/6936005 bytes Thread net_thread 00:01:33 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:33 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30133 block 53 block_size 0 bloc_loc 5655831 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 6936144 block_read_tot 0.0823323726654 tape_rate 84245646.9485 Thread tape_thread 00:01:33 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 6936144/6936144 bytes Thread tape_thread 00:01:33 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.000894546508789 read 0.0467412471771 crc_check 0.0342309474945 push 0.000465631484985 block_read 0.0823323726654 idle 0.0010986328125 Thread tape_thread 00:01:33 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:34 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV read bytes 6233969476/6233969476, blocks 47562 header 0 Thread tape_thread 00:01:34 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV write_client: wrote 6233969476/6233969476 bytes Thread net_thread 00:01:34 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV write_client waits for tape thread to exit Thread net_thread 00:01:34 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV drive stats after read. Tape VOV146 position 383 block 47562 block_size 0 bloc_loc 3477554 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 6233969476 block_read_tot 55.9561100006 tape_rate 111408199.675 Thread tape_thread 00:01:34 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV read_tape exiting, read 6233969476/6233969476 bytes Thread tape_thread 00:01:34 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV read_tape timing:get_space 0.438705444336 read 25.1840932369 crc_check 29.9232435226 push 0.410067796707 block_read 55.9561100006 idle 1.07869720459 Thread tape_thread 00:01:34 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:34 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30134 current 30133 53 Thread seek_thread 00:01:34 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:34 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=15083429 Thread tape_thread 00:01:34 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 15083568/15083568, blocks 116 header 139 Thread tape_thread 00:01:34 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 15083429/15083429 bytes Thread net_thread 00:01:34 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:34 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30134 block 116 block_size 0 bloc_loc 5655948 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 15083568 block_read_tot 0.150798797607 tape_rate 100024458.015 Thread tape_thread 00:01:34 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 15083568/15083568 bytes Thread tape_thread 00:01:34 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00181841850281 read 0.0737016201019 crc_check 0.0742788314819 push 0.000999927520752 block_read 0.150798797607 idle 0.00247597694397 Thread tape_thread 00:01:34 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV filemarks written. Tape VON279 absolute location in blocks 1206747 Thread tape_thread 00:01:34 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV write_tape timing:pull 0.170402526855 write 7.01430583 crc_check 9.00618100166 freespace 0.144290685654 block_write 16.3351800442 idle 0.259359121323 Thread tape_thread 00:01:34 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV drive stats after write. Tape VON279 position 85 block 0 block_size 0 bloc_loc 1206747 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 1966080512 block_write_tot 16.3351800442 tape_rate 120358668.021 Thread tape_thread 00:01:34 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV remaining bytes info in DB 632858624000 estimated 630892543488 reported from drive 631010304000 position 86 Thread tape_thread 00:01:34 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV new bitfile request {'uid': 0, 'external_label': 'VON279', 'gid': 0, 'location_cookie': '0000_000000000_0000085', 'complete_crc': 1909037629L, 'sanity_cookie': (65536L, 994841672L), 'size': 1966080000L} Thread tape_thread 00:01:34 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:35 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV set remaining: VON279 631010304000 0000_000000000_0000086 Thread tape_thread 00:01:35 last message repeated 1 times 00:01:35 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set last_access = '2009-11-07 00:01:35', eod_cookie = '0000_000000000_0000086', non_del_files = '85', remaining_bytes = 631010304000 where label = 'VON279'; 00:01:35 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV transfer complete volume=VON279 location=86 Thread tape_thread 00:01:35 last message repeated 1 times 00:01:35 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_wr_access = '85', modification_time = '2009-11-07 00:01:35' where label = 'VON279'; 00:01:35 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV Sending done to client: {'vc': {'comment': '', 'storage_group': 'exp-db', 'blocksize': 131072, 'declared': 1248994100.0, 'library': 'CD-LTO4G1', 'si_time': [0.0, 1243106842.0], 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'none'], 'external_label': 'VON279', 'wrapper': 'cpio_odc', 'remaining_bytes': 632858624000L, 'sum_mounts': 46, 'sum_rd_access': 0, 'file_family_width': 2, 'last_access': 1257573629.0, 'media_type': 'LTO4', 'status': ('ok', None), 'eod_cookie': '0000_000000000_0000085', 'capacity_bytes': 858993459200L, 'non_del_files': 84, 'sum_wr_err': 0, 'sum_wr_access': 84, 'file_family': 'daily-d0-offline', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'modification_time': 1257573629.0, 'write_protected': 'n', 'sum_rd_err': 0, 'first_access': 1257567920.0}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel01/D0OFPRD1_t702245888_s103346_p1', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7868, 'mc_device': '0,5,1,11', 'compression': 0, 'serial_num': '1310154353', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr208a', 'max_time_in_state': 1200, 'device': 'stkenmvr208a:/dev/rmt/tps0d0n', 'name': 'LTO4_108.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_108.stat', 'mover_address': ('131.225.13.204', 7868), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.204', 49959), 'data_ip': 'stkenmvr208a', 'logname': 'LTO4_108MV', 'hostip': '131.225.13.204', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel01/D0OFPRD1_t702245888_s103346_p1', 'uid': 0, 'uname': 'root', 'mtime': 1257573613, 'rmajor': 0, 'machine': ('Linux', 'stkendca20a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 820620024L, 20L, 1, 1602, 2752, 512L, 1257573594, 1257573594, 1254398419), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 1966080000L, 'gname': 'root', 'gid': 0, 'fullname': '/diskc/write-pool-3/data/003000000000000006222A10', 'inode': 807545360L, 'minor': 20}, 'version': 'v3_8b CVS $Revision: 1.922 $ ', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskc/write-pool-3/data/003000000000000006222A10', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel01/.(access)(003000000000000006222A10)', 'fc': {'uid': 0, 'external_label': 'VON279', 'gid': 0, 'location_cookie': '0000_000000000_0000085', 'complete_crc': 1909037629L, 'bfid': 'CDMS125757369400000', 'sanity_cookie': (65536L, 994841672L), 'size': 1966080000L}, 'infilepath': '/diskc/write-pool-3/data/003000000000000006222A10', 'at_the_top': 270, 'local_inode': 270526672L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.12', 49618), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257573611.889703, 'seek_time': 0.030288934707641602, 'transfer_time': 64.863438844680786, 't0': 1257573611, 'mount_time': 0.017022132873535156, 'lm_dequeued': 1257573630.2049761, 'job_queued': 1257573613.9431551, 'drive_call_time': 6.8156704902648926, 'in_queue': 16.261560916900635, 'drive_transfer_time': 18.352673530578613}, 'user_level': 'user2', 'unique_id': 'stkendca20a.fnal.gov-1257(trunc. 4096) Thread tape_thread 00:01:35 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV Done is sent Thread tape_thread 00:01:35 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV maybe_clean Thread tape_thread 00:01:35 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV returned from maybe_clean Thread tape_thread 00:01:35 stkendca20a.fnal.gov 023640 root I ENCP Received final dialog for stkendca20a.fnal.gov-1257573613-23640-0. elapsed=83.330sec 00:01:35 stkendca20a.fnal.gov 023640 root I ENCP leaving transfer_file(). 00:01:35 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:35 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm next work on vol=VON279 mover=LTO4_108.mover requester:root 00:01:35 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending write_to_hsm /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel02/D0OFPRD1_t702243438_s103342_p12 to mover LTO4_108.mover ('131.225.13.204', 7868) DEL_DISM None 00:01:35 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : exp-db 00:01:35 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM HAVE_BOUND:sending {'vc': {'storage_group': 'exp-db', 'external_label': 'VON279', 'library': 'CD-LTO4G1', 'file_family': 'daily-d0-offline', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'file_family_width': 2}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel02/D0OFPRD1_t702243438_s103342_p12', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': 'LTO4_108.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel02/D0OFPRD1_t702243438_s103342_p12', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257573613, 'rmajor': 0, 'machine': ('Linux', 'stkendca20a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (16877, 820620168L, 20L, 1, 1602, 2752, 512L, 1257573601, 1257573601, 1254398419), 'mode_octal': '0100016', 'mode': 33188, 'gname': 'root', 'size_bytes': 1966080000L, 'fullname': '/diskc/write-pool-3/data/003000000000000006222948', 'inode': 807545160L, 'minor': 20}, 'version': 'v3_8b CVS $Revision: 1.922 $ ', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskc/write-pool-3/data/003000000000000006222948', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel02/.(access)(003000000000000006222948)', 'fc': {'size': 1966080000L, 'external_label': 'VON279', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diskc/write-pool-3/data/003000000000000006222948', 'at_the_top': 980, 'local_inode': 270526700L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.12', 49619), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573695.268332, 'encp_start_time': 1257573611.8976369, 'job_queued': 1257573614.067333, 't0': 1257573611, 'in_queue': 81.200711965560913}, 'user_level': 'user2', 'unique_id': 'stkendca20a.fnal.gov-1257573613-23649-0'} to mover 00:01:35 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:35 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV WRITE_TO_HSM Thread MainThread 00:01:35 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV SENDING {'vc': {'storage_group': 'exp-db', 'external_label': 'VON279', 'library': 'CD-LTO4G1', 'file_family': 'daily-d0-offline', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'exp-db.daily-d0-offline.cpio_odc', 'file_family_width': 2}, 'outfilepath': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel02/D0OFPRD1_t702243438_s103342_p12', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': None, 'curpri': 1}, 'file_size': 1966080000L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7868, 'mc_device': '0,5,1,11', 'compression': 0, 'serial_num': '1310154353', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr208a', 'max_time_in_state': 1200, 'device': 'stkenmvr208a:/dev/rmt/tps0d0n', 'name': 'LTO4_108.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_108.stat', 'mover_address': ('131.225.13.204', 7868), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.204', 49962), 'data_ip': 'stkenmvr208a', 'logname': 'LTO4_108MV', 'hostip': '131.225.13.204', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel02/D0OFPRD1_t702243438_s103342_p12', 'uid': 0, 'uname': 'root', 'mtime': 1257573613, 'rmajor': 0, 'machine': ('Linux', 'stkendca20a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16877, 820620168L, 20L, 1, 1602, 2752, 512L, 1257573601, 1257573601, 1254398419), 'mode_octal': '0100016', 'mode': 33188, 'size_bytes': 1966080000L, 'gname': 'root', 'gid': 0, 'fullname': '/diskc/write-pool-3/data/003000000000000006222948', 'inode': 807545160L, 'minor': 20}, 'version': 'v3_8b CVS $Revision: 1.922 $ ', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/diskc/write-pool-3/data/003000000000000006222948', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel02/.(access)(003000000000000006222948)', 'fc': {'size': 1966080000L, 'external_label': 'VON279', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/diskc/write-pool-3/data/003000000000000006222948', 'at_the_top': 980, 'local_inode': 270526700L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.12', 49619), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573695.268332, 'encp_start_time': 1257573611.8976369, 'job_queued': 1257573614.067333, 't0': 1257573611, 'in_queue': 81.200711965560913}, 'user_level': 'user2', 'unique_id': 'stkendca20a.fnal.gov-1257573613-23649-0'} Thread client_connect_thread 00:01:35 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV write_client exits Thread net_thread 00:01:35 stkendca20a.fnal.gov 023649 root I ENCP Control socket ('131.225.13.12', 49619) is connected to ('131.225.13.204', 49963) for stkendca20a.fnal.gov-1257573613-23649-0. elapsed=83.410sec 00:01:35 stkendca20a.fnal.gov 023649 root I ENCP Received callback ticket from mover LTO4_108.mover for transfer stkendca20a.fnal.gov-1257573613-23649-0. elapsed=83.410sec 00:01:35 stkendca20a.fnal.gov 023649 root I ENCP Atempting to connect data socket to mover at ('131.225.13.204', 49962). 00:01:35 stkendca20a.fnal.gov 023649 root I ENCP Connecting data socket to mover (('131.225.13.204', 49962)) with interface 131.225.13.12. elapsed=83.410sec 00:01:35 stkendca20a.fnal.gov 023649 root I ENCP Data socket ('131.225.13.12', 49626) is connected to ('131.225.13.204', 49962). elapsed=83.420sec 00:01:35 stkendca20a.fnal.gov 023649 root I ENCP Opening local file. 00:01:35 stkendca20a.fnal.gov 023640 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:01:35 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV tape position: target 386 current 383 47562 Thread seek_thread 00:01:35 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV tape position: target 86 current 86 0 Thread seek_thread 00:01:35 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV write_tape starting, bytes_to_write=1966080512 Thread tape_thread 00:01:35 stkendca20a.fnal.gov 023649 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:01:35 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV skip_fm 3 Thread seek_thread 00:01:35 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV read_tape starting, bytes_to_read=188467948 Thread tape_thread 00:01:35 stkenmvr208a.fnal.gov 006111 root I LTO4_108MV Write starting. Tape VON279 absolute location in blocks 1206747 Thread tape_thread 00:01:35 stkensrv0n.fnal.gov 028796 enstore I FILSRV Updating update file set deleted = 'n', pnfs_path = '/pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel01/D0OFPRD1_t702245888_s103346_p1', drive = 'stkenmvr208a:/dev/rmt/tps0d0n:1310154353', pnfs_id = '003000000000000006222A10' where bfid = 'CDMS125757369400000'; 00:01:35 stkendca20a.fnal.gov 023640 root I ENCP MSG_TYPE=ENCP_XFER write_to_hsm /diskc/write-pool-3/data/003000000000000006222A10 -> /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel01/D0OFPRD1_t702245888_s103346_p1: 1966080000 bytes copied to VON279 at 28.9 MB/S (64.1 MB/S network) (102 MB/S drive) (33.4 MB/S disk) mover=LTO4_108.mover drive_id=ULTRIUM-TD4 drive_sn=1310154353 drive_vendor=IBM elapsed=83.644 {'storage_group': 'exp-db', 'encp_crc': 1909037629L, 'disk_rate': 33.422315122613462, 'drive_rate': 102.16495143750787, 'driver': 'FTTDriver', 'transfer_rate': 28.893117098950462, 'mover_interface': 'stkenmvr208a', 'overall_rate': 28.72247389866477, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 28.893117098950462, 'encp_ip': '131.225.13.12', 'unique_id': 'stkendca20a.fnal.gov-1257573613-23640-0'} 00:01:35 stkenmvr109a.fnal.gov 031352 root I LTO09MV read bytes 1498719687/1498719687, blocks 11435 header 232 Thread tape_thread 00:01:35 stkendca20a.fnal.gov 023640 root I ENCP File /pnfs/fnal.gov/usr/exp-db/daily/d0-offline/d0ofprd1/2009/11-November/06/channel01/D0OFPRD1_t702245888_s103346_p1 copied successfully. 00:01:35 stkendca20a.fnal.gov 023640 root I ENCP Completed transferring 1966080000 bytes in 1 files in 83.660875082 sec. Overall rate = 28.7 MB/sec. Transfer rate = 28.9 MB/sec. Network rate = 64.1 MB/sec. Drive rate = 102 MB/sec. Disk rate = 33.4 MB/sec. Exit status = 0. 00:01:35 stkenmvr109a.fnal.gov 031352 root I LTO09MV drive stats after read. Tape VOF410 position 83 block 11435 block_size 0 bloc_loc 1029489 tot_blocks 381469000 BOT 0 read_err 0 write_err 0 bytes 1498719687 block_read_tot 18.8496522903 tape_rate 79509142.3393 Thread tape_thread 00:01:35 stkenmvr109a.fnal.gov 031352 root I LTO09MV read_tape exiting, read 1498719687/1498719687 bytes Thread tape_thread 00:01:35 stkenmvr109a.fnal.gov 031352 root I LTO09MV read_tape timing:get_space 0.225145101547 read 12.6866858006 crc_check 5.76063513756 push 0.177186250687 block_read 18.8496522903 idle 0.44608259201 Thread tape_thread 00:01:35 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:35 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:35 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30135 current 30134 116 Thread seek_thread 00:01:35 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:35 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=14238344 Thread tape_thread 00:01:36 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 14238483/14238483, blocks 109 header 139 Thread tape_thread 00:01:36 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 14238344/14238344 bytes Thread net_thread 00:01:36 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:36 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30135 block 109 block_size 0 bloc_loc 5656058 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 14238483 block_read_tot 0.143661737442 tape_rate 99111170.8206 Thread tape_thread 00:01:36 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 14238483/14238483 bytes Thread tape_thread 00:01:36 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00153541564941 read 0.0704336166382 crc_check 0.07075548172 push 0.000937223434448 block_read 0.143661737442 idle 0.00236797332764 Thread tape_thread 00:01:36 stkenmvr109a.fnal.gov 031352 root I LTO09MV write_client: wrote 1498719455/1498719455 bytes Thread net_thread 00:01:36 stkensrv3n.fnal.gov 007327 root I ENCP File /pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/F815C1A7-907E-DC11-86AA-003048726DD9.root transfered for stkensrv3n.fnal.gov-1257573636-7327-0 in 20.7650110722 seconds. 00:01:36 stkenmvr109a.fnal.gov 031352 root I LTO09MV write_client exits Thread net_thread 00:01:36 stkenmvr109a.fnal.gov 031352 root I LTO09MV transfer complete volume=VOF410 location=83 Thread net_thread 00:01:36 last message repeated 1 times 00:01:36 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_rd_access = '26', last_access = '2009-11-07 00:01:36' where label = 'VOF410'; 00:01:36 stkenmvr109a.fnal.gov 031352 root I LTO09MV Sending done to client: {'vc': {'comment': '', 'declared': 1190942903.0, 'blocksize': 131072, 'storage_group': 'cms', 'library': 'CD-LTO3', 'eod_cookie': '0000_000000000_0000258', 'current_location': '0000_000000000_0000083', 'user_inhibit': ['none', 'none'], 'sum_rd_access': 25, 'system_inhibit': ['none', 'full'], 'external_label': 'VOF410', 'wrapper': 'cpio_odc', 'remaining_bytes': 1097543680L, 'sum_mounts': 122, 'capacity_bytes': 429496729600L, 'media_type': 'LTO3', 'last_access': 1257573635.0, 'status': ('ok', None), 'si_time': [0.0, 1192921340.0], 'non_del_files': 257, 'sum_wr_err': 0, 'sum_wr_access': 256, 'file_family': 'CSA07-Skims', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.CSA07-Skims.cpio_odc', 'modification_time': 1200327609.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1192731952.0}, 'outfilepath': '/dev/null', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': 1, 'curpri': 1}, 'file_size': 1498719455L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 300, 'max_failures': 5, 'library': ['CD-LTO3.library_manager', 'CD-LTO3_test.library_manager'], 'local_mover': 0, 'dismount_delay': 20, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7809, 'mc_device': '0,0,1,14', 'compression': 0, 'serial_num': '9210007592', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 57671680.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr109a', 'max_time_in_state': 1200, 'device': 'stkenmvr109a:/dev/rmt/tps0d0n', 'name': 'LTO3_09.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD3', 'statistics_path': '/tmp/enstore/enstore/LTO3_09.stat', 'mover_address': ('131.225.13.139', 7809), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.139', 46907), 'data_ip': 'stkenmvr109a', 'logname': 'LTO09MV', 'hostip': '131.225.13.139', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'lm': {'address': ('131.225.13.4', 7506)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/F815C1A7-907E-DC11-86AA-003048726DD9.root', 'uid': 0, 'uname': 'root', 'rmajor': 0, 'machine': ('Linux', 'stkensrv3n.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 08:50:41 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (33188, 124757760L, 20L, 1, 9811, 5063, 1498719455L, 1247963569, 1192845783, 1192845374), 'mode_octal': '0100744', 'mode': 33252, 'size_bytes': 1498719455L, 'gname': 'root', 'gid': 0, 'fullname': '/dev/null', 'inode': 124757760L, 'minor': 20}, 'version': 'v3_7d CVS $Revision: 1.929 $ encp', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/.(access)(0007000000000000096FA700)', 'status': ('ok', None), 'volume': 'VOF410', 'outfile': '/dev/null', 'fc': {'status': ('ok', None), 'pnfsid': '0007000000000000096FA700', 'pnfs_name0': '/pnfs/fnal.gov/usr/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/F815C1A7-907E-DC11-86AA-003048726DD9.root', 'deleted': 'no', 'external_label': 'VOF410', 'drive': 'stkenmvr106a:/dev/rmt/tps2d0n:1210116666', 'update': '2007-10-19 21:02:57.784073', 'gid': 0, 'location_cookie': '0000_000000000_0000083', 'complete_crc': 3596548126L, 'address': ('131.225.13.59', 7501), 'uid': 0, 'bfid': 'CDMS119284576600000', 'sanity_cookie': (65536L, 1854190849L), 'size': 1498719455L}, 'infilepath': '/pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/F815C1A7-907E-DC11-86AA-003048726DD9.root', 'at_the_top': 14, 'bfid': 'CDMS119284576600000', 'local_inode': 2469L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': 7, 'retry': 0}, 'callback_addr': ('131.225.13.51', 43476), 'work': 'read_from_hsm', 'override_ro_mount': 1, 'times': {'encp_start_time': 1257573635.401355, 'seek_time': 37.1211919(trunc. 4439) Thread net_thread 00:01:36 stkenmvr109a.fnal.gov 031352 root I LTO09MV Done is sent Thread net_thread 00:01:36 stkenmvr109a.fnal.gov 031352 root I LTO09MV maybe_clean Thread net_thread 00:01:36 stkensrv3n.fnal.gov 007327 root I ENCP Received final dialog for stkensrv3n.fnal.gov-1257573636-7327-0. elapsed=60.820sec 00:01:36 stkensrv3n.fnal.gov 007327 root I ENCP leaving transfer_file(). 00:01:36 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:36 stkensrv3n.fnal.gov 007327 root I ENCP MSG_TYPE=ENCP_XFER read_from_hsm /pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/F815C1A7-907E-DC11-86AA-003048726DD9.root -> /dev/null: 1498719455 bytes copied from VOF410 at 68.6 MB/S (68.8 MB/S network) (75.1 MB/S drive) (68.8 MB/S disk) mover=LTO3_09.mover drive_id=ULTRIUM-TD3 drive_sn=9210007592 drive_vendor=IBM elapsed=60.861 {'storage_group': 'cms', 'encp_crc': 3596548126L, 'disk_rate': 68.832160762826376, 'drive_rate': 75.063479329162476, 'driver': 'FTTDriver', 'transfer_rate': 68.636216494052888, 'mover_interface': 'stkenmvr109a', 'overall_rate': 24.617622911007373, 'media_changer': 'SL8500G1.media_changer', 'network_rate': 68.636216494052888, 'encp_ip': '131.225.13.51', 'unique_id': 'stkensrv3n.fnal.gov-1257573636-7327-0'} 00:01:36 stkenmvr109a.fnal.gov 031352 root I LTO09MV returned from maybe_clean Thread net_thread 00:01:36 stkensrv3n.fnal.gov 007327 root I ENCP File /dev/null copied successfully. 00:01:36 stkensrv3n.fnal.gov 007327 root I ENCP Completed transferring 1498719455 bytes in 1 files in 60.8680400848 sec. Overall rate = 24.6 MB/sec. Transfer rate = 68.6 MB/sec. Network rate = 68.8 MB/sec. Drive rate = 75.1 MB/sec. Disk rate = 68.8 MB/sec. Exit status = 0. 00:01:36 stkensrv3n.fnal.gov 007903 root I ENCP Version: v3_7d CVS $Revision: 1.929 $ encp OS: Linux 2.6.9-89.0.9.ELsmp x86_64 Release: Scientific Linux SL release 4.4 (Beryllium), User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Current working directory: stkensrv3n.fnal.gov:/root Command line: /opt/enstore/bin/encp --verbose 10 --bypass-filesystem-max-filesize-check --max-resubmit 7 --delayed-dismount 1 --override-ro-mount --get-bfid CDMS119292132200000 /dev/null 00:01:36 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV skip_fm 75 Thread seek_thread 00:01:36 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV write_tape starting, bytes_to_write=1966080512 Thread tape_thread 00:01:37 stkenmvr194a.fnal.gov 029623 root I LTO4_94MV Write starting. Tape VON278 absolute location in blocks 1050886 Thread tape_thread 00:01:37 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:37 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV read bytes 188467948/188467948, blocks 1438 header 0 Thread tape_thread 00:01:37 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV write_client: wrote 188467948/188467948 bytes Thread net_thread 00:01:37 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV write_client waits for tape thread to exit Thread net_thread 00:01:37 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30136 current 30135 109 Thread seek_thread 00:01:37 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV drive stats after read. Tape VOV146 position 386 block 1438 block_size 0 bloc_loc 3478997 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 188467948 block_read_tot 1.63155245781 tape_rate 115514488.73 Thread tape_thread 00:01:37 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV read_tape exiting, read 188467948/188467948 bytes Thread tape_thread 00:01:37 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV read_tape timing:get_space 0.0162105560303 read 0.677639961243 crc_check 0.925774097443 push 0.0119278430939 block_read 1.63155245781 idle 0.032618522644 Thread tape_thread 00:01:37 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:37 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=18743308 Thread tape_thread 00:01:37 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:01:37 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 18743447/18743447, blocks 144 header 139 Thread tape_thread 00:01:37 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 18743308/18743308 bytes Thread net_thread 00:01:37 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:37 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30136 block 144 block_size 0 bloc_loc 5656203 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 18743447 block_read_tot 0.185482263565 tape_rate 101052503.025 Thread tape_thread 00:01:37 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 18743447/18743447 bytes Thread tape_thread 00:01:37 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00146079063416 read 0.0901906490326 crc_check 0.0925941467285 push 0.0012366771698 block_read 0.185482263565 idle 0.00343632698059 Thread tape_thread 00:01:37 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:37 stkensrv3n.fnal.gov 007903 root I ENCP Creating zero length output files. 00:01:37 stkensrv3n.fnal.gov 007903 root I ENCP Sending /pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/D05F61DD-477F-DC11-859F-001731AF677B.root read request to LM: unique_id: stkensrv3n.fnal.gov-1257573697-7903-0 inputfile: /pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/.(access)(00070000000000000977ED18) outputfile: /dev/null 00:01:37 stkensrv4n.fnal.gov 005806 enstore I LTO3LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:01:37 stkensrv4n.fnal.gov 005806 enstore I LTO3LM read Q'd /pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/D05F61DD-477F-DC11-859F-001731AF677B.root (stkensrv3n.fnal.gov-1257573697-7903-0) -> /dev/null : library=CD-LTO3 family=cms.CSA07-Skims.cpio_odc requester:root 00:01:38 stkenmvr27a.fnal.gov 005886 root I DBT27MV read bytes 9405442304/9405442304, blocks 71758 header 0 Thread tape_thread 00:01:38 stkenmvr27a.fnal.gov 005886 root I DBT27MV write_client: wrote 9405442304/9405442304 bytes Thread net_thread 00:01:38 stkenmvr27a.fnal.gov 005886 root I DBT27MV write_client waits for tape thread to exit Thread net_thread 00:01:38 stkenmvr27a.fnal.gov 005886 root I DBT27MV drive stats after read. Tape VO6043 position 881 block 71758 block_size 0 bloc_loc 1236301 tot_blocks 0 BOT 0 read_err 0 write_err 0 bytes 9405442304 block_read_tot 299.48465538 tape_rate 31405423.066 Thread tape_thread 00:01:38 stkenmvr27a.fnal.gov 005886 root I DBT27MV read_tape exiting, read 9405442304/9405442304 bytes Thread tape_thread 00:01:38 stkenmvr27a.fnal.gov 005886 root I DBT27MV read_tape timing:get_space 1.15006828308 read 252.968675613 crc_check 43.9209849834 push 1.44492650032 block_read 299.48465538 idle 2.75103664398 Thread tape_thread 00:01:38 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV write_client exits Thread net_thread 00:01:38 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV tape position: target 389 current 386 1438 Thread seek_thread 00:01:38 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV skip_fm 3 Thread seek_thread 00:01:38 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV read_tape starting, bytes_to_read=547872928 Thread tape_thread 00:01:38 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:38 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30137 current 30136 144 Thread seek_thread 00:01:38 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:38 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=14930013 Thread tape_thread 00:01:38 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 14930152/14930152, blocks 114 header 139 Thread tape_thread 00:01:38 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 14930013/14930013 bytes Thread net_thread 00:01:38 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:38 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30137 block 114 block_size 0 bloc_loc 5656318 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 14930152 block_read_tot 0.159860134125 tape_rate 93395092.4147 Thread tape_thread 00:01:38 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 14930152/14930152 bytes Thread tape_thread 00:01:38 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00406813621521 read 0.0818994045258 crc_check 0.0729265213013 push 0.00096607208252 block_read 0.159860134125 idle 0.00258493423462 Thread tape_thread 00:01:38 stkendca22a.fnal.gov 005716 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: garbage Storage Group: test File Family: dcache FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0001 --put-cache 0005000000000000005FA590 /diskb/write-pool-2/data/0005000000000000005FA590 00:01:38 stkendca22a.fnal.gov 005737 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: garbage Storage Group: test File Family: dcache FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0003 --put-cache 0005000000000000005FA5F8 /diskb/write-pool-2/data/0005000000000000005FA5F8 00:01:38 stkendca22a.fnal.gov 005749 root I ENCP Version: v3_8b CVS $Revision: 1.922 $ OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: garbage Storage Group: test File Family: dcache FF Wrapper: cpio_odc FF Width: 1 Current working directory: stkendca22a.fnal.gov:/home/enstore/dcache-log Command line: /fnal/ups/prd/encp/v3_8b/Linux-2-6-dcache/encp --verbose 4 --threaded --ecrc --bypass-filesystem-max-filesize-check --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0002 --put-cache 0005000000000000005FA5C8 /diskb/write-pool-2/data/0005000000000000005FA5C8 00:01:39 stkenmvr27a.fnal.gov 005886 root I DBT27MV write_client exits Thread net_thread 00:01:39 stkenmvr27a.fnal.gov 005886 root I DBT27MV transfer complete volume=VO6043 location=881 Thread net_thread 00:01:39 last message repeated 1 times 00:01:39 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_rd_access = '396', last_access = '2009-11-07 00:01:39' where label = 'VO6043'; 00:01:39 stkenmvr27a.fnal.gov 005886 root I DBT27MV Sending done to client: {'vc': {'comment': '', 'declared': 1080687098.0, 'blocksize': 131072, 'storage_group': 'ktev', 'library': 'CD-9940B', 'eod_cookie': '0000_000000000_0001084', 'current_location': '0000_000000000_0000881', 'user_inhibit': ['none', 'none'], 'sum_rd_access': 395, 'system_inhibit': ['none', 'migrating'], 'external_label': 'VO6043', 'wrapper': 'cern', 'remaining_bytes': 968444416L, 'sum_mounts': 9, 'capacity_bytes': 214748364800L, 'media_type': '9940B', 'last_access': 1257573387.0, 'status': ('ok', None), 'si_time': [0.0, 1257568219.0], 'non_del_files': 361, 'sum_wr_err': 0, 'sum_wr_access': 361, 'file_family': 'ktev_migration', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration.cern', 'modification_time': 1221190342.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1082654321.0}, 'outfilepath': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000881', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 9405442304L, 'mover': {'max_buffer': 1048576000L, 'check_written_file': 10, 'max_failures': 5, 'library': 'CD-9940B.library_manager', 'local_mover': 0, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7827, 'mc_device': '0,1,10,2', 'compression': 0, 'serial_num': '479002016447', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 28311552.0, 'vendor_id': 'STK', 'driver': 'FTTDriver', 'host': 'stkenmvr27a', 'max_time_in_state': 1200, 'device': 'stkenmvr27a:/dev/rmt/tps2d0n', 'name': '9940B27.mover', 'send_stats': 1, 'product_id': 'T9940B', 'statistics_path': '/tmp/enstore/enstore/DBT27MV.stat', 'mover_address': ('131.225.13.96', 7827), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.96', 51703), 'data_ip': 'stkenmvr27a', 'logname': 'DBT27MV', 'hostip': '131.225.13.96', 'do_cleaning': 'No', 'media_changer': 'stk.media_changer'}, 'lm': {'address': ('131.225.13.4', 7522)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/UP/UPA2/UPA255/f1', 'uid': 0, 'uname': 'root', 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (33188, 528388840L, 20L, 1, 5406, 2539, 1L, 1082667703, 1082667703, 1082663571), 'mode_octal': '0100744', 'mode': 33252, 'size_bytes': 9405442304L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000881', 'inode': 528388840L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/fs/usr/ktev/migrated_from_dlt/UP/UPA2/UPA255/.(access)(001F000000000000007E92E8)', 'status': ('ok', None), 'volume': 'VO6043', 'outfile': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000881', 'fc': {'status': ('ok', None), 'pnfsid': '001F000000000000007E92E8', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/UP/UPA2/UPA255/f1', 'deleted': 'no', 'external_label': 'VO6043', 'drive': 'stkenmvr11a:/dev/rmt/tps0d0n:4790000019', 'update': None, 'gid': -1, 'location_cookie': '0000_000000000_0000881', 'complete_crc': 2656649566L, 'address': ('131.225.13.59', 7501), 'uid': -1, 'bfid': 'CDMS108266770200001', 'sanity_cookie': (65536L, 932325300L), 'size': 9405442304L}, 'infilepath': '/pnfs/fs/usr/ktev/migrated_from_dlt/UP/UPA2/UPA255/f1', 'at_the_top': 5, 'bfid': 'CDMS108266770200001', 'local_inode': 39503341L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40445), 'work': 'read_from_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257573384.5625391, 'seek_time': 0.0068039894104003906, 'transfer_time': 310.39857006072998, 't0': 1257573384, 'mount_time': 0.019881010055541992, 'lm_dequeued': 1257573388.5330999, 'job_queued': 1257573384.766789, 'drive_call_time': 250.64530420303345, 'in_queue': 3.7658929824829102, 'drive_transfer_time': 300.87385058403015}, 'user_l(trunc. 4149) Thread net_thread 00:01:39 stkenmvr27a.fnal.gov 005886 root I DBT27MV Done is sent Thread net_thread 00:01:39 stkenmvr27a.fnal.gov 005886 root I DBT27MV maybe_clean Thread net_thread 00:01:39 stkenmvr27a.fnal.gov 005886 root I DBT27MV returned from maybe_clean Thread net_thread 00:01:39 stkendca22a.fnal.gov 005716 root I ENCP Configuration Server: no such name: 'garbage.library_manager' 00:01:39 stkendca22a.fnal.gov 005716 root E ENCP INFILE=/diskb/write-pool-2/data/0005000000000000005FA590 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0001 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=0.87 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager' 00:01:39 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:01:39 stkensrv4n.fnal.gov 005806 enstore I LTO3LM read_from_hsm next work on vol=VOF410 mover=LTO3_09.mover requester:root 00:01:39 stkensrv4n.fnal.gov 005806 enstore I LTO3LM HAVE_BOUND:sending read_from_hsm /pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/D05F61DD-477F-DC11-859F-001731AF677B.root to mover LTO3_09.mover ('131.225.13.139', 7809) DEL_DISM 1 00:01:39 stkensrv4n.fnal.gov 005806 enstore I LTO3LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:01:39 stkensrv4n.fnal.gov 005806 enstore I LTO3LM HAVE_BOUND:sending {'vc': {'comment': '', 'declared': 1190942903.0, 'blocksize': 131072, 'storage_group': 'cms', 'library': 'CD-LTO3', 'eod_cookie': '0000_000000000_0000258', 'current_location': '0000_000000000_0000257', 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'full'], 'external_label': 'VOF410', 'wrapper': 'cpio_odc', 'remaining_bytes': 1097543680L, 'sum_mounts': 122, 'sum_rd_access': 26, 'media_type': 'LTO3', 'last_access': 1257573696.0, 'status': ('ok', None), 'si_time': [0.0, 1192921340.0], 'capacity_bytes': 429496729600L, 'non_del_files': 257, 'sum_wr_err': 0, 'sum_wr_access': 256, 'file_family': 'CSA07-Skims', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.CSA07-Skims.cpio_odc', 'modification_time': 1200327609.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1192731952.0}, 'outfilepath': '/dev/null', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': 1, 'curpri': 1}, 'file_size': 1742852427L, 'mover': 'LTO3_09.mover', 'lm': {'address': ('131.225.13.4', 7506)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/D05F61DD-477F-DC11-859F-001731AF677B.root', 'uid': 0, 'uname': 'root', 'gid': 0, 'rmajor': 0, 'machine': ('Linux', 'stkensrv3n.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 08:50:41 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (33188, 125299992L, 20L, 1, 9811, 5063, 1742852427L, 1247963682, 1192921338, 1192920951), 'mode_octal': '0100744', 'mode': 33252, 'gname': 'root', 'size_bytes': 1742852427L, 'fullname': '/dev/null', 'inode': 125299992L, 'minor': 20}, 'version': 'v3_7d CVS $Revision: 1.929 $ encp', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/.(access)(00070000000000000977ED18)', 'status': ('ok', None), 'volume': 'VOF410', 'outfile': '/dev/null', 'fc': {'status': ('ok', None), 'pnfsid': '00070000000000000977ED18', 'pnfs_name0': '/pnfs/fnal.gov/usr/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/D05F61DD-477F-DC11-859F-001731AF677B.root', 'deleted': 'no', 'external_label': 'VOF410', 'drive': 'stkenmvr120a:/dev/rmt/tps4d0n:1210245200', 'update': '2007-10-20 18:02:16.183401', 'gid': 0, 'location_cookie': '0000_000000000_0000257', 'complete_crc': 2815434212L, 'address': ('131.225.13.59', 7501), 'uid': 0, 'bfid': 'CDMS119292132200000', 'sanity_cookie': (65536L, 3167746981L), 'size': 1742852427L}, 'infilepath': '/pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/D05F61DD-477F-DC11-859F-001731AF677B.root', 'at_the_top': 17, 'bfid': 'CDMS119292132200000', 'local_inode': 2469L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': 7, 'retry': 0}, 'callback_addr': ('131.225.13.51', 43484), 'work': 'read_from_hsm', 'override_ro_mount': 1, 'times': {'lm_dequeued': 1257573699.2376521, 'encp_start_time': 1257573696.443423, 'job_queued': 1257573697.7335241, 't0': 1257573696, 'in_queue': 1.5040550231933594}, 'user_level': 'admin', 'unique_id': 'stkensrv3n.fnal.gov-1257573697-7903-0'} to mover 00:01:39 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:01:39 stkendca22a.fnal.gov 005737 root I ENCP Configuration Server: no such name: 'garbage.library_manager' 00:01:39 stkendca22a.fnal.gov 005737 root E ENCP INFILE=/diskb/write-pool-2/data/0005000000000000005FA5F8 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0003 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=0.92 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager' 00:01:39 stkenmvr109a.fnal.gov 031352 root I LTO09MV READ FROM HSM Thread MainThread 00:01:39 stkensrv3n.fnal.gov 007903 root I ENCP Listening for control socket at: ('131.225.13.51', 43484) 00:01:39 stkensrv3n.fnal.gov 007903 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.51', 43484) elapsed=2.830sec 00:01:39 stkenmvr109a.fnal.gov 031352 root I LTO09MV SENDING {'vc': {'comment': '', 'declared': 1190942903.0, 'blocksize': 131072, 'storage_group': 'cms', 'library': 'CD-LTO3', 'eod_cookie': '0000_000000000_0000258', 'current_location': '0000_000000000_0000257', 'user_inhibit': ['none', 'none'], 'sum_rd_access': 26, 'system_inhibit': ['none', 'full'], 'external_label': 'VOF410', 'wrapper': 'cpio_odc', 'remaining_bytes': 1097543680L, 'sum_mounts': 122, 'capacity_bytes': 429496729600L, 'media_type': 'LTO3', 'last_access': 1257573696.0, 'status': ('ok', None), 'si_time': [0.0, 1192921340.0], 'non_del_files': 257, 'sum_wr_err': 0, 'sum_wr_access': 256, 'file_family': 'CSA07-Skims', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.CSA07-Skims.cpio_odc', 'modification_time': 1200327609.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1192731952.0}, 'outfilepath': '/dev/null', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 1, 'delayed_dismount': 1, 'curpri': 1}, 'file_size': 1742852427L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 300, 'max_failures': 5, 'library': ['CD-LTO3.library_manager', 'CD-LTO3_test.library_manager'], 'local_mover': 0, 'dismount_delay': 20, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7809, 'mc_device': '0,0,1,14', 'compression': 0, 'serial_num': '9210007592', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 57671680.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr109a', 'max_time_in_state': 1200, 'device': 'stkenmvr109a:/dev/rmt/tps0d0n', 'name': 'LTO3_09.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD3', 'statistics_path': '/tmp/enstore/enstore/LTO3_09.stat', 'mover_address': ('131.225.13.139', 7809), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.139', 46910), 'data_ip': 'stkenmvr109a', 'logname': 'LTO09MV', 'hostip': '131.225.13.139', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer', 'min_buffer': 67108864L}, 'lm': {'address': ('131.225.13.4', 7506)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/D05F61DD-477F-DC11-859F-001731AF677B.root', 'uid': 0, 'uname': 'root', 'rmajor': 0, 'machine': ('Linux', 'stkensrv3n.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 08:50:41 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (33188, 125299992L, 20L, 1, 9811, 5063, 1742852427L, 1247963682, 1192921338, 1192920951), 'mode_octal': '0100744', 'mode': 33252, 'size_bytes': 1742852427L, 'gname': 'root', 'gid': 0, 'fullname': '/dev/null', 'inode': 125299992L, 'minor': 20}, 'version': 'v3_7d CVS $Revision: 1.929 $ encp', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/.(access)(00070000000000000977ED18)', 'status': ('ok', None), 'volume': 'VOF410', 'outfile': '/dev/null', 'fc': {'status': ('ok', None), 'pnfsid': '00070000000000000977ED18', 'pnfs_name0': '/pnfs/fnal.gov/usr/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/D05F61DD-477F-DC11-859F-001731AF677B.root', 'deleted': 'no', 'external_label': 'VOF410', 'drive': 'stkenmvr120a:/dev/rmt/tps4d0n:1210245200', 'update': '2007-10-20 18:02:16.183401', 'gid': 0, 'location_cookie': '0000_000000000_0000257', 'complete_crc': 2815434212L, 'address': ('131.225.13.59', 7501), 'uid': 0, 'bfid': 'CDMS119292132200000', 'sanity_cookie': (65536L, 3167746981L), 'size': 1742852427L}, 'infilepath': '/pnfs/cms/WAX/11/store/CSA07/skims/2007/10/18/CSA07-CSA07AllEvents-Chowder-B4-PDAllEvents-Skims1/0001/D05F61DD-477F-DC11-859F-001731AF677B.root', 'at_the_top': 17, 'bfid': 'CDMS119292132200000', 'local_inode': 2469L, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': 7, 'retry': 0}, 'callback_addr': ('131.225.13.51', 43484), 'work': 'read_from_hsm', 'override_ro_mount': 1, 'times': {'lm_dequeued': 1257573699.2376521, 'encp_start_time': 1257573696.443423, 'job(trunc. 4237) Thread client_connect_thread 00:01:39 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:01:39 stkendca22a.fnal.gov 005749 root I ENCP Configuration Server: no such name: 'garbage.library_manager' 00:01:39 stkendca22a.fnal.gov 005749 root E ENCP INFILE=/diskb/write-pool-2/data/0005000000000000005FA5C8 OUTFILE=/pnfs/fnal.gov/usr/fermigrid/volatile/fermilab/timur-test/broken-lib/timur-test-0002 FILESIZE=801512 LABEL= LOCATION= DRIVE= DRIVE_SN= TRANSFER_TIME=0.00 SEEK_TIME=0.00 MOUNT_TIME=0.00 QWAIT_TIME=0.00 TIME2NOW=0.94 CRC= STATUS=KEYERROR Configuration Server: no such name: 'garbage.library_manager' 00:01:39 stkensrv3n.fnal.gov 007903 root I ENCP Control socket ('131.225.13.51', 43484) is connected to ('131.225.13.139', 46911) for stkensrv3n.fnal.gov-1257573697-7903-0. elapsed=2.850sec 00:01:39 stkensrv3n.fnal.gov 007903 root I ENCP Received callback ticket from mover LTO3_09.mover for transfer stkensrv3n.fnal.gov-1257573697-7903-0. elapsed=2.850sec 00:01:39 stkensrv3n.fnal.gov 007903 root I ENCP Attempting to connect data socket to mover at ('131.225.13.139', 46910). 00:01:39 stkensrv3n.fnal.gov 007903 root I ENCP Connecting data socket to mover (('131.225.13.139', 46910)) with interface 131.225.13.51. elapsed=2.850sec 00:01:39 stkensrv3n.fnal.gov 007903 root I ENCP Data socket ('131.225.13.51', 43485) is connected to ('131.225.13.139', 46910). elapsed=2.850sec 00:01:39 last message repeated 1 times 00:01:39 stkensrv3n.fnal.gov 007903 root I ENCP Opening local file. 00:01:39 stkenmvr109a.fnal.gov 031352 root I LTO09MV tape position: target 257 current 83 11435 Thread seek_thread 00:01:39 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:39 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:39 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30138 current 30137 114 Thread seek_thread 00:01:39 stkendm7a.fnal.gov 000642 root I ENCP File /pnfs/fs/usr/ktev/migrated_from_dlt/UP/UPA2/UPA255/f1 transfered for stkendm7a.fnal.gov-1257573384-642-423 in 311.047068834 seconds. Thread READ_2 00:01:39 stkendm7a.fnal.gov 000642 root I ENCP Received final dialog for stkendm7a.fnal.gov-1257573384-642-423. elapsed=5479.390sec Thread READ_2 00:01:39 stkendm7a.fnal.gov 000642 root I ENCP leaving transfer_file(). Thread READ_2 00:01:39 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:39 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=15086054 Thread tape_thread 00:01:39 stkensrv4n.fnal.gov 005795 enstore I 9940BLM 00:01:39 stkensrv4n.fnal.gov 005795 enstore I 9940BLM read_from_hsm next work on vol=VO6043 mover=9940B27.mover requester:root 00:01:39 stkensrv4n.fnal.gov 005795 enstore I 9940BLM HAVE_BOUND:sending read_from_hsm /pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/f7 to mover 9940B27.mover ('131.225.13.96', 7827) DEL_DISM 2 00:01:39 stkensrv4n.fnal.gov 005795 enstore I 9940BLM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : ktev 00:01:39 stkensrv4n.fnal.gov 005795 enstore I 9940BLM HAVE_BOUND:sending {'vc': {'comment': '', 'declared': 1080687098.0, 'blocksize': 131072, 'storage_group': 'ktev', 'library': 'CD-9940B', 'eod_cookie': '0000_000000000_0001084', 'current_location': '0000_000000000_0000884', 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'migrating'], 'external_label': 'VO6043', 'wrapper': 'cern', 'remaining_bytes': 968444416L, 'sum_mounts': 9, 'sum_rd_access': 394, 'media_type': '9940B', 'last_access': 1257573386.0, 'status': ('ok', None), 'si_time': [0.0, 1257568219.0], 'capacity_bytes': 214748364800L, 'non_del_files': 361, 'sum_wr_err': 0, 'sum_wr_access': 361, 'file_family': 'ktev_migration', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration.cern', 'modification_time': 1221190342.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1082654321.0}, 'outfilepath': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000884', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 125779816L, 'mover': '9940B27.mover', 'lm': {'address': ('131.225.13.4', 7522)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/f7', 'uid': 0, 'uname': 'root', 'gid': 0, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (33188, 528391288L, 20L, 1, 5406, 2539, 125779816L, 1082667723, 1082667723, 1082666001), 'mode_octal': '0100744', 'mode': 33252, 'gname': 'root', 'size_bytes': 125779816L, 'fullname': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000884', 'inode': 528391288L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/.(access)(001F000000000000007E9C78)', 'status': ('ok', None), 'volume': 'VO6043', 'outfile': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000884', 'fc': {'status': ('ok', None), 'pnfsid': '001F000000000000007E9C78', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/f7', 'deleted': 'no', 'external_label': 'VO6043', 'drive': 'stkenmvr11a:/dev/rmt/tps0d0n:4790000019', 'update': None, 'gid': -1, 'location_cookie': '0000_000000000_0000884', 'complete_crc': 51423070L, 'address': ('131.225.13.59', 7501), 'uid': -1, 'bfid': 'CDMS108266772200000', 'sanity_cookie': (65536L, 3360354394L), 'size': 125779816L}, 'infilepath': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/f7', 'at_the_top': 1897, 'bfid': 'CDMS108266772200000', 'local_inode': 39503342L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40448), 'work': 'read_from_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573699.7895319, 'encp_start_time': 1257573386.623035, 'job_queued': 1257573386.829046, 't0': 1257573386, 'in_queue': 312.96004605293274}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573386-642-424'} to mover 00:01:39 stkensrv4n.fnal.gov 005795 enstore I 9940BLM 00:01:39 stkenmvr27a.fnal.gov 005886 root I DBT27MV READ FROM HSM Thread MainThread 00:01:39 stkenmvr27a.fnal.gov 005886 root I DBT27MV SENDING {'vc': {'comment': '', 'declared': 1080687098.0, 'blocksize': 131072, 'storage_group': 'ktev', 'library': 'CD-9940B', 'eod_cookie': '0000_000000000_0001084', 'current_location': '0000_000000000_0000884', 'user_inhibit': ['none', 'none'], 'sum_rd_access': 394, 'system_inhibit': ['none', 'migrating'], 'external_label': 'VO6043', 'wrapper': 'cern', 'remaining_bytes': 968444416L, 'sum_mounts': 9, 'capacity_bytes': 214748364800L, 'media_type': '9940B', 'last_access': 1257573386.0, 'status': ('ok', None), 'si_time': [0.0, 1257568219.0], 'non_del_files': 361, 'sum_wr_err': 0, 'sum_wr_access': 361, 'file_family': 'ktev_migration', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration.cern', 'modification_time': 1221190342.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1082654321.0}, 'outfilepath': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000884', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 125779816L, 'mover': {'max_buffer': 1048576000L, 'check_written_file': 10, 'max_failures': 5, 'library': 'CD-9940B.library_manager', 'local_mover': 0, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7827, 'mc_device': '0,1,10,2', 'compression': 0, 'serial_num': '479002016447', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 28311552.0, 'vendor_id': 'STK', 'driver': 'FTTDriver', 'host': 'stkenmvr27a', 'max_time_in_state': 1200, 'device': 'stkenmvr27a:/dev/rmt/tps2d0n', 'name': '9940B27.mover', 'send_stats': 1, 'product_id': 'T9940B', 'statistics_path': '/tmp/enstore/enstore/DBT27MV.stat', 'mover_address': ('131.225.13.96', 7827), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.96', 51715), 'data_ip': 'stkenmvr27a', 'logname': 'DBT27MV', 'hostip': '131.225.13.96', 'do_cleaning': 'No', 'media_changer': 'stk.media_changer'}, 'lm': {'address': ('131.225.13.4', 7522)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/f7', 'uid': 0, 'uname': 'root', 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (33188, 528391288L, 20L, 1, 5406, 2539, 125779816L, 1082667723, 1082667723, 1082666001), 'mode_octal': '0100744', 'mode': 33252, 'size_bytes': 125779816L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000884', 'inode': 528391288L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/.(access)(001F000000000000007E9C78)', 'status': ('ok', None), 'volume': 'VO6043', 'outfile': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000884', 'fc': {'status': ('ok', None), 'pnfsid': '001F000000000000007E9C78', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/f7', 'deleted': 'no', 'external_label': 'VO6043', 'drive': 'stkenmvr11a:/dev/rmt/tps0d0n:4790000019', 'update': None, 'gid': -1, 'location_cookie': '0000_000000000_0000884', 'complete_crc': 51423070L, 'address': ('131.225.13.59', 7501), 'uid': -1, 'bfid': 'CDMS108266772200000', 'sanity_cookie': (65536L, 3360354394L), 'size': 125779816L}, 'infilepath': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/f7', 'at_the_top': 1897, 'bfid': 'CDMS108266772200000', 'local_inode': 39503342L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40448), 'work': 'read_from_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573699.7895319, 'encp_start_time': 1257573386.623035, 'job_queued': 1257573386.829046, 't0': 1257573386, 'in_queue': 312.96004605293274}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573386-642-424'} Thread client_connect_thread 00:01:39 stkendm7a.fnal.gov 000642 root I ENCP Listening for control socket at: ('131.225.13.19', 40448) Thread READ 00:01:39 stkendm7a.fnal.gov 000642 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.19', 40448) elapsed=5479.540sec Thread READ 00:01:39 stkendm7a.fnal.gov 000642 root I ENCP Control socket ('131.225.13.19', 40448) is connected to ('131.225.13.96', 51716) for stkendm7a.fnal.gov-1257573386-642-424. elapsed=5479.540sec Thread READ 00:01:39 stkendm7a.fnal.gov 000642 root I ENCP Received callback ticket from mover 9940B27.mover for transfer stkendm7a.fnal.gov-1257573386-642-424. elapsed=5479.550sec Thread READ 00:01:39 stkendm7a.fnal.gov 000642 root I ENCP Attempting to connect data socket to mover at ('131.225.13.96', 51715). Thread READ 00:01:39 stkendm7a.fnal.gov 000642 root I ENCP Connecting data socket to mover (('131.225.13.96', 51715)) with interface 131.225.13.19. elapsed=5479.550sec Thread READ 00:01:39 stkendm7a.fnal.gov 000642 root I ENCP Data socket ('131.225.13.19', 40545) is connected to ('131.225.13.96', 51715). elapsed=5479.550sec Thread READ 00:01:39 stkendm7a.fnal.gov 000642 root I ENCP Opening local file. Thread READ 00:01:39 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 15086193/15086193, blocks 116 header 139 Thread tape_thread 00:01:39 stkenmvr27a.fnal.gov 005886 root I DBT27MV tape position: target 884 current 881 71758 Thread seek_thread 00:01:39 stkenmvr27a.fnal.gov 005886 root I DBT27MV skip_fm 3 Thread seek_thread 00:01:39 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 15086054/15086054 bytes Thread net_thread 00:01:39 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:39 stkenmvr27a.fnal.gov 005886 root I DBT27MV read_tape starting, bytes_to_read=125779816 Thread tape_thread 00:01:39 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30138 block 116 block_size 0 bloc_loc 5656435 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 15086193 block_read_tot 0.15095448494 tape_rate 99938686.8568 Thread tape_thread 00:01:39 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 15086193/15086193 bytes Thread tape_thread 00:01:39 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00199818611145 read 0.0736215114594 crc_check 0.0743560791016 push 0.000978708267212 block_read 0.15095448494 idle 0.00276112556458 Thread tape_thread 00:01:40 stkendm7a.fnal.gov 000642 root I ENCP MSG_TYPE=ENCP_XFER read_from_hsm /pnfs/fs/usr/ktev/migrated_from_dlt/UP/UPA2/UPA255/f1 -> /data/data1/Migration_Spool/VO6043:0000_000000000_0000881: 9405442304 bytes copied from VO6043 at 28.8 MB/S (38.2 MB/S network) (29.8 MB/S drive) (62.1 MB/S disk) mover=9940B27.mover drive_id=T9940B drive_sn=479002016447 drive_vendor=STK elapsed=315.51 {'storage_group': 'ktev', 'encp_crc': 2656649566L, 'disk_rate': 62.064573991180431, 'drive_rate': 29.812257669964897, 'driver': 'FTTDriver', 'transfer_rate': 28.836003058594176, 'mover_interface': 'stkenmvr27a', 'overall_rate': 28.798755227216262, 'media_changer': 'stk.media_changer', 'network_rate': 28.836003058594176, 'encp_ip': '131.225.13.19', 'unique_id': 'stkendm7a.fnal.gov-1257573384-642-423'} Thread READ_2 00:01:40 stkendm7a.fnal.gov 000642 root I ENCP File /data/data1/Migration_Spool/VO6043:0000_000000000_0000881 copied successfully. Thread READ_2 00:01:40 stkendm7a.fnal.gov 000642 root I ENCP Completed transferring 9405442304 bytes in 1 files in 315.524330854 sec. Overall rate = 28.8 MB/sec. Transfer rate = 28.8 MB/sec. Network rate = 38.2 MB/sec. Drive rate = 29.8 MB/sec. Disk rate = 62.1 MB/sec. Exit status = 0. Thread READ_2 00:01:40 stkendm7a.fnal.gov 000642 root I ENCP Version: v3_8 CVS $Revision: 1.963 $ encp.pyc OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson), User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Current working directory: stkendm7a.fnal.gov:/opt/enstore/src Command line: encp --delayed-dismount 2 --ignore-fair-share --bypass-filesystem-max-filesize-check --priority 0 --threaded /pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM10/f0 /data/data1/Migration_Spool/VO6043:0000_000000000_0000890 Thread READ_2 00:01:40 stkendm7a.fnal.gov 000642 root I ENCP Creating zero length output files. Thread READ_2 00:01:40 stkendm7a.fnal.gov 000642 root I ENCP Sending /pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM10/f0 read request to LM: unique_id: stkendm7a.fnal.gov-1257573700-642-432 inputfile: /pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM10/.(access)(001F000000000000007E9800) outputfile: /data/data1/Migration_Spool/VO6043:0000_000000000_0000890 Thread READ_2 00:01:40 stkensrv4n.fnal.gov 005795 enstore I 9940BLM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : ktev 00:01:40 stkensrv4n.fnal.gov 005795 enstore I 9940BLM read Q'd /pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM10/f0 (stkendm7a.fnal.gov-1257573700-642-432) -> /data/data1/Migration_Spool/VO6043:0000_000000000_0000890 : library=CD-9940B family=ktev.ktev_migration.cern requester:root 00:01:40 stkensrv4n.fnal.gov 005795 enstore I 9940BLM 00:01:40 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:40 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30139 current 30138 116 Thread seek_thread 00:01:40 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:40 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=56550460 Thread tape_thread 00:01:41 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 56550599/56550599, blocks 432 header 139 Thread tape_thread 00:01:41 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 56550460/56550460 bytes Thread net_thread 00:01:41 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:41 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30139 block 432 block_size 0 bloc_loc 5656868 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 56550599 block_read_tot 0.515256166458 tape_rate 109752396.344 Thread tape_thread 00:01:41 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 56550599/56550599 bytes Thread tape_thread 00:01:41 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00410318374634 read 0.230164051056 crc_check 0.277226686478 push 0.00376224517822 block_read 0.515256166458 idle 0.00933957099915 Thread tape_thread 00:01:41 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:42 stkenmvr27a.fnal.gov 005886 root I DBT27MV read bytes 125779816/125779816, blocks 960 header 0 Thread tape_thread 00:01:42 stkenmvr27a.fnal.gov 005886 root I DBT27MV drive stats after read. Tape VO6043 position 884 block 960 block_size 0 bloc_loc 1237266 tot_blocks 0 BOT 0 read_err 0 write_err 0 bytes 125779816 block_read_tot 2.1272251606 tape_rate 59128586.0706 Thread tape_thread 00:01:42 stkenmvr27a.fnal.gov 005886 root I DBT27MV read_tape exiting, read 125779816/125779816 bytes Thread tape_thread 00:01:42 stkenmvr27a.fnal.gov 005886 root I DBT27MV read_tape timing:get_space 0.0638384819031 read 1.45084166527 crc_check 0.592219829559 push 0.0203251838684 block_read 2.1272251606 idle 0.0372474193573 Thread tape_thread 00:01:42 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:42 stkenmvr27a.fnal.gov 005886 root I DBT27MV write_client: wrote 125779816/125779816 bytes Thread net_thread 00:01:42 stkenmvr27a.fnal.gov 005886 root I DBT27MV write_client exits Thread net_thread 00:01:42 stkenmvr27a.fnal.gov 005886 root I DBT27MV transfer complete volume=VO6043 location=884 Thread net_thread 00:01:42 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30140 current 30139 432 Thread seek_thread 00:01:42 last message repeated 1 times 00:01:42 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_rd_access = '397', last_access = '2009-11-07 00:01:42' where label = 'VO6043'; 00:01:42 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:42 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=9439829 Thread tape_thread 00:01:42 stkenmvr27a.fnal.gov 005886 root I DBT27MV Sending done to client: {'vc': {'comment': '', 'declared': 1080687098.0, 'blocksize': 131072, 'storage_group': 'ktev', 'library': 'CD-9940B', 'eod_cookie': '0000_000000000_0001084', 'current_location': '0000_000000000_0000884', 'user_inhibit': ['none', 'none'], 'sum_rd_access': 396, 'system_inhibit': ['none', 'migrating'], 'external_label': 'VO6043', 'wrapper': 'cern', 'remaining_bytes': 968444416L, 'sum_mounts': 9, 'capacity_bytes': 214748364800L, 'media_type': '9940B', 'last_access': 1257573699.0, 'status': ('ok', None), 'si_time': [0.0, 1257568219.0], 'non_del_files': 361, 'sum_wr_err': 0, 'sum_wr_access': 361, 'file_family': 'ktev_migration', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration.cern', 'modification_time': 1221190342.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1082654321.0}, 'outfilepath': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000884', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 125779816L, 'mover': {'max_buffer': 1048576000L, 'check_written_file': 10, 'max_failures': 5, 'library': 'CD-9940B.library_manager', 'local_mover': 0, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7827, 'mc_device': '0,1,10,2', 'compression': 0, 'serial_num': '479002016447', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 28311552.0, 'vendor_id': 'STK', 'driver': 'FTTDriver', 'host': 'stkenmvr27a', 'max_time_in_state': 1200, 'device': 'stkenmvr27a:/dev/rmt/tps2d0n', 'name': '9940B27.mover', 'send_stats': 1, 'product_id': 'T9940B', 'statistics_path': '/tmp/enstore/enstore/DBT27MV.stat', 'mover_address': ('131.225.13.96', 7827), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.96', 51715), 'data_ip': 'stkenmvr27a', 'logname': 'DBT27MV', 'hostip': '131.225.13.96', 'do_cleaning': 'No', 'media_changer': 'stk.media_changer'}, 'lm': {'address': ('131.225.13.4', 7522)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/f7', 'uid': 0, 'uname': 'root', 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (33188, 528391288L, 20L, 1, 5406, 2539, 125779816L, 1082667723, 1082667723, 1082666001), 'mode_octal': '0100744', 'mode': 33252, 'size_bytes': 125779816L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000884', 'inode': 528391288L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/.(access)(001F000000000000007E9C78)', 'status': ('ok', None), 'volume': 'VO6043', 'outfile': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000884', 'fc': {'status': ('ok', None), 'pnfsid': '001F000000000000007E9C78', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/f7', 'deleted': 'no', 'external_label': 'VO6043', 'drive': 'stkenmvr11a:/dev/rmt/tps0d0n:4790000019', 'update': None, 'gid': -1, 'location_cookie': '0000_000000000_0000884', 'complete_crc': 51423070L, 'address': ('131.225.13.59', 7501), 'uid': -1, 'bfid': 'CDMS108266772200000', 'sanity_cookie': (65536L, 3360354394L), 'size': 125779816L}, 'infilepath': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/f7', 'at_the_top': 1897, 'bfid': 'CDMS108266772200000', 'local_inode': 39503342L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40448), 'work': 'read_from_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257573386.623035, 'seek_time': 0.0068609714508056641, 'transfer_time': 2.6823248863220215, 't0': 1257573386, 'mount_time': 0.022632122039794922, 'lm_dequeued': 1257573699.7895319, 'job_queued': 1257573386.829046, 'drive_call_time': 1.4187531471252441, 'in_queue': 312.96004605293274, 'drive_transfer_time': 2.1463158130645752}, '(trunc. 4155) Thread net_thread 00:01:42 stkenmvr27a.fnal.gov 005886 root I DBT27MV Done is sent Thread net_thread 00:01:42 stkenmvr27a.fnal.gov 005886 root I DBT27MV maybe_clean Thread net_thread 00:01:42 stkenmvr27a.fnal.gov 005886 root I DBT27MV returned from maybe_clean Thread net_thread 00:01:42 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 9439968/9439968, blocks 73 header 139 Thread tape_thread 00:01:42 stkendm7a.fnal.gov 000642 root I ENCP File /pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/f7 transfered for stkendm7a.fnal.gov-1257573386-642-424 in 2.81859707832 seconds. Thread READ 00:01:42 stkendm7a.fnal.gov 000642 root I ENCP Received final dialog for stkendm7a.fnal.gov-1257573386-642-424. elapsed=5482.410sec Thread READ 00:01:42 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 9439829/9439829 bytes Thread net_thread 00:01:42 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:42 stkendm7a.fnal.gov 000642 root I ENCP leaving transfer_file(). Thread READ 00:01:42 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30140 block 73 block_size 0 bloc_loc 5656942 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 9439968 block_read_tot 0.101886510849 tape_rate 92651793.8571 Thread tape_thread 00:01:42 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 9439968/9439968 bytes Thread tape_thread 00:01:42 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00133109092712 read 0.053459405899 crc_check 0.0464670658112 push 0.00062894821167 block_read 0.101886510849 idle 0.00186419487 Thread tape_thread 00:01:42 stkensrv4n.fnal.gov 005795 enstore I 9940BLM 00:01:42 stkensrv4n.fnal.gov 005795 enstore I 9940BLM read_from_hsm next work on vol=VO6043 mover=9940B27.mover requester:root 00:01:42 stkensrv4n.fnal.gov 005795 enstore I 9940BLM HAVE_BOUND:sending read_from_hsm /pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/f6 to mover 9940B27.mover ('131.225.13.96', 7827) DEL_DISM 2 00:01:42 stkendm7a.fnal.gov 000642 root I ENCP MSG_TYPE=ENCP_XFER read_from_hsm /pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/f7 -> /data/data1/Migration_Spool/VO6043:0000_000000000_0000884: 125779816 bytes copied from VO6043 at 42.4 MB/S (60.8 MB/S network) (55.9 MB/S drive) (68.5 MB/S disk) mover=9940B27.mover drive_id=T9940B drive_sn=479002016447 drive_vendor=STK elapsed=316.17 {'storage_group': 'ktev', 'encp_crc': 51423070L, 'disk_rate': 68.539585924583051, 'drive_rate': 55.887851783671749, 'driver': 'FTTDriver', 'transfer_rate': 42.352616181969033, 'mover_interface': 'stkenmvr27a', 'overall_rate': 40.913375364097618, 'media_changer': 'stk.media_changer', 'network_rate': 42.352616181969033, 'encp_ip': '131.225.13.19', 'unique_id': 'stkendm7a.fnal.gov-1257573386-642-424'} Thread READ 00:01:42 stkensrv4n.fnal.gov 005795 enstore I 9940BLM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : ktev 00:01:42 stkensrv4n.fnal.gov 005795 enstore I 9940BLM HAVE_BOUND:sending {'vc': {'comment': '', 'declared': 1080687098.0, 'blocksize': 131072, 'storage_group': 'ktev', 'library': 'CD-9940B', 'eod_cookie': '0000_000000000_0001084', 'current_location': '0000_000000000_0000887', 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'migrating'], 'external_label': 'VO6043', 'wrapper': 'cern', 'remaining_bytes': 968444416L, 'sum_mounts': 9, 'sum_rd_access': 395, 'media_type': '9940B', 'last_access': 1257573387.0, 'status': ('ok', None), 'si_time': [0.0, 1257568219.0], 'capacity_bytes': 214748364800L, 'non_del_files': 361, 'sum_wr_err': 0, 'sum_wr_access': 361, 'file_family': 'ktev_migration', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration.cern', 'modification_time': 1221190342.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1082654321.0}, 'outfilepath': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000887', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 652733196L, 'mover': '9940B27.mover', 'lm': {'address': ('131.225.13.4', 7522)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/f6', 'uid': 0, 'uname': 'root', 'gid': 0, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (33188, 528391024L, 20L, 1, 5406, 2539, 652733196L, 1082667776, 1082667776, 1082665884), 'mode_octal': '0100744', 'mode': 33252, 'gname': 'root', 'size_bytes': 652733196L, 'fullname': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000887', 'inode': 528391024L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/.(access)(001F000000000000007E9B70)', 'status': ('ok', None), 'volume': 'VO6043', 'outfile': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000887', 'fc': {'status': ('ok', None), 'pnfsid': '001F000000000000007E9B70', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/f6', 'deleted': 'no', 'external_label': 'VO6043', 'drive': 'stkenmvr11a:/dev/rmt/tps0d0n:4790000019', 'update': None, 'gid': -1, 'location_cookie': '0000_000000000_0000887', 'complete_crc': 2842112764L, 'address': ('131.225.13.59', 7501), 'uid': -1, 'bfid': 'CDMS108266777500000', 'sanity_cookie': (65536L, 256914739L), 'size': 652733196L}, 'infilepath': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/f6', 'at_the_top': 15, 'bfid': 'CDMS108266777500000', 'local_inode': 39503343L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40450), 'work': 'read_from_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573702.7943521, 'encp_start_time': 1257573388.4220531, 'job_queued': 1257573388.568013, 't0': 1257573388, 'in_queue': 314.22586607933044}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573388-642-425'} to mover 00:01:42 stkensrv4n.fnal.gov 005795 enstore I 9940BLM 00:01:42 stkendm7a.fnal.gov 000642 root I ENCP File /data/data1/Migration_Spool/VO6043:0000_000000000_0000884 copied successfully. Thread READ 00:01:42 stkendm7a.fnal.gov 000642 root I ENCP Completed transferring 125779816 bytes in 1 files in 316.181791067 sec. Overall rate = 40.9 MB/sec. Transfer rate = 42.4 MB/sec. Network rate = 60.8 MB/sec. Drive rate = 55.9 MB/sec. Disk rate = 68.5 MB/sec. Exit status = 0. Thread READ 00:01:42 stkenmvr27a.fnal.gov 005886 root I DBT27MV READ FROM HSM Thread MainThread 00:01:42 stkendm7a.fnal.gov 000642 root I MIGRATION Listening for control socket at: ('131.225.13.19', 40450) Thread READ_1 00:01:42 stkendm7a.fnal.gov 000642 root I MIGRATION Waiting for mover to connect control socket at address: ('131.225.13.19', 40450) elapsed=5482.530sec Thread READ_1 00:01:42 stkenmvr27a.fnal.gov 005886 root I DBT27MV SENDING {'vc': {'comment': '', 'declared': 1080687098.0, 'blocksize': 131072, 'storage_group': 'ktev', 'library': 'CD-9940B', 'eod_cookie': '0000_000000000_0001084', 'current_location': '0000_000000000_0000887', 'user_inhibit': ['none', 'none'], 'sum_rd_access': 395, 'system_inhibit': ['none', 'migrating'], 'external_label': 'VO6043', 'wrapper': 'cern', 'remaining_bytes': 968444416L, 'sum_mounts': 9, 'capacity_bytes': 214748364800L, 'media_type': '9940B', 'last_access': 1257573387.0, 'status': ('ok', None), 'si_time': [0.0, 1257568219.0], 'non_del_files': 361, 'sum_wr_err': 0, 'sum_wr_access': 361, 'file_family': 'ktev_migration', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration.cern', 'modification_time': 1221190342.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1082654321.0}, 'outfilepath': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000887', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 652733196L, 'mover': {'max_buffer': 1048576000L, 'check_written_file': 10, 'max_failures': 5, 'library': 'CD-9940B.library_manager', 'local_mover': 0, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7827, 'mc_device': '0,1,10,2', 'compression': 0, 'serial_num': '479002016447', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 28311552.0, 'vendor_id': 'STK', 'driver': 'FTTDriver', 'host': 'stkenmvr27a', 'max_time_in_state': 1200, 'device': 'stkenmvr27a:/dev/rmt/tps2d0n', 'name': '9940B27.mover', 'send_stats': 1, 'product_id': 'T9940B', 'statistics_path': '/tmp/enstore/enstore/DBT27MV.stat', 'mover_address': ('131.225.13.96', 7827), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.96', 51717), 'data_ip': 'stkenmvr27a', 'logname': 'DBT27MV', 'hostip': '131.225.13.96', 'do_cleaning': 'No', 'media_changer': 'stk.media_changer'}, 'lm': {'address': ('131.225.13.4', 7522)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/f6', 'uid': 0, 'uname': 'root', 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (33188, 528391024L, 20L, 1, 5406, 2539, 652733196L, 1082667776, 1082667776, 1082665884), 'mode_octal': '0100744', 'mode': 33252, 'size_bytes': 652733196L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000887', 'inode': 528391024L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/.(access)(001F000000000000007E9B70)', 'status': ('ok', None), 'volume': 'VO6043', 'outfile': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000887', 'fc': {'status': ('ok', None), 'pnfsid': '001F000000000000007E9B70', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/f6', 'deleted': 'no', 'external_label': 'VO6043', 'drive': 'stkenmvr11a:/dev/rmt/tps0d0n:4790000019', 'update': None, 'gid': -1, 'location_cookie': '0000_000000000_0000887', 'complete_crc': 2842112764L, 'address': ('131.225.13.59', 7501), 'uid': -1, 'bfid': 'CDMS108266777500000', 'sanity_cookie': (65536L, 256914739L), 'size': 652733196L}, 'infilepath': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/f6', 'at_the_top': 15, 'bfid': 'CDMS108266777500000', 'local_inode': 39503343L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40450), 'work': 'read_from_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573702.7943521, 'encp_start_time': 1257573388.4220531, 'job_queued': 1257573388.568013, 't0': 1257573388, 'in_queue': 314.22586607933044}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573388-642-425'} Thread client_connect_thread 00:01:42 stkendm7a.fnal.gov 000642 root I MIGRATION Control socket ('131.225.13.19', 40450) is connected to ('131.225.13.96', 51718) for stkendm7a.fnal.gov-1257573388-642-425. elapsed=5482.540sec Thread READ_1 00:01:42 stkendm7a.fnal.gov 000642 root I MIGRATION Received callback ticket from mover 9940B27.mover for transfer stkendm7a.fnal.gov-1257573388-642-425. elapsed=5482.550sec Thread READ_1 00:01:42 stkendm7a.fnal.gov 000642 root I MIGRATION Attempting to connect data socket to mover at ('131.225.13.96', 51717). Thread READ_1 00:01:42 stkendm7a.fnal.gov 000642 root I MIGRATION Connecting data socket to mover (('131.225.13.96', 51717)) with interface 131.225.13.19. elapsed=5482.550sec Thread READ_1 00:01:42 stkendm7a.fnal.gov 000642 root I MIGRATION Data socket ('131.225.13.19', 40548) is connected to ('131.225.13.96', 51717). elapsed=5482.550sec Thread READ_1 00:01:42 stkendm7a.fnal.gov 000642 root I MIGRATION Opening local file. Thread READ_1 00:01:42 stkenmvr27a.fnal.gov 005886 root I DBT27MV tape position: target 887 current 884 960 Thread seek_thread 00:01:42 stkenmvr27a.fnal.gov 005886 root I DBT27MV skip_fm 3 Thread seek_thread 00:01:42 stkenmvr27a.fnal.gov 005886 root I DBT27MV read_tape starting, bytes_to_read=652733196 Thread tape_thread 00:01:43 stkendm7a.fnal.gov 000642 root I ENCP Version: v3_8 CVS $Revision: 1.963 $ encp.pyc OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson), User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Current working directory: stkendm7a.fnal.gov:/opt/enstore/src Command line: encp --delayed-dismount 2 --ignore-fair-share --bypass-filesystem-max-filesize-check --priority 0 --threaded /pnfs/fs/usr/ktev/migrated_from_dlt/UP/UPA2/UPA253/f1 /data/data1/Migration_Spool/VO6043:0000_000000000_0000893 Thread READ 00:01:43 stkendm7a.fnal.gov 000642 root I ENCP Creating zero length output files. Thread READ 00:01:43 stkendm7a.fnal.gov 000642 root I ENCP Sending /pnfs/fs/usr/ktev/migrated_from_dlt/UP/UPA2/UPA253/f1 read request to LM: unique_id: stkendm7a.fnal.gov-1257573703-642-433 inputfile: /pnfs/fs/usr/ktev/migrated_from_dlt/UP/UPA2/UPA253/.(access)(001F000000000000007E9250) outputfile: /data/data1/Migration_Spool/VO6043:0000_000000000_0000893 Thread READ 00:01:43 stkensrv4n.fnal.gov 005795 enstore I 9940BLM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : ktev 00:01:43 stkensrv4n.fnal.gov 005795 enstore I 9940BLM read Q'd /pnfs/fs/usr/ktev/migrated_from_dlt/UP/UPA2/UPA253/f1 (stkendm7a.fnal.gov-1257573703-642-433) -> /data/data1/Migration_Spool/VO6043:0000_000000000_0000893 : library=CD-9940B family=ktev.ktev_migration.cern requester:root 00:01:43 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:43 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:43 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30141 current 30140 73 Thread seek_thread 00:01:43 stkensrv4n.fnal.gov 005795 enstore I 9940BLM 00:01:43 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:43 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=1014952 Thread tape_thread 00:01:43 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 1015091/1015091, blocks 8 header 139 Thread tape_thread 00:01:43 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 1014952/1014952 bytes Thread net_thread 00:01:43 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:43 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30141 block 8 block_size 0 bloc_loc 5656951 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 1015091 block_read_tot 0.0300180912018 tape_rate 33815974.2795 Thread tape_thread 00:01:43 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 1015091/1015091 bytes Thread tape_thread 00:01:43 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.000596761703491 read 0.0242643356323 crc_check 0.00508689880371 push 7.00950622559e-05 block_read 0.0300180912018 idle 0.000179052352905 Thread tape_thread 00:01:44 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV read bytes 547872928/547872928, blocks 4180 header 0 Thread tape_thread 00:01:44 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV write_client: wrote 547872928/547872928 bytes Thread net_thread 00:01:44 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV write_client waits for tape thread to exit Thread net_thread 00:01:44 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV drive stats after read. Tape VOV146 position 389 block 4180 block_size 0 bloc_loc 3483182 tot_blocks 762939000 BOT 0 read_err 0 write_err 0 bytes 547872928 block_read_tot 6.09674715996 tape_rate 89863153.8467 Thread tape_thread 00:01:44 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV read_tape exiting, read 547872928/547872928 bytes Thread tape_thread 00:01:44 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV read_tape timing:get_space 0.0337023735046 read 3.51461434364 crc_check 2.50984549522 push 0.0385849475861 block_read 6.09674715996 idle 0.0899572372437 Thread tape_thread 00:01:44 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:44 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30142 current 30141 8 Thread seek_thread 00:01:44 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:44 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=6952389 Thread tape_thread 00:01:44 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 6952528/6952528, blocks 54 header 139 Thread tape_thread 00:01:45 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 6952389/6952389 bytes Thread net_thread 00:01:45 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:45 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30142 block 54 block_size 0 bloc_loc 5657006 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 6952528 block_read_tot 0.082581281662 tape_rate 84190119.8441 Thread tape_thread 00:01:45 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 6952528/6952528 bytes Thread tape_thread 00:01:45 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.000932455062866 read 0.0471985340118 crc_check 0.0339767932892 push 0.000473499298096 block_read 0.082581281662 idle 0.0013439655304 Thread tape_thread 00:01:45 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV write_client exits Thread net_thread 00:01:45 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV tape position: target 392 current 389 4180 Thread seek_thread 00:01:45 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:45 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV skip_fm 3 Thread seek_thread 00:01:45 stkenmvr211a.fnal.gov 006256 root I LTO4_111MV read_tape starting, bytes_to_read=3794533608 Thread tape_thread 00:01:46 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:46 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30143 current 30142 54 Thread seek_thread 00:01:46 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:46 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=14438531 Thread tape_thread 00:01:46 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 14438670/14438670, blocks 111 header 139 Thread tape_thread 00:01:46 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 14438531/14438531 bytes Thread net_thread 00:01:46 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:46 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30143 block 111 block_size 0 bloc_loc 5657118 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 14438670 block_read_tot 0.144810438156 tape_rate 99707384.2456 Thread tape_thread 00:01:46 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 14438670/14438670 bytes Thread tape_thread 00:01:46 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00194573402405 read 0.0706272125244 crc_check 0.0712921619415 push 0.000945329666138 block_read 0.144810438156 idle 0.00254130363464 Thread tape_thread 00:01:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30144 current 30143 111 Thread seek_thread 00:01:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=18804599 Thread tape_thread 00:01:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 18804738/18804738, blocks 144 header 139 Thread tape_thread 00:01:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 18804599/18804599 bytes Thread net_thread 00:01:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30144 block 144 block_size 0 bloc_loc 5657263 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 18804738 block_read_tot 0.181670427322 tape_rate 103510176.517 Thread tape_thread 00:01:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 18804738/18804738 bytes Thread tape_thread 00:01:47 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00253367424011 read 0.0852644443512 crc_check 0.0926532745361 push 0.00121903419495 block_read 0.181670427322 idle 0.00321817398071 Thread tape_thread 00:01:47 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:48 last message repeated 1 times 00:01:48 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:48 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30145 current 30144 144 Thread seek_thread 00:01:48 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:48 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=17557780 Thread tape_thread 00:01:48 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 17557919/17557919, blocks 134 header 139 Thread tape_thread 00:01:48 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 17557780/17557780 bytes Thread net_thread 00:01:48 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:48 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30145 block 134 block_size 0 bloc_loc 5657398 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 17557919 block_read_tot 0.174828767776 tape_rate 100429232.69 Thread tape_thread 00:01:48 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 17557919/17557919 bytes Thread tape_thread 00:01:48 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00163578987122 read 0.0852112770081 crc_check 0.0868525505066 push 0.00112915039062 block_read 0.174828767776 idle 0.00327062606812 Thread tape_thread 00:01:49 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:49 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:49 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30146 current 30145 134 Thread seek_thread 00:01:49 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:49 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=15073055 Thread tape_thread 00:01:50 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 15073194/15073194, blocks 115 header 139 Thread tape_thread 00:01:50 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 15073055/15073055 bytes Thread net_thread 00:01:50 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:50 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30146 block 115 block_size 0 bloc_loc 5657514 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 15073194 block_read_tot 0.148619413376 tape_rate 101421433.833 Thread tape_thread 00:01:50 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 15073194/15073194 bytes Thread tape_thread 00:01:50 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.0020751953125 read 0.0710442066193 crc_check 0.0745351314545 push 0.000964879989624 block_read 0.148619413376 idle 0.00286459922791 Thread tape_thread 00:01:51 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:51 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30147 current 30146 115 Thread seek_thread 00:01:51 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:51 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=51103 Thread tape_thread 00:01:51 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 51242/51242, blocks 1 header 139 Thread tape_thread 00:01:51 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 51103/51103 bytes Thread net_thread 00:01:51 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:51 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30147 block 1 block_size 0 bloc_loc 5657517 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 51242 block_read_tot 0.019788980484 tape_rate 2589420.91743 Thread tape_thread 00:01:51 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 51242/51242 bytes Thread tape_thread 00:01:51 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 4.31537628174e-05 read 0.0191869735718 crc_check 0.000545978546143 push 1.28746032715e-05 block_read 0.019788980484 idle 2.19345092773e-05 Thread tape_thread 00:01:51 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:52 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:52 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30148 current 30147 1 Thread seek_thread 00:01:52 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:52 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=51099 Thread tape_thread 00:01:52 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 51238/51238, blocks 1 header 139 Thread tape_thread 00:01:52 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 51099/51099 bytes Thread net_thread 00:01:52 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:52 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30148 block 1 block_size 0 bloc_loc 5657519 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 51238 block_read_tot 0.0207600593567 tape_rate 2468104.6966 Thread tape_thread 00:01:52 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 51238/51238 bytes Thread tape_thread 00:01:52 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 4.19616699219e-05 read 0.0201570987701 crc_check 0.000548839569092 push 1.21593475342e-05 block_read 0.0207600593567 idle 2.19345092773e-05 Thread tape_thread 00:01:52 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:53 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:53 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30149 current 30148 1 Thread seek_thread 00:01:53 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:53 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=8044551 Thread tape_thread 00:01:53 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 8044690/8044690, blocks 62 header 139 Thread tape_thread 00:01:53 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 8044551/8044551 bytes Thread net_thread 00:01:53 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:53 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30149 block 62 block_size 0 bloc_loc 5657582 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 8044690 block_read_tot 0.0882513523102 tape_rate 91156563.4907 Thread tape_thread 00:01:53 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 8044690/8044690 bytes Thread tape_thread 00:01:53 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00156378746033 read 0.0462799072266 crc_check 0.0398836135864 push 0.000524044036865 block_read 0.0882513523102 idle 0.00138163566589 Thread tape_thread 00:01:53 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:54 stkensrv4n.fnal.gov 005795 enstore I 9940BLM 00:01:54 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:54 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30150 current 30149 62 Thread seek_thread 00:01:54 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:54 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=14846394 Thread tape_thread 00:01:54 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 14846533/14846533, blocks 114 header 139 Thread tape_thread 00:01:54 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 14846394/14846394 bytes Thread net_thread 00:01:54 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:54 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30150 block 114 block_size 0 bloc_loc 5657697 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 14846533 block_read_tot 0.149288654327 tape_rate 99448501.7424 Thread tape_thread 00:01:54 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 14846533/14846533 bytes Thread tape_thread 00:01:54 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00173473358154 read 0.0735294818878 crc_check 0.0730612277985 push 0.00096321105957 block_read 0.149288654327 idle 0.00260066986084 Thread tape_thread 00:01:55 stkensrv4n.fnal.gov 005806 enstore I LTO3LM 00:01:55 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:55 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:55 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30151 current 30150 114 Thread seek_thread 00:01:55 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:55 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=51209 Thread tape_thread 00:01:55 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 51348/51348, blocks 1 header 139 Thread tape_thread 00:01:55 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 51209/51209 bytes Thread net_thread 00:01:55 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:55 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30151 block 1 block_size 0 bloc_loc 5657699 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 51348 block_read_tot 0.0208189487457 tape_rate 2466406.95585 Thread tape_thread 00:01:55 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 51348/51348 bytes Thread tape_thread 00:01:55 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 3.60012054443e-05 read 0.0202260017395 crc_check 0.000545024871826 push 1.19209289551e-05 block_read 0.0208189487457 idle 2.19345092773e-05 Thread tape_thread 00:01:56 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:56 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30152 current 30151 1 Thread seek_thread 00:01:56 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:56 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=10735686 Thread tape_thread 00:01:56 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 10735825/10735825, blocks 82 header 139 Thread tape_thread 00:01:56 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 10735686/10735686 bytes Thread net_thread 00:01:56 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:57 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30152 block 82 block_size 0 bloc_loc 5657782 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 10735825 block_read_tot 0.111536026001 tape_rate 96254325.9333 Thread tape_thread 00:01:57 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 10735825/10735825 bytes Thread tape_thread 00:01:57 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00169396400452 read 0.0558569431305 crc_check 0.0532646179199 push 0.000720500946045 block_read 0.111536026001 idle 0.00182700157166 Thread tape_thread 00:01:57 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV skip_fm 232 Thread seek_thread 00:01:57 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV write_tape starting, bytes_to_write=2147475968 Thread tape_thread 00:01:57 stkenmvr210a.fnal.gov 011436 root I LTO4_110MV Write starting. Tape VON267 absolute location in blocks 3415905 Thread tape_thread 00:01:57 cmsstor63.fnal.gov 030546 root I ENCP Version: v3_7d CVS $Revision: 1.922 $ encp OS: Linux 2.6.31 x86_64 Release: Rocks release 5.1 (V.I) Scientific Linux SL release 5.2 (Boron) User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Library: CD-LTO4G1 Storage Group: cms File Family: cmsusers FF Wrapper: cpio_odc FF Width: 7 Current working directory: cmsstor63.fnal.gov:/ Command line: /home/enstore/enstore/bin/encp --verbose 4 --threaded --ecrc --resubmit-timeout 1800 --age-time 30 --delpri 1 --priority 25 --pnfs-mount /pnfs/fs --shortcut --override-path /pnfs/fnal.gov/usr/cms/WAX/11/store/user/andrey/WW/WW_Summer09_NWU_MC_31X_V3_v1/6c95c1cdbd259435ef6b1d3324c0519f/WW_Summer09_NWU_MC_31X_V3_v1_306.root --put-cache 0011000000000000011B21C0 /storage/data3/write-pool-3/data/0011000000000000011B21C0 00:01:57 cmsstor63.fnal.gov 030546 root I ENCP Creating zero length output files. 00:01:57 cmsstor63.fnal.gov 030546 root I ENCP Sending /pnfs/fnal.gov/usr/cms/WAX/11/store/user/andrey/WW/WW_Summer09_NWU_MC_31X_V3_v1/6c95c1cdbd259435ef6b1d3324c0519f/WW_Summer09_NWU_MC_31X_V3_v1_306.root write request to LM: unique_id: cmsstor63.fnal.gov-1257573717-30546-0 inputfile: /storage/data3/write-pool-3/data/0011000000000000011B21C0 outputfile: /pnfs/fnal.gov/usr/cms/WAX/11/store/user/andrey/WW/WW_Summer09_NWU_MC_31X_V3_v1/6c95c1cdbd259435ef6b1d3324c0519f/.(access)(0011000000000000011B21C0) 00:01:57 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : cms 00:01:57 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write Q'd /storage/data3/write-pool-3/data/0011000000000000011B21C0 (cmsstor63.fnal.gov-1257573717-30546-0) -> /pnfs/fnal.gov/usr/cms/WAX/11/store/user/andrey/WW/WW_Summer09_NWU_MC_31X_V3_v1/6c95c1cdbd259435ef6b1d3324c0519f/WW_Summer09_NWU_MC_31X_V3_v1_306.root : library=CD-LTO4G1 family=cmsusers requester:root volume_family:cms.cmsusers.cpio_odc 00:01:57 cmsstor63.fnal.gov 030546 root I ENCP Listening for control socket at: ('131.225.204.239', 41026) 00:01:57 cmsstor63.fnal.gov 030546 root I ENCP Waiting for mover to connect control socket at address: ('131.225.204.239', 41026) elapsed=1.070sec 00:01:57 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:01:57 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Assigning volume VOM116 from storage group cms.cmsusers.cpio_odc to library CD-LTO4G1, volume family cms.cmsusers.cpio_odc 00:01:58 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM write_to_hsm work on vol=VOM116 mover=LTO4_109.mover requester:root 00:01:58 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : cms 00:01:58 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM IDLE:sending {'vc': {'storage_group': 'cms', 'library': 'CD-LTO4G1', 'file_family': 'cmsusers', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cmsusers.cpio_odc', 'file_family_width': 7}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX/11/store/user/andrey/WW/WW_Summer09_NWU_MC_31X_V3_v1/6c95c1cdbd259435ef6b1d3324c0519f/WW_Summer09_NWU_MC_31X_V3_v1_306.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2977640624L, 'mover': 'LTO4_109.mover', 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX/11/store/user/andrey/WW/WW_Summer09_NWU_MC_31X_V3_v1/6c95c1cdbd259435ef6b1d3324c0519f/WW_Summer09_NWU_MC_31X_V3_v1_306.root', 'uid': 0, 'uname': 'root', 'type': 'cpio_odc', 'gid': 0, 'mtime': 1257573717, 'rmajor': 0, 'machine': ('Linux', 'cmsstor63.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'pstat': (16893, 286963976, 27L, 1, 44342, 5063, 512, 1257573566, 1257573566, 1257546280), 'mode_octal': '0100036', 'mode': 33188, 'gname': 'root', 'size_bytes': 2977640624L, 'fullname': '/storage/data3/write-pool-3/data/0011000000000000011B21C0', 'inode': 286990784L, 'minor': 27}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/0011000000000000011B21C0', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX/11/store/user/andrey/WW/WW_Summer09_NWU_MC_31X_V3_v1/6c95c1cdbd259435ef6b1d3324c0519f/.(access)(0011000000000000011B21C0)', 'fc': {'size': 2977640624L, 'external_label': 'VOM116', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data3/write-pool-3/data/0011000000000000011B21C0', 'at_the_top': 2, 'local_inode': 1073742002, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.204.239', 41026), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573717.9910929, 'encp_start_time': 1257573716.5320389, 'job_queued': 1257573717.5441561, 't0': 1257573716, 'in_queue': 0.44687581062316895}, 'user_level': 'admin', 'unique_id': 'cmsstor63.fnal.gov-1257573717-30546-0'} to mover 00:01:58 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:58 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV WRITE_TO_HSM Thread MainThread 00:01:58 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM can not remove from sg LTO4_109.mover VOM116 00:01:58 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM SG: {'exp-db': [('LTO4_110.mover', 'VON267'), ('LTO4_108.mover', 'VON279'), ('LTO4_94.mover', 'VON278')]} 00:01:58 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM can not remove from vf LTO4_109.mover VOM116 00:01:58 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM VF: {'exp-db.daily-d0-offline.cpio_odc': [('LTO4_108.mover', 'VON279'), ('LTO4_94.mover', 'VON278')], 'exp-db.daily-cdf-offline.cpio_odc': [('LTO4_110.mover', 'VON267')]} 00:01:58 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:58 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV SENDING {'vc': {'storage_group': 'cms', 'library': 'CD-LTO4G1', 'file_family': 'cmsusers', 'wrapper': 'cpio_odc', 'address': ('131.225.13.59', 7502), 'volume_family': 'cms.cmsusers.cpio_odc', 'file_family_width': 7}, 'outfilepath': '/pnfs/fnal.gov/usr/cms/WAX/11/store/user/andrey/WW/WW_Summer09_NWU_MC_31X_V3_v1/6c95c1cdbd259435ef6b1d3324c0519f/WW_Summer09_NWU_MC_31X_V3_v1_306.root', 'encp': {'agetime': 30, 'adminpri': -1, 'delpri': 1, 'basepri': 25, 'delayed_dismount': None, 'curpri': 25}, 'file_size': 2977640624L, 'mover': {'max_buffer': 2621440000L, 'check_written_file': 600, 'max_failures': 5, 'library': 'CD-LTO4G1.library_manager', 'local_mover': 0, 'single_filemark': 1, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7869, 'mc_device': '0,5,1,7', 'compression': 0, 'serial_num': '9310025229', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 73400320.0, 'vendor_id': 'IBM', 'driver': 'FTTDriver', 'host': 'stkenmvr209a', 'max_time_in_state': 1200, 'device': 'stkenmvr209a:/dev/rmt/tps0d0n', 'name': 'LTO4_109.mover', 'send_stats': 1, 'product_id': 'ULTRIUM-TD4', 'statistics_path': '/tmp/enstore/enstore/LTO4_109.stat', 'mover_address': ('131.225.13.205', 7869), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.205', 48904), 'data_ip': 'stkenmvr209a', 'logname': 'LTO4_109MV', 'hostip': '131.225.13.205', 'do_cleaning': 'No', 'media_changer': 'SL8500G1.media_changer'}, 'ignore_fair_share': None, 'retry': None, 'lm': {'address': ('131.225.13.4', 8541)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fnal.gov/usr/cms/WAX/11/store/user/andrey/WW/WW_Summer09_NWU_MC_31X_V3_v1/6c95c1cdbd259435ef6b1d3324c0519f/WW_Summer09_NWU_MC_31X_V3_v1_306.root', 'uid': 0, 'uname': 'root', 'mtime': 1257573717, 'rmajor': 0, 'machine': ('Linux', 'cmsstor63.fnal.gov', '2.6.31', '#1 SMP Mon Sep 14 09:43:20 CDT 2009', 'x86_64'), 'sanity_size': 65536, 'type': 'cpio_odc', 'pstat': (16893, 286963976, 27L, 1, 44342, 5063, 512, 1257573566, 1257573566, 1257546280), 'mode_octal': '0100036', 'mode': 33188, 'size_bytes': 2977640624L, 'gname': 'root', 'gid': 0, 'fullname': '/storage/data3/write-pool-3/data/0011000000000000011B21C0', 'inode': 286990784L, 'minor': 27}, 'version': 'v3_7d CVS $Revision: 1.922 $ encp', 'encp_daq': None, 'client_crc': 1, 'crc_seed': 0, 'infile': '/storage/data3/write-pool-3/data/0011000000000000011B21C0', 'status': ('ok', None), 'outfile': '/pnfs/fnal.gov/usr/cms/WAX/11/store/user/andrey/WW/WW_Summer09_NWU_MC_31X_V3_v1/6c95c1cdbd259435ef6b1d3324c0519f/.(access)(0011000000000000011B21C0)', 'fc': {'size': 2977640624L, 'external_label': 'VOM116', 'address': ('131.225.13.59', 7501)}, 'infilepath': '/storage/data3/write-pool-3/data/0011000000000000011B21C0', 'at_the_top': 2, 'local_inode': 1073742002, 'resend': {'max_retry': 3, 'resubmit': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.204.239', 41026), 'work': 'write_to_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573717.9910929, 'encp_start_time': 1257573716.5320389, 'job_queued': 1257573717.5441561, 't0': 1257573716, 'in_queue': 0.44687581062316895}, 'user_level': 'admin', 'unique_id': 'cmsstor63.fnal.gov-1257573717-30546-0'} Thread client_connect_thread 00:01:58 stkensrv4n.fnal.gov 005357 enstore I LTO4G1LM 00:01:58 cmsstor63.fnal.gov 030546 root I ENCP Control socket ('131.225.204.239', 41026) is connected to ('131.225.13.205', 48905) for cmsstor63.fnal.gov-1257573717-30546-0. elapsed=1.520sec 00:01:58 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30153 current 30152 82 Thread seek_thread 00:01:58 cmsstor63.fnal.gov 030546 root I ENCP Received callback ticket from mover LTO4_109.mover for transfer cmsstor63.fnal.gov-1257573717-30546-0. elapsed=1.520sec 00:01:58 cmsstor63.fnal.gov 030546 root I ENCP Atempting to connect data socket to mover at ('131.225.13.205', 48904). 00:01:58 cmsstor63.fnal.gov 030546 root I ENCP Connecting data socket to mover (('131.225.13.205', 48904)) with interface 131.225.204.239. elapsed=1.530sec 00:01:58 cmsstor63.fnal.gov 030546 root I ENCP Data socket ('131.225.204.239', 44992) is connected to ('131.225.13.205', 48904). elapsed=1.530sec 00:01:58 cmsstor63.fnal.gov 030546 root I ENCP Opening local file. 00:01:58 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:58 stkenmvr209a.fnal.gov 006118 root I LTO4_109MV MSG_TYPE=MC_LOAD_REQ mounting VOM116 ULTRIUM-TD4 00:01:58 Thread media_thread 00:01:58 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=14245751 Thread tape_thread 00:01:58 stkensrv4n.fnal.gov 003689 enstore I SL8500G1MC PROCESSING mount VOM116 0,5,1,7 00:01:58 cmsstor63.fnal.gov 030546 root I ENCP pf_status = ('ok', None) skip_layer_cleanup = False 00:01:58 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 14245890/14245890, blocks 109 header 139 Thread tape_thread 00:01:58 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 14245751/14245751 bytes Thread net_thread 00:01:58 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:58 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30153 block 109 block_size 0 bloc_loc 5657892 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 14245890 block_read_tot 0.145349979401 tape_rate 98010953.0028 Thread tape_thread 00:01:58 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 14245890/14245890 bytes Thread tape_thread 00:01:58 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00156235694885 read 0.072637796402 crc_check 0.0702204704285 push 0.000929355621338 block_read 0.145349979401 idle 0.0027449131012 Thread tape_thread 00:01:59 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:01:59 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30154 current 30153 109 Thread seek_thread 00:01:59 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:01:59 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=15261137 Thread tape_thread 00:01:59 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 15261276/15261276, blocks 117 header 139 Thread tape_thread 00:01:59 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 15261137/15261137 bytes Thread net_thread 00:01:59 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:01:59 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30154 block 117 block_size 0 bloc_loc 5658010 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 15261276 block_read_tot 0.153756856918 tape_rate 99255905.1081 Thread tape_thread 00:01:59 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 15261276/15261276 bytes Thread tape_thread 00:01:59 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.0018470287323 read 0.0757691860199 crc_check 0.0751421451569 push 0.000998497009277 block_read 0.153756856918 idle 0.00290322303772 Thread tape_thread 00:01:59 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:02:00 last message repeated 1 times 00:02:00 stkensrv4n.fnal.gov 032536 enstore I SL8500G1MC query drive 0,5,1,7 => 0,0,5,1,7 online available IBM-LTO4 00:02:00 stkensrv4n.fnal.gov 032536 enstore I SL8500G1MC Dismount 14 ignored: Drive 0,5,1,7 is empty. Thought VOM116 was there =>. dismount VOLUME 0,5,1,7 force => ok,0,5,1,7 online available IBM-LTO4 00:02:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:02:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30155 current 30154 117 Thread seek_thread 00:02:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:02:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=10741963 Thread tape_thread 00:02:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 10742102/10742102, blocks 82 header 139 Thread tape_thread 00:02:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 10741963/10741963 bytes Thread net_thread 00:02:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:02:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30155 block 82 block_size 0 bloc_loc 5658093 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 10742102 block_read_tot 0.113325834274 tape_rate 94789524.9904 Thread tape_thread 00:02:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 10742102/10742102 bytes Thread tape_thread 00:02:00 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00121355056763 read 0.058159828186 crc_check 0.0532467365265 push 0.000705718994141 block_read 0.113325834274 idle 0.00180125236511 Thread tape_thread 00:02:01 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:02:01 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:02:01 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30156 current 30155 82 Thread seek_thread 00:02:01 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:02:01 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=11499401 Thread tape_thread 00:02:01 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 11499540/11499540, blocks 88 header 139 Thread tape_thread 00:02:01 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 11499401/11499401 bytes Thread net_thread 00:02:01 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:02:01 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30156 block 88 block_size 0 bloc_loc 5658182 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 11499540 block_read_tot 0.121948480606 tape_rate 94298345.8494 Thread tape_thread 00:02:01 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 11499540/11499540 bytes Thread tape_thread 00:02:01 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.000929832458496 read 0.0631384849548 crc_check 0.0570650100708 push 0.000815153121948 block_read 0.121948480606 idle 0.00179243087769 Thread tape_thread 00:02:02 stkensrv4n.fnal.gov 032536 enstore I SL8500G1MC query vol VOM116 => 0,VOM116 home 0, 6,42, 4, 0 LTO-800G 00:02:02 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:02:03 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30157 current 30156 88 Thread seek_thread 00:02:03 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:02:03 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=5649374 Thread tape_thread 00:02:03 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 5649513/5649513, blocks 44 header 139 Thread tape_thread 00:02:03 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 5649374/5649374 bytes Thread net_thread 00:02:03 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:02:03 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30157 block 44 block_size 0 bloc_loc 5658227 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 5649513 block_read_tot 0.0686604976654 tape_rate 82281853.3523 Thread tape_thread 00:02:03 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 5649513/5649513 bytes Thread tape_thread 00:02:03 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00119519233704 read 0.0391798019409 crc_check 0.0279111862183 push 0.000374317169189 block_read 0.0686604976654 idle 0.000907182693481 Thread tape_thread 00:02:03 stkenmvr27a.fnal.gov 005886 root I DBT27MV read bytes 652733196/652733196, blocks 4980 header 0 Thread tape_thread 00:02:03 stkenmvr27a.fnal.gov 005886 root I DBT27MV drive stats after read. Tape VO6043 position 887 block 4980 block_size 0 bloc_loc 1242251 tot_blocks 0 BOT 0 read_err 0 write_err 0 bytes 652733196 block_read_tot 19.8975908756 tape_rate 32804634.4947 Thread tape_thread 00:02:03 stkenmvr27a.fnal.gov 005886 root I DBT27MV read_tape exiting, read 652733196/652733196 bytes Thread tape_thread 00:02:03 stkenmvr27a.fnal.gov 005886 root I DBT27MV read_tape timing:get_space 0.10256934166 read 16.6446959972 crc_check 3.0510892868 push 0.0992362499237 block_read 19.8975908756 idle 0.187571763992 Thread tape_thread 00:02:03 stkenmvr27a.fnal.gov 005886 root I DBT27MV write_client: wrote 652733196/652733196 bytes Thread net_thread 00:02:03 stkenmvr27a.fnal.gov 005886 root I DBT27MV write_client exits Thread net_thread 00:02:03 stkenmvr27a.fnal.gov 005886 root I DBT27MV transfer complete volume=VO6043 location=887 Thread net_thread 00:02:03 last message repeated 1 times 00:02:03 stkensrv0n.fnal.gov 028762 enstore I VOLSRV Updating update volume set sum_rd_access = '398', last_access = '2009-11-07 00:02:03' where label = 'VO6043'; 00:02:03 stkenmvr27a.fnal.gov 005886 root I DBT27MV Sending done to client: {'vc': {'comment': '', 'declared': 1080687098.0, 'blocksize': 131072, 'storage_group': 'ktev', 'library': 'CD-9940B', 'eod_cookie': '0000_000000000_0001084', 'current_location': '0000_000000000_0000887', 'user_inhibit': ['none', 'none'], 'sum_rd_access': 397, 'system_inhibit': ['none', 'migrating'], 'external_label': 'VO6043', 'wrapper': 'cern', 'remaining_bytes': 968444416L, 'sum_mounts': 9, 'capacity_bytes': 214748364800L, 'media_type': '9940B', 'last_access': 1257573702.0, 'status': ('ok', None), 'si_time': [0.0, 1257568219.0], 'non_del_files': 361, 'sum_wr_err': 0, 'sum_wr_access': 361, 'file_family': 'ktev_migration', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration.cern', 'modification_time': 1221190342.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1082654321.0}, 'outfilepath': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000887', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 652733196L, 'mover': {'max_buffer': 1048576000L, 'check_written_file': 10, 'max_failures': 5, 'library': 'CD-9940B.library_manager', 'local_mover': 0, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7827, 'mc_device': '0,1,10,2', 'compression': 0, 'serial_num': '479002016447', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 28311552.0, 'vendor_id': 'STK', 'driver': 'FTTDriver', 'host': 'stkenmvr27a', 'max_time_in_state': 1200, 'device': 'stkenmvr27a:/dev/rmt/tps2d0n', 'name': '9940B27.mover', 'send_stats': 1, 'product_id': 'T9940B', 'statistics_path': '/tmp/enstore/enstore/DBT27MV.stat', 'mover_address': ('131.225.13.96', 7827), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.96', 51717), 'data_ip': 'stkenmvr27a', 'logname': 'DBT27MV', 'hostip': '131.225.13.96', 'do_cleaning': 'No', 'media_changer': 'stk.media_changer'}, 'lm': {'address': ('131.225.13.4', 7522)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/f6', 'uid': 0, 'uname': 'root', 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (33188, 528391024L, 20L, 1, 5406, 2539, 652733196L, 1082667776, 1082667776, 1082665884), 'mode_octal': '0100744', 'mode': 33252, 'size_bytes': 652733196L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000887', 'inode': 528391024L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/.(access)(001F000000000000007E9B70)', 'status': ('ok', None), 'volume': 'VO6043', 'outfile': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000887', 'fc': {'status': ('ok', None), 'pnfsid': '001F000000000000007E9B70', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/f6', 'deleted': 'no', 'external_label': 'VO6043', 'drive': 'stkenmvr11a:/dev/rmt/tps0d0n:4790000019', 'update': None, 'gid': -1, 'location_cookie': '0000_000000000_0000887', 'complete_crc': 2842112764L, 'address': ('131.225.13.59', 7501), 'uid': -1, 'bfid': 'CDMS108266777500000', 'sanity_cookie': (65536L, 256914739L), 'size': 652733196L}, 'infilepath': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/f6', 'at_the_top': 15, 'bfid': 'CDMS108266777500000', 'local_inode': 39503343L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40450), 'work': 'read_from_hsm', 'override_ro_mount': 0, 'times': {'encp_start_time': 1257573388.4220531, 'seek_time': 0.0069169998168945312, 'transfer_time': 20.591706991195679, 't0': 1257573388, 'mount_time': 0.022955894470214844, 'lm_dequeued': 1257573702.7943521, 'job_queued': 1257573388.568013, 'drive_call_time': 16.480720520019531, 'in_queue': 314.22586607933044, 'drive_transfer_time': 19.993417978286743}, '(trunc. 4155) Thread net_thread 00:02:03 stkenmvr27a.fnal.gov 005886 root I DBT27MV Done is sent Thread net_thread 00:02:03 stkenmvr27a.fnal.gov 005886 root I DBT27MV maybe_clean Thread net_thread 00:02:03 stkenmvr27a.fnal.gov 005886 root I DBT27MV returned from maybe_clean Thread net_thread 00:02:03 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:02:03 stkensrv4n.fnal.gov 005795 enstore I 9940BLM 00:02:03 stkensrv4n.fnal.gov 005795 enstore I 9940BLM read_from_hsm next work on vol=VO6043 mover=9940B27.mover requester:root 00:02:03 stkensrv4n.fnal.gov 005795 enstore I 9940BLM HAVE_BOUND:sending read_from_hsm /pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM10/f0 to mover 9940B27.mover ('131.225.13.96', 7827) DEL_DISM 2 00:02:03 stkensrv4n.fnal.gov 005795 enstore I 9940BLM MSG_TYPE=ADD_TO_LMQ request added to work at movers queue for storage group : ktev 00:02:03 stkensrv4n.fnal.gov 005795 enstore I 9940BLM HAVE_BOUND:sending {'vc': {'comment': '', 'declared': 1080687098.0, 'blocksize': 131072, 'storage_group': 'ktev', 'library': 'CD-9940B', 'eod_cookie': '0000_000000000_0001084', 'current_location': '0000_000000000_0000890', 'user_inhibit': ['none', 'none'], 'system_inhibit': ['none', 'migrating'], 'external_label': 'VO6043', 'wrapper': 'cern', 'remaining_bytes': 968444416L, 'sum_mounts': 9, 'sum_rd_access': 396, 'media_type': '9940B', 'last_access': 1257573699.0, 'status': ('ok', None), 'si_time': [0.0, 1257568219.0], 'capacity_bytes': 214748364800L, 'non_del_files': 361, 'sum_wr_err': 0, 'sum_wr_access': 361, 'file_family': 'ktev_migration', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration.cern', 'modification_time': 1221190342.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1082654321.0}, 'outfilepath': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000890', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 622601828L, 'mover': '9940B27.mover', 'lm': {'address': ('131.225.13.4', 7522)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM10/f0', 'uid': 0, 'uname': 'root', 'gid': 0, 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (33188, 528390144L, 20L, 1, 5406, 2539, 622601828L, 1082667827, 1082667827, 1082665433), 'mode_octal': '0100744', 'mode': 33252, 'gname': 'root', 'size_bytes': 622601828L, 'fullname': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000890', 'inode': 528390144L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM10/.(access)(001F000000000000007E9800)', 'status': ('ok', None), 'volume': 'VO6043', 'outfile': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000890', 'fc': {'status': ('ok', None), 'pnfsid': '001F000000000000007E9800', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM10/f0', 'deleted': 'no', 'external_label': 'VO6043', 'drive': 'stkenmvr11a:/dev/rmt/tps0d0n:4790000019', 'update': None, 'gid': -1, 'location_cookie': '0000_000000000_0000890', 'complete_crc': 2728993383L, 'address': ('131.225.13.59', 7501), 'uid': -1, 'bfid': 'CDMS108266782700000', 'sanity_cookie': (65536L, 1019612907L), 'size': 622601828L}, 'infilepath': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM10/f0', 'at_the_top': 63, 'bfid': 'CDMS108266782700000', 'local_inode': 39503208L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40547), 'work': 'read_from_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573723.8124461, 'encp_start_time': 1257573700.2607069, 'job_queued': 1257573700.4751339, 't0': 1257573700, 'in_queue': 23.336892127990723}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573700-642-432'} to mover 00:02:03 stkensrv4n.fnal.gov 005795 enstore I 9940BLM 00:02:03 stkenmvr27a.fnal.gov 005886 root I DBT27MV READ FROM HSM Thread MainThread 00:02:03 stkendm7a.fnal.gov 000642 root I ENCP Listening for control socket at: ('131.225.13.19', 40547) Thread READ_2 00:02:03 stkendm7a.fnal.gov 000642 root I ENCP Waiting for mover to connect control socket at address: ('131.225.13.19', 40547) elapsed=5503.550sec Thread READ_2 00:02:03 stkenmvr27a.fnal.gov 005886 root I DBT27MV SENDING {'vc': {'comment': '', 'declared': 1080687098.0, 'blocksize': 131072, 'storage_group': 'ktev', 'library': 'CD-9940B', 'eod_cookie': '0000_000000000_0001084', 'current_location': '0000_000000000_0000890', 'user_inhibit': ['none', 'none'], 'sum_rd_access': 396, 'system_inhibit': ['none', 'migrating'], 'external_label': 'VO6043', 'wrapper': 'cern', 'remaining_bytes': 968444416L, 'sum_mounts': 9, 'capacity_bytes': 214748364800L, 'media_type': '9940B', 'last_access': 1257573699.0, 'status': ('ok', None), 'si_time': [0.0, 1257568219.0], 'non_del_files': 361, 'sum_wr_err': 0, 'sum_wr_access': 361, 'file_family': 'ktev_migration', 'address': ('131.225.13.59', 7502), 'volume_family': 'ktev.ktev_migration.cern', 'modification_time': 1221190342.0, 'write_protected': 'y', 'sum_rd_err': 0, 'first_access': 1082654321.0}, 'outfilepath': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000890', 'encp': {'agetime': 30, 'adminpri': 6, 'delpri': 1, 'basepri': 1000007, 'delayed_dismount': 2, 'curpri': 1000007}, 'file_size': 622601828L, 'mover': {'max_buffer': 1048576000L, 'check_written_file': 10, 'max_failures': 5, 'library': 'CD-9940B.library_manager', 'local_mover': 0, 'max_consecutive_failures': 3, 'norestart': 'INQ', 'port': 7827, 'mc_device': '0,1,10,2', 'compression': 0, 'serial_num': '479002016447', 'update_interval_in_bound': 2, 'update_interval': 5, 'syslog_entry': 'sense|st[0-9]', 'status': ('ok', None), 'max_rate': 28311552.0, 'vendor_id': 'STK', 'driver': 'FTTDriver', 'host': 'stkenmvr27a', 'max_time_in_state': 1200, 'device': 'stkenmvr27a:/dev/rmt/tps2d0n', 'name': '9940B27.mover', 'send_stats': 1, 'product_id': 'T9940B', 'statistics_path': '/tmp/enstore/enstore/DBT27MV.stat', 'mover_address': ('131.225.13.96', 7827), 'mount_delay': 15, 'check_first_written_file': 0, 'callback_addr': ('131.225.13.96', 51720), 'data_ip': 'stkenmvr27a', 'logname': 'DBT27MV', 'hostip': '131.225.13.96', 'do_cleaning': 'No', 'media_changer': 'stk.media_changer'}, 'lm': {'address': ('131.225.13.4', 7522)}, 'wrapper': {'major': 0, 'rminor': 0, 'pnfsFilename': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM10/f0', 'uid': 0, 'uname': 'root', 'rmajor': 0, 'machine': ('Linux', 'stkendm7a.fnal.gov', '2.6.9-89.0.9.ELsmp', '#1 SMP Mon Aug 24 13:47:43 CDT 2009', 'i686'), 'sanity_size': 65536, 'pstat': (33188, 528390144L, 20L, 1, 5406, 2539, 622601828L, 1082667827, 1082667827, 1082665433), 'mode_octal': '0100744', 'mode': 33252, 'size_bytes': 622601828L, 'gname': 'root', 'gid': 0, 'fullname': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000890', 'inode': 528390144L, 'minor': 20}, 'version': 'v3_8 CVS $Revision: 1.963 $ encp.pyc', 'encp_daq': None, 'client_crc': 1, 'infile': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM10/.(access)(001F000000000000007E9800)', 'status': ('ok', None), 'volume': 'VO6043', 'outfile': '/data/data1/Migration_Spool/VO6043:0000_000000000_0000890', 'fc': {'status': ('ok', None), 'pnfsid': '001F000000000000007E9800', 'pnfs_name0': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM10/f0', 'deleted': 'no', 'external_label': 'VO6043', 'drive': 'stkenmvr11a:/dev/rmt/tps0d0n:4790000019', 'update': None, 'gid': -1, 'location_cookie': '0000_000000000_0000890', 'complete_crc': 2728993383L, 'address': ('131.225.13.59', 7501), 'uid': -1, 'bfid': 'CDMS108266782700000', 'sanity_cookie': (65536L, 1019612907L), 'size': 622601828L}, 'infilepath': '/pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM10/f0', 'at_the_top': 63, 'bfid': 'CDMS108266782700000', 'local_inode': 39503208L, 'resend': {'max_retry': 3, 'resubmits': 0, 'max_resubmits': None, 'retry': 0}, 'callback_addr': ('131.225.13.19', 40547), 'work': 'read_from_hsm', 'override_ro_mount': 0, 'times': {'lm_dequeued': 1257573723.8124461, 'encp_start_time': 1257573700.2607069, 'job_queued': 1257573700.4751339, 't0': 1257573700, 'in_queue': 23.336892127990723}, 'user_level': 'admin', 'unique_id': 'stkendm7a.fnal.gov-1257573700-642-432'} Thread client_connect_thread 00:02:03 stkendm7a.fnal.gov 000642 root I ENCP Control socket ('131.225.13.19', 40547) is connected to ('131.225.13.96', 51721) for stkendm7a.fnal.gov-1257573700-642-432. elapsed=5503.550sec Thread READ_2 00:02:03 stkendm7a.fnal.gov 000642 root I ENCP Received callback ticket from mover 9940B27.mover for transfer stkendm7a.fnal.gov-1257573700-642-432. elapsed=5503.560sec Thread READ_2 00:02:03 stkendm7a.fnal.gov 000642 root I ENCP Attempting to connect data socket to mover at ('131.225.13.96', 51720). Thread READ_2 00:02:03 stkendm7a.fnal.gov 000642 root I ENCP Connecting data socket to mover (('131.225.13.96', 51720)) with interface 131.225.13.19. elapsed=5503.560sec Thread READ_2 00:02:03 stkendm7a.fnal.gov 000642 root I ENCP Data socket ('131.225.13.19', 40551) is connected to ('131.225.13.96', 51720). elapsed=5503.560sec Thread READ_2 00:02:04 last message repeated 1 times 00:02:04 stkendm7a.fnal.gov 000642 root I ENCP Opening local file. Thread READ_2 00:02:04 stkenmvr27a.fnal.gov 005886 root I DBT27MV tape position: target 890 current 887 4980 Thread seek_thread 00:02:04 stkenmvr27a.fnal.gov 005886 root I DBT27MV skip_fm 3 Thread seek_thread 00:02:04 stkenmvr27a.fnal.gov 005886 root I DBT27MV read_tape starting, bytes_to_read=622601828 Thread tape_thread 00:02:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:02:04 stkendm7a.fnal.gov 000642 root I ENCP File /pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/f6 transfered for stkendm7a.fnal.gov-1257573388-642-425 in 21.2027029991 seconds. Thread READ_1 00:02:04 stkendm7a.fnal.gov 000642 root I ENCP Received final dialog for stkendm7a.fnal.gov-1257573388-642-425. elapsed=5503.800sec Thread READ_1 00:02:04 stkendm7a.fnal.gov 000642 root I ENCP leaving transfer_file(). Thread READ_1 00:02:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30158 current 30157 44 Thread seek_thread 00:02:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:02:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=23944176 Thread tape_thread 00:02:04 stkendm7a.fnal.gov 000642 root I ENCP MSG_TYPE=ENCP_XFER read_from_hsm /pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQMM/NQMM07/f6 -> /data/data1/Migration_Spool/VO6043:0000_000000000_0000887: 652733196 bytes copied from VO6043 at 29.3 MB/S (35.6 MB/S network) (31.1 MB/S drive) (74.2 MB/S disk) mover=9940B27.mover drive_id=T9940B drive_sn=479002016447 drive_vendor=STK elapsed=335.77 {'storage_group': 'ktev', 'encp_crc': 2842112764L, 'disk_rate': 74.242424031910105, 'drive_rate': 31.134990783817397, 'driver': 'FTTDriver', 'transfer_rate': 29.344053430704303, 'mover_interface': 'stkenmvr27a', 'overall_rate': 29.184147107447384, 'media_changer': 'stk.media_changer', 'network_rate': 29.344053430704303, 'encp_ip': '131.225.13.19', 'unique_id': 'stkendm7a.fnal.gov-1257573388-642-425'} Thread READ_1 00:02:04 stkendm7a.fnal.gov 000642 root I ENCP File /data/data1/Migration_Spool/VO6043:0000_000000000_0000887 copied successfully. Thread READ_1 00:02:04 stkendm7a.fnal.gov 000642 root I ENCP Completed transferring 652733196 bytes in 1 files in 335.783460855 sec. Overall rate = 29.2 MB/sec. Transfer rate = 29.3 MB/sec. Network rate = 35.6 MB/sec. Drive rate = 31.1 MB/sec. Disk rate = 74.2 MB/sec. Exit status = 0. Thread READ_1 00:02:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 23944315/23944315, blocks 183 header 139 Thread tape_thread 00:02:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 23944176/23944176 bytes Thread net_thread 00:02:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:02:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30158 block 183 block_size 0 bloc_loc 5658411 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 23944315 block_read_tot 0.228114128113 tape_rate 104966383.267 Thread tape_thread 00:02:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 23944315/23944315 bytes Thread tape_thread 00:02:04 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00207781791687 read 0.105796337128 crc_check 0.118699789047 push 0.001540184021 block_read 0.228114128113 idle 0.00425553321838 Thread tape_thread 00:02:04 stkendm7a.fnal.gov 000642 root I ENCP Version: v3_8 CVS $Revision: 1.963 $ encp.pyc OS: Linux 2.6.9-89.0.9.ELsmp i686 Release: Scientific Linux Fermi LTS release 4.4 (Wilson), User: root(0) Group: root(0) Euser: root(0) Egroup: root(0) Current working directory: stkendm7a.fnal.gov:/opt/enstore/src Command line: encp --delayed-dismount 2 --ignore-fair-share --bypass-filesystem-max-filesize-check --priority 0 --threaded /pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQNB/NQNB11/f14 /data/data1/Migration_Spool/VO6043:0000_000000000_0000896 Thread READ_1 00:02:04 stkendm7a.fnal.gov 000642 root I ENCP Creating zero length output files. Thread READ_1 00:02:04 stkendm7a.fnal.gov 000642 root I ENCP Sending /pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQNB/NQNB11/f14 read request to LM: unique_id: stkendm7a.fnal.gov-1257573724-642-434 inputfile: /pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQNB/NQNB11/.(access)(001F000000000000007E9CE0) outputfile: /data/data1/Migration_Spool/VO6043:0000_000000000_0000896 Thread READ_1 00:02:04 stkensrv4n.fnal.gov 005795 enstore I 9940BLM MSG_TYPE=ADD_TO_LMQ request added to pending queue for storage group : ktev 00:02:04 stkensrv4n.fnal.gov 005795 enstore I 9940BLM read Q'd /pnfs/fs/usr/ktev/migrated_from_dlt/NQ/NQNB/NQNB11/f14 (stkendm7a.fnal.gov-1257573724-642-434) -> /data/data1/Migration_Spool/VO6043:0000_000000000_0000896 : library=CD-9940B family=ktev.ktev_migration.cern requester:root 00:02:04 stkensrv4n.fnal.gov 005795 enstore I 9940BLM 00:02:05 stkensrv4n.fnal.gov 032536 enstore I SL8500G1MC query drive 0,5,1,7 => 0,0,5,1,7 online available IBM-LTO4 00:02:05 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:02:05 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30159 current 30158 183 Thread seek_thread 00:02:05 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:02:05 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=3522082 Thread tape_thread 00:02:05 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 3522221/3522221, blocks 27 header 139 Thread tape_thread 00:02:05 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 3522082/3522082 bytes Thread net_thread 00:02:05 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:02:05 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30159 block 27 block_size 0 bloc_loc 5658439 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 3522221 block_read_tot 0.0495865345001 tape_rate 71031804.007 Thread tape_thread 00:02:05 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 3522221/3522221 bytes Thread tape_thread 00:02:05 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.000841379165649 read 0.0318992137909 crc_check 0.0166170597076 push 0.000228881835938 block_read 0.0495865345001 idle 0.00056004524231 Thread tape_thread 00:02:05 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:02:06 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:02:06 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30160 current 30159 27 Thread seek_thread 00:02:06 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:02:06 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=5641392 Thread tape_thread 00:02:06 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 5641531/5641531, blocks 44 header 139 Thread tape_thread 00:02:06 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 5641392/5641392 bytes Thread net_thread 00:02:06 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:02:06 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30160 block 44 block_size 0 bloc_loc 5658484 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 5641531 block_read_tot 0.0679497718811 tape_rate 83025017.5066 Thread tape_thread 00:02:06 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 5641531/5641531 bytes Thread tape_thread 00:02:06 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00105547904968 read 0.0387806892395 crc_check 0.0277338027954 push 0.000379800796509 block_read 0.0679497718811 idle 0.00115251541138 Thread tape_thread 00:02:07 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:02:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:02:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30161 current 30160 44 Thread seek_thread 00:02:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:02:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=525440 Thread tape_thread 00:02:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 525579/525579, blocks 5 header 139 Thread tape_thread 00:02:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 525440/525440 bytes Thread net_thread 00:02:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:02:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30161 block 5 block_size 0 bloc_loc 5658490 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 525579 block_read_tot 0.0319480895996 tape_rate 16451030.6121 Thread tape_thread 00:02:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 525579/525579 bytes Thread tape_thread 00:02:07 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.000532865524292 read 0.0286264419556 crc_check 0.00274395942688 push 4.48226928711e-05 block_read 0.0319480895996 idle 0.000102043151855 Thread tape_thread 00:02:08 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:02:08 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30162 current 30161 5 Thread seek_thread 00:02:08 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:02:08 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=10423224 Thread tape_thread 00:02:09 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 10423363/10423363, blocks 80 header 139 Thread tape_thread 00:02:09 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 10423224/10423224 bytes Thread net_thread 00:02:09 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:02:09 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30162 block 80 block_size 0 bloc_loc 5658571 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 10423363 block_read_tot 0.109810829163 tape_rate 94921084.555 Thread tape_thread 00:02:09 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 10423363/10423363 bytes Thread tape_thread 00:02:09 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.0014533996582 read 0.0560998916626 crc_check 0.0515823364258 push 0.000675201416016 block_read 0.109810829163 idle 0.00179100036621 Thread tape_thread 00:02:09 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM 00:02:10 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:02:10 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30163 current 30162 80 Thread seek_thread 00:02:10 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV skip_fm 1 Thread seek_thread 00:02:10 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape starting, bytes_to_read=48955583 Thread tape_thread 00:02:10 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read bytes 48955722/48955722, blocks 374 header 139 Thread tape_thread 00:02:10 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client: wrote 48955583/48955583 bytes Thread net_thread 00:02:10 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client waits for tape thread to exit Thread net_thread 00:02:10 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV drive stats after read. Tape VOO329 position 30163 block 374 block_size 0 bloc_loc 5658946 tot_blocks 762939000 BOT 0 read_err 3 write_err 0 bytes 48955722 block_read_tot 0.44425535202 tape_rate 110197258.8 Thread tape_thread 00:02:10 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape exiting, read 48955722/48955722 bytes Thread tape_thread 00:02:10 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV read_tape timing:get_space 0.00492477416992 read 0.194490194321 crc_check 0.241708993912 push 0.00313138961792 block_read 0.44425535202 idle 0.00840020179749 Thread tape_thread 00:02:11 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV write_client exits Thread net_thread 00:02:11 stkenmvr213a.fnal.gov 006246 root I LTO4_113MV tape position: target 30164 current 30163 374 Thread seek_thread 00:02:11 stkensrv4n.fnal.gov 005828 enstore I LTO4F1LM