Table of Contents

Blackeagle Data

To find out we need some data about what is going on.

Bullets are action items to address.

Scratch

After fixing the reclamation schedule and letting it run all day long with a reclamation threshold of 25%, blackeagle's scratch count went up to 100 or so. Since then, mid summer, it has continually shown a downward trend. Like so (it is accelerating a bit with everybody back):

NR DATE
42 Mon Sep 5 10:00:01 EDT 2011
42 Mon Sep 5 11:00:01 EDT 2011
41 Mon Sep 5 12:00:01 EDT 2011
43 Mon Sep 5 13:00:01 EDT 2011
43 Mon Sep 5 14:00:01 EDT 2011
43 Mon Sep 5 15:00:01 EDT 2011
43 Mon Sep 5 16:00:01 EDT 2011
43 Mon Sep 5 17:00:01 EDT 2011
43 Mon Sep 5 18:00:01 EDT 2011
43 Mon Sep 5 19:00:01 EDT 2011
42 Mon Sep 5 20:00:01 EDT 2011
39 Mon Sep 5 21:00:01 EDT 2011
39 Mon Sep 5 22:00:01 EDT 2011
39 Mon Sep 5 23:00:01 EDT 2011
39 Tue Sep 6 00:00:01 EDT 2011
38 Tue Sep 6 01:00:01 EDT 2011
38 Tue Sep 6 02:00:01 EDT 2011
36 Tue Sep 6 03:00:01 EDT 2011
36 Tue Sep 6 04:00:01 EDT 2011
36 Tue Sep 6 05:00:01 EDT 2011
35 Tue Sep 6 06:00:01 EDT 2011
35 Tue Sep 6 07:00:01 EDT 2011
33 Tue Sep 6 08:00:01 EDT 2011
32 Tue Sep 6 09:00:02 EDT 2011

Occasionally we see the disappearance of tapes/volumes. Like below. This happens when two or more tapes/volumes undergo a “relabel” event because of our setting RELABELSCRATCH=YES. Hong does not remember why we have this set, might have been an earlier problem with VTL. Not damaging as tapes/volumes will relabel at next interval. I initially thought it was related to session load.

NR Volumes Date Note
1120 Thu Sep 1 10:00:01 EDT 2011
1120 Thu Sep 1 11:00:01 EDT 2011
1120 Thu Sep 1 12:00:01 EDT 2011 laptop window starts
1117 Thu Sep 1 13:00:01 EDT 2011
1112 Thu Sep 1 14:00:01 EDT 2011
1112 Thu Sep 1 15:00:01 EDT 2011
1120 Thu Sep 1 16:00:01 EDT 2011
1120 Thu Sep 1 17:00:01 EDT 2011
1118 Thu Sep 1 18:00:02 EDT 2011
1120 Thu Sep 1 19:00:01 EDT 2011
1120 Thu Sep 1 20:00:01 EDT 2011
NR Sessions Date Note
5 Thu Sep 1 10:00:01 EDT 2011
2 Thu Sep 1 11:00:01 EDT 2011
3 Thu Sep 1 12:00:01 EDT 2011
218 Thu Sep 1 13:00:01 EDT 2011 max connections=225
206 Thu Sep 1 14:00:01 EDT 2011
167 Thu Sep 1 15:00:01 EDT 2011
88 Thu Sep 1 16:00:01 EDT 2011
43 Thu Sep 1 17:00:01 EDT 2011
23 Thu Sep 1 18:00:02 EDT 2011
31 Thu Sep 1 19:00:01 EDT 2011
15 Thu Sep 1 20:00:01 EDT 2011

PM Scedule

We need to relieve the early afternoon sched1216, it is simply too full. It puts the server under tremendous loads (loads above 15 have been observed, 8 is business as usual)

NR of sessions Date
1 Thu Sep 8 12:00:01 EDT 2011
266 Thu Sep 8 13:00:01 EDT 2011 *
221 Thu Sep 8 14:00:01 EDT 2011
164 Thu Sep 8 15:00:01 EDT 2011
141 Thu Sep 8 16:00:01 EDT 2011
84 Thu Sep 8 17:00:01 EDT 2011
24 Thu Sep 8 18:00:01 EDT 2011
25 Thu Sep 8 19:00:01 EDT 2011
10 Thu Sep 8 20:00:01 EDT 2011

It appears most servers are simply stuffed into the first schedule listed. Since the first window is for laptops we need to move nodes out of that slot into others, surely not all of these are laptops. 1,048 nodes so the ideal number is to have about 200 per schedule window.

UPDATE 09/20/2011:

SCHED1216 node count = 550

AADZENYAHVTL AAHMAD-11 ABALDWINXPLP ABERGEN-10 ABERLIND-GX620 ABRUNO-11 ACABAL07 ACBURKE-2010 ACOOKE-11 ADUPUY-09 FIELDSTEEL-10PC AGILORDONEZ-2011 AGLAUZTODRAN-06 AGONZALEZ-LAPTOP AHADLER-10 AHAENSCH-08 AHORNSTEIN-10 AINFANTE-2008 AKELLY-11 AKNAUERHASE09 ALEVEY-07 AMACQUEEN-DESKTOP AMCLANE-10 AMILLER01-12 AMILLER01XP AMULLINS-06 AOSTOR-08 APATALANO-11 APEREZGIRONES APETERS-10 ARCHPGIS ARUBACHA-10 ARUSHDY-2011 ASANTANIELLO07 ASCHMIDTLAP-11 ASHARMA-09 ASHUSTERMAN-10 ASTUERZEL-11 ASZEGEDYMASZ ATAMHANKAR-11 ATANAKA-106 ATANAKA-2009 ATANG-10 AVANDERBURG07 AWANG01-11 AWANG01-2010 AWARREN-11 AWIGHTMAN-10 BALU-2007 BCHERNOFF-27IMAC BCHERNOFF-329ESC BCHERNOFF-LABGX620 BCHERNOFF-LABGX620B BCHERNOFF-PC BCRAIGHEAD-11 BHOOD-10 BJUHASZ-11 BJWILSON-DESKTOP BKATTEN-10 BMASTERS-11 BMESLIN-2007 BNANGLE-08 BNORTHROP-2009 BSALAFIA-10 BSTEWART-2008 BSTEWART-GRADG5 BSTEWART-MACBOOKPRO BSTRICKLAND-HP CANDREWSXP06 CASHTON09 CBARAW-IMAC CBRODOWSKI-06 CCRUZ01-08 CDANIELS-07 CDANIELS-08 CFHVISITOR-05 CFHVISITOR2-05 CHENOWETHLABMAC3 CHENOWETHLABMAC5 CJPOTTER09 CKEARNEYVTL CLANSERHP07 CLAZOR-VM CLECHOWICZ09 CLEIDY-07 CMAINES-10 CMCCANNHP-08 CMILLER-08 CMILLER-12 COSTROW CPARENTE CPINA-07 CPOTTER01-11 CRASMUSSEN-08 CROCKWELL-MAC CRODRIGUE-08 CSALAS-WIN7 CSANISLOW-HM24 CSANISLOWLAB4-11 CSCULLYHP07 CSMITH03-12 CSORKIN-2011 CTHORNTONHP07 CZINSER-2008 DABROWN-NETBK DACSLIDESCANNER DALBERT-2005 DALEXSON-2008 DALLEN-2010 DALLEN-LCQHA033 DALLEN-POSIM DALONG-11 DALONG08 DBEVERIDGE-37HA DBEVERIDGE-66PC4 DBEVERIDGE-DESKTOP DBEVERIDGE-LABIMAC DBODZNICK-2008 DBODZNICK-LABGX280 DBOULE-OSXPOSIM DEUDELL-12 DGANDOLFO-11 DHENRY-08 DHIEZMAN-VM DJBLOOM-11 DKATZ01-10 DLABONTE-NETBK DLAWRENCELOAN DMATTHEWS-09 DMEREDITH-GX280 DMEYERXP06 DMOON-10 DOLIVER-2008 DOLIVER-LABMINI DPFLEDERER-10 DPFLEDERERHP-11 DPHILLIPS-08 DPNELSON DRICH-VM DROYER-445 DSCHORRVTL DSHEEHAN-08 DSHOREVTL DSIERPINSKI-10 DSIERPINSKI-10A DSIMSHP07 DSTRICKLAND-HP DWHITE-2009 DWINAKOR-10 EATAYLOR-LABAKTA EATAYLOR-LABDELL EATAYLOR-LAPTOP EATAYLOR-OFFICE EBELOW08 ECHARRY-2010 ECHENOWETHLABMAC2 ECONSTUWORK-09 EFOWLER-10 EGERSHON-10 EKLEINBERG-09 ELIM-09 EMCALISTER-11 EMCALISTERIMAC-09 EMCCORMICK-08 EMEREDITH09 EMORAN2006 ENERENBERG EPARIS-2010 ESAVAGE-PHYS ESLUSSER-11 ESZADY-07 ETAKAHASHI ETHOMAS-G5 ETHOMAS-LAPTOP ETRAUBE-12 EVISVARDI-10 FCARNEGIENET FCOHAN-2008 FELLIS-2004 FGRAMUGLIAXP07 FHAUSER09 FKOERTING-09 FKUAN-09 FSTARR-SHAKTI FWILLIAMS-10 GAARON-2008 GBERNARD-07 GEBSTEINHP-08 GGALLAROTTI-10 GGIBBONS-VM GGOLDBERG-12 GOVVISITOR-07 GPETERSSON-2009 GPYKE-12 GSHAW-10 GSKILLMAN-10 GULYSSE-10 GVOTH-2008B GWHEELER-MAC GYOHE-10 HBARTHLAB-MBP HERIC HFCHAMBERS-PC HMCNEILVTL HNEEDLER-08 HPMIMIME-10 HPSTEVENS-2008 HWHEELER09 IALESHKOVSKY-2011 IHARJITOVTL IKARAMCHETI-11 IMUKERJI-2009 IMUKERJI-273HA-PR IRUSSU-2006 ISAAKA JADAMS-2009 JALDEN01-2011 JAMCKENNA-MAC JBAIRDBURNET-12 JBIDDISCOMBE-MAC JBONIN-09 JBRUNO-2010 JBSILVERHP07 JBURNETTHPLAP JCASTRO-10 JCROOKE2XP JCUTLER-11 JDANDREA-11 JDEBOER-IMAC JDONADY-2008 JDRISCOLL-08 JDRISCOLLHP08 JFINN-11 JFOUNTAIN-10 JGILARDE-2006 JGOODALE-VM JGREENWOOD-IMAC JHICKS-11 JJACOBSEN-10 JKAMM-10 JKAMMLOAN-01 JKAUANUI-08 JKAUANUI-12 JKAUANUI-LT JKNEE-2008 JLABELLA-G5 JLEONE-08 JMCGUIRE-10 JMCINTOSH-2005 JMEERTS-2008 JMILLER04XP JMORAWSKI-MINI JMORAWSKILAB-05 JMORAWSKILABG5-06 JNAEGELE-2008 JNAEGELE-MACBOOK JOSEPHREED-11 JPECK-4500 JPFISTER-10 JPOWZYK-2006 JQUIJADA-12 JRAITI-11 JRANDALLVTL JRIDER-2011 JRKIRN-2011 JROSE01DELL-09 JROUSE-11 JSCHIFF JSEAMON-09 JSHORTELL-11 JSLANE-MAC JSLANEMAC JTOPSHE09 JTUCKER-08 JVAREKAMP-MARSHGX280 JVODAK09 JWALTER-10 JWANG03-11 JWESTMACBOOK JWILLIS-11 JWOLFE-LAPTOP JZWELLING-07 KANDERSON-10 KATTENINTERN-10 KBIRNEY-12 KKOLCIO KNICHOLS-08 KNORRISELITE-10 KRIM-12 KROBERTS-11 KSAVINELLI07 KSMITH-VM KTENEYCKVTL KTHORPE-MBP KTOLOLYAN-10 KVANBLARCOM-10 KVOELKELMEIM-LAPTOP KWARREN-MAC KWEIL-LAPTOP KWEIL-SOPHIE KWHALEN-07 LAPPEL-2006 LCALDWELL-11 LCARRASQUILL-10 LCOHEN01-12 LCOHEN01-MB LDOMBROWSKI-2011 LGRABEL-LABIMAC LGUENOVA-11 LJSTARK-10 LLAPLANT-08 LLENSING LLUKENS-DESKTOP LMCQUEENEY-VM LMIMS-07 LMWRIGHT LNUSSDORFER-11 LSACKS-11 LSBROWN-08 LSCOTT-09 LSTARRMAC-10 LWEINBERG LWESTLING-2007 MAHADDAD-08BE MANJON-RES-VM MARMSTRONG-2011 MATH613B MATH630B-GX270 MATHTEMP-07 MBNGUYEN-12 MCALNEN-2009 MCALTER-2007 MCALTER-2012 MCGARRETT-09 MEISNER-11 MEMLABIMAC-05 MEMLABIMAC-09 MFITZGIBBONS-2007 MGILMORE-2008 MGILMORE-PLANETARY1 MGILMORE-PLANETARY2 MGKELLY09 MHERLIHY-6930 MHINGORANI-2010 MHINGORANI-LABG5 MHINGORANI-NEWKINTEK MIMAI-10 MIMAI-EAS MISLEIB-10 MKEANELAPTOP MKINGSLEY-09 MKLANESKI-MAC MKURTZ07 MLOOMER-03 MMCALEAR-2012 MMCALEAR-LABOS9 MMCALEAR-OFFICE MMELENDEZ-10 MMUROLO-LAPTOP MMYERSBROWN-08 MNGUYEN-12 MOLERUDVTL MOTEIZA-2011 MRESNICKHP-08 MROBERTS-10 MROTH-2010 MRUBENSTEIN-11 MSCHUG-09 MSCIOLA-HP MSCIOLAHP-11 MSIMONSONHP07 MSINGER-2008 MSINGER-LAPTOP MSNOW-2009 MTETER-10 MWEIR-2008 MWEIR-LABG5 MWHALEN-MACBOOK MWHALEY-2009 MWHITCOMB-10 MZOCCO-08 NAKSAMIJA-2010 NGARRETT-10 NGARRETT07 NGREENE-11 NKORDA-10 NMEISLAHN-2009 NSHAPIRO-2011 OBOOKAS08 OHOLMES-09 OLEWIS-10 PABRAHAMXP PBOLTON-LABIMAC21 PBOLTON-LABIMAC27 PCARNEYMACBOOK-2009 PDOWDEY-2010 PDSMITH-09 PGOTTSCHALK-11 PHALL09 PHILL-09 PHOLYCROSS-06 PKLECHAPORTE-09 PLANETARY3 PMACKLP PMARK PMATTHUSEN-2011 PMELLEY09 PPATTON-07 PRESOR-2008 PRESOR-GIS2 PRODRIGUEZMO-09 PRUTLAND-11 PSCHWABER-11 PSMATHERS-MAC PTATGE PTATGE-2011 PTIEN-09 PTURENNE2007 PWAGONER RADELSTEIN-09 RAUTRY-10 RBRAULT-CHEMMACBOOK RBRAULT-CHEMPCLAPTOP RBRAULT-DESKTOP RCAMERON-09 RCULLITONXP07 RELPHICK-10 RGAUDALUPE-11 RGORDON-2007 RGROSSMAN-10 RGUADALUPEXP06 RHOWARD-11 RHOWARD05 RIJOHNSON-2012 RJENKINS RJENKINS-IMAC RJOHNSON-MAC RMBERMAN-07 RMEHMET-09 RMURPHY-09 RNISSE-10 ROLSON-2009 ROLSON-LABIMAC RPRATT-2009 RPRATT-G5 RRASMUSSEN-MACBOOK RROLLEFSON-2007 RROSENTHAL-09 RROSENTHAL-WIN7 RSCHATZ-09 RSCHMIDT-10 RSCHUBEL-07 RSILLASENXP RSTEELE-10 RSTRIEGELTEST SBACHNER-11 SBACKER07 SBROUGHVTL SBUNAEVVTL SCAYETANO-2008 SCHAKRAVARTI-10 SCHYUNG08 SCOLLINSVTL SCROUCHER-08 SDAILEY-07 SDEVOTO-2008 SDEVOTO-HALIBUT SDEVOTO-JUDAS-PRIEST SESULTAN-2007 SESULTAN-LABIMAC SFAZZINO-2005 SGUHA-12 SHIGGINS SHILL-DESKTOP SHOLMES-2008 SHOLMES-G5 SHUSTLABIMAC2-09 SHUSTLABIMAC3-10 SJACARUSO09 SKHALIGHI_MO-PHYSGRAD SKIRSCHE-11 SLASTRINA-2007 SLAZAREXP07 SLOURIE SMANJON-2008 SMBROWN-NETBOOK SMCCANN-09 SMCCANNHP-09 SMCQUEEN-10 SNOVICK-2009 SOCONNELL-2008 SPARSAMOGHAD-PHYSGRAD239 SPASSMAN-2010 SPASSMANSERVER SPAVIS08 SPHELPS-10 SPRINGERHP-07 SRUDEN-06 SSADLERVTL SSOLOVEI-ANIMALCARE SSTEMLERLABMAC-09 SSTEMLERVSTA-09 SVANEK-2008 SVONSCHOELER-GX280 SWEINER-11 SWILIARTY-09 SWU03-XP SZHENG-2011 TCKU-2007 TCKU-LABIMAC TCKU-LABIMAC2 TIMADA-11 TIRANI-09 TKANNAM-2009 TKANNAM-XP06 TKAWASHIMA-2011 TLWILSON10 TMACRI04 TMORGAN-2010 TMORGAN-XP06 TOVERTON-12 TSABO-10 TTELFAIR-2011 VDIAZ-11 VGILLISPIE-10 VGRAHAM-10 VHARRIS-2008 VHARRIS-EESOFFICEWORKSTATION VMARINELLI-2008 VNYE-10 VSABINLAPTOP VSMOLKIN-11 WCOMFORT-10 WESTMORELAND-2007 WHOLDERMACBOOK-2009 WJOHNSTON-09 WJOHNSTON-MBP WKCHAN-09 WNELLIGAN-10 WPINCH-11 WPRINGLE-2007 WRAYACK-10 WRITWORKS-06 WSTOWE-08 WSTOWE-11 WSTOWE-12 WSTRACKINGUPPER XZHU-WIN7 YKORDONSKIY-XP YMALIN

SCHED1821 node count = 126

ACOOKEGW07 AFRY-2010 AFRY-LABG5 AFRY-ROOM137 AMARZOUCA-WIN7 AMCFADZEN-11 AROJAS-06 ATANASI-VM AWALSH-09 AWENSINGER BDANIEL-06 BLOOMBERG-11 BNORTHROP-745 BSTEWART-LAB08 BTRAVERS-11 CCOLFER-VM CKAUFMAN-DELL CMATTZA-2011 COTHON-2010 CROGAN DBEVERIDGE-66PC2 DFOYLE-11 DHINTONXP DKLARE-10 DKOEPF-HP DLOW-10 DMEYER-11 DOLIVER-LABPC DRHENRY-OBHS-VM DSCHNAIDT EAS-MACMINI EATAYLOR-LABIMAC ELEE01-10 EMILROY EMMORAN ETINKER-11 ETINKER-VM FCOHAN-HOT-GREEN-SLIME FDEGIOVANNI-2011 GAARON-LAB1 GVOTH-LAB4 GVOTH-LABPC3 GVOTH-VIDEO3 HEPSTEIN-09 HEPSTEIN-10 HTRELOAR-2012 IMUKERJI-LAB2 JBURNETTXP JCONNARY07 JCORMACK-10 JCROOKE2KP JESTEVESXP-07 JFIELDS-MAC JGILARDE-LSMMETA JGYURITS-09 JHUEBLER07 JKEHOE09 JKNEE-LABPC JLOWRIE-GW JNAEGELE-LAB1 JNAEGELE-LAB990 JRAFFERTY-11 JRAFFERTY07 JRKIRN-PC2 JVAREKAMP-2010 KKUENZLI-2010 KLOGSDON-09 KMULLEN-10 KONEILL-11 KPUGLIESE-07 KPUGLIESE-08 KSAVINELLI11 KSCHEIBE-10 KSCHEIBEXP KSHINOHARA KSTEFANOWICZ-09 KVECCHITO-VM KWOLFE-09 LBORHMAN-09 LCARLSON LGRABEL-TEMPMAC LHUWEL-2010 LJSTARK-VM LKEISER03 LNEARY-2011 MDAVIS-11 MFRISCH-2010 MGILMORE-GX260 MGILMORE-LABBOOTCAMP MHINGORANI-LABDELL8100 MICAHWIN7 MNAKAMURA-2011 MNEALE-10 MOSPINA-2011 MWEIR-LCQ NBRUCE-2011 NICKDAVID-08 NSCHWARTZ-11 ODRAKE-MAC PABRAHAMXP1 PHOTOID-10 PRESOR-MACMAC PSTAYE-10 PTYNANMAC RBRAULT-STORM153HA RCHIAPETTA-MACMINI RKISSINGER-VM RLANE-165 RMCKENNA-VM ROBINZUP-WIN7 RPRATT-LABG5 SANTONIO2-09 SBACKER11 SCWIRKA09 SIGNSHOP-11 SJAVORSKI-10 SMCGOWAN-XP SNOVICK-LABGX620 SNOVICK-MACHINE THARLEY07 TLINDROS-11 TPARKER-IMAC VLANE-2010 WESTMORELAND-LABXW6600 WSUSANTHAWIJ-239ESC YPRAMUDYA-PHYSGRAD235

SCHED2100 node count = 131

AAISSA-2010 ACHATFIELD-2011 AGERTZ-ASIAN ARUDENSKY-2011 ASTARON-VM BALLY-2011 BCHERNOFF-2009MAC BCHERNOFF-FISH2 BCHERNOFF-LABIMAC BFAY-11 BSTEWART-LABGX260 CCARR-VM CFA-MARKETING CFLANAGAN-10 CHAGNERXP07 CHENOWETHLABPC1 CNASCIMENTO-2010 COTHON-LABPC2 CPIKE01-08 DBEVERIDGE-66PC1 DCANALIA-04 DHUTSON-11 DISPATCH1-10 DOLIVER-LABHP DSKURA01-10 DTREISTER-11 DWRIGHT08A ECAREY-08 EDELVALLE-GW EHARRISXP07 EMAINELLA-10 FCOHAN-LABHP FMARSILLIXP07 GAARON-LAB2 GAARON-TEMP GGDOVIN-VM GGIBBONSWIN7-VM GGRUBBS-2011 GRODRIGUEZ07 GUN-11 GVOTH-LAB2 GVOTH-VIDEO7 HORAVEC-2011 IMUKERJI-HPLC INVESTINTERN-1 INVESTMENT1-11 JBEST-2011 JBUCKLEY-11 JCHIARI-2011 JENXUTOWIN7-VM JFIELDSGWE06 JGILARDE-2012 JGILARDE-SEM JGOETZ-2011 JGUERRERO-2010 JHAMILTON07 JKNEE-RM025 JMAGUDA-GX280 JNAEGELE-LAB2 JOUTLAW-10 JSEADEEK-10 JSTEELE2-09 JTHOM-10 KCOLLINS-2011 KHOOK-10 KHOOKXP06 KMURPHYHP08 KSRUBASWIN7-VM KTAILLONWIN7 KWINSTON LANTOSZWIN7-VM LCUMMINGS-VM LDIERKER08 LGAMBELL-09 LHUWEL-LAB745 LMASCARO-VM LNEARY LSPRINGER-11 LSTETHERS-10 MBLAKE-VM MCALTER-LABGX620 MDAMEWIN7-VM MGILMORE-PLANETARY MGILMORE-PLANETARYBC MGLYNN-09 MHUMPHRIES-2011 MPERALTA-11 MSINGER203 MSNOW-2012 PBOLTON-HA038A PCBEST-10 PCROWLEY-10 PGRANDE-07 PJACKSON-09 PLEONE-IMAC PRESOR-MACPRO PTULLY-10 QRODE-11 RBRAULT-153HADELL RCONNOLLY09 RCULLITON11 RLANE-2010 RLANE-LABIMAC RLANE-RTPCR RMCCALLUM-10 RPRATT-BEAST RPRATT-LABIMAC2 RPRATT-LABPC RSCHMIDTXP07 RWILSON-09 RZUPXP07 SANTONIO-09 SASMITHWIN7-VM SBENSONWIN7-VM SDEVOTO-ZZTOP SFERRIS-VM SHCOHEN-08 SHOLMES-LABWIN7 SJHIGGINS-10 SKULESZA-09 SMARTINXP06 SNORTHGRAVES08 SOCONNELL-LAB419 SOPHAT-VM SSARVER-10 SUMARSAM TMORGAN-LABPC2 TSALAFIA-VM USDAN-4WINDS WSAADMIN3-10 YSAMUELMAC

SCHED0003 node count = 131

ABALDWIN09 ABLOOM-2010 ABRUNO-VM ACBURKE-LABG5 ACURRAN-IMAC AFRY-LAB620 AKIDDLEWIN7-VM AKLEIN01-10 AMARTIN-2010 AMORINWIN7-VM ARUSHDY BCHERNOFF-2009 BCHERNOFF-LABBC7 BSCHUKOSKE-VM CCOLFER04 CCPFELLOW-VM CFLORES-11 CKAUFMAN09 CLINICXP CPOISSON CRIZZA08 CSCHUKOSKE-10 CWOOD2005 D2WORKHORSE DBEVERIDGE-66LABPC3 DDYER-DELL DHINTONGW DISPATCH2-10 DISPATCH3-10 DMOORE07 DOLIVER-LABDIM EBROWN02-09 EKONERDING-10 ELECSHOPXP ESZADY-VM FCOHAN-LABVISTA GAARON-LAB3 GBROWNXP GVOTH-VIDEO8 HAIELLO-10 HMINETTI-MAC HSTEGMAIERWIN7 HWEIDNER-08 IADEBOYE-2012 IMUKERJI-FLUOROMAX4 IMUKERJI-LABPC1 IYORK-11 IYORK-411 JDANDREA-06 JESTEVESXP07 JGILARDE-3DRECON JGILARDE-AMT1 JGILARDE-CZV JHEIDORN-WIN7 JJURGEN-11 JKNEE-TEMP JMAGUDA-GX280 JMLOZANOWSKI-10 JMORAWSKI-10 JMORAWSKI-VM JPLATT-GX280 JRKIRN-G5 JSIRY-2011 JTILLEY-11 JVODAK-09 KBEAUDETTE-2011 KBORDNER-06 KCATALDI-09 KHOBBS-10 KKARPA-08 KMACKOWIN7-VM LBERMAN-2011 LCALHOUN08 LCARTA-VM LFLANNIGAN-VM LGRABEL-IMAGEPC LHUWEL-LAB05 LHUWEL-LABGX270 LMIMS-11 LSACKS07 MBIDDLE-10 MCLAWSON-12 MFAYWIN7-VM MGAWLAK MGILMORE-PLAN3TBC MGILMOREBOOT MLINK-VM MLINKWIN7-VM MMCALEAR-IMAC MMCALEAR-LABIMAC MSINGER-LABPC MWEIR-PPA NDABBSWIN7-VM NDANNER-2010 NMCLEES-411 OBARTLETTHP-08 PBOLTON-LAB038PC PHOLYCROSS-11 PMEYER-2007 RBLUMEL-ORIGIN RCONN RCROPPERXP RLANE-EMAC RLANGER7 RMCCALLUM-09 RPRATT-274TYPHOON RPRATT-PC RSHAW-2010 RSRUSSO SATWELL-VM SCONDONXP SDEVOTO-OZZY SHINER-TEST SHORST-11 SLAZARE11 SMARTIN-11 SNOVICK-LABHP SPRIVITERA-2011 SSTEMLER-09 STARABA-10 TBOSTICK-11 TCKU-2012 TKOTTOS-HAMIDWI TMAHEU-09 TPOLICKI-07 TSTANLEY-HP WBURKHART-MAC WESTMORELAND-HA42 WFIRSHEIN-09 WSAADMIN4-10 YSAMUELMACMINI-09

SCHED0306 node count = 110

ACBURKE-LABHP ACOOKE-PC11 AFRY-128HA AMARTIN-WORKSTATION ASCHOLAR-GX260 ATURCO-06 BCAHILLXP07 BCHAPPELL-WIN7 BNANGLE11 BNORTHROP-LAB360 CDOLANSKY-10 CGONZALES-VM CHAGNER-12 CLALANDE CLWILKINS-12 COTHON-LAB1 CPINA-11 CRACEWIN7-VM DBEVERIDGE-LABPC1 DBLACKMINI-11 DGRASSO-VM DHINTON-10 DKELLY-09 DKOEPF-2009 DLAWRENCE07 DLEIPZIGER08 DLEISAWITZ-2011 DMEYER-10 DSCHORR EGLICK-09 EMALMSKOG-2012 EROOSBROWN-IMAC FCOHAN-EGGS FCOHAN-LABVISTA2 FELLIS-2012 GAARON-XW6600 GDRISCOLLWIN7 GFOUNTAINXP07 GVOTH-VIDEO4 IRUSSU-2012 JDRECHSLER-10 JGILARDE-EDAX JIMILLER-2011 JJYLKKAHPLAP07 JKAMMWIN7-VM JLAWTON-10 JMAGUDA-12 JRABAMAC JRKIRN-LABMICRO JRKIRN-LABSILVER JSCHENKER-09 JSTEELE-10 JTCALLAGHAN-08 JTHOM2-09 JVAREKAMP-MLABGX270 JWAREHAM-11 JWATSON-HP KMCQUEENEY-WIN7 KNICHOLSVTL KPORTER01-11 KWILIARTYPC-07 LANTOSZ-VM LAWESTLING-2010 LCALHOUNVTL LDERUSHA-09 LHENDRIX-10 LPINETTE-09 LVILLARD07 LWOODARD-IMAC MARMSTRONG-2006 MBAILEY-09 MDATREDELL-11 MGILMORE-BOOTCAMPPC MHANSON-09 MHINGORANI-MAX3 MHINGORANI-XW6600 MKHAMIS-12 MKOZIOL-DESKTOP MLOOMER-VM MSINGER-2010 MSLOBIN-2011MINI MSNOW-2009 NANGELETTI07 NMCLEES-10 OFLORES PPAIGE09 RDARNA-10 RLANE-108 RLOONEY-06 RPANDOLFO-09 RPRATT-LABPC1 RYORK SDEVOTO-ACDC SELLIOTT09 SESULTAN-LABPC2 SFUSSO SGRUCAN-10 SHETTLEWORTH-VM SLIN-HA286BIO SNOVICK-GRAD2 SNOVICK-GRAD3 SOCONNELL-LABNEW SPRIVITERA TCKU-LAB1 TKIEFER-10 TMORGAN-LABPC1 TWEST-09 VINYLCUTTER VOICERECORDING WESTMORELAND-12

Size

Total backup volume is currently 45 TB, up from 42 TB mid summer. Top 25 nodes account for 15.3 TB or 34%.

NODE_NAME TOTAL_MB Note
BSTEWART-2008 1967245 external drives with family vacation pictures
MGILMORE-PLAN3TBC 1270853 new replacement for mgilmore-planetbc
DROYER-445 1097047 just suddenly appeared overnight in > 1TB group
MGILMORE-PLANETARYBC 1038708 has not contacted TSM in 150+ days
MGILMORE-PLANETARY2 975637 top5 = 5373853 or 12%
MGILMORE-2008 778021
KANDERSON-10 710883 view filespace data
BTRAVERS-11 636657
PRESOR-MACMAC 615274
SCOLLINSVTL 547504
PRESOR-MACPRO 533161 top10 = 9090325 or 20%
SHUSTLABIMAC2-09 497669
RBRAULT-DESKTOP 492860
JWAREHAM-11 461870
MGILMOREBOOT 453614
GVOTH-LAB4 433816
RPRATT-BEAST 391407
PRESOR-2008 360682
DSCHNAIDT 356824 view files data
SNOVICK-MACHINE 341067
IHARJITOVTL 337055
MKLANESKI-MAC 330143
JGILARDE-LSMMETA 328862
SDEVOTO-2008 314268 top25 = 15272127 or 34%

Files

The number of files has a significant impact on client (metadata needs to be transferred) as well as on database (each file is a metadata record in multiple tables). What gets backed up is impacted by this, for example, some analyses reveals that perhaps 15% of the number of files reaching the BACKUPPOOL (incremental, incoming) has a filename that contains the path …/*CACHE*/…

There are a total of 118 million files backed up.

NODE_NAME NUM_FILES 9/2/2011
DSCHNAIDT 4459413 node size is 357 GB, avg file size is 84 KB
AMACQUEEN-DESKTOP 4234858 node size is 256 GB, avg file size is 62 KB
JBEST-2011 3551854
JFIELDS-MAC 3514101
KANDERSON-10 3312859 top5 = 19073085 or 16%
FSTARR-SHAKTI 1706581
KANDERSON-10 1498570
SNOVICK-MACHINE 1473626
MWEIR-LABG5 1395251
MWEIR-LCQ 1086678
BSTEWART-2008 1036992 top10 = 27270783 or 23%
SNOVICK-2009 963130
PCARNEYMACBOOK-2009 900007
HPSTEVENS-2008 893613
JAMCKENNA-MAC 890057
KANDERSON-10 850155
RPRATT-BEAST 742613
KOS 713745
SDEVOTO-2008 698238
PRESOR-MACPRO 679111
KANDERSON-10 674675
WJOHNSTON-09 671055
WHOLDERMACBOOK-2009 608875
SNOVICK-MACHINE 569640 top25 = 37125697 or 31%

Filespace

Lots of different filespaces are being backed up. Here is an example.

This node is currently 710 GB but could expand to slightly over 6 TB.

Mostly a Mac problem.

Node Name Filespace FSID Platform Filespace Is Files- Capacity Pct
Name Type pace (MB) Util
Unicode?
KANDERSON-10 / 1 Mac HFS Yes 476,612.0 26.4
KANDERSON-10 /Volumes/B- 3 Mac HFS Yes 953,741.3 24.0
lueGlow
KANDERSON-10 /Volumes/My 4 Mac HFS Yes 476,812.0 96.7
Passport
KANDERSON-10 /Volumes/U- 5 Mac HFS Yes 476,940.0 21.1
ntitled 1
KANDERSON-10 /Volumes/R- 6 Mac HFS Yes 476,940.0 27.9
edEye
KANDERSON-10 /Volumes/I- 7 Mac HFS Yes 1,907,601.0 5.2
omega HDD
KANDERSON-10 /Volumes/S- 8 Mac HFS Yes 1,907,601.0 7.0
ilver
Bullet
Potential Total 6,676,247

ANSWER http://www.oucs.ox.ac.uk/hfs/help/excluding.xml.ID=excludedrives

explains nicely what Jim was talking about excluding drives via GUI and also points out you can do it via client dsm.opt settings. but it also shows me how we can do this server side wise, so i tried with bstewart-2008 after making sure /Volumes/WD1.5TB1 and /Volumes/WD1.5TB-2 were unique, added excludes for them and boom they were gone, node back to 350GB.

so if we adopt a convention to force / to mount at some unique mountpoint, which it already does, we could do, evaluate from bottom up

exclude /Volumes/…/*
include /Volumes/SomeApprovedVolume/…/*
include.dir /Volumes/Macintosh HD

In the meantime i can set up global excludes for some of our problem cases whose mountpoint are unique and if they do not need to be backed up.

Mac Drives

report of 09/19/2011: roughly 45 drives with a total utilized capacity of 7.7 TB

It may be that some of these drives are not being backed up, as they may have been excluded before, but were not delete from TSM.

node_name volume fsid platform fs type unicode capacity MB pct util GB util
AMACQUEEN-DESKTOP /Volumes/amacqueen-extra-drive 3 Mac HFS Yes 610,152.3 73.9451
BSTEWART-2008 /Volumes/LACIE 8 Mac HFS Yes 152,625.3 89.1136
BSTEWART-MACBOOKPRO /Volumes/RAID02 2 Mac HFS Yes 956,699.4 39.6379
DMATTHEWS-09 /Volumes/LaCie 2 Mac HFS Yes 953,869.3 30.5291
EMEREDITH09 /Volumes/My Passport 3 Mac HFS Yes 476,812.0 52.0248
ETAKAHASHI /Volumes/LaCie 8 Mac HFS Yes 476,939.6 52.4250
FHAUSER09 /Volumes/My Passport 2 Mac HFS Yes 238,414.4 1.2
FHAUSER09 /Volumes/CHARLES SCH 3 Mac HFS Yes 245.2 74.8
GWHEELER-MAC /Volumes/Google Chrome 4 Mac HFS Yes 90.5 100.0
GWHEELER-MAC /Volumes/TurboTax2010 5 Mac HFS Yes 185.3 100.0
GWHEELER-MAC /Volumes/TurboTax Premier 2010 6 Mac HFS Yes 650.0 35.3
HWHEELER09 /Volumes/LaCie 2 Mac HFS Yes 152,299.8 66.3101
HWHEELER09 /Volumes/HandBrake-0.9.5-MacOSX.5_GUI_x86_64 3 Mac HFS Yes 30.5 62.9
IRUSSU-2012 /Volumes/Backup Drive 2 Mac HFS Yes 476,812.0 6.029
JBEST-2011 /Volumes/TRAVELDRIVE 2 Mac HFS Yes 978.8 95.6
JBEST-2011 /Volumes/jbest-Backup Drive 3 Mac HFS Yes 476,811.1 8.139
JBIDDISCOMBE-MAC /Volumes/Macintosh HD 1 2 Mac HFS Yes 76,191.0 74.957
JFIELDS-MAC /Volumes/Macintosh HD 2 2 Mac HFS Yes 953,541.7 25.9247
JFIELDS-MAC /Volumes/Time Machine Backups 3 Mac HFSX Yes 953,541.7 5.956
JNAEGELE-MACBOOK /Volumes/NaegeleLab3 2 Mac HFS Yes 304,917.3 24.875
JRKIRN-2011 /Volumes/Macup 1 Mac HFS Yes 238,135.0 69.8166
JVAREKAMP-2010 /Volumes/MINI HDD 3 Mac HFS Yes 3,890.9 88.6
JWAREHAM-11 /Volumes/LaCie 2 Mac HFS Yes 953,741.7 22.9218
JWAREHAM-11 /Volumes/Kanguru 3 Mac HFS Yes 476,937.5 19.191
MCALTER-2012 /Volumes/Macup 1 Mac HFS Yes 238,135.0 70.0167
MGILMORE-2008 /Volumes/marthagilmore 29 Mac HFSX Yes 10,268.0 0.6
MGILMORE-2008 /Volumes/mgilmoredesk2 33 Mac HFS Yes 715,404.8 7.654
MKLANESKI-MAC /Volumes/Projects 1 Mac HFS Yes 953,741.7 21.5205
MSINGER-2010 /Volumes/Nola triquetrana 2 Mac HFS Yes 953,867.2 0.0
MWEIR-2008 /Volumes/igs-share-1 3 Mac HFS Yes 5,628,857.4 5.1287
MWEIR-2008 /Volumes/Python 2.7.1 4 Mac HFS Yes 33.4 66.9
MWEIR-LABG5 /Volumes/mweir-lab2 2 Mac HFS Yes 114,344.6 59.267
PLANETARY3 /Volumes/BOOTCAMP 2 Mac HFS Yes 466,244.0 80.8377
PRESOR-2008 /Volumes/ONTRACK 4 Mac Unknown Yes 152,625.3 87.9134
PRESOR-2008 /Volumes/My Book 5 Mac HFS Yes 476,821.0 50.9242
PRESOR-MACMAC /Volumes/My Passport 4 Mac HFS Yes 305,168.3 54.7167
PRESOR-MACMAC /Volumes/NO NAME 5 Mac HFS Yes 32,308.2 24.2
PRESOR-MACMAC /Volumes/TRAVEL 6 Mac HFS Yes 476,153.7 66.7317
PRESOR-MACMAC /Volumes/JULIA 7 Mac HFS Yes 476,823.6 5.0
PTYNANMAC /Volumes/Macintosh HD 1 2 Mac HFS Yes 190,654.2 70.1134
RBRAULT-DESKTOP /Volumes/LaCie 9 Mac HFS Yes 953,869.3 2.0
RBRAULT-DESKTOP /Volumes/My Book 11 Mac FS Yes 476,821.0 37.1177
RBRAULT-DESKTOP /Volumes/CHEMdata 12 Mac HFS Yes 2,097,149.2 96.82030
SDEVOTO-2008 /Volumes/EXCHANGE 3 Mac HFS Yes 31,984.3 1.6
SDEVOTO-2008 /Volumes/LaCie 4 Mac HFS Yes 273,245.3 19.453
SDEVOTO-2008 /Volumes/Orange Roughie 10 Mac HFS Yes 953,741.7 27.2259
SGUHA-12 /Volumes/Storage Drive 1 Mac HFS Yes 715,076.8 3.2
WBURKHART-MAC /Volumes/Little Max 2 Mac HFS Yes 238,347.1 34.268

Growth

One measure of growth is the calculation between scratch tapes/volumes wanted and reclaimed.

Based on a moving average, in this case the last 7 days so weekends are included, total bytes migrated from the BACKUPPOOL to tapes/volumes can be summed for all those days. Then an average daily rate can be calculated to smooth over hills and valleys. Next we look at how full are tapes/volumes that can not receive any more data (tapes are of size 50 GB).

This leads to a calculation how many tapes/volumes TSM wants to store the “migrated” volume out of the BACKUPPOOL to to tapes/volume based on “full” tape usage efficiency. That can then be followed up by searching for the rate of tapes/volume reclamation.

The difference is the rate of growth.

Weekly Moving Average ending week of Sept 5
Days Total MB Daily MB Avg MB Scratch Difference
“migrated” “migrated” per full Tape Wanted/Week Reclaimed/Week per Week
7 6183.35 883.34 46.12 134.07 115.01 +19.06
7 6826.59 975.23 46.12 148.02 121.03 +26.99
7 5860.81 837.26 46.12 127.08 128.03 -0.95
7 5535.35 790.76 46.12 120.02 94.01 +26.01
7 7636.51 1090.93 46.12 165.58 98.00 +67.58
7 8928.74 1275.53 46.13 193.56 101.01 +92.55
7 9780.65 1397.24 46.13 212.02 102.97 +109.05
7 9688.42 1384.06 46.13 210.02 100.00 +110.02
7 9790.58 1398.65 46.13 212.24 78.00 +134.24
7 10726.68 1532.38 46.13 232.53 121.00 +111.53
7 10371.81 1481.69 46.14 224.79 125.00 +99.79
7 10617.39 1516.77 46.14 230.11 132.00 +98.11
7 10211.63 1458.80 46.14 221.32 132.00 +89.32
7 9395.18 1342.17 46.14 203.62 132.00 +71.62
7 10539.49 1505.64 46.14 228.42 137.00 +91.42
7 9429.37 1347.05 46.14 204.36 154.00 +50.36
7 10090.96 1441.57 46.14 218.70 188.00 +30.70

We need to get the grow rate down by

BACKUPPOOL

Incremental backup volume, as well as first backups volumes of course, arrive at the BACKUPPOOL before being packaged up and migrated to tapes/volumes. So one interesting view is which nodes backup large amounts of volumes up repeatedly under incremental backup? And if so what?

For a running log comparing day to day behavior look at blackeagle:/tmp/tsm_backup_yesterday_report.log. For example:

Report: Tue Sep 6 14:39:30 EDT 2011
top25 (MB): 09/05 Repeaters
JNAEGELE-LAB2 77477.80
MKLANESKI-MAC 29921.20
JBIDDISCOMBE-MAC 12490.24
PTURENNE2007 11084.86
DMEYER-11 10686.49
SCROUCHER-08 7655.16
LSCOTT-09 6678.20
LWEINBERG 6565.15
JRABAMAC 6268.96 *
SGRUCAN-10 6145.57 *
RCULLITON11 6145.19 *
FCOHAN-LABHP 5073.08
SBACHNER-11 4518.37
SMARTIN-11 4473.26 *
AGONZALEZ-LAPTOP 4425.20
SSARVER-10 3883.46 *
AFRY-2010 3881.87
MSINGER-2008 3277.08
HEPSTEIN-10 3276.59
KVECCHITO-VM 2921.09
JKNEE-TEMP 2747.52 *
LGRABEL-TEMPMAC 2415.17
JTHOM-10 2342.72
MGLYNN-09 2287.40 *
CMAINES-10 2276.51
Summary: total volume last 24 hours 300077.36 MB
of which top25 total volume was 228918.14 MB or 76.29%
Report: Wed Sep 7 09:11:46 EDT 2011
top25 (MB):
MWHALEN-MACBOOK 49955.10
VMARINELLI-2008 44645.09
SLASTRINA-2007 42506.51
MKLANESKI-MAC 29921.20
FWILLIAMS-10 29700.91
BSTEWART-2008 21676.59
JGREENWOOD-IMAC 18583.40
BNORTHROP-2009 16427.23
JWESTMACBOOK 13884.93
CLEIDY-07 13045.73
RSCHMIDT-10 12356.27
DALEXSON-2008 10189.79
IMUKERJI-2009 8803.82
DHUTSON-11 8131.34
EAS-MACMINI 8038.98
VSMOLKIN-11 8015.53
PTURENNE2007 7528.64
BHOOD-10 7528.56
LMIMS-07 7262.16
LSBROWN-08 7120.80
VHARRIS-2008 6156.25
RCULLITON11 6109.16
PTULLY-10 5956.06
DTREISTER-11 5179.13
ABERGEN-10 5128.32
Summary: total volume last 24 hours 665830.81 MB
of which top25 total volume was 393851.5 MB or 59.15%

BACKUPPOOL 2

So the question now is the same content repeatedly backed up? Probably easier to determine from client side logs. Looking at content arriving at the BACKUPPOOL the following will be added to the server side exclude filters … this does not tell us per node if same content arrives day after day but gives us a clue if there is junk we can avoid backing up.

tsm> q con /blackeagle/stgeql/stg?.dsm

PC (done 06sept/2011)
*:\DOCUMENTS AND SETTINGS\…\COOKIES
*:\DOCUMENTS AND SETTINGS\…\LOCAL SETTINGS\APPLICATION DATA\MOZILLA\FIREFOX\PROFILES\…\CACHE
*:\DOCUMENTS AND SETTINGS\…\LOCAL SETTINGS\APPLICATION DATA\XYTHOS\CACHE
*:\DOCUMENTS AND SETTINGS\…\LOCAL SETTINGS\APPLICATION DATA\GOOGLE\CHROME\USER DATA\DEFAULT\CACHE
*:\DOCUMENTS AND SETTINGS\…\LOCAL SETTINGS\APPLICATION DATA\MICROSOFT\TOOLBAR\ISOLATEDSTORAGE
*:\USERS\…\APPDATA\LOCAL\TEMP
*:\USERS\…\APPDATA\LOCAL\MOZILLA\FIREFOX\PROFILES\…\CACHE
*:\USERS\…\APPDATA\LOCAL\GOOGLE\CHROME\USER DATA\DEFAULT\CACHE
*:\USERS\…\APPDATA\LOCAL\XYTHOS\CACHE
*:\USERS\…\APPDATA\LOCALLOW\MICROSOFT\CRYPTNETURLCACHE
*:\USERS\…\APPDATA\LOCALLOW\SUN\JAVA\DEPLOYMENT\CACHE
*:\USERS\…\APPDATA\ROAMING\ADOBE\FLASH PLAYER\ASSETCACHE
*:\PS\CACHE
*:\PROGRAMDATA\ESET
*:\TIMESPROSQL\TPROSERV\LOGS
*:\CYGWYN\…\.CPAN
*:\CYGWIN\…\USR\SHARE

exclude

Mac (done 06sept/2011)
/.MOBILEBACKUPS
/USERS/…/LIBRARY/APPLICATION SUPPORT/MOBILESYNC
/USERS/…/LIBRARY/APPLICATION SUPPORT/XYTHOS/DRIVE/CACHE
/USERS/…/.TRASH
/USERS/…/LIBRARY/CACHES
/USERS/…/LIBRARY/PUBSUB/FEEDS
Mac (done 16sept/2011)
/Users/*/Documents/Microsoft User Data/Main Identity*/Data Records/Message Sources/*
/Users/*/Documents/Microsoft User Data/Main Identity*/Data Records/Messages/*
/Users/*/Documents/Microsoft User Data/Main Identity*/Data Records/Message Attachments/*
/Users/*/Documents/Microsoft User Data/Main Identity*/Database*
/Users/*/Library/Mail/…/*@*wesleyan.edu*
/USERS/…/DOWNLOAD

Filters

Since there is so much junk arriving at the BACKUPPOOL i have implemented new filters for windows and macintoshes (Sept 09)

The filter definitions can be found https://wiki.wesleyan.edu/display/itstss/TSM+recommended+excludes

IBM's Recommended section and the Optional Trash section have been both implemented.

CONTENT

It would be nice to be able to query the content of nodes of large size or with huge numbers of files. The first step is displaying their filespaces looking for external drives. The second step is to get an inventory of all files and their file sizes for a particular node.

However the table CONTENTS is so large that any SQL attempts via a script fail to return data (select file_name, file_size from contents where node_name='this_node_name')

Policy

Both approaches below are too taxing on TSM and the SQL generate no output in desired time window. We could consider:

What is the file space used by inactive files by node?

What is the file space used by deleted files by node?