DokuWiki

User Tools

Site Tools

Trace:
tsm:blackeagle:06sep11

Warning: Undefined array key -1 in /usr/share/dokuwiki/inc/html.php on line 1458

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
tsm:blackeagle:06sep11 [2011/09/09 15:30]
hmeij [Growth] 		tsm:blackeagle:06sep11 [2011/09/22 13:58] (current)
hmeij [Filespace]
Line 82: Line 82:
  
 It appears most servers are simply stuffed into the first schedule listed.  Since the first window is for laptops we need to move nodes out of that slot into others, surely not all of these are laptops.  1,048 nodes so the ideal number is to have about 200 per schedule window. It appears most servers are simply stuffed into the first schedule listed.  Since the first window is for laptops we need to move nodes out of that slot into others, surely not all of these are laptops.  1,048 nodes so the ideal number is to have about 200 per schedule window.
 +
 +
 +UPDATE 09/20/2011:
 +  * SCHED1216 has been extended to run till 18:00 hours (did not change the schedule name ...)
 +  * SCHED0912 has been created
 +  * Database backup and maintenance window is now 6am - 9am (reclamation might happen but migration is stopped)
  
 ^  SCHED1216 node count = 550  ^ ^  SCHED1216 node count = 550  ^
Line 197: Line 203:
  
  
-  * Can we find these drives and exclude them?+  * Can we find these drives and exclude them? (ie, include /Volume/Macintosh HD, exclude rest)
   * Exclude external drives /Volumes/MyPassport? (western digital)   * Exclude external drives /Volumes/MyPassport? (western digital)
   * Exclude /Volumes/Iomega HDD?   * Exclude /Volumes/Iomega HDD?
-  * Can we block backup server side if FSID>X?+ 
 +ANSWER 
 +http://www.oucs.ox.ac.uk/hfs/help/excluding.xml.ID=excludedrives 
 + 
 +explains nicely what Jim was talking about excluding drives via GUI and also points out you can do it via client dsm.opt settings.  but it also shows me how we can do this server side wise, so i tried with bstewart-2008 after making sure /Volumes/WD1.5TB1 and /Volumes/WD1.5TB-2 were unique, added excludes for them and boom they were gone, node back to 350GB. 
 + 
 +so if we adopt a convention to force / to mount at some unique mountpoint, which it already does, we could do, evaluate from bottom up 
 + 
 +exclude /Volumes/.../*\\ 
 +include /Volumes/SomeApprovedVolume/.../*\\ 
 +include.dir /Volumes/Macintosh HD\\ 
 + 
 +In the meantime i can set up global excludes for some of our problem cases whose mountpoint are unique and if they do not need to be backed up.  
 + 
 +===== Mac Drives ===== 
 + 
 +report of 09/19/2011: roughly 45 drives with a total utilized capacity of 7.7 TB 
 + 
 +  * last column will help you find the hogs: GB util = (capacity MB/1024) * (pct util/100) 
 +  * RBRAULT-DESKTOP needs to be dealt with 
 +  * MWEIR-2008 has a, gasp, 5 TB array on it 
 + 
 +It may be that some of these drives are not being backed up, as they may have been excluded before, but were not delete from TSM. 
 + 
 +^node_name  ^volume  ^fsid  ^platform^ fs type  ^ unicode  ^ capacity MB  ^ pct util  ^ GB util^ 
 +|AMACQUEEN-DESKTOP       |/Volumes/amacqueen-extra-drive  |3       |Mac     |HFS     |Yes     |610,152.3       |73.9|451| 
 +|BSTEWART-2008   |/Volumes/LACIE  |8       |Mac     |HFS     |Yes     |152,625.3       |89.1|136| 
 +|BSTEWART-MACBOOKPRO     |/Volumes/RAID02 |2      |Mac     |HFS    | Yes    | 956,699.4     |  39.6|379| 
 +|DMATTHEWS-09    |/Volumes/LaCie  |2      | Mac    | HFS     |Yes     |953,869.3       |30.5|291| 
 +|EMEREDITH09     |/Volumes/My Passport    |3       |Mac     |HFS     |Yes     |476,812.0       |52.0|248| 
 +|ETAKAHASHI      |/Volumes/LaCie  |8       |Mac     |HFS     |Yes     |476,939.6       |52.4|250| 
 +|FHAUSER09       |/Volumes/My Passport    |2       |Mac     |HFS     |Yes     |238,414.4       |1.2| | 
 +|FHAUSER09       |/Volumes/CHARLES SCH    |3       |Mac     |HFS     |Yes     |245.2   |74.8| | 
 +|GWHEELER-MAC    |/Volumes/Google Chrome  |4       |Mac     |HFS     |Yes     |90.5    |100.0| | 
 +|GWHEELER-MAC    |/Volumes/TurboTax2010   |5       |Mac     |HFS     |Yes     |185.3   |100.0| | 
 +|GWHEELER-MAC    |/Volumes/TurboTax Premier 2010  |6       |Mac     |HFS     |Yes     |650.0   |35.3| | 
 +|HWHEELER09      |/Volumes/LaCie  |2       |Mac     |HFS     |Yes     |152,299.8       |66.3|101| 
 +|HWHEELER09      |/Volumes/HandBrake-0.9.5-MacOSX.5_GUI_x86_64    |3       |Mac     |HFS     |Yes     |30.5    |62.9| | 
 +|IRUSSU-2012     |/Volumes/Backup Drive   |2       |Mac    |HFS     |Yes     |476,812.0       |6.0|29| 
 +|JBEST-2011      |/Volumes/TRAVELDRIVE    |2       |Mac    |HFS     |Yes     |978.8   |95.6| | 
 +|JBEST-2011      |/Volumes/jbest-Backup Drive     |3       |Mac     |HFS     |Yes     |476,811.1      | 8.1|39| 
 +|JBIDDISCOMBE-MAC        |/Volumes/Macintosh HD 1 |2       |Mac     |HFS     |Yes     |76,191.0        |74.9|57| 
 +|JFIELDS-MAC     |/Volumes/Macintosh HD 2 |2       |Mac    | HFS     |Yes     |953,541.7       |25.9|247| 
 +|JFIELDS-MAC     |/Volumes/Time Machine Backups   |3       |Mac     |HFSX    |Yes     |953,541.7      | 5.9|56| 
 +|JNAEGELE-MACBOOK        |/Volumes/NaegeleLab3    |2       |Mac     |HFS     |Yes     |304,917.3       |24.8|75| 
 +|JRKIRN-2011     |/Volumes/Macup  |1       |Mac     |HFS     |Yes     |238,135.0       |69.8|166| 
 +|JVAREKAMP-2010  |/Volumes/MINI HDD       |3       |Mac     |HFS     |Yes     |3,890.9 |88.6| | 
 +|JWAREHAM-11     |/Volumes/LaCie  |2       |Mac     |HFS     |Yes     |953,741.7       |22.9|218| 
 +|JWAREHAM-11     |/Volumes/Kanguru        |3       |Mac     |HFS     |Yes     |476,937.5       |19.1|91| 
 +|MCALTER-2012    |/Volumes/Macup  |1       |Mac     |HFS    |Yes     |238,135.0       |70.0|167| 
 +|MGILMORE-2008   |/Volumes/marthagilmore  |29      |Mac     |HFSX    |Yes     |10,268.0        |0.6| | 
 +|MGILMORE-2008   |/Volumes/mgilmoredesk2  |33      |Mac     |HFS     |Yes     |715,404.8      | 7.6|54| 
 +|MKLANESKI-MAC   |/Volumes/Projects       |1       |Mac     |HFS     |Yes     |953,741.7       |21.5|205| 
 +|MSINGER-2010    |/Volumes/Nola triquetrana       |2       |Mac     |HFS     |Yes     |953,867.2       |0.0| | 
 +|MWEIR-2008      |/Volumes/igs-share-1    |3       |Mac     |HFS     |Yes     |5,628,857.4    | 5.1|287| 
 +|MWEIR-2008      |/Volumes/Python 2.7.1   |4       |Mac     |HFS     |Yes     |33.4    |66.9| | 
 +|MWEIR-LABG5     |/Volumes/mweir-lab2     |2       |Mac     |HFS     |Yes    |114,344.6       |59.2|67| 
 +|PLANETARY3      |/Volumes/BOOTCAMP       |2       |Mac     |HFS     |Yes     |466,244.0      | 80.8|377| 
 +|PRESOR-2008     |/Volumes/ONTRACK        |4       |Mac    |Unknown |Yes     |152,625.3      | 87.9|134| 
 +|PRESOR-2008     |/Volumes/My Book        |5       |Mac    |HFS     |Yes     |476,821.0       |50.9|242| 
 +|PRESOR-MACMAC   |/Volumes/My Passport    |4       |Mac     |HFS     |Yes     |305,168.3       |54.7|167| 
 +|PRESOR-MACMAC   |/Volumes/NO NAME        |5       |Mac     |HFS     |Yes     |32,308.2        |24.2| | 
 +|PRESOR-MACMAC   |/Volumes/TRAVEL |6       |Mac     |HFS     |Yes    | 476,153.7      | 66.7|317| 
 +|PRESOR-MACMAC   |/Volumes/JULIA  |7       |Mac     |HFS     |Yes     |476,823.6       |5.0| | 
 +|PTYNANMAC       |/Volumes/Macintosh HD 1 |2       |Mac     |HFS     |Yes     |190,654.2       |70.1|134| 
 +|RBRAULT-DESKTOP |/Volumes/LaCie  |9       |Mac     |HFS     |Yes     |953,869.3       |2.0| | 
 +|RBRAULT-DESKTOP |/Volumes/My Book        |11      |Mac     |FS     |Yes    |476,821.0       |37.1|177| 
 +|RBRAULT-DESKTOP |/Volumes/CHEMdata       |12      |Mac     |HFS     |Yes     |2,097,149.2     |96.8|2030| 
 +|SDEVOTO-2008    |/Volumes/EXCHANGE       |3       |Mac     |HFS     |Yes     |31,984.3        |1.6| | 
 +|SDEVOTO-2008    |/Volumes/LaCie  |4       |Mac     |HFS    | Yes     |273,245.3       |19.4|53| 
 +|SDEVOTO-2008    |/Volumes/Orange Roughie |10      |Mac    | HFS    | Yes     |953,741.7       |27.2|259| 
 +|SGUHA-12        |/Volumes/Storage Drive  |1       |Mac     |HFS    | Yes     |715,076.8       |3.2| | 
 +|WBURKHART-MAC   |/Volumes/Little Max     |2       |Mac    | HFS    | Yes     |238,347.1       |34.2|68|
  
  
Line 226: Line 304:
 |7       |8928.74   |1275.53 |46.13   |193.56  |101.01  |+92.55   | |7       |8928.74   |1275.53 |46.13   |193.56  |101.01  |+92.55   |
 |7       |9780.65   |1397.24 |46.13   |212.02  |102.97  |+109.05   | |7       |9780.65   |1397.24 |46.13   |212.02  |102.97  |+109.05   |
 +|7       |9688.42   |1384.06 |46.13   |210.02  |100.00  |+110.02   |
 +|7       |9790.58   |1398.65 |46.13   |212.24  |78.00   |+134.24   |
 +|7       |10726.68  |1532.38 |46.13   |232.53  |121.00  |+111.53   |
 +|7       |10371.81  |1481.69 |46.14   |224.79  |125.00  |+99.79   |
 +|7       |10617.39  |1516.77 |46.14   |230.11  |132.00  |+98.11   |
 +|7       |10211.63  |1458.80 |46.14   |221.32  |132.00  |+89.32   |
 +|7       |9395.18   |1342.17 |46.14   |203.62  |132.00  |+71.62   |
 +|7       |10539.49  |1505.64 |46.14   |228.42  |137.00  |+91.42   |
 +|7       |9429.37   |1347.05 |46.14   |204.36  |154.00  |+50.36   |
 +|7       |10090.96  |1441.57 |46.14   |218.70  |188.00  |+30.70   |
 +
 +
 +We need to get the grow rate down by
 +
 +  * reducing volume of content backed up (large size nodes, large files nodes)
 +  * reducing junk being backed up by applying exclude filters
 +  * eliminate external drives form being backed up
 +  * ???
 +  * 
  
  
Line 294: Line 391:
 ^Summary: total volume last 24 hours 665830.81 MB ^^ ^Summary: total volume last 24 hours 665830.81 MB ^^
 ^         of which top25 total volume was 393851.5 MB or 59.15% ^^ ^         of which top25 total volume was 393851.5 MB or 59.15% ^^
 +
 +  * Analyze these top nodes, look at client logs for last few days, come up with filters.
 +
  
 ===== BACKUPPOOL 2 ===== ===== BACKUPPOOL 2 =====
Line 319: Line 419:
 |*:\TIMESPROSQL\TPROSERV\LOGS| |*:\TIMESPROSQL\TPROSERV\LOGS|
 |*:\CYGWYN\...\.CPAN| |*:\CYGWYN\...\.CPAN|
 +|*:\CYGWIN\...\USR\SHARE|
 +exclude
 +
 +
  
  
Line 328: Line 432:
 |/USERS/.../LIBRARY/CACHES| |/USERS/.../LIBRARY/CACHES|
 |/USERS/.../LIBRARY/PUBSUB/FEEDS| |/USERS/.../LIBRARY/PUBSUB/FEEDS|
 +^Mac (done 16sept/2011)^ 
 +|/Users/*/Documents/Microsoft User Data/Main Identity*/Data Records/Message Sources/*|
 +|/Users/*/Documents/Microsoft User Data/Main Identity*/Data Records/Messages/*|
 +|/Users/*/Documents/Microsoft User Data/Main Identity*/Data Records/Message Attachments/*|
 +|/Users/*/Documents/Microsoft User Data/Main Identity*/Database*|
 +|/Users/*/Library/Mail/.../*@*wesleyan.edu*|
 +|/USERS/.../DOWNLOAD|
 +
 +  * Filters have been applied.  Wait for 60 days to take effect, sigh.
 +
  
    
 ===== Filters ===== ===== Filters =====
  
-Since there is so much junk arriving at the BACKUPPOOL i have implemented the "recommended" filters for windows and macintoshes.  +Since there is so much junk arriving at the BACKUPPOOL i have implemented new filters for windows and macintoshes (Sept 09)  
  
 The filter definitions can be found https://wiki.wesleyan.edu/display/itstss/TSM+recommended+excludes The filter definitions can be found https://wiki.wesleyan.edu/display/itstss/TSM+recommended+excludes
  
-IBM's recommendation section and the Optional Trash section have been implemented.+IBM's Recommended section and the Optional Trash section have been both implemented.
  
-  * We should consider sticking the rest in.+  * We should consider sticking the rest in, and wait 60 days, sigh. APPLIED 9/19/2011
  
  
Line 360: Line 474:
  
 Both approaches below are too taxing on TSM and the SQL generate no output in desired time window. We could consider: Both approaches below are too taxing on TSM and the SQL generate no output in desired time window. We could consider:
 +
   * changing inactive version retention to 60 days   * changing inactive version retention to 60 days
   * lower deleted file retention to 30 days   * lower deleted file retention to 30 days
tsm/blackeagle/06sep11.1315596611.txt.gz · Last modified: 2011/09/09 15:30 by hmeij

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
CC Attribution-Share Alike 4.0 International Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki