DokuWiki

Keeping it all together on a single page for reference.

• 1) review client option set, 1st email on this was sent in September;
• 2) setup local disk pool (however not that urgent as the initial clients can go directly to VTL);
• 3) setup schedules for clients, including morning schedules as on blackhawk;
• 4) adding SAN LUN space to Linux host for db volumes (not urgent as db can stay on local disk to start with).

For creating nodes on new TSM, we can expand or clone the eP app so that new nodes can be created on TSM instance via eP app. We need to re-visit TSM node password node change policy. I recall you recommend setting password for 3 days (current 14 days on blackhawk), I am unsure whether should make this change.

Important to point out that we're using a dedicated library and only port 0a for host access.

All jobs crashed roughly after 17 hrs and 20 mins when database exceeded it's allocation. That yields a throughput of (17*60)+20=1040 mins or (2,919.8*1024)/(1040*60) = 48 MB/s or 2.8 GB/min or 4.1 TB/day.

We observed total drive throughput in the order of 100 Mb/s so this deteriorated quite a bit. According to the VTL statistics page a total of 3131.51 GB was written taking up 2580.67 GB of space yielding a compression ratio of 1.2

That is using one host and one fiber card. Throughput could be doubbled.

But if we take TSM total bytes transfer the compression ratio becomes 2,919/2,581 or 1.13

From Monitoring Page

From Virtual Lib Page | Statistics Page

Note: condor_home picked up the .snapshot dirs so bombed, /tmp for logging filled.

• server
  • we will set up a Dell Power Edge 1955 blade (2.6 gzh, dual-quad core xeon 5355 with 16 gb RAM footprint and 2×36 gb 15K rpm disks)
  • install redhat AS v4 Linux, x86 32-bit
  • install Linux based TSM server, perhaps v5.5 for potential vmware backup
  • connect linux host to vtl device via fiber using the dell enclosure

• note: doing it this way does imply that the IO of TSM traffic will compete with the other blades in the enclosure.
  • if this turns into a problem we will have to grab a desktop pc
  • Matt Stoller appeared ok with this for testing, but production should be a standalone server (PE 2950?)

• tsm clients
  • we will be creating a separate dsm.opt of existing desktop clients and in addition to regular backup schedule point the clients to the VTL for backup
  • we will initially stagger the first backup events to avoid overloading parts of the network
  • the goal is to request ITS employees to join in testing
  • target is 100 clients in a representative distribution: 45% windows, 45% macs, 10% linux

1. – Zhu, Hong
   • we currently already have 50Tb data, when VTL in service, the virtual tapes will not be 100% utilized, so it is unrealistic to assume max storage capacity.
   • disappointed with disabling client compression as this will put more data to our 10/100 network, so client backups might take longer to complete if network bandwidth is saturated.

• total backup volume of first backup by clients (compare windows vs macs vs linux)
• deduplication rate over time as new clients perform a first backup

• total incremental backup volume by clients (compare windows vs macs vs linux)
• deduplication rate over time

• measure any throughput? probably not relevant

• estimate total volume of backup of production system (how to find the median value?)

note … the average value of the desktops is heavily skewed as there is at least one very large clients (as in one faculty cluster)

Total (uncompressed) Volume: 27tb + 13tb + 23tb = 63 tb

So, our desktops would at 2x use up 13.5/18 or 75% of the available disk space on the dxi barriing even thinking about the incremental volume.

SELECT NODE_NAME, FILESPACE_NAME, FILESPACE_TYPE, CAPACITY AS "File System Size in MB", PCT_UTIL, 
DECIMAL((CAPACITY * (PCT_UTIL / 100.0)), 10, 2) AS "MB of Active Files" 
FROM FILESPACES ORDER BY NODE_NAME, FILESPACE_NAME

• apply deduplication ratios and estimate total needed addressable space

Assessing TSM blackhawk makeup. If we left the large servers listed on the right behind on blackhawk (for now), the VTL front end host could have 2 instances of about 15 TB, one for macs, the other for windows clients. Roughly balanced, except for the database object counts which seems heavily skewed towards macs.

To fill out the site survey, a few bits of information:

• Where will the DXi physically be located?
  • Wesleyan University, Information Technology Services, Excley Science Center (Room 516, Data Center), 265 Church Street, Middletown, Connecticut 06459

• Who will be the primary contact (name/phone/email)?
  • Administrative: Henk Meij, 860.685.3783, hmeij@wes
  • Technical: Hong Zhu, 860.685.2542, hzhu@wes

• Are you directly connecting the fibre from the blade center to the DXI?
  • Yes.

• How many virtual tape drives you plan to use in the DXI (up to 64)?
  • If tape drive doesn't consume capacity, we would like to start with 50. What is Quantum's recommendation for production? As we like to backup clients directly to DXi during test period, i.e., no need to go to disk pool first. However, when in production, we would like clients go to diskpool then migrate to VTL as our nodes are beyond 64.

• Do you have a preferred virtual cartridge size in the DXI? (smaller cartridges tend to improve the granularity of restore - if you create 1TB cartridges, they are likely to be involved in a backup when you get a restore request, and you'll have to wait until the backup is done to get that cartridge; smaller cartridges limit this issue … for example configure 40GB cartridges).
  • 40Gb

• For the test, do you plan to mount this in a data center rack cabinet?
  • Yes. If installed in top of rack, we have enough persons to lift it in place.

• What threshold do you use for tape reclamation?
  • 55% on physical L3584, any recommendation for VTL tapes?

During the last 3 months we experienced 300GB to 1TB of incremental data backup with average at ~850Gb… (most of them are compressed).

Also, here goes our backup policy:

Desktop clients:

• Files currently on client file system: 2
• Files deleted from client file system: 1
• Number of days to retain inactive version: No limit
• Number of days to retain the deleted version: 60

Server clients, mainly two: 1)

• Files currently on client file system: 2
• Files deleted from client file system: 1
• Number of days to retain inactive version: 30
• Number of days to retain the deleted version: 60

2)

• Files currently on client file system: 3
• Files deleted from client file system: 2
• Number of days to retain inactive version: No limit
• Number of days to retain the deleted version: 121

• evaluate compression and deduplication ratios of the unit given 2 scenarios: tsm clients send data to be backed up compressed and not compressed. the hope is to achieve 2x for compression and 10x for deduplication, yielding a 20x factor for the useable space using client compression or not.

• evaluate throughput using inline deduplication and asses if our backup schedules would/could shorten; perhaps assess post processing throughput since that is an available feature

• hopefully, 100 client desktops will participate (45% windows, 45% macs, 10% unix) yielding data that can be scaled out to assess if our target of 10TB usable space (200TB with a 20x factor) gives us room to grow with this unit for an anticipated time period of 3-6 years (this is less important now with the dxi7500)

• unit should be able to replicate to another unit (one to one or selectively) in a future DR site to be build

Quantum reports:
-------------------
Capacity: 17.81TB
Free: 15.81TB

Used: 2.00TB/11.21%
Date reduced by: 43.56%



TSM reports:
--------------
tsm: VTLTSM>q occ

Node Name      Type     Filespace       FSID     Storage        Number of      Physical       Logical
                        Name                     Pool Name          Files         Space         Space
                                                                               Occupied      Occupied
                                                                                   (MB)          (MB)
----------     ----     ----------     -----     ----------     ---------     ---------     ---------
EXCHANGEW-     Bkup     \\monet\g$         1     VTLPOOL        1,336,817     1,056,386     1,056,386
 ES5                                                                                .79           .79

This was not so useful other than the throughput statistics. 1 TB of files were send over. Q7500 reports 2TB is used (presumably 1 TB of staged data, and 1 TB for block pool). Data was send uncompressed. Unfortunately the before/after reduction info was not grabbed from the screen. Recall something like 1.16x. That's compess + dedup !?

Is the above overhead? We did recycle the cartridges from previous data dump and initiated a space reclamation. Then recreated cartridges.

Not too realistic … we might be running close to 50/50 windows vs macs in the field.

The q7500 received 885GB of data and deduped/compressed that down to 426GB. Hmm…that looks suspiciously like compression only.

Next we re-submitted the desktops above. Initially the nodes were called username-vtl and now we renamed them username-vtl2 and inititated a full backup. So if the numbers hold, and very changes occur on these desktops within this week, then we expect:

We did not manage to send all nodes again (2 linux, 17 windows, 3 mac … 1 large mac desktop missing). The block pool grew by roughly 10% (new unique blocks) while resending 88% of the original data. So that means the dedup is working. The key here is that this may represent nightly incremental backups (as in backing up files that changed and have already been processed by the VTL at the block level). So 1640-885= 775 GB flowed to VTL in 2nd backup which added 479-426= 53.56 GB to the pattern pool which generates a factor of 1:14 … clearly working.

The baseline of 2x across this mix of desktops is surprising.

Side bar: if this were compression only, then the 1.7 GB volume would yield an “After” value of 850MB assuming a 50% (2x) reduction which is close to what TSM achieves.

#7 Observed when re-submitting same nodes (under different name) in uncompressed data stream mode

So sending compressed client data over is a nono. When sending the same client data uncompressed we a compress/dedup positive effect 1.38x over 1x. Backup time is about the same, so the only caveat is sending too much uncompressed data over the network for first backups. Incremental backup volume should be ok.

200 cartidges of LTO-3 type with custom size of 50 GB.

In this effort we're trying to export existig nodes from “blackhawk” to out “vtltsm” servers. The idea being that when a node is exported and imported we have both the original data and the inactive/deleted data for a node. That would represent the real world.

We still do not know the rough volume of original data versus inactive/deleted data volume.

• Source = size of node file space on TSM server “blackhawk”
• Cum. Source = sum of source, total size of the source nodes
• Target = estimated by taking previous before reduction and current before reduction
• Cum. Target = sum of target, total size of target nodes (before reduction)

Aborted … the export/import fails between the TSM instances … since hong is out, i'm pretty unable to fix this.

200 cartridges of LTO-3 type with custom size of 50 GB.

In order to get a clearer picture of the results of compression *and* deduplication of *first backups*, we're tackling some filesystems with different content. We'll back these serially and keep track of the progress as we compile the final statistic.

The filesystems: (<hi #ffff00>Note: dragon's home and data are backed up via node miscnode, cluster is backed up via node miscnode2 … these 3 were initiated simultaneously on sunday 8/17 AM</hi>) … i submitted miscnode twice and send up data first, edited dsm.sys, and send up home. that appears to not trigger the use of another drive (becuase of same node??). created another node and send up cluster. now we do see 2 drives being used.

tsm: VTLTSM>q sess

  Sess Comm.  Sess     Wait   Bytes   Bytes Sess  Platform Client Name         
Number Method State    Time    Sent   Recvd Type           
------ ------ ------ ------ ------- ------- ----- -------- --------------------
    59 Tcp/Ip IdleW  7.0 M      960     677 Node  Linux86  MISCNODE            
    60 Tcp/Ip RecvW    0 S    4.4 K   9.1 G Node  Linux86  MISCNODE            
    61 Tcp/Ip IdleW   15 S  183.5 K     829 Node  Linux86  MISCNODE            
    62 Tcp/Ip RecvW    0 S   41.0 K   8.6 G Node  Linux86  MISCNODE            
    69 Tcp/Ip IdleW   42 S      898     674 Node  Linux86  MISCNODE2           
    70 Tcp/Ip Run      0 S    1.2 K  38.9 G Node  Linux86  MISCNODE2           
    72 Tcp/Ip Run      0 S      126     163 Admin Linux86  ADMIN  

Vertical axis (Data in KB):
0-34995 digitization
0-48059 openmedia
0-52428 media	… 9 hrs, too long to catch …

This may be useful info later on …

tsm: VTLTSM>q db

Available     Assigned       Maximum       Maximum        Page         Total          Used       Pct      Max.
    Space     Capacity     Extension     Reduction        Size        Usable         Pages      Util       Pct
     (MB)         (MB)          (MB)          (MB)     (bytes)         Pages                              Util
---------     --------     ---------     ---------     -------     ---------     ---------     -----     -----
   13,328       12,000         1,328         5,448       4,096     3,072,000     1,677,880      54.6      54.6

tsm: VTLTSM>q filespace

Node Name           Filespace       FSID     Platform     Filespace     Is Files-        Capacity       Pct
                    Name                                  Type            pace               (MB)      Util
                                                                        Unicode?
---------------     -----------     ----     --------     ---------     ---------     -----------     -----
MISCNODE            /mnt/digit-        2     Linux86      NFS              No           277,181.6      78.5
                     ization
MISCNODE            /mnt/openm-        3     Linux86      NFS              No           352,776.6      71.2
                     edia
MISCNODE            /mnt/media         4     Linux86      NFS              No           774,092.6      95.8
MISCNODE            /mnt/strea-        5     Linux86      NFS              No         1,209,519.4      85.4
                     ming
MISCNODE            /dragon_da-        6     Linux86      UNKNOWN          No         1,433,605.1      97.9
                     ta
MISCNODE            /dragon_ho-        7     Linux86      UNKNOWN          No         1,228,964.6      98.3
                     me
MISCNODE2           /mnt/clust-        1     Linux86      NFS              No         5,242,880.0      70.7
                     er

Notes on the evaluation activities are listed below. Trying to cross reference what we stated at the beginning of this page. We got nowhere near the 100 desktop client participation, more like 24. Turned out to be a lot of leg work visiting employees desktops. The VLAN location prevented us from using labs as desktop clients (although they also do not represent our clients as there is no local data on these desktops).

We configure cartridge media type to be LTO-3 with a custom size of 50 GB. We assigned 20 of our 40 drives to this single partition holding the media.

• Positives
  1. DXi7500 rack unit installation went smoothly barring some network complexities establishing connectivity
  2. Support is responsive
  3. Backups to the unit are fast from the desktops
  4. Restores are really fast, although this may be the best performance we see until we hit the 70% threshold, although with 40 drives total it probably will still be much better than what we currently experience
• Negatives
  1. many bugs (known and unknown ones) were quickly found
  2. tapes got stuck in “virtual mount” state while tsm showed no mounted tapes; a reboot is the only way to fix that (although partition unload/load might do it too) … to be fixed in upcoming firmware.
  3. upon one of two reboots to get rid of virtual tape mount problems, the unit “hung” overnight and engineer needed to be called in
  4. unit also spontaneously rebooted twice by itself; one of those times apparently was triggered by a fiber disconnnect at circa 11pm 8/7 when nobody was in the data center (although mice and ghosts are not ruled out), cable reconnected in 4 mins !?

Our biggest disappointment is the compress/dedup ratios. Some of the above tests do point out that we probably can achieve ratios greater than 1:10 looking at the nightly incremental data stream only; that is the backup of files by TSM that have already been backed up.

But my biggest concern is new data. Hard disks are now 250GB (instead fo 40 or 80GB), email quotas go from 100MB to 2GB, more pictures, movies, digital books, pdfs, you name it. My unquantified impression is that new data is the significant growth factor, not modified files. So … we really need to establish a dent in the first backup data stream to get ahead. If compress+dedup is no better than compress only, perhaps our solution should focus on “Massive Array of Idle Disks” (MAID technology).

We start with …

And create one cartridge …

Now we have … (i always wanted to know the overhead … added another 99 5 gb cartridges)

Finally, reset TSM db and checkin library …

[root@miscellaneous bin]# ./dsmserv restore db todate=today

VTLTSM>checkin libv q7500 search=yes checklabel=bar status=scratch

Good to go. Export/Import nodes (sizes in MB) …

TSM / DXI reports …

Chris requested the re-export of this node. This in essence is a full backup again. The results are the same … around 1.5x for a full backup. We see nothing in terms of the dedup performance of the inactive/delete file space withing the first backup. This node is uncompressed.

DXI repeorts …

It appears the GUI is not reporting correct info. So scripts were installed and we did some small tests. Found that compression is flat and dedup may run in the order of 1:4 to 1:6 backing up singular instances of c:\windows\systems32 and c:\temp_old … that is a wild mix of windows files and the results are interesting.

So Chris asked me to back up some large file systems. Here is our file share, both for students and departments. Roughly 2.5 TB of windows and mac files of all kinds. We call these fie systems dragon_home and dragon_data. They are backed up in sequence, home first then data.

Presented graphs (3hrs, 15 hrs then in 24 hour increments). When done all graphs in the “Full Overview” menu. Hover over the images to get the ImageName_DateTime information.

Media: 60 LTO-3 50 GB Cartriges and 20 drives (only one used at any point in time)