The Practice of System and Network Administration: The DevOps and other Best Practices for Enterprise IT (Volume 1) (3rd Edition)

Description

With 28 new chapters, the third edition of The Practice of System and Network Administration innovates yet again! Revised with thousands of updates and clarifications based on reader feedback, this new edition also incorporates DevOps strategies even for non-DevOps environments.

Whether you use Linux, Unix, or Windows, this new edition describes the essential practices previously handed down only from mentor to protégé. This wonderfully lucid, often funny cornucopia of information introduces beginners to advanced frameworks valuable for their entire career, yet is structured to help even experts through difficult projects.

Other books tell you what commands to type. This book teaches you the cross-platform strategies that are timeless!

• DevOps techniques: Apply DevOps principles to enterprise IT infrastructure, even in environments without developers
• Game-changing strategies: New ways to deliver results faster with less stress
• Fleet management: A comprehensive guide to managing your fleet of desktops, laptops, servers and mobile devices
• Service management: How to design, launch, upgrade and migrate services
• Measurable improvement: Assess your operational effectiveness; a forty-page, pain-free assessment system you can start using today to raise the quality of all services
• Design guides: Best practices for networks, data centers, email, storage, monitoring, backups and more
  Management skills: Organization design, communication, negotiation, ethics, hiring and firing, and more

Have you ever had any of these problems?

• Have you been surprised to discover your backup tapes are blank?
• Ever spent a year launching a new service only to be told the users hate it?
• Do you have more incoming support requests than you can handle?
• Do you spend more time fixing problems than building the next awesome thing?
• Have you suffered from a botched migration of thousands of users to a new service?
• Does your company rely on a computer that, if it died, can’t be rebuilt?
• Is your network a fragile mess that breaks any time you try to improve it?
• Is there a periodic “hell month” that happens twice a year? Twelve times a year?
• Do you find out about problems when your users call you to complain?
• Does your corporate “Change Review Board” terrify you?
• Does each division of your company have their own broken way of doing things?
• Do you fear that automation will replace you, or break more than it fixes?
• Are you underpaid and overworked?

No vague “management speak” or empty platitudes. This comprehensive guide provides real solutions that prevent these problems and more!

Table of Contents

1. Preface xxxix
2. Acknowledgments xlvii
3. About the Authors li
4. Part I: Game-Changing Strategies 1
   1. Chapter 1: Climbing Out of the Hole 3
      1. Organizing WIP 5
      2. Eliminating Time Sinkholes 12
      3. DevOps 16
      4. DevOps Without Devs 16
      5. Bottlenecks 18
      6. Getting Started 20
      7. Summary 21
      8. Exercises 22
   2. Chapter 2: The Small Batches Principle 23
      1. The Carpenter Analogy 23
      2. Fixing Hell Month 24
      3. Improving Emergency Failovers 26
      4. Launching Early and Often 29
      5. Summary 34
      6. Exercises 34
   3. Chapter 3: Pets and Cattle 37
      1. The Pets and Cattle Analogy 37
      2. Scaling 39
      3. Desktops as Cattle 40
      4. Server Hardware as Cattle 41
      5. Pets Store State 43
      6. Isolating State 44
      7. Generic Processes 47
      8. Moving Variations to the End 51
      9. Automation 53
      10. Summary 53
      11. Exercises 54
   4. Chapter 4: Infrastructure as Code 55
      1. Programmable Infrastructure 56
      2. Tracking Changes 57
      3. Benefits of Infrastructure as Code 59
      4. Principles of Infrastructure as Code 62
      5. Configuration Management Tools 63
      6. Example Infrastructure as Code Systems 67
      7. Bringing Infrastructure as Code to Your Organization 71
      8. Infrastructure as Code for Enhanced Collaboration 72
      9. Downsides to Infrastructure as Code 73
      10. Automation Myths 74
      11. Summary 75
      12. Exercises 76
5. Part II: Workstation Fleet Management 77
   1. Chapter 5: Workstation Architecture 79
      1. Fungibility 80
      2. Hardware 82
      3. Operating System 82
      4. Network Configuration 84
      5. Accounts and Authorization 86
      6. Data Storage 89
      7. OS Updates 93
      8. Security 94
      9. Logging 97
      10. Summary 98
      11. Exercises 99
   2. Chapter 6: Workstation Hardware Strategies 101
      1. Physical Workstations 101
      2. Virtual Desktop Infrastructure 105
      3. Bring Your Own Device 110
      4. Summary 113
      5. Exercises 114
   3. Chapter 7: Workstation Software Life Cycle 117
      1. Life of a Machine 117
      2. OS Installation 120
      3. OS Configuration 120
      4. Updating the System Software and Applications 123
      5. Rolling Out Changes . . . Carefully 128
      6. Disposal 130
      7. Summary 134
      8. Exercises 135
   4. Chapter 8: OS Installation Strategies 137
      1. Consistency Is More Important Than Perfection 138
      2. Installation Strategies 142
      3. Test-Driven Configuration Development 147
      4. Automating in Steps 148
      5. When Not to Automate 152
      6. Vendor Support of OS Installation 152
      7. Should You Trust the Vendor’s Installation? 154
      8. Summary 154
      9. Exercises 155
   5. Chapter 9: Workstation Service Definition 157
      1. Basic Service Definition 157
      2. Refresh Cycles 161
      3. Tiered Support Levels 165
      4. Workstations as a Managed Service 168
      5. Summary 170
      6. Exercises 171
   6. Chapter 10: Workstation Fleet Logistics 173
      1. What Employees See 173
      2. What Employees Don’t See 174
      3. Configuration Management Database 183
      4. Small-Scale Fleet Logistics 186
      5. Summary 188
      6. Exercises 188
   7. Chapter 11: Workstation Standardization 191
      1. Involving Customers Early 192
      2. Releasing Early and Iterating 193
      3. Having a Transition Interval (Overlap) 193
      4. Ratcheting 194
      5. Setting a Cut-Off Date 195
      6. Adapting for Your Corporate Culture 195
      7. Leveraging the Path of Least Resistance 196
      8. Summary 198
      9. Exercises 199
   8. Chapter 12: Onboarding 201
      1. Making a Good First Impression 201
      2. IT Responsibilities 203
      3. Five Keys to Successful Onboarding 203
      4. Cadence Changes 212
      5. Case Studies 212
      6. Summary 216
      7. Exercises 217
6. Part III: Servers 219
   1. Chapter 13: Server Hardware Strategies 221
      1. All Eggs in One Basket 222
      2. Beautiful Snowflakes 224
      3. Buy in Bulk, Allocate Fractions 228
      4. Grid Computing 235
      5. Blade Servers 237
      6. Cloud-Based Compute Services 238
      7. Server Appliances 241
      8. Hybrid Strategies 242
      9. Summary 243
      10. Exercises 244
   2. Chapter 14: Server Hardware Features 245
      1. Workstations Versus Servers 246
      2. Server Reliability 249
      3. Remotely Managing Servers 254
      4. Separate Administrative Networks 257
      5. Maintenance Contracts and Spare Parts 258
      6. Selecting Vendors with Server Experience 261
      7. Summary 263
      8. Exercises 263
   3. Chapter 15: Server Hardware Specifications 265
      1. Models and Product Lines 266
      2. Server Hardware Details 266
      3. Things to Leave Out 278
      4. Summary 278
      5. Exercises 279
7. Part IV: Services 281
   1. Chapter 16: Service Requirements 283
      1. Services Make the Environment 284
      2. Starting with a Kick-Off Meeting 285
      3. Gathering Written Requirements 286
      4. Customer Requirements 288
      5. Scope, Schedule, and Resources 291
      6. Operational Requirements 292
      7. Open Architecture 298
      8. Summary 302
      9. Exercises 303
   2. Chapter 17: Service Planning and Engineering 305
      1. General Engineering Basics 306
      2. Simplicity 307
      3. Vendor-Certified Designs 308
      4. Dependency Engineering 309
      5. Decoupling Hostname from Service Name 313
      6. Support 315
      7. Summary 319
      8. Exercises 319
   3. Chapter 18: Service Resiliency and Performance Patterns 321
      1. Redundancy Design Patterns 322
      2. Performance and Scaling 326
      3. Summary 333
      4. Exercises 334
   4. Chapter 19: Service Launch: Fundamentals 335
      1. Planning for Problems 335
      2. The Six-Step Launch Process 336
      3. Launch Readiness Review 345
      4. Launch Calendar 348
      5. Common Launch Problems 349
      6. Summary 351
      7. Exercises 351
   5. Chapter 20: Service Launch: DevOps 353
      1. Continuous Integration and Deployment 354
      2. Minimum Viable Product 357
      3. Rapid Release with Packaged Software 359
      4. Cloning the Production Environment 362
      5. Example: DNS/DHCP Infrastructure Software 363
      6. Launch with Data Migration 366
      7. Controlling Self-Updating Software 369
      8. Summary 370
      9. Exercises 371
   6. Chapter 21: Service Conversions 373
      1. Minimizing Intrusiveness 374
      2. Layers Versus Pillars 376
      3. Vendor Support 377
      4. Communication 378
      5. Training 379
      6. Gradual Roll-Outs 379
      7. Flash-Cuts: Doing It All at Once 380
      8. Backout Plan 383
      9. Summary 385
      10. Exercises 385
   7. Chapter 22: Disaster Recovery and Data Integrity 387
      1. Risk Analysis 388
      2. Legal Obligations 389
      3. Damage Limitation 390
      4. Preparation 391
      5. Data Integrity 392
      6. Redundant Sites 393
      7. Security Disasters 394
      8. Media Relations 394
      9. Summary 395
      10. Exercises 395
8. Part V: Infrastructure 397
   1. Chapter 23: Network Architecture 399
      1. Physical Versus Logical 399
      2. The OSI Model 400
      3. Wired Office Networks 402
      4. Wireless Office Networks 406
      5. Datacenter Networks 408
      6. WAN Strategies 413
      7. Routing 419
      8. Internet Access 420
      9. Corporate Standards 422
      10. Software-Defined Networks 425
      11. IPv6 426
      12. Summary 428
      13. Exercises 429
   2. Chapter 24: Network Operations 431
      1. Monitoring 431
      2. Management 432
      3. Documentation 437
      4. Support 440
      5. Summary 446
      6. Exercises 447
   3. Chapter 25: Datacenters Overview 449
      1. Build, Rent, or Outsource 450
      2. Requirements 452
      3. Summary 456
      4. Exercises 457
   4. Chapter 26: Running a Datacenter 459
      1. Capacity Management 459
      2. Life-Cycle Management 465
      3. Patch Cables 468
      4. Labeling 471
      5. Console Access 475
      6. Workbench 476
      7. Tools and Supplies 477
      8. Summary 480
      9. Exercises 481
9. Part VI: Helpdesks and Support 483
   1. Chapter 27: Customer Support 485
      1. Having a Helpdesk 485
      2. Offering a Friendly Face 488
      3. Reflecting Corporate Culture 488
      4. Having Enough Staff 488
      5. Defining Scope of Support 490
      6. Specifying How to Get Help 493
      7. Defining Processes for Staff 493
      8. Establishing an Escalation Process 494
      9. Defining “Emergency” in Writing 495
      10. Supplying Request-Tracking Software 496
      11. Statistical Improvements 498
      12. After-Hours and 24/7 Coverage 499
      13. Better Advertising for the Helpdesk 500
      14. Different Helpdesks for Different Needs 501
      15. Summary 502
      16. Exercises 503
   2. Chapter 28: Handling an Incident Report 505
      1. Process Overview 506
      2. Phase A—Step 1: The Greeting 508
      3. Phase B: Problem Identification 509
      4. Phase C: Planning and Execution 515
      5. Phase D: Verification 518
      6. Perils of Skipping a Step 519
      7. Customer Care 521
      8. Summary 525
      9. Exercises 527
   3. Chapter 29: Debugging 529
      1. Understanding the Customer’s Problem 529
      2. Fixing the Cause, Not the Symptom 531
      3. Being Systematic 532
      4. Having the Right Tools 533
      5. End-to-End Understanding of the System 538
      6. Summary 540
      7. Exercises 540
   4. Chapter 30: Fixing Things Once 541
      1. Story: The Misconfigured Servers 541
      2. Avoiding Temporary Fixes 543
      3. Learn from Carpenters 545
      4. Automation 547
      5. Summary 549
      6. Exercises 550
   5. Chapter 31: Documentation 551
      1. What to Document 552
      2. A Simple Template for Getting Started 553
      3. Easy Sources for Documentation 554
      4. The Power of Checklists 556
      5. Wiki Systems 557
      6. Findability 559
      7. Roll-Out Issues 559
      8. A Content-Management System 560
      9. A Culture of Respect 561
      10. Taxonomy and Structure 561
      11. Additional Documentation Uses 562
      12. Off-Site Links 562
      13. Summary 563
      14. Exercises 564
10. Part VII: Change Processes 565
    1. Chapter 32: Change Management 567
       1. Change Review Boards 568
       2. Process Overview 570
       3. Change Proposals 570
       4. Change Classifications 571
       5. Risk Discovery and Quantification 572
       6. Technical Planning 573
       7. Scheduling 574
       8. Communication 576
       9. Tiered Change Review Boards 578
       10. Change Freezes 579
       11. Team Change Management 581
       12. Starting with Git 583
       13. Summary 585
       14. Exercises 585
    2. Chapter 33: Server Upgrades 587
       1. The Upgrade Process 587
       2. Step 1: Develop a Service Checklist 588
       3. Step 2: Verify Software Compatibility 591
       4. Step 3: Develop Verification Tests 592
       5. Step 4: Choose an Upgrade Strategy 595
       6. Step 5: Write a Detailed Implementation Plan 598
       7. Step 6: Write a Backout Plan 600
       8. Step 7: Select a Maintenance Window 600
       9. Step 8: Announce the Upgrade 602
       10. Step 9: Execute the Tests 603
       11. Step 10: Lock Out Customers 604
       12. Step 11: Do the Upgrade with Someone 605
       13. Step 12: Test Your Work 605
       14. Step 13: If All Else Fails, Back Out 605
       15. Step 14: Restore Access to Customers 606
       16. Step 15: Communicate Completion/Backout 606
       17. Summary 608
       18. Exercises 610
    3. Chapter 34: Maintenance Windows 611
       1. Process Overview 612
       2. Getting Management Buy-In 613
       3. Scheduling Maintenance Windows 614
       4. Planning Maintenance Tasks 615
       5. Selecting a Flight Director 616
       6. Managing Change Proposals 617
       7. Developing the Master Plan 620
       8. Disabling Access 621
       9. Ensuring Mechanics and Coordination 622
       10. Change Completion Deadlines 628
       11. Comprehensive System Testing 628
       12. Post-maintenance Communication 630
       13. Reenabling Remote Access 631
       14. Be Visible the Next Morning 631
       15. Postmortem 631
       16. Mentoring a New Flight Director 632
       17. Trending of Historical Data 632
       18. Providing Limited Availability 633
       19. High-Availability Sites 634
       20. Summary 636
       21. Exercises 637
    4. Chapter 35: Centralization Overview 639
       1. Rationale for Reorganizing 640
       2. Approaches and Hybrids 642
       3. Summary 643
       4. Exercises 644
    5. Chapter 36: Centralization Recommendations 645
       1. Architecture 645
       2. Security 645
       3. Infrastructure 648
       4. Support 654
       5. Purchasing 655
       6. Lab Environments 656
       7.  Summary 656
       8. Exercises 657
    6. Chapter 37: Centralizing a Service 659
       1. Understand the Current Solution 660
       2. Make a Detailed Plan 661
       3. Get Management Support 662
       4. Fix the Problems 662
       5. Provide an Excellent Service 663
       6. Start Slowly 663
       7. Look for Low-Hanging Fruit 664
       8. When to Decentralize 665
       9. Managing Decentralized Services 666
       10. Summary 667
       11. Exercises 668
11. Part VIII: Service Recommendations 669
    1. Chapter 38: Service Monitoring 671
       1. Types of Monitoring 672
       2. Building a Monitoring System 673
       3. Historical Monitoring 67438.4
       4. Real-Time Monitoring 676
       5. Scaling 684
       6. Centralization and Accessibility 685
       7. Pervasive Monitoring 686
       8. End-to-End Tests 687
       9. Application Response Time Monitoring 688
       10. Compliance Monitoring 689
       11. Meta-monitoring 690
       12. Summary 690
       13. Exercises 691
    2. Chapter 39: Namespaces 693
       1. What Is a Namespace? 693
       2. Basic Rules of Namespaces 694
       3. Defining Names 694
       4. Merging Namespaces 698
       5. Life-Cycle Management 699
       6. Reuse 700
       7. Usage 701
       8. Federated Identity 708
       9. Summary 709
       10. Exercises 710
    3. Chapter 40: Nameservices 711
       1. Nameservice Data 711
       2. Reliability 714
       3. Access Policy 721
       4. Change Policies 723
       5. Change Procedures 724
       6. Centralized Management 726
       7. Summary 728
       8. Exercises 728
    4. Chapter 41: Email Service 729
       1. Privacy Policy 730
       2. Namespaces 730
       3. Reliability 731
       4. Simplicity 733
       5. Spam and Virus Blocking 735
       6. Generality 736
       7. Automation 737
       8. Monitoring 738
       9. Redundancy 738
       10. Scaling 739
       11. Security Issues 742
       12. Encryption 743
       13. Email Retention Policy 743
       14. Communication 744
       15. High-Volume List Processing 745
       16. Summary 746
       17. Exercises 747
    5. Chapter 42: Print Service 749
       1. Level of Centralization 750
       2. Print Architecture Policy 751
       3. Documentation 754
       4. Monitoring 755
       5. Environmental Issues 756
       6. Shredding 757
       7. Summary 758
       8. Exercises 758
    6. Chapter 43: Data Storage 759
       1. Terminology 760
       2. Managing Storage 765
       3. Storage as a Service 772
       4. Performance 780
       5. Evaluating New Storage Solutions 784
       6. Common Data Storage Problems 787
       7. Summary 789
       8. Exercises 790
    7. Chapter 44: Backup and Restore 793
       1. Getting Started 794
       2. Reasons for Restores 795
       3. Corporate Guidelines 799
       4. A Data-Recovery SLA and Policy 800
       5. The Backup Schedule 801
       6. Time and Capacity Planning 807
       7. Consumables Planning 809
       8. Restore-Process Issues 815
       9. Backup Automation 816
       10. Centralization 819
       11. Technology Changes 820
       12. Summary 821
       13. Exercises 822
    8. Chapter 45: Software Repositories 825
       1. Types of Repositories 826
       2. Benefits of Repositories 827
       3. Package Management Systems 829
       4. Anatomy of a Package 829
       5. Anatomy of a Repository 833
       6. Managing a Repository 837
       7. Repository Client 841
       8. Build Environment 843
       9. Repository Examples 845
       10. Summary 848
       11. Exercises 849
    9. Chapter 46: Web Services 851
       1. Simple Web Servers 852
       2. Multiple Web Servers on One Host 853
       3. Service Level Agreements 854
       4. Monitoring 855
       5. Scaling for Web Services 855
       6. Web Service Security 859
       7. Content Management 866
       8. Summary 868
       9. Exercises 869
12. Part IX: Management Practices 871
    1. Chapter 47: Ethics 873
       1. Informed Consent 873
       2. Code of Ethics 875
       3. Customer Usage Guidelines 875
       4. Privileged-Access Code of Conduct 877
       5. Copyright Adherence 878
       6. Working with Law Enforcement 881
       7. Setting Expectations on Privacy and Monitoring 885
       8. Being Told to Do Something Illegal/Unethical 887
       9. Observing Illegal Activity 888
       10. Summary 889
       11. Exercises 889
    2. Chapter 48: Organizational Structures 891
       1. Sizing 892
       2. Funding Models 894
       3. Management Chain’s Influence 897
       4. Skill Selection 898
       5. Infrastructure Teams 900
       6. Customer Support 902
       7. Helpdesk 904
       8. Outsourcing 904
       9. Consultants and Contractors 906
       10. Sample Organizational Structures 907
       11. Summary 911
       12. Exercises 911
    3. Chapter 49: Perception and Visibility 913
       1. Perception 913
       2. Visibility 925
       3. Summary 933
       4. Exercises 934
    4. Chapter 50: Time Management 935
       1. Interruptions 935
       2. Follow-Through 937
       3. Basic To-Do List Management 938
       4. Setting Goals 939
       5. Handling Email Once 940
       6. Precompiling Decisions 942
       7. Finding Free Time 943
       8. Dealing with Ineffective People 944
       9. Dealing with Slow Bureaucrats 944
       10. Summary 946
       11. Exercises 946
    5. Chapter 51: Communication and Negotiation 949
       1. Communication 949
       2. I Statements 950
       3. Active Listening 950
       4. Negotiation 954
       5. Additional Negotiation Tips 958
       6. Further Reading 960
       7. Summary 961
       8. Exercises 961
    6. Chapter 52: Being a Happy SA 963
       1. Happiness 963
       2. Accepting Criticism 965
       3. Your Support Structure 965
       4. Balancing Work and Personal Life 966
       5. Professional Development 967
       6. Staying Technical 968
       7. Loving Your Job 969
       8. Motivation 970
       9. Managing Your Manager 972
       10. Self-Help Books 976
       11. Summary 976
       12. Exercises 977
    7. Chapter 53: Hiring System Administrators 979
       1. Job Description 980
       2. Skill Level 982
       3. Recruiting 983
       4. Timing 985
       5. Team Considerations 987
       6. The Interview Team 990
       7. Interview Process 991
       8. Technical Interviewing 994
       9. Nontechnical Interviewing 998
       10. Selling the Position 1000
       11. Employee Retention 1000
       12. Getting Noticed 1001
       13. Summary 1002
       14. Exercises 1003
    8. Chapter 54: Firing System Administrators 1005
       1. Cooperate with Corporate HR 1006
       2. The Exit Checklist 1007
       3. Removing Access 1007
       4. Logistics 1011
       5. Examples 1011
       6. Supporting Infrastructure 1014
       7. Summary 1015
       8. Exercises 1016
13. Part X: Being More Awesome 1017
    1. Chapter 55: Operational Excellence 1019
       1. What Does Operational Excellence Look Like? 1019
       2. How to Measure Greatness 1020
       3. Assessment Methodology 1021
       4. Service Assessments 1025
       5. Organizational Assessments 1029
       6. Levels of Improvement 1030
       7. Getting Started 1031
       8. Summary 1032
       9. Exercises 1033
    2. Chapter 56: Operational Assessments 1035
       1. Regular Tasks (RT) 1036
       2. Emergency Response (ER) 1039
       3. Monitoring and Metrics (MM) 1041
       4. Capacity Planning (CP) 1043
       5. Change Management (CM) 1045
       6. New Product Introduction and Removal (NPI/NPR) 1047
       7. Service Deployment and Decommissioning (SDD) 1049
       8. Performance and Efficiency (PE) 1051
       9. Service Delivery: The Build Phase 1054
       10. Service Delivery: The Deployment Phase 1056
       11. Toil Reduction 1058
       12. Disaster Preparedness 1060
       13. Epilogue 1063
14. Part XI: Appendices 1065
    1. Appendix A: What to Do When . . . 1067
    2. Appendix B: The Many Roles of a System Administrator 1089
    3. B.1 Common Positive Roles 1090
    4. B.2 Negative Roles 1107
    5. B.3 Team Roles 1109
    6. B.4 Summary 1112
    7. Exercises 1112
    8. Bibliography 1115
    9. Index 1121

Authors Biography

Thomas A. Limoncelli is an internationally recognized author, speaker, and system administrator with more than twenty years of experience at companies like Google, Bell Labs, and StackOverflow.com.

Christina J. Hogan has more than twenty years of experience in system administration and network engineering, from Silicon Valley to Italy and Switzerland. She has a master’s degree in computer science, a doctorate in aeronautical engineering, and has been part of a Formula 1 racing team.

Strata R. Chalup has more than twenty-five years of experience in Silicon Valley, focusing on IT strategy, best-practices, and scalable infrastructures at firms that include Apple, Sun, Cisco, McAfee, and Palm.