Modern Computer Vision with PyTorch: Explore deep learning concepts and implement over 50 real-world image applications

Description

Deep learning is the driving force behind many recent advances in various computer vision (CV) applications. This book takes a hands-on approach to help you to solve over 50 CV problems using PyTorch1.x on real-world datasets.

You’ll start by building a neural network (NN) from scratch using NumPy and PyTorch and discover best practices for tweaking its hyperparameters. You’ll then perform image classification using convolutional neural networks and transfer learning and understand how they work. As you progress, you’ll implement multiple use cases of 2D and 3D multi-object detection, segmentation, human-pose-estimation by learning about the R-CNN family, SSD, YOLO, U-Net architectures, and the Detectron2 platform. The book will also guide you in performing facial expression swapping, generating new faces, and manipulating facial expressions as you explore autoencoders and modern generative adversarial networks. You’ll learn how to combine CV with NLP techniques, such as LSTM and transformer, and RL techniques, such as Deep Q-learning, to implement OCR, image captioning, object detection, and a self-driving car agent. Finally, you’ll move your NN model to production on the AWS Cloud.

By the end of this book, you’ll be able to leverage modern NN architectures to solve over 50 real-world CV problems confidently.

What you will learn

• Train a NN from scratch with NumPy and PyTorch
• Implement 2D and 3D multi-object detection and segmentation
• Generate digits and DeepFakes with autoencoders and advanced GANs
• Manipulate images using CycleGAN, Pix2PixGAN, StyleGAN2, and SRGAN
• Combine CV with NLP to perform OCR, image captioning, and object detection
• Combine CV with reinforcement learning to build agents that play pong and self-drive a car
• Deploy a deep learning model on the AWS server using FastAPI and Docker
• Implement over 35 NN architectures and common OpenCV utilities

Who this book is for

This book is for beginners to PyTorch and intermediate-level machine learning practitioners who are looking to get well-versed with computer vision techniques using deep learning and PyTorch. If you are just getting started with neural networks, you’ll find the use cases accompanied by notebooks in GitHub present in this book useful. Basic knowledge of the Python programming language and machine learning is all you need to get started with this book.

Key Features

• Implement solutions to 50 real-world computer vision applications using PyTorch
• Understand the theory and working mechanisms of neural network architectures and their implementation
• Discover best practices using a custom library created especially for this book

Table of Contents

1. Title Page
2. Copyright and Credits
3. Modern Computer Vision with PyTorch
4. Dedication
5. About Packt
6. Why subscribe?
7. Contributors
8. About the authors
9. About the reviewer
10. Packt is searching for authors like you
11. Preface
12. Who this book is for
13. What this book covers
14. To get the most out of this book
15. Download the example code files
16. Download the color images
17. Conventions used
18. Get in touch
19. Reviews
20. Section 1 – Fundamentals of Deep Learning for Computer Vision
    1. Artificial Neural Network Fundamentals
    2. Comparing AI and traditional machine learning
    3. Learning about the artificial neural network building blocks
    4. Implementing feedforward propagation
    5. Calculating the hidden layer unit values
    6. Applying the activation function
    7. Calculating the output layer values
    8. Calculating loss values
    9. Calculating loss during continuous variable prediction
    10. Calculating loss during categorical variable prediction
    11. Feedforward propagation in code
    12. Activation functions in code
    13. Loss functions in code
    14. Implementing backpropagation
    15. Gradient descent in code
    16. Implementing backpropagation using the chain rule
    17. Putting feedforward propagation and backpropagation together
    18. Understanding the impact of the learning rate
    19. Summarizing the training process of a neural network
    20. Summary
    21. Questions
    22. PyTorch Fundamentals
    23. Installing PyTorch
    24. PyTorch tensors
    25. Initializing a tensor
    26. Operations on tensors
    27. Auto gradients of tensor objects
    28. Advantages of PyTorch’s tensors over NumPy’s ndarrays
    29. Building a neural network using PyTorch
    30. Dataset, DataLoader, and batch size
    31. Predicting on new data points
    32. Implementing a custom loss function
    33. Fetching the values of intermediate layers
    34. Using a sequential method to build a neural network
    35. Saving and loading a PyTorch model
    36. state dict
    37. Saving
    38. Loading
    39. Summary
    40. Questions
    41. Building a Deep Neural Network with PyTorch
    42. Representing an image
    43. Converting images into structured arrays and scalars
    44. Why leverage neural networks for image analysis?
    45. Preparing our data for image classification
    46. Training a neural network
    47. Scaling a dataset to improve model accuracy
    48. Understanding the impact of varying the batch size
    49. Batch size of 32
    50. Batch size of 10,000
    51. Understanding the impact of varying the loss optimizer
    52. Understanding the impact of varying the learning rate
    53. Impact of the learning rate on a scaled dataset
    54. High learning rate
    55. Medium learning rate
    56. Low learning rate
    57. Parameter distribution across layers for different learning rates
    58. Impact of varying the learning rate on a non-scaled dataset
    59. Understanding the impact of learning rate annealing
    60. Building a deeper neural network
    61. Understanding the impact of batch normalization
    62. Very small input values without batch normalization
    63. Very small input values with batch normalization
    64. The concept of overfitting
    65. Impact of adding dropout
    66. Impact of regularization
    67. L1 regularization
    68. L2 regularization
    69. Summary
    70. Questions
21. Section 2 – Object Classification and Detection
    1. Introducing Convolutional Neural Networks
    2. The problem with traditional deep neural networks
    3. Building blocks of a CNN
    4. Convolution
    5. Filter
    6. Strides and padding
    7. Strides
    8. Padding
    9. Pooling
    10. Putting them all together
    11. How convolution and pooling help in image translation
    12. Implementing a CNN
    13. Building a CNN-based architecture using PyTorch
    14. Forward propagating the output in Python
    15. Classifying images using deep CNNs
    16. Implementing data augmentation
    17. Image augmentations
    18. Affine transformations
    19. Changing the brightness
    20. Adding noise
    21. Performing a sequence of augmentations
    22. Performing data augmentation on a batch of images and the need for collate_fn
    23. Data augmentation for image translation
    24. Visualizing the outcome of feature learning
    25. Building a CNN for classifying real-world images
    26. Impact on the number of images used for training
    27. Summary
    28. Questions
    29. Transfer Learning for Image Classification
    30. Introducing transfer learning
    31. Understanding VGG16 architecture
    32. Understanding ResNet architecture
    33. Implementing facial key point detection
    34. 2D and 3D facial key point detection
    35. Multi-task learning – Implementing age estimation and gender classification
    36. Introducing the torch_snippets library
    37. Summary
    38. Questions
    39. Practical Aspects of Image Classification
    40. Generating CAMs
    41. Understanding the impact of data augmentation and batch normalization
    42. Coding up road sign detection
    43. Practical aspects to take care of during model implementation
    44. Dealing with imbalanced data
    45. The size of the object within an image
    46. Dealing with the difference between training and validation data
    47. The number of nodes in the flatten layer
    48. Image size
    49. Leveraging OpenCV utilities
    50. Summary
    51. Questions
    52. Basics of Object Detection
    53. Introducing object detection
    54. Creating a bounding box ground truth for training
    55. Installing the image annotation tool
    56. Understanding region proposals
    57. Leveraging SelectiveSearch to generate region proposals
    58. Implementing SelectiveSearch to generate region proposals
    59. Understanding IoU
    60. Non-max suppression
    61. Mean average precision
    62. Training R-CNN-based custom object detectors
    63. Working details of R-CNN
    64. Implementing R-CNN for object detection on a custom dataset
    65. Downloading the dataset
    66. Preparing the dataset
    67. Fetching region proposals and the ground truth of offset
    68. Creating the training data
    69. R-CNN network architecture
    70. Predict on a new image
    71. Training Fast R-CNN-based custom object detectors
    72. Working details of Fast R-CNN
    73. Implementing Fast R-CNN for object detection on a custom dataset
    74. Summary
    75. Questions
    76. Advanced Object Detection
    77. Components of modern object detection algorithms
    78. Anchor boxes
    79. Region Proposal Network
    80. Classification and regression
    81. Training Faster R-CNN on a custom dataset
    82. Working details of YOLO
    83. Training YOLO on a custom dataset
    84. Installing Darknet
    85. Setting up the dataset format
    86. Configuring the architecture
    87. Training and testing the model
    88. Working details of SSD
    89. Components in SSD code
    90. SSD300
    91. MultiBoxLoss
    92. Training SSD on a custom dataset
    93. Summary
    94. Test your understanding
    95. Image Segmentation
    96. Exploring the U-Net architecture
    97. Performing upscaling
    98. Implementing semantic segmentation using U-Net
    99. Exploring the Mask R-CNN architecture
    100. RoI Align
    101. Mask head
    102. Implementing instance segmentation using Mask R-CNN
    103. Predicting multiple instances of multiple classes
    104. Summary
    105. Questions
    106. Applications of Object Detection and Segmentation
    107. Multi-object instance segmentation
    108. Fetching and preparing data
    109. Training the model for instance segmentation
    110. Making inferences on a new image
    111. Human pose detection
    112. Crowd counting
    113. Coding up crowd counting
    114. Image colorization
    115. 3D object detection with point clouds
    116. Theory
    117. Input encoding
    118. Output encoding
    119. Training the YOLO model for 3D object detection
    120. Data format
    121. Data inspection
    122. Training
    123. Testing
    124. Summary
22. Section 3 – Image Manipulation
    1. Autoencoders and Image Manipulation
    2. Understanding autoencoders
    3. Implementing vanilla autoencoders
    4. Understanding convolutional autoencoders
    5. Grouping similar images using t-SNE
    6. Understanding variational autoencoders
    7. Working of VAE
    8. KL divergence
    9. Building a VAE
    10. Performing an adversarial attack on images
    11. Performing neural style transfer
    12. Generating deep fakes
    13. Summary
    14. Questions
    15. Image Generation Using GANs
    16. Introducing GANs
    17. Using GANs to generate handwritten digits
    18. Using DCGANs to generate face images
    19. Implementing conditional GANs
    20. Summary
    21. Questions
    22. Advanced GANs to Manipulate Images
    23. Leveraging the Pix2Pix GAN
    24. Leveraging CycleGAN
    25. Leveraging StyleGAN on custom images
    26. Super-resolution GAN
    27. Architecture
    28. Coding SRGAN
    29. Summary
    30. Questions
23. Section 4 – Combining Computer Vision with Other Techniques
    1. Training with Minimal Data Points
    2. Implementing zero-shot learning
    3. Coding zero-shot learning
    4. Implementing few-shot learning
    5. Building a Siamese network
    6. Coding Siamese networks
    7. Working details of prototypical networks
    8. Working details of relation networks
    9. Summary
    10. Questions
    11. Combining Computer Vision and NLP Techniques
    12. Introducing RNNs
    13. The idea behind the need for RNN architecture
    14. Exploring the structure of an RNN
    15. Why store memory?
    16. Introducing LSTM architecture
    17. The working details of LSTM
    18. Implementing LSTM in PyTorch
    19. Implementing image captioning
    20. Image captioning in code
    21. Transcribing handwritten images
    22. The working details of CTC loss
    23. Calculating the CTC loss value
    24. Handwriting transcription in code
    25. Object detection using DETR
    26. The working details of transformers
    27. Basics of transformers
    28. The working details of DETR
    29. Detection with transformers in code
    30. Summary
    31. Questions
    32. Combining Computer Vision and Reinforcement Learning
    33. Learning the basics of reinforcement learning
    34. Calculating the state value
    35. Calculating the state-action value
    36. Implementing Q-learning
    37. Q-value
    38. Understanding the Gym environment
    39. Building a Q-table
    40. Leveraging exploration-exploitation
    41. Implementing deep Q-learning
    42. Implementing deep Q-learning with the fixed targets model
    43. Coding up an agent to play Pong
    44. Implementing an agent to perform autonomous driving
    45. Installing the CARLA environment
    46. Install the CARLA binaries
    47. Installing the CARLA Gym environment
    48. Training a self-driving agent
    49. model.py
    50. actor.py
    51. Training DQN with fixed targets
    52. Summary
    53. Questions
    54. Moving a Model to Production
    55. Understanding the basics of an API
    56. Creating an API and making predictions on a local server
    57. Installing the API module and dependencies
    58. Serving an image classifier
    59. fmnist.py
    60. server.py
    61. Running the server
    62. Moving the API to the cloud
    63. Comparing Docker containers and Docker images
    64. Creating a Docker container
    65. Creating the requirements.txt file
    66. Creating a Dockerfile
    67. Building a Docker image and creating a Docker container
    68. Shipping and running the Docker container in the cloud
    69. Configuring AWS
    70. Creating a Docker repository on AWS ECR and pushing the image
    71. Creating an EC2 instance
    72. Pulling the image and building the Docker container
    73. Summary
    74. Using OpenCV Utilities for Image Analysis
    75. Drawing bounding boxes around words in an image
    76. Detecting lanes in an image of a road
    77. Detecting objects based on color
    78. Building a panoramic view of images
    79. Detecting the number plate of a car
    80. Summary
    81. Appendix
    82. Chapter 1 – Artificial Neural Network Fundamentals
    83. Chapter 2 – PyTorch Fundamentals
    84. Chapter 3 – Building a Deep Neural Network with PyTorch
    85. Chapter 4 – Introducing Convolutional Neural Networks
    86. Chapter 5 – Transfer Learning for Image Classification
    87. Chapter 6 – Practical Aspects of Image Classification
    88. Chapter 7 – Basics of Object Detection
    89. Chapter 8 – Advanced Object Detection
    90. Chapter 9 – Image Segmentation
    91. Chapter 11 – Autoencoders and Image Manipulation
    92. Chapter 12 – Image Generation Using GANs
    93. Chapter 13 – Advanced GANs to Manipulate Images
    94. Chapter 14 – Training with Minimal Data Points
    95. Chapter 15 – Combining Computer Vision and NLP Techniques
    96. Chapter 16 – Combining Computer Vision and Reinforcement Learning

Authors Biography

V Kishore Ayyadevara leads a team focused on using AI to solve problems in the healthcare space. He has more than 10 years’ experience in the field of data science with prominent technology companies. In his current role, he is responsible for developing a variety of cutting-edge analytical solutions that have an impact at scale while building strong technical teams. Kishore has filed 8 patents at the intersection of machine learning, healthcare, and operations. Prior to this book, he authored four books in the fields of machine learning and deep learning. Kishore got his MBA from IIM Calcutta and his engineering degree from Osmania University.

Yeshwanth Reddy is a senior data scientist with a strong focus on the research and implementation of cutting-edge technologies to solve problems in the health and computer vision domains. He has filed four patents in the field of OCR. He also has 2 years of teaching experience, where he delivered sessions to thousands of students in the fields of statistics, machine learning, AI, and natural language processing. He has completed his MTech and BTech at IIT Madras.