Image Recognition: Dogs vs Cats! (92%)

Running this tutorial ¶

Unlike most posts on this blog, I do not provide a recipe to run this notebook on Google Colab. I tried it, but it appears that:

• the disk transfers on the Google Colab virtual machines are too slow. This very much slows down the training of neural networks on relatively large datasets such as the dogs and cats dataset.
• the number of CPU cores on these machines is too low to handle the heavy image preprocessing that we will need to apply.

As a consequence, you'll need to run on your machine.

First, install TensorFlow for your Linux or Windows PC. With these recipes, you will also install anaconda, including the keras package.

Then, install these packages with conda:

conda install numpy matplotlib

Clone my github repo locally, and start the notebook:

git clone https://github.com/cbernet/maldives.git
cd maldives/dogs_vs_cats
jupyter notebook

And open the notebook dogs_vs_cats_local.ipynb .

I haven't tested this recipe, so if it doesn't work, tell me in the comments and I'll help right away.

The dogs and cats dataset ¶

The dogs and cats dataset was first introduced for a Kaggle competition in 2013. To access the dataset, you will need to create a Kaggle account and to log in. No pressure, we're not here for the competition, but to learn!

The dataset is available here . You can use the kaggle utility to get the dataset, or simply download the train.zip file (about 540 MB). Don't forget to log in to Kaggle first.

The instructions to prepare the dataset are for Linux or macOS. If you work on Windows, I'm sure you can easily find a way to do that (e.g. use 7-zip do unpack the archive, and Windows Explorer to create directories and move files around).

Once downloaded, unzip the archive:

unzip train.zip

List the contents of the train directory:

ls train

You will see a lot of images of dogs and cats.

In the next sections, we will use Keras to retrieve images from disk with the flow_from_directory method of the ImageDataGenerator class.

This method however requires the images from the different classes to be sorted in different directories. So we will put all dog images in dogs , and all cat images in cats :

mkdir cats 
mkdir dogs
find train -name 'dog.*' -exec mv {} dogs/ \;
find train -name 'cat.*' -exec mv {} cats/ \;

You might be wondering why I used find instead of a simple mv to move the files. It's because with mv , the shell needs to pass a very large number of arguments to the command (all the file names), and there is a limitation on this number on macOS (on Linux, it's working fine). With find , we can work around this limitation.

Initialization ¶

Now, enter in the cell below the location of your dataset directory, the one that contains the dogs and cats subdirectories, and execute it:

A first look at the dogs and cats dataset ¶

We can start investigating the dataset by plotting the first picture in each category:

Cute. But let's be more specific and print some information about our images:

In the image shape, the first two columns correspond to the height and width of the image in pixels, respectively, and the third one to the three color channels. So each pixel contains three values (for red, green, and blue, respectively). We also print the maximum color level in each channel, and we can conclude that the RGB levels are in the range 0-255.

Pet cleaning: improving the dataset quality ¶

If there is only one thing that you should take away from this tutorial it is this:

You Should Never Trust Your Data

The data are always dirty.

To get a close look at this dataset, I used a fast image browser to check all images in the dogs and cats directories. Actually, I simply used the Preview application on my mac to browse through the small preview icons. The brain is very fast to spot obvious issues even if you just let your eyes wander on a large number of pictures. So this (tedious) work took me no more than 20 minutes. But of course, I have certainly missed a lot of less obvious issues.

Anyway, here is what I found.

First, here are the indices for the bad images I could find in each category:

We can then retrieve the images with these ids from the cats and dogs directories:

Some of these images are completely meaningless, like 5604 and 8736. For 10401 and 10797, we actually see a cat in the picture! sigh... Keeping or not cartoon dogs is debatable. My feeling is that it's going to be better to remove them. Same for 6413, we could keep it, but I'm afraid that the network would focus on the drawings around the dog picture.

Now let's look at the bad cats:

Again, I'm not a fan of keeping cartoon cats for training, but who knows, it could be ok. In image 4688, we have a small cartoon cat and a big cartoon dog... this one should clearly be rejected. In image 6215, we just see fur, so it could be either a cat or a dog, though this does look like cat fur. And why is the guy in image 7377 labeled as a cat? no idea...

It should however be noted that even if we decide to reject cartoon images for training, the trained network might be able to identify them correctly.

Now let's implement a small function to clean up the dataset:

If you want to restore your dataset to its original version before cleaning, just uncomment and execute the following:

Load the dogs and cats image dataset with Keras ¶

Neural networks are trained by presenting them with batches of images, each of them with a label identifying the true nature of the image (either cat or dog in our case). A batch may contain of the order of a few tenths to a couple hundred images. If you want an introduction to neural networks and supervised learning for classification, you can check out my post on Handwritten Digit Recognition with scikit-learn .

For each image, the prediction of the network is compared with the corresponding label, and the distance between the predictions of the network and the truth is evaluated for the whole batch. Then, after the processing of the batch, the network parameters are changed in a way to minimize this distance, therefore improving the prediction capability of the network. The training then proceeds iteratively, batch after batch.

So we need a way to turn our images, now files on disk, into batches of data arrays in memory that can be fed to the network during training.

The ImageDataGenerator can readily be used for this purpose. Let's import this class and create an instance of the generator:

Now, we will use the flow_from_directory method of the gen object to start generating batches.

This method will return an iterator that returns a batch everytime it's iterated on. To see how the data is organized, we can simply create this iterator, and get a first batch to look at it:

The batch has two elements. What's their type?

Two numpy arrays! well, it means we can print their shape and type:

Obviously, the first element is an array of 32 images with 256x256 pixels, and 3 color channels, encoded as floats in the range 0 to 255. So the ImageDataGenerator did force the image to 256x256 pixels as requested, but didn't normalize the color levels between 0 and 1. We will have to do that later.

The second element contains the 32 corresponding labels.

Before having a detailed look at the labels, we can plot the first image:

And here is the corresponding label:

We see that the ImageDataGenerator automatically produced a label for each image depending on the directory in which it was found. One-hot encoding is used for the labels, and this is precisely what we need for the classification task we are up to. If you want to know more about one-hot encoding, check my post about a First Neural Network with Keras .

We can also guess that the label [0., 1.] corresponds to a true cat, and [1.,0.] to a true dog. The prediction of the network for a given image will lie somewhere in between, like [0.6,0.4] for a dog. Still, that's only a guess, and a guess is not enough! we need to be sure, or we take the risk to feed our network with mislabeled images (and garbage in, garbage out).

Since it might be the first time you use the ImageDataGenerator, you probably want to get more confident about this tool. For that, we are going to develop a small function in the next section to validate the input dataset.

Pet plotting for label validation ¶

To validate the dataset labels, we want to check that, for a few batches, the labels are correctly set. So we need a function that can plot a fairly large number of images and label them. Here it is:

Please repeat the previous cell until you are confident that the labels are correct.

Split the training and validation samples with ImageDataGenerator ¶

We will train our neural network on a subset of the dog and cat images called the training dataset .

If a network is complex enough (if it has enough parameters), it can start overfitting , which means that it is able to learn the specific features of the images in the training dataset. In other words, the network loses its generality, and its ability to classify an arbitrary image as a dog or cat image.

To make sure that overfitting does not occur, we will evaluate the performance of the network on a validation dataset , disjoint from the training dataset.

Let's first create a new ImageDataGenerator . With respect to the previous one, we request to:

• rescale all color levels to be in the range 0-1. We're doing this because neural nets behave better on input variables of the order of unity.
• use 20% of the images for validation, and thus 80% for the training.

Next, we define our iterators for the training and validation datasets. We use a batch size of 30 because, typically, networks are not able to learn if the batch size is too large or too small. You may try again with a different batch size after completing this tutorial.

The images are forced to 256x256 pixels. Here, we just need to make sure that the format of all images is the same, as the convolutional neural network that we're going to use has a fixed number of inputs. I chose a square shape to avoid getting too much distortion on both landscape and portrait pictures. But if the vast majority of our images is portrait, maybe a shape that is closer to portrait would be a better choice. I didn't check that.

A simple convolutional neural network ¶

Deep convolutional neural networks are the go-to choice when it comes to classifying images. For a detailed introduction to this kind of networks, see my post on Tuning such a network for handwritten digit recognition . The model below is very similar to the one we have used in this article.

Here are the main differences with respect to the network used for handwritten digits recognition:

• we have two neurons in the last softmax layer instead of 10, as we now have two categories.
• our input images are much larger, 256x256 instead of 28x28. Note that the input_shape of the first layer has to match the shape given to the generator.
• I have added a third convolutional layer, and I'm extracting more features at each layer.
• the dropout rate was increased from 0.4 to 0.9 to help reduce overfitting.

The first two points are kind of technical, we just need to do this or the network will not run.

The last two points are far from being obvious. These choices come from a long optimization. I started with two convolutional layers and less features, but the training accuracy was plateauing, which is a sign that the network was underfitting . This means that the network did not have enough parameters to describe the training dataset. So let alone the validation dataset...

So I increased the complexity by adding a layer, and by increasing the number of extracted features in each layer until the training accuracy could reach almost 100%.

At that point, the network was overfitting : the validation accuracy was much lower than the training accuracy. So I increased the dropout rate slowly from 0.4 to 0.9 to tame overfitting, and actually reached this fairly high value. This means that before entering the last dense layer for classification, the dropout layer drops 90% of the variables from the previous stages of the network on a random basis. That's a lot!

To train a neural network, we need to use an optimizer. This tool decides how to change the parameters of the network after each batch to minimize the distance between the network output and the truth. Among the optimizers implemented in Keras , people often choose Adam, or RMSProp, often as a matter of habit.

But in this case, these optimizers are not working well. The network very often starts training in a bad configuration very far from the optimal set of parameters, and is not able to learn. The loss starts around 8, and does not improve. As a result, the training accuracy remains at 50%, which is equivalent to a random guess.

So I fell back to Stochastic Gradient Descent (SGD). It works well, and the network learns. But the training is veeery long. And, by the way, this is the reason why people invented boosted optimizers such as Adam or RMSProp.

In all these studies, I tried to vary wildly the learning rate, to no avail.

I then read the paper about Adam, A Method for Stochastic Optimization , and decided to give a try to AdaMax, a variant of Adam:

After compiling the model, we train it on the training dataset, validating the results at the end of each epoch with the validation dataset. I'm using 10 cores of my CPU to handle the ImageDataGenerator tasks, and two GeForce GTX 1080 Ti for TensorFlow.

Each epoch should take about one minute. If it takes much more, like twenty minutes, you should make sure that you are indeed making use of your GPU. Check out your installation of the nvidia drivers and TensorFlow on Linux or Windows .

Now that the training is done, we need a way to see how the training worked. For this, we'll write a small function to plot the loss and the accuracy for both the training and validation datasets, as a function of the epoch:

And here are the results:

We can conclude the following:

• The training accuracy is not able to approach 100% because of the large dropout rate.
• The network starts overfitting at epoch 6, even with such a large dropout rate.
• The validation accuracy plateaus around 87%, which is not too bad.

We cannot increase dropout further, so we would need more data to do a better job. In the next section, we will see how to use data augmentation to generate more training images from the ones we already have. That's going to be much easier and much faster than collecting and tagging new cat and dog images.

Data augmentation ¶

Data augmentation consists in generating new training examples from the ones we already have, in such a way as to increase artificially the size of the training sample. That's very easy to do with the ImageDataGenerator. For example, we can start by randomly flipping left and right in our images:

Let's see the effect of this transformation on a given image:

You should be able to see the effect of the horizontal flip, except if you're really unlucky!

Now, let's make the transformation a bit more complex. This time, the ImageDataGenerator will flip, zoom, and rotate the images on a random basis:

We have seen that these transformations produce new images that are perfectly acceptable. So let's retrain the network with data augmentation. It is important to note that, due to the random nature of the transformations, the network will see each image only once. We can therefore expect that it will be difficult for the network to overfit.

With data augmentation, there is no need for a large dropout rate anymore. I reduced it from 0.9 to 0.2 but I haven't tried to tune this parameter:

As you can see, with data augmentation, the training takes longer, but overfitting is much reduced, and we could reach a classification accuracy of 92%.

We could keep tuning the network to limit overfitting by maybe increasing a bit the dropout rate, and train more. But I guess we're not going to be able to reach 95% on this dataset.

Fortunately, there are other ways, as we're going to see.

Using a pre-trained model : ResNet50 ¶

Many clever people have been working on image recognition, with powerful hardware and very large datasets. I don't know about you, but I don't have that.

However, what we can do is to use their networks directly. Such networks are called pre-trained models . Because these models could be trained on very large datasets, they usually have a deep and complex architecture, and are extremely powerful. These models can be downloaded with their parameters set to the values they had at the end of the training. In this way, we can easily get excellent performance without training a new model ourselves.

Here is the list of pre-trained models available in keras .

I decided to use ResNet50 , a model trained on the ImageNet dataset, which contains 14 million images belonging to 1000 categories.

First, we download and build the model in just a few lines of code:

Then, we define a small utility function to evaluate the model on an input image, and we call this function on a couple pictures in our dataset:

As you can see, the network gives dogs in the top categories, and is even to predict, to a large extent, the race of the dog!

What about cats?

Here is does not work that well. The top category is television, presumably because of the low quality of the image. And then we get dogs that are of this color, and finally a cat. For the next picture, the network is behaving in a better way:

Now let's try with dog cartoons:

The top-probability category is indeed a dog! ... closely followed by a chain saw. Now what about the cover picture of this blog post?

The network did not get fooled by the cosplay: it's a dog!

Ok we had fun with ResNet50, but the network is not doing exactly what we want, which is to classify pictures as dog or cat. To do this with a model based on ImageNet, we would need to be able to find out that a fine-grained category such as Great_Dane actually belongs to the coarser dog category. Until we do that, we cannot quantify the performance of this model in the context of our problem.

We could do that by hand, but we'll instead learn a more elegant solution in the next article.