Ggtcf

I have met two ways of standardization before feeding data into the TensorFlow model.
The first way is using tf.dataset.per_image_standardization().
This function computes mean and stddev for each image individually. I find this way in the official TensorFlow resnet cifar10 tutorial.
https://github.com/tensorflow/models/tree/master/official/resnet
In the testing phase, each image is standardized individually.

The second way is computing the mean and stddev of the whole dataset in the per channel style. I find this way at the following densenet implementation.https://github.com/taki0112/Densenet-Tensorflow
In testing phase, the test dataset is also preprocessed as whole batch.

These two standardization ways are not equivalent.
My question is: for the second standarization method, how to preprocess a single image for inference? What mean and stddev we should use? Do we need to use the mean and stddev computed for the training dataset as that in batch normalization?

Yes, you should use the mean and std computed from the training phase.

In general, there are 2 approaches for normalization. Let's say we have an input X of shape [B, H, W, C]

• The per feature approach normalizes every point of the image separately. For this to be done, matrices of shape [H, W, C] that estimate mean and std per feature must be computed at training phase.


• The per channel approach normalizes every channel of the image separately. This can be done in 3 ways:
  
  
  • Compute mean and std per channel across training set
  
  
  • Get statistics from a big collection of images and use these at evaluation phase (e.g. imagenet: 'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225])
  
  
  • Normalize each channel on the fly. Compute mean and std of each example (testing phase) or each batch (training phase) and normalize each channel separately.
  
  

The majority of models uses the "per channel" approach, but there is not a correct answer. The important thing is to be consistent between training and test phase. Check also here for more details.

edit: For transfer learning purposes the best choice is to gradually adopting to new dataset statistics. Hence, init your statistics from the old dataset and throughout finetuning update them with the ones from the new dataset. In the end of the training phase, mean and std must have adjusted to the new dataset.