Add ConvNeXt models

[sayakpaul]

Closes #16321

Conversion scripts and ImageNet-1k evaluation are available here: https://github.com/sayakpaul/keras-convnext-conversion.

Comparison to the actual reported numbers

name	original acc@1	keras acc@1
convnext_tiny_1k_224	82.1	81.312
convnext_small_1k_224	83.1	82.392
convnext_base_21k_1k_224	85.8	85.364
convnext_large_21k_1k_224	86.6	86.36
convnext_xlarge_21k_1k_224	87.0	86.732

@LukeWood

[fchollet]

Thanks for the PR! This is a great addition to applications.

Comments may not be exhaustive.

[sayakpaul]

Once the implementation looks good, I will add the other components. Please find the model conversion and the evaluation details in this comment: #16421 (comment).

By "conversion" I mean the following:

• I first implemented the models in Keras.
• I then populated them with the pre-trained parameters.

Implementation correctness needs to be ensured in these cases, hence the evaluation.

[sayakpaul]

I have also converted the ImageNet-21 checkpoints which would likely be better for transfer learning than the checkpoints from ImageNet-1k pre-training.

But to add those checkpoints we need the following:

• ImageNet-21k models are supposed to be multi-label classifiers. So the activation should be "sigmoid". So when weights="imagenet21k" && include_top=True, classifier_activation is supposed to be sigmoid.

This would require changes to imagenet_utils.validate_activation. As I understand it, it only supports softmax at the moment.

[sayakpaul]

IMO, a better idea would be to keep the PR as it is. Once it's done, we could work on another PR setting up validation for sigmoid when loading pre-trained models like the one mentioned above. After that, I'm happy to work on another PR to incorporate the ImageNet-21k checkpoints making the changes necessary.

[fchollet]

Sounds good, it's fine to only have support for imagenet1k for now (this is consistent with the other applications). We can add more checkpoints in the future.

[fchollet]

Sounds good, it's fine to only have support for imagenet1k for now (this is consistent with the other applications). We can add more checkpoints in the future.

[sayakpaul]

@fchollet just incorporated the changes you asked for.

These changes will require a re-conversion of the pre-trained parameters since the model structure has been changed a bit now. I will do that (as well as the ImageNet-1k evaluation) after others have had a chance to review the PR.

[LukeWood]

Looks good to me - just need to figure out the right end action per Francois' comments and add tests when that is done. Thanks @sayakpaul for the contribution!

[LukeWood]

Performed a more thorough pass to get this moving along. A few minor changes, then a big question regarding normalization that I think @fchollet will have context on.

[LukeWood]

So, these are initialized based on imagenet: this is required for use with the pretrained weights. Is there a way we can allow users to configure this for custom datasets?

@fchollet

[sayakpaul]

It's inspired from ResNet-RS and RegNets. For non-ImageNet weights, it's necessary to disable it.

[LukeWood]

We will also need to add a test case to applications_test.py

[sayakpaul]

@LukeWood with the current setup, we have a problem.

https://github.com/sayakpaul/keras/blob/feat/convnext/keras/applications/applications_test.py#L129 will fail being unable to get instantiated from the config.

https://github.com/sayakpaul/keras/blob/feat/convnext/keras/applications/applications_test.py#L134-#L137 does not help.

Error trace (used separately to test the component separately):

Traceback (most recent call last):
  File "convert.py", line 249, in <module>
    main(args)
  File "convert.py", line 111, in main
    reconstructed_model = convnext_model_tf.__class__.from_config(config)
  File "/Users/sayakpaul/.local/bin/.virtualenvs/pytorch/lib/python3.8/site-packages/keras/engine/functional.py", line 708, in from_config
    input_tensors, output_tensors, created_layers = reconstruct_from_config(
  File "/Users/sayakpaul/.local/bin/.virtualenvs/pytorch/lib/python3.8/site-packages/keras/engine/functional.py", line 1326, in reconstruct_from_config
    process_layer(layer_data)
  File "/Users/sayakpaul/.local/bin/.virtualenvs/pytorch/lib/python3.8/site-packages/keras/engine/functional.py", line 1308, in process_layer
    layer = deserialize_layer(layer_data, custom_objects=custom_objects)
  File "/Users/sayakpaul/.local/bin/.virtualenvs/pytorch/lib/python3.8/site-packages/keras/layers/serialization.py", line 207, in deserialize
    return generic_utils.deserialize_keras_object(
  File "/Users/sayakpaul/.local/bin/.virtualenvs/pytorch/lib/python3.8/site-packages/keras/utils/generic_utils.py", line 679, in deserialize_keras_object
    deserialized_obj = cls.from_config(
  File "/Users/sayakpaul/.local/bin/.virtualenvs/pytorch/lib/python3.8/site-packages/keras/engine/training.py", line 2641, in from_config
    functional.reconstruct_from_config(config, custom_objects))
  File "/Users/sayakpaul/.local/bin/.virtualenvs/pytorch/lib/python3.8/site-packages/keras/engine/functional.py", line 1325, in reconstruct_from_config
    for layer_data in config['layers']:
KeyError: 'layers'

I am currently trying to wrap LayerScale as a separate layer so that ConvNeXtBlock class could be turned into a nested function such as done in RegNets (example).

If this works out well, then the problem is solved otherwise we'll have to brainstorm more.

[sayakpaul]

@LukeWood I think I was able to make things work. I ran bazel test keras/applications/applications_test and it's passing successfully (Colab Notebook).

Here's what changed:

LayerScale has now become a layer so that we can stay with the Functional API. This simplifies the model design and stays in line with the other Keras applications.

One thing I couldn't understand is that without this with context, the test fails. It should have also complained about the StochasticDepth layer since it's also custom. Both these custom layers have get_config() overridden. So, I'm not sure if I defined the get_config() for LayerScale in the wrong way.

Let me know your thoughts on the recent changes.

P.S.: Weight conversion and verification have been performed on ImageNet-1k as well and they are all good. Refer here.

[LukeWood]

Hey @sayakpaul ! I uploaded the weights to our bucket. Now we can update the path in your code and merge the PR. Thanks!

[sayakpaul]

@LukeWood done. Thank you!

[AdityaKane2001]

This is great to have in keras.applications!

[LukeWood]

This is great to have in keras.applications!

super excited to have it in keras.applications!