Expand the model zoo (example model set)

[nudles]

SINGA has multiple example models at http://singa.apache.org/docs/examples/
Some are implemented from scratch and some are converted from ONNX, which has a bigger model zoo https://github.com/onnx/models.
The task is to convert more onnx models and implement some popular (and interesting) models that are not in onnx model zoo.
Here are some reference model zoos https://modelzoo.co/, https://gluon-nlp.mxnet.io/model_zoo/index.html

[nudles]

Examples (To be updated):
MobileNetV2
ShuffleNetV2
YOLO
Mask RCNN
Faster RCNN
InceptionNetV3

[Shashankwer]

Will Start working with MobileNetV2, InceptionNetV3

[Alvinnyk]

I will work on Mask RCNN

[joddiy]

Hi, guys, @Shashankwer , @Alvinnyk , thanks for your help.

Before you start, please check this Jira, https://issues.apache.org/jira/projects/SINGA/issues/SINGA-509?filter=allopenissues

It lists the models which require support and for some of the models, SINGA may lack the necessary operators now.

So I hope you:

1. check the Jira carefully, to see which model we can do now(I guess is: super_resolution, ResNet101_DUC_HDC, ShuffleNet_V2, ShuffleNet_V1, inception_v2, squeezenet).
2. find these models in ONNX model zoo, download them and open them by a tool called Netron, try to understand the model and check each part refer the IR doc of ONNX.
3. try to read the code of SingaRep and SingaBackend in python/singa/soonx.py to understand how we parse and construct the SINGA model.
4. try to read the code examples/onnx, to make out how we load and do the inference for a ONNX model.
5. then you can try to support a model(in first point) by yourself.

[agnesnatasya]

I will try to work on the ShuffleNet

[agnesnatasya]

Hi, may I ask how do we get the Amazon AWS link to download the model itself? For example, for ResNet, the link is https://s3.amazonaws.com/onnx-model-zoo/resnet/resnet18v1/resnet18v1.tar.gz. How do we get the link for the others? Because if I copy the link from ONNX's github, the link to download is Github's link, not Amazon AWS link. Thank you!

[Shashankwer]

For the models conversion do we also nee to train model on SINGA after conversion from onnx, and have a comparison matrix?

For example super_resolution via subpixel convolution layer was trained on BSD 300. Model is originally present in pytorch and can be converted well into SINGA using ONNX. It gives similar result for an image across pytorch, onnxruntime and SINGA. Do we need to compare the performance of the model across these platforms with the dataset they were trained on or SINGA compatibility is what we are checking at the moment?

Thanks and Regards,
Shashank Nigam

[nudles]

There is no need to retrain the model.
But we need to evaluate the model over a test dataset to make sure the conversion is correct.

[nudles]

Hi, may I ask how do we get the Amazon AWS link to download the model itself? For example, for ResNet, the link is https://s3.amazonaws.com/onnx-model-zoo/resnet/resnet18v1/resnet18v1.tar.gz. How do we get the link for the others? Because if I copy the link from ONNX's github, the link to download is Github's link, not Amazon AWS link. Thank you!

https://github.com/onnx/models
I think you need to go to the github page for each model and then find the link for the model checkpoint file, which may not all be on aws.
@joddiy any comments?

[joddiy]

Hi, may I ask how do we get the Amazon AWS link to download the model itself? For example, for ResNet, the link is https://s3.amazonaws.com/onnx-model-zoo/resnet/resnet18v1/resnet18v1.tar.gz. How do we get the link for the others? Because if I copy the link from ONNX's github, the link to download is Github's link, not Amazon AWS link. Thank you!

Hi, @agnesnatasya , it's not necessary to use the AWS download link. As you said, for the image classification models, I found the download link from the backend test cases here. But if for other models, you can use the download link within GitHub repo directly.

[joddiy]

For the models conversion do we also nee to train model on SINGA after conversion from onnx, and have a comparison matrix?

For example super_resolution via subpixel convolution layer was trained on BSD 300. Model is originally present in pytorch and can be converted well into SINGA using ONNX. It gives similar result for an image across pytorch, onnxruntime and SINGA. Do we need to compare the performance of the model across these platforms with the dataset they were trained on or SINGA compatibility is what we are checking at the moment?

Thanks and Regards,
Shashank Nigam

For this version of SINGA, we don't need to consider the matter of training(or retaining), we just need to test if we can run the model without errors, and the result is correct. In the next version(what we are doing now), the ONNX of SINGA will support retraining.

So for now, we just need to do three things:

1. load the model correctly without errors.
2. verify the model by using the test dataset within the model download package. If you Download (with sample test data), you can see a directory called test_data_set_0, and you can verify the result of the model by using these test data.
3. for demo, we need to do the inference by using a human-readable data(or image), so you need to find a public image and then use the model to do the inference, finally show the result.

[agnesnatasya]

@joddiy Thank you for the reply, I could find the link to download, and I have completed SqueezeNet. May I ask how do I test if my conversion is already correct? Thank you!

[joddiy]

@joddiy Thank you for the reply, I could find the link to download, and I have completed SqueezeNet. May I ask how do I test if my conversion is already correct? Thank you!

you can use this code to verify the model by using its test data set:

    # verifty the test dataset
    from utils import load_dataset
    inputs, ref_outputs = load_dataset(
        os.path.join('/tmp', 'resnet100', 'test_data_set_0'))
    x_batch = tensor.Tensor(device=dev, data=inputs[0])
    outputs = model.forward(x_batch)
    for ref_o, o in zip(ref_outputs, outputs):
        np.testing.assert_almost_equal(ref_o, tensor.to_numpy(o), 4)

[Shashankwer]

Hi,

Can anyone verify if for SuperResolutionNet, attached notebook would be sufficient?

Thanks and Regards,
Shashank Nigam
SuperResolution.zip