Import a custom model to percepti

[tito]

Hi all, I am loading a custom model into percepti and facing a few errors:

Error 1:
I loaded a new object and used keras load_model(path/to/model) however I am getting the below errors , I loaded a pth and a pt files types: looking into the code of other objects it seems I should also add some "template code such as build and get_conf etc… " to build the object itself so I did that also:

LayerCustom_LayerCustom_1Keras(tf.keras.layers.Layer, PerceptiLabsVisualizer):
def call(self, inputs, training=True):
""" Takes a tensor and one-hot encodes it """

raise TypeError("Missing input connection 'input'")


input_ = inputs['input']
input_ = self.preprocess_input(input_)
#output = preview = input_['input']
output = self.Model(input_, training=False)

self._outputs = {
'output': output,
'preview': output,
}
return self._outputs

def load_model():
model = keras.models.load_model('/home/user/Desktop/Models Trained/Model-C10.pth')
model = get_model()
return model def get_config(self):
"""Any variables belonging to this layer that should be rendered in the frontend.

Returns:
A dictionary with tensor names for keys and picklable for values.
"""
return {} @property
def visualized_trainables(self):
""" Returns two tf.Variables (weights, biases) to be visualized in the frontend """
return tf.constant(0), tf.constant(0) class LayerCustom_LayerCustom_1(Tf2xLayer):
def __init__(self):
super().__init__(
keras_class=LayerCustom_LayerCustom_1Keras
) t rac Errors:
File "perceptilabs/lwcore/strategies/tf2x.py", line 36, in perceptilabs.lwcore.strategies.tf2x.Tf2xInnerStrategy._run_internal
File "perceptilabs/lwcore/strategies/tf2x.py", line 54, in perceptilabs.lwcore.strategies.tf2x.Tf2xInnerStrategy._make_results
File "perceptilabs/lwcore/strategies/tf2x.py", line 45, in perceptilabs.lwcore.strategies.tf2x.Tf2xInnerStrategy._make_results
File "perceptilabs/layers/legacy.py", line 19, in perceptilabs.layers.legacy.Tf2xLayer.__call__
File "/home/tarek/.local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1030, in __call__
outputs = call_fn(inputs, *args, **kwargs)
File "<rendered-code: 1636785172006 [LayerCustom]>", line 5, in call

@robertl advised me to remove this line of code:

It looks like this line of code still exist in your component, you might want to remove it:
`raise TypeError("Missing input connection 'input'")` 

I removed this line of code and checked and one error gone thank you @robertl,

Error2:
However, I am still getting one error:

Error in layer targets. Expected shape (10,) but got None

During training model is saved via pytorsch, I paste the code below:

   model=utils.get_model(self.model)
    torch.save({'model_state_dict': model,
                }, os.path.join(self.checkpoint, 'model_{}.pth.tar'.format(task_id)))

My question is if I am saving the model in this way Iwill affects on how I am loading the model in my case via keras?
As far as I know that if i saved the model via keras API I should load it via keras, and If i save it via tensorflow I should loaded via tensorflow API I read this info on this documentation:
https://www.tensorflow.org/tutorials/distribute/save_and_load

And in your advise this will be reason that it is failing to be loaded properly in pertcepti or there is another reason?
Thank you for your assistance in advance

[birdstream]

You are trying to load a pytorch model? That wont work, you need to convert it first.
Check this out: pytorch2keras

Then, to use that model as a component in Perceptilabs you should be able to use hub.KerasLayer to wrap the model in a Keras layer (look at some examples using tf hub in Perceptilabs like Using the new ViT from TF Hub )

[JulianSMoore]

Thanks @birdstream for the pytorch2keras info; I wasn’t aware of that - anything that opens up more sources to exploit is good!

And welcome @tito I’m looking forward to seeing what comes of this - especially since you dived right in to custom layers! I expect you have an interesting dev/ml background to share sometime

[birdstream]

EDIT: Did a few adjustments in the code! (like importing numpy)

Just some additional thoughts here

Looking closer at your post it seems to me as in the pytorch saving part you’re saving the models weights & biases rather than the whole model? For saving the entire model one should only have to do

torch.save(model, PATH)

Converting the model should be something in the likes of:

import torch
from torch.autograd import Variable
import numpy as np
from pytorch2keras import pytorch_to_keras
model_path = 'path/to/pytorch/model.pt'
# load the pytorch model
pytorch_model = torch.load(model_path)
# create dummy variable with the correct shape
input_np = np.random.uniform(0, 1, (1, 10, 32, 32))
input_var = Variable(torch.FloatTensor(input_np))
# we should specify shape of the input tensor
keras_model = pytorch_to_keras(pytorch_model, input_var, [(10, 32, 32,)], verbose=True, change_ordering=True, name_policy="keep")
# Save the keras model. This will create a directory in the specified location from which we'll load the model later.
keras_model.save('path/to/model')

Then, to use the model in Perceptilabs we should be able to leverage hub.KerasLayer to wrap the model in a layer. The CustomLayer should look like:

import tensorflow_hub as hub
class LayerCustom_LayerCustom_1Keras(tf.keras.layers.Layer, PerceptiLabsVisualizer):
    def call(self, inputs, training=True):
        """ Takes a tensor and one-hot encodes it """
        input_ = inputs['input']
        output = self.custom_model(input_)
        self._outputs = {            
            'output': output,
            'preview': output,
        }
        return self._outputs
    def build(self, input_shape):
        self.custom_model = hub.KerasLayer('path/to/model')
    def get_config(self):
        """Any variables belonging to this layer that should be rendered in the frontend.
        Returns:
            A dictionary with tensor names for keys and picklable for values.
        """
        return {}
    @property
    def visualized_trainables(self):
        """ Returns two tf.Variables (weights, biases) to be visualized in the frontend """
        return tf.constant(0), tf.constant(0)
class LayerCustom_LayerCustom_1(Tf2xLayer):
    def __init__(self):
        super().__init__(
            keras_class=LayerCustom_LayerCustom_1Keras
        )

Note the import of tensorflow_hub at the beginning. So you need to install tensorflow_hub in the same environment as you’re running Perceptilabs:

pip install tensorflow_hub

I can’t garantuee that the above will work right out of the box, because I’ve personally haven’t tried it out (yet)
And, seconding @JulianSMoore here, I’d love to hear how it turns out

[tito]

Hello @birdstream, what you are proposing makes sense to me, let me test it / adjust and I will update you on the outcome thank you so much for your reply. You are 100% right on the saving (original model) I am facing some issues once I am loading the model on other platforms your guidance is very helpful cheers , @JulianSMoore thank you so much for your warm welcoming, I will let you know how it will go for sure plus share my experience, keep in mind that I am still on the learning track but with a very enthusiastic interest and approach once it comes to ML

[JulianSMoore]

@birdstream’s step-by-step advice on converting etc. deserves to be in a FAQ somewhere - just the sort of thing that others would really benefit from and should be able to find easily.

@tito I’ve just discovered that there’s space in the profile pages for us to say a bit more about ourselves (not exactly prominent, but it is there).

See e.g. https://forum.perceptilabs.com/u/YOURNAME/preferences/profile

I’ll share some stuff about me and look forward to finding out a bit more about some of the other active users here

[birdstream]

You’re welcome Please note that i did some adjustments to the code (import numpy, fixed so NCHW gets reordered to NHWC and such)

Also a tip along the way is later versions of onnx does not work with pytorch2keras, so you’ll need to downgrade it to 1.8.0

pip install --upgrade onnx==1.8.0

Of course, I couldn’t resist actually trying this myself… and ta-daa!

I grabbed the pytorch ResNet18 model with pretrained weights from torch hub and converted it to a TensorFlow model for use in PerceptiLabs

[robertl]

Awesome @birdstream!
As per @JulianSMoore’s suggestion we did also add this thread into the FAQ: https://docs.perceptilabs.com/perceptilabs/support/perceptilab-faqs#how-do-i-convert-a-pytorch-model-to-tensorflow
We’ll probably do some tutorial video on it as well, a few people has asked about loading existing models (and a lot about PyTorch) in the past

[tito]

That is a great Idea @JulianSMoore I will add my profile there looking forward to read yours and other members:)

[tito]

@birdstream I added this def to my code and training it again. I will take the first one saved on the first task. I forgot to mention that I am training a multitask model:
This is what I did:
on fires
def save_model(self, task_id):
print(“Saving all models for task {} …”.format(task_id + 1))
torch.save(model, os.path.join(self.checkpoint, ‘model_NEW{}.pth’.format(task_id)))

And I have now a new saved model. Just to confirm this is the way that we should be saving as a whole model looks good for you? I can see that the size is a bit different between my two saved outputs / models but in term of structure archive it is the same, I will know for sure when I load it again in percepti.

[tito]

Hi @birdstream @JulianSMoore these are my steps, I took my model output and working on converting from PyTorch to Keras:

INFO:pytorch2keras:Converter is called.
WARNING:pytorch2keras:Name policy isn’t supported now.
WARNING:pytorch2keras:Custom shapes isn’t supported now.
keras_model = pytorch_to_keras(pytorch_model, input_var, [(10, 32, 32,1)], verbose=True, change_ordering=True, name_policy=“keep”)
TypeError: forward() missing 3 required positional arguments: ‘x_p’, ‘tt’, and ‘task_id’

hmm thank you gents in advance

[birdstream]

Hmm, i’m not entirely sure, but it may be that the model you’re saving is in training mode…?
try model.eval() prior to saving it (and to continue training you must set it back to training with model.train()
At least this has to be done according to the docs. I found another good guide here

Hope this helps

PS. I’ve updated my post to properly use the converted model in PL (the old version i wrote surely worked in the model view but would throw errors if trying to actually train ) Thanks again, @mukund_s @robertl

[JulianSMoore]

Hi @birdstream

Thanks for the .eval/.train info: I need to train for an epoch, run predictions, and then repeat for n epochs… I was wondering about how to do that - thanks for the timely info.

(Why do I want to run predictions after each epoch? To evaluate the progression of convergence/generalisation, except I don’t want to have special “validation” data - I want that to be standard so I need another input path )

[robertl]

@JulianSMoore, one trick within PL for wanting to evaluate each epoch independently, there’s a little checkbox in the training settings called “Save checkpoint every epoch” that will do just as it’s called, which means that you later can load the checkpoints for each epoch and run the model in that “state” for deeper analysis

[JulianSMoore]

That’s interesting. I’d like to chat about the nuts-and-bolts of that offline…

[birdstream]

That sounds interesting what kind of model are you training? I guess it’s pytorch related?

[birdstream]

Now that we’re on the subject: When i was doing pytorch i usually had my app making predictions on a specific image/set of images every x iteration to see how the model was doing for that particular image. I think that could be a nice addition to PL

[robertl]

@birdstream, thanks for the suggestion, and I agree That would give a nice baseline to check against as you are training to progressively see if it’s improving (quantitatively rather than the usual qualitative metrics such as Accuracy).
Added a feature request for it here: https://perceptilabs.canny.io/feature-requests/p/i-want-to-have-the-option-to-see-how-the-model-predicts-on-a-few-samples-every-x