Looking for feedback - We are aiming to build a serving feature

Feature Requests
#1

Hello everyone who reads this

We are looking to start building out a feature in the tool which will let you use your models easily.
However, there are a few different options on how this can be built out and I would love any opinion on what would work best for you :slight_smile:

The four different options (so far) are:

  • Run data through the tool using a CSV file
  • Start serving an endpoint you can send data to
  • Run Gradio (or a similar app)
  • Export a serving script

I’ll go through and describe them briefly below.

Run data through the tool using a CSV
This variation is pretty simple; you select what model you want to run, then you select what dataset you want to run through the model and finally you select where you want to export the data and press Run.
The model will now run through the new dataset and will produce a folder will all the predictions in them in a relevant format.
The dataset you select to run through the model can be a CSV file which has not been used in the tool before, as long as the columns match with the ones in the model.

Start serving an endpoint you can send data to
In this variation, you would like before select what model you want to run, but that’s everything.
After you have selected the model and pressed Run, you would get an IP address which could look something like 127.0.0.1:8500 (not necessarily a local address).
Now you can send POST requests to this address with data (images for example) inside them and it will respond with the predictions. You can send both one image at a time or many at once.

Run Gradio (or a similar app)
Here we would utilize an OpenSource tool like Gradio (https://gradio.app/) to create an app for your model.
Like before, you would only need to select what model you would want to run, but this time it would open a Gradio app (probably as a new tab in your browser) that you can interact with.
One of those apps could look something like this:

image1029Ă—735 260 KB

This would have some restrictions though, such as:

  • It seems like you can only run one sample (one image for example) through it at once.
  • It would be limited to Image Classification and Segmentation to start with.

Export a serving script
This would work very similar to “Start serving an endpoint you can send data to”, except instead of starting it automatically for you, we just give you the script and the model so you can run it where you wish to.

.

Thanks for reading as well as any opinions on what you would like to see the most!
All the best,
Robert

1 Like
#2

Run data through the tool using CSV requires nothing that the user doesn’t already know how to do and has no dependencies, so that would be my #1 choice since it should be able to handle e.g. multiple images specified in a single CSV. CSV also has the advantage that for non-image models pure data can be input.

#2 would be serving an endpoint that data can be posted to… provided there is a minimal HTML page with it to select e.g. a source file to post. But what if there is just “data” (another row of parameters from a table, sheet in excel…) from which a prediction should be made?

I don’t have a view on gradio per se; export a serving script is viable for users with sufficient skills (but see #1 above)

Question: suppose you are in a “professional” (corporate, academic, govt) environment with internal wifi… is there any option that would allow a user to set up a model and then walk to his boss’s/colleague’s office and show them the result on the user (or the boss’s?) mobile/cell?

Given the data wizard setup that already exists, CSV has most flexibility, lowest barrier to use, fewest dependencies - and with current screen-sharing capabilities even showing the boss/colleague remotely would not be a problem… just not quite as cool as showing them on their phone.

PS But speaking of phone… image processing that could receive input from the phone camera… wow… production line quality checking, diagnostic; taking input from another computer screen (so some capability even if the “raw” data can’t be handled.

Oooh! PyDroid anyone?? It says it even supports TF!

2 Likes
#3

i’d like to see “Export a serving script” as a priority, because it would allow me to automate my code. I don’t like to manually click through a form. That would also make “Start serving an endpoint you can send data to” me second choise.

The other ones are not of much interest for me.

Thanks for the feedback request and your great work!

1 Like
#4

Thank you both for the feedback so far!

To answer some of @JulianSMoore’s questions;
For the serving endpoint, there may be some minimal HTML page, but more likely than not we won’t invest time in it for the first version, as the Gradio solution just does that better. Instead, the endpoint would be used with a script that may look something like this:

import requests
import json
import os
from PIL import Image
import numpy as np
def load_image(path):
    image = Image.open(path)
    image = np.array(image, dtype=np.float32)
    image = np.expand_dims(image, axis=0)
    return image
defected = load_image("C:/Users/Robert/Documents/PerceptiLabs/Default/pills/defect/pic.6.571.0.png")
data = json.dumps({"signature_name": "serving_default", "instances": defected.tolist()})
headers = {"content-type": "application/json"}
import pdb; pdb.set_trace()
json_response = requests.post('http://localhost:8501/v1/models/my_model:predict', data=data, headers=headers)
predictions = json.loads(json_response.text)['predictions']
print(predictions)

And using this (or similar methods, you could use Postman instead as well) you would be able to send arbitrary data to the endpoint. This allows you to hook an app up to this endpoint to get continuous predictions for example.

For you question on walking to the boss office, that sounds exactly like Gradio (maybe minus the phone part and instead just enter the address on a laptop) and is what Gradio is used for a lot.

PyDroid looks pretty cool! Maybe we will look at adding some export option for that as well at some point :slight_smile:

As it looks right now, we may choose 1 “easy to use” solution, which would be either Gradio or “Run data through the tool”. And then 1 “production” solution which would be starting an endpoint or exporting a script. We would in that case build them out in close succession so that both cases are covered.

2 Likes
#5

Update here, it looks like we will start with these 2:

  • Gradio
  • Export serving script

As Gradio is a great prototyping/showcasing tool, while exporting the serving script just is a more flexible version of running the serving script. The serving will serve up a REST endpoint that you can send POST requests to like described above.
Depending on the feedback on those versions we may quickly implement some other alternatives as well :slight_smile: