Thanks for the reply and for looking into it.
Looks like app.py:1390 is throwing the exception. But maybe tensorflow-related. Not sure.
Here’s the log output:
[14/Jul/2021 21:11:41] "OPTIONS /models/ HTTP/1.1" 200 0
[14/Jul/2021 21:11:41] "POST /models/ HTTP/1.1" 201 235
2021-07-14 21:11:41,539 - INFO - app.py:1884 - Request to endpoint 'model_recommendations' took 0.00010570000000598156s
INFO:perceptilabs.applogger:Request to endpoint 'model_recommendations' took 0.00010570000000598156s
2021-07-14 21:11:41.577228: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-07-14 21:11:41.584335: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
WARNING:tensorflow:AutoGraph could not transform <bound method ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader.call of <perceptilabs.data.pipelines.image.ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader object at 0x000001D5952EB908>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING:tensorflow:AutoGraph could not transform <bound method ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader.call of <perceptilabs.data.pipelines.image.ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader object at 0x000001D5952EB908>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
2021-07-14 21:11:41.611886: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-07-14 21:12:09.570584: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
WARNING:tensorflow:AutoGraph could not transform <bound method ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader.call of <perceptilabs.data.pipelines.image.ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader object at 0x000001D5954BB548>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING:tensorflow:AutoGraph could not transform <bound method ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader.call of <perceptilabs.data.pipelines.image.ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader object at 0x000001D5954BB548>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
2021-07-14 21:12:12,014 - INFO - views.py:83 - Building dataset pipelines took 30.4439943 seconds
INFO:perceptilabs.applogger:Building dataset pipelines took 30.4439943 seconds
2021-07-14 21:12:16,078 - ERROR - app.py:1390 - Error in request 'model_recommendations'
ERROR:perceptilabs.applogger:Error in request 'model_recommendations'
2021-07-14 21:12:16,085 - INFO - app.py:1884 - Request to endpoint 'model_recommendations' took 34.5365257s
INFO:perceptilabs.applogger:Request to endpoint 'model_recommendations' took 34.5365257s
[14/Jul/2021 21:12:16] "OPTIONS /models/6/ HTTP/1.1" 200 0
[14/Jul/2021 21:12:16] "DELETE /models/6/ HTTP/1.1" 204 0
[14/Jul/2021 21:12:16] "GET /projects/ HTTP/1.1" 200 269