Error creating model with image as input & target

[ThatDude]

Hi!

Came across this error when doodling around with PL:

Couldn't get model recommendation due to: Error: Request failed with status code 500

Input/target is simply one image, a white 64x64 jpg.

Here the output of verbosity level 3:
(pl) C:\Users[REDACTED]>perceptilabs -v=3
2021-06-28 14:53:29.609642: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-06-28 14:53:32.624122: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-06-28 14:53:32.625081: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll
2021-06-28 14:53:32.641182: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 1080 computeCapability: 6.1
coreClock: 1.771GHz coreCount: 20 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 298.32GiB/s
2021-06-28 14:53:32.641367: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-06-28 14:53:32.649951: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-06-28 14:53:32.650114: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-06-28 14:53:32.653993: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-06-28 14:53:32.655259: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-06-28 14:53:32.663553: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-06-28 14:53:32.667420: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-06-28 14:53:32.668321: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-06-28 14:53:32.668533: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
Operations to perform:
Apply all migrations: admin, api, auth, contenttypes, sessions
Running migrations:
No migrations to apply.
PerceptiLabs: Starting
2021-06-28 14:53:48.351326: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-06-28 14:53:48.360003: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
Performing system checks…

Performing system checks...
System check identified no issues (0 silenced).
June 28, 2021 - 14:53:49
Django version 3.2, using settings 'static_file_server.settings'
Starting development server at http://127.0.0.1:8080/
Quit the server with CTRL-BREAK.
System check identified no issues (0 silenced).
June 28, 2021 - 14:53:49
Django version 3.2, using settings 'fileserver.settings'
Starting development server at http://127.0.0.1:8011/
Quit the server with CTRL-BREAK.
Performing system checks...
System check identified no issues (0 silenced).
June 28, 2021 - 14:53:49
Django version 3.2, using settings 'rygg.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.
[28/Jun/2021 14:53:50] "GET /?token=jNo3mvx2ssyocDQcuQZXFoZnBZWGv0ykyOraBRI8rto HTTP/1.1" 200 493
[28/Jun/2021 14:53:50] "GET /static/styles/vendors.df1267d6ff8457d42086.css HTTP/1.1" 304 0
[28/Jun/2021 14:53:50] "GET /static/js/0.65773a032b9e270e374c.js HTTP/1.1" 304 0
[28/Jun/2021 14:53:50] "GET /static/js/app.787e50e4dbab3976d321.js HTTP/1.1" 304 0
[28/Jun/2021 14:53:51] "GET /fileserver_url HTTP/1.1" 200 0
[28/Jun/2021 14:53:51] "GET /rendering_kernel_url HTTP/1.1" 200 0
[28/Jun/2021 14:53:51] "GET /rygg_url HTTP/1.1" 200 0
[28/Jun/2021 14:53:51] "GET /kernel_url HTTP/1.1" 200 0
[28/Jun/2021 14:53:51] "GET /keycloak_url HTTP/1.1" 200 0
[28/Jun/2021 14:53:51,301] - Broken pipe from ('127.0.0.1', 51621)
2021-06-28 14:53:51.649209: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-06-28 14:53:51.649919: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll
2021-06-28 14:53:51.662292: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-06-28 14:53:51.662986: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll
2021-06-28 14:53:51.665498: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 1080 computeCapability: 6.1
coreClock: 1.771GHz coreCount: 20 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 298.32GiB/s
2021-06-28 14:53:51.665739: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-06-28 14:53:51.670744: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-06-28 14:53:51.671199: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-06-28 14:53:51.676110: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-06-28 14:53:51.677194: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-06-28 14:53:51.679577: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 1080 computeCapability: 6.1
coreClock: 1.771GHz coreCount: 20 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 298.32GiB/s
2021-06-28 14:53:51.681229: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-06-28 14:53:51.683516: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-06-28 14:53:51.685626: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-06-28 14:53:51.686798: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-06-28 14:53:51.687092: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-06-28 14:53:51.688027: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-06-28 14:53:51.688518: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-06-28 14:53:51.693013: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-06-28 14:53:51.694052: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-06-28 14:53:51.700733: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-06-28 14:53:51.702918: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-06-28 14:53:51.703881: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-06-28 14:53:51.704118: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
[28/Jun/2021 14:53:51] "GET /keycloak_url HTTP/1.1" 200 0
[28/Jun/2021 14:53:52] "GET /?token=jNo3mvx2ssyocDQcuQZXFoZnBZWGv0ykyOraBRI8rto HTTP/1.1" 200 493
[28/Jun/2021 14:53:52] "GET /fileserver_url HTTP/1.1" 200 0
[28/Jun/2021 14:53:52] "GET /rendering_kernel_url HTTP/1.1" 200 0
[28/Jun/2021 14:53:52] "GET /rygg_url HTTP/1.1" 200 0
[28/Jun/2021 14:53:52] "GET /kernel_url HTTP/1.1" 200 0
[28/Jun/2021 14:53:52] "GET /keycloak_url HTTP/1.1" 200 0
[28/Jun/2021 14:53:53] "GET /keycloak_url HTTP/1.1" 200 0
[28/Jun/2021 14:53:53] "GET /static/fonts/nunito-sans-v5-latin-600.b56c7e1.woff2 HTTP/1.1" 304 0
[28/Jun/2021 14:53:53] "GET /static/fonts/nunito-sans-v5-latin-700.d0b6d0f.woff2 HTTP/1.1" 304 0
[28/Jun/2021 14:53:53] "GET /static/fonts/nunito-sans-v5-latin-regular.9c6a1ff.woff2 HTTP/1.1" 304 0
[28/Jun/2021 14:53:53] "GET /static/img/tutorial-background.ac2e517.png HTTP/1.1" 200 186377
[28/Jun/2021 14:53:53] "GET /rygg_url HTTP/1.1" 200 0
[28/Jun/2021 14:53:53] "OPTIONS /version?token=jNo3mvx2ssyocDQcuQZXFoZnBZWGv0ykyOraBRI8rto HTTP/1.1" 200 0
[28/Jun/2021 14:53:53] "GET /projects/ HTTP/1.1" 200 264
[28/Jun/2021 14:53:53] "GET /version?token=jNo3mvx2ssyocDQcuQZXFoZnBZWGv0ykyOraBRI8rto HTTP/1.1" 200 22
[28/Jun/2021 14:53:53] "GET /app/updates_available/ HTTP/1.1" 200 22
[28/Jun/2021 14:53:53] "POST /mixpanel/track/?ip=1&_=1624884833561 HTTP/1.1" 200 0
2021-06-28 14:53:53,955 - INFO - perceptilabs\__main__:4 - Started kernel in mode '{args.mode}'
2021-06-28 14:53:53,955 - INFO - perceptilabs\__main__:4 - Started kernel in mode '{args.mode}'
2021-06-28 14:53:53,955 - INFO - perceptilabs\__main__:4 - Reporting errors with commit id: 2d0477bff7e114a20241b90fc7b2edab35a9ebc4
PerceptiLabs is ready...
2021-06-28 14:53:53,983 - INFO - perceptilabs\__main__:4 - Trying to listen to: 0.0.0.0 5000
[28/Jun/2021 14:53:54] "GET /mixpanel/decide/?verbose=1&version=1&lib=web&token=1480b2244fdd4d821227a29e2637f922&ip=1&_=1624884833556 HTTP/1.1" 200 0
PerceptiLabs: PerceptiLabs Started
PerceptiLabs: PerceptiLabs is running at http://localhost:8080/?token=jNo3mvx2ssyocDQcuQZXFoZnBZWGv0ykyOraBRI8rto
PerceptiLabs: Use Control-C to stop this server and shut down all PerceptiLabs processes.
[28/Jun/2021 14:53:58] "POST /mixpanel/engage/?ip=1&_=1624884838502 HTTP/1.1" 200 0
[28/Jun/2021 14:53:58] "GET /mixpanel/decide/?verbose=1&version=3&lib=web&token=1480b2244fdd4d821227a29e2637f922&distinct_id=[REDACTED]%40gmail.com&ip=1&_=1624884838498 HTTP/1.1" 200 0
[28/Jun/2021 14:53:58] "POST /mixpanel/engage/?ip=1&_=1624884838500 HTTP/1.1" 200 0
2021-06-28 14:53:59,111 - INFO - server.py:293 - Created coreLogic for network 'None'
2021-06-28 14:53:59,155 - INFO - server.py:293 - Frontend receiver: None , Frontend request: checkVersions
2021-06-28 14:53:59,177 - INFO - server.py:293 - Frontend receiver:  , Frontend request: setUser
2021-06-28 14:53:59,177 - INFO - server.py:293 - Created coreLogic for network ''
2021-06-28 14:53:59,186 - INFO - server.py:293 - User has been set to [REDACTED]@gmail.com
2021-06-28 14:54:01,097 - INFO - server.py:293 - Frontend receiver:  , Frontend request: setUser
2021-06-28 14:54:01,098 - INFO - server.py:293 - User has been set to [REDACTED]@gmail.com
[28/Jun/2021 14:54:22] "GET /static/fonts/icomoon.1b9c004.woff HTTP/1.1" 304 0
[28/Jun/2021 14:54:22] "GET /static/img/file-picker/home.svg HTTP/1.1" 304 0
[28/Jun/2021 14:54:22] "OPTIONS /directories/get_folder_content?path=D:/tai&token=jNo3mvx2ssyocDQcuQZXFoZnBZWGv0ykyOraBRI8rto HTTP/1.1" 200 0
[28/Jun/2021 14:54:22] "GET /directories/get_folder_content?path=D:/tai&token=jNo3mvx2ssyocDQcuQZXFoZnBZWGv0ykyOraBRI8rto HTTP/1.1" 200 96
[28/Jun/2021 14:54:22] "GET /static/img/file-picker/folder.svg HTTP/1.1" 304 0
[28/Jun/2021 14:54:22] "GET /static/img/file-picker/file.svg HTTP/1.1" 304 0
[28/Jun/2021 14:54:26] "OPTIONS /files/get_file_content?path=D:/tai/tset.csv&token=jNo3mvx2ssyocDQcuQZXFoZnBZWGv0ykyOraBRI8rto HTTP/1.1" 200 0
('D:/tai/tset.csv', 4)
[28/Jun/2021 14:54:26] "GET /files/get_file_content?path=D:/tai/tset.csv&token=jNo3mvx2ssyocDQcuQZXFoZnBZWGv0ykyOraBRI8rto HTTP/1.1" 200 58
2021-06-28 14:54:26,173 - INFO - app.py:1884 - Request to endpoint 'type_inference' took 0.00010979999999705115s
INFO:perceptilabs.applogger:Request to endpoint 'type_inference' took 0.00010979999999705115s
2021-06-28 14:54:29,309 - INFO - app.py:1884 - Request to endpoint 'type_inference' took 3.134523999999999s
INFO:perceptilabs.applogger:Request to endpoint 'type_inference' took 3.134523999999999s
[28/Jun/2021 14:54:29] "GET /static/fonts/roboto-regular-webfont.736b705.woff2 HTTP/1.1" 200 20000
[28/Jun/2021 14:54:29] "OPTIONS /directories/resolved_dir?path=C:\Users\[REDACTED]\Documents\Perceptilabs\Default&token=jNo3mvx2ssyocDQcuQZXFoZnBZWGv0ykyOraBRI8rto HTTP/1.1" 200 0
[28/Jun/2021 14:54:29] "GET /directories/resolved_dir?path=C:\Users\[REDACTED]\Documents\Perceptilabs\Default&token=jNo3mvx2ssyocDQcuQZXFoZnBZWGv0ykyOraBRI8rto HTTP/1.1" 200 61
[28/Jun/2021 14:54:29] "OPTIONS /directories/get_folder_content?path=C:\Users\[REDACTED]\Documents\Perceptilabs\Default&token=jNo3mvx2ssyocDQcuQZXFoZnBZWGv0ykyOraBRI8rto HTTP/1.1" 200 0
[28/Jun/2021 14:54:29] "GET /directories/get_folder_content?path=C:\Users\[REDACTED]\Documents\Perceptilabs\Default&token=jNo3mvx2ssyocDQcuQZXFoZnBZWGv0ykyOraBRI8rto HTTP/1.1" 200 112
[28/Jun/2021 14:54:34] "OPTIONS /directories?path=C:\Users\[REDACTED]\Documents\Perceptilabs\Default/Model%201&token=jNo3mvx2ssyocDQcuQZXFoZnBZWGv0ykyOraBRI8rto HTTP/1.1" 200 0
[28/Jun/2021 14:54:34] "HEAD /directories?path=C:\Users\[REDACTED]\Documents\Perceptilabs\Default/Model%201&token=jNo3mvx2ssyocDQcuQZXFoZnBZWGv0ykyOraBRI8rto HTTP/1.1" 204 34
[28/Jun/2021 14:54:35] "POST /models/ HTTP/1.1" 201 237
2021-06-28 14:54:35,026 - INFO - app.py:1884 - Request to endpoint 'model_recommendations' took 0.00011920000000031905s
INFO:perceptilabs.applogger:Request to endpoint 'model_recommendations' took 0.00011920000000031905s
2021-06-28 14:54:35.044967: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-06-28 14:54:35.046006: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 1080 computeCapability: 6.1
coreClock: 1.771GHz coreCount: 20 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 298.32GiB/s
2021-06-28 14:54:35.046161: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-06-28 14:54:35.046575: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-06-28 14:54:35.047054: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-06-28 14:54:35.047506: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-06-28 14:54:35.047952: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-06-28 14:54:35.048456: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-06-28 14:54:35.049071: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-06-28 14:54:35.050265: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-06-28 14:54:35.050895: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-06-28 14:54:35.645714: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-06-28 14:54:35.645876: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0
2021-06-28 14:54:35.646932: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N
2021-06-28 14:54:35.647769: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6692 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1)
2021-06-28 14:54:35.651093: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-06-28 14:54:35,654 - ERROR - views.py:83 - Error getting model recommendation!
Traceback (most recent call last):
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\eager\context.py", line 2113, in execution_mode
    yield
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 733, in _next_internal
    output_shapes=self._flat_output_shapes)
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\ops\gen_dataset_ops.py", line 2578, in iterator_get_next
    _ops.raise_from_not_ok_status(e, name)
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\framework\ops.py", line 6862, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence [Op:IteratorGetNext]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 747, in __next__
    return self._next_internal()
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 739, in _next_internal
    return structure.from_compatible_tensor_list(self._element_spec, ret)
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\eager\context.py", line 2116, in execution_mode
    executor_new.wait()
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\eager\executor.py", line 69, in wait
    pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle)
tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "perceptilabs\endpoints\model_recommendations\base.py", line 22, in perceptilabs.endpoints.model_recommendations.base.ModelRecommendations.dispatch_request
  File "perceptilabs\data\base.py", line 67, in perceptilabs.data.base.DataLoader.from_dict
  File "perceptilabs\data\base.py", line 85, in perceptilabs.data.base.DataLoader.from_features
  File "perceptilabs\data\base.py", line 44, in perceptilabs.data.base.DataLoader.__init__
  File "perceptilabs\data\base.py", line 235, in perceptilabs.data.base.DataLoader._build_datasets_and_pipelines
  File "perceptilabs\data\base.py", line 110, in perceptilabs.data.base.DataLoader._create_datasets_and_pipelines_from_dataframe
  File "perceptilabs\data\base.py", line 139, in perceptilabs.data.base.DataLoader._load_and_preprocess_feature
  File "perceptilabs\data\base.py", line 169, in perceptilabs.data.base.DataLoader._build_pipelines
  File "perceptilabs\data\pipelines\image.py", line 23, in perceptilabs.data.pipelines.image.ImagePipelineBuilder.build
  File "perceptilabs\data\pipelines\image.py", line 204, in perceptilabs.data.pipelines.image.ImagePipelineBuilder._get_file_loader_pipeline
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 749, in __next__
    raise StopIteration
StopIteration
ERROR:perceptilabs.applogger:Error getting model recommendation!
Traceback (most recent call last):
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\eager\context.py", line 2113, in execution_mode
    yield
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 733, in _next_internal
    output_shapes=self._flat_output_shapes)
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\ops\gen_dataset_ops.py", line 2578, in iterator_get_next
    _ops.raise_from_not_ok_status(e, name)
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\framework\ops.py", line 6862, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence [Op:IteratorGetNext]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 747, in __next__
    return self._next_internal()
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 739, in _next_internal
    return structure.from_compatible_tensor_list(self._element_spec, ret)
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\eager\context.py", line 2116, in execution_mode
    executor_new.wait()
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\eager\executor.py", line 69, in wait
    pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle)
tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "perceptilabs\endpoints\model_recommendations\base.py", line 22, in perceptilabs.endpoints.model_recommendations.base.ModelRecommendations.dispatch_request
  File "perceptilabs\data\base.py", line 67, in perceptilabs.data.base.DataLoader.from_dict
  File "perceptilabs\data\base.py", line 85, in perceptilabs.data.base.DataLoader.from_features
  File "perceptilabs\data\base.py", line 44, in perceptilabs.data.base.DataLoader.__init__
  File "perceptilabs\data\base.py", line 235, in perceptilabs.data.base.DataLoader._build_datasets_and_pipelines
  File "perceptilabs\data\base.py", line 110, in perceptilabs.data.base.DataLoader._create_datasets_and_pipelines_from_dataframe
  File "perceptilabs\data\base.py", line 139, in perceptilabs.data.base.DataLoader._load_and_preprocess_feature
  File "perceptilabs\data\base.py", line 169, in perceptilabs.data.base.DataLoader._build_pipelines
  File "perceptilabs\data\pipelines\image.py", line 23, in perceptilabs.data.pipelines.image.ImagePipelineBuilder.build
  File "perceptilabs\data\pipelines\image.py", line 204, in perceptilabs.data.pipelines.image.ImagePipelineBuilder._get_file_loader_pipeline
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 749, in __next__
    raise StopIteration
StopIteration
ERROR:perceptilabs.endpoints.base:Exception on /model_recommendations [POST]
Traceback (most recent call last):
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\eager\context.py", line 2113, in execution_mode
    yield
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 733, in _next_internal
    output_shapes=self._flat_output_shapes)
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\ops\gen_dataset_ops.py", line 2578, in iterator_get_next
    _ops.raise_from_not_ok_status(e, name)
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\framework\ops.py", line 6862, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence [Op:IteratorGetNext]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 747, in __next__
    return self._next_internal()
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 739, in _next_internal
    return structure.from_compatible_tensor_list(self._element_spec, ret)
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\eager\context.py", line 2116, in execution_mode
    executor_new.wait()
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\eager\executor.py", line 69, in wait
    pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle)
tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\flask\app.py", line 2070, in wsgi_app
    response = self.full_dispatch_request()
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\flask\app.py", line 1515, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\flask\app.py", line 1513, in full_dispatch_request
    rv = self.dispatch_request()
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\flask\app.py", line 1499, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\flask\views.py", line 83, in view
    return self.dispatch_request(*args, **kwargs)
  File "perceptilabs\endpoints\model_recommendations\base.py", line 28, in perceptilabs.endpoints.model_recommendations.base.ModelRecommendations.dispatch_request
  File "perceptilabs\endpoints\model_recommendations\base.py", line 22, in perceptilabs.endpoints.model_recommendations.base.ModelRecommendations.dispatch_request
  File "perceptilabs\data\base.py", line 67, in perceptilabs.data.base.DataLoader.from_dict
  File "perceptilabs\data\base.py", line 85, in perceptilabs.data.base.DataLoader.from_features
  File "perceptilabs\data\base.py", line 44, in perceptilabs.data.base.DataLoader.__init__
  File "perceptilabs\data\base.py", line 235, in perceptilabs.data.base.DataLoader._build_datasets_and_pipelines
  File "perceptilabs\data\base.py", line 110, in perceptilabs.data.base.DataLoader._create_datasets_and_pipelines_from_dataframe
  File "perceptilabs\data\base.py", line 139, in perceptilabs.data.base.DataLoader._load_and_preprocess_feature
  File "perceptilabs\data\base.py", line 169, in perceptilabs.data.base.DataLoader._build_pipelines
  File "perceptilabs\data\pipelines\image.py", line 23, in perceptilabs.data.pipelines.image.ImagePipelineBuilder.build
  File "perceptilabs\data\pipelines\image.py", line 204, in perceptilabs.data.pipelines.image.ImagePipelineBuilder._get_file_loader_pipeline
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 749, in __next__
    raise StopIteration
StopIteration
2021-06-28 14:54:35,695 - INFO - app.py:1884 - Request to endpoint 'model_recommendations' took 0.6608693999999957s
INFO:perceptilabs.applogger:Request to endpoint 'model_recommendations' took 0.6608693999999957s
[28/Jun/2021 14:54:35] "OPTIONS /models/68/ HTTP/1.1" 200 0
[28/Jun/2021 14:54:35] "DELETE /models/68/ HTTP/1.1" 204 0
[28/Jun/2021 14:54:35] "GET /projects/ HTTP/1.1" 200 264

Edit: idk why the formatting is wacky, sorry

[robertl]

Hi @ThatDude!
It looks like the preprocessing pipelines failed, most likely because they were trying to divide your single image into Training, Validation and Testing partitions. Does it work better if you have more images (at least 10) in your dataset?

All the best,
Robert

[ThatDude]

I tried with part of the MNIST dataset just to test (512 images for both input and target), got the same error.

(pl) C:\Users\[REDACTED]>perceptilabs -v=3
2021-06-28 17:07:23.240954: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-06-28 17:07:26.077301: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-06-28 17:07:26.078226: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll
2021-06-28 17:07:26.092400: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 1080 computeCapability: 6.1
coreClock: 1.771GHz coreCount: 20 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 298.32GiB/s
2021-06-28 17:07:26.092551: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-06-28 17:07:26.101100: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-06-28 17:07:26.101223: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-06-28 17:07:26.105064: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-06-28 17:07:26.106310: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-06-28 17:07:26.114527: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-06-28 17:07:26.117275: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-06-28 17:07:26.118222: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-06-28 17:07:26.118694: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
Operations to perform:
  Apply all migrations: admin, api, auth, contenttypes, sessions
Running migrations:
  No migrations to apply.
PerceptiLabs: Starting
2021-06-28 17:07:28.518744: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-06-28 17:07:28.523350: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
Performing system checks...
Performing system checks...
System check identified no issues (0 silenced).
June 28, 2021 - 17:07:29
Django version 3.2, using settings 'static_file_server.settings'
Starting development server at http://127.0.0.1:8080/
Quit the server with CTRL-BREAK.
System check identified no issues (0 silenced).
June 28, 2021 - 17:07:29
Django version 3.2, using settings 'fileserver.settings'
Starting development server at http://127.0.0.1:8011/
Quit the server with CTRL-BREAK.
Performing system checks...
System check identified no issues (0 silenced).
June 28, 2021 - 17:07:30
Django version 3.2, using settings 'rygg.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.
[28/Jun/2021 17:07:31] "GET /?token=mhuXbTL8GyseTqQQbifMg3WVQqr8U5no8iC4vPyWcRw HTTP/1.1" 200 493
[28/Jun/2021 17:07:31] "GET /static/styles/vendors.df1267d6ff8457d42086.css HTTP/1.1" 304 0
[28/Jun/2021 17:07:31] "GET /static/js/0.65773a032b9e270e374c.js HTTP/1.1" 304 0
[28/Jun/2021 17:07:31] "GET /static/js/app.787e50e4dbab3976d321.js HTTP/1.1" 304 0
[28/Jun/2021 17:07:31] "GET /fileserver_url HTTP/1.1" 200 0
[28/Jun/2021 17:07:31] "GET /rendering_kernel_url HTTP/1.1" 200 0
[28/Jun/2021 17:07:31] "GET /rygg_url HTTP/1.1" 200 0
[28/Jun/2021 17:07:31] "GET /kernel_url HTTP/1.1" 200 0
[28/Jun/2021 17:07:31] "GET /keycloak_url HTTP/1.1" 200 0
[28/Jun/2021 17:07:31,517] - Broken pipe from ('127.0.0.1', 49486)
2021-06-28 17:07:31.667943: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-06-28 17:07:31.668087: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-06-28 17:07:31.668614: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll
2021-06-28 17:07:31.668614: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll
2021-06-28 17:07:31.686560: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 1080 computeCapability: 6.1
coreClock: 1.771GHz coreCount: 20 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 298.32GiB/s
2021-06-28 17:07:31.686560: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 1080 computeCapability: 6.1
coreClock: 1.771GHz coreCount: 20 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 298.32GiB/s
2021-06-28 17:07:31.686792: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-06-28 17:07:31.687370: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-06-28 17:07:31.692522: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-06-28 17:07:31.692533: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-06-28 17:07:31.692653: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-06-28 17:07:31.693106: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-06-28 17:07:31.696319: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-06-28 17:07:31.696688: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-06-28 17:07:31.697782: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-06-28 17:07:31.698004: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-06-28 17:07:31.702861: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-06-28 17:07:31.703270: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-06-28 17:07:31.704659: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-06-28 17:07:31.705133: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-06-28 17:07:31.705671: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-06-28 17:07:31.705983: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-06-28 17:07:31.706311: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-06-28 17:07:31.706526: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
[28/Jun/2021 17:07:32] "GET /keycloak_url HTTP/1.1" 200 0
2021-06-28 17:07:33,705 - INFO - perceptilabs\__main__:4 - Started kernel in mode '{args.mode}'
2021-06-28 17:07:33,705 - INFO - perceptilabs\__main__:4 - Reporting errors with commit id: 2d0477bff7e114a20241b90fc7b2edab35a9ebc4
2021-06-28 17:07:33,708 - INFO - perceptilabs\__main__:4 - Started kernel in mode '{args.mode}'
PerceptiLabs is ready...
2021-06-28 17:07:33,737 - INFO - perceptilabs\__main__:4 - Trying to listen to: 0.0.0.0 5000
PerceptiLabs: PerceptiLabs Started
PerceptiLabs: PerceptiLabs is running at http://localhost:8080/?token=mhuXbTL8GyseTqQQbifMg3WVQqr8U5no8iC4vPyWcRw
PerceptiLabs: Use Control-C to stop this server and shut down all PerceptiLabs processes.
[28/Jun/2021 17:07:37] "GET /?token=mhuXbTL8GyseTqQQbifMg3WVQqr8U5no8iC4vPyWcRw HTTP/1.1" 200 493
[28/Jun/2021 17:07:37] "GET /fileserver_url HTTP/1.1" 200 0
[28/Jun/2021 17:07:37] "GET /rendering_kernel_url HTTP/1.1" 200 0
[28/Jun/2021 17:07:37] "GET /rygg_url HTTP/1.1" 200 0
[28/Jun/2021 17:07:37] "GET /kernel_url HTTP/1.1" 200 0
[28/Jun/2021 17:07:37] "GET /keycloak_url HTTP/1.1" 200 0
[28/Jun/2021 17:07:37] "GET /keycloak_url HTTP/1.1" 200 0
[28/Jun/2021 17:07:37] "GET /static/fonts/nunito-sans-v5-latin-600.b56c7e1.woff2 HTTP/1.1" 304 0
[28/Jun/2021 17:07:37] "GET /static/fonts/nunito-sans-v5-latin-700.d0b6d0f.woff2 HTTP/1.1" 304 0
[28/Jun/2021 17:07:37] "GET /static/fonts/nunito-sans-v5-latin-regular.9c6a1ff.woff2 HTTP/1.1" 304 0
[28/Jun/2021 17:07:37] "GET /static/img/tutorial-background.ac2e517.png HTTP/1.1" 200 186377
2021-06-28 17:07:37,803 - INFO - server.py:293 - Frontend receiver: None , Frontend request: checkVersions
2021-06-28 17:07:37,803 - INFO - server.py:293 - Created coreLogic for network 'None'
[28/Jun/2021 17:07:37] "GET /rygg_url HTTP/1.1" 200 0
[28/Jun/2021 17:07:37] "OPTIONS /version?token=mhuXbTL8GyseTqQQbifMg3WVQqr8U5no8iC4vPyWcRw HTTP/1.1" 200 0
[28/Jun/2021 17:07:37] "GET /projects/ HTTP/1.1" 200 264
[28/Jun/2021 17:07:37] "GET /version?token=mhuXbTL8GyseTqQQbifMg3WVQqr8U5no8iC4vPyWcRw HTTP/1.1" 200 22
2021-06-28 17:07:37,853 - INFO - server.py:293 - Frontend receiver:  , Frontend request: setUser
2021-06-28 17:07:37,853 - INFO - server.py:293 - Created coreLogic for network ''
2021-06-28 17:07:37,855 - INFO - server.py:293 - User has been set to [REDACTED]@gmail.com
[28/Jun/2021 17:07:37] "POST /mixpanel/track/?ip=1&_=1624892857829 HTTP/1.1" 200 0
[28/Jun/2021 17:07:38] "GET /app/updates_available/ HTTP/1.1" 200 22
[28/Jun/2021 17:07:39] "GET /mixpanel/decide/?verbose=1&version=1&lib=web&token=1480b2244fdd4d821227a29e2637f922&ip=1&_=1624892857826 HTTP/1.1" 200 0
[28/Jun/2021 17:07:40] "GET /static/fonts/icomoon.1b9c004.woff HTTP/1.1" 304 0
[28/Jun/2021 17:07:40] "GET /static/img/file-picker/home.svg HTTP/1.1" 304 0
[28/Jun/2021 17:07:40] "OPTIONS /directories/get_folder_content?path=D:/tai&token=mhuXbTL8GyseTqQQbifMg3WVQqr8U5no8iC4vPyWcRw HTTP/1.1" 200 0
[28/Jun/2021 17:07:40] "GET /directories/get_folder_content?path=D:/tai&token=mhuXbTL8GyseTqQQbifMg3WVQqr8U5no8iC4vPyWcRw HTTP/1.1" 200 96
[28/Jun/2021 17:07:40] "GET /static/img/file-picker/folder.svg HTTP/1.1" 304 0
[28/Jun/2021 17:07:40] "GET /static/img/file-picker/file.svg HTTP/1.1" 304 0
[28/Jun/2021 17:07:42] "POST /mixpanel/engage/?ip=1&_=1624892862790 HTTP/1.1" 200 0
[28/Jun/2021 17:07:42] "POST /mixpanel/engage/?ip=1&_=1624892862789 HTTP/1.1" 200 0
[28/Jun/2021 17:07:42] "GET /mixpanel/decide/?verbose=1&version=3&lib=web&token=1480b2244fdd4d821227a29e2637f922&distinct_id=[REDACTED]%40gmail.com&ip=1&_=1624892862787 HTTP/1.1" 200 0
[28/Jun/2021 17:07:43] "OPTIONS /files/get_file_content?path=D:/tai/tset.csv&token=mhuXbTL8GyseTqQQbifMg3WVQqr8U5no8iC4vPyWcRw HTTP/1.1" 200 0
('D:/tai/tset.csv', 4)
[28/Jun/2021 17:07:43] "GET /files/get_file_content?path=D:/tai/tset.csv&token=mhuXbTL8GyseTqQQbifMg3WVQqr8U5no8iC4vPyWcRw HTTP/1.1" 200 126
2021-06-28 17:07:43,700 - INFO - app.py:1884 - Request to endpoint 'type_inference' took 0.00011179999999910706s
INFO:perceptilabs.applogger:Request to endpoint 'type_inference' took 0.00011179999999910706s
2021-06-28 17:07:46,966 - INFO - app.py:1884 - Request to endpoint 'type_inference' took 3.2620655000000003s
INFO:perceptilabs.applogger:Request to endpoint 'type_inference' took 3.2620655000000003s
[28/Jun/2021 17:07:47] "GET /static/fonts/roboto-regular-webfont.736b705.woff2 HTTP/1.1" 200 20000
[28/Jun/2021 17:07:47] "OPTIONS /directories/resolved_dir?path=C:\Users\[REDACTED]\Documents\Perceptilabs\Default&token=mhuXbTL8GyseTqQQbifMg3WVQqr8U5no8iC4vPyWcRw HTTP/1.1" 200 0
[28/Jun/2021 17:07:47] "GET /directories/resolved_dir?path=C:\Users\[REDACTED]\Documents\Perceptilabs\Default&token=mhuXbTL8GyseTqQQbifMg3WVQqr8U5no8iC4vPyWcRw HTTP/1.1" 200 61
[28/Jun/2021 17:07:47] "OPTIONS /directories/get_folder_content?path=C:\Users\[REDACTED]\Documents\Perceptilabs\Default&token=mhuXbTL8GyseTqQQbifMg3WVQqr8U5no8iC4vPyWcRw HTTP/1.1" 200 0
[28/Jun/2021 17:07:47] "GET /directories/get_folder_content?path=C:\Users\[REDACTED]\Documents\Perceptilabs\Default&token=mhuXbTL8GyseTqQQbifMg3WVQqr8U5no8iC4vPyWcRw HTTP/1.1" 200 112
[28/Jun/2021 17:07:53] "OPTIONS /directories?path=C:\Users\[REDACTED]\Documents\Perceptilabs\Default/Model%201&token=mhuXbTL8GyseTqQQbifMg3WVQqr8U5no8iC4vPyWcRw HTTP/1.1" 200 0
[28/Jun/2021 17:07:53] "HEAD /directories?path=C:\Users\[REDACTED]\Documents\Perceptilabs\Default/Model%201&token=mhuXbTL8GyseTqQQbifMg3WVQqr8U5no8iC4vPyWcRw HTTP/1.1" 204 34
[28/Jun/2021 17:07:53] "POST /models/ HTTP/1.1" 201 237
2021-06-28 17:07:53,144 - INFO - app.py:1884 - Request to endpoint 'model_recommendations' took 0.00010630000000233508s
INFO:perceptilabs.applogger:Request to endpoint 'model_recommendations' took 0.00010630000000233508s
2021-06-28 17:07:53.160185: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-06-28 17:07:53.161121: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 1080 computeCapability: 6.1
coreClock: 1.771GHz coreCount: 20 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 298.32GiB/s
2021-06-28 17:07:53.161298: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-06-28 17:07:53.161788: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-06-28 17:07:53.162223: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-06-28 17:07:53.162651: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-06-28 17:07:53.163208: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-06-28 17:07:53.164350: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-06-28 17:07:53.165999: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-06-28 17:07:53.166429: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-06-28 17:07:53.166912: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-06-28 17:07:53.744853: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-06-28 17:07:53.745056: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0
2021-06-28 17:07:53.745935: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N
2021-06-28 17:07:53.746495: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6692 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1)
2021-06-28 17:07:53.749499: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
WARNING:tensorflow:AutoGraph could not transform <bound method ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader.call of <perceptilabs.data.pipelines.image.ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader object at 0x000002E3CBF67DC8>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING:tensorflow:AutoGraph could not transform <bound method ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader.call of <perceptilabs.data.pipelines.image.ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader object at 0x000002E3CBF67DC8>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
2021-06-28 17:07:53.782491: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
WARNING:tensorflow:AutoGraph could not transform <bound method ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader.call of <perceptilabs.data.pipelines.image.ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader object at 0x000002E3CBFFE748>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING:tensorflow:AutoGraph could not transform <bound method ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader.call of <perceptilabs.data.pipelines.image.ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader object at 0x000002E3CBFFE748>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
2021-06-28 17:08:05,803 - INFO - views.py:83 - Building dataset pipelines took 12.6467812 seconds
INFO:perceptilabs.applogger:Building dataset pipelines took 12.6467812 seconds
ERROR:perceptilabs.endpoints.base:Exception on /model_recommendations [POST]
Traceback (most recent call last):
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\flask\app.py", line 2070, in wsgi_app
    response = self.full_dispatch_request()
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\flask\app.py", line 1515, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\flask\app.py", line 1513, in full_dispatch_request
    rv = self.dispatch_request()
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\flask\app.py", line 1499, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\flask\views.py", line 83, in view
    return self.dispatch_request(*args, **kwargs)
  File "perceptilabs\endpoints\model_recommendations\base.py", line 31, in perceptilabs.endpoints.model_recommendations.base.ModelRecommendations.dispatch_request
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\flask\json\__init__.py", line 348, in jsonify
    f"{dumps(data, indent=indent, separators=separators)}\n",
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\flask\json\__init__.py", line 129, in dumps
    rv = _json.dumps(obj, **kwargs)
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\json\__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\json\encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\json\encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\site-packages\flask\json\__init__.py", line 56, in default
    return super().default(o)
  File "d:\[REDACTED]\anaconda3\envs\pl\lib\json\encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type int32 is not JSON serializable
2021-06-28 17:08:08,761 - INFO - app.py:1884 - Request to endpoint 'model_recommendations' took 15.612382799999999s
INFO:perceptilabs.applogger:Request to endpoint 'model_recommendations' took 15.612382799999999s
[28/Jun/2021 17:08:08] "OPTIONS /models/69/ HTTP/1.1" 200 0
[28/Jun/2021 17:08:08] "DELETE /models/69/ HTTP/1.1" 204 0
[28/Jun/2021 17:08:08] "GET /projects/ HTTP/1.1" 200 264

[anton.k]

Hi @ThatDude, I’m looking into this for you.

May I ask which versions of Python and PerceptiLabs you’re running?

[ThatDude]

PL version 0.12.11
Python version 3.7.10

[Pixels128]

I’m having the same issue. I have version 0.12.14 installed and I can’t create from my own, MNIST or Wildfire datasets. I don’t know if this is important, but I’m using perceptilabs in virtualenv. Below is the output from Wildfires.

perceptilabs -v=3
2021-07-12 00:58:37.265243: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] 
Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-07-12 00:58:37.265568: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above 
cudart dlerror if you do not have a GPU set up on your machine.
2021-07-12 00:58:40.480775: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] 
Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: 
No such file or directory
2021-07-12 00:58:40.481044: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to 
cuInit: UNKNOWN ERROR (303)
2021-07-12 00:58:40.481268: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel 
driver does not appear to be running on this host (DESKTOP-HDOJV0F): /proc/driver/nvidia/version 
does not exist
Operations to perform:
Apply all migrations: admin, api, auth, contenttypes, sessions
Running migrations:
 No migrations to apply.
PerceptiLabs: Starting
2021-07-12 00:58:43.239017: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] 
Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared 
object file: No such file or directory
2021-07-12 00:58:43.259428: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above 
cudart dlerror if you do not have a GPU set up on your machine.
2021-07-12 00:58:43.466827: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-07-12 00:58:43.467874: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Performing system checks...
Performing system checks...
Performing system checks...
System check identified no issues (0 silenced).
July 12, 2021 - 04:58:44
Django version 3.2, using settings 'rygg.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
System check identified no issues (0 silenced).
July 12, 2021 - 04:58:44
Django version 3.2, using settings 'fileserver.settings'
Starting development server at http://127.0.0.1:8011/
Quit the server with CONTROL-C.
System check identified no issues (0 silenced).
July 12, 2021 - 04:58:44
Django version 3.2, using settings 'static_file_server.settings'
Starting development server at http://127.0.0.1:8080/
Quit the server with CONTROL-C.
PerceptiLabs: Waiting for services to listen on these ports:
PerceptiLabs:    kernel on port 5000
PerceptiLabs:    rendering-kernel on port 5001
sh: 1: xdg-open: not found
2021-07-12 00:58:46.803859: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2021-07-12 00:58:46.804082: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
2021-07-12 00:58:46.804297: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (DESKTOP-HDOJV0F): /proc/driver/nvidia/version does not exist
2021-07-12 00:58:46.972627: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2021-07-12 00:58:46.972907: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
2021-07-12 00:58:46.973121: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (DESKTOP-HDOJV0F): /proc/driver/nvidia/version does not exist
PerceptiLabs: Waiting for services to listen on these ports:
PerceptiLabs:    kernel on port 5000
PerceptiLabs:    rendering-kernel on port 5001
2021-07-12 00:58:48,964 - INFO - perceptilabs.__main__:4 - Started kernel in mode 'training'
2021-07-12 00:58:49,126 - INFO - perceptilabs.__main__:4 - Started kernel in mode 'rendering'
2021-07-12 00:58:49,424 - INFO - perceptilabs.__main__:4 - Reporting errors with commit id: e70ab3ba51534b65e7168c9a3cd0d79c32a212a6
PerceptiLabs is ready...
2021-07-12 00:58:49,451 - INFO - perceptilabs.__main__:4 - Trying to listen to: 0.0.0.0 5000
2021-07-12 00:58:49,599 - INFO - <frozen importlib._bootstrap>:219 - Initialized sentry for environment 'production' and release '0.12.14'
2021-07-12 00:58:49,763 - INFO - server.py:293 - Created coreLogic for network 'None'
2021-07-12 00:58:50,082 - INFO - server.py:293 - Frontend receiver:  , Frontend request: setUser
2021-07-12 00:58:50,083 - INFO - server.py:293 - Created coreLogic for network ''
2021-07-12 00:58:50,084 - INFO - server.py:293 - User has been set to {email!}
/mnt/c/Users/Joey/Projects/labs/lib/python3.8/site-packages/psutil/_pslinux.py:564: RuntimeWarning: 'sin' and 'sout' swap memory stats couldn't be determined and were set to 0 ([Errno 2] No such file or directory: '/proc/vmstat')
  warnings.warn(msg, RuntimeWarning)
PerceptiLabs: PerceptiLabs Started
PerceptiLabs: PerceptiLabs is running at http://localhost:8080/?token=n_crndNz5E-9umc2URk3QHfzwjaOBLDKrn7cqncf4lE
PerceptiLabs: Use Control-C to stop this server and shut down all PerceptiLabs processes.
[12/Jul/2021 04:58:54] "GET /?token=n_crndNz5E-9umc2URk3QHfzwjaOBLDKrn7cqncf4lE HTTP/1.1" 200 493
[12/Jul/2021 04:58:54] "GET /fileserver_url HTTP/1.1" 200 0
[12/Jul/2021 04:58:54] "GET /rendering_kernel_url HTTP/1.1" 200 0
[12/Jul/2021 04:58:54] "GET /rygg_url HTTP/1.1" 200 0
[12/Jul/2021 04:58:54] "GET /kernel_url HTTP/1.1" 200 0
[12/Jul/2021 04:58:54] "GET /keycloak_url HTTP/1.1" 200 0
[12/Jul/2021 04:58:54] "GET /keycloak_url HTTP/1.1" 200 0
[12/Jul/2021 04:58:54] "GET /?token=n_crndNz5E-9umc2URk3QHfzwjaOBLDKrn7cqncf4lE HTTP/1.1" 200 493
[12/Jul/2021 04:58:54] "GET /rendering_kernel_url HTTP/1.1" 200 0
[12/Jul/2021 04:58:54] "GET /fileserver_url HTTP/1.1" 200 0
[12/Jul/2021 04:58:54] "GET /rygg_url HTTP/1.1" 200 0
[12/Jul/2021 04:58:54] "GET /kernel_url HTTP/1.1" 200 0
[12/Jul/2021 04:58:55] "GET /keycloak_url HTTP/1.1" 200 0
[12/Jul/2021 04:58:55] "GET /keycloak_url HTTP/1.1" 200 0
[12/Jul/2021 04:58:55] "GET /rygg_url HTTP/1.1" 200 0
2021-07-12 00:58:55,579 - INFO - server.py:293 - Frontend receiver: None , Frontend request: checkVersions
[12/Jul/2021 04:58:55] "OPTIONS /version?token=n_crndNz5E-9umc2URk3QHfzwjaOBLDKrn7cqncf4lE HTTP/1.1" 200 0
[12/Jul/2021 04:58:55] "GET /version?token=n_crndNz5E-9umc2URk3QHfzwjaOBLDKrn7cqncf4lE HTTP/1.1" 200 22
[12/Jul/2021 04:58:55] "GET /projects/ HTTP/1.1" 200 251
[12/Jul/2021 04:58:55] "GET /app/updates_available/ HTTP/1.1" 200 22
[12/Jul/2021 04:58:55] "GET /mixpanel/decide/?verbose=1&version=1&lib=web&token=1480b2244fdd4d821227a29e2637f922&ip=1&_=1626065935313 HTTP/1.1" 200 0
[12/Jul/2021 04:58:55] "POST /mixpanel/track/?ip=1&_=1626065935318 HTTP/1.1" 200 0
2021-07-12 00:58:55,900 - INFO - server.py:293 - Frontend receiver:  , Frontend request: setUser
2021-07-12 00:58:55,901 - INFO - server.py:293 - User has been set to {email!}
/mnt/c/Users/Joey/Projects/labs/lib/python3.8/site-packages/psutil/_pslinux.py:564: RuntimeWarning: 'sin' and 'sout' swap memory stats couldn't be determined and were set to 0 ([Errno 2] No such file or directory: '/proc/vmstat')
  warnings.warn(msg, RuntimeWarning)
[12/Jul/2021 04:58:58] "OPTIONS /directories/get_folder_content?path=/mnt/c/Users/Joey/Projects/labs/lib/python3.8/site-packages/perceptilabs/tutorial_data/Wildfires&token=n_crndNz5E-9umc2URk3QHfzwjaOBLDKrn7cqncf4lE HTTP/1.1" 200 0
[12/Jul/2021 04:58:58] "GET /directories/get_folder_content?path=/mnt/c/Users/Joey/Projects/labs/lib/python3.8/site-packages/perceptilabs/tutorial_data/Wildfires&token=n_crndNz5E-9umc2URk3QHfzwjaOBLDKrn7cqncf4lE HTTP/1.1" 200 190
[12/Jul/2021 04:59:00] "GET /mixpanel/decide/?verbose=1&version=3&lib=web&token=1480b2244fdd4d821227a29e2637f922&distinct_id={email!}&ip=1&_=1626065940272 HTTP/1.1" 200 0
[12/Jul/2021 04:59:00] "POST /mixpanel/engage/?ip=1&_=1626065940284 HTTP/1.1" 200 0
[12/Jul/2021 04:59:00] "POST /mixpanel/engage/?ip=1&_=1626065940281 HTTP/1.1" 200 0
[12/Jul/2021 04:59:00] "OPTIONS /files/get_file_content?path=/mnt/c/Users/Joey/Projects/labs/lib/python3.8/site-packages/perceptilabs/tutorial_data/Wildfires/data.csv&token=n_crndNz5E-9umc2URk3QHfzwjaOBLDKrn7cqncf4lE HTTP/1.1" 200 0
('/mnt/c/Users/Joey/Projects/labs/lib/python3.8/site-packages/perceptilabs/tutorial_data/Wildfires/data.csv', 4)
[12/Jul/2021 04:59:00] "GET /files/get_file_content?path=/mnt/c/Users/Joey/Projects/labs/lib/python3.8/site-packages/perceptilabs/tutorial_data/Wildfires/data.csv&token=n_crndNz5E-9umc2URk3QHfzwjaOBLDKrn7cqncf4lE HTTP/1.1" 200 119
2021-07-12 00:59:01,165 - INFO - app.py:1884 - Request to endpoint 'type_inference' took 0.00025290000030508963s
INFO:perceptilabs.applogger:Request to endpoint 'type_inference' took 0.00025290000030508963s
2021-07-12 00:59:05,658 - INFO - app.py:1884 - Request to endpoint 'type_inference' took 4.488045799999782s
INFO:perceptilabs.applogger:Request to endpoint 'type_inference' took 4.488045799999782s
[12/Jul/2021 04:59:05] "OPTIONS /directories/resolved_dir?path=/root/Documents/Perceptilabs/Default&token=n_crndNz5E-9umc2URk3QHfzwjaOBLDKrn7cqncf4lE HTTP/1.1" 200 0
[12/Jul/2021 04:59:05] "GET /directories/resolved_dir?path=/root/Documents/Perceptilabs/Default&token=n_crndNz5E-9umc2URk3QHfzwjaOBLDKrn7cqncf4lE HTTP/1.1" 200 48
[12/Jul/2021 04:59:05] "OPTIONS /directories/get_folder_content?path=/root/Documents/Perceptilabs/Default&token=n_crndNz5E-9umc2URk3QHfzwjaOBLDKrn7cqncf4lE HTTP/1.1" 200 0
[12/Jul/2021 04:59:05] "GET /directories/get_folder_content?path=/root/Documents/Perceptilabs/Default&token=n_crndNz5E-9umc2URk3QHfzwjaOBLDKrn7cqncf4lE HTTP/1.1" 200 102
[12/Jul/2021 04:59:12] "OPTIONS /directories?path=/root/Documents/Perceptilabs/Default/Model%201&token=n_crndNz5E-9umc2URk3QHfzwjaOBLDKrn7cqncf4lE HTTP/1.1" 200 0
[12/Jul/2021 04:59:12] "HEAD /directories?path=/root/Documents/Perceptilabs/Default/Model%201&token=n_crndNz5E-9umc2URk3QHfzwjaOBLDKrn7cqncf4lE HTTP/1.1" 204 34
[12/Jul/2021 04:59:12] "POST /models/ HTTP/1.1" 201 224
2021-07-12 00:59:12,450 - INFO - app.py:1884 - Request to endpoint 'model_recommendations' took 0.0001235999998243642s
INFO:perceptilabs.applogger:Request to endpoint 'model_recommendations' took 0.0001235999998243642s
2021-07-12 00:59:12.468895: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:AutoGraph could not transform <bound method ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader.call of <perceptilabs.data.pipelines.image.ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader object at 0x7f92eb7bd670>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING:tensorflow:AutoGraph could not transform <bound method ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader.call of <perceptilabs.data.pipelines.image.ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader object at 0x7f92eb7bd670>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
2021-07-12 00:59:12.513134: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-07-12 00:59:12.514231: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 3301000000 Hz
2021-07-12 00:59:12.524650: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at whole_file_read_ops.cc:116 : Not found: /mnt/c/Users/Joey/Projects/labs/lib/python3.8/site-packages/perceptilabs/tutorial_data/Wildfires/nofire/nofire_0470.jpg; No such file or directory
2021-07-12 00:59:12.524966: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at whole_file_read_ops.cc:116 : Not found: /mnt/c/Users/Joey/Projects/labs/lib/python3.8/site-packages/perceptilabs/tutorial_data/Wildfires/fire/fire_0360.jpg; No such file or directory
2021-07-12 00:59:12.525192: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at whole_file_read_ops.cc:116 : Not found: /mnt/c/Users/Joey /Projects/labs/lib/python3.8/site-packages/perceptilabs/tutorial_data/Wildfires/fire/fire_0404.jpg; No such file or directory
2021-07-12 00:59:12.525392: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at whole_file_read_ops.cc:116 : Not found: /mnt/c/Users/Joey/Projects/labs/lib/python3.8/site-packages/perceptilabs/tutorial_data/Wildfires/nofire/nofire_0625.jpg; No such file or directory
2021-07-12 00:59:12,532 - ERROR - app.py:1390 - Error in request 'model_recommendations'
ERROR:perceptilabs.applogger:Error in request 'model_recommendations'
2021-07-12 00:59:12,535 - INFO - app.py:1884 - Request to endpoint 'model_recommendations' took 0.08255959999951301s
INFO:perceptilabs.applogger:Request to endpoint 'model_recommendations' took 0.08255959999951301s
[12/Jul/2021 04:59:12] "OPTIONS /models/14/ HTTP/1.1" 200 0
[12/Jul/2021 04:59:12] "DELETE /models/14/ HTTP/1.1" 204 0
[12/Jul/2021 04:59:12] "GET /projects/ HTTP/1.1" 200 251

[birdstream]

I’m having this issue, too!

[robertl]

Hi @Pixels128,

For your issue it seems like it’s not finding the images. Would you mind sending a quick sample of how your folder structure/CSV file looks like (a screenshot for each would work well)?

[robertl]

@birdstream,

If you are running the tool as “perceptilabs -v=3”, how does the printout look like when the error occurs?
Just want to check which of the types of error you seem to have

[birdstream]

This is from start to the actual error

(plabs) joakim@joakim-All-Series:~$ perceptilabs -v=3
2021-07-14 19:31:10.494869: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-07-14 19:31:11.517507: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
2021-07-14 19:31:11.521537: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-14 19:31:11.522066: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 980 computeCapability: 5.2
coreClock: 1.2155GHz coreCount: 16 deviceMemorySize: 3.95GiB deviceMemoryBandwidth: 208.91GiB/s
2021-07-14 19:31:11.522082: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-07-14 19:31:11.523766: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
2021-07-14 19:31:11.523795: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
2021-07-14 19:31:11.524505: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
2021-07-14 19:31:11.524661: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
2021-07-14 19:31:11.526441: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11
2021-07-14 19:31:11.526848: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
2021-07-14 19:31:11.526954: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2021-07-14 19:31:11.527032: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-14 19:31:11.527596: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-14 19:31:11.527959: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
Operations to perform:
  Apply all migrations: admin, api, auth, contenttypes, sessions
Running migrations:
  No migrations to apply.
PerceptiLabs: Starting
Performing system checks...
2021-07-14 19:31:13.798720: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-07-14 19:31:13.818810: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
System check identified no issues (0 silenced).
July 14, 2021 - 17:31:13
Django version 3.2, using settings 'static_file_server.settings'
Starting development server at http://127.0.0.1:8080/
Quit the server with CONTROL-C.
Performing system checks...
System check identified no issues (0 silenced).
July 14, 2021 - 17:31:14
Django version 3.2, using settings 'rygg.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
2021-07-14 19:31:15.057017: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
2021-07-14 19:31:15.062739: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-14 19:31:15.063333: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 980 computeCapability: 5.2
coreClock: 1.2155GHz coreCount: 16 deviceMemorySize: 3.95GiB deviceMemoryBandwidth: 208.91GiB/s
2021-07-14 19:31:15.063363: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-07-14 19:31:15.065872: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
2021-07-14 19:31:15.065935: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
2021-07-14 19:31:15.067018: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
2021-07-14 19:31:15.067245: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
2021-07-14 19:31:15.070050: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11
2021-07-14 19:31:15.070701: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
2021-07-14 19:31:15.070862: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2021-07-14 19:31:15.070987: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-14 19:31:15.071627: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-14 19:31:15.072052: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-07-14 19:31:15.072794: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
2021-07-14 19:31:15.078686: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-14 19:31:15.079260: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 980 computeCapability: 5.2
coreClock: 1.2155GHz coreCount: 16 deviceMemorySize: 3.95GiB deviceMemoryBandwidth: 208.91GiB/s
2021-07-14 19:31:15.079285: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-07-14 19:31:15.081030: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
2021-07-14 19:31:15.081073: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
2021-07-14 19:31:15.081811: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
2021-07-14 19:31:15.081991: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
2021-07-14 19:31:15.083877: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11
2021-07-14 19:31:15.084317: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
2021-07-14 19:31:15.084461: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2021-07-14 19:31:15.084578: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-14 19:31:15.084915: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-14 19:31:15.085173: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
[14/Jul/2021 17:31:15] "GET /?token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 200 493
[27721:27721:0714/193115.930319:ERROR:shared_context_state.cc(73)] Skia shader compilation error
------------------------
Errors:
Program binary could not be loaded. Binary is not compatible with current driver/hardware combination. Driver build date Jun 15 2021. Please check build information of source that generated the binary.
[27721:27721:0714/193115.935623:ERROR:shared_context_state.cc(73)] Skia shader compilation error
------------------------
Errors:
Program binary could not be loaded. Binary is not compatible with current driver/hardware combination. Driver build date Jun 15 2021. Please check build information of source that generated the binary.
[14/Jul/2021 17:31:16] "GET /rendering_kernel_url HTTP/1.1" 200 0
[14/Jul/2021 17:31:16] "GET /rygg_url HTTP/1.1" 200 0
[14/Jul/2021 17:31:16] "GET /rygg_url HTTP/1.1" 200 0
[14/Jul/2021 17:31:16] "GET /rendering_kernel_url HTTP/1.1" 200 0
2021-07-14 19:31:16,302 - INFO - perceptilabs.__main__:4 - Started kernel in mode 'training'
2021-07-14 19:31:16,349 - INFO - perceptilabs.__main__:4 - Started kernel in mode 'rendering'
[14/Jul/2021 17:31:16] "GET /kernel_url HTTP/1.1" 200 0
[14/Jul/2021 17:31:16] "GET /kernel_url HTTP/1.1" 200 0
[14/Jul/2021 17:31:16] "GET /keycloak_url HTTP/1.1" 200 0
[14/Jul/2021 17:31:16] "GET /keycloak_url HTTP/1.1" 200 0
PerceptiLabs: Waiting for services to listen on these ports:
PerceptiLabs:    kernel on port 5000
PerceptiLabs:    rendering-kernel on port 5001
2021-07-14 19:31:16,659 - INFO - perceptilabs.__main__:4 - Reporting errors with commit id: 8cd24977c467d7e7a3340a69f0888af48db5d735
PerceptiLabs is ready...
2021-07-14 19:31:16,670 - INFO - perceptilabs.__main__:4 - Trying to listen to: 0.0.0.0 5000
2021-07-14 19:31:16,691 - INFO - <frozen importlib._bootstrap>:219 - Initialized sentry for environment 'production' and release '0.12.13.20210714.1'
[14/Jul/2021 17:31:17] "GET /keycloak_url HTTP/1.1" 200 0
[14/Jul/2021 17:31:17] "GET /keycloak_url HTTP/1.1" 200 0
[14/Jul/2021 17:31:17] "GET /?token=si8jC3zhiBXvre6P7pe2czsHVuMvvSicCSSukW1iGN0 HTTP/1.1" 200 493
[14/Jul/2021 17:31:17] "GET /?token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 200 493
[14/Jul/2021 17:31:18] "GET /rygg_url HTTP/1.1" 200 0
[14/Jul/2021 17:31:18] "GET /rendering_kernel_url HTTP/1.1" 200 0
[14/Jul/2021 17:31:18] "GET /rendering_kernel_url HTTP/1.1" 200 0
[14/Jul/2021 17:31:18] "GET /rygg_url HTTP/1.1" 200 0
[14/Jul/2021 17:31:18] "GET /kernel_url HTTP/1.1" 200 0
[14/Jul/2021 17:31:18] "GET /kernel_url HTTP/1.1" 200 0
[14/Jul/2021 17:31:18] "GET /keycloak_url HTTP/1.1" 200 0
[14/Jul/2021 17:31:18] "GET /keycloak_url HTTP/1.1" 200 0
[14/Jul/2021 17:31:18] "GET /keycloak_url HTTP/1.1" 200 0
[14/Jul/2021 17:31:18] "GET /keycloak_url HTTP/1.1" 200 0
[14/Jul/2021 17:31:18] "GET /rygg_url HTTP/1.1" 200 0
[14/Jul/2021 17:31:18] "OPTIONS /app/version?token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 200 0
[14/Jul/2021 17:31:18] "OPTIONS /app/updates_available/?token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 200 0
[14/Jul/2021 17:31:18] "OPTIONS /projects/?token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 200 0
[14/Jul/2021 17:31:18] "GET /app/version?token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 301 0
[27721:27721:0714/193118.629207:ERROR:shared_context_state.cc(73)] Skia shader compilation error
------------------------
Errors:
Program binary could not be loaded. Binary is not compatible with current driver/hardware combination. Driver build date Jun 15 2021. Please check build information of source that generated the binary.
[27721:27721:0714/193118.632166:ERROR:shared_context_state.cc(73)] Skia shader compilation error
------------------------
Errors:
Program binary could not be loaded. Binary is not compatible with current driver/hardware combination. Driver build date Jun 15 2021. Please check build information of source that generated the binary.
[14/Jul/2021 17:31:18] "GET /projects/?token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 200 258
[14/Jul/2021 17:31:18] "OPTIONS /app/version/?token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 200 0
2021-07-14 19:31:18,643 - INFO - server.py:293 - Frontend receiver: None , Frontend request: checkVersions
2021-07-14 19:31:18,643 - INFO - server.py:293 - Created coreLogic for network 'None'
[14/Jul/2021 17:31:18] "GET /app/version/?token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 200 33
Bad Request: /mixpanel/decide/
[14/Jul/2021 17:31:18] "GET /mixpanel/decide/?verbose=1&version=1&lib=web&token=1480b2244fdd4d821227a29e2637f922&ip=1&_=1626283878670 HTTP/1.1" 400 28
Bad Request: /mixpanel/track/
[14/Jul/2021 17:31:18] "POST /mixpanel/track/?ip=1&_=1626283878677 HTTP/1.1" 400 23
[14/Jul/2021 17:31:18] "GET /app/updates_available/?token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 200 22
2021-07-14 19:31:18,698 - INFO - server.py:293 - Frontend receiver:  , Frontend request: setUser
2021-07-14 19:31:18,698 - INFO - server.py:293 - Created coreLogic for network ''
2021-07-14 19:31:18,699 - INFO - server.py:293 - User has been set to [email protected]
[14/Jul/2021 17:31:18] "GET /rygg_url HTTP/1.1" 200 0
[14/Jul/2021 17:31:18] "GET /projects/?token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 200 258
[14/Jul/2021 17:31:18] "GET /app/version/?token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 200 33
Bad Request: /mixpanel/decide/
[14/Jul/2021 17:31:18] "GET /mixpanel/decide/?verbose=1&version=1&lib=web&token=1480b2244fdd4d821227a29e2637f922&ip=1&_=1626283878822 HTTP/1.1" 400 28
Bad Request: /mixpanel/track/
[14/Jul/2021 17:31:18] "POST /mixpanel/track/?ip=1&_=1626283878827 HTTP/1.1" 400 23
[14/Jul/2021 17:31:18] "GET /app/updates_available/?token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 200 22
PerceptiLabs: PerceptiLabs Started
PerceptiLabs: PerceptiLabs is running at http://localhost:8080/?token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM
PerceptiLabs: Use Control-C to stop this server and shut down all PerceptiLabs processes.
2021-07-14 19:31:19,472 - INFO - server.py:293 - Frontend receiver: None , Frontend request: checkVersions
2021-07-14 19:31:19,494 - INFO - server.py:293 - Frontend receiver:  , Frontend request: setUser
2021-07-14 19:31:19,494 - INFO - server.py:293 - User has been set to [email protected]
Bad Request: /mixpanel/decide/
Bad Request: /mixpanel/engage/
[14/Jul/2021 17:31:23] "GET /mixpanel/decide/?verbose=1&version=3&lib=web&token=1480b2244fdd4d821227a29e2637f922&distinct_id=birdstream%40hotmail.com&ip=1&_=1626283883600 HTTP/1.1" 400 28
Bad Request: /mixpanel/engage/
[14/Jul/2021 17:31:23] "POST /mixpanel/engage/?ip=1&_=1626283883601 HTTP/1.1" 400 23
[14/Jul/2021 17:31:23] "POST /mixpanel/engage/?ip=1&_=1626283883602 HTTP/1.1" 400 23
Bad Request: /mixpanel/decide/
[14/Jul/2021 17:31:23] "GET /mixpanel/decide/?verbose=1&version=3&lib=web&token=1480b2244fdd4d821227a29e2637f922&distinct_id=birdstream%40hotmail.com&ip=1&_=1626283883916 HTTP/1.1" 400 28
Bad Request: /mixpanel/engage/
Bad Request: /mixpanel/engage/
[14/Jul/2021 17:31:23] "POST /mixpanel/engage/?ip=1&_=1626283883918 HTTP/1.1" 400 23
[14/Jul/2021 17:31:23] "POST /mixpanel/engage/?ip=1&_=1626283883920 HTTP/1.1" 400 23
[27721:27721:0714/193240.656621:ERROR:shared_context_state.cc(73)] Skia shader compilation error
------------------------
Errors:
Program binary could not be loaded. Binary is not compatible with current driver/hardware combination. Driver build date Jun 15 2021. Please check build information of source that generated the binary.
[14/Jul/2021 17:32:42] "OPTIONS /directories/get_folder_content?path=/home/joakim&token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 200 0
[14/Jul/2021 17:32:42] "GET /directories/get_folder_content?path=/home/joakim&token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 200 5175
[14/Jul/2021 17:32:51] "OPTIONS /files/get_file_content?path=/home/joakim/test2014.csv&token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 200 0
('/home/joakim/test2014.csv', 4)
[14/Jul/2021 17:32:51] "GET /files/get_file_content?path=/home/joakim/test2014.csv&token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 200 293
[27721:27721:0714/193251.758104:ERROR:shared_context_state.cc(73)] Skia shader compilation error
------------------------
Errors:
Program binary could not be loaded. Binary is not compatible with current driver/hardware combination. Driver build date Jun 15 2021. Please check build information of source that generated the binary.
2021-07-14 19:32:51,800 - INFO - app.py:1884 - Request to endpoint 'type_inference' took 0.00011407400000962298s
INFO:perceptilabs.applogger:Request to endpoint 'type_inference' took 0.00011407400000962298s
2021-07-14 19:32:52,641 - INFO - app.py:1884 - Request to endpoint 'type_inference' took 0.8374802469999736s
INFO:perceptilabs.applogger:Request to endpoint 'type_inference' took 0.8374802469999736s
[14/Jul/2021 17:32:52] "OPTIONS /directories/resolved_dir?path=/home/joakim/Documents/Perceptilabs/Default&token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 200 0
[14/Jul/2021 17:32:52] "GET /directories/resolved_dir?path=/home/joakim/Documents/Perceptilabs/Default&token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 200 55
[14/Jul/2021 17:32:52] "OPTIONS /directories/get_folder_content?path=/home/joakim/Documents/Perceptilabs/Default&token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 200 0
[14/Jul/2021 17:32:52] "GET /directories/get_folder_content?path=/home/joakim/Documents/Perceptilabs/Default&token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 200 151
[14/Jul/2021 17:33:06] "OPTIONS /directories?path=/home/joakim/Documents/Perceptilabs/Default/Model%205&token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 200 0
[14/Jul/2021 17:33:06] "HEAD /directories?path=/home/joakim/Documents/Perceptilabs/Default/Model%205&token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 204 34
[14/Jul/2021 17:33:06] "OPTIONS /models/?token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 200 0
[14/Jul/2021 17:33:06] "POST /models/?token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 201 231
2021-07-14 19:33:06,947 - INFO - app.py:1884 - Request to endpoint 'model_recommendations' took 0.0001523839999890697s
INFO:perceptilabs.applogger:Request to endpoint 'model_recommendations' took 0.0001523839999890697s
2021-07-14 19:33:06.969298: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-07-14 19:33:06.969794: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-14 19:33:06.970090: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 980 computeCapability: 5.2
coreClock: 1.2155GHz coreCount: 16 deviceMemorySize: 3.95GiB deviceMemoryBandwidth: 208.91GiB/s
2021-07-14 19:33:06.970181: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-14 19:33:06.970497: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-14 19:33:06.970737: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-07-14 19:33:06.971336: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-07-14 19:33:08.136163: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-07-14 19:33:08.136192: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      0 
2021-07-14 19:33:08.136209: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0:   N 
2021-07-14 19:33:08.136421: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-14 19:33:08.136711: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-14 19:33:08.136972: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-14 19:33:08.137214: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3102 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce GTX 980, pci bus id: 0000:01:00.0, compute capability: 5.2)
WARNING:tensorflow:AutoGraph could not transform <bound method ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader.call of <perceptilabs.data.pipelines.image.ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader object at 0x7f772012bf10>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING:tensorflow:AutoGraph could not transform <bound method ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader.call of <perceptilabs.data.pipelines.image.ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader object at 0x7f772012bf10>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
2021-07-14 19:33:08.202313: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-07-14 19:33:08.202646: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 3997920000 Hz
WARNING:tensorflow:AutoGraph could not transform <bound method ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader.call of <perceptilabs.data.pipelines.image.ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader object at 0x7f771871af50>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING:tensorflow:AutoGraph could not transform <bound method ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader.call of <perceptilabs.data.pipelines.image.ImagePipelineBuilder._get_file_loader_pipeline.<locals>.Loader object at 0x7f771871af50>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
2021-07-14 19:33:08,496 - INFO - views.py:83 - Building dataset pipelines took 1.5311603149999655 seconds
INFO:perceptilabs.applogger:Building dataset pipelines took 1.5311603149999655 seconds
2021-07-14 19:33:09,171 - ERROR - app.py:1390 - Error in request 'model_recommendations'
ERROR:perceptilabs.applogger:Error in request 'model_recommendations'
2021-07-14 19:33:09,174 - INFO - app.py:1884 - Request to endpoint 'model_recommendations' took 2.225105880000001s
INFO:perceptilabs.applogger:Request to endpoint 'model_recommendations' took 2.225105880000001s
[14/Jul/2021 17:33:09] "OPTIONS /models/54/?token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 200 0
[14/Jul/2021 17:33:09] "DELETE /models/54/?token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 204 0
[14/Jul/2021 17:33:09] "GET /projects/?token=794WMWv9_Jm3M4zfivR3bzpvksgCr6DAhrjLTX651FM HTTP/1.1" 200 258

[birdstream]

Nothing in the works to fix this?

I’ve installed the latest version in a fresh conda environment on linux (Ubuntu 18.04)
Some errors are gone, but it still will not let me use images as output…
It seems like something takes a wrong turn at “TypeError: Object of type int64 is not JSON serializable”?

2021-07-26 15:28:49.768532: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-07-26 15:28:50.724170: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
2021-07-26 15:28:50.728381: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-26 15:28:50.728775: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 980 computeCapability: 5.2
coreClock: 1.2155GHz coreCount: 16 deviceMemorySize: 3.95GiB deviceMemoryBandwidth: 208.91GiB/s
2021-07-26 15:28:50.728790: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-07-26 15:28:50.731755: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
2021-07-26 15:28:50.731789: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
2021-07-26 15:28:50.732453: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
2021-07-26 15:28:50.732615: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
2021-07-26 15:28:50.732989: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11
2021-07-26 15:28:50.733591: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
2021-07-26 15:28:50.733695: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2021-07-26 15:28:50.733792: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-26 15:28:50.734355: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-26 15:28:50.734751: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
Operations to perform:
  Apply all migrations: admin, api, auth, contenttypes, sessions
Running migrations:
  No migrations to apply.
PerceptiLabs: Starting
Performing system checks...
2021-07-26 15:28:52.114564: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-07-26 15:28:52.152006: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
System check identified no issues (0 silenced).
July 26, 2021 - 13:28:52
Django version 3.2, using settings 'static_file_server.settings'
Starting development server at http://127.0.0.1:8080/
Quit the server with CONTROL-C.
Performing system checks...
System check identified no issues (0 silenced).
July 26, 2021 - 13:28:52
Django version 3.2, using settings 'rygg.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
2021-07-26 15:28:53.464890: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
2021-07-26 15:28:53.469646: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-26 15:28:53.469944: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 980 computeCapability: 5.2
coreClock: 1.2155GHz coreCount: 16 deviceMemorySize: 3.95GiB deviceMemoryBandwidth: 208.91GiB/s
2021-07-26 15:28:53.469977: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-07-26 15:28:53.473111: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
2021-07-26 15:28:53.473166: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
2021-07-26 15:28:53.473909: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
2021-07-26 15:28:53.474163: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
2021-07-26 15:28:53.474606: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11
2021-07-26 15:28:53.475294: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
2021-07-26 15:28:53.475435: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2021-07-26 15:28:53.475549: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-26 15:28:53.475928: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-26 15:28:53.476191: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-07-26 15:28:53.494681: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
2021-07-26 15:28:53.501177: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-26 15:28:53.501574: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 980 computeCapability: 5.2
coreClock: 1.2155GHz coreCount: 16 deviceMemorySize: 3.95GiB deviceMemoryBandwidth: 208.91GiB/s
2021-07-26 15:28:53.501605: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-07-26 15:28:53.504862: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
2021-07-26 15:28:53.504933: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
2021-07-26 15:28:53.505649: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
2021-07-26 15:28:53.505863: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
2021-07-26 15:28:53.506298: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11
2021-07-26 15:28:53.506944: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
2021-07-26 15:28:53.507086: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2021-07-26 15:28:53.507210: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-26 15:28:53.507584: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-26 15:28:53.507859: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
[26/Jul/2021 13:28:54] "GET /?token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34 HTTP/1.1" 200 493
[26/Jul/2021 13:28:54] "GET /rygg_url HTTP/1.1" 200 0
[26/Jul/2021 13:28:54] "GET /rendering_kernel_url HTTP/1.1" 200 0
[26/Jul/2021 13:28:54] "GET /kernel_url HTTP/1.1" 200 0
2021-07-26 15:28:54,674 - INFO - perceptilabs.__main__:4 - Started kernel in mode 'training'
2021-07-26 15:28:54,690 - INFO - perceptilabs.__main__:4 - Started kernel in mode 'rendering'
PerceptiLabs: Waiting for services to listen on these ports:
PerceptiLabs:    kernel on port 5000
PerceptiLabs:    rendering-kernel on port 5001
[26/Jul/2021 13:28:54] "GET /keycloak_url HTTP/1.1" 200 0
2021-07-26 15:28:54,833 - INFO - perceptilabs.__main__:4 - Reporting errors with commit id: 89e2abc3915ae49d8739c72d32da86ac4c8b85a9
PerceptiLabs is ready...
2021-07-26 15:28:54,842 - INFO - perceptilabs.__main__:4 - Trying to listen to: 0.0.0.0 5000
2021-07-26 15:28:54,856 - INFO - <frozen importlib._bootstrap>:219 - Initialized sentry for environment 'production' and release '0.12.16'
[26/Jul/2021 13:28:55] "GET /keycloak_url HTTP/1.1" 200 0
PerceptiLabs: PerceptiLabs Started
PerceptiLabs: PerceptiLabs is running at http://localhost:8080/?token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34
PerceptiLabs: Use Control-C to stop this server and shut down all PerceptiLabs processes.
[26/Jul/2021 13:29:01] "GET /?token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34 HTTP/1.1" 200 493
[26/Jul/2021 13:29:01] "GET /rygg_url HTTP/1.1" 200 0
[26/Jul/2021 13:29:01] "GET /rendering_kernel_url HTTP/1.1" 200 0
[26/Jul/2021 13:29:01] "GET /kernel_url HTTP/1.1" 200 0
[26/Jul/2021 13:29:01] "GET /keycloak_url HTTP/1.1" 200 0
[26/Jul/2021 13:29:01] "GET /keycloak_url HTTP/1.1" 200 0
[26/Jul/2021 13:29:02] "GET /is_enterprise HTTP/1.1" 200 28
[26/Jul/2021 13:29:02] "GET /rygg_url HTTP/1.1" 200 0
[26/Jul/2021 13:29:02] "OPTIONS /app/updates_available/?token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34 HTTP/1.1" 200 0
[26/Jul/2021 13:29:02] "OPTIONS /app/version?token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34 HTTP/1.1" 200 0
[26/Jul/2021 13:29:02] "OPTIONS /projects/?token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34 HTTP/1.1" 200 0
[26/Jul/2021 13:29:02] "GET /app/version?token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34 HTTP/1.1" 301 0
[26/Jul/2021 13:29:02] "GET /projects/?token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34 HTTP/1.1" 200 258
[26/Jul/2021 13:29:02] "OPTIONS /app/version/?token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34 HTTP/1.1" 200 0
2021-07-26 15:29:02,275 - INFO - server.py:293 - Frontend receiver: None , Frontend request: checkVersions
2021-07-26 15:29:02,275 - INFO - server.py:293 - Created coreLogic for network 'None'
2021-07-26 15:29:02,276 - INFO - server.py:293 - Frontend receiver: None , Frontend request: isRunning
[26/Jul/2021 13:29:02] "GET /app/version/?token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34 HTTP/1.1" 200 22
Bad Request: /mixpanel/decide/
[26/Jul/2021 13:29:02] "GET /mixpanel/decide/?verbose=1&version=1&lib=web&token=1480b2244fdd4d821227a29e2637f922&ip=1&_=1627306142310 HTTP/1.1" 400 28
Bad Request: /mixpanel/track/
2021-07-26 15:29:02,321 - INFO - server.py:293 - Frontend receiver:  , Frontend request: setUser
2021-07-26 15:29:02,322 - INFO - server.py:293 - Created coreLogic for network ''
2021-07-26 15:29:02,322 - INFO - server.py:293 - User has been set to [email protected]
[26/Jul/2021 13:29:02] "POST /mixpanel/track/?ip=1&_=1627306142317 HTTP/1.1" 400 23
[26/Jul/2021 13:29:02] "GET /app/updates_available/?token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34 HTTP/1.1" 200 22
Bad Request: /mixpanel/decide/
Bad Request: /mixpanel/engage/
[26/Jul/2021 13:29:07] "GET /mixpanel/decide/?verbose=1&version=3&lib=web&token=1480b2244fdd4d821227a29e2637f922&distinct_id=birdstream%40hotmail.com&ip=1&_=1627306147259 HTTP/1.1" 400 28
Bad Request: /mixpanel/engage/
[26/Jul/2021 13:29:07] "POST /mixpanel/engage/?ip=1&_=1627306147261 HTTP/1.1" 400 23
[26/Jul/2021 13:29:07] "POST /mixpanel/engage/?ip=1&_=1627306147262 HTTP/1.1" 400 23
[26/Jul/2021 13:29:25] "OPTIONS /directories/get_folder_content?path=/home/joakim/gray-to-color&token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34 HTTP/1.1" 200 0
[26/Jul/2021 13:29:25] "GET /directories/get_folder_content?path=/home/joakim/gray-to-color&token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34 HTTP/1.1" 200 135
[26/Jul/2021 13:29:28] "OPTIONS /files/get_file_content?path=/home/joakim/gray-to-color/data.csv&token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34 HTTP/1.1" 200 0
('/home/joakim/gray-to-color/data.csv', 4)
[26/Jul/2021 13:29:28] "GET /files/get_file_content?path=/home/joakim/gray-to-color/data.csv&token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34 HTTP/1.1" 200 289
2021-07-26 15:29:28,020 - INFO - app.py:1857 - Request to endpoint 'type_inference
INFO:perceptilabs.applogger:Request to endpoint 'type_inference
2021-07-26 15:29:28,020 - INFO - app.py:1884 - Request to endpoint 'type_inference' took 0.0001s
INFO:perceptilabs.applogger:Request to endpoint 'type_inference' took 0.0001s
2021-07-26 15:29:28,022 - INFO - app.py:1857 - Request to endpoint 'type_inference
INFO:perceptilabs.applogger:Request to endpoint 'type_inference
2021-07-26 15:29:28,757 - INFO - app.py:1884 - Request to endpoint 'type_inference' took 0.7340s
INFO:perceptilabs.applogger:Request to endpoint 'type_inference' took 0.7340s
[26/Jul/2021 13:29:28] "OPTIONS /directories/resolved_dir?path=/home/joakim/Documents/Perceptilabs/Default&token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34 HTTP/1.1" 200 0
[26/Jul/2021 13:29:28] "GET /directories/resolved_dir?path=/home/joakim/Documents/Perceptilabs/Default&token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34 HTTP/1.1" 200 55
[26/Jul/2021 13:29:28] "OPTIONS /directories/get_folder_content?path=/home/joakim/Documents/Perceptilabs/Default&token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34 HTTP/1.1" 200 0
[26/Jul/2021 13:29:28] "GET /directories/get_folder_content?path=/home/joakim/Documents/Perceptilabs/Default&token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34 HTTP/1.1" 200 171
[26/Jul/2021 13:29:37] "OPTIONS /directories?path=/home/joakim/Documents/Perceptilabs/Default/Model%205&token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34 HTTP/1.1" 200 0
[26/Jul/2021 13:29:37] "HEAD /directories?path=/home/joakim/Documents/Perceptilabs/Default/Model%205&token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34 HTTP/1.1" 204 34
[26/Jul/2021 13:29:37] "OPTIONS /models/?token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34 HTTP/1.1" 200 0
[26/Jul/2021 13:29:37] "POST /models/?token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34 HTTP/1.1" 201 232
2021-07-26 15:29:37,183 - INFO - app.py:1857 - Request to endpoint 'put_data
INFO:perceptilabs.applogger:Request to endpoint 'put_data
2021-07-26 15:29:37,183 - INFO - app.py:1884 - Request to endpoint 'put_data' took 0.0002s
INFO:perceptilabs.applogger:Request to endpoint 'put_data' took 0.0002s
2021-07-26 15:29:37,186 - INFO - app.py:1857 - Request to endpoint 'put_data
INFO:perceptilabs.applogger:Request to endpoint 'put_data
2021-07-26 15:29:37,187 - INFO - app.py:1884 - Request to endpoint 'put_data' took 0.0008s
INFO:perceptilabs.applogger:Request to endpoint 'put_data' took 0.0008s
2021-07-26 15:29:37.195474: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-07-26 15:29:37.195844: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-07-26 15:29:37.195858: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      
2021-07-26 15:29:37,197 - INFO - app.py:1857 - Request to endpoint 'put_data
INFO:perceptilabs.applogger:Request to endpoint 'put_data
2021-07-26 15:29:37,197 - INFO - app.py:1884 - Request to endpoint 'put_data' took 0.0001s
INFO:perceptilabs.applogger:Request to endpoint 'put_data' took 0.0001s
2021-07-26 15:29:37,201 - INFO - thread.py:57 - Created indexed TensorFlow dataset. Duration: 0.008s
INFO:perceptilabs.applogger:Created indexed TensorFlow dataset. Duration: 0.008s
2021-07-26 15:29:37,202 - INFO - app.py:1857 - Request to endpoint 'is_data_ready
INFO:perceptilabs.applogger:Request to endpoint 'is_data_ready
2021-07-26 15:29:37,202 - INFO - app.py:1884 - Request to endpoint 'is_data_ready' took 0.0001s
INFO:perceptilabs.applogger:Request to endpoint 'is_data_ready' took 0.0001s
2021-07-26 15:29:37,203 - INFO - thread.py:57 - Partitioned dataset into training (286), validation (81) and test (42). Duration: 0.002s
INFO:perceptilabs.applogger:Partitioned dataset into training (286), validation (81) and test (42). Duration: 0.002s
2021-07-26 15:29:37,204 - INFO - thread.py:57 - Repeated dataset 1 times. Duration: 0.000s
INFO:perceptilabs.applogger:Repeated dataset 1 times. Duration: 0.000s
2021-07-26 15:29:37,204 - INFO - thread.py:57 - Built and partitioned TensorFlow dataset. Size of training: 286, validation: 81, test: 42 
INFO:perceptilabs.applogger:Built and partitioned TensorFlow dataset. Size of training: 286, validation: 81, test: 42 
2021-07-26 15:29:37.227103: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-07-26 15:29:37.227351: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 3997880000 Hz
WARNING:tensorflow:AutoGraph could not transform <bound method Loader.call of <perceptilabs.data.pipelines.image.loader.Loader object at 0x7f35744e45d0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING:tensorflow:AutoGraph could not transform <bound method Loader.call of <perceptilabs.data.pipelines.image.loader.Loader object at 0x7f35744e45d0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING:tensorflow:AutoGraph could not transform <bound method Augmenter.call of <perceptilabs.data.pipelines.image.augmenter.Augmenter object at 0x7f356c2b52d0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING:tensorflow:AutoGraph could not transform <bound method Augmenter.call of <perceptilabs.data.pipelines.image.augmenter.Augmenter object at 0x7f356c2b52d0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
2021-07-26 15:29:37,451 - INFO - thread.py:57 - Built pipelines. Durations for loader=0.189s, augmenter=0.022s, pre- and post-processing=0.022s
INFO:perceptilabs.applogger:Built pipelines. Durations for loader=0.189s, augmenter=0.022s, pre- and post-processing=0.022s
2021-07-26 15:29:37,452 - INFO - thread.py:57 - Built pipelines for grayscale. Duration: 0.247s
INFO:perceptilabs.applogger:Built pipelines for grayscale. Duration: 0.247s
2021-07-26 15:29:37,686 - INFO - thread.py:57 - Built pipelines. Durations for loader=0.192s, augmenter=0.019s, pre- and post-processing=0.020s
INFO:perceptilabs.applogger:Built pipelines. Durations for loader=0.192s, augmenter=0.019s, pre- and post-processing=0.020s
2021-07-26 15:29:37,687 - INFO - thread.py:57 - Built pipelines for color. Duration: 0.235s
INFO:perceptilabs.applogger:Built pipelines for color. Duration: 0.235s
2021-07-26 15:29:37,687 - INFO - thread.py:57 - Built preprocessing pipelines.
INFO:perceptilabs.applogger:Built preprocessing pipelines.
2021-07-26 15:29:37,688 - INFO - thread.py:57 - Inserted metadata with hash '0620ca719339c75c1b6b844dc322a697a8b09e08a3c76b07e6d87f7a6a6108ee'
INFO:perceptilabs.applogger:Inserted metadata with hash '0620ca719339c75c1b6b844dc322a697a8b09e08a3c76b07e6d87f7a6a6108ee'
2021-07-26 15:29:38,209 - INFO - app.py:1857 - Request to endpoint 'is_data_ready
INFO:perceptilabs.applogger:Request to endpoint 'is_data_ready
2021-07-26 15:29:38,209 - INFO - views.py:83 - Found metadata with hash '0620ca719339c75c1b6b844dc322a697a8b09e08a3c76b07e6d87f7a6a6108ee'
INFO:perceptilabs.applogger:Found metadata with hash '0620ca719339c75c1b6b844dc322a697a8b09e08a3c76b07e6d87f7a6a6108ee'
2021-07-26 15:29:38,209 - INFO - app.py:1884 - Request to endpoint 'is_data_ready' took 0.0005s
INFO:perceptilabs.applogger:Request to endpoint 'is_data_ready' took 0.0005s
2021-07-26 15:29:38,215 - INFO - app.py:1857 - Request to endpoint 'model_recommendations
INFO:perceptilabs.applogger:Request to endpoint 'model_recommendations
2021-07-26 15:29:38,216 - INFO - app.py:1884 - Request to endpoint 'model_recommendations' took 0.0003s
INFO:perceptilabs.applogger:Request to endpoint 'model_recommendations' took 0.0003s
2021-07-26 15:29:38,219 - INFO - app.py:1857 - Request to endpoint 'model_recommendations
INFO:perceptilabs.applogger:Request to endpoint 'model_recommendations
2021-07-26 15:29:38,222 - INFO - views.py:83 - Initializing data loader...
INFO:perceptilabs.applogger:Initializing data loader...
2021-07-26 15:29:38,228 - INFO - views.py:83 - Created indexed TensorFlow dataset. Duration: 0.003s
INFO:perceptilabs.applogger:Created indexed TensorFlow dataset. Duration: 0.003s
2021-07-26 15:29:38,229 - INFO - views.py:83 - Partitioned dataset into training (286), validation (81) and test (42). Duration: 0.001s
INFO:perceptilabs.applogger:Partitioned dataset into training (286), validation (81) and test (42). Duration: 0.001s
2021-07-26 15:29:38,229 - INFO - views.py:83 - Repeated dataset 1 times. Duration: 0.000s
INFO:perceptilabs.applogger:Repeated dataset 1 times. Duration: 0.000s
2021-07-26 15:29:38,229 - INFO - views.py:83 - Built and partitioned TensorFlow dataset. Size of training: 286, validation: 81, test: 42 
INFO:perceptilabs.applogger:Built and partitioned TensorFlow dataset. Size of training: 286, validation: 81, test: 42 
2021-07-26 15:29:38,239 - INFO - views.py:83 - Loaded preprocessing pipelines from metadata.
INFO:perceptilabs.applogger:Loaded preprocessing pipelines from metadata.
2021-07-26 15:29:38,239 - INFO - views.py:83 - Data loader initialized
INFO:perceptilabs.applogger:Data loader initialized
2021-07-26 15:29:38,885 - ERROR - app.py:1390 - Error in request 'model_recommendations'
Traceback (most recent call last):
  File "/home/joakim/anaconda3/envs/perceptilabs/lib/python3.7/site-packages/flask/app.py", line 1513, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/joakim/anaconda3/envs/perceptilabs/lib/python3.7/site-packages/flask/app.py", line 1499, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/home/joakim/anaconda3/envs/perceptilabs/lib/python3.7/site-packages/flask/views.py", line 83, in view
    return self.dispatch_request(*args, **kwargs)
  File "perceptilabs/endpoints/model_recommendations/base.py", line 35, in perceptilabs.endpoints.model_recommendations.base.ModelRecommendations.dispatch_request
  File "/home/joakim/anaconda3/envs/perceptilabs/lib/python3.7/site-packages/flask/json/__init__.py", line 348, in jsonify
    f"{dumps(data, indent=indent, separators=separators)}\n",
  File "/home/joakim/anaconda3/envs/perceptilabs/lib/python3.7/site-packages/flask/json/__init__.py", line 129, in dumps
    rv = _json.dumps(obj, **kwargs)
  File "/home/joakim/anaconda3/envs/perceptilabs/lib/python3.7/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/home/joakim/anaconda3/envs/perceptilabs/lib/python3.7/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/home/joakim/anaconda3/envs/perceptilabs/lib/python3.7/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/home/joakim/anaconda3/envs/perceptilabs/lib/python3.7/site-packages/flask/json/__init__.py", line 56, in default
    return super().default(o)
  File "/home/joakim/anaconda3/envs/perceptilabs/lib/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type int64 is not JSON serializable
ERROR:perceptilabs.applogger:Error in request 'model_recommendations'
Traceback (most recent call last):
  File "/home/joakim/anaconda3/envs/perceptilabs/lib/python3.7/site-packages/flask/app.py", line 1513, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/joakim/anaconda3/envs/perceptilabs/lib/python3.7/site-packages/flask/app.py", line 1499, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/home/joakim/anaconda3/envs/perceptilabs/lib/python3.7/site-packages/flask/views.py", line 83, in view
    return self.dispatch_request(*args, **kwargs)
  File "perceptilabs/endpoints/model_recommendations/base.py", line 35, in perceptilabs.endpoints.model_recommendations.base.ModelRecommendations.dispatch_request
  File "/home/joakim/anaconda3/envs/perceptilabs/lib/python3.7/site-packages/flask/json/__init__.py", line 348, in jsonify
    f"{dumps(data, indent=indent, separators=separators)}\n",
  File "/home/joakim/anaconda3/envs/perceptilabs/lib/python3.7/site-packages/flask/json/__init__.py", line 129, in dumps
    rv = _json.dumps(obj, **kwargs)
  File "/home/joakim/anaconda3/envs/perceptilabs/lib/python3.7/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/home/joakim/anaconda3/envs/perceptilabs/lib/python3.7/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/home/joakim/anaconda3/envs/perceptilabs/lib/python3.7/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/home/joakim/anaconda3/envs/perceptilabs/lib/python3.7/site-packages/flask/json/__init__.py", line 56, in default
    return super().default(o)
  File "/home/joakim/anaconda3/envs/perceptilabs/lib/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type int64 is not JSON serializable
2021-07-26 15:29:39,002 - INFO - app.py:1884 - Request to endpoint 'model_recommendations' took 0.7832s
INFO:perceptilabs.applogger:Request to endpoint 'model_recommendations' took 0.7832s
[26/Jul/2021 13:29:39] "OPTIONS /models/134/?token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34 HTTP/1.1" 200 0
[26/Jul/2021 13:29:39] "DELETE /models/134/?token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34 HTTP/1.1" 204 0
[26/Jul/2021 13:29:39] "GET /projects/?token=O6N8-ffKzuqKuxnzXo2ng5KpwEC31gU-B_pxcYXbu34 HTTP/1.1" 200 258

[robertl]

Hi @birdstream,
We finally managed to reproduce it on our side and will rush out a fix for it.
Expect to see it included in either this weeks or next weeks update.
Sorry for the long wait on this one.

[birdstream]

It’s cool very much appreciated!
Thanks for an awesome piece of software!