Nodes preview doesn't render statistics\data

Bugs and Issues
#1

Hi! I’ve encountered issue where Perceptilabs cannot produce preview output for model’s node:

2021-05-24 16_29_19-PerceptiLabs2088×384 79.6 KB

2021-05-24 16_35_53-Desktop2511×872 81.4 KB

And console output says:

AutoGraph could not transform <bound method build_image_pipelines..Pipeline.call of <perceptilabs.data.pipelines.image.build_image_pipelines..Pipeline object at 0x000001E9509CD3C8>> and will run it as-is. Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert

And It’s many of these.

I have tried this combinations of versions:
python 3.7.10 + cudnn 8.0, cudatools 11.0 with perceptilabs 0.12.3, 0.12.4.
Older versions of perceptilabs (0.11.11, 0.11.15) with GPU support and corresponded cuda packages produced similar results.
The last attempt on packages was simply:

conda env create -n plabs python=3.7
pip install perceptilabs

Even try it on different hardware (AMD, Intel, Nvidia GPU 1080ti\1050), but nothing of it doesn’t help.

Preview still can shows images, though across the standart MNIST tutorial. But I doesn’t understand, I am clearly see people doesn’t have this issue with graph\data rendering. So what it can be wrong?
I do not find the problem in this forum, although StackOverflow forwarded the problem to possible Django versioning. Please, help.

#2

Hi @Loki,
I just recently saw this error happen for another user as well, but we have not yet figured out why it happens nor been able to reproduce it.

I doubt it’s a GPU thing as the model does start running, but if you want to try running it as CPU that could be interesting.

Would you be willing to run PerceptiLabs as “perceptilabs -v=3” and copy the entire log to here?
Additionally, if you can say what system you have tried on (both OS and browser) as well as the results from “pip list” in one of your environment that would help as well.

Sorry that this is happening for you, we will try to fix it asap.

All the best,
Robert

#3

Hi, Robert!
I put reports with GPU support:
conda_env_GpuSupport.txt (12.0 KB) console_output_GpuSupport.txt (62.5 KB)

and without it:
console_output_noGpuSupport.txt (16.0 KB)

Nvidia setup:

2021-05-25 23_05_18-Administrator_ Windows PowerShell806×621 56.1 KB

Google Chrome: Version 90.0.4430.212 (Official Build) (64-bit) with disabled Addblock
Firefox: 88.0.1 (64-bit)
Windows 10 64bit 20H2 19042.985

Don’t know if it counts, but I use Ramdisk solution for Chrome and Firefox, and I previously install system-wide Cuda support driver 11.3 and remove it after realize, it can be installed via “conda install”

#4

Thank you for the details!
We’ll dig into it and come back shortly :slight_smile:

#5

I have got the same problem.
Interestingly, I have GTX1080ti too. I get exactly the same errors like yours.
however, on Firefox, all nodes show nothing, while on chrome the first nodes (Input + conv) show some info. nothing else. It was all working fine till the last update. I tried downgrading all dependencies, but didn’t help. uninstall and reinstall… nothing.
did you figure out a solution ? did you update Nvidia drivers recently ? which version of drivers are you using (Studio or Game ready) ?

it’s very critical timing to lose this tool as we are participating in a competition

PerceptiLabs1349×643 85.5 KB

#6

Hi! Nvidia v. 466.11 Game ready driver.
Interesting, but none of my 4! instances of windows on 2! different devices with different GPU couldn’t work properly (same issue)
Wish the team Perceptilabs comes with solution soon.

I come to conclusion, that my apartment is cursed and not Perceptilabs compatible.

#7

No your apartment is likely fine . it seems something is within the update itself especially it’s related dependencies. It was working perfectly the day before.
My Nvidia driver is the Studio version which is suited for simulations better.
I also noted something a bit strange. the very first time i load the model, all graphs keeps loading for sometime then the kernel disconnect (the little red light next to Tensorflow indicator). then the problem becomes permanent. The same problem I had it when I tried to export the model into Jupyter notebook. [kernel offline]
not sure… I am literally trying everything XD even changing my coffee machine location.

#8

Hi @Loki - I wanted to try out TF 2.5 recently so this caught my eye because on this TF page it says

  • CUDA® Toolkit —TensorFlow supports CUDA® 11.2 (TensorFlow >= 2.5.0)

PL is only using TF2.4 at present, and my experience with a TF2.5 install is that there are package incompatibilities between TF2.5 dependencies and PL dependencies.

Does the problem persist if you use a cuda toolkit known to be compatible with TF2.4 and your OS, e.g. for me on Windows 10: 11.0.221 + cuDNN 8.0.5.39 ?

#9

Pretty interesting. I have CUDA 11.2 while @Loki has 11.3.
both shouldn’t be compatible with TF2.4 in PL
Ok, so I will downgrade to Cuda toolkit 11.1 and see if it will work again.
BRB

#10

let me keep this link here just for compatibility list between CUDA, CuDNN, TF, PL

https://www.tensorflow.org/install/source#gpu

#11

Hey
Sorry for not returning to this topic yet, we have not managed to reproduce it properly on our end.

You mentioned that it was the latest update (0.12.8) where you noticed this issue @DrGM, and it was working fine just before?
Was it the same for you @Loki where it stopped working after an update and was working fine before then?

I believe @Loki also tried on a CPU only setup so it may not be related to the GPUs.

#12

Hi @DrGM, would you be willing to share the log output you get when running PerceptiLabs in verbose mode?

I.e., run the tool as “perceptilabs -v=3” and then copy the full log output here.

Thanks for using the tool and sorry about the issues :slight_smile:

#13

Hi,
yes it was working fine before the update.
however, after the update, downgrading didn’t solve the issue.

#14

PL report.txt (81.7 KB)

Check attached report.

1 Like
#15

Ok, what version did you have before updating? 0.12.7 or an older one?

#16

I am not very sure, but it’s 7-8 months old as i don’t update regularly. probably in version 0.11.xx
but i can’t be sure.

**Side note: i downgraded CUDA toolkit to version 11.0 after clean installation.
now when i use command “nvidia-smi” i get cuda version 11.2
but when i use command “nvcc --version” i get cuda version 11.0

sorry, SOS, HELP!

#17

Hmm, I’m not sure about the CUDA, looks like you got duplicate installations running? Maybe changing the environment variables can help.
For the PL error, would you mind trying two things?

  1. Press F12 to open the browser console, then press the Console tab at the top and send anything you see in there after the previews are not showing.
  2. Is your current installation inside an environment? Can you try creating a new environment and installing PL in that one, and then start up PL in any browser in Incognito/Private mode?
#18

Someone else who had this issue managed to solve it by disabling some browser features:

image751×340 30.7 KB

Maybe something similar could help you?

#19

I have already disabled this function, but with enabling it - all preview is disappear:

image1107×325 41 KB

image1338×1283 126 KB

Also with cudnn and cudatools 11.0, 8.05 - it does not use GPU acceleration while training:

#20

Hi Anton! I have logs too with similar issue:
output.txt (42.7 KB)

My steps:

  1. Run Perceptilabs -v=3
  2. Create MNIST model
  3. Open preview