I have a bug with GPU

Issues/Feedback
#1

I used this post A CUDA Environment Runbook for 0.11.8 for activating perceptilabs-gpu. However, perceptilabs was not able to run any project for training. I think that something was wrong with python because I had visual UI but without visualization of a specific node. (Probably, it is because it’s using python for it).

After truing to run any project. I got this error
"Internal error in threading:926: Training server timed out!

This will be reported as a bug."Bug.txt (94 Bytes)

#2

Hi @ilya

I know how frustrating initial setup can be! I have no knowledge of that specific error, but since the environment and Perceptilabs should work from those instructions, let’s see whether we can find out a bit more…

Firstly, just so you know, as just another user like you I don’t have access to the bug report that can be filed from with Perceptilabs, so I’ll let Perceptilabs people come back if they have any insight from that.

Some things you can try to produce more info to help diagnose this:

  • What particular GPU are you using? (general system description might be helpful)
  • Start Perceptilabs with the command -perceptilabs -v=3. The -v=3 enables verbose logging and there may be some useful information in the console window to share
  • If you followed the recipe for setting up the environment exactly you have also installed jupyterlab. I also uploaded a jupyter notebook on this post. Does running that notebook indicate any specific issues?
  • Can you clarify what you meant by this:

I think that something was wrong with python because I had visual UI but without visualization of a specific node. (Probably, it is because it’s using python for it).

Can you say a bit more to clarify the statement, “Probably, it is because it’s using python for it”? I’m not sure how to interpret that.

Cheers, Julian

#3

Thank you for your help. I am using NVIDIA GTX970M. I tried reinstalling and using the command
-perceptilabs-v=3. When, I was truing to run jupyter notebook I had error “permission denied”. So, maybe my computer just refusing to give access to my GPU.

I will provide a video for better understanding. (this video is montaged 0047.png0001-0808.zip (2.2 MB) in reality it was taking much more time)

#4

OK, I see no immediate problem with the GPU; it is on the CUDA list here.

Diagnosis will be faster and more efficient if you can provide details rather than summaries.

Which cell did it fail at in the notebook and what was the detail of the error “permission denied”? (It is not something I would expect to be the whole report.)

And if you can upload the log of startup with perceptilabs -v=3 until the point where it goes wrong that will tell the perceptilabs guys a lot.

If you have more information about the notebook error I will let you know whether I can shed any light on that issue. You could re-save the notebook with the output generated - including error messages - & upload it here, for example.

Julian

1 Like
#5

I think I found the problem, in console there are few messages “cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version”. I will try to update driver for my GPU.
(It worked)

#6

Glad you have made progress!

1 Like