報錯內容：
Epoch 1/1
2020-01-13 21:39:19.392806: I tensorflow/stream_executor/dso_loader.cc:152] 成功打開 CUDA library libcublas.so.10.0
2020-01-13 21:39:22.432074: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES 在 tensor_array_ops.cc:122 失敗 : 找不到: 資源 __per_step_10/_tensor_arraysinput_ta_0_4/N10tensorflow11TensorArrayE 不存在。
追蹤回溯 (最近的呼叫最先):
File "/home/hans/WorkSpace/client.py", line 90, in <module>
hist = client.train(convertDict(dictItem), dictItem.get(f"DirName"), dictItem.get(f"round"), CRNNconfig["ESTIMATION"]["SAVE_CLIENT_MODEL"])
File "/home/hans/WorkSpace/FLutils/client.py", line 44, in train
verbose=1)
File "/home/hans/WorkSpace/venv/Rand2AI/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/home/hans/WorkSpace/venv/Rand2AI/lib/python3.6/site-packages/keras/engine/training.py", line 1732, in fit_generator
initial_epoch=initial_epoch)
File "/home/hans/WorkSpace/venv/Rand2AI/lib/python3.6/site-packages/keras/engine/training_generator.py", line 220, in fit_generator
reset_metrics=False)
File "/home/hans/WorkSpace/venv/Rand2AI/lib/python3.6/site-packages/keras/engine/training.py", line 1514, in train_on_batch
outputs = self.train_function(ins)
File "/home/hans/WorkSpace/venv/Rand2AI/lib/python3.6/site-packages/tensorflow/python/keras/backend.py", line 3076, in __call__
run_metadata=self.run_metadata)
File "/home/hans/WorkSpace/venv/Rand2AI/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1439, in __call__
run_metadata_ptr)
File "/home/hans/WorkSpace/venv/Rand2AI/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: 資源 __per_step_10/_tensor_arraysinput_ta_0_4/N10tensorflow11TensorArrayE 不存在。
[[{{node training/Adadelta/gradients/replica_0/model_1/rnn1_bgru1/while_1/TensorArrayReadV3_grad/TensorArrayGrad/TensorArrayGradV3}}]]
[[{{node training/Adadelta/gradients/b_count_26}}]]
花了一天時間，經歷重裝驅動到 CUDA 到系統到虛擬環境，最終最確定是因為 keras 版本問題。我其他兩台機器的版本是 2.2.4，出問題這台電腦是 2.3.1，版本降到 2.2.4 就不報這個錯誤了。
我是第一個遇到這個問題的人，至少之前在網上根本搜不到相關問題的帖子。