OpenStack – Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance…

Error goes like this – Failed to perform requested operation on instance “TestUbuntu2”, the instance has an error status: Please try again later [Error: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 7bf9297f-bcdc-4c4a-a425-a70d2c4739d3.].

I logged into my block storage node (cinder) and went to see logs /var/log/volume.log

Everything was completely ok

After that I went to compute node (nova) and again, went to see logs /var/log/nova/nova-compute.log

There were quite a few warnings and errors

INFO os_brick.initiator.connectors.iscsi [req-172c4913-748f-4892-b951-2c6f115e26e7 e9bf7a3b5599497ab3e6c145094b1b4f 339f310a18084fc1a4a3b4b85b27c516 – default default] Trying to connect to iSCSI portal 10.0.0.41:3260

2020-04-24 08:53:22.602 1962 WARNING os_brick.initiator.connectors.iscsi [req-172c4913-748f-4892-b951-2c6f115e26e7 e9bf7a3b5599497ab3e6c145094b1b4f 339f310a18084fc1a4a3b4b85b27c516 – default default] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions.

WARNING os_brick.initiator.connectors.iscsi [req-172c4913-748f-4892-b951-2c6f115e26e7 e9bf7a3b5599497ab3e6c145094b1b4f 339f310a18084fc1a4a3b4b85b27c516 – default default] Failed to login iSCSI target iqn.2010-10.org.openstack:volume-325b8c0f-4c52-470f-bd76-7de58c8f586c on portal 10.0.0.41:3260 (exit code 8).: ProcessExecutionError: Unexpected error while running command.

2020-04-24 08:55:22.710 1962 WARNING os_brick.initiator.connectors.iscsi [req-172c4913-748f-4892-b951-2c6f115e26e7 e9bf7a3b5599497ab3e6c145094b1b4f 339f310a18084fc1a4a3b4b85b27c516 – default default] Failed to connect to iSCSI portal 10.0.0.41:3260.

2020-04-24 08:55:22.721 1962 WARNING os_brick.initiator.connectors.iscsi [req-172c4913-748f-4892-b951-2c6f115e26e7 e9bf7a3b5599497ab3e6c145094b1b4f 339f310a18084fc1a4a3b4b85b27c516 – default default] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions.

WARNING os_brick.initiator.connectors.iscsi [req-172c4913-748f-4892-b951-2c6f115e26e7 e9bf7a3b5599497ab3e6c145094b1b4f 339f310a18084fc1a4a3b4b85b27c516 – default default] Couldn’t find iSCSI nodes because iscsiadm err: iscsiadm: No records found

: VolumeDeviceNotFound: Volume device not found at .

2020-04-24 08:59:29.027 1962 WARNING os_brick.initiator.connectors.iscsi [req-172c4913-748f-4892-b951-2c6f115e26e7 e9bf7a3b5599497ab3e6c145094b1b4f 339f310a18084fc1a4a3b4b85b27c516 – default default] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions.

Ok, I did my whole lab with firewall on, and SELinux enabled – that is not recommendation in official documentation.

Again, I did it, because I wanted to learn as much as I can, and again, when I put OpenStack in production it is secure solution.

So, on my block storage node 10.0.0.41 (cinder) I failed to open port 3260

Lets check firewall opened ports on cinder node

firewall-cmd --list-all

Ok, so no opened ports

sudo firewall-cmd --add-port=3260/tcp --permanent
sudo firewall-cmd --reload 

Again, I checked for opened ports

After that I tried to create new instance with CirrOS image

Success

So, in my case firewall was the problem. It may be different in your case. I wanted to show you procedure for troubleshooting more or less everything in OpenStack – check logs first and try to go through them, there should be your problem and potentially – solution.

Disclaimer