Due to an unexpected connectivity issue between cluster and azure container registry, older nodes were unable to pull images, which in turn caused failed MLSkill deployment issue at AICenter side. We worked with our cloud provider team to diagnose and fix the issue at backend.
The deployments are working fine now and we are marking this incident resolved. We will continue to keep the system under monitoring and work on preventing this issue to resurface.
Posted Feb 02, 2023 - 18:31 UTC
Monitoring
We have applied a fix on the cluster and all ML Skills are getting deployed fine now. We are currently monitoring the results and testing a few different use cases as well.
Posted Feb 02, 2023 - 18:14 UTC
Update
We are making progress in resolving the issues with backend compute instances. Now, only few of the instances are pending to be in healthy state now. Once completed, we should see improvement in ML Skill deployment state.
Posted Feb 02, 2023 - 17:22 UTC
Identified
The issue has been identified and a fix is being implemented.