kapp should work with kubernetes APIs that are affected by faulty APIService resources

[nocquidant]

Hi,

I've recently played with certmanager and mess up my environment a little bit (my namespaces refused to be deleted so I had to force the deletion). But, since then, I couldn't launch kapp anymore:

kapp deploy -n hello -a app-hello -f ./sandbox/config.app.yml                                                                  
Error: unable to retrieve the complete list of server APIs: admission.certmanager.k8s.io/v1beta1: the server is currently unable to handle the request

Delete the orphaned apiservices did the trick: kubectl delete apiservices v1beta1.admission.certmanager.k8s.io

Could kapp be more resilient here and start even in case of errors in server APIs?

Thanks
--nick

[cppforlife]

did you install that apiservice resource via kapp as well?

…

Sent from my iPhone

On May 31, 2019, at 9:05 AM, Nicolas Ocquidant ***@***.***> wrote: Hi, I've recently played with certmanager and mess up my environment a little bit (my namespaces refused to be deleted so I had to force the deletion). But, since then, I couldn't launch kapp anymore: kapp deploy -n hello -a app-hello -f ./sandbox/config.app.yml Error: unable to retrieve the complete list of server APIs: admission.certmanager.k8s.io/v1beta1: the server is currently unable to handle the request Deleted the orphaned apiservices did the trick: kubectl delete apiservices v1beta1.admission.certmanager.k8s.io Could kapp be more resilient here? Thanks --nick — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[nocquidant]

Don't remember if I used helm or not, but it wasn't kapp.

[cppforlife]

Could kapp be more resilient here and start even in case of errors in server APIs?

in theory we could try and ignore certain types of resources if they are not used; however, i do not know if there repercussions from that down the line (some random controller may fail to create some resource somewhere, etc.)

this seems to be similar to #8 where someone else also had to bad config in the cluster. ill dig in more to see if there are downsides. cc @nimakaviani may have an opinion.

[nimakaviani]

My take is similar to the following comment from #8. It is hard to draw the line between which resources to ignore and which ones not to.

wouldnt ignoring that error potentially hide some resources? seems like fixing underlying problem is better.

[cppforlife]

@nocquidant another question for you, do you foresee this problem cropping up in your cluster periodically due to other tools' activity, or it was more of a one off?

[nocquidant]

It was more of a one off. Feel free to close the issue. Thanks

[cppforlife]

closing for now. i included issue to this link in the error [1] to lead people here to gather more feedback. please +1 and reopen if issue persists.

[1] a6ee5c8

[cppforlife]

reopening since im thinking more about this. someone else just recently reported this in slack: https://kubernetes.slack.com/archives/CH8KCCKA5/p1565900345299800. i've also run into this myself a few days ago with certmgr as well.

thinking about putting some apiservice ordering constraints that by default would make ux better... (may be something like this but more generic https://github.com/k14s/k8s-resource-protection/blob/bb6e2411f0ed35c4b759bddfe7d3a4defe527f67/config-certmanager/patches.yml)