Configure Hosts for Harbor (MSDC)#64
Conversation
There was a problem hiding this comment.
This seems to create vlan0 on top of en0 but doesn't remove the IP from en0, so they both end up with the same IP.
Full ifconfig from an example MSVDI host from the BeOneMed, then a ping showing no outbound internet access:
administrator@BNMDCN-D-001-Mini002 ~ % ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
nd6 options=201<PERFORMNUD,DAD>
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280
anpi3: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=400<CHANNEL_IO>
ether 0a:20:51:04:df:d0
media: none
status: inactive
anpi0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=400<CHANNEL_IO>
ether 0a:20:51:04:df:cd
media: none
status: inactive
anpi1: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=400<CHANNEL_IO>
ether 0a:20:51:04:df:ce
media: none
status: inactive
en5: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=400<CHANNEL_IO>
ether 0a:20:51:04:df:ad
nd6 options=201<PERFORMNUD,DAD>
media: none
status: inactive
en6: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=400<CHANNEL_IO>
ether 0a:20:51:04:df:ae
nd6 options=201<PERFORMNUD,DAD>
media: none
status: inactive
en7: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=400<CHANNEL_IO>
ether 0a:20:51:04:df:b0
nd6 options=201<PERFORMNUD,DAD>
media: none
status: inactive
en2: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
options=460<TSO4,TSO6,CHANNEL_IO>
ether 36:11:ea:ef:ca:c0
media: autoselect <full-duplex>
status: inactive
en3: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
options=460<TSO4,TSO6,CHANNEL_IO>
ether 36:11:ea:ef:ca:c4
media: autoselect <full-duplex>
status: inactive
en4: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
options=460<TSO4,TSO6,CHANNEL_IO>
ether 36:11:ea:ef:ca:cc
media: autoselect <full-duplex>
status: inactive
ap1: flags=8822<BROADCAST,SMART,SIMPLEX,MULTICAST> mtu 1500
options=400<CHANNEL_IO>
ether aa:51:0e:59:9d:f5
media: autoselect (none)
en1: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=6460<TSO4,TSO6,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
ether 4e:38:e3:28:8b:82
nd6 options=201<PERFORMNUD,DAD>
media: autoselect
status: inactive
awdl0: flags=8822<BROADCAST,SMART,SIMPLEX,MULTICAST> mtu 1500
options=400<CHANNEL_IO>
ether aa:11:b5:b9:92:3d
media: autoselect (none)
llw0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=400<CHANNEL_IO>
ether aa:11:b5:b9:92:3d
inet6 fe80::a811:b5ff:feb9:923d%llw0 prefixlen 64 scopeid 0x10
nd6 options=201<PERFORMNUD,DAD>
media: autoselect (none)
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=567<RXCSUM,TXCSUM,VLAN_MTU,TSO4,TSO6,AV,CHANNEL_IO>
ether d0:11:e5:c6:86:01
inet6 fe80::4c7:e056:23cc:3662%en0 prefixlen 64 secured scopeid 0x11
inet 10.221.188.12 netmask 0xfffffe00 broadcast 10.221.189.255
nd6 options=201<PERFORMNUD,DAD>
media: autoselect (10Gbase-T <full-duplex>)
status: active
bridge0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=63<RXCSUM,TXCSUM,TSO4,TSO6>
ether 36:11:ea:ef:ca:c0
Configuration:
id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
ipfilter disabled flags 0x0
member: en2 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 10 priority 0 path cost 0
member: en3 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 11 priority 0 path cost 0
member: en4 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 12 priority 0 path cost 0
nd6 options=201<PERFORMNUD,DAD>
media: <unknown type>
status: inactive
vlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=63<RXCSUM,TXCSUM,TSO4,TSO6>
ether d0:11:e5:c6:86:01
inet6 fe80::8fd:374b:4c91:a1fa%vlan0 prefixlen 64 secured scopeid 0x13
inet 10.221.188.12 netmask 0xfffffe00 broadcast 10.221.189.255
nd6 options=201<PERFORMNUD,DAD>
vlan: 2593 parent interface: en0
media: autoselect (10Gbase-T <full-duplex>)
status: active
utun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
inet6 fe80::9d3b:ab1d:9b21:3655%utun0 prefixlen 64 scopeid 0x14
nd6 options=201<PERFORMNUD,DAD>
utun1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1500
inet6 fe80::d55d:3ab4:7fc:ff43%utun1 prefixlen 64 scopeid 0x15
nd6 options=201<PERFORMNUD,DAD>
utun2: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 2000
inet6 fe80::5dcb:f0ef:7d32:398f%utun2 prefixlen 64 scopeid 0x16
nd6 options=201<PERFORMNUD,DAD>
utun3: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1000
inet6 fe80::ce81:b1c:bd2c:69e%utun3 prefixlen 64 scopeid 0x17
nd6 options=201<PERFORMNUD,DAD>
vlan1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=63<RXCSUM,TXCSUM,TSO4,TSO6>
ether d0:11:e5:c6:86:01
inet6 fe80::14ec:9dff:cb8c:b743%vlan1 prefixlen 64 secured scopeid 0x18
inet 172.16.158.12 netmask 0xfffffe00 broadcast 172.16.159.255
nd6 options=201<PERFORMNUD,DAD>
vlan: 3520 parent interface: en0
media: autoselect (10Gbase-T <full-duplex>)
status: active
administrator@BNMDCN-D-001-Mini002 ~ % ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
Request timeout for icmp_seq 0
ping: sendto: No route to host
Request timeout for icmp_seq 1
Hey @ybenchouaf this is odd. Can you share the command that you ran and what variables you set? The Essentially - only a new VLAN interface should be added for the storage VLAN. In the output, I can see it is there ( Also - see if you can reach
|
|
With the management IP now exclusively on |
|
Flagging a few things I ran into while troubleshooting the BeOneMed PoC hosts at Dublin... Worth noting: I'm not confident the most recent version of this playbook has ever been run against these hosts. Some of what I'm seeing could be leftover state from an older run or manual changes made along the way, so take this with that caveat. The storage VLAN and A few open questions:
|
In an initial state (when the host is handed over from service delivery), the networking should be setup with a single VLAN interface with So it should look similar to the following: devadmin@mini-arm-33 ~ % ifconfig en0
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=567<RXCSUM,TXCSUM,VLAN_MTU,TSO4,TSO6,AV,CHANNEL_IO>
ether 14:98:77:6b:52:88
inet6 fe80::1cb2:2f68:9c7c:f11b%en0 prefixlen 64 secured scopeid 0xc
inet 169.254.169.250 netmask 0xffff0000 broadcast 169.254.255.255
nd6 options=201<PERFORMNUD,DAD>
media: autoselect (10Gbase-T <full-duplex>)
status: active
devadmin@mini-arm-33 ~ % ifconfig vlan0
vlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=63<RXCSUM,TXCSUM,TSO4,TSO6>
ether 14:98:77:6b:52:88
inet6 fe80::c52:203f:d8cd:57cd%vlan0 prefixlen 64 secured scopeid 0xe
inet 10.221.192.33 netmask 0xfffffe00 broadcast 10.221.193.255
nd6 options=201<PERFORMNUD,DAD>
vlan: 2869 parent interface: en0
media: autoselect (10Gbase-T <full-duplex>)
status: activeNothing in the Ansible in this repository (especially and including this PR) touches this. It is assumed that the host is already configured correctly for the existing primary VLAN interface The additional commit that was pushed since the last time you tested includes changes so that the router IP is never set for the storage VLAN interface, meaning that the default route should never be overridden. To answer the questions:
tl;dr: something is not right with the initial host configuration there |
The network interface to bind to should be ansible-playbook deploy.yml -i dev/inventory -e "vm_name=foo" -e "vm_image=orkl10000001-01.oci.las1.macstadiumcloud.com/sb/sequoia:test" -e "network_interface=vlan0"
|
|
Ahhh, that makes sense. Looking back at the state of those hosts, I'm fairly confident the config got mucked up by manual changes made by a few people during troubleshooting along the way. So the playbook was never running against a clean baseline. I'll make sure the next deployment gets hosts delivered in the correct initial state (vlan0 configured, en0 link-local only) before anyone runs the playbook. If the platform team's handover process needs to be updated to codify that, I'll work with them to get it in there. Also confirmed: network_interface=vlan0 for MSDC bridge networking. Got it documented internally. From the product side, no more blockers on my end. Thanks for the thorough walkthrough. This looks good to me then! |
3 participants
Description
Introduce the configuration of Mac hosts at MSDC to reach Harbor.
For deployments at MSDC, the Mac hosts must be able to:
Testing
Make sure to set all the required vars under
dev/group_vars/all/main.yml:msdc_storage_network: The first 3 octets of the storage network, e.g. 172.16.180msdc_storage_vlan: The VLAN tag, e.g. 2870configure_harbor_msdc_region: The MSDC region, must be one of 'atl', 'las' or 'dub'configure_harbor_msdc_domain: The domain for Harbor, e.g. my-harbor-01.oci.las1.macstadiumcloud.comconfigure_harbor_msdc_ip: The IP address for the Harbor server, e.g. 10.221.189.254Then run
ansible-playbook configure_harbor_msdc.yml -i dev/inventory.Pull an image from Harbor, e.g.:
It should work