ZStack源码剖析之二次开发——在Utility上堆代码

 约 64 分钟
本文首发于泊浮目的专栏:https://segmentfault.com/blog...

背景

在上篇文章中(ZStack源码剖析之二次开发——可扩展框架
),我们简单的了解了一下ZStack核心引擎的二次开发技巧。在这篇文章中,我们将一起来了解ZStack-Utility(即ZStack的Agent端)的二开姿势。

例子

我们以ZStack管理节点调用startVm这个api为例子,一起来看一下在agent上的执行逻辑。

    def start(self):
        http_server = kvmagent.get_http_server()

        http_server.register_async_uri(self.KVM_START_VM_PATH, self.start_vm)

首先,得注册一个http path用来接受reqeust。

    @kvmagent.replyerror
    def start_vm(self, req):
        cmd = jsonobject.loads(req[http.REQUEST_BODY])
        rsp = StartVmResponse()
        try:
            self._record_operation(cmd.vmInstanceUuid, self.VM_OP_START)

            self._start_vm(cmd)
            logger.debug('successfully started vm[uuid:%s, name:%s]' % (cmd.vmInstanceUuid, cmd.vmName))
        except kvmagent.KvmError as e:
            e_str = linux.get_exception_stacktrace()
            logger.warn(e_str)
            if "burst" in e_str and "Illegal" in e_str and "rate" in e_str:
                rsp.error = "QoS exceed max limit, please check and reset it in zstack"
            elif "cannot set up guest memory" in e_str:
                logger.warn('unable to start vm[uuid:%s], %s' % (cmd.vmInstanceUuid, e_str))
                rsp.error = "No enough physical memory for guest"
            else:
                rsp.error = e_str
            err = self.handle_vfio_irq_conflict(cmd.vmInstanceUuid)
            if err != "":
                rsp.error = "%s, details: %s" % (err, rsp.error)
            rsp.success = False
        return jsonobject.dumps(rsp)

直接进入主干逻辑,self._start_vm(cmd)

    @lock.lock('libvirt-startvm')
    def _start_vm(self, cmd):
        try:
            vm = get_vm_by_uuid_no_retry(cmd.vmInstanceUuid, False)

            if vm:
                if vm.state == Vm.VM_STATE_RUNNING:
                    raise kvmagent.KvmError(
                        'vm[uuid:%s, name:%s] is already running' % (cmd.vmInstanceUuid, vm.get_name()))
                else:
                    vm.destroy()

            vm = Vm.from_StartVmCmd(cmd)
            vm.start(cmd.timeout)
        except libvirt.libvirtError as e:
            logger.warn(linux.get_exception_stacktrace())
            if "Device or resource busy" in str(e.message):
                raise kvmagent.KvmError(
                    'unable to start vm[uuid:%s, name:%s], libvirt error: %s' % (
                    cmd.vmInstanceUuid, cmd.vmName, str(e)))

            try:
                vm = get_vm_by_uuid(cmd.vmInstanceUuid)
                if vm and vm.state != Vm.VM_STATE_RUNNING:
                    raise kvmagent.KvmError(
                       'vm[uuid:%s, name:%s, state:%s] is not in running state, libvirt error: %s' % (
                        cmd.vmInstanceUuid, cmd.vmName, vm.state, str(e)))

            except kvmagent.KvmError:
                raise kvmagent.KvmError(
                    'unable to start vm[uuid:%s, name:%s], libvirt error: %s' % (cmd.vmInstanceUuid, cmd.vmName, str(e)))

关键逻辑:

            vm = Vm.from_StartVmCmd(cmd)
            vm.start(cmd.timeout)

先看from_StartVmCmd

    @staticmethod
    def from_StartVmCmd(cmd):
        use_virtio = cmd.useVirtio
        use_numa = cmd.useNuma

        elements = {}

        def make_root():
            root = etree.Element('domain')
            root.set('type', 'kvm')
            # self._root.set('type', 'qemu')
            root.set('xmlns:qemu', 'http://libvirt.org/schemas/domain/qemu/1.0')
            elements['root'] = root

        def make_cpu():
            if use_numa:
                root = elements['root']
                e(root, 'vcpu', '128', {'placement': 'static', 'current': str(cmd.cpuNum)})
                # e(root,'vcpu',str(cmd.cpuNum),{'placement':'static'})
                tune = e(root, 'cputune')
                e(tune, 'shares', str(cmd.cpuSpeed * cmd.cpuNum))
                # enable nested virtualization
                if cmd.nestedVirtualization == 'host-model':
                    cpu = e(root, 'cpu', attrib={'mode': 'host-model'})
                    e(cpu, 'model', attrib={'fallback': 'allow'})
                elif cmd.nestedVirtualization == 'host-passthrough':
                    cpu = e(root, 'cpu', attrib={'mode': 'host-passthrough'})
                    e(cpu, 'model', attrib={'fallback': 'allow'})
                elif IS_AARCH64:
                    cpu = e(root, 'cpu', attrib={'mode': 'host-passthrough'})
                    e(cpu, 'model', attrib={'fallback': 'allow'})
                else:
                    cpu = e(root, 'cpu')
                    # e(cpu, 'topology', attrib={'sockets': str(cmd.socketNum), 'cores': str(cmd.cpuOnSocket), 'threads': '1'})
                mem = cmd.memory / 1024
                e(cpu, 'topology', attrib={'sockets': str(32), 'cores': str(4), 'threads': '1'})
                numa = e(cpu, 'numa')
                e(numa, 'cell', attrib={'id': '0', 'cpus': '0-127', 'memory': str(mem), 'unit': 'KiB'})
            else:
                root = elements['root']
                # e(root, 'vcpu', '128', {'placement': 'static', 'current': str(cmd.cpuNum)})
                e(root, 'vcpu', str(cmd.cpuNum), {'placement': 'static'})
                tune = e(root, 'cputune')
                e(tune, 'shares', str(cmd.cpuSpeed * cmd.cpuNum))
                # enable nested virtualization
                if cmd.nestedVirtualization == 'host-model':
                    cpu = e(root, 'cpu', attrib={'mode': 'host-model'})
                    e(cpu, 'model', attrib={'fallback': 'allow'})
                elif cmd.nestedVirtualization == 'host-passthrough':
                    cpu = e(root, 'cpu', attrib={'mode': 'host-passthrough'})
                    e(cpu, 'model', attrib={'fallback': 'allow'})
                elif IS_AARCH64:
                    cpu = e(root, 'cpu', attrib={'mode': 'host-passthrough'})
                    e(cpu, 'model', attrib={'fallback': 'allow'})
                else:
                    cpu = e(root, 'cpu')
                e(cpu, 'topology', attrib={'sockets': str(cmd.socketNum), 'cores': str(cmd.cpuOnSocket), 'threads': '1'})

        def make_memory():
            root = elements['root']
            mem = cmd.memory / 1024
            if use_numa:
                e(root, 'maxMemory', str(68719476736), {'slots': str(16), 'unit': 'KiB'})
                # e(root,'memory',str(mem),{'unit':'k'})
                e(root, 'currentMemory', str(mem), {'unit': 'k'})
            else:
                e(root, 'memory', str(mem), {'unit': 'k'})
                e(root, 'currentMemory', str(mem), {'unit': 'k'})

        def make_os():
            root = elements['root']
            os = e(root, 'os')
            if IS_AARCH64:
                e(os, 'type', 'hvm', attrib={'arch': 'aarch64'})
                e(os, 'loader', '/usr/share/edk2.git/aarch64/QEMU_EFI-pflash.raw', attrib={'readonly': 'yes', 'type': 'pflash'})
            else:
                e(os, 'type', 'hvm', attrib={'machine': 'pc'})
            # if not booting from cdrom, don't add any boot element in os section
            if cmd.bootDev[0] == "cdrom":
                for boot_dev in cmd.bootDev:
                    e(os, 'boot', None, {'dev': boot_dev})

            if cmd.useBootMenu:
                e(os, 'bootmenu', attrib={'enable': 'yes'})

        def make_features():
            root = elements['root']
            features = e(root, 'features')
            for f in ['acpi', 'apic', 'pae']:
                e(features, f)
            if cmd.kvmHiddenState == True:
                kvm = e(features, "kvm")
                e(kvm, 'hidden', None, {'state': 'on'})

        def make_devices():
            root = elements['root']
            devices = e(root, 'devices')
            if cmd.addons and cmd.addons['qemuPath']:
                e(devices, 'emulator', cmd.addons['qemuPath'])
            else:
                e(devices, 'emulator', kvmagent.get_qemu_path())
            tablet = e(devices, 'input', None, {'type': 'tablet', 'bus': 'usb'})
            e(tablet, 'address', None, {'type':'usb', 'bus':'0', 'port':'1'})
            if IS_AARCH64:
                keyboard = e(devices, 'input', None, {'type': 'keyboard', 'bus': 'usb'})
            elements['devices'] = devices

        def make_cdrom():
            devices = elements['devices']

            MAX_CDROM_NUM = len(Vm.ISO_DEVICE_LETTERS)
            EMPTY_CDROM_CONFIGS = None

            if IS_AARCH64:
                # AArch64 Does not support the attachment of multiple iso
                EMPTY_CDROM_CONFIGS = [
                    EmptyCdromConfig(None, None, None)
                ]
            else:
                # bus 0 unit 0 already use by root volume
                EMPTY_CDROM_CONFIGS = [
                    EmptyCdromConfig('hd%s' % Vm.ISO_DEVICE_LETTERS[0], '0', '1'),
                    EmptyCdromConfig('hd%s' % Vm.ISO_DEVICE_LETTERS[1], '1', '0'),
                    EmptyCdromConfig('hd%s' % Vm.ISO_DEVICE_LETTERS[2], '1', '1')
                ]

            if len(EMPTY_CDROM_CONFIGS) != MAX_CDROM_NUM:
                logger.error('ISO_DEVICE_LETTERS or EMPTY_CDROM_CONFIGS config error')

            def makeEmptyCdrom(targetDev, bus, unit):
                cdrom = e(devices, 'disk', None, {'type': 'file', 'device': 'cdrom'})
                e(cdrom, 'driver', None, {'name': 'qemu', 'type': 'raw'})
                if IS_AARCH64:
                    e(cdrom, 'target', None, {'dev': 'sdc', 'bus': 'scsi'})
                else:
                    e(cdrom, 'target', None, {'dev': targetDev, 'bus': 'ide'})
                    e(cdrom, 'address', None,{'type' : 'drive', 'bus' : bus, 'unit' : unit})
                e(cdrom, 'readonly', None)
                return cdrom

            if not cmd.bootIso:
                for config in EMPTY_CDROM_CONFIGS:
                    makeEmptyCdrom(config.targetDev, config.bus, config.unit)
                return

            notEmptyCdrom = set([])
            for iso in cmd.bootIso:
                notEmptyCdrom.add(iso.deviceId)
                cdromConfig = EMPTY_CDROM_CONFIGS[iso.deviceId]
                if iso.path.startswith('ceph'):
                    ic = IsoCeph()
                    ic.iso = iso
                    devices.append(ic.to_xmlobject(cdromConfig.targetDev, cdromConfig.bus , cdromConfig.unit))
                elif iso.path.startswith('fusionstor'):
                    ic = IsoFusionstor()
                    ic.iso = iso
                    devices.append(ic.to_xmlobject(cdromConfig.targetDev, cdromConfig.bus , cdromConfig.unit))
                else:
                    cdrom = makeEmptyCdrom(cdromConfig.targetDev, cdromConfig.bus , cdromConfig.unit)
                    e(cdrom, 'source', None, {'file': iso.path})

            emptyCdrom = set(range(MAX_CDROM_NUM)).difference(notEmptyCdrom)
            for i in emptyCdrom:
                cdromConfig = EMPTY_CDROM_CONFIGS[i]
                makeEmptyCdrom(cdromConfig.targetDev, cdromConfig.bus, cdromConfig.unit)

        def make_volumes():
            devices = elements['devices']
            volumes = [cmd.rootVolume]
            volumes.extend(cmd.dataVolumes)

            def filebased_volume(_dev_letter, _v):
                disk = etree.Element('disk', {'type': 'file', 'device': 'disk', 'snapshot': 'external'})
                e(disk, 'driver', None, {'name': 'qemu', 'type': linux.get_img_fmt(_v.installPath), 'cache': _v.cacheMode})
                e(disk, 'source', None, {'file': _v.installPath})

                if _v.shareable:
                    e(disk, 'shareable')

                if _v.useVirtioSCSI:
                    e(disk, 'target', None, {'dev': 'sd%s' % _dev_letter, 'bus': 'scsi'})
                    e(disk, 'wwn', _v.wwn)
                    e(disk, 'address', None, {'type': 'drive', 'controller': '0', 'unit': str(_v.deviceId)})
                    return disk

                if _v.useVirtio:
                    e(disk, 'target', None, {'dev': 'vd%s' % _dev_letter, 'bus': 'virtio'})
                elif IS_AARCH64:
                    e(disk, 'target', None, {'dev': 'sd%s' % _dev_letter, 'bus': 'scsi'})
                else:
                    e(disk, 'target', None, {'dev': 'sd%s' % _dev_letter, 'bus': 'ide'})
                return disk

            def iscsibased_volume(_dev_letter, _v):
                def blk_iscsi():
                    bi = BlkIscsi()
                    portal, bi.target, bi.lun = _v.installPath.lstrip('iscsi://').split('/')
                    bi.server_hostname, bi.server_port = portal.split(':')
                    bi.device_letter = _dev_letter
                    bi.volume_uuid = _v.volumeUuid
                    bi.chap_username = _v.chapUsername
                    bi.chap_password = _v.chapPassword

                    return bi.to_xmlobject()

                def virtio_iscsi():
                    vi = VirtioIscsi()
                    portal, vi.target, vi.lun = _v.installPath.lstrip('iscsi://').split('/')
                    vi.server_hostname, vi.server_port = portal.split(':')
                    vi.device_letter = _dev_letter
                    vi.volume_uuid = _v.volumeUuid
                    vi.chap_username = _v.chapUsername
                    vi.chap_password = _v.chapPassword

                    return vi.to_xmlobject()

                if _v.useVirtio:
                    return virtio_iscsi()
                else:
                    return blk_iscsi()

            def ceph_volume(_dev_letter, _v):
                def ceph_virtio():
                    vc = VirtioCeph()
                    vc.volume = _v
                    vc.dev_letter = _dev_letter
                    return vc.to_xmlobject()

                def ceph_blk():
                    if not IS_AARCH64:
                        ic = IdeCeph()
                    else:
                        ic = ScsiCeph()
                    ic.volume = _v
                    ic.dev_letter = _dev_letter
                    return ic.to_xmlobject()

                def ceph_virtio_scsi():
                    vsc = VirtioSCSICeph()
                    vsc.volume = _v
                    vsc.dev_letter = _dev_letter
                    return vsc.to_xmlobject()

                if _v.useVirtioSCSI:
                    disk = ceph_virtio_scsi()
                    if _v.shareable:
                        e(disk, 'shareable')
                    return disk

                if _v.useVirtio:
                    return ceph_virtio()
                else:
                    return ceph_blk()

            def fusionstor_volume(_dev_letter, _v):
                def fusionstor_virtio():
                    vc = VirtioFusionstor()
                    vc.volume = _v
                    vc.dev_letter = _dev_letter
                    return vc.to_xmlobject()

                def fusionstor_blk():
                    ic = IdeFusionstor()
                    ic.volume = _v
                    ic.dev_letter = _dev_letter
                    return ic.to_xmlobject()

                def fusionstor_virtio_scsi():
                    vsc = VirtioSCSIFusionstor()
                    vsc.volume = _v
                    vsc.dev_letter = _dev_letter
                    return vsc.to_xmlobject()

                if _v.useVirtioSCSI:
                    disk = fusionstor_virtio_scsi()
                    if _v.shareable:
                        e(disk, 'shareable')
                    return disk

                if _v.useVirtio:
                    return fusionstor_virtio()
                else:
                    return fusionstor_blk()

            def volume_qos(volume_xml_obj):
                if not cmd.addons:
                    return

                vol_qos = cmd.addons['VolumeQos']
                if not vol_qos:
                    return

                qos = vol_qos[v.volumeUuid]
                if not qos:
                    return

                if not qos.totalBandwidth and not qos.totalIops:
                    return

                iotune = e(volume_xml_obj, 'iotune')
                if qos.totalBandwidth:
                    e(iotune, 'total_bytes_sec', str(qos.totalBandwidth))
                if qos.totalIops:
                    # e(iotune, 'total_iops_sec', str(qos.totalIops))
                    e(iotune, 'read_iops_sec', str(qos.totalIops))
                    e(iotune, 'write_iops_sec', str(qos.totalIops))
                    # e(iotune, 'read_iops_sec_max', str(qos.totalIops))
                    # e(iotune, 'write_iops_sec_max', str(qos.totalIops))
                    # e(iotune, 'total_iops_sec_max', str(qos.totalIops))

            volumes.sort(key=lambda d: d.deviceId)
            scsi_device_ids = [v.deviceId for v in volumes if v.useVirtioSCSI]
            for v in volumes:
                if v.deviceId >= len(Vm.DEVICE_LETTERS):
                    err = "exceeds max disk limit, it's %s but only 26 allowed" % v.deviceId
                    logger.warn(err)
                    raise kvmagent.KvmError(err)

                dev_letter = Vm.DEVICE_LETTERS[v.deviceId]
                if v.useVirtioSCSI:
                    dev_letter = Vm.DEVICE_LETTERS[scsi_device_ids.pop()]

                if v.deviceType == 'file':
                    vol = filebased_volume(dev_letter, v)
                elif v.deviceType == 'iscsi':
                    vol = iscsibased_volume(dev_letter, v)
                elif v.deviceType == 'ceph':
                    vol = ceph_volume(dev_letter, v)
                elif v.deviceType == 'fusionstor':
                    vol = fusionstor_volume(dev_letter, v)
                else:
                    raise Exception('unknown volume deviceType: %s' % v.deviceType)

                assert vol is not None, 'vol cannot be None'
                # set boot order for root volume when boot from hd
                if v.deviceId == 0 and cmd.bootDev[0] == 'hd' and cmd.useBootMenu:
                    e(vol, 'boot', None, {'order': '1'})
                volume_qos(vol)
                devices.append(vol)

        def make_nics():
            if not cmd.nics:
                return

            def nic_qos(nic_xml_object):
                if not cmd.addons:
                    return

                nqos = cmd.addons['NicQos']
                if not nqos:
                    return

                qos = nqos[nic.uuid]
                if not qos:
                    return

                if not qos.outboundBandwidth and not qos.inboundBandwidth:
                    return

                bandwidth = e(nic_xml_object, 'bandwidth')
                if qos.outboundBandwidth:
                    e(bandwidth, 'outbound', None, {'average': str(qos.outboundBandwidth / 1024 / 8)})
                if qos.inboundBandwidth:
                    e(bandwidth, 'inbound', None, {'average': str(qos.inboundBandwidth / 1024 / 8)})

            devices = elements['devices']
            for nic in cmd.nics:
                interface = e(devices, 'interface', None, {'type': 'bridge'})
                e(interface, 'mac', None, {'address': nic.mac})
                if nic.ip is not None and nic.ip != "":
                    filterref = e(interface, 'filterref', None, {'filter':'clean-traffic'})
                    e(filterref, 'parameter', None, {'name':'IP', 'value': nic.ip})
                e(interface, 'alias', None, {'name': 'net%s' % nic.nicInternalName.split('.')[1]})
                e(interface, 'source', None, {'bridge': nic.bridgeName})
                if use_virtio:
                    e(interface, 'model', None, {'type': 'virtio'})
                else:
                    e(interface, 'model', None, {'type': 'e1000'})
                e(interface, 'target', None, {'dev': nic.nicInternalName})

                nic_qos(interface)

        def make_meta():
            root = elements['root']

            e(root, 'name', cmd.vmInstanceUuid)
            e(root, 'uuid', uuidhelper.to_full_uuid(cmd.vmInstanceUuid))
            e(root, 'description', cmd.vmName)
            e(root, 'on_poweroff', 'destroy')
            e(root, 'on_crash', 'restart')
            e(root, 'on_reboot', 'restart')
            meta = e(root, 'metadata')
            zs = e(meta, 'zstack', usenamesapce=True)
            e(zs, 'internalId', str(cmd.vmInternalId))
            e(zs, 'hostManagementIp', str(cmd.hostManagementIp))
            clock = e(root, 'clock', None, {'offset': cmd.clock})
            if cmd.clock == 'localtime':
                e(clock, 'timer', None, {'name': 'rtc', 'tickpolicy': 'catchup'})
                e(clock, 'timer', None, {'name': 'pit', 'tickpolicy': 'delay'})
                e(clock, 'timer', None, {'name': 'hpet', 'present': 'no'})
                e(clock, 'timer', None, {'name': 'hypervclock', 'present': 'yes'})

        def make_vnc():
            devices = elements['devices']
            if cmd.consolePassword == None:
                vnc = e(devices, 'graphics', None, {'type': 'vnc', 'port': '5900', 'autoport': 'yes'})
            else:
                vnc = e(devices, 'graphics', None,
                        {'type': 'vnc', 'port': '5900', 'autoport': 'yes', 'passwd': str(cmd.consolePassword)})
            e(vnc, "listen", None, {'type': 'address', 'address': '0.0.0.0'})

        def make_spice():
            devices = elements['devices']
            spice = e(devices, 'graphics', None, {'type': 'spice', 'port': '5900', 'autoport': 'yes'})
            e(spice, "listen", None, {'type': 'address', 'address': '0.0.0.0'})
            e(spice, "image", None, {'compression': 'auto_glz'})
            e(spice, "jpeg", None, {'compression': 'always'})
            e(spice, "zlib", None, {'compression': 'never'})
            e(spice, "playback", None, {'compression': 'off'})
            e(spice, "streaming", None, {'mode': cmd.spiceStreamingMode})
            e(spice, "mouse", None, {'mode': 'client'})
            e(spice, "filetransfer", None, {'enable': 'no'})
            e(spice, "clipboard", None, {'copypaste': 'no'})

        def make_usb_redirect():
            if cmd.usbRedirect == "true":
                devices = elements['devices']
                e(devices, 'controller', None, {'type': 'usb', 'model': 'ich9-ehci1'})
                e(devices, 'controller', None, {'type': 'usb', 'model': 'ich9-uhci1', 'multifunction': 'on'})
                e(devices, 'controller', None, {'type': 'usb', 'model': 'ich9-uhci2'})
                e(devices, 'controller', None, {'type': 'usb', 'model': 'ich9-uhci3'})

                chan = e(devices, 'channel', None, {'type': 'spicevmc'})
                e(chan, 'target', None, {'type': 'virtio', 'name': 'com.redhat.spice.0'})
                e(chan, 'address', None, {'type': 'virtio-serial'})

                redirdev2 = e(devices, 'redirdev', None, {'type': 'spicevmc', 'bus': 'usb'})
                e(redirdev2, 'address', None, {'type': 'usb', 'bus': '0', 'port': '2'})
                redirdev3 = e(devices, 'redirdev', None, {'type': 'spicevmc', 'bus': 'usb'})
                e(redirdev3, 'address', None, {'type': 'usb', 'bus': '0', 'port': '3'})
                redirdev4 = e(devices, 'redirdev', None, {'type': 'spicevmc', 'bus': 'usb'})
                e(redirdev4, 'address', None, {'type': 'usb', 'bus': '0', 'port': '4'})
                redirdev5 = e(devices, 'redirdev', None, {'type': 'spicevmc', 'bus': 'usb'})
                e(redirdev5, 'address', None, {'type': 'usb', 'bus': '0', 'port': '6'})
            else:
                # make sure there are three default usb controllers, for usb 1.1/2.0/3.0
                devices = elements['devices']
                e(devices, 'controller', None, {'type': 'usb', 'index': '0'})
                if not IS_AARCH64:
                    e(devices, 'controller', None, {'type': 'usb', 'index': '1', 'model': 'ehci'})
                    e(devices, 'controller', None, {'type': 'usb', 'index': '2', 'model': 'nec-xhci'})

        def make_video():
            devices = elements['devices']
            if IS_AARCH64:
                video = e(devices, 'video')
                e(video, 'model', None, {'type': 'virtio'})
            elif cmd.videoType != "qxl":
                video = e(devices, 'video')
                e(video, 'model', None, {'type': str(cmd.videoType)})
            else:
                for monitor in range(cmd.VDIMonitorNumber):
                    video = e(devices, 'video')
                    e(video, 'model', None, {'type': str(cmd.videoType)})


        def make_audio_microphone():
            if cmd.consoleMode == 'spice':
                devices = elements['devices']
                e(devices, 'sound',None,{'model':'ich6'})
            else:
                return

        def make_graphic_console():
            if cmd.consoleMode == 'spice':
                make_spice()
            else:
                make_vnc()

        def make_addons():
            if not cmd.addons:
                return

            devices = elements['devices']
            channel = cmd.addons['channel']
            if channel:
                basedir = os.path.dirname(channel.socketPath)
                linux.mkdir(basedir, 0777)
                chan = e(devices, 'channel', None, {'type': 'unix'})
                e(chan, 'source', None, {'mode': 'bind', 'path': channel.socketPath})
                e(chan, 'target', None, {'type': 'virtio', 'name': channel.targetName})

            cephSecretKey = cmd.addons['ceph_secret_key']
            cephSecretUuid = cmd.addons['ceph_secret_uuid']
            if cephSecretKey and cephSecretUuid:
                VmPlugin._create_ceph_secret_key(cephSecretKey, cephSecretUuid)

            pciDevices = cmd.addons['pciDevice']
            if pciDevices:
                make_pci_device(pciDevices)

            usbDevices = cmd.addons['usbDevice']
            if usbDevices:
                make_usb_device(usbDevices)

        def make_pci_device(addresses):
            devices = elements['devices']
            for addr in addresses:
                if match_pci_device(addr):
                    hostdev = e(devices, "hostdev", None, {'mode': 'subsystem', 'type': 'pci', 'managed': 'yes'})
                    e(hostdev, "driver", None, {'name': 'vfio'})
                    source = e(hostdev, "source")
                    e(source, "address", None, {
                        "domain": hex(0) if len(addr.split(":")) == 2 else hex(int(addr.split(":")[0], 16)),
                        "bus": hex(int(addr.split(":")[-2], 16)),
                        "slot": hex(int(addr.split(":")[-1].split(".")[0], 16)),
                        "function": hex(int(addr.split(":")[-1].split(".")[1], 16))
                    })
                else:
                    raise kvmagent.KvmError(
                       'can not find pci device for address %s' % addr)

        def make_usb_device(usbDevices):
            next_uhci_port = 2
            next_ehci_port = 1
            next_xhci_port = 1
            devices = elements['devices']
            for usb in usbDevices:
                if match_usb_device(usb):
                    hostdev = e(devices, "hostdev", None, {'mode': 'subsystem', 'type': 'usb', 'managed': 'yes'})
                    source = e(hostdev, "source")
                    e(source, "address", None, {
                        "bus": str(int(usb.split(":")[0])),
                        "device": str(int(usb.split(":")[1]))
                    })
                    e(source, "vendor", None, {
                        "id": hex(int(usb.split(":")[2], 16))
                    })
                    e(source, "product", None, {
                        "id": hex(int(usb.split(":")[3], 16))
                    })

                    # get controller index from usbVersion
                    # eg. 1.1 -> 0
                    # eg. 2.0.0 -> 1
                    # eg. 3 -> 2
                    bus = int(usb.split(":")[4][0]) - 1
                    if bus == 0:
                        address = e(hostdev, "address", None, {'type': 'usb', 'bus': str(bus), 'port': str(next_uhci_port)})
                        next_uhci_port += 1
                    elif bus == 1:
                        address = e(hostdev, "address", None, {'type': 'usb', 'bus': str(bus), 'port': str(next_ehci_port)})
                        next_ehci_port += 1
                    elif bus == 2:
                        address = e(hostdev, "address", None, {'type': 'usb', 'bus': str(bus), 'port': str(next_xhci_port)})
                        next_xhci_port += 1
                    else:
                        raise kvmagent.KvmError('unknown usb controller %s', bus)
                else:
                    raise kvmagent.KvmError('cannot find usb device %s', usb)

        # TODO(WeiW) Validate here
        def match_pci_device(addr):
            return True

        def match_usb_device(addr):
            if len(addr.split(':')) == 5:
                return True
            else:
                return False

        def make_balloon_memory():
            devices = elements['devices']
            b = e(devices, 'memballoon', None, {'model': 'virtio'})
            e(b, 'stats', None, {'period': '10'})

        def make_console():
            devices = elements['devices']
            serial = e(devices, 'serial', None, {'type': 'pty'})
            e(serial, 'target', None, {'port': '0'})
            console = e(devices, 'console', None, {'type': 'pty'})
            e(console, 'target', None, {'type': 'serial', 'port': '0'})

        def make_sec_label():
            root = elements['root']
            e(root, 'seclabel', None, {'type': 'none'})

        def make_controllers():
            devices = elements['devices']
            e(devices, 'controller', None, {'type': 'scsi', 'model': 'virtio-scsi'})

        make_root()
        make_meta()
        make_cpu()
        make_memory()
        make_os()
        make_features()
        make_devices()
        make_video()
        make_audio_microphone()
        make_nics()
        make_volumes()
        make_cdrom()
        make_graphic_console()
        make_usb_redirect()
        make_addons()
        make_balloon_memory()
        make_console()
        make_sec_label()
        make_controllers()

        root = elements['root']
        xml = etree.tostring(root)

        vm = Vm()
        vm.uuid = cmd.vmInstanceUuid
        vm.domain_xml = xml
        vm.domain_xmlobject = xmlobject.loads(xml)
        return vm

显然,上述逻辑是在组装一份xml,便于之后的libvirt使用。

然后是

 vm.start(cmd.timeout)

可以看到,这里是直接调用了libvirt的sdk。

这仅仅是一个调用流程。而在很多地方,来自MN的请求会直接调用linux的shell命令,详情见linux.py。(获取云盘大小、主存储容量等)。

问题

在基于扩展ZStack的Agent时,如果是一个全新的功能模块,可能并不会造成和原有代码的深度耦合。但如果在原有功能上的增强, 对原有代码进行修改可能会导致我们的业务逻辑和Utility的上游代码耦合。而在没有足够人力来维护、开发ZStack时,我们会将目标定为能够及时跟上发布版本。 因此,我们要尽量减少冲突。

举个例子:我们要对启动vm的逻辑进行增强,添加一个自己的配置写入xml。这段代码如果写进了vm_plugin.py,那么就是一个耦合。耦合多了以后,跟上发布版本就会很困难。

解决方案

这是一个参考方案:

如果是引入一个全新的功能模块,建议重写一个项目。无论是代码规范还是自动化测试,都可以有一个很好的实践。

如果是基于Utility的扩展,比如对于扩展的api——APIStartVmInstanceExMsg。由上游发送http request时,将指定v2版本的agent。比如原有start vm会发送至path:AGENT_IP:7070/vm/start;而如果我们增强了这部分逻辑,将这段代码copy至vm_plugin_ex.py,并注册一个path,ex/vm/start。当然port也要重新注册一个,就像这样::AGENT_IP:7071/ex/vm/start

同样的,对linux.py扩展时,复制一个linux2.py来存放属于我们自己的扩展逻辑。

阅读 1.6k

推荐阅读
泊浮说
用户专栏

作者是个热爱分享交流的人,所以有了这个专栏。你的点赞是我最大的更新动力。

56 人关注
45 篇文章
专栏主页
目录