diff options
| -rw-r--r-- | README | 52 | ||||
| -rw-r--r-- | README.md | 283 |
2 files changed, 283 insertions, 52 deletions
@@ -1,52 +0,0 @@ - _ -| | ____ ___ __ ___ -| |/ /\ \ / / '_ ` _ \ -| < \ V /| | | | | | -|_|\_\ \_/ |_| |_| |_| - - -BUILDING - -Building the kvm driver is pretty easy and compared to building O/N it will -finish in a snap. - -Preparation: - -Edit the Makefile and appropriately set the path for the KERNEL_SOURCE directory -that points to the root of a checked out and built illumos directory. As we -have not had the chance to upstream a change to uts/intel/sys/tss.h ensure that -line 19 of the Makfile, which adds the -D_NEED_KVM_TSS option, is uncommented. - -Verify that gcc is installed and the version you'd like to use is in your path. -Note that this has only been tested against the SFW version of gcc -- version -3.4. - -Verify that you either have SUNWmake or GNU make installed. - -Building and Development: - -To build, simply use the default make target: - -$ make - -To check style, header files, and other various nits: - -$ make check - - -INSTALLING - -There are two mandatory artifacts to install and one optional component: - -kvm The driver -kvm.so The mdb module -kvm.conf The driver configuration file - -On the target machine: -Place kvm in /kernel/drv/amd64 -Place kvm.conf in /kernel/drv -add_drv kvm -ln -s /devices/pseudo/kvm@0:kvm /dev/kvm - -That's all there is too it. Good luck, make sure you have a dump device, and -don't panic! diff --git a/README.md b/README.md new file mode 100644 index 0000000..76cc12e --- /dev/null +++ b/README.md @@ -0,0 +1,283 @@ +illumos-kvm: KVM for illumos +============================ + +KVM is the kernel virtual machine, a framework for the in-kernel acceleration +of QEMU. illumos-kvm is a port of KVM to illumos, taking advantage of +illumos-specific constructs like DTrace, cyclics, mdb, kstat, OS +virtualization, network virtualization, ZFS, etc. It is derived from the KVM +source for Linux 2.6.34, the longterm source for which may be found here: + + git://git.kernel.org/pub/scm/linux/kernel/git/longterm/linux-2.6.34.y.git + +To date, this implementation has been verified with a wide range of guest +operating systems including illumos itself (both SmartOS and OpenIndiana +distributions), FreeBSD, Plan 9, QNX, ChromeOS, HaikuOS, Microsoft Windows +and Linux. + +The design center for this work is to use the virtualization features made +available in the microprocessor -- and in particular, Intel's VMX. As such, +behavior on microprocessors that do not support VMX -- and more specifically, +the extended page tables (EPT) found in second generation VMX support -- +should be graceful failure, not degraded operation. + +Divergences from KVM +-------------------- + +Divergences from KVM fall into several broad categories: some functionality +has been removed or not implemented because it is obviated by features of +illumos (e.g., the custom tracing facility built into KVM); some functionality +has been removed because it is only relevant to hardware that lacks +virtualization support (e.g., older x86 hardware) or on hardware for which +illumos lacks support (e.g., PPC, s390); and some functionality has been +removed because the implementation complexity was simply too great relative +to its value. + +Of this latter category, three areas of divergence merit special note. First, +there is no support for pageable guest memory (that is, guest memory is locked +down). While this is an opinionated decision at some level (in our +experience, memory oversell leads to unacceptable pathologies in all but the +idlest of workloads), we would welcome the work to integrate the KVM MMU +notifier support into illumos-kvm. + +Second (and relatedly), illumos itself has no support for kernel same-page +mapping (KSM) as found in Linux. While illumos could in principle add such +support, it is our experience that the memory that accrues from this is not +sufficiently significant to pay for the increase in implementation and +operator complexity. + +Finally, there is no support currently for AMD SVM. This is not a value +judgement of AMD's technology, but rather a reflection of limited engineering +and testing resources. (In the spirit of full disclosure, it should be said +that the sponsor of illumos-kvm, Joyent, is an Intel-funded company -- but the +lack of AMD support reflects only engineering prioritization and lack of +testing infrastructure; AMD SVM support would be most welcome should someone +in the community be so motivated as to port and test it.) + +Building illumos-kvm +-------------------- + +### Preparation + +Edit the Makefile and appropriately set the path for the `KERNEL_SOURCE` +directory to point to the root of a checked out and built illumos directory. +Building illumos KVM requires several recent additions to illumos, +so be sure your illumos is up to date. + +Verify that gcc is installed and the version you'd like to use is in your path. +Note that this has only been tested against the SFW version of gcc -- version +3.4. + +Verify that you either have SUNWmake or GNU make installed. + +### Building + +To build, simply use the default make target: + + $ make + +To check style, header files, and other various nits: + + $ make check + +Installing illumos-kvm +---------------------- + +### System requirements + +To run illumos-kvm, you will need an illumos that has the fix for issue +1347 (integrated on 2011-08-11). Further, your machine will need to +support VMX. To see if your machine supports VMX, run `isainfo -v` and +look for `vmx`, e.g.: + + % isainfo -v + 64-bit amd64 applications + vmx sse4.2 sse4.1 ssse3 popcnt tscp cx16 sse3 sse2 sse fxsr mmx + cmov amd_sysc cx8 tsc fpu + 32-bit i386 applications + vmx sse4.2 sse4.1 ssse3 popcnt tscp ahf cx16 sse3 sse2 sse fxsr mmx + cmov sep cx8 tsc fpu + +If you do not see `vmx` in this output, the `kvm` driver will be unable to +attach. + +### Required binaries + +There are two mandatory artifacts to install, and one optional component: + +* `kvm` is the driver itself +* `kvm.conf` is the driver configuration file +* `kvm.so` is the mdb module + +On the target machine, place `kvm` in `/kernel/drv/amd64` and `kvm.conf` +in `/kernel/drv` then: + + # add_drv kvm + # ln -s /devices/pseudo/kvm@0:kvm /dev/kvm + +Running illumos-kvm +------------------- + +To run KVM, you will need the build product of the illumos-kvm-cmd repo: +`qemu-system-x86_64`; please follow the instructions in the illumos-kvm-cmd +repo to execute QEMU such that KVM is enabled. + +Monitoring illumos-kvm +---------------------- + +Once one or more VMs are running, there is a variety of tooling to help +understand the operating characteristics of the system. + +### kvmstat + +The `kvmstat` command, found in the illumos repository, can be used to monitor +VMs. For example, here is one second of `kvmstat` output from a machine +running two VMs (one 2 VCPU instance running Linux; another 4 VCPU instance +running the illumos-derived SmartOS): + + pid vcpu | exits : haltx irqx irqwx iox mmiox | irqs emul eptv + 4668 0 | 23 : 6 0 0 1 0 | 6 16 0 + 4668 1 | 25 : 6 1 0 1 0 | 6 16 0 + 5026 0 | 17833 : 223 2946 707 106 0 | 3379 13315 0 + 5026 1 | 18687 : 244 2761 512 0 0 | 3085 14803 0 + 5026 2 | 15696 : 194 3452 542 0 0 | 3568 11230 0 + 5026 3 | 16822 : 244 2817 487 0 0 | 3100 12963 0 + +As for the meaning of the columns, they are explained with `kvmstat -h`: + + # kvmstat -h + Usage: kvmstat [interval [count]] + + Displays statistics for running kernel virtual machines, with one line + per virtual CPU. All statistics are reported as per-second rates. + + The columns are as follows: + + pid => identifier of process controlling the virtual CPU + vcpu => virtual CPU identifier relative to its virtual machine + exits => virtual machine exits for the virtual CPU + haltx => virtual machine exits due to the HLT instruction + irqx => virtual machine exits due to a pending external interrupt + irqwx => virtual machine exits due to an open interrupt window + iox => virtual machine exits due to an I/O instruction + mmiox => virtual machine exits due to memory mapped I/O + irqs => interrupts injected into the virtual CPU + emul => instructions emulated in the kernel + eptv => extended page table violations + +### kstat + +As one might expect, `kvmstat` is implemented in terms of kstat. You +can use `kstat(1)` to browse the kstats from the `kvm` module: + + # kstat -m kvm + ... + module: kvm instance: 0 + name: vcpu-4 class: misc + crtime 4407.142410068 + exits 5367443 + fpu-reload 57302 + halt-exits 317275 + halt-wakeup 8991 + host-state-reload 503920 + hypercalls 0 + insn-emulation 3043881 + inst-emulation-fail 0 + invlpg 0 + io-exits 237191 + irq-exits 1668 + irq-injections 320339 + irq-window-exits 1635 + mmio-exits 617 + nmi-injections 0 + nmi-window-exits 0 + pf-fixed 163629 + pf-guest 0 + pid 3949 + request-irq-exits 0 + signal-exits 460 + snaptime 43219.723435123 + zonename global + + module: kvm instance: 4 + name: vm class: misc + crtime 4407.1241134 + lpages 0 + mmu-cache-miss 950 + mmu-flooded 0 + mmu-pte-updated 0 + mmu-pte-write 56360 + mmu-pte-zapped 0 + mmu-recycled 0 + mmu-unsync-page 0 + pid 3949 + remote-tlb-flush 1511 + snaptime 43219.723875091 + zonename global + + +### DTrace + +While there is not currently a stable KVM provider, there are many SDT probes +in KVM; `dtrace -l -m sdt:kvm` to list these. + +Of these, of particular note are the `kvm-guest-entry` and `kvm-guest-exit` +probes, which fire upon entry to and exit from a guest virtual machine. To +determine context, one can use the `vmregs` variable present in illumos. + +For example, here's a simple script that shows histograms of time spent in VM +guests on a per-PID and per-VCPU basis: + + #pragma D option quiet + + kvm-guest-entry + { + self->entry = timestamp; + } + + kvm-guest-exit + /self->entry/ + { + @[pid, vmregs[VMX_VIRTUAL_PROCESSOR_ID]] = + quantize(timestamp - self->entry); + } + + END + { + printa("pid %d, vcpu %d: %@d\n", @); + } + +Here's what the output of running the above might look like: + + pid 3949, vcpu 1: + value ------------- Distribution ------------- count + 512 | 0 + 1024 |@@@@@@@@@@@@@ 26805 + 2048 |@@@@@ 11641 + 4096 |@@@@@@@ 14187 + 8192 |@ 1559 + 16384 |@ 2931 + 32768 |@@@ 5653 + 65536 |@@@@ 8385 + 131072 |@@@ 6926 + 262144 |@@@ 6639 + 524288 | 785 + 1048576 | 0 + +There are many other ways in which DTrace can be used to understand either +host or guest behavior; see the `tools` subdirectory from some sample D +scripts. + +### mdb + +The `kvm.so` build product is an mdb module that contains several useful +commands, including a `kvm` walker to iterate over all `struct kvm` +structures. + +Contributing to illumos-kvm +--------------------------- + +Unless and until its volume dictate that it be elsewhere, illumos KVM +discussion should be on the `illumos-developer` mailing list. +Contributions are happily accepted; please send patches to +`illumos-developer`. + |