summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
-rw-r--r--README52
-rw-r--r--README.md283
2 files changed, 283 insertions, 52 deletions
diff --git a/README b/README
deleted file mode 100644
index 479696f..0000000
--- a/README
+++ /dev/null
@@ -1,52 +0,0 @@
- _
-| | ____ ___ __ ___
-| |/ /\ \ / / '_ ` _ \
-| < \ V /| | | | | |
-|_|\_\ \_/ |_| |_| |_|
-
-
-BUILDING
-
-Building the kvm driver is pretty easy and compared to building O/N it will
-finish in a snap.
-
-Preparation:
-
-Edit the Makefile and appropriately set the path for the KERNEL_SOURCE directory
-that points to the root of a checked out and built illumos directory. As we
-have not had the chance to upstream a change to uts/intel/sys/tss.h ensure that
-line 19 of the Makfile, which adds the -D_NEED_KVM_TSS option, is uncommented.
-
-Verify that gcc is installed and the version you'd like to use is in your path.
-Note that this has only been tested against the SFW version of gcc -- version
-3.4.
-
-Verify that you either have SUNWmake or GNU make installed.
-
-Building and Development:
-
-To build, simply use the default make target:
-
-$ make
-
-To check style, header files, and other various nits:
-
-$ make check
-
-
-INSTALLING
-
-There are two mandatory artifacts to install and one optional component:
-
-kvm The driver
-kvm.so The mdb module
-kvm.conf The driver configuration file
-
-On the target machine:
-Place kvm in /kernel/drv/amd64
-Place kvm.conf in /kernel/drv
-add_drv kvm
-ln -s /devices/pseudo/kvm@0:kvm /dev/kvm
-
-That's all there is too it. Good luck, make sure you have a dump device, and
-don't panic!
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..76cc12e
--- /dev/null
+++ b/README.md
@@ -0,0 +1,283 @@
+illumos-kvm: KVM for illumos
+============================
+
+KVM is the kernel virtual machine, a framework for the in-kernel acceleration
+of QEMU. illumos-kvm is a port of KVM to illumos, taking advantage of
+illumos-specific constructs like DTrace, cyclics, mdb, kstat, OS
+virtualization, network virtualization, ZFS, etc. It is derived from the KVM
+source for Linux 2.6.34, the longterm source for which may be found here:
+
+ git://git.kernel.org/pub/scm/linux/kernel/git/longterm/linux-2.6.34.y.git
+
+To date, this implementation has been verified with a wide range of guest
+operating systems including illumos itself (both SmartOS and OpenIndiana
+distributions), FreeBSD, Plan 9, QNX, ChromeOS, HaikuOS, Microsoft Windows
+and Linux.
+
+The design center for this work is to use the virtualization features made
+available in the microprocessor -- and in particular, Intel's VMX. As such,
+behavior on microprocessors that do not support VMX -- and more specifically,
+the extended page tables (EPT) found in second generation VMX support --
+should be graceful failure, not degraded operation.
+
+Divergences from KVM
+--------------------
+
+Divergences from KVM fall into several broad categories: some functionality
+has been removed or not implemented because it is obviated by features of
+illumos (e.g., the custom tracing facility built into KVM); some functionality
+has been removed because it is only relevant to hardware that lacks
+virtualization support (e.g., older x86 hardware) or on hardware for which
+illumos lacks support (e.g., PPC, s390); and some functionality has been
+removed because the implementation complexity was simply too great relative
+to its value.
+
+Of this latter category, three areas of divergence merit special note. First,
+there is no support for pageable guest memory (that is, guest memory is locked
+down). While this is an opinionated decision at some level (in our
+experience, memory oversell leads to unacceptable pathologies in all but the
+idlest of workloads), we would welcome the work to integrate the KVM MMU
+notifier support into illumos-kvm.
+
+Second (and relatedly), illumos itself has no support for kernel same-page
+mapping (KSM) as found in Linux. While illumos could in principle add such
+support, it is our experience that the memory that accrues from this is not
+sufficiently significant to pay for the increase in implementation and
+operator complexity.
+
+Finally, there is no support currently for AMD SVM. This is not a value
+judgement of AMD's technology, but rather a reflection of limited engineering
+and testing resources. (In the spirit of full disclosure, it should be said
+that the sponsor of illumos-kvm, Joyent, is an Intel-funded company -- but the
+lack of AMD support reflects only engineering prioritization and lack of
+testing infrastructure; AMD SVM support would be most welcome should someone
+in the community be so motivated as to port and test it.)
+
+Building illumos-kvm
+--------------------
+
+### Preparation
+
+Edit the Makefile and appropriately set the path for the `KERNEL_SOURCE`
+directory to point to the root of a checked out and built illumos directory.
+Building illumos KVM requires several recent additions to illumos,
+so be sure your illumos is up to date.
+
+Verify that gcc is installed and the version you'd like to use is in your path.
+Note that this has only been tested against the SFW version of gcc -- version
+3.4.
+
+Verify that you either have SUNWmake or GNU make installed.
+
+### Building
+
+To build, simply use the default make target:
+
+ $ make
+
+To check style, header files, and other various nits:
+
+ $ make check
+
+Installing illumos-kvm
+----------------------
+
+### System requirements
+
+To run illumos-kvm, you will need an illumos that has the fix for issue
+1347 (integrated on 2011-08-11). Further, your machine will need to
+support VMX. To see if your machine supports VMX, run `isainfo -v` and
+look for `vmx`, e.g.:
+
+ % isainfo -v
+ 64-bit amd64 applications
+ vmx sse4.2 sse4.1 ssse3 popcnt tscp cx16 sse3 sse2 sse fxsr mmx
+ cmov amd_sysc cx8 tsc fpu
+ 32-bit i386 applications
+ vmx sse4.2 sse4.1 ssse3 popcnt tscp ahf cx16 sse3 sse2 sse fxsr mmx
+ cmov sep cx8 tsc fpu
+
+If you do not see `vmx` in this output, the `kvm` driver will be unable to
+attach.
+
+### Required binaries
+
+There are two mandatory artifacts to install, and one optional component:
+
+* `kvm` is the driver itself
+* `kvm.conf` is the driver configuration file
+* `kvm.so` is the mdb module
+
+On the target machine, place `kvm` in `/kernel/drv/amd64` and `kvm.conf`
+in `/kernel/drv` then:
+
+ # add_drv kvm
+ # ln -s /devices/pseudo/kvm@0:kvm /dev/kvm
+
+Running illumos-kvm
+-------------------
+
+To run KVM, you will need the build product of the illumos-kvm-cmd repo:
+`qemu-system-x86_64`; please follow the instructions in the illumos-kvm-cmd
+repo to execute QEMU such that KVM is enabled.
+
+Monitoring illumos-kvm
+----------------------
+
+Once one or more VMs are running, there is a variety of tooling to help
+understand the operating characteristics of the system.
+
+### kvmstat
+
+The `kvmstat` command, found in the illumos repository, can be used to monitor
+VMs. For example, here is one second of `kvmstat` output from a machine
+running two VMs (one 2 VCPU instance running Linux; another 4 VCPU instance
+running the illumos-derived SmartOS):
+
+ pid vcpu | exits : haltx irqx irqwx iox mmiox | irqs emul eptv
+ 4668 0 | 23 : 6 0 0 1 0 | 6 16 0
+ 4668 1 | 25 : 6 1 0 1 0 | 6 16 0
+ 5026 0 | 17833 : 223 2946 707 106 0 | 3379 13315 0
+ 5026 1 | 18687 : 244 2761 512 0 0 | 3085 14803 0
+ 5026 2 | 15696 : 194 3452 542 0 0 | 3568 11230 0
+ 5026 3 | 16822 : 244 2817 487 0 0 | 3100 12963 0
+
+As for the meaning of the columns, they are explained with `kvmstat -h`:
+
+ # kvmstat -h
+ Usage: kvmstat [interval [count]]
+
+ Displays statistics for running kernel virtual machines, with one line
+ per virtual CPU. All statistics are reported as per-second rates.
+
+ The columns are as follows:
+
+ pid => identifier of process controlling the virtual CPU
+ vcpu => virtual CPU identifier relative to its virtual machine
+ exits => virtual machine exits for the virtual CPU
+ haltx => virtual machine exits due to the HLT instruction
+ irqx => virtual machine exits due to a pending external interrupt
+ irqwx => virtual machine exits due to an open interrupt window
+ iox => virtual machine exits due to an I/O instruction
+ mmiox => virtual machine exits due to memory mapped I/O
+ irqs => interrupts injected into the virtual CPU
+ emul => instructions emulated in the kernel
+ eptv => extended page table violations
+
+### kstat
+
+As one might expect, `kvmstat` is implemented in terms of kstat. You
+can use `kstat(1)` to browse the kstats from the `kvm` module:
+
+ # kstat -m kvm
+ ...
+ module: kvm instance: 0
+ name: vcpu-4 class: misc
+ crtime 4407.142410068
+ exits 5367443
+ fpu-reload 57302
+ halt-exits 317275
+ halt-wakeup 8991
+ host-state-reload 503920
+ hypercalls 0
+ insn-emulation 3043881
+ inst-emulation-fail 0
+ invlpg 0
+ io-exits 237191
+ irq-exits 1668
+ irq-injections 320339
+ irq-window-exits 1635
+ mmio-exits 617
+ nmi-injections 0
+ nmi-window-exits 0
+ pf-fixed 163629
+ pf-guest 0
+ pid 3949
+ request-irq-exits 0
+ signal-exits 460
+ snaptime 43219.723435123
+ zonename global
+
+ module: kvm instance: 4
+ name: vm class: misc
+ crtime 4407.1241134
+ lpages 0
+ mmu-cache-miss 950
+ mmu-flooded 0
+ mmu-pte-updated 0
+ mmu-pte-write 56360
+ mmu-pte-zapped 0
+ mmu-recycled 0
+ mmu-unsync-page 0
+ pid 3949
+ remote-tlb-flush 1511
+ snaptime 43219.723875091
+ zonename global
+
+
+### DTrace
+
+While there is not currently a stable KVM provider, there are many SDT probes
+in KVM; `dtrace -l -m sdt:kvm` to list these.
+
+Of these, of particular note are the `kvm-guest-entry` and `kvm-guest-exit`
+probes, which fire upon entry to and exit from a guest virtual machine. To
+determine context, one can use the `vmregs` variable present in illumos.
+
+For example, here's a simple script that shows histograms of time spent in VM
+guests on a per-PID and per-VCPU basis:
+
+ #pragma D option quiet
+
+ kvm-guest-entry
+ {
+ self->entry = timestamp;
+ }
+
+ kvm-guest-exit
+ /self->entry/
+ {
+ @[pid, vmregs[VMX_VIRTUAL_PROCESSOR_ID]] =
+ quantize(timestamp - self->entry);
+ }
+
+ END
+ {
+ printa("pid %d, vcpu %d: %@d\n", @);
+ }
+
+Here's what the output of running the above might look like:
+
+ pid 3949, vcpu 1:
+ value ------------- Distribution ------------- count
+ 512 | 0
+ 1024 |@@@@@@@@@@@@@ 26805
+ 2048 |@@@@@ 11641
+ 4096 |@@@@@@@ 14187
+ 8192 |@ 1559
+ 16384 |@ 2931
+ 32768 |@@@ 5653
+ 65536 |@@@@ 8385
+ 131072 |@@@ 6926
+ 262144 |@@@ 6639
+ 524288 | 785
+ 1048576 | 0
+
+There are many other ways in which DTrace can be used to understand either
+host or guest behavior; see the `tools` subdirectory from some sample D
+scripts.
+
+### mdb
+
+The `kvm.so` build product is an mdb module that contains several useful
+commands, including a `kvm` walker to iterate over all `struct kvm`
+structures.
+
+Contributing to illumos-kvm
+---------------------------
+
+Unless and until its volume dictate that it be elsewhere, illumos KVM
+discussion should be on the `illumos-developer` mailing list.
+Contributions are happily accepted; please send patches to
+`illumos-developer`.
+