HVM-596 need README.md for illumos-kvm

1 files changed, 283 insertions, 0 deletions

diff --git a/README.md b/README.md
new file mode 100644
index 0000000..76cc12e
--- /dev/null
+++ b/README.md
@@ -0,0 +1,283 @@
+illumos-kvm: KVM for illumos
+============================
+
+KVM is the kernel virtual machine, a framework for the in-kernel acceleration
+of QEMU.  illumos-kvm is a port of KVM to illumos, taking advantage of
+illumos-specific constructs like DTrace, cyclics, mdb, kstat, OS
+virtualization, network virtualization, ZFS, etc.  It is derived from the KVM
+source for Linux 2.6.34, the longterm source for which may be found here:
+
+    git://git.kernel.org/pub/scm/linux/kernel/git/longterm/linux-2.6.34.y.git
+
+To date, this implementation has been verified with a wide range of guest
+operating systems including illumos itself (both SmartOS and OpenIndiana
+distributions), FreeBSD, Plan 9, QNX, ChromeOS, HaikuOS, Microsoft Windows
+and Linux.
+
+The design center for this work is to use the virtualization features made
+available in the microprocessor -- and in particular, Intel's VMX.  As such,
+behavior on microprocessors that do not support VMX -- and more specifically,
+the extended page tables (EPT) found in second generation VMX support --
+should be graceful failure, not degraded operation.
+
+Divergences from KVM
+--------------------
+
+Divergences from KVM fall into several broad categories:  some functionality
+has been removed or not implemented because it is obviated by features of
+illumos (e.g., the custom tracing facility built into KVM); some functionality
+has been removed because it is only relevant to hardware that lacks
+virtualization support (e.g., older x86 hardware) or on hardware for which
+illumos lacks support (e.g., PPC, s390); and some functionality has been
+removed because the implementation complexity was simply too great relative
+to its value.
+
+Of this latter category, three areas of divergence merit special note.  First,
+there is no support for pageable guest memory (that is, guest memory is locked
+down).  While this is an opinionated decision at some level (in our
+experience, memory oversell leads to unacceptable pathologies in all but the
+idlest of workloads), we would welcome the work to integrate the KVM MMU
+notifier support into illumos-kvm.
+
+Second (and relatedly), illumos itself has no support for kernel same-page
+mapping (KSM) as found in Linux.  While illumos could in principle add such
+support, it is our experience that the memory that accrues from this is not
+sufficiently significant to pay for the increase in implementation and
+operator complexity.
+
+Finally, there is no support currently for AMD SVM.  This is not a value
+judgement of AMD's technology, but rather a reflection of limited engineering
+and testing resources.  (In the spirit of full disclosure, it should be said
+that the sponsor of illumos-kvm, Joyent, is an Intel-funded company -- but the
+lack of AMD support reflects only engineering prioritization and lack of
+testing infrastructure; AMD SVM support would be most welcome should someone
+in the community be so motivated as to port and test it.)
+
+Building illumos-kvm
+--------------------
+
+### Preparation
+
+Edit the Makefile and appropriately set the path for the `KERNEL_SOURCE`
+directory to point to the root of a checked out and built illumos directory.
+Building illumos KVM requires several recent additions to illumos,
+so be sure your illumos is up to date.
+
+Verify that gcc is installed and the version you'd like to use is in your path.
+Note that this has only been tested against the SFW version of gcc -- version
+3.4.
+
+Verify that you either have SUNWmake or GNU make installed.
+
+### Building
+
+To build, simply use the default make target:
+
+    $ make
+
+To check style, header files, and other various nits:
+
+    $ make check
+
+Installing illumos-kvm
+----------------------
+
+### System requirements
+
+To run illumos-kvm, you will need an illumos that has the fix for issue
+1347 (integrated on 2011-08-11). Further, your machine will need to
+support VMX.  To see if your machine supports VMX, run `isainfo -v` and
+look for `vmx`, e.g.:
+
+      % isainfo -v
+      64-bit amd64 applications
+            vmx sse4.2 sse4.1 ssse3 popcnt tscp cx16 sse3 sse2 sse fxsr mmx 
+            cmov amd_sysc cx8 tsc fpu 
+      32-bit i386 applications
+            vmx sse4.2 sse4.1 ssse3 popcnt tscp ahf cx16 sse3 sse2 sse fxsr mmx 
+            cmov sep cx8 tsc fpu 
+
+If you do not see `vmx` in this output, the `kvm` driver will be unable to
+attach.
+
+### Required binaries
+
+There are two mandatory artifacts to install, and one optional component:
+
+* `kvm` is the driver itself
+* `kvm.conf` is the driver configuration file
+* `kvm.so` is the mdb module
+
+On the target machine, place `kvm` in `/kernel/drv/amd64` and `kvm.conf`
+in `/kernel/drv` then:
+
+    # add_drv kvm
+    # ln -s /devices/pseudo/kvm@0:kvm /dev/kvm
+
+Running illumos-kvm
+-------------------
+
+To run KVM, you will need the build product of the illumos-kvm-cmd repo:
+`qemu-system-x86_64`; please follow the instructions in the illumos-kvm-cmd
+repo to execute QEMU such that KVM is enabled.
+
+Monitoring illumos-kvm
+----------------------
+
+Once one or more VMs are running, there is a variety of tooling to help
+understand the operating characteristics of the system.
+
+### kvmstat
+
+The `kvmstat` command, found in the illumos repository, can be used to monitor
+VMs. For example, here is one second of `kvmstat` output from a machine
+running two VMs (one 2 VCPU instance running Linux; another 4 VCPU instance
+running the illumos-derived SmartOS):
+
+       pid vcpu |  exits :  haltx   irqx  irqwx    iox  mmiox |   irqs   emul   eptv
+      4668    0 |     23 :      6      0      0      1      0 |      6     16      0
+      4668    1 |     25 :      6      1      0      1      0 |      6     16      0
+      5026    0 |  17833 :    223   2946    707    106      0 |   3379  13315      0
+      5026    1 |  18687 :    244   2761    512      0      0 |   3085  14803      0
+      5026    2 |  15696 :    194   3452    542      0      0 |   3568  11230      0
+      5026    3 |  16822 :    244   2817    487      0      0 |   3100  12963      0
+
+As for the meaning of the columns, they are explained with `kvmstat -h`:
+
+      # kvmstat -h
+      Usage: kvmstat [interval [count]]
+
+        Displays statistics for running kernel virtual machines, with one line
+        per virtual CPU.  All statistics are reported as per-second rates.
+
+        The columns are as follows:
+
+          pid    =>  identifier of process controlling the virtual CPU
+          vcpu   =>  virtual CPU identifier relative to its virtual machine
+          exits  =>  virtual machine exits for the virtual CPU
+          haltx  =>  virtual machine exits due to the HLT instruction
+          irqx   =>  virtual machine exits due to a pending external interrupt
+          irqwx  =>  virtual machine exits due to an open interrupt window
+          iox    =>  virtual machine exits due to an I/O instruction
+          mmiox  =>  virtual machine exits due to memory mapped I/O 
+          irqs   =>  interrupts injected into the virtual CPU
+          emul   =>  instructions emulated in the kernel
+          eptv   =>  extended page table violations
+
+### kstat
+
+As one might expect, `kvmstat` is implemented in terms of kstat.  You
+can use `kstat(1)` to browse the kstats from the `kvm` module:
+
+       # kstat -m kvm
+       ...
+       module: kvm                      instance: 0     
+       name:   vcpu-4                   class:    misc
+        crtime                          4407.142410068
+        exits                           5367443
+        fpu-reload                      57302
+        halt-exits                      317275
+        halt-wakeup                     8991
+        host-state-reload               503920
+        hypercalls                      0
+        insn-emulation                  3043881
+        inst-emulation-fail             0
+        invlpg                          0
+        io-exits                        237191
+        irq-exits                       1668
+        irq-injections                  320339
+        irq-window-exits                1635
+        mmio-exits                      617
+        nmi-injections                  0
+        nmi-window-exits                0
+        pf-fixed                        163629
+        pf-guest                        0
+        pid                             3949
+        request-irq-exits               0
+        signal-exits                    460
+        snaptime                        43219.723435123
+        zonename                        global
+       
+       module: kvm                      instance: 4     
+       name:   vm                       class:    misc
+        crtime                          4407.1241134
+        lpages                          0
+        mmu-cache-miss                  950
+        mmu-flooded                     0
+        mmu-pte-updated                 0
+        mmu-pte-write                   56360
+        mmu-pte-zapped                  0
+        mmu-recycled                    0
+        mmu-unsync-page                 0
+        pid                             3949
+        remote-tlb-flush                1511
+        snaptime                        43219.723875091
+        zonename                        global
+       
+
+### DTrace
+
+While there is not currently a stable KVM provider, there are many SDT probes
+in KVM; `dtrace -l -m sdt:kvm` to list these.
+
+Of these, of particular note are the `kvm-guest-entry` and `kvm-guest-exit`
+probes, which fire upon entry to and exit from a guest virtual machine.  To
+determine context, one can use the `vmregs` variable present in illumos.
+
+For example, here's a simple script that shows histograms of time spent in VM
+guests on a per-PID and per-VCPU basis:
+
+        #pragma D option quiet
+        
+        kvm-guest-entry
+        {
+                self->entry = timestamp;
+        }
+        
+        kvm-guest-exit
+        /self->entry/
+        {
+                @[pid, vmregs[VMX_VIRTUAL_PROCESSOR_ID]] =
+                    quantize(timestamp - self->entry);
+        }
+        
+        END
+        {
+                printa("pid %d, vcpu %d: %@d\n", @);
+        }
+
+Here's what the output of running the above might look like:
+
+      pid 3949, vcpu 1: 
+           value  ------------- Distribution ------------- count    
+             512 |                                         0        
+            1024 |@@@@@@@@@@@@@                            26805    
+            2048 |@@@@@                                    11641    
+            4096 |@@@@@@@                                  14187    
+            8192 |@                                        1559     
+           16384 |@                                        2931     
+           32768 |@@@                                      5653     
+           65536 |@@@@                                     8385     
+          131072 |@@@                                      6926     
+          262144 |@@@                                      6639     
+          524288 |                                         785      
+         1048576 |                                         0        
+
+There are many other ways in which DTrace can be used to understand either
+host or guest behavior; see the `tools` subdirectory from some sample D
+scripts.
+
+### mdb
+
+The `kvm.so` build product is an mdb module that contains several useful
+commands, including a `kvm` walker to iterate over all `struct kvm`
+structures.
+
+Contributing to illumos-kvm
+---------------------------
+
+Unless and until its volume dictate that it be elsewhere, illumos KVM
+discussion should be on the `illumos-developer` mailing list.
+Contributions are happily accepted; please send patches to 
+`illumos-developer`.
+