diff options
Diffstat (limited to 'doc/articles')
53 files changed, 5435 insertions, 0 deletions
diff --git a/doc/articles/c_go_cgo.html b/doc/articles/c_go_cgo.html new file mode 100644 index 000000000..ac6bb29a2 --- /dev/null +++ b/doc/articles/c_go_cgo.html @@ -0,0 +1,180 @@ +<!--{ +"Title": "C? Go? Cgo!", +"Template": true +}--> + +<p> +Cgo lets Go packages call C code. Given a Go source file written with some +special features, cgo outputs Go and C files that can be combined into a +single Go package. +</p> + +<p> +To lead with an example, here's a Go package that provides two functions - +<code>Random</code> and <code>Seed</code> - that wrap C's <code>random</code> +and <code>srandom</code> functions. +</p> + +{{code "/doc/progs/cgo1.go" `/package rand/` `/END/`}} + +<p> +Let's look at what's happening here, starting with the import statement. +</p> + +<p> +The <code>rand</code> package imports <code>"C"</code>, but you'll find there's +no such package in the standard Go library. That's because <code>C</code> is a +"pseudo-package", a special name interpreted by cgo as a reference to C's +name space. +</p> + +<p> +The <code>rand</code> package contains four references to the <code>C</code> +package: the calls to <code>C.random</code> and <code>C.srandom</code>, the +conversion <code>C.uint(i)</code>, and the <code>import</code> statement. +</p> + +<p> +The <code>Random</code> function calls the standard C library's <code>random</code> +function and returns the result. In C, <code>random</code> returns a value of the +C type <code>long</code>, which cgo represents as the type <code>C.long</code>. +It must be converted to a Go type before it can be used by Go code outside this +package, using an ordinary Go type conversion: +</p> + +{{code "/doc/progs/cgo1.go" `/func Random/` `/STOP/`}} + +<p> +Here's an equivalent function that uses a temporary variable to illustrate +the type conversion more explicitly: +</p> + +{{code "/doc/progs/cgo2.go" `/func Random/` `/STOP/`}} + +<p> +The <code>Seed</code> function does the reverse, in a way. It takes a +regular Go <code>int</code>, converts it to the C <code>unsigned int</code> +type, and passes it to the C function <code>srandom</code>. +</p> + +{{code "/doc/progs/cgo1.go" `/func Seed/` `/END/`}} + +<p> +Note that cgo knows the <code>unsigned int</code> type as <code>C.uint</code>; +see the <a href="/cmd/cgo">cgo documentation</a> for a complete list of +these numeric type names. +</p> + +<p> +The one detail of this example we haven't examined yet is the comment +above the <code>import</code> statement. +</p> + +{{code "/doc/progs/cgo1.go" `/\/\*/` `/STOP/`}} + +<p> +Cgo recognizes this comment. Any lines starting +with <code>#cgo</code> +followed +by a space character are removed; these become directives for cgo. +The remaining lines are used as a header when compiling the C parts of +the package. In this case those lines are just a +single <code>#include</code> +statement, but they can be almost any C code. The <code>#cgo</code> +directives are +used to provide flags for the compiler and linker when building the C +parts of the package. +</p> + +<p> +There is a limitation: if your program uses any <code>//export</code> +directives, then the C code in the comment may only include declarations +(<code>extern int f();</code>), not definitions (<code>int f() { +return 1; }</code>). You can use <code>//export</code> directives to +make Go functions accessible to C code. +</p> + +<p> +The <code>#cgo</code> and <code>//export</code> directives are +documented in +the <a href="/cmd/cgo/">cgo documentation</a>. +</p> + +<p> +<b>Strings and things</b> +</p> + +<p> +Unlike Go, C doesn't have an explicit string type. Strings in C are +represented by a zero-terminated array of chars. +</p> + +<p> +Conversion between Go and C strings is done with the +<code>C.CString</code>, <code>C.GoString</code>, and +<code>C.GoStringN</code> functions. These conversions make a copy of the +string data. +</p> + +<p> +This next example implements a <code>Print</code> function that writes a +string to standard output using C's <code>fputs</code> function from the +<code>stdio</code> library: +</p> + +{{code "/doc/progs/cgo3.go" `/package print/` `/END/`}} + +<p> +Memory allocations made by C code are not known to Go's memory manager. +When you create a C string with <code>C.CString</code> (or any C memory +allocation) you must remember to free the memory when you're done with it +by calling <code>C.free</code>. +</p> + +<p> +The call to <code>C.CString</code> returns a pointer to the start of the +char array, so before the function exits we convert it to an +<a href="/pkg/unsafe/#Pointer"><code>unsafe.Pointer</code></a> and release +the memory allocation with <code>C.free</code>. A common idiom in cgo programs +is to <a href="/doc/articles/defer_panic_recover.html"><code>defer</code></a> +the free immediately after allocating (especially when the code that follows +is more complex than a single function call), as in this rewrite of +<code>Print</code>: +</p> + +{{code "/doc/progs/cgo4.go" `/func Print/` `/END/`}} + +<p> +<b>Building cgo packages</b> +</p> + +<p> +To build cgo packages, just use <a href="/cmd/go/#Compile_packages_and_dependencies">" +<code>go build</code>"</a> or +<a href="/cmd/go/#Compile_and_install_packages_and_dependencies">"<code>go install</code> +"</a> as usual. The go tool recognizes the special <code>"C"</code> import and automatically +uses cgo for those files. +</p> + +<p> +<b>More cgo resources</b> +</p> + +<p> +The <a href="/cmd/cgo/">cgo command</a> documentation has more detail about +the C pseudo-package and the build process. The <a href="/misc/cgo/">cgo examples</a> +in the Go tree demonstrate more advanced concepts. +</p> + +<p> +For a simple, idiomatic example of a cgo-based package, see Russ Cox's <a +href="http://code.google.com/p/gosqlite/source/browse/sqlite/sqlite.go">gosqlite</a>. +Also, the Go Project Dashboard lists <a +href="https://godashboard.appspot.com/project?tag=cgo">several other +cgo packages</a>. +</p> + +<p> +Finally, if you're curious as to how all this works internally, take a look +at the introductory comment of the runtime package's <a href="/src/pkg/runtime/cgocall.c">cgocall.c</a>. +</p> diff --git a/doc/articles/concurrency_patterns.html b/doc/articles/concurrency_patterns.html new file mode 100644 index 000000000..63c8cd59e --- /dev/null +++ b/doc/articles/concurrency_patterns.html @@ -0,0 +1,79 @@ +<!--{ +"Title": "Go Concurrency Patterns: Timing out, moving on", +"Template": true +}--> + +<p> +Concurrent programming has its own idioms. A good example is timeouts. Although +Go's channels do not support them directly, they are easy to implement. Say we +want to receive from the channel <code>ch</code>, but want to wait at most one +second for the value to arrive. We would start by creating a signalling channel +and launching a goroutine that sleeps before sending on the channel: +</p> + +{{code "/doc/progs/timeout1.go" `/timeout :=/` `/STOP/`}} + +<p> +We can then use a <code>select</code> statement to receive from either +<code>ch</code> or <code>timeout</code>. If nothing arrives on <code>ch</code> +after one second, the timeout case is selected and the attempt to read from +<cde>ch</cde> is abandoned. +</p> + +{{code "/doc/progs/timeout1.go" `/select {/` `/STOP/`}} + +<p> +The <code>timeout</code> channel is buffered with space for 1 value, allowing +the timeout goroutine to send to the channel and then exit. The goroutine +doesn't know (or care) whether the value is received. This means the goroutine +won't hang around forever if the <code>ch</code> receive happens before the +timeout is reached. The <code>timeout</code> channel will eventually be +deallocated by the garbage collector. +</p> + +<p> +(In this example we used <code>time.Sleep</code> to demonstrate the mechanics +of goroutines and channels. In real programs you should use <code> +<a href="/pkg/time/#After">time.After</a></code>, a function that returns +a channel and sends on that channel after the specified duration.) +</p> + +<p> +Let's look at another variation of this pattern. In this example we have a +program that reads from multiple replicated databases simultaneously. The +program needs only one of the answers, and it should accept the answer that +arrives first. +</p> + +<p> +The function <code>Query</code> takes a slice of database connections and a +<code>query</code> string. It queries each of the databases in parallel and +returns the first response it receives: +</p> + +{{code "/doc/progs/timeout2.go" `/func Query/` `/STOP/`}} + +<p> +In this example, the closure does a non-blocking send, which it achieves by +using the send operation in <code>select</code> statement with a +<code>default</code> case. If the send cannot go through immediately the +default case will be selected. Making the send non-blocking guarantees that +none of the goroutines launched in the loop will hang around. However, if the +result arrives before the main function has made it to the receive, the send +could fail since no one is ready. +</p> + +<p> +This problem is a textbook of example of what is known as a +<a href="https://en.wikipedia.org/wiki/Race_condition">race condition</a>, but +the fix is trivial. We just make sure to buffer the channel <code>ch</code> (by +adding the buffer length as the second argument to <a href="/pkg/builtin/#make">make</a>), +guaranteeing that the first send has a place to put the value. This ensures the +send will always succeed, and the first value to arrive will be retrieved +regardless of the order of execution. +</p> + +<p> +These two examples demonstrate the simplicity with which Go can express complex +interactions between goroutines. +</p> diff --git a/doc/articles/defer_panic_recover.html b/doc/articles/defer_panic_recover.html new file mode 100644 index 000000000..c964cd368 --- /dev/null +++ b/doc/articles/defer_panic_recover.html @@ -0,0 +1,197 @@ +<!--{ + "Title": "Defer, Panic, and Recover", + "Template": true +}--> + +<p> +Go has the usual mechanisms for control flow: if, for, switch, goto. It also +has the go statement to run code in a separate goroutine. Here I'd like to +discuss some of the less common ones: defer, panic, and recover. +</p> + +<p> +A <b>defer statement</b> pushes a function call onto a list. The list of saved +calls is executed after the surrounding function returns. Defer is commonly +used to simplify functions that perform various clean-up actions. +</p> + +<p> +For example, let's look at a function that opens two files and copies the +contents of one file to the other: +</p> + +{{code "/doc/progs/defer.go" `/func CopyFile/` `/STOP/`}} + +<p> +This works, but there is a bug. If the call to os.Create fails, the +function will return without closing the source file. This can be easily +remedied by putting a call to src.Close before the second return statement, +but if the function were more complex the problem might not be so easily +noticed and resolved. By introducing defer statements we can ensure that the +files are always closed: +</p> + +{{code "/doc/progs/defer2.go" `/func CopyFile/` `/STOP/`}} + +<p> +Defer statements allow us to think about closing each file right after opening +it, guaranteeing that, regardless of the number of return statements in the +function, the files <i>will</i> be closed. +</p> + +<p> +The behavior of defer statements is straightforward and predictable. There are +three simple rules: +</p> + +<p> +1. <i>A deferred function's arguments are evaluated when the defer statement is +evaluated.</i> +</p> + +<p> +In this example, the expression "i" is evaluated when the Println call is +deferred. The deferred call will print "0" after the function returns. +</p> + +{{code "/doc/progs/defer.go" `/func a/` `/STOP/`}} + +<p> +2. <i>Deferred function calls are executed in Last In First Out order +</i>after<i> the surrounding function returns.</i> +</p> + +<p> +This function prints "3210": +</p> + +{{code "/doc/progs/defer.go" `/func b/` `/STOP/`}} + +<p> +3. <i>Deferred functions may read and assign to the returning function's named +return values.</i> +</p> + +<p> +In this example, a deferred function increments the return value i <i>after</i> +the surrounding function returns. Thus, this function returns 2: +</p> + +{{code "/doc/progs/defer.go" `/func c/` `/STOP/`}} + +<p> +This is convenient for modifying the error return value of a function; we will +see an example of this shortly. +</p> + +<p> +<b>Panic</b> is a built-in function that stops the ordinary flow of control and +begins <i>panicking</i>. When the function F calls panic, execution of F stops, +any deferred functions in F are executed normally, and then F returns to its +caller. To the caller, F then behaves like a call to panic. The process +continues up the stack until all functions in the current goroutine have +returned, at which point the program crashes. Panics can be initiated by +invoking panic directly. They can also be caused by runtime errors, such as +out-of-bounds array accesses. +</p> + +<p> +<b>Recover</b> is a built-in function that regains control of a panicking +goroutine. Recover is only useful inside deferred functions. During normal +execution, a call to recover will return nil and have no other effect. If the +current goroutine is panicking, a call to recover will capture the value given +to panic and resume normal execution. +</p> + +<p> +Here's an example program that demonstrates the mechanics of panic and defer: +</p> + +{{code "/doc/progs/defer2.go" `/package main/` `/STOP/`}} + +<p> +The function g takes the int i, and panics if i is greater than 3, or else it +calls itself with the argument i+1. The function f defers a function that calls +recover and prints the recovered value (if it is non-nil). Try to picture what +the output of this program might be before reading on. +</p> + +<p> +The program will output: +</p> + +<pre>Calling g. +Printing in g 0 +Printing in g 1 +Printing in g 2 +Printing in g 3 +Panicking! +Defer in g 3 +Defer in g 2 +Defer in g 1 +Defer in g 0 +Recovered in f 4 +Returned normally from f.</pre> + +<p> +If we remove the deferred function from f the panic is not recovered and +reaches the top of the goroutine's call stack, terminating the program. This +modified program will output: +</p> + +<pre>Calling g. +Printing in g 0 +Printing in g 1 +Printing in g 2 +Printing in g 3 +Panicking! +Defer in g 3 +Defer in g 2 +Defer in g 1 +Defer in g 0 +panic: 4 + +panic PC=0x2a9cd8 +[stack trace omitted]</pre> + +<p> +For a real-world example of <b>panic</b> and <b>recover</b>, see the +<a href="/pkg/encoding/json/">json package</a> from the Go standard library. +It decodes JSON-encoded data with a set of recursive functions. +When malformed JSON is encountered, the parser calls panic to unwind the +stack to the top-level function call, which recovers from the panic and returns +an appropriate error value (see the 'error' and 'unmarshal' methods of +the decodeState type in +<a href="/src/pkg/encoding/json/decode.go">decode.go</a>). +</p> + +<p> +The convention in the Go libraries is that even when a package uses panic +internally, its external API still presents explicit error return values. +</p> + +<p> +Other uses of <b>defer</b> (beyond the file.Close example given earlier) +include releasing a mutex: +</p> + +<pre>mu.Lock() +defer mu.Unlock()</pre> + +<p> +printing a footer: +</p> + +<pre>printHeader() +defer printFooter()</pre> + +<p> +and more. +</p> + +<p> +In summary, the defer statement (with or without panic and recover) provides an +unusual and powerful mechanism for control flow. It can be used to model a +number of features implemented by special-purpose structures in other +programming languages. Try it out. +</p> diff --git a/doc/articles/error_handling.html b/doc/articles/error_handling.html new file mode 100644 index 000000000..8f4fffb48 --- /dev/null +++ b/doc/articles/error_handling.html @@ -0,0 +1,316 @@ +<!--{ + "Title": "Error Handling and Go", + "Template": true +}--> + +<p> +If you have written any Go code you have probably encountered the built-in +<code>error</code> type. Go code uses <code>error</code> values to +indicate an abnormal state. For example, the <code>os.Open</code> function +returns a non-nil <code>error</code> value when it fails to open a file. +</p> + +{{code "/doc/progs/error.go" `/func Open/`}} + +<p> +The following code uses <code>os.Open</code> to open a file. If an error +occurs it calls <code>log.Fatal</code> to print the error message and stop. +</p> + +{{code "/doc/progs/error.go" `/func openFile/` `/STOP/`}} + +<p> +You can get a lot done in Go knowing just this about the <code>error</code> +type, but in this article we'll take a closer look at <code>error</code> and +discuss some good practices for error handling in Go. +</p> + +<p> +<b>The error type</b> +</p> + +<p> +The <code>error</code> type is an interface type. An <code>error</code> +variable represents any value that can describe itself as a string. Here is the +interface's declaration: +</p> + +<pre>type error interface { + Error() string +}</pre> + +<p> +The <code>error</code> type, as with all built in types, is +<a href="/doc/go_spec.html#Predeclared_identifiers">predeclared</a> in the +<a href="/doc/go_spec.html#Blocks">universe block</a>. +</p> + +<p> +The most commonly-used <code>error</code> implementation is the +<a href="/pkg/errors/">errors</a> package's unexported <code>errorString</code> type. +</p> + +{{code "/doc/progs/error.go" `/errorString/` `/STOP/`}} + +<p> +You can construct one of these values with the <code>errors.New</code> +function. It takes a string that it converts to an <code>errors.errorString</code> +and returns as an <code>error</code> value. +</p> + +{{code "/doc/progs/error.go" `/New/` `/STOP/`}} + +<p> +Here's how you might use <code>errors.New</code>: +</p> + +{{code "/doc/progs/error.go" `/func Sqrt/` `/STOP/`}} + +<p> +A caller passing a negative argument to <code>Sqrt</code> receives a non-nil +<code>error</code> value (whose concrete representation is an +<code>errors.errorString</code> value). The caller can access the error string +("math: square root of...") by calling the <code>error</code>'s +<code>Error</code> method, or by just printing it: +</p> + +{{code "/doc/progs/error.go" `/func printErr/` `/STOP/`}} + +<p> +The <a href="/pkg/fmt/">fmt</a> package formats an <code>error</code> value +by calling its <code>Error() string</code> method. +</p> + +<p> +It is the error implementation's responsibility to summarize the context. +The error returned by <code>os.Open</code> formats as "open /etc/passwd: +permission denied," not just "permission denied." The error returned by our +<code>Sqrt</code> is missing information about the invalid argument. +</p> + +<p> +To add that information, a useful function is the <code>fmt</code> package's +<code>Errorf</code>. It formats a string according to <code>Printf</code>'s +rules and returns it as an <code>error</code> created by +<code>errors.New</code>. +</p> + +{{code "/doc/progs/error.go" `/fmtError/` `/STOP/`}} + +<p> +In many cases <code>fmt.Errorf</code> is good enough, but since +<code>error</code> is an interface, you can use arbitrary data structures as +error values, to allow callers to inspect the details of the error. +</p> + +<p> +For instance, our hypothetical callers might want to recover the invalid +argument passed to <code>Sqrt</code>. We can enable that by defining a new +error implementation instead of using <code>errors.errorString</code>: +</p> + +{{code "/doc/progs/error.go" `/type NegativeSqrtError/` `/STOP/`}} + +<p> +A sophisticated caller can then use a +<a href="/doc/go_spec.html#Type_assertions">type assertion</a> to check for a +<code>NegativeSqrtError</code> and handle it specially, while callers that just +pass the error to <code>fmt.Println</code> or <code>log.Fatal</code> will see +no change in behavior. +</p> + +<p> +As another example, the <a href="/pkg/encoding/json/">json</a> package specifies a +<code>SyntaxError</code> type that the <code>json.Decode</code> function +returns when it encounters a syntax error parsing a JSON blob. +</p> + +{{code "/doc/progs/error.go" `/type SyntaxError/` `/STOP/`}} + +<p> +The <code>Offset</code> field isn't even shown in the default formatting of the +error, but callers can use it to add file and line information to their error +messages: +</p> + +{{code "/doc/progs/error.go" `/func decodeError/` `/STOP/`}} + +<p> +(This is a slightly simplified version of some +<a href="http://camlistore.org/code/?p=camlistore.git;a=blob;f=lib/go/camli/jsonconfig/eval.go#l68">actual code</a> +from the <a href="http://camlistore.org">Camlistore</a> project.) +</p> + +<p> +The <code>error</code> interface requires only a <code>Error</code> method; +specific error implementations might have additional methods. For instance, the +<a href="/pkg/net/">net</a> package returns errors of type +<code>error</code>, following the usual convention, but some of the error +implementations have additional methods defined by the <code>net.Error</code> +interface: +</p> + +<pre>package net + +type Error interface { + error + Timeout() bool // Is the error a timeout? + Temporary() bool // Is the error temporary? +}</pre> + +<p> +Client code can test for a <code>net.Error</code> with a type assertion and +then distinguish transient network errors from permanent ones. For instance, a +web crawler might sleep and retry when it encounters a temporary error and give +up otherwise. +</p> + +{{code "/doc/progs/error.go" `/func netError/` `/STOP/`}} + +<p> +<b>Simplifying repetitive error handling</b> +</p> + +<p> +In Go, error handling is important. The language's design and conventions +encourage you to explicitly check for errors where they occur (as distinct from +the convention in other languages of throwing exceptions and sometimes catching +them). In some cases this makes Go code verbose, but fortunately there are some +techniques you can use to minimize repetitive error handling. +</p> + +<p> +Consider an <a href="http://code.google.com/appengine/docs/go/">App Engine</a> +application with an HTTP handler that retrieves a record from the datastore and +formats it with a template. +</p> + +{{code "/doc/progs/error2.go" `/func init/` `/STOP/`}} + +<p> +This function handles errors returned by the <code>datastore.Get</code> +function and <code>viewTemplate</code>'s <code>Execute</code> method. In both +cases, it presents a simple error message to the user with the HTTP status code +500 ("Internal Server Error"). This looks like a manageable amount of code, but +add some more HTTP handlers and you quickly end up with many copies of +identical error handling code. +</p> + +<p> +To reduce the repetition we can define our own HTTP <code>appHandler</code> +type that includes an <code>error</code> return value: +</p> + +{{code "/doc/progs/error3.go" `/type appHandler/`}} + +<p> +Then we can change our <code>viewRecord</code> function to return errors: +</p> + +{{code "/doc/progs/error3.go" `/func viewRecord/` `/STOP/`}} + +<p> +This is simpler than the original version, but the <a +href="/pkg/net/http/">http</a> package doesn't understand functions that return +<code>error</code>. +To fix this we can implement the <code>http.Handler</code> interface's +<code>ServeHTTP</code> method on <code>appHandler</code>: +</p> + +{{code "/doc/progs/error3.go" `/ServeHTTP/` `/STOP/`}} + +<p> +The <code>ServeHTTP</code> method calls the <code>appHandler</code> function +and displays the returned error (if any) to the user. Notice that the method's +receiver, <code>fn</code>, is a function. (Go can do that!) The method invokes +the function by calling the receiver in the expression <code>fn(w, r)</code>. +</p> + +<p> +Now when registering <code>viewRecord</code> with the http package we use the +<code>Handle</code> function (instead of <code>HandleFunc</code>) as +<code>appHandler</code> is an <code>http.Handler</code> (not an +<code>http.HandlerFunc</code>). +</p> + +{{code "/doc/progs/error3.go" `/func init/` `/STOP/`}} + +<p> +With this basic error handling infrastructure in place, we can make it more +user friendly. Rather than just displaying the error string, it would be better +to give the user a simple error message with an appropriate HTTP status code, +while logging the full error to the App Engine developer console for debugging +purposes. +</p> + +<p> +To do this we create an <code>appError</code> struct containing an +<code>error</code> and some other fields: +</p> + +{{code "/doc/progs/error4.go" `/type appError/` `/STOP/`}} + +<p> +Next we modify the appHandler type to return <code>*appError</code> values: +</p> + +{{code "/doc/progs/error4.go" `/type appHandler/`}} + +<p> +(It's usually a mistake to pass back the concrete type of an error rather than +<code>error</code>, +for reasons discussed in <a href="/doc/go_faq.html#nil_error">the Go FAQ</a>, +but it's the right thing to do here because <code>ServeHTTP</code> is the only +place that sees the value and uses its contents.) +</p> + +<p> +And make <code>appHandler</code>'s <code>ServeHTTP</code> method display the +<code>appError</code>'s <code>Message</code> to the user with the correct HTTP +status <code>Code</code> and log the full <code>Error</code> to the developer +console: +</p> + +{{code "/doc/progs/error4.go" `/ServeHTTP/` `/STOP/`}} + +<p> +Finally, we update <code>viewRecord</code> to the new function signature and +have it return more context when it encounters an error: +</p> + +{{code "/doc/progs/error4.go" `/func viewRecord/` `/STOP/`}} + +<p> +This version of <code>viewRecord</code> is the same length as the original, but +now each of those lines has specific meaning and we are providing a friendlier +user experience. +</p> + +<p> +It doesn't end there; we can further improve the error handling in our +application. Some ideas: +</p> + +<ul> +<li>give the error handler a pretty HTML template, +<li>make debugging easier by writing the stack trace to the HTTP response when +the user is an administrator, +<li>write a constructor function for <code>appError</code> that stores the +stack trace for easier debugging, +<li>recover from panics inside the <code>appHandler</code>, logging the error +to the console as "Critical," while telling the user "a serious error +has occurred." This is a nice touch to avoid exposing the user to inscrutable +error messages caused by programming errors. +See the <a href="defer_panic_recover.html">Defer, Panic, and Recover</a> +article for more details. +</ul> + +<p> +<b>Conclusion</b> +</p> + +<p> +Proper error handling is an essential requirement of good software. By +employing the techniques described in this post you should be able to write +more reliable and succinct Go code. +</p> diff --git a/doc/articles/go_command.html b/doc/articles/go_command.html new file mode 100644 index 000000000..1e9e70fd8 --- /dev/null +++ b/doc/articles/go_command.html @@ -0,0 +1,265 @@ +<!--{ + "title": "About the go command" +}--> + +<p>The Go distribution includes a command, named +"<code><a href="/cmd/go/">go</a></code>", that +automates the downloading, building, installation, and testing of Go packages +and commands. This document talks about why we wrote a new command, what it +is, what it's not, and how to use it.</p> + +<h2>Motivation</h2> + +<p>You might have seen early Go talks in which Rob Pike jokes that the idea +for Go arose while waiting for a large Google server to compile. That +really was the motivation for Go: to build a language that worked well +for building the large software that Google writes and runs. It was +clear from the start that such a language must provide a way to +express dependencies between code libraries clearly, hence the package +grouping and the explicit import blocks. It was also clear from the +start that you might want arbitrary syntax for describing the code +being imported; this is why import paths are string literals.</p> + +<p>An explicit goal for Go from the beginning was to be able to build Go +code using only the information found in the source itself, not +needing to write a makefile or one of the many modern replacements for +makefiles. If Go needed a configuration file to explain how to build +your program, then Go would have failed.</p> + +<p>At first, there was no Go compiler, and the initial development +focused on building one and then building libraries for it. For +expedience, we postponed the automation of building Go code by using +make and writing makefiles. When compiling a single package involved +multiple invocations of the Go compiler, we even used a program to +write the makefiles for us. You can find it if you dig through the +repository history.</p> + +<p>The purpose of the new go command is our return to this ideal, that Go +programs should compile without configuration or additional effort on +the part of the developer beyond writing the necessary import +statements.</p> + +<h2>Configuration versus convention</h2> + +<p>The way to achieve the simplicity of a configuration-free system is to +establish conventions. The system works only to the extent that those conventions +are followed. When we first launched Go, many people published packages that +had to be installed in certain places, under certain names, using certain build +tools, in order to be used. That's understandable: that's the way it works in +most other languages. Over the last few years we consistently reminded people +about the <code>goinstall</code> command +(now replaced by <a href="/cmd/go/#Download_and_install_packages_and_dependencies"><code>go get</code></a>) +and its conventions: first, that the import path is derived in a known way from +the URL of the source code; second, that that the place to store the sources in +the local file system is derived in a known way from the import path; third, +that each directory in a source tree corresponds to a single package; and +fourth, that the package is built using only information in the source code. +Today, the vast majority of packages follow these conventions. +The Go ecosystem is simpler and more powerful as a result.</p> + +<p>We received many requests to allow a makefile in a package directory to +provide just a little extra configuration beyond what's in the source code. +But that would have introduced new rules. Because we did not accede to such +requests, we were able to write the go command and eliminate our use of make +or any other build system.</p> + +<p>It is important to understand that the go command is not a general +build tool. It cannot be configured and it does not attempt to build +anything but Go packages. These are important simplifying +assumptions: they simplify not only the implementation but also, more +important, the use of the tool itself.</p> + +<h2>Go's conventions</h2> + +<p>The <code>go</code> command requires that code adheres to a few key, +well-established conventions.</p> + +<p>First, the import path is derived in an known way from the URL of the +source code. For Bitbucket, GitHub, Google Code, and Launchpad, the +root directory of the repository is identified by the repository's +main URL, without the <code>http://</code> prefix. Subdirectories are named by +adding to that path. For example, the supplemental networking +libraries for Go are obtained by running</p> + +<pre> +hg clone http://code.google.com/p/go.net +</pre> + +<p>and thus the import path for the root directory of that repository is +"<code>code.google.com/p/go.net</code>". The websocket package is stored in a +subdirectory, so its import path is +"<code>code.google.com/p/go.net/websocket</code>".</p> + +<p>These paths are on the long side, but in exchange we get an +automatically managed name space for import paths and the ability for +a tool like the go command to look at an unfamiliar import path and +deduce where to obtain the source code.</p> + +<p>Second, the place to store sources in the local file system is derived +in a known way from the import path. Specifically, the first choice +is <code>$GOPATH/src/<import-path></code>. If <code>$GOPATH</code> is +unset, the go command will fall back to storing source code alongside the +standard Go packages, in <code>$GOROOT/src/pkg/<import-path></code>. +If <code>$GOPATH</code> is set to a list of paths, the go command tries +<code><dir>/src/<import-path></code> for each of the directories in +that list.</p> + +<p>Each of those trees contains, by convention, a top-level directory named +"<code>bin</code>", for holding compiled executables, and a top-level directory +named "<code>pkg</code>", for holding compiled packages that can be imported, +and the "<code>src</code>" directory, for holding package source files. +Imposing this structure lets us keep each of these directory trees +self-contained: the compiled form and the sources are always near each +other.</p> + +<p>These naming conventions also let us work in the reverse direction, +from a directory name to its import path. This mapping is important +for many of the go command's subcommands, as we'll see below.</p> + +<p>Third, each directory in a source tree corresponds to a single +package. By restricting a directory to a single package, we don't have +to create hybrid import paths that specify first the directory and +then the package within that directory. Also, most file management +tools and UIs work on directories as fundamental units. Tying the +fundamental Go unit—the package—to file system structure means +that file system tools become Go package tools. Copying, moving, or +deleting a package corresponds to copying, moving, or deleting a +directory.</p> + +<p>Fourth, each package is built using only the information present in +the source files. This makes it much more likely that the tool will +be able to adapt to changing build environments and conditions. For +example, if we allowed extra configuration such as compiler flags or +command line recipes, then that configuration would need to be updated +each time the build tools changed; it would also be inherently tied +to the use of a specific tool chain.</p> + +<h2>Getting started with the go command</h2> + +<p>Finally, a quick tour of how to use the go command, to supplement +the information in <a href="/doc/code.html">How to Write Go Code</a>, +which you might want to read first. Assuming you want +to keep your source code separate from the Go distribution source +tree, the first step is to set <code>$GOPATH</code>, the one piece of global +configuration that the go command needs. The <code>$GOPATH</code> can be a +list of directories, but by far the most common usage should be to set it to a +single directory. In particular, you do not need a separate entry in +<code>$GOPATH</code> for each of your projects. One <code>$GOPATH</code> can +support many projects.</p> + +<p>Here’s an example. Let’s say we decide to keep our Go code in the directory +<code>$HOME/mygo</code>. We need to create that directory and set +<code>$GOPATH</code> accordingly.</p> + +<pre> +$ mkdir $HOME/mygo +$ export GOPATH=$HOME/mygo +$ +</pre> + +<p>Into this directory, we now add some source code. Suppose we want to use +the indexing library from the codesearch project along with a left-leaning +red-black tree. We can install both with the "<code>go get</code>" +subcommand:</p> + +<pre> +$ go get code.google.com/p/codesearch/index +$ go get github.com/petar/GoLLRB/llrb +$ +</pre> + +<p>Both of these projects are now downloaded and installed into our +<code>$GOPATH</code> directory. The one tree now contains the two directories +<code>src/code.google.com/p/codesearch/index/</code> and +<code>src/github.com/petar/GoLLRB/llrb/</code>, along with the compiled +packages (in <code>pkg/</code>) for those libraries and their dependencies.</p> + +<p>Because we used version control systems (Mercurial and Git) to check +out the sources, the source tree also contains the other files in the +corresponding repositories, such as related packages. The "<code>go list</code>" +subcommand lists the import paths corresponding to its arguments, and +the pattern "<code>./...</code>" means start in the current directory +("<code>./</code>") and find all packages below that directory +("<code>...</code>"):</p> + +<pre> +$ go list ./... +code.google.com/p/codesearch/cmd/cgrep +code.google.com/p/codesearch/cmd/cindex +code.google.com/p/codesearch/cmd/csearch +code.google.com/p/codesearch/index +code.google.com/p/codesearch/regexp +code.google.com/p/codesearch/sparse +github.com/petar/GoLLRB/example +github.com/petar/GoLLRB/llrb +$ +</pre> + +<p>We can also test those packages:</p> + +<pre> +$ go test ./... +? code.google.com/p/codesearch/cmd/cgrep [no test files] +? code.google.com/p/codesearch/cmd/cindex [no test files] +? code.google.com/p/codesearch/cmd/csearch [no test files] +ok code.google.com/p/codesearch/index 0.239s +ok code.google.com/p/codesearch/regexp 0.021s +? code.google.com/p/codesearch/sparse [no test files] +? github.com/petar/GoLLRB/example [no test files] +ok github.com/petar/GoLLRB/llrb 0.231s +$ +</pre> + +<p>If a go subcommand is invoked with no paths listed, it operates on the +current directory:</p> + +<pre> +$ cd $GOPATH/src/code.google.com/p/codesearch/regexp +$ go list +code.google.com/p/codesearch/regexp +$ go test -v +=== RUN TestNstateEnc +--- PASS: TestNstateEnc (0.00 seconds) +=== RUN TestMatch +--- PASS: TestMatch (0.01 seconds) +=== RUN TestGrep +--- PASS: TestGrep (0.00 seconds) +PASS +ok code.google.com/p/codesearch/regexp 0.021s +$ go install +$ +</pre> + +<p>That "<code>go install</code>" subcommand installs the latest copy of the +package into the pkg directory. Because the go command can analyze the +dependency graph, "<code>go install</code>" also installs any packages that +this package imports but that are out of date, recursively.</p> + +<p>Notice that "<code>go install</code>" was able to determine the name of the +import path for the package in the current directory, because of the convention +for directory naming. It would be a little more convenient if we could pick +the name of the directory where we kept source code, and we probably wouldn't +pick such a long name, but that ability would require additional configuration +and complexity in the tool. Typing an extra directory name or two is a small +price to pay for the increased simplicity and power.</p> + +<p>As the example shows, it’s fine to work with packages from many different +projects at once within a single <code>$GOPATH</code> root directory.</p> + +<h2>Limitations</h2> + +<p>As mentioned above, the go command is not a general-purpose build +tool. In particular, it does not have any facility for generating Go +source files during a build. Instead, if you want to use a tool like +yacc or the protocol buffer compiler, you will need to write a +makefile (or a configuration file for the build tool of your choice) +to generate the Go files and then check those generated source files +into your repository. This is more work for you, the package author, +but it is significantly less work for your users, who can use +"<code>go get</code>" without needing to obtain and build +any additional tools.</p> + +<h2>More information</h2> + +<p>For more information, read <a href="/doc/code.html">How to Write Go Code</a> +and see the <a href="/cmd/go/">go command documentation</a>.</p> diff --git a/doc/articles/gobs_of_data.html b/doc/articles/gobs_of_data.html new file mode 100644 index 000000000..6b836b2c3 --- /dev/null +++ b/doc/articles/gobs_of_data.html @@ -0,0 +1,315 @@ +<!--{ +"Title": "Gobs of data", +"Template": true +}--> + +<p> +To transmit a data structure across a network or to store it in a file, it must +be encoded and then decoded again. There are many encodings available, of +course: <a href="http://www.json.org/">JSON</a>, +<a href="http://www.w3.org/XML/">XML</a>, Google's +<a href="http://code.google.com/p/protobuf">protocol buffers</a>, and more. +And now there's another, provided by Go's <a href="/pkg/encoding/gob/">gob</a> +package. +</p> + +<p> +Why define a new encoding? It's a lot of work and redundant at that. Why not +just use one of the existing formats? Well, for one thing, we do! Go has +<a href="/pkg/">packages</a> supporting all the encodings just mentioned (the +<a href="http://code.google.com/p/goprotobuf">protocol buffer package</a> is in +a separate repository but it's one of the most frequently downloaded). And for +many purposes, including communicating with tools and systems written in other +languages, they're the right choice. +</p> + +<p> +But for a Go-specific environment, such as communicating between two servers +written in Go, there's an opportunity to build something much easier to use and +possibly more efficient. +</p> + +<p> +Gobs work with the language in a way that an externally-defined, +language-independent encoding cannot. At the same time, there are lessons to be +learned from the existing systems. +</p> + +<p> +<b>Goals</b> +</p> + +<p> +The gob package was designed with a number of goals in mind. +</p> + +<p> +First, and most obvious, it had to be very easy to use. First, because Go has +reflection, there is no need for a separate interface definition language or +"protocol compiler". The data structure itself is all the package should need +to figure out how to encode and decode it. On the other hand, this approach +means that gobs will never work as well with other languages, but that's OK: +gobs are unashamedly Go-centric. +</p> + +<p> +Efficiency is also important. Textual representations, exemplified by XML and +JSON, are too slow to put at the center of an efficient communications network. +A binary encoding is necessary. +</p> + +<p> +Gob streams must be self-describing. Each gob stream, read from the beginning, +contains sufficient information that the entire stream can be parsed by an +agent that knows nothing a priori about its contents. This property means that +you will always be able to decode a gob stream stored in a file, even long +after you've forgotten what data it represents. +</p> + +<p> +There were also some things to learn from our experiences with Google protocol +buffers. +</p> + +<p> +<b>Protocol buffer misfeatures</b> +</p> + +<p> +Protocol buffers had a major effect on the design of gobs, but have three +features that were deliberately avoided. (Leaving aside the property that +protocol buffers aren't self-describing: if you don't know the data definition +used to encode a protocol buffer, you might not be able to parse it.) +</p> + +<p> +First, protocol buffers only work on the data type we call a struct in Go. You +can't encode an integer or array at the top level, only a struct with fields +inside it. That seems a pointless restriction, at least in Go. If all you want +to send is an array of integers, why should you have to put it into a +struct first? +</p> + +<p> +Next, a protocol buffer definition may specify that fields <code>T.x</code> and +<code>T.y</code> are required to be present whenever a value of type +<code>T</code> is encoded or decoded. Although such required fields may seem +like a good idea, they are costly to implement because the codec must maintain a +separate data structure while encoding and decoding, to be able to report when +required fields are missing. They're also a maintenance problem. Over time, one +may want to modify the data definition to remove a required field, but that may +cause existing clients of the data to crash. It's better not to have them in the +encoding at all. (Protocol buffers also have optional fields. But if we don't +have required fields, all fields are optional and that's that. There will be +more to say about optional fields a little later.) +</p> + +<p> +The third protocol buffer misfeature is default values. If a protocol buffer +omits the value for a "defaulted" field, then the decoded structure behaves as +if the field were set to that value. This idea works nicely when you have +getter and setter methods to control access to the field, but is harder to +handle cleanly when the container is just a plain idiomatic struct. Required +fields are also tricky to implement: where does one define the default values, +what types do they have (is text UTF-8? uninterpreted bytes? how many bits in a +float?) and despite the apparent simplicity, there were a number of +complications in their design and implementation for protocol buffers. We +decided to leave them out of gobs and fall back to Go's trivial but effective +defaulting rule: unless you set something otherwise, it has the "zero value" +for that type - and it doesn't need to be transmitted. +</p> + +<p> +So gobs end up looking like a sort of generalized, simplified protocol buffer. +How do they work? +</p> + +<p> +<b>Values</b> +</p> + +<p> +The encoded gob data isn't about <code>int8</code>s and <code>uint16</code>s. +Instead, somewhat analogous to constants in Go, its integer values are abstract, +sizeless numbers, either signed or unsigned. When you encode an +<code>int8</code>, its value is transmitted as an unsized, variable-length +integer. When you encode an <code>int64</code>, its value is also transmitted as +an unsized, variable-length integer. (Signed and unsigned are treated +distinctly, but the same unsized-ness applies to unsigned values too.) If both +have the value 7, the bits sent on the wire will be identical. When the receiver +decodes that value, it puts it into the receiver's variable, which may be of +arbitrary integer type. Thus an encoder may send a 7 that came from an +<code>int8</code>, but the receiver may store it in an <code>int64</code>. This +is fine: the value is an integer and as a long as it fits, everything works. (If +it doesn't fit, an error results.) This decoupling from the size of the variable +gives some flexibility to the encoding: we can expand the type of the integer +variable as the software evolves, but still be able to decode old data. +</p> + +<p> +This flexibility also applies to pointers. Before transmission, all pointers are +flattened. Values of type <code>int8</code>, <code>*int8</code>, +<code>**int8</code>, <code>****int8</code>, etc. are all transmitted as an +integer value, which may then be stored in <code>int</code> of any size, or +<code>*int</code>, or <code>******int</code>, etc. Again, this allows for +flexibility. +</p> + +<p> +Flexibility also happens because, when decoding a struct, only those fields +that are sent by the encoder are stored in the destination. Given the value +</p> + +{{code "/doc/progs/gobs1.go" `/type T/` `/STOP/`}} + +<p> +the encoding of <code>t</code> sends only the 7 and 8. Because it's zero, the +value of <code>Y</code> isn't even sent; there's no need to send a zero value. +</p> + +<p> +The receiver could instead decode the value into this structure: +</p> + +{{code "/doc/progs/gobs1.go" `/type U/` `/STOP/`}} + +<p> +and acquire a value of <code>u</code> with only <code>X</code> set (to the +address of an <code>int8</code> variable set to 7); the <code>Z</code> field is +ignored - where would you put it? When decoding structs, fields are matched by +name and compatible type, and only fields that exist in both are affected. This +simple approach finesses the "optional field" problem: as the type +<code>T</code> evolves by adding fields, out of date receivers will still +function with the part of the type they recognize. Thus gobs provide the +important result of optional fields - extensibility - without any additional +mechanism or notation. +</p> + +<p> +From integers we can build all the other types: bytes, strings, arrays, slices, +maps, even floats. Floating-point values are represented by their IEEE 754 +floating-point bit pattern, stored as an integer, which works fine as long as +you know their type, which we always do. By the way, that integer is sent in +byte-reversed order because common values of floating-point numbers, such as +small integers, have a lot of zeros at the low end that we can avoid +transmitting. +</p> + +<p> +One nice feature of gobs that Go makes possible is that they allow you to define +your own encoding by having your type satisfy the +<a href="/pkg/encoding/gob/#GobEncoder">GobEncoder</a> and +<a href="/pkg/encoding/gob/#GobDecoder">GobDecoder</a> interfaces, in a manner +analogous to the <a href="/pkg/encoding/json/">JSON</a> package's +<a href="/pkg/encoding/json/#Marshaler">Marshaler</a> and +<a href="/pkg/encoding/json/#Unmarshaler">Unmarshaler</a> and also to the +<a href="/pkg/fmt/#Stringer">Stringer</a> interface from +<a href="/pkg/fmt/">package fmt</a>. This facility makes it possible to +represent special features, enforce constraints, or hide secrets when you +transmit data. See the <a href="/pkg/encoding/gob/">documentation</a> for +details. +</p> + +<p> +<b>Types on the wire</b> +</p> + +<p> +The first time you send a given type, the gob package includes in the data +stream a description of that type. In fact, what happens is that the encoder is +used to encode, in the standard gob encoding format, an internal struct that +describes the type and gives it a unique number. (Basic types, plus the layout +of the type description structure, are predefined by the software for +bootstrapping.) After the type is described, it can be referenced by its type +number. +</p> + +<p> +Thus when we send our first type <code>T</code>, the gob encoder sends a +description of <code>T</code> and tags it with a type number, say 127. All +values, including the first, are then prefixed by that number, so a stream of +<code>T</code> values looks like: +</p> + +<pre> +("define type id" 127, definition of type T)(127, T value)(127, T value), ... +</pre> + +<p> +These type numbers make it possible to describe recursive types and send values +of those types. Thus gobs can encode types such as trees: +</p> + +{{code "/doc/progs/gobs1.go" `/type Node/` `/STOP/`}} + +<p> +(It's an exercise for the reader to discover how the zero-defaulting rule makes +this work, even though gobs don't represent pointers.) +</p> + +<p> +With the type information, a gob stream is fully self-describing except for the +set of bootstrap types, which is a well-defined starting point. +</p> + +<p> +<b>Compiling a machine</b> +</p> + +<p> +The first time you encode a value of a given type, the gob package builds a +little interpreted machine specific to that data type. It uses reflection on +the type to construct that machine, but once the machine is built it does not +depend on reflection. The machine uses package unsafe and some trickery to +convert the data into the encoded bytes at high speed. It could use reflection +and avoid unsafe, but would be significantly slower. (A similar high-speed +approach is taken by the protocol buffer support for Go, whose design was +influenced by the implementation of gobs.) Subsequent values of the same type +use the already-compiled machine, so they can be encoded right away. +</p> + +<p> +Decoding is similar but harder. When you decode a value, the gob package holds +a byte slice representing a value of a given encoder-defined type to decode, +plus a Go value into which to decode it. The gob package builds a machine for +that pair: the gob type sent on the wire crossed with the Go type provided for +decoding. Once that decoding machine is built, though, it's again a +reflectionless engine that uses unsafe methods to get maximum speed. +</p> + +<p> +<b>Use</b> +</p> + +<p> +There's a lot going on under the hood, but the result is an efficient, +easy-to-use encoding system for transmitting data. Here's a complete example +showing differing encoded and decoded types. Note how easy it is to send and +receive values; all you need to do is present values and variables to the +<a href="/pkg/encoding/gob/">gob package</a> and it does all the work. +</p> + +{{code "/doc/progs/gobs2.go" `/package main/` `$`}} + +<p> +You can compile and run this example code in the +<a href="http://play.golang.org/p/_-OJV-rwMq">Go Playground</a>. +</p> + +<p> +The <a href="/pkg/net/rpc/">rpc package</a> builds on gobs to turn this +encode/decode automation into transport for method calls across the network. +That's a subject for another article. +</p> + +<p> +<b>Details</b> +</p> + +<p> +The <a href="/pkg/encoding/gob/">gob package documentation</a>, especially the +file <a href="/src/pkg/encoding/gob/doc.go">doc.go</a>, expands on many of the +details described here and includes a full worked example showing how the +encoding represents data. If you are interested in the innards of the gob +implementation, that's a good place to start. +</p> diff --git a/doc/articles/godoc_documenting_go_code.html b/doc/articles/godoc_documenting_go_code.html new file mode 100644 index 000000000..ca66076ad --- /dev/null +++ b/doc/articles/godoc_documenting_go_code.html @@ -0,0 +1,139 @@ +<!--{ +"Title": "Godoc: documenting Go code", +"Template": true +}--> + +<p> +The Go project takes documentation seriously. Documentation is a huge part of +making software accessible and maintainable. Of course it must be well-written +and accurate, but it also must be easy to write and to maintain. Ideally, it +should be coupled to the code itself so the documentation evolves along with the +code. The easier it is for programmers to produce good documentation, the better +for everyone. +</p> + +<p> +To that end, we have developed the <a href="/cmd/godoc/">godoc</a> documentation +tool. This article describes godoc's approach to documentation, and explains how +you can use our conventions and tools to write good documentation for your own +projects. +</p> + +<p> +Godoc parses Go source code - including comments - and produces documentation as +HTML or plain text. The end result is documentation tightly coupled with the +code it documents. For example, through godoc's web interface you can navigate +from a function's <a href="/pkg/strings/#HasPrefix">documentation</a> to its +<a href="/src/pkg/strings/strings.go?#L312">implementation</a> with one click. +</p> + +<p> +Godoc is conceptually related to Python's +<a href="http://www.python.org/dev/peps/pep-0257/">Docstring</a> and Java's +<a href="http://www.oracle.com/technetwork/java/javase/documentation/index-jsp-135444.html">Javadoc</a>, +but its design is simpler. The comments read by godoc are not language +constructs (as with Docstring) nor must they have their own machine-readable +syntax (as with Javadoc). Godoc comments are just good comments, the sort you +would want to read even if godoc didn't exist. +</p> + +<p> +The convention is simple: to document a type, variable, constant, function, or +even a package, write a regular comment directly preceding its declaration, with +no intervening blank line. Godoc will then present that comment as text +alongside the item it documents. For example, this is the documentation for the +<code>fmt</code> package's <a href="/pkg/fmt/#Fprint"><code>Fprint</code></a> +function: +</p> + +{{code "/src/pkg/fmt/print.go" `/Fprint formats using the default/` `/func Fprint/`}} + +<p> +Notice this comment is a complete sentence that begins with the name of the +element it describes. This important convention allows us to generate +documentation in a variety of formats, from plain text to HTML to UNIX man +pages, and makes it read better when tools truncate it for brevity, such as when +they extract the first line or sentence. +</p> + +<p> +Comments on package declarations should provide general package documentation. +These comments can be short, like the <a href="/pkg/sort/"><code>sort</code></a> +package's brief description: +</p> + +{{code "/src/pkg/sort/sort.go" `/Package sort provides/` `/package sort/`}} + +<p> +They can also be detailed like the <a href="/pkg/encoding/gob/">gob package</a>'s +overview. That package uses another convention for packages +that need large amounts of introductory documentation: the package comment is +placed in its own file, <a href="/src/pkg/encoding/gob/doc.go">doc.go</a>, which +contains only those comments and a package clause. +</p> + +<p> +When writing package comments of any size, keep in mind that their first +sentence will appear in godoc's <a href="/pkg/">package list</a>. +</p> + +<p> +Comments that are not adjacent to a top-level declaration are omitted from +godoc's output, with one notable exception. Top-level comments that begin with +the word <code>"BUG(who)”</code> are recognized as known bugs, and included in +the "Bugs” section of the package documentation. The "who” part should be the +user name of someone who could provide more information. For example, this is a +known issue from the <a href="/pkg/bytes/#bugs">bytes package</a>: +</p> + +<pre> +// BUG(r): The rule Title uses for word boundaries does not handle Unicode punctuation properly. +</pre> + +<p> +Godoc treats executable commands somewhat differently. Instead of inspecting the +command source code, it looks for a Go source file belonging to the special +package "documentation”. The comment on the "package documentation” clause is +used as the command's documentation. For example, see the +<a href="/cmd/godoc/">godoc documentation</a> and its corresponding +<a href="/src/cmd/godoc/doc.go">doc.go</a> file. +</p> + +<p> +There are a few formatting rules that Godoc uses when converting comments to +HTML: +</p> + +<ul> +<li> +Subsequent lines of text are considered part of the same paragraph; you must +leave a blank line to separate paragraphs. +</li> +<li> +Pre-formatted text must be indented relative to the surrounding comment text +(see gob's <a href="/src/pkg/encoding/gob/doc.go">doc.go</a> for an example). +</li> +<li> +URLs will be converted to HTML links; no special markup is necessary. +</li> +</ul> + +<p> +Note that none of these rules requires you to do anything out of the ordinary. +</p> + +<p> +In fact, the best thing about godoc's minimal approach is how easy it is to use. +As a result, a lot of Go code, including all of the standard library, already +follows the conventions. +</p> + +<p> +Your own code can present good documentation just by having comments as +described above. Any Go packages installed inside <code>$GOROOT/src/pkg</code> +and any <code>GOPATH</code> work spaces will already be accessible via godoc's +command-line and HTTP interfaces, and you can specify additional paths for +indexing via the <code>-path</code> flag or just by running <code>"godoc ."</code> +in the source directory. See the <a href="/cmd/godoc/">godoc documentation</a> +for more details. +</p> diff --git a/doc/articles/gos_declaration_syntax.html b/doc/articles/gos_declaration_syntax.html new file mode 100644 index 000000000..455cced1d --- /dev/null +++ b/doc/articles/gos_declaration_syntax.html @@ -0,0 +1,348 @@ +<!--{ +"Title": "Go's Declaration Syntax" +}--> + +<p> +Newcomers to Go wonder why the declaration syntax is different from the +tradition established in the C family. In this post we'll compare the +two approaches and explain why Go's declarations look as they do. +</p> + +<p> +<b>C syntax</b> +</p> + +<p> +First, let's talk about C syntax. C took an unusual and clever approach +to declaration syntax. Instead of describing the types with special +syntax, one writes an expression involving the item being declared, and +states what type that expression will have. Thus +</p> + +<pre> +int x; +</pre> + +<p> +declares x to be an int: the expression 'x' will have type int. In +general, to figure out how to write the type of a new variable, write an +expression involving that variable that evaluates to a basic type, then +put the basic type on the left and the expression on the right. +</p> + +<p> +Thus, the declarations +</p> + +<pre> +int *p; +int a[3]; +</pre> + +<p> +state that p is a pointer to int because '*p' has type int, and that a +is an array of ints because a[3] (ignoring the particular index value, +which is punned to be the size of the array) has type int. +</p> + +<p> +What about functions? Originally, C's function declarations wrote the +types of the arguments outside the parens, like this: +</p> + +<pre> +int main(argc, argv) + int argc; + char *argv[]; +{ /* ... */ } +</pre> + +<p> +Again, we see that main is a function because the expression main(argc, +argv) returns an int. In modern notation we'd write +</p> + +<pre> +int main(int argc, char *argv[]) { /* ... */ } +</pre> + +<p> +but the basic structure is the same. +</p> + +<p> +This is a clever syntactic idea that works well for simple types but can +get confusing fast. The famous example is declaring a function pointer. +Follow the rules and you get this: +</p> + +<pre> +int (*fp)(int a, int b); +</pre> + +<p> +Here, fp is a pointer to a function because if you write the expression +(*fp)(a, b) you'll call a function that returns int. What if one of fp's +arguments is itself a function? +</p> + +<pre> +int (*fp)(int (*ff)(int x, int y), int b) +</pre> + +<p> +That's starting to get hard to read. +</p> + +<p> +Of course, we can leave out the name of the parameters when we declare a +function, so main can be declared +</p> + +<pre> +int main(int, char *[]) +</pre> + +<p> +Recall that argv is declared like this, +</p> + +<pre> +char *argv[] +</pre> + +<p> +so you drop the name from the <em>middle</em> of its declaration to construct +its type. It's not obvious, though, that you declare something of type +char *[] by putting its name in the middle. +</p> + +<p> +And look what happens to fp's declaration if you don't name the +parameters: +</p> + +<pre> +int (*fp)(int (*)(int, int), int) +</pre> + +<p> +Not only is it not obvious where to put the name inside +</p> + +<pre> +int (*)(int, int) +</pre> + +<p> +it's not exactly clear that it's a function pointer declaration at all. +And what if the return type is a function pointer? +</p> + +<pre> +int (*(*fp)(int (*)(int, int), int))(int, int) +</pre> + +<p> +It's hard even to see that this declaration is about fp. +</p> + +<p> +You can construct more elaborate examples but these should illustrate +some of the difficulties that C's declaration syntax can introduce. +</p> + +<p> +There's one more point that needs to be made, though. Because type and +declaration syntax are the same, it can be difficult to parse +expressions with types in the middle. This is why, for instance, C casts +always parenthesize the type, as in +</p> + +<pre> +(int)M_PI +</pre> + +<p> +<b>Go syntax</b> +</p> + +<p> +Languages outside the C family usually use a distinct type syntax in +declarations. Although it's a separate point, the name usually comes +first, often followed by a colon. Thus our examples above become +something like (in a fictional but illustrative language) +</p> + +<pre> +x: int +p: pointer to int +a: array[3] of int +</pre> + +<p> +These declarations are clear, if verbose - you just read them left to +right. Go takes its cue from here, but in the interests of brevity it +drops the colon and removes some of the keywords: +</p> + +<pre> +x int +p *int +a [3]int +</pre> + +<p> +There is no direct correspondence between the look of [3]int and how to +use a in an expression. (We'll come back to pointers in the next +section.) You gain clarity at the cost of a separate syntax. +</p> + +<p> +Now consider functions. Let's transcribe the declaration for main, even +though the main function in Go takes no arguments: +</p> + +<pre> +func main(argc int, argv *[]byte) int +</pre> + +<p> +Superficially that's not much different from C, but it reads well from +left to right: +</p> + +<p> +<em>function main takes an int and a pointer to a slice of bytes and returns an int.</em> +</p> + +<p> +Drop the parameter names and it's just as clear - they're always first +so there's no confusion. +</p> + +<pre> +func main(int, *[]byte) int +</pre> + +<p> +One value of this left-to-right style is how well it works as the types +become more complex. Here's a declaration of a function variable +(analogous to a function pointer in C): +</p> + +<pre> +f func(func(int,int) int, int) int +</pre> + +<p> +Or if f returns a function: +</p> + +<pre> +f func(func(int,int) int, int) func(int, int) int +</pre> + +<p> +It still reads clearly, from left to right, and it's always obvious +which name is being declared - the name comes first. +</p> + +<p> +The distinction between type and expression syntax makes it easy to +write and invoke closures in Go: +</p> + +<pre> +sum := func(a, b int) int { return a+b } (3, 4) +</pre> + +<p> +<b>Pointers</b> +</p> + +<p> +Pointers are the exception that proves the rule. Notice that in arrays +and slices, for instance, Go's type syntax puts the brackets on the left +of the type but the expression syntax puts them on the right of the +expression: +</p> + +<pre> +var a []int +x = a[1] +</pre> + +<p> +For familiarity, Go's pointers use the * notation from C, but we could +not bring ourselves to make a similar reversal for pointer types. Thus +pointers work like this +</p> + +<pre> +var p *int +x = *p +</pre> + +<p> +We couldn't say +</p> + +<pre> +var p *int +x = p* +</pre> + +<p> +because that postfix * would conflate with multiplication. We could have +used the Pascal ^, for example: +</p> + +<pre> +var p ^int +x = p^ +</pre> + +<p> +and perhaps we should have (and chosen another operator for xor), +because the prefix asterisk on both types and expressions complicates +things in a number of ways. For instance, although one can write +</p> + +<pre> +[]int("hi") +</pre> + +<p> +as a conversion, one must parenthesize the type if it starts with a *: +</p> + +<pre> +(*int)(nil) +</pre> + +<p> +Had we been willing to give up * as pointer syntax, those parentheses +would be unnecessary. +</p> + +<p> +So Go's pointer syntax is tied to the familiar C form, but those ties +mean that we cannot break completely from using parentheses to +disambiguate types and expressions in the grammar. +</p> + +<p> +Overall, though, we believe Go's type syntax is easier to understand +than C's, especially when things get complicated. +</p> + +<p> +<b>Notes</b> +</p> + +<p> +Go's declarations read left to right. It's been pointed out that C's +read in a spiral! See <a href="http://c-faq.com/decl/spiral.anderson.html"> +The "Clockwise/Spiral Rule"</a> by David Anderson. +</p> diff --git a/doc/articles/image-20.png b/doc/articles/image-20.png Binary files differnew file mode 100644 index 000000000..063e43064 --- /dev/null +++ b/doc/articles/image-20.png diff --git a/doc/articles/image-2a.png b/doc/articles/image-2a.png Binary files differnew file mode 100644 index 000000000..3f1c0afff --- /dev/null +++ b/doc/articles/image-2a.png diff --git a/doc/articles/image-2b.png b/doc/articles/image-2b.png Binary files differnew file mode 100644 index 000000000..32b247011 --- /dev/null +++ b/doc/articles/image-2b.png diff --git a/doc/articles/image-2c.png b/doc/articles/image-2c.png Binary files differnew file mode 100644 index 000000000..f9abce5b5 --- /dev/null +++ b/doc/articles/image-2c.png diff --git a/doc/articles/image-2d.png b/doc/articles/image-2d.png Binary files differnew file mode 100644 index 000000000..ed0a9f92c --- /dev/null +++ b/doc/articles/image-2d.png diff --git a/doc/articles/image-2e.png b/doc/articles/image-2e.png Binary files differnew file mode 100644 index 000000000..483b208e3 --- /dev/null +++ b/doc/articles/image-2e.png diff --git a/doc/articles/image-2f.png b/doc/articles/image-2f.png Binary files differnew file mode 100644 index 000000000..3dce02d5f --- /dev/null +++ b/doc/articles/image-2f.png diff --git a/doc/articles/image-package-01.png b/doc/articles/image-package-01.png Binary files differnew file mode 100644 index 000000000..aad9b1243 --- /dev/null +++ b/doc/articles/image-package-01.png diff --git a/doc/articles/image-package-02.png b/doc/articles/image-package-02.png Binary files differnew file mode 100644 index 000000000..3dd4692f3 --- /dev/null +++ b/doc/articles/image-package-02.png diff --git a/doc/articles/image-package-03.png b/doc/articles/image-package-03.png Binary files differnew file mode 100644 index 000000000..5bc0bf732 --- /dev/null +++ b/doc/articles/image-package-03.png diff --git a/doc/articles/image-package-04.png b/doc/articles/image-package-04.png Binary files differnew file mode 100644 index 000000000..393dc1207 --- /dev/null +++ b/doc/articles/image-package-04.png diff --git a/doc/articles/image-package-05.png b/doc/articles/image-package-05.png Binary files differnew file mode 100644 index 000000000..54c47b67b --- /dev/null +++ b/doc/articles/image-package-05.png diff --git a/doc/articles/image_draw.html b/doc/articles/image_draw.html new file mode 100644 index 000000000..ba735ad2b --- /dev/null +++ b/doc/articles/image_draw.html @@ -0,0 +1,222 @@ +<!--{ + "Title": "The Go image/draw package", + "Template": true +}--> + +<p> +<a href="/pkg/image/draw/">Package image/draw</a> defines +only one operation: drawing a source image onto a destination +image, through an optional mask image. This one operation is +surprisingly versatile and can perform a number of common image +manipulation tasks elegantly and efficiently. +</p> + +<p> +Composition is performed pixel by pixel in the style of the Plan 9 +graphics library and the X Render extension. The model is based on +the classic "Compositing Digital Images" paper by Porter and Duff, +with an additional mask parameter: <code>dst = (src IN mask) OP dst</code>. +For a fully opaque mask, this reduces to the original Porter-Duff +formula: <code>dst = src OP dst</code>. In Go, a nil mask image is equivalent +to an infinitely sized, fully opaque mask image. +</p> + +<p> +The Porter-Duff paper presented +<a href="http://www.w3.org/TR/SVGCompositing/examples/compop-porterduff-examples.png">12 different composition operators</a>, +but with an explicit mask, only 2 of these are needed in practice: +source-over-destination and source. In Go, these operators are +represented by the <code>Over</code> and <code>Src</code> constants. The <code>Over</code> operator +performs the natural layering of a source image over a destination +image: the change to the destination image is smaller where the +source (after masking) is more transparent (that is, has lower +alpha). The <code>Src</code> operator merely copies the source (after masking) +with no regard for the destination image's original content. For +fully opaque source and mask images, the two operators produce the +same output, but the <code>Src</code> operator is usually faster. +</p> + +<p><b>Geometric Alignment</b></p> + +<p> +Composition requires associating destination pixels with source and +mask pixels. Obviously, this requires destination, source and mask +images, and a composition operator, but it also requires specifying +what rectangle of each image to use. Not every drawing should write +to the entire destination: when updating an animating image, it is +more efficient to only draw the parts of the image that have +changed. Not every drawing should read from the entire source: when +using a sprite that combines many small images into one large one, +only a part of the image is needed. Not every drawing should read +from the entire mask: a mask image that collects a font's glyphs is +similar to a sprite. Thus, drawing also needs to know three +rectangles, one for each image. Since each rectangle has the same +width and height, it suffices to pass a destination rectangle `r` +and two points <code>sp</code> and <code>mp</code>: the source rectangle is equal to <code>r</code> +translated so that <code>r.Min</code> in the destination image aligns with +<code>sp</code> in the source image, and similarly for <code>mp</code>. The effective +rectangle is also clipped to each image's bounds in their +respective co-ordinate space. +</p> + +<p> +<img src="image-20.png"> +</p> + +<p> +The <a href="/pkg/image/draw/#DrawMask"><code>DrawMask</code></a> +function takes seven arguments, but an explicit mask and mask-point +are usually unnecessary, so the +<a href="/pkg/image/draw/#Draw"><code>Draw</code></a> function takes five: +</p> + +<pre> +// Draw calls DrawMask with a nil mask. +func Draw(dst Image, r image.Rectangle, src image.Image, sp image.Point, op Op) +func DrawMask(dst Image, r image.Rectangle, src image.Image, sp image.Point, + mask image.Image, mp image.Point, op Op) +</pre> + +<p> +The destination image must be mutable, so the image/draw package +defines a <a href="/pkg/image/draw/#Image"><code>draw.Image</code></a> +interface which has a <code>Set</code> method. +</p> + +{{code "../src/pkg/image/draw/draw.go" `/type Image/` `/}/`}} + +<p><b>Filling a Rectangle</b></p> + +<p> +To fill a rectangle with a solid color, use an <code>image.Uniform</code> +source. The <code>ColorImage</code> type re-interprets a <code>Color</code> as a +practically infinite-sized <code>Image</code> of that color. For those +familiar with the design of Plan 9's draw library, there is no need +for an explicit "repeat bit" in Go's slice-based image types; the +concept is subsumed by <code>Uniform</code>. +</p> + +{{code "/doc/progs/image_draw.go" `/ZERO/` `/STOP/`}} + +<p> +To initialize a new image to all-blue: +</p> + +{{code "/doc/progs/image_draw.go" `/BLUE/` `/STOP/`}} + +<p> +To reset an image to transparent (or black, if the destination +image's color model cannot represent transparency), use +<code>image.Transparent</code>, which is an <code>image.Uniform</code>: +</p> + +{{code "/doc/progs/image_draw.go" `/RESET/` `/STOP/`}} + +<p> +<img src="image-2a.png"> +</p> + + +<p><b>Copying an Image</b></p> + +<p> +To copy from a rectangle <code>sr</code> in the source image to a rectangle +starting at a point <code>dp</code> in the destination, convert the source +rectangle into the destination image's co-ordinate space: +</p> + +{{code "/doc/progs/image_draw.go" `/RECT/` `/STOP/`}} + +<p> +Alternatively: +</p> + +{{code "/doc/progs/image_draw.go" `/RECT2/` `/STOP/`}} + +<p> +To copy the entire source image, use <code>sr = src.Bounds()</code>. +</p> + +<p> +<img src="image-2b.png"> +</p> + +<p><b>Scrolling an Image</b></p> + +<p> +Scrolling an image is just copying an image to itself, with +different destination and source rectangles. Overlapping +destination and source images are perfectly valid, just as Go's +built-in copy function can handle overlapping destination and +source slices. To scroll an image m by 20 pixels: +</p> + +{{code "/doc/progs/image_draw.go" `/SCROLL/` `/STOP/`}} + +<p><img src="image-2c.png"></p> + +<p><b>Converting an Image to RGBA</b></p> + +<p> +The result of decoding an image format might not be an +<code>image.RGBA</code>: decoding a GIF results in an <code>image.Paletted</code>, +decoding a JPEG results in a <code>ycbcr.YCbCr</code>, and the result of +decoding a PNG depends on the image data. To convert any image to +an <code>image.RGBA</code>: +</p> + +{{code "/doc/progs/image_draw.go" `/CONV/` `/STOP/`}} + +<p> +<img src="image-2d.png"> +</p> + +<p><b>Drawing Through a Mask</b></p> + +<p> +To draw an image through a circular mask with center <code>p</code> and radius +<code>r</code>: +</p> + +{{code "/doc/progs/image_draw.go" `/CIRCLESTRUCT/` `/STOP/`}} +{{code "/doc/progs/image_draw.go" `/CIRCLE2/` `/STOP/`}} + +<p> +<img src="image-2e.png"> +</p> + +<p><b>Drawing Font Glyphs</b></p> + +<p> +To draw a font glyph in blue starting from a point <code>p</code>, draw with +an <code>image.ColorImage</code> source and an <code>image.Alpha mask</code>. For +simplicity, we aren't performing any sub-pixel positioning or +rendering, or correcting for a font's height above a baseline. +</p> + +{{code "/doc/progs/image_draw.go" `/GLYPH/` `/STOP/`}} + +<p> +<img src="image-2f.png"> +</p> + +<p><b>Performance</b></p> + +<p> +The image/draw package implementation demonstrates how to provide +an image manipulation function that is both general purpose, yet +efficient for common cases. The <code>DrawMask</code> function takes arguments +of interface types, but immediately makes type assertions that its +arguments are of specific struct types, corresponding to common +operations like drawing one <code>image.RGBA</code> image onto another, or +drawing an <code>image.Alpha</code> mask (such as a font glyph) onto an +<code>image.RGBA</code> image. If a type assertion succeeds, that type +information is used to run a specialized implementation of the +general algorithm. If the assertions fail, the fallback code path +uses the generic <code>At</code> and <code>Set</code> methods. The fast-paths are purely +a performance optimization; the resultant destination image is the +same either way. In practice, only a small number of special cases +are necessary to support typical applications. +</p> + + diff --git a/doc/articles/image_package.html b/doc/articles/image_package.html new file mode 100644 index 000000000..a9d2f3581 --- /dev/null +++ b/doc/articles/image_package.html @@ -0,0 +1,312 @@ +<!--{ + "Title": "The Go image package", + "Template": true +}--> + +<p> +The <a href="/pkg/image/">image</a> and +<a href="/pkg/image/color/">image/color</a> packages define a number of types: +<code>color.Color</code> and <code>color.Model</code> describe colors, +<code>image.Point</code> and <code>image.Rectangle</code> describe basic 2-D +geometry, and <code>image.Image</code> brings the two concepts together to +represent a rectangular grid of colors. A +<a href="/doc/articles/image_draw.html">separate article</a> covers image +composition with the <a href="/pkg/image/draw/">image/draw</a> package. +</p> + +<p> +<b>Colors and Color Models</b> +</p> + +<p> +<a href="/pkg/image/color/#Color">Color</a> is an interface that defines the minimal +method set of any type that can be considered a color: one that can be converted +to red, green, blue and alpha values. The conversion may be lossy, such as +converting from CMYK or YCbCr color spaces. +</p> + +{{code "/src/pkg/image/color/color.go" `/type Color interface/` `/^}/`}} + +<p> +There are three important subtleties about the return values. First, the red, +green and blue are alpha-premultiplied: a fully saturated red that is also 25% +transparent is represented by RGBA returning a 75% r. Second, the channels have +a 16-bit effective range: 100% red is represented by RGBA returning an r of +65535, not 255, so that converting from CMYK or YCbCr is not as lossy. Third, +the type returned is <code>uint32</code>, even though the maximum value is 65535, to +guarantee that multiplying two values together won't overflow. Such +multiplications occur when blending two colors according to an alpha mask from a +third color, in the style of +<a href="https://en.wikipedia.org/wiki/Alpha_compositing">Porter and Duff's</a> +classic algebra: +</p> + +<pre> +dstr, dstg, dstb, dsta := dst.RGBA() +srcr, srcg, srcb, srca := src.RGBA() +_, _, _, m := mask.RGBA() +const M = 1<<16 - 1 +// The resultant red value is a blend of dstr and srcr, and ranges in [0, M]. +// The calculation for green, blue and alpha is similar. +dstr = (dstr*(M-m) + srcr*m) / M +</pre> + +<p> +The last line of that code snippet would have been more complicated if we worked +with non-alpha-premultiplied colors, which is why <code>Color</code> uses +alpha-premultiplied values. +</p> + +<p> +The image/color package also defines a number of concrete types that implement +the <code>Color</code> interface. For example, +<a href="/pkg/image/color/#RGBA"><code>RGBA</code></a> is a struct that represents +the classic "8 bits per channel" color. +</p> + +{{code "/src/pkg/image/color/color.go" `/type RGBA struct/` `/^}/`}} + +<p> +Note that the <code>R</code> field of an <code>RGBA</code> is an 8-bit +alpha-premultiplied color in the range [0, 255]. <code>RGBA</code> satisfies the +<code>Color</code> interface by multiplying that value by 0x101 to generate a +16-bit alpha-premultiplied color in the range [0, 65535]. Similarly, the +<a href="/pkg/image/color/#NRGBA"><code>NRGBA</code></a> struct type represents +an 8-bit non-alpha-premultiplied color, as used by the PNG image format. When +manipulating an <code>NRGBA</code>'s fields directly, the values are +non-alpha-premultiplied, but when calling the <code>RGBA</code> method, the +return values are alpha-premultiplied. +</p> + +<p> +A <a href="/pkg/image/color/#Model"><code>Model</code></a> is simply +something that can convert <code>Color</code>s to other <code>Color</code>s, possibly lossily. For +example, the <code>GrayModel</code> can convert any <code>Color</code> to a +desaturated <a href="/pkg/image/color/#Gray"><code>Gray</code></a>. A +<code>Palette</code> can convert any <code>Color</code> to one from a +limited palette. +</p> + +{{code "/src/pkg/image/color/color.go" `/type Model interface/` `/^}/`}} + +{{code "/src/pkg/image/color/color.go" `/type Palette \[\]Color/`}} + +<p> +<b>Points and Rectangles</b> +</p> + +<p> +A <a href="/pkg/image/#Point"><code>Point</code></a> is an (x, y) co-ordinate +on the integer grid, with axes increasing right and down. It is neither a pixel +nor a grid square. A <code>Point</code> has no intrinsic width, height or +color, but the visualizations below use a small colored square. +</p> + +{{code "/src/pkg/image/geom.go" `/type Point struct/` `/^}/`}} + +<p> +<img src="image-package-01.png" width="400" height="300"> +</p> + +{{code "/doc/progs/image_package1.go" `/p := image.Point/`}} + +<p> +A <a href="/pkg/image/#Rectangle"><code>Rectangle</code></a> is an axis-aligned +rectangle on the integer grid, defined by its top-left and bottom-right +<code>Point</code>. A <code>Rectangle</code> also has no intrinsic color, but +the visualizations below outline rectangles with a thin colored line, and call +out their <code>Min</code> and <code>Max</code> <code>Point</code>s. +</p> + +{{code "/src/pkg/image/geom.go" `/type Rectangle struct/` `/^}/`}} + +<p> +For convenience, <code>image.Rect(x0, y0, x1, y1)</code> is equivalent to +<code>image.Rectangle{image.Point{x0, y0}, image.Point{x1, y1}}</code>, but is +much easier to type. +</p> + +<p> +A <code>Rectangle</code> is inclusive at the top-left and exclusive at the +bottom-right. For a <code>Point p</code> and a <code>Rectangle r</code>, +<code>p.In(r)</code> if and only if +<code>r.Min.X <= p.X && p.X < r.Max.X</code>, and similarly for <code>Y</code>. This is analagous to how +a slice <code>s[i0:i1]</code> is inclusive at the low end and exclusive at the +high end. (Unlike arrays and slices, a <code>Rectangle</code> often has a +non-zero origin.) +</p> + +<p> +<img src="image-package-02.png" width="400" height="300"> +</p> + +{{code "/doc/progs/image_package2.go" `/r := image.Rect/` `/fmt.Println/`}} + +<p> +Adding a <code>Point</code> to a <code>Rectangle</code> translates the +<code>Rectangle</code>. Points and Rectangles are not restricted to be in the +bottom-right quadrant. +</p> + +<p> +<img src="image-package-03.png" width="400" height="300"> +</p> + +{{code "/doc/progs/image_package3.go" `/r := image.Rect/` `/fmt.Println/`}} + +<p> +Intersecting two Rectangles yields another Rectangle, which may be empty. +</p> + +<p> +<img src="image-package-04.png" width="400" height="300"> +</p> + +{{code "/doc/progs/image_package4.go" `/r := image.Rect/` `/fmt.Printf/`}} + +<p> +Points and Rectangles are passed and returned by value. A function that takes a +<code>Rectangle</code> argument will be as efficient as a function that takes +two <code>Point</code> arguments, or four <code>int</code> arguments. +</p> + +<p> +<b>Images</b> +</p> + +<p> +An <a href="/pkg/image/#Image">Image</a> maps every grid square in a +<code>Rectangle</code> to a <code>Color</code> from a <code>Model</code>. +"The pixel at (x, y)" refers to the color of the grid square defined by the +points (x, y), (x+1, y), (x+1, y+1) and (x, y+1). +</p> + +{{code "/src/pkg/image/image.go" `/type Image interface/` `/^}/`}} + +<p> +A common mistake is assuming that an <code>Image</code>'s bounds start at (0, +0). For example, an animated GIF contains a sequence of Images, and each +<code>Image</code> after the first typically only holds pixel data for the area +that changed, and that area doesn't necessarily start at (0, 0). The correct +way to iterate over an <code>Image</code> m's pixels looks like: +</p> + +<pre> +b := m.Bounds() +for y := b.Min.Y; y < b.Max.Y; y++ { + for x := b.Min.X; y < b.Max.X; x++ { + doStuffWith(m.At(x, y)) + } +} +</pre> + +<p> +<code>Image</code> implementations do not have to be based on an in-memory +slice of pixel data. For example, a +<a href="/pkg/image/#Uniform"><code>Uniform</code></a> is an +<code>Image</code> of enormous bounds and uniform color, whose in-memory +representation is simply that color. +</p> + +{{code "/src/pkg/image/names.go" `/type Uniform struct/` `/^}/`}} + +<p> +Typically, though, programs will want an image based on a slice. Struct types +like <a href="/pkg/image/#RGBA"><code>RGBA</code></a> and +<a href="/pkg/image/#Gray"><code>Gray</code></a> (which other packages refer +to as <code>image.RGBA</code> and <code>image.Gray</code>) hold slices of pixel +data and implement the <code>Image</code> interface. +</p> + +{{code "/src/pkg/image/image.go" `/type RGBA struct/` `/^}/`}} + +<p> +These types also provide a <code>Set(x, y int, c color.Color)</code> method +that allows modifying the image one pixel at a time. +</p> + +{{code "/doc/progs/image_package5.go" `/m := image.New/` `/m.Set/`}} + +<p> +If you're reading or writing a lot of pixel data, it can be more efficient, but +more complicated, to access these struct type's <code>Pix</code> field directly. +</p> + +<p> +The slice-based <code>Image</code> implementations also provide a +<code>SubImage</code> method, which returns an <code>Image</code> backed by the +same array. Modifying the pixels of a sub-image will affect the pixels of the +original image, analagous to how modifying the contents of a sub-slice +<code>s[i0:i1]</code> will affect the contents of the original slice +<code>s</code>. +</p> + +<img src="image-package-05.png" width="400" height="300"> + +{{code "/doc/progs/image_package6.go" `/m0 := image.New/` `/fmt.Println\(m0.Stride/`}} + +<p> +For low-level code that works on an image's <code>Pix</code> field, be aware +that ranging over <code>Pix</code> can affect pixels outside an image's bounds. +In the example above, the pixels covered by <code>m1.Pix</code> are shaded in +blue. Higher-level code, such as the <code>At</code> and <code>Set</code> +methods or the <a href="/pkg/image/draw/">image/draw package</a>, will clip +their operations to the image's bounds. +</p> + +<p> +<b>Image Formats</b> +</p> + +<p> +The standard package library supports a number of common image formats, such as +GIF, JPEG and PNG. If you know the format of a source image file, you can +decode from an <a href="/pkg/io/#Reader"><code>io.Reader</code></a> directly. +</p> + +<pre> +import ( + "image/jpeg" + "image/png" + "io" +) + +// convertJPEGToPNG converts from JPEG to PNG. +func convertJPEGToPNG(w io.Writer, r io.Reader) error { + img, err := jpeg.Decode(r) + if err != nil { + return err + } + return png.Encode(w, img) +} +</pre> + +<p> +If you have image data of unknown format, the +<a href="/pkg/image/#Decode"><code>image.Decode</code></a> function can detect +the format. The set of recognized formats is constructed at run time and is not +limited to those in the standard package library. An image format package +typically registers its format in an init function, and the main package will +"underscore import" such a package solely for the side effect of format +registration. +</p> + +<pre> +import ( + "image" + "image/png" + "io" + + _ "code.google.com/p/vp8-go/webp" + _ "image/jpeg" +) + +// convertToPNG converts from any recognized format to PNG. +func convertToPNG(w io.Writer, r io.Reader) error { + img, _, err := image.Decode(r) + if err != nil { + return err + } + return png.Encode(w, img) +} +</pre> diff --git a/doc/articles/index.html b/doc/articles/index.html new file mode 100644 index 000000000..5f70734ec --- /dev/null +++ b/doc/articles/index.html @@ -0,0 +1,7 @@ +<!--{ + "Title": "/doc/articles/" +}--> + +<p> +See the <a href="/doc/#articles">Documents page</a> for a complete list of Go articles. +</p> diff --git a/doc/articles/json_and_go.html b/doc/articles/json_and_go.html new file mode 100644 index 000000000..af7776c0a --- /dev/null +++ b/doc/articles/json_and_go.html @@ -0,0 +1,356 @@ +<!--{ +"Title": "JSON and Go", +"Template": true +}--> + +<p> +JSON (JavaScript Object Notation) is a simple data interchange format. +Syntactically it resembles the objects and lists of JavaScript. It is most +commonly used for communication between web back-ends and JavaScript programs +running in the browser, but it is used in many other places, too. Its home page, +<a href="http://json.org">json.org</a>, provides a wonderfully clear and concise +definition of the standard. +</p> + +<p> +With the <a href="/pkg/encoding/json/">json package</a> it's a snap to read and +write JSON data from your Go programs. +</p> + +<p> +<b>Encoding</b> +</p> + +<p> +To encode JSON data we use the +<a href="/pkg/encoding/json/#Marshal"><code>Marshal</code></a> function. +</p> + +<pre> +func Marshal(v interface{}) ([]byte, error) +</pre> + +<p> +Given the Go data structure, <code>Message</code>, +</p> + +{{code "/doc/progs/json1.go" `/type Message/` `/STOP/`}} + +<p> +and an instance of <code>Message</code> +</p> + +{{code "/doc/progs/json1.go" `/m :=/`}} + +<p> +we can marshal a JSON-encoded version of m using <code>json.Marshal</code>: +</p> + +{{code "/doc/progs/json1.go" `/b, err :=/`}} + +<p> +If all is well, <code>err</code> will be <code>nil</code> and <code>b</code> +will be a <code>[]byte</code> containing this JSON data: +</p> + +<pre> +b == []byte(`{"Name":"Alice","Body":"Hello","Time":1294706395881547000}`) +</pre> + +<p> +Only data structures that can be represented as valid JSON will be encoded: +</p> + +<ul> +<li> +JSON objects only support strings as keys; to encode a Go map type it must be +of the form <code>map[string]T</code> (where <code>T</code> is any Go type +supported by the json package). +</li> +<li> +Channel, complex, and function types cannot be encoded. +</li> +<li> +Cyclic data structures are not supported; they will cause <code>Marshal</code> +to go into an infinite loop. +</li> +<li> +Pointers will be encoded as the values they point to (or 'null' if the pointer +is <code>nil</code>). +</li> +</ul> + +<p> +The json package only accesses the exported fields of struct types (those that +begin with an uppercase letter). Therefore only the the exported fields of a +struct will be present in the JSON output. +</p> + +<p> +<b>Decoding</b> +</p> + +<p> +To decode JSON data we use the +<a href="/pkg/encoding/json/#Unmarshal"><code>Unmarshal</code></a> function. +</p> + +<pre> +func Unmarshal(data []byte, v interface{}) error +</pre> + +<p> +We must first create a place where the decoded data will be stored +</p> + +{{code "/doc/progs/json1.go" `/var m Message/`}} + +<p> +and call <code>json.Unmarshal</code>, passing it a <code>[]byte</code> of JSON +data and a pointer to <code>m</code> +</p> + +{{code "/doc/progs/json1.go" `/err := json.Unmarshal/`}} + +<p> +If <code>b</code> contains valid JSON that fits in <code>m</code>, after the +call <code>err</code> will be <code>nil</code> and the data from <code>b</code> +will have been stored in the struct <code>m</code>, as if by an assignment +like: +</p> + +{{code "/doc/progs/json1.go" `/m = Message/` `/STOP/`}} + +<p> +How does <code>Unmarshal</code> identify the fields in which to store the +decoded data? For a given JSON key <code>"Foo"</code>, <code>Unmarshal</code> +will look through the destination struct's fields to find (in order of +preference): +</p> + +<ul> +<li> +An exported field with a tag of <code>"Foo"</code> (see the +<a href="/ref/spec#Struct_types">Go spec</a> for more on struct tags), +</li> +<li> +An exported field named <code>"Foo"</code>, or +</li> +<li> +An exported field named <code>"FOO"</code> or <code>"FoO"</code> or some other +case-insensitive match of <code>"Foo"</code>. +</li> +</ul> + +<p> +What happens when the structure of the JSON data doesn't exactly match the Go +type? +</p> + +{{code "/doc/progs/json1.go" `/"Food":"Pickle"/` `/STOP/`}} + +<p> +<code>Unmarshal</code> will decode only the fields that it can find in the +destination type. In this case, only the Name field of m will be populated, +and the Food field will be ignored. This behavior is particularly useful when +you wish to pick only a few specific fields out of a large JSON blob. It also +means that any unexported fields in the destination struct will be unaffected +by <code>Unmarshal</code>. +</p> + +<p> +But what if you don't know the structure of your JSON data beforehand? +</p> + +<p> +<b>Generic JSON with interface{}</b> +</p> + +<p> +The <code>interface{}</code> (empty interface) type describes an interface with +zero methods. Every Go type implements at least zero methods and therefore +satisfies the empty interface. +</p> + +<p> +The empty interface serves as a general container type: +</p> + +{{code "/doc/progs/json2.go" `/var i interface{}/` `/STOP/`}} + +<p> +A type assertion accesses the underlying concrete type: +</p> + +{{code "/doc/progs/json2.go" `/r := i/` `/STOP/`}} + +<p> +Or, if the underlying type is unknown, a type switch determines the type: +</p> + +{{code "/doc/progs/json2.go" `/switch v/` `/STOP/`}} + + +The json package uses <code>map[string]interface{}</code> and +<code>[]interface{}</code> values to store arbitrary JSON objects and arrays; +it will happily unmarshal any valid JSON blob into a plain +<code>interface{}</code> value. The default concrete Go types are: + +<ul> +<li> +<code>bool</code> for JSON booleans, +</li> +<li> +<code>float64</code> for JSON numbers, +</li> +<li> +<code>string</code> for JSON strings, and +</li> +<li> +<code>nil</code> for JSON null. +</li> +</ul> + +<p> +<b>Decoding arbitrary data</b> +</p> + +<p> +Consider this JSON data, stored in the variable <code>b</code>: +</p> + +{{code "/doc/progs/json3.go" `/b :=/`}} + +<p> +Without knowing this data's structure, we can decode it into an +<code>interface{}</code> value with <code>Unmarshal</code>: +</p> + +{{code "/doc/progs/json3.go" `/var f interface/` `/STOP/`}} + +<p> +At this point the Go value in <code>f</code> would be a map whose keys are +strings and whose values are themselves stored as empty interface values: +</p> + +{{code "/doc/progs/json3.go" `/f = map/` `/STOP/`}} + +<p> +To access this data we can use a type assertion to access <code>f</code>'s +underlying <code>map[string]interface{}</code>: +</p> + +{{code "/doc/progs/json3.go" `/m := f/`}} + +<p> +We can then iterate through the map with a range statement and use a type switch +to access its values as their concrete types: +</p> + +{{code "/doc/progs/json3.go" `/for k, v/` `/STOP/`}} + +<p> +In this way you can work with unknown JSON data while still enjoying the +benefits of type safety. +</p> + +<p> +<b>Reference Types</b> +</p> + +<p> +Let's define a Go type to contain the data from the previous example: +</p> + +{{code "/doc/progs/json4.go" `/type FamilyMember/` `/STOP/`}} + +{{code "/doc/progs/json4.go" `/var m FamilyMember/` `/STOP/`}} + +<p> +Unmarshaling that data into a <code>FamilyMember</code> value works as +expected, but if we look closely we can see a remarkable thing has happened. +With the var statement we allocated a <code>FamilyMember</code> struct, and +then provided a pointer to that value to <code>Unmarshal</code>, but at that +time the <code>Parents</code> field was a <code>nil</code> slice value. To +populate the <code>Parents</code> field, <code>Unmarshal</code> allocated a new +slice behind the scenes. This is typical of how <code>Unmarshal</code> works +with the supported reference types (pointers, slices, and maps). +</p> + +<p> +Consider unmarshaling into this data structure: +</p> + +<pre> +type Foo struct { + Bar *Bar +} +</pre> + +<p> +If there were a <code>Bar</code> field in the JSON object, +<code>Unmarshal</code> would allocate a new <code>Bar</code> and populate it. +If not, <code>Bar</code> would be left as a <code>nil</code> pointer. +</p> + +<p> +From this a useful pattern arises: if you have an application that receives a +few distinct message types, you might define "receiver" structure like +</p> + +<pre> +type IncomingMessage struct { + Cmd *Command + Msg *Message +} +</pre> + +<p> +and the sending party can populate the <code>Cmd</code> field and/or the +<code>Msg</code> field of the top-level JSON object, depending on the type of +message they want to communicate. <code>Unmarshal</code>, when decoding the +JSON into an <code>IncomingMessage</code> struct, will only allocate the data +structures present in the JSON data. To know which messages to process, the +programmer need simply test that either <code>Cmd</code> or <code>Msg</code> is +not <code>nil</code>. +</p> + +<p> +<b>Streaming Encoders and Decoders</b> +</p> + +<p> +The json package provides <code>Decoder</code> and <code>Encoder</code> types +to support the common operation of reading and writing streams of JSON data. +The <code>NewDecoder</code> and <code>NewEncoder</code> functions wrap the +<a href="/pkg/io/#Reader"><code>io.Reader</code></a> and +<a href="/pkg/io/#Writer"><code>io.Writer</code></a> interface types. +</p> + +<pre> +func NewDecoder(r io.Reader) *Decoder +func NewEncoder(w io.Writer) *Encoder +</pre> + +<p> +Here's an example program that reads a series of JSON objects from standard +input, removes all but the <code>Name</code> field from each object, and then +writes the objects to standard output: +</p> + +{{code "/doc/progs/json5.go" `/package main/` `$`}} + +<p> +Due to the ubiquity of Readers and Writers, these <code>Encoder</code> and +<code>Decoder</code> types can be used in a broad range of scenarios, such as +reading and writing to HTTP connections, WebSockets, or files. +</p> + +<p> +<b>References</b> +</p> + +<p> +For more information see the <a href="/pkg/encoding/json/">json package documentation</a>. For an example usage of +json see the source files of the <a href="/pkg/net/rpc/jsonrpc/">jsonrpc package</a>. +</p> diff --git a/doc/articles/json_rpc_tale_of_interfaces.html b/doc/articles/json_rpc_tale_of_interfaces.html new file mode 100644 index 000000000..a545f55f6 --- /dev/null +++ b/doc/articles/json_rpc_tale_of_interfaces.html @@ -0,0 +1,78 @@ +<!--{ +"Title": "JSON-RPC: a tale of interfaces" +}--> + +<p> +Here we present an example where Go's +<a href="/doc/effective_go.html#interfaces_and_types">interfaces</a> made it +easy to refactor some existing code to make it more flexible and extensible. +Originally, the standard library's <a href="/pkg/net/rpc/">RPC package</a> used +a custom wire format called <a href="/pkg/encoding/gob/">gob</a>. For a +particular application, we wanted to use <a href="/pkg/encoding/json/">JSON</a> +as an alternate wire format. +</p> + +<p> +We first defined a pair of interfaces to describe the functionality of the +existing wire format, one for the client, and one for the server (depicted +below). +</p> + +<pre> +type ServerCodec interface { + ReadRequestHeader(*Request) error + ReadRequestBody(interface{}) error + WriteResponse(*Response, interface{}) error + Close() error +} +</pre> + +<p> +On the server side, we then changed two internal function signatures to accept +the <code>ServerCodec</code> interface instead of our existing +<code>gob.Encoder</code>. Here's one of them: +</p> + +<pre> +func sendResponse(sending *sync.Mutex, req *Request, + reply interface{}, enc *gob.Encoder, errmsg string) +</pre> + +<p> +became +</p> + +<pre> +func sendResponse(sending *sync.Mutex, req *Request, + reply interface{}, enc ServerCodec, errmsg string) +</pre> + +<p> +We then wrote a trivial <code>gobServerCodec</code> wrapper to reproduce the +original functionality. From there it is simple to build a +<code>jsonServerCodec</code>. +</p> + +<p> +After some similar changes to the client side, this was the full extent of the +work we needed to do on the RPC package. This whole exercise took about 20 +minutes! After tidying up and testing the new code, the +<a href="http://code.google.com/p/go/source/diff?spec=svn9daf796ebf1cae97b2fcf760a4ab682f1f063f29&r=9daf796ebf1cae97b2fcf760a4ab682f1f063f29&format=side&path=/src/pkg/rpc/server.go">final changeset</a> +was submitted. +</p> + +<p> +In an inheritance-oriented language like Java or C++, the obvious path would be +to generalize the RPC class, and create JsonRPC and GobRPC subclasses. However, +this approach becomes tricky if you want to make a further generalization +orthogonal to that hierarchy. (For example, if you were to implement an +alternate RPC standard). In our Go package, we took a route that is both +conceptually simpler and requires less code be written or changed. +</p> + +<p> +A vital quality for any codebase is maintainability. As needs change, it is +essential to adapt your code easily and cleanly, lest it become unwieldy to work +with. We believe Go's lightweight, composition-oriented type system provides a +means of structuring code that scales. +</p> diff --git a/doc/articles/laws_of_reflection.html b/doc/articles/laws_of_reflection.html new file mode 100644 index 000000000..826a054f2 --- /dev/null +++ b/doc/articles/laws_of_reflection.html @@ -0,0 +1,649 @@ +<!--{ + "Title": "The Laws of Reflection", + "Template": true +}--> + +<p> +Reflection in computing is the +ability of a program to examine its own structure, particularly +through types; it's a form of metaprogramming. It's also a great +source of confusion. +</p> + +<p> +In this article we attempt to clarify things by explaining how +reflection works in Go. Each language's reflection model is +different (and many languages don't support it at all), but +this article is about Go, so for the rest of this article the word +"reflection" should be taken to mean "reflection in Go". +</p> + +<p><b>Types and interfaces</b></p> + +<p> +Because reflection builds on the type system, let's start with a +refresher about types in Go. +</p> + +<p> +Go is statically typed. Every variable has a static type, that is, +exactly one type known and fixed at compile time: <code>int</code>, +<code>float32</code>, <code>*MyType</code>, <code>[]byte</code>, +and so on. If we declare +</p> + +{{code "/doc/progs/interface.go" `/type MyInt/` `/STOP/`}} + +<p> +then <code>i</code> has type <code>int</code> and <code>j</code> +has type <code>MyInt</code>. The variables <code>i</code> and +<code>j</code> have distinct static types and, although they have +the same underlying type, they cannot be assigned to one another +without a conversion. +</p> + +<p> +One important category of type is interface types, which represent +fixed sets of methods. An interface variable can store any concrete +(non-interface) value as long as that value implements the +interface's methods. A well-known pair of examples is +<code>io.Reader</code> and <code>io.Writer</code>, the types +<code>Reader</code> and <code>Writer</code> from the +<a href="/pkg/io/">io package</a>: +</p> + +{{code "/doc/progs/interface.go" `/// Reader/` `/STOP/`}} + +<p> +Any type that implements a <code>Read</code> (or +<code>Write</code>) method with this signature is said to implement +<code>io.Reader</code> (or <code>io.Writer</code>). For the +purposes of this discussion, that means that a variable of type +<code>io.Reader</code> can hold any value whose type has a +<code>Read</code> method: +</p> + +{{code "/doc/progs/interface.go" `/func readers/` `/STOP/`}} + +<p> +It's important to be clear that whatever concrete value +<code>r</code> may hold, <code>r</code>'s type is always +<code>io.Reader</code>: Go is statically typed and the static type +of <code>r</code> is <code>io.Reader</code>.</p> + +<p> +An extremely important example of an interface type is the empty +interface: +</p> + +<pre> +interface{} +</pre> + +<p> +It represents the empty set of methods and is satisfied by any +value at all, since any value has zero or more methods. +</p> + +<p> +Some people say that Go's interfaces are dynamically typed, but +that is misleading. They are statically typed: a variable of +interface type always has the same static type, and even though at +run time the value stored in the interface variable may change +type, that value will always satisfy the interface. +</p> + +<p> +We need to be precise about all this because reflection and +interfaces are closely related. +</p> + +<p><b>The representation of an interface</b></p> + +<p> +Russ Cox has written a +<a href="http://research.swtch.com/2009/12/go-data-structures-interfaces.html">detailed blog post</a> +about the representation of interface values in Go. It's not necessary to +repeat the full story here, but a simplified summary is in order. +</p> + +<p> +A variable of interface type stores a pair: the concrete value +assigned to the variable, and that value's type descriptor. +To be more precise, the value is the underlying concrete data item +that implements the interface and the type describes the full type +of that item. For instance, after +</p> + +{{code "/doc/progs/interface.go" `/func typeAssertions/` `/STOP/`}} + +<p> +<code>r</code> contains, schematically, the (value, type) pair, +(<code>tty</code>, <code>*os.File</code>). Notice that the type +<code>*os.File</code> implements methods other than +<code>Read</code>; even though the interface value provides access +only to the <code>Read</code> method, the value inside carries all +the type information about that value. That's why we can do things +like this: +</p> + +{{code "/doc/progs/interface.go" `/var w io.Writer/` `/STOP/`}} + +<p> +The expression in this assignment is a type assertion; what it +asserts is that the item inside <code>r</code> also implements +<code>io.Writer</code>, and so we can assign it to <code>w</code>. +After the assignment, <code>w</code> will contain the pair +(<code>tty</code>, <code>*os.File</code>). That's the same pair as +was held in <code>r</code>. The static type of the interface +determines what methods may be invoked with an interface variable, +even though the concrete value inside may have a larger set of +methods. +</p> + +<p> +Continuing, we can do this: +</p> + +{{code "/doc/progs/interface.go" `/var empty interface{}/` `/STOP/`}} + +<p> +and our empty interface value <code>e</code> will again contain +that same pair, (<code>tty</code>, <code>*os.File</code>). That's +handy: an empty interface can hold any value and contains all the +information we could ever need about that value. +</p> + +<p> +(We don't need a type assertion here because it's known statically +that <code>w</code> satisfies the empty interface. In the example +where we moved a value from a <code>Reader</code> to a +<code>Writer</code>, we needed to be explicit and use a type +assertion because <code>Writer</code>'s methods are not a +subset of <code>Reader</code>'s.) +</p> + +<p> +One important detail is that the pair inside an interface always +has the form (value, concrete type) and cannot have the form +(value, interface type). Interfaces do not hold interface +values. +</p> + +<p> +Now we're ready to reflect. +</p> + +<p><b>The first law of reflection</b></p> + +<p><b>1. Reflection goes from interface value to reflection object.</b></p> + +<p> +At the basic level, reflection is just a mechanism to examine the +type and value pair stored inside an interface variable. To get +started, there are two types we need to know about in +<a href="/pkg/reflect/">package reflect</a>: +<a href="/pkg/reflect/#Type">Type</a> and +<a href="/pkg/reflect/#Value">Value</a>. Those two types +give access to the contents of an interface variable, and two +simple functions, called <code>reflect.TypeOf</code> and +<code>reflect.ValueOf</code>, retrieve <code>reflect.Type</code> +and <code>reflect.Value</code> pieces out of an interface value. +(Also, from the <code>reflect.Value</code> it's easy to get +to the <code>reflect.Type</code>, but let's keep the +<code>Value</code> and <code>Type</code> concepts separate for +now.) +</p> + +<p> +Let's start with <code>TypeOf</code>: +</p> + +{{code "/doc/progs/interface2.go" `/package main/` `/STOP main/`}} + +<p> +This program prints +</p> + +<pre> +type: float64 +</pre> + +<p> +You might be wondering where the interface is here, since the program looks +like it's passing the <code>float64</code> variable <code>x</code>, not an +interface value, to <code>reflect.TypeOf</code>. But it's there; as +<a href="/pkg/reflect/#Type.TypeOf">godoc reports</a>, the signature of +<code>reflect.TypeOf</code> includes an empty interface: +</p> + +<pre> +// TypeOf returns the reflection Type of the value in the interface{}. +func TypeOf(i interface{}) Type +</pre> + +<p> +When we call <code>reflect.TypeOf(x)</code>, <code>x</code> is +first stored in an empty interface, which is then passed as the +argument; <code>reflect.TypeOf</code> unpacks that empty interface +to recover the type information. +</p> + +<p> +The <code>reflect.ValueOf</code> function, of course, recovers the +value (from here on we'll elide the boilerplate and focus just on +the executable code): +</p> + +{{code "/doc/progs/interface2.go" `/START f9/` `/STOP/`}} + +<p> +prints +</p> + +<pre> +value: <float64 Value> +</pre> + +<p> +Both <code>reflect.Type</code> and <code>reflect.Value</code> have +lots of methods to let us examine and manipulate them. One +important example is that <code>Value</code> has a +<code>Type</code> method that returns the <code>Type</code> of a +<code>reflect.Value</code>. Another is that both <code>Type</code> +and <code>Value</code> have a <code>Kind</code> method that returns +a constant indicating what sort of item is stored: +<code>Uint</code>, <code>Float64</code>, <code>Slice</code>, and so +on. Also methods on <code>Value</code> with names like +<code>Int</code> and <code>Float</code> let us grab values (as +<code>int64</code> and <code>float64</code>) stored inside: +</p> + +{{code "/doc/progs/interface2.go" `/START f1/` `/STOP/`}} + +<p> +prints +</p> + +<pre> +type: float64 +kind is float64: true +value: 3.4 +</pre> + +<p> +There are also methods like <code>SetInt</code> and +<code>SetFloat</code> but to use them we need to understand +settability, the subject of the third law of reflection, discussed +below. +</p> + +<p> +The reflection library has a couple of properties worth singling +out. First, to keep the API simple, the "getter" and "setter" +methods of <code>Value</code> operate on the largest type that can +hold the value: <code>int64</code> for all the signed integers, for +instance. That is, the <code>Int</code> method of +<code>Value</code> returns an <code>int64</code> and the +<code>SetInt</code> value takes an <code>int64</code>; it may be +necessary to convert to the actual type involved: +</p> + +{{code "/doc/progs/interface2.go" `/START f2/` `/STOP/`}} + +<p> +The second property is that the <code>Kind</code> of a reflection +object describes the underlying type, not the static type. If a +reflection object contains a value of a user-defined integer type, +as in +</p> + +{{code "/doc/progs/interface2.go" `/START f3/` `/STOP/`}} + +<p> +the <code>Kind</code> of <code>v</code> is still +<code>reflect.Int</code>, even though the static type of +<code>x</code> is <code>MyInt</code>, not <code>int</code>. In +other words, the <code>Kind</code> cannot discriminate an int from +a <code>MyInt</code> even though the <code>Type</code> can. +</p> + +<p><b>The second law of reflection</b></p> + +<p><b>2. Reflection goes from reflection object to interface +value.</b></p> + +<p> +Like physical reflection, reflection in Go generates its own +inverse. +</p> + +<p> +Given a <code>reflect.Value</code> we can recover an interface +value using the <code>Interface</code> method; in effect the method +packs the type and value information back into an interface +representation and returns the result: +</p> + +<pre> +// Interface returns v's value as an interface{}. +func (v Value) Interface() interface{} +</pre> + +<p> +As a consequence we can say +</p> + +{{code "/doc/progs/interface2.go" `/START f3b/` `/STOP/`}} + +<p> +to print the <code>float64</code> value represented by the +reflection object <code>v</code>. +</p> + +<p> +We can do even better, though. The arguments to +<code>fmt.Println</code>, <code>fmt.Printf</code> and so on are all +passed as empty interface values, which are then unpacked by the +<code>fmt</code> package internally just as we have been doing in +the previous examples. Therefore all it takes to print the contents +of a <code>reflect.Value</code> correctly is to pass the result of +the <code>Interface</code> method to the formatted print +routine: +</p> + +{{code "/doc/progs/interface2.go" `/START f3c/` `/STOP/`}} + +<p> +(Why not <code>fmt.Println(v)</code>? Because <code>v</code> is a +<code>reflect.Value</code>; we want the concrete value it holds.) +Since our value is a <code>float64</code>, we can even use a +floating-point format if we want: +</p> + +{{code "/doc/progs/interface2.go" `/START f3d/` `/STOP/`}} + +<p> +and get in this case +</p> + +<pre> +3.4e+00 +</pre> + +<p> +Again, there's no need to type-assert the result of +<code>v.Interface()</code> to <code>float64</code>; the empty +interface value has the concrete value's type information inside +and <code>Printf</code> will recover it. +</p> + +<p> +In short, the <code>Interface</code> method is the inverse of the +<code>ValueOf</code> function, except that its result is always of +static type <code>interface{}</code>. +</p> + +<p> +Reiterating: Reflection goes from interface values to reflection +objects and back again. +</p> + +<p><b>The third law of reflection</b></p> + +<p><b>3. To modify a reflection object, the value must be settable.</b></p> + +<p> +The third law is the most subtle and confusing, but it's easy +enough to understand if we start from first principles. +</p> + +<p> +Here is some code that does not work, but is worth studying. +</p> + +{{code "/doc/progs/interface2.go" `/START f4/` `/STOP/`}} + +<p> +If you run this code, it will panic with the cryptic message +</p> + +<pre> +panic: reflect.Value.SetFloat using unaddressable value +</pre> + +<p> +The problem is not that the value <code>7.1</code> is not +addressable; it's that <code>v</code> is not settable. Settability +is a property of a reflection <code>Value</code>, and not all +reflection <code>Values</code> have it. +</p> + +<p> +The <code>CanSet</code> method of <code>Value</code> reports the +settability of a <code>Value</code>; in our case, +</p> + +{{code "/doc/progs/interface2.go" `/START f5/` `/STOP/`}} + +<p> +prints +</p> + +<pre> +settability of v: false +</pre> + +<p> +It is an error to call a <code>Set</code> method on an non-settable +<code>Value</code>. But what is settability? +</p> + +<p> +Settability is a bit like addressability, but stricter. It's the +property that a reflection object can modify the actual storage +that was used to create the reflection object. Settability is +determined by whether the reflection object holds the original +item. When we say +</p> + +{{code "/doc/progs/interface2.go" `/START f6/` `/STOP/`}} + +<p> +we pass a <em>copy</em> of <code>x</code> to +<code>reflect.ValueOf</code>, so the interface value created as the +argument to <code>reflect.ValueOf</code> is a <em>copy</em> of +<code>x</code>, not <code>x</code> itself. Thus, if the +statement +</p> + +{{code "/doc/progs/interface2.go" `/START f6b/` `/STOP/`}} + +<p> +were allowed to succeed, it would not update <code>x</code>, even +though <code>v</code> looks like it was created from +<code>x</code>. Instead, it would update the copy of <code>x</code> +stored inside the reflection value and <code>x</code> itself would +be unaffected. That would be confusing and useless, so it is +illegal, and settability is the property used to avoid this +issue. +</p> + +<p> +If this seems bizarre, it's not. It's actually a familiar situation +in unusual garb. Think of passing <code>x</code> to a +function: +</p> + +<pre> +f(x) +</pre> + +<p> +We would not expect <code>f</code> to be able to modify +<code>x</code> because we passed a copy of <code>x</code>'s value, +not <code>x</code> itself. If we want <code>f</code> to modify +<code>x</code> directly we must pass our function the address of +<code>x</code> (that is, a pointer to <code>x</code>):</p> + +<p> +<code>f(&x)</code> +</p> + +<p> +This is straightforward and familiar, and reflection works the same +way. If we want to modify <code>x</code> by reflection, we must +give the reflection library a pointer to the value we want to +modify. +</p> + +<p> +Let's do that. First we initialize <code>x</code> as usual +and then create a reflection value that points to it, called +<code>p</code>. +</p> + +{{code "/doc/progs/interface2.go" `/START f7/` `/STOP/`}} + +<p> +The output so far is +</p> + +<pre> +type of p: *float64 +settability of p: false +</pre> + +<p> +The reflection object <code>p</code> isn't settable, but it's not +<code>p</code> we want to set, it's (in effect) <code>*p</code>. To +get to what <code>p</code> points to, we call the <code>Elem</code> +method of <code>Value</code>, which indirects through the pointer, +and save the result in a reflection <code>Value</code> called +<code>v</code>: +</p> + +{{code "/doc/progs/interface2.go" `/START f7b/` `/STOP/`}} + +<p> +Now <code>v</code> is a settable reflection object, as the output +demonstrates, +</p> + +<pre> +settability of v: true +</pre> + +<p> +and since it represents <code>x</code>, we are finally able to use +<code>v.SetFloat</code> to modify the value of +<code>x</code>: +</p> + +{{code "/doc/progs/interface2.go" `/START f7c/` `/STOP/`}} + +<p> +The output, as expected, is +</p> + +<pre> +7.1 +7.1 +</pre> + +<p> +Reflection can be hard to understand but it's doing exactly what +the language does, albeit through reflection <code>Types</code> and +<code>Values</code> that can disguise what's going on. Just keep in +mind that reflection Values need the address of something in order +to modify what they represent. +</p> + +<p><b>Structs</b></p> + +<p> +In our previous example <code>v</code> wasn't a pointer itself, it +was just derived from one. A common way for this situation to arise +is when using reflection to modify the fields of a structure. As +long as we have the address of the structure, we can modify its +fields. +</p> + +<p> +Here's a simple example that analyzes a struct value, <code>t</code>. We create +the reflection object with the address of the struct because we'll want to +modify it later. Then we set <code>typeOfT</code> to its type and iterate over +the fields using straightforward method calls +(see <a href="/pkg/reflect/">package reflect</a> for details). +Note that we extract the names of the fields from the struct type, but the +fields themselves are regular <code>reflect.Value</code> objects. +</p> + +{{code "/doc/progs/interface2.go" `/START f8/` `/STOP/`}} + +<p> +The output of this program is +</p> + +<pre> +0: A int = 23 +1: B string = skidoo +</pre> + +<p> +There's one more point about settability introduced in +passing here: the field names of <code>T</code> are upper case +(exported) because only exported fields of a struct are +settable. +</p> + +<p> +Because <code>s</code> contains a settable reflection object, we +can modify the fields of the structure. +</p> + +{{code "/doc/progs/interface2.go" `/START f8b/` `/STOP/`}} + +<p> +And here's the result: +</p> + +<pre> +t is now {77 Sunset Strip} +</pre> + +<p> +If we modified the program so that <code>s</code> was created from +<code>t</code>, not <code>&t</code>, the calls to +<code>SetInt</code> and <code>SetString</code> would fail as the +fields of <code>t</code> would not be settable. +</p> + +<p><b>Conclusion</b></p> + +<p> +Here again are the laws of reflection: +</p> + +<ol> +<li>Reflection goes from interface value to reflection +object.</li> +<li>Reflection goes from reflection object to interface +value.</li> +<li>To modify a reflection object, the value must be settable.</li> +</ol> + +<p> +Once you understand these laws reflection in Go becomes much easier +to use, although it remains subtle. It's a powerful tool that +should be used with care and avoided unless strictly +necessary. +</p> + +<p> +There's plenty more to reflection that we haven't covered — +sending and receiving on channels, allocating memory, using slices +and maps, calling methods and functions — but this post is +long enough. We'll cover some of those topics in a later +article. +</p> diff --git a/doc/articles/slice-1.png b/doc/articles/slice-1.png Binary files differnew file mode 100644 index 000000000..ba465cf71 --- /dev/null +++ b/doc/articles/slice-1.png diff --git a/doc/articles/slice-2.png b/doc/articles/slice-2.png Binary files differnew file mode 100644 index 000000000..a57581e8c --- /dev/null +++ b/doc/articles/slice-2.png diff --git a/doc/articles/slice-3.png b/doc/articles/slice-3.png Binary files differnew file mode 100644 index 000000000..64ece5e87 --- /dev/null +++ b/doc/articles/slice-3.png diff --git a/doc/articles/slice-array.png b/doc/articles/slice-array.png Binary files differnew file mode 100644 index 000000000..a533702cf --- /dev/null +++ b/doc/articles/slice-array.png diff --git a/doc/articles/slice-struct.png b/doc/articles/slice-struct.png Binary files differnew file mode 100644 index 000000000..f9141fc59 --- /dev/null +++ b/doc/articles/slice-struct.png diff --git a/doc/articles/slices_usage_and_internals.html b/doc/articles/slices_usage_and_internals.html new file mode 100644 index 000000000..810b0a41f --- /dev/null +++ b/doc/articles/slices_usage_and_internals.html @@ -0,0 +1,438 @@ +<!--{ + "Title": "Slices: usage and internals", + "Template": true +}--> + +<p> +Go's slice type provides a convenient and efficient means of working with +sequences of typed data. Slices are analogous to arrays in other languages, but +have some unusual properties. This article will look at what slices are and how +they are used. +</p> + +<p> +<b>Arrays</b> +</p> + +<p> +The slice type is an abstraction built on top of Go's array type, and so to +understand slices we must first understand arrays. +</p> + +<p> +An array type definition specifies a length and an element type. For example, +the type <code>[4]int</code> represents an array of four integers. An array's +size is fixed; its length is part of its type (<code>[4]int</code> and +<code>[5]int</code> are distinct, incompatible types). Arrays can be indexed in +the usual way, so the expression <code>s[n]</code> accesses the <i>n</i>th +element: +</p> + +<pre> +var a [4]int +a[0] = 1 +i := a[0] +// i == 1 +</pre> + +<p> +Arrays do not need to be initialized explicitly; the zero value of an array is +a ready-to-use array whose elements are themselves zeroed: +</p> + +<pre> +// a[2] == 0, the zero value of the int type +</pre> + +<p> +The in-memory representation of <code>[4]int</code> is just four integer values laid out sequentially: +</p> + +<p> +<img src="slice-array.png"> +</p> + +<p> +Go's arrays are values. An array variable denotes the entire array; it is not a +pointer to the first array element (as would be the case in C). This means +that when you assign or pass around an array value you will make a copy of its +contents. (To avoid the copy you could pass a <i>pointer</i> to the array, but +then that's a pointer to an array, not an array.) One way to think about arrays +is as a sort of struct but with indexed rather than named fields: a fixed-size +composite value. +</p> + +<p> +An array literal can be specified like so: +</p> + +<pre> +b := [2]string{"Penn", "Teller"} +</pre> + +<p> +Or, you can have the compiler count the array elements for you: +</p> + +<pre> +b := [...]string{"Penn", "Teller"} +</pre> + +<p> +In both cases, the type of <code>b</code> is <code>[2]string</code>. +</p> + +<p> +<b>Slices</b> +</p> + +<p> +Arrays have their place, but they're a bit inflexible, so you don't see them +too often in Go code. Slices, though, are everywhere. They build on arrays to +provide great power and convenience. +</p> + +<p> +The type specification for a slice is <code>[]T</code>, where <code>T</code> is +the type of the elements of the slice. Unlike an array type, a slice type has +no specified length. +</p> + +<p> +A slice literal is declared just like an array literal, except you leave out +the element count: +</p> + +<pre> +letters := []string{"a", "b", "c", "d"} +</pre> + +<p> +A slice can be created with the built-in function called <code>make</code>, +which has the signature, +</p> + +<pre> +func make([]T, len, cap) []T +</pre> + +<p> +where T stands for the element type of the slice to be created. The +<code>make</code> function takes a type, a length, and an optional capacity. +When called, <code>make</code> allocates an array and returns a slice that +refers to that array. +</p> + +<pre> +var s []byte +s = make([]byte, 5, 5) +// s == []byte{0, 0, 0, 0, 0} +</pre> + +<p> +When the capacity argument is omitted, it defaults to the specified length. +Here's a more succinct version of the same code: +</p> + +<pre> +s := make([]byte, 5) +</pre> + +<p> +The length and capacity of a slice can be inspected using the built-in +<code>len</code> and <code>cap</code> functions. +</p> + +<pre> +len(s) == 5 +cap(s) == 5 +</pre> + +<p> +The next two sections discuss the relationship between length and capacity. +</p> + +<p> +The zero value of a slice is <code>nil</code>. The <code>len</code> and +<code>cap</code> functions will both return 0 for a nil slice. +</p> + +<p> +A slice can also be formed by "slicing" an existing slice or array. Slicing is +done by specifying a half-open range with two indices separated by a colon. For +example, the expression <code>b[1:4]</code> creates a slice including elements +1 through 3 of <code>b</code> (the indices of the resulting slice will be 0 +through 2). +</p> + +<pre> +b := []byte{'g', 'o', 'l', 'a', 'n', 'g'} +// b[1:4] == []byte{'o', 'l', 'a'}, sharing the same storage as b +</pre> + +<p> +The start and end indices of a slice expression are optional; they default to zero and the slice's length respectively: +</p> + +<pre> +// b[:2] == []byte{'g', 'o'} +// b[2:] == []byte{'l', 'a', 'n', 'g'} +// b[:] == b +</pre> + +<p> +This is also the syntax to create a slice given an array: +</p> + +<pre> +x := [3]string{"Лайка", "Белка", "Стрелка"} +s := x[:] // a slice referencing the storage of x +</pre> + +<p> +<b>Slice internals</b> +</p> + +<p> +A slice is a descriptor of an array segment. It consists of a pointer to the +array, the length of the segment, and its capacity (the maximum length of the +segment). +</p> + +<p> +<img src="slice-struct.png"> +</p> + +<p> +Our variable <code>s</code>, created earlier by <code>make([]byte, 5)</code>, +is structured like this: +</p> + +<p> +<img src="slice-1.png"> +</p> + +<p> +The length is the number of elements referred to by the slice. The capacity is +the number of elements in the underlying array (beginning at the element +referred to by the slice pointer). The distinction between length and capacity +will be made clear as we walk through the next few examples. +</p> + +<p> +As we slice <code>s</code>, observe the changes in the slice data structure and +their relation to the underlying array: +</p> + +<pre> +s = s[2:4] +</pre> + +<p> +<img src="slice-2.png"> +</p> + +<p> +Slicing does not copy the slice's data. It creates a new slice value that +points to the original array. This makes slice operations as efficient as +manipulating array indices. Therefore, modifying the <i>elements</i> (not the +slice itself) of a re-slice modifies the elements of the original slice: +</p> + +<pre> +d := []byte{'r', 'o', 'a', 'd'} +e := d[2:] +// e == []byte{'a', 'd'} +e[1] == 'm' +// e == []byte{'a', 'm'} +// d == []byte{'r', 'o', 'a', 'm'} +</pre> + +<p> +Earlier we sliced <code>s</code> to a length shorter than its capacity. We can +grow s to its capacity by slicing it again: +</p> + +<pre> +s = s[:cap(s)] +</pre> + +<p> +<img src="slice-3.png"> +</p> + +<p> +A slice cannot be grown beyond its capacity. Attempting to do so will cause a +runtime panic, just as when indexing outside the bounds of a slice or array. +Similarly, slices cannot be re-sliced below zero to access earlier elements in +the array. +</p> + +<p> +<b>Growing slices (the copy and append functions)</b> +</p> + +<p> +To increase the capacity of a slice one must create a new, larger slice and +copy the contents of the original slice into it. This technique is how dynamic +array implementations from other languages work behind the scenes. The next +example doubles the capacity of <code>s</code> by making a new slice, +<code>t</code>, copying the contents of <code>s</code> into <code>t</code>, and +then assigning the slice value <code>t</code> to <code>s</code>: +</p> + +<pre> +t := make([]byte, len(s), (cap(s)+1)*2) // +1 in case cap(s) == 0 +for i := range s { + t[i] = s[i] +} +s = t +</pre> + +<p> +The looping piece of this common operation is made easier by the built-in copy +function. As the name suggests, copy copies data from a source slice to a +destination slice. It returns the number of elements copied. +</p> + +<pre> +func copy(dst, src []T) int +</pre> + +<p> +The <code>copy</code> function supports copying between slices of different +lengths (it will copy only up to the smaller number of elements). In addition, +<code>copy</code> can handle source and destination slices that share the same +underlying array, handling overlapping slices correctly. +</p> + +<p> +Using <code>copy</code>, we can simplify the code snippet above: +</p> + +<pre> +t := make([]byte, len(s), (cap(s)+1)*2) +copy(t, s) +s = t +</pre> + +<p> +A common operation is to append data to the end of a slice. This function +appends byte elements to a slice of bytes, growing the slice if necessary, and +returns the updated slice value: +</p> + +{{code "/doc/progs/slices.go" `/AppendByte/` `/STOP/`}} + +<p> +One could use <code>AppendByte</code> like this: +</p> + +<pre> +p := []byte{2, 3, 5} +p = AppendByte(p, 7, 11, 13) +// p == []byte{2, 3, 5, 7, 11, 13} +</pre> + +<p> +Functions like <code>AppendByte</code> are useful because they offer complete +control over the way the slice is grown. Depending on the characteristics of +the program, it may be desirable to allocate in smaller or larger chunks, or to +put a ceiling on the size of a reallocation. +</p> + +<p> +But most programs don't need complete control, so Go provides a built-in +<code>append</code> function that's good for most purposes; it has the +signature +</p> + +<pre> +func append(s []T, x ...T) []T +</pre> + +<p> +The <code>append</code> function appends the elements <code>x</code> to the end +of the slice <code>s</code>, and grows the slice if a greater capacity is +needed. +</p> + +<pre> +a := make([]int, 1) +// a == []int{0} +a = append(a, 1, 2, 3) +// a == []int{0, 1, 2, 3} +</pre> + +<p> +To append one slice to another, use <code>...</code> to expand the second +argument to a list of arguments. +</p> + +<pre> +a := []string{"John", "Paul"} +b := []string{"George", "Ringo", "Pete"} +a = append(a, b...) // equivalent to "append(a, b[0], b[1], b[2])" +// a == []string{"John", "Paul", "George", "Ringo", "Pete"} +</pre> + +<p> +Since the zero value of a slice (<code>nil</code>) acts like a zero-length +slice, you can declare a slice variable and then append to it in a loop: +</p> + +{{code "/doc/progs/slices.go" `/Filter/` `/STOP/`}} + +<p> +<b>A possible "gotcha"</b> +</p> + +<p> +As mentioned earlier, re-slicing a slice doesn't make a copy of the underlying +array. The full array will be kept in memory until it is no longer referenced. +Occasionally this can cause the program to hold all the data in memory when +only a small piece of it is needed. +</p> + +<p> +For example, this <code>FindDigits</code> function loads a file into memory and +searches it for the first group of consecutive numeric digits, returning them +as a new slice. +</p> + +{{code "/doc/progs/slices.go" `/digit/` `/STOP/`}} + +<p> +This code behaves as advertised, but the returned <code>[]byte</code> points +into an array containing the entire file. Since the slice references the +original array, as long as the slice is kept around the garbage collector can't +release the array; the few useful bytes of the file keep the entire contents in +memory. +</p> + +<p> +To fix this problem one can copy the interesting data to a new slice before +returning it: +</p> + +{{code "/doc/progs/slices.go" `/CopyDigits/` `/STOP/`}} + +<p> +A more concise version of this function could be constructed by using +<code>append</code>. This is left as an exercise for the reader. +</p> + +<p> +<b>Further Reading</b> +</p> + +<p> +<a href="/doc/effective_go.html">Effective Go</a> contains an +in-depth treatment of <a href="/doc/effective_go.html#slices">slices</a> +and <a href="/doc/effective_go.html#arrays">arrays</a>, +and the Go <a href="/doc/go_spec.html">language specification</a> +defines <a href="/doc/go_spec.html#Slice_types">slices</a> and their +<a href="/doc/go_spec.html#Length_and_capacity">associated</a> +<a href="/doc/go_spec.html#Making_slices_maps_and_channels">helper</a> +<a href="/doc/go_spec.html#Appending_and_copying_slices">functions</a>. +</p> diff --git a/doc/articles/wiki/Makefile b/doc/articles/wiki/Makefile new file mode 100644 index 000000000..0cb907185 --- /dev/null +++ b/doc/articles/wiki/Makefile @@ -0,0 +1,20 @@ +# Copyright 2010 The Go Authors. All rights reserved. +# Use of this source code is governed by a BSD-style +# license that can be found in the LICENSE file. + +all: index.html + +CLEANFILES:=srcextract.bin htmlify.bin get.bin + +index.html: wiki.html srcextract.bin htmlify.bin + PATH=.:$$PATH awk '/^!/{system(substr($$0,2)); next} {print}' < wiki.html | tr -d '\r' > index.html + +test: get.bin + bash ./test.sh + rm -f get.6 get.bin + +%.bin: %.go + go build -o $@ $^ + +clean: + rm -f $(CLEANFILES) diff --git a/doc/articles/wiki/edit.html b/doc/articles/wiki/edit.html new file mode 100644 index 000000000..c14953b17 --- /dev/null +++ b/doc/articles/wiki/edit.html @@ -0,0 +1,6 @@ +<h1>Editing {{.Title |html}}</h1> + +<form action="/save/{{.Title |html}}" method="POST"> +<div><textarea name="body" rows="20" cols="80">{{printf "%s" .Body |html}}</textarea></div> +<div><input type="submit" value="Save"></div> +</form> diff --git a/doc/articles/wiki/final-noclosure.go b/doc/articles/wiki/final-noclosure.go new file mode 100644 index 000000000..a23cf7a27 --- /dev/null +++ b/doc/articles/wiki/final-noclosure.go @@ -0,0 +1,104 @@ +// Copyright 2010 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package main + +import ( + "errors" + "html/template" + "io/ioutil" + "net/http" + "regexp" +) + +type Page struct { + Title string + Body []byte +} + +func (p *Page) save() error { + filename := p.Title + ".txt" + return ioutil.WriteFile(filename, p.Body, 0600) +} + +func loadPage(title string) (*Page, error) { + filename := title + ".txt" + body, err := ioutil.ReadFile(filename) + if err != nil { + return nil, err + } + return &Page{Title: title, Body: body}, nil +} + +func viewHandler(w http.ResponseWriter, r *http.Request) { + title, err := getTitle(w, r) + if err != nil { + return + } + p, err := loadPage(title) + if err != nil { + http.Redirect(w, r, "/edit/"+title, http.StatusFound) + return + } + renderTemplate(w, "view", p) +} + +func editHandler(w http.ResponseWriter, r *http.Request) { + title, err := getTitle(w, r) + if err != nil { + return + } + p, err := loadPage(title) + if err != nil { + p = &Page{Title: title} + } + renderTemplate(w, "edit", p) +} + +func saveHandler(w http.ResponseWriter, r *http.Request) { + title, err := getTitle(w, r) + if err != nil { + return + } + body := r.FormValue("body") + p := &Page{Title: title, Body: []byte(body)} + err = p.save() + if err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } + http.Redirect(w, r, "/view/"+title, http.StatusFound) +} + +func renderTemplate(w http.ResponseWriter, tmpl string, p *Page) { + t, err := template.ParseFiles(tmpl + ".html") + if err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } + err = t.Execute(w, p) + if err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + } +} + +const lenPath = len("/view/") + +var titleValidator = regexp.MustCompile("^[a-zA-Z0-9]+$") + +func getTitle(w http.ResponseWriter, r *http.Request) (title string, err error) { + title = r.URL.Path[lenPath:] + if !titleValidator.MatchString(title) { + http.NotFound(w, r) + err = errors.New("Invalid Page Title") + } + return +} + +func main() { + http.HandleFunc("/view/", viewHandler) + http.HandleFunc("/edit/", editHandler) + http.HandleFunc("/save/", saveHandler) + http.ListenAndServe(":8080", nil) +} diff --git a/doc/articles/wiki/final-noerror.go b/doc/articles/wiki/final-noerror.go new file mode 100644 index 000000000..e11d268e2 --- /dev/null +++ b/doc/articles/wiki/final-noerror.go @@ -0,0 +1,55 @@ +// Copyright 2010 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package main + +import ( + "html/template" + "io/ioutil" + "net/http" +) + +type Page struct { + Title string + Body []byte +} + +func (p *Page) save() error { + filename := p.Title + ".txt" + return ioutil.WriteFile(filename, p.Body, 0600) +} + +func loadPage(title string) (*Page, error) { + filename := title + ".txt" + body, err := ioutil.ReadFile(filename) + if err != nil { + return nil, err + } + return &Page{Title: title, Body: body}, nil +} + +const lenPath = len("/view/") + +func editHandler(w http.ResponseWriter, r *http.Request) { + title := r.URL.Path[lenPath:] + p, err := loadPage(title) + if err != nil { + p = &Page{Title: title} + } + t, _ := template.ParseFiles("edit.html") + t.Execute(w, p) +} + +func viewHandler(w http.ResponseWriter, r *http.Request) { + title := r.URL.Path[lenPath:] + p, _ := loadPage(title) + t, _ := template.ParseFiles("view.html") + t.Execute(w, p) +} + +func main() { + http.HandleFunc("/view/", viewHandler) + http.HandleFunc("/edit/", editHandler) + http.ListenAndServe(":8080", nil) +} diff --git a/doc/articles/wiki/final-parsetemplate.go b/doc/articles/wiki/final-parsetemplate.go new file mode 100644 index 000000000..6234c08f2 --- /dev/null +++ b/doc/articles/wiki/final-parsetemplate.go @@ -0,0 +1,93 @@ +// Copyright 2010 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package main + +import ( + "html/template" + "io/ioutil" + "net/http" + "regexp" +) + +type Page struct { + Title string + Body []byte +} + +func (p *Page) save() error { + filename := p.Title + ".txt" + return ioutil.WriteFile(filename, p.Body, 0600) +} + +func loadPage(title string) (*Page, error) { + filename := title + ".txt" + body, err := ioutil.ReadFile(filename) + if err != nil { + return nil, err + } + return &Page{Title: title, Body: body}, nil +} + +func viewHandler(w http.ResponseWriter, r *http.Request, title string) { + p, err := loadPage(title) + if err != nil { + http.Redirect(w, r, "/edit/"+title, http.StatusFound) + return + } + renderTemplate(w, "view", p) +} + +func editHandler(w http.ResponseWriter, r *http.Request, title string) { + p, err := loadPage(title) + if err != nil { + p = &Page{Title: title} + } + renderTemplate(w, "edit", p) +} + +func saveHandler(w http.ResponseWriter, r *http.Request, title string) { + body := r.FormValue("body") + p := &Page{Title: title, Body: []byte(body)} + err := p.save() + if err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } + http.Redirect(w, r, "/view/"+title, http.StatusFound) +} + +func renderTemplate(w http.ResponseWriter, tmpl string, p *Page) { + t, err := template.ParseFiles(tmpl + ".html") + if err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } + err = t.Execute(w, p) + if err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + } +} + +const lenPath = len("/view/") + +var titleValidator = regexp.MustCompile("^[a-zA-Z0-9]+$") + +func makeHandler(fn func(http.ResponseWriter, *http.Request, string)) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + title := r.URL.Path[lenPath:] + if !titleValidator.MatchString(title) { + http.NotFound(w, r) + return + } + fn(w, r, title) + } +} + +func main() { + http.HandleFunc("/view/", makeHandler(viewHandler)) + http.HandleFunc("/edit/", makeHandler(editHandler)) + http.HandleFunc("/save/", makeHandler(saveHandler)) + http.ListenAndServe(":8080", nil) +} diff --git a/doc/articles/wiki/final-template.go b/doc/articles/wiki/final-template.go new file mode 100644 index 000000000..f295b9d60 --- /dev/null +++ b/doc/articles/wiki/final-template.go @@ -0,0 +1,67 @@ +// Copyright 2010 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package main + +import ( + "html/template" + "io/ioutil" + "net/http" +) + +type Page struct { + Title string + Body []byte +} + +func (p *Page) save() error { + filename := p.Title + ".txt" + return ioutil.WriteFile(filename, p.Body, 0600) +} + +func loadPage(title string) (*Page, error) { + filename := title + ".txt" + body, err := ioutil.ReadFile(filename) + if err != nil { + return nil, err + } + return &Page{Title: title, Body: body}, nil +} + +const lenPath = len("/view/") + +func editHandler(w http.ResponseWriter, r *http.Request) { + title := r.URL.Path[lenPath:] + p, err := loadPage(title) + if err != nil { + p = &Page{Title: title} + } + renderTemplate(w, "edit", p) +} + +func viewHandler(w http.ResponseWriter, r *http.Request) { + title := r.URL.Path[lenPath:] + p, _ := loadPage(title) + renderTemplate(w, "view", p) +} + +func saveHandler(w http.ResponseWriter, r *http.Request) { + title := r.URL.Path[lenPath:] + body := r.FormValue("body") + p := &Page{Title: title, Body: []byte(body)} + p.save() + http.Redirect(w, r, "/view/"+title, http.StatusFound) +} + +func renderTemplate(w http.ResponseWriter, tmpl string, p *Page) { + t, _ := template.ParseFiles(tmpl + ".html") + t.Execute(w, p) +} + +func main() { + http.HandleFunc("/view/", viewHandler) + http.HandleFunc("/edit/", editHandler) + http.HandleFunc("/save/", saveHandler) + http.ListenAndServe(":8080", nil) +} diff --git a/doc/articles/wiki/final.go b/doc/articles/wiki/final.go new file mode 100644 index 000000000..e93cdee47 --- /dev/null +++ b/doc/articles/wiki/final.go @@ -0,0 +1,90 @@ +// Copyright 2010 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package main + +import ( + "html/template" + "io/ioutil" + "net/http" + "regexp" +) + +type Page struct { + Title string + Body []byte +} + +func (p *Page) save() error { + filename := p.Title + ".txt" + return ioutil.WriteFile(filename, p.Body, 0600) +} + +func loadPage(title string) (*Page, error) { + filename := title + ".txt" + body, err := ioutil.ReadFile(filename) + if err != nil { + return nil, err + } + return &Page{Title: title, Body: body}, nil +} + +func viewHandler(w http.ResponseWriter, r *http.Request, title string) { + p, err := loadPage(title) + if err != nil { + http.Redirect(w, r, "/edit/"+title, http.StatusFound) + return + } + renderTemplate(w, "view", p) +} + +func editHandler(w http.ResponseWriter, r *http.Request, title string) { + p, err := loadPage(title) + if err != nil { + p = &Page{Title: title} + } + renderTemplate(w, "edit", p) +} + +func saveHandler(w http.ResponseWriter, r *http.Request, title string) { + body := r.FormValue("body") + p := &Page{Title: title, Body: []byte(body)} + err := p.save() + if err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } + http.Redirect(w, r, "/view/"+title, http.StatusFound) +} + +var templates = template.Must(template.ParseFiles("edit.html", "view.html")) + +func renderTemplate(w http.ResponseWriter, tmpl string, p *Page) { + err := templates.ExecuteTemplate(w, tmpl+".html", p) + if err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + } +} + +const lenPath = len("/view/") + +var titleValidator = regexp.MustCompile("^[a-zA-Z0-9]+$") + +func makeHandler(fn func(http.ResponseWriter, *http.Request, string)) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + title := r.URL.Path[lenPath:] + if !titleValidator.MatchString(title) { + http.NotFound(w, r) + return + } + fn(w, r, title) + } +} + +func main() { + http.HandleFunc("/view/", makeHandler(viewHandler)) + http.HandleFunc("/edit/", makeHandler(editHandler)) + http.HandleFunc("/save/", makeHandler(saveHandler)) + http.ListenAndServe(":8080", nil) +} diff --git a/doc/articles/wiki/get.go b/doc/articles/wiki/get.go new file mode 100644 index 000000000..c6e9bf28b --- /dev/null +++ b/doc/articles/wiki/get.go @@ -0,0 +1,54 @@ +// Copyright 2011 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package main + +import ( + "flag" + "fmt" + "io" + "log" + "net" + "net/http" + "os" + "strings" +) + +var ( + post = flag.String("post", "", "urlencoded form data to POST") + addr = flag.Bool("addr", false, "find open address and print to stdout") +) + +func main() { + flag.Parse() + if *addr { + l, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + log.Fatal(err) + } + defer l.Close() + fmt.Print(l.Addr()) + return + } + url := flag.Arg(0) + if url == "" { + log.Fatal("no url supplied") + } + var r *http.Response + var err error + if *post != "" { + b := strings.NewReader(*post) + r, err = http.Post(url, "application/x-www-form-urlencoded", b) + } else { + r, err = http.Get(url) + } + if err != nil { + log.Fatal(err) + } + defer r.Body.Close() + _, err = io.Copy(os.Stdout, r.Body) + if err != nil { + log.Fatal(err) + } +} diff --git a/doc/articles/wiki/htmlify.go b/doc/articles/wiki/htmlify.go new file mode 100644 index 000000000..2a845a174 --- /dev/null +++ b/doc/articles/wiki/htmlify.go @@ -0,0 +1,16 @@ +// Copyright 2010 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package main + +import ( + "io/ioutil" + "os" + "text/template" +) + +func main() { + b, _ := ioutil.ReadAll(os.Stdin) + template.HTMLEscape(os.Stdout, b) +} diff --git a/doc/articles/wiki/http-sample.go b/doc/articles/wiki/http-sample.go new file mode 100644 index 000000000..ac8cc4f2d --- /dev/null +++ b/doc/articles/wiki/http-sample.go @@ -0,0 +1,15 @@ +package main + +import ( + "fmt" + "net/http" +) + +func handler(w http.ResponseWriter, r *http.Request) { + fmt.Fprintf(w, "Hi there, I love %s!", r.URL.Path[1:]) +} + +func main() { + http.HandleFunc("/", handler) + http.ListenAndServe(":8080", nil) +} diff --git a/doc/articles/wiki/index.html b/doc/articles/wiki/index.html new file mode 100644 index 000000000..52bf7e798 --- /dev/null +++ b/doc/articles/wiki/index.html @@ -0,0 +1,721 @@ +<!--{ + "Title": "Writing Web Applications", + "Template": true +}--> + +<h2>Introduction</h2> + +<p> +Covered in this tutorial: +</p> +<ul> +<li>Creating a data structure with load and save methods</li> +<li>Using the <code>net/http</code> package to build web applications +<li>Using the <code>html/template</code> package to process HTML templates</li> +<li>Using the <code>regexp</code> package to validate user input</li> +<li>Using closures</li> +</ul> + +<p> +Assumed knowledge: +</p> +<ul> +<li>Programming experience</li> +<li>Understanding of basic web technologies (HTTP, HTML)</li> +<li>Some UNIX/DOS command-line knowledge</li> +</ul> + +<h2>Getting Started</h2> + +<p> +At present, you need to have a FreeBSD, Linux, OS X, or Windows machine to run Go. +We will use <code>$</code> to represent the command prompt. +</p> + +<p> +Install Go (see the <a href="/doc/install">Installation Instructions</a>). +</p> + +<p> +Make a new directory for this tutorial inside your <code>GOPATH</code> and cd to it: +</p> + +<pre> +$ mkdir gowiki +$ cd gowiki +</pre> + +<p> +Create a file named <code>wiki.go</code>, open it in your favorite editor, and +add the following lines: +</p> + +<pre> +package main + +import ( + "fmt" + "io/ioutil" +) +</pre> + +<p> +We import the <code>fmt</code> and <code>ioutil</code> packages from the Go +standard library. Later, as we implement additional functionality, we will +add more packages to this <code>import</code> declaration. +</p> + +<h2>Data Structures</h2> + +<p> +Let's start by defining the data structures. A wiki consists of a series of +interconnected pages, each of which has a title and a body (the page content). +Here, we define <code>Page</code> as a struct with two fields representing +the title and body. +</p> + +{{code "doc/articles/wiki/part1.go" `/^type Page/` `/}/`}} + +<p> +The type <code>[]byte</code> means "a <code>byte</code> slice". +(See <a href="/doc/articles/slices_usage_and_internals.html">Slices: usage and +internals</a> for more on slices.) +The <code>Body</code> element is a <code>[]byte</code> rather than +<code>string</code> because that is the type expected by the <code>io</code> +libraries we will use, as you'll see below. +</p> + +<p> +The <code>Page</code> struct describes how page data will be stored in memory. +But what about persistent storage? We can address that by creating a +<code>save</code> method on <code>Page</code>: +</p> + +{{code "doc/articles/wiki/part1.go" `/^func.*Page.*save/` `/}/`}} + +<p> +This method's signature reads: "This is a method named <code>save</code> that +takes as its receiver <code>p</code>, a pointer to <code>Page</code> . It takes +no parameters, and returns a value of type <code>error</code>." +</p> + +<p> +This method will save the <code>Page</code>'s <code>Body</code> to a text +file. For simplicity, we will use the <code>Title</code> as the file name. +</p> + +<p> +The <code>save</code> method returns an <code>error</code> value because +that is the return type of <code>WriteFile</code> (a standard library function +that writes a byte slice to a file). The <code>save</code> method returns the +error value, to let the application handle it should anything go wrong while +writing the file. If all goes well, <code>Page.save()</code> will return +<code>nil</code> (the zero-value for pointers, interfaces, and some other +types). +</p> + +<p> +The octal integer constant <code>0600</code>, passed as the third parameter to +<code>WriteFile</code>, indicates that the file should be created with +read-write permissions for the current user only. (See the Unix man page +<code>open(2)</code> for details.) +</p> + +<p> +We will want to load pages, too: +</p> + +{{code "doc/articles/wiki/part1-noerror.go" `/^func loadPage/` `/^}/`}} + +<p> +The function <code>loadPage</code> constructs the file name from +<code>Title</code>, reads the file's contents into a new +<code>Page</code>, and returns a pointer to that new <code>page</code>. +</p> + +<p> +Functions can return multiple values. The standard library function +<code>io.ReadFile</code> returns <code>[]byte</code> and <code>error</code>. +In <code>loadPage</code>, error isn't being handled yet; the "blank identifier" +represented by the underscore (<code>_</code>) symbol is used to throw away the +error return value (in essence, assigning the value to nothing). +</p> + +<p> +But what happens if <code>ReadFile</code> encounters an error? For example, +the file might not exist. We should not ignore such errors. Let's modify the +function to return <code>*Page</code> and <code>error</code>. +</p> + +{{code "doc/articles/wiki/part1.go" `/^func loadPage/` `/^}/`}} + +<p> +Callers of this function can now check the second parameter; if it is +<code>nil</code> then it has successfully loaded a Page. If not, it will be an +<code>error</code> that can be handled by the caller (see the +<a href="/ref/spec#Errors">language specification</a> for details). +</p> + +<p> +At this point we have a simple data structure and the ability to save to and +load from a file. Let's write a <code>main</code> function to test what we've +written: +</p> + +{{code "doc/articles/wiki/part1.go" `/^func main/` `/^}/`}} + +<p> +After compiling and executing this code, a file named <code>TestPage.txt</code> +would be created, containing the contents of <code>p1</code>. The file would +then be read into the struct <code>p2</code>, and its <code>Body</code> element +printed to the screen. +</p> + +<p> +You can compile and run the program like this: +</p> + +<pre> +$ go build wiki.go +$ ./wiki +This is a sample page. +</pre> + +<p> +(If you're using Windows you must type "<code>wiki</code>" without the +"<code>./</code>" to run the program.) +</p> + +<p> +<a href="part1.go">Click here to view the code we've written so far.</a> +</p> + +<h2>Introducing the <code>net/http</code> package (an interlude)</h2> + +<p> +Here's a full working example of a simple web server: +</p> + +{{code "doc/articles/wiki/http-sample.go"}} + +<p> +The <code>main</code> function begins with a call to +<code>http.HandleFunc</code>, which tells the <code>http</code> package to +handle all requests to the web root (<code>"/"</code>) with +<code>handler</code>. +</p> + +<p> +It then calls <code>http.ListenAndServe</code>, specifying that it should +listen on port 8080 on any interface (<code>":8080"</code>). (Don't +worry about its second parameter, <code>nil</code>, for now.) +This function will block until the program is terminated. +</p> + +<p> +The function <code>handler</code> is of the type <code>http.HandlerFunc</code>. +It takes an <code>http.ResponseWriter</code> and an <code>http.Request</code> as +its arguments. +</p> + +<p> +An <code>http.ResponseWriter</code> value assembles the HTTP server's response; by writing +to it, we send data to the HTTP client. +</p> + +<p> +An <code>http.Request</code> is a data structure that represents the client +HTTP request. The string <code>r.URL.Path</code> is the path component +of the request URL. The trailing <code>[1:]</code> means +"create a sub-slice of <code>Path</code> from the 1st character to the end." +This drops the leading "/" from the path name. +</p> + +<p> +If you run this program and access the URL: +</p> +<pre>http://localhost:8080/monkeys</pre> +<p> +the program would present a page containing: +</p> +<pre>Hi there, I love monkeys!</pre> + +<h2>Using <code>net/http</code> to serve wiki pages</h2> + +<p> +To use the <code>net/http</code> package, it must be imported: +</p> + +<pre> +import ( + "fmt" + <b>"net/http"</b> + "io/ioutil" +) +</pre> + +<p> +Let's create a handler to view a wiki page: +</p> + +{{code "doc/articles/wiki/part2.go" `/^const lenPath/`}} + +{{code "doc/articles/wiki/part2.go" `/^func viewHandler/` `/^}/`}} + +<p> +First, this function extracts the page title from <code>r.URL.Path</code>, +the path component of the request URL. The global constant +<code>lenPath</code> is the length of the leading <code>"/view/"</code> +component of the request path. +The <code>Path</code> is re-sliced with <code>[lenPath:]</code> to drop the +first 6 characters of the string. This is because the path will invariably +begin with <code>"/view/"</code>, which is not part of the page title. +</p> + +<p> +The function then loads the page data, formats the page with a string of simple +HTML, and writes it to <code>w</code>, the <code>http.ResponseWriter</code>. +</p> + +<p> +Again, note the use of <code>_</code> to ignore the <code>error</code> +return value from <code>loadPage</code>. This is done here for simplicity +and generally considered bad practice. We will attend to this later. +</p> + +<p> +To use this handler, we create a <code>main</code> function that +initializes <code>http</code> using the <code>viewHandler</code> to handle +any requests under the path <code>/view/</code>. +</p> + +{{code "doc/articles/wiki/part2.go" `/^func main/` `/^}/`}} + +<p> +<a href="part2.go">Click here to view the code we've written so far.</a> +</p> + +<p> +Let's create some page data (as <code>test.txt</code>), compile our code, and +try serving a wiki page. +</p> + +<p> +Open <code>test.txt</code> file in your editor, and save the string "Hello world" (without quotes) +in it. +</p> + +<pre> +$ go build wiki.go +$ ./wiki +</pre> + +<p> +With this web server running, a visit to <code><a +href="http://localhost:8080/view/test">http://localhost:8080/view/test</a></code> +should show a page titled "test" containing the words "Hello world". +</p> + +<h2>Editing Pages</h2> + +<p> +A wiki is not a wiki without the ability to edit pages. Let's create two new +handlers: one named <code>editHandler</code> to display an 'edit page' form, +and the other named <code>saveHandler</code> to save the data entered via the +form. +</p> + +<p> +First, we add them to <code>main()</code>: +</p> + +{{code "doc/articles/wiki/final-noclosure.go" `/^func main/` `/^}/`}} + +<p> +The function <code>editHandler</code> loads the page +(or, if it doesn't exist, create an empty <code>Page</code> struct), +and displays an HTML form. +</p> + +{{code "doc/articles/wiki/notemplate.go" `/^func editHandler/` `/^}/`}} + +<p> +This function will work fine, but all that hard-coded HTML is ugly. +Of course, there is a better way. +</p> + +<h2>The <code>html/template</code> package</h2> + +<p> +The <code>html/template</code> package is part of the Go standard library. +We can use <code>html/template</code> to keep the HTML in a separate file, +allowing us to change the layout of our edit page without modifying the +underlying Go code. +</p> + +<p> +First, we must add <code>html/template</code> to the list of imports: +</p> + +<pre> +import ( + "http" + "io/ioutil" + "os" + <b>"html/template"</b> +) +</pre> + +<p> +Let's create a template file containing the HTML form. +Open a new file named <code>edit.html</code>, and add the following lines: +</p> + +{{code "doc/articles/wiki/edit.html"}} + +<p> +Modify <code>editHandler</code> to use the template, instead of the hard-coded +HTML: +</p> + +{{code "doc/articles/wiki/final-noerror.go" `/^func editHandler/` `/^}/`}} + +<p> +The function <code>template.ParseFiles</code> will read the contents of +<code>edit.html</code> and return a <code>*template.Template</code>. +</p> + +<p> +The method <code>t.Execute</code> executes the template, writing the +generated HTML to the <code>http.ResponseWriter</code>. +The <code>.Title</code> and <code>.Body</code> dotted identifiers refer to +<code>p.Title</code> and <code>p.Body</code>. +</p> + +<p> +Template directives are enclosed in double curly braces. +The <code>printf "%s" .Body</code> instruction is a function call +that outputs <code>.Body</code> as a string instead of a stream of bytes, +the same as a call to <code>fmt.Printf</code>. +The <code>|html</code> part of each directive pipes the value through the +<code>html</code> formatter before outputting it, which escapes HTML +characters (such as replacing <code>></code> with <code>&gt;</code>), +preventing user data from corrupting the form HTML. +</p> + +<p> +Now that we've removed the <code>fmt.Fprintf</code> statement, we can remove +<code>"fmt"</code> from the <code>import</code> list. +</p> + +<p> +While we're working with templates, let's create a template for our +<code>viewHandler</code> called <code>view.html</code>: +</p> + +{{code "doc/articles/wiki/view.html"}} + +<p> +Modify <code>viewHandler</code> accordingly: +</p> + +{{code "doc/articles/wiki/final-noerror.go" `/^func viewHandler/` `/^}/`}} + +<p> +Notice that we've used almost exactly the same templating code in both +handlers. Let's remove this duplication by moving the templating code +to its own function: +</p> + +{{code "doc/articles/wiki/final-template.go" `/^func viewHandler/` `/^}/`}} +{{code "doc/articles/wiki/final-template.go" `/^func editHandler/` `/^}/`}} +{{code "doc/articles/wiki/final-template.go" `/^func renderTemplate/` `/^}/`}} + +<p> +The handlers are now shorter and simpler. +</p> + +<h2>Handling non-existent pages</h2> + +<p> +What if you visit <a href="http://localhost:8080/view/APageThatDoesntExist"> +<code>/view/APageThatDoesntExist</code></a>? The program will crash. This is +because it ignores the error return value from <code>loadPage</code>. Instead, +if the requested Page doesn't exist, it should redirect the client to the edit +Page so the content may be created: +</p> + +{{code "doc/articles/wiki/final-noclosure.go" `/^func viewHandler/` `/^}/`}} + +<p> +The <code>http.Redirect</code> function adds an HTTP status code of +<code>http.StatusFound</code> (302) and a <code>Location</code> +header to the HTTP response. +</p> + +<h2>Saving Pages</h2> + +<p> +The function <code>saveHandler</code> will handle the form submission. +</p> + +{{code "doc/articles/wiki/final-template.go" `/^func saveHandler/` `/^}/`}} + +<p> +The page title (provided in the URL) and the form's only field, +<code>Body</code>, are stored in a new <code>Page</code>. +The <code>save()</code> method is then called to write the data to a file, +and the client is redirected to the <code>/view/</code> page. +</p> + +<p> +The value returned by <code>FormValue</code> is of type <code>string</code>. +We must convert that value to <code>[]byte</code> before it will fit into +the <code>Page</code> struct. We use <code>[]byte(body)</code> to perform +the conversion. +</p> + +<h2>Error handling</h2> + +<p> +There are several places in our program where errors are being ignored. This +is bad practice, not least because when an error does occur the program will +crash. A better solution is to handle the errors and return an error message +to the user. That way if something does go wrong, the server will continue to +function and the user will be notified. +</p> + +<p> +First, let's handle the errors in <code>renderTemplate</code>: +</p> + +{{code "doc/articles/wiki/final-parsetemplate.go" `/^func renderTemplate/` `/^}/`}} + +<p> +The <code>http.Error</code> function sends a specified HTTP response code +(in this case "Internal Server Error") and error message. +Already the decision to put this in a separate function is paying off. +</p> + +<p> +Now let's fix up <code>saveHandler</code>: +</p> + +{{code "doc/articles/wiki/final-noclosure.go" `/^func saveHandler/` `/^}/`}} + +<p> +Any errors that occur during <code>p.save()</code> will be reported +to the user. +</p> + +<h2>Template caching</h2> + +<p> +There is an inefficiency in this code: <code>renderTemplate</code> calls +<code>ParseFiles</code> every time a page is rendered. +A better approach would be to call <code>ParseFiles</code> once at program +initialization, parsing all templates into a single <code>*Template</code>. +Then we can use the +<a href="/pkg/html/template/#Template.ExecuteTemplate"><code>ExecuteTemplate</code></a> +method to render a specific template. +</p> + +<p> +First we create a global variable named <code>templates</code>, and initialize +it with <code>ParseFiles</code>. +</p> + +{{code "doc/articles/wiki/final.go" `/var templates/`}} + +<p> +The function <code>template.Must</code> is a convenience wrapper that panics +when passed a non-nil <code>error</code> value, and otherwise returns the +<code>*Template</code> unaltered. A panic is appropriate here; if the templates +can't be loaded the only sensible thing to do is exit the program. +</p> + +<p> +A <code>for</code> loop is used with a <code>range</code> statement to iterate +over an array constant containing the names of the templates we want parsed. +If we were to add more templates to our program, we would add their names to +that array. +</p> + +<p> +We then modify the <code>renderTemplate</code> function to call the +<code>templates.ExecuteTemplate</code> method with the name of the appropriate +template: +</p> + +{{code "doc/articles/wiki/final.go" `/func renderTemplate/` `/^}/`}} + +<p> +Note that the template name is the template file name, so we must +append <code>".html"</code> to the <code>tmpl</code> argument. +</p> + +<h2>Validation</h2> + +<p> +As you may have observed, this program has a serious security flaw: a user +can supply an arbitrary path to be read/written on the server. To mitigate +this, we can write a function to validate the title with a regular expression. +</p> + +<p> +First, add <code>"regexp"</code> to the <code>import</code> list. +Then we can create a global variable to store our validation regexp: +</p> + +{{code "doc/articles/wiki/final-noclosure.go" `/^var titleValidator/`}} + +<p> +The function <code>regexp.MustCompile</code> will parse and compile the +regular expression, and return a <code>regexp.Regexp</code>. +<code>MustCompile</code> is distinct from <code>Compile</code> in that it will +panic if the expression compilation fails, while <code>Compile</code> returns +an <code>error</code> as a second parameter. +</p> + +<p> +Now, let's write a function that extracts the title string from the request +URL, and tests it against our <code>TitleValidator</code> expression: +</p> + +{{code "doc/articles/wiki/final-noclosure.go" `/func getTitle/` `/^}/`}} + +<p> +If the title is valid, it will be returned along with a <code>nil</code> +error value. If the title is invalid, the function will write a +"404 Not Found" error to the HTTP connection, and return an error to the +handler. +</p> + +<p> +Let's put a call to <code>getTitle</code> in each of the handlers: +</p> + +{{code "doc/articles/wiki/final-noclosure.go" `/^func viewHandler/` `/^}/`}} +{{code "doc/articles/wiki/final-noclosure.go" `/^func editHandler/` `/^}/`}} +{{code "doc/articles/wiki/final-noclosure.go" `/^func saveHandler/` `/^}/`}} + +<h2>Introducing Function Literals and Closures</h2> + +<p> +Catching the error condition in each handler introduces a lot of repeated code. +What if we could wrap each of the handlers in a function that does this +validation and error checking? Go's +<a href="/ref/spec#Function_declarations">function +literals</a> provide a powerful means of abstracting functionality +that can help us here. +</p> + +<p> +First, we re-write the function definition of each of the handlers to accept +a title string: +</p> + +<pre> +func viewHandler(w http.ResponseWriter, r *http.Request, title string) +func editHandler(w http.ResponseWriter, r *http.Request, title string) +func saveHandler(w http.ResponseWriter, r *http.Request, title string) +</pre> + +<p> +Now let's define a wrapper function that <i>takes a function of the above +type</i>, and returns a function of type <code>http.HandlerFunc</code> +(suitable to be passed to the function <code>http.HandleFunc</code>): +</p> + +<pre> +func makeHandler(fn func (http.ResponseWriter, *http.Request, string)) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + // Here we will extract the page title from the Request, + // and call the provided handler 'fn' + } +} +</pre> + +<p> +The returned function is called a closure because it encloses values defined +outside of it. In this case, the variable <code>fn</code> (the single argument +to <code>makeHandler</code>) is enclosed by the closure. The variable +<code>fn</code> will be one of our save, edit, or view handlers. +</p> + +<p> +Now we can take the code from <code>getTitle</code> and use it here +(with some minor modifications): +</p> + +{{code "doc/articles/wiki/final.go" `/func makeHandler/` `/^}/`}} + +<p> +The closure returned by <code>makeHandler</code> is a function that takes +an <code>http.ResponseWriter</code> and <code>http.Request</code> (in other +words, an <code>http.HandlerFunc</code>). +The closure extracts the <code>title</code> from the request path, and +validates it with the <code>TitleValidator</code> regexp. If the +<code>title</code> is invalid, an error will be written to the +<code>ResponseWriter</code> using the <code>http.NotFound</code> function. +If the <code>title</code> is valid, the enclosed handler function +<code>fn</code> will be called with the <code>ResponseWriter</code>, +<code>Request</code>, and <code>title</code> as arguments. +</p> + +<p> +Now we can wrap the handler functions with <code>makeHandler</code> in +<code>main</code>, before they are registered with the <code>http</code> +package: +</p> + +{{code "doc/articles/wiki/final.go" `/func main/` `/^}/`}} + +<p> +Finally we remove the calls to <code>getTitle</code> from the handler functions, +making them much simpler: +</p> + +{{code "doc/articles/wiki/final.go" `/^func viewHandler/` `/^}/`}} +{{code "doc/articles/wiki/final.go" `/^func editHandler/` `/^}/`}} +{{code "doc/articles/wiki/final.go" `/^func saveHandler/` `/^}/`}} + +<h2>Try it out!</h2> + +<p> +<a href="final.go">Click here to view the final code listing.</a> +</p> + +<p> +Recompile the code, and run the app: +</p> + +<pre> +$ go build wiki.go +$ ./wiki +</pre> + +<p> +Visiting <a href="http://localhost:8080/view/ANewPage">http://localhost:8080/view/ANewPage</a> +should present you with the page edit form. You should then be able to +enter some text, click 'Save', and be redirected to the newly created page. +</p> + +<h2>Other tasks</h2> + +<p> +Here are some simple tasks you might want to tackle on your own: +</p> + +<ul> +<li>Store templates in <code>tmpl/</code> and page data in <code>data/</code>. +<li>Add a handler to make the web root redirect to + <code>/view/FrontPage</code>.</li> +<li>Spruce up the page templates by making them valid HTML and adding some + CSS rules.</li> +<li>Implement inter-page linking by converting instances of + <code>[PageName]</code> to <br> + <code><a href="/view/PageName">PageName</a></code>. + (hint: you could use <code>regexp.ReplaceAllFunc</code> to do this) + </li> +</ul> diff --git a/doc/articles/wiki/notemplate.go b/doc/articles/wiki/notemplate.go new file mode 100644 index 000000000..33006ac95 --- /dev/null +++ b/doc/articles/wiki/notemplate.go @@ -0,0 +1,58 @@ +// Copyright 2010 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package main + +import ( + "fmt" + "io/ioutil" + "net/http" +) + +type Page struct { + Title string + Body []byte +} + +func (p *Page) save() error { + filename := p.Title + ".txt" + return ioutil.WriteFile(filename, p.Body, 0600) +} + +func loadPage(title string) (*Page, error) { + filename := title + ".txt" + body, err := ioutil.ReadFile(filename) + if err != nil { + return nil, err + } + return &Page{Title: title, Body: body}, nil +} + +const lenPath = len("/view/") + +func viewHandler(w http.ResponseWriter, r *http.Request) { + title := r.URL.Path[lenPath:] + p, _ := loadPage(title) + fmt.Fprintf(w, "<h1>%s</h1><div>%s</div>", p.Title, p.Body) +} + +func editHandler(w http.ResponseWriter, r *http.Request) { + title := r.URL.Path[lenPath:] + p, err := loadPage(title) + if err != nil { + p = &Page{Title: title} + } + fmt.Fprintf(w, "<h1>Editing %s</h1>"+ + "<form action=\"/save/%s\" method=\"POST\">"+ + "<textarea name=\"body\">%s</textarea><br>"+ + "<input type=\"submit\" value=\"Save\">"+ + "</form>", + p.Title, p.Title, p.Body) +} + +func main() { + http.HandleFunc("/view/", viewHandler) + http.HandleFunc("/edit/", editHandler) + http.ListenAndServe(":8080", nil) +} diff --git a/doc/articles/wiki/part1-noerror.go b/doc/articles/wiki/part1-noerror.go new file mode 100644 index 000000000..7577b7b46 --- /dev/null +++ b/doc/articles/wiki/part1-noerror.go @@ -0,0 +1,33 @@ +// Copyright 2010 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package main + +import ( + "fmt" + "io/ioutil" +) + +type Page struct { + Title string + Body []byte +} + +func (p *Page) save() error { + filename := p.Title + ".txt" + return ioutil.WriteFile(filename, p.Body, 0600) +} + +func loadPage(title string) *Page { + filename := title + ".txt" + body, _ := ioutil.ReadFile(filename) + return &Page{Title: title, Body: body} +} + +func main() { + p1 := &Page{Title: "TestPage", Body: []byte("This is a sample page.")} + p1.save() + p2 := loadPage("TestPage") + fmt.Println(string(p2.Body)) +} diff --git a/doc/articles/wiki/part1.go b/doc/articles/wiki/part1.go new file mode 100644 index 000000000..d7bf1be97 --- /dev/null +++ b/doc/articles/wiki/part1.go @@ -0,0 +1,36 @@ +// Copyright 2010 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package main + +import ( + "fmt" + "io/ioutil" +) + +type Page struct { + Title string + Body []byte +} + +func (p *Page) save() error { + filename := p.Title + ".txt" + return ioutil.WriteFile(filename, p.Body, 0600) +} + +func loadPage(title string) (*Page, error) { + filename := title + ".txt" + body, err := ioutil.ReadFile(filename) + if err != nil { + return nil, err + } + return &Page{Title: title, Body: body}, nil +} + +func main() { + p1 := &Page{Title: "TestPage", Body: []byte("This is a sample Page.")} + p1.save() + p2, _ := loadPage("TestPage") + fmt.Println(string(p2.Body)) +} diff --git a/doc/articles/wiki/part2.go b/doc/articles/wiki/part2.go new file mode 100644 index 000000000..dd4365c82 --- /dev/null +++ b/doc/articles/wiki/part2.go @@ -0,0 +1,43 @@ +// Copyright 2010 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package main + +import ( + "fmt" + "io/ioutil" + "net/http" +) + +type Page struct { + Title string + Body []byte +} + +func (p *Page) save() error { + filename := p.Title + ".txt" + return ioutil.WriteFile(filename, p.Body, 0600) +} + +func loadPage(title string) (*Page, error) { + filename := title + ".txt" + body, err := ioutil.ReadFile(filename) + if err != nil { + return nil, err + } + return &Page{Title: title, Body: body}, nil +} + +const lenPath = len("/view/") + +func viewHandler(w http.ResponseWriter, r *http.Request) { + title := r.URL.Path[lenPath:] + p, _ := loadPage(title) + fmt.Fprintf(w, "<h1>%s</h1><div>%s</div>", p.Title, p.Body) +} + +func main() { + http.HandleFunc("/view/", viewHandler) + http.ListenAndServe(":8080", nil) +} diff --git a/doc/articles/wiki/srcextract.go b/doc/articles/wiki/srcextract.go new file mode 100644 index 000000000..813e25283 --- /dev/null +++ b/doc/articles/wiki/srcextract.go @@ -0,0 +1,76 @@ +// Copyright 2010 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package main + +import ( + "bytes" + "flag" + "go/ast" + "go/parser" + "go/printer" + "go/token" + "log" + "os" + "text/template" +) + +var ( + srcFn = flag.String("src", "", "source filename") + getName = flag.String("name", "", "func/type name to output") + html = flag.Bool("html", true, "output HTML") + showPkg = flag.Bool("pkg", false, "show package in output") +) + +func main() { + // handle input + flag.Parse() + if *srcFn == "" || *getName == "" { + flag.Usage() + os.Exit(2) + } + // load file + fs := token.NewFileSet() + file, err := parser.ParseFile(fs, *srcFn, nil, 0) + if err != nil { + log.Fatal(err) + } + // create filter + filter := func(name string) bool { + return name == *getName + } + // filter + if !ast.FilterFile(file, filter) { + os.Exit(1) + } + // print the AST + var b bytes.Buffer + printer.Fprint(&b, fs, file) + // drop package declaration + if !*showPkg { + for { + c, err := b.ReadByte() + if c == '\n' || err != nil { + break + } + } + } + // drop leading newlines + for { + b, err := b.ReadByte() + if err != nil { + break + } + if b != '\n' { + os.Stdout.Write([]byte{b}) + break + } + } + // output + if *html { + template.HTMLEscape(os.Stdout, b.Bytes()) + } else { + b.WriteTo(os.Stdout) + } +} diff --git a/doc/articles/wiki/test.bash b/doc/articles/wiki/test.bash new file mode 100755 index 000000000..5c2cb60dc --- /dev/null +++ b/doc/articles/wiki/test.bash @@ -0,0 +1,30 @@ +#!/usr/bin/env bash +# Copyright 2010 The Go Authors. All rights reserved. +# Use of this source code is governed by a BSD-style +# license that can be found in the LICENSE file. + +set -e +wiki_pid= +cleanup() { + kill $wiki_pid + rm -f test_*.out Test.txt final-test.bin final-test.go +} +trap cleanup 0 INT + +go build -o get.bin get.go +addr=$(./get.bin -addr) +sed s/:8080/$addr/ < final.go > final-test.go +go build -o final-test.bin final-test.go +(./final-test.bin) & +wiki_pid=$! + +sleep 1 + +./get.bin http://$addr/edit/Test > test_edit.out +diff -u test_edit.out test_edit.good +./get.bin -post=body=some%20content http://$addr/save/Test +diff -u Test.txt test_Test.txt.good +./get.bin http://$addr/view/Test > test_view.out +diff -u test_view.out test_view.good + +echo PASS diff --git a/doc/articles/wiki/test_Test.txt.good b/doc/articles/wiki/test_Test.txt.good new file mode 100644 index 000000000..f0eec86f6 --- /dev/null +++ b/doc/articles/wiki/test_Test.txt.good @@ -0,0 +1 @@ +some content
\ No newline at end of file diff --git a/doc/articles/wiki/test_edit.good b/doc/articles/wiki/test_edit.good new file mode 100644 index 000000000..36c6dbb73 --- /dev/null +++ b/doc/articles/wiki/test_edit.good @@ -0,0 +1,6 @@ +<h1>Editing Test</h1> + +<form action="/save/Test" method="POST"> +<div><textarea name="body" rows="20" cols="80"></textarea></div> +<div><input type="submit" value="Save"></div> +</form> diff --git a/doc/articles/wiki/test_view.good b/doc/articles/wiki/test_view.good new file mode 100644 index 000000000..07e8edb22 --- /dev/null +++ b/doc/articles/wiki/test_view.good @@ -0,0 +1,5 @@ +<h1>Test</h1> + +<p>[<a href="/edit/Test">edit</a>]</p> + +<div>some content</div> diff --git a/doc/articles/wiki/view.html b/doc/articles/wiki/view.html new file mode 100644 index 000000000..023391577 --- /dev/null +++ b/doc/articles/wiki/view.html @@ -0,0 +1,5 @@ +<h1>{{.Title |html}}</h1> + +<p>[<a href="/edit/{{.Title |html}}">edit</a>]</p> + +<div>{{printf "%s" .Body |html}}</div> |
