net/http: new HTTP client package

[bradfitz]

Tracking bug for higher-level more usable HTTP client functionality.

• take contexts
• more timeout control per-request, rather than per-transport/client
• ...Options pattern?
• decode into JSON/etc easily?
• treat non-2xx as error (by default?).
• slurp entire response from server (by default?)
• control over max response size (some sane default limit when slurping is enabled)

I thought we had a bug for this, but maybe not.

[pciet]

This is a pattern in a test client I wrote:

// or Post, PostForm
r, err := client.Get(addr)
if err != nil {
    panic(err)
}
// do some parsing here then discard the rest
_, err = io.Copy(ioutil.Discard, r.Body)
if err != nil {
    panic(err)
}
err = r.Body.Close()
if err != nil {
    panic(err)
}
if r.StatusCode != http.StatusOK {
    panic(r)
}

Are you looking at adding new identifiers?

type BodyParser func(io.Reader) error

// http.ErrBadStatus for non-2xx, discards unread response body, and closes the connection.
func (c *Client) ClosingGet(url string, resp BodyParser) error

// repeated for the other methods

var t MyStruct
err := c.ClosingGet(addr, http.JSONBodyParser(&t))

[bradfitz]

Are you looking at adding new identifiers?

There would be new API for this, yes. I don't have a concrete proposal yet.

[awreece]

We wrote a wrapper for the http client internally that uses the ...Options pattern and achieves a lot of these goals:

https://github.com/AlphaFlow/gohttp

GET with parameters:

cli := http.NewClient()

var ret map[string]interface{}
err := cli.Get(context.Background(),
    "http://example.com",
    http.WithParam("debug", "1),
    http.WithJSONResponse(&ret))

POST with body:

ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
defer cancel()

body = map[string]string{
   "hello": "world",
}
err := cli.Post(ctx, "http://example.com",
    http.WithJSONBody(body),
    http.WithHeader("Authorization", "Bearer my-token"))
if bse, ok := err.(*http.BadStatusError); ok && bse.Code == 404 {
    fmt.Println(bse)
}

Mocking the HTTP request:

	return http.NewMockClient(func(ctx context.Context, req *http.Request) error {
		if e != nil {
			return e
		}
		if len(body) == 0 {
			return nil
		}
		if req.Output != nil {
			_, err := req.Output.Write(body)
			return err
		} else if req.JSONOutput != nil {
			return json.Unmarshal(body, req.JSONOutput)
		} else {
			panic("no place to write response")
		}
	})

This has greatly simplified our error handling/json marshaling.

Quirks

• Mocking the client using this interface is a bit weird -- the mock needs to know an awful lot about how the code works. This could be simplified a bit, maybe.
• We inherit standard go http client "follow redirect" idiom, when it would be nice for that to be an option.