Enforcing API Design Rules for High Quality Code Generation

Enforcing API Design
Rules for High-Quality
Code Generation
Mike Kistler, IBM
Tim Burks, Google

The API Economy
API
API API
API

Open API
Industry standard format for describing for REST APIs

Open API Tooling
Originally designed for documentation, tools are now available to automate or assist
with many common tasks:
● API Authoring
● Validation
● Documentation
● Testing
● Mocking
● Management
● Code Generation
● Validation

OpenAPI-based Tooling at IBM
● SDK Generation for Watson Developer Cloud
○ Java
○ Node
○ Python
○ Swift
○ .NET
● Benefits
○ Faster time to Market
○ Improved Quality
○ Enforce Compliance
○ Enforce Consistency
○ Increase Adoption
https://github.com/watson-developer-cloud

Some challenges for OpenAPI-based tools
● Some elements of an OAS API definition are "optional" or "flexible"
○ Parameter and property type and format
○ OperationId
○ Response schema
● Some styles of definitions are problematic for tools
○ Inline responses
○ Content-type, accept-type
● Focus implementation effort on main use cases

Parameter and property type and format
● OpenAPI v2 does not restrict the
values for type and format
○ except they must be strings
● But some values are "defined"
● Many tools can have unexpected
results when type and format are not
one of these combinations
"properties": {
"level": {
"type": "number",
"format": "integer"
},

API Design Guides
To promote a consistent style for APIs, and also to address the requirements and
constraints of tooling, many companies have developed "design guides" for their APIs:
● Google
○ https://cloud.google.com/apis/design
● Microsoft
○ https://github.com/Microsoft/api-guidelines
○ https://docs.microsoft.com/en-us/azure/architecture/best-practices/api-design
● IBM (in development)
○ http://watson-developer-cloud.github.io/api-guidelines/
○ http://watson-developer-cloud.github.io/api-guidelines/swagger-coding-style

Google API Design Guide
● Collection IDs
○ valid C/C++ identifiers
○ clear and concise English terms
○ avoid overly general terms
○ plural form with lowerCamel case
● Naming conventions
○ Field definitions (property names) must use lower_case_underscore_separated_names.
● Standard field names and types
● Error Model
○ code: integer
○ message: string
○ details: array of object

Microsoft API Design Guide
● All APIs MUST support explicit versioning.
● Services SHOULD provide JSON as the default encoding.
● JSON property names SHOULD be camelCased.
● Error Model
○ error:
■ code: string
■ message: string
■ target: string
■ details: array of object
■ innererror: object

IBM API Design Guide
○ Parameter and property names must be lower snake-case
● Data types
○ Parameters and properties should use well-defined data types (type and format)
● Operations
○ Every operation should have a unique operationId
○ List all required parameters before any optional parameters
● Error Model
○ code: integer
○ error: string
○ help: string

API Design -- the devil is in the details
In a perfect world, everyone would agree on the design guidelines for APIs.
Even if everyone agreed on the aesthetics, the disparity of tools and methodologies
used across today's Cloud companies drives each company to establish specific
requirements to match their tooling.
● Approach to versioning
● Error models

Validating Compliance
● Trust, but verify.
● There are validators available to flag deviances from OpenAPI.
○ swagger-editor
○ swagger-parser
○ swagger-tools
○ go-swagger
● But the design guidelines we're talking about are generally beyond simple
compliance to the OpenAPI spec.
● And as we've said, each company -- maybe even different divisions within the
same company -- have different API guidelines.

Solution: Linters
Let’s use a solution borrowed from programming languages: linters
● Java: maven checkstyle
● Python: pylint
● Javascript: eslint
● Go: go-tools
The best of these tools are:
● Configurable
○ Let users choose which rules to enforce and which rules to ignore
● Extensible
○ Let users implement their own rules

A Configurable and Extensible Linter for OpenAPI
● All the common linters are written in the same language that they consume.
● So what language should we use for the OpenAPI linter?
● Answer (borrowing from microservices): A Polyglot Linter
○ Linter extensions (plugins) in any language can detect and report violations

JSON/YAML: in-between language and data structure
Humans can write it easily
but find it tedious and need validators to get it right.
Dynamically-typed languages can read it easily
but can crash on unexpected inputs.
Statically-typed languages can read it easily(?)
but require lots of casting (ugly!) or explicit models.

How to write OpenAPI tooling in static languages
Step 1: Read the OpenAPI spec.
Step 2: Start handwriting data structures and a reader.
Step 3: Notice two things:
1. This is tedious.
2. There’s a JSON schema for OpenAPI.

The OpenAPI JSON Schema
{
"title": "Schema for Swagger 2.0 API.",
"id": "http://swagger.io/v2/schema.json#",
"type": "object",
"required": [
"swagger",
"info",
"paths"
],
"additionalProperties": false,
"patternProperties": {
"^x-": {
"$ref": "#/definitions/vendorExtension"
}
},
"properties": {
"swagger": {
"type": "string",
"enum": [
"2.0"
],
"description": "The version of this document."
},
"info": {
"$ref": "#/definitions/info"
},
"host": {
"type": "string",
"pattern": "^[^{}/ :]+(?::d+)?$",
"description": "The host of the API.'"
},
"definitions": {
"info": {
"type": "object",
"description": "General information”,
"required": [
"version",
"title"
],
"additionalProperties": false,
"patternProperties": {
"^x-": {
"$ref": "#/definitions/vendorExtension"
}
},
"properties": {
"title": {
"type": "string",
"description": "A unique and precise title."
},
"version": {
"type": "string",
"description": "A semantic version number."
},

Can we use JSON Schema to build a data structure factory?
OpenAPI 2.0
JSON Schema
OpenAPI 3.0
JSON Schema
OpenAPI 3.1
JSON Schema
OpenAPI 3.2
JSON Schema
OpenAPI.go
OpenAPI.swift
OpenAPI.rust
OpenAPI.dart
OpenAPI.ex
OpenAPI.kt

Protocol Buffers
a language-neutral, platform-neutral, extensible mechanism
for serializing structured data.

Protocol Buffers?
Serialization mechanism
Interface description language
Methodology

Protocol Buffers, a.k.a Google’s API Factory
search.proto
mail.proto
maps.proto
youtube.proto
search.cpp
search.py
search.java
search.go
......
protoc
protoc-gen-cpp
protoc-gen-py
protoc-gen-java
protoc-gen-go

A data structure factory for OpenAPI
OpenAPI 2.0
JSON Schema
OpenAPI 3.0
JSON Schema
OpenAPI 3.1
JSON Schema
OpenAPIv2.go
OpenAPIv2.swift
OpenAPIv2.rust
OpenAPIv2.dart
OpenAPIv2.ex
OpenAPIv2.kt
protoc-gen-go
protoc-gen-swift
protocgnostic-
generator
OpenAPIv2
.proto

gnostic, the OpenAPI Compiler
gnostic-
generator
OpenAPI
.proto and
support
code
OpenAPI
JSON
schema
protoc +
pluginsOpenAPI
.proto
reusable data structures
and reader for protobuf
OpenAPI descriptions
gnostic apps
and pluginsOpenAPI
description
gnostic parsed and
verified binary
protobuf of the
OpenAPI description

Kubernetes OpenAPI: .json vs .pb
Format Size Deserialization time Download time
(at 80 Mbps)
Json 1653 KB >500 ms 165.3 ms
Proto binary 914 KB 9.3 ms 91.4 ms
Proto binary
compressed
96 KB 13.5 ms 1.3 ms

Inside OpenAPI.proto: it starts with a Document
From OpenAPIv2/OpenAPIv2.proto:
message Document {
string swagger = 1;
Info info = 2;
string host = 3;
string base_path = 4;
repeated string schemes = 5;
// A list of MIME types accepted by the API.
repeated string consumes = 6;
// A list of MIME types the API can produce.
repeated string produces = 7;
Paths paths = 8;
Definitions definitions = 9;
ParameterDefinitions parameters = 10;
ResponseDefinitions responses = 11;
repeated SecurityRequirement security = 12;
SecurityDefinitions security_definitions = 13;
repeated Tag tags = 14;
ExternalDocs external_docs = 15;
repeated NamedAny vendor_extension = 16;

Inside OpenAPI.proto: preserving ordering in maps
// One or more JSON representations for parameters
message ParameterDefinitions {
repeated NamedParameter additional_properties = 1;
}
// Automatically-generated message used to represent maps of Parameter as ordered (name,value) pairs.
message NamedParameter {
// Map key
string name = 1;
// Mapped value
Parameter value = 2;
}
message Parameter {
oneof oneof {
BodyParameter body_parameter = 1;
NonBodyParameter non_body_parameter = 2;
}
}

Inside OpenAPI.proto: representing paths
// Relative paths to the individual endpoints. They must be relative to the 'basePath'.
message Paths {
repeated NamedAny vendor_extension = 1;
repeated NamedPathItem path = 2;
}
// Automatically-generated message used to represent maps of PathItem as ordered (name,value) pairs.
message NamedPathItem {
// Map key
string name = 1;
// Mapped value
PathItem value = 2;
}
message PathItem {
string _ref = 1;
Operation get = 2;
Operation put = 3;
Operation post = 4;
...
}

gnostic writes a “Request” message to plugins
From plugins/plugin.proto:
// A parameter passed to the plugin from (or through) gnostic.
message Parameter {
// The name of the parameter as specified in the option string
string name = 1;
// The parameter value as specified in the option string
string value = 2;
}
// An encoded Request is written to the plugin's stdin.
message Request {
// filename or URL of the original source document
string source_name = 1;
// Output path specified in the plugin invocation.
string output_path = 2;
// Plugin parameters parsed from the invocation string.
repeated Parameter parameters = 3;
// The version number of gnostic.
Version compiler_version = 4;
// API models
repeated google.protobuf.Any models = 5;
From google/protobuf/any.proto:
message Any {
// A URL/resource name that uniquely
// identifies the type of the serialized
// protocol buffer message.
string type_url = 1;
// Must be a valid serialized protocol
// buffer of the above specified type.
bytes value = 2;
}

plugins write “Response” messages back to gnostic
// The plugin writes an encoded Response to stdout.
message Response {
// error messages. If non-empty, the plugin failed.
repeated string errors = 1;
// file output, each file will be written by gnostic to an appropriate location.
repeated File files = 2;
// informational messages to be collected and reported by gnostic.
repeated Message messages = 3;
}
// File describes a file generated by a plugin.
message File {
// name of the file
string name = 1;
// data to be written to the file
bytes data = 2;
}

New for linters: message responses
// Plugins can return messages to be collated and reported by gnostic.
message Message {
enum Level {
UNKNOWN = 0;
INFO = 1;
WARNING = 2;
ERROR = 3;
FATAL = 4;
}
// message severity
Level level = 1;
// a unique message identifier
string code = 2;
// message text
string text = 3;
// an associated key path in an API description
repeated string keys = 4;
}

Linter examples
● Go
○ gnostic-lint-descriptions
○ gnostic-lint-paths
● Node.js
○ gnostic-lint-operations
○ gnostic-lint-responses
● Swift
○ gnostic-lint-responses-swift

Let’s run one.
$ gnostic examples/v2.0/yaml/petstore.yaml --lint-responses
level:ERROR code:"NO_ARRAY_RESPONSES" text:"Arrays MUST NOT be returned as the
top-level structure in a response body." keys:"paths" keys:"/pets" keys:"get"
keys:"responses" keys:"200" keys:"schema"
level:ERROR code:"NO_ARRAY_RESPONSES" text:"Arrays MUST NOT be returned as the
top-level structure in a response body." keys:"paths" keys:"/pets/{petId}"
keys:"get" keys:"responses" keys:"200" keys:"schema"
$ gnostic examples/v2.0/yaml/petstore.yaml --lint-responses --messages-out=.
$ report-messages petstore.messages.pb
ERROR NO_ARRAY_RESPONSES Arrays MUST NOT be returned as the top-level
structure in a response body. [paths /pets get responses 200 schema]
ERROR NO_ARRAY_RESPONSES Arrays MUST NOT be returned as the top-level
structure in a response body. [paths /pets/{petId} get responses 200 schema]

Let’s run ALL of them. (1 / 2)
$ gnostic examples/v2.0/yaml/petstore.yaml
--lint-responses
--lint-descriptions
--lint-paths
--lint-operations
--lint-responses-swift
--messages-out=.
--time-plugins

Let’s run ALL of them. (2 / 2)
$ report-messages petstore.messages.pb
ERROR NO_ARRAY_RESPONSES Arrays MUST NOT be returned as the top-level structure in a response body. [paths ...
WARNING NODESCRIPTION Operation has no description. [paths /pets get]
WARNING NODESCRIPTION Response has no description. [paths /pets get responses 200]
WARNING NODESCRIPTION Response has no description. [paths /pets get responses default]
WARNING NODESCRIPTION Operation has no description. [paths /pets post]
WARNING NODESCRIPTION Response has no description. [paths /pets post responses default]
WARNING NODESCRIPTION Operation has no description. [paths /pets/{petId} get]
WARNING NODESCRIPTION Response has no description. [paths /pets/{petId} get responses 200]
WARNING NODESCRIPTION Response has no description. [paths /pets/{petId} get responses default]
WARNING NODESCRIPTION Definition has no description. [definitions Pet]
WARNING NODESCRIPTION Property has no description. [definitions Pet properties id]
WARNING NODESCRIPTION Property has no description. [definitions Pet properties name]
WARNING NODESCRIPTION Property has no description. [definitions Pet properties tag]
WARNING NODESCRIPTION Definition has no description. [definitions Pets]
WARNING NODESCRIPTION Definition has no description. [definitions Error]
WARNING NODESCRIPTION Property has no description. [definitions Error properties code]
WARNING NODESCRIPTION Property has no description. [definitions Error properties message]
INFO PATH /pets [paths /pets]
INFO PATH /pets/{petId} [paths /pets/{petId}]

Runtime comparisons
Plugin Language Plugin Run Time
gnostic-lint-operations Node.js 195 ms
gnostic-lint-responses Node.js 221 ms
gnostic-lint-responses-swift Swift 17.8 ms
gnostic-lint-descriptions Go 18.5 ms
gnostic-lint-paths Go 18.6 ms

Where is this going?
● Organization-specific linters, e.g.
○ gnostic-lint-ibm
○ gnostic-lint-google
○ gnostic-lint-supercodegen
● Per-language linter-helper libraries, e.g.
○ Messaging helpers
○ Reference resolution
● More API tools: code generators, API management, smart clients, …
Better API experiences!

https://github.com/googleapis/gnostic

Enforcing API Design Rules for High Quality Code Generation

More Related Content

What's hot

Similar to Enforcing API Design Rules for High Quality Code Generation

More from Tim Burks

Recently uploaded

Enforcing API Design Rules for High Quality Code Generation