Monday, 26 June 2017

Migrating Balrog to OpenAPI Specification

After working with Address Intelligence and Experience at Amazon to help out-migrate their legacy Address Authority Service to a new design, little did I know that few months after finishing up the internship, I would be involved in another migration task. Only this time, it can be technically described as more of migration to a new standard rather than changing the entire service layout and schema design.

My Google Summer of Code project involves migrating the existing admin APIs of Mozilla's update service called Balrog to the OpenAPI Specification format. Below is brief abstract for the project:

Balrog is the software that runs the server side component of the update system used by Firefox and other Mozilla products. It helps ship updates faster and with much more flexibility than it's predecessor AUS (Application Update Service). Balrog's admin API is mostly hand-written code,
with some usage of wtforms for validation. It largely works, but it has some rough spots and
inconsistencies. The move to an OpenAPI 2.0 Specification will have multiple benefits including less
code maintenance, better documentation and easier for clients to visualize and interact with API’s
resources. Connexion, a framework designed for flask based applications, will be used to handle
HTTP requests based on OpenAPI 2.0 specification of the API to map the endpoints to the
Balrog’s existing methods to serve the requests.

After reading up on the documentations for OpenAPI 2.0, JsonSchema, Connexion and Balrog's, the high-level approach to the project was determined by performing the following steps in an incremental development cycle model:

Note:  Since there was no coupling between many (but not all) APIs, hence almost each API could be independently migrated without affecting any other API operation or any precedence order.
  1. Discuss the changes before writing up the openAPI specification. These changes need to be discussed on IRC with the mentor so that the both functional and nonfunctional requirements of the change in request or responses are clear and understood by both.
    1. This can involve determining the expected request and response formats by clients and the UI. A design session with the mentor may be required.
    2. Document relevant request or response JSON form and URL parameter fields and make use of swagger's models to re-use these fields like OOP classes. If the requirements necessitate split the model into a composition or hierarchy before writing up the swagger spec file.
      1. Example: Determining when an operation parameter can/can not have null values( link ) or whether a field that accepts specific Regex patterns only may accept any string in other API operations. This will require the specific field to be inherited and overwritten in the new model. 
      2. Another Example: In Rules model, priority and backgroundRate are both required when creating a new rule, but not required when updating a rule. Hence this is an ideal scenario where hierarchical/composite Rule models can be used for two separate use-cases.

Here's the link to publicly hosted swagger document for visualizing and interacting with the APIs. The listed APIs aren't exhaustive and right now only contain the APIs migrated by the end of first evaluation period of GSoC.  Will update the link in future before second evaluation.

Note: During the coding period, swaggerhub updated their swagger-ui vendor which throws a semantic error if 'type' is listed as an array of multiple data type (ex: string, int) instead of a single basic data type. However, this doesn't cause any issues in connexion's swagger parser as JSON schema draft4 allows this.