Defining the data types
Once you've taken care of defining the REST API requests that expose the data from your 3rd party system, the next step is to define the data types in Quantive Results that will correspond to the objects your REST APi returns. To define your data types you must:
- Describe the schema
- Describe the fields for each data type
- Add data download instructions
- Specify the plugin version
Describe the schema
The schema of your Data app lists all data types available for syncing over to Quantive Results. Each entry in the schema collection is called a type. All entries are listed in the UI on the Select which item you want to connect screen.
You must describe your schema inside a schema object in the root of your YAML file. The schema object consists of a types collection. Each type must have a:
- title - this is the name of your type that appears for selection in the UI
- key - this is the programmatic name of your type. It determines the name of the table where the data for this type will be stored in the Quantive Results Data warehouse. It's visible for users in the SQL insight editor.
For the purposes of this tutorial our Data app schema will contain only one type – planets. This is what it must look like in your YAML file:
schema:
types:
- title: "Planets"
key: "swapi_planets"
NOTE: The key property for each type must be unique across all Data apps in a Quantive Results account. Best practice advise - prefix your keys with the name of your app or another unique identifier. In the example above we did not just name our key palnets, but swapi_planets instead.
Describe the data type fields
In addition to title and key, each data type must have a list of the fields it stores the data into. If you think about this programmatically, the key property determines the name of the table that will store the information for this data type, and the fields determine the table columns. The data type fields are described in a staticFields object inside each type. Each field must have a:
- name - the programmatic name of the field. This name is visible in the SQL insight editor.
- title - the name of the field that appears in the UI
- type - the programmatic type (for example string, integer, float)
If we go back to the example with the Planets object form the Star Wars API, when you request the /planets endpoint, the information it returns something like this:
{
"count": 60,
"next": "https://swapi.dev/api/planets/?page=2",
"previous": null,
"results": [
{
"name": "Tatooine",
"terrain": "desert",
"surface_water": "1",
"population": "200000",
...
},
{
"name": "Alderaan",
"terrain": "grasslands, mountains",
"surface_water": "40",
"population": "2000000000",
...
},
...
This way we know that our Data app will be working with planets. A planet has fields like name, terrain, surface_water, population and so on. This is what the staticFields definition for these fields looks like in your YAML file for the example above:
staticFields:
- name: "name"
title: "Name"
type: "string"
- name: "terrain"
title: "Terrain"
type: "string"
- name: "surface_water"
title: "Surface Water"
type: "string"
- name: "population"
title: "Population"
type: "string"
Querying the data
Each data type defined in your Data app’s schema must provide instructions how to download its data. This is done via the download object in your YAML file. It contains properties that define the:
- uri - the endpoint where the request is made. NOTE: specify only the endpoint uri, which must be appended to the baseUrl you already specified in the root of your YAML file.
- verb - the HTTP method (verb) to be used
- headers - any specific HTTP headers you must pass
- response - if your REST API response wraps the results in an object (for example, results, items) specify the name of that object, so Quantive Results knows where to look for your data in the response.
- paging - If your REST API endpoint implements paging, specify the paging options that Quantive Results will use. For more information see Anatomy of a YAML File - paging
For example, this is how the download object for the planets type must look like in your YAML file:
download:
uri: "/planets"
verb: "GET"
headers:
Accept: "application/json"
response:
collection: "results"
paging:
kind: "custom-rest"
settings:
skipParamName: "skip"
takeParamName: "top"
NOTE: If the 3rd party system REST API returns the actual data within a collection, you must specify the name of that collection in the response object. The /planets endpoint returns the planets data in the results collection, that’s why we have specified it explicitly above. If the data was just returned as an array we would leave the collection property empty.
Complete YAML
If you've gotten this far, your YAML file should look something like this:
kind: rest
baseUrl: "https://swapi.dev/api"
connectionTest:
uri: "/planets"
verb: "GET"
headers:
Accept: "application/json"
schema:
types:
- title: "Planets"
key: "swapi_planets"
staticFields:
- name: "name"
title: "Name"
type: "string"
- name: "terrain"
title: "Terrain"
type: "string"
- name: "surface_water"
title: "Surface Water"
type: "string"
- name: "population"
title: "Population"
type: "string"
download:
uri: "/planets"
verb: "GET"
headers:
Accept: "application/json"
response:
collection: "results"
paging:
kind: "custom-rest"
settings:
skipParamName: "skip"
takeParamName: "top"
Now that your YAML file is complete, follow the instructions on Build and publish an app to submit your app to the Quantive Results Marketplace and test it in your account.
Appendix I - Taking care of authorization
This tutorial uses an API that is open to the public and requires no authorization. If your REST API requires authorization you must pass an additional Authorization header in the headers collection of each download object, where authorization is required for the corresponding endpoint. For example, if the /planets endpoint was secured and required a JWT token, the download object would have looked like this:
download:
uri: "/planets"
verb: "GET"
headers:
Accept: "application/json"
Authorization: "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c"
response:
collection: "results"
paging:
kind: "custom-rest"
settings:
skipParamName: "skip"
takeParamName: "top"
Using setup parameters
When you submit your Data app to the Quantive Results Marketplace, you can configure setup parameters. These parameters appear for users to fill in, when adding a new data source via up your Data App. This mechanism enables you to get user-specific input and use it in your logic. For example, instead of hard-coding an authorization token, you can expose a setup parameter called authorization and ask users to put in their personal API code. All parameters are available from the connectorSettings object.
For example, to use your authorization parameter from the above example, you can say:
authToken: "{connectorSettings.authorization}"
Using individual tokens
Hard-coding the API key is not a good idea, especially if it's going to change, or if you want to enable users to use their own token when setting up your Data app. You can use setup parameters to ask users to provide an authorization token on the set-up screen. For example, if you had a setting parameter called authorization the above example would look like this:
download:
uri: "/planets"
verb: "GET"
headers:
Accept: "application/json"
Authorization: "Bearer {connectorSettings.authorization}"
response:
collection: "results"
paging:
kind: "custom-rest"
settings:
skipParamName: "skip"
takeParamName: "top"
