You are here

Home / Developers

Developers


1. Introduction

This section of the Open Data BCN portal is intended to provide useful information for the application developers and the target audience with a technical profile.

2. APIs

The APIs (Application Programming Interface) are interfaces that specify how the different components of computer programs should interact. It is a set of functions and procedures offered to be used by another program in order to interact with the program in question.

We can distinguish three types of APIs available in the Open Data BCN portal:

  • Catalogue management API: This API is used to consult and manage the catalogue information, such as datasets, resources, labels, etc...
  • Query APIs on CSV resources: Allows you to check and filter the information contained within a specific CSV
  • City Council APIs: These are APIs enabled by information sources and tailor-made

Both the Open Data catalogue management API and the CSV resource query are offered as
REST web services in JSON (input and output) format.

Many of the methods can also be called directly by URL adding the necessary parameters. Including the callback parameter the response of which is returned in JSON format for use with Javascript.

2.1. Catalogue management API

Using the API the datasets and resources published in the Open Data BCN portal can be accessed.

This is the API offered by the CKAN technology to manage the catalogue of datasets. The complete documentation can be found in this link.

The base URL of the API is https://opendata-ajuntament.barcelona.cat/data/api/ and can be accessed from the URL https://opendata-ajuntament.barcelona.cat/data/api/action/ by adding the name of the method we would like to access.

These methods are public and do not require the use of a token or any other identification method, being examples:

  • current_package_list_with_resources: returns the list of current datasets and their resources
  • package_search: performs a search within the dataset catalogue
  • resource_show: returns the metadata of the dataset from the id of the dataset and among the data, the resource_id of the resource
    Example: https://opendata-ajuntament.barcelona.cat/data/api/3/action/package_show?id=arbrat-zona
  • datastore_search: returns each resource's information
    Example: https://opendata-ajuntament.barcelona.cat/data/api/action/datastore_search?resource_id=f7134b6f-95ce-4b83-bacd-6f411bd9c1f0
    and if we would want to search for the valor=14 also within the resource:
    https://opendata-ajuntament.barcelona.cat/data/api/action/datastore_search?resource_id=f7134b6f-95ce-4b83-bacd-6f411bd9c1f0&q=14

You can find sample queries and the complete list of methods in the CKAN API guide.

Furthermore, to facilitate interoperability between data catalogues published on the web, we use the DCAT extension.

Up

2.2. Query APIs on CSV resources

The CKAN API available filter allows selecting datasets with this interface enabled.

These interfaces allow you to act on a specific CSV resource. In order for the platform to show this service, the resources must be uploaded to the Open Data BCN portal with a suitable format and coding. When a CSV is uploaded on the platform, a table is created internally in the database of this resource. This storage within CKAN is called datastore. The API of the datastore allows us to make queries directly on these tables.

It should be noted that the use of the query API is at the individual CSV resource level, not the dataset catalogue. The resources with CKAN API enabled can be selected with the catalogue filter 'CKAN API available'.

There are only two common endpoints for all platform CSVs that allow you to search within the resource based on the search criteria added as parameters to the call or by making a SQL query:

The endpoints are unique to all the resources of the platform, to make a query about a specific resource, we must define as a parameter or within the query the ID of the resource on which we want to consult. For example, to make a query about the 2017 resource of the dataset '50 Most common names of the inhabitants of Barcelona, disaggregated by decade of birth and sex' we would use the following URL: https://opendata-ajuntament.barcelona.cat/data/api/action/datastore_search?resource_id=51b78472-cd11-4426-912a-1d26240491e7

2.2.1 How to get the resource ID

The URL of the resource, which is of the type: https://opendata-ajuntament.barcelona.cat/data/en/dataset/nom/resource/identifier already includes its ID

identifier is the resource ID, we only have to navigate the portal and select the resource in question to obtain the ID from the URL.

2.2.2 How to find the names of the fields in the portal

To find the names of the fields in the portal in order to use the query API, we only have to access the CSV resource preview. For example, let's go to the resource Fonts_201610.csv from the dataset Barcelona public fountains and we look at the headers of the resource's preview: INVENTARI_NOM, INVENTARI_TIPUS, INVENTARI_CARRER, INVENTARI_NUMERO_CARRER, INVENTARI_COORDENADA_X, INVENTARI_COORDENADA_Y we will be able to make the queries as explained in the following sections

2.2.3 datastore_search

URL: https://opendata-ajuntament.barcelona.cat/data/api/action/datastore_search

  • resource_id (string - required) - id of the CSV resource we want to search
  • filters (dictionary) - conditions to be met. Example: {“direccio”: “%carrer%”}
  • q (string or dictionary) - search in full text. You can specify a string to search for in all fields or a dictionary to search for specific fields. Example: {“key1”: “a”, “key2”: “b”}
  • distinct (boolean) - returns only the different rows
  • plain (boolean) - treat the query as if it were a plain text
  • language (string) - query language in full text
  • limit (int) - maximum number of results. By default it returns 100 results.
  • offset (int) - number of results to be skipped. Useful for making a results page.
  • fields (string) - list of fields to be included in the response. By default, returns all fields in the same order as in the CSV
  • sort (string) - name of the fields for which to sort separated by commas: “nom, codi desc”

Sample call to datastore_search

We want to look for the public drinking fountains that are located on Mallorca street. To do this, we first need to know the resource ID and then the name of the field we want to search for. This information can be obtained from the preview page of the resource on the portal.

Input Parameters:

  • resource_id: This is the last part of the resource URL (21f7a4df-2e73-45f8-8c6d-0b3db8c21527)
  • q: We consult it in the fields of the resource preview {“INVENTARI_CARRER” : “Mallorca”}

Calling URL:

https://opendata-ajuntament.barcelona.cat/data/api/action/datastore_search?resource_id=21f7a4df-2e73-45f8-8c6d-0b3db8c21527&limit=5&q={%22INVENTARI_CARRER%22:%22%Mallorca%%22}

Up

2.2.4 datastore_search_sql

URL: https://opendata-ajuntament.barcelona.cat/data/api/action/datastore_search_sql

Queries of this type have a limited timeout so that unintentional DOS attacks do not occur.

Input parameters:

  • sql (string) - SELECT to be executed on the data. The name of the table on which the query is made is the resource ID.

Sample call to datastore_search_sql

We want to search the public drinking fountains on Mallorca street. We need to know the resource ID (which is the name of the table) and the name of the field we want to filter.

Input parameters:

  • sql: SELECT * from "21f7a4df-2e73-45f8-8c6d-0b3db8c21527" WHERE "INVENTARI_CARRER" LIKE '%MALLORCA%'

Calling URL:
https://opendata-ajuntament.barcelona.cat/data/api/action/datastore_search_sql?sql=SELECT%20*%20from%20%2221f7a4df-2e73-45f8-8c6d-0b3db8c21527%22%20WHERE%20%22INVENTARI_CARRER%22%20LIKE%20%27%MALLORCA%%27

2.3. APIs owned by the City Council

The own APIS ar those provided by the teams that manage the sources of information. These APIs are linked form the Open Data BCN portal but they are not maintained or validated from the same team that manages the portal. In the portal they are treated as one more resource, as if it were a link to a CSV or RDF file.

Currently, the following API resources are offered in the Open Data BCN catalogue:

  • Geoservices WMS and WMTS of the City Council Spatial Data Infrastructure

3. Tokens

The access tokens are identifiers which contain the security credentials to sign in a a session, making it so it identifies the user, the groups, the permits and in some cases, a specific application. Regarding the Open Data BCN portal, a user is identified.

You can find here all information regarding its obtaining and use.

Likewise, the Token required filter, available in the catalogue, allows selecting datasets which have resources with this option enabled which have a key icon in the 'Download' button.

4. Limitation of the number of accesses to datasets and resources

In order to offer maximum availability and quality of service, the access limits detailed below will be applied in online dataset searches and resource downloads:

  • Datasets:
    On-line search: 30 petitions/minute

  • Resources:*
    Download: 1 petition/second or 60 petitions/minute
    Access via API: 30 petitions/minute

These limits are subject to modification to always offer the best service to users.
In the event that the access limit is reached, it will be indicated by returning the code 503.

5. Barcelona City Council's GITHUB

In the Barcelona City Council's GITHUB the CKAN and Drupal extensions that have been developed to implement new portal features are published:

  • CKAN:
    • odatabcn: appearance customization of the catalogue and other functionalities.
    • federagobes: generation of the RDF file with the Open Data BCN catalog to federate the datasets with the data.gob.es portal.
  • Drupal:
    • opendata_tokens: generation, storage and regeneration of access tokens

6. Programación

  • How to transform a file with .csv extension from an initial WIDE format to a LONG format.

We describe with an example how to transform a file published in WIDE format to another in LONG format using Python.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

#To use the script, make a request with the number of id_vars(fix columns) Example - 2 id_vars: python wide_to_long.py 2
import pandas as pd
import sys

df = pd.read_csv("in_WIDE.csv")
numero_ids = int(sys.argv[1])

id_vars = []

for i in range(numero_ids):
	id_vars.append(df.columns[i])

df = pd.melt(df, id_vars=id_vars, var_name='Concepte', value_name='Valor')
id_vars.append('Concepte')
id_vars.append('Valor')
df = df.reindex_axis(id_vars, axis=1)

dt = df.dtypes
print (dt)
print(df)

df.to_csv("out_LONG.csv", index = False)


7. Interesting links

The open data portal of the European Union has a Develpers section where it offers links to clients for the CKAN API:

8. Others

  • In the Open Data BCN portal we have a FAQ section with frequently asked questions about the Open Data and the use of the data
  • You can send any questions or queries through Contact us