This project involved creating a Python web application using a Flask server with REST API to perform CRUD operations on several database tables using AJAX calls. The application links to a third party API - the Irish open data portal at https://data.gov.ie using the CKAN API.
This post contains an overview of a data representation project which was completed for the Data Representation and Querying module at GMIT as part of the Higher Diploma in Computing and Data Analytics.
The Data Representation module focused on investigating and operating the protocols, standards and architectures used in representing data, with a focus on interacting with data services across the internet with practical experience in developing applications that interact with such data.
The project involved writing a Flask server program that consumes a REST API and creating a web interface. The project links to the Ireland’s open data portal at https://data.gov.ie, populates a MySql database with some simple tables, then allows a user to interact with the data stored in the database through a web interface. It also allows the user to request additional data from the Irish open data portal using additional API actions.
The project repository is on GitHub and contains several files including:
A Flask server program application.py
DAO files written in Python providing an interface to the database
DAO files written in Python for retrieving JSON data from the https://data.gov.ie and populates some database tables in MySQL.
A requirements.txt file containing the Python packages required to run the application
initdb.sql file containing the SQL code for creating the database.
An overview of the instructions for downloading and running the project is included on this website here.
There were two choices in the project instructions for this project with some overlap.
Web application project. Create a web application project with a Flask server that has a REST API to perform CRUD operations on one or more database tables. The accompanying web interface should use AJAX calls to perform these CRUD operations.
Third party API project. Link to a third party API, store some data in database and create a web interface to view the data and perform some CRUD operations through the API.
I chose to use the Irish Government’s Open data portal at https://data.gov.ie as I wanted to use real data and to explore the Irish Open-Data portal more. This portal is built using CKAN. Using the CKAN API you can get (JSON-formatted) lists of all the datasets that are hosted on the open data portal. You can also get lists of the organisations that publish the datasets as well as lists of tags used. (Note The term package is used for datasets).
Some notes on the Irish Open data portal are included here.
Once you have retrieved the list of packages, tags or organizations you can then programmatically retrieve the lists of actual dataset resources using further API actions.
Figure 2: Retrieving the list of datasets / packagesFigure 3: Retrieving the list of tags
From the list of dataset resources you can extract the URLs to the actual datasets and retrieve them using Python, Curl or by going to the URLs.
Figure 4: Search for a dataset listed in the databaseFigure 5: Search for a dataset using the external API
Through this project I gained experience with the following:
Flask which is a web application framework written in Python
MySQL, a database management system that allows you to manage relational databases
Bootstrap - a framework for building responsive mobile-first sites.
JQuery methods for Ajax functionality.
AJAX (Asynchronous JavaScript and XML) allows for data to be loaded in the background without reloading the whole page.
